Entry: 5135940c< < Blog
Big data is on everyone's mind lately in the tech press. With computer power having reached such a high level, and with the cloud becoming ubiquitous, data has increased at a tremendous pace. This means companies like Google, Facebook and others have an immense amount of information pouring into their databases which can cause a lot of issues when trying to deal with all of these incoming signals. These last couple of years, many have been focused on trying to solve this problem with things like NoSQL and other data processing concepts to deal with exabytes of data.
Big data is hard to deal with because traditional databases and data visualization software often rely on loading the whole data set in memory, or have enough processing power to go through the information in an efficient way. Google's BigQuery is one of the popular technologies used to deal with such large data sets, and many other companies have developed similar mechanisms. Using highly scalable architectures like Heroku is no longer something left to very large corporations, with even small web startups predicting that they will be getting so much information back from their users that they need to plan for big data from the very start.
But now that companies can receive and store this data, everything from logs, to usage tracking, location coordinates, patterns and so on, the next step is to make sense of all of this. And early this year we're starting to see many companies investing large amounts of money in data analysis. This is just a snippet from Google News:
The top news right now in data analysis isn't even what these companies are analyzing, but simply the fact that so many data analysis companies are popping up and being acquired by businesses who need to make sense of their big data. A Microsoft researcher gave an interesting talk last week about some of the pitfalls of big data and how you need proper context and depth in order to reach the right conclusions. One example she gives is that tweets during natural disasters are influenced by areas that have lost power, population concentrations, differences in wealth and thus the use of mobile devices, etc. Simply gathering and storing data is not enough and could be dangerous since a quick analysis may miss some key facts and give you the wrong conclusions. Of course, once we do have the right analysis, then more worries can appear.
In a recent interview, an Oxford professor gives an example of someone who feels this sort of obsession on big data is going too far. He says that an art professor came to see him and complain that to get an art grant these days, artists are required to quantify what they are doing. But since they are artists, how are they supposed to quantify success? They believe that this quest for quantification has gone too far. However, some see it differently. He says that data can be gathered showing how many people see a particular painting or share it online, and thus reach conclusions on how successful an artist is. With big data and the proper analysis, now all these little details about every single piece of content can be known and made easily accessible and searchable.
The problem of big data has been solved. We know how to gather data and store it. Now companies are scrambling to analyze it in a way that makes sense and brings them useful results, to get a good overall picture of their business without losing fine resolution, which is an incredibly hard problem. This is where the money is going to go over the coming years.