We see references to big data in newspapers, magazine articles and analyst reports. It reminds me of the term “cloud” from a few years ago – it had a similar arc in terms of origins, adoption and usage. And just like “cloud”, big data seems to have a broad definition and depending on who you ask there seems to be a different perspective on what it really is:
- Industry analysts and popular media seem to define big data as an opportunity or a new class of problems that didn’t exist before. Fueled primarily by the availability of large volumes of data being generated by mobile devices, sensors, social media etc. Harnessing this gives organizations new capabilities and an ability to get ahead of their competitors.
- Technologists seem to define big data as a new means of solving problems that could not be addressed using “old technology”. They tend to use the 3 V’s (Volume, Variety and Velocity) to describe characteristics of data and suggest new mechanisms (Hadoop, NoSQL, Machine Learning) that enables us manage and analyze them. The phrase “big data problem” identifies situations that need to be solved differently and the phrase “big data platform” typically refers to a set of tools and technologies that help solving those problems.
- Data scientists and analysts tend to define big data as a methodology. A different way of thinking about data and gleaning insights from it. When data sets are available in huge quantity, applying advanced analytical techniques to analyze them using cost effective mechanisms need a very different mindset and approach.
Whatever be the specific definition of big data, there are certain shifts and transformations over the last decade that are fueling the conversation around big data. In this video clip (courtesy Key Info Systems) I discuss some of the main ones.