Big Data: Gospel or Myth?

For the past few years, many IT evangelists have been preaching the εὐαγγέλιον/gospel/”Good News” about Big Data. Big Data has been promoted in books, trade magazines, and conference papers.

The image projected concerning Big Data, has been salvific in nature. It represents deliverance to all those who are in bondage regarding the optimization their data assets. It represents freedom to all those who are oppressed regarding the range of their analytic options. It represents redemption to all whose bosses have questioned their database investments. For some, Big Data is the corporate deity; Big Data is the way, the truth and the life. Even the Pointy-Haired Boss (Dilbert by Scott Adams) has become a disciple. But let’s explore this belief system a little; let’s assess whether Big Data should be canonized as gospel or classified as a myth.

Big Data as “gospel” suggests that the people who walked in darkness have seen a great light. Those who could not make sense of the data at a granular level, can now see the big picture. Those who were limited to hypothesis testing, can now let the data tell its own stories. Those who had curbed their work to structured data, can now expand their options with unstructured data. Those who had to restrict the number of files for storage on their local workstations, can now work unrestrained on the cloud server. Big Data simplifies sentiment analysis. Big Data enables research. Big Data eases forensic accounting. Big Data is the gateway to information heaven.

Big Data as “myth” suggests that the firestorm surrounding this technology may produce more heat than light. It suggests that the concept is less of a divine inspiration and more of marketing strategy that projects abstract benefits vis-à-vis usable results. Big Data as myth means that prerequisites such as a solid platform, the ability to integrate with multiple applications and high data storage, fast data transmission and tight data security must be well-planned, financed, implemented and tested. It suggests that this is not a shrink wrapped/plug and play technical solution. It suggests that at the end of ETL, analysis, and visual analytics, the human element must still make a decision on if and how to use the information produced. Like the Farmer’s Almanac, Big Data is a decision support tool that is dependent upon human judgment.

So how does one discern whether Big Data is gospel or myth? One way is to ask three questions:

In short, knowing whether Big Data is a producer or a consumer of human, information, and capital resources will inform your opinion.

Big Data -- gospel or myth? Should we embrace its virtues literally or interpret them as allegory of potential uses? Is Big Data solving problems or just being promoted as a revenue stream for vendors? Are you a believer or a skeptic? Please share your thoughts.