Big data will never eliminate uncertainty or answer all your questions. But treat it as a fad at your own peril.
There's no shortage of noise surrounding big data. Today it seems that every software vendor, consulting firm and thought leader has developed the "right" definition of the term. While I'd argue there is no such definition, I would like to dispel a few of the most commonly held myths about the subject, many of which I explore in Too Big to Ignore: The Business Case for Big Data.
Myth 1: You Can Get To All Of The Data
On many levels, we are living in unprecedented times. Never before has so much data been available to us. Forget megabytes and petabytes, exabytes of data now exist. I read recently that the average person in an industrialized society today consumes more information in one day than his fifteenth century counterpart did in his lifetime.
Despite this unfathomable amount of data, no person or organization can store and retrieve all of the data on a particular subject, much less overall. And yes, that includes Google. Its software indexes the Surface Web, not the Deep Web. Some estimates put the latter at 25 times the size of the former. As a result, when you search, you are accessing anywhere from 4% to 6% of all information on the Internet.
Taking it down a level or 30, individual authors like me cannot access some very valuable information, such as which specific customers are buying my books. Sites like Amazon and stores like Barnes & Noble keep that information. Nothing would make me happier than knowing my customers, but even in a big data world, that information eludes me. You will never get all of the data. Period.
Deal with it.
Myth 2: You Need All Of The Data
No doubt that more data helps, but don't for a minute think that you need all data to make an informed business decision. Organizations that are effectively leveraging the power of big data realize that they will never capture all relevant information.
New sources of data spring up seemingly every day, and it's not as if they're all valuable. For instance, email messages often contain extremely valuable insights into the state of an enterprise. Smart companies are mining individual messages to gauge employee sentiment and potentially determine who might be exiting.
This is a far cry from saying that all emails are equally valuable. It's hard to make the argument that using text analytics on spam makes much sense.
You don't need all of the data. Yes, more is better than less, but don't waste time trying to achieve the impossible.
Myth 3: Big Data Yields Certainty
A well-trodden business aphorism is, "I have all of the data I can handle. I just need more information." In Too Big to Ignore, I write about the difficulty of being truly certain about business decisions of any import. It's virtually impossible to be completely sure about a merger, product launch, new venture or even an individual employee hire.
But isn't big data supposed to help us with uncertainty? Yes, but don't confuse reducing uncertainty with eliminating it. That day isn't here yet, and I suspect that it won't arrive anytime soon.
Analyzing petabytes of unstructured data may well help companies better understand customer sentiment. However, don't make the mistake of assuming that big data eliminates all variability. The fluctuations of life and business will still throw a wrench into the best laid plans.
Myth 4: Big Data Is A Temporary Fad
Arguably, the current face of big data is Nate Silver -- or at least it was until he left the New York Times. The blogger and statistician famously predicted that Barack Obama was a 90% favorite to win the 2012 U.S. presidential election, despite the fact that polls put Obama in a virtual dead heat with Mitt Romney. Silver's model was remarkably accurate, and now everyone is asking for his take on everything.
To be sure, the terms big data and data science may vanish into the ether over the next few years. We do like our buzzwords and jargon. Foolish is the professional, however, who believes that data is a fad. I'm sure of very few things, but I know that in 2013 we will collectively generate and consume more data than we did in 2012.
It's high time that organizations recognize the importance of big data. Refuse, and your company may not be around when the light bulb finally goes off.
The big data market is not just about technologies and platforms -- it's about creating new opportunities and solving problems. The Big Data Conference provides three days of comprehensive content for business and technology professionals seeking to capitalize on the boom in data volume, variety and velocity. The Big Data Conference happens in Chicago, Oct. 21-23.
6 Tools to Protect Big DataMost IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Big Data Brings Big Security ProblemsWhy should big data be more difficult to secure? In a word, variety. But the business won’t wait to use it to predict customer behavior, find correlations across disparate data sources, predict fraud or financial risk, and more.
Join us for a roundup of the top stories on InformationWeek.com for the week of December 14, 2014. Be here for the show and for the incredible Friday Afternoon Conversation that runs beside the program.