EMC says that between 2005 and 2020, total stored information, what it calls the digital universe, will increase by a factor of 300 -- not 300%, but 300 times. By 2015, the storage vendor says, we'll be sitting on a collective 7,910 exabytes. Now, it's true that number likely reflects some aspirational thinking. But it's also true that in 2011, when EMC last issued a digital universe update, the Internet of things wasn't mainstream, and few of us thought much about instrumenting every corner of our worlds and saving every scrap of data for time eternal, the way we are now.
For the sake of argument, let's take EMC at its word. Even if it's off by a factor of 10, the implications are profound on a number of levels -- cost, analysis methods, security, management, how cloud plays in. Companies that are sitting tight waiting for that mythical perfect big data wave should know that many of these areas have matured. Don't hesitate too long: Emerging tools now put analysis within reach, without drowning users in complexity, and costs can be kept under control. There still are risks in investing in emerging big data analytics, but there's also risk in missing business opportunities that might have bubbled up from those oceans of data.
The thing that scares IT pros most about big data is that it sounds darned expensive. In our InformationWeek2013 Big Data Survey, when asked about the top barrier to successful use of big data, the No. 1 answer by far was budget constraints, at 38%. Just 13% cite a lack of a business interest, the No. 2 response.
One factor to remember is that the predominant vendor of the previous era usually isn't the leader in the next era. Cost drives that changing of the guard whenever users want to do [insert new IT trend] right this minute but find it's prohibitively expensive with current-generation stuff. Mainframes were too expensive to support most scientific and engineering needs, so we got minis and workstations. Those were too expensive for personal productivity, so enter the PC. PCs weren't portable enough, so $500 tablets appeared and suddenly we're all supporting iPads.
Software seems to move more slowly than hardware in these shifts, sometimes taking decades for a transformation away from a dominant technology. But when it comes to managing and analyzing big data, that kind of important transition is happening right now. Everything is in transition, from servers to endpoints, from dominant operating systems to databases to presentation technology. It's also why now is simultaneously an exciting time for IT pros who like cruising a wave of change and a scary ride for those who prefer to stay safely on shore.
6 Tools to Protect Big DataMost IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Big Data Brings Big Security ProblemsWhy should big data be more difficult to secure? In a word, variety. But the business won’t wait to use it to predict customer behavior, find correlations across disparate data sources, predict fraud or financial risk, and more.
InformationWeek Must Reads Oct. 21, 2014InformationWeek's new Must Reads is a compendium of our best recent coverage of digital strategy. Learn why you should learn to embrace DevOps, how to avoid roadblocks for digital projects, what the five steps to API management are, and more.