EMC says that between 2005 and 2020, total stored information, what it calls the digital universe, will increase by a factor of 300 -- not 300%, but 300 times. By 2015, the storage vendor says, we'll be sitting on a collective 7,910 exabytes. Now, it's true that number likely reflects some aspirational thinking. But it's also true that in 2011, when EMC last issued a digital universe update, the Internet of things wasn't mainstream, and few of us thought much about instrumenting every corner of our worlds and saving every scrap of data for time eternal, the way we are now.
For the sake of argument, let's take EMC at its word. Even if it's off by a factor of 10, the implications are profound on a number of levels -- cost, analysis methods, security, management, how cloud plays in. Companies that are sitting tight waiting for that mythical perfect big data wave should know that many of these areas have matured. Don't hesitate too long: Emerging tools now put analysis within reach, without drowning users in complexity, and costs can be kept under control. There still are risks in investing in emerging big data analytics, but there's also risk in missing business opportunities that might have bubbled up from those oceans of data.
What you'll find:
- Table of the likely annual data volume for a $250 million, 500-person organization
- Three "little" big data lies IT must address (including security)
The thing that scares IT pros most about big data is that it sounds darned expensive. In our InformationWeek 2013 Big Data Survey, when asked about the top barrier to successful use of big data, the No. 1 answer by far was budget constraints, at 38%. Just 13% cite a lack of a business interest, the No. 2 response.
One factor to remember is that the predominant vendor of the previous era usually isn't the leader in the next era. Cost drives that changing of the guard whenever users want to do [insert new IT trend] right this minute but find it's prohibitively expensive with current-generation stuff. Mainframes were too expensive to support most scientific and engineering needs, so we got minis and workstations. Those were too expensive for personal productivity, so enter the PC. PCs weren't portable enough, so $500 tablets appeared and suddenly we're all supporting iPads.
Software seems to move more slowly than hardware in these shifts, sometimes taking decades for a transformation away from a dominant technology. But when it comes to managing and analyzing big data, that kind of important transition is happening right now. Everything is in transition, from servers to endpoints, from dominant operating systems to databases to presentation technology. It's also why now is simultaneously an exciting time for IT pros who like cruising a wave of change and a scary ride for those who prefer to stay safely on shore.
download the InformationWeek October special issue on big data analytics.