Think Today's Data Is Big? Wait 10 Years

16 Top Big Data Analytics Platforms

At what point does data make the leap to big data? There's no internationally sanctioned demarcation, of course, and it would appear that the rapid growth of unstructured data, particularly video, has the imaginary boundary on the move.

According to Chris Gladwin, founder and vice chairman of the object storage vendor Cleversafe, storage requirements have soared dramatically over the past decade, but that's just the start. Emerging technologies such as 4K video and cloud services will make today's big data storage needs seem laughably small by 2024.

Today, a petabyte of data -- 1,024 terabytes, to be exact -- probably meets many people's definition of "big data." The storage manufacturer Aberdeen, for example, sells a one-petabyte storage rack, aptly named the Petarack, for a cool $375,000.

Fast-forward 10 years, however, and a petabyte no longer will qualify as big -- at least not in the enterprise. "You think of it as a pretty big system today, but in 10 years, you won't even be able to buy systems that small," Gladwin told InformationWeek. "Today, a petabyte is half a cabinet, but you go out 10 years, and that's like part of a server. It won't even qualify as an enterprise-scale system."

[UK startup tackles the challenge of enabling computers to understand the spoken word. Read Natural Language Processing: Big Data's Role.]

Since Gladwin founded Cleversafe in 2004, the Chicago company has always focused on storage systems with very large capacity. A decade ago, only 60-70 companies in the world acquired a petabyte or more of new storage each year. "There was maybe one organization at that time that was at a 100-petabyte scale, and that was it. Now, 10 years later, the number of companies in the world that are deploying a petabyte or more of storage every year is around 7,000."

A decade from now, "when you look at the capacity-optimized segment of enterprise storage -- which is the big enterprise storage systems -- we're projecting that zero percent of the market will be systems that are a petabyte or less."

So if a petabyte won't qualify as big data, what will?

Perhaps an exabyte, or 1,024 petabytes. "Today's petabytes are tomorrow's exabytes. There are 5-10 organizations this year that will deploy at exabyte scale."

Currently, only a small group of companies need that much storage. "It's the Internet-deployed services that are storage-intensive. It can include social networking and companies that provide storage-as-a-service to others. The really big ones are at that scale."

The growing demand from cloud storage, particularly user-generated content, is driving this trend, as well. "You take a bunch of video files -- you get a hundred million customers making them -- and start storing them. The next thing you know, you're at exabyte-scale pretty quick." Government installations are reaching that level, too.

The arrival of 4K video -- and eventually 8K -- will also hasten the arrival of exabyte-level storage systems. "We're beginning the transition to 4K, and you can see the transition to 8K after that. Even with compression, each is basically a 4x jump" in storage. "If you really want to get to massive scale, you're not going to get there with numbers, text, and documents. You're really talking about big objects, and lots of them -- like videos or genomes, things like that."

Google is already at that scale, Gladwin said. "There's a billion people making videos, and they all land on YouTube."

Engage with Oracle president Mark Hurd, NFL CIO Michelle McKenna-Doyle, General Motors CIO Randy Mott, Box founder Aaron Levie, UPMC CIO Dan Drawbaugh, GE Power CIO Jim Fowler, and other leaders of the Digital Business movement at the InformationWeek Conference and Elite 100 Awards Ceremony, to be held in conjunction with Interop in Las Vegas, March 31 to April 1, 2014. See the full agenda here.