Big Data // Software Platforms
Commentary
3/20/2014
11:34 AM
Doug Henschen
Doug Henschen
Commentary
Connect Directly
LinkedIn
Twitter
Google+
RSS
E-Mail

Big Data Reaches Inflection Point

Enterprises see the light on big data opportunities. It's only a matter of time before mainstream data-management environments evolve.
2 of 2

Pivotal CEO Paul Maritz.
Pivotal CEO Paul Maritz.

2 of 2
Comment  | 
Print  | 
Comments
Newest First  |  Oldest First  |  Threaded View
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
3/24/2014 | 12:21:57 PM
Re: Revolution or Evolution
My latest example of a data lake user -- coming in a story next week -- is a major drug manufacturer that is running Hortonworks on Amazon's cloud. The deployment lets them study all data related to the production of drugs -- 10+ years worth of information including plant-floor temperature and pressure sensor readings -- in order to spot yeild variations by batch and determine how to optimize production yeild. It's the sort of lighthouse implementation that shows others the way and makes adoption much more likely in the five-year time frame than the 15-year time frame. Stay tuned for the full story next week.
weckerson
50%
50%
weckerson,
User Rank: Apprentice
3/21/2014 | 9:16:25 AM
Revolution or Evolution
The estimates are high but we'll get there eventually -- 15-20 years? The data lake is an aspirational architecture that is for bleeding-edge companies only right now. We'll see slow evolution for some time to come. 
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
3/21/2014 | 8:26:20 AM
Calculating the rate of infection
I'm told that Cloudera and Hortonworks are each logging deals at a rate of 50 to 70 per quarter. Throw in all the other Hadoop distributors and call it 500 to 700 per year. That's at current rates, which we can assume will accelerate. In 2013, for instance, sales teams at HP, Microsoft, SAP, and Teradata added Hortonworks' distribution to the portfolio, and we can confidently guess that having more feet on the street will increase sales. Cloudera notes that deal sizes and deployments are growting. And let's assume also that these suppliers are pursuing the biggest companies possible.

With these stats, it won't take long to see big adoption among the Global 2000. From there the question is how much serious, production-grade work are they doing on the platform and to what degree are they moving existing workloads onto Hadoop? In 2014 we're still in the early days. By 2020, I think the data-management arena is going to be very different.  
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
3/20/2014 | 5:35:06 PM
Re: How much data moves to Hadoop?
Those were figures for 2017 into 2020. That's an eternity in a fast-moving market.
Laurianne
50%
50%
Laurianne,
User Rank: Author
3/20/2014 | 5:23:16 PM
Re: How much data moves to Hadoop?
Hadoop as parking garage. Is that efficient?
tunvall01
50%
50%
tunvall01,
User Rank: Apprentice
3/20/2014 | 3:22:26 PM
Re: How much data moves to Hadoop?
Rob Bearden's numbers are extremely high. Hadoop techonology and skills are way to immature to succesfully be adopted in the majority of operational and production systems. 
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
3/20/2014 | 2:31:40 PM
Hadoop data vs. Know SQL data
Hadoop is something like a big, undifferentiated resevoir. It collects data from everywhere. The water coming out of your faucet, on the other hand, is charcoal-filtered and chlorinated. Yes, there's a lot of water in the resevoir, but remember where the stuff good to drink is coming from. Sorry, we're having a drought here in Calif.
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
3/20/2014 | 1:18:14 PM
Re: How much data moves to Hadoop?
The $50 billion figure is the one that sounds high. The Data Hub/Data Lake concept foresees Hadoop as a place where you store many types of data, including high-scale machine data (log files, clickstreams) that many companies haven't been collecting or keeping until recently. That you have copies of such data in a hub does NOT mean that you are necessarily replacing transactional systems or even data warehouses. It's a cheaper platform on which to gain new insights and perhaps replace some workloads. If the Hub has high-scale data and lots of historcal data long since deleted from operatoinal systems, it's not at all unrealistic to see 60%, 70%, 80% of the total data footprint in an organization being stored on Hadoop.

 
Laurianne
50%
50%
Laurianne,
User Rank: Author
3/20/2014 | 12:52:25 PM
How much data moves to Hadoop?
"Rob Bearden, CEO of Cloudera-rival Hortonworks, said that he expects see "60%, 70%, 80%" of enterprise data moving into Hadoop over the coming years." Does this sound high to you, Doug? Sounds pretty high to me given state of tools and Hadoop talent.
In A Fever For Big Data
In A Fever For Big Data
Healthcare orgs are relentlessly accumulating data, and a growing array of tools are becoming available to manage it.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Must Reads Oct. 21, 2014
InformationWeek's new Must Reads is a compendium of our best recent coverage of digital strategy. Learn why you should learn to embrace DevOps, how to avoid roadblocks for digital projects, what the five steps to API management are, and more.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
A roundup of the top stories and trends on InformationWeek.com
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.