Hadoop At 10: Milestones And Momentum - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Data Management // Software Platforms
News
2/3/2016
07:06 AM
Jessica Davis
Jessica Davis
Slideshows
Connect Directly
Twitter
RSS
E-Mail
50%
50%

Hadoop At 10: Milestones And Momentum

Hadoop, an open source framework for wrangling unstructured data and analytics, celebrated its 10th birthday in January. Here's a look at the milestones, players, and events that marked the growth of this groundbreaking technology.
Previous
1 of 11
Next

(Image: Cloudera)

(Image: Cloudera)

The year was 2006. Facebook was a two-year-old startup company, run by a 21-year-old in a hoodie. Some entrepreneurs had joined together to launch a new social media service called Twitter, and in December the world was still six months away from seeing the introduction of the iPhone. It was a different time.

Consumers weren't carrying around sensor-laden, camera-equipped data collection devices everywhere they went and posting every thought, emotion, and meal to social media. When companies thought about data, they thought about structured data in ERP and CRM systems, and how they could create better business intelligence reports for executives.

It was in this environment that a new technology called Hadoop was born. It started as a framework to support a search engine project called Nutch. Nutch's creators needed a way to store and process the massive amount of data collected for their search engine to use, so they created a new software framework based on inspiration gained from a couple of papers published by growing Silicon Valley upstart Google.

[Check out our interview with Hadoop creator Doug Cutting as he talks about what this birthday means for big data and the software framework. Read Hadoop At 10: Doug Cutting On Making Big Data Work.]

These two developers, Doug Cutting and Mike Cafarella, eventually joined a different company called Yahoo, which then was struggling to retain its lead in the metric of site visits against this upstart Google. At Yahoo, their work on the distributed file system and framework for parallel processing was named Hadoop, after a toy stuffed elephant that Cutting's son had. Yahoo eventually sent Hadoop to open source organization the Apache Foundation. And work continued on making this not-quite-ready-for-prime-time distributed storage and processing system more scalable.

Today, Hadoop has entered a new stage. Load improvements, as well as add-on projects, have turned the software framework into a powerful tool used in a number of big companies, including Facebook, Twitter, eBay, and Salesforce. Hadoop indeed seems like it's getting ready for prime time.

In the Forrester Wave: Big Data Hadoop Distributions, Q1 2016 report, the analyst firm said: "Enterprise Hadoop is a market that is not even 10 years old, but Forrester estimates that 100% of all large enterprises will adopt [Hadoop and related technologies such as Spark] for big data analytics within the next two years."

To celebrate Hadoop's 10-year anniversary, come with us as we look back at some of the milestones, key players, and important developments in Hadoop's history.

Are you a Hadoop user? Is it something you're considering for your enterprise? Is there an important milestone we missed? Tell us all about it in the comments section below.

Rising stars wanted. Are you an IT professional under age 30 who's making a major contribution to the field? Do you know someone who fits that description? Submit your entry now for InformationWeek's Pearl Award. Full details and a submission form can be found here.

Jessica Davis has spent a career covering the intersection of business and technology at titles including IDG's Infoworld, Ziff Davis Enterprise's eWeek and Channel Insider, and Penton Technology's MSPmentor. She's passionate about the practical use of business intelligence, ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Previous
1 of 11
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
2/4/2016 | 7:49:31 PM
Production version is relatively new
Interesting points on Hadoop. I think of Hadoop as one of the earliest and most straight forward examples of how a cloud software system is supposed to work. Hard to believe the 1.0 version was as recent at 2012.
batye
50%
50%
batye,
User Rank: Ninja
2/3/2016 | 9:46:56 AM
Re: 100%
@Ariella I trust you are right, changes in the way, we just not seeing them yet but Hadoop is here to stay and grow... - how I see it...
Ariella
100%
0%
Ariella,
User Rank: Author
2/3/2016 | 8:55:55 AM
100%
I'm used to seeing fairly bullish prediction, but usually there is at least some qualifier involved. I'm surprised they go all out and say 100% here: "Enterprise Hadoop is a market that is not even 10 years old, but Forrester estimates that 100% of all large enterprises will adopt [Hadoop and related technologies such as Spark] for big data analytics within the next two years."
Slideshows
What Digital Transformation Is (And Isn't)
Cynthia Harvey, Freelance Journalist, InformationWeek,  12/4/2019
Commentary
Watch Out for New Barriers to Faster Software Development
Lisa Morgan, Freelance Writer,  12/3/2019
Commentary
If DevOps Is So Awesome, Why Is Your Initiative Failing?
Guest Commentary, Guest Commentary,  12/2/2019
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
Getting Started With Emerging Technologies
Looking to help your enterprise IT team ease the stress of putting new/emerging technologies such as AI, machine learning and IoT to work for their organizations? There are a few ways to get off on the right foot. In this report we share some expert advice on how to approach some of these seemingly daunting tech challenges.
Slideshows
Flash Poll