Hadoop At 10: Milestones And Momentum

Hadoop, an open source framework for wrangling unstructured data and analytics, celebrated its 10th birthday in January. Here's a look at the milestones, players, and events that marked the growth of this groundbreaking technology.
Cloudera Founded
MapR Founded
Hortonworks Spun Out Of Yahoo
Other Hadoop Distributions
Apache Hadoop Version 1.0
Let's All Get Together And Talk Hadoop
Spark As A Catalyst
Enterprise Adoption
Digital Natives Embrace Hadoop
Hadoop's Many Applications And Future

The year was 2006. Facebook was a two-year-old startup company, run by a 21-year-old in a hoodie. Some entrepreneurs had joined together to launch a new social media service called Twitter, and in December the world was still six months away from seeing the introduction of the iPhone. It was a different time.

Consumers weren't carrying around sensor-laden, camera-equipped data collection devices everywhere they went and posting every thought, emotion, and meal to social media. When companies thought about data, they thought about structured data in ERP and CRM systems, and how they could create better business intelligence reports for executives.

It was in this environment that a new technology called Hadoop was born. It started as a framework to support a search engine project called Nutch. Nutch's creators needed a way to store and process the massive amount of data collected for their search engine to use, so they created a new software framework based on inspiration gained from a couple of papers published by growing Silicon Valley upstart Google.

[Check out our interview with Hadoop creator Doug Cutting as he talks about what this birthday means for big data and the software framework. Read Hadoop At 10: Doug Cutting On Making Big Data Work.]

These two developers, Doug Cutting and Mike Cafarella, eventually joined a different company called Yahoo, which then was struggling to retain its lead in the metric of site visits against this upstart Google. At Yahoo, their work on the distributed file system and framework for parallel processing was named Hadoop, after a toy stuffed elephant that Cutting's son had. Yahoo eventually sent Hadoop to open source organization the Apache Foundation. And work continued on making this not-quite-ready-for-prime-time distributed storage and processing system more scalable.

Today, Hadoop has entered a new stage. Load improvements, as well as add-on projects, have turned the software framework into a powerful tool used in a number of big companies, including Facebook, Twitter, eBay, and Salesforce. Hadoop indeed seems like it's getting ready for prime time.

In the Forrester Wave: Big Data Hadoop Distributions, Q1 2016 report, the analyst firm said: "Enterprise Hadoop is a market that is not even 10 years old, but Forrester estimates that 100% of all large enterprises will adopt [Hadoop and related technologies such as Spark] for big data analytics within the next two years."

To celebrate Hadoop's 10-year anniversary, come with us as we look back at some of the milestones, key players, and important developments in Hadoop's history.

Are you a Hadoop user? Is it something you're considering for your enterprise? Is there an important milestone we missed? Tell us all about it in the comments section below.

Rising stars wanted. Are you an IT professional under age 30 who's making a major contribution to the field? Do you know someone who fits that description? Submit your entry now for InformationWeek's Pearl Award. Full details and a submission form can be found here.

Next slide