Hadoop, an open source framework for wrangling unstructured data and analytics, celebrated its 10th birthday in January. Here's a look at the milestones, players, and events that marked the growth of this groundbreaking technology.
1 of 11
The year was 2006. Facebook was a two-year-old startup company, run by a 21-year-old in a hoodie. Some entrepreneurs had joined together to launch a new social media service called Twitter, and in December the world was still six months away from seeing the introduction of the iPhone. It was a different time.
Consumers weren't carrying around sensor-laden, camera-equipped data collection devices everywhere they went and posting every thought, emotion, and meal to social media. When companies thought about data, they thought about structured data in ERP and CRM systems, and how they could create better business intelligence reports for executives.
It was in this environment that a new technology called Hadoop was born. It started as a framework to support a search engine project called Nutch. Nutch's creators needed a way to store and process the massive amount of data collected for their search engine to use, so they created a new software framework based on inspiration gained from a couple of papers published by growing Silicon Valley upstart Google.
These two developers, Doug Cutting and Mike Cafarella, eventually joined a different company called Yahoo, which then was struggling to retain its lead in the metric of site visits against this upstart Google. At Yahoo, their work on the distributed file system and framework for parallel processing was named Hadoop, after a toy stuffed elephant that Cutting's son had. Yahoo eventually sent Hadoop to open source organization the Apache Foundation. And work continued on making this not-quite-ready-for-prime-time distributed storage and processing system more scalable.
Today, Hadoop has entered a new stage. Load improvements, as well as add-on projects, have turned the software framework into a powerful tool used in a number of big companies, including Facebook, Twitter, eBay, and Salesforce. Hadoop indeed seems like it's getting ready for prime time.
In the Forrester Wave: Big Data Hadoop Distributions, Q1 2016 report, the analyst firm said: "Enterprise Hadoop is a market that is not even 10 years old, but Forrester estimates that 100% of all large enterprises will adopt [Hadoop and related technologies such as Spark] for big data analytics within the next two years."
To celebrate Hadoop's 10-year anniversary, come with us as we look back at some of the milestones, key players, and important developments in Hadoop's history.
Are you a Hadoop user? Is it something you're considering for your enterprise? Is there an important milestone we missed? Tell us all about it in the comments section below.
Rising stars wanted. Are you an IT professional under age 30 who's making a major contribution to the field? Do you know someone who fits that description? Submit your entry now for InformationWeek's Pearl Award. Full details and a submission form can be found here.
Jessica Davis has spent a career covering the intersection of business and technology at titles including IDG's Infoworld, Ziff Davis Enterprise's eWeek and Channel Insider, and Penton Technology's MSPmentor. She's passionate about the practical use of business intelligence, ... View Full Bio
How Enterprises Are Attacking the IT Security EnterpriseTo learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
2017 State of IT ReportIn today's technology-driven world, "innovation" has become a basic expectation. IT leaders are tasked with making technical magic, improving customer experience, and boosting the bottom line -- yet often without any increase to the IT budget. How are organizations striking the balance between new initiatives and cost control? Download our report to learn about the biggest challenges and how savvy IT executives are overcoming them.