Software // Information Management
News
6/15/2012
01:57 AM
Connect Directly
Twitter
RSS
E-Mail
50%
50%

Yahoo And Hadoop: In It For The Long Term

Yahoo made an early bet that it could use Hadoop to monetize the big data the content site was collecting. Now it's betting the company on it.

12 Hadoop Vendors To Watch In 2012
12 Hadoop Vendors To Watch In 2012
(click image for larger view and for slideshow)
As Hadoop's first large-scale user, Yahoo still holds a special place in Hadoop's rapidly expanding universe.

Yahoo started out using Hadoop to speed up indexing of Web crawl results for its search engine. Now it relies on Microsoft's Bing for search results, and a Bing team does the heavy lifting with Web data.

Where does that leave Yahoo's relationship with Hadoop today? The company is still betting the business on it, according to Scott Burke, senior VP of advertising and data platforms. "Yahoo remains committed to Hadoop, more than ever before. It's the only platform we use globally. We have the largest Hadoop footprint in the world," said Burke in his keynote address at the Hadoop Summit 2012 Thursday in San Jose.

Yahoo runs Hadoop on 42,000 servers--that's 1,200 racks--in four data centers. Its largest Hadoop cluster is 4,000 nodes, but that will increase to 10,000 with the release of Apache Hadoop 2.0, expected shortly.

[ Why did VMware launch another Hadoop open source project? It needs to be virtualized. Read VMware Launch Open Source Project To Virtualize Hadoop . ]

After his keynote, Burke explained some of the key ways in which Hadoop has become integral to how Yahoo does business. For starters, Yahoo uses Hadoop to block spam trying to get into its email servers. Burke says it screens out 20.5 billion spam messages a day. "It's really transformed our spam detection capabilities," he said. Hadoop sits at the base of a powerful software stack inside Yahoo and powers the "personalization" Yahoo strives to give its visitors. Hadoop already knows something about you based on your previous visits, so if you request sports or financial news, for example, you'll get a "high-value package" reflecting your interests. (Yahoo uses a combination of automated analysis and human editors to define packages. If left only to automation, everyone would get packages about celebrities, which draw the highest traffic on the site.)

"Our goal is to give every consumer a personalized experience," said Burke. That means Yahoo can give visitors who've just read a particular story a list of related story choices that may also appeal to them.

Yahoo isn't using Hadoop as a stand-alone system. Rather, it serves as an information foundation for an Oracle database system, which pulls presorted, indexed data out and feeds it into a Microsoft SQL Server cube for highly detailed analysis. The resulting data is displayed in either Tableau or Microstrategy visualization systems to Yahoo business analysts. They in turn use it to advise advertisers how their campaigns are faring soon after launch.

Yahoo can advise advertisers on the demographic that responds best to their campaigns so that advertisers can adjust their ads as necessary to appeal to the widest audience. According to Burke, this can lead to improved results, sometimes even doubling expected response rates.

Yahoo's data indicates not only who is viewing advertising, but also how much time they're spending on it, what their click-through rate is, and what they do on the Yahoo site after viewing an ad. This is all valuable feedback to advertisers, as it lets advertisers make decisions early in an ad campaign and helps them plan their future advertising strategies.

In addition, Yahoo has also developed a set of open source code called Cocktails, which gives advertisers the tools to extract information in Yahoo's Hadoop systems. Cocktails are written in JavaScript so they can run in browser windows; Manhattan is a server-side cocktail and Mojito is client-side. Essentially, Burke explained, each cocktail could run the same reporting program on a server or an end-user laptop, respectively, to get information that an advertiser is seeking. "We believe over the years, Cocktails will replace PHP in the enterprise. JavaScript is a better, more interactive language," Burke said.

Cocktails work with a Yahoo service, Advertiser Insights, to find information. The reporting system works on a Hadoop-Oracle-Pentaho Mondrian (open source code for building multi-view cubes) stack. "Yahoo has a lot of valuable data. Advertisers can see all the information that we've learned about their customers," noted Burke.

Finally, Yahoo uses Hadoop for its own internal analysis of information captured from user interactions. It stores 140 petabytes in Hadoop. Since Hadoop keeps all data sets in triplicate, over 400 petabytes of storage are needed to sustain its systems.

Burke, a seven-year veteran at Yahoo, said the company started using Hadoop when it was a rudimentary system. Yahoo developed it, made it open source code, and at the same time bet the business on it by developing a business model for monetizing content. In the process, Burke conceded that Yahoo has "gotten a lot of attention, sometimes for the wrong reasons." Over the years, the company has been rumored to be for sale, or undergoing significant changes in management or its business model.

Burke is confident that Yahoo's success with Hadoop will lead the company to the top in its various business initiatives. Yahoo is becoming an expert at capitalizing on "the big shift from the off-line analysis model to something much closer to a predictive model: Get a response in front of the customer with the right offer at the right time," said Burke.

To achieve this goal, Yahoo has had to implement "science at scale," or invest in an open source system that seemed to have broad potential but wasn't yet proven in prime time. "We've bet the business on this platform at on a global scale," Burke says, "and there's no turning back."

At the Big Data Analytics interactive InformationWeek Virtual Event, experts and solution providers will offer detailed insight into how to put big data to use in ad hoc analyses, what-if scenario planning, customer sentiment analysis, and the building of highly accurate data models to drive better predictions about fraud, risk, and customer behavior. It happens June 28.

Comment  | 
Print  | 
More Insights
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Must Reads Oct. 21, 2014
InformationWeek's new Must Reads is a compendium of our best recent coverage of digital strategy. Learn why you should learn to embrace DevOps, how to avoid roadblocks for digital projects, what the five steps to API management are, and more.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
A roundup of the top stories and trends on InformationWeek.com
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.