Software // Information Management
News
11/7/2011
11:20 AM
Connect Directly
Google+
LinkedIn
Twitter
RSS
E-Mail
50%
50%

Hadoop Spurs Big Data Revolution

Open source data processing platform has won over Web giants for its low cost, scalability, and flexibility. Now Hadoop will make its way into more enterprises.

Analyzing The Internet

Another company rolling out a large-scale Hadoop deployment is digital media measurement company ComScore. It's planning to use Hadoop as its main platform for raw data analysis, replacing a homegrown, grid-based system built on commodity hardware that it has used since 2004. The grid preprocesses raw data, boiling down hundreds of terabytes of Web clickstream data into orderly data sets that can be loaded onto ComScore's 150-TB Sybase IQ data warehouse, a row-oriented, relational database best suited to analytics.

Sybase IQ lets ComScore measure the traffic of the world's leading websites and do marketing segmentation based on the surfing habits of its panel of more than 2 million Web users. (ComScore's panel is a Web version of the Nielsen households used to track TV viewing.)

ComScore's Hadoop platform is expected to scale better than its grid system, while providing higher utilization rates and reducing operations costs, says CTO Michael Brown. It will also free the company's developers to work on business problems rather than having to maintain and scale a proprietary stack, Brown says.

NoSQL's Driving Factors
What factors are driving, or would drive, your company's interest in using alternative data platforms such as Hadoop?
31%
Ability to manage and process nonrelational and unstructured data
30%
Ability to manage and process massive volumes of data
23%
Lower software and deployment costs than commercial products
23%
Lower hardware and storage scaling costs than commercial products
16%
Interest in new insights, such as social media analysis
47%
Such platforms aren't a priority for my company
Data: InformationWeek 2012 Business Intelligence, Analytics, and Information Management Survey of 431 business technology pros involved with information management tech, October 2011
ComScore first put Hadoop to work for Social Essentials, a service it introduced in June that processes the 5 TB of panelist data the company collects each day to determine the extent to which top social networks, social network brand pages, and influential people on social networks boost visits to and purchases from specific websites.

ComScore's panelists visit more than 140 million social network pages a day. "The Facebook API gives you basic statistics, but marketers have a huge need to know the impact of influencers, the Facebook news feed, the Facebook wall, and branded pages," Brown says.

Using algorithms running on top of Hadoop, ComScore determines which friends, influencers, and pages panelists visited on a given social network. ComScore also has profile information on its panelists and their Web activities, and it uses that information to develop broader insights about social network usage.

Social Essentials is geared to help marketers understand the effectiveness of their social networking activities. If you're Southwest Airlines, for example, the service can tell you that 3% of Web users are likely to visit your site, whereas 12% of those who are fans of the airline's Facebook page are likely to visit and 8% of friends of Facebook fans are likely to visit, Brown says.

Previous
4 of 5
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
6/15/2012 | 10:10:09 PM
re: Hadoop Spurs Big Data Revolution
That's per core, but these stats have all been surpassed with the latest hardware.
molloy
50%
50%
molloy,
User Rank: Apprentice
12/6/2011 | 4:08:09 AM
re: Hadoop Spurs Big Data Revolution
Reading through the whole document I see only one mention of Yahoo, and no mention of Yahoo as the originator of Hadoop. It sometimes appears that the Press is intent on highlighting all of Yahoo's weaknesses, and none of it's strengths. Perhaps you think this information is already well-known, but the pie-chart showing that 74% have "no current or planned use" would suggest otherwise. For those who wish to read more meaty detail, see http://developer.yahoo.com/had....
IKODUKULA945
50%
50%
IKODUKULA945,
User Rank: Apprentice
12/4/2011 | 8:59:29 PM
re: Hadoop Spurs Big Data Revolution
Matspca - we're working on establishing a benchmark for Hadoop. If you'd like to participate, please let me know at indu.kodukula@sungard.com
matspca
50%
50%
matspca,
User Rank: Apprentice
11/30/2011 | 11:01:45 PM
re: Hadoop Spurs Big Data Revolution
Not everyone believes in the Hype of Hadoop. See http://www.vertica.com/2011/09... The big organizations mentioned here can afford to use non optimal solutions. I have seen no benchmark showing Hadoop beating say Oracle. My own noSQL database beats Hadoop by a large margin using $330 PC verses $1 million (or so) used by Hadoop for the same benchmark. See http://www.velocitydb.com/Comp...

I will continue following the Hype of Hadoop and if there really is some substance behind it then I look forward to a .NET version of the distribution mechanism.
RodneyG79
50%
50%
RodneyG79,
User Rank: Apprentice
11/10/2011 | 8:48:32 PM
re: Hadoop Spurs Big Data Revolution
128 MB of RAM for 16 cores? That has to be typo.
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest, Dec. 9, 2014
Apps will make or break the tablet as a work device, but don't shortchange critical factors related to hardware, security, peripherals, and integration.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of December 7, 2014. Be here for the show and for the incredible Friday Afternoon Conversation that runs beside the program!
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.