ComScore's Big Data Deployment In Detail - InformationWeek
IoT
IoT
Software // Information Management
Commentary
11/24/2010
11:04 AM
Doug Henschen
Doug Henschen
Commentary
Connect Directly
LinkedIn
Twitter
Google+
RSS
E-Mail
50%
50%

ComScore's Big Data Deployment In Detail

Crunching tens of terabytes, a leading digital marketing measurement firm delivers insight -- quickly.

"If I had gone with the big Unix-box approach, at some point there would be a major cost," he explains. "We went with the commodity approach because we like buying things with five zeros rather than six zeros."

Compression is a key advantage for column-store databases. comScore gets anywhere between 25% and 40% compression, depending on the type of data being analyzed, according to Smith. In terms of performance, increases in processing power and data input/output speed and throughput have brought steady improvements. As an example, an analysis of Internet search engine traffic that took five hours on comScore's last Sybase IQ deployment now takes 11 minutes in the new deployment, Smith says.

Sybase IQ runs on symmetric multiprocessor (SMP) hardware, a contrast with the many appliances and databases that now run on massively parallel processing (MPP) architectures. MPP architectures have advantages when it comes to scalability, but Smith says Sybase IQ's Multiplex (grid scaling) approach has advantages in terms of flexibility.

"I can do parallelized loads and the things that MPP is good at, but I also have the throttle of being able to do multiple tasks at the same time with Multiplex," he says, contrasting the multiple, independent database nodes of the IQ grid with the single-database-controller/worker-node approach of MPP.

Smith says comScore has looked at MPP alternatives from the likes of Vertica and ParAccel, column-store competitors to Sybase IQ, but a switch would also mean the company couldn't leverage prior investments.

"I have all this commodity hardware that I can just move over to the new warehouse, so why wouldn't I do that?" he says.

In the never-ending challenge of analyzing more data and doing it more quickly, Smith says Sybase IQ has always provided what he describes as an extensible model.

"We don't want to design into a box, so I'm always interested in where Sybase is planning to go and how I'm going to leverage the platform."

Incremental investment, the ability to leverage existing hardware, extensibility when it comes to processing power and storage: these aren't capabilities that are exclusive to Sybase IQ, but not all data warehousing approaches and technologies give you this flexibility. Thus, take a cue from comScore's experience and explore these system capabilities in detail in your next data warehousing RFP.

Previous
2 of 2
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of the Cloud Report
As the use of public cloud becomes a given, IT leaders must navigate the transition and advocate for management tools or architectures that allow them to realize the benefits they seek. Download this report to explore the issues and how to best leverage the cloud moving forward.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of November 6, 2016. We'll be talking with the InformationWeek.com editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll