Software // Information Management
Commentary
11/24/2010
11:04 AM
Doug Henschen
Doug Henschen
Commentary
Connect Directly
LinkedIn
Twitter
Google+
RSS
E-Mail
50%
50%

ComScore's Big Data Deployment In Detail

Crunching tens of terabytes, a leading digital marketing measurement firm delivers insight -- quickly.

"If I had gone with the big Unix-box approach, at some point there would be a major cost," he explains. "We went with the commodity approach because we like buying things with five zeros rather than six zeros."

Compression is a key advantage for column-store databases. comScore gets anywhere between 25% and 40% compression, depending on the type of data being analyzed, according to Smith. In terms of performance, increases in processing power and data input/output speed and throughput have brought steady improvements. As an example, an analysis of Internet search engine traffic that took five hours on comScore's last Sybase IQ deployment now takes 11 minutes in the new deployment, Smith says.

Sybase IQ runs on symmetric multiprocessor (SMP) hardware, a contrast with the many appliances and databases that now run on massively parallel processing (MPP) architectures. MPP architectures have advantages when it comes to scalability, but Smith says Sybase IQ's Multiplex (grid scaling) approach has advantages in terms of flexibility.

"I can do parallelized loads and the things that MPP is good at, but I also have the throttle of being able to do multiple tasks at the same time with Multiplex," he says, contrasting the multiple, independent database nodes of the IQ grid with the single-database-controller/worker-node approach of MPP.

Smith says comScore has looked at MPP alternatives from the likes of Vertica and ParAccel, column-store competitors to Sybase IQ, but a switch would also mean the company couldn't leverage prior investments.

"I have all this commodity hardware that I can just move over to the new warehouse, so why wouldn't I do that?" he says.

In the never-ending challenge of analyzing more data and doing it more quickly, Smith says Sybase IQ has always provided what he describes as an extensible model.

"We don't want to design into a box, so I'm always interested in where Sybase is planning to go and how I'm going to leverage the platform."

Incremental investment, the ability to leverage existing hardware, extensibility when it comes to processing power and storage: these aren't capabilities that are exclusive to Sybase IQ, but not all data warehousing approaches and technologies give you this flexibility. Thus, take a cue from comScore's experience and explore these system capabilities in detail in your next data warehousing RFP.

Previous
2 of 2
Next
Comment  | 
Print  | 
More Insights
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest September 18, 2014
Enterprise social network success starts and ends with integration. Here's how to finally make collaboration click.
Flash Poll
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
The weekly wrap-up of the top stories from InformationWeek.com this week.
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.