ComScore's Big Data Deployment In Detail

Crunching tens of terabytes, a leading digital marketing measurement firm delivers insight -- quickly.

Doug Henschen, Executive Editor, Enterprise Apps

November 24, 2010

2 Min Read
InformationWeek logo in a gray background | InformationWeek

"If I had gone with the big Unix-box approach, at some point there would be a major cost," he explains. "We went with the commodity approach because we like buying things with five zeros rather than six zeros."

Compression is a key advantage for column-store databases. comScore gets anywhere between 25% and 40% compression, depending on the type of data being analyzed, according to Smith. In terms of performance, increases in processing power and data input/output speed and throughput have brought steady improvements. As an example, an analysis of Internet search engine traffic that took five hours on comScore's last Sybase IQ deployment now takes 11 minutes in the new deployment, Smith says.

Sybase IQ runs on symmetric multiprocessor (SMP) hardware, a contrast with the many appliances and databases that now run on massively parallel processing (MPP) architectures. MPP architectures have advantages when it comes to scalability, but Smith says Sybase IQ's Multiplex (grid scaling) approach has advantages in terms of flexibility.

"I can do parallelized loads and the things that MPP is good at, but I also have the throttle of being able to do multiple tasks at the same time with Multiplex," he says, contrasting the multiple, independent database nodes of the IQ grid with the single-database-controller/worker-node approach of MPP.

Smith says comScore has looked at MPP alternatives from the likes of Vertica and ParAccel, column-store competitors to Sybase IQ, but a switch would also mean the company couldn't leverage prior investments.

"I have all this commodity hardware that I can just move over to the new warehouse, so why wouldn't I do that?" he says.

In the never-ending challenge of analyzing more data and doing it more quickly, Smith says Sybase IQ has always provided what he describes as an extensible model.

"We don't want to design into a box, so I'm always interested in where Sybase is planning to go and how I'm going to leverage the platform."

Incremental investment, the ability to leverage existing hardware, extensibility when it comes to processing power and storage: these aren't capabilities that are exclusive to Sybase IQ, but not all data warehousing approaches and technologies give you this flexibility. Thus, take a cue from comScore's experience and explore these system capabilities in detail in your next data warehousing RFP.

About the Author

Doug Henschen

Executive Editor, Enterprise Apps

Doug Henschen is Executive Editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data and analytics. He previously served as editor in chief of Intelligent Enterprise, editor in chief of Transform Magazine, and Executive Editor at DM News. He has covered IT and data-driven marketing for more than 15 years.

Never Miss a Beat: Get a snapshot of the issues affecting the IT industry straight to your inbox.

You May Also Like


More Insights