ComScore's Big Data Deployment In Detail

"If I had gone with the big Unix-box approach, at some point there would be a major cost," he explains. "We went with the commodity approach because we like buying things with five zeros rather than six zeros."

Compression is a key advantage for column-store databases. comScore gets anywhere between 25% and 40% compression, depending on the type of data being analyzed, according to Smith. In terms of performance, increases in processing power and data input/output speed and throughput have brought steady improvements. As an example, an analysis of Internet search engine traffic that took five hours on comScore's last Sybase IQ deployment now takes 11 minutes in the new deployment, Smith says.

Sybase IQ runs on symmetric multiprocessor (SMP) hardware, a contrast with the many appliances and databases that now run on massively parallel processing (MPP) architectures. MPP architectures have advantages when it comes to scalability, but Smith says Sybase IQ's Multiplex (grid scaling) approach has advantages in terms of flexibility.

"I can do parallelized loads and the things that MPP is good at, but I also have the throttle of being able to do multiple tasks at the same time with Multiplex," he says, contrasting the multiple, independent database nodes of the IQ grid with the single-database-controller/worker-node approach of MPP.

Smith says comScore has looked at MPP alternatives from the likes of Vertica and ParAccel, column-store competitors to Sybase IQ, but a switch would also mean the company couldn't leverage prior investments.

"I have all this commodity hardware that I can just move over to the new warehouse, so why wouldn't I do that?" he says.

In the never-ending challenge of analyzing more data and doing it more quickly, Smith says Sybase IQ has always provided what he describes as an extensible model.

"We don't want to design into a box, so I'm always interested in where Sybase is planning to go and how I'm going to leverage the platform."

Incremental investment, the ability to leverage existing hardware, extensibility when it comes to processing power and storage: these aren't capabilities that are exclusive to Sybase IQ, but not all data warehousing approaches and technologies give you this flexibility. Thus, take a cue from comScore's experience and explore these system capabilities in detail in your next data warehousing RFP.