Ingres Unveils VectorWise Database Engine

Ingres has launched a new kind of analytics database system, Ingres VectorWise, that it says cuts in half the time needed to process complex queries. It does so by understanding and exploiting the capabilities of the latest generation of computer chips in a manner similar to the game software. If Ingres' claims are borne out, then the data warehousing and analytics market, which includes several tightly balanced database machines, may be about to turn in a new direction.

"It's done without the mind-numbing complexity previously required for business intelligence systems," said Roger Burkhardt in an interview in advance of today's announcement. "We've been getting extraordinary results."

Burkhardt said the database system code is new, but customers do not have to change their existing relational business intelligence queries and SQL statements. They will work as is on VectorWise.

Perhaps most significant, it takes advantage of the parallel processing capabilities built into the latest generation of commodity, x86 architecture chips. "Intel is spending $6 billion a year on research and development. The database world hasn't taken advantage of that," Burkhardt said.

VectorWise taps the chips' ability to process an instruction along with multiple streams of data, called Single Instruction, Multiple Data (SIMD), and Streaming SIMD Extensions, which allows the processing of SIMD-type instructions with greater parallelism, according to an Ingres/Intel joint white paper. In doing so, the new system, can process 1,000 rows at a time where an OLTP database would process one.

Game machines take advantage of this array processing and parallel processing in order to show the complex backgrounds and vivid scene interactions that is their lifeblood. But relational databases that were borne in the late 1970s and early 1980s haven't previously taken advantage of the capability, Burkhardt said.

VectorWise was co-founded by Peter Boncz, a researcher at the Amsterdam-based CWI national computer science institute of the Netherlands. Ingres acquired all commercial rights to the technology two years ago and Ingres and VectorWise together continue research into the technology. The CWI institute in the past has produced the Python language and MonetDB, one of the first column-oriented database systems.

"The VectorWise technology is unique in the database area in fully exploiting modern chip features… something usually only achieved in video-editing software or games," said Marcin Zukowski , CEO of VectorWise and a former researcher at CWI (Centrum voor Wiskunde en Informatica). Other features of VectorWise help it tackle analytical tasks. It functions like a column-oriented database rather than a transaction processing relational system. An analysis query frequently deals with a large amount of data, all found in the same column, and a column-oriented system retrieves it faster. VectorWise uses compression, which also speeds up the process. Compression is particularly adapted to the uniform data types found in database columns; once a compression scheme is constructed for the type, it works throughout the column, while compression in a transaction processing system has to span the many different data types found in a row.

VectorWise works across a server cluster, while relational systems tend to be most efficient if they move to a larger single server when it's time to scale up. Burkhardt said VectorWise, borrowing a phrase from the NoSQL movement, is able to "scale out" by adding servers. Burkhardt calls the capability "my YesSQL movement." Burkhardt says VectorWise is now a column-oriented system sitting atop a standard row-oriented relational Ingres database.

Today's eight megabytes of cache on a modern CPU, a small fraction of the memory available on a server, exceeds the total memory available to the first relational database systems when they came out. Multiple CPU cores are available per CPU, and each core can process two threads at a time in Intel's latest Nehalem designs. These chip features are taken advantage of as well.

Warren Master, CTO at the Rohatyn Group, an asset management company in New York, is an early user of VectorWise and said in the announcement that his firm had been searching for a high performance analytics system.

"VectorWise lets us plow through millions and millions of rows of data with seemingly infinite width and depth without the need for new, expensive hardware, complicated schemas, explicit indexing, pre-aggregation, or specially hand crafted, DBA-tuned SQL," he said.

Burkhardt said VectorWise can yield 70X faster query results on a simple queries than standard relational systems.

"The young folks in NoSQL will like the no knobs, just get the job done quickly" results of VectorWise, he claimed.