Startups Tackle Real-Time Data Analysis

Two startup companies with roots in academia have developed software that monitors and analyzes high-volume streams of real-time data, an increasingly important yet difficult goal for many businesses.

Business intelligence today is largely an offline affair. Data such as financial or sales information is pulled from operational systems and loaded into a data warehouse for analysis. But that's too slow for an increasing number of applications where data must be analyzed in real time, such as in equities trading and supply-chain management.

RiverGlass Inc. has developed software that merges data from multiple, disparate data streams--including unstructured text and numeric data--and applies real-time data modeling and analysis techniques to those streams. The goal is to detect patterns in the data to identify potential investment risks and opportunities, for example.

"The flow of this kind of information is increasing far, far faster than people can analyze with traditional tools," says Brian Buck, RiverGlass' chief technology officer.

The RiverGlass system uses real-time text and data- analysis technology developed at the National Center for Supercomputing Applications, a University of Illinois-based lab that developed the Mosaic Web browser. RiverGlass' technology is nearly complete and will be generally available midyear.

The other startup, StreamBase Systems Inc., has developed an extended version of SQL, the structured query language for developing database queries, and a special query engine that analyzes data as it streams by. The genesis of the software was the Aurora project conducted at Brandeis University, Brown University, and MIT.

The main brain behind StreamBase is Michael Stonebraker, an MIT professor who was a pioneer of relational and object-oriented database technology, and created the Ingres and Postgres databases, while he was a professor at University of California at Berkeley in the 1980s and '90s.

Though StreamBase was founded in 2003 and its first products began shipping last year, this week marks the formal debut of the company and its Streambase Stream Processing Engine (with a typical starting price of $60,000 per year) at Demo@15, an annual conference that focuses on technology innovation that's taking place this week in Scottsdale, Ariz. About a dozen customers have pilot projects in the works using the StreamBase technology, CEO Barry Morris says.

Archipelago Holdings Inc., which operates the Archipelago all-electronic stock exchange, is among StreamBase's early adopters. Archipelago developed a prototype application late last year based on Streambase that monitors the response time between every trading order that comes into the company and every booking confirmation that goes out in reply--as many as 20,000 messages per second.

The goal is to reduce the latency time between incoming and outgoing messages as even tiny delays are critical for program traders, says Steve Rubinow, Archipelago's CTO. "In our business, a few milliseconds can make the difference between people making money and not making money," he says. The trading company plans to have the application in full production next month.

Both companies say their technology can be used to monitor network traffic to detect intrusions; for law-enforcement and anti-terrorism applications; for industrial control tasks; and for monitoring and analyzing the huge volumes of data expected to be generated by radio-frequency identification tags in supply-chain-management systems.

While RiverGlass and StreamBase would seem to be competitors at first glance, executives at each company say they are taking different technology and business approaches to the challenge of analyzing real-time data. StreamBase is selling its technology as a server/data-stream processing engine and development tools, for example, while RiverGlass is developing industry and role-specific applications based on its technology. RiverGlass' Buck says the two companies' products might even be complementary in some situations.