Tackling Real-Time Data Analysis

Two startup companies with roots in academia have developed software that monitors and analyzes high-volume streams of real-time data, an increasingly important yet difficult goal for many businesses.

Business intelligence today is largely an offline affair. Data such as financial or sales information is pulled from operational systems and loaded into a data warehouse for analysis. But that's too slow for an increasing number of applications where data must be analyzed in real time, such as in equities trading and supply-chain management.

RiverGlass Inc. has developed software that merges data from multiple disparate data streams, including unstructured text and numeric data, and applies real-time data-modeling and -analysis techniques to those streams. The goal is to detect patterns in the data to identify potential investment risks and opportunities, for example.

"The flow of this kind of information is increasing far, far faster than people can analyze with traditional tools," says Brian Buck, chief technology officer at RiverGlass.

The RiverGlass system uses real-time text and data-analysis technology developed at the National Center for Supercomputing Applications, a University of Il-linois-based lab that developed the Mosaic Web browser. RiverGlass' technology is nearly complete and will be generally available midyear. RiverGlass says a major technology vendor is trying out RiverGlass' software for network- intrusion applications while a state police agency is experimenting with it to analyze streaming text data.

The other startup, StreamBase Systems Inc., has developed an extended version of SQL, the structured query language for developing database queries, and a special query engine that analyzes data as it streams by. The genesis of the software was the Aurora project conducted at Brandeis University, Brown University, and MIT.

The main brain behind StreamBase is Michael Stonebraker, an MIT adjunct professor who was a pioneer in relational and object-oriented database technology and created the Ingres and Postgres databases. Stonebraker also served as CTO for several years at database vendor Informix Software Inc. after it acquired Illustra in 1996, another Stonebraker startup.

Though StreamBase's first products began shipping last year, last week marked the formal debut of the company and its StreamBase Stream Processing Engine (with a typical starting price of $60,000 per year) at Demo@15, an annual conference on technology innovation held in Scottsdale, Ariz. About a dozen customers have StreamBase pilot projects in the works, CEO Barry Morris says.

Archipelago Holdings Inc., which operates the Archipelago all-electronic stock exchange, is among StreamBase's early adopters. Archipelago developed a prototype application late last year based on StreamBase that monitors the response time between every incoming trading order and every booking confirmation that goes out in reply--as many as 20,000 messages per second.

The goal is to reduce the latency time between incoming and outgoing messages, as even tiny delays are critical for program traders, says Steve Rubinow, Archipelago's CTO. "In our business, a few milliseconds can make the difference between people making money and not making money," he says. The trading company plans to have the application in full production next month.

Both companies say their technology can be used to monitor network traffic to detect intrusions, for law-enforcement and anti-terrorism applications, for industrial control tasks, and for monitoring and analyzing the huge volumes of data expected to be generated by radio-frequency identification tags in supply-chain-management systems.