Real-Time Ambition: Reaching the Potential of Event Processing - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Software // Information Management
News
6/10/2007
04:38 PM
Connect Directly
Google+
LinkedIn
Twitter
RSS
E-Mail
50%
50%

Real-Time Ambition: Reaching the Potential of Event Processing

Complex event processing (CEP) software delivers on the promise of real-time insight, but is the technology too green for mainstream success? CEP was once available only to big financial institutions and government agencies that could afford custom development projects. That's no longer the case, as off-the-shelf products and implementations have proliferated. Intelligent Enterprise shares success stories and explores the potential of CEP as your next competitive edge.

"Real-time" is now one of the buzz phrases that shows up all over the place, but if there's any technology that can truly deliver on these words, it's the emerging category of complex event processing (CEP). With so many organizations now using conventional business intelligence technologies and approaches, CEP offers a speed-to-insight advantage that may be the next competitive differentiator, but is the technology ready for mainstream adoption?

Developed in the computer science labs of Stanford, Caltech, M.I.T., Cambridge and other universities, CEP was first put into use on Wall Street and in super-secret intelligence applications. The technology spots patterns in complex, high-volume data while it's streaming through business systems and applications, rather than after it's stored and a matter of history. Conventional data warehouse technologies have come a long way in reducing time to analysis, thanks to techniques such as trickle feeding, but CEP latency is measured in milliseconds and the volumes of data processed can exceed 50,000 messages per second.

It's not every application that requires this kind of speed and processing power, but CEP is quickly moving out of limited, high-end applications. The technology is far from mature and major vendors are still waiting on the sidelines, but as you'll discover in this article, CEP is being productized, embedded in other applications and pioneered in more mainstream industries such as shipping. Open-source and "community edition" offerings have even made the technology affordable to small and midsized business seeking rapid detection of changes in customer behavior or system performance.

CEP's Starting Point

The sweet spot for CEP is applications involving high volumes of data, complex analyses and time sensitivity (as in millisecond or sub-milliseconds). Algorithmic trading in the financial services industry is a classic example in which all three requirements are present. Algorithms are basically used to search patterns in data indicating buying or selling opportunities. Wall Street firms have used trading algorithms for decades, but CEP puts the approach on steroids, applying thousands of algorithms simultaneously against tens of thousands of trading data points per second. With CEP, a single broker monitoring a dashboard can now achieve what used to require armies of traders and platoons of PhDs.

CEP applications were once rare and always custom-built, but the technology is now common on Wall Street, and it more often starts with off-the-shelf products. "Best practices are falling into place and costs are starting to come down," says Mary Knox, a Gartner analyst who serves the investment services industry. "We're also seeing CEP bundled with applications such as order management systems and market data distribution systems."

CEP has also been a boon to intelligence agencies, so much so that In-Q-Tel, the technology investment firm set up by the Central Intelligence Agency, has backed CEP vendor StreamBase. Intelligence and military applications include surveillance and alerting, monitoring network traffic and screening message queues or e-mail.

"Two big challenges are coping with the volume and velocity of information as it's streaming," says Troy M. Pearsall, executive vice president of technology initiatives at In-Q-Tel. "It's important to be able to examine the data and see the patterns as they're coming toward you rather than after the fact."

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Previous
1 of 3
Next
Comment  | 
Print  | 
More Insights
Commentary
AI Regulation: Has the Time Arrived?
John Edwards, Technology Journalist & Author,  2/24/2020
News
Fighting the Coronavirus with Analytics and GIS
Jessica Davis, Senior Editor, Enterprise Apps,  2/3/2020
Slideshows
IT Careers: 10 Job Skills in High Demand This Year
Cynthia Harvey, Freelance Journalist, InformationWeek,  2/3/2020
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
IT Careers: Tech Drives Constant Change
Advances in information technology and management concepts mean that IT professionals must update their skill sets, even their career goals on an almost yearly basis. In this IT Trend Report, experts share advice on how IT pros can keep up with this every-changing job market. Read it today!
Slideshows
Flash Poll