Real-Time Ambition: Reaching the Potential of Event Processing - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Software // Information Management
04:38 PM
Connect Directly

Real-Time Ambition: Reaching the Potential of Event Processing

Complex event processing (CEP) software delivers on the promise of real-time insight, but is the technology too green for mainstream success? CEP was once available only to big financial institutions and government agencies that could afford custom development projects. That's no longer the case, as off-the-shelf products and implementations have proliferated. Intelligent Enterprise shares success stories and explores the potential of CEP as your next competitive edge.

"Real-time" is now one of the buzz phrases that shows up all over the place, but if there's any technology that can truly deliver on these words, it's the emerging category of complex event processing (CEP). With so many organizations now using conventional business intelligence technologies and approaches, CEP offers a speed-to-insight advantage that may be the next competitive differentiator, but is the technology ready for mainstream adoption?

Developed in the computer science labs of Stanford, Caltech, M.I.T., Cambridge and other universities, CEP was first put into use on Wall Street and in super-secret intelligence applications. The technology spots patterns in complex, high-volume data while it's streaming through business systems and applications, rather than after it's stored and a matter of history. Conventional data warehouse technologies have come a long way in reducing time to analysis, thanks to techniques such as trickle feeding, but CEP latency is measured in milliseconds and the volumes of data processed can exceed 50,000 messages per second.

It's not every application that requires this kind of speed and processing power, but CEP is quickly moving out of limited, high-end applications. The technology is far from mature and major vendors are still waiting on the sidelines, but as you'll discover in this article, CEP is being productized, embedded in other applications and pioneered in more mainstream industries such as shipping. Open-source and "community edition" offerings have even made the technology affordable to small and midsized business seeking rapid detection of changes in customer behavior or system performance.

CEP's Starting Point

The sweet spot for CEP is applications involving high volumes of data, complex analyses and time sensitivity (as in millisecond or sub-milliseconds). Algorithmic trading in the financial services industry is a classic example in which all three requirements are present. Algorithms are basically used to search patterns in data indicating buying or selling opportunities. Wall Street firms have used trading algorithms for decades, but CEP puts the approach on steroids, applying thousands of algorithms simultaneously against tens of thousands of trading data points per second. With CEP, a single broker monitoring a dashboard can now achieve what used to require armies of traders and platoons of PhDs.

CEP applications were once rare and always custom-built, but the technology is now common on Wall Street, and it more often starts with off-the-shelf products. "Best practices are falling into place and costs are starting to come down," says Mary Knox, a Gartner analyst who serves the investment services industry. "We're also seeing CEP bundled with applications such as order management systems and market data distribution systems."

CEP has also been a boon to intelligence agencies, so much so that In-Q-Tel, the technology investment firm set up by the Central Intelligence Agency, has backed CEP vendor StreamBase. Intelligence and military applications include surveillance and alerting, monitoring network traffic and screening message queues or e-mail.

"Two big challenges are coping with the volume and velocity of information as it's streaming," says Troy M. Pearsall, executive vice president of technology initiatives at In-Q-Tel. "It's important to be able to examine the data and see the patterns as they're coming toward you rather than after the fact."

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
1 of 3
Comment  | 
Print  | 
More Insights
10 Ways to Transition Traditional IT Talent to Cloud Talent
Lisa Morgan, Freelance Writer,  11/23/2020
What Comes Next for the COVID-19 Computing Consortium
Joao-Pierre S. Ruth, Senior Writer,  11/24/2020
Top 10 Data and Analytics Trends for 2021
Jessica Davis, Senior Editor, Enterprise Apps,  11/13/2020
White Papers
Register for InformationWeek Newsletters
Current Issue
Why Chatbots Are So Popular Right Now
In this IT Trend Report, you will learn more about why chatbots are gaining traction within businesses, particularly while a pandemic is impacting the world.
Flash Poll