Real-Time Ambition: Reaching the Potential of Event Processing - InformationWeek
Software // Information Management
04:38 PM
Connect Directly

Real-Time Ambition: Reaching the Potential of Event Processing

Complex event processing (CEP) software delivers on the promise of real-time insight, but is the technology too green for mainstream success? CEP was once available only to big financial institutions and government agencies that could afford custom development projects. That's no longer the case, as off-the-shelf products and implementations have proliferated. Intelligent Enterprise shares success stories and explores the potential of CEP as your next competitive edge.

"Real-time" is now one of the buzz phrases that shows up all over the place, but if there's any technology that can truly deliver on these words, it's the emerging category of complex event processing (CEP). With so many organizations now using conventional business intelligence technologies and approaches, CEP offers a speed-to-insight advantage that may be the next competitive differentiator, but is the technology ready for mainstream adoption?

Developed in the computer science labs of Stanford, Caltech, M.I.T., Cambridge and other universities, CEP was first put into use on Wall Street and in super-secret intelligence applications. The technology spots patterns in complex, high-volume data while it's streaming through business systems and applications, rather than after it's stored and a matter of history. Conventional data warehouse technologies have come a long way in reducing time to analysis, thanks to techniques such as trickle feeding, but CEP latency is measured in milliseconds and the volumes of data processed can exceed 50,000 messages per second.

It's not every application that requires this kind of speed and processing power, but CEP is quickly moving out of limited, high-end applications. The technology is far from mature and major vendors are still waiting on the sidelines, but as you'll discover in this article, CEP is being productized, embedded in other applications and pioneered in more mainstream industries such as shipping. Open-source and "community edition" offerings have even made the technology affordable to small and midsized business seeking rapid detection of changes in customer behavior or system performance.

CEP's Starting Point

The sweet spot for CEP is applications involving high volumes of data, complex analyses and time sensitivity (as in millisecond or sub-milliseconds). Algorithmic trading in the financial services industry is a classic example in which all three requirements are present. Algorithms are basically used to search patterns in data indicating buying or selling opportunities. Wall Street firms have used trading algorithms for decades, but CEP puts the approach on steroids, applying thousands of algorithms simultaneously against tens of thousands of trading data points per second. With CEP, a single broker monitoring a dashboard can now achieve what used to require armies of traders and platoons of PhDs.

CEP applications were once rare and always custom-built, but the technology is now common on Wall Street, and it more often starts with off-the-shelf products. "Best practices are falling into place and costs are starting to come down," says Mary Knox, a Gartner analyst who serves the investment services industry. "We're also seeing CEP bundled with applications such as order management systems and market data distribution systems."

CEP has also been a boon to intelligence agencies, so much so that In-Q-Tel, the technology investment firm set up by the Central Intelligence Agency, has backed CEP vendor StreamBase. Intelligence and military applications include surveillance and alerting, monitoring network traffic and screening message queues or e-mail.

"Two big challenges are coping with the volume and velocity of information as it's streaming," says Troy M. Pearsall, executive vice president of technology initiatives at In-Q-Tel. "It's important to be able to examine the data and see the patterns as they're coming toward you rather than after the fact."

1 of 3
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
IT Strategies to Conquer the Cloud
Chances are your organization is adopting cloud computing in one way or another -- or in multiple ways. Understanding the skills you need and how cloud affects IT operations and networking will help you adapt.
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on for the week of November 6, 2016. We'll be talking with the editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll