The problem is that CEP is mature technology that has not lived up to its potential. It sprang from '90s academic and industrial research projects. The rationale? A 2003 paper written by the principals of the Brown/Brandeis/MIT Aurora project explains it well: "The fact that a software system must process and react to continual inputs from many sources (e.g., sensors) rather than from human operators requires one to rethink the fundamental architecture of a DBMS for this application area."
Informatica's very recent acquisition of CEP vendor Agent Logic, as well as the continuous transformation-type messaging of CEP vendors SQLstream and Truviso, testifies that this statement can and should be applied to data-integration as well as to DBMSes.
Breaking out of market niches
CEP has won a strong reputation in important niche applications such as communications network monitoring and automated, algorithmic trading but broader, enterprise up-take has been slow. StreamBase has notably found it difficult to break out of capital (financial-trading) markets, and SeeWhy narrowed its own focus to Web-site session abandonment. TIBCO, Progress Software, and Aleri are leading vendors that have made significant efforts to break into the mainstream BI market.
I'll cite Forrester analysts Mike Gualtieri and John R. Rymeras as other authorities who see vendor-driven breakout potential. They write in a recent Forrester Wave report, "Complex Event Processing (CEP) Platforms, Q3 2009" --
"Although event processing features and high throughput are still extremely important to CEP platform customers, the number of vendors that have built advanced tools for both developers and business end users is a sign that the market is maturing. Many of the vendors are broadening their sales and marketing efforts beyond capital market applications, the first big market for CEP platforms."
Two webinars and a study
Progress and Aleri recently sponsored Intelligent Enterprise/TechWeb webinars -- Progress's featured IE Editor Doug Henschen and I presented findings from my Aleri-sponsored market-perception study during Aleri's.
The first thing I determined from study (which was vendor neutral, by the way) is that the 48 valid respondents seemed a fair sampling of the universe of BI users. Responses to the question, "Where does your organization apply data analysis?" were consistent with broad application of analytics within the enterprise, across many disparate business functions, that reflects today's enterprise analytics world.
- Operational or transactional system, 72.7%
- Data warehouse or data mart, 68.2%
- Spreadsheets, 45.5%
- Reporting, 66.7%
- Dashboards, 59.0%
- Excel or another spreadsheet, 59.0%
- OLAP or a pivot-table interface, 51.3%
- Data or event streams, 36.4%
- Messaging system/bus, 20.5%
- Content feed (e.g., via RSS or Atom), 15.9%
- Better integrated with line-of-business/operational systems, 51.3%
- Handle real-time data, 41.0%
- Provider faster, real-time response, 41.0%
- Access additional data sources, 28.2%
- Handle data and message streams, 20.5%
These latter areas -- data-analysis software gaps -- are in CEP's sweet spot, yet almost 50% of respondents had not heard of Complex Event Processing. This figure shows that vendor and thought-leader education efforts have made progress but that a lot more work is needed. (That figure also shows that my survey successfully reached outside a core CEP audience in eliciting survey participation.)
Business value in real-time, continuous analysis
The figures that 34% of respondents believe that a "CEP or Continuous Intelligence solution would improve [their] organization's analytical capabilities" added to the 38% who responded "might improve" -- as against only 10% who responded "unlikely to improve" -- do indicate the positive direction experiences and perceptions of CEP and Continuous Intelligence (Aleri's trademarked term for CEP-extended BI) are taking.
Responses to the question, "What business value can you see in real-time, continuous analysis?" demonstrate not only that current and potential users see value in "faster decisions" (68%) and "more accurate and timely information" (61%) but also that these capabilities are directly linked to concrete business measures that affect the profitability such as "higher customer satisfaction" (42%), "lower personnel and operational costs" (32%), and "increased sales" (26%).
Responses to "If you implement real-time or continuous analysis, how would information be delivered to business people?" communicate three interesting points. First, respondents expect to continue using interfaces users are already familiar with from the BI world such as dashboards (74%), reports (48%), and ad-hoc queries (48%). Secondly, respondents expect to bring on-line near or enhanced communications: "alerts (e-mail, phone, other messaging)" (68%). Lastly, respondents see data warehouses and data marts, even if enhanced, and spreadsheets, as not so promising for delivery of real-time or continuous analysis, with only 29% and 13% of responses, respectively.
Conversely, "enhanced BI tools" (58%), "OLAP cubes refreshed with live, real-time data" (35%), and faster data warehouse loading (32%) are seen as possibly drivers for real-time analysis. Only 23% of respondents thought existing BI tools would be up to the job.
Findings in context
The findings I have reported here are presented in a richer, narrative context in my TechWeb/IE webinar, which was recorded and is available on-line. You can also download my slides, if you wish, from the webinar interface. My goal was to help folks in the BI world understand how CEP -- event-driven, real-time BI using streaming and database sources -- can benefit them. I hope I've succeeded!Event-driven analytics aims to facilitate business decisions and actions as opportunities (and threats) emerge. Move the concept into the Now, into the worlds of on-line commerce and real-time fraud detection, and old, DBMS-reliant architectures just aren't fast enough. Complex event processing (CEP) added to BI promises to close the speed gap, to enable real-time event analytics...