CEP + BI = Real-Time Event Analytics

Event-driven analytics aims to facilitate business decisions and actions as opportunities (and threats) emerge. Move the concept into the Now, into the worlds of (for instance) on-line commerce and real-time detection of suspect financial transactions, and it becomes apparent that old, DBMS-reliant architectures, with their load-index-query overhead, just aren't fast enough. Complex event processing (CEP) technology does deliver required sub-second responses, combining real-time BI capabilities -- what my twitter friend @communicating describes as "the act of identifying 'pain' & 'profit' signals as they happen" -- with the ability to tap both DBMS-stored historical data and real-time "data in flight." Yet the mainstream BI world doesn't have a clear picture of the possibilities afforded by CEP-enabled event-driven analytics, so I set out to study perceptions with the support of one of the space's leading vendors.Why CEP?

The problem is that CEP is mature technology that has not lived up to its potential. It sprang from '90s academic and industrial research projects. The rationale? A 2003 paper written by the principals of the Brown/Brandeis/MIT Aurora project explains it well: "The fact that a software system must process and react to continual inputs from many sources (e.g., sensors) rather than from human operators requires one to rethink the fundamental architecture of a DBMS for this application area."

Informatica's very recent acquisition of CEP vendor Agent Logic, as well as the continuous transformation-type messaging of CEP vendors SQLstream and Truviso, testifies that this statement can and should be applied to data-integration as well as to DBMSes.

Breaking out of market niches

CEP has won a strong reputation in important niche applications such as communications network monitoring and automated, algorithmic trading but broader, enterprise up-take has been slow. StreamBase has notably found it difficult to break out of capital (financial-trading) markets, and SeeWhy narrowed its own focus to Web-site session abandonment. TIBCO, Progress Software, and Aleri are leading vendors that have made significant efforts to break into the mainstream BI market.

I'll cite Forrester analysts Mike Gualtieri and John R. Rymeras as other authorities who see vendor-driven breakout potential. They write in a recent Forrester Wave report, "Complex Event Processing (CEP) Platforms, Q3 2009" --

Two webinars and a study

Progress and Aleri recently sponsored Intelligent Enterprise/TechWeb webinars -- Progress's featured IE Editor Doug Henschen and I presented findings from my Aleri-sponsored market-perception study during Aleri's.

The first thing I determined from study (which was vendor neutral, by the way) is that the 48 valid respondents seemed a fair sampling of the universe of BI users. Responses to the question, "Where does your organization apply data analysis?" were consistent with broad application of analytics within the enterprise, across many disparate business functions, that reflects today's enterprise analytics world.

Top responses to "What sources do your data analyses draw from?" were --

-- and respondents reported using --

-- again surely consistent with the enterprise analytics experience. Yet a significant minority of respondents indicated use of data sources for analyses that challenge mainstream BI and data warehousing tools --

-- which explains, at least in part, responses to the question, "How could your data-analysis software be improved?" such as

These latter areas -- data-analysis software gaps -- are in CEP's sweet spot, yet almost 50% of respondents had not heard of Complex Event Processing. This figure shows that vendor and thought-leader education efforts have made progress but that a lot more work is needed. (That figure also shows that my survey successfully reached outside a core CEP audience in eliciting survey participation.)

Business value in real-time, continuous analysis

The figures that 34% of respondents believe that a "CEP or Continuous Intelligence solution would improve [their] organization's analytical capabilities" added to the 38% who responded "might improve" -- as against only 10% who responded "unlikely to improve" -- do indicate the positive direction experiences and perceptions of CEP and Continuous Intelligence (Aleri's trademarked term for CEP-extended BI) are taking.

Responses to the question, "What business value can you see in real-time, continuous analysis?" demonstrate not only that current and potential users see value in "faster decisions" (68%) and "more accurate and timely information" (61%) but also that these capabilities are directly linked to concrete business measures that affect the profitability such as "higher customer satisfaction" (42%), "lower personnel and operational costs" (32%), and "increased sales" (26%).

Responses to "If you implement real-time or continuous analysis, how would information be delivered to business people?" communicate three interesting points. First, respondents expect to continue using interfaces users are already familiar with from the BI world such as dashboards (74%), reports (48%), and ad-hoc queries (48%). Secondly, respondents expect to bring on-line near or enhanced communications: "alerts (e-mail, phone, other messaging)" (68%). Lastly, respondents see data warehouses and data marts, even if enhanced, and spreadsheets, as not so promising for delivery of real-time or continuous analysis, with only 29% and 13% of responses, respectively.

Conversely, "enhanced BI tools" (58%), "OLAP cubes refreshed with live, real-time data" (35%), and faster data warehouse loading (32%) are seen as possibly drivers for real-time analysis. Only 23% of respondents thought existing BI tools would be up to the job.

Findings in context

The findings I have reported here are presented in a richer, narrative context in my TechWeb/IE webinar, which was recorded and is available on-line. You can also download my slides, if you wish, from the webinar interface. My goal was to help folks in the BI world understand how CEP -- event-driven, real-time BI using streaming and database sources -- can benefit them. I hope I've succeeded!Event-driven analytics aims to facilitate business decisions and actions as opportunities (and threats) emerge. Move the concept into the Now, into the worlds of on-line commerce and real-time fraud detection, and old, DBMS-reliant architectures just aren't fast enough. Complex event processing (CEP) added to BI promises to close the speed gap, to enable real-time event analytics...