Software // Information Management
Commentary
9/4/2009
09:20 AM
Seth Grimes
Seth Grimes
Commentary
Connect Directly
Twitter
RSS
E-Mail
50%
50%

CEP + BI = Real-Time Event Analytics

Event-driven analytics aims to facilitate business decisions and actions as opportunities (and threats) emerge. Move the concept into the Now, into the worlds of on-line commerce and real-time fraud detection, and old, DBMS-reliant architectures just aren't fast enough. Complex event processing (CEP) added to BI promises to close the speed gap, to enable real-time event analytics...

Event-driven analytics aims to facilitate business decisions and actions as opportunities (and threats) emerge. Move the concept into the Now, into the worlds of (for instance) on-line commerce and real-time detection of suspect financial transactions, and it becomes apparent that old, DBMS-reliant architectures, with their load-index-query overhead, just aren't fast enough. Complex event processing (CEP) technology does deliver required sub-second responses, combining real-time BI capabilities -- what my twitter friend @communicating describes as "the act of identifying 'pain' & 'profit' signals as they happen" -- with the ability to tap both DBMS-stored historical data and real-time "data in flight." Yet the mainstream BI world doesn't have a clear picture of the possibilities afforded by CEP-enabled event-driven analytics, so I set out to study perceptions with the support of one of the space's leading vendors.Why CEP?

The problem is that CEP is mature technology that has not lived up to its potential. It sprang from '90s academic and industrial research projects. The rationale? A 2003 paper written by the principals of the Brown/Brandeis/MIT Aurora project explains it well: "The fact that a software system must process and react to continual inputs from many sources (e.g., sensors) rather than from human operators requires one to rethink the fundamental architecture of a DBMS for this application area."

Informatica's very recent acquisition of CEP vendor Agent Logic, as well as the continuous transformation-type messaging of CEP vendors SQLstream and Truviso, testifies that this statement can and should be applied to data-integration as well as to DBMSes.

Breaking out of market niches

CEP has won a strong reputation in important niche applications such as communications network monitoring and automated, algorithmic trading but broader, enterprise up-take has been slow. StreamBase has notably found it difficult to break out of capital (financial-trading) markets, and SeeWhy narrowed its own focus to Web-site session abandonment. TIBCO, Progress Software, and Aleri are leading vendors that have made significant efforts to break into the mainstream BI market.

I'll cite Forrester analysts Mike Gualtieri and John R. Rymeras as other authorities who see vendor-driven breakout potential. They write in a recent Forrester Wave report, "Complex Event Processing (CEP) Platforms, Q3 2009" --

"Although event processing features and high throughput are still extremely important to CEP platform customers, the number of vendors that have built advanced tools for both developers and business end users is a sign that the market is maturing. Many of the vendors are broadening their sales and marketing efforts beyond capital market applications, the first big market for CEP platforms."

Two webinars and a study

Progress and Aleri recently sponsored Intelligent Enterprise/TechWeb webinars -- Progress's featured IE Editor Doug Henschen and I presented findings from my Aleri-sponsored market-perception study during Aleri's.

The first thing I determined from study (which was vendor neutral, by the way) is that the 48 valid respondents seemed a fair sampling of the universe of BI users. Responses to the question, "Where does your organization apply data analysis?" were consistent with broad application of analytics within the enterprise, across many disparate business functions, that reflects today's enterprise analytics world.

Chart: Where does your organization apply data analysis?
Top responses to "What sources do your data analyses draw from?" were --
  1. Operational or transactional system, 72.7%
  2. Data warehouse or data mart, 68.2%
  3. Spreadsheets, 45.5%
-- and respondents reported using --
  1. Reporting, 66.7%
  2. Dashboards, 59.0%
  3. Excel or another spreadsheet, 59.0%
  4. OLAP or a pivot-table interface, 51.3%
-- again surely consistent with the enterprise analytics experience. Yet a significant minority of respondents indicated use of data sources for analyses that challenge mainstream BI and data warehousing tools --
  1. Data or event streams, 36.4%
  2. Messaging system/bus, 20.5%
  3. Content feed (e.g., via RSS or Atom), 15.9%
-- which explains, at least in part, responses to the question, "How could your data-analysis software be improved?" such as
  1. Better integrated with line-of-business/operational systems, 51.3%
  2. Handle real-time data, 41.0%
  3. Provider faster, real-time response, 41.0%
  4. Access additional data sources, 28.2%
  5. Handle data and message streams, 20.5%
Chart: How could your data analysis software be improved?

These latter areas -- data-analysis software gaps -- are in CEP's sweet spot, yet almost 50% of respondents had not heard of Complex Event Processing. This figure shows that vendor and thought-leader education efforts have made progress but that a lot more work is needed. (That figure also shows that my survey successfully reached outside a core CEP audience in eliciting survey participation.)

Business value in real-time, continuous analysis

The figures that 34% of respondents believe that a "CEP or Continuous Intelligence solution would improve [their] organization's analytical capabilities" added to the 38% who responded "might improve" -- as against only 10% who responded "unlikely to improve" -- do indicate the positive direction experiences and perceptions of CEP and Continuous Intelligence (Aleri's trademarked term for CEP-extended BI) are taking.

Responses to the question, "What business value can you see in real-time, continuous analysis?" demonstrate not only that current and potential users see value in "faster decisions" (68%) and "more accurate and timely information" (61%) but also that these capabilities are directly linked to concrete business measures that affect the profitability such as "higher customer satisfaction" (42%), "lower personnel and operational costs" (32%), and "increased sales" (26%).

Chart: What business value can you see in real-time, continuous analysis?

Responses to "If you implement real-time or continuous analysis, how would information be delivered to business people?" communicate three interesting points. First, respondents expect to continue using interfaces users are already familiar with from the BI world such as dashboards (74%), reports (48%), and ad-hoc queries (48%). Secondly, respondents expect to bring on-line near or enhanced communications: "alerts (e-mail, phone, other messaging)" (68%). Lastly, respondents see data warehouses and data marts, even if enhanced, and spreadsheets, as not so promising for delivery of real-time or continuous analysis, with only 29% and 13% of responses, respectively.

Conversely, "enhanced BI tools" (58%), "OLAP cubes refreshed with live, real-time data" (35%), and faster data warehouse loading (32%) are seen as possibly drivers for real-time analysis. Only 23% of respondents thought existing BI tools would be up to the job.

Findings in context

The findings I have reported here are presented in a richer, narrative context in my TechWeb/IE webinar, which was recorded and is available on-line. You can also download my slides, if you wish, from the webinar interface. My goal was to help folks in the BI world understand how CEP -- event-driven, real-time BI using streaming and database sources -- can benefit them. I hope I've succeeded!Event-driven analytics aims to facilitate business decisions and actions as opportunities (and threats) emerge. Move the concept into the Now, into the worlds of on-line commerce and real-time fraud detection, and old, DBMS-reliant architectures just aren't fast enough. Complex event processing (CEP) added to BI promises to close the speed gap, to enable real-time event analytics...

Comment  | 
Print  | 
More Insights
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest, Dec. 9, 2014
Apps will make or break the tablet as a work device, but don't shortchange critical factors related to hardware, security, peripherals, and integration.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of December 14, 2014. Be here for the show and for the incredible Friday Afternoon Conversation that runs beside the program.
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.