Event-driven analytics aims to facilitate business decisions and actions as opportunities (and threats) emerge. Move the concept into the Now, into the worlds of on-line commerce and real-time fraud detection, and old, DBMS-reliant architectures just aren't fast enough. Complex event processing (CEP) added to BI promises to close the speed gap, to enable real-time event analytics...
Event-driven analytics aims to facilitate business decisions and actions as opportunities (and threats) emerge. Move the concept into the Now, into the worlds of (for instance) on-line commerce and real-time detection of suspect financial transactions, and it becomes apparent that old, DBMS-reliant architectures, with their load-index-query overhead, just aren't fast enough. Complex event processing (CEP) technology does deliver required sub-second responses, combining real-time BI capabilities -- what my twitter friend @communicating describes as "the act of identifying 'pain' & 'profit' signals as they happen" -- with the ability to tap both DBMS-stored historical data and real-time "data in flight." Yet the mainstream BI world doesn't have a clear picture of the possibilities afforded by CEP-enabled event-driven analytics, so I set out to study perceptions with the support of one of the space's leading vendors.Why CEP?
The problem is that CEP is mature technology that has not lived up to its potential. It sprang from '90s academic and industrial research projects. The rationale? A 2003 paper written by the principals of the Brown/Brandeis/MIT Aurora project explains it well: "The fact that a software system must process and react to continual inputs from many sources (e.g., sensors) rather than from human operators requires one to rethink the fundamental architecture of a DBMS for this application area."
CEP has won a strong reputation in important niche applications such as communications network monitoring and automated, algorithmic trading but broader, enterprise up-take has been slow. StreamBase has notably found it difficult to break out of capital (financial-trading) markets, and SeeWhy narrowed its own focus to Web-site session abandonment. TIBCO, Progress Software, and Aleri are leading vendors that have made significant efforts to break into the mainstream BI market.
"Although event processing features and high throughput are still extremely important to CEP platform customers, the number of vendors that have built advanced tools for both developers and business end users is a sign that the market is maturing. Many of the vendors are broadening their sales and marketing efforts beyond capital market applications, the first big market for CEP platforms."
Two webinars and a study
Progress and Aleri recently sponsored Intelligent Enterprise/TechWeb webinars -- Progress's featured IE Editor Doug Henschen and I presented findings from my Aleri-sponsored market-perception study during Aleri's.
The first thing I determined from study (which was vendor neutral, by the way) is that the 48 valid respondents seemed a fair sampling of the universe of BI users. Responses to the question, "Where does your organization apply data analysis?" were consistent with broad application of analytics within the enterprise, across many disparate business functions, that reflects today's enterprise analytics world.
Top responses to "What sources do your data analyses draw from?" were --
Operational or transactional system, 72.7%
Data warehouse or data mart, 68.2%
-- and respondents reported using --
Excel or another spreadsheet, 59.0%
OLAP or a pivot-table interface, 51.3%
-- again surely consistent with the enterprise analytics experience.
Yet a significant minority of respondents indicated use of data sources for analyses that challenge mainstream BI and data warehousing tools --
Data or event streams, 36.4%
Messaging system/bus, 20.5%
Content feed (e.g., via RSS or Atom), 15.9%
-- which explains, at least in part, responses to the question, "How could your data-analysis software be improved?" such as
Better integrated with line-of-business/operational systems, 51.3%
Handle real-time data, 41.0%
Provider faster, real-time response, 41.0%
Access additional data sources, 28.2%
Handle data and message streams, 20.5%
These latter areas -- data-analysis software gaps -- are in CEP's sweet spot, yet almost 50% of respondents had not heard of Complex Event Processing. This figure shows that vendor and thought-leader education efforts have made progress but that a lot more work is needed. (That figure also shows that my survey successfully reached outside a core CEP audience in eliciting survey participation.)
Business value in real-time, continuous analysis
The figures that 34% of respondents believe that a "CEP or Continuous Intelligence solution would improve [their] organization's analytical capabilities" added to the 38% who responded "might improve" -- as against only 10% who responded "unlikely to improve" -- do indicate the positive direction experiences and perceptions of CEP and Continuous Intelligence (Aleri's trademarked term for CEP-extended BI) are taking.
Responses to the question, "What business value can you see in real-time, continuous analysis?" demonstrate not only that current and potential users see value in "faster decisions" (68%) and "more accurate and timely information" (61%) but also that these capabilities are directly linked to concrete business measures that affect the profitability such as "higher customer satisfaction" (42%), "lower personnel and operational costs" (32%), and "increased sales" (26%).
Responses to "If you implement real-time or continuous analysis, how would information be delivered to business people?" communicate three interesting points. First, respondents expect to continue using interfaces users are already familiar with from the BI world such as dashboards (74%), reports (48%), and ad-hoc queries (48%). Secondly, respondents expect to bring on-line near or enhanced communications: "alerts (e-mail, phone, other messaging)" (68%). Lastly, respondents see data warehouses and data marts, even if enhanced, and spreadsheets, as not so promising for delivery of real-time or continuous analysis, with only 29% and 13% of responses, respectively.
Conversely, "enhanced BI tools" (58%), "OLAP cubes refreshed with live, real-time data" (35%), and faster data warehouse loading (32%) are seen as possibly drivers for real-time analysis. Only 23% of respondents thought existing BI tools would be up to the job.
Findings in context
The findings I have reported here are presented in a richer, narrative context in my TechWeb/IE webinar, which was recorded and is available on-line. You can also download my slides, if you wish, from the webinar interface. My goal was to help folks in the BI world understand how CEP -- event-driven, real-time BI using streaming and database sources -- can benefit them. I hope I've succeeded!Event-driven analytics aims to facilitate business decisions and actions as opportunities (and threats) emerge. Move the concept into the Now, into the worlds of on-line commerce and real-time fraud detection, and old, DBMS-reliant architectures just aren't fast enough. Complex event processing (CEP) added to BI promises to close the speed gap, to enable real-time event analytics...
How Enterprises Are Attacking the IT Security EnterpriseTo learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
IT Strategies to Conquer the CloudChances are your organization is adopting cloud computing in one way or another -- or in multiple ways. Understanding the skills you need and how cloud affects IT operations and networking will help you adapt.