Q&A: Gartner's Roy Schulte on Choosing BI, BAM or CEP

You offered a keynote on event processing (EP) at a recent Gartner BPM Summit. How does the technology show up in business process management?

It's fairly common to see EP used to monitor business processes. That's one of the big reasons people use BPM in the first place — to give end users visibility into what's happening in those processes. BAM monitors are something that end users can see, and they're attractive to them because they're easy to use and understand.

In most cases BAM vendors have developed their own EP engines that operate behind the scenes to correlate multiple incoming events. If you're going to put up a dashboard and turn the lights red, yellow or green, or show a gauge or a bar chart, you almost have to be using EP under the covers because you're trying to spot changes in seconds or even milliseconds.

Many BI vendors offer dashboards with the kinds of displays you describe, but I take it they're not in the same latency league as BAM products?

The BI vendors are certainly moving toward BAM. If you put up a BI dashboard and you're refreshing it, say, every ten minutes, which you can do with any BI dashboard, then you are doing BAM with traditional BI tools. BAM is really a style of deployment rather than a distinct technology type, so I would say BAM can be done with traditional BI tools.

When it comes to low-latency BAM, where you're trying to refresh the dashboard every few seconds, traditional BI tools aren't fast enough. That's when you need BAM products that are purpose-built to run fast.

Can you name some leading low-latency BAM products?

There's Oracle BAM and Sun has the SeeBeyond eBAM product. IBM has several BAM products, one of which is Cognos Now, which Cognos acquired from Celequest... Information Builder's iWay [integration unit] is getting there, too...

What about the ultimate in event processing, namely the kind of complex-event-processing (CEP) technologies used on Wall Street and in the intelligence community? Can you describe the latency demands in that arena?

These products are processing 10,000 or more incoming events per second, applying rules, coming up with an analysis and taking action within about 20 milliseconds.

So when do you step up from BAM into CEP?

If you're doing something that involves a human being, you probably don't need CEP. People can't work that fast, so in those applications you may be able to use BI or BAM. When you're dealing with machine speeds, that's where CEP comes in. Are you seeing signs that CEP is finally moving into mainstream business use?

Absolutely. The biggest area today is around customer experience management. We're not necessarily talking about rarified speeds in this case, but low-latency processing is not the only reason to use CEP. The technology can also be used when the application is very complex. One customer-experience-management example is monitoring Web traffic. So you're seeing products like Tealeaf integrated with CEP engines like Coral8. Tealeaf is used to listen to the http traffic coming in and Coral8 finds the patterns in that traffic.

Customer experience management is a great use for CEP because you can't control it, you can't predict it, it's high volume and it's continuous. At any given moment you may have thousands or tens of thousands or hundreds of thousands of customers clicking through your Web site. You don't know when they're going to arrive and when they're going to leave. There are rush periods and slow periods, yet you need to find correlations and patterns.

Can you cite other customer experience management examples?

Some companies are using CEP to track call center loads. If the call load is suddenly much different than usual, you might find a pattern in the data that tells you that a competitor just dropped prices. Or maybe you see that call volumes are higher than expected, so you'll bring in extra people for the second shift.

A call center obviously involves human beings, so is this one of those medium-latency scenarios?

Call centers don't demand millisecond latency, for sure, but you might still use a CEP engine due to the volume of messages. The application might also involve very complex rules where you're keeping track of thousands of customers.

This is one area where BAM and CEP are starting to overlap. For example, one vendor known for BAM is Systar, but in some of these customer experience management sales scenarios they've been introducing themselves as a CEP vendor. They compete with the likes of TIBCO Business Events and Progress Apama, which are both so-called CEP products. At the same time, many of the so-called CEP products now have dashboards and analytical capabilities. So increasingly, BAM tools can do CEP and CEP tools can be used to build BAM.

You're speaking on EP application selection and design at this week's Gartner Event Processing Summit. What's your key advice?

My advice is that the business analysts need to learn about BAM and CEP because middleware people don't understand the business processes well enough to be able to identify where to use BAM and CEP. You have to educate your business analysts on characteristics that are a good fit for BAM and CEP.

And what are those characteristics?

It has to involve part of the company where you have continuous operation. It's well proven in stock trading and risk detection, but it might be any situation in which things are changing continuously and you can't predict how or when conditions will change. Transportation operations is a good example, as is customer experience management. Factory floor automation is another good area. It's anyplace where there are ongoing activities and where there's a business value in gaining visibility into what is happening at this moment so you can make smarter decisions. What are some of the best practices you'll share on application design?

It's partly the classic advice to start small and grow, and to talk to the business people and get them involved. You have to get business people to identify key performance indicators that would help them make better decisions if they could see up-to-the-minute information.

You're also speaking this week on event-driven architecture and mixing CEP with SOA. What's your advice?

The key point there, if you're an architect or a CIO implementing SOA, is don't just put in an communications backbone that only knows remote-procedure-style SOA. Make sure you can do both remote-procedure-style and event-driven SOA.

There will inevitably be parts of your application that are asynchronous, meaning they happen sequentially instead of interactively. If you have those asynchronous kinds of elements, where step one follows step two follows step three, you should be doing it with event-driven SOA. If you don't, then you'll have to step outside of SOA and you'll have a mix of SOA and traditional messaging or traditional file transfer. If part of your application is non-SOA, it will be clumsy and messy.

How do you ensure that you have event-compatible SOA?

Make sure that you can use the same communication backbone for asynchronous communication as well as request-reply synchronous communication. Otherwise you'll have to use separate middleware products for the asynchronous stuff. You end up with two communication pipes instead of one.

Any final thoughts on the coming era of event processing?

I'm excited about BAM — not necessarily the term "BAM," because most people don't use it, but the concept. I think this whole approach of creating dashboards and giving visibility to business people is such a big win.

My parting thought is that if I sit down at my desk and bring up an application, I can't tell if that application was built with SOA or not. I can't tell if it was built with BPM or not. That's invisible to the end user, and only the IT people know or care. However, I sure can tell if I have a BAM dashboard, because that is something that changes minute by minute, and it gives me information I want. BPM and SOA? Yeah, you should be using those two things because that will make the application itself easy to reengineer. But what really helps users — and what may be the only thing they can see — is BAM. It's the equity in visibility.