Embedded BI: Intelligence at Your Service

Outsourcing individual business operations and services is a major trend in global business operation models today. In such scenarios, often the only legal obligations that govern the service provider are the terms stated in the service-level agreement (SLA) for a specific service performed and rendered.

What companies most commonly want to outsource are service- and manufacturing-oriented business operations. Manufacturing processes, although complex, have less direct interaction with customers; current SLAs are normally sufficient regardless of where the actual manufacturing, assembly, and delivery process takes place. Customer service operations, however, require service providers to have direct interaction with customers. Beyond the technical aspects of problem resolution and support, it matters a great deal how "tuned" a service provider is to the customer's psyche. It's very hard to spell out cultural, linguistic, and psychological characteristics in SLAs.

Today, when customers call to address problems or follow up on unresolved problems, they usually end up interacting with voice-response prompting systems, which put them through sequences of prompts, and eventually to customer service representatives. By the time customers reach live representatives, they're already frustrated. Further delays and unsatisfactory answers stoke agitation, eroding customer confidence and possibly causing the company to lose business. Is it possible to turn such a situation around?

Let's fast forward a few years. In a future version of this scenario, before the customer gets upset, embedded mobile intelligent agents pick up the conversation pattern and voice tone change. The system proactively addresses the rising anger by playing an appropriate subliminal message that calms both the caller and the customer rep before it gets too late. This "intelligence" is built right in the "communication" layer, where an intelligent agent performs voice mining on the fly to understand not only the language, voice envelope, and culture involved, but also the psyche of the caller. The system understands the customer's behavior with far more sensitivity than what's possible with today's customer profiling techniques.

Globally distributed intelligent agents can search and aggregate information from heterogeneous sources. Internet devices, making extensive use of in-memory database technologies, broadcast and receive information continuously through mobile agents as part of business process service chains. In a number of scenarios, such agents and devices will eliminate the need for traditional, central enterprise data warehouses where the SQL engine determines what "intelligence" can be derived from the warehouse's relational DBMS.

The key shift that will bring this vision into reality is the new wave of embedded business intelligence (BI) agent technologies, which are well suited for service-oriented business models. By definition, network-based services are decentralized; this new services environment changes the requirements for implementing intelligent business solutions.

Embedded BI Today

The buzz around "embedded BI" is growing among BI vendors. However, merely generating and publishing reports on a scheduled basis isn't embedded intelligence. Some vendors also describe embedded BI as the integration of information into portals to implement dashboards. I would classify current discussion about such BI solutions into four categories:

BI-centered analytics. BI vendors provide views against data warehouses. These views (primarily reports) are embedded in decision-support solutions and portals. This sort of "embedding" simply renders data into formats best suited for end-user views. Analytics aren't woven into business processes. Emerging technology for integrated OLAP, online analytic mining (OLAM), text mining, and visualization from vendors such as PolyVista will improve stand-alone, BI-centered analytics. However, rather than close integration with business processes, the focus remains on traditional, after-the-fact information processing from available data warehouses.

Callable APIs. With this approach, BI vendors expose intelligence via programming interfaces. MicroStrategy, for example, offers robust APIs, which are called from within an application to add reporting and analytic capabilities. Such APIs are good for accessing data objects (such as reports, charts, and key performance indicators [KPIs]), developing dashboards and other stand-alone decision-support solutions, or blending such objects into business applications. The API approach enables customers to design their own BI applications. The primary use of this approach has been to build a wide array of analytic applications — but still with very little integration into business processes.

Database features and extensions. IBM, Microsoft, NCR's Teradata Division, and Oracle offer built-in data mining techniques to integrate analytic capabilities with traditional data warehouse-based BI solutions. Database vendors have much more control over "embedding" intelligence due to native data mining algorithms available within their respective database platforms. As long as you stay within the homogeneous world of each vendor, this approach to embedded intelligence can be fruitful.

Business applications with "integrated intelligence." Major enterprise application vendors generally offer a common infrastructure for both transaction and BI systems, regardless of the database platform. The benefit is that you have a greater possibility of a closed loop between decision-support and operational systems. Implementing this vision requires a central, usually single-vendor data warehouse system shared across all business applications. SAP's Business Information Warehouse offers a good example. In recent months, SAP has announced even tighter, closed-loop BI integration among business processes through its NetWeaver initiative, which is intended to span SAP and non-SAP applications as well as structured and unstructured content. NetWeaver offers a promising infrastructure, but it doesn't yet bring the truly embedded BI capability needed to drive distributed, service-oriented business processes through agents.

Integrated or Embedded?

Before defining embedded BI (EBI), let me flesh out the term "integrated BI" as I have used it thus far. You have business information residing in its own respective environment; integrated BI is when you can link the information to specific business processes to support specific business activities. As an example, consider a credit card transaction processing system, where a business application calls an external service provider (such as Visa Network) to validate a credit card and check for fraudulent activity. Another example would be publishing a product sales status report on a company sales dashboard, or pulling several KPIs from diverse sources and publishing them on the sales executive dashboard with very little visibility or knowledge required of the full business processes associated with the KPIs. This common scenario is served by the first three categories discussed earlier; that is, the solutions rely on separate environments integrated through APIs or at the portal level as aggregated content.

In contrast to integrated BI, true EBI is an integral part of a business process that drives networked business activities for effective business operations. EBI is deeply woven into the business process and gives full visibility into the processes that drive business applications. You may find that this sounds similar to business activity monitoring (BAM). However, with EBI, I'm talking about acting on BI right then and there, when and where it's needed within the business process — not after the fact.

Most BAM solutions look at the business activities from the outside in; BAM systems offer no built-in actions other than to raise alarms or display business activity status. EBI looks at activities from the inside-out. EBI brings full visibility to actual business processes so that organizations can act on specific decision-making points without stepping out of the business process flow — crucial for emerging business models that depend on distributed services. SLAs will have to include statements about the actions agreed upon by participating business services as necessary to fulfill a task effectively.

FIGURE 1 The evolution of embedded business intelligence.

EBI relies heavily on distributed, autonomous intelligent agents that travel along the business process streams, facilitating the collection of key information and making the right decisions along the way. Thus, along with distributed, autonomous intelligent agents, EBI requires a second key technological advance: distributed information mining algorithms that can work with both structured and unstructured content. These algorithms could ultimately eliminate the need for many data warehouses because they could detect anomalies and patterns and react as specified in SLAs. Figure 1 offers a view of how EBI is evolving with these new technologies.

Intelligence gathering and "connect the dots" requirements from intelligence organizations, the U.S. Department of Homeland Security, and other public and private entities are driving research into distributed information mining and sophisticated autonomous agent technology. As I've stated, such advances are going to be important for service-oriented business applications. Traditional data warehouses could become nothing more than document and legal records repositories. As Figure 1 shows, new "intelligence repositories" will contain rules, records of observed discoveries, and records of actions taken by autonomous agents participating in the business services chain.

The Intelligence Flow

The futuristic CRM example I described earlier would happen through collaboration of several autonomous intelligent agents: mobile agents on the caller's device, the voice recognition agents, possibly visual agents, and human "psyche" agents. To accomplish the task, there would be no time to go back to a data warehouse to pull a customer profile, blend it with the business process, send data back to the data warehouse for analysis, and then make a decision. The CRM scenario offers just one example of why autonomous intelligent agent technology is well suited for distributed business processes that depend on individual business activities to carry some intelligence along with them in their path to hand over to the next business service in the process.

In Figure 1, you can see that when the system records an activity such as a new order, it launches an agent at the client device to carry basic information to facilitate a business activity. Based on that agent's request, the application decides how to process the order by looking into the business processes, at past intelligence rules, and at available SLAs. The application compiles its needed set of services, key decision-making checkpoints, and SLAs associated with the individual subscribing services.

The application then takes this complete set of information and issues a request for services that in turn launches autonomous agents associated with individual services. Remember, these services may or may not be offered with the same application or company; they may be drawn from a services provider based on SLA agreements. Therefore, embedded intelligence solutions also provide SLA-related information to agents so that agents can automatically "adapt" to the changing business environment without going back to the originating application to inquire about next appropriate action(s).

Although most of the business Web services in my scenario exchange information using some XML flavor, services can also exchange all the key information described through the distributed business process chain. Carrying the intelligence through the chain eliminates the need for unnecessary tasks that require the system to look back in the centralized data warehouse for whatever additional content might be needed to perform a service task intelligently.

Distributed agent technology for business processes is at an early stage. New vendors, such as Global IDs, Agenik, and InfoGlide are exploiting network-based agents to address BI without going back to data warehouses at every step in a business process. Technology providers must still solve how to orchestrate autonomous intelligent agents and keep track of security and privacy. Researchers are exploring distributed data mining algorithms to safeguard privacy and security.

Organizations must also consider legal ramifications of autonomous, intelligent mobile agents acting as part of a business service chain that surely travels through service providers in different countries. EBI scenarios such as the one I've described amplify the challenges, but legal concerns about information ownership are inherent in global distributed services business operation models. Who is legally responsible for the business information being exchanged when the originator may not be visible in the stream of services working in concert to complete a business task? And where and when does that responsibility begin and end? Web services organizations have begun to address the orchestration problems, but have yet to establish any clear standards and protocols.

Application Vendors' Progress

Most BI solution providers are still very much devoted to stand-alone analytics or providing BI through APIs that allow business applications vendors to integrate reports, charts, KPIs, and more into their role-based portal environments. The portals bring users information aggregated from diverse sources, which enables them to perform some collaborative actions. However, the portal integration is largely just an extension of traditional reporting or analysis; "intelligent actions" are still driven by humans themselves. These systems can do little in the way of proactive decision-making, even when actionable information is available.

Fortunately, major business applications providers (particularly Oracle, PeopleSoft, and SAP) have recently embarked on technology directions that will break their tightly integrated business applications and processes into loosely coupled services orchestrated by a Web service-oriented architecture. The new approaches align well not only with how organizations want to manage business operations today, but also with the future of distributed networked services. The "grain" of the individual service will matter a lot when it comes to EBI. For example, if a company chooses to outsource a large-grain service such as human resources, it will need the SLA to state many intelligence requirements. A small-grained service, such as "validate a product against a corporate part master file" would demand fewer EBI requirements.

Business processes will determine the time sensitivity of EBI: In other words, EBI doesn't always have to mean "real-time" BI. To some degree, time sensitivity will depend upon how the business might define individual business service granularity: that is, whether the service performs an entire transaction from start to end, or just a portion. For this reason, the EBI infrastructure must be flexible, so that it can define actionable events in the business process and SLA obligations and orchestrate intelligent agents appropriately. A book order-to-cash process at Amazon.com may only last a few minutes; an order-to-cash process for a Boeing jet may last more than a year.

Driving Adaptive Businesses

In contrast to today's business portals and dashboards, the purpose of intelligent agents isn't simply to provide integrated information to users but to eliminate the need for such portals except where users still have to direct which actions to take. BI agents are intended to make the needed adjustments and decisions whenever and wherever they must be made, thereby eliminating the need for massive data movement across the business-process chain.

No data warehouses? Is this revolutionary idea possible? Is EBI possible? The answer is yes: And the technology will become reality faster than we realize. BI has come a long way, but it still presents primarily the historical view of what happened — and more recently, a present view of what's happening in business operations. However, as we move from a database-centered to a network-centered services paradigm, we'll need a different approach to BI, one that moves away from centralized data warehouse models.

Distributed agent technology will play a pivotal role. True EBI, through autonomous intelligent agents, will make business applications much more adaptive, allowing them to learn from the operating environment and act appropriately to run global business operations more efficiently.

Naeem Hashmi is CTO of Information Frameworks and strategic advisor to International Technologies Inc. He's an expert in emerging e-business intelligence, enterprise information architectures, and portal technologies.