Business Process Management 101: The Basics of BPM and How to Choose the Right Suite

IT giants are either buying business process management (BPM) vendors or expanding on existing BPM capabilities. Business Intelligence vendors are partnering with BPM vendors, as are purveyors of business activity monitoring and business rule vendors. Plenty of businesses are adopting the technology, too, so just what is BPM, why is it gaining attention and how do you choose the right technology?

You would think that defining BPM would be straight-forward, but that's not the case, in part because business process management suite (BPMS) capabilities are now being found in many solutions solving different needs. Thus, the terminology is sometimes used rather loosely (not to mention the confusion with the other BPM, business performance management). Regardless, one thing is clear: BPM is getting a lot of attention. Forrester Research estimates that BPMS license, services and maintenance revenue will grow from $1.2 billion in 2005 to more than $2.7 billion by 2009 – an adoption trend vendors and practitioners alike can't ignore.

A common confusion about BPM surrounds the difference between the workflow systems of the 1990s and today's BPMS. Older, proprietary workflow systems managed document-based processes where people executed the workflow steps of the process. Today's BPM systems manage processes that include person-to-person work steps, system-to-system communications or combinations of both. In addition, BPM systems include integrated features such as enhanced (and portable) process modeling, simulation, code generation, process execution, process monitoring, customizable industry-specific templates and UI components, and out-of-box integration capabilities along with support for Web-services-based integration.

All of these ingredients translate to increased interest today in BPM suites because they bring businesses a higher level of flexibility for business processes while reducing risks and cost. Think of BPM suites as offering a way to build, execute and monitor automated processes that may go across organizational boundaries - a kind of next-generation workflow. Read on to learn what's inside the two styles of BPMS, what makes the technology tick and how to choose the right system for your organization and process needs.

The Components of BPM Suites

BPM suites incorporate multiple technologies, as shown in the diagram to the right.

Another area of confusion is about the terminology used to refer to the different types of BPMS. For example, you may encounter the term "pure-play BPM," describing vendors that focus solely on business process management, but given vendor consolidation and an increasing number of solutions with similar capabilities, this terminology should soon disappear.

Lightweight BPM functionality is sometimes found in packaged enterprise applications such as ERP, CRM and financial software. This is a type of "embedded BPM," since it is not intended to be used outside of the application domain.

BPM features have also emerged from the enterprise application integration (EAI) vendors. These are enhanced versions of Enterprise Service Bus (ESB) products. These systems support the design, development and execution of system-to-system processes while integrating them by means of Web services and asynchronous messaging.

In a nutshell, it is much easier to view BPM solutions in two basic categories: Front-Office BPM (FO-BPM) and Back-Office BPM (BO-BPM). While vendors are racing to provide both types of BPM solutions, the maturity of their capabilities in each area tends to depend on their history and depth of experience in each domain.

The FO-BPM solutions (with roots in human-centric workflow products) provide capabilities for person-to-person processes in which "work items" are created and routed along with any attached documents. A partial list of vendors that provide FO-BPM capabilities would include TIBCO, FileNet, IBM, PegaSystems, Global360, Oracle, DST Systems, BEA (with recently acquired Fuego) , Computer Associates, Ultimus, Savvion and MetaStorm, among others.

On the other hand, the back-office-oriented BPM solutions (such as ESB products) enable system-to-system integration in which a single process may rely on multiple external processes (and possibly heterogeneous platforms) to complete its work. These suites also manage aggregation and composition of all services for the enterprise. Vendors such as IBM, Oracle, TIBCO, Sun, CapeClear, WebMethods (soon to be acquired by Software AG), Fiorano Software, Sonic Software and BEA Systems, among others, have specialized products in this space.

Front-office- and back-office-oriented BPM systems are aimed at different problems. For example, front-office BPM applications typically involve transactions that are "short running." This may not be the case when an internal process is integrated with Web-service-based processes (possibly outside of an enterprise), which can result in "long-running" transactions. Managing the transaction context of a process involving multiple resource managers requires an alternative approach in BO-BPM. (We will cover this in more detail later on).

One major differentiation among BPM suites is in support business process lifecycles. Therefore, it's important to understand what's involved in each phase of a process. The diagram on the left depicts the basic BPM Lifecycle. The basic flow is similar for both types of BPMS, but we'll describe some specific differences later in the article.

Process Modeling

The modeling phase is typically supported by a visual process design tool. Some vendors have their own proprietary tools while others provide specialized plug-ins that extend existing IDEs or products such as Microsoft Visio or Eclipse-based modules.

A process consists of multiple activities (also know as "steps" or "tasks").These are created and linked to each other to form the flow of the process. Conditions that define how and when an activity must be called are also are defined during the modeling phase. Process activities can include attachments, such as documents or images, that are passed to the next activity along with associated data. In addition, process can be split into parallel execution paths and joined at a predetermined step. If needed, an instance of a process can be started by an external event, such as the arrival of an e-mail, message or document. These capabilities can be graphically depicted using the same visual design tool. The activities within a process either require human involvement or are processed without any human interaction. Therefore, the modeling tool needs to support person-to-person and system-to-system processes.

Since different types of activities usually have their own visual representations (such as activity icons and graphics), this area of design and presented an opportunity for standardization. One such standard that is widely accepted is Business Process Modeling Notation (BPMN).

The outcome of the process modeling phase is usually a pseudo-executable process flow model that can be exported in an executable language format supported by the BPM engine. Thus, there needs to be a mapping from a modeling language (such as BPMN) to a process execution language (such as XPDL or BPEL) understood by the BPM engine of your choice (more on this later).

To streamline the transition from business design to technical design, some process modeling tools are also capable of producing application design diagrams in Unified Modeling Language (UML). Advanced modeling tools also let you simulate processes and changing activity parameters such duration and resource utilization to test impacts on process efficiency.

Most BPMS vendors supply modeling tools that are optimized to work with their solution, but there are a number of vendors that specialize in process modeling (and analysis, to be covered later), including IDS Scheer, Proforma, Mega, iGrafx, Telelogic and Casewise.

Process Activity Implementation

To create the business logic of the modeled activities, one must provide the necessary instructions (beyond those required to manage the execution flow and pass necessary data). Further, the complex activities within a process often need to be programmed at a lower level to do such things as accessing databases or performing business calculations.

The development environment varies by vendor strategy, but this typically comes in the form of either an integrated development environment (IDE) that is integrated with the modeling tool or a separate IDE. Regardless, all vendors try to minimize the need for hard-coding by supporting a visual programming environment. The most common characteristic of BPM IDEs are meta-data-driven development and code-generation aimed at speeding implementation. The programming language used might be an inference-based business rule engine or a standard language such as Java or C++. To implement more complex business logic, it might be necessary to rely on different programming tools. To ensure flexibility, almost all vendors provide a set of APIs and Web service interfaces.

Front-office-oriented BPM systems typically let you design forms, define data fields, customize process templates, set up access control lists, configure integration capabilities and manage deadlines. Since back-office-oriented BPM solutions aren't intended to support human-centric workflow (and therefore don't require form capabilities), they focus instead on features such as back-end process composition, data/message transformation and automatic creation of service interfaces to ease implementation.

If your application requires integration with external systems, look for out-of-the-box support first. Usually, vendors offer prebuilt components to integrate with document/content management systems, security repositories, third-party packaged applications and middleware products.

Once process activities are implemented, the application is deployable and ready to execute on the vendor's chosen platform, such as J2EE or .Net.

Process Execution

The process engine that runs on the BPMS vendor's server is responsible for executing the process model exported for execution and activities programmed. A typical run-time environment also needs other low-level services, such as security, transaction and concurrency management. These services are either provided by the same vendor or the vendor relies on another technology, such as applications or servers. If the transactions involve multiple business partners, the challenge of managing those transactions gets more complex.

To control the execution of process activities, there are basically two approaches: orchestration and choreography. The distinction lies in the point of view on "who is the process owner" of an internal corporate process, an external public process (peer-to-peer/B2B) or combinations of both.

The orchestration approach mainly assumes that the process owner is always a single-entity that has total control of all message exchanges between activities. An example is a workflow application for an in-house loan-approval process.

The choreography approach, on the other hand, implies that the process is owned "jointly" by all participants, and the sequence of multiple message exchanges among participants are driven by mutually agreed "contracts." Examples of this include the fulfillment processes of a purchase order in a B2B environment.

Among process execution languages, BPEL more closely resembles the orchestration approach. Another standard, WS-CDL, mostly addresses the needs of the choreography approach. XPDL, on the other hand, is mostly seen in human-centric workflow tools/applications. In the future, expect BPEL to include similar capabilities from XPDL and WS-CDL.

Process Monitoring and Analysis

The ability to report business performance metrics is increasingly important and is a major differentiator among BPM suites. Given the large potential impact on process efficiency, BPM vendors are rushing to include this type of business activity monitoring (BAM) capability in their suite through acquisitions, partnerships or enhancements of existing tools.

Unfortunately, would-be customers are usually confused about how to best use BAM capabilities. Part of the reason for this confusion is that there are two types of BAM: real-time and historical. Yet the distinctions between the two are not clear to most.

Real-time BAM components are tightly integrated within the BPM solution to monitor metrics for work-in-progress (such as the number of process instances started, execution times of activities and so on). Advanced real-time systems can do such things as record an execution pattern signaling process failure so you can avoid them in the future. Real-time BAM also predicts work loads for better resource utilization. Filtering and reacting to business events is also a key feature. For example, based on a certain business condition (such as receiving an electronic order), the following actions can start: send e-mail to a user, automatically start another process instance and send alerts to dashboard/console.

Historical BAM components, in contrast, provide analysis reports to draw conclusions from purely historical execution of already completed processes. Process-specific key performance indicators (KPIs) and service level agreements (SLAs) can then be compared against this data. These tools can also integrate with existing data warehouses and may require a different server platform than the BPM server.

Collecting execution metrics also enables "closed loop" modeling in which process simulations can be adjusted to improve efficiency.

Are We There Yet?

The short answer to this question is "not completely," but we are getting rather close. Even though the industry has make much progress in foundational areas, it is still a highly dynamic market. Here are just a few of the areas that are in flux:

There is currently a lack of strong support for a standard way of working with processes that involve human-interactions, even though there are recommendations to address them. BPEL currently does not handle person-to-person processes, so an extension has been proposed called "BPEL4People." Similar capabilities can also be found in XPDL.

The security issues are about message integrity, confidentiality and authentication. In SOAP-based communication, XML extensions are created to support the use of multiple security tokens (such as username, certificates, SAML) within the message. The example of such effort comes from the WS-Security specification.

Communication standards are a priority given the need to handle communication problems gracefully between any two systems (due to, say, a network or application outage). In SOAP-based Web-service integration, the two competing standards, backed by different vendors, are WS-RM and WS-R. They are both XML-based protocols that attempt to ensure that the sender of the message gets a reliable acknowledgement when the receiver has received the message (and preferably ignored any duplicate messages). Currently WS-RM seems to have broader vendor support, but reconciliation between the two standards is necessary.

One other area in flux involves how transactions are managed during system-to-system interactions with Web services. BPEL and WS-* extensions are becoming widely supported in this type of back-office BPMS. BPEL workflow transactions, for example, can be short or long running. WS-Transaction and WS-Coordination services are ways of supporting transactional Web services. While another standard, BTP (Business Transaction Protocol) supports similar needs, currently BPEL and WS-* extensions are getting more visibility.

On the BAM front, BPM vendors are determining how to best collect and present information from the customer's BAM tool of choice. While most BPM products provide operational metrics on dashboards, it's not easy to use BAM and BPM products from different vendors. Creating historical reports from BPM data and analyzing KPIs using existing reporting tools requires customization effort.

When disparate process engines (XPDL- or BPEL-compliant) from different vendors need to be integrated, it may be hard to reconcile different levels of standards, vendor-specific features and proprietary interfaces. For example, to start a process in one process engine and hand it off to another, you need proper communication. The WfXML initiative is an effort to address issues specific to this scenario.

Before You Buy

The selection of a BPMS demands unique analyses you won't find on a typical software evaluation checklist. Here are 6 things to look for in determining the BPMS that's right for your organization:

1. BPM systems aren't meant for all application needs. BPM systems are especially good for processes in which all activities are predetermined in order. Validate the business value of taking a BPM approach by developing a deep understanding of both the business problem and the capabilities of the BPMS. Document the detailed activities of your process and be sure the BPMS addresses core needs before committing to a product.

2. Next, consider the type of BPM solution needed: front-office-oriented BPM, for human-centric processes, or back-office-oriented BPM, for integration centric processes. It's possible that your environment may benefit from both types of BPM. If so, focus on products from either the same vendor or from vendors that have partnered to minimize integration efforts. If you use solutions from multiple vendors, keep in mind that you could end up with multiple process modeling and monitoring tools as well as server platforms.

3. Once you decide on the type of BPMS you're after, refer to the BPM lifecycle activities and architectural components covered earlier and analyze how your requirements are supported, in detail, by the prospective products. Make your product selection based on the total value of the BPMS rather than just standards support, particularly if you're considering front-office BPMS (an area in which standards aren't settled).

4. For front-office-oriented BPMS, ease of customization and out-of-box support for integration (such as with a content management or imaging systems) should be high on your list. If the same vendor provides content-management and collaboration modules as well, then confirm that their repositories, business modeling tools and administration consoles are integrated with all these components.

5. Among back-office-oriented BPM suites, you will notice that BPEL and WS-* support are widely implemented. Therefore, it's important to pay more close attention to the vendor's direction. Consider the vendor's support message transformation, routing and multiple-transport protocols. While most front-office-oriented BPM systems can be deployed on application servers, BO-BPM server requirements can vary. They may require installation of messaging-oriented middleware (MOM), application servers, subsets of application servers (such as a Web container) or some combination. Make sure you understand how the vendor solution is architected in order to wisely plan for its technology fit and impact on infrastructure.

6. Another issue you may want to consider is how rule engines are used in the BPMS. Rule engines are typically an optional component in most BPM suites, but you may find that certain vendors requires everything to run under a rule engine that needs a proprietary language. Determine whether the benefits of using that rule engine for the entire application are conclusive while also considering potential hiring and training needs tied to that environment and language.

BPM solutions bundle a lot of capabilities, so you should expect a learning curve to before you can take full advantage of a suite. Focus on proper training and establish best-practices and guidelines to create manageable deployments. Remember that matching the right type of BPMS to the processes considered is the most important decision you will need to make early on.

BIO: Tulu Tanrikorur is a corporate vice president, enterprise architecture at New York Life. He has written numerous articles including Who Are You? and Great Expectations. Write him at [email protected]