First the good news: More than ever, business and technical users have access to a broad supply of information coming from across the enterprise. Organizations are utilizing and leveraging unique knowledge assets, thanks to growing deployment of enterprise information portals (EIPs), which provide a single gateway to heterogeneous data sources and applications. EIPs also serve as a platform for personalization and security tailored to the user's functional business role in the organization.
EIPs have broken ground for the development of performance management dashboards, scorecards, alerts, business process notifications, business intelligence (BI) reporting, e-mail and other mission-critical technologies for information-driven organizations. This reliance is driving business and technical users to collaborate to establish enterprise information management and architecture initiatives, such as master data management, data warehousing, metadata management, data quality management and data distribution services.
But now the bad — or at least challenging — news: The difficulties in supporting enterprise information management today and into the future are growing in size and scope, outpacing many organizations' ability to keep up. Users need information 24 X 7; especially for global organizations, data integration must occur around the clock. Maintenance and upgrade windows are shrinking. Creating more redundancy (typically by using failover systems) is an obvious option, but the cost and complexity associated with this approach lessens the appeal. IT management is in a quandary.
In this article, I'll propose a solution that builds on a key investment already in place at many organizations: the operational data store (ODS). Experience with ODS implementations is growing at a propitious time — a time when IT desperately needs to think outside the box about where to take its enterprise information strategy. After considering where things stand now, I'll discuss how the ODS may be repositioned to serve broader enterprise information requirements.
A traditional ODS most often comes into being as a component of an overall data warehouse environment. The motivation is generally to support faster operational decision making with more focused, subject-oriented and time-sensitive data than the historical snapshots found in a traditional data warehouse. An insurance company might deploy an ODS to speed up its claims processes; an e-commerce retailer might use an ODS to interpret, in real time, what customers are doing on its Web site.
An ODS integrates detailed, operational data from different sources into a single store, without the history, summarization and extensive transformation typical of a data warehouse. It's set up to serve many standard, simpler queries against minimally transformed (if not atomic) data rather than complex, ad hoc queries against often extensively transformed and aggregated data. With its emphasis on frequently updated information, an ODS is usually smaller than a data warehouse, which might contain many years — and many terabytes — of data. Where a data warehouse might store an entire history of customer contacts, for example, the ODS would store just the names needed currently by marketing or accounting business functions.
Some companies try to combine ODS and data warehouse activity into one system. Business requirements, the number of operational systems and complexity of data transformation and integration processes, however, have led organizations to separate the structures schematically and often physically. Some ODS deployments serve nearly automated tasks, such as determining whether a credit card transaction is fraudulent or whether a customer meets profitability or other criteria for loans. An ODS frees both the transaction-oriented operational systems and the data warehouse from having to balance information delivery processing loads and prioritize operational, tactical and strategic queries.
An ODS might be the perfect antidote for the pressing enterprise information availability challenges I discussed at the beginning of the article. However, a traditional ODS still lives downstream from the operational data, which means that goals of producing immediately "actionable" information will always hit an architectural wall. But what if we could reposition the ODS to improve overall information availability?
IT managers should consider positioning their ODSs to be at the center of their enterprise information environment, as depicted in the diagram below. This would involve thinking of the ODS as more than just an adjunct to BI and data warehousing systems. A reengineered and redeployed ODS, in conjunction with an EIP, would help organizations address the greater availability, scalability, performance and security issues that challenge managers of heterogeneous business information systems.
How do you make the transition? Let's first consider more closely what we're dealing with. Numerous applications, involving many business areas, make up a typical enterprise information environment. Each one contains its own unique set of data, organized to fulfill the purposes of the business function the application supports. Recruiting systems create requisitions; payroll systems create pay runs; and billing systems create bills. In addition to the unique data, each application has a core set of information that's essentially common to most of the operational systems; these data objects include customers, employees, contacts, products and locations.
The purpose of a reengineered ODS, coupled with an EIP, would be to aggregate the required transaction data into a central store, making the data available to more users and applications across the enterprise than is possible with a traditional placement of an ODS or data warehouse. This new positioning would address the wider problem of availability: If any one of the operational systems goes down due to maintenance, upgrade or performance problems, the ODS would give EIP users and applications access to the integrated selection of operational data. Business requirements would dictate how much transactional data the ODS would store; the amount might be similar to that for a traditional decision-support ODS.
Business requirements would also determine how "just in time" (JIT) the data integration would need to be. Organizations could establish JIT data integration by deploying data adapters between the ODS and enterprise operational systems, as shown in the diagram. Enterprise application integration (EAI) software from vendors such as webMethods and Tibco are sources of such adapters, although note that these vendors and others are moving quickly to support nonproprietary standards that function within Java and/or .Net environments. Full-fledged EAI tools provide advantages of scalability, failover, security and guaranteed delivery in both synchronous and asynchronous modes.
You could also look at other current data integration technology to link the ODS with operational sources, including extract, transform and load (ETL) and enterprise information integration (EII) technology. However, remember that an ODS is designed to integrate information coming from all relevant operational systems though extraction and replication. This is fundamentally different from an EII approach, which unifies views of systems at a metadata level, and typically sends queries out to data sources, only bringing back result sets. While your business requirements will guide you to choose the most appropriate technology for your organization, the centralization route is best suited for the "ODS reinvented" approach I'm describing here.
In addition to a centralized database, the repositioned ODS would have a message broker that serves as a distribution hub for messages that carry data and establish sharing between the ODS and operational applications. Here again, availability is better because the ODS database can maintain the data for the EIP. You could use the same EAI approach to replicate transaction data into the ODS itself. In some business scenarios, transaction data is crucial for integrating multiple operational systems; two good examples would be a multiple-order entry system and a set of regional operational systems for human resources management.
Operational systems could use the message brokers and bidirectional data adapters to share core data with other systems. A new item in the order management system, for example, could also update customer information in the billing system. The ODS wouldn't serve as the system of record for core data: That responsibility remains with the source systems. The ODS would serve as an integration point, from which updates could be sent from one operational system to another via the EAI architecture I've described here or through alternative data integration tools. EAI tools would provide message routing, delivery rules, monitoring and support. Business and IT managers would set the data governance rules that determine update frequency.
The primary purpose of this repositioned ODS is information availability. The data warehouse would assume responsibility for all reporting. It would continue to extract data from the source operational systems. However, the data warehouse could take advantage of the data already integrated within the ODS to cut down on sourcing and integrating steps.
Organizations face availability and flexibility difficulties throughout their enterprises, not just those within the sphere of BI and data warehousing systems. The growing presence of EIPs and related user interfaces only increases the pressure to provide a shared resource of timely data coming from a wide range of sources and applications. The reengineered ODS approach can provide an information access layer that can serve EIPs, especially when the source operational systems are unavailable for one reason or another.
The repositioned ODS, with the EIP, also helps organizations standardize how they deliver information to users. The combination can also shield users from changes to ERP or other operational systems. In addition, as a centralized database supported by EAI middleware, the new ODS approach will reduce the number of data interfaces between operational systems.
What are the risks? Organizations must always consider each potential point of failure: and the repositioned ODS and EIP do become just that. Thus, failover and performance considerations such as load balancing are important to consider in the context of the high-availability demands the refashioned system is supposed to meet.
The scalability of a messaging approach to moving and integrating data is also a potential risk. Depending on the volume and design, backlogs could form, which will delay enterprise data integration. Significant backlogs or failures could be challenging to resolve; it may be tough to find a point to restart processing with the message broker approach. Mass or bulk changes to operational systems could bring about bandwidth problems and severely degrade the messaging backbone. An example of how this might occur is if your company performed a divisional realignment, merger or acquisition. This would necessitate a major change to the organizational structure of the HR or financial operational application. Thus, it would be critical for data governance rules to set out which applications receive versus distribute specific data objects from the ODS.
Nevertheless, organizations should consider how they might expand the potential of current and future ODS and EIP investments to serve the entire enterprise with greater data availability. Enterprise information is now mission critical; with careful design and deployment, this new approach can help your business flourish in a competitive economy that rewards organizations that share knowledge effectively by leveraging the full value of information assets.
Michael Jennings is VP of Professional Services for EW Solutions. He is co-author (with David Marco) of Universal MetaData Models (Wiley, 2004). Write to him at [email protected].