Let The Data Flow - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Software // Information Management

Let The Data Flow

Integrating operational data has long been one of the toughest IT problems. One choice is ripping and replacing legacy systems and consolidating on a single application platform, but that's neither affordable nor practical. You'll find a better answer in business integration convergence, a trend toward combining techniques including metadata repositories and discovery, data modeling and data quality services, ETL and EII, and integration with message brokers and ESBs for timely data synchronizat

The more complex an organization's business relationships and activity, the louder corporate leaders demand that IT do one thing: simplify. And when it comes to strategic business applications and intelligence systems, the word both business and IT executives use most often to describe how to achieve simplicity is common. Business users want common interfaces, common business processes and common application functionality, tools and services.

Executive Summary

Information silos are the bane of most organizations as they try to integrate and share data to get a clearer understanding of customers, products and other objects of interest. Regulatory compliance, performance management and collaborative business relationships demand a single view of relevant information. The question is: What's the best way to get the flow going so that users can see a consistent, quality view?

Data warehousing has solved many problems but cannot respond easily to changes in the organizational context of the data, such as new product hierarchies or altered reporting requirements brought on by mergers and changing business relationships. Gathering and managing metadata, or data about the data, is critical but only part of the solution. Often, each application or data store has its own metadata, which must be coordinated through a shared vocabulary. In the end, users need a richer understanding of the data and its context.

Master data management (MDM) is gathering interest as a comprehensive approach to information integration. Current technologies and methods are part of MDM, but IT managers should look at new tools that are in step with service orientation. Managers must focus MDM first on integration and synchronization of operational data both to support automated processes that cross applications and to inform decision makers about business conditions that demand immediate action.

Moving from where we are now to this "common" destination is the business integration equivalent of going on the Atkins Diet. Old habits and "high-carb" business and IT behaviors must change. The potential benefits, however, are clear: significant cost reductions, increased agility through better separation of processes from their application silos and overall improvements in business performance.

Efforts to establish more commonality are driving — and impacting — data and information integration. For many, today's Holy Grail is metadata — that is, a common source of definitions and context about the data. Metadata plays an important role in master data management, which is the buzz among vendors and user organizations as a way of getting to the next level in establishing shared information resources. You can read about MDM's potential for business intelligence reporting and analysis in "The Search for One Truth." Here, let's focus on operational data integration.

Integration: The Great Web Hope

With many business integration efforts converging, no project can proceed in a vacuum. There are five major levels of integration that most businesses are trying to tackle using their respective software tools:

  • User interfaces (enterprise portals)
  • People (collaboration tools)
  • Business processes (business process management [BPM] systems)
  • Applications (enterprise application integration [EAI], message brokers and enterprise service bus [ESB] software)
  • Data and metadata (extract, transformation and loading [ETL], enterprise information integration [EII] and enterprise content management [ECM] tools)

Each area is important — and complicated. Businesses need the technologies to work together to solve integration problems. Pressures from the other four areas are changing how we look at solving data and metadata integration.

The journey began with the introduction of the data warehouse as a separate store for integrating operational data. From the warehouse, analysts could create single, historical views of customer, product and asset data. Businesses could analyze activity over time according to different views, or dimensions. Tools and techniques that account for and interpret "slowing changing dimensions" are critical to gaining value from data warehouses but also are challenging from a data management, integration and reporting perspective.

Dimensional data (that which is specifically about customers, products or other objects of interest) housed in BI and data warehousing systems has in many cases become the "master" data, integrated from multiple systems, that users rely on to measure, report and analyze performance across their organizations. The dimensional data also offers an important resource for other levels of integration.

Data integration tools and techniques are essential, because operational systems have sprouted up on all platforms. Duplicated information and functionality through fractured subsets of operational data spread across these systems has led to inefficiency and high costs.

In recent years, organizations have employed EAI and asynchronous message queuing products from IBM, Sonic Software, Tibco Software and others to synchronize operational data, with some success. However, systems without proper APIs haven't been able to play, which has meant that batch file update processing goes on anyway.

The Web has given business executives a glimmer of hope that simplicity may be attainable. If the Web can allow access to common processes, application services and information through a browser, organizations may not need to support multiple interfaces and versions, not to mention all the duplicate functionality and fractured operational data versions.

Is it possible to share common operational data across applications and access common services using the Web? The answer is key for organizations trying to link systems for end-to-end BPM. Processes require monitoring, spurring demand for integrated, event-driven data, which in many cases must be gathered and/or disseminated in real time.

Business integration convergence also encompasses content management. Organizations must bring together information from unstructured documents, Web content, e-mail and other collaborative content, as well as other media. Established enterprise content management systems vendors and innovators such as Kapow Technologies and Vamosa are using Web standards and platforms to provide fuller access to data and let users manipulate it from all sources.

So, activities in all five levels of business integration require a mix of data and information integration solutions:

  • Metadata repositories to hold a shared business vocabulary (SBV), such as data names, definitions and integrity rules
  • Metadata discovery and mapping services to identify and map differing data definitions to the SBV
  • Data modeling services
  • Data quality profiling and cleansing services
  • Batch and event-driven ETL for data consolidation
  • EII for data federation and replication across heterogeneous systems
  • Integration with message brokers and ESB systems for data synchronization

No longer does the subject of data and information integration begin and end with ETL and data warehousing. As business objectives change the landscape of applications and services, information integration must evolve.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
1 of 3
Comment  | 
Print  | 
More Insights
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

Becoming a Self-Taught Cybersecurity Pro
Jessica Davis, Senior Editor, Enterprise Apps,  6/9/2021
Ancestry's DevOps Strategy to Control Its CI/CD Pipeline
Joao-Pierre S. Ruth, Senior Writer,  6/4/2021
IT Leadership: 10 Ways to Unleash Enterprise Innovation
Lisa Morgan, Freelance Writer,  6/8/2021
White Papers
Register for InformationWeek Newsletters
Current Issue
Planning Your Digital Transformation Roadmap
Download this report to learn about the latest technologies and best practices or ensuring a successful transition from outdated business transformation tactics.
Flash Poll