The Myth and the Magic of Enterprise Information Integration

As the relatively new boy in town, enterprise information integration (EII) has attracted some attention. He's powerful and attractive: so quite a few of the young ladies are keen to be seen with him. By the same token, some of the older guys see him as a definite threat.

Among these guys, the more traditional data warehousing experts and vendors have been lining up to take a few pot shots: What about data integrity? Won't this impact operational systems performance? How will it cope with missing data? they ask. Isn't this just the Virtual Data Warehouse come back to haunt us?

Well, clearly these guys are not your typical playground bullies — their questions do point at genuine concerns. Enterprise information integration does indeed allow direct, on demand access to data, wherever it lies. As a result, one could encounter the problems they describe. But let's stand back a moment. There is an urban myth at play here. Those issues would arise only if EII was being used in place of a data warehouse. Now, there may be a few vendors who are promoting EII as a replacement for data warehousing, but those who know what they’re talking about see EII as extending the data warehouse — not replacing it.

So, this is what the young ladies see in EII. If we already have a data warehouse — or some other repository of reconciled, integrated data, such as an operational data store (ODS) — then we can use that store to ensure data integrity, provide the historical view, and then branch out into some interesting and valuable new areas.

Here are a number of ways that EII can extend the data warehouse and bring significant and early business value:

In short, EII is a vital component of the real-time enterprise. This is the magic! It allows us to get at extensive and significant information that really has no place in the data warehouse. Now we can avoid bloating our warehouses with unnecessary data. And we can get instant access to the data we really need. But EII does not do away with the warehouse, and it does not replace it: it augments and extends the warehouse. Of course, EII may have an impact on the operational systems. But what we need to recognize is that as business moves increasingly to 24x7, the impact of the traditional Extract, Transform and Load (ETL) approach also becomes an issue. In either case, IT has to find ways to manage in this new world.

And when it comes to data quality, history, and similar issues, it still remains the responsibility of the data warehouse, ODS, or other data reconciliation approach to fix those problems. In fact, the data warehouse or equivalent is the key source used by EII to obtain cleansed and reconciled data and as the basis for creating those cross-system queries.

The question is not about EII vs. data warehousing at all, but rather how can we combine the strengths of both enterprise information integration and data warehousing for maximum business value? Personally, I think the young ladies have got it right! After all, which would you choose — the myth or the magic?

Dr. Barry Devlin was responsible for the definition of IBMs warehouse architecture in the mid 1980s and is a widely respected consultant and lecturer on this and related topics, and author of a comprehensive book on the subject "Data Warehouse from Architecture to Implementation." Barry currently works on DB2 Information Integrator, where he is part of the team defining IBM's information integration architecture as well as industry solutions and applications of the technology. Barry is 20 years in the IT business, mainly with IBM's Software and Solutions Centre in Dublin. He is a Council Member of the IBM Academy of Technology and an IBM Distinguished Engineer.