Along with globalization and integrity, information integration and systems consolidation of are driving forces in 2004 -- but for new reasons. For BI/DW, 2003 was the Cover-Your-Rear Year, during which operational cost savings, compliance monitoring, risk assessment, and fraud detection were key trends. That experience revitalized the old realization that the enterprise must manage information and systems as intellectual assets that are just as important as patents and trade secrets. In other words, not only must you protect these assets, you must also invest in them so that they can better contribute to corporate wealth.
A good example is "super vigilance," a phrase coined by independent consultant Frank Teklitz. "Super vigilance is practiced every day in the airline industry. It is what keeps airplanes in the sky." This involves comprehensive and continuous information about a key business process (such as airplane maintenance) beyond just a few dimensions to, instead, "a disco ball reflecting thousands of facets."
During 2003, we all heard much discussion about real-time (or "right-time") data warehousing. During 2004, this discussion will go deeper to focus on subtransactional data warehousing. Sybase's Paul Krneta has observed that atomic-level data (which captures a single transaction as a row) is too coarse for today's super-vigilant applications. For example, shipping a critical package overnight was once considered a single transaction. Now, we can track that package in detail at every point where it is touched. This expands the transaction into multiple sub-transactions, all of which should be managed by a data warehouse.
Another example is "enterprise memory": that is, the concept of recording, into a single store, all data over all time at all granularity about the entire enterprise. Storage has become essentially free; but, as Microsoft's Jim Gray observes, "It's free like a puppy!" To realize any business benefit, you must organize and structure data according to key business concepts and objectives. A common schema must be a template for the data to give each item a context. Gray is working on the World-Wide Telescope project, which is collecting all astronomy data into a single store. The project highlights the challenge of sharing complex data sets. Participants have created more than 1,500 context descriptors to describe the taxonomy of astronomy data adequately.
What to do? Invest in infrastructure that protects all intellectual assets of your enterprise and focuses these assets on generating revenue and controlling expenses. In particular, give more priority to better managing data warehouse metadata so that users, applications, and global systems can understand the rich complexity of your business. Metadata is not static; within the enterprise memory, metadata has as much volatility as the data itself.
This year and beyond, we must nurture corporate wealth broadly while rethinking many basic assumptions about BI/DW. Globalization, integration, and integrity are driving forces. Focus on the challenges they present and you will ensure that IT does indeed "matter."
Guest columnist Richard Hackathorn, [[email protected]] is president and founder of Bolder Technology. He has more than 30 years of experience in the IT industry as an innovator, educator, and author. His most recent book is Web Farming for the Data Warehouse (Morgan Kaufmann, 1998).
The author appreciates insights gained in interviews with the following people: Robert Bolds, Computer Associates; Andrew Cardno, Compudigm; Darren Cunningham, Business Objects; Kerry Gilger, Identitech; Dan Graham, IBM; Jim Gray, Microsoft Research; Dave Henry, TemTec USA; Claudia Imhoff, Intelligent Solutions; Paul Krneta, Sybase; Justin Langseth, Claraview; William McKnight, McKnight Associates; Diaz Nesamoney, Celequest; Sanjay Poonen, Informatica; Bill Prentice, SAS; Neil Raden, Hired Brains; Chris Sherman, Searchwise; Kim Stanick, NCR/Teradata; Rob Stephens, SAS; Frank Teklitz, Inmon Data Systems; John Ulery, Computer Associates; Ian Walsh, Approva; Colin White, BI Research; and Bob Zurek, Ascential Software.