The Making of a Real-Time Hero 2

Data warehousing has prospered in support of strategic decision-making. Now, as business intelligence expands, the world looks to the sky, not for a bird or a plane - but for a data warehouse that meets real-time, tactical demands.

Passivity is Passe

Within the data warehouse edifice itself, however, changes are afoot to stamp out the latency nemesis. Availability must increase for updating and query activity. Again, middleware, federation, and EII are shaping up as answers — ones championed by IBM, among others. Oracle has traditionally focused on greater centralization to reduce the number of sources and the technology required to make distributed queries work. The balance desired between strategic and tactical data warehouse activity will likely dictate whether a federated or centralized approach is appropriate.

Richard Winter, president of Winter Corp., along with Stephen Brobst, now chief technology officer with NCR's Teradata Division, developed the concept of "active" data warehousing, which has become an important part of Teradata's vision. "The active data warehouse addresses a type of automated decision-making that has rarely been accomplished in the past," Winter wrote in the Fall 2000 Teradata Review. "It is too complex and far reaching in its data needs to fit within the picture for operational systems; it is too time sensitive to have been attempted in most data warehouses. And yet it is emerging as the critical factor in enterprise performance as e-business takes hold and pervasive computing promises to speed the pace of many businesses even further."

"As an engineer by trade, I can't stand this 'real-time' expression, even though I get forced to use it," Brobst says now. "By engineering standards of flight control systems and so forth, we're not really talking about 'real time.' I prefer the term 'right time' because what we're talking about is aligning business processes with data availability, freshness, performance, and delivery. The concept of an active data warehouse is broader than just 'real time.' As opposed to passive reporting, an active data warehouse is actively involved in business decision-making. That might mean the user needs data within two seconds; or if 20 minutes is appropriate, that's fine, too. We shouldn't be driving what we do by technology, but by business processes."

"Plus, when we talk about real time, there are really two types," Brobst continued. "The first is where you need information provided to tactical decision-support applications in real time, but the data doesn't have to be updated in real time. A good example would be a call center such as BCP, one of the largest communications providers in Brazil. BCP's system will retrieve call detail records and perform analysis so that customer representatives can offer the best plan. The data doesn't have to be calls you made in the last 13 minutes; the real-time response is about getting data through the scoring algorithms and other analysis."

Brobst said that the second type of real time data warehousing is where the data itself is as up-to-date as possible. "But obviously, it doesn't make any difference if I can't deliver the data fast enough. If the data is only two seconds old but it takes me longer than two seconds to deliver the data, that's not very interesting. The big issue becomes cost justification. Is it worth the technology that you need to get that last two minutes of data?"

Middleware technology is changing how companies assess whether they should try to achieve real-time data warehousing for tactical users. "But middleware is an enabler, not a driver," Brobst said. "During the early adopter phase of active data warehousing, unless you were Wal-Mart, or some other gargantuan company, you couldn't afford to build the middleware yourself. Now, we're moving beyond this early phase because tools are coming on the market that will enable a greater number of companies to achieve active data warehousing goals. When middleware was largely file-based, it was totally inappropriate for active data warehousing. Now, we're seeing companies such as Ab Initio produce stream-based tools, which are highly useful."

Strategic and Tactical

Transaction databases and data warehouses — whether for strategic or tactical decision-making — will always be tough to support within one system because they serve different purposes. "The data warehouse will never replace transactional bookkeeping — that's the golden copy," said Brobst. However, the technology goal of most data warehouse solution providers is to support a mixed workload of strategic and tactical decision-making.

By developing an intelligent I/O subsystem and query optimizer that can determine whether the nature of the query is strategic or tactical — and then adjust how the database responds internally — Teradata has produced "the only commercial databases that can handle a mixed data warehouse workload," Brobst asserts. The company is working on enhancing the current release with the ability to do data mining and run scoring models that reveal "propensity to buy," for example, and send the results to assist in tactical decisions.

Discussions with user organizations and leading technology providers reveal the sense that an expanding universe of BI and data warehousing applications is about to unfold. Tactical data warehousing applications will encounter "green field" opportunities in many organizations, especially those frustrated by the limitations of current call center and CRM applications. "Real time" may remain in the eye of the beholder: but if the data warehouse can save the day by delivering higher efficiency and more profitable customer relationships, it will indeed be a superhero.

David Stodder [[email protected]] is the Editorial Director and Editor-in-Chief of Intelligent Enterprise.