3 min read

Q&A: A Practical Path to Real-Time Data Warehousing

Teradata's Stephen Brobst and GoldenGate's Alok Pareek describe an evolutionary path to real-time data warehousing and operational BI. It's an incremental journey that starts with a simple question about business process change.
You've talked about process automation and human-facing dashboards. Which approach will dominate in the real-time era?

Brobst: There's a case for both. But in the long term, the software event detectives will play a big role because of the volumes of data and complexity of the decision-making. The more you can automate, the better, in terms of efficiency and consistency.

You used the example of fraud detection, but that seems like it has been around quite awhile. What are some of the emerging examples of automated event detection?

Pareek: Detection is used in telecommunications to look out for network overloads. When they're spotted, events are triggered to reconfigure the network and balance the load. Energy companies are another example. Several leading utilities are building smart grids that detect fluctuations in supply and demand and make decisions on rerouting energy sources on the fly. In manufacturing, we've seen several cases where real-time detection of defects has been integrated with factory automation software. If you can correct a problem in real time rather than reacting half an hour or an hour after the problem crops up, it can significantly improve the productivity of the entire assembly line.

Brobst: Freescale is a great example of that. They are a spinoff of Motorola, and they do real-time analytics around quality control. They're not just looking for failures; they are spotting the quality trends before they have to shut down the assembly line and start throwing away silicon.

Another opportunity for real-time information is related to customer events. I'm talking about event detectives that look for customer defections or leads that can trigger corresponding offers. For example, Travelocity monitors continuous, real-time pricing feeds from external providers such as airlines, hotels and rental car agencies. As new terms become available, Travelocity can bundle deals that can be presented to visitors to the Web site or promoted via outbound e-mail. These prices are offered to everybody, so the faster Travelocity can react and get new offers out, the more likely it is to get the business of those customers.

What's your advice to companies that are just starting to look for real-time opportunities?

Brobst: I would encourage them to identify and partner with the business visionary who is going to lead the charge. I would go through the "business discovery" exercise of getting the key leaders in a room and asking the simple question, "If you could have the data today that you're used to seeing tomorrow, what would you do with it?" What business process changes would be required and what would the value be? If it's just a twinkling report, it's useless. But if you get the right creative people in the room, you can identify some interesting opportunities.

Is this a big-budget, long-term investment? For instance, many aging data warehouses may not be up to the rigors of real-time data, so is this a rip-and-replace proposition?

Brobst: I'd encourage evolution not revolution. You don't have to make everything real-time all at once. You can pick particular areas of data and phase in real-time information and decisioning. That said, it does assume certain fundamentals in terms of warehouse design. For example, trying to use a data warehouse that is highly de-normalized and highly star schema-oriented is more difficult than using a warehouse that has a more extensible, relational design. So you do have to have some fundamentals in place, but it should be an incremental, subject-area-by-subject-area initiative with relatively short project timeframes of 90 to 100 days. You definitely don't want a multi-year, blow-it-up-and-start-over-again approach.