The BI Gap in Moore's Law, SOA and DB Performance

You can't swing a dead cat by the tail in this industry and not hit a story about exploding data volumes, service-oriented architecture (SOA), pervasive/operational BI and software-as-a-service (SaaS). Moore's Law is supposed to handle that first one, but can it really? And what about the others? Are we really ready for them? Moore's Law, it's true, has driven the cost of computer hardware down, relatively speaking, but in one area, hot storage, there is a monster under the bed. While the density of disk drives has doubled every eighteen months or so, and the price per megabyte (really, of gigabyte now) has followed a similar pattern, there is a component of disk drives that is not electronic and doesn't follow the same trend. The actual data transfer rates have not been improving at a rate anywhere close to the increases in drive capacities. So we may have bigger, cheaper drives, but it doesn't mean that we can read or write more data any faster, or, at least not at the dizzying rate of increase of CPU's, memory and disk drive platters. That's the first problem.SOA and SaaS are wonderful innovations, there is no question about it. The problem, though, is how will they affect data warehousing and BI? Having loosely coupled services that are components of applications, that can be located through open directory services and accessed through open standards will ultimately provide the tools for all sorts of new and innovative applications that weren't possible before. It's likely they will be easier and less expensive to deploy and maintain. That's the promise and I more or less believe it, just without the breathless enthusiasm of some. There will be speed bumps along the way, rest assured.

But here is the rub. SOA and the standards that go around it were not invented for BI. They were invented for e-commerce and transactional processing. It's one thing to ship a url or a 141-digit credit card transaction string or even 50k of XML around the network, but SOAP was not designed to handle shipping a 10 GB result set from one service to another. The whole idea of loosely coupled begs the question, "Where is the data?" If there are three different services, each one optimized to perform a certain kind of function for analytical work, how do we move these rapidly exploding amounts of data from one to the other? Consider Operational BI (a term which is highly ambiguous). Assume there is a data warehouse that already provides the information for reporting and analysis, OLAP and performance management. Operational BI will jack up the number of users, change the profile of usage and really mix up the numbers and types and queries the database has to service. Is it really likely that this data warehouse will become "virtual," with bits and pieces of the data spread across the landscape, owned by numerous and sundry services? Will all of the different kinds of queries be satisfied by spreading the work across all of these presumably asymmetric processors? Will there be effective orchestration to manage the federation of the queries to provide the kind of service that the applications require? I doubt it. I was ready to write data warehousing off not long ago, but I think it's just heading in a new direction supporting operational processes and the growing disparity and externalization of business. Relational database vendors have a pretty mixed record when it comes to supporting BI, but they are the only product group that has the R&D funds to ramp up performance. Improvements in optimizer technology for BI over the past decade are impressive, but that's only one piece of the puzzle, albeit an important piece. I would pose this question to the vendors, though: If you are going to support business decision making, which going forward includes decision automation and the implementation of predictive models, when will you be able to implement it in your own products? When will workload managers develop enough intelligence to be able to handle all that the industry is promising for the next few years?

Neil Raden is the founder of Hired Brains, providers of consulting, research and analysis in Business Intelligence, Performance Management, real-time analytics and information/semantic integration. Neil is co-author of the just-released book "Smart Enough Systems," with business rules expert James Taylor.Here is the rub. SOA and the standards that go around it were not invented for BI. It's one thing to ship a url or a 141-digit credit card transaction string or even 50k of XML around the network, but SOAP was not designed to handle shipping a 10 GB result set from one service to another. The whole idea of loosely coupled begs the question, "Where is the data?"