Scaling The Data Warehouse

ULTIMATE SCALABILITY
The new technology trend designed to deal with multidimensional data warehouse growth is toward highly parallel architectures. The HP Oracle Exadata Storage Server, announced last month, is designed to keep data flowing to and from more disks at once, increasing the pace at which I/O-intensive tasks can be performed. And Microsoft has just revealed that it will incorporate the DATAllegro technology acquired earlier this year into the next release of SQL Server, thereby increasing both I/O bandwidth and processor parallelism. Almost everyone is moving to exploit lower-cost hardware. Though big symmetric multiprocessor servers aren't about to disappear, there's an ever greater emphasis on scale-out architectures.

In the 1990s, conventional wisdom had it that massively parallel processing would never be more than a niche architecture, used for extreme requirements at the margins. But MPP has become reliable, manageable, and affordable--and suddenly it seems that nearly everyone is hungry for scalability. So highly parallel architectures--whether you call them MPP, cluster, or something else--have become part of the mainstream.

NEED ADVICE ON GETTING MORE FROM YOUR DATA?

Bookmark our sister site:
intelligententerprise.com

A lot of data warehouse practitioners are struggling with the changes brought on by rising data warehouse scale and rapidly evolving architectures. The most important thing to remember is that business problems aren't solved by buying new hardware or introducing new architectures. They're solved by determining the requirements of a solution and then implementing systems that meet those requirements.

To do that, follow these three recommendations in any data warehouse development project: Introduce a systematic management process to deal with the scalability problems. Avoid the seven gotchas of scalability management. Emphasize quantitative requirements and use measurements or tests at every stage of the development life cycle. With a systematic approach, you will meet business expectations and have a scalable data warehouse with long-term business value.

Richard Winter is the president and founder of WinterCorp, a consulting firm focused on large-scale data management. In addition to advising companies in industries including retail, health care, financial services, and distribution, WinterCorp provides consulting services to vendors including Hewlett-Packard, IBM, Microsoft, Netezza, Oracle, and Teradata.

Illustration by Sek Leung

Continue to the sidebars:
Microsoft And Oracle Are Scaling Out,
EBay Turns To Analytics As A Service
and
7 Gotchas That Wreck Data Warehouse Scalability