Data warehouses with as much as 100 to 200 terabytes of raw data will be operational by next year, performing nearly 2,000 concurrent queries and occupying nearly 1 petabyte (1,000 terabytes) of disk space. In the same time period, transaction-processing databases will handle workloads of nearly 66,000 transactions per second.
Those predictions are in a new report from the Winter Corp. consulting firm, based on surveys of operators of some of the largest decision-support (data warehouse) and transaction-processing databases in the world. "We're moving into the area, in the next year or two, of petabyte systems," president Richard Winter says.
Despite the slowdown in IT spending since early 2001, companies have continued to invest in large-scale database projects. They say the systems raise productivity, enhance customer-relationship management and customer-satisfaction efforts, provide a single view of the business, and improve the bottom line, according to the report.
Designing such rapidly growing databases so that they are manageable and useful is a difficult challenge, especially as more databases are used for real-time applications, Winter says. While computer hardware can provide the needed processing power for giant databases, faster interconnects to move data in and out are needed, as are increasingly sophisticated database design and management tools.