Big Databases Are Big Challenges

The size and workload of transaction-processing and decision-support databases have increased rapidly--doubling in some cases--in the last two years, according to a Winter Corp. survey of the operators of the world's largest, most-heavily used database systems. That continuing growth poses challenges for those charged with managing these behemoths.

At peak workloads, the U.S. Bureau of Customs and Border Protection's database processes 51,448 transactions per second, nearly twice its peak of 26,655 TPS in 2001. It has the distinction of being the hardest-working transactional database in the world, according to survey results released earlier this month.

Experian Marketing Services Inc. has the busiest decision-support database, handling 887 simultaneous queries at its peak workload.

"Transactions and queries are getting richer and more complex as time goes on," says Richard Winter, president of market research and consulting firm Winter Corp. Companies are making more use of data for cross-selling, and the number of transactions is increasing as more business is conducted in real time. Rising workloads are a symptom of success for decision-support data warehouses, Winter says, "because it means [businesses] are getting more value out of them."

The survey, conducted in September, also identified the largest decision-support and online transaction databases in the world. France Telecom topped the list of big decision-support databases with its 29.2-terabyte data warehouse, followed by AT&T and SBC Communications. Land Registry, a U.K. government agency, has the largest transaction-processing database with its 18.3-terabyte system, followed by telecom firm BT plc and United Parcel Service. HARD-WORKING DATABASES Top 10 databases for online transaction processing, in transactions per second Company Peak workload (TPS) Vendor Bureau of Customs and Border Protection 51,448 Computer Associates Anonymous 4,010 Microsoft Internet Auction 3,634 Microsoft Commander Communications 2,604 Microsoft Scottish and Southern Energy 2,152 Microsoft 2001Outlet 1,204 Microsoft Anonymous 1,200 Microsoft Caixa EconÙmica Federal 1,133 Computer Associates CheckFree 739 IBM Land Registry 702 IBM Data: Winter Corp.

As databases get larger and their workloads increase, managing them becomes more difficult. The job demands administering access for tens of thousands of users, maintaining data integrity, and managing the thousands of disk drives that actually store the data. Vendors have been scrambling to develop software to automate more of these chores, but Winter believes database growth may be outstripping those capabilities. "It's a huge management challenge," he says.

Winter, who provides database-consulting services to government organizations, including intelligence agencies, declined to say why the database at the Customs Bureau grew so quickly. But with increased scrutiny of people leaving and entering the country in the wake of the Sept. 11, 2001, attacks, the reasons may be obvious. The Customs Bureau, part of the U.S. Department of Homeland Security, has a transaction-processing system based on an Advantage CA-Datacom database from Computer Associates running on IBM eServer zSeries hardware.

Experian's decision-support system is based on an Oracle database running on Sun Microsystems Sun Fire servers running the Solaris operating system. Almost all of the decision-support databases with the largest peak workloads run on Unix systems, according to the survey. But six of the 10 transaction databases with the largest peak workloads run on Microsoft's SQL Server database (see chart). SQL Server was once considered fit only as a departmental database.

The survey provides a good snapshot of the current state of database usage, Winter says. But the survey is voluntary, and some companies known to have huge database systems--Wal-Mart Stores Inc. being one example--don't participate.