Software // Enterprise Applications
06:09 PM
Rick Whiting
Rick Whiting

Big Databases Are Big Challenges

As databases bulk up, tasks such as administering access and managing data integrity become more difficult

The size and workload of transaction-processing and decision-support databases have increased rapidly--doubling in some cases--in the last two years, according to a Winter Corp. survey of the operators of the world's largest, most-heavily used database systems. That continuing growth poses challenges for those charged with managing these behemoths.

At peak workloads, the U.S. Bureau of Customs and Border Protection's database processes 51,448 transactions per second, nearly twice its peak of 26,655 TPS in 2001. It has the distinction of being the hardest-working transactional database in the world, according to survey results released earlier this month.

Experian Marketing Services Inc. has the busiest decision-support database, handling 887 simultaneous queries at its peak workload.

"Transactions and queries are getting richer and more complex as time goes on," says Richard Winter, president of market research and consulting firm Winter Corp. Companies are making more use of data for cross-selling, and the number of transactions is increasing as more business is conducted in real time. Rising workloads are a symptom of success for decision-support data warehouses, Winter says, "because it means [businesses] are getting more value out of them."

The survey, conducted in September, also identified the largest decision-support and online transaction databases in the world. France Telecom topped the list of big decision-support databases with its 29.2-terabyte data warehouse, followed by AT&T and SBC Communications. Land Registry, a U.K. government agency, has the largest transaction-processing database with its 18.3-terabyte system, followed by telecom firm BT plc and United Parcel Service.

Top 10 databases for online transaction processing, in transactions per second
Company Peak workload (TPS) Vendor
Bureau of Customs and Border Protection 51,448 Computer Associates
Anonymous 4,010 Microsoft
Internet Auction 3,634 Microsoft
Commander Communications 2,604 Microsoft
Scottish and Southern Energy 2,152 Microsoft
2001Outlet 1,204 Microsoft
Anonymous 1,200 Microsoft
Caixa EconÙmica Federal 1,133 Computer Associates
CheckFree 739 IBM
Land Registry 702 IBM
Data: Winter Corp.

As databases get larger and their workloads increase, managing them becomes more difficult. The job demands administering access for tens of thousands of users, maintaining data integrity, and managing the thousands of disk drives that actually store the data. Vendors have been scrambling to develop software to automate more of these chores, but Winter believes database growth may be outstripping those capabilities. "It's a huge management challenge," he says.

Winter, who provides database-consulting services to government organizations, including intelligence agencies, declined to say why the database at the Customs Bureau grew so quickly. But with increased scrutiny of people leaving and entering the country in the wake of the Sept. 11, 2001, attacks, the reasons may be obvious. The Customs Bureau, part of the U.S. Department of Homeland Security, has a transaction-processing system based on an Advantage CA-Datacom database from Computer Associates running on IBM eServer zSeries hardware.

Experian's decision-support system is based on an Oracle database running on Sun Microsystems Sun Fire servers running the Solaris operating system. Almost all of the decision-support databases with the largest peak workloads run on Unix systems, according to the survey. But six of the 10 transaction databases with the largest peak workloads run on Microsoft's SQL Server database (see chart). SQL Server was once considered fit only as a departmental database.

The survey provides a good snapshot of the current state of database usage, Winter says. But the survey is voluntary, and some companies known to have huge database systems--Wal-Mart Stores Inc. being one example--don't participate.

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
Building A Mobile Business Mindset
Building A Mobile Business Mindset
Among 688 respondents, 46% have deployed mobile apps, with an additional 24% planning to in the next year. Soon all apps will look like mobile apps and it's past time for those with no plans to get cracking.
Register for InformationWeek Newsletters
White Papers
Current Issue
Top IT Trends to Watch in Financial Services
IT pros at banks, investment houses, insurance companies, and other financial services organizations are focused on a range of issues, from peer-to-peer lending to cybersecurity to performance, agility, and compliance. It all matters.
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on for the week of July 17, 2016. We'll be talking with the editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.