Software // Enterprise Applications
News
2/19/2004
06:09 PM
Rick Whiting
Rick Whiting
Features
Connect Directly
RSS
E-Mail
50%
50%
Repost This

Big Databases Are Big Challenges

As databases bulk up, tasks such as administering access and managing data integrity become more difficult

The size and workload of transaction-processing and decision-support databases have increased rapidly--doubling in some cases--in the last two years, according to a Winter Corp. survey of the operators of the world's largest, most-heavily used database systems. That continuing growth poses challenges for those charged with managing these behemoths.

At peak workloads, the U.S. Bureau of Customs and Border Protection's database processes 51,448 transactions per second, nearly twice its peak of 26,655 TPS in 2001. It has the distinction of being the hardest-working transactional database in the world, according to survey results released earlier this month.

Experian Marketing Services Inc. has the busiest decision-support database, handling 887 simultaneous queries at its peak workload.

"Transactions and queries are getting richer and more complex as time goes on," says Richard Winter, president of market research and consulting firm Winter Corp. Companies are making more use of data for cross-selling, and the number of transactions is increasing as more business is conducted in real time. Rising workloads are a symptom of success for decision-support data warehouses, Winter says, "because it means [businesses] are getting more value out of them."

The survey, conducted in September, also identified the largest decision-support and online transaction databases in the world. France Telecom topped the list of big decision-support databases with its 29.2-terabyte data warehouse, followed by AT&T and SBC Communications. Land Registry, a U.K. government agency, has the largest transaction-processing database with its 18.3-terabyte system, followed by telecom firm BT plc and United Parcel Service.

HARD-WORKING DATABASES
Top 10 databases for online transaction processing, in transactions per second
Company Peak workload (TPS) Vendor
Bureau of Customs and Border Protection 51,448 Computer Associates
Anonymous 4,010 Microsoft
Internet Auction 3,634 Microsoft
Commander Communications 2,604 Microsoft
Scottish and Southern Energy 2,152 Microsoft
2001Outlet 1,204 Microsoft
Anonymous 1,200 Microsoft
Caixa EconÙmica Federal 1,133 Computer Associates
CheckFree 739 IBM
Land Registry 702 IBM
Data: Winter Corp.

As databases get larger and their workloads increase, managing them becomes more difficult. The job demands administering access for tens of thousands of users, maintaining data integrity, and managing the thousands of disk drives that actually store the data. Vendors have been scrambling to develop software to automate more of these chores, but Winter believes database growth may be outstripping those capabilities. "It's a huge management challenge," he says.

Winter, who provides database-consulting services to government organizations, including intelligence agencies, declined to say why the database at the Customs Bureau grew so quickly. But with increased scrutiny of people leaving and entering the country in the wake of the Sept. 11, 2001, attacks, the reasons may be obvious. The Customs Bureau, part of the U.S. Department of Homeland Security, has a transaction-processing system based on an Advantage CA-Datacom database from Computer Associates running on IBM eServer zSeries hardware.

Experian's decision-support system is based on an Oracle database running on Sun Microsystems Sun Fire servers running the Solaris operating system. Almost all of the decision-support databases with the largest peak workloads run on Unix systems, according to the survey. But six of the 10 transaction databases with the largest peak workloads run on Microsoft's SQL Server database (see chart). SQL Server was once considered fit only as a departmental database.

The survey provides a good snapshot of the current state of database usage, Winter says. But the survey is voluntary, and some companies known to have huge database systems--Wal-Mart Stores Inc. being one example--don't participate.

Comment  | 
Print  | 
More Insights
Building A Mobile Business Mindset
Building A Mobile Business Mindset
Among 688 respondents, 46% have deployed mobile apps, with an additional 24% planning to in the next year. Soon all apps will look like mobile apps and it's past time for those with no plans to get cracking.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Elite 100 - 2014
Our InformationWeek Elite 100 issue -- our 26th ranking of technology innovators -- shines a spotlight on businesses that are succeeding because of their digital strategies. We take a close at look at the top five companies in this year's ranking and the eight winners of our Business Innovation awards, and offer 20 great ideas that you can use in your company. We also provide a ranked list of our Elite 100 innovators.
Video
Slideshows
Twitter Feed
Audio Interviews
Archived Audio Interviews
GE is a leader in combining connected devices and advanced analytics in pursuit of practical goals like less downtime, lower operating costs, and higher throughput. At GIO Power & Water, CIO Jim Fowler is part of the team exploring how to apply these techniques to some of the world's essential infrastructure, from power plants to water treatment systems. Join us, and bring your questions, as we talk about what's ahead.