InformationWeek

November 10, 1997

Transaction Time

Computer-based transactions are hitting new heights. Can your enterprise handle the load?

Discuss this story in InformationWeek Online Shop Talk

he recent record-breaking trading volumes on Wall Street are serving as a wake-up call to businesses in many industries about their transaction-processing capacity. A booming economy, market deregulation, and corporate acquisitions are driving computer-based transactions at many companies to new, unpredictable heights. And in an electronic society where cash withdrawals, retail purchases, and airline bookings are now just a few key strokes away for many consumers, IT departments could quickly find themselves suffering transaction overload.

Is your enterprise prepared?
The trading surge, experts say, is a reminder that sudden spikes in transaction volumes can strain most any company's computer systems. Securities exchanges and brokerages were moving quickly last week to bolster their transaction-processing systems in the wake of the billion-plus stock-trading volumes on Oct. 28, the busiest trading day in Wall Street history.

Nasdaq, the electronic stock exchange, intends to upgrade to a Unisys IX4800 mainframe next month, and plans to expand its battery of Tandem Computers Himalaya K20000 servers. These moves will let the exchange handle 1,000 transactions per second at peak capacity-or 30% more than its busiest period on Oct. 28. The policy of the New York Stock Exchange is to have five times the capacity normally needed to accommodate peaks. The exchange's support for 500 messages per second (MPS) is adequate for now, says operations and technology VP Bob Britz, but the NYSE had already planned to raise capacity to 750 MPS by the middle of next year and is evaluating a plan to go to 1,000 MPS.

Among brokerage firms, Merrill Lynch & Co. has rushed a standing order for a mainframe that will boost its ability to process transactions by a third. Financial firms that do business internationally could see another surge in transaction volumes next month, when the Federal Reserve Bank increases the number of hours it processes cross-border electronic funds transfers. "Volumes are growing very quickly," says Ed Goldberg, executive VP of operations, systems, and technology at Merrill Lynch. "It won't take another five years to get a 2-billion-share day."

But the escalation of transactions isn't just a financial industry challenge. On Galileo International's computer airline-reservation system, transactions are growing at more than 10% a year, spurring the company to start retiring its bipolar mainframes in favor of IBM's CMOS machines. Galileo runs IBM's Transaction Processing Facility, specialized software for high-end transaction processing, on most of its 15 mainframes. This week, the company will announce that it's using IBM's Visual Age TPF for Windows NT, a new development environment designed to improve the productivity of its 400 TPF programmers and provide Web access to the reservation system.

Telecom-related transactions are also soaring, as deregulation pushes carriers to become one-stop providers of local, long-distance, cellular, and other network services. AT&T, whose call volume is growing 10% a year, estimates that it handles 240 million voice calls on an average day-with each call requiring 10 to 20 transactions as it gets routed over the carrier's network. In addition, each call triggers a transaction against AT&T's billing database.

The carrier hopes to put procedures in place to minimize the call-routing transactions, says Dan Sheinbein, AT&T's VP of network architecture, but he adds: "As people create more and more bundled, complex services, the back-office transactions will increase ."

Visa International is upgrading its mainframe systems to support credit-card transaction volumes that are growing 28% a year-and must be processed in an instant, says Mike Sandlin, senior VP of transaction switching services with Visa, in Foster City, Calif. Visa, which runs IBM's TPF, has already rewritten applications so they will run in a clustered mainframe environment. The switch from monolithic bipolar mainframes to clustered CMOS hardware will begin early next year.

Merger activity is also driving transactions higher at many companies. Pharmacy chain Rite-Aid Corp., for instance, expects its peak volumes to grow from 475 transactions per second to 530 per second as it acquires its way from 3,000 to nearly 4,000 stores. The load is handled by four IBM S/390 mainframes, running IBM's DB2 database, CICS transaction-processing software, and Parallel Sysplex clustering technology. On an average day, Rite-Aid processes between 15 million and 22 million transactions, each of which entails several database actions. All told, the system executes about 700 million SQL calls to the database each day.

Capacity Problems
Where is this growth in transaction volumes leading? To big problems, in some cases.

Merrill Lynch was caught off guard on Oct. 28-and will end up paying for it. About 3,000 of the 50,000 Nasdaq trades it handled that day weren't executed promptly, and the firm expects to pay up to $10 million to compensate individual customers. Merrill Lynch says the problem arose partly because quotes are posted at the same workstations where trades are executed, creating a bottleneck.

Other brokerage firms that do business online, including Charles Schwab & Co. and ETrade Securities Inc., saw their systems choke on the trading volume, leaving anxious customers unable to retrieve quotes or conduct trades. Both companies quickly sent apologies to customers-and both promised to increase their capacity for simultaneous online users from 10,000 to 25,000 by year's end

Inadequate tr ansaction-processing systems are partly to blame for cash-flow problems and a third-quarter loss at fast-growing Oxford Health Plans Inc., costing the chief financial officer his job last week and leading to an IS reshuffling at the Norwalk, Conn., managed-care company. Oxford last week named a new VP of operations, and it has retained IS consultants to review the shortcomings in the company's processing of claims to support its growing membership.

The lesson for IS departments: Be prepared. Lori Hricik, senior VP of technology and operations with the treasury and global trade finance unit of Chase Manhattan Bank, notes that transaction-processing capacity "is one thing you can control." The Chase unit, which processes more than 1 million transactions a day valued at $3 billion, is talking with Tandem about upgrading the bank unit's Himalaya servers. "We have adequate capacity, but we anticipate growth," Hricik says. "If we get to 60% of capacity, I say buy more."

Microsoft chairman Bill Gates predict ed last year that companies would eventually need systems that support a billion transactions per day. On May 20, Microsoft demonstrated a mockup cash-machine network running on 45 Compaq Computer ProLiant servers that processed 1 billion transactions. IBM claims to have demonstrated support for a billion transactions per day as well.

But few, if any, real-world computer systems process that kind of traffic. Some of the airline reservation systems come close. The Sabre reservation system, which handles more than 5,000 transactions per second during peak periods, is supporting transaction-volume growth of 10% to 20% a year, says Brad Boston, executive VP with Sabre Technology Solutions in Dallas. "We pretty much see high sustained volumes from early in the morning until early evening," he says. Volume is going up at night as well, as Sabre increasingly serves overseas travel agents.

While the New York Stock Exchange handled 1.2 billion shares on Oct. 28, the number of transactions required to process those shares was lower. During peak activity that day, the exchange processed 270 messages a second, about half its capacity of 500 MPS. While each message typically involves several database transactions, a billion transactions per day requires 11,500 transactions per second.

Even the top-performing system audited by the Transactions Processing Performance Council (TPC), a benchmarking organization in San Jose, Calif., falls far short of supporting a billion transactions per day. That system, a 44-processor Sun Microsystems Enterprise 6000 cluster running Oracle's Oracle8 database and BEA Systems' Tuxedo transaction manager, sustained throughput of about 865 transactions per second-only about 7% of the run rate needed for a billion transactions per day.

Cost Is Dropping
Still, it's only a matter of time before systems hit the billion-transactions milestone. "It's a matter of investing in the right infrastructure to support it," says Jeff Jones, program manager of data management marketin g with IBM Software Solutions.

The cost of transaction processing is dropping. While Microsoft-based systems still don't scale for heavy-duty transaction-processing applications, they offer average users a big bang for their buck. In conjunction with its server hardware partners, Microsoft is beginning to make good on promises it made last spring to deliver increasingly faster transaction rates while driving down the price of those machines. Microsoft delivered Windows NT 4.0 Enterprise Edition in September. It's expected to roll out in the next few weeks a complementary version of its database, SQL Server 6.5 Enterprise Edition, that takes advantage of NT/E's transaction-processing monitor and two-node clustering.

Using the new Microsoft software, Compaq posted a TPC-audited test in mid-October for a four-processor Pentium Pro machine priced at $433,000 and producing 11,000 transactions per minute, for a cost of $43 per TPM. Days later, Unisys beat that mark with an audited figure of 13,000 transact ions per minute at just $38 per TPM. ALR raised the bar again last week with a six-processor Pentium Pro system that handled 13,000 transactions per minute at $35 per TPM. Only six months ago, comparable performance cost nearly $1 million and it wasn't possible to come close to those numbers of transactions on Windows NT-based systems.

Bigger Systems
At the higher end, transaction-processing systems can involve huge databases and thousands of users. Database consulting firm Winter Corp. identifies four online transaction processing systems that support more than 2 terabytes of data, and two others that support more than 10,000 concurrent users, all running on IBM mainframes. Other large OLTP systems cited run on Amdahl, Digital Equipment, Hewlett-Packard, NCR, Sequent, and Siemens Pyramid computers, and on databases from Computer Associates, Digital, IBM, Informix, Oracle, NCR, and Software AG.

At its DECUS user conference last week, Digital demonstrated Galaxy, a system optimized for tran saction processing that runs both the OpenVMS operating system and Windows NT, and combines symmetric multiprocessing and clustering. Also last week, NCR announced that it's adding RSA Data Security's encryption technology to its popular Top End transaction monitor, providing a higher level of security to Top End transactions. NCR will announce next week that SunSoft plans to use Top End in its Internet products, making it possible to extend data-center-class transaction management to Web environments.

Increasingly, Web servers are taking on transaction-processing functions. Steve Ariana, VP of architecture and strategy for electronic commerce with Charles Schwab, says the brokerage is learning to apply some of the principles of mainframe transaction processing to Web commerce. But the Web, he says, is a "very different environment because there's no state. The whole user interaction is different."

Schwab has raised the basis for capacity planning for its Web-based stock trading service from 2-1/2 tim es to five times the trading volume on a normal day. "We think the aberration that happened last month is where we are going to be in 12 to 18 months," says Ariana.

The number of trades handled by Schwab online surged from its 40,000-a-day average to 90,000 on the stock market's record day. The company didn't track the total transactions-which can include buy/sell orders, requests for quotes, or account access-but transactions surely rose by a higher multiple than trades did because many transactions entail requests for quotes before customers buy or sell a stock.

Schwab, which runs Netscape Enterprise Server on IBM RS/6000 SP2 servers, is talking with IBM, Sun, and Netscape about solving its transaction-processing problems. "Some of these solutions work but they don't scale. Other things scale but they don't work for us," Ariana says. Schwab is also considering developing its own solutions.

Merrill Lynch processed 750,000 trading transactions on Oct. 28, nearly twice its normal volume. In addit ion to boosting hardware, application performance is an issue, notes Elizabeth Byrnes, a first VP for the firm's equities trading floor operations. Not all the trading applications are multithreaded or capable of processing transactions simultaneously, Byrnes explains, so "we are aggressively going after that additional capability."

Within a year, Nasdaq wants to be prepared for more than 2 billion shares a day, says Robert Bloom, VP of telecommunications. The exchange's previous record of 970 million shares was shattered Oct. 28, when it handled 1.34 billion. Government regulators have suggested planning for as many as 10 billion shares a day, but Bloom says that's overkill. "There is a point where capacity planning becomes ludicrous," he says. "You can't spend an infinite amount of money provisioning systems. You need to be conservatively reasonable."

From a business perspective, that makes sense. But it may prove difficult for IT capacity planners to remain conservative when their companies' liveli hoods depend on handling an ever-soaring number of computer-based transactions.

-with additional reporting by Bruce Caldwell , Gregory Dalton , Stuart J. Johnston , Mary E. Thyfault , Marianne Kolbasuk McGee , and Mary Hayes

Back to News in Review