Welcome Guest. | Log In| Register | Membership Benefits
News In Review

September 6, 1999

Print this story
Print this story
Coping With An E-Business Emergency

The high-stakes battle for online customers and market share has turned crisis management into a top priority

By Gregory Dalton

Illustration by John Bleck
Related links:
  • sidebar: Auction Site's Bid For High Availability

  • Network Pressure
  • July 14 starts like every other trading day at Charles Schwab & Co. The stock markets on the East Coast have just opened, and 25 IT and business staffers in the brokerage's San Francisco headquarters and Phoenix data center are on a conference call discussing how their systems are handling the first wave of orders. At 9:39 a.m. EDT, there are about 20,000 customers on Schwab's Web site, which is humming along. Then the trouble starts.

    At 9:44, a red alert flashes at the data center, indicating a problem with one of Schwab's IBM mainframes. Customers experiencing delays accessing accounts and executing trades online begin flooding the company's call center with complaints. A marketing and public relations nightmare looms.

    Following prescribed procedures, members of the Internet team run diagnostic tests using mainframe hardware and software tools. When these tests turn up nothing, team members check to see which programs were running, if there were any anomalies before the problem occurred, or if a staffer did anything unusual.

    By 9:50, they've identified three possible causes related to the software. At 9:55 a.m.--exactly 11 minutes after the first sign of trouble--the team decides to recycle some regions of the mainframe. That does the trick, and by 10 a.m., the Web site's performance is back to normal. Calls from customers begin to subside.

    Schwab's crisis-management procedures helped avert a more serious problem that day. But the brokerage learned the hard way; it refined those procedures only after its Web site went down during trading several times earlier this year, sometimes for more than an hour. Like Schwab, many companies are learning that having well-honed crisis-management procedures in place is more critical than ever in the intense battle for online customers and market share.

    E-commerce veterans Borders Online, eBay, Ticketmaster, and others, including Schwab, that have reexamined their procedures say crisis management involves well-thought-out strategies. These include striving for simplicity in information systems and carefully managing changes to those systems. Then, to react quickly when problems do occur, IT managers need to make sure each person in a department knows his or her role during a crisis and follows defined procedures for notifying people up the chain of command. Informing customers of problems early and often is also important.

    The stakes are high. At Ticketmaster, for instance, 13% of all sales of concert tickets in the second quarter were made through the Internet, up from about 8% during the previous three months. In some cases, as much as 35% of all tickets for a performance are sold online. "We can't afford to have a glitch," says John Pleasants, president of Ticketmaster.com. "We would lose consumer confidence. We would lose market share."

    Customers have high expectations. They can easily jump to a competitor if a company's site goes down. And if that company has built a very popular site, outages can lead to scathing attacks on Internet bulletin boards, online news articles--and more than likely, a sharp fall in the company's stock price. All that can happen within a matter of hours, putting extreme pressure on IT departments.

    continued...page 2, 3, 4

    Illustration by John Bleck


    Back to This Week's Issue

    Send Us Your Feedback

    Top of the Page