InformationWeek.com

Learn IT strategies from the 500 most innovative companies in North America. Get the InformationWeek 500 Analytics Report FREE today!

Welcome Guest. | Log In| Register | Membership Benefits

September 6, 1999

Print this story

Coping With An E-Business Emergency

continued....page 3 of 4

Illustration by John Bleck

Related links:

sidebar: Auction Site's Bid For High Availability

Network Pressure
Complex changes rank high because they're more likely to cause problems that are difficult to fix. When making changes, IT staffers try to evaluate how many variables they'd have to look at if something went wrong. "The more variables you have, the more time-consuming it is to resolve the problem," Hier-King says.

Every change also has defined back-out procedures, so Schwab can quickly revert to doing things the old way if necessary. Fifteen people from various parts of the IT organization are involved in a change-management review process that approves or denies all modifications to the company's systems. When a change is approved, it's implemented around 8 p.m. Pacific time, and all members of the IT department involved in that particular change are required to be on hand in case something goes wrong.

IT departments also have to establish lines of communication with a company's other departments. Borders Online discovered that last spring, when it E-mailed tens of thousands of $5 gift certificates to customers on a Friday--hours before Borders' Web site was scheduled to go down for maintenance. About 5% of the customers tried to redeem the coupons overnight, but couldn't get through to the site, creating an embarrassing situation for the fledgling online bookseller.

"That was a breakdown between marketing and IT," Vanzura says. "It doesn't sound like rocket science, but there's so much going on at a major Web site that it's pretty easy to have something fall through the cracks unless you're methodical." To make sure that fiasco wasn't repeated, Vanzura started holding meetings on Monday mornings with the heads of marketing, content acquisition and management, customer service, production, and public relations.

A crisis can occur at any time--Schwab experienced another hourlong outage Aug. 30, due to a problem with its storage systems--so it's important that each member of an IT department knows what to do when the unexpected happens. Hier-King says one reason Schwab was able to handle the July 14 crisis was that every IT person knew how to respond. "Everybody knows precisely what his or her role is," she says. "Before it was more of an all-hands-on-deck mentality." Some system administrators now look at the operational costs of the hardware. Others are assigned to look at system software and network configurations, session counts on the Web site, load-balancing software, security, and incoming network lines.

Communication within the IT department is critical when a crisis occurs. "You're always going to run into problems," says Barnesandnoble.com CIO Gary King. "The activities that you need to go through there are built around key escalation paths and key contact paths."

Escalation paths are rules about who in the company gets contacted and when. Since Web sites operate around-the-clock, that means having three shifts for the IT department and clearly defined communication procedures. Barnesand- noble.com encountered a problem earlier this summer when air-conditioning units in its New York data center went down one Sunday around 1 a.m. during an intense heat wave. The staff shut down some development servers to reduce the heat being generated in the facility and prepared to send some processing work to a second site in Virginia. King was notified Sunday morning and arrived at 9 a.m. to manage the problem, which was ranked a Level Four in its escalation path.

"Our integrity wasn't compromised, but we were running at a higher risk level," King says. "I need to know that." In this case, the hierarchy worked. King says time sensitivity is key in the escalation processes. At Barnesandnoble.com, a problem is raised in 15 minutes from Level One to Level Two, which may mean calling an IT manager at home, or bringing in a vendor to fix the system. Level Three occurs after two hours and involves Barnesandnoble.com's VP of operations, while Level Four is notifying the CIO within five hours of the original problem. At Level Five, the CEO is alerted.

continued...page 4
return to page 1, 2