Ready For Anything?

When the towers of the World Trade Center collapsed, the IT assets of many businesses-as well as, tragically, IT professionals themselves-disappeared with them. Now, as the recovery continues, the magnitude and violence of the attacks are causing many companies to re-evaluate their IT contingency plans. Some are finding that new approaches are in order.

The Pentagon suffered its own debilitating attack, but New York businesses were the hardest hit. In addition to the loss of life and office space, computers were destroyed or abandoned and telecommunications systems damaged or jammed. IBM estimates it had 1,200 customers in the immediate area of the World Trade Center. Disaster-recovery specialist Comdisco Inc. says 48 New York businesses moved into its backup facilities in Queens, N.Y., and in two New Jersey locations in the hours after the terrorist attacks.

While all New York companies were caught off guard by the immediacy and scope of the destruction, some were better prepared than others to reboot their IT systems-and ultimately their businesses. Joseph Walton, senior VP of global services at storage vendor EMC Corp., says some of the company's smaller accounts "were just put out of business."

For the most part, the critical systems of major financial institutions and other large businesses seem to have recovered quickly and according to plan, either at backup locations or after repairs and testing at affected sites. In a report on the aftermath, Morgan Stanley Dean Witter analyst Charles Phillips says the 1993 terrorist bombing of the World Trade Center taught tenants to be ready. "It was probably one of the best-prepared office facilities from a systems and data-recovery perspective," Phillips says. He estimates the replacement costs for computers, storage, and networking equipment lost in the World Trade Center alone at $500 million.

The New York Stock Exchange and the Nasdaq exchange, closed for four business days after the attack, handled record volume when they reopened on Sept. 17, with no reports of technical problems. The NYSE wasn't directly affected by the catastrophe, says Robert Britz, group executive VP of technology and operations at the exchange. On Tuesday, Sept. 11, when it became clear the exchange wouldn't open, IT systems were brought down in an "orderly fashion," he says. When trading resumed on Sept. 17, some 2.3 billion shares changed hands without incident. "We're in very good shape," Britz says.

Still, there are lessons to be learned. For instance, infrastructure planners should probably reassess potential points of failure in the concentration of telecommunications and electric cables in Manhattan, Britz says.

EMC had 76 customers in and around the World Trade Center, and the largest and most technically sophisticated were protected by investments they had made in systems that support real-time data backup at secondary data centers. "In the most high-risk, high-exposure environments, we had great success," Walton says.

Business-technology managers at those same companies, however, spent many hours re-creating noncritical applications-departmental servers running E-mail, for example-that hadn't been automatically backed up. As a result, EMC officials expect IT managers to consider giving those types of applications the same kind of backup protection that's now used for data centers.

The experience was similar in Comdisco's facilities. Only a few companies required help with mainframe recovery; the real challenge was restoring midrange systems and servers, says Damian Walch, senior VP of professional services. And work space for displaced workers was at a premium. Companies seemed better-prepared to revive back-office operations than other business processes, especially those that dealt with customers and required a lot of personnel, Walch says. "It highlighted the fact that we need to think more about what is customer-facing and what information is required to support customers," he says.

Other concerns revolved around the people and chain-of-command processes involved in emergency response. Venkat Gopalan, VP and CIO at DynCorp, an IT outsourcer to the Department of Defense and other federal agencies, says the company's disaster-recovery plan, including the backup of a main data center in Virginia to a facility in Texas, worked as intended when a section of the Pentagon exploded in flames after being hit by a hijacked American Airlines jet on Sept. 11. But it was a reminder of the need to have clear lines of communication, role assignments, and relocation plans for displaced workers. "We have already begun replanning, mostly on the people issue," he says.

Other companies would be well-advised to do the same, Comdisco's Walch says. That includes having backup plans in case something should happen to the IT staff who have been designated to manage emergencies. "Unfortunately, we have a couple of companies that literally lost their disaster-recovery teams," he says. Another lesson: Action plans should be summarized in a few pages and always carried in a briefcase or PDA.

In general, small and midsize businesses in lower Manhattan were more vulnerable because few have the budgets for real-time data backup or standby facilities. Their ability to get back to business depends on whether they recently saved essential data to disk, tape, or other media for storage off-site-a well-understood practice that's not always followed. "The smaller the customer, the bigger the exposure," says Michael McLay, VP of sales and service with Storage Technology Corp., a Louisville, Colo., storage vendor.

StorageTek, which had 62 customers in a four-block radius of the World Trade Center, was directly affected. More than 300 employees had to be evacuated because the company's New York branch, at One World Financial Center, was located across the street from the World Trade Center.

The New York Board of Trade had to move to the Comdisco site in Queens after its facilities at 4 World Trade Center were decimated. The $300,000 annual cost of maintaining space with Comdisco raised eyebrows among some Board of Trade members in the past, but "they're not questioning that decision anymore," says board president and CEO Mark Fichtel.

For East Coast Options Services Inc., a broker of coffee, sugar, and cocoa that does business at the New York Board of Trade, the transition to the backup location went smoothly. "Having this Comdisco facility ready means we survived as a company and can move forward," says Anthony Compagnino, a principal owner of East Coast Options Services.

The cataclysm had impact beyond New York and Washington. United Parcel Service Inc. activated its business-continuity planning group, says John Nallin, VP of information systems. UPS was well-prepared because the group recently put the finishing touches on a revised IT contingency plan and tested its routing, tracking, and other operational software at a backup data center outside Atlanta. It's the same software used at the company's technology headquarters in Mahwah, N.J.

The duplicated facilities let UPS operate and manage its global business from either New Jersey or Georgia, something the company was unable to do when Hurricane Floyd flooded parts of New Jersey in 1999, knocking out voice lines. "That was an intense lesson," Nallin says, and it prompted UPS to undertake a two-year project to mirror its systems in real time at the two facilities.

The events in New York and Washington didn't affect either UPS data center directly, but some of the company's facilities in the Northeast encountered difficulty connecting to the UPS data network for about two hours-a problem that was corrected automatically in most areas, but one that required manual rerouting of communications links in New York and New Jersey.

Just as UPS revisited its contingency plans after Hurricane Floyd, other companies are now asking questions about their own readiness for a new kind of threat. "We're going to take a real hard look at real-time recovery," says Ken Smith, CIO and VP with PolyOne Corp., a chemical company in Cleveland.

In the week after the attack, Comdisco received inquiries from approximately 25 companies looking to reassess their IT and business contingency plans-or develop them for the first time. "It's really scary," Walch says of companies that are ill-prepared.

The telecommunications infrastructure in New York was severely strained. Verizon's switching center at 140 West St., which served as a major wiring hub for the World Trade Center, was heavily damaged, and the phone company's Broad Street office, which provides 80% of the data circuits serving the New York Stock Exchange, lost electricity. Other carriers, including AT&T, and Genuity, an Internet service provider and hosting company, experienced serious outages, too. Technicians at those companies worked around the clock to restore service. "When you think about the ability to get this back up and running in the space of three or four days, it's nothing short of a miracle," Ivan Seidenberg, Verizon's co-CEO and president, said Monday after the stock exchange was back in business.

The experience will cause many companies to reassess their communications backbones. Equinix Inc., which provides diversified network connectivity and hosting facilities in six cities to ISPs and business customers, received inquiries from four financial-services firms in the wake of the attacks. "One very important requirement for these firms is geographical diversity," says Jay Adelson, co-founder and chief technology officer.

Equinix, like some other hosting companies, goes to great lengths to ensure a secure environment. The company does background checks on potential customers, and carefully restricts access to its facilities. Its 100,000-plus-square-foot buildings have concrete walls up to 2 feet thick, and they're surrounded by concrete embankments. The buildings are lined with Kevlar, a material used in bulletproof vests.

Once inside, "man traps" and biometric devices limit movement. "We have an extraordinary audit trail of every employee and customer," Adelson says. All of a sudden, such measures no longer seem extreme.

--With Eileen Colkin, Robin Gareiss, Larry Greenemeier, Christopher T. Heun, Steve Konicki, John Rendleman, and Rick Whiting