Partners You Can Count On

Eight years ago, when terrorists set off a bomb in the basement of the World Trade Center, IT vendors did their part to help companies affected by the blast get back in business. Not surprisingly, those efforts paled in comparison to what vendors did this fall, in the face of much greater devastation.

"This seemed more of a united effort," says DeWayne Bobo, global telecommunications manager at Deloitte Consulting. Until Sept. 11, the company, which along with Deloitte & Touche LLP is part of Deloitte Touche Tohmatsu, maintained offices and its main communications hub in the now-inaccessible World Financial Center, next door to the twin towers. This time around, Bobo says, vendors "worked the extra hours and bypassed normal routines to make things happen."

Businesses displaced by September's attacks on the World Trade Center found out fast that the effectiveness of their own disaster-recovery plans depended in large part on the performance of their IT vendors. Many companies that had been located at Ground Zero and the surrounding areas in lower Manhattan credit their vendors for coming through in the crisis.

But one lesson learned was the value of maintaining close communication with IT vendors about their roles in a business' continuity plan, long before disaster strikes. All too often, that doesn't happen. An InformationWeek Research survey conducted in November reveals that of 250 companies with business-continuity plans, only 17% have alerted primary vendors about their role in those plans, and just 15% have informed their service providers. Smart businesses are beginning to revisit their relationships with technology providers to ensure that their vendors will be at their sides even in less-arduous circumstances.

At Deloitte Consulting, director of global technology infrastructure Eric Erikson is working with the firm's vendor contacts on a new disaster-recovery plan that outlines in advance the role partners are expected to play. "There's now a need to have the vendors involved in the emergency planning," he says. The division's Manhattan office consults with at least 1,500 customers about their IT, human-resources, and enterprise-development projects, and staying in contact with clients is a key goal, even during an emergency.

It was a hard goal to live up to in the aftermath of the Sept. 11 attacks, with the communications hub used by the division's 1,000 employees in the New York metropolitan area out of commission. Workers were forced to use cell phones to communicate until the company's service provider, NextiraOne LLC, duplicated the downtown landline communications center at Deloitte & Touche's midtown Manhattan operations. The service provider began the rebuilding project the weekend after the attacks; after that, it switched over Deloitte Consulting's three locations to the new system. The project wrapped up in early November.

It was no easy feat for NextiraOne. Deloitte Consulting was one of 800 customers that called the service provider's emergency hot line right after the attacks. "The nightmare was that we were trying to do what everyone else was trying to do at the same time--get connectivity so we could get working," Erikson says.

Adding to the difficulty was the need for NextiraOne, which installed the 81C Nortel Networks switch on the PBX in Deloitte & Touche's midtown offices, to lead the coordination effort with a slew of other vendors. Avaya provided an Octel 350 voice-mail system, AT&T provided the network, and Sprint established long-distance connections. Coordinating it all called for close collaboration. "We had to rely on manufacturers to go 24-by-7 in order to manufacture equipment in the quantities we were requesting," says William McGlashan, who manages the Deloitte Consulting account for NextiraOne.

Placing orders was difficult, as the recovery plans were constantly in flux. "It was an imperfect situation because we were continually updating, but manufacturers were wonderful in trying to produce to our needs," McGlashan says. For example, Nortel helped expedite the process by cutting the paperwork usually required to process orders. "We all did what we had to do to get the job done. We'll take care of the paperwork issues later," McGlashan says.

All things considered, the reconstruction went swiftly, but Erikson says he's taking steps to ensure a less-harried and quicker recovery in the event of future problems. "What we've done is clearly spell out the roles and responsibilities to an individual level," Erikson says. That includes outlining how each of the vendors will communicate down Deloitte's chain of command and among themselves in the event of an emergency.

Most vendors, including large companies such as Compaq, EMC, IBM, and Sun Microsystems, responded immediately to the attacks, setting up command centers to centralize control over relief efforts. But there was an unquestionable need for these intense rivals to collaborate to aid their customers.

"It's strange to see IBM, Compaq, and Sun working together as one harmonious team. Usually it's more of a competitive environment," says Ed Taylor, managing director of global infrastructure at Marsh Inc., a subsidiary of $10 billion professional-services firm Marsh & McLennan Cos.

Tragically, Marsh lost 129 IT staff members in the attacks. It also lost two data centers, 255 servers, and valuable applications and data--about $70 million worth of technology assets. Vendors were on the scene immediately to help with the rebuilding. "Overnight we regenerated as much of that data center as we could," Taylor says. "That was only accomplished with the work of all the vendors acting as partners."

There was another factor that helped the company get back on its feet as quickly as possible. Two years ago, CIO Ellen Clarke created a list of 10 to 15 technology providers that were key to Marsh's operations and with whom Marsh needed to work as closely as possible. Clarke also made it a policy for her IT team to tour vendor facilities annually and select technology based on quality rather than price.

"Our strategy is to move our relationships toward vendors who will understand our needs and will come to the table. That has really worked for us," Clarke says. The requirements for any technology implementation at Marsh include a disaster-recovery test, security standards, and remote-access capability. As a result, Marsh's vendors were intimately familiar with its processes, applications, and data requirements and were able to spring quickly into action.

It's a smart strategy, says Kevin O'Mara, a research principal at AMR Research. "If the vendor is your partner," he says, "it has to know not only what your applications are, but the data that was captured and how to work it." But few businesses are willing to pay to engage vendors on that level. "If you're going to bargain down the contract, you won't get that," he says.

J.P. Morgan Chase & Co. also believes that vendors should be viewed as partners--a concept that paid off when the crisis struck. Storage provider EMC rushed to get millions of dollars in hardware to the company the day after the attacks and also provided senior technical staff to get J.P. Morgan Chase's 60 Wall St. data center online again. EMC's understanding of J.P. Morgan Chase's environment was key to the successful installation, say managing director Michael Poser and executive VP Thomas Fogarty. Just as important, they say, EMC "provided all of this on a handshake--with the message from EMC to J.P. Morgan Chase being, 'Get whole again and send us back what you don't need long-term.'"

But not every company is gearing up to form closer partnerships with all its business-continuity vendors. One major Wall Street securities firm is looking at establishing a business-recovery plan that will leave it better-prepared should another outage occur. One option is bringing disaster-recovery operations in-house.

With a data center about two blocks from the Trade Center--an area that's indefinitely inaccessible--the firm, which requested anonymity, had to quickly move operations to its service provider's disaster-recovery facility so it could provide services to its more than 1 million clients when the markets reopened Sept. 17. But because of the way its contract with the service provider (which the securities firm declined to name) was written, the firm wasn't confident it would be able to stay in the disaster-recovery facility longer than six weeks.

Facing that uncertainty, executives decided to quickly build a data center in New Jersey while operating out of the backup location. The firm called on its network service provider, Greenwich Technology Partners, which immediately provided engineering and operations staff to get the project rolling. "They all went above and beyond the call," says a high-ranking technology executive at the securities firm. "We said, 'We need you to jump,' and they said, 'How high?'"

Greenwich Technology immediately began rebuilding the company's network lines, pulling in senior tech staff to lay wires, says Johna Till Johnson, senior VP and chief technology officer at the service provider. The data center was operating within three weeks of the attacks. The firm is working on a long-term goal of building a network that will connect all of its facilities.

As the securities firm ponders its future business-recovery options, it's unsure whether it will again sign with a disaster-recovery provider or build a secondary facility. It expects to make the decision within the next two months.

But one thing is clear: The vendor who responded so effectively to the Sept. 11 emergency won't be forgotten, the executive says. "When a company provides extraordinary levels of support, it only strengthens the partnership and increases the likelihood that I'll go back to them in the future," the executive says. "In the current economic environment, there's no tolerance for screwing up."