Deloitte Won't Get Caught Short

When Deloitte Consulting lost its main telecommunications hub in the World Financial Center on Sept. 11, the experience provided invaluable lessons for the design of a new global data center in Spring Valley, Pa., which the firm completed on Feb. 1.

The devastating terrorist attacks knocked out communications for some 1,000 Deloitte Consulting employees in the New York area. In the weeks following the attacks, workers had to use cell phones to communicate internally and with customers while the company scrambled to duplicate its downtown landline communications center.

Those weeks were tough. Losing communications is intolerable for a firm whose core business is sharing information with clients. "We don't produce anything; our product is intellectual property, phone connections, and E-mail," says Eric Eriksen, director of global technology infrastructure at Deloitte Consulting. "That got brought to light on Sept. 11 that this is absolutely critical."

The firm, which is in the midst of divesting itself from parent company Deloitte Touche Tohmatsu, is determined never again to be without telecommunications. So Deloitte Consulting has added to its new data center larger PBX boxes that will enable faster network transfers in the event of an outage and self-healing network functions that can automatically repair problems. It's also contracting with multiple vendors to improve disaster recovery, and has consolidated into a single repository all its client data that had previously been stored in local offices. "Before 9/11, we were pretty far in planning the facility, but after 9/11 we had to add things like overcapacity capabilities and redundancy," Eriksen says. "It was more costly, and we had to add them on quickly, but if we had another disaster in the tristate area, we could be up in the facility in a matter of days as opposed to weeks."

Deloitte Consulting had decided a year ago--well before the Sept. 11 attacks--to build the data center to consolidate its infrastructure. The firm had servers spread across the country and believed it needed to consolidate technical operations into a single campus.

Before building even started, Bruce Short, manager of global networks at Deloitte Consulting, went on a fact-finding mission, surveying data-center managers in at least a dozen of the world's largest companies to find out what they would do differently if they had the opportunity to rebuild their facilities.

Common responses to the survey included improving wiring and communications cabinets, moving power supplies to the front of racks and data to the back, implementing dedicated networks for labs and staging rooms, and running backup networks for transaction processes.

Deloitte Consulting has incorporated all these recommendations into its new facility. Since the beginning of February, the firm has spent every Wednesday and Saturday relocating servers and applications to the facility; Compaq was hired to oversee the equipment moves. More than 300 servers were moved, the VPN was rehubbed, and remote-access capabilities were transferred to the new site, all with the stipulations that each server had to have a backup and each application had to be tested and running again within a 24-hour-period.

Now that the project is complete, Deloitte Consulting's next step is to build contingency plans for the new data center. Currently, the firm backs up data by transferring it to tapes daily and manually moving them off-site. But automatically backing up data to another facility has become a much more pressing priority.

"It's something that we have had on our budget every year, and it usually gets cut quickly, but this year it's staying on," Eriksen says. "Our focus is on protecting what we need to service clients. If we're out of human-resources systems we can live with that--but if we're out of E-mail, we're dead in the water."

Photo by Dominic Episcopo