HIP has invested carefully in some high-end hardware as well. During the last three years, the company has turned an office in Manhattan's financial district into a critical data repository, the core of its operations. Two mainframes and 127 servers containing 15 terabytes of data are host to the company's services, applications, and files. HIP is building a large storage area network, based on EMC Corp.'s Clariion technology, that mirrors all of its data. The terrorist attacks expedited work on the system, Villalba says.

Three years ago, the only disaster-recovery plan for this critical data was "the traditional take-the-tapes-and-go" approach, Villalba says. In the event of an emergency, HIP would have its data, but not much else. So the company created a business-recovery plan to go with its disaster-recovery steps. "When we recover the system, the corresponding operational units know exactly the process to follow, where to go, and who to call," Villalba says. In addition to backing up its data daily and storing it off-site, twice a year HIP practices its recovery plan, assuring that business processes will work as well as databases.

Little did HIP know that only three months after the latest test, Steber and his IT staff would be forced to put the plan into action. After watching the events of Sept. 11 unfold on TV at HIP's 34th Street headquarters, Steber immediately got his disaster-recovery team together. "We told the staff to repeat everything we did in July, and they did, and they actually did it ahead of schedule," he says.

The first problem the disaster-recovery team encountered was a lack of communication with the data center, since phone service was knocked out and cell phones were constantly busy. "This was a lifesaver," Villalba says, pulling out a small Skytel two-way pager. The devices had been deployed the previous June to all IT staffers with a role in disaster recovery.

Minutes after the second jet hit the World Trade Center, city officials ordered every building south of 14th Street evacuated, so the data-center employees fled the area, designating one operator to stay behind. The recovery team responded quickly, rerouting data traffic and some data-center functions to an HIP subsidiary in Florida. The pharmacy system was one of the first to be moved over, Steber says. "It was down less than an hour."

As the day progressed, the team scrambled to get the rest of HIP's infrastructure online. "We recovered all our critical systems, some within one hour, all within 24 hours," Villalba says. On the morning of Sept. 12, HIP officially declared a disaster, which assured it a space on Comdisco Inc.'s recovery floor in North Bergen, N.J.

HIP's business-recovery plan requires backup tapes to be made every night and sent out of the city for storage. The team had the tapes from Sept. 10 installed at Comdisco's facility and was running much of HIP's enterprise from that site. But a significant number of transactions had been processed between the time the backups were made and when the attacks occurred. Steber figured they should retrieve the latest data, still on the racks at the Wall Street data center, and recover that as well. Since HIP's facility in Florida had control of the data center's equipment, it was able to use a robotic storage system to dump all that data onto tapes. On the morning of Sept. 13, a few staff members got a police escort back to the building to remove those tapes, recovering all data up until the attacks. By the end of that day, HIP had all its critical systems and mainframes running, 24 hours ahead of schedule.

To HIP's top managers, it's clear the investments in technology have been a success, saving the organization in a crisis and giving it a competitive edge in a tough economy. They're convinced IT will continue to provide the edge HIP needs to succeed. "We can go toe to toe with any health-care insurance agency in terms of where we are," Villalba says. "And we can go one step better in terms of where we're going to be."

Return to 2002 InformationWeek 500 homepage