Hurricane Sandy: Disaster Recovery Improv Tales

That meant Internap customers, using the Eighth Street site as major connection point to the Internet, were off the Internet until power was restored. "We continue to work with vendors to bring the entire site back online," he wrote Wednesday.

No cause was listed but a building's fuel lines and pumps, which might function fine during periodic short term tests, could have debris build up in the line, a clogged filter or a failed a mechanical component, causing a failure during sustained, 24-hour operations. To anticipate such a development is hard, although more recently constructed buildings are built with instruments to report on the functioning of their systems that can help spot developing weak points. Even then, experienced building engineers need to be on hand to analyze and respond when a malady is identified. For whatever reason, that response wasn't available for the building's fuel system at 111 Eighth Street.

But even SunGard acknowledged there could be unforeseen eventualities that threaten to disrupt the best laid disaster recovery plan. Nick Magliato said he had recovery teams both on premises and on the ground with walkie talkies along the likely path of flooding as the storm surge drove what would have been a seasonal high tide to an even higher than usual mark.

Monday night the surge raised a nearby waterway 11 feet above flood stage but still within its earthen containment levees. The SunGard ground team found water leaking out and slowly creeping as a two-inch flood toward the industrial complex where SunGard's three data centers were housed. But it was rising slowly and not moving fast. Everyone hoped the tide would recede before causing any damage.

Then information from the ground teams, entered into a spread sheet at the data center where Magliato was located, indicated that the depth of the creeping flood had risen one-and-a half feet in ten minutes. The earthen levee had broken and water was flooding out. As that assessment came in, Magliato, said he realized "at that rate, it wasn't going to take very long before the whole place was going to be under water."

The observers were calling in reports from far enough away that the data center had advance warning. Still it wasn't long before water began appearing at the edge of the industrial park's parking lot, and then began to intrude upon them. Magliato still had the basic defense of having put all data center equipment on a raised floor above the level of the building foundation. The water level would have to increase a total of 6 feet to 7 feet before any of his operations would be affected and it never reached that height.

Nevertheless, the SunGard recovery teams found themselves working farther down their disaster recovery check lists than they ever had before at the Carlstadt location. The full executive team was at work and members of the board of directors were being alerted. Customers were alerted and placed on standing bridge conference calls. They would need to be kept abreast and might have to quickly approve transfer or shutdown procedures, if the water reached a certain mark.

It was at that point, Magliato said in an interview, that he realized his own disaster recovery plan had a gap in it. Fuel was needed to replenish the backup generators. As he looked at his map, Magliato realized the delivery truck's route would take it close to where the levee had broken and the water was the deepest. With myriad things to do, he and staffers nevertheless looked up an alternative route to ensure the truck did end up mired at some low lying intersection. Soon two 7,500-gallon trucks were continuously delivering fuel in shifts to keep the complex's 40,000 gallons of reserves stocked up.

The plan assumed "any flood water would recede by the time the replacement fuel was needed," but that turned out not to be the case. "We had to think of a path from another direction for them to come on," Magliato recalled.

High tides along coastal areas occur twice a day and the water lingers for varying periods -- usually an hour or less at the high water mark before receding and becoming a slack tide. For Magliato and his teams, "it was long high tide. We didn't see any significant decrease in the water level until lunchtime" Tuesday, he said.

On the whole, the disaster plan worked as expected and the SunGard facilities continued operating continuously. But afterward, no one claimed that the plan had foreseen every challenge or didn't require a little improvisation along the way.