Hurricane Sandy: Disaster Recovery Improv Tales - InformationWeek
Mobile // Mobile Applications
02:00 AM
Connect Directly
Moving UEBA Beyond the Ground Floor
Sep 20, 2017
This webinar will provide the details you need about UEBA so you can make the decisions on how bes ...Read More>>

Hurricane Sandy: Disaster Recovery Improv Tales

In lower Manhattan, Peer1 Hosting used a bucket brigade to replenish fuel for diesel generators on the 18th floor after pumps and the elevator broke down. In New Jersey, SunGard rerouted fuel trucks to avoid flooded intersections.

That meant Internap customers, using the Eighth Street site as major connection point to the Internet, were off the Internet until power was restored. "We continue to work with vendors to bring the entire site back online," he wrote Wednesday.

No cause was listed but a building's fuel lines and pumps, which might function fine during periodic short term tests, could have debris build up in the line, a clogged filter or a failed a mechanical component, causing a failure during sustained, 24-hour operations. To anticipate such a development is hard, although more recently constructed buildings are built with instruments to report on the functioning of their systems that can help spot developing weak points. Even then, experienced building engineers need to be on hand to analyze and respond when a malady is identified. For whatever reason, that response wasn't available for the building's fuel system at 111 Eighth Street.

But even SunGard acknowledged there could be unforeseen eventualities that threaten to disrupt the best laid disaster recovery plan. Nick Magliato said he had recovery teams both on premises and on the ground with walkie talkies along the likely path of flooding as the storm surge drove what would have been a seasonal high tide to an even higher than usual mark.

Monday night the surge raised a nearby waterway 11 feet above flood stage but still within its earthen containment levees. The SunGard ground team found water leaking out and slowly creeping as a two-inch flood toward the industrial complex where SunGard's three data centers were housed. But it was rising slowly and not moving fast. Everyone hoped the tide would recede before causing any damage.

Then information from the ground teams, entered into a spread sheet at the data center where Magliato was located, indicated that the depth of the creeping flood had risen one-and-a half feet in ten minutes. The earthen levee had broken and water was flooding out. As that assessment came in, Magliato, said he realized "at that rate, it wasn't going to take very long before the whole place was going to be under water."

The observers were calling in reports from far enough away that the data center had advance warning. Still it wasn't long before water began appearing at the edge of the industrial park's parking lot, and then began to intrude upon them. Magliato still had the basic defense of having put all data center equipment on a raised floor above the level of the building foundation. The water level would have to increase a total of 6 feet to 7 feet before any of his operations would be affected and it never reached that height.

Nevertheless, the SunGard recovery teams found themselves working farther down their disaster recovery check lists than they ever had before at the Carlstadt location. The full executive team was at work and members of the board of directors were being alerted. Customers were alerted and placed on standing bridge conference calls. They would need to be kept abreast and might have to quickly approve transfer or shutdown procedures, if the water reached a certain mark.

It was at that point, Magliato said in an interview, that he realized his own disaster recovery plan had a gap in it. Fuel was needed to replenish the backup generators. As he looked at his map, Magliato realized the delivery truck's route would take it close to where the levee had broken and the water was the deepest. With myriad things to do, he and staffers nevertheless looked up an alternative route to ensure the truck did end up mired at some low lying intersection. Soon two 7,500-gallon trucks were continuously delivering fuel in shifts to keep the complex's 40,000 gallons of reserves stocked up.

The plan assumed "any flood water would recede by the time the replacement fuel was needed," but that turned out not to be the case. "We had to think of a path from another direction for them to come on," Magliato recalled.

High tides along coastal areas occur twice a day and the water lingers for varying periods -- usually an hour or less at the high water mark before receding and becoming a slack tide. For Magliato and his teams, "it was long high tide. We didn't see any significant decrease in the water level until lunchtime" Tuesday, he said.

On the whole, the disaster plan worked as expected and the SunGard facilities continued operating continuously. But afterward, no one claimed that the plan had foreseen every challenge or didn't require a little improvisation along the way.

2 of 2
Comment  | 
Print  | 
More Insights
Threaded  |  Newest First  |  Oldest First
User Rank: Ninja
11/5/2012 | 3:53:13 PM
re: Hurricane Sandy: Disaster Recovery Improv Tales
Some great stories of people adapting to the circumstances. It is unrealistic to believe you can think of everything that could possibly go wrong ahead of time. I realize that 20-20 hindsight makes people say "Well they should have thought of that", but things that are obvious after that happen just aren't so obvious beforehand. Bottom line you have to be adaptable to get through events like this. That doesn't mean you don't plan for what you can think of but also don't beat yourself up very badly afterwards. Adjust your plan, but don't fret over the unforeseeable.
Javier Jones
Javier Jones,
User Rank: Apprentice
11/15/2012 | 3:16:51 AM
re: Hurricane Sandy: Disaster Recovery Improv Tales
I think you mean 111 Eight Avenue, not street.

- Javier
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
IT Strategies to Conquer the Cloud
Chances are your organization is adopting cloud computing in one way or another -- or in multiple ways. Understanding the skills you need and how cloud affects IT operations and networking will help you adapt.
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll