7 Data Center Disasters You'll Never See Coming

Hurricane Sandy: Generator Fail

Manhattan, like much of the East Coast, lost power as Hurricane Sandy came ashore over Virginia, Delaware, Maryland, and New Jersey in late October 2012. A storm surge of salt water followed, rushing up the streets and flooding the lower Manhattan and many other sites around the Tri-State area.

At 75 Broad Street in Lower Manhattan, home of Peer 1 hosting, it was a disaster recovery planner's nightmare. There were backup generators ready to go well above the water line on the building's 18th floor. But the same storm surge that poured into the building's lobby and filled its basement, knocked out the emergency generator fuel pumping system located there. Once under water, its electrical circuits no longer worked. (Part of New York's response to 9/11 was to limit the amount of fuel oil stored in office buildings.). So, as the generators began running out their limited supply of fuel, the company couldn't get fresh oil to them. Peer 1 advised customers of a planned shutdown of their systems within hours, as several employees made their way to the facility to help prevent any data loss.

Instead of a shutdown, Peer 1 engineered a bucket brigade to carry fuel oil for the generators. The fuel was lined up on the street, above, and carried by hand up to the up to the 17th floor, to where the generator's day tank was located. That tank and its pumps could deliver fuel to the generators on the floor above. Peer 1 customers -- including such as SquareSpace, a Web site development firm, and Fog Creek Software, a supplier of online project management -- provided manpower for the 25-member team carrying fuel oil up stairs to the generators the night of Oct. 30 and into Oct. 31.

By lunchtime on Oct. 31, they'd filled the day tank and could take a break, eating a lunch that had to be delivered by foot over the Brooklyn Bridge, due to Manhattan's clogged streets. Neither the need for a bucket brigade, nor on-foot delivery of lunch, had been put in the Peer 1 disaster recovery plan. But no shutdown resulted from the storm.

(Image: Fog Creek Software)