Hurricane Sandy Surge Challenges NYC Data Centers

As Hurricane Sandy sent a heavy storm surge into lower Manhattan Monday night, Peer 1 Hosting's status went from "Hurricane Sandy has not impacted operations," to "facility has made the transition over to generator power," to "basement is flooded," to "we are going to implement a controlled shutdown" over the space of a few hours.

Peer 1, a managed hosting service, is located three blocks from the tip of Battery Park. Water from the hurricane's unprecedented storm surge reached its location at 75 Broad Street. Both Peer 1 and the Internap colocation and managed hosting service have data center facilities in the building. They were reported to be knocked out.

Nearby at 33 Whitehall Street, a Datagram data center was also knocked out, taking down popular websites Gawker, Huffington Post and BuzzFeed.

Peter Feldman, CEO of DataGryd, a data center designer with operations on higher ground at 60 Hudson Street, described 75 Broad Street as "a converted office building with a basement that isn't as deep" as 60 Hudson's. His location wasn't affected by the surge and his firm was able to continue operations overnight Monday and Tuesday without incident. But he said he went home where he "paced like an expectant father" as he monitored reports on the area late into the night.

Many data centers had to convert from utility power to emergency backup and diesel generators as the public power supply disappeared throughout the evening. Feldman said his own emergency generators and diesel fuel are in the basement of 60 Hudson, but its basements sit atop a set of still deeper "catacomb" channels beneath them that can carry water away from the building if any drains into the basements.

But Feldman said reliance on diesel generators, now commonplace around the city, is fraught with its own problems. In many cases, they are located on the lowest floors or in basements. They also have emergency delivery contracts for more fuel, but those contracts did not anticipate that the city's bridges and tunnels would be closed down when the fuel was needed.

His only hiccup was that "it took four hours instead of two hours" to get his fuel tanks replenished, a delay that did not endanger his operations but might have been fatal in some other cases with smaller reserves.

As it was, the status reports by Peer 1 throughout Monday offer a replay of events that at first seemed to be well in hand, then led to unexpected shutdown.

At 2:10 p.m., Peer 1 sent a message to customers: "We don't expect to encounter any issues caused by Hurricane Sandy, but we want to ensure you know that we are prepared. We know your IT infrastructure is critical to your business. We strongly recommend that you back up your data and take any other necessary precautions to avoid disruption to your business."

That turned out to be good advice.

At 5:40 p.m., Peer 1 customers were notified that the data center had shifted to generator power "due to a drop in commercial power."

At 8:36 p.m., customers learned that operations continued on diesel power, but the building lobby had taken in some water, the first direct cause for alarm.

Two hours later, at 10:27 p.m., the good news was that observers in the building could see "some lowering of the water level outside the building." But the bad news was flooding was being detected in both first and second basement levels, though there was still plenty of fuel on hand. "Extent of the damage will not be determined until the basement is accessible. The fuel system has a header with 5000 gallons of fuel and will be the primary supply for the next 12-24 hours," the update said.

The next update at 3:30 a.m. Tuesday said: "We are still running from emergency generator power. Water has receded and we are currently waiting for a report back from building engineers on the status of the fuel and power systems that were located in the basement."

At 8 a.m. Tuesday, the declining fuel situation began to look dire: "At this point we have an estimate of 4 hours for the fuel left on our generators. Our techs and facility are continuously working to get emergency fuel delivery on time and was looking to set-up a temporary tank and pump since the basement is still flooded. In the event of not receiving the fuel on time, worst case scenario is we will have to gracefully shutdown the facility."

And at 9:30 a.m., the hammer fell: "We are going to implement a controlled shutdown of NY Data Center at 10:45 a.m. ET. Customer communications is being prepped."

It was a message that had to be repeated by several other facilities. Peer 1 appears to have continued operating until the flooding affected its ability to keep its basement diesel fuel pumps operating, as reported by Data Center Knowledge.

Download the new issue of Must Reads, a compendium of our best recent coverage on IT-as-a-service. It includes articles on cloud computing myths, how to build an IT service catalog, security problems, and more. (Free registration required.)