7 Data Center Disasters You'll Never See Coming - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Cloud
News
6/7/2015
12:06 PM
Charles Babcock
Charles Babcock
Slideshows
Connect Directly
Twitter
RSS
E-Mail

7 Data Center Disasters You'll Never See Coming

These are the kinds of random events that keep data center operators up at night. Is your disaster recovery plan prepared to handle these freak accidents?
2 of 9

Fire In Iowa
On the afternoon of Feb. 18, 2014, a day when the state of Iowa normally produced the state payroll, an electrical fire broke out in its primary data center. Far from anticipating such an incident, the IT staff had been preparing contingency plans for a blizzard that was predicted for that evening, recalled Robert von Wolffradt, CIO for the state of Iowa, in a blog posted March 25, 2014 on GovTech.com.
When the fire alarm sounded at 3 p.m., the data center lost power, smoke invaded the building, and the staff had to evacuate. The alarm triggered the data center's gas-powered FM-200 fire suppression system, and the fire was contained to the inside of a wall-mounted, transient voltage suppression box (pictured above). The unit, which controlled the flow of power into the data center, had overheated and melted down. The state's General Services team built a bypass, and power was restored several hours later.
With the power back on, doors could be opened, fans turned on and the building vented, although police and fire crews were reluctant to permit IT staffers back into the building. Three-and-a-half hours after the incident, state officials determined the data center was fit to re-enter. 
Wolffradt had to decide whether $162 million in state payments to citizens and vendors, along with employees' paychecks, could be processed. Working quickly, crews had the data center cleaned of residue, and IT staffers restored the storage attached network, firewalls and network core by 9 p.m. Turning them back on meant putting the equipment at risk without the transient surge suppression unit being replaced. Wolffradt decided to do so anyway, but he staffed a secondary data center as a precaution.
By 11 p.m. additional systems were back on line, including the service desk, and the Department of Transportation cameras needed for monitoring bridges and highways in the impending snowstorm. 
Also restored: financial systems and virtualized applications. Additional systems were brought on line through the night, eliminating the prospect by morning that the secondary data center would have to take over state processing. 
'We leveraged our Homeland Security's voice notification system to update agency directors and key staff twice during the event,' recalled Wolffradt, who noted that rumors fly about wildly in the aftermath of a data center fire and the CIO must communicate frequently with other responsible parties. He had personally kept the governor and key state officials informed as events unfolded.
One lesson Wolffradt shared in his blog post:  Diversify major enterprise systems from each other, such as putting email in a separate facility from payroll. Another: In a fire, General Services and Human Resources 'are your best friends,' and will help you get through it. One of the most difficult barriers to restoring operations was convincing police and fire crews that IT staff could re-enter the data center, he wrote. The building where the center was located housed a total of 1,000 state employees, most of whom waited longer than IT staffers for the all clear.
(Image: Robert von Wolffradt, CIO, State of Iowa via GovTech.com)

Fire In Iowa

On the afternoon of Feb. 18, 2014, a day when the state of Iowa normally produced the state payroll, an electrical fire broke out in its primary data center. Far from anticipating such an incident, the IT staff had been preparing contingency plans for a blizzard that was predicted for that evening, recalled Robert von Wolffradt, CIO for the state of Iowa, in a blog posted March 25, 2014 on GovTech.com.

When the fire alarm sounded at 3 p.m., the data center lost power, smoke invaded the building, and the staff had to evacuate. The alarm triggered the data center's gas-powered FM-200 fire suppression system, and the fire was contained to the inside of a wall-mounted, transient voltage suppression box (pictured above). The unit, which controlled the flow of power into the data center, had overheated and melted down. The state's General Services team built a bypass, and power was restored several hours later.

With the power back on, doors could be opened, fans turned on and the building vented, although police and fire crews were reluctant to permit IT staffers back into the building. Three-and-a-half hours after the incident, state officials determined the data center was fit to re-enter.

Wolffradt had to decide whether $162 million in state payments to citizens and vendors, along with employees’ paychecks, could be processed. Working quickly, crews had the data center cleaned of residue, and IT staffers restored the storage attached network, firewalls and network core by 9 p.m. Turning them back on meant putting the equipment at risk without the transient surge suppression unit being replaced. Wolffradt decided to do so anyway, but he staffed a secondary data center as a precaution.

By 11 p.m. additional systems were back on line, including the service desk, and the Department of Transportation cameras needed for monitoring bridges and highways in the impending snowstorm.

Also restored: financial systems and virtualized applications. Additional systems were brought on line through the night, eliminating the prospect by morning that the secondary data center would have to take over state processing. "We leveraged our Homeland Security’s voice notification system to update agency directors and key staff twice during the event," recalled Wolffradt, who noted that rumors fly about wildly in the aftermath of a data center fire and the CIO must communicate frequently with other responsible parties. He had personally kept the governor and key state officials informed as events unfolded.

One lesson Wolffradt shared in his blog post: Diversify major enterprise systems from each other, such as putting email in a separate facility from payroll. Another: In a fire, General Services and Human Resources "are your best friends," and will help you get through it. One of the most difficult barriers to restoring operations was convincing police and fire crews that IT staff could re-enter the data center, he wrote. The building where the center was located housed a total of 1,000 state employees, most of whom waited longer than IT staffers for the all clear.

(Image: Robert von Wolffradt, CIO, State of Iowa via GovTech.com)

2 of 9
Comment  | 
Print  | 
Comments
Oldest First  |  Newest First  |  Threaded View
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
6/8/2015 | 12:48:52 PM
When the fire fighting system gets triggered by accident....
DanaRothlock, Yes, part of the problem of disaster preparedness is preventing the fire fighting system, especially when it's triggered by accident, from destroying what it's supposed to save. There's been no easy answer for years. Halon was meant to prevent water damage to the equipment. Sprinklers, on the other hand, prevent Halon damage. It's a fool's bargain with fate.
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
6/8/2015 | 9:07:39 PM
Move the backup site further away!
Dave Kramer, yes, it's a good idea to move the backup data center to a different site. But Hurricane Sandy told us just how far away that second site might have to be. Moving it across town or across the state might not have been enough in that case. With Sandy, disaster recovery specialist Sungard, had flood waters lapping at the edges of its parking lots on the high ground in N.J. The advent of disaster recovery based on virtual machines makes it more feasible to move recovery to a distant site (but still doesn't solve all problems).
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
6/11/2015 | 3:02:16 PM
Diesel fuel stored at NYC data centers reduced by 9/11
KBartle, what about this? One of the unreported aspects of the Hurricane Sandy disaster, when New York and many places along the East Coast went dark, was that every data center in the city had a limited supply of diesel fuel on premises. That was due to new regulations, I believe from a former mayor's office after 9/11, that the flamable liquids stored inside an office building must be reduced. In some cases, that made the investment in generators irrelevant. Public transit was down, city streets were clogged and fuel delivery trucks had great difficulty getting through. There goes the disaster recovery plan.
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
6/11/2015 | 3:15:00 PM
A narrow margin separates "chilled" from "too hot"
In no. 6, Outage by SUV, a commenter on Lew Moorman's blog post noted that a data center has about five minutes between the loss of its chillers and the start of equipment overheating. Does anyone know, is the margin really that narrow? I understand that computer equipment can operate at up to 100 degrees OK, but after that overheating starts to get dicey.
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

Slideshows
10 Things Your Artificial Intelligence Initiative Needs to Succeed
Lisa Morgan, Freelance Writer,  4/20/2021
News
Tech Spending Climbs as Digital Business Initiatives Grow
Jessica Davis, Senior Editor, Enterprise Apps,  4/22/2021
Commentary
Optimizing the CIO and CFO Relationship
Mary E. Shacklett, Technology commentator and President of Transworld Data,  4/13/2021
White Papers
Register for InformationWeek Newsletters
2021 State of ITOps and SecOps Report
2021 State of ITOps and SecOps Report
This new report from InformationWeek explores what we've learned over the past year, critical trends around ITOps and SecOps, and where leaders are focusing their time and efforts to support a growing digital economy. Download it today!
Video
Current Issue
Planning Your Digital Transformation Roadmap
Download this report to learn about the latest technologies and best practices or ensuring a successful transition from outdated business transformation tactics.
Slideshows
Flash Poll