Hurricane Sandy Lesson: VM Migration Can Stop Outages - InformationWeek
Cloud // Platform as a Service
12:16 PM
Connect Directly
Ransomware: Latest Developments & How to Defend Against Them
Nov 01, 2017
Ransomware is one of the fastest growing types of malware, and new breeds that escalate quickly ar ...Read More>>

Hurricane Sandy Lesson: VM Migration Can Stop Outages

When a hurricane or other disaster threatens, why not just move critical systems out of the way? It can be done -- but not at the last minute.

Who Is Hacking U.S. Banks? 8 Facts
Who Is Hacking U.S. Banks? 8 Facts
(click image for larger view and for slideshow)
When it comes to disaster recovery, there's nothing like having learned your lesson before the disaster arrives. One of the most useful lessons is that virtualized systems now enable IT managers to move entire systems out of harm's way, even if that means half way across the country -- although it's necessary to have an alternative site set up well in advance.

Another lesson is that moderate-cost backup telephony services based on IP networks, including the Internet, and open-source code can serve as host substitutes for your local carrier and office PBX service, if those get knocked offline.

Asterisk and BroadSoft are providers of what's come to be known as SIP trunkline services. Another, Evolve IP, played a role in ensuring patients needing oxygen in the aftermath of Hurricane Sandy could still place their calls to Apria Health Care and get their deliveries.

Apria is a national deliverer of home health care services, which learned in 2005 when Katrina hit New Orleans that it could lose telephone service at the moment it was most needed. Apria supplies hospital beds, drug infusion equipment, and other medical goods and nursing services to homes of elderly, incapacitated and convalescing patients. But one of the most frequently needed services, especially in a disaster, is simply the personal oxygen tank.

[ Read how data centers reacted to Hurricane Sandy, including bucket brigades. Hurricane Sandy Disaster Recovery Improv Tales. ]

During Katrina, Apria employees were incapacitated by the failure of office phone systems as the storm and its flood waters swept through the New Orleans region. That meant oxygen users who had left home to move in with relatives or find a room on higher ground had only what they had been able to carry away with them, often reflecting a hurried departure at the last minute. But when they picked up the phone -- their primary way of placing orders -- they often found the line to the Apria Health Care office was dead.

"Patients couldn't get through to us after Katrina," said David Slack, VP of IT network engineering for Apria. There was an alternative -- calling a national toll free number -- but few patients had it when they needed it. Something had to be done.

Apria looked at backups to its phone system from the major carriers, who expected the firm to spend $2,000 to $4,000 per office to upgrade to SIP trunkline services. Apria has 550 offices nationwide, which would have brought the total bill to over $2 million. It found a lower-cost substitute in the firms that carry enterprise voice and data over IP networks, including the Internet. BroadSoft and Asterisk use private branch exchange open-source code to provide such a service. Apria settled on Evolve IP, another provider, an enterprise version of the Vonage VOIP for consumers.

The Evolve IP system, providing a virtual PBX for Apria hosted in an Evolve IP data center, was installed a year ago, based on the lessons learned from Katrina. The hosted PBX needed to be accessible from any branch and have a call forwarding capability in case a Katrina-like storm should knock out a branch. As Sandy gathered strength and was projected to veer into Middle Atlantic states, Apria's executives knew the system was going to get its first test. Twenty of its offices were directly in Sandy's path. The system needed to be programmed where to forward calls if the primary destination wasn't available, and Apria made those adjustments. If the Middletown, N.Y., branch lost telephone service, the system was to shift calls to the nearest well-staffed hub, in Cromwell, Conn.

"We lost both phone and data connectivity in Middletown. Immediately the phones started ringing in Cromwell, the backup site. All the intelligence on (emergency) routing was in the cloud. It understood instantly if a branch office was down. It worked fantastically," said Slack in an interview. In this case, the "cloud" is three private Apria data centers in Philadelophia; Wayne, Pa.; and Las Vegas, connected through SIP gateways to the branches.

Apria offices in Brooklyn, N.Y., and Elmsford, N.J., also lost voice and data service and the backup system took over their calls as well.

In all, 1,006 messages dealing with customers' needs were handled by Apria's remote, virtual PBX system. The system could be reprogrammed on the fly to select a new backup location if a designated one was knocked out. "Our product is not a nicety that people come and pick up. To maintain themselves, clients need our services," Slack said. And those 1,006 rerouted calls was proof of that after Sandy.

Not everyone was fortunate enough to have had a prior hurricane as an instructor. One hard lesson taught by Sandy was that the magnitude of the disaster can change the terms on which you thought your recovery plan was operating. The top priority of a plan is to keep a data center running; the top priority of government authorities might lie somewhere else.

1 of 2
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
Digital Transformation Myths & Truths
Transformation is on every IT organization's to-do list, but effectively transforming IT means a major shift in technology as well as business models and culture. In this IT Trend Report, we examine some of the misconceptions of digital transformation and look at steps you can take to succeed technically and culturally.
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll