Hurricane Sandy Lesson: VM Migration Can Stop Outages
When a hurricane or other disaster threatens, why not just move critical systems out of the way? It can be done -- but not at the last minute.
Who Is Hacking U.S. Banks? 8 Facts
(click image for larger view and for slideshow)
When it comes to disaster recovery, there's nothing like having learned your lesson before the disaster arrives. One of the most useful lessons is that virtualized systems now enable IT managers to move entire systems out of harm's way, even if that means half way across the country -- although it's necessary to have an alternative site set up well in advance.
Another lesson is that moderate-cost backup telephony services based on IP networks, including the Internet, and open-source code can serve as host substitutes for your local carrier and office PBX service, if those get knocked offline.
Asterisk and BroadSoft are providers of what's come to be known as SIP trunkline services. Another, Evolve IP, played a role in ensuring patients needing oxygen in the aftermath of Hurricane Sandy could still place their calls to Apria Health Care and get their deliveries.
Apria is a national deliverer of home health care services, which learned in 2005 when Katrina hit New Orleans that it could lose telephone service at the moment it was most needed. Apria supplies hospital beds, drug infusion equipment, and other medical goods and nursing services to homes of elderly, incapacitated and convalescing patients. But one of the most frequently needed services, especially in a disaster, is simply the personal oxygen tank.
During Katrina, Apria employees were incapacitated by the failure of office phone systems as the storm and its flood waters swept through the New Orleans region. That meant oxygen users who had left home to move in with relatives or find a room on higher ground had only what they had been able to carry away with them, often reflecting a hurried departure at the last minute. But when they picked up the phone -- their primary way of placing orders -- they often found the line to the Apria Health Care office was dead.
"Patients couldn't get through to us after Katrina," said David Slack, VP of IT network engineering for Apria. There was an alternative -- calling a national toll free number -- but few patients had it when they needed it. Something had to be done.
Apria looked at backups to its phone system from the major carriers, who expected the firm to spend $2,000 to $4,000 per office to upgrade to SIP trunkline services. Apria has 550 offices nationwide, which would have brought the total bill to over $2 million. It found a lower-cost substitute in the firms that carry enterprise voice and data over IP networks, including the Internet. BroadSoft and Asterisk use private branch exchange open-source code to provide such a service. Apria settled on Evolve IP, another provider, an enterprise version of the Vonage VOIP for consumers.
The Evolve IP system, providing a virtual PBX for Apria hosted in an Evolve IP data center, was installed a year ago, based on the lessons learned from Katrina. The hosted PBX needed to be accessible from any branch and have a call forwarding capability in case a Katrina-like storm should knock out a branch.
As Sandy gathered strength and was projected to veer into Middle Atlantic states, Apria's executives knew the system was going to get its first test. Twenty of its offices were directly in Sandy's path. The system needed to be programmed where to forward calls if the primary destination wasn't available, and Apria made those adjustments. If the Middletown, N.Y., branch lost telephone service, the system was to shift calls to the nearest well-staffed hub, in Cromwell, Conn.
"We lost both phone and data connectivity in Middletown. Immediately the phones started ringing in Cromwell, the backup site. All the intelligence on (emergency) routing was in the cloud. It understood instantly if a branch office was down. It worked fantastically," said Slack in an interview. In this case, the "cloud" is three private Apria data centers in Philadelophia; Wayne, Pa.; and Las Vegas, connected through SIP gateways to the branches.
Apria offices in Brooklyn, N.Y., and Elmsford, N.J., also lost voice and data service and the backup system took over their calls as well.
In all, 1,006 messages dealing with customers' needs were handled by Apria's remote, virtual PBX system. The system could be reprogrammed on the fly to select a new backup location if a designated one was knocked out. "Our product is not a nicety that people come and pick up. To maintain themselves, clients need our services," Slack said. And those 1,006 rerouted calls was proof of that after Sandy.
Not everyone was fortunate enough to have had a prior hurricane as an instructor. One hard lesson taught by Sandy was that the magnitude of the disaster can change the terms on which you thought your recovery plan was operating. The top priority of a plan is to keep a data center running; the top priority of government authorities might lie somewhere else.
Google in the Enterprise SurveyThere's no doubt Google has made headway into businesses: Just 28 percent discourage or ban use of its productivity products, and 69 percent cite Google Apps' good or excellent mobility. But progress could still stall: 59 percent of nonusers distrust the security of Google's cloud. Its data privacy is an open question, and 37 percent worry about integration.