9 Crucial Steps Toward Effective Disaster Recovery
A strong disaster recovery plan is key to keeping your business going in the face of a very bad, no good, horrible day. Here are nine steps IT pros should not skip in their planning and preparation for the worst.
![](https://eu-images.contentstack.com/v3/assets/blt69509c9116440be8/blt2a4771e1939fe6e6/64cb455b29c6a2574692ceff/Disaster-Plan_designer491_iStock_72643555_MEDIUM.jpg?width=700&auto=webp&quality=80&disable=upscale)
Everyone knows that, when it comes to computers, the question is not whether bad things will happen to good companies, but what those good companies do when bad things happen. If the company is very good, indeed, then it relies upon its IT disaster recovery plan to pull it through to the other side.
But the bad news is that many businesses are not prepared. According to survey results released last year by Ohio-based Nationwide Insurance, 75% of the 500 small businesses polled said they don't have a disaster plan, and an estimated 25% of businesses do not reopen following a major disaster.
Disaster recovery plans are insurance against very bad things, and like all forms of insurance they fall into the category of things that you pay for but hope against hope you never have to use. Since you hope you never have to use the disaster plan, some executives take them lightly, figuring that they've hired smart people who can think on their feet, perform heroic acts of improvisation, and make sure that the bits keep rolling along no matter how things get.
When a true disaster hits, executives like these are called "unemployed."
[See 9 Ways IT Can Ruin Its Relationship With the Business.]
I have spent a lot of time in various types of disaster response and business recovery activities. I live in a hurricane-prone state. I've been on governor-appointed emergency response committees.
I've worked in emergency communications during natural disasters, and I've talked to more than my share of people who have planned for or been the victim of business disasters. In all of these situations, a handful of things have come to light as key ingredients that make it much more likely that your business IT will come through the problems intact.
When you get right down to it, most of the following tips fall into one of two broad categories: The first is, "Know what you're going to do before you have to do it." The second is, "Don't keep your plan a secret."
Read on to get the details on these points and more.
This is one of those deceptively simple points. You might think that it's obvious why you would want a disaster preparedness plan, but let's start with a basic question: Is your IT disaster plan a stand-alone document, or is it part of a broader business continuity plan? That's the first and most important fork in the road you'll find when you start asking "why."
The reason a question like this matters is that it gets to issues like who will be in control of disaster response, how many employees are involved, and (perhaps most important) whether the goal of the plan is to help quickly pick up the pieces following a disaster, or keep operating seamlessly through the disaster. Those are two very different scenarios, and everyone involved in the plan should be clear about the goal before you have to push the big red "emergency" button in the control center.
Among other things, there can be a huge gulf in the costs between response and continuity. In a business continuity plan that includes IT, permissible response time to catastrophic events might be measured in seconds (or fractions of a second), while response time in disaster recovery might be measured in hours or even days. Ask why you're creating the plan, how big the plan is, and who's involved. The answers will inform pretty much every aspect of your plan.
Once in a while I'll run across an IT department that feels inclined to "simplify" disaster recovery plans by keeping them on a strict "need to know" basis. In some cases, that means that, if an individual or department doesn't have an active role to play in recovery, that party gets no information about what the plan contains. This is a huge mistake with potentially huge consequences.
Every stakeholder -- that is, every business unit that will feel the impact of your disaster recovery plan -- should be involved in creating the plan itself. Ask the units which applications are most critical on a day-to-day basis. Ask whether the choice of critical applications changes based on the day of the month (or of the quarter). Ask what their plans are when it comes to disaster recovery or business continuity, and how IT can best fit within those plans.
When you have involved all the stakeholders, you can formulate a plan that best meets the needs of the business. When the stakeholders are all involved, you have a much better chance of creating partners, rather than victims, when the worst happens.
When you're building your disaster recovery plan you typically work on a statistically-most-likely scenario and also on a worst-case scenario. Here's my advice: Whatever your worst-case scenario looks like, make it worse.
Let me give you a couple of examples of plans that didn't make "worst-case" nearly bad enough. The first involves doctoral and post-doc students at the University of Hawaii. Many of these students had been doing research for years, carefully gathering data, archiving that data, and putting copies of the archived files in fireproof safes in their department offices.
That was great until a 500-year flood came through the valley in which the school is located. In a number of cases, the researcher's computers were washed away, as were the fireproof safes. With no off-site copies, those students were faced with restarting projects and trying to recreate years of data.
The other example leaves academia for commerce. In coastal Mississippi towns like Bay St. Louis, many businesses had followed the best practice of keeping a-b-c backups, with master copies kept off-site in a bank safe-deposit box. The best practice failed when Hurricane Katrina hit, washing away the businesses, the banks, and the safe-deposit boxes. Data recovery was not an option when everything on which recovered data sets might be built was washed out to sea.
In both of these examples, organizations had taken reasonable, or even best practice, precautions. In both of these examples, those precautions weren't enough because they weren't based on a scenario that was sufficiently bad. Use your collective imaginations to create outlandishly bad scenarios, then plan to recover from those.
OK, I was just telling you that you should involve the stakeholders. You should communicate with them, too. Let them know what the plan is, make sure they get copies highlighting their roles in the plan, and be sure they're aware of changes to the plan and rehearsals of the plan. Yes, we're all aware of operational security and all that, but these are your colleagues in the business we're talking about. Involve them in the plan and, if anything, over-communicate the importance of their roles and the impact the plan will have on them. Remember, you want them working by your side, not standing in your way, when things go south in spectacular fashion.
When you're building your disaster recovery plan it will very quickly become obvious that you need help if you're going to pull everything off. Whether it's help in communicating with all your employees, help dealing with certain types of emergencies, or help in relocating people and equipment, you're going to have to work with outside companies and agencies to make it happen. You don't want those companies and agencies wondering about precisely what they're supposed to do when you call on them for help.
The first thing to do is figure out exactly what you need the partners to do. Once you have that list, you can begin contacting potential partners, choosing the partners, and ultimately working with them to find out how they can best work with you within your plan. You will want to find out how they can work with your other partners, how much advanced warning they need before beginning to act on your plan, and what their business continuity plans look like. There's nothing worse than having a critical partner too busy saving their own business to help save yours.
One of the funny things about disasters is that they so rarely occur at convenient times. No, they're often at their peak at 0300, or on a Sunday night -- any hour that makes it difficult to find and communicate with the employees who might be part of your disaster recovery plan or who will be effected by the plan going into action. I would pretty much bet that your first plan for communicating rests on the cell phone system, with some combination of text and voice messages for most employees and direct human-to-human calls for the most critical.
Since we're including worst-case scenarios here, you should assume that some employees won't have their smartphones, that cell towers will go down, and that some things won't work at all, because all those things happen in a disaster. What is your second layer of communication? Your third?
You should both confirm that the secondary and tertiary means of communications work and establish the conditions in which employees will turn to those channels. The last thing you want is to be broadcasting on your second and third layers of communication while your employees pay rapt attention to the defunct primary channel waiting for a message that will never come.
Let's start with something simple. On the last page, I wrote about having a second and third layer of communication should something happen to your primary methods. How will your employees know to look to those second and third layers, rather than simply waiting for a first-layer message that never comes? You have to figure out what your employees should do, and then tell them what to do.
I know that you've hired creative, innovative people for your IT staff, but a disaster is really not the time to be exercising that creativity unnecessarily. At the very least, your disaster recovery plan should provide limits within which the creativity can be exercised. There should be procedures in the plan for how everyone will communicate and work together if (when?) events get ahead of the plan in one significant area or another.
For all of this to work you must train your employees in what's expected. Train them in ways to communicate during an emergency. Train them on the hierarchy of priorities (e.g. human life is above everything else), and on how to work with your partners. Train them. Train them thoroughly. Train employees you don't think you should have to train.
This is important. Once you have the plan developed, approved, communicated, and enforced by training, you're about halfway home. Now, you have to practice the plan. Don't just rehearse activities that the plan has been designed for; you want to make sure that your plan is effective for disasters that you haven't specifically planned for. An external auditor or facilitator may be useful, since that party has no vested interest in making the plan's creators look good.
You want to practice the plan more than once. At least once a year, plan on a day (or more) to run through a SET (Simulated Emergency Test). Involve your partners and, if necessary, local first responders to make sure that all the extra-organizational players know their roles and are able to do what's required to keep your business up and running (or able to quickly recover) no matter what the universe throws at it.
After your SETs, after your conversations with all the stakeholders, and after you truly understand why you're developing the plan to begin with, be brutal with what you think you've learned. Be clear that you're criticizing faults in the plan, not with individuals, but pick through all the weaknesses and points of confusion.
Know that whatever stress you've applied to the plan and its execution is mild compared to the stress that the real world will apply. The more you learn (and apply the knowledge you've gained), the better your plan will be in protecting your company when things go wrong.
If you ever have to put your disaster recovery plan into effect, it will be in the middle of a very bad, no good, horrible day. The more thorough your planning and preparation, and the more complete your testing and rehearsal, the more likely it will be that the plan's execution will keep your business alive and running through the worst that can befall it.
Have you had to act on your organization's disaster recovery plan? How do you make sure that the worst won't be the last for your organization? I'd love to hear about your experience with disaster recovery and business continuity plans. Please leave a comment to let me know what you've done.
After your SETs, after your conversations with all the stakeholders, and after you truly understand why you're developing the plan to begin with, be brutal with what you think you've learned. Be clear that you're criticizing faults in the plan, not with individuals, but pick through all the weaknesses and points of confusion.
Know that whatever stress you've applied to the plan and its execution is mild compared to the stress that the real world will apply. The more you learn (and apply the knowledge you've gained), the better your plan will be in protecting your company when things go wrong.
If you ever have to put your disaster recovery plan into effect, it will be in the middle of a very bad, no good, horrible day. The more thorough your planning and preparation, and the more complete your testing and rehearsal, the more likely it will be that the plan's execution will keep your business alive and running through the worst that can befall it.
Have you had to act on your organization's disaster recovery plan? How do you make sure that the worst won't be the last for your organization? I'd love to hear about your experience with disaster recovery and business continuity plans. Please leave a comment to let me know what you've done.
-
Read more about:
Business Continuity/Disaster RecoveryAbout the Author(s)
You May Also Like