It Is Time to Rethink Disaster Recovery

On April 19, 1995, Timothy McVeigh and others destroyed the Murrah federal building in Oklahoma City, Oklahoma. It was, to date, the worst case of terrorism in the United States since the Civil War. On that day a number of things changed, but the biggest lessons were not well learned.

On September, 11, 2001, Al-Qaeda crashed air craft into the financial district of New York City and the Pentagon in Washington, DC. It was the worst case of terrorism to date since Oklahoma City. A number of things changed, but some of the largest lessons were still not well learned.

Since early 2003, a strain of the influenza virus (H5N1) has been making its way around the world. Call it bird (avian) flu, call it swine flu (which is really a different strain, H1N1, and more virulent to humans), but in the late summer of 2009, it is still with us and according to the World Health Organization we are officially in the grips of a pandemic. Perhaps now is the time to review the lessons of both Oklahoma City and September 11, because a full-blown influenza outbreak could be more telling than either Oklahoma or September 11 of just how prepared our systems really are.

Following the bombing of the Murrah building, a long investigation ensued. During this time a number of companies went out of business. Following the collapse of the Twin Towers, a long investigation ensued and a number of companies went out of business. In the event of a pandemic, it is safe to say a number of businesses will go out of business. Why? Not because of loss of life, but because the people working for these companies could not reach their facilities, and the IT infrastructure, to conduct the routine business they were employed to do, everything from CPAs to fulfillment houses. The damage done in 1995 closed several city blocks in Oklahoma City and a number of companies, mostly small, lost access to their facilities during this time because it was a crime scene. Similarly, large parts of downtown New York were cordoned off for safety reasons, as well as because it was a crime scene. A pandemic flu could have similar implication without a single piece of yellow tape.

As an IT architect, it is my job to build a robust, redundant system. But, like most, my assumptions were based on the availability of people to be able to go to the disaster recovery site or take the tapes to XYZ recovery company or make sure my disaster site is x number of miles away from my primary site. These are some of the lessons, and the financial industry learned them and executed them quite successfully in the days following September 11, 2001, but in every picture and description I saw, makeshift tables were layered with machines and wires, clearly set up on the fly by IT professionals – in many cases after working long hours to get the job done. Disaster recovery of that scale worked. But what happens when the disaster is not a loss of systems, but a loss of access to the systems and a loss of the manpower to run them?

In the event of a pandemic, the experts have made the following predictions. First, absenteeism could be as high as 40%. For an IT staff of 10, that is 4 people out sick, either sick themselves or caring for someone who is sick. Second, depending on severity, mandatory separation may be instituted. The standard is six feet. Think about how far from your co-worker your desk is right now. Think about how you get to work, and how you would get to work if you could not sit within six feet of someone. It puts a whole new spin on the issues of mass transit. Finally, depending on the management of your company, rotation schedules might be implemented where half the staff is at home while half the staff is in the office. What sort of impact would that have on your IT services and your ability to manage your IT infrastructure? And are you ready for the level of remote access requests that will come flooding into the department and the issues of fulfilling these requests?

As I have said a number of times, those of us who work in IT just cannot win. When things are humming along smoothly, the bean counters are wondering why they are paying us, and when things are crashing down around you, the bean counters are wondering why they are paying us. In tight times, IT is almost always the first department to suffer cuts. Usually, those cut are at the top and the bottom of the stack, leaving those in the middle to bear the load, often without being properly briefed on the various back doors, trap doors and the ever popular what does that box do?. In the late 1990s and early 2000s, a number of companies, in cost cutting moves, dumped real estate and went to remote access. Over the last five odd years, those telecommuting trends have reversed as management and employees want to be seen as valuable (and thus remain employed), and the communications lines have been slashed as a useless expense, without the forethought of disaster preparedness. As IT people, we are beholden to the budgets. Most of us work for companies that are more concerned with the quarterly stock price and how it can be boosted for the next quarter, with very little long term strategic planning being done. But it does not have to be this way.

It is our responsibility to exercise the disaster recovery plans. So we have the opportunity to apply new tactics to the disaster recovery scenarios. Suggest that the next DR test include a 40% staff cut. Roll dice, generate the names randomly, whatever works for you, and tell them just not to show up. Can you put the systems back on line? What happens if you cannot get the tapes to the DR site? What happens if your remote access systems do not work? These are only some of the things we should be thinking about and preparing for.

As IT professionals, we tend to get bore sighted on hardware and software, in many cases down in the weeds so deep we do not see how all the parts go together, or what other parts are needed, or as one former boss used to remind me, for want of a nail…. So, as we sit in the middle of hurricane season, with tornadoes popping up in unusual places and with increased ferocity, remember that the winter is coming, that there are other concerns out there and we should be considering an all-hazards approach in our disaster planning. And sometimes that means nothing happens to the equipment.


David Lane, KG4GIY is a member of Linux Journal's Editorial Advisory Panel and the Control Op for Linux Journal's Virtual Ham Shack


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Hey Dave

sms kostenlos's picture

Hello Dave

Very good contribution. But I find that somehow very unfair. I once read somewhere a contribution, it is claimed that the 11 September was caused by the Americans themselves.
In 1960 there were ever such a case. Codename: Fluctuating flag.

Gregor S.

Website: Free SMS

H5N1 vs H1N1

DavidC's picture

Somewhat minor point, but H1N1 caused the 1918 pandemic. It has also been a strain of seasonal flu for decades. The current H1N1 is a novel form of the virus, but it NOT the strain that emerged in 2003. That was H5N1 avian flu or bird flu. It is still out there perculating and could merge with the new H1N1 and really make things interesting.


David Lane's picture

Yes, sorry, the Avian flu was H5N1...I will correct that in the is hard enough to keep the acronyms in our field straight!

David Lane, KG4GIY is a member of Linux Journal's Editorial Advisory Panel and the Control Op for Linux Journal's Virtual Ham Shack


Anonymous's picture

Sorry, but it is way overkill to prepare for something as specific as a terrorist attack. You are FAR MORE LIKELY to lose all your employees because you sent them all to a training session on a single plane or bus and it is destroyed in an accident.

Reply to specific as Terrorist attack!

Pete G's picture

Maybe the thought isn't that specific.

I think it's a great idea. I now make a living at it!
Maybe it is just an example of many things that could be as disastrous or more so.

Send them all on the same plane or bus!
Damned if that wouldn't be as stupid as it gets!
In Corporate Aviation where I used to work, "in Avionics then later as a pilot" never is a department or more than two of the top executives allowed on a single plane. Even two planes are used at double cost for two top execs going to the same destination for the safety of the companies assets. Yes an employee is an asset.
Here you had better be an asset!
By the way corporate aircraft are usually maintained way above the requirements for the Airline industry.
All my backup servers in environmentally controlled space which is also a double Faraday cage. The cages are electrically isolated from each other than the common ground.
All the drives are in another Faraday cage within the first two making it basically triple RF protected. All data is transferred in via optical system, "that would be wireless. How is my business."
No it isn't radio! No it isn't IR either. After all IR is radio that is just below the frequencies you can see anyway.
The power is protected via being generated inside the Faraday Cage. Yep in that cage is a large UPS and some large APC UPS boxes as a backup to the big one.
The power is transferred via a wireless system also.
Hint on the power transfer.
Contrary to popular belief, H2O, "water for those who need a description" is not a conductor! It is an insulator! Believe it!
Yes I have a large unit to power my house. That's where the business is now days.
It powered us through IKE. We were told power would be off for six weeks. Fortunately it was only about two and a half weeks. We had power 24/7 with full AC, lights, Satellite, Refrigerator, Freezers, electric stove and two microwaves.
Hopefully our generator will survive an EMP!
It is older with no electronics, "solid state at least."
No ignition electrics/electronics at all.
It is regulated by magnetic feed back system.
It is, believe it or not in a Faraday cage just because I'm anal about it!
I have several desktop and laptop computers stored in sealed metal cases in the Faraday cages.
The cases have been flushed with dry nitrogen until completely dry then stored with 1.5 PSI dry nitrogen. They are connected to a large bottle of nitrogen via two stage regulator and the bottle has an alarm via sensor that will detect pressure lower than 300 PSI and a change in flow to the boxes.
Guns are also stored in dry nitrogen. No they are not on my property! When they come for them. Sooner than later! Count on it. I will not own any but the one I carry. I'll give it to them!
I have no fear of being laid off or fired.
You have to get a job before you can be laid off.
I can't get a job.
Age discrimination is alive and well.
The excuses are infinite but the truth is, I'm older than most of you in this business and experience and qualifications do not count in this world any more. Quality now is defined by quantity not how well you do your job, but how fast you do it.
Remember that and prepare for old age! It is coming!
Licenses don't count. Experience doesn't count, a promise to get the license and being young will get you the job even though you came to the interview that states you must have the license to qualify for the interview/job with just a promise.
That will get you shot out the door here as you are dishonest and obviously have no regard for the rules! First thing here is honesty ! Without that, I/we do not want you!
I bet he is a good little lying Democrat!
In general though it probably will help get you hired in this new environment where it is OK for the, Alleged President, "where is your Birth Certificate Obama" and the Socialist lying Democrats in Congress to lie and get caught every day with no consequences.

No! I'm not a damned Republican either. Never have been.
I sure would vote for Palin though! John McCain can kiss my butt! He is a damned Socialist and liar!
I knew him on the USS Forrestal. The little ButtWipe!
Being older and having years of experience and the license in hand does not count!
I know it happened to me!
Just so you can see that Affirmative Action works, yes the guy who came to the interview stating, "I'll be taking the test in a week" was hired.
He is black, I'm not.

"After listening to him" for about thirty minutes my bet is he has not passed the test, "and never will!"
After all there is about a 70% failure rate on the required tests. Most people taking the tests are College Engineering Seniors required by the course to take the tests.
He was or is a nice guy but he thinks BS , lying and promises are more important than the rules.
In his case, “HE'S RIGHT!” I also came away feeling as if his IQ was some where around the SQ Root of his shoe size, “which from observation is probably less than a two digit number to start with!”

I've had the required license since 1968 when the only thing you could bring in to the test was a slide rule. The Post Versalog slide rule I took had functions that put it in the category of not allowed.
Fortunately I did have a basic six inch slide rule in my shirt pocket I was allowed to use.
I'm not going to tell you what a slide rule is.
If you don't know your too young or too ignorant to work here anyway.
Besides if you can't do a square root or square in your head we don't need you.
If you can't state and use Ohm's law and a few other basic electronics formula and law, "we don't need you!"

So, no I'm not worried about being laid off.
I work for myself.
Thanks Affirmative Action for pushing me off the ledge to get it done myself!

Should have done it years ago!

Am I angry?
Hell yes! My country is going to hell.
My friends and family have died for it and I nearly did!

Am I prejudiced? Hell no!

One of my three best friends is black!
His daughter is my God Daughter.
We've been there for each other through the good times and the bad times.
No he does not work here!
If you tell me you have more than one to three so called best friends, you are delusional.
No one has that many real friends. Just wait until you are in trouble. You'll find out!

I make the rules here!
You lie you get kicked out the door.
Maybe even kicked in the ass!
Can I do that?
I'm sure willing to start the contest and "SOMEBODY'S ASS WILL BE KICKED AT THE END OF THE CONTEST!"
You bring a MicroSnot system in here and hook it to my network, wired/wireless........ read the above about lying.
When you arrive here you first read and sign our normal company agreement.
In that agreement you are required to have a Concealed Hand Gun License!
If you do not have a CHL from Texas or a Reciprocating state. You will not finish the interview until you have obtained one.
If you otherwise are qualified for the job, we will pay for the class. An all day Saturday class. After about six weeks if you passed you will have your CHL and the interview will be continued and you may be hired at that time.

Because I can't legally do much of a back ground check on you.
I really can't afford it either!
They can and I'll take the license as a back ground check.

Most companies do not allow hand guns on the property even with a CHL.
To work here you will be required to have a CHL and carry at all times while on company property.
While on company property you may choose to open carry or go concealed. We don't care.
I'll assume you are carrying and I may ask and you had better be, "if you want to keep your job!"
If you can't afford a Hand Gun we will finance one of the proper type to match your CHL but you will carry on this property!
Come on punks, drop by here and start shooting people!

You will be required to join the NRA!
We'll pay for it!
You have a problem with that!
There's the door!
Even mention the word, "Labor Union" in a positive way and you're out of here. I may go postal on that one.
If that pisses you off just remember when you arrive to go postal that everyone in the building is carrying a hand gun, or they better be!

Obama Ain't my Moma!

Both God and Obama have no birth certificate.
The difference is God does not think he is Obama!

Do I have a problem with Obama?
Obviously I do!

Is it because he is black! NO!

Linux is Unix on Steroids!
MicroSnot is Unix on LSD!
MicroSnot Security is an Oxymoron!

Good luck!
Pete G

Dammmmmm I love TOR!


David Lane's picture

I would agree that the chances of losing your staff in a bus crash is much higher. And you are one of the people that get it. Most DR plans are not taking the human quotient into consideration, which is why they fail more often than succeed.

David Lane, KG4GIY is a member of Linux Journal's Editorial Advisory Panel and the Control Op for Linux Journal's Virtual Ham Shack

Online Data Backup

MikeGNLE's picture

I work for an online data backup company, and I was just curious as to what your opinion is of online backup? Do you see this service as something that can be of value to you DR planning? Are IT professionals ever concerned that an automated type solution could lead to jobs being eliminated? Thanks, Mike


David Lane's picture

Backing up the data serves (generally) two purposes:

1) Recovering data that the users delete, either intentionally or accidentally.

2) Recovery in the event of a catastrophic failure.

Any solution that allows you to get back up and running in a short period of time is good. The question then becomes How many moving parts do you have to have in place to get back up and running?

Online backups are good if your connection to the online company is robust and stable. If you were one of millions of customers that went through the central office that was beneath the Twin Towers in New York, you might be telling me that online backups stink...

If you live in DC, where it can take upwards of 60 days to provision a new circuit...then you know online backups probably are not a great solution for the near-line data, but might be sufficient, if you can wait the 60 days, for some of the off-line stuff.

I can see advantages if you are a multi-site operation. I can also see serious, critical disadvantages too.

David Lane, KG4GIY is a member of Linux Journal's Editorial Advisory Panel and the Control Op for Linux Journal's Virtual Ham Shack