Scary Backup Stories

The dangers of not testing your backup procedures and some common pitfalls to avoid.

Backups. We all know the importance of making a backup of our most important systems. Unfortunately, some of us also know that realizing the importance of performing backups often is a lesson learned the hard way. Everyone has their scary backup stories. Here are mine.

Scary Story #1

Like a lot of people, my professional career started out in technical support. In my case, I was part of a help-desk team for a large professional practice. Among other things, we were responsible for performing PC LAN backups for a number of systems used by other departments. For one especially important system, we acquired fancy new tape-backup equipment and a large collection of tapes. A procedure was put in place, and before-you-go-home-at-night backups became a standard. Some months later, a crash brought down the system, and all the data was lost. Shortly thereafter, a call came in for the latest backup tape. It was located and dispatched, and a recovery was attempted. The recovery failed, however, as the tape was blank. A call came in for the next-to-last backup tape. Nervously, it was located and dispatched, and a recovery was attempted. It also failed because this tape also was blank. Amid long silences and pink-slip glares, panic started to set in as the tape from three nights prior was called up. This attempt resulted in a lot of shouting.

All the tapes were then checked, and they were all blank. To add insult to injury, the problem wasn't only that the tapes were blank--they weren't even formatted! The fancy new backup equipment wasn't smart enough to realize the tapes were not formatted, so it allowed them to be used. Note: writing good data to an unformatted tape is never a good idea.

Now, don't get me wrong, the backup procedures themselves were good. The problem was that no one had ever tested the whole process--no one had ever attempted a recovery. Was it no small wonder then that each recovery failed?

For backups to work, you need to do two things: (1) define and implement a good procedure and (2) test that it works.

To this day, I can't fathom how my boss (who had overall responsibility for the backup procedures) managed not to get fired over this incident. And what happened there has always stayed with me.

A Good Solution

When it comes to doing backups on Linux systems, a number of standard tools can help avoid the problems discussed above. Marcel Gagné's excellent book (see Resources) contains a simple yet useful script that not only performs the backup but verifies that things went well. Then, after each backup, the script sends an e-mail to root detailing what occurred.

I'll run through the guts of a modified version of Marcel's script here, to show you how easy this process actually is. This bash script starts by defining the location of a log and an error file. Two mv commands then copy the previous log and error files to allow for the examination of the next-to-last backup (if required):

    #! /bin/bash
    backup_log=/usr/local/.Backups/backup.log
    backup_err=/usr/local/.Backups/backup.err
    mv $backup_log $backup_log.old
    mv $backup_err $backup_err.old

With the log and error files ready, a few echo commands append messages (note the use of >>) to each of the files. The messages include the current date and time (which is accessed using the back-ticked date command). The cd command then changes to the location of the directory to be backed up. In this example, that directory is /mnt/data, but it could be any location:

    echo "Starting backup of /mnt/data: `date`." >> $backup_log
    echo "Errors reported for backup/verify: `date`." >> $backup_err
    cd /mnt/data

The backup then starts, using the tried and true tar command. The -cvf options request the creation of a new archive (c), verbose mode (v) and the name of the file/device to backup to (f). In this example, we backup to /dev/st0, the location of an attached SCSI tape drive:

    tar -cvf /dev/st0 . 2>>$backup_err

Any errors produced by this command are sent to STDERR (standard error). The above command exploits this behaviour by appending anything sent to STDERR to the error file as well (using the 2>> directive).

When the backup completes, the script then rewinds the tape using the mt command, before listing the files on the tape with another tar command (the -t option lists the files in the named archive). This is a simple way of verifying the contents of the tape. As before, we append any errors reported during this tar command to the error file. Additionally, informational messages are added to the log file at appropriate times:

    mt -f /dev/st0 rewind
    echo "Verifying this backup: `date`" >>$backup_log
    tar -tvf /dev/st0 2>>$backup_err
    echo "Backup complete: `date`" >>$backup_log

To conclude the script, we concatenate the error file to the log file (with cat), then e-mail the log file to root (where the -s option to the mail command allows the specification of an appropriate subject line):

    cat $backup_err >> $backup_log
    mail -s "Backup status report for /mnt/data" root < $backup_log

And there you have it, Marcel's deceptively simple solution to performing a verified backup and e-mailing the results to an interested party. If only we'd had something similar all those years ago.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

I have been using Outlook Exp

outlook express errors's picture

I have been using Outlook Express errors repair tool for the past few months and LOVE it.
I've even recommended it to all my friends. Outlook Express more stable and secure than other programs.

Where are your recovery tools?

Anonymous's picture

I've been working on a backup-to-CD/DVD tool - it's designed more for backups of the critical GB or so of data, not the stuff that you should have on original distribution media, etc. Anyway, I've tried to incorporate the hard-earned lessons from others - every CD is stamped with a unique ID and a metadata file that's conveniently also a Java resource file, the data format is standard (tar with zlib compression for the data, Berkeley DB for an index). I even reset the compression engine before each file so that most files can be recovered even if the media is damaged.

All sounds fine, until you realize that the main reason I'm using CD-R instead of tapes (besides availability of CD-R burners vs tape drives on consumer-grade hardware) is the fact that it's a random access media. My compressed tar file is SEEKABLE. This means that I don't have to read and decompress the entire disc to get a single file from the archive - my tools can read the index and extract the right file within no perceptable delay over the time required to copy any file from CD.

`

The killer problem I hit during testing? Where are the extraction tools? I can use tar if I need to, but reading and decompressing an archive that fills a CD puts an incredible load on the system.

My solution is to have the archive put the extract tools on the CD itself. Fortunately that's trivial since I just need to copy the program and libraries into the source directory crunched by mkisofs.

Re: Where are your recovery tools?

Anonymous's picture

hmm. would it not be possible to make a 700M image of something like LNX-BBC (itself less than 50 mb)?
The image is copied (given the time, date as name) then mounted -o loop,
A backup is then written to "/backupdir" on that image (about 650M free) with compression of choise, then unmounted and written with cdrecord...
imho something like that should be possible, or am i way out.
that would give you a working system to restore things with... AND your backupfiles.

Re: Where are your recovery tools?

brianlane's picture

This is a good solution, I tried out a couple of bootable backup scripts, but they were all too darn hard to get to work. Currently I'm using multiple extra harddrives on seperate computers. I've got about 4G per backup, so CDs are out the question now.

Brian

Re: Scary Backup Stories

Anonymous's picture

-:- Many moons ago, a predesessor at an electronics shop (which shall remain nameless ;-)) sold a system

To a client (an 8088 based system, dos 2.2) - then admonished the client to do regular backups

showing him how, and making sure the client knew. So the

tech finished up and left feeling sure everything was

O.K. and it seemed to be the client was dutifully doing as he had been instructed backing up to the 5 1/4 floppies.

Well, time came came to use the disks, and they were blank! he brought them into the shop and they were replaced

since he was doing the backup as had been tought to him,

this happened a couple of times, so the previous tech

went out to check what was going on -- Only to find the client had been storing the disks in the most

convenient place he could possibly find --- sticking them to the side of the computer case with a large speaker magnet!

'Doh

Ken

Re: Scary Backup Stories

Anonymous's picture

Take a page out of Walmart's book regarding backups.. Somehow on their AIX system they have it set up so that it checks the date on the tape every hour, If it is yesterday's backup it will send a wall message (as root) to all terminals. In one supercenter alone there are 6+ desk terminals and 40 or so hand held terminals which also get the message.

The managers really hate it when they suddenly get 30 calls reporting their oversight. I usually never get that message more than 3 times a day on very busy days.

DRACO-

Re: Scary Stories topics

Anonymous's picture

I need Ideas

Re: Scary Backup Stories

Anonymous's picture

Well if you still use the "long ago" tape back up, then you should at the least look at the data on the disk before you assume the back up has gone as planned, dont get me wrong they were good in their day, about 15 years ago.

l8er

Re: Scary Backup Stories

Anonymous's picture

Sometimes even checking your backups isn't enough. You also have to take your environment into account.
A few years back I worked at a military school which taught the use of various systems, radar included. My office was located directly under a very old and never-used radar tower (a small one). We used best practices and used a combination of full and incremental backups with multiple sets of tapes. After each backup we even test restored random files.
One summer, we were directly in the path of a hurricane. We packed up all of our tapes and sent them out of state with our IT manager. Upon return, we had immediate need for backups for two systems that we'd left online (no choice) and had subsequently lost.
Yep, you guessed it. Each and every tape was compeletely blank. Seems that during the one week I chosen to take vacation, the techs had decided to do maintenance on the radar and had "radiated" for about 1/2 hour. Made for one heck of a bulk eraser.
After that, we used CDROM. Cost more, but it was safer, given the environment we were in.

Re: Scary Backup Stories

Anonymous's picture

At an unnamed location it happened thus...

The customer had been using a home built 'tar' -based backup system for a long

time. They were informed enough to have even tested and verified that recovery

would work also.

Everything had been working fine, and they even had to do a recovery which

went fine.

Well, one day something evil happened to a disk and they had to replace the

unit and do a full recovery. Things looked fine until someone noticed that

a directory with critically important and sensitive data was missing.

Turned out that some manager had decided to 'secure' the directory by

doing 'chmod 000 dir' to protect the data from inquisitive eyes when the

data was not being used.

Of course, tar complained about the situation and returned with non-null

status, but since the backup procedure had seemed to work fine, no one

thought it necessary to view the logs...

Re: Scary Backup Stories

Anonymous's picture

To this day, I can't fathom how my boss (who had overall responsibility for the backup procedures) managed not to get fired over this incident. And what happened there has always stayed with me.

Maybe his boss had a clue. :-)

Assuming your boss just made a poor decision, and wasn't terminally stupid, don't you think that after this, he'll be probably the most careful person in the world about backups? And maybe his boss was smart enough not to make the knee-jerk reaction of firing him and hiring someone else who may not have learned this lesson the hard way.

Dale Carnegie writes in How to Make Friends and Influence People of a pilot who used to fly vintage World War II-era planes at air shows. One day, while flying at an air show, the engine sputtered and died. Luckily, he made a safe emergency landing, but the engine was wrecked--the mechanic at the airfield, not used to servicing old planes, had filled the tank with jet fuel instead of gasoline.

When they found out, the mechanic pretty much feared for his life, and apologized profusely The pilot would have been justified in demanding that the mechanic be fired--after all, the mistake could have killed him. Instead, he asked that that mechanic be assigned to his planes for the remainder of the show. He knew that after that incident, the mechanic would be twice as careful as anyone else at the airfield.

Naturally, this won't work with everyone. Some people, sadly, just don't care, and some really can't learn from mistakes. But the majority of people can. I'd rather hire or keep someone who's learned the hard way than someone who hasn't learned at all.

Re: Scary Backup Stories

barryp's picture

Yeah, that was a bit of a throw-away line. I agree, most people should be allowed to make *** one *** mistake. Thanks for the comment.Cheers, Paul.

Re: Scary Backup Stories

Anonymous's picture

My startup company had just been acquired by a much larger company, who wanted our flagship product. At the startup, we didn't have a network, let alone source control (this was in 1993). The manager from the new company tried to convince me to put the source code in the corporate standard source control system, but I kept pushing him off, citing deadline reasons. I finally agreed to at least keep a copy of the sources a backed-up network server.

One day I was trying an approach to fixing a bug, and it didn't work out. No problem, I thought, I'll just wipe the local copy and pull it down from the server. I wiped the local copy, changed directories to the network server, and was about to copy when some co-workers came by to go to lunch.

After lunch, I remembered what I was doing. Ah, yes, that bug fix that didn't work out. No problem, I thought. I'll just wipe the local directory and pull it down from the server. Wipe... umm... wait... that WAS the server... well, I still have the local, umm, no, I wiped that before lunch. Ummm....

One server tape restore (from the previous night) and six hours later, and the code was in source control. Nothing like your own carelessness to teach humility.

Home backup solutions

Anonymous's picture

If you want to back up data at home, you might want to consider back up to hard drive. Large hard disks are very affordable now, and with software like amanda-tapeio (it's a CVS branch), you can perform automated backup to the hard disk. Since it's essentially amanda software (that just happens to write data to files instead of to a block device), you can implement any sort of backup policy that amanda allows. Recovery can be done with "amrecover" tool just like in tape-based amanda.

The ULTIMATE Horrors story with recovery!

Anonymous's picture

Its here.. Unbeliveable..

[I had intended to leave the discussion of "rm -r *" behind after

the compendium I sent earlier, but I couldn't resist this one.

I also received a response from rutgers!seismo!hadron!jsdy (Joseph S. D. Yao)

that described building a list of "dangerous" commands into a shell

and dropping into a query when a glob turns up. They built it in

so it couldn't be removed, like an alias. Anyway, on to the story! RWH.]

I didn't see the message that opened up the discussion on rm, but

thought you might like to read this sorry tale about the perils of

rm....

(It was posted to net.unix some time ago, but I think our postnews

didn't send it as far as it should have!)

----------------------------------------------------------------

Have you ever left your terminal logged in, only to find when you came

back to it that a (supposed) friend had typed "rm -rf ~/*" and was

hovering over the keyboard with threats along the lines of "lend me a

fiver 'til Thursday, or I hit return"? Undoubtedly the person in

question would not have had the nerve to inflict such a trauma upon

you, and was doing it in jest. So you've probably never experienced the

worst of such disasters....

It was a quiet Wednesday afternoon. Wednesday, 1st October, 15:15

BST, to be precise, when Peter, an office-mate of mine, leaned away

from his terminal and said to me, "Mario, I'm having a little trouble

sending mail." Knowing that msg was capable of confusing even the

most capable of people, I sauntered over to his terminal to see what

was wrong. A strange error message of the form (I forget the exact

details) "cannot access /foo/bar for userid 147" had been issued by

msg. My first thought was "Who's userid 147?; the sender of the

message, the destination, or what?" So I leant over to another

terminal, already logged in, and typed

grep 147 /etc/passwd

only to receive the response

/etc/passwd: No such file or directory.

Instantly, I guessed that something was amiss. This was confirmed

when in response to

ls /etc

I got

ls: not found.

I suggested to Peter that it would be a good idea not to try anything

for a while, and went off to find our system manager.

When I arrived at his office, his door was ajar, and within ten

seconds I realised what the problem was. James, our manager, was

sat down, head in hands, hands between knees, as one whose world has

just come to an end. Our newly-appointed system programmer, Neil, was

beside him, gazing listlessly at the screen of his terminal. And at

the top of the screen I spied the following lines:

# cd

# rm -rf *

Oh, *****, I thought. That would just about explain it.

I can't remember what happened in the succeeding minutes; my memory is

just a blur. I do remember trying ls (again), ps, who and maybe a few

other commands beside, all to no avail. The next thing I remember was

being at my terminal again (a multi-window graphics terminal), and

typing

cd /

echo *

I owe a debt of thanks to David Korn for making echo a built-in of his

shell; needless to say, /bin, together with /bin/echo, had been

deleted. What transpired in the next few minutes was that /dev, /etc

and /lib had also gone in their entirety; fortunately Neil had

interrupted rm while it was somewhere down below /news, and /tmp, /usr

and /users were all untouched.

Meanwhile James had made for our tape cupboard and had retrieved what

claimed to be a dump tape of the root filesystem, taken four weeks

earlier. The pressing question was, "How do we recover the contents

of the tape?". Not only had we lost /etc/restore, but all of the

device entries for the tape deck had vanished. And where does mknod

live? You guessed it, /etc. How about recovery across Ethernet of

any of this from another VAX? Well, /bin/tar had gone, and

thoughtfully the Berkeley people had put rcp in /bin in the 4.3

distribution. What's more, none of the Ether stuff wanted to know

without /etc/hosts at least. We found a version of cpio in

/usr/local, but that was unlikely to do us any good without a tape

deck.

Alternatively, we could get the boot tape out and rebuild the root

filesystem, but neither James nor Neil had done that before, and we

weren't sure that the first thing to happen would be that the whole

disk would be re-formatted, losing all our user files. (We take dumps

of the user files every Thursday; by Murphy's Law this had to happen

on a Wednesday). Another solution might be to borrow a disk from

another VAX, boot off that, and tidy up later, but that would have

entailed calling the DEC engineer out, at the very least. We had a

number of users in the final throes of writing up PhD theses and the

loss of a maybe a weeks' work (not to mention the machine down time)

was unthinkable.

So, what to do? The next idea was to write a program to make a device

descriptor for the tape deck, but we all know where cc, as and ld

live. Or maybe make skeletal entries for /etc/passwd, /etc/hosts and

so on, so that /usr/bin/ftp would work. By sheer luck, I had a

gnuemacs still running in one of my windows, which we could use to

create passwd, etc., but the first step was to create a directory to

put them in. Of course /bin/mkdir had gone, and so had /bin/mv, so we

couldn't rename /tmp to /etc. However, this looked like a reasonable

line of attack.

By now we had been joined by Alasdair, our resident UNIX guru, and as

luck would have it, someone who knows VAX assembler. So our plan

became this: write a program in assembler which would either rename

/tmp to /etc, or make /etc, assemble it on another VAX, uuencode it,

type in the uuencoded file using my gnu, uudecode it (some bright

spark had thought to put uudecode in /usr/bin), run it, and hey

presto, it would all be plain sailing from there. By yet another

miracle of good fortune, the terminal from which the damage had been

done was still su'd to root (su is in /bin, remember?), so at least we

stood a chance of all this working.

Off we set on our merry way, and within only an hour we had managed to

concoct the dozen or so lines of assembler to create /etc. The

stripped binary was only 76 bytes long, so we converted it to hex

(slightly more readable than the output of uuencode), and typed it in

using my editor. If any of you ever have the same problem, here's the

hex for future reference:

070100002c000000000000000000000000000000000000000000000000000000

0000dd8fff010000dd8f27000000fb02ef07000000fb01ef070000000000bc8f

8800040000bc012f65746300

I had a handy program around (doesn't everybody?) for converting ASCII

hex to binary, and the output of /usr/bin/sum tallied with our

original binary. But hang on---how do you set execute permission

without /bin/chmod? A few seconds thought (which as usual, lasted a

couple of minutes) suggested that we write the binary on top of an

already existing binary, owned by me...problem solved.

So along we trotted to the terminal with the root login, carefully

remembered to set the umask to 0 (so that I could create files in it

using my gnu), and ran the binary. So now we had a /etc, writable by

all. From there it was but a few easy steps to creating passwd,

hosts, services, protocols, (etc), and then ftp was willing to play

ball. Then we recovered the contents of /bin across the ether (it's

amazing how much you come to miss ls after just a few, short hours),

and selected files from /etc. The key file was /etc/rrestore, with

which we recovered /dev from the dump tape, and the rest is history.

Now, you're asking yourself (as I am), what's the moral of this story?

Well, for one thing, you must always remember the immortal words,

DON'T PANIC. Our initial reaction was to reboot the machine and try

everything as single user, but it's unlikely it would have come up

without /etc/init and /bin/sh. Rational thought saved us from this

one.

The next thing to remember is that UNIX tools really can be put to

unusual purposes. Even without my gnuemacs, we could have survived by

using, say, /usr/bin/grep as a substitute for /bin/cat.

And the final thing is, it's amazing how much of the system you can

delete without it falling apart completely. Apart from the fact that

nobody could login (/bin/login?), and most of the useful commands

had gone, everything else seemed normal. Of course, some things can't

stand life without say /etc/termcap, or /dev/kmem, or /etc/utmp, but

by and large it all hangs together.

I shall leave you with this question: if you were placed in the same

situation, and had the presence of mind that always comes with

hindsight, could you have got out of it in a simpler or easier way?

Answers on a postage stamp to:

Mario Wolczko

------------------------------------------------------------------------

Dept. of Computer Science ARPA: miw%uk.ac.man.cs.ux@cs.ucl.ac.uk

The University USENET: mcvax!ukc!man.cs.ux!miw

Manchester M13 9PL JANET: miw@uk.ac.man.cs.ux

U.K. 061-273 7121 x 5699

------------------------------------------------------------------------

Re: Scary Backup Stories

Anonymous's picture

The Subject, not the content, really brings back memories.

Imagine this, your tasked with complete control over the network in a multi-million dollar company. You've had some experience in the real world of network maintaince, but mostly you've learned from breaking things at home.

Time comes to implement (yes this was a startup company), a backup routine. You carefully consider the best way to do it and decide copying data to a holding disk before the tape run would be perfect in the situation, faster restore if the holding disk is still alive.

So off you go configuring all your servers for ssh pass through, and create the rsync scripts. Then before the trial run you think it would be a good idea to create a local backup of all the websites. You logon to the web server, create a temp directory and start testing your newly advance rsync skills. After a couple of goes, you think your ready for the real thing, but you decide to run the test one more time. Everything seems fine so you delete the temp directory. You pause for a second and your month drops open wider than it has ever opened before, and a feeling of terror overcomes you. You want to hide in a hole and hope you didn't see what you saw.

I RECURSIVLY DELETED ALL THE LIVE CORPORTE WEBSITES ON FRIDAY AFTERNOON AT 4PM!

Re: Scary Backup Stories

Anonymous's picture

LOL!!!!

Re: Scary Backup Stories

Anonymous's picture

This is why it's ALWAYS A GOOD IDEA to use Midnight Commander or something similar to delete directories!!

...Root for (5) years and never trashed a filesystem yet (knockwoody)...

.

Re: Scary Backup Stories

Anonymous's picture

rsync with ssh as the transport mechanism works very well with my nightly LAN backups. I've found this page to be very helpful:

http://www.mikerubel.org/computers/rsync_snapshots/

Re: Scary Backup Stories

Anonymous's picture

If you need to use ssh on your lan, you have bigger problems than data backup. You need some physical security.

Re: Scary Backup Stories

Anonymous's picture

SSH should be used to avoid a whole host of problems that can result in sending "clear text" user-ids and passwords around a LAN. This is the case with POP, Telnet, FTP, etc., etc. Even if you trust everyone on the LAN (which is unlikely), SSH is still the way to go.Cheers, Paul.

Re: Scary Backup Stories

Anonymous's picture

So anybody on your LAN is welcom on your servers when you log in from the LAN? Every machine should have ssh on it to begin with so you might as well use it..

Re: Scary Backup Stories

Anonymous's picture

For one customer, we had an automated nightly backup procedure: you just had to put in a tape, and at 3am a backup would be dumped on the tape (QIC cartridges)

After more than a year, a casualty required the use of a backup. Strangely, the latest backup turned out blank and unreadable. Where is the previous tape? Blank stare. They never switched cartridges, and the tape was blank because it was blank from thousands of revolutions. No magnetic particles worth noting remaining on it.

And then there was the customer that found out that when the backup volume went multi-tape, he could just keep the same cartridge in the drive when the prompt came up to insert a tape and press RET.

Re: Scary Backup Stories

Anonymous's picture

I realize that this stuff is old now, but in searching for info I stumbled onto this site, and just had to read what was written. Your reply post is extremely funny! The really scary part about it is.... I know people like that.

Re: Scary Backup Stories

Anonymous's picture

Some years ago, I've seen a couple of people that did something like that. Back then it was common to perform backup using tools like pkzip and arj (Yeah, I know... Not Linux, but it applies to the subject anyway).

I've carefully explained them on how to do a multi-volume archival of their files. "When the program says: Insert next disk and press any key, you take off the disk on the floppy drive and put another one in there and then press any key, OK?" and the morons said "Yeah! Piece o' cake!"

But, as most of you already realized, they never used more than two floppy disks to do the whole thing: one in the drive and the other to replace. :-)) When the archival utility said "Please insert...", they took the one in the drive and put the other one in there and they keep doing it until the program finishes. Of course, the program complained about the lack of space but this kind of people never mind in _reading_ error messages that pops right in front of them. They just answered anything just to get rid of those messages...

At least, they couldn't read error messages in English (I'm writing from Brazil) so nobody could really blame them. But how could I possibly imagine that someone would do such stupid thing even back at that time? :-))

Fortunately, they did it using their files and not mine so I could stay cool after all... and give that great laugh. Like the author said, "Never underestimate the human factor".

Cheers,

DeadFish Man

PS: Sorry about my poor english... :-)

Re: Scary Backup Stories

barryp's picture

As I said in the article: never underestimate the human factor. :-( Regards, Paul.

What sane IT Manager would do Story #2?

Anonymous's picture

Never , ever, ever let a "user" perform a critical(payroll) backup!!!!

You should have sent one of your cronies over to do the backup or

login remotely if possible.

That is too damn funny.... Sorry buy you deserve a spankin for that one :)

Re: What sane IT Manager would do Story #2?

Anonymous's picture

That is the ideal, yes, but I'm sorry to inform you that you are not worthy of accessing all data, especially payroll data. I'm sure there were bean counters and managers VERY concerned about limiting access to that information. Hence the locked cabinet ana tape sign-out procedures.

Now, not having the result logs mailed to the admin staff, or just not checking to see that the 10G of data was actually backed up not just 50k. I'm working on adding a new-file/deleted-file listing to my weekly backup script. That way I can keep a better eye on what's happening on the shares and spot mass deletions or unintentional moves. Hmm, should probably keep an eye on the modifieds too, at least the count to spot viral file-fondling.

- RustyTaco

Re: What sane IT Manager would do Story #2?

Anonymous's picture

This is where cryptographic hashes come to your rescue. I don't know to tie this into your system (I essentially wrote my own tools to do this), but what you want is a listing of every file and it's MD5 or SHA1 checksum to be sent to you outside of the normal archive. If necessary, the name of the file can be obscured, what you need is the checksum. It would also be nice to have the file size (for capacity planning), but it's not critical.

These checksums then go into a database - basically you want a tape id x date entry (when the backup was performed), and a tape id x hash entry (which tape the file was written to). This allows you to find all instances of any particular file (e.g., fairly static user data), detect changes that shouldn't be occuring (e.g., in executable files, if you know the filename), determine that the raw count or capacity of files is in the range you expected, etc. All without revealing much, if anything, about what's being backed up. (In the most hardcore environments, you identify files by their inode number!)

Re: What sane IT Manager would do Story #2?

Anonymous's picture

Or at least put in as part of the script commands to rewind the tape, then dump a catalog!!! You can then (in the same script) examine the catalog. For example, if the typical size of the catalog is 5KB, and suddenly it's 2KB, send an email...

Re: What sane IT Manager would do Story #2?

Anonymous's picture

The user in question was working in payroll, and so should be perfectly able to take responsibility for their own data and backups.

IT implements the solution, but the user should carry it out, unless one's able to centralize backups.

Or do you wish we were still having users come to IT with their data on paper or maybe even punchcards and pick up the results the next day. ;-)

Andreas

Re: What sane IT Manager would do Story #2?

barryp's picture

Yes, I agree. And, I'm a firm believer in "whomever owns the data, backs up the data". Thanks, Paul.

Re: What sane IT Manager would do Story #2?

barryp's picture

One with too little staff, and too many systems to look after. Cheers, Paul.

Re: Scary Backup Stories

Anonymous's picture

I work at this small Community College in upstate NY. No money is spent on linux, yet we run three linux servers. lately we have been moving faculty web accounts to our web server, and we have got a surge of requests for streaming video, and that run on one of the linux servers. The last is the mail relay for a lot of our web apps that need a server to relay mail (he he he, we use exchange and no one can figure out how to make it relay mail). So anyways, with no money for backus, we did a simple tar of /home since we can restore the rest of the base os with norton ghost. Then each server ftp's its /home tar file to the other two. Works great. E-mail me at seth@bcvoice.com if you want my scripts.

Re: Scary Backup Stories

Anonymous's picture

And there I was thinking that bc meant british columbia...

"Formatting" tapes?

Anonymous's picture

What sort of tapes need to be formatted? Certainly DDS and DLT tapes don't need to be.

And to the poeple who don't use tapes for backup: this is not possible when you need to backup >100 gigabytes on a regular basis.

Re:

Anonymous's picture

Tapes do not have to be formatted to work, they are simple devices. But just like your hard disk, if you want special features, you have to format. For example an index that points to individual files or backups can be placed at the front of a tape, but the tape has to have space set aside for the index.

Re: formatting tapes??

barryp's picture

The first story happened about 10 years ago, the second about five years ago. Nowadays, tapes are preformatted. This was not always the case. And I'd agree, tapes are still useful. Paul.

Re:

Anonymous's picture

The first story happened about 10 years ago. The second one about 5 years ago. Nowadays, tapes don't need to be formatted. Back then, things were different. Thanks. (Paul).

Re: Scary Backup Stories

Anonymous's picture

I am looking for a way to back-up 2 home computers, and I would still rather use tape back-up, rather than burning a CD-ROM or copying saved data to other machines. However, it seems like there are very few tape drives on the market that are not targeted to large enterprises, rather than home users. Could anyone point me to tape drives that are afforable and appropriate for the home user?

Re: Scary Backup Stories

Anonymous's picture

amanda-tapeio :-) go read up on it on amanda mailing lists and/or newsgroups. Basically, buy a big 160GB hard disk (or whatever you can afford) and back up your data to this disk (all automated, of course) and keep weekly/monthly/yearly rotation on it.

No

Anonymous's picture

There aren't any.

In the early/mid 1990s, for under a thousand dollars you could buy a tape drive that could back up a thousand dollar hard disk. In the 2000s, a thousand dollars will not get you a tape drive big enough to back up even a hundred dollar disk.

The only way tape still works, is if you are only using a tiny fraction of your available storage. Buy an 80 Gig drive and only use 9 Gig of it, and DDS3 will work fine. Use the rest of the drive for /tmp or swap or porn or something else that you you can afford to lose.

Re: Scary Backup Stories

Anonymous's picture

1. do a CD for the basic operating system and ALL the basic progs you use for each system. The new USB 2.0 externals are a good portable choice.

2. most any travan and software will handle network mapped drives of the new/changed files (mostly data, I hope) for both pc's

3. I do this in a university environment for my department's specific shares. When the Super, Special, Expensive IT system backups fail repeatedly (Scarry stories 5,6,7, & 8) my seagate travan 20Gb restores production data that otherwise would be lost.

Course I ain't no IT pro.....

Re: Scary Backup Stories

Anonymous's picture

I have a 4mm DAT I bought in '93 for something like $700. it's SCSI and I had an Adaptec 1542 to run my Syquest drive. I also had macintoshes and suns at work.

It stores 2GB which ain't enough by today's standards. But, I can still read old tapes. And still use it on all my machines because I have SCSI in them (including my laptop). I have macintoshes, PCs, and Sun systems now.

You can get a SCSI card real cheap nowadays. Sadly, 4mm DAT still cost around $700 but they have lots more capacity.

I'll also occasionally burn CDs (SCSI burner too). CDs will last longer and the format is more portable. The capacity is too low though.

Re: Scary Backup Stories

Anonymous's picture

Why don't you just buy an extra hard disk and have a copy of your important data there. With today's prices it doesn't cost anything.

Re: Scary Backup Stories

Anonymous's picture

A lot of people seams to have this idea, and in many situations it should work fine.

However, there is the human factor. Sometimes simple things go wrong (as simple as copying a file), and it takes a while before anybody notices that the contents of this file is not what is expected. This means you have to have many "generations" of backup of the file in order to be able to restore it, and in order to not put all the "eggs in the same basket" each of the file backups should be on a physical device.

Also, backing up to another disk in the same computer will probably not save you when lighting strikes, as the backup disk is just as likely to be fried as the main disk.

In real life, the backup strategy and hardware/software choices to support it is (as most other things) a balancing act. The important thing is that you have a strategy, and that you test it regularly to make sure it works as intended (as the main point is in the article). Also, realizing that achieving 100% backup security is impossible might save a lot of time in setting up the strategy.

(I.e. you have to say that this strategy has certain specified limits, like not being able to restore a file to its intermediate state sometime during a workday, only to the state it had when it was last backed up, which should be a maximum of xxx hours ago and so on...)

--

Hallvard P

Re: Scary Backup Stories

Anonymous's picture

Well, the travan-type drives have traditionally been more small-office oriented than DAT/DDS drives. These are the thicker tapes that have the metal bottom. They are formatted tapes, which gives the benefit of putting multiple backups on a single tape, much like a disk drive.

DAT tapes generally are not formatted; the tape rewinds and you basically stream the data onto the tape as a raw image. While you generally can't put multiple backups on a DAT tape, the benefit is that the backups are generally faster, and Travan drives tend to whir and wind quite a bit when you insert the tape before you can actually use the tape (they have to read in the format information first, presumably).

DAT drives tend to me more expensive and run off of a SCSI interface, but DAT tapes are usually a lot cheaper than Travans, since they are simpler and don't have the metal base.

A coworker of mine has a recent Travan drive and gets good performance out of it. The newer ones run off the IDE interface so they are much faster than the drives of old that ran on the floppy interface. I believe he pays around $25-$30 per tape, but they last a while and you can get by with fewer tapes by putting multiple backups on a single tape. DAT tapes should be less than $10. Buy.com, pricewatch.com, newegg, etc, should have both styles of drives and media.

Re: Scary Backup Stories

Anonymous's picture

Travan drives will die after a year of regular usage, DDS will hold on for a couple more.

As well, DDS-4 is far faster than the 20gb travan drives

And the most important factor

You can eject a DDS tape after a backup. If someone breaks into your backup machine and you're using Travan, better pray someone pulled the tape out or your cracker will just wipe your backup tape.

Re: Scary Backup Stories

Anonymous's picture

yeah forget tape backups.. i backup my data to multiple servers at night in different locations. I have a Seagate sidewinder 200 and its useless it fails way too often.

I first ccrypt all my files, tar them, gzip them, and sftp them to multiple servers. if transfer or an error is encountered im emailed with the errors.

im dependant on sftp ; raid 1 and 5 systems in multiple locations for my backup.. the only way i would loose them is if a huge meteor fell on the US..

Re: Scary Backup Stories

Anonymous's picture

That Seagate Sidewinder is really a Sony product which was rebranded and sold by Seagate for a time. Seagate never manufactured any AIT drives.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix