LVM and Removable IDE Drives Backup System

Backing up data is a problem we all face, and as the size and complexity of the systems for which we are responsible grow, so do our backup problems.

When the company I work for, a civil engineering and surveying firm, decided to move all its AutoCad drawings onto a central fileserver, we were presented with a backup situation orders of magnitude larger than anything we had confronted before. We had at that time (now considerably larger) about 120,000 files, totaling 200GB, that were in active change and needed to be backed up at least daily.

My first thoughts were of some sort of tape backup system, but as I began to research them, I was shocked at the prices I encountered. A tape autoloader large enough to contain our filesystem ran about $12,000 and a 40Gig tape was $89. When I first convinced my boss to let me run Linux on our servers, cheap was a big selling point. So, what are the alternatives?

I had been using a removable 120GB IDE drive to transfer data back and forth from home; it had cost me $101 for the drive and $17 for the removable bay. I also had a retired 1GHz P4 box that had cost about $800--the math was starting to look interesting. If I could find a way to stick 120GB drives together, I would be home free. That's when I discovered Linux LVM (Logical Volume Manager).

With LVM you can, if you like, treat the removable drives as large, cheap tapes and use any of the traditional rotation schemes of incremental and full backups, such as the Tower of Hanoi. We took a somewhat different course for our needs. It has always seemed awkward to me that in order to recover a file from tape you had to search for it sequentially, restore the full backup, apply the incremental backups in order and, finally, arrive at the version you wanted. I wanted a backup system that would let me open and examine drawings to determine if they were the appropriate version before restoration, and with LVM that's what I got. It takes a few more drives to do it this way, but the 12,000 bucks I saved on the autoloader buys a whole lot of drives.

I probably should take this opportunity to warn everybody that I'm still new enough to Linux that I do things that doubtless make more experienced people shudder in horror. All I can say in my own defense is I'm still learning, and many of these things have worked so well in the real world I have found no need to change them.

We started by setting up the P4 box with one IDE drive for the operating system, a CD-ROM drive (to load the operating system) and two removable drive bays, each containing a 120GB IDE drive. It has since grown to four drives on a RAID controller as JBOD, but let's keep this example simple. We loaded Red Hat 8.0 on the backup server--the first Red Hat version to include LVM--and gave it an IP address on the network. Configure your box in a way appropriate to your own network.

Because we were going to be moving these drives in and out of the box and off site, we needed a way to keep track of which drive was which. I used stick-on letters and labelled the part of the first drive bay that remained in the case A and the part that came out A1. The part of the second drive bay that remained in the case was B and the part that came out was B1. These were our backups for one day. Subsequent days were called A2B2, A3B3 and so on.

Having done this, we were ready to configure our drives, in this case /dev/hdb and /dev/hdd, for LVM. Start by logging in as root and type fdisk /dev/hdb. fdisk returns a helpful warning that because we are using 120GB drives, the number of cylinders is set higher than a certain number, but we can probably safely ignore this. Trust me, we can.

Assuming we are starting with a new unformatted disk, we wish to create a new partition. Type n and then press Enter. fdisk now asks if we wish to create an extended or primary partition. We want a primary, so type p. fdisk now wants to know the number of the partition (1-4), so type 1.

Next, fdisk wants to know the beginning cylinder of our partition. The default is 1; accept it by pressing Enter. fdisk now wants to know the ending cylinder of our partition. The default is the last available cylinder on the drive; again, accept this by pressing Enter.

Let's take a look at what we've created so far. To see the partition table, type p and press Enter. We should see something like the following:



DEVICE   BOOT  START  END    BLOCKS    ID    SYSTEM

/dev/hdb1       1    15017   (large    83     Linux
                             number)


Up to this point we haven't actually modified the partition table. These changes are being held in memory by fdisk until we tell it to write them. At any time, we can exit fdisk without making permanent changes by typing q.

So far everything looks good except for the partition type, which seems to be 83 Linux. We need to change this to Linux LVM, which we do by typing t and then 8e. Now, when we read the partition table by typing p, we should see:



DEVICE   BOOT  START  END    BLOCKS    ID    SYSTEM

/dev/hdb1       1    15017   (large    8e    Linux LVM
                             number)


This is what we want, so we now officially change the partition table by typing w. At this point, fdisk becomes quite excited and tells us The partition table has been changed and returns some ongoing information about what it's doing before unceremoniously dumping us back at the command prompt.

We're done with the hairy part, except that we need to repeat the process for any other drives we are going to include in our logical volume set.

At this point we are ready to begin creating the logical volume. Much useful information can be found in the lvm man pages. Typing man lvm puts you at the parent page, which provides directions to the subheading man pages. Now that our disks are formatted as Linux LVM, we need to create them as physical volumes for inclusion in a volume group. We do this by typing pvcreate /dev/hdb1 /dev/hdd1.

We now have to join the physical volumes we created into a volume group from which one or more logical volumes can be created. A peculiarity of LVM is we must at this time set a physical extents size for our volume group. That is, we have to define a certain sized unit in terms of megabytes, called a physical extent. Any logical volume we create from this volume group can contain no more than 64k of these physical extents. If we suspect our logical volume eventually may contain a terabyte of data, we need to set our physical extents to 16MB. We do this while creating our volume group by typing:



vgcreate -s 16M a1b1 /dev/hdb1 /dev/hdd1     

We now have a volume group called a1b1 composed of the two physical volumes, /dev/hdb1 and /dev/hdd1. Let's create a logical volume that uses all the available room on this volume group by typing lvcreate -L 226g a1b1.

The -L option specifies the size (226GB). For some reason, I've never been able to get a drive advertised as 120GB to yield more than 113GB at this point. In the lvcreate line above we actually accepted the default logical volume name (lvol1), so our logical volume is now /dev/a1b1/lvol1. Notice that the last character is the numeral one and not a lower case l. Now we need to create a filesystem on our logical volume. I prefer to use the ext3 filesystem for our drawing files because of their large size. We can create this by typing mke2fs -j /dev/a1b1/lvol1.

It takes a little while for the filesystem to be created, but when it's done the only thing left to do is to create a mountpoint. We do this by typing mkdir /mnt/back. We mount our logical volume by typing:



mount /dev/a1b1/lvol1 /mnt/back  


We can test it by typing df -h /mnt/back; the operating system should report that we have about 226GB available on /dev/a1b1/lvol1.

If we need to add drives to this volume group in the future, we can do so on the fly with vgextend and e2fsadm. We now need to repeat the process for each volume group that we wish to create. I use one for each day of the week, and once a month I pull one out of rotation and save it for a year. This doesn't have to be done all at once; I did one a day until all were complete.

Our network is composed of mostly Windows workstations connecting to Linux servers through Samba. So, we set up a Samba configuration to make our backup server available in the same way. First, copy the default Samba configuration file to a safe place. On Red Hat systems this is /etc/samba/smb.conf. Now, we compose a new simpler smb.conf. Using your favorite editor, type:



encrypt passwords=yes

netbios name=yourbackupserver

workgroup=yourworkgroupname

[back]

read only=no

path=/mnt/back


Save this as /etc/samba/smb.conf. Now we'll restart the Samba server by typing /etc/rc.d/init.d/smb restart. On the first night, we copy the entire filesystem to a1b1 with cp. You may have to unalias cp on your system to make sure the -i flag is not set. This cp process may take 10-12 hours for 200GB. On the next night, we copy the entire filesystem to a2b2, and so on, until we have a complete copy for each day of the week. On subsequent nights, we use cp -au and only copy files that have changed since the previous backup. This only takes an hour or two and has the virtue that any files accidentally deleted from the main server are preserved on the backup.

When changing drives each day, it is somewhat tedious to activate the volume group and mount the logical volume, so we wrote a little Perl script to do it for us. Using our favorite editor, we put this together: like this:



#!/usr/bin/perl


$hold = `vgscan | grep a[0-9]b[0-9]`;

$hold2 = substr($hold, 39, 4);

`vgchange -a y $hold2`;

`mount /dev/$hold2/lvol1 /mnt/back`;

print "$hold2 mounted\n";


and saved it as /home/me/mountback.pl. We make this executable by typing chmod 700 /home/me/mountback.pl.

On the first line, our script creates a variable called $hold, runs the operating system command vgscan to search for volume groups, pipes the output of this command to grep, which searches for a line containing an a, followed by a digit from 0-9, followed by a b, followed by a digit from 0-9, and places it in $hold. On the next line we create a variable called $hold2, run the substring command on the line of text placed in $hold, grab the four characters after the first 39 and place them in $hold2.This should be the name of our volume group, a1b1.

After running vgscan manually a few times it becomes obvious where the name will appear in the line of text. On the third line we run vgchange with the -a option (activate) and give it the y option for yes and the name of the current volume group, which should be in $hold2. On the fourth line we mount the logical volume at our previously created mount point. On the last line we print to the screen the name of the volume we have mounted. If we have done something wrong, we get some strange messages here. I have always intended to go back and add some error checking to this script, but somehow or other I've never gotten around to it. It's been working fine day after day, year after year.

So, to sum up our procedure, to change out our backup drives we type shutdown -h now. When the machine is finished shutting down, we remove the drives labeled a1 and b1 and replace them with those labeled a2 and b2. Then, we start up the machine and log on as root. Type /home/me/mountback.pl, and we should see a2b2 mounted.

We're now ready for the current night's backup to happen. This is best run as a cron job from the main server, which presumably has more and faster processors, RAM and so on. I've been contentedly using a shell script based on cp -au for some time and have recently begun experimenting with rsync. I have every reason to believe tar would work fine as well. A simple script would be something like:



#!/bin/bash

echo $0 >> /var/spool/mail/me

echo "START:" >> /var/spool/mail/me

date >> /var/spool/mail/me

mount -t smbfs -o username=me,password=mepassword

//backupserver/back /mnt/mountpoint

(above 2 lines should be on 1 line in your script)

cp -au /files /mnt/mountpoint

umount /mnt/mountpoint

echo "FINISH:" >> /var/spool/mail/me

date >> /var/spool/mail/me


This mounts the logical volume from our backup server to a mountpoint on the main server, copies all files that have changed since the last backup and sends us an e-mail message giving the name of our backup cron job, the time the backup started and the time it finished. The script then unmounts the logical volume.

I hope this article is helpful to anyone who has found him or herself in a similar backup situation and is considering removable IDE drives as a possible solution. When I first started in this line of work, I would study articles that showed how to do things I desperately needed to learn, follow them carefully and get a good grasp on the process, only to reach the end and have the author say something like, "Now all you have to do is make a few changes to the usual configuration files, recompile the kernel and you're done", which would render the whole thing useless to me. I have tried not to let that happen here, but if anyone tries to implement this solution and runs into a snag, drop me an e-mail at mikef@farnerbarley.com, and I will try to help you over the hump.

Mike Fogarty is the System Administrator for Farner, Barley and Associates, a civil engineering and surveying company in central Florida. He has a Linux System Administration Certificate from the University of Illinois and is a member of SAGE, the System Administrators Guild.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Step up quality of the EMP discussion

Anonymous's picture

The reckless way that folks in this forum seem to disregard the seriousness of the EMP threat is akin to the reckless way that people for decades regarded the hurricane threat for New Orleans. May I remind everyone that our country published the Report of the Commission to Assess the Threat to the Unites Sates from EMP Attack, and that this report is about 200 pages? This matter is not for ostriches, it is for those are courageous enough to face reality.

The fact of the matter is that unless responsible data center managers take prudent steps, your technology is toast. It would only take a single nuke launched from the Gulf of Mexico or the Atlantic, exploded somewhere above the heartland. Short of a nuke, a Skud type missile launched from a commercial freighter would cause major Eastern seaboard destruction. Read the report from the Sage Policy Group which describes this threat. Short of malicious EMP attack, solar storms could destroy a major part of the grid including the large scale transformers. Our country no longer manufactures large scale transformers.

Let's please stop the jokes, or move this discussion to a more intelligent forum.

Larger LVM Backup

Anonymous's picture

Great Article.

I have not yet used LVM although I intend to soon and have given backups a lot of thought. My LVM set up will be fairly large, consisting of several physical drives in the machine - lets say 5, and several sets of removable drives for backup, each of which would be 5 similar drives. I would back up from a snapshot to one of my backup drive sets. Is it necessary to mount all 5 backup drives at the same time to do the backup? What I would like is to have one removeable drive bay, and to be able to plug in each of the 5 backup drives one after the other from one backup set and copy the data there. Even better would be if the data on the backup drives was already useable so that it could be just updated.

If I had a non-LVM system, I could have 5 seperate partitions and do the backup very easily in this way with rsync, copying only the changed data, so doing an incremental backup while keeping a full backup of each of the 5 partitions available. Obviously this doesnt have any of the advantages of LVM, like combining the drives into one logical volume, or being able to add more drives easily.

If Ive made the decision to use LVM does this automatically lead to me having 5 monted drives and 5 removable drive bays, 10 drives in all or is there another way?

MarkW.

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

Nice, but nothing beats a proper backup software solution such as Amanda or Arkeia, and backup to disk instead of tape. Its the media which is costly, not the software.

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

As far as Arkeia goes, it is the software that is costly, not the hardware. Try getting pricing for corporate use - single machine licenses cost ~$600 USD. That can buy a second PC!

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

Good article to get some people thinking about backups. Just like most everything in the opensource (and real) world we put ourselves in, there is more than one "right" way to do any of the things we do. Some have mentioned logical ways to enhance what you have done (eg rsync to increase efficiency). I would put looking into a NBD (network block device) high on your research list. Redundant mirrored pairs are alos a good idea. Others have mentioned packages to others have spent much of their time creating for our collective good. I would like to mention rdiff-backup as one of those great options. It's availible at http://rdiff-backup.stanford.edu/.

I do have to agree with greatly with one of your statements: ..."I do things that doubtless make more experienced people shudder in horror."... Indeed, indeed (lol). First and formost, you have zero error checking or return code status checks for any part of the scripts (let alone the critical sections). Generally, you don't write directly to your own mail file (a log file is understandable)...that's kinda wierd.

Realy, in the end, it seems to work for you and that makes it great. Again, good article.

How to advertise hard disk size.

Anonymous's picture

113MB ==> 120M

First start with a 113Mb harddrive. That will hold:
113*1024*1024 = 118,489,088 bytes.
118,489,088 / 1,000,000 = 118M (maybe close enough for advertising purposes to 120M)

Starting with 120M advertised size:
((120,000,000/1,024)/1024) = 114.44Mb of data.

So maybe try 114M in the future.

Re: How to advertise hard disk size.

Anonymous's picture

Wow, 120MB HDD's in 2004?

112GB * 1024 * 1024 * 1024 = 120,259,084,288 B

Which is typically advertised as 120GB.

The difference between a ok admin and a great one

Anonymous's picture

One word for you my friend "snapshot" , if you do not know what it is look it up.

Re: The difference between a ok admin and a great one

Anonymous's picture

Wheee! Linux Server Hacks (O'Reilly) tells you how to create a snapshot.

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

Once upon a time, Sun had a file system known as TFS, the "Transparent File System". TFS was designed to "overlay" a real file system, and store only changes made to the original files. So, say you have a read-only media. You could mount an empty hard disk over the read-only media using TFS. The files would allow write access(!), but saves would be written back to TFS (since the underlying media was read-only anyway).

In the classic context, the read-only system was NTFS. In a more modern context, the read-only system would be DVD or CD.

I sought far and wide for "TFS for Linux" to no avail for exactly this purpose: archival of low usage files to optical media while still enabling changes. Any info on such a technique would be appreciated.

too late but anyway: plasticFS by Peter miller

Anonymous's picture

do the googleing yourself. Oh wait here it is already.

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

I sought far and wide for "TFS for Linux" to no avail

Obviously not far and wide enough

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

for what period of time are the backups maintained?
what is the total number of backup ide drives required?
do you have an estimate of how many GB the 200GB would bzip2 into - are they ASCII/binary?
how long a time does a backup run require?

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

Once upon a time, Sun had a file system known as TFS, the "Transparent File System". TFS was designed to "overlay" a real file system, and store only changes made to the original files. So, say you have a read-only media. You could mount an empty hard disk over the read-only media using TFS. The files would allow write access(!), but saves would be written back to TFS (since the underlying media was read-only anyway).

In the classic context, the read-only system was NTFS. In a more modern context, the read-only system would be DVD or CD.

I sought far and wide for "TFS for Linux" to no avail for exactly this purpose: archival of low usage files to optical media while still enabling changes. Any info on such a technique would be appreciated.

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

Using LVM to get around the nastyness of tapes is a nice idea, have not seen that before.

Have been using 2 approaches over the last 2 years at a small business; fortunately the data sizes are not so big - we only need to keep about 10gig under control.

The first one is a thing I call TIABS (Transparent Incremental Archiver Backup System or TIABS Is Another Backup Scheme) which is an unholy collection of shell scripts.

The 10gig workspace is a samba share on the same box as TIABS. This space has about 150meg churn a day. The hardware is an old Soyo / Intel BX / Celeron box running Mandrake 8.1; this gives us zero problems (but we have a hardware twin just in case).

TIABS works using incrementals against a master image of the share, which I recreate manually every few months. Essentially overnight TIABS does a cp -au type thing to identify changes, populates a blank folder with these then goes back and makes _many_ links back to previous day's image for the unchanged files.

That means the archive area has N folders, one per day, each apparently with a "complete image" of the samba share workspace being backed-up. All browsable (read-only) across the share; that's nice as it allows (for example) our web designer to go look at her website from 6 months ago and run it live right out of the archive area. Everyone can see the archive using Explorer on their own PC; very little intervention from me (though I do check it's pulse etc regularly) and best of all - no evil tapes :))

The biggest problems came from dealing with MS filenames which allow all sorts of embedded $, multiple spaces etc. I could not find *nix tools which would not trip up somewhere (cp, cpio, tar, rsync - my mind just glazes over thinking about the problems). All the incremental detection finally had to be done by script (ouch). It got ugly.

The second approach is using a rotation of big disks. The workspace plus every partition on each PC gets imaged across the local network twice monthly (samba mounts + tar etc); these are just compressed images of complete partitions; very handy if a complete rebuild (=reversion to a working PC) is needed.

TIABS itself uses ext3 and sits on it's own disk perminantly in the server; the disk is an 60gig unit split into 4 partitions. This holds about 10 months worth of daily images.

I'll just mention that we are in deep coutryside with pathetic overhead powerlines - we get 2 or 3 outages a month. Thank goodness for ext3! Floods? Should I mention the river? and there's also.... :)

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

And your offsite backups??? You would be severely screwed w/out them if the office were to catch fire.

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

ahh, there is a fire safe in another building (200 yards away) where the rotation big-disks are kept.

Another option: external USB2 disks

Anonymous's picture

Another similar backup option which I'm about to try is to use external USB2 (or FireWire) hard disks. They're hot pluggable, not too expensive, and you can get small portable ones if you want to take them off-site. Data throughput may not be quite as high as IDE (especially with a 2.4 kernel - USB2 support appears to be much faster in 2.6), but that may not matter if you're happy to leave backups chugging away overnight. Presumably they could be combined using LVM or RAID in just the same way as described in the article?

Re: Another option: external USB2 disks

Anonymous's picture

I use a usb-ide enclosure for one of those cheap ide bays. In windows, i can simply "eject" the hard drive bay device, replace the hard drive, and then plug the drive back in without rebooting. My guess is that you could do the same thing in linux by re-loading the usb mass storage driver. It wouldn't cost very much to do, also it would save the trouble of rebooting

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

A nice article indeed. Rsync has already been mentioned, so I don't need to do that again.
There is one thing that scares me a little: what happens if you find out a file has been corrupted two weeks ago? You will have no backups before that do you? Would a setup with CVS or a more modern version control system not be a better solution then?

Another thing: RH 8 is quite old. In fact it has already been EOL-ed by redhat. It uses LVM 1, which uses a dedicated kernel module. In 2.6 kernels, this has been replaced by LVM 2 and a smaller more generic solution in the form of the device mapper module.

Re: LVM and Removable IDE Drives Backup System

dx100's picture

We have been using a similar approach for several years. But there are still a few remaining issues unresolved.

1. We give each user a removable IDE to backup their own data. But the removable IDE is not Hot Swappable. The BIOS has to find out that there is a disk exists in the IDE channel, and pass such information to the Linux boot procedure, which, in turn, would enable the Linux kernel to have access to the disk. This means that EVERYTIME a user has to reboot the workstation he/she is using. Over the years, we found such approach is not very user friendly. Sometimes there are other people using that workstation via ssh or other remote login process. Sometime other user left a heavy number crunching process to run on the workstation and it is not ethical to reboot the workstation. Furthermore, users sometime accidentally switch off the removable disk (the removable caddy has a power switch) before a complete shutdown of the machine and hence resulting in data corruption (on disks using ext2 system). As a result, our users do not want to use the removable disk as a means of backup.
2. At the institution level, we are providing a monthly full backup and daily incremental backup to ensure that a user can retrieve his/her data of any version over a period of a month. Four year ago we built a backup server using 8 80GB IDE disks (the largest at that time) in RAID 5 mode (a net capacity about 540 GB). An open source package (afbackup http://sourceforge.net/projects/afbackup) is configured to implement our backup policy to backup data from 2 NFS servers over the LAN. The backup process has been running trouble free for 4 years (great Linux system

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

Hi.

1.5 GB should be possible with a 12-port 3ware SATA-controller (3Ware Escalade 8506-12MI) and some large disks.

Reading the story, my first concern is the JBOD-like structuring of the drives, if 1 (one) drive fails the backup is toast.

I have implementet something similar on a 250Gb (net) scale for a customer. I used a h/w (p-)ata raid controller and running the drives in RAID5 (ok, this does cost approx 50% more gross-space) and a very fault-tolerant file system (reiserfs) as e2fs in my opionion is too fragile for backup purposes.

I second the rsync comment above, I implemented it with hourly versioning so that there is always data for each hour the last day, then each day the last week and each week for the last two months and each month one year back.

This huge number of backup takes up approx. 130% of the data, but that is highly dependant on your data's update frequency.

After all for most companies their data is their business today, so the good old "better safe than sorry" applies more than ever.

The system has been running automated without human intervention for more than one year now and the client is happy.

Svenne (svenne(at)kracon.dk)

Why not have a look at Bacula?

Anonymous's picture

I recently implemented Bacula for our backup needs. We don't have such a high volume to backup (only 15 GB), but I'm sure Bacula can manage volumes of 200 GB.
The added advantage is that a catalog is maintained in an SQL database, so finding a version backed up on a particular date is very easy.It can back up to files or to tape. We chose to backup to removable USB2.0 storage (to be carried off site) and to DVD+R (smaller set of files (compressed) for historical backup).
Bacula can backup files from Linux, BSD, Irix and Windows clients over the network.
In your case I would create a RAID5 array with 7 or 8 200 GB disks for the daily backups and for holding the other ones before writing them to external media. Daily backups don't need more maintenance than checking the logs to see whether everything went fine. On a weekly basis the backup made on Friday can be copied to removable storage. (If you want, incrementals or differentials against these weekly backups can also be made parallel to the daily full backups).
That leaves the historical backup for which you could use removable storage or fixed IDE disks, as you do now.
Bacula takes care of maintaining the database.
One thing that is not possible, is to access your files directly without restoring them first. This means you can't accidentally modify them either though, while you are busy, trying to find the correct file.

Of course, if you are happy with your current solution, don't change a winning horse. I'm just trying to describe an alternative. One with which I'm very pleased and that I think the world should know about.

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

Why you are using 'cp' ? why not 'rsync' ? With rsync you will save time.

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

A very useful tool is rdiff-backup which will do much the same
as rsync only with increments (ie. it will allow you to maintain
a mirror of the last backup and then increments for the last 7 days
or whatever).

A nice touch is to NFS mount it read only on workstations so that
users can restore their own backups when they accidentally
delete a file etc.

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

Great article. This concept could be combined with the rsync backup method discussed in the latest Linux Focus: http://www.linuxfocus.org/English/March2004/article326.shtml

Re: LVM and Removable IDE Drives Backup System

Stomil's picture

I have a similar solution here, but we took another approach: 5 120GB drives in a RAID5 array (software Linux kernel driver) on a dedicated machine used as a 'data dump'. We also have a tape library (a big one) but due to the tapes' failure rate the thing is next to useless - sometimes takes days of retries to read a tape written a year of two ago (we store meteorological simulations data there). And the tape library absolutely sucks at file management - files aren't erased when deleted from the library virtual filesystem, library admin needs to trigger a 'compaction' process to actually erace the data. So think what happens at any approach to manage differential backups.
OTOH the RAID array, cheap and not very fast is a good medium to do differential backups and store data that may beome useless (and should be deleted) after a short period of time.

Re: LVM and Removable IDE Drives Backup System

Anonymous's picture

Hmmm...

Wouldn't it be easier to create a 2 or 3 submirror mirror set with software raid, and just pull one submirror each night?

Drives are (relatively) cheap, and you wouldn't have to go through all the time waiting for the copy to take place.

Also, once a 'new' set of drives was inserted after the old ones were pulled, the system should 'sync them up' all by itself...

Not only is it much faster; but you also have multiple layers of protection against a disk failure while the system is up and in use.

EMP Attacks kill HDs/Tapes

Anonymous's picture

An EMP attack would ruin all that backupped data on a harddrive or tape. So that's why I'd buy DVDs for backing up data. It's optical and should be more resistent to EMP attacks. Which I think will happen someday.

EMP Attack?

Anonymous's picture

I think if we ever experienced an EMP attack it would most likely be followed by some other sort of attack. Like by guys with guns and bombs and s*!t! I don't know about anyone else but, if that happens, the VERY LAST thing on my mind will be the safty on my safety of my backups.

Where do you live? Like Iran or something?

Re: EMP Attacks kill people too

Anonymous's picture

Have you an EMP-safe people-backup-and-restore system? Without it, your point is ... pointless: who would be there to :
a) still suffer from not being able to restore
b) try to do it anyhow, if the media are there

BTW: those "EMP attacks" are often mentioned together with nuclear war. Will your DVD be heat-resistant? Or do you burn DVD's in heat-resistant glass? (asbestos?)

Re: EMP Attacks kill people too

Anonymous's picture

EMP attacks do not kill people. EMP stands for electromagnetic pulse. One does not need to deploy nukes to create an EMP either.

bonkers

Anonymous's picture

EMP weapons have been tested since the '50, and no unclassified weapon has come out of it. If you do not live in a political hot contry, your fear of EMP attack should be equal to fear of asteroid impact or other natural disasters; not very likely.

EMP weapons have been hyped by many on the net, but never has any failed or succesful attack been reported anywhere. Computers by design hare quite good at withstanding an EMP attack: the are enclosed in a faraday cage with all wires coming in fused at some point. Computers usually need to keep 1-3 GHz frequencies inside, so they are pretty resistant. An succesfull EMP attack might fry keyboards and other periferals, and it may even kill the mobo, it will not erase the hd, that is a little faradays cage (the hd casing) sitting in a bigger faraday cage.

You need a device using high explosives or massive capacitor banks to do any damage against computers; the much hyped magnetron dish on a truck will generate a narrow spectrum that will not penetrate computercases at all.

A simple question

Anonymous's picture

Hello,

I'm attempting to setup a small network for my research lab. I'm interested in distributed simulation. Currently I have three computers, including a laptop. The operating systems are Win2000 and WinXP (laptop). I want to add a linux server and want to run win2000 in at least one of the computer. Is this possible? How can I get a quick start?

Thank you very much,

SC

Re: EMP Attacks kill HDs/Tapes

Anonymous's picture

So that's why I'd buy DVDs for backing up data.

Must have failed 4th grade math, eh?

200GB / 4.7GB = 42.55 rounded up to 43. So, they'd need 43 disks!!!!!! I'm not even going to comment on how utterly stupid that idea is. Oh wait, I just did.Guess it's because that's such a stupid idea.

Re: EMP Attacks kill HDs/Tapes

Anonymous's picture

And is necessary to make such comments without any education? Only say that is not a good solution because the larger number of disk is sufficient.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix