Easy Backup and Restore
Until recently the extent of my backup efforts was to take the occasional CD copy of my home directory and keep copies of important files somewhere else, usually on another disk partition or a floppy disk. All of this changed with the need to run some Windows legacy applications. The only machine really suitable for this work was my main workstation, a 1.2GHz Athlon machine, multibooted with four distributions. I decided to free up the first primary partition, which held Mandrake 9.0, and set up a Windows partition.
I freed up the first primary partition by transferring its contents to the seventh partition, overwriting an expendable Vector Linux 3.0 Distribution. To be totally safe, I booted into Debian 3.0 and mounted both partitions to individual mount points in /mnt. Then, as root, I used tar and a pipe to copy everything, including all links and permissions, from the source partition to the target partition. A few minutes later, after changing my GRUB boot menu, I was able to boot into Mandrake 9.0 Linux in the seventh partition and verify that everything worked as expected.
At this point, one normally would DOS-format the now free first partition and install Windows. I began to feel a little uneasy, however. Windows could format the whole darn drive or some other similar screwup could happen, in which case I would be in the position of having to fdisk the partitions and reinstall everything from scratch. The original disks, of course, would have all the applications except for those extra packages installed by me, but all custom configurations would be lost.
The machine now was running Mandrake 9.0, Debian 3.0 and Slackware 8.1. Of these, only losing my Slackware install would cause me grief, as it has been running like a top, boots to KDE 3.0 in less than 30 seconds--including my sign on--and is absolutely rock-solid stable. It also has the CUPS printing system set up perfectly for all of my printers on the LAN. So I must retain this setup at all costs. The solution, of course, is to back up fully everything from the Slackware install.
At this point, the desire to have a simple, easy and foolproof backup and recovery method took hold.
If you are a home or SOHO Linux user, I suggest that your backup and recovery system should:
Require no equipment or software other than what you already have
Be cost effective in backup media
Be easy to use regularly, or it will not be used at all
Be easy to verify, or it may be useless when the time comes
Require only the media and a working machine, in the hardware sense
Require only minimal knowledge of the recovery process when the crunch comes
A quick review of past Linux Gazette articles and a search of the Web turn up hundreds of backup solutions. Many are aimed specifically at the backup function; others are aimed at the repair and system recovery part of the overall effort to get back to some predefined state. Virtually none are customized for your system or your specific requirements, so why not roll your own solution? That is what we do here.
Most home or SOHO users do not have a tape drive system and are unlikely to purchase one for the sole purpose of backup, given that the cost of the tape system and software probably exceeds that of the computer itself. This essentially narrows our options to backup to removable disk, backup to the same or another hard drive, backup to a CD or backup over a network to some other hard drive. This last option essentially is a more complicated backup to local hard drive option, except there is zero chance of it being lost when your system goes down. So let us look at these options:
Floppy: Good for incremental backups on a daily basis and perhaps the best solution for saving work as it progresses, but useless for system-wide restoration. The LS120 Disk and the Zip disk are not large enough or common enough to be considered for the sort of simple but complete backup considered here.
Hard Drive: One can back up to a separate partition on the same drive, which is of little use if that drive fails, or one can back up to another hard drive in the same computer. This is good, except there is a fair chance that a power supply failure or nearby lightning strike could fry both drives--or somebody could steal the computer--leaving nothing to restore.
Network Filesystem Transfer: This is a good solution to back up and restore files for someone interested enough to correctly install it. however, it does nothing for the process of getting the system up again to the point where one can restore the files. In short, it's too complicated for most people to institute.
CD-ROM: This is where things begin to look interesting. These days most Linux users have a CD burner, and the availability of cheap CD-RW disks means the cost of maintaining something akin to the traditional rotating backup system is definitely manageable. This is the one for us.
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Sponsored by AMD
If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.
Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.
Sponsored by ActiveState
| Speed Up Your Web Site with Varnish | Jun 19, 2013 |
| Non-Linux FOSS: libnotify, OS X Style | Jun 18, 2013 |
| Containers—Not Virtual Machines—Are the Future Cloud | Jun 17, 2013 |
| Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer | Jun 12, 2013 |
| Weechat, Irssi's Little Brother | Jun 11, 2013 |
| One Tail Just Isn't Enough | Jun 07, 2013 |
- Speed Up Your Web Site with Varnish
- Containers—Not Virtual Machines—Are the Future Cloud
- Linux Systems Administrator
- Non-Linux FOSS: libnotify, OS X Style
- Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer
- Senior Perl Developer
- Technical Support Rep
- UX Designer
- Web & UI Developer (JavaScript & j Query)
- RSS Feeds
- Reply to comment | Linux Journal
2 hours 51 min ago - Yeah, user namespaces are
4 hours 8 min ago - Cari Uang
7 hours 39 min ago - user namespaces
10 hours 32 min ago - yea
10 hours 58 min ago - One advantage with VMs
13 hours 27 min ago - about info
14 hours 27 sec ago - info
14 hours 1 min ago - info
14 hours 2 min ago - info
14 hours 4 min ago
Featured Jobs
| Linux Systems Administrator | Houston and Austin, Texas | Host Gator |
| Senior Perl Developer | Austin, Texas | Host Gator |
| Technical Support Rep | Houston and Austin, Texas | Host Gator |
| UX Designer | Austin, Texas | Host Gator |
| Web & UI Developer (JavaScript & j Query) | Austin, Texas | Host Gator |
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?



Comments
backing up files in linux or ubuntu
I am a new user in linux, have to take it for class. How do you back up files?
After restore linux won't boot
I am trying to recover my SLES ENT Server 9 from a tar backup. I have successfully restored the MBR and created the paritions and restored the full system using tar. When I try to boot the system after a restart I get the following error:
graphics file (hd0,1) /Message" Missing, press any key to continue"
I hit a key to continue and get the grub boot menu. I try to boot to linux but I get the following error:
Error 15: File not Found
Any ideas I can't seem to find any information on how to go further.
Thanks
Hard disk
I think the new hard disks are the best way to recover your data..
Backup
I think as alternative for backup it is possible also to use Flash Memory Cards.
10 Years average guaranteed term of a storage of the data.
Use Flash Memory Cards
Nice work on top of that it’s really helpful..
re
Shouldn’t a mistress city give some sort of seclusion and msytery to its visitor? If so how ‘bout Park City, UT? High up in the mountains but still a very engaging and inspiring city for creatives. Dear old Philly would never think to look there…
Thanks for this!
Thanks for this!
more flexible way to check size of cd/dvd in the script
Great script my-man!
One minor thing I might add though, the line:
Size=`du -m $Image_File | cut -c 1-3`
is better served with the -f flag:
Size=`du -m $Image_File | cut -f1`
This way if your dvd is let's say 4700 mb's it still works. -f1 picks up the first field instead of the -c which parses characters.
shwoot!
-nicholas biondi
My way
I keep a second identical hard drive on a removable drawer. Once a week I install it in the PC and run G4U disk mirroring software from a bootable CD. For a local disk copy I do the command "copydisk wd0 wdx" (x is the ordinal number of the destination disk) and it takes about 15 minutes. Free, quick, simple, OS-independant, and backs up all partitions at once.
Advice for young players: G4U can be dangerous if you have multiple working hard drives. You must be VERY careful to get the drive names (wd0, wd1, wd2) correct : otherwise you may find yourself backing up over a working drive, and the moment you realise its happening, it's too late.
If I need to restore from a drive failure, I would have to physically swap the backup drive for the failed drive, changing jumpers to make it the master drive along the way. If you need to restore individual files, just plug in the backup drive, and copy your backup files over to the working drive.
G4U also provides for backing up to another drive across a network, which avoids moving the backup drive around. Personally I like using the removable disk drawer as it provides protection from theft and other calamities. USB connected external hard disk housings would also work well and be a tad easier.
DVD instead of CD?
Please help me...
What if I want to use DVD instead of CD?
How do I change the script? Does cdrecord write to DVD as well as CD?
If not, what do I use?
Thanks and take care,
Allen
Use rdiff-backup and pyBackpack
Here is my setup:
This setup works quite well. I can save data from my PCs into the network mounted drive on the linux server and all the data is transparently backed up. It takes incrementals and is able to restore history.
Free professional backup solution from Arkeia
Arkeia offers a no-cost license (up to 50GB) of its Linux based backup software, allowing both backup to disk and to tapes. This solution also brings network backup of Windows, xBSDs, Mac...
Download available from arkeia.org
ZIP as an alternative
I have been using "find" and "zip" in combination for backups for years. And my main motivation for using zip instead of tar/gzip is your observation that "even a single-bit error conceivably could corrupt the entire compressed archive". Tar/gzip archives are great for distributing source packages, where you can simply get a new copy if yours goes bad, but they're not as good for backups.
Bit errors in a zip archive typically will make only the one file unrecoverable. And note that bit errors may develop over time -- just because your archive passes the md5sum check today, it might not in a year. CDs have an expected shelf life of 100 years, but remember, that hard drive that failed had a MTBF of 100,000 hours.
True
Yea I have yet to recover a bad file yet!
mbr equivalent for lvm?
What would be the equivalent to the 'dd' statement for storing the layout of an lvm setup? Is it stored i a regular file, or is there a chunk of disk that needs copying?
(I don't have access to a linux box at the moment to investigate the answer for myself.)
Rsnapshot
How could you have not mentioned rsnapshot? ;-)
www.rsnapshot.org
Good tips on rolling your own
Thanks for a fine article on how to roll your own solution.
I have been using Hugo Rabson's MONDO from www.mondorescue.org to create backup disks for years. I think Hugo's code started with what you are doing here but has expanded. It still retains the kind of flexability and ease of use for personal backups. It has the capability to create CD-R, CD-RW, DVD-RW, ISO images, backups to NFS volumes and even to tape.
A CD-RW is far too small.
Im sure it's nice, and all, but what about people with 160GB drives?
My collection of digital photos I've taken takes about 10GB alone.
I just have to trust in XFS and Western Digital :(
Well, if you value your
Well, if you value your data, you should be buying a backup solution as part of your computer investment. So, if you can spend extra on a 300G hard drive, you should probably instead buy 2 160G hard drives, for instance. The price vs. size vs. quantity thing isn't quite that simple, but you get the idea. Backup solutions are not an "extra" thing; they're part of your basic costs. If you can't afford a backup system, then you can't afford the computer at all.
But, at the very least, partition the drive, and backups to the other partition with rsync or something like dirvish, as mentioned elsewhere.
re... too small
I have the same problem. I have about 2 GB of saved docs and email, 3 GB of digital pics and about 10 GB of music that I don't want to get lost. I have the cd's for most of the music (yes I do own it) but to get it back onto the hard disk I would have to rip for a week.
My solution:
1) Grab an old pc out of the garage (PPro 200MZ in this case)
2) Install 3 of the biggest disk drives I had laying around.
2) Install Linux (RedHat 7.3 with all the Fedora Lagacy patches)
3) Plug it into the network. ( We now have NAS)
4) Set up rsync to backup nightly. (rsync only sends files that have changed)
5) Set up Samba to allow the wife to save her "My Documents"
Scripts:
$ cat sunday_sync
#!/bin/bash
sun_sync_doc_with_ast
sun_sync_pic_with_ast
sync_dudley_with_ast
sync_mp3_with_ast
$ cat sun_sync_doc_with_ast
#!/bin/bash
cd /home/me/
rsync -avz ./Documents me@192.168.1.50:./Doc1
$ cat sync_mp3_with_ast
#!/bin/bash
cd /home/me/Music
rsync -avz ./mp3 me@192.168.1.50:./Music
(you get the idea... I rsync nightly to this machine. I also have more then one directory on the target machine for doc and pic ... ie. ./Doc1 ./Doc2 ./Doc3 .... and rotate through the week. )
Notes:
You will notice that I have several copies of ./Doc on the backup server but only one copy of ./Music . Music doesn't change that often and if I did lose it it would be more of a hassle then a disaster. (You get extra points for making sure that ./Doc1 is on a different hard disk then ./Doc2 is on a different hard disk then ./Doc3)
In order to make the rsync happen with out my having to imput passwords I created an ssh key with no password. This is a little bit of a security hole but since this is sitting in my home office and you would have to hack my main pc to access the key... oh well. Even if you did this key is only used for the backup machine.
I choose RedHat 7.3 for the following reasons.
1) I had the disks sitting in my drawer.
2) With the Fedora Lagacy project putting out security updates it is as secure as any other version of RedHat.
3) Version 7.3 runs great on this old Pentium Pro 200. 7.3 was the high point for RedHat. It was fast, complete, stable, and ran well. Every version since has gotten more and more bloated and slower and slower on anything except "state of the art" hardware. In my opinion they are close to if not already living in Microsoft Bloatsville. RedHat needs to force their developers use 500MZ Pentium II machines for their desktops.
if you want rsync web
if you want rsync web interface or gui tools check this very useful
rsync web interface or gui tools
Have a look at dirvish.you
Have a look at dirvish.
you can use it on your local computer and via the network. it also uses rsync for transferring the data, but you can keep multiple backups of your data wihout wasting space for common files. this is achieved by using hardlinks for common files. it also provides indexing and search functionality. I use it for several months on our servers and had no problems...
Yes. Something like Dirvish
Yes. Something like Dirvish is ideal, I think. Optical media is far too unreliable for backups, imho. At best, they're useful for shipping data or transferring to a non-networked machine. I was using faubackup for a while, to transfer from one machine's HD to another's, but dirvish is similar, only faster; just got finished setting it up myself. It's a little more complex than faubackup (which would still be useful for very simple backups or backups to the same machine), but not too difficult to get your head around, and quite flexible and powerful too.
A CD-RW is far too small.
For large amounts of data, an external USB2 or firewire drive is the only practical way to go.
It should be possible to modify this script to use an external USB disk instead. For rotation and "security" use multiple directories or partitions.
I think you switched the bs
I think you switched the bs and count #'s to restore the mbr.
D#ont use a CD-RW
Correct, the only way is to use an external USB2, don't use a CD-RW, it's not suitable for data in large amounts.
External USb2 and firewir drive
Hi Bob, you are correct in your statement, but a firewire drive is also
possible.
Best regards Hilde from Germany
I read some articles and
I read some articles and learned a lot from your journal. Thank you and success for the future.
Best wishes
Dobby
Interesting articles
By reviewing your articles I can found out the same as Dobby(see his comment on September 9). I like your journal, especially the content but also the design.
Success in the future
Carsten
Thanks
A helpful script for external USB!
By modifying the script use
By modifying the script use an external USB disk instead.
best regards
Heinz
my comment before
sorry, my homepage behind my name heinz in not running. therefore
i try it again.
best regards
heinz
Easy backup
Easy backup and restore is a neverending story. I like the comments, helpful for my work.