Copying a Filesystem between Computers

October 27th, 2008 by Dashamir Hoxha in

Your rating: None Average: 4.8 (25 votes)

If you need to transfer an entire filesystem from one machine to another, for example, when you get a new computer, do the following steps.

1) Boot both PCs with any Linux live CD (for example, Knoppix), and make sure they can access each other via the network.

2) On the source machine, mount the partition containing the filesystem to be copied, and start the transfer using netcat and tar:

cd /mnt/sda1
tar -czpsf - . | pv -b | nc -l 3333

3) On the destination machine, mount the partition to receive the filesystem, and start the process:

cd /mnt/sda1
nc 192.168.10.101 3333 | pv -b | tar -xzpsf -

The nc (netcat) command is used for any kind of TCP connections between two hosts. The pv (progress viewer) command is used to display the progress of the transfer. tar is used to archive the files on the source machine and un-archive them on the destination.

__________________________


Special Magazine Offer -- Free Gift with Subscription
Receive a free digital copy of Linux Journal's System Administration Special Edition as well as instant online access to current and past issues. CLICK HERE for offer

Linux Journal: delivering readers the advice and inspiration they need to get the most out of their Linux systems since 1994.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
D. Joe's picture

nc may need -p

On November 30th, 2008 D. Joe (not verified) says:

I found it necessary to invoke netcat on the source side of the connection with the -p switch to specify the port, that is

tar cvf - . | nc -l -p 3333

instead of

tar cvf - . | nc -l 3333

This may be due to version differences--I'm using Ubuntu 8.10's "netcat-traditional" package, the description for which notes that it is different from the more featureful "netcat-openbsd" package. YMMV.

Anonymous's picture

cat /dev/hda1 >

On November 3rd, 2008 Anonymous (not verified) says:

cat /dev/hda1 > /dev/hdb1

That's all you need for a perfect clone of your file system. Of course you can pipe it through netcat/ssh/whatever if you need to do it across the network.

Peter L's picture

My recent experience of

On November 1st, 2008 Peter L (not verified) says:

My recent experience of migrating data was when updating PC due to over zealous cleaning of the old box's cpu fan with a vacuum cleaner...
Hence no original PC to connect to. I put the original hard drives in an enclosure with usb connection and then used rsync to copy the data.
Our household computer has several users. An important thing to remember to do when setting up the users on a new system is to specify the same numeric user and group IDs as on the previous system (write them down!). This ensures the permissions preserved by "rsync -a" make sense.
Finally, any suggestions on life cycle management of user data? The way "home" is used for application and user data, and lack of robust migration tools for data, results in an eclectic mess. I copied the old hard drives' data to "/archive" for now....ouch. Manage a home network over a period of a decade or more and the real issues are way deeper than just copying the files!

Mohsen's picture

file system

On October 31st, 2008 Mohsen (not verified) says:

WoW!
very nice

Paul Bartholdi's picture

Copying a Filesystem between Computers, tar + nc vs rsync

On October 30th, 2008 Paul Bartholdi (not verified) says:

I see many drawbacks to the use of tar+nc+pv:
- hard links are not conserved;
- it is difficult to filter out some files (exclude file) or add extra files (include file);
- connection is not encryted, nor compressed (default with rsync, but not necessary);
- rsync can also be used for updating a [sub]partition, without having to copy everything, just the modified files.
- rsync is used without connecting to both computers, either in sink or feed direction.

The only drawback of rsync is the large number of options. The documentation is well done, rather clear, but most options are never used. A typical use of rsync would not take more space than the introduction to tar+nc+pv above, and would be more usefull, in particular to newcomers. Some thing like:

rsync -alvH source_path [user@]host:target_path
in your case:
rsync -alvH /mnt/sda1 192.168.10.101:/mnt/sda1

On the other hand, I agree that nc and in particular pv can be nice little tools, worth knowing about.

Regards, Paul

Felipe Alvarez's picture

rsync

On October 30th, 2008 Felipe Alvarez (not verified) says:

rsync + ssh = really easy, fast, and saves time.

Theo Parmakis's picture

If you want to actually copy

On October 29th, 2008 Theo Parmakis (not verified) says:

If you want to actually copy the *filesystem* (as opposed to simply the files_, you could do something like:

first on machine 2:
nc -l -p 7777 | dd of=PC_one.img

and then on machine 1:
dd if=/dev/hda1 | nc pc2 7777

Where machine 1 is the source and machine 2 is the destination.

Anonymous's picture

I used to use tar, it's

On October 29th, 2008 Anonymous (not verified) says:

I used to use tar, it's good, but it doesn't maintain hard links. Now I've switched to rsync and I am using rsync -avHhx --progress to backup and/or duplicate whole filesystems.

catclub's picture

-z option for tar

On October 29th, 2008 catclub (not verified) says:

I disagree with the -z option.

Given two machines essentially bonded together, isn't the compression a time waster in this situation?

Especially so if you have lots of mostly compressed files - such as jpegs or .tgz files.

Anonymous's picture

Host keys

On October 28th, 2008 Anonymous (not verified) says:

Having just performed one of these mirrorings (personally, I used 'tar ... | ssh', yet another way of accomplishing it ;) ) I stumbled across one more little "gotcha!" -- if you don't generate new host keys on the mirrored machine, you will have two machines sharing one set of access tokens. That's probably not desireable...

jeffc's picture

another way

On October 28th, 2008 jeffc (not verified) says:

Albeit more indepth: I married a geek so now when Jeff gets a new laptop, the missus gets the same one. For the longest time I would get new hardware, drop Linux on it, spend time figuring out all the stuff to install, hardware tweaks, etc, then have to repeat the process on her box. Now, I get the first computer, burn it with whatever distro I want, trick it out so everything is cool, then add her account. Then I boot each machine off of a pair of el-cheapo 256M usb keys, basically do the netcat trick to move it over the gigabit network but I us dd as the in and out on each side. The drives are the same since we always buy two of the same (well, the odds are in our favor). I start the process before we crash for the night and in the morning she reboots into her new system. The one thing you want to remember to do is set the new hostname to something else.

Happy Cloning.

JeffC

goll's picture

pv ?

On October 28th, 2008 goll (not verified) says:

Hi

Where can I find this 'pv' tool ?

Andreas Schamanek's picture

PV:

On October 29th, 2008 Andreas Schamanek (not verified) says:

PV: http://www.ivarch.com/programs/pv.shtml
It's also available as packages in Debian and Debian based systems such as Ubuntu. HTH.

Simon Faulkner's picture

ntfsclone

On October 27th, 2008 Simon Faulkner (not verified) says:

Don't forget ntfsclone for all those Windows boxes out there...

Anonymous's picture

alternative way

On October 27th, 2008 Anonymous (not verified) says:

Why not just use rysnc after booting the target from a livecd (assuming it's a new install), enabling ssh and mounting the target partition?

original:
mkdir /mnt/fake
mount -o bind / /mnt/fake
cd /mnt/fake
rsync -avz --progress ./* user@other.systems.ip.address:/some/mount

/boot /home or /var other other partitions can be done the same way, and attributes are preserved. Bind mounting will keep rsync from trying to copy /proc /dev or /sys from your original / partition, too. Only / has to be bind mounted before copying.

Anonymous's picture

Extended attributes

On October 27th, 2008 Anonymous (not verified) says:

This method won't preserve any extended attributes of the filesystem in question. For that matter, dump can be used, although not supported by all filesystems. For ext2/3, there is dumpe2fs which works just fine.

Post new comment

Please note that comments may not appear immediately, so there is no need to repost your comment.
The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <pre> <ul> <ol> <li> <dl> <dt> <dd> <i> <b>
  • Lines and paragraphs break automatically.

More information about formatting options

Newsletter

Each week Linux Journal editors will tell you what's hot in the world of Linux. You will receive late breaking news, technical tips and tricks, and links to in-depth stories featured on www.linuxjournal.com.
Sign up for our Email Newsletter

Tech Tip Videos

From the Magazine

July 2009, #183

News Flash: Linux Kernel 3.0 to include an on-the-go Expresso machine interface! Ok, maybe not, but Linux is definitely going mobile, from phones to e-readers. Find out more inside about Android, the Kindle 2, the Western Digital MyBook II, The Bug, and Indamixx (a portable recording studio). And if you've gone mobile and you been wanting more Emacs in your life then check out Conkeror.


To compliment the mobile we've got the stationary: parsing command line options with getopt, checking your Ruby code with metric_fu, and building a secure Squid proxy. How is this stationary you ask? What can we say? It's not. We just wanted to see if anybody actually read this part of the page :) .


All this and more, and all you have to do is get your hot sweaty hands on the latest copy of Linux Journal.





Read this issue