File Synchronization with Unison

Keeping directories in sync on multiple machines can be difficult. Running Unison is one way to make the task easier.

Unison is a file-synchronization tool that runs on Linux, UNIX and Microsoft Windows. Those of you who've used IBM Lotus Notes or Intellisync Mobile Suite probably have an idea of what synchronization is good for, as compared to one-way mirroring options such as rsync. You might have mirrored a company document directory to your laptop, for example, and then modified a document or two. Other people might have modified other documents in the same directory by the time you get back. With rsync, you'd need to reconcile the differences between the two directories manually or risk overwriting someone's changes. Unison can sort out what has changed where, propagate the changed files and even merge different changes to the same file if you tell it how.

Think of Unison as two-way rsync with a bit of revision control mixed in. The most common use is keeping your local and remote home directory, or some data directory you often use in different contexts, in sync. It uses the rsync algorithm to keep network traffic down and should be tunneled through SSH over untrusted networks. No extra work is needed—simply specify ssh:// when adding a directory location. Quite a bit of extra disk space often is needed for Unison, though, because the synchronizer needs to keep track of what the files looked like on the last run.

Getting, Compiling and Installing Unison

Unison's home page is maintained at the University of Pennsylvania; the project leader, Benjamin C. Pierce, is a professor in the Department of Computer and Information Science. See the on-line Resources for the URL.

Unison isn't as widely deployed as rsync, so you might not be able to find a precompiled package for your distribution. But the binaries downloadable from the Unison home page should work for most people.

If you'd like to compile from source, you can. A few extra hoops must be jumped through, however, because Unison is programmed in OCaml, not the most common language. See Resources if there is no handy package for your distribution.

Compiling and installing Unison is simple; type make UISTYLE=xxx. The GTK user interface needs additional OCaml bindings for GTK, so I use the text interface in this article. Typing make UISTYLE=text or make UISTYLE=gtk should give you a Unison executable. Simply copy the executable to somewhere in the path on both machines you want to synchronize.

In this article, I'm using the current stable version of Unison, 2.9.1, unless otherwise noted. You need to use the latest betas if you're going to synchronize files larger than 2GB.

The developer versions tend to work well. They are what the developers run themselves on their own precious data. Sign up for the unison-hackers mailing list if you feel a bit adventurous. Jerome Vouillon, Benjamin C. Pierce and Trevor Jim tend to hang out there discussing improvements. Commit logs also float by, so you can track what is going on.

Configuring and Using Unison

Unison keeps its config and working files in a .unison directory in your home directory or wherever you want to put it. Set the UNISON environment variable to specify an alternate location.

The default configuration is stored in .unison/default.prf. Listing 1 shows a plain config file suitable for testing. Synchronizing two directories is now as simple as:

$ unison /nfsmount/dir1 /home/me/dir1

Unison then asks the user about any differences between the directories and offers reasonable defaults. It does take a bit of time to get used to Unison's way of thinking, however. And, Unison is no substitute for backups. Unison happily propagates back the deletion of all the files in one replica, for example, which can be a rude awakening for programmers used to CVS. For example:

rm dir1/* ; unison ssh://server/dir1 dir1

doesn't do what you expect from a:

rm dir1/*; cvs update dir1

Deleting a file is an action that is replicated on the other side upon synchronization. So, this example command removes all files in dir1 on both sides.

Once you feel comfortable, consider adding auto = true to the Unison profile. This skips questions about any non-conflicting changes but gives you a chance to back out at the end.

The Unison manual is recommended reading. It is clear and well written and explains what happens at most corner cases.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Reg: Unison on Linux

pbr's picture

Can any one help me to install unison on Linux, i have tried it but it shows some error. Through ocaml i have combiled it eventhough the executable unison is not get created.
Thanks in advance

A new product to do cross operating system data sync

Anonymous's picture

I think you all will like this product "Antida" that I just used. It is from Dimatas Technologies can can do data sync across win, Mac, Linux, Solaris. Also it does allow you to send and receive data from computer which does not have Antida installed. You can check the software at www.dimatas.com

References??

Anonymous's picture

I don't find any references about this tool or the company. Someone has tried it out?

Antida -- from www.dimatas.com

Anonymous's picture

Yes, I tried. It works and is free. The product is in Beta phase and I think they are trying to improve and make it better. It is designed on peer to peer concept

multiple websites ecommerce system

Anonymous's picture

Hi,

Im running many ecomerce websites from the same source code, just few config files keep site variables and images ect , but im manually updating them all at the moment when i make a change, ive looked at subversion which seems to be over kill , and unison, but is unisonone to one relationship ?

what im looking for is a process whereby i alter one set files of codes and this updates many diffrent webservers, the code is currently ASP running on iis

can anyone suggest tools or process ?

Steven M

There is a nicely convenient

Johnathan's picture

There is a nicely convenient tool called usbsync which has specially been desgined for the purpose of keeping a usb flash drive in sync with multiple Unix/Linux hosts. At the time it's just beta but looks like it's going somewhere.

I found it quite useful as it allows me to perform my simple daily synchronization tasks without having to remember lengthy unison or rsync commands.

You might wanna take a look at it at http://klingspor-thueringen.de/usbsync

Fails

Anonymous's picture

That example won't work..

File "default", line 10: bad pattern: diff3 -m CURRENT1 OLD CURRENT2 > NEW
A pattern must be introduced by one of the following keywords:
Name, Path, or Regex.

Does n't work for me either

Anonymous's picture

Does n't work for me either ...

Missing PathSpec

Mitch Frazier's picture

The option is missing the part that tells unison what types of files this particular merge command applies to. Try something like this:

merge = Name *.txt -> diff3 -m CURRENT1 CURRENTARCH CURRENT2 > NEW || echo "differences detected"

See the unison docs for more information.

Mitch Frazier is an Associate Editor for Linux Journal.

PortXPort - FREE service

Anonymous's picture

I use it few month and it's great.

try it www.portxport.com

Unison - issues for using it a backup

Anonymous's picture

Unison has one serious limitation compared to rsync when you use it for backups (rather then
keeping 2 copies in sync): I would like to simply backup my files and not remove deleted files on the backup even if they had been deleted on the source. The reason is that occasionally I delete a file by mistake and if
I realize it after the unison backup happened then it will have been deleted on the backup.
This option is very easy to support and I am surprised that Unison does not offer it

Another missing feature (also wrt using it for backups) is that it does have the ability to use hard links on the
target; this feature would have allowed one to use it as a time machine.

Due to this I decided to stick with rsync.

You can specify master root

Anonymous's picture

You can specify master root in unison (-force xxx), thus propagating changes only in the way you want. I think it's what you want.

Unison develpment has ben stoped

Luis Medina's picture

That's a shame, a some and grate pice of sw showld be supported and this is a case. It make a easy way to sync files and keep backups and save my life several times.

kernel level unison/rsync

Anonymous's picture

Does anyone know of an open source tool to multi-directionally any number of servers in sync. Basically what I am looking for is something like unison but at the kernel level, so that as soon as a change happens, it propagates the change, and doesnt have to be manually run from the command line or in a crontab. Also something that propagates posix locks would be very helpful. I am looking to mirror samba shares in real time in 3 different cities. There are commercial products that do this (peerfs etc), but I am a big proponent of open source.

Thank you.

incron + unison?

Odin / Velmont's picture

Depending on how fast you want it to be, etc.

You could use incron and unison. incron watches for events, so when you e.g. save a file, or make a new one, you can run commands. So you could easily set that up and have it sync your files quite fast.

you most use DRDB

Anonymous's picture

you most use DRDB

end of life for unison?

David Hartley's picture

From the unison docs?!

http://www.cis.upenn.edu/~bcpierce/unison/download/stable/latest/unison-...

Unison is no longer under active development as a research project.(Our research efforts are now focused on a follow-on project called Harmony, described at http://www.cis.upenn.edu/~bcpierce/harmony.) At
this point, there is no one whose job it is to maintain Unison, fix
bugs, or answer questions.

2 points missing and worth noting

James's picture

Since so many have to live in mixed environments Unsion is capable of backing up that other OS's data to a linux box as easily as linux to linux. It's also a gui tool for the windows and non techy Linux desktop user, They can backup their personal data as often as they to the same repository you use for the main backup, and you don't have to listen to them complaining that the last backup was 10 hours before they downloaded that movie in the e-mail. (Hey you should have backed it up yourself!) I have found that no one uses the added feature until the first person has a meltdown. Suddenly it's a very popular feature since all they have to do is click.

GUI

Mark E.'s picture

Instead I prefer to use a visual file synchronization tool "ViceVersa" http://www.tgrmn.com, which makes the job much easier. ViceVersa works on Windows only, but can sync to linux.

synco

Ruud Mantingh's picture

There is now a visual sync tool for Linux that does something similar to ViceVersa except that it doesn't do automated syncs yet. Manual only. Have a look and see if may be useful. sourceforge.net/projects/synco/

GUI

Mark E.'s picture

Instead I prefer to use a visual file synchronization tool "ViceVersa" www.tgrmn.com, which makes the job much easier. ViceVersa works on Windows only, but can sync to linux.

VisaVersa

Scott Bohler's picture

I use VisaVersa too and it is a great synchronization tool, but not for remote filesystems. Unison works much better for remote file systems separated by slow links.

Example: I synch two large active filesystems, one on the east coast and one in the midwest, daily, via the Internet (encrypted).

VisaVersa:
Perfect synch, but very slow and bandwidth-intensive because it scans the remote filesystem from the executing computer.

Unison:
WHAM!!! Perfect synch, FAST! Fast and non-bandwidth intensive because Unison only communicates and transfers filesystem changes between two remote nodes. Superb tool for an admins and geeks.

Unison has a host of advanced features, but you have to do some technical reading in order to understand and use them properly.

VisaVersa is a great tool but it's not comparable to Unison for remote filesystems applications.

...Rsync? It answers bot

Anonymous's picture

...Rsync?

It answers both comments, I think the biggest issue here is the Diff algorythm nd rsync is taking care ( if not the best out there anyway ) the rest can easily be done via script, both on windows and linus. And yes there is a not only a client but a GUI for windows, and on linux a one line cron script take care of it.

iFolder?

Anonymous's picture

Not even *remotely* as cool as Novell iFolder...

Novell is like living in the

Anonymous's picture

Novell is like living in the stone age...

If you're making a comment

Anonymous's picture

If you're making a comment like that - YOU'RE living in the stone age...

ifolder

Anonymous's picture

I have tried to install ifolder on SUSE as well as UBUNTU. It just is very hard to make it work..... And documentation is very poor.

Compare that to Drupal. I had it working the first time and each time even stupid people could install it a variety of flavors of Linux.

ifolder = hackers dream of busy doing nothing!!!

Re: iFolder?

Anonymous's picture

iFolder in its current open source state is unusable. When there is a server backend and you are using the full version it is excellent but there is a use for unison - its easy to setup and there are Windows clients for it also.

I've been using unison...

Anonymous's picture

I've been using unison for about six months to sync my work dev server and work live server. I spent a fair amount of time looking for a solution to my problems, which were that I needed the sync the servers BOTH ways (the websites allows uploads, which I need synced to me and obviously I need to sync script updates from dev to live). When I finally found unison, I had it installed and setup within mintues. The fact that the text interface is idiot proof is great too, especially considering my dev servers HD crashed yesterday! Reinstalled OS, install unison, sync servers, job done :)

Using Unison for about 3 years now ...

Mike Bright's picture

Just my 2 cents ...

I've been using Unison for about 3 years now. It's saved my butt many times. I use it to synchronize about 7 different machines running various linux flavours, Windows 2000 and XP and even on a Linksys NSLU2.
Especially useful for synchronizing (over ssh) between home and work. or even anywhere and home.

I gave up on the graphical interface as I found it a bit heavy on resources. I thought I'd never use the text interface which I found horrible but one day I tried it again and now always use that. I have a perl script wrapped around the unison text client to simplify creating the profile files for different machines/OS on the fly using DISPLAY/SSH_CLIENT variables to figure out where the server is.

That's not to say I'm not on the lookout for a replacement.
Unison development is slow now (I'm sure it's not helped by the fact that it's written in CAML but that's just my opinion) probably as the team seem to have moved onto something bigger and better....

As a replacement I guess I want:
- multi-OS support as unison
- open source as unison
- a more active community around the tool,
more plugin tools
- better handling of file and directory moves/renames
it's a pain if you just rename a large file to have to manually rename it on the other machine OR put up with a long transfer.
Of course this may not be an easy problem to solve but having a
special move tool, e.g. unisonrename which logs the move
- ability to synchronize against different sources, e.g. imagine synchronizing against a tar archive or a CD/DVD-based archive (and maybe just logging or saving the differences) for example.

Interested to hear your views on what could be improved in unison-like tools, or what alternatives exist or are in development.

Rename of files

Knut-Olav Hoven's picture

Tracking a file is not that hard at all since the file has an inode that doesn't change when moving/renaming the file as long as it is on the same media.

How did you get ocaml compiled on nslu2?

Anonymous's picture

How did you get ocaml compiled on nslu2?

The NSLU2 would likely be a

Anonymous's picture

The NSLU2 would likely be a Debian Slug. It makes the NSLU2 way more worth the money. The overclock hack is nice as well. No heat probs on mine.

On second thoughts ...

Anonymous's picture

Replying to my own entry here - a correction: unison is still advancing much more than I thought.
I'm currently using version 2.9.1 but new versions have been announced with quite a lot of changes:

unison 2.12.0 now available; 2.10.2 promoted to stable

To see full change lists (may not mean much to you if you haven't already used unison) sign up to read messages on the unison-announce list at yahoogroups.com

(You can sign up to the group without being obliged to rev\ceive the e-mails, anyway I think before last night the previous posting was last September).

Personally, I'll be upgrading to 2.10.2 as soon as I get hold of i386 windows and linux binaries (no problem) and arm for my NSLU2.

Still looking for the perfect sync tool!

Anonymous's picture

I have my documents on a usb hard drive which I use to sync between a linux (home) and a windows PC (work). To make the drive readable by both linux and windows, the drive is formatted in fat32.

When connecting the drive to the linux machine for synchronization, however, all the changed files will get the executable bit set under linux. To avoid this, I have to connect the usb drive to a windows pc at home and sync through the samba server to linux (I use vuBrief on Windows). This works, but it is one of the few reasons why I maintain a windows PC at home. One reason to much from my point of view.

A linux sync tool which could deal with the executable bit (set by the windows OS) would be very welcome!

Give ExtX FS Windows Driver a try ! ;-)

Tarax's picture

Hi,

It may be interesting to give Ext2 IFS a try ( http://www.fs-driver.org ) ! (see also FAQ entry about accessing Ext3 FS at http://www.fs-driver.org/faq.html#acc_ext3).
FYI, I'm using it for weeks now on a WinXP SP2 box with an external 160Go firewire HDD. This disk was an hawfull warehouse (145Go) of files of any types and any sizes up to 5Go, coming from many computers, from different times. I had to cleanup and organize it with no Linux box available... three full days of moving, deleting, and renaming later (with valuable help of dualXplorer BTW) I had my job done :)
The only trouble I encountered was some files or folders locking, all solved with a remount.

Hope it helps
Bests
Tarax

the already mentioned unison

Anonymous's picture

the already mentioned unison has a option to decide which bits are synced.

gerhard

Mount options

Anonymous's picture

Have you tried different mount options to get the permissions you want?

see umask, dmask and fmask in the "mount" man page.

did you try the unison

Anonymous's picture

did you try the unison 'perms = 0' option to avoid synchronizing permissions ??

unison: from linux to windows Fat32, workaround

asturbcn's picture

Hello

Synchornize from ext3 fs to vfat (windows Fat32)
I got error chmod error, when unison creates files in Fat32.
Even with perms = 0 option, unison use chmod?? and get

Workaround: mount vfat system with "quiet" option, get works.
quiet mounting option make chmod and chgrp return ok even if error in that file system.

union default.prf:
...
auto = true
batch = true
force = /
prefer = /
owner = false
perms = 0
....

unison version 2.13.16, Linux 2.6.8.1-12mdk, Mandriva (Mandrake) 10.1

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState