Kernel Korner - Unionfs: Bringing Filesystems Together

Unionfs merges several directories into a single unified view. We describe applications of Unionfs and also interesting implementation aspects.
Conclusion

Unionfs recursively merges several underlying directories or branches into a single virtual view. The efficient fan-out structure of Unionfs makes it suitable for many applications. Unionfs can be used to provide merged distribution ISOs, a single /home namespace and more. Unionfs's copy-on-write semantics make it useful for source code versioning, snapshotting and patching CD-ROMs. We benchmarked Unionfs's performance under Linux 2.4. For a compile benchmark with one to four branches, Unionfs overhead is only 12%. For an I/O intensive workload, the overhead ranges from 10% for a single branch to 12% for four branches.

Acknowledgements

Thanks to Puja Gupta, Harikesavan Krishnan, Mohammad Zubair and Jay Dave, who also were on the Unionfs development team. Special thanks go to Mohammad for helping to get the software for this article prepared for release.

Resources for this article: www.linuxjournal.com/article/7812.

Charles P. Wright (cwright@cs.stonybrook.edu) is a Computer Science PhD student at Stony Brook University. Charles conducts operating systems research with a focus on filesystems, security and extensibility. He also is active in the Linux Users Group at Stony Brook (LUGSB).

Erez Zadok (ezk@cs.stonybrook.edu) is on the Computer Science faculty at Stony Brook University, the author of Linux NFS and Automounter Administration (Sybex, 2001), the creator and maintainer of the FiST stackable templates system and the primary maintainer of the Am-utils (aka Amd) automounter. Erez conducts operating systems research with a focus on filesystems, security, versatility and portability.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

"I think UnionFS is great

Thinking's picture

"I think UnionFS is great for Live CD applications(e.g. DSL/Linux - many others). Also, in general UnionFS is great for small simple applications like remembering writes (COWs - copy on writes) to any read-only file system (e.g. CDROM). However, beyond that, and especially the tomato example I think UnionFS is actually far too dangerous for serious computing. Why? Because there is no intelligent way to remember/decide the duplicate file issues. That being said I'm actually very pro-UnionFS; but like seasoning a fine stew - a little seasoning goes a long way, too much seasoning spoils the stew - re: a little UnionFS is great, beyond that it is DANGEROUS".
adipex phentermine
buy cheap phentermine online
buy cheap phentermine
buy phentermine online

UnionFS - great for some apps, very dangerous for others

mjc777's picture

I think UnionFS is great for Live CD applications(e.g. DSL/Linux - many others). Also, in general UnionFS is great for small simple applications like remembering writes (COWs - copy on writes) to any read-only file system (e.g. CDROM). However, beyond that, and especially the tomato example I think UnionFS is actually far too dangerous for serious computing. Why? Because there is no intelligent way to remember/decide the duplicate file issues. That being said I'm actually very pro-UnionFS; but like seasoning a fine stew - a little seasoning goes a long way, too much seasoning spoils the stew - re: a little UnionFS is great, beyond that it is DANGEROUS (IMHO).

unionfs snapshots of live mounted systems

Cesar's picture

Is it possible to create snapshots of live mounted systems
using unionfs? If so how to do it? I would like some help
on this issue. I currently use LVM2 snapshots to do
consistent backups of live running systems and are having
some problems: system hangs, hard locks, etc. If I could
do the same as with LVM2 snapshots but with the unionfs
it would be great.
thanks

ISO-File from unionfs

pmt's picture

Is it possible to make a iso-file from unionfs?

I have tried it with mkisofs, but the directory depth is too high.
What i can do?

re:ISO-File from unionfs

Jake Miller's picture

Hi Joost,
Is it the unionfs stuff discussed recently here used for slax?

Unknown:
A little more involved to modify the image, perhaps check the slax
site for hints. Note: .iso images are built with mkisofs, not written
to like a read/write filesystem. This _is_ documented, seek ;)

Cheers,

re:ISO-File from unionfs

Randall M.'s picture

Yes, NetBSD 3.0 has support for compressed VND images - they work exactly
the same way as e.g. in Knoppix on Linux or Freesbie on FreeBSD.
The kernel needs to have support enabled via the VND_COMPRESSION kernel
option (see options(4)). After that, you can create a disk image using
vnd(4), and then compress that image with vndcompress(1), put it on the
CD, configure it with vnconfig(8)'s -z option, then mount as usual.
You can use UFS or CD9660 as filesystem, your choice, vnd(4) won't care.

re:ISO-File from unionfs

Bill Manson's picture

apparently, unionfs is a standard part of slax. but of course it is only
operating when you're booting from the slax cd, not if you're mounting it
as an iso. and even then, it only creates the illusion of a modifiable fs,
the cd itself isn't changed.

Re:ISO-File from unionfs

Anonymous's picture

Do it with ex2fs . An Image using ex2fs (or other native Linux fs) can be mounted/read ro in Linux without a problem

error in code examples?

Anonymous's picture

In some code examples there is a "," and in some a ":" between the two "dirs=". I get an error when using ",", with ":" it works.

NFS exports

Anonymous's picture

Most problems reported above (compile-time error, Unknown error 524 while doing ls, etc) are fixed in v1.0.4.

However, although the article claims that it is possible to NFS-export and mount a unionfs FS, I get a "getfh failed: Operation not permitted" from mountd when I try to mount it from remote. If I export a parent directory of the unionfs dir, I can mount the parent but cd-ing to the unionfs dir shows an empty directory. Has anyone ever suceeded in NFS-exporting and remote mounting a unionfs directory?

Cheers,
Gabor

Re: NFS exports

Geoff Mishkin's picture

I have a similar problem in version 1.0.9. When I try to mount the unionized directory, I get:

mount: amsa.dnsalias.org:/mnt/amsa-e failed, reason given by server: Permission denied

However, in the server logs, it would seem that the mount is successful:

Mar 15 18:21:12 [rpc.mountd] authenticated mount request from comm700-c0501-dhcp088.bu.edu:707 for /mnt/amsa-e (/mnt/amsa-e)

However, the packet trace indicates a return status of ERR_ACCESS to the mount request.

Anyone else have a problem like this?

The official NEWS file in the

Anonymous's picture

The official NEWS file in the current download (1.0.11) admits that NFS was broken in 1.0.9 on 2.6 kernels, and claims this to be fixed in 1.0.11.

NFS Exports

Ian Rawlings's picture

Yep, also having exactly the same problem, it appears to be impossible to export a unionfs via NFS. Just to test I wasn't doing anything wrong I have tried exporting the parent directory of a unionfs mountpoint and was able to see the directory and the unionfs mountpoint, but trying to list the contents of the unionfs mountpoint (i.e. the contents of the unionfs itself) lead to a frozen command that eventually freed up with a "permission denied" once I'd restarted the NFS server.

A great shame, UnionFS would be very useful for me and my discless clients, but it appears that it's not compatible with NFS. Note that the initial "authenticated mount request" in the logs means that mountd was happy with the mount request, so it's the NFS daemons/kernel threads that are refusing to co-operate.

RE: NFS

derFlo's picture

official mailinglist: http://www.fsl.cs.sunysb.edu/pipermail/unionfs/2005-June/000610.html
cite:knfsd will not export Unionfs unless you
have an fsid option in /etc/exports

http://www.fsl.cs.sunysb.edu/pipermail/unionfs/2005-April/000350.html
cite:/mnt/unionfs *(rw,fsid=42,async)

i dont know what this fsid means (goin to sleep now), but just (touch) created a file throu a nfs mounted directory that showed up in the rw branch of my union.. (debian kernel-image-2.6-11-1k7 , module-assistent,unionfs unionfs-source 1.0.11-1)

unionfs for 2.6

Anonymous's picture

Is unionFS is available for 2.6 kernels?

Article and related home page lack download link

Anonymous's picture

I was quite interested in the unionfs article in the newest issue
but can not find any link to download the unionfs software, even
from the project's homepage. Please talk with the author to
resolve this oversight.

Software available here

Anonymous's picture

ftp://ftp.fsl.cs.sunysb.edu/pub/unionfs

Though, I've been trying to get it to compile with 2.4.27/28 and had
no luck so far...

fix f. kernel 2.4.27

Anonymous's picture

if it's about an error concerning implicit declaration of
IS_IMMUTABLE, try this:

--- namei.c.org Thu Jan 13 03:18:06 2005
+++ namei.c Thu Jan 13 03:38:17 2005
@@ -67,7 +67,7 @@
if (IS_APPEND(dir))
return -EPERM;
if (check_sticky(dir, victim->d_inode)||IS_APPEND(victim->d_inode)||
- IS_IMMUTABLE(victim->d_inode))
+ IS_IMMUTABLE_FILE(victim->d_inode)||IS_IMMUTABLE_LINK(victim->d_inode))
return -EPERM;
if (isdir) {
if (!S_ISDIR(victim->d_inode->i_mode))

make CC=gcc-3.0

http://www.fsl.cs.sunysb.edu/

Anonymous's picture

http://www.fsl.cs.sunysb.edu/project-unionfs.html

Oops. This is the author's page for the above link. Sorry

/src/unionfs-1.0.3# make all

Anonymous's picture

/src/unionfs-1.0.3# make all
gcc -D__KERNEL__ -DMODULE -DFISTGEN -I. -I/lib/modules/`uname -r`/build/include -Wall -Wno-unused-label -Werror -g -O2 -c -o subr.o subr.c
In file included from /lib/modules/2.4.27-1-k7/build/include/linux/spinlock.h:6,
from /lib/modules/2.4.27-1-k7/build/include/linux/wait.h:16,
from /lib/modules/2.4.27-1-k7/build/include/linux/fs.h:12,
from /lib/modules/2.4.27-1-k7/build/include/linux/capability.h:17,
from /lib/modules/2.4.27-1-k7/build/include/linux/binfmts.h:5,
from /lib/modules/2.4.27-1-k7/build/include/linux/sched.h:9,
from fist.h:32,
from subr.c:22:
/lib/modules/2.4.27-1-k7/build/include/asm/system.h: In function `__set_64bit_var':
/lib/modules/2.4.27-1-k7/build/include/asm/system.h:190: warning: dereferencing type-punned pointer will break strict-aliasing rules
/lib/modules/2.4.27-1-k7/build/include/asm/system.h:190: warning: dereferencing type-punned pointer will break strict-aliasing rules
make: *** [subr.o] Error 1

I get the same results with a virgin 2.4.28 source code too.

Anyone?

Got unionfs working

Steve's picture

I finally got unionfs working after...

Downgrading from 2.6.9 to 2.4.28
and
Downgrading from gcc 3.3.4 to gcc 3.2.3 (just for the build)

I hope this helps.

Steve

Spoke too soon...

Anonymous's picture

I've tried searching google, but with no luck. First of I discovered that the command to mount a unionfs is mount -t unionfs -o dir=/dirone:/dirtwo none /combined ... you'll note the : as the article uses a , separator. The other problem I'm having is running 'ls -l' on the /combined directory give me multiple "Unknown error 524" for all subdirectoried, thought it seems to be working.

Perfect! It works here too. J

Anonymous's picture

Perfect! It works here too. Just incase anyone is interested. I'm using Debian Sarge and just installed the gcc-3.2 compiler. Then changed the symlink gcc to point to gcc-3.2.

Thanks Steve! :-)

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState