The standard protocol for sharing files between Linux boxes is the Network File System (NFS). This protocol, which originated with Sun in the mid 80s, does the job, but it has many deficiencies that can cause trouble for a systems administrator. Though there are alternatives, such as the Andrew File System (AFS) that are much nicer, most of us are stuck with NFS at this time—it is standard, available on every platform under the sun and free. Fortunately, the program AMD (AutoMount Daemon) exists to make living with NFS much easier.
AMD is an automounter—i.e., it maintains a cache of mounted file systems. At a minimum, AMD should be used wherever you use a normal NFS mount, since AMD makes your network more reliable. Because of the stateless design of NFS, any process trying to access data on an NFS partition will blocked if the partition's server goes down. AMD improves the situation by keeping track of which machines are down and which are inaccessible. Since AMD doesn't mount every partition immediately or keep them mounted, as does NFS, you save overhead that otherwise would be used for kernel and network traffic from the unused partitions, and thus improve machine performance.
Configuration and administration become much easier with AMD. Instead of requiring a different fstab file on each host, you can have a single, centrally maintained AMD map which can be distributed as a file with rdist or NIS maps or even Hesiod. As an example, we have over 100 machines with one centrally maintained AMD map. One map file is certainly easier to edit than 100.
Another convenient feature of AMD is dynamic maps that change depending on any number of criteria. A single map can point to multiple places, allowing you to do operations unavailable with normal NFS. For instance, if you have multiple replicated servers, you can set up a map so that if one server goes down, AMD will automatically mount files using one of the others.
AMD operates by mimicking an NFS server. When a file is accessed, AMD uses its map to decide where that file actually resides. It then mounts that partition, if necessary, using regular NFS, and mimics a symlink to the actual location. All AMD actions are done transparently, so that from the user's point of view she is simply accessing a regular Unix symlink that points to a regular user's file. AMD maintains its NFS mounts beneath a temporary directory, by default called “a”, a name choice that can cause problems. For example, the actual physical path of the directory /home/crosby is /a/home/crosby, but /a/home/crosby exists only if someone has recently accessed /home/crosby (or some other path on the same partition). Therefore, users should never explicitly access files through /a.
Diagram 1 demonstrates the three types of mounts involved: the native partition, the AMD pseudo partition and the behind-the-scenes NFS partition.
.................. . NFS Partition . \-+-a---home . | .....^............ +-bin : | : |......... |. AMD . +-home . .........
AMD does a few other things behind the scenes to keep operations healthy. First of all, it sends out rpc requests at regular intervals to all the servers it knows to see if they are alive. If one isn't, AMD will not try to mount it. This checking also allows AMD to offer access to replicated file systems; that is, you can set up multiple redundant servers, and if one goes down, AMD will try to mount another one.
To use AMD, you must first of all build one or more AMD maps. These maps are the configuration files that tell AMD exactly what to do. Many tasks can be done from an AMD map, and documenting them all would take more than one article. Listing 1 provides a sample AMD map with some common tasks, and with comments under each entry to explain it. In general, a map consists of two fields: the name, which is translated to the path name underneath the AMD mount point, and the options, which specify what to do with this path name.
I have merely touched the surface of AMD features in Listing 1. The uses of AMD are almost endless—as the man page says, “A weird imagination is most useful to gain full advantage of all the features.” The documentation that comes with the package gives complete instructions for writing a map.
|Speed Up Your Web Site with Varnish||Jun 19, 2013|
|Non-Linux FOSS: libnotify, OS X Style||Jun 18, 2013|
|Containers—Not Virtual Machines—Are the Future Cloud||Jun 17, 2013|
|Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer||Jun 12, 2013|
|Weechat, Irssi's Little Brother||Jun 11, 2013|
|One Tail Just Isn't Enough||Jun 07, 2013|
- Speed Up Your Web Site with Varnish
- Containers—Not Virtual Machines—Are the Future Cloud
- Non-Linux FOSS: libnotify, OS X Style
- Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer
- Linux Systems Administrator
- Senior Perl Developer
- Technical Support Rep
- UX Designer
- Android's Limits
- Weechat, Irssi's Little Brother
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?