How-To: Release Stuck NFS Mounts without a Reboot

Computing environments may revolve around heavy usage of NFS infrastructure. Network areas are hosted and provided by storage file servers, with compute servers mounting the exported areas into their directory tree. Periodically, the mounts expire when not in use and are removed from the directory tree on local machines.

If NFS traffic between file servers and compute servers is disrupted, machines holding active mounts will not be able to release them. This could happen in several cases. A potential root cause for the problem might be a network outage; however, this issue will be resolved the moment network traffic is resumed. A far more serious scenario is when a file server stops its service due to malfunction or coordinated maintenance, including an EOL (End of Life) cycle. Any host holding active mounts of areas exported by a file server that is no longer reachable will not release them.

This situation leads to so-called stuck mounts. Any process running on an affected computer server waiting for NFS activity will remain in uninterruptible sleep state without any way to kill it. Up to date, the only sensible solution was to reboot affected hosts. However, critical computer servers, including infrastructure and interactive-use machines, cannot be rebooted without prior coordination. This leaves a time window until the next reboot, during which the affected hosts will continue running in an unstable state.

We provide a rebootless solution to this problem by bringing up a fake file server, a dedicated host assigned the same IP like an unreachable file server, which then rejects the NFS requests from compute servers, freeing them. The special host uses its own network architecture, allowing it to be moved to any VLAN (Virtual LAN) used by file servers. The IP change is done using a Web interface. (A VLAN is a group of hosts with a common set of requirements that communicate as if they were attached to the same broadcast domain, regardless of their physical location. A VLAN has the same attributes as a physical LAN, but it allows for end stations to be grouped together, even if they are not located on the same network switch. Network reconfiguration can be done through software instead of physically relocating devices.)

The stuck NFS mounts solution consists of several key components:

  • A dedicated host used for faking the IP addresses—we use a virtual machine with minimal hardware requirements. The virtual machine is assigned a separate NIC in the virtualization server. This host is known as the Admin server.

  • A network infrastructure that supports on-the-fly assignment of the "fake" file server (Admin) to the file server VLANs. Admin is connected directly to the NFS router using a single standard NFS connection, with a 3Gb channel to primary port and 1Gb channel to backup port (Figure 1).

  • A Web interface for assigning Admin to file server VLANs.

  • A mechanism to change the IP address on Admin, which can be automated (scripted) or manual.

Figure 1. Connection to NFS Router

In our site, Admin is a virtual machine hosted on a virtualization server. The host runs a standard enterprise Linux image; however, it is used only for IP changes and no other purposes, and is configured with minimal disk space and memory. The virtual machine is assigned its own NIC in the virtual server network configuration.

The NFS stuck mounts VLAN change is executed using a Web interface. The GUI allows system administrators with no networking knowledge or access permissions to change the VLAN allocation for the specific router port where the dedicated server is connected to the desired VLAN. The GUI has a static configuration file, with the server name, router name and dedicated port. Once called through the Web interface, it uses SNMP to collect the data from the router and present the table in the middle, the current status and the VLAN drop-down menu. Figure 2 shows a mockup of the Web interface.

Figure 2. Part of the Web Interface for VLAN Change

On clicking the Change VLAN button, the script, using SNMP, changes the VLAN allocation for the specific port. Figure 3 shows the network flow.

Figure 3. VLAN Change Diagram

Stuck mounts usually are identified by indirect symptoms exhibited by affected hosts, such as increased load due to a large number of processes in an uninterruptible sleep state or lsof not being able to complete its run. The first step is to open the Web interface and change the VLAN to that matching the unreachable file server. Once that's done, data-center operations technicians are instructed to change the IP address and default gateway on the Admin server to match that of the unreachable file server. Alternatively, the IP address and default gateway can be configured manually by logging in to the machine via the virtualization console and performing the change. The host IP address can be changed under /etc/sysconfig/network/ifcfg-ethx or ifcfg-eth-idxxx; the default gateway is configured under /etc/sysconfig/network/routes.

Once those two steps are complete, the network service on Admin needs to be restarted with /etc/init.d/network restart. The "fake" file server will come up and start rejecting NFS requests from affected hosts, releasing the stuck processes. At this stage, system administrators also can try to umount the stuck mounts.

We encountered a number of challenges in setting up this solution. On the network side, file servers are connected to routers; whereas compute servers are connected to switches. This necessitated an innovative approach by the network operations team, which resulted in the dedicated connection to the NFS router.

Furthermore, this solution requires allocating site assets to be used only rarely. We chose the virtual machine because of reduced cost and higher flexibility in the setup rather than using a physical machine. Finally, the IP/VLAN change procedure requires discipline and some skill on behalf of system administrators to use it effectively.

We first tried the stuck NFS solution as a hack by manually changing the IP address of a standard computer server located on the same segment as the unreachable file server more than a year ago. The attempt proved successful, and we were able to save a number of important infrastructure machines with stuck mounts, preventing downtime and reboots.

Prompted by this early success, we implemented the full solution and have used it on a number of occasions. In July 2010, we prevented a number of Samba servers from being rebooted, alongside another 21 compute servers.

The stuck NFS mounts solution has a direct impact on our ability to recover from unexpected file server outages. We also are able to mitigate scheduled file server EOL leftovers without reboots. In fact, one might argue that the solution creates leniency that may lead to lesser discipline in using strict file server EOL procedures, which requires all mounts to be removed before the file servers are shut down. We believe that the stuck NFS mounts solution does not replace hard work and adherence to procedures and should be used only in an emergency.

The stuck NFS mounts solution is a valuable asset and a necessity in large and dynamic environments. It allows us to maintain high uptime of important machines, without unnecessary reboots due to mistakes or file server traffic disruptions.

Our solution is simple and effective, and it has proven its worth on several occasions in real-life circumstances. The implementation is transparent to users and allows system administrators with no network operations of NFS permissions or extensive knowledge to recover systems quickly and easily. We recommend its use as a standard in computing environments that rely on NFS infrastructure.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Use an Firewall?

Lord_Pinhead's picture

Can´t you just use the firewall on the NFS Client an reset any traffic to the NFS Server and try to unmount the mounts with umount -f ? Didn´t have ever problems with stuck NFS Mounts, so i cant tell.

lazy unmount

Gavin's picture

Have you tried using umount -l ??? This is much easier than bringing up a fake server.

I'll second this option,

Anonymous's picture

I'll second this option, umount -l has worked great for me. I've actually used it on a production system recently, to avoid the dreaded reboot.

sm-notify should do the job

Ugis's picture

Actually, sm-notify should release locked NFS shares, at least after server reboots.
http://manpages.ubuntu.com/manpages/natty/man8/sm-notify.8.html
I have experimented with it a little, but it seems it does not help much(find on nfs share resumed after server reboot, but in case of dd->nfs share still locked the system).
It would be nice to find out how to use this tool correctly - so it really unlocked the nfs client.

Isn't this what the "intr" option on NFS mounts is for?

Jesse Pollard's picture

If the client mounts with the intr option, then the waits are no longer non-interruptable...

The intr option I believe

Anonymous's picture

The intr option I believe determines whether the mount operation can be interrupted or not. I don't think that there is the concept of interrupting the nfs connection of a mounted filesystem.

This is closer to the hard or soft options And even though the soft option is more
convenient it is dangerous to use for certain types is uses (binaries or databases for example you want certain types of io to wait for completion).

I used umount to dismount nfs - even stuck one.

Jesse Pollard's picture

I used to do a "umount" on stuck filesystems. The first one would report the error.

The second one would force the dismount.

This only worked if the NFS filesystem were mounted "intr".

Igor, This has worked for me

Jayson Broughton's picture

Igor,

This has worked for me in the past, but I assign a virtual interface with the IP of the nfs server, then unmount the nfs mount

ifconfig eth0:1 192.168.0.15 netmask 255.255.252.0
ifconfig eth0:1 up
umount /nfsdirectory
vim /etc/fstab (remove entry)
ifconfig eth0:1 down

The virtual interface mimics the connection of the nfs server enough where the client thinks it's still there and can do an unmount. It's been awhile since I've done it but I do remember it's worked on several production servers without having to do a reboot when the primary NFS server changed IP addresses or went away.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix