Re: Optimal NFS mount options to safely allow interrupts and timeouts on newer kernels

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 3989 bytes --]

On Wed, 5 Mar 2014 23:03:43 -0600 (CST) Andrew Martin <amartin@xes-inc.com>
wrote:

> ----- Original Message -----
> > From: "NeilBrown" <neilb@suse.de>
> > To: "Andrew Martin" <amartin@xes-inc.com>
> > Cc: linux-nfs@vger.kernel.org
> > Sent: Wednesday, March 5, 2014 9:50:42 PM
> > Subject: Re: Optimal NFS mount options to safely allow interrupts and timeouts on newer kernels
> > 
> > On Wed, 5 Mar 2014 11:45:24 -0600 (CST) Andrew Martin <amartin@xes-inc.com>
> > wrote:
> > 
> > > Hello,
> > > 
> > > Is it safe to use the "soft" mount option with proto=tcp on newer kernels
> > > (e.g
> > > 3.2 and newer)? Currently using the "defaults" nfs mount options on Ubuntu
> > > 12.04 results in processes blocking forever in uninterruptable sleep if
> > > they
> > > attempt to access a mountpoint while the NFS server is offline. I would
> > > prefer
> > > that NFS simply return an error to the clients after retrying a few times,
> > > however I also cannot have data loss. From the man page, I think these
> > > options
> > > will give that effect?
> > > soft,proto=tcp,timeo=10,retrans=3
> > > 
> > > >From my understanding, this will cause NFS to retry the connection 3 times
> > > >(once
> > > per second), and then if all 3 are unsuccessful return an error to the
> > > application. Is this correct? Is there a risk of data loss or corruption by
> > > using "soft" in this way? Or is there a better way to approach this?
> > 
> > I think your best bet is to use an auto-mounter so that the filesystem gets
> > unmounted if the server isn't available.
> Would this still succeed in unmounting the filesystem if there are already
> processes requesting files from it (and blocking in uninterruptable sleep)?

The kernel would allow a 'lazy' unmount in this case.  I don't know if any
automounter would try a lazy unmount though - I suspect not.

A long time ago I used "amd" which would create syslinks to a separate tree
where the filesystems were mounted.  I'm pretty sure that when a server went
away the symlink would disappear even if the unmount failed.
So while any processes accessing the filesystem would block, new processes
would not be able to find the filesystem and so would not block.


> 
> > "soft" always implies the risk of data loss.  "Nulls Frequently Substituted"
> > as it was described to very many years ago.
> > 
> > Possibly it would be good to have something between 'hard' and 'soft' for
> > cases like yours (you aren't the first to ask).
> > 
> >  From http://docstore.mik.ua/orelly/networking/puis/ch20_01.htm
> > 
> >    BSDI and OSF /1 also have a spongy option that is similar to hard , except
> >    that the stat, lookup, fsstat, readlink, and readdir operations behave
> >    like a soft MOUNT .
> > 
> > Linux doesn't have 'spongy'.  Maybe it could.  Or maybe it was a failed
> > experiment and there are good reasons not to want it.
> 
> The problem that sparked this question is a webserver where apache can serve
> files from an NFS mount. If the NFS server becomes unavailable, then the apache
> processes block in uninterruptable sleep and drive the load very high, forcing
> a server restart. It would be better for this case if the mount would simply 
> return an error to apache, so that it would give up rather than blocking 
> forever and taking down the system. Can such behavior be achieved safely?

If you have a monitoring program that notices this high load you can try
  umount -f /mount/point

The "-f" should cause outstanding requests to fail.  That doesn't stop more
requests being made though so it might not be completely successful.
Possibly running it several times would help.

  mount --move /mount/point /somewhere/safe
  for i in {1..15}; do umount -f /somewhere/safe; done

might be even better, if you can get "mount --move" to work.  It doesn't work
for me, probably the fault of systemd (isn't everything :-)).

NeilBrown



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]