Re: NFSv4 grace time

["J. Bruce Fields" <bfields@fieldses.org>]

On Fri, Jul 22, 2011 at 07:41:08AM -0400, Jeff Layton wrote:
> On Fri, 22 Jul 2011 13:24:22 +0200
> Michael Schwartzkopff <misch@schwartzkopff.org> wrote:
> 
> > > On Thu, 21 Jul 2011 15:40:05 +0200
> > > 
> > > Michael Schwartzkopff <misch@schwartzkopff.org> wrote:
> > > > Hi,
> > > > 
> > > > setting up a linux NFS server cluster I experience a strange problem
> > > > during a failover. A client can access a file only after 60 to 90
> > > > seconds. On the line I see a NFS4ERR_GRACE from the server to the
> > > > client.
> > > > 
> > > > I already set the /proc/fs/nfsd/nfsv4leastime to 10.
> > > > 
> > > > Any other idea?
> > > > 
> > > > For the tests I have a simple setup:
> > > > 
> > > > One host with a simple script that simulates failover:
> > > > 
> > > > #!/bin/bash
> > > > 
> > > > ip a d 1.2.3.4/24 dev eth0
> > > > exportfs -u *:/srv/nfs/home
> > > > exportfs -u *:/srv/nfs
> > > > /etc/init.d/nfsserver stop
> > > > #
> > > > /etc/init.d/nfsserver start
> > > > exportfs -o fsid=0,rw,crossmnt,no_root_squash *:/srv/nfs
> > > > exportfs -o fsid=1000,rw,mountpoint,no_root_squash *:/srv/nfs/home
> > > > ip a a 1.2.3.4/24 dev eth0
> > > 
> > > This is normal. The grace period is there to allow clients to reclaim
> > > their state without other clients racing in and grabbing their locks on
> > > the new server, etc.
> > 
> > Ok. But this is the same client that cannot its own data.
> >  
> 
> That may be the case in your situation, but the protocol has to allow for that.

We could be a bit smarter here:

	- Use the NFSv4.1 RECLAIM_COMPLETE to identify when all clients
	  have reclaimed state, if possible.
	- Skip the grace period when we *know* no clients held open
	  state on the previous boot.  (But we'd have to think about
	  v2/v3 clients: the kernel doesn't know about them.  Possibly
	  we can assume they don't use share locks, hence don't need to
	  wait on them for v4 opens.  But we *do* actually have share lock
	  procedures in NLM--I wonder if anyone uses them?  They aren't
	  consistent with v4 opens, argh.)

> > > You can play with /proc/fs/nfsd/nfsv4gracetime too but I'd be very
> > > leery of setting that too low. It should never be lower than the
> > > previous leasetime (see the comments on write_gracetime in the kernel).
> > 
> > I did already set this to 10 seconds without success. The client still has to 
> > wait 60 seconds to access its data.
> > 
> 
> You may need to restart the server and remount the clients in order for it to work.

Note /proc/sys/fs/nfs/nlm_grace_period needs to be shrunk too.

--b.

> 
> > By the way: Is there a nice way to set this during startup of the nfsserver, 
> > i.e. a mount option for nfsd?
> > 
> 
> Not currently, but a module parm for this would seem to make more sense
> than this file in /proc.
> 
> -- 
> Jeff Layton <jlayton@redhat.com>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html