NFSv4 grace time

NFSv4 grace time

[Michael Schwartzkopff]

[-- Attachment #1: Type: text/plain, Size: 820 bytes --]

Hi,

setting up a linux NFS server cluster I experience a strange problem during a 
failover. A client can access a file only after 60 to 90 seconds. On the line I 
see a NFS4ERR_GRACE from the server to the client.

I already set the /proc/fs/nfsd/nfsv4leastime to 10.

Any other idea?

For the tests I have a simple setup:

One host with a simple script that simulates failover:

#!/bin/bash

ip a d 1.2.3.4/24 dev eth0
exportfs -u *:/srv/nfs/home
exportfs -u *:/srv/nfs
/etc/init.d/nfsserver stop
#
/etc/init.d/nfsserver start
exportfs -o fsid=0,rw,crossmnt,no_root_squash *:/srv/nfs
exportfs -o fsid=1000,rw,mountpoint,no_root_squash *:/srv/nfs/home
ip a a 1.2.3.4/24 dev eth0

-- 
Dr. Michael Schwartzkopff
Guardinistr. 63
81375 München

Tel: (0163) 172 50 98
Fax: (089) 620 304 13

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: NFSv4 grace time

[Jeff Layton]

On Thu, 21 Jul 2011 15:40:05 +0200
Michael Schwartzkopff <misch@schwartzkopff.org> wrote:

> Hi,
> 
> setting up a linux NFS server cluster I experience a strange problem during a 
> failover. A client can access a file only after 60 to 90 seconds. On the line I 
> see a NFS4ERR_GRACE from the server to the client.
> 
> I already set the /proc/fs/nfsd/nfsv4leastime to 10.
> 
> Any other idea?
> 
> For the tests I have a simple setup:
> 
> One host with a simple script that simulates failover:
> 
> #!/bin/bash
> 
> ip a d 1.2.3.4/24 dev eth0
> exportfs -u *:/srv/nfs/home
> exportfs -u *:/srv/nfs
> /etc/init.d/nfsserver stop
> #
> /etc/init.d/nfsserver start
> exportfs -o fsid=0,rw,crossmnt,no_root_squash *:/srv/nfs
> exportfs -o fsid=1000,rw,mountpoint,no_root_squash *:/srv/nfs/home
> ip a a 1.2.3.4/24 dev eth0
> 

This is normal. The grace period is there to allow clients to reclaim
their state without other clients racing in and grabbing their locks on
the new server, etc.

You can play with /proc/fs/nfsd/nfsv4gracetime too but I'd be very
leery of setting that too low. It should never be lower than the
previous leasetime (see the comments on write_gracetime in the kernel).

Note that setting the leasetime that small has an effect on performance
too as the clients have to renew their state more frequently.

It also affects the resiliency of your entire setup in the face of
network partitions (including the network partition that occurs during
failover). If the client can't reclaim its state during the grace
period, then it's lost and nasty things can occur (like other clients
stealing posix locks out from under you).

-- 
Jeff Layton <jlayton@redhat.com>

Re: NFSv4 grace time

[Michael Schwartzkopff]

[-- Attachment #1: Type: Text/Plain, Size: 1721 bytes --]

> On Thu, 21 Jul 2011 15:40:05 +0200
> 
> Michael Schwartzkopff <misch@schwartzkopff.org> wrote:
> > Hi,
> > 
> > setting up a linux NFS server cluster I experience a strange problem
> > during a failover. A client can access a file only after 60 to 90
> > seconds. On the line I see a NFS4ERR_GRACE from the server to the
> > client.
> > 
> > I already set the /proc/fs/nfsd/nfsv4leastime to 10.
> > 
> > Any other idea?
> > 
> > For the tests I have a simple setup:
> > 
> > One host with a simple script that simulates failover:
> > 
> > #!/bin/bash
> > 
> > ip a d 1.2.3.4/24 dev eth0
> > exportfs -u *:/srv/nfs/home
> > exportfs -u *:/srv/nfs
> > /etc/init.d/nfsserver stop
> > #
> > /etc/init.d/nfsserver start
> > exportfs -o fsid=0,rw,crossmnt,no_root_squash *:/srv/nfs
> > exportfs -o fsid=1000,rw,mountpoint,no_root_squash *:/srv/nfs/home
> > ip a a 1.2.3.4/24 dev eth0
> 
> This is normal. The grace period is there to allow clients to reclaim
> their state without other clients racing in and grabbing their locks on
> the new server, etc.

Ok. But this is the same client that cannot its own data.
 
> You can play with /proc/fs/nfsd/nfsv4gracetime too but I'd be very
> leery of setting that too low. It should never be lower than the
> previous leasetime (see the comments on write_gracetime in the kernel).

I did already set this to 10 seconds without success. The client still has to 
wait 60 seconds to access its data.

By the way: Is there a nice way to set this during startup of the nfsserver, 
i.e. a mount option for nfsd?

Greetings,

-- 
Dr. Michael Schwartzkopff
Guardinistr. 63
81375 München

Tel: (0163) 172 50 98
Fax: (089) 620 304 13

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: NFSv4 grace time

[Jeff Layton]

On Fri, 22 Jul 2011 13:24:22 +0200
Michael Schwartzkopff <misch@schwartzkopff.org> wrote:

> > On Thu, 21 Jul 2011 15:40:05 +0200
> > 
> > Michael Schwartzkopff <misch@schwartzkopff.org> wrote:
> > > Hi,
> > > 
> > > setting up a linux NFS server cluster I experience a strange problem
> > > during a failover. A client can access a file only after 60 to 90
> > > seconds. On the line I see a NFS4ERR_GRACE from the server to the
> > > client.
> > > 
> > > I already set the /proc/fs/nfsd/nfsv4leastime to 10.
> > > 
> > > Any other idea?
> > > 
> > > For the tests I have a simple setup:
> > > 
> > > One host with a simple script that simulates failover:
> > > 
> > > #!/bin/bash
> > > 
> > > ip a d 1.2.3.4/24 dev eth0
> > > exportfs -u *:/srv/nfs/home
> > > exportfs -u *:/srv/nfs
> > > /etc/init.d/nfsserver stop
> > > #
> > > /etc/init.d/nfsserver start
> > > exportfs -o fsid=0,rw,crossmnt,no_root_squash *:/srv/nfs
> > > exportfs -o fsid=1000,rw,mountpoint,no_root_squash *:/srv/nfs/home
> > > ip a a 1.2.3.4/24 dev eth0
> > 
> > This is normal. The grace period is there to allow clients to reclaim
> > their state without other clients racing in and grabbing their locks on
> > the new server, etc.
> 
> Ok. But this is the same client that cannot its own data.
>  

That may be the case in your situation, but the protocol has to allow for that.

> > You can play with /proc/fs/nfsd/nfsv4gracetime too but I'd be very
> > leery of setting that too low. It should never be lower than the
> > previous leasetime (see the comments on write_gracetime in the kernel).
> 
> I did already set this to 10 seconds without success. The client still has to 
> wait 60 seconds to access its data.
> 

You may need to restart the server and remount the clients in order for it to work.

> By the way: Is there a nice way to set this during startup of the nfsserver, 
> i.e. a mount option for nfsd?
> 

Not currently, but a module parm for this would seem to make more sense
than this file in /proc.

-- 
Jeff Layton <jlayton@redhat.com>

Re: NFSv4 grace time

[J. Bruce Fields]

On Fri, Jul 22, 2011 at 07:41:08AM -0400, Jeff Layton wrote:
> On Fri, 22 Jul 2011 13:24:22 +0200
> Michael Schwartzkopff <misch@schwartzkopff.org> wrote:
> 
> > > On Thu, 21 Jul 2011 15:40:05 +0200
> > > 
> > > Michael Schwartzkopff <misch@schwartzkopff.org> wrote:
> > > > Hi,
> > > > 
> > > > setting up a linux NFS server cluster I experience a strange problem
> > > > during a failover. A client can access a file only after 60 to 90
> > > > seconds. On the line I see a NFS4ERR_GRACE from the server to the
> > > > client.
> > > > 
> > > > I already set the /proc/fs/nfsd/nfsv4leastime to 10.
> > > > 
> > > > Any other idea?
> > > > 
> > > > For the tests I have a simple setup:
> > > > 
> > > > One host with a simple script that simulates failover:
> > > > 
> > > > #!/bin/bash
> > > > 
> > > > ip a d 1.2.3.4/24 dev eth0
> > > > exportfs -u *:/srv/nfs/home
> > > > exportfs -u *:/srv/nfs
> > > > /etc/init.d/nfsserver stop
> > > > #
> > > > /etc/init.d/nfsserver start
> > > > exportfs -o fsid=0,rw,crossmnt,no_root_squash *:/srv/nfs
> > > > exportfs -o fsid=1000,rw,mountpoint,no_root_squash *:/srv/nfs/home
> > > > ip a a 1.2.3.4/24 dev eth0
> > > 
> > > This is normal. The grace period is there to allow clients to reclaim
> > > their state without other clients racing in and grabbing their locks on
> > > the new server, etc.
> > 
> > Ok. But this is the same client that cannot its own data.
> >  
> 
> That may be the case in your situation, but the protocol has to allow for that.

We could be a bit smarter here:

	- Use the NFSv4.1 RECLAIM_COMPLETE to identify when all clients
	  have reclaimed state, if possible.
	- Skip the grace period when we *know* no clients held open
	  state on the previous boot.  (But we'd have to think about
	  v2/v3 clients: the kernel doesn't know about them.  Possibly
	  we can assume they don't use share locks, hence don't need to
	  wait on them for v4 opens.  But we *do* actually have share lock
	  procedures in NLM--I wonder if anyone uses them?  They aren't
	  consistent with v4 opens, argh.)

> > > You can play with /proc/fs/nfsd/nfsv4gracetime too but I'd be very
> > > leery of setting that too low. It should never be lower than the
> > > previous leasetime (see the comments on write_gracetime in the kernel).
> > 
> > I did already set this to 10 seconds without success. The client still has to 
> > wait 60 seconds to access its data.
> > 
> 
> You may need to restart the server and remount the clients in order for it to work.

Note /proc/sys/fs/nfs/nlm_grace_period needs to be shrunk too.

--b.

> 
> > By the way: Is there a nice way to set this during startup of the nfsserver, 
> > i.e. a mount option for nfsd?
> > 
> 
> Not currently, but a module parm for this would seem to make more sense
> than this file in /proc.
> 
> -- 
> Jeff Layton <jlayton@redhat.com>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html