Re: grace period

[Bruce Fields <bfields@fieldses.org>]

On Tue, Jul 05, 2016 at 04:05:56PM -0700, Marc Eshel wrote:
> Can you please point me to the kernel that you are using so I can check if 
> it is an obvious problem before I open an RHEL bug?

I've tried it on the latest upstream and on rhel 3.10.0-327.13.1.el7.

--b.

> Thanks, Marc. 
> 
> 
> 
> From:   Bruce Fields <bfields@fieldses.org>
> To:     Marc Eshel/Almaden/IBM@IBMUS
> Cc:     linux-nfs@vger.kernel.org, Tomer Perry <TOMP@il.ibm.com>
> Date:   07/05/2016 01:52 PM
> Subject:        Re: grace period
> Sent by:        linux-nfs-owner@vger.kernel.org
> 
> 
> 
> On Sat, Jul 02, 2016 at 10:30:11PM -0700, Marc Eshel wrote:
> > I tried again NFSv3 locks with xfs export. "echo 0 > 
> > /proc/fs/nfsd/threads" releases locks on rhel7.0 but not on rhel7.2
> > What else can I show you to find the problem?
> 
> Sorry, I can't reproduce, though I've only tried a slightly later kernel
> than that.  Could you submit a RHEL bug?
> 
> --b.
> 
> > Marc.
> > 
> > works:
> > [root@boar11 ~]# uname -a
> > Linux boar11 3.10.0-123.el7.x86_64 #1 SMP Mon May 5 11:16:57 EDT 2014 
> > x86_64 x86_64 x86_64 GNU/Linux
> > [root@boar11 ~]# cat /etc/redhat-release 
> > Red Hat Enterprise Linux Server release 7.0 (Maipo)
> > 
> > not working:
> > [root@sonascl21 ~]# uname -a
> > Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP Thu 
> 
> > Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
> > [root@sonascl21 ~]# cat /etc/redhat-release 
> > Red Hat Enterprise Linux Server release 7.2 (Maipo)
> > [root@sonascl21 ~]# cat /proc/fs/nfsd/threads 
> > 0
> > [root@sonascl21 ~]# cat /proc/locks
> > 1: POSIX  ADVISORY  WRITE 2346 fd:00:1612092569 0 9999
> > 
> > 
> > 
> > From:   Bruce Fields <bfields@fieldses.org>
> > To:     Marc Eshel/Almaden/IBM@IBMUS
> > Cc:     linux-nfs@vger.kernel.org, Tomer Perry <TOMP@il.ibm.com>
> > Date:   07/01/2016 05:58 PM
> > Subject:        Re: grace period
> > 
> > 
> > 
> > On Fri, Jul 01, 2016 at 03:42:43PM -0700, Marc Eshel wrote:
> > > Yes, the locks are requested from another node, what fs are you using, 
> I 
> > 
> > > don't think it should make any difference, but I can try it with the 
> > same 
> > > fs. 
> > > Make sure you are using v3, it does work for v4.
> > 
> > I tested v3 on upstream.--b.
> > 
> > > Marc.
> > > 
> > > 
> > > 
> > > From:   Bruce Fields <bfields@fieldses.org>
> > > To:     Marc Eshel/Almaden/IBM@IBMUS
> > > Cc:     linux-nfs@vger.kernel.org, Tomer Perry <TOMP@il.ibm.com>
> > > Date:   07/01/2016 02:01 PM
> > > Subject:        Re: grace period
> > > 
> > > 
> > > 
> > > On Fri, Jul 01, 2016 at 01:46:42PM -0700, Marc Eshel wrote:
> > > > This is my v3 test that show the lock still there after echo 0 > 
> > > > /proc/fs/nfsd/threads
> > > > 
> > > > [root@sonascl21 ~]# cat /etc/redhat-release 
> > > > Red Hat Enterprise Linux Server release 7.2 (Maipo)
> > > > 
> > > > [root@sonascl21 ~]# uname -a
> > > > Linux sonascl21.sonasad.almaden.ibm.com 3.10.0-327.el7.x86_64 #1 SMP 
> 
> > Thu 
> > > 
> > > > Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
> > > > 
> > > > [root@sonascl21 ~]# cat /proc/locks | grep 999
> > > > 3: POSIX  ADVISORY  WRITE 2349 00:2a:489486 0 999
> > > > 
> > > > [root@sonascl21 ~]# echo 0 > /proc/fs/nfsd/threads
> > > > [root@sonascl21 ~]# cat /proc/fs/nfsd/threads
> > > > 0
> > > > 
> > > > [root@sonascl21 ~]# cat /proc/locks | grep 999
> > > > 3: POSIX  ADVISORY  WRITE 2349 00:2a:489486 0 999
> > > 
> > > Huh, that's not what I see.  Are you positive that's the lock on the
> > > backend filesystem and not the client-side lock (in case you're doing 
> a
> > > loopback mount?)
> > > 
> > > --b.
> > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > From:   Bruce Fields <bfields@fieldses.org>
> > > > To:     Marc Eshel/Almaden/IBM@IBMUS
> > > > Cc:     linux-nfs@vger.kernel.org
> > > > Date:   07/01/2016 01:07 PM
> > > > Subject:        Re: grace period
> > > > 
> > > > 
> > > > 
> > > > On Fri, Jul 01, 2016 at 10:31:55AM -0700, Marc Eshel wrote:
> > > > > It used to be that sending KILL signal to lockd would free locks 
> and 
> > 
> > > > start 
> > > > > Grace period, and when setting nfsd threads to zero, 
> > > nfsd_last_thread() 
> > > > > calls nfsd_shutdown that called lockd_down that I believe was 
> > causing 
> > > > both 
> > > > > freeing of locks and starting grace period or maybe it was setting 
> 
> > it 
> > > > back 
> > > > > to a value > 0 that started the grace period.
> > > > 
> > > > OK, apologies, I didn't know (or forgot) that.
> > > > 
> > > > > Any way starting with the kernels that are in RHEL7.1 and up echo 
> 0 
> > > 
> > > > > /proc/fs/nfsd/threads doesn't do it anymore, I assume going to 
> > common 
> > > > > grace period for NLM and NFSv4 changed things.
> > > > > The question is how to do IP fail-over, so when a node fails and 
> the 
> > 
> > > IP 
> > > > is 
> > > > > moving to another node, we need to go into grace period on all the 
> 
> > > nodes 
> > > > 
> > > > > in the cluster so the locks of the failed node are not given to 
> > anyone 
> > > 
> > > > > other than the client that is reclaiming his locks. Restarting NFS 
> 
> > > > server 
> > > > > is to distractive.
> > > > 
> > > > What's the difference?  Just that clients don't have to reestablish 
> > tcp
> > > > connections?
> > > > 
> > > > --b.
> > > > 
> > > > > For NFSv3 KILL signal to lockd still works but for 
> > > > > NFSv4 have no way to do it for v4.
> > > > > Marc. 
> > > > > 
> > > > > 
> > > > > 
> > > > > From:   Bruce Fields <bfields@fieldses.org>
> > > > > To:     Marc Eshel/Almaden/IBM@IBMUS
> > > > > Cc:     linux-nfs@vger.kernel.org
> > > > > Date:   07/01/2016 09:09 AM
> > > > > Subject:        Re: grace period
> > > > > 
> > > > > 
> > > > > 
> > > > > On Thu, Jun 30, 2016 at 02:46:19PM -0700, Marc Eshel wrote:
> > > > > > I see that setting the number of nfsd threads to 0 (echo 0 > 
> > > > > > /proc/fs/nfsd/threads) is not releasing the locks and putting 
> the 
> > > > server 
> > > > > 
> > > > > > in grace mode.
> > > > > 
> > > > > Writing 0 to /proc/fs/nfsd/threads shuts down knfsd.  So it should
> > > > > certainly drop locks.  If that's not happening, there's a bug, but 
> 
> > > we'd
> > > > > need to know more details (version numbers, etc.) to help.
> > > > > 
> > > > > That alone has never been enough to start a grace period--you'd 
> have 
> > 
> > > to
> > > > > start knfsd again to do that.
> > > > > 
> > > > > > What is the best way to go into grace period, in new version of 
> > the
> > > > > > kernel, without restarting the nfs server?
> > > > > 
> > > > > Restarting the nfs server is the only way.  That's true on older 
> > > kernels
> > > > > true, as far as I know.  (OK, you can apparently make lockd do 
> > > something
> > > > > like this with a signal, I don't know if that's used much, and I 
> > doubt
> > > > > it works outside an NFSv3-only environment.)
> > > > > 
> > > > > So if you want locks dropped and a new grace period, then you 
> should 
> > 
> > > run
> > > > > "systemctl restart nfs-server", or your distro's equivalent.
> > > > > 
> > > > > But you're probably doing something more complicated than that. 
> I'm 
> > 
> > > not
> > > > > sure I understand the question....
> > > > > 
> > > > > --b.
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > > 
> > > 
> > > 
> > 
> > 
> > 
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 
>