From: James Pearson <jcpearson@gmail.com>
To: Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: NFSv4.0: client stuck looping on RENEW + NFSERR_STALE_CLIENTID
Date: Thu, 2 Jul 2020 11:28:18 +0100 [thread overview]
Message-ID: <CAK3fRr_GR3bY5d6GC-cQYGN4UTR0nmkCyJa9AOgmSbxSxYWYEw@mail.gmail.com> (raw)
In-Reply-To: <CAK3fRr-cV_0c-sHhA2rMswrHEcLrkNfHs-=ejHRS8-j8zzObhA@mail.gmail.com>
On Wed, 24 Jun 2020 at 09:54, James Pearson <jcpearson@gmail.com> wrote:
>
> On Wed, 10 Jun 2020 at 17:24, Robert Milkowski <rmilkowski@gmail.com> wrote:
> >
> > I was hoping someone here might get back with "hey, this has been fixed by
> > commit...".
> > We also did see it on centos 7.6
> >
> > I will try to get it re-produce it and once I can then I'll try to reproduce
> > it against upstream.
>
> I haven't been monitoring this list recently - but we are still having
> the same problem with CentOS 7.7 clients and Isilon filers
>
> I've just 'fixed' one client with the issue in the way you described
> earlier in this thread by looking for a process in nfs4_state_manager
> via a stack trace - killing that pid with -9 and all the stuck mount
> pops back to life
I have tcpdumps of when this RENEW/NFSERR_STALE_CLIENTID loop starts
taken on different clients - and all show a similar behaviour - from
one of these tcpdumps:
27803 2020-06-23 13:40:36 10.78.202.217 -> 10.78.196.220 NFS 194 V4
Call RENEW CID: 0x3091
27804 2020-06-23 13:40:36 10.78.196.220 -> 10.78.202.217 NFS 114 V4
Reply (Call In 27803) RENEW
27815 2020-06-23 13:41:06 10.78.202.217 -> 10.78.196.220 NFS 194 V4
Call RENEW CID: 0x3091
27816 2020-06-23 13:41:06 10.78.196.220 -> 10.78.202.217 NFS 114 V4
Reply (Call In 27815) RENEW
27827 2020-06-23 13:41:36 10.78.202.217 -> 10.78.196.220 NFS 194 V4
Call RENEW CID: 0x3091
27828 2020-06-23 13:41:36 10.78.196.220 -> 10.78.202.217 NFS 114 V4
Reply (Call In 27827) RENEW
27856 2020-06-23 13:42:06 10.78.202.217 -> 10.78.196.220 NFS 194 V4
Call RENEW CID: 0x3091
27857 2020-06-23 13:42:06 10.78.196.220 -> 10.78.202.217 NFS 114 V4
Reply (Call In 27856) RENEW
28071 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 110 V4
NULL Call[Malformed Packet]
28073 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 94 V4
NULL Reply (Call In 28071)[Malformed Packet]
28082 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 110 V4
NULL Call[Malformed Packet]
28084 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 94 V4
NULL Reply (Call In 28082)[Malformed Packet]
28086 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 370 V4
Call EXCHANGE_ID
28087 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 106 V4
Reply (Call In 28086) Status: NFS4ERR_MINOR_VERS_MISMATCH
28092 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 206 V4
Call PUTROOTFH | GETATTR
28093 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 342 V4
Reply (Call In 28092) PUTROOTFH | GETATTR
28095 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 258 V4
Call GETATTR FH: 0x2e47f02d
28096 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 162 V4
Reply (Call In 28095) GETATTR
28097 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 262 V4
Call GETATTR FH: 0x2e47f02d
28098 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 178 V4
Reply (Call In 28097) GETATTR
28099 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 258 V4
Call GETATTR FH: 0x2e47f02d
28100 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 162 V4
Reply (Call In 28099) GETATTR
...
39660 2020-06-23 13:42:53 10.78.202.217 -> 10.78.196.220 NFS 346 V4
Call LOOKUP DH: 0x19cf3646/shared
39661 2020-06-23 13:42:53 10.78.196.220 -> 10.78.202.217 NFS 342 V4
Reply (Call In 39660) LOOKUP
39663 2020-06-23 13:42:53 10.78.202.217 -> 10.78.196.220 NFS 314 V4
Call READLINK
39665 2020-06-23 13:42:53 10.78.196.220 -> 10.78.202.217 NFS 162 V4
Reply (Call In 39663) READLINK
44182 2020-06-23 13:42:56 10.78.202.217 -> 10.78.196.220 NFS 326 V4
Call GETATTR FH: 0xff288ce9
44183 2020-06-23 13:42:56 10.78.196.220 -> 10.78.202.217 NFS 266 V4
Reply (Call In 44182) GETATTR
68152 2020-06-23 13:42:59 10.78.202.217 -> 10.78.196.220 NFS 194 V4
Call RENEW CID: 0x3091
68153 2020-06-23 13:42:59 10.78.196.220 -> 10.78.202.217 NFS 114 V4
Reply (Call In 68152) RENEW Status: NFS4ERR_STALE_CLIENTID
68155 2020-06-23 13:42:59 10.78.202.217 -> 10.78.196.220 NFS 194 V4
Call RENEW CID: 0x3091
68156 2020-06-23 13:42:59 10.78.196.220 -> 10.78.202.217 NFS 114 V4
Reply (Call In 68155) RENEW Status: NFS4ERR_STALE_CLIENTID
73012 2020-06-23 13:43:04 10.78.202.217 -> 10.78.196.220 NFS 194 V4
Call RENEW CID: 0x3091
73013 2020-06-23 13:43:04 10.78.196.220 -> 10.78.202.217 NFS 114 V4
Reply (Call In 73012) RENEW Status: NFS4ERR_STALE_CLIENTID
73270 2020-06-23 13:43:09 10.78.202.217 -> 10.78.196.220 NFS 194 V4
Call RENEW CID: 0x3091
73271 2020-06-23 13:43:09 10.78.196.220 -> 10.78.202.217 NFS 114 V4
Reply (Call In 73270) RENEW Status: NFS4ERR_STALE_CLIENTID
73273 2020-06-23 13:43:14 10.78.202.217 -> 10.78.196.220 NFS 194 V4
Call RENEW CID: 0x3091
73274 2020-06-23 13:43:14 10.78.196.220 -> 10.78.202.217 NFS 114 V4
Reply (Call In 73273) RENEW Status: NFS4ERR_STALE_CLIENTID
73276 2020-06-23 13:43:19 10.78.202.217 -> 10.78.196.220 NFS 194 V4
Call RENEW CID: 0x3091
73277 2020-06-23 13:43:19 10.78.196.220 -> 10.78.202.217 NFS 114 V4
Reply (Call In 73276) RENEW Status: NFS4ERR_STALE_CLIENTID
73279 2020-06-23 13:43:24 10.78.202.217 -> 10.78.196.220 NFS 194 V4
Call RENEW CID: 0x3091
73280 2020-06-23 13:43:24 10.78.196.220 -> 10.78.202.217 NFS 114 V4
Reply (Call In 73279) RENEW Status: NFS4ERR_STALE_CLIENTID
i.e. all clients have:
28071 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 110 V4
NULL Call[Malformed Packet]
28073 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 94 V4
NULL Reply (Call In 28071)[Malformed Packet]
28082 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 110 V4
NULL Call[Malformed Packet]
28084 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 94 V4
NULL Reply (Call In 28082)[Malformed Packet]
28086 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 370 V4
Call EXCHANGE_ID
28087 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 106 V4
Reply (Call In 28086) Status: NFS4ERR_MINOR_VERS_MISMATCH
then 30 seconds later, the RENEW/NFSERR_STALE_CLIENTID loop starts:
68152 2020-06-23 13:42:59 10.78.202.217 -> 10.78.196.220 NFS 194 V4
Call RENEW CID: 0x3091
68153 2020-06-23 13:42:59 10.78.196.220 -> 10.78.202.217 NFS 114 V4
Reply (Call In 68152) RENEW Status: NFS4ERR_STALE_CLIENTID
68155 2020-06-23 13:42:59 10.78.202.217 -> 10.78.196.220 NFS 194 V4
Call RENEW CID: 0x3091
68156 2020-06-23 13:42:59 10.78.196.220 -> 10.78.202.217 NFS 114 V4
Reply (Call In 68155) RENEW Status: NFS4ERR_STALE_CLIENTID
Does anyone have any idea if the above might point to something that
has already been fixed in the mainline kernel?
Thanks
James Pearson
prev parent reply other threads:[~2020-07-02 10:28 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-08 12:34 NFSv4.0: client stuck looping on RENEW + NFSERR_STALE_CLIENTID Robert Milkowski
2020-06-10 16:19 ` Chuck Lever
2020-06-10 16:24 ` Robert Milkowski
2020-06-24 8:54 ` James Pearson
2020-07-02 10:28 ` James Pearson [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAK3fRr_GR3bY5d6GC-cQYGN4UTR0nmkCyJa9AOgmSbxSxYWYEw@mail.gmail.com \
--to=jcpearson@gmail.com \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).