Question on RPC_TASK_NO_RETRANS_TIMEOUT / NFS_CS_NO_RETRANS_TIMEOUT for NFSv3

Question on RPC_TASK_NO_RETRANS_TIMEOUT / NFS_CS_NO_RETRANS_TIMEOUT for NFSv3

[Pedro Principeza]

Hi, there!

We have some shares that use NFSv3 with TCP and Kerberos and have been
hitting an intriguing issue with those. We have noticed that network
instabilities have been causing some 'Permission denied' errors on
files.

The current scenario we have is based on NFSv3 over TCP clients,
configured with Kerberos (krb5p) authentication against a NetApp NFS
Server (ONTAP).  This is happening regardless of the Kernel we use
(our main tests bear 4.15 and 5.15 generic Ubuntu Kernels - from
Bionic and Jammy), and we have not found any interesting commits in
either components upstream that would change the behaviour in hand.

We tracked those issues down and found out that the 'Permission
denied' happens because our packets are failing the GSS checksum[1].
We kept investigating and discovered, after some tcpdump, that this
happens because the client retransmits RPC packets, which increases
the GSS sequence number. Meanwhile, the response to the original
packet gets received, but the checksum fails because the client is
expecting a different GSS sequence number.

This can be avoided with NFSv4 because the RPC client is created with
a "no retrans timeout" flag[2]. Such a flag is not set and is
impossible to set on NFSv3. We did some investigation and thought that
setting this flag would fix our problems without the need to move to
NFSv4.

Our question is: is there a reason this flag is not being set nor is
it possible to set it for NFSv3? Is there something on NFSv3 that
demands RPC retransmissions even with TCP?  One "hint" we have come
across is that it is *explicitly mentioned* in NFSv4's RFC [3], and
there is nothing in NFSv3 at all - most likely due to the fact we're
dealing with a stateless protocol.

Any comments would be greatly appreciated here!

Thank you,

[1] https://github.com/torvalds/linux/blob/v5.15/net/sunrpc/auth_gss/gss_krb5_unseal.c#L194
[2] https://github.com/torvalds/linux/blob/v5.15/fs/nfs/client.c#L521
[3] https://datatracker.ietf.org/doc/html/rfc7530#section-3.1.1

--
Pedro Principeza
Senior Sustaining Operations Engineer

Re: Question on RPC_TASK_NO_RETRANS_TIMEOUT / NFS_CS_NO_RETRANS_TIMEOUT for NFSv3

[Trond Myklebust]

On Wed, 2023-08-23 at 16:59 -0300, Pedro Principeza wrote:
> [You don't often get email from pedro.principeza@canonical.com. Learn
> why this is important at
> https://aka.ms/LearnAboutSenderIdentification ]
> 
> Hi, there!
> 
> We have some shares that use NFSv3 with TCP and Kerberos and have
> been
> hitting an intriguing issue with those. We have noticed that network
> instabilities have been causing some 'Permission denied' errors on
> files.
> 
> The current scenario we have is based on NFSv3 over TCP clients,
> configured with Kerberos (krb5p) authentication against a NetApp NFS
> Server (ONTAP).  This is happening regardless of the Kernel we use
> (our main tests bear 4.15 and 5.15 generic Ubuntu Kernels - from
> Bionic and Jammy), and we have not found any interesting commits in
> either components upstream that would change the behaviour in hand.
> 
> We tracked those issues down and found out that the 'Permission
> denied' happens because our packets are failing the GSS checksum[1].
> We kept investigating and discovered, after some tcpdump, that this
> happens because the client retransmits RPC packets, which increases
> the GSS sequence number. Meanwhile, the response to the original
> packet gets received, but the checksum fails because the client is
> expecting a different GSS sequence number.
> 
> This can be avoided with NFSv4 because the RPC client is created with
> a "no retrans timeout" flag[2]. Such a flag is not set and is
> impossible to set on NFSv3. We did some investigation and thought
> that
> setting this flag would fix our problems without the need to move to
> NFSv4.
> 
> Our question is: is there a reason this flag is not being set nor is
> it possible to set it for NFSv3? Is there something on NFSv3 that
> demands RPC retransmissions even with TCP?  One "hint" we have come
> across is that it is *explicitly mentioned* in NFSv4's RFC [3], and
> there is nothing in NFSv3 at all - most likely due to the fact we're
> dealing with a stateless protocol.
> 
> Any comments would be greatly appreciated here!
> 
> Thank you,
> 
> [1]
> https://github.com/torvalds/linux/blob/v5.15/net/sunrpc/auth_gss/gss_krb5_unseal.c#L194
> [2] https://github.com/torvalds/linux/blob/v5.15/fs/nfs/client.c#L521
> [3] https://datatracker.ietf.org/doc/html/rfc7530#section-3.1.1

NFSv3 servers are allowed to drop requests, and NFSv3 clients are
expected to retransmit them when this happens. NFSv4 servers may not
drop requests, and NFSv4 clients are expected never to retransmit
(unless the connection breaks). For that reason we do set
RPC_TASK_NO_RETRANS_TIMEOUT on NFSv4 and do not on NFSv3.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com

Re: Question on RPC_TASK_NO_RETRANS_TIMEOUT / NFS_CS_NO_RETRANS_TIMEOUT for NFSv3

[Russell Cattelan]

> > Hi, there!
> > 
> > We have some shares that use NFSv3 with TCP and Kerberos and have
> > been
> > hitting an intriguing issue with those. We have noticed that network
> > instabilities have been causing some 'Permission denied' errors on
> > files.
> > 
> > The current scenario we have is based on NFSv3 over TCP clients,
> > configured with Kerberos (krb5p) authentication against a NetApp NFS
> > Server (ONTAP).  This is happening regardless of the Kernel we use
> > (our main tests bear 4.15 and 5.15 generic Ubuntu Kernels - from
> > Bionic and Jammy), and we have not found any interesting commits in
> > either components upstream that would change the behaviour in hand.
> > 
> > We tracked those issues down and found out that the 'Permission
> > denied' happens because our packets are failing the GSS checksum.
> > We kept investigating and discovered, after some tcpdump, that this
> > happens because the client retransmits RPC packets, which increases
> > the GSS sequence number. Meanwhile, the response to the original
> > packet gets received, but the checksum fails because the client is
> > expecting a different GSS sequence number.
> > 
> > This can be avoided with NFSv4 because the RPC client is created with
> > a "no retrans timeout" flag. Such a flag is not set and is
> > impossible to set on NFSv3. We did some investigation and thought
> > that
> > setting this flag would fix our problems without the need to move to
> > NFSv4.
> > 
> > Our question is: is there a reason this flag is not being set nor is
> > it possible to set it for NFSv3? Is there something on NFSv3 that
> > demands RPC retransmissions even with TCP?  One "hint" we have come
> > across is that it is *explicitly mentioned* in NFSv4's RFC, and
> > there is nothing in NFSv3 at all - most likely due to the fact we're
> > dealing with a stateless protocol.
> > 
> > Any comments would be greatly appreciated here!
> > 
> > Thank you,
> > 
> > [1]
> > https://github.com/torvalds/linux/blob/v5.15/net/sunrpc/auth_gss/gss_krb5_unseal.c#L194
> > [2] https://github.com/torvalds/linux/blob/v5.15/fs/nfs/client.c#L521
> > [3] https://datatracker.ietf.org/doc/html/rfc7530#section-3.1.1
>
> NFSv3 servers are allowed to drop requests, and NFSv3 clients are
> expected to retransmit them when this happens. NFSv4 servers may not
> drop requests, and NFSv4 clients are expected never to retransmit
> (unless the connection breaks). For that reason we do set
> RPC_TASK_NO_RETRANS_TIMEOUT on NFSv4 and do not on NFSv3.
>
We have been doing a bunch of debugging on this issue and the key point / problem we are
running into is that because this is a kerberos enabled mount when the client does a
re-transmit it ends up generating a new MIC header / checksum since the krb5 context
sequence number has moved on.

If that retrans happens before the original response is received then the mic verification
fails since the client is now expecting a response to the second packet and not the first.
mic header verification fails which then results in an EACCES error which ends up as an IO
error at the application.

What we have found that is it easy to repro in our environment adding an iptables
rule to drop responses from the nfs server for 55-63 seconds.
Less than 55 sec and the retrans does not happen things recover
More than 63 sec and the rpc code goes down the reconnect path before doing the retrans and
things recover.

It seems like kerberos enabled mounts should be using RPC_TASK_NO_RETRANS_TIMEOUT since doing
a retrans changes the GSS checksum from the original checksum.


--Russell Cattelan

Re: Question on RPC_TASK_NO_RETRANS_TIMEOUT / NFS_CS_NO_RETRANS_TIMEOUT for NFSv3

[Trond Myklebust]

On Fri, 2023-08-25 at 11:12 -0500, Russell Cattelan wrote:
> [You don't often get email from cattelan@thebarn.com. Learn why this
> is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> > > Hi, there!
> > > 
> > > We have some shares that use NFSv3 with TCP and Kerberos and have
> > > been
> > > hitting an intriguing issue with those. We have noticed that
> > > network
> > > instabilities have been causing some 'Permission denied' errors
> > > on
> > > files.
> > > 
> > > The current scenario we have is based on NFSv3 over TCP clients,
> > > configured with Kerberos (krb5p) authentication against a NetApp
> > > NFS
> > > Server (ONTAP).  This is happening regardless of the Kernel we
> > > use
> > > (our main tests bear 4.15 and 5.15 generic Ubuntu Kernels - from
> > > Bionic and Jammy), and we have not found any interesting commits
> > > in
> > > either components upstream that would change the behaviour in
> > > hand.
> > > 
> > > We tracked those issues down and found out that the 'Permission
> > > denied' happens because our packets are failing the GSS checksum.
> > > We kept investigating and discovered, after some tcpdump, that
> > > this
> > > happens because the client retransmits RPC packets, which
> > > increases
> > > the GSS sequence number. Meanwhile, the response to the original
> > > packet gets received, but the checksum fails because the client
> > > is
> > > expecting a different GSS sequence number.
> > > 
> > > This can be avoided with NFSv4 because the RPC client is created
> > > with
> > > a "no retrans timeout" flag. Such a flag is not set and is
> > > impossible to set on NFSv3. We did some investigation and thought
> > > that
> > > setting this flag would fix our problems without the need to move
> > > to
> > > NFSv4.
> > > 
> > > Our question is: is there a reason this flag is not being set nor
> > > is
> > > it possible to set it for NFSv3? Is there something on NFSv3 that
> > > demands RPC retransmissions even with TCP?  One "hint" we have
> > > come
> > > across is that it is *explicitly mentioned* in NFSv4's RFC, and
> > > there is nothing in NFSv3 at all - most likely due to the fact
> > > we're
> > > dealing with a stateless protocol.
> > > 
> > > Any comments would be greatly appreciated here!
> > > 
> > > Thank you,
> > > 
> > > [1]
> > > https://github.com/torvalds/linux/blob/v5.15/net/sunrpc/auth_gss/gss_krb5_unseal.c#L194
> > > [2]
> > > https://github.com/torvalds/linux/blob/v5.15/fs/nfs/client.c#L521
> > > [3] https://datatracker.ietf.org/doc/html/rfc7530#section-3.1.1
> > 
> > NFSv3 servers are allowed to drop requests, and NFSv3 clients are
> > expected to retransmit them when this happens. NFSv4 servers may
> > not
> > drop requests, and NFSv4 clients are expected never to retransmit
> > (unless the connection breaks). For that reason we do set
> > RPC_TASK_NO_RETRANS_TIMEOUT on NFSv4 and do not on NFSv3.
> > 
> We have been doing a bunch of debugging on this issue and the key
> point / problem we are
> running into is that because this is a kerberos enabled mount when
> the client does a
> re-transmit it ends up generating a new MIC header / checksum since
> the krb5 context
> sequence number has moved on.
> 
> If that retrans happens before the original response is received then
> the mic verification
> fails since the client is now expecting a response to the second
> packet and not the first.
> mic header verification fails which then results in an EACCES error
> which ends up as an IO
> error at the application.
> 
> What we have found that is it easy to repro in our environment adding
> an iptables
> rule to drop responses from the nfs server for 55-63 seconds.
> Less than 55 sec and the retrans does not happen things recover
> More than 63 sec and the rpc code goes down the reconnect path before
> doing the retrans and
> things recover.
> 
> It seems like kerberos enabled mounts should be using
> RPC_TASK_NO_RETRANS_TIMEOUT since doing
> a retrans changes the GSS checksum from the original checksum.
> 

No, that is not an option. NFSv3 servers are allowed to drop any
incoming RPC request without needing a reason, so turning on
RPC_TASK_NO_RETRANS_TIMEOUT would just lead to client hangs.

The right thing to do is to just fix up rpc_decode_header() to retry
instead of firing off an error in this case.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com

Re: Question on RPC_TASK_NO_RETRANS_TIMEOUT / NFS_CS_NO_RETRANS_TIMEOUT for NFSv3

[Russell Cattelan]

On 8/25/23 2:49 PM, Trond Myklebust wrote:

>>> NFSv3 servers are allowed to drop requests, and NFSv3 clients are
>>> expected to retransmit them when this happens. NFSv4 servers may
>>> not
>>> drop requests, and NFSv4 clients are expected never to retransmit
>>> (unless the connection breaks). For that reason we do set
>>> RPC_TASK_NO_RETRANS_TIMEOUT on NFSv4 and do not on NFSv3.
>>>
>> We have been doing a bunch of debugging on this issue and the key
>> point / problem we are
>> running into is that because this is a kerberos enabled mount when
>> the client does a
>> re-transmit it ends up generating a new MIC header / checksum since
>> the krb5 context
>> sequence number has moved on.
>>
>> If that retrans happens before the original response is received then
>> the mic verification
>> fails since the client is now expecting a response to the second
>> packet and not the first.
>> mic header verification fails which then results in an EACCES error
>> which ends up as an IO
>> error at the application.
>>
>> What we have found that is it easy to repro in our environment adding
>> an iptables
>> rule to drop responses from the nfs server for 55-63 seconds.
>> Less than 55 sec and the retrans does not happen things recover
>> More than 63 sec and the rpc code goes down the reconnect path before
>> doing the retrans and
>> things recover.
>>
>> It seems like kerberos enabled mounts should be using
>> RPC_TASK_NO_RETRANS_TIMEOUT since doing
>> a retrans changes the GSS checksum from the original checksum.
>>
> 
> No, that is not an option. NFSv3 servers are allowed to drop any
> incoming RPC request without needing a reason, so turning on
> RPC_TASK_NO_RETRANS_TIMEOUT would just lead to client hangs.
I can see that for UDP but is that true for TCP as well?
Wouldn't the rpc code behave the same as v4 and setup a new connection
before doing the retrans?
At least in our experimentation if we leave the connection down for more 
63 seconds we can see from the rpc traces that is what is happening.
Once there is a new connection then old message is ignored and 
processing continues with the new set request / responses.

> 
> The right thing to do is to just fix up rpc_decode_header() to retry
> instead of firing off an error in this case.
So you are thinking that rpc_decode_header just returns EAGAIN if the 
checksum fails?
What happens if the GSS context actually goes bad (times out etc) 
wouldn't that also result in the client get stuck just doing re-sends
over and over?

I'm really not that up to speed on subtleties of NFS kerberos.

Oh note this isn't even krb5p just krb5 mounts. (not that should matter 
all that much)

--Russell Cattelan

Re: Question on RPC_TASK_NO_RETRANS_TIMEOUT / NFS_CS_NO_RETRANS_TIMEOUT for NFSv3

[Trond Myklebust]

On Sat, 2023-08-26 at 09:32 -0500, Russell Cattelan wrote:
> [You don't often get email from cattelan@thebarn.com. Learn why this
> is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> On 8/25/23 2:49 PM, Trond Myklebust wrote:
> 
> > > > NFSv3 servers are allowed to drop requests, and NFSv3 clients
> > > > are
> > > > expected to retransmit them when this happens. NFSv4 servers
> > > > may
> > > > not
> > > > drop requests, and NFSv4 clients are expected never to
> > > > retransmit
> > > > (unless the connection breaks). For that reason we do set
> > > > RPC_TASK_NO_RETRANS_TIMEOUT on NFSv4 and do not on NFSv3.
> > > > 
> > > We have been doing a bunch of debugging on this issue and the key
> > > point / problem we are
> > > running into is that because this is a kerberos enabled mount
> > > when
> > > the client does a
> > > re-transmit it ends up generating a new MIC header / checksum
> > > since
> > > the krb5 context
> > > sequence number has moved on.
> > > 
> > > If that retrans happens before the original response is received
> > > then
> > > the mic verification
> > > fails since the client is now expecting a response to the second
> > > packet and not the first.
> > > mic header verification fails which then results in an EACCES
> > > error
> > > which ends up as an IO
> > > error at the application.
> > > 
> > > What we have found that is it easy to repro in our environment
> > > adding
> > > an iptables
> > > rule to drop responses from the nfs server for 55-63 seconds.
> > > Less than 55 sec and the retrans does not happen things recover
> > > More than 63 sec and the rpc code goes down the reconnect path
> > > before
> > > doing the retrans and
> > > things recover.
> > > 
> > > It seems like kerberos enabled mounts should be using
> > > RPC_TASK_NO_RETRANS_TIMEOUT since doing
> > > a retrans changes the GSS checksum from the original checksum.
> > > 
> > 
> > No, that is not an option. NFSv3 servers are allowed to drop any
> > incoming RPC request without needing a reason, so turning on
> > RPC_TASK_NO_RETRANS_TIMEOUT would just lead to client hangs.
> I can see that for UDP but is that true for TCP as well?

Yes. For instance the Linux knfsd server will drop requests if the GSS
sequence number lies outside the window (no, I don't know why it
doesn't just return RPCSEC_GSS_CTXPROBLEM). It will also happily drop
deferred requests (i.e. requests waiting for a reply to an upcall) once
they start piling up. Finally, if the knfsd reply cache says that an
earlier transmission of the NFSv3 RPC request is still being processed,
then the new request gets dropped.

> Wouldn't the rpc code behave the same as v4 and setup a new
> connection
> before doing the retrans?
> At least in our experimentation if we leave the connection down for
> more
> 63 seconds we can see from the rpc traces that is what is happening.
> Once there is a new connection then old message is ignored and
> processing continues with the new set request / responses.
> 
> > 
> > The right thing to do is to just fix up rpc_decode_header() to
> > retry
> > instead of firing off an error in this case.
> So you are thinking that rpc_decode_header just returns EAGAIN if the
> checksum fails?
> What happens if the GSS context actually goes bad (times out etc)
> wouldn't that also result in the client get stuck just doing re-sends
> over and over?

If the GSS context goes bad, then the server is supposed to return
either RPCSEC_GSS_CREDPROBLEM (if the server no longer has context for
that handle) or RPCSEC_GSS_CTXPROBLEM (context is stale due to ticket
expiry, etc).

If it just times out, then surely the replay cache should ensure that
it gets processed quickly after a retry, no?

> 
> I'm really not that up to speed on subtleties of NFS kerberos.
> 
> Oh note this isn't even krb5p just krb5 mounts. (not that should
> matter
> all that much)
> 
> --Russell Cattelan

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com

Re: Question on RPC_TASK_NO_RETRANS_TIMEOUT / NFS_CS_NO_RETRANS_TIMEOUT for NFSv3

[Russell Cattelan]

On 8/26/23 10:30 AM, Trond Myklebust wrote:

> Yes. For instance the Linux knfsd server will drop requests if the GSS
> sequence number lies outside the window (no, I don't know why it
> doesn't just return RPCSEC_GSS_CTXPROBLEM). It will also happily drop
> deferred requests (i.e. requests waiting for a reply to an upcall) once
> they start piling up. Finally, if the knfsd reply cache says that an
> earlier transmission of the NFSv3 RPC request is still being processed,
> then the new request gets dropped.
> 
>> Wouldn't the rpc code behave the same as v4 and setup a new
>> connection
>> before doing the retrans?
>> At least in our experimentation if we leave the connection down for
>> more
>> 63 seconds we can see from the rpc traces that is what is happening.
>> Once there is a new connection then old message is ignored and
>> processing continues with the new set request / responses.
>>
>>>
>>> The right thing to do is to just fix up rpc_decode_header() to
>>> retry
>>> instead of firing off an error in this case.
>> So you are thinking that rpc_decode_header just returns EAGAIN if the
>> checksum fails?
>> What happens if the GSS context actually goes bad (times out etc)
>> wouldn't that also result in the client get stuck just doing re-sends
>> over and over?
> 
> If the GSS context goes bad, then the server is supposed to return
> either RPCSEC_GSS_CREDPROBLEM (if the server no longer has context for
> that handle) or RPCSEC_GSS_CTXPROBLEM (context is stale due to ticket
> expiry, etc).

Test environment got reset so took a day or so try this.

I change the return when rpcauth_checkverf fails to be a EKEYREJECTED 
error vs EAGAIN

Which puts the decode failure down this path vs just re-transmitting the 
same XID but with a new GSS sequence / checksum.

	case -EKEYREJECTED:
		task->tk_action = call_reserve;
		rpc_check_timeout(task);
		rpcauth_invalcred(task);
		/* Ensure we obtain a new XID if we retry! */
		xprt_release(task);

This does appear to work / address the failure that we are able
to introduce with iptable down ; iptables up.

But question is this a valid thing to do when the gss checksum fails?


diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index ad3e9a40b061..d0bcb6c6b3df 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -2645,7 +2645,8 @@ rpc_decode_header(struct rpc_task *task, struct 
xdr_stream *xdr)

  out_verifier:
  	trace_rpc_bad_verifier(task);
-	goto out_garbage;
+	return -EKEYREJECTED;
+	//goto out_garbage;

  out_msg_denied:
  	error = -EACCES;



> 
> If it just times out, then surely the replay cache should ensure that
> it gets processed quickly after a retry, no?
> 
>>
>> I'm really not that up to speed on subtleties of NFS kerberos.
>>
>> Oh note this isn't even krb5p just krb5 mounts. (not that should
>> matter
>> all that much)
>>
>> --Russell Cattelan
>