Re: SOFT + NO_RETRANS_TIMEOUT semantics

[Trond Myklebust <trondmy@hammerspace.com>]

On Mon, 2021-07-12 at 17:48 +0000, Chuck Lever III wrote:
> 
> 
> > On Jul 12, 2021, at 1:36 PM, Trond Myklebust
> > <trondmy@hammerspace.com> wrote:
> > 
> > On Mon, 2021-07-12 at 17:07 +0000, Chuck Lever III wrote:
> > > Hi Trond-
> > > 
> > > I'm seeing some interesting client hangs that arise from a well-
> > > timed server crash or network partition.
> > > 
> > > The easiest to see is gss_destroy() on an Kerberized NFSv4 mount.
> > > 
> > > NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter
> > > I'll
> > > refer to it as NORTO) when creating a new rpc_clnt. The initial
> > > rpc_ping() for that rpc_clnt is done before the logic that sets
> > > cl_noretranstimeo, thus that ping works as expected (SOFT |
> > > SOFTCONN) and can time out properly if the server isn't
> > > responsive.
> > > 
> > > However, once that ping succeeds, cl_noretranstimeo is asserted,
> > > and all subsequent RPC requests on that rpc_clnt are with NORTO
> > > semantics.
> > > 
> > > When it comes time to destroy the GSS context for that rpc_clnt,
> > > the NULL procedure with the GSS decorations is sent with SOFT |
> > > SOFTCONN | NORTO. If the server isn't responding at that point,
> > > the client continues to retransmit the GSS context destruction
> > > request forever, and the xprt and possibly the nfs_client are
> > > pinned.
> > > 
> > > The problem also arises for lease management operations such as
> > > singleton SEQUENCE or RENEW requests. These are also done with
> > > SOFT, as I recall they need to time out properly. But with
> > > NORTO + SOFT, they will be retried until a connection loss that
> > > might never come.
> > > 
> > > I've thought of some ways to modify the cl_noretranstimeo logic
> > > such that it can be disabled for particular RPC tasks, though
> > > none is really striking me as exceptionally clever:
> > > 
> > >  - Add a field to struct rpc_procinfo that contains a mask of
> > >    RPC_TASK flags to clear for each procedure.
> > >  - Add logic to rpc_task_set_client() that clears NORTO in
> > >    some special cases.
> > >  - Reverse the meaning of NORTO (e.g., make it
> > >    RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller
> > >    for particular RPC tasks if the rpc_clnt-default behavior
> > >    is NORTO.
> > > 
> > > Any thoughts?
> > > 
> > 
> > Why would the connection not break when the server goes down?
> 
> The server can't actively RST or FIN the connection if a network
> partition occurs; and some servers might crash while their kernel
> is still alive to respond to keep-alive.
> 
> 
> > Aren't
> > the TCP_USER_TIMEOUT or the TCP_KEEPALIVE kicking in as they
> > should?
> 
> I don't see them kicking in, but I let the test run only for about
> 12 minutes. 
> 

TCP_USER_TIMEOUT should kick in any time when the server is failing to
read the socket contents, and should close the connection.

A more likely scenario is that the server is actually reading the
socket, but is just dropping the requests on the floor. I agree that
needs to be handled correctly.
One way to do that could be to add a flag that says "don't apply any
other default task flags" for special cases like this one?

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com