Re: SOFT + NO_RETRANS_TIMEOUT semantics

[Trond Myklebust <trondmy@hammerspace.com>]

On Mon, 2021-07-12 at 17:07 +0000, Chuck Lever III wrote:
> Hi Trond-
> 
> I'm seeing some interesting client hangs that arise from a well-
> timed server crash or network partition.
> 
> The easiest to see is gss_destroy() on an Kerberized NFSv4 mount.
> 
> NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter I'll
> refer to it as NORTO) when creating a new rpc_clnt. The initial
> rpc_ping() for that rpc_clnt is done before the logic that sets
> cl_noretranstimeo, thus that ping works as expected (SOFT |
> SOFTCONN) and can time out properly if the server isn't
> responsive.
> 
> However, once that ping succeeds, cl_noretranstimeo is asserted,
> and all subsequent RPC requests on that rpc_clnt are with NORTO
> semantics.
> 
> When it comes time to destroy the GSS context for that rpc_clnt,
> the NULL procedure with the GSS decorations is sent with SOFT |
> SOFTCONN | NORTO. If the server isn't responding at that point,
> the client continues to retransmit the GSS context destruction
> request forever, and the xprt and possibly the nfs_client are
> pinned.
> 
> The problem also arises for lease management operations such as
> singleton SEQUENCE or RENEW requests. These are also done with
> SOFT, as I recall they need to time out properly. But with
> NORTO + SOFT, they will be retried until a connection loss that
> might never come.
> 
> I've thought of some ways to modify the cl_noretranstimeo logic
> such that it can be disabled for particular RPC tasks, though
> none is really striking me as exceptionally clever:
> 
>  - Add a field to struct rpc_procinfo that contains a mask of
>    RPC_TASK flags to clear for each procedure.
>  - Add logic to rpc_task_set_client() that clears NORTO in
>    some special cases.
>  - Reverse the meaning of NORTO (e.g., make it
>    RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller
>    for particular RPC tasks if the rpc_clnt-default behavior
>    is NORTO.
> 
> Any thoughts?
> 

Why would the connection not break when the server goes down? Aren't
the TCP_USER_TIMEOUT or the TCP_KEEPALIVE kicking in as they should?

Is this an RDMA problem?

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com