Re: Spurious instability with NFSoRDMA under moderate load

[Chuck Lever III <chuck.lever@oracle.com>]

> On Aug 11, 2021, at 2:38 PM, Olga Kornievskaia <aglo@umich.edu> wrote:
> 
> On Wed, Aug 11, 2021 at 1:30 PM Chuck Lever III <chuck.lever@oracle.com> wrote:
>> 
>> 
>> 
>>> On Aug 11, 2021, at 12:20 PM, Timo Rothenpieler <timo@rothenpieler.org> wrote:
>>> 
>>> resulting dmesg and trace logs of both client and server are attached.
>>> 
>>> Test procedure:
>>> 
>>> - start tracing on client and server
>>> - mount NFS on client
>>> - immediately run 'xfs_io -fc "copy_range testfile" testfile.copy' (which succeeds)
>>> - wait 10~15 minutes for the backchannel to time out (still running 5.12.19 with the fix for that reverted)
>>> - run xfs_io command again, getting stuck now
>>> - let it sit there stuck for a minute, then cancel it
>>> - run the command again
>>> - while it's still stuck, finished recording the logs and traces
>> 
>> The server tries to send CB_OFFLOAD when the offloaded copy
>> completes, but finds the backchannel transport is not connected.
>> 
>> The server can't report the problem until the client sends a
>> SEQUENCE operation, but there's really no other traffic going
>> on, so it just waits.
>> 
>> The client eventually sends a singleton SEQUENCE to renew its
>> lease. The server replies with the SEQ4_STATUS_BACKCHANNEL_FAULT
>> flag set at that point. Client's recovery is to destroy that
>> session and create a new one. That appears to be successful.
>> 
>> But the server doesn't send another CB_OFFLOAD to let the client
>> know the copy is complete, so the client hangs.
>> 
>> This seems to be peculiar to COPY_OFFLOAD, but I wonder if the
>> other CB operations suffer from the same "failed to retransmit
>> after the CB path is restored" issue. It might not matter for
>> some of them, but for others like CB_RECALL, that could be
>> important.
> 
> Thank you for the analysis Chuck (btw I haven't seen any attachments
> with Timo's posts so I'm assuming some offline communication must have
> happened).
> ?
> I'm looking at the code and wouldn't the mentioned flags be set on the
> CB_SEQUENCE operation?

CB_SEQUENCE is sent from server to client, and that can't work if
the callback channel is down.

So the server waits for the client to send a SEQUENCE and it sets
the SEQ4_STATUS_BACKCHANNEL_FAULT in its reply.


> nfsd4_cb_done() has code to mark the channel
> and retry (or another way of saying this, this code should generically
> handle retrying whatever operation it is be it CB_OFFLOAD or
> CB_RECALL)?

cb_done() marks the callback fault, but as far as I can tell the
RPC is terminated at that point and there is no subsequent retry.
The RPC_TASK flags on the CB_OFFLOAD operation cause that RPC to
fail immediately if there's no connection.

And in the BACKCHANNEL_FAULT case, the bc_xprt is destroyed as
part of recovery. I think that would kill all pending RPC tasks.


> Is that not working (not sure if this is  a question or a
> statement).... I would think that would be the place to handle this
> problem.

IMHO the OFFLOAD code needs to note that the CB_OFFLOAD RPC
failed and then try the call again once the new backchannel is
available.


--
Chuck Lever