Re: [PATCH 2/3] NFSD: Do not fence the client on NFS4ERR_RETRY_UNCACHED_REP error

[Jeff Layton <jlayton@kernel.org>]

On Mon, 2025-11-03 at 10:50 -0800, Dai Ngo wrote:
> On 11/3/25 6:16 AM, Chuck Lever wrote:
> > On 11/3/25 6:45 AM, Christoph Hellwig wrote:
> > > On Sat, Nov 01, 2025 at 11:51:34AM -0700, Dai Ngo wrote:
> > > > NFS4ERR_RETRY_UNCACHED_REP error means client has seen and replied
> > > > to the layout recall, no fencing is needed.
> > > RFC 5661 specifies that error as:
> > > 
> > >    The requester has attempted a retry of a Compound that it previously
> > >    requested not be placed in the reply cache.
> > > 
> > > which to me doesn't imply a positive action here.
> > Agreed, this status code seems like a loss of synchronization of session
> > state between the client and server, or an implementation bug. Ie, it
> > seems to mean that at the very least, session re-negotiation is needed,
> > at first blush. Should the server mark a callback channel FAULT, for
> > instance?
> > 
> > 
> > > But I'm not an
> > > expert at reply cache semantics, so I'll leave others to correct me.
> > > But please add a more detailed comment and commit log as this is
> > > completely unintuitive.
> > The session state and the state of the layout are at two different
> > and separate layers. Connect the dots to show that not fencing is
> > the correct action and will result in recovery of full backchannel
> > operation while maintaining the integrity of the file's content.
> > 
> > So IMHO reviewers need this patch description to provide:
> > 
> > - How this came up during your testing (and maybe a small reproducer)
> > 
> > - An explanation of why leaving the client unfenced is appropriate
> > 
> > - A discussion of what will happen when the server subsequently sends
> >    another operation on this session slot
> 
> Here is the sequence of events that leads to NFS4ERR_RETRY_UNCACHED_REP:
> 
> 1. Server sends CB_LAYOUTRECALL with stateID seqid 2
> 2. Client replies NFS4ERR_NOMATCHING_LAYOUT
> 3. Server does not receive the reply due to hard hang - no server thread
>     available to service the reply (I will post a fix for this problem)
> 4. Server RPC times out waiting for the reply, nfsd4_cb_sequence_done
>     is called with cb_seq_status == 1, nfsd4_mark_cb_fault is called
>     and the request is re-queued.
> 5. Client receives the same CB_LAYOUTRECALL with stateID seqid 2
>     again and this time client replies with NFS4ERR_RETRY_UNCACHED_REP.
> 
> This process repeats forever from step 4.
> 

I'm a little confused here. I could see that you might not be able to
process a LAYOUTRETURN if all nfsd threads were blocked waiting for the
break_layout(), but I don't get why that would blocks a CB_LAYOUTRECALL
reply.

For the server, CB_LAYOUTRECALL is a client RPC (server acts as client
and vice versa). A CB_LAYOUTRECALL shouldn't depend on having a nfsd
thread available, since it runs in the context of a workqueue thread.

What am I missing?

-- 
Jeff Layton <jlayton@kernel.org>