Re: [PATCH 2/2] NFSD: Fix server hang when there are multiple layout conflicts

[Jeff Layton <jlayton@kernel.org>]

On Thu, 2025-11-06 at 09:17 -0800, Dai Ngo wrote:
> On 11/6/25 9:14 AM, Jeff Layton wrote:
> > On Thu, 2025-11-06 at 08:47 -0800, Dai Ngo wrote:
> > > When a layout conflict triggers a call to __break_lease, the function
> > > nfsd4_layout_lm_break clears the fl_break_time timeout before sending
> > > the CB_LAYOUTRECALL. As a result, __break_lease repeatedly restarts
> > > its loop, waiting indefinitely for the conflicting file lease to be
> > > released.
> > > 
> > > If the number of lease conflicts matches the number of NFSD threads
> > > (which defaults to 8), all available NFSD threads become occupied.
> > > Consequently, there are no threads left to handle incoming requests
> > > or callback replies, leading to a total hang of the NFS server.
> > > 
> > > This issue is reliably reproducible by running the Git test suite
> > > on a configuration using SCSI layout.
> > > 
> > > This patch addresses the problem by using the break lease timeout
> > > and ensures that the unresponsive client is fenced, preventing it from
> > > accessing the data server directly.
> > > 
> > > Fixes: f99d4fbdae67 ("nfsd: add SCSI layout support")
> > > Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
> > > ---
> > >   fs/nfsd/nfs4layouts.c | 25 +++++++++++++++++++++----
> > >   1 file changed, 21 insertions(+), 4 deletions(-)
> > > 
> > > diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
> > > index 683bd1130afe..b9b1eb32624c 100644
> > > --- a/fs/nfsd/nfs4layouts.c
> > > +++ b/fs/nfsd/nfs4layouts.c
> > > @@ -747,11 +747,10 @@ static bool
> > >   nfsd4_layout_lm_break(struct file_lease *fl)
> > >   {
> > >   	/*
> > > -	 * We don't want the locks code to timeout the lease for us;
> > > -	 * we'll remove it ourself if a layout isn't returned
> > > -	 * in time:
> > > +	 * Enforce break lease timeout to prevent starvation of
> > > +	 * NFSD threads in __break_lease that causes server to
> > > +	 * hang.
> > >   	 */
> > > -	fl->fl_break_time = 0;
> > I guess this ends up with whatever the default fl_break_time is which
> > is:
> > 
> > 	jiffies + lease_break_time * HZ;
> 
> Yes, currently is 45 secs which is, I think, is way too long.
> 
> > 
> > I wonder if this should be based around some multiple of the grace
> > period instead?
> 
> I think the time to allow for recall reply should be in milliseconds.
> 
> -Dai
> 

I don't think that's at all reasonable. We'll be fencing slow machines
all over the place. Clients expect that they can be out of contact for
a little while (a lease period) and not lose their state. Fencing them
on a timeout substantially less than that will violate that
expectation.

> > 
> > >   	nfsd4_recall_file_layout(fl->c.flc_owner);
> > >   	return false;
> > >   }
> > > @@ -764,9 +763,27 @@ nfsd4_layout_lm_change(struct file_lease *onlist, int arg,
> > >   	return lease_modify(onlist, arg, dispose);
> > >   }
> > >   
> > > +static void nfsd_layout_breaker_timedout(struct file_lease *fl)
> > > +{
> > > +	struct nfs4_layout_stateid *ls = fl->c.flc_owner;
> > > +	struct nfsd_file *nf;
> > > +
> > > +	rcu_read_lock();
> > > +	nf = nfsd_file_get(ls->ls_file);
> > > +	rcu_read_unlock();
> > > +	if (nf) {
> > > +		int type = ls->ls_layout_type;
> > > +
> > > +		if (nfsd4_layout_ops[type]->fence_client)
> > > +			nfsd4_layout_ops[type]->fence_client(ls, nf);
> > > +		nfsd_file_put(nf);
> > > +	}
> > > +}
> > > +
> > >   static const struct lease_manager_operations nfsd4_layouts_lm_ops = {
> > >   	.lm_break	= nfsd4_layout_lm_break,
> > >   	.lm_change	= nfsd4_layout_lm_change,
> > > +	.lm_breaker_timedout	= nfsd_layout_breaker_timedout,
> > >   };
> > >   
> > >   int

-- 
Jeff Layton <jlayton@kernel.org>