Re: [PATCH 2/2] NFSD: Fix server hang when there are multiple layout conflicts

[Dai Ngo <dai.ngo@oracle.com>]

On 11/6/25 9:36 AM, Jeff Layton wrote:
> On Thu, 2025-11-06 at 09:17 -0800, Dai Ngo wrote:
>> On 11/6/25 9:14 AM, Jeff Layton wrote:
>>> On Thu, 2025-11-06 at 08:47 -0800, Dai Ngo wrote:
>>>> When a layout conflict triggers a call to __break_lease, the function
>>>> nfsd4_layout_lm_break clears the fl_break_time timeout before sending
>>>> the CB_LAYOUTRECALL. As a result, __break_lease repeatedly restarts
>>>> its loop, waiting indefinitely for the conflicting file lease to be
>>>> released.
>>>>
>>>> If the number of lease conflicts matches the number of NFSD threads
>>>> (which defaults to 8), all available NFSD threads become occupied.
>>>> Consequently, there are no threads left to handle incoming requests
>>>> or callback replies, leading to a total hang of the NFS server.
>>>>
>>>> This issue is reliably reproducible by running the Git test suite
>>>> on a configuration using SCSI layout.
>>>>
>>>> This patch addresses the problem by using the break lease timeout
>>>> and ensures that the unresponsive client is fenced, preventing it from
>>>> accessing the data server directly.
>>>>
>>>> Fixes: f99d4fbdae67 ("nfsd: add SCSI layout support")
>>>> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
>>>> ---
>>>>    fs/nfsd/nfs4layouts.c | 25 +++++++++++++++++++++----
>>>>    1 file changed, 21 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
>>>> index 683bd1130afe..b9b1eb32624c 100644
>>>> --- a/fs/nfsd/nfs4layouts.c
>>>> +++ b/fs/nfsd/nfs4layouts.c
>>>> @@ -747,11 +747,10 @@ static bool
>>>>    nfsd4_layout_lm_break(struct file_lease *fl)
>>>>    {
>>>>    	/*
>>>> -	 * We don't want the locks code to timeout the lease for us;
>>>> -	 * we'll remove it ourself if a layout isn't returned
>>>> -	 * in time:
>>>> +	 * Enforce break lease timeout to prevent starvation of
>>>> +	 * NFSD threads in __break_lease that causes server to
>>>> +	 * hang.
>>>>    	 */
>>>> -	fl->fl_break_time = 0;
>>> I guess this ends up with whatever the default fl_break_time is which
>>> is:
>>>
>>> 	jiffies + lease_break_time * HZ;
>> Yes, currently is 45 secs which is, I think, is way too long.
>>
>>> I wonder if this should be based around some multiple of the grace
>>> period instead?
>> I think the time to allow for recall reply should be in milliseconds.
>>
>> -Dai
>>
> I don't think that's at all reasonable. We'll be fencing slow machines
> all over the place. Clients expect that they can be out of contact for
> a little while (a lease period) and not lose their state. Fencing them
> on a timeout substantially less than that will violate that
> expectation.

That is fine. But the way __break_lease is implemented now is that the
NFSD thread (limited resource) is tied up for a very long time to wait
or the client, which does not seem right.

Also, in __break_lease, if the first lease in the flc_lease list is not
the one that causes the conflict then its fl_break_time is 0 which causes
wait in wait_event_interruptible_timeout to be 1 sec. And that thread
just chews up CPU cycles in 1 second interval.

For long term, perhaps we should find other ways to implement __break_lease.

-Dai

>
>>>>    	nfsd4_recall_file_layout(fl->c.flc_owner);
>>>>    	return false;
>>>>    }
>>>> @@ -764,9 +763,27 @@ nfsd4_layout_lm_change(struct file_lease *onlist, int arg,
>>>>    	return lease_modify(onlist, arg, dispose);
>>>>    }
>>>>    
>>>> +static void nfsd_layout_breaker_timedout(struct file_lease *fl)
>>>> +{
>>>> +	struct nfs4_layout_stateid *ls = fl->c.flc_owner;
>>>> +	struct nfsd_file *nf;
>>>> +
>>>> +	rcu_read_lock();
>>>> +	nf = nfsd_file_get(ls->ls_file);
>>>> +	rcu_read_unlock();
>>>> +	if (nf) {
>>>> +		int type = ls->ls_layout_type;
>>>> +
>>>> +		if (nfsd4_layout_ops[type]->fence_client)
>>>> +			nfsd4_layout_ops[type]->fence_client(ls, nf);
>>>> +		nfsd_file_put(nf);
>>>> +	}
>>>> +}
>>>> +
>>>>    static const struct lease_manager_operations nfsd4_layouts_lm_ops = {
>>>>    	.lm_break	= nfsd4_layout_lm_break,
>>>>    	.lm_change	= nfsd4_layout_lm_change,
>>>> +	.lm_breaker_timedout	= nfsd_layout_breaker_timedout,
>>>>    };
>>>>    
>>>>    int