Re: extremely long cl_tasks list

[Trond Myklebust <trondmy@hammerspace.com>]

On Fri, 2024-11-08 at 15:20 -0800, Dai Ngo wrote:
> Hi Trond,
> 
> Currently cl_tasks is used to maintain the list of all rpc_task's
> for each rpc_clnt.
> 
> Under heavy write load, we've seen this list grows to millions
> of entries. Even though the list is extremely long, the system
> still runs fine until the user wants to get the information of
> all active RPC tasks by doing:
> 
> #  cat /sys/kernel/debug/sunrpc/rpc_clnt/N/tasks
> 
> When this happens, tasks_start() is called and it acquires the
> rpc_clnt.cl_lock to walk the cl_tasks list, returning one entry
> at a time to the caller. The cl_lock is held until all tasks on
> this list have been processed.
>       
> While the cl_lock is held, completed RPC tasks have to spin wait
> in rpc_task_release_client for the cl_lock. If there are millions
> of entries in the cl_tasks list it will take a long time before
> tasks_stop is called and the cl_lock is released.
> 
> Under heavy load condition the rpc_task_release_client threads
> will use up all the available CPUs in the system, preventing other
> jobs to run and this causes the system to temporarily lock up.
>   
> I'm looking for suggestions on how to address this issue. I think
> one option is to convert the cl_tasks list to use xarray to eliminate
> the contention on the cl_lock and would like to get the opinion
> from the community.


No. We are definitely not going to add a gravity-challenged solution
like xarray to solve a corner-case problem of list iteration.

Firstly, this is really only a problem for NFSv3 and NFSv4.0 because
they don't actually throttle at the NFS layer.

Secondly, having millions of entries associated with a single struct
rpc_clnt, means living in latency hell, where waking up a sleeping task
can mean living on the rpciod queue for several 100ms before execution
starts due to the shear volume of tasks in the queue.

So IMHO a better question would be: "What is a sensible throttling
scheme for NFSv3 and NFSv4.0?"

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com