Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Tim Menninger <tmenninger@everpuredata.com>]

I am seeing a significant throughput regression from the rpciod SMT
affinity change on a high-throughput NFS/RDMA workload.

This is a 96-CPU, two-socket system with 48 physical cores (2 threads per
core). I bisected the regression to the patch that changes rpciod to use
WQ_AFFN_SMT.

With the default cache_shard scope, the workload sustains approximately 45
GB/s. With the SMT scope, some runs fall to approximately 15-25 GB/s.

The failure is intermittent across workload starts and appears easier to
reproduce shortly after boot. However, once I have a bad run, the
dependency on the rpciod affinity scope is reproducible without restarting
the workload.

For example, during one continuously running workload with the regression
actively reproducing, throughput recovers to ~45 GB/s immediately when I
change /sys/bus/workqueue/devices/rpciod/affinity_scope to cache_shard,
then regresses again immediately when I restore smt.

Nothing else about the workload, mount, RPC connections, or RDMA
connections is changed between those transitions.

On this machine, wq_dump.py reports:

    SMT:          48 affinity pods
    CACHE_SHARD:   6 affinity pods

The SMT pods correspond to one physical core / two sibling CPUs, while each
cache_shard pod contains eight physical cores / sixteen logical CPUs.

I have not yet identified the exact mechanism that causes the SMT
configuration to lose throughput, so I don't want to speculate about the
specific lock or scheduler interaction involved. But the live smt ->
cache_shard -> smt transition seems to isolate the regression to this
affinity-scope change.

Given the magnitude of the regression, I think this needs to be understood
before the rpciod SMT affinity change is merged.

I can collect additional workqueue or scheduler traces if there is
something specific that would help characterize why the SMT scope performs
poorly here.