[PATCH RFC 0/8] Reduce lock contention in the NFS client

[PATCH RFC 0/8] Reduce lock contention in the NFS client

[Chuck Lever]

Under a 4KB NFSv3 workload on 100GbE RDMA, roughly 150 RPC worker
threads drive the client, and lock contention dominates its CPU
profile: up to 53% of non-idle cycles are spent in
native_queued_spin_lock_slowpath.

An earlier RFC put this fix in the workqueue core, falling back
automatically to a finer scope when the configured scope
degenerates to a single pod. Tejun rejected a finer default for
unbound workqueues and suggested two alternatives: a sharded
scope between CACHE and SMT, and a scope change on the NFS
workqueues themselves [1]. The first became the
WQ_AFFN_CACHE_SHARD default, whose 8-core shards still leave a
12-core single-LLC system with only two pools. This series does
the second.

My test harness is transport-limited, so the freed cycles appear as
reduced submission latency variance and lower tail completion
latency rather than higher IOPS.

[1] https://lore.kernel.org/all/aYUVVuIidMpuYy3j@slm.duckdns.org/

---
Chuck Lever (8):
      SUNRPC: Use atomic_t for XID allocation
      SUNRPC: Execute initial async RPC states in caller's context
      SUNRPC: Split recv_lock out of xprt->queue_lock
      Set WQ_SYSFS on key NFS-related workqueues
      workqueue: Export the functions needed for WQ attribute modification
      SUNRPC: Reduce rpciod workqueue contention
      NFS: Reduce nfsiod workqueue contention
      SUNRPC: Reduce xprtiod workqueue contention

 fs/nfs/inode.c                             | 18 ++++++++-
 include/linux/sunrpc/xprt.h                |  8 +++-
 kernel/workqueue.c                         |  3 ++
 net/sunrpc/sched.c                         | 42 ++++++++++++++------
 net/sunrpc/svcsock.c                       |  6 +--
 net/sunrpc/xprt.c                          | 63 +++++++++++++++---------------
 net/sunrpc/xprtrdma/rpc_rdma.c             | 14 +++----
 net/sunrpc/xprtrdma/svc_rdma_backchannel.c |  8 ++--
 net/sunrpc/xprtsock.c                      | 18 ++++-----
 9 files changed, 111 insertions(+), 69 deletions(-)
---
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
change-id: 20260831-performance-e465e621c1c0

Best regards,
--  
Chuck Lever <cel@kernel.org>

[PATCH RFC 1/8] SUNRPC: Use atomic_t for XID allocation

[Chuck Lever]

xprt_alloc_xid() acquires reserve_lock to increment a simple
counter. Under a high-IOPS NFSv3 workload on 100GbE RDMA,
profiling shows 1.06% of system-wide CPU cycles contending on
this lock in xprt_request_init, as ~150 RPC worker threads
serialize on the counter.

reserve_lock protects the slot table and backlog queue, but
XID allocation is an independent operation that does not require
synchronization with either.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 include/linux/sunrpc/xprt.h | 2 +-
 net/sunrpc/xprt.c           | 9 ++-------
 2 files changed, 3 insertions(+), 8 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index a82045804d34..0d6c3f6bf97e 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -273,7 +273,7 @@ struct rpc_xprt {
 	spinlock_t		transport_lock;	/* lock transport info */
 	spinlock_t		reserve_lock;	/* lock slot table */
 	spinlock_t		queue_lock;	/* send/receive queue lock */
-	u32			xid;		/* Next XID value to use */
+	atomic_t		xid;		/* Most recently issued XID */
 	struct rpc_task *	snd_task;	/* Task blocked in send */
 
 	struct list_head	xmit_queue;	/* Send queue */
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 48a3618cbb29..186c14f0f928 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1882,18 +1882,13 @@ xprt_init_connect_cookie(struct rpc_rqst *req, struct rpc_xprt *xprt)
 static __be32
 xprt_alloc_xid(struct rpc_xprt *xprt)
 {
-	__be32 xid;
-
-	spin_lock(&xprt->reserve_lock);
-	xid = (__force __be32)xprt->xid++;
-	spin_unlock(&xprt->reserve_lock);
-	return xid;
+	return (__force __be32)atomic_inc_return(&xprt->xid);
 }
 
 static void
 xprt_init_xid(struct rpc_xprt *xprt)
 {
-	xprt->xid = get_random_u32();
+	atomic_set(&xprt->xid, get_random_u32());
 }
 
 static void

-- 
2.55.0

[PATCH RFC 2/8] SUNRPC: Execute initial async RPC states in caller's context

[Chuck Lever]

Each async RPC task is dispatched to the rpciod workqueue before its
first FSM state runs, though the initial states (call_start through
call_allocate) typically complete without blocking. Under high
concurrency this adds a workqueue enqueue, dequeue, and context
switch per RPC and contributes to rpciod pool lock contention.

Have rpc_execute() call __rpc_execute() directly for async tasks as
well. The initial states then run in the caller's context, and
__rpc_execute() returns to the caller when the task first sleeps on
a wait queue. Later wake-ups still dispatch to rpciod via
rpc_make_runnable().

Commit d6a1ed08c6ac ("SUNRPC: Reduce asynchronous RPC task stack
usage") moved async dispatch onto rpciod to bound caller stack
depth. Inline execution preserves that bound because __rpc_execute()
still runs each state from the same stack frame.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 net/sunrpc/sched.c | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)

diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
index 016f16ca5779..9b2800f6bfc7 100644
--- a/net/sunrpc/sched.c
+++ b/net/sunrpc/sched.c
@@ -1003,8 +1003,9 @@ static void __rpc_execute(struct rpc_task *task)
 	current_restore_flags(pflags, PF_MEMALLOC);
 }
 
-/*
- * User-visible entry point to the scheduler.
+/**
+ * rpc_execute - Consumer entry point to the RPC scheduler
+ * @task: RPC task to be scheduled
  *
  * This may be called recursively if e.g. an async NFS task updates
  * the attributes and finds that dirty pages must be flushed.
@@ -1014,15 +1015,12 @@ static void __rpc_execute(struct rpc_task *task)
  */
 void rpc_execute(struct rpc_task *task)
 {
-	bool is_async = RPC_IS_ASYNC(task);
+	unsigned int pflags = memalloc_nofs_save();
 
 	rpc_set_active(task);
-	rpc_make_runnable(rpciod_workqueue, task);
-	if (!is_async) {
-		unsigned int pflags = memalloc_nofs_save();
-		__rpc_execute(task);
-		memalloc_nofs_restore(pflags);
-	}
+	rpc_test_and_set_running(task);
+	__rpc_execute(task);
+	memalloc_nofs_restore(pflags);
 }
 
 static void rpc_async_schedule(struct work_struct *work)

-- 
2.55.0

[PATCH RFC 3/8] SUNRPC: Split recv_lock out of xprt->queue_lock

[Chuck Lever]

xprt->queue_lock protects two independent structures: the recv_queue
rb-tree for reply matching and the xmit_queue list for transmit
draining. No hot path touches both in one critical section, yet every
RPC submit and completion contends on the same lock. Under a 4KB
NFSv3 READ workload on 100GbE RDMA, 53% of non-idle CPU cycles are
spent in native_queued_spin_lock_slowpath: the CQ completion worker
running rpcrdma_reply_handler serializes against ~150 kworker threads
enqueuing receives and transmits.

Introduce xprt->recv_lock for the receive path -- recv_queue
operations, request lookup, receive-side pinning, and completion --
leaving queue_lock to the xmit_queue and the xprt_transmit drain
loop.

Also move the rq_private_buf memcpy in xprt_request_enqueue_receive
above the lock acquisition: until the rb-tree insert publishes the
request, the reply handler cannot see it, so the copy is safe
unlocked and the submitter's critical section shrinks to the insert
alone.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 include/linux/sunrpc/xprt.h                |  6 +++-
 net/sunrpc/svcsock.c                       |  6 ++--
 net/sunrpc/xprt.c                          | 54 ++++++++++++++++--------------
 net/sunrpc/xprtrdma/rpc_rdma.c             | 14 ++++----
 net/sunrpc/xprtrdma/svc_rdma_backchannel.c |  8 ++---
 net/sunrpc/xprtsock.c                      | 18 +++++-----
 6 files changed, 57 insertions(+), 49 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index 0d6c3f6bf97e..ed1e28b74f02 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -272,7 +272,7 @@ struct rpc_xprt {
 	atomic_long_t		queuelen;
 	spinlock_t		transport_lock;	/* lock transport info */
 	spinlock_t		reserve_lock;	/* lock slot table */
-	spinlock_t		queue_lock;	/* send/receive queue lock */
+	spinlock_t		queue_lock;	/* send queue lock */
 	atomic_t		xid;		/* Most recently issued XID */
 	struct rpc_task *	snd_task;	/* Task blocked in send */
 
@@ -292,6 +292,10 @@ struct rpc_xprt {
 						 * backchannel rpc_rqst's */
 #endif /* CONFIG_SUNRPC_BACKCHANNEL */
 
+	/*
+	 * Receive stuff
+	 */
+	spinlock_t		recv_lock;	/* receive queue lock */
 	struct rb_root		recv_queue;	/* Receive queue */
 
 	struct {
diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index 50e5e7f5b762..8939ba604385 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -1102,7 +1102,7 @@ static int receive_cb_reply(struct svc_sock *svsk, struct svc_rqst *rqstp)
 
 	if (!bc_xprt)
 		return -EAGAIN;
-	spin_lock(&bc_xprt->queue_lock);
+	spin_lock(&bc_xprt->recv_lock);
 	req = xprt_lookup_rqst(bc_xprt, xid);
 	if (!req)
 		goto unlock_eagain;
@@ -1120,10 +1120,10 @@ static int receive_cb_reply(struct svc_sock *svsk, struct svc_rqst *rqstp)
 	memcpy(dst->iov_base, src->iov_base, src->iov_len);
 	xprt_complete_rqst(req->rq_task, rqstp->rq_arg.len);
 	rqstp->rq_arg.len = 0;
-	spin_unlock(&bc_xprt->queue_lock);
+	spin_unlock(&bc_xprt->recv_lock);
 	return 0;
 unlock_eagain:
-	spin_unlock(&bc_xprt->queue_lock);
+	spin_unlock(&bc_xprt->recv_lock);
 	return -EAGAIN;
 }
 
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 186c14f0f928..42c66464d8f3 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1061,7 +1061,7 @@ xprt_request_rb_remove(struct rpc_xprt *xprt, struct rpc_rqst *req)
  * @xprt: transport on which the original request was transmitted
  * @xid: RPC XID of incoming reply
  *
- * Caller holds xprt->queue_lock.
+ * Caller holds xprt->recv_lock.
  */
 struct rpc_rqst *xprt_lookup_rqst(struct rpc_xprt *xprt, __be32 xid)
 {
@@ -1092,8 +1092,9 @@ xprt_is_pinned_rqst(struct rpc_rqst *req)
  * xprt_pin_rqst - Pin a request on the transport receive list
  * @req: Request to pin
  *
- * Caller must ensure this is atomic with the call to xprt_lookup_rqst()
- * so should be holding xprt->queue_lock.
+ * Caller must hold the lock that protects the queue through which
+ * it found the request: xprt->recv_lock for the receive path,
+ * xprt->queue_lock for the transmit drain path.
  */
 void xprt_pin_rqst(struct rpc_rqst *req)
 {
@@ -1105,14 +1106,10 @@ EXPORT_SYMBOL_GPL(xprt_pin_rqst);
  * xprt_unpin_rqst - Unpin a request on the transport receive list
  * @req: Request to pin
  *
- * Caller should be holding xprt->queue_lock.
+ * Caller holds the lock it held for the matching xprt_pin_rqst().
  */
 void xprt_unpin_rqst(struct rpc_rqst *req)
 {
-	if (!test_bit(RPC_TASK_MSG_PIN_WAIT, &req->rq_task->tk_runstate)) {
-		atomic_dec(&req->rq_pin);
-		return;
-	}
 	if (atomic_dec_and_test(&req->rq_pin))
 		wake_up_var(&req->rq_pin);
 }
@@ -1155,16 +1152,16 @@ xprt_request_enqueue_receive(struct rpc_task *task)
 	ret = xprt_request_prepare(task->tk_rqstp, &req->rq_rcv_buf);
 	if (ret)
 		return ret;
-	spin_lock(&xprt->queue_lock);
-
-	/* Update the softirq receive buffer */
+	/* Reply handlers cannot find the request until the rb-tree
+	 * insert below publishes it, so the copy needs no lock.
+	 */
 	memcpy(&req->rq_private_buf, &req->rq_rcv_buf,
 			sizeof(req->rq_private_buf));
 
-	/* Add request to the receive list */
+	spin_lock(&xprt->recv_lock);
 	xprt_request_rb_insert(xprt, req);
 	set_bit(RPC_TASK_NEED_RECV, &task->tk_runstate);
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 
 	/* Turn off autodisconnect */
 	timer_delete_sync(&xprt->timer);
@@ -1175,7 +1172,7 @@ xprt_request_enqueue_receive(struct rpc_task *task)
  * xprt_request_dequeue_receive_locked - Remove a request from the receive queue
  * @task: RPC task
  *
- * Caller must hold xprt->queue_lock.
+ * Caller must hold xprt->recv_lock.
  */
 static void
 xprt_request_dequeue_receive_locked(struct rpc_task *task)
@@ -1190,7 +1187,7 @@ xprt_request_dequeue_receive_locked(struct rpc_task *task)
  * xprt_update_rtt - Update RPC RTT statistics
  * @task: RPC request that recently completed
  *
- * Caller holds xprt->queue_lock.
+ * Caller holds xprt->recv_lock.
  */
 void xprt_update_rtt(struct rpc_task *task)
 {
@@ -1212,7 +1209,7 @@ EXPORT_SYMBOL_GPL(xprt_update_rtt);
  * @task: RPC request that recently completed
  * @copied: actual number of bytes received from the transport
  *
- * Caller holds xprt->queue_lock.
+ * Caller holds xprt->recv_lock.
  */
 void xprt_complete_rqst(struct rpc_task *task, int copied)
 {
@@ -1309,7 +1306,7 @@ void xprt_request_wait_receive(struct rpc_task *task)
 	 * The spinlock ensures atomicity between the test of
 	 * req->rq_reply_bytes_recvd, and the call to rpc_sleep_on().
 	 */
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	if (test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate)) {
 		xprt->ops->wait_for_reply_request(task);
 		/*
@@ -1321,7 +1318,7 @@ void xprt_request_wait_receive(struct rpc_task *task)
 			rpc_wake_up_queued_task_set_status(&xprt->pending,
 					task, -ENOTCONN);
 	}
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 }
 
 static bool
@@ -1439,7 +1436,11 @@ xprt_request_dequeue_transmit(struct rpc_task *task)
  * @task: pointer to rpc_task
  *
  * Remove a task from the transmit and receive queues, and ensure that
- * it is not pinned by the receive work item.
+ * it is not pinned by any concurrent work item.
+ *
+ * Dequeuing from both queues prevents new pins: xprt_lookup_rqst
+ * and xprt_transmit can no longer find the request. The wait for
+ * in-flight pins to drain then needs neither lock.
  */
 void
 xprt_request_dequeue_xprt(struct rpc_task *task)
@@ -1451,16 +1452,18 @@ xprt_request_dequeue_xprt(struct rpc_task *task)
 	    test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate) ||
 	    xprt_is_pinned_rqst(req)) {
 		spin_lock(&xprt->queue_lock);
-		while (xprt_is_pinned_rqst(req)) {
+		xprt_request_dequeue_transmit_locked(task);
+		spin_unlock(&xprt->queue_lock);
+
+		spin_lock(&xprt->recv_lock);
+		xprt_request_dequeue_receive_locked(task);
+		spin_unlock(&xprt->recv_lock);
+
+		if (xprt_is_pinned_rqst(req)) {
 			set_bit(RPC_TASK_MSG_PIN_WAIT, &task->tk_runstate);
-			spin_unlock(&xprt->queue_lock);
 			xprt_wait_on_pinned_rqst(req);
-			spin_lock(&xprt->queue_lock);
 			clear_bit(RPC_TASK_MSG_PIN_WAIT, &task->tk_runstate);
 		}
-		xprt_request_dequeue_transmit_locked(task);
-		xprt_request_dequeue_receive_locked(task);
-		spin_unlock(&xprt->queue_lock);
 		xdr_free_bvec(&req->rq_rcv_buf);
 	}
 }
@@ -2038,6 +2041,7 @@ static void xprt_init(struct rpc_xprt *xprt, struct net *net)
 	spin_lock_init(&xprt->transport_lock);
 	spin_lock_init(&xprt->reserve_lock);
 	spin_lock_init(&xprt->queue_lock);
+	spin_lock_init(&xprt->recv_lock);
 
 	INIT_LIST_HEAD(&xprt->free);
 	xprt->recv_queue = RB_ROOT;
diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c
index 1285f04cdac1..a82d3d9bc7ae 100644
--- a/net/sunrpc/xprtrdma/rpc_rdma.c
+++ b/net/sunrpc/xprtrdma/rpc_rdma.c
@@ -1321,9 +1321,9 @@ void rpcrdma_unpin_rqst(struct rpcrdma_rep *rep)
 	req->rl_reply = NULL;
 	rep->rr_rqst = NULL;
 
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	xprt_unpin_rqst(rqst);
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 }
 
 /**
@@ -1363,10 +1363,10 @@ void rpcrdma_complete_rqst(struct rpcrdma_rep *rep)
 		goto out_badheader;
 
 out:
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	xprt_complete_rqst(rqst->rq_task, status);
 	xprt_unpin_rqst(rqst);
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 	return;
 
 out_badheader:
@@ -1492,12 +1492,12 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep)
 	/* Match incoming rpcrdma_rep to an rpcrdma_req to
 	 * get context for handling any incoming chunks.
 	 */
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	rqst = xprt_lookup_rqst(xprt, rep->rr_xid);
 	if (!rqst)
 		goto out_norqst;
 	xprt_pin_rqst(rqst);
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 
 	if (buf->rb_credits != credits)
 		rpcrdma_update_cwnd(r_xprt, credits);
@@ -1524,7 +1524,7 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep)
 	return;
 
 out_norqst:
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 	trace_xprtrdma_reply_rqst_err(rep);
 	rpcrdma_rep_put(buf, rep);
 	goto out_post;
diff --git a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
index e5a78b761012..3c7b85427f33 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
@@ -28,7 +28,7 @@ void svc_rdma_handle_bc_reply(struct svc_rqst *rqstp,
 	struct rpc_rqst *req;
 	u32 credits;
 
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	req = xprt_lookup_rqst(xprt, *rdma_resp);
 	if (!req)
 		goto out_unlock;
@@ -39,7 +39,7 @@ void svc_rdma_handle_bc_reply(struct svc_rqst *rqstp,
 		goto out_unlock;
 	memcpy(dst->iov_base, src->iov_base, src->iov_len);
 	xprt_pin_rqst(req);
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 
 	credits = be32_to_cpup(rdma_resp + 2);
 	if (credits == 0)
@@ -50,13 +50,13 @@ void svc_rdma_handle_bc_reply(struct svc_rqst *rqstp,
 	xprt->cwnd = credits << RPC_CWNDSHIFT;
 	spin_unlock(&xprt->transport_lock);
 
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	xprt_complete_rqst(req->rq_task, rcvbuf->len);
 	xprt_unpin_rqst(req);
 	rcvbuf->len = 0;
 
 out_unlock:
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 }
 
 /* Send a reverse-direction RPC Call.
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 7f60723fa64d..1454da9575b3 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -673,25 +673,25 @@ xs_read_stream_reply(struct sock_xprt *transport, struct msghdr *msg, int flags)
 	ssize_t ret = 0;
 
 	/* Look up and lock the request corresponding to the given XID */
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	req = xprt_lookup_rqst(xprt, transport->recv.xid);
 	if (!req || (transport->recv.copied && !req->rq_private_buf.len)) {
 		msg->msg_flags |= MSG_TRUNC;
 		goto out;
 	}
 	xprt_pin_rqst(req);
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 
 	ret = xs_read_stream_request(transport, msg, flags, req);
 
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	if (msg->msg_flags & (MSG_EOR|MSG_TRUNC))
 		xprt_complete_rqst(req->rq_task, transport->recv.copied);
 	else
 		req->rq_private_buf.len = transport->recv.copied;
 	xprt_unpin_rqst(req);
 out:
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 	return ret;
 }
 
@@ -1398,13 +1398,13 @@ static void xs_udp_data_read_skb(struct rpc_xprt *xprt,
 		return;
 
 	/* Look up and lock the request corresponding to the given XID */
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	rovr = xprt_lookup_rqst(xprt, *xp);
 	if (!rovr)
 		goto out_unlock;
 	xprt_pin_rqst(rovr);
 	xprt_update_rtt(rovr->rq_task);
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 	task = rovr->rq_task;
 
 	if ((copied = rovr->rq_private_buf.buflen) > repsize)
@@ -1412,7 +1412,7 @@ static void xs_udp_data_read_skb(struct rpc_xprt *xprt,
 
 	/* Suck it into the iovec, verify checksum if not done by hw. */
 	if (csum_partial_copy_to_xdr(&rovr->rq_private_buf, skb)) {
-		spin_lock(&xprt->queue_lock);
+		spin_lock(&xprt->recv_lock);
 		__UDPX_INC_STATS(sk, UDP_MIB_INERRORS);
 		goto out_unpin;
 	}
@@ -1421,13 +1421,13 @@ static void xs_udp_data_read_skb(struct rpc_xprt *xprt,
 	spin_lock(&xprt->transport_lock);
 	xprt_adjust_cwnd(xprt, task, copied);
 	spin_unlock(&xprt->transport_lock);
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	xprt_complete_rqst(task, copied);
 	__UDPX_INC_STATS(sk, UDP_MIB_INDATAGRAMS);
 out_unpin:
 	xprt_unpin_rqst(rovr);
  out_unlock:
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 }
 
 static void xs_udp_data_receive(struct sock_xprt *transport)

-- 
2.55.0

[PATCH RFC 4/8] Set WQ_SYSFS on key NFS-related workqueues

[Chuck Lever]

WQ_SYSFS exposes settable workqueue attributes under
/sys/devices/virtual/workqueue/. For NFS workloads, this is useful
for tuning affinity scope at runtime without reloading modules.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 fs/nfs/inode.c     | 3 ++-
 net/sunrpc/sched.c | 5 +++--
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c
index 3022454f7698..107a2135029d 100644
--- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c
@@ -2619,7 +2619,8 @@ static void nfsiod_stop(void)
 static int nfsiod_start(void)
 {
 	dprintk("RPC:       creating workqueue nfsiod\n");
-	nfsiod_workqueue = alloc_workqueue("nfsiod", WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
+	nfsiod_workqueue = alloc_workqueue("nfsiod",
+				WQ_MEM_RECLAIM | WQ_UNBOUND | WQ_SYSFS, 0);
 	if (nfsiod_workqueue == NULL)
 		return -ENOMEM;
 #if IS_ENABLED(CONFIG_NFS_LOCALIO)
diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
index 9b2800f6bfc7..c31cf55b933f 100644
--- a/net/sunrpc/sched.c
+++ b/net/sunrpc/sched.c
@@ -1271,16 +1271,17 @@ void rpciod_down(void)
  */
 static int rpciod_start(void)
 {
+	const unsigned int wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND | WQ_SYSFS;
 	struct workqueue_struct *wq;
 
 	/*
 	 * Create the rpciod thread and wait for it to start.
 	 */
-	wq = alloc_workqueue("rpciod", WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
+	wq = alloc_workqueue("rpciod", wq_flags, 0);
 	if (!wq)
 		goto out_failed;
 	rpciod_workqueue = wq;
-	wq = alloc_workqueue("xprtiod", WQ_UNBOUND | WQ_MEM_RECLAIM, 0);
+	wq = alloc_workqueue("xprtiod", wq_flags, 0);
 	if (!wq)
 		goto free_rpciod;
 	xprtiod_workqueue = wq;

-- 
2.55.0

[PATCH RFC 5/8] workqueue: Export the functions needed for WQ attribute modification

[Chuck Lever]

Modules that create unbound workqueues currently have no way
to modify workqueue attributes at run time. Export the
allocation, application, and teardown functions so that
modules such as sunrpc can adjust affinity scope without
reloading.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 kernel/workqueue.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 3c034cbc5bb3..fd76a875cf2d 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -4813,6 +4813,7 @@ void free_workqueue_attrs(struct workqueue_attrs *attrs)
 		kfree(attrs);
 	}
 }
+EXPORT_SYMBOL_GPL(free_workqueue_attrs);
 
 /**
  * alloc_workqueue_attrs - allocate a workqueue_attrs
@@ -4841,6 +4842,7 @@ struct workqueue_attrs *alloc_workqueue_attrs_noprof(void)
 	free_workqueue_attrs(attrs);
 	return NULL;
 }
+EXPORT_SYMBOL_GPL(alloc_workqueue_attrs_noprof);
 
 static void copy_workqueue_attrs(struct workqueue_attrs *to,
 				 const struct workqueue_attrs *from)
@@ -5609,6 +5611,7 @@ int apply_workqueue_attrs(struct workqueue_struct *wq,
 
 	return ret;
 }
+EXPORT_SYMBOL_GPL(apply_workqueue_attrs);
 
 /**
  * unbound_wq_update_pwq - update a pwq slot for CPU hot[un]plug

-- 
2.55.0

[PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

rpciod drives the RPC client state machine. Under heavy NFS
workloads, multiple CPUs queue RPC task completions concurrently and
contend on the UNBOUND worker pool lock. perf profiles on a 12-core
system show 30-40% of cycles lost to
native_queued_spin_lock_slowpath in the rpciod pool at the
WQ_AFFN_CACHE scope (one pool per LLC). The WQ_AFFN_CACHE_SHARD
default helps little here, because its 8-core shards split this
system into just two pools of six cores each.

Set WQ_AFFN_SMT on rpciod so each SMT group gets its own pool and
lock. Most UNBOUND workqueues never contend on the pool lock and
profit from a coarser scope's cache locality. rpciod's sustained
completion traffic makes the lock a first-order bottleneck, so the
override belongs on this workqueue rather than in the system-wide
default. Idle kworkers are culled, so the extra pools cost little on
large systems.

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Chuck Lever <cel@kernel.org>
---
 net/sunrpc/sched.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
index c31cf55b933f..b84af2c0b104 100644
--- a/net/sunrpc/sched.c
+++ b/net/sunrpc/sched.c
@@ -1266,6 +1266,25 @@ void rpciod_down(void)
 	module_put(THIS_MODULE);
 }
 
+static void rpc_set_wq_smt_affinity(struct workqueue_struct *wq,
+				    const char *name)
+{
+	struct workqueue_attrs *attrs;
+	int err;
+
+	attrs = alloc_workqueue_attrs();
+	if (!attrs) {
+		pr_warn("%s: failed to allocate workqueue attrs\n", name);
+		return;
+	}
+	attrs->affn_scope = WQ_AFFN_SMT;
+	err = apply_workqueue_attrs(wq, attrs);
+	free_workqueue_attrs(attrs);
+	if (err)
+		pr_warn("%s: failed to set SMT affinity scope: %d\n",
+			name, err);
+}
+
 /*
  * Start up the rpciod workqueue.
  */
@@ -1280,6 +1299,7 @@ static int rpciod_start(void)
 	wq = alloc_workqueue("rpciod", wq_flags, 0);
 	if (!wq)
 		goto out_failed;
+	rpc_set_wq_smt_affinity(wq, "rpciod");
 	rpciod_workqueue = wq;
 	wq = alloc_workqueue("xprtiod", wq_flags, 0);
 	if (!wq)

-- 
2.55.0

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Tim Menninger]

I am seeing a significant throughput regression from the rpciod SMT
affinity change on a high-throughput NFS/RDMA workload.

This is a 96-CPU, two-socket system with 48 physical cores (2 threads per
core). I bisected the regression to the patch that changes rpciod to use
WQ_AFFN_SMT.

With the default cache_shard scope, the workload sustains approximately 45
GB/s. With the SMT scope, some runs fall to approximately 15-25 GB/s.

The failure is intermittent across workload starts and appears easier to
reproduce shortly after boot. However, once I have a bad run, the
dependency on the rpciod affinity scope is reproducible without restarting
the workload.

For example, during one continuously running workload with the regression
actively reproducing, throughput recovers to ~45 GB/s immediately when I
change /sys/bus/workqueue/devices/rpciod/affinity_scope to cache_shard,
then regresses again immediately when I restore smt.

Nothing else about the workload, mount, RPC connections, or RDMA
connections is changed between those transitions.

On this machine, wq_dump.py reports:

    SMT:          48 affinity pods
    CACHE_SHARD:   6 affinity pods

The SMT pods correspond to one physical core / two sibling CPUs, while each
cache_shard pod contains eight physical cores / sixteen logical CPUs.

I have not yet identified the exact mechanism that causes the SMT
configuration to lose throughput, so I don't want to speculate about the
specific lock or scheduler interaction involved. But the live smt ->
cache_shard -> smt transition seems to isolate the regression to this
affinity-scope change.

Given the magnitude of the regression, I think this needs to be understood
before the rpciod SMT affinity change is merged.

I can collect additional workqueue or scheduler traces if there is
something specific that would help characterize why the SMT scope performs
poorly here.

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

On Wed, Sep 2, 2026, at 4:40 PM, Tim Menninger wrote:
> I am seeing a significant throughput regression from the rpciod SMT
> affinity change on a high-throughput NFS/RDMA workload.

Thanks for the report!

I'd like to learn a bit more about the setup before drawing
any firm conclusions.

1. Does v2 of the series behave the same way? v2 dropped the
   patch that ran the first RPC states in the submitter's
   context, so the rpciod traffic pattern differs from v1.
   The same smt -> cache_shard -> smt toggle on v2 would tell
   us whether that matters.
  
2. How are the RDMA device's completion interrupts placed?
   Please share, on a bad run:
   - /proc/interrupts lines for the device's completion vectors
   - the smp_affinity_list for each of those IRQs
   - whether irqbalance is running, and whether you've pinned
     the IRQs by hand

3. Where is CPU time going on a bad run versus a good one?
   A short capture of each would help:
     perf record -a -g -- sleep 10
     perf report --sort comm,cpu --stdio | head -80
   In particular I'm interested in which CPUs the kworker
   threads for rpciod and the ib-comp-wq threads run on under
   each scope.

4. tools/workqueue/wq_monitor.py rpciod, sampled for a few
   seconds under each scope, would show whether the pools are
   evenly loaded.

5. The workload itself: thread count, I/O size and direction,
   number of mounts and RDMA connections, and the mount
   options (nconnect in particular).

6. The base kernel the series was applied to, and the RDMA
   device and driver.
  
If you have a way to capture a bad run reliably shortly after
boot, a "before and after" of items 2 through 4 across a single
scope toggle would be the most direct evidence.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Tim Menninger]

Thanks. I did some more testing with v2 and collected the requested
information.

> 1. Does v2 of the series behave the same way? v2 dropped the
>    patch that ran the first RPC states in the submitter's
>    context, so the rpciod traffic pattern differs from v1.
>    The same smt -> cache_shard -> smt toggle on v2 would tell
>    us whether that matters.

The v2 series behaves similarly insofar as some runs see ~15 GB/s and
other runs see ~45 GB/s, but the low-throughput state no longer appears
to be directly controlled by the rpciod affinity scope.

I have started runs with both smt and cache_shard, and with both scopes
I have seen all three of:

    1. start and remain at ~45 GB/s
    2. start and remain at ~15 GB/s
    3. start at ~45 GB/s, then abruptly drop to ~15 GB/s

I have not found a discernible pattern for how long a run remains at
~45 GB/s before dropping.

cache_shard does still seem somewhat more likely to give me a ~45 GB/s
run, particularly as uptime increases, but unlike v1 I can reproduce
both good and bad runs with either scope. The deterministic live
smt -> cache_shard -> smt behavior I reported for v1 is no longer
present in v2.

So the low-throughput state remains with v2, but it no longer seems to be
from the SMT affinity change alone.

All captures below are from separate runs with the scope set as indicated.

> 2. How are the RDMA device's completion interrupts placed?
>    Please share, on a bad run:
>
>    * /proc/interrupts lines for the device's completion vectors
>    * the smp_affinity_list for each of those IRQs
>    * whether irqbalance is running, and whether you've pinned
>      the IRQs by hand

There are two ConnectX-7 devices:

    mlx5_0 port 1 ==> ens3np0
    mlx5_1 port 1 ==> ens6np0

at PCI addresses 0000:2a:00.0 and 0000:ab:00.0 respectively.

The substantial completion traffic in these captures is on
0000:2a:00.0. Its mlx5 completion IRQs are individually affinitized to
CPUs. For example:

    mlx5_comp0   -> CPU 0
    mlx5_comp1   -> CPU 1
    ...
    mlx5_comp23  -> CPU 23
    mlx5_comp24  -> CPU 48
    ...
    mlx5_comp47  -> CPU 71
    mlx5_comp48  -> CPU 24
    ...
    mlx5_comp62  -> CPU 38

The affinity mapping was the same in the good and bad captures I took.

I have not manually pinned any IRQs, and irqbalance is inactive.

I have the complete /proc/interrupts and smp_affinity_list captures
available if there are particular vectors or deltas that would be useful
to see.

> 3. Where is CPU time going on a bad run versus a good one?
>    A short capture of each would help:
>    perf record -a -g -- sleep 10
>    perf report --sort comm,cpu --stdio | head -80
>    In particular I'm interested in which CPUs the kworker
>    threads for rpciod and the ib-comp-wq threads run on under
>    each scope.

The perf results aren't showing anything useful yet. With the requested
--sort comm,cpu, the top of the report is dominated by perf itself. Without
that sort, the top entries are things like cpuidle_enter_state and
cpuidle_enter.

I'll keep working on the perf capture and follow up if/when I get something
useful, but I didn't want to hold up the rest of this on that.

> 4. tools/workqueue/wq_monitor.py rpciod, sampled for a few
>    seconds under each scope, would show whether the pools are
>    evenly loaded.

I captured rpciod monitoring data from four v2 runs. Here are windows from
each.

cache_shard good (~46 GB/s):
                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        27613426     3    534.5       - 1699488      16       0 

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        27817822     5    537.3       - 1713539      16       0 

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        28027937     2    540.1       - 1728063      16       0 

cache_shard bad (~15 GB/s):
                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        71754056     2    932.1       - 4767531       0       0 

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        71825047     0    933.9       - 4771333       0       0 

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        71901102     3    935.8       - 4775677       0       0 

smt good (~46 GB/s):
                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        17036137     2    144.9       - 4993757       0       0

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        17220514     0    146.8       - 5064827       0       0

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        17397194     3    148.8       - 5132231       0       0

smt bad (~26 GB/s):
                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod         4937072     1     43.2       - 1407321       0       0 

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod         5037193     4     44.0       - 1432686       0       0 

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod         5138509     4     44.8       - 1458079       0       0 

> 5. The workload itself: thread count, I/O size and direction,
>    number of mounts and RDMA connections, and the mount
>    options (nconnect in particular).

Workload:
    elbencho --iter 1 --threads 160 --files 16 --size 256G \
        --block 1m --dropcache --iodepth 64 --direct --read \
        --lat --latpercent --log 1 ...

Mount options:
    /home/ir/exapurity from 10.71.61.129:/exapurity
     Flags:	rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,fatal_neterrors=none,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=10.230.36.66,local_lock=none,write=eager,addr=10.71.61.129

It's pNFS over RDMA with one client, one MDS, one DS.

This is a pNFS flexfiles workload. The MDS connection is TCP, and I see
16 RDMA rpc_xprt instances for the data-server traffic during these
tests.

> 6. The base kernel the series was applied to, and the RDMA
>    device and driver.

For all of the data above, I applied the full v2 series on top of:

    940de590b839 Merge tag 'hardening-v7.3-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux

The RDMA hardware is NVIDIA/Mellanox ConnectX-7 (MT2910, PCI ID
15b3:1021), using mlx5_core/mlx5_ib.

For the active interface:

    driver: mlx5_core
    version: 7.3.0-rc1-mainline-bad-v2+
    firmware-version: 28.47.2682 (MT_0000000838)
    bus-info: 0000:2a:00.0

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

On Thu, 3 Sep 2026, Tim Menninger wrote:
> The v2 series behaves similarly insofar as some runs see ~15 GB/s and
> other runs see ~45 GB/s, but the low-throughput state no longer appears
> to be directly controlled by the rpciod affinity scope.

Based on your follow-up, I have a theory about the bimodal performance
behavior:

> The substantial completion traffic in these captures is on
> 0000:2a:00.0. Its mlx5 completion IRQs are individually affinitized to
> CPUs. For example:
>
>     mlx5_comp0   -> CPU 0
>     mlx5_comp1   -> CPU 1
>     ...
>     mlx5_comp23  -> CPU 23
>     mlx5_comp24  -> CPU 48
>     ...
>     mlx5_comp47  -> CPU 71
>     mlx5_comp48  -> CPU 24
>     ...
>     mlx5_comp62  -> CPU 38

That is the mlx5 default spread. It fills the NIC's local NUMA node
first, both SMT threads, and overflows the remaining vectors onto the
other socket. If I read the numbering right, CPUs 0-23 and 48-71 are
the two threads of the node that 0000:2a:00.0 hangs off, and comp48
through comp62 land on the remote socket. Please confirm with:

    cat /sys/bus/pci/devices/0000:2a:00.0/numa_node
    lscpu | grep 'NUMA node'

rpcrdma.ko allocates a send and a receive completion queue for each
transport, and the RDMA core assigns the completion vector from a
host-global round-robin counter. The completion vector is selected at
transport connect time and is fixed for the life of the connection.

Completions on a remote vector run the CQ handler, the RPC reply
processing, and the rpciod wakeup all on the wrong socket. My
suggestion is that you should pin the NIC's interrupt vectors to
CPUs on the NIC's local node so that the round-robin cannot select
a remote one. Something like the following, adjusted once you have
confirmed the local CPU list:

    for irq in $(awk '/mlx5_comp(4[89]|5[0-9]|6[0-2])@pci:0000:2a:00.0/ \
                      { sub(":", "", $1); print $1 }' /proc/interrupts); do
        echo 0-23,48-71 > /proc/irq/$irq/smp_affinity_list
    done

Keep irqbalance disabled, since it will undo the interrupt steering.


-- 
Chuck Lever

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Tim Menninger]

I realized the bad state correlates with where the RDMA CQs' completion
work runs.

This client has 16 RDMA transports and 32 CQs. The CQs receive 32
consecutive completion vectors out of the mlx5 device's 63-vector ring.
Depending on the global round-robin starting point, all 32 CQ workers can
execute on NUMA node 0, or they can split 17/15 between the two NUMA nodes.

On unpatched mainline (940de590b839 without your patch set), I tested five
consecutive allocation windows. On three of those, all 32 CQs executed on
NUMA node 0. On the other two they fell with a 17/15 split. All five
sustained ~46 GB/s.

With your v2 applied, all-local windows fall to roughly 24-28 GB/s while
split windows sustain ~46 GB/s.

I also tested this by changing only mlx5 IRQ affinity. I redirected
completion vectors 20-34 from node 0 to CPUs 24-38 on node 1. CQ allocation
windows 5-36 and 6-37, which would otherwise be entirely local and slow,
then sustained ~46 GB/s with their CQs split 17/15 across the nodes.

So the CQ allocation determines whether the regression is exposed, but
mainline is insensitive to that placement. The series introduces the
performance sensitivity. This also explains why v2 appeared
nondeterministic with respect to the rpciod affinity scope alone.

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

On Fri, 4 Sep 2026, Tim Menninger wrote:
> With your v2 applied, all-local windows fall to roughly 24-28 GB/s while
> split windows sustain ~46 GB/s.

So on your system the interrupt pinning I suggested would put every
mount into the "slow" placement.

(Pinning completion vectors to the local node still helps where the
dominant cost is cross-socket reply traffic, but your data says that
on your configuration, with this series applied, spreading the CQ
handlers across both sockets matters more than keeping them local).

> So the CQ allocation determines whether the regression is exposed, but
> mainline is insensitive to that placement. The series introduces the
> performance sensitivity.

Agreed. What it does not yet tell us is which part of the series.

Now that you have a reliable way to force an all-local window,
would you run these on top of 940de590b839, each in the all-local
placement:

1. Patches 1-2 only (the XID and recv_lock changes).

2. Patches 3-8 only (WQ_SYSFS, workqueue_set_affn_scope, and the
   three scope changes).

3. The full v2 series, with all three workqueues set to
   cache_shard at once:

     for wq in rpciod xprtiod nfsiod; do
         echo cache_shard > /sys/bus/workqueue/devices/$wq/affinity_scope
     done

If 1 reproduces the slow state on its own, the scope patches are
not the cause and the question becomes why removing queue_lock
contention makes node 0 concentration visible. If only 2
reproduces it, the scope change is the cause and 3 should recover
it.

Also: per-node CPU utilization in a good run and a bad run, e.g.
mpstat -P ALL 1 for a few seconds, or the per-CPU lines from
/proc/stat. Your perf capture showed a lot of idle CPU, so I
would like to know whether node 0 is saturated while node 1
sleeps, or whether both are idle but completions are serializing.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[tmenninger]

I reran the matrix. I'm at a loss for why, but I can no longer reproduce
the "bad" throughput with cache_shard. I saw that a few times last week.
Furthermore, the "bad" throughput is now a 10% degradation rather than 50%.
Nevertheless, it's still reliably reproducible when CQs all stack on NUMA 0
and using smt affinity_scope.

Patches 1-2 alone do not reproduce it across 5 trials where 3 of them fell
32/0 and 2 fell 17/15 on NUMA 0/1. All had full, unchanged throughput.

Patches 3-8 do reproduce it (as well as the full 1-8): throughput dips by
10% when all CQs fall on NUMA 0. Another behavior that I'm seeing again is,
within the same run, toggling affinity_scope to cache_shard fully restores
throughput, then back to smt and the throughput falls again.

The mpstat results, CPU idle times:

                        All NUMA 0          17/15 Split
    aggregate           20%                 70%
    CPU 0-23,48-71      0-2%                60-80%
    CPU 24-47,72-95     40-46%              70-90%

This shape is generally consistent among all three patch subsets: patches
1-2, patches 3-8, and patches 1-8. All of these are without modifying
affinity_scope for the respective defaults.

For patches 3-8 I reran the same experiment but captured mpstat on each
segment of smt -> cache_shard -> smt, on a run where all CQs were on NUMA
0, all idle times again:

                        smt         cache_shard     smt
    aggregate           15%         32%             19%
    CPU 0-23,48-71      0-1%        6-9%            0-2%
    CPU 24-47,72-95     28-32%      55-61%          35-40%

I've also seen all CQs land on NUMA 1, which seems less common than NUMA 0,
but the same behavior appears there with the node roles reversed.

Would there be a downside to using WQ_AFFN_CACHE_SHARD here instead of
WQ_AFFN_SMT?

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

On Tue, 8 Sep 2026, Tim Menninger wrote:

> Patches 1-2 alone do not reproduce it across 5 trials ...
>
> Patches 3-8 do reproduce it (as well as the full 1-8) ...

Thanks, that confirms which part of the series is responsible. The
scope change on rpciod is the cause, and the XID and recv_lock
changes are not involved.


>                         smt         cache_shard     smt
>     aggregate           15%         32%             19%
>     CPU 0-23,48-71      0-1%        6-9%            0-2%
>     CPU 24-47,72-95     28-32%      55-61%          35-40%

Hrm. Under smt, node 0 is saturated and node 1 is busier too, yet
throughput is lower than under cache_shard, where both nodes have
more idle time. The smt scope is burning CPU on something that is
not moving data. That is a different failure than the one I
expected (node 0 starved while node 1 sleeps), so the next step
is to find out what those cycles are doing.


> Would there be a downside to using WQ_AFFN_CACHE_SHARD here instead of
> WQ_AFFN_SMT?

cache_shard is the system default, and that is what the workqueue
used before this series. On the 12-core system where I developed
the series, cache_shard produces only two pools of six cores each,
and 30-40% of cycles went to the pool lock's spinlock slowpath,
accompanied with a measurable loss in throughput.

Your machine gets six pods of sixteen threads, which is a different
regime, and it is possible that cache_shard is simply the right
answer there. What I'm ultimately shooting for is something that
automatically configures the correct behavior. I thought smt would
be that configuration.

A few experiments, all with the full series applied and all CQs on
node 0. Each is a single sysfs write on rpciod except the last.

1. Toggle affinity_strict under smt:

       echo 1 > /sys/bus/workqueue/devices/rpciod/affinity_strict

   Strict pins each pool's kworkers to its SMT pair. If strict
   recovers throughput, the loss comes from non-strict workers
   being wake-affined or migrated onto the saturated node. If
   strict makes it worse, node 0's pools are starved and the fix
   is to let work spill to node 1. Either result cuts the
   hypothesis space in half, so if you have time for only one of
   these, this is the one.

2. Walk the scope ladder: cpu, smt, cache, cache_shard, numa, and
   report throughput for each. If cpu is as bad as smt, pool
   granularity itself is the problem. If cache already recovers,
   the threshold sits between 2-thread and 16-thread pods.

3. Profile the smt and cache_shard windows of one run, node 0 CPUs
   only, so the two captures differ in nothing but the scope:

       perf record -a -g -C 0-23,48-71 -- sleep 10
       perf lock contention -a -C 0-23,48-71 -- sleep 10
       perf stat -a -C 0-23,48-71 \
           -e context-switches,cpu-migrations,sched:sched_wakeup \
           -- sleep 10

   The two candidates I have in mind are a downstream lock, such as
   the transport's queue_lock or recv_lock, contended by 24 small
   pools running completions in parallel; or scheduler overhead
   from each pool waking its own kworkers. The %sys, %irq, and
   %soft columns from the same mpstat runs would help too, since
   idle alone does not say what the busy CPUs are doing.

4. With the even 17/15 CQ split and rpciod at cache_shard, a perf
   profile shows whether the pool-lock slowpath the series targets
   appears on your box at all. If it does not, cache_shard is
   correct for your system, and the series needs a way to express
   that rather than one hardcoded scope.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[tmenninger]

> cache_shard is the system default, and that is what the workqueue
> used before this series

Right, bummer, I was thinking default was WQ_AFFN_CACHE.

> 1. Toggle affinity_strict under smt:
> 
>        echo 1 > /sys/bus/workqueue/devices/rpciod/affinity_strict
> 
>    Strict pins each pool's kworkers to its SMT pair. If strict
>    recovers throughput, the loss comes from non-strict workers
>    being wake-affined or migrated onto the saturated node. If
>    strict makes it worse, node 0's pools are starved and the fix
>    is to let work spill to node 1. Either result cuts the
>    hypothesis space in half, so if you have time for only one of
>    these, this is the one.

Strict makes it worse. Across five runs, the three all-local CQ
placements ran at 38, 36, and 39 GB/s, while the two split placements
both ran at 46 GB/s.

> 2. Walk the scope ladder: cpu, smt, cache, cache_shard, numa, and
>    report throughput for each. If cpu is as bad as smt, pool
>    granularity itself is the problem. If cache already recovers,
>    the threshold sits between 2-thread and 16-thread pods.

    cpu:         37 GBps
    smt:         43 GBps
    cache:       47 GBps
    cache_shard: 47 GBps
    numa:        47 GBps

> 3. Profile the smt and cache_shard windows of one run, node 0 CPUs
>    only, so the two captures differ in nothing but the scope:
> 
>        perf record -a -g -C 0-23,48-71 -- sleep 10
>        perf lock contention -a -C 0-23,48-71 -- sleep 10
>        perf stat -a -C 0-23,48-71 \
>            -e context-switches,cpu-migrations,sched:sched_wakeup \
>            -- sleep 10

The lock profile is dominated by __slab_free in both cases.

For smt, perf report shows:

    native_queued_spin_lock_slowpath     87.07% self

and its callchain is almost entirely:

    rpc_async_release
      -> rpc_free_task
      -> ff_layout_read_release
      -> pnfs_generic_rw_release
      -> nfs_pgio_release
      -> nfs_direct_read_completion
      -> nfs_release_request
      -> nfs_free_request
      -> kmem_cache_free
      -> __slab_free
      -> _raw_spin_lock_irqsave
      -> native_queued_spin_lock_slowpath

cache_shard looks surprisingly similar:

    native_queued_spin_lock_slowpath     85.84% self

with the same nfs_release_request -> nfs_free_request ->
kmem_cache_free -> __slab_free path dominating the profile.

perf lock reports the same general picture. For smt:

    __slab_free:
        2.07M contentions
        3.83 minutes aggregate wait
        111 us average wait

and for cache_shard:

    __slab_free:
        2.27M contentions
        3.65 minutes aggregate wait
        96 us average wait

process_one_work itself is much smaller:

                        smt             cache_shard
    contentions         34              2087
    total wait          91 us           7.13 ms

Scheduler counters:

                        smt             cache_shard
    context switches    2,184,584       3,107,991
    CPU migrations        129,706         499,400
    sched_wakeup        1,176,787       1,776,031

>    The two candidates I have in mind are a downstream lock, such as
>    the transport's queue_lock or recv_lock, contended by 24 small
>    pools running completions in parallel; or scheduler overhead
>    from each pool waking its own kworkers. The %sys, %irq, and
>    %soft columns from the same mpstat runs would help too, since
>    idle alone does not say what the busy CPUs are doing.

For the all-local case, essentially all of the busy time on that node
is %sys. %irq and %soft are both approximately zero. A few CPUs have
low-single-digit %usr.

For the 17/15 split, CPUs 0-2 have low-single-digit %soft, but aggregate
%soft is only about 0.10%. Otherwise it looks the same: the busy time
is overwhelmingly %sys, with a few CPUs showing low-single-digit %usr.

> 4. With the even 17/15 CQ split and rpciod at cache_shard, a perf
>    profile shows whether the pool-lock slowpath the series targets
>    appears on your box at all. If it does not, cache_shard is
>    correct for your system, and the series needs a way to express
>    that rather than one hardcoded scope.

It is present, but it does not appear significant compared with the
slab contention.

With the 17/15 split and cache_shard:

    process_one_work:
        5,598 contentions
        29.33 ms aggregate wait
        5.24 us average wait

In the same 5-second capture, __slab_free has about 1.61M contentions
and 2.66 minutes aggregate wait. get_partial_node_bulk and
__refill_objects_node are both around 19K contentions and ~200 ms total
wait.

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

On Tue, 8 Sep 2026, Tim Menninger wrote:
> Strict makes it worse. Across five runs, the three all-local CQ
> placements ran at 38, 36, and 39 GB/s, while the two split placements
> both ran at 46 GB/s.
>
>     cpu:         37 GBps
>     smt:         43 GBps
>     cache:       47 GBps
>     cache_shard: 47 GBps
>     numa:        47 GBps

Thanks, that settles the scope question. Anything at LLC size or
larger is OK on your box, finer is worse, and the pool-lock cost
the series was after does not show up there at all. I'll drop the
patches that change the WQ affinity, for now.


> The lock profile is dominated by __slab_free in both cases.
> ...
>     __slab_free:
>         2.07M contentions
>         3.83 minutes aggregate wait
>         111 us average wait

That is on the order of twenty CPUs on node 0 doing nothing but
spinning on one lock, in every configuration you measured. It is a
much bigger target than the workqueue scope.

The lock is the SLUB per-node list_lock for the nfs_page cache.
Since 7.3-rc1, every ordinary kmem_cache gets per-CPU sheaves by
default (for nfs_page, 60 objects per sheaf), and a plain free
lands in the local sheaf without touching list_lock. It reaches
__slab_free only when a CPU's sheaves are full and the per-node
barn, which holds at most ten full sheaves, is full too. Then the
whole sheaf gets flushed to the slabs under list_lock.

Your workload appears to defeat that optimization: nfs_page
objects are allocated on the application CPUs across both nodes and
freed in bursts by rpciod on the CQ CPUs. The freeing CPUs never
allocate, so their sheaves stay full, and a 600-object barn is
microseconds of buffering at your free rate. If that is what is
going on, it is an allocator scaling issue rather than an NFS one,
and the right place to take it is linux-mm and Vlastimil.

Before that, I'd like to confirm this analysis. Could you:

1. Check that sheaves are actually on for that cache:

       cat /sys/kernel/slab/nfs_page/sheaf_capacity

   A zero means your kernel predates sheaves or the cache has debug
   flags, and the contention is the older remote-free slab
   transition path. That is a different report.

2. Build with CONFIG_SLUB_STATS=y and sample these before and after
   a 10-second all-local window:

       cd /sys/kernel/slab/nfs_page
       grep . free_fastpath free_slowpath sheaf_flush barn_put \
           barn_put_fail barn_get barn_get_fail alloc_slowpath

   If barn_put_fail and sheaf_flush track the contention count, the
   barn limit is the bottleneck. If free_slowpath dominates and the
   sheaf counters are quiet, the frees are bypassing sheaves
   entirely and my analysis is wrong.

3. Run perf lock contention with -l on the same window so the lock
   shows up by address, and note whether the callers under
   list_lock are sheaf_flush_unused / __kmem_cache_free_bulk or
   direct __slab_free from kmem_cache_free.

4. Tell me the I/O pattern: direct reads, I assume, but the I/O
   size, number of threads, and how they are spread across the two
   nodes. I would like to attempt to reproduce the
   alloc-here-free-there imbalance rather than speculate.

I don't have a NUMA system in my lab, so I cannot recreate the
per-node barn directly. What I do have is a 24-core/48-thread Ryzen,
which is a single node but has several CCDs. The per-CPU part of the
imbalance exists without NUMA. If barn_put_fail climbs there too, I
would have a reproducer that does not need specialist hardware.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)

[PATCH RFC 7/8] NFS: Reduce nfsiod workqueue contention

[Chuck Lever]

The default affinity scope for unbound workqueues is now
WQ_AFFN_CACHE_SHARD, which splits each LLC into shards of about
eight cores. On a single-socket system whose LLC fits in one
shard, every NFS I/O completion serializes on one nfsiod pool
lock. Profiling 4KB random writes over NFSv3/RDMA with nconnect=3
shows that lock consuming 17% of CPU cycles: 8% dequeuing work and
9% enqueuing follow-on work from rpciod and nfsiod workers.

Set nfsiod's affinity scope to WQ_AFFN_SMT so each CPU gets its
own pool and queue_work_on() no longer takes a lock on another
CPU. Enqueue contention disappears and dequeue contention drops
to 1.4%. Throughput is unchanged because the workload is
transport-limited, but the freed cycles cut submission latency
variance by 67% (slat stdev 31.6 us to 10.5 us), IOPS stdev by
31%, and p99.9 completion latency by 11%.

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Chuck Lever <cel@kernel.org>
---
 fs/nfs/inode.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c
index 107a2135029d..8e0d2bebba54 100644
--- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c
@@ -2618,11 +2618,26 @@ static void nfsiod_stop(void)
  */
 static int nfsiod_start(void)
 {
+	struct workqueue_attrs *attrs;
+
 	dprintk("RPC:       creating workqueue nfsiod\n");
 	nfsiod_workqueue = alloc_workqueue("nfsiod",
 				WQ_MEM_RECLAIM | WQ_UNBOUND | WQ_SYSFS, 0);
 	if (nfsiod_workqueue == NULL)
 		return -ENOMEM;
+	attrs = alloc_workqueue_attrs();
+	if (attrs) {
+		int err;
+
+		attrs->affn_scope = WQ_AFFN_SMT;
+		err = apply_workqueue_attrs(nfsiod_workqueue, attrs);
+		free_workqueue_attrs(attrs);
+		if (err)
+			pr_warn("nfsiod: failed to set SMT affinity scope: %d\n",
+				err);
+	} else {
+		pr_warn("nfsiod: failed to allocate workqueue attrs\n");
+	}
 #if IS_ENABLED(CONFIG_NFS_LOCALIO)
 	/*
 	 * localio writes need to use a normal (non-memreclaim) workqueue.

-- 
2.55.0

[PATCH RFC 8/8] SUNRPC: Reduce xprtiod workqueue contention

[Chuck Lever]

xprtiod handles transport-level operations: socket receive
processing, error recovery, and connection lifecycle. On systems
driving heavy NFS traffic, these operations contend on the UNBOUND
worker pool lock just as rpciod does.

Set WQ_AFFN_SMT on the xprtiod workqueue to give each SMT group its
own pool and lock.

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Chuck Lever <cel@kernel.org>
---
 net/sunrpc/sched.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
index b84af2c0b104..3236dfdb4096 100644
--- a/net/sunrpc/sched.c
+++ b/net/sunrpc/sched.c
@@ -1304,6 +1304,7 @@ static int rpciod_start(void)
 	wq = alloc_workqueue("xprtiod", wq_flags, 0);
 	if (!wq)
 		goto free_rpciod;
+	rpc_set_wq_smt_affinity(wq, "xprtiod");
 	xprtiod_workqueue = wq;
 	return 1;
 free_rpciod:

-- 
2.55.0