[PATCH RFC 0/8] Reduce lock contention in the NFS client

[PATCH RFC 0/8] Reduce lock contention in the NFS client

[Chuck Lever]

Under a 4KB NFSv3 workload on 100GbE RDMA, roughly 150 RPC worker
threads drive the client, and lock contention dominates its CPU
profile: up to 53% of non-idle cycles are spent in
native_queued_spin_lock_slowpath.

An earlier RFC put this fix in the workqueue core, falling back
automatically to a finer scope when the configured scope
degenerates to a single pod. Tejun rejected a finer default for
unbound workqueues and suggested two alternatives: a sharded
scope between CACHE and SMT, and a scope change on the NFS
workqueues themselves [1]. The first became the
WQ_AFFN_CACHE_SHARD default, whose 8-core shards still leave a
12-core single-LLC system with only two pools. This series does
the second.

My test harness is transport-limited, so the freed cycles appear as
reduced submission latency variance and lower tail completion
latency rather than higher IOPS.

[1] https://lore.kernel.org/all/aYUVVuIidMpuYy3j@slm.duckdns.org/

---
Chuck Lever (8):
      SUNRPC: Use atomic_t for XID allocation
      SUNRPC: Execute initial async RPC states in caller's context
      SUNRPC: Split recv_lock out of xprt->queue_lock
      Set WQ_SYSFS on key NFS-related workqueues
      workqueue: Export the functions needed for WQ attribute modification
      SUNRPC: Reduce rpciod workqueue contention
      NFS: Reduce nfsiod workqueue contention
      SUNRPC: Reduce xprtiod workqueue contention

 fs/nfs/inode.c                             | 18 ++++++++-
 include/linux/sunrpc/xprt.h                |  8 +++-
 kernel/workqueue.c                         |  3 ++
 net/sunrpc/sched.c                         | 42 ++++++++++++++------
 net/sunrpc/svcsock.c                       |  6 +--
 net/sunrpc/xprt.c                          | 63 +++++++++++++++---------------
 net/sunrpc/xprtrdma/rpc_rdma.c             | 14 +++----
 net/sunrpc/xprtrdma/svc_rdma_backchannel.c |  8 ++--
 net/sunrpc/xprtsock.c                      | 18 ++++-----
 9 files changed, 111 insertions(+), 69 deletions(-)
---
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
change-id: 20260831-performance-e465e621c1c0

Best regards,
--  
Chuck Lever <cel@kernel.org>

[PATCH RFC 1/8] SUNRPC: Use atomic_t for XID allocation

[Chuck Lever]

xprt_alloc_xid() acquires reserve_lock to increment a simple
counter. Under a high-IOPS NFSv3 workload on 100GbE RDMA,
profiling shows 1.06% of system-wide CPU cycles contending on
this lock in xprt_request_init, as ~150 RPC worker threads
serialize on the counter.

reserve_lock protects the slot table and backlog queue, but
XID allocation is an independent operation that does not require
synchronization with either.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 include/linux/sunrpc/xprt.h | 2 +-
 net/sunrpc/xprt.c           | 9 ++-------
 2 files changed, 3 insertions(+), 8 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index a82045804d34..0d6c3f6bf97e 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -273,7 +273,7 @@ struct rpc_xprt {
 	spinlock_t		transport_lock;	/* lock transport info */
 	spinlock_t		reserve_lock;	/* lock slot table */
 	spinlock_t		queue_lock;	/* send/receive queue lock */
-	u32			xid;		/* Next XID value to use */
+	atomic_t		xid;		/* Most recently issued XID */
 	struct rpc_task *	snd_task;	/* Task blocked in send */
 
 	struct list_head	xmit_queue;	/* Send queue */
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 48a3618cbb29..186c14f0f928 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1882,18 +1882,13 @@ xprt_init_connect_cookie(struct rpc_rqst *req, struct rpc_xprt *xprt)
 static __be32
 xprt_alloc_xid(struct rpc_xprt *xprt)
 {
-	__be32 xid;
-
-	spin_lock(&xprt->reserve_lock);
-	xid = (__force __be32)xprt->xid++;
-	spin_unlock(&xprt->reserve_lock);
-	return xid;
+	return (__force __be32)atomic_inc_return(&xprt->xid);
 }
 
 static void
 xprt_init_xid(struct rpc_xprt *xprt)
 {
-	xprt->xid = get_random_u32();
+	atomic_set(&xprt->xid, get_random_u32());
 }
 
 static void

-- 
2.55.0

[PATCH RFC 2/8] SUNRPC: Execute initial async RPC states in caller's context

[Chuck Lever]

Each async RPC task is dispatched to the rpciod workqueue before its
first FSM state runs, though the initial states (call_start through
call_allocate) typically complete without blocking. Under high
concurrency this adds a workqueue enqueue, dequeue, and context
switch per RPC and contributes to rpciod pool lock contention.

Have rpc_execute() call __rpc_execute() directly for async tasks as
well. The initial states then run in the caller's context, and
__rpc_execute() returns to the caller when the task first sleeps on
a wait queue. Later wake-ups still dispatch to rpciod via
rpc_make_runnable().

Commit d6a1ed08c6ac ("SUNRPC: Reduce asynchronous RPC task stack
usage") moved async dispatch onto rpciod to bound caller stack
depth. Inline execution preserves that bound because __rpc_execute()
still runs each state from the same stack frame.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 net/sunrpc/sched.c | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)

diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
index 016f16ca5779..9b2800f6bfc7 100644
--- a/net/sunrpc/sched.c
+++ b/net/sunrpc/sched.c
@@ -1003,8 +1003,9 @@ static void __rpc_execute(struct rpc_task *task)
 	current_restore_flags(pflags, PF_MEMALLOC);
 }
 
-/*
- * User-visible entry point to the scheduler.
+/**
+ * rpc_execute - Consumer entry point to the RPC scheduler
+ * @task: RPC task to be scheduled
  *
  * This may be called recursively if e.g. an async NFS task updates
  * the attributes and finds that dirty pages must be flushed.
@@ -1014,15 +1015,12 @@ static void __rpc_execute(struct rpc_task *task)
  */
 void rpc_execute(struct rpc_task *task)
 {
-	bool is_async = RPC_IS_ASYNC(task);
+	unsigned int pflags = memalloc_nofs_save();
 
 	rpc_set_active(task);
-	rpc_make_runnable(rpciod_workqueue, task);
-	if (!is_async) {
-		unsigned int pflags = memalloc_nofs_save();
-		__rpc_execute(task);
-		memalloc_nofs_restore(pflags);
-	}
+	rpc_test_and_set_running(task);
+	__rpc_execute(task);
+	memalloc_nofs_restore(pflags);
 }
 
 static void rpc_async_schedule(struct work_struct *work)

-- 
2.55.0

[PATCH RFC 3/8] SUNRPC: Split recv_lock out of xprt->queue_lock

[Chuck Lever]

xprt->queue_lock protects two independent structures: the recv_queue
rb-tree for reply matching and the xmit_queue list for transmit
draining. No hot path touches both in one critical section, yet every
RPC submit and completion contends on the same lock. Under a 4KB
NFSv3 READ workload on 100GbE RDMA, 53% of non-idle CPU cycles are
spent in native_queued_spin_lock_slowpath: the CQ completion worker
running rpcrdma_reply_handler serializes against ~150 kworker threads
enqueuing receives and transmits.

Introduce xprt->recv_lock for the receive path -- recv_queue
operations, request lookup, receive-side pinning, and completion --
leaving queue_lock to the xmit_queue and the xprt_transmit drain
loop.

Also move the rq_private_buf memcpy in xprt_request_enqueue_receive
above the lock acquisition: until the rb-tree insert publishes the
request, the reply handler cannot see it, so the copy is safe
unlocked and the submitter's critical section shrinks to the insert
alone.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 include/linux/sunrpc/xprt.h                |  6 +++-
 net/sunrpc/svcsock.c                       |  6 ++--
 net/sunrpc/xprt.c                          | 54 ++++++++++++++++--------------
 net/sunrpc/xprtrdma/rpc_rdma.c             | 14 ++++----
 net/sunrpc/xprtrdma/svc_rdma_backchannel.c |  8 ++---
 net/sunrpc/xprtsock.c                      | 18 +++++-----
 6 files changed, 57 insertions(+), 49 deletions(-)

diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index 0d6c3f6bf97e..ed1e28b74f02 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -272,7 +272,7 @@ struct rpc_xprt {
 	atomic_long_t		queuelen;
 	spinlock_t		transport_lock;	/* lock transport info */
 	spinlock_t		reserve_lock;	/* lock slot table */
-	spinlock_t		queue_lock;	/* send/receive queue lock */
+	spinlock_t		queue_lock;	/* send queue lock */
 	atomic_t		xid;		/* Most recently issued XID */
 	struct rpc_task *	snd_task;	/* Task blocked in send */
 
@@ -292,6 +292,10 @@ struct rpc_xprt {
 						 * backchannel rpc_rqst's */
 #endif /* CONFIG_SUNRPC_BACKCHANNEL */
 
+	/*
+	 * Receive stuff
+	 */
+	spinlock_t		recv_lock;	/* receive queue lock */
 	struct rb_root		recv_queue;	/* Receive queue */
 
 	struct {
diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index 50e5e7f5b762..8939ba604385 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -1102,7 +1102,7 @@ static int receive_cb_reply(struct svc_sock *svsk, struct svc_rqst *rqstp)
 
 	if (!bc_xprt)
 		return -EAGAIN;
-	spin_lock(&bc_xprt->queue_lock);
+	spin_lock(&bc_xprt->recv_lock);
 	req = xprt_lookup_rqst(bc_xprt, xid);
 	if (!req)
 		goto unlock_eagain;
@@ -1120,10 +1120,10 @@ static int receive_cb_reply(struct svc_sock *svsk, struct svc_rqst *rqstp)
 	memcpy(dst->iov_base, src->iov_base, src->iov_len);
 	xprt_complete_rqst(req->rq_task, rqstp->rq_arg.len);
 	rqstp->rq_arg.len = 0;
-	spin_unlock(&bc_xprt->queue_lock);
+	spin_unlock(&bc_xprt->recv_lock);
 	return 0;
 unlock_eagain:
-	spin_unlock(&bc_xprt->queue_lock);
+	spin_unlock(&bc_xprt->recv_lock);
 	return -EAGAIN;
 }
 
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 186c14f0f928..42c66464d8f3 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1061,7 +1061,7 @@ xprt_request_rb_remove(struct rpc_xprt *xprt, struct rpc_rqst *req)
  * @xprt: transport on which the original request was transmitted
  * @xid: RPC XID of incoming reply
  *
- * Caller holds xprt->queue_lock.
+ * Caller holds xprt->recv_lock.
  */
 struct rpc_rqst *xprt_lookup_rqst(struct rpc_xprt *xprt, __be32 xid)
 {
@@ -1092,8 +1092,9 @@ xprt_is_pinned_rqst(struct rpc_rqst *req)
  * xprt_pin_rqst - Pin a request on the transport receive list
  * @req: Request to pin
  *
- * Caller must ensure this is atomic with the call to xprt_lookup_rqst()
- * so should be holding xprt->queue_lock.
+ * Caller must hold the lock that protects the queue through which
+ * it found the request: xprt->recv_lock for the receive path,
+ * xprt->queue_lock for the transmit drain path.
  */
 void xprt_pin_rqst(struct rpc_rqst *req)
 {
@@ -1105,14 +1106,10 @@ EXPORT_SYMBOL_GPL(xprt_pin_rqst);
  * xprt_unpin_rqst - Unpin a request on the transport receive list
  * @req: Request to pin
  *
- * Caller should be holding xprt->queue_lock.
+ * Caller holds the lock it held for the matching xprt_pin_rqst().
  */
 void xprt_unpin_rqst(struct rpc_rqst *req)
 {
-	if (!test_bit(RPC_TASK_MSG_PIN_WAIT, &req->rq_task->tk_runstate)) {
-		atomic_dec(&req->rq_pin);
-		return;
-	}
 	if (atomic_dec_and_test(&req->rq_pin))
 		wake_up_var(&req->rq_pin);
 }
@@ -1155,16 +1152,16 @@ xprt_request_enqueue_receive(struct rpc_task *task)
 	ret = xprt_request_prepare(task->tk_rqstp, &req->rq_rcv_buf);
 	if (ret)
 		return ret;
-	spin_lock(&xprt->queue_lock);
-
-	/* Update the softirq receive buffer */
+	/* Reply handlers cannot find the request until the rb-tree
+	 * insert below publishes it, so the copy needs no lock.
+	 */
 	memcpy(&req->rq_private_buf, &req->rq_rcv_buf,
 			sizeof(req->rq_private_buf));
 
-	/* Add request to the receive list */
+	spin_lock(&xprt->recv_lock);
 	xprt_request_rb_insert(xprt, req);
 	set_bit(RPC_TASK_NEED_RECV, &task->tk_runstate);
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 
 	/* Turn off autodisconnect */
 	timer_delete_sync(&xprt->timer);
@@ -1175,7 +1172,7 @@ xprt_request_enqueue_receive(struct rpc_task *task)
  * xprt_request_dequeue_receive_locked - Remove a request from the receive queue
  * @task: RPC task
  *
- * Caller must hold xprt->queue_lock.
+ * Caller must hold xprt->recv_lock.
  */
 static void
 xprt_request_dequeue_receive_locked(struct rpc_task *task)
@@ -1190,7 +1187,7 @@ xprt_request_dequeue_receive_locked(struct rpc_task *task)
  * xprt_update_rtt - Update RPC RTT statistics
  * @task: RPC request that recently completed
  *
- * Caller holds xprt->queue_lock.
+ * Caller holds xprt->recv_lock.
  */
 void xprt_update_rtt(struct rpc_task *task)
 {
@@ -1212,7 +1209,7 @@ EXPORT_SYMBOL_GPL(xprt_update_rtt);
  * @task: RPC request that recently completed
  * @copied: actual number of bytes received from the transport
  *
- * Caller holds xprt->queue_lock.
+ * Caller holds xprt->recv_lock.
  */
 void xprt_complete_rqst(struct rpc_task *task, int copied)
 {
@@ -1309,7 +1306,7 @@ void xprt_request_wait_receive(struct rpc_task *task)
 	 * The spinlock ensures atomicity between the test of
 	 * req->rq_reply_bytes_recvd, and the call to rpc_sleep_on().
 	 */
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	if (test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate)) {
 		xprt->ops->wait_for_reply_request(task);
 		/*
@@ -1321,7 +1318,7 @@ void xprt_request_wait_receive(struct rpc_task *task)
 			rpc_wake_up_queued_task_set_status(&xprt->pending,
 					task, -ENOTCONN);
 	}
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 }
 
 static bool
@@ -1439,7 +1436,11 @@ xprt_request_dequeue_transmit(struct rpc_task *task)
  * @task: pointer to rpc_task
  *
  * Remove a task from the transmit and receive queues, and ensure that
- * it is not pinned by the receive work item.
+ * it is not pinned by any concurrent work item.
+ *
+ * Dequeuing from both queues prevents new pins: xprt_lookup_rqst
+ * and xprt_transmit can no longer find the request. The wait for
+ * in-flight pins to drain then needs neither lock.
  */
 void
 xprt_request_dequeue_xprt(struct rpc_task *task)
@@ -1451,16 +1452,18 @@ xprt_request_dequeue_xprt(struct rpc_task *task)
 	    test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate) ||
 	    xprt_is_pinned_rqst(req)) {
 		spin_lock(&xprt->queue_lock);
-		while (xprt_is_pinned_rqst(req)) {
+		xprt_request_dequeue_transmit_locked(task);
+		spin_unlock(&xprt->queue_lock);
+
+		spin_lock(&xprt->recv_lock);
+		xprt_request_dequeue_receive_locked(task);
+		spin_unlock(&xprt->recv_lock);
+
+		if (xprt_is_pinned_rqst(req)) {
 			set_bit(RPC_TASK_MSG_PIN_WAIT, &task->tk_runstate);
-			spin_unlock(&xprt->queue_lock);
 			xprt_wait_on_pinned_rqst(req);
-			spin_lock(&xprt->queue_lock);
 			clear_bit(RPC_TASK_MSG_PIN_WAIT, &task->tk_runstate);
 		}
-		xprt_request_dequeue_transmit_locked(task);
-		xprt_request_dequeue_receive_locked(task);
-		spin_unlock(&xprt->queue_lock);
 		xdr_free_bvec(&req->rq_rcv_buf);
 	}
 }
@@ -2038,6 +2041,7 @@ static void xprt_init(struct rpc_xprt *xprt, struct net *net)
 	spin_lock_init(&xprt->transport_lock);
 	spin_lock_init(&xprt->reserve_lock);
 	spin_lock_init(&xprt->queue_lock);
+	spin_lock_init(&xprt->recv_lock);
 
 	INIT_LIST_HEAD(&xprt->free);
 	xprt->recv_queue = RB_ROOT;
diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c
index 1285f04cdac1..a82d3d9bc7ae 100644
--- a/net/sunrpc/xprtrdma/rpc_rdma.c
+++ b/net/sunrpc/xprtrdma/rpc_rdma.c
@@ -1321,9 +1321,9 @@ void rpcrdma_unpin_rqst(struct rpcrdma_rep *rep)
 	req->rl_reply = NULL;
 	rep->rr_rqst = NULL;
 
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	xprt_unpin_rqst(rqst);
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 }
 
 /**
@@ -1363,10 +1363,10 @@ void rpcrdma_complete_rqst(struct rpcrdma_rep *rep)
 		goto out_badheader;
 
 out:
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	xprt_complete_rqst(rqst->rq_task, status);
 	xprt_unpin_rqst(rqst);
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 	return;
 
 out_badheader:
@@ -1492,12 +1492,12 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep)
 	/* Match incoming rpcrdma_rep to an rpcrdma_req to
 	 * get context for handling any incoming chunks.
 	 */
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	rqst = xprt_lookup_rqst(xprt, rep->rr_xid);
 	if (!rqst)
 		goto out_norqst;
 	xprt_pin_rqst(rqst);
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 
 	if (buf->rb_credits != credits)
 		rpcrdma_update_cwnd(r_xprt, credits);
@@ -1524,7 +1524,7 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep)
 	return;
 
 out_norqst:
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 	trace_xprtrdma_reply_rqst_err(rep);
 	rpcrdma_rep_put(buf, rep);
 	goto out_post;
diff --git a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
index e5a78b761012..3c7b85427f33 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
@@ -28,7 +28,7 @@ void svc_rdma_handle_bc_reply(struct svc_rqst *rqstp,
 	struct rpc_rqst *req;
 	u32 credits;
 
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	req = xprt_lookup_rqst(xprt, *rdma_resp);
 	if (!req)
 		goto out_unlock;
@@ -39,7 +39,7 @@ void svc_rdma_handle_bc_reply(struct svc_rqst *rqstp,
 		goto out_unlock;
 	memcpy(dst->iov_base, src->iov_base, src->iov_len);
 	xprt_pin_rqst(req);
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 
 	credits = be32_to_cpup(rdma_resp + 2);
 	if (credits == 0)
@@ -50,13 +50,13 @@ void svc_rdma_handle_bc_reply(struct svc_rqst *rqstp,
 	xprt->cwnd = credits << RPC_CWNDSHIFT;
 	spin_unlock(&xprt->transport_lock);
 
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	xprt_complete_rqst(req->rq_task, rcvbuf->len);
 	xprt_unpin_rqst(req);
 	rcvbuf->len = 0;
 
 out_unlock:
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 }
 
 /* Send a reverse-direction RPC Call.
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 7f60723fa64d..1454da9575b3 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -673,25 +673,25 @@ xs_read_stream_reply(struct sock_xprt *transport, struct msghdr *msg, int flags)
 	ssize_t ret = 0;
 
 	/* Look up and lock the request corresponding to the given XID */
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	req = xprt_lookup_rqst(xprt, transport->recv.xid);
 	if (!req || (transport->recv.copied && !req->rq_private_buf.len)) {
 		msg->msg_flags |= MSG_TRUNC;
 		goto out;
 	}
 	xprt_pin_rqst(req);
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 
 	ret = xs_read_stream_request(transport, msg, flags, req);
 
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	if (msg->msg_flags & (MSG_EOR|MSG_TRUNC))
 		xprt_complete_rqst(req->rq_task, transport->recv.copied);
 	else
 		req->rq_private_buf.len = transport->recv.copied;
 	xprt_unpin_rqst(req);
 out:
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 	return ret;
 }
 
@@ -1398,13 +1398,13 @@ static void xs_udp_data_read_skb(struct rpc_xprt *xprt,
 		return;
 
 	/* Look up and lock the request corresponding to the given XID */
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	rovr = xprt_lookup_rqst(xprt, *xp);
 	if (!rovr)
 		goto out_unlock;
 	xprt_pin_rqst(rovr);
 	xprt_update_rtt(rovr->rq_task);
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 	task = rovr->rq_task;
 
 	if ((copied = rovr->rq_private_buf.buflen) > repsize)
@@ -1412,7 +1412,7 @@ static void xs_udp_data_read_skb(struct rpc_xprt *xprt,
 
 	/* Suck it into the iovec, verify checksum if not done by hw. */
 	if (csum_partial_copy_to_xdr(&rovr->rq_private_buf, skb)) {
-		spin_lock(&xprt->queue_lock);
+		spin_lock(&xprt->recv_lock);
 		__UDPX_INC_STATS(sk, UDP_MIB_INERRORS);
 		goto out_unpin;
 	}
@@ -1421,13 +1421,13 @@ static void xs_udp_data_read_skb(struct rpc_xprt *xprt,
 	spin_lock(&xprt->transport_lock);
 	xprt_adjust_cwnd(xprt, task, copied);
 	spin_unlock(&xprt->transport_lock);
-	spin_lock(&xprt->queue_lock);
+	spin_lock(&xprt->recv_lock);
 	xprt_complete_rqst(task, copied);
 	__UDPX_INC_STATS(sk, UDP_MIB_INDATAGRAMS);
 out_unpin:
 	xprt_unpin_rqst(rovr);
  out_unlock:
-	spin_unlock(&xprt->queue_lock);
+	spin_unlock(&xprt->recv_lock);
 }
 
 static void xs_udp_data_receive(struct sock_xprt *transport)

-- 
2.55.0

[PATCH RFC 4/8] Set WQ_SYSFS on key NFS-related workqueues

[Chuck Lever]

WQ_SYSFS exposes settable workqueue attributes under
/sys/devices/virtual/workqueue/. For NFS workloads, this is useful
for tuning affinity scope at runtime without reloading modules.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 fs/nfs/inode.c     | 3 ++-
 net/sunrpc/sched.c | 5 +++--
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c
index 3022454f7698..107a2135029d 100644
--- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c
@@ -2619,7 +2619,8 @@ static void nfsiod_stop(void)
 static int nfsiod_start(void)
 {
 	dprintk("RPC:       creating workqueue nfsiod\n");
-	nfsiod_workqueue = alloc_workqueue("nfsiod", WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
+	nfsiod_workqueue = alloc_workqueue("nfsiod",
+				WQ_MEM_RECLAIM | WQ_UNBOUND | WQ_SYSFS, 0);
 	if (nfsiod_workqueue == NULL)
 		return -ENOMEM;
 #if IS_ENABLED(CONFIG_NFS_LOCALIO)
diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
index 9b2800f6bfc7..c31cf55b933f 100644
--- a/net/sunrpc/sched.c
+++ b/net/sunrpc/sched.c
@@ -1271,16 +1271,17 @@ void rpciod_down(void)
  */
 static int rpciod_start(void)
 {
+	const unsigned int wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND | WQ_SYSFS;
 	struct workqueue_struct *wq;
 
 	/*
 	 * Create the rpciod thread and wait for it to start.
 	 */
-	wq = alloc_workqueue("rpciod", WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
+	wq = alloc_workqueue("rpciod", wq_flags, 0);
 	if (!wq)
 		goto out_failed;
 	rpciod_workqueue = wq;
-	wq = alloc_workqueue("xprtiod", WQ_UNBOUND | WQ_MEM_RECLAIM, 0);
+	wq = alloc_workqueue("xprtiod", wq_flags, 0);
 	if (!wq)
 		goto free_rpciod;
 	xprtiod_workqueue = wq;

-- 
2.55.0

[PATCH RFC 5/8] workqueue: Export the functions needed for WQ attribute modification

[Chuck Lever]

Modules that create unbound workqueues currently have no way
to modify workqueue attributes at run time. Export the
allocation, application, and teardown functions so that
modules such as sunrpc can adjust affinity scope without
reloading.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 kernel/workqueue.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 3c034cbc5bb3..fd76a875cf2d 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -4813,6 +4813,7 @@ void free_workqueue_attrs(struct workqueue_attrs *attrs)
 		kfree(attrs);
 	}
 }
+EXPORT_SYMBOL_GPL(free_workqueue_attrs);
 
 /**
  * alloc_workqueue_attrs - allocate a workqueue_attrs
@@ -4841,6 +4842,7 @@ struct workqueue_attrs *alloc_workqueue_attrs_noprof(void)
 	free_workqueue_attrs(attrs);
 	return NULL;
 }
+EXPORT_SYMBOL_GPL(alloc_workqueue_attrs_noprof);
 
 static void copy_workqueue_attrs(struct workqueue_attrs *to,
 				 const struct workqueue_attrs *from)
@@ -5609,6 +5611,7 @@ int apply_workqueue_attrs(struct workqueue_struct *wq,
 
 	return ret;
 }
+EXPORT_SYMBOL_GPL(apply_workqueue_attrs);
 
 /**
  * unbound_wq_update_pwq - update a pwq slot for CPU hot[un]plug

-- 
2.55.0

[PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

rpciod drives the RPC client state machine. Under heavy NFS
workloads, multiple CPUs queue RPC task completions concurrently and
contend on the UNBOUND worker pool lock. perf profiles on a 12-core
system show 30-40% of cycles lost to
native_queued_spin_lock_slowpath in the rpciod pool at the
WQ_AFFN_CACHE scope (one pool per LLC). The WQ_AFFN_CACHE_SHARD
default helps little here, because its 8-core shards split this
system into just two pools of six cores each.

Set WQ_AFFN_SMT on rpciod so each SMT group gets its own pool and
lock. Most UNBOUND workqueues never contend on the pool lock and
profit from a coarser scope's cache locality. rpciod's sustained
completion traffic makes the lock a first-order bottleneck, so the
override belongs on this workqueue rather than in the system-wide
default. Idle kworkers are culled, so the extra pools cost little on
large systems.

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Chuck Lever <cel@kernel.org>
---
 net/sunrpc/sched.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
index c31cf55b933f..b84af2c0b104 100644
--- a/net/sunrpc/sched.c
+++ b/net/sunrpc/sched.c
@@ -1266,6 +1266,25 @@ void rpciod_down(void)
 	module_put(THIS_MODULE);
 }
 
+static void rpc_set_wq_smt_affinity(struct workqueue_struct *wq,
+				    const char *name)
+{
+	struct workqueue_attrs *attrs;
+	int err;
+
+	attrs = alloc_workqueue_attrs();
+	if (!attrs) {
+		pr_warn("%s: failed to allocate workqueue attrs\n", name);
+		return;
+	}
+	attrs->affn_scope = WQ_AFFN_SMT;
+	err = apply_workqueue_attrs(wq, attrs);
+	free_workqueue_attrs(attrs);
+	if (err)
+		pr_warn("%s: failed to set SMT affinity scope: %d\n",
+			name, err);
+}
+
 /*
  * Start up the rpciod workqueue.
  */
@@ -1280,6 +1299,7 @@ static int rpciod_start(void)
 	wq = alloc_workqueue("rpciod", wq_flags, 0);
 	if (!wq)
 		goto out_failed;
+	rpc_set_wq_smt_affinity(wq, "rpciod");
 	rpciod_workqueue = wq;
 	wq = alloc_workqueue("xprtiod", wq_flags, 0);
 	if (!wq)

-- 
2.55.0

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Tim Menninger]

I am seeing a significant throughput regression from the rpciod SMT
affinity change on a high-throughput NFS/RDMA workload.

This is a 96-CPU, two-socket system with 48 physical cores (2 threads per
core). I bisected the regression to the patch that changes rpciod to use
WQ_AFFN_SMT.

With the default cache_shard scope, the workload sustains approximately 45
GB/s. With the SMT scope, some runs fall to approximately 15-25 GB/s.

The failure is intermittent across workload starts and appears easier to
reproduce shortly after boot. However, once I have a bad run, the
dependency on the rpciod affinity scope is reproducible without restarting
the workload.

For example, during one continuously running workload with the regression
actively reproducing, throughput recovers to ~45 GB/s immediately when I
change /sys/bus/workqueue/devices/rpciod/affinity_scope to cache_shard,
then regresses again immediately when I restore smt.

Nothing else about the workload, mount, RPC connections, or RDMA
connections is changed between those transitions.

On this machine, wq_dump.py reports:

    SMT:          48 affinity pods
    CACHE_SHARD:   6 affinity pods

The SMT pods correspond to one physical core / two sibling CPUs, while each
cache_shard pod contains eight physical cores / sixteen logical CPUs.

I have not yet identified the exact mechanism that causes the SMT
configuration to lose throughput, so I don't want to speculate about the
specific lock or scheduler interaction involved. But the live smt ->
cache_shard -> smt transition seems to isolate the regression to this
affinity-scope change.

Given the magnitude of the regression, I think this needs to be understood
before the rpciod SMT affinity change is merged.

I can collect additional workqueue or scheduler traces if there is
something specific that would help characterize why the SMT scope performs
poorly here.

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

On Wed, Sep 2, 2026, at 4:40 PM, Tim Menninger wrote:
> I am seeing a significant throughput regression from the rpciod SMT
> affinity change on a high-throughput NFS/RDMA workload.

Thanks for the report!

I'd like to learn a bit more about the setup before drawing
any firm conclusions.

1. Does v2 of the series behave the same way? v2 dropped the
   patch that ran the first RPC states in the submitter's
   context, so the rpciod traffic pattern differs from v1.
   The same smt -> cache_shard -> smt toggle on v2 would tell
   us whether that matters.
  
2. How are the RDMA device's completion interrupts placed?
   Please share, on a bad run:
   - /proc/interrupts lines for the device's completion vectors
   - the smp_affinity_list for each of those IRQs
   - whether irqbalance is running, and whether you've pinned
     the IRQs by hand

3. Where is CPU time going on a bad run versus a good one?
   A short capture of each would help:
     perf record -a -g -- sleep 10
     perf report --sort comm,cpu --stdio | head -80
   In particular I'm interested in which CPUs the kworker
   threads for rpciod and the ib-comp-wq threads run on under
   each scope.

4. tools/workqueue/wq_monitor.py rpciod, sampled for a few
   seconds under each scope, would show whether the pools are
   evenly loaded.

5. The workload itself: thread count, I/O size and direction,
   number of mounts and RDMA connections, and the mount
   options (nconnect in particular).

6. The base kernel the series was applied to, and the RDMA
   device and driver.
  
If you have a way to capture a bad run reliably shortly after
boot, a "before and after" of items 2 through 4 across a single
scope toggle would be the most direct evidence.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Tim Menninger]

Thanks. I did some more testing with v2 and collected the requested
information.

> 1. Does v2 of the series behave the same way? v2 dropped the
>    patch that ran the first RPC states in the submitter's
>    context, so the rpciod traffic pattern differs from v1.
>    The same smt -> cache_shard -> smt toggle on v2 would tell
>    us whether that matters.

The v2 series behaves similarly insofar as some runs see ~15 GB/s and
other runs see ~45 GB/s, but the low-throughput state no longer appears
to be directly controlled by the rpciod affinity scope.

I have started runs with both smt and cache_shard, and with both scopes
I have seen all three of:

    1. start and remain at ~45 GB/s
    2. start and remain at ~15 GB/s
    3. start at ~45 GB/s, then abruptly drop to ~15 GB/s

I have not found a discernible pattern for how long a run remains at
~45 GB/s before dropping.

cache_shard does still seem somewhat more likely to give me a ~45 GB/s
run, particularly as uptime increases, but unlike v1 I can reproduce
both good and bad runs with either scope. The deterministic live
smt -> cache_shard -> smt behavior I reported for v1 is no longer
present in v2.

So the low-throughput state remains with v2, but it no longer seems to be
from the SMT affinity change alone.

All captures below are from separate runs with the scope set as indicated.

> 2. How are the RDMA device's completion interrupts placed?
>    Please share, on a bad run:
>
>    * /proc/interrupts lines for the device's completion vectors
>    * the smp_affinity_list for each of those IRQs
>    * whether irqbalance is running, and whether you've pinned
>      the IRQs by hand

There are two ConnectX-7 devices:

    mlx5_0 port 1 ==> ens3np0
    mlx5_1 port 1 ==> ens6np0

at PCI addresses 0000:2a:00.0 and 0000:ab:00.0 respectively.

The substantial completion traffic in these captures is on
0000:2a:00.0. Its mlx5 completion IRQs are individually affinitized to
CPUs. For example:

    mlx5_comp0   -> CPU 0
    mlx5_comp1   -> CPU 1
    ...
    mlx5_comp23  -> CPU 23
    mlx5_comp24  -> CPU 48
    ...
    mlx5_comp47  -> CPU 71
    mlx5_comp48  -> CPU 24
    ...
    mlx5_comp62  -> CPU 38

The affinity mapping was the same in the good and bad captures I took.

I have not manually pinned any IRQs, and irqbalance is inactive.

I have the complete /proc/interrupts and smp_affinity_list captures
available if there are particular vectors or deltas that would be useful
to see.

> 3. Where is CPU time going on a bad run versus a good one?
>    A short capture of each would help:
>    perf record -a -g -- sleep 10
>    perf report --sort comm,cpu --stdio | head -80
>    In particular I'm interested in which CPUs the kworker
>    threads for rpciod and the ib-comp-wq threads run on under
>    each scope.

The perf results aren't showing anything useful yet. With the requested
--sort comm,cpu, the top of the report is dominated by perf itself. Without
that sort, the top entries are things like cpuidle_enter_state and
cpuidle_enter.

I'll keep working on the perf capture and follow up if/when I get something
useful, but I didn't want to hold up the rest of this on that.

> 4. tools/workqueue/wq_monitor.py rpciod, sampled for a few
>    seconds under each scope, would show whether the pools are
>    evenly loaded.

I captured rpciod monitoring data from four v2 runs. Here are windows from
each.

cache_shard good (~46 GB/s):
                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        27613426     3    534.5       - 1699488      16       0 

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        27817822     5    537.3       - 1713539      16       0 

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        28027937     2    540.1       - 1728063      16       0 

cache_shard bad (~15 GB/s):
                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        71754056     2    932.1       - 4767531       0       0 

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        71825047     0    933.9       - 4771333       0       0 

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        71901102     3    935.8       - 4775677       0       0 

smt good (~46 GB/s):
                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        17036137     2    144.9       - 4993757       0       0

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        17220514     0    146.8       - 5064827       0       0

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod        17397194     3    148.8       - 5132231       0       0

smt bad (~26 GB/s):
                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod         4937072     1     43.2       - 1407321       0       0 

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod         5037193     4     44.0       - 1432686       0       0 

                     total  infl  CPUtime CPUitsv CMW/RPR  mayday rescued
    rpciod         5138509     4     44.8       - 1458079       0       0 

> 5. The workload itself: thread count, I/O size and direction,
>    number of mounts and RDMA connections, and the mount
>    options (nconnect in particular).

Workload:
    elbencho --iter 1 --threads 160 --files 16 --size 256G \
        --block 1m --dropcache --iodepth 64 --direct --read \
        --lat --latpercent --log 1 ...

Mount options:
    /home/ir/exapurity from 10.71.61.129:/exapurity
     Flags:	rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,fatal_neterrors=none,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,clientaddr=10.230.36.66,local_lock=none,write=eager,addr=10.71.61.129

It's pNFS over RDMA with one client, one MDS, one DS.

This is a pNFS flexfiles workload. The MDS connection is TCP, and I see
16 RDMA rpc_xprt instances for the data-server traffic during these
tests.

> 6. The base kernel the series was applied to, and the RDMA
>    device and driver.

For all of the data above, I applied the full v2 series on top of:

    940de590b839 Merge tag 'hardening-v7.3-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux

The RDMA hardware is NVIDIA/Mellanox ConnectX-7 (MT2910, PCI ID
15b3:1021), using mlx5_core/mlx5_ib.

For the active interface:

    driver: mlx5_core
    version: 7.3.0-rc1-mainline-bad-v2+
    firmware-version: 28.47.2682 (MT_0000000838)
    bus-info: 0000:2a:00.0

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

On Thu, 3 Sep 2026, Tim Menninger wrote:
> The v2 series behaves similarly insofar as some runs see ~15 GB/s and
> other runs see ~45 GB/s, but the low-throughput state no longer appears
> to be directly controlled by the rpciod affinity scope.

Based on your follow-up, I have a theory about the bimodal performance
behavior:

> The substantial completion traffic in these captures is on
> 0000:2a:00.0. Its mlx5 completion IRQs are individually affinitized to
> CPUs. For example:
>
>     mlx5_comp0   -> CPU 0
>     mlx5_comp1   -> CPU 1
>     ...
>     mlx5_comp23  -> CPU 23
>     mlx5_comp24  -> CPU 48
>     ...
>     mlx5_comp47  -> CPU 71
>     mlx5_comp48  -> CPU 24
>     ...
>     mlx5_comp62  -> CPU 38

That is the mlx5 default spread. It fills the NIC's local NUMA node
first, both SMT threads, and overflows the remaining vectors onto the
other socket. If I read the numbering right, CPUs 0-23 and 48-71 are
the two threads of the node that 0000:2a:00.0 hangs off, and comp48
through comp62 land on the remote socket. Please confirm with:

    cat /sys/bus/pci/devices/0000:2a:00.0/numa_node
    lscpu | grep 'NUMA node'

rpcrdma.ko allocates a send and a receive completion queue for each
transport, and the RDMA core assigns the completion vector from a
host-global round-robin counter. The completion vector is selected at
transport connect time and is fixed for the life of the connection.

Completions on a remote vector run the CQ handler, the RPC reply
processing, and the rpciod wakeup all on the wrong socket. My
suggestion is that you should pin the NIC's interrupt vectors to
CPUs on the NIC's local node so that the round-robin cannot select
a remote one. Something like the following, adjusted once you have
confirmed the local CPU list:

    for irq in $(awk '/mlx5_comp(4[89]|5[0-9]|6[0-2])@pci:0000:2a:00.0/ \
                      { sub(":", "", $1); print $1 }' /proc/interrupts); do
        echo 0-23,48-71 > /proc/irq/$irq/smp_affinity_list
    done

Keep irqbalance disabled, since it will undo the interrupt steering.


-- 
Chuck Lever

Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Tim Menninger]

I realized the bad state correlates with where the RDMA CQs' completion
work runs.

This client has 16 RDMA transports and 32 CQs. The CQs receive 32
consecutive completion vectors out of the mlx5 device's 63-vector ring.
Depending on the global round-robin starting point, all 32 CQ workers can
execute on NUMA node 0, or they can split 17/15 between the two NUMA nodes.

On unpatched mainline (940de590b839 without your patch set), I tested five
consecutive allocation windows. On three of those, all 32 CQs executed on
NUMA node 0. On the other two they fell with a 17/15 split. All five
sustained ~46 GB/s.

With your v2 applied, all-local windows fall to roughly 24-28 GB/s while
split windows sustain ~46 GB/s.

I also tested this by changing only mlx5 IRQ affinity. I redirected
completion vectors 20-34 from node 0 to CPUs 24-38 on node 1. CQ allocation
windows 5-36 and 6-37, which would otherwise be entirely local and slow,
then sustained ~46 GB/s with their CQs split 17/15 across the nodes.

So the CQ allocation determines whether the regression is exposed, but
mainline is insensitive to that placement. The series introduces the
performance sensitivity. This also explains why v2 appeared
nondeterministic with respect to the rpciod affinity scope alone.

[PATCH RFC 7/8] NFS: Reduce nfsiod workqueue contention

[Chuck Lever]

The default affinity scope for unbound workqueues is now
WQ_AFFN_CACHE_SHARD, which splits each LLC into shards of about
eight cores. On a single-socket system whose LLC fits in one
shard, every NFS I/O completion serializes on one nfsiod pool
lock. Profiling 4KB random writes over NFSv3/RDMA with nconnect=3
shows that lock consuming 17% of CPU cycles: 8% dequeuing work and
9% enqueuing follow-on work from rpciod and nfsiod workers.

Set nfsiod's affinity scope to WQ_AFFN_SMT so each CPU gets its
own pool and queue_work_on() no longer takes a lock on another
CPU. Enqueue contention disappears and dequeue contention drops
to 1.4%. Throughput is unchanged because the workload is
transport-limited, but the freed cycles cut submission latency
variance by 67% (slat stdev 31.6 us to 10.5 us), IOPS stdev by
31%, and p99.9 completion latency by 11%.

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Chuck Lever <cel@kernel.org>
---
 fs/nfs/inode.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c
index 107a2135029d..8e0d2bebba54 100644
--- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c
@@ -2618,11 +2618,26 @@ static void nfsiod_stop(void)
  */
 static int nfsiod_start(void)
 {
+	struct workqueue_attrs *attrs;
+
 	dprintk("RPC:       creating workqueue nfsiod\n");
 	nfsiod_workqueue = alloc_workqueue("nfsiod",
 				WQ_MEM_RECLAIM | WQ_UNBOUND | WQ_SYSFS, 0);
 	if (nfsiod_workqueue == NULL)
 		return -ENOMEM;
+	attrs = alloc_workqueue_attrs();
+	if (attrs) {
+		int err;
+
+		attrs->affn_scope = WQ_AFFN_SMT;
+		err = apply_workqueue_attrs(nfsiod_workqueue, attrs);
+		free_workqueue_attrs(attrs);
+		if (err)
+			pr_warn("nfsiod: failed to set SMT affinity scope: %d\n",
+				err);
+	} else {
+		pr_warn("nfsiod: failed to allocate workqueue attrs\n");
+	}
 #if IS_ENABLED(CONFIG_NFS_LOCALIO)
 	/*
 	 * localio writes need to use a normal (non-memreclaim) workqueue.

-- 
2.55.0

[PATCH RFC 8/8] SUNRPC: Reduce xprtiod workqueue contention

[Chuck Lever]

xprtiod handles transport-level operations: socket receive
processing, error recovery, and connection lifecycle. On systems
driving heavy NFS traffic, these operations contend on the UNBOUND
worker pool lock just as rpciod does.

Set WQ_AFFN_SMT on the xprtiod workqueue to give each SMT group its
own pool and lock.

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Chuck Lever <cel@kernel.org>
---
 net/sunrpc/sched.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
index b84af2c0b104..3236dfdb4096 100644
--- a/net/sunrpc/sched.c
+++ b/net/sunrpc/sched.c
@@ -1304,6 +1304,7 @@ static int rpciod_start(void)
 	wq = alloc_workqueue("xprtiod", wq_flags, 0);
 	if (!wq)
 		goto free_rpciod;
+	rpc_set_wq_smt_affinity(wq, "xprtiod");
 	xprtiod_workqueue = wq;
 	return 1;
 free_rpciod:

-- 
2.55.0