[PATCH 2/3] svcrdma: Clear XPT_DATA when the last receive context is consumed

[Chuck Lever <cel@kernel.org>]

svc_rdma_wc_receive() and svc_rdma_wc_read_done() set XPT_DATA after
adding a completed context to sc_rq_dto_q or sc_read_complete_q.
svc_rdma_recvfrom() dequeues one context and leaves XPT_DATA set, so
the svc_xprt_received() that follows re-enqueues the transport and
svc_xprt_enqueue() dispatches a second thread. That thread finds both
queues empty and returns zero.

Recheck the receive queues after each dequeue and clear XPT_DATA when
the last context is taken, rather than only when a dequeue finds
nothing. svc_xprt_received()'s kernel-doc no longer describes every
transport, so relax its note about when XPT_DATA is cleared.

Measured on one NFSv4.2 connection over 100GbE RoCE. A 4KB random read
at queue depth 1 falls from 2.997 transport dequeues per RPC to 1.998,
and from 96,629 to 82,398 server cycles per RPC. A 256KB random write
falls from 3.270 dequeues to 2.004, and from 259,936 to 246,059 cycles.
Each dispatch removed is worth about 10,000 cycles.

The gain shrinks as the receive queues fill, since a leftover XPT_DATA
then dispatches a thread that finds real work. An 8KB random write at
queue depth 512 already runs at the two dequeues an RPC with a Read
chunk requires, and shows no change. Throughput moves only where the
server has no idle CPU to absorb the saving, so only the queue depth 1
read gains, by 1.8%.

One dispatch per RPC remains. svc_rdma_send_ctxt_put() sets XPT_DATA to
schedule a drain of sc_send_release_list, and svc_rdma_recvfrom() does
not service that list.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 net/sunrpc/svc_xprt.c                   |  5 +++--
 net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 18 ++++++++++++++----
 2 files changed, 17 insertions(+), 6 deletions(-)

diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
index 40040af588fb..c0e6772c6683 100644
--- a/net/sunrpc/svc_xprt.c
+++ b/net/sunrpc/svc_xprt.c
@@ -218,8 +218,9 @@ EXPORT_SYMBOL_GPL(svc_xprt_init);
  * The caller must hold the XPT_BUSY bit and must
  * not thereafter touch transport data.
  *
- * Note: XPT_DATA only gets cleared when a read-attempt finds no (or
- * insufficient) data.
+ * Note: xpo_recvfrom decides when to clear XPT_DATA. A transport may
+ * leave the bit set until a read attempt finds no (or insufficient)
+ * data, or clear it as soon as it consumes the last queued receive.
  */
 void svc_xprt_received(struct svc_xprt *xprt)
 {
diff --git a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
index fdfed1be97da..d029bcb7a5c0 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
@@ -925,8 +925,9 @@ static noinline void svc_rdma_read_complete(struct svc_rqst *rqstp,
  *	%-ENOTCONN if posting failed (connection is lost),
  *	%-EIO if rdma_rw initialization failed (DMA mapping, etc).
  *
- * Called in a loop when XPT_DATA is set. XPT_DATA is cleared only
- * when there are no remaining ctxt's to process.
+ * Called in a loop when XPT_DATA is set. XPT_DATA is cleared as
+ * soon as both receive queues are empty, so a consumed ctxt does
+ * not leave a stale bit behind.
  *
  * The next ctxt is removed from the "receive" lists.
  *
@@ -960,6 +961,15 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp)
 	ctxt = svc_rdma_next_recv_ctxt(&rdma_xprt->sc_read_complete_q);
 	if (ctxt) {
 		list_del(&ctxt->rc_list);
+		/* Producers add to these queues and set XPT_DATA under
+		 * this lock, so the clear cannot race one. The clear can
+		 * drop the XPT_DATA that svc_rdma_send_ctxt_put() sets
+		 * for sc_send_release_list. svc_xprt_release() drains
+		 * that list before this thread looks for more work.
+		 */
+		if (list_empty(&rdma_xprt->sc_read_complete_q) &&
+		    list_empty(&rdma_xprt->sc_rq_dto_q))
+			clear_bit(XPT_DATA, &xprt->xpt_flags);
 		spin_unlock(&rdma_xprt->sc_rq_dto_lock);
 		svc_xprt_received(xprt);
 		svc_rdma_read_complete(rqstp, ctxt);
@@ -968,8 +978,8 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp)
 	ctxt = svc_rdma_next_recv_ctxt(&rdma_xprt->sc_rq_dto_q);
 	if (ctxt)
 		list_del(&ctxt->rc_list);
-	else
-		/* No new incoming requests, terminate the loop */
+	/* sc_read_complete_q was empty above, under this same lock. */
+	if (list_empty(&rdma_xprt->sc_rq_dto_q))
 		clear_bit(XPT_DATA, &xprt->xpt_flags);
 	spin_unlock(&rdma_xprt->sc_rq_dto_lock);
 
-- 
2.54.0