From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A54F835AC20 for ; Sat, 28 Feb 2026 18:19:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772302741; cv=none; b=ESyNIX3+WcuhPNRJXklF0i2kOvqMe47i6MfyIk3xtUMpUCpkQpU9sbHb6ZphjTqdXDMEOqpFEzemkdU94QOsHruc8gtNTljQw7PhEwKAWoEQBUoyfiByV7Qy638RM32CrGcc3OkZJoJhzD+WzgsO0Iq4UkeZWGM7nl/lfsjcm58= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772302741; c=relaxed/simple; bh=2RwZfdDEWNax/Zy7aJNzl8VlobMCWnLmWWNbRk2g6O0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pC/7ao26iwirkhN2mH9AIE3gSHw7z7DKEYRpeRsPdZ2mnHtpmmuh1t1fViweSjDV4TQIx2nhLw2z3uFRt8Mhn90XSaCKWbhvMDa0DLrL8+jKW3hsjSw6LsUGVRl8BMKVykHwLxzH5SrbvOF2Y+Tg/goEbZR0fmIO7RUHb88HLy4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=qMWdkhfr; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="qMWdkhfr" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 00B87C19423; Sat, 28 Feb 2026 18:19:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1772302741; bh=2RwZfdDEWNax/Zy7aJNzl8VlobMCWnLmWWNbRk2g6O0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qMWdkhfrOx1Qux7jK+fyf74TUgvahgQ/N1Xxzmb5voMVqQOQDL4bS/Zmudx5fQVrA D54wz6nMXstyhj9Lll/Gz2ctz9OIOB/3I/QCmKc+fUA6drJWENhG7HZ5DFaX88TzTy +6NV58+8gQJodqTK5V/qtFVAZGE7MLSjlh6YNI4FxcfZqq2sVkH4f5fqLr2kgVEnzz lwp8SYBCzpAQs3ecTcm1wXkiWNde9DvUQB/Pu7A7l6ka1zOH3G9qVKYrFxOINGdhk+ nxZ1zTR5kYXVnOkEG2WWL2BAYt1/BXRkOCbKR1VUJXgKURM1VwYAb6PlZTYq62JW7C Fb4N980GwXduA== From: Sasha Levin To: patches@lists.linux.dev Cc: Chuck Lever , NeilBrown , Christoph Hellwig , Sasha Levin Subject: [PATCH 5.10 107/147] svcrdma: Reduce the number of rdma_rw contexts per-QP Date: Sat, 28 Feb 2026 13:16:55 -0500 Message-ID: <20260228181736.1605592-107-sashal@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260228181736.1605592-1-sashal@kernel.org> References: <20260228181736.1605592-1-sashal@kernel.org> Precedence: bulk X-Mailing-List: patches@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-stable: review X-Patchwork-Hint: Ignore Content-Transfer-Encoding: 8bit From: Chuck Lever [ Upstream commit 59243315890578a040a2d50ae9e001a2ef2fcb62 ] There is an upper bound on the number of rdma_rw contexts that can be created per QP. This invisible upper bound is because rdma_create_qp() adds one or more additional SQEs for each ctxt that the ULP requests via qp_attr.cap.max_rdma_ctxs. The QP's actual Send Queue length is on the order of the sum of qp_attr.cap.max_send_wr and a factor times qp_attr.cap.max_rdma_ctxs. The factor can be up to three, depending on whether MR operations are required before RDMA Reads. This limit is not visible to RDMA consumers via dev->attrs. When the limit is surpassed, QP creation fails with -ENOMEM. For example: svcrdma's estimate of the number of rdma_rw contexts it needs is three times the number of pages in RPCSVC_MAXPAGES. When MAXPAGES is about 260, the internally-computed SQ length should be: 64 credits + 10 backlog + 3 * (3 * 260) = 2414 Which is well below the advertised qp_max_wr of 32768. If RPCSVC_MAXPAGES is increased to 4MB, that's 1040 pages: 64 credits + 10 backlog + 3 * (3 * 1040) = 9434 However, QP creation fails. Dynamic printk for mlx5 shows: calc_sq_size:618:(pid 1514): send queue size (9326 * 256 / 64 -> 65536) exceeds limits(32768) Although 9326 is still far below qp_max_wr, QP creation still fails. Because the total SQ length calculation is opaque to RDMA consumers, there doesn't seem to be much that can be done about this except for consumers to try to keep the requested rdma_rw ctxt count low. Fixes: 2da0f610e733 ("svcrdma: Increase the per-transport rw_ctx count") Reviewed-by: NeilBrown Reviewed-by: Christoph Hellwig Signed-off-by: Chuck Lever Stable-dep-of: afcae7d7b8a2 ("RDMA/core: add rdma_rw_max_sge() helper for SQ sizing") Signed-off-by: Sasha Levin --- net/sunrpc/xprtrdma/svc_rdma_transport.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c index 9f66b57125042..7b7d2add99a4c 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_transport.c +++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c @@ -365,12 +365,12 @@ static struct svc_xprt *svc_rdma_create(struct svc_serv *serv, */ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt) { + unsigned int ctxts, rq_depth, maxpayload; struct svcxprt_rdma *listen_rdma; struct svcxprt_rdma *newxprt = NULL; struct rdma_conn_param conn_param; struct rpcrdma_connect_private pmsg; struct ib_qp_init_attr qp_attr; - unsigned int ctxts, rq_depth; struct ib_device *dev; int ret = 0; RPC_IFDEBUG(struct sockaddr *sap); @@ -418,12 +418,14 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt) newxprt->sc_max_bc_requests = 2; } - /* Arbitrarily estimate the number of rw_ctxs needed for - * this transport. This is enough rw_ctxs to make forward - * progress even if the client is using one rkey per page - * in each Read chunk. + /* Arbitrary estimate of the needed number of rdma_rw contexts. */ - ctxts = 3 * RPCSVC_MAXPAGES; + maxpayload = min(xprt->xpt_server->sv_max_payload, + RPCSVC_MAXPAYLOAD_RDMA); + ctxts = newxprt->sc_max_requests * 3 * + rdma_rw_mr_factor(dev, newxprt->sc_port_num, + maxpayload >> PAGE_SHIFT); + newxprt->sc_sq_depth = rq_depth + ctxts; if (newxprt->sc_sq_depth > dev->attrs.max_qp_wr) newxprt->sc_sq_depth = dev->attrs.max_qp_wr; -- 2.51.0