All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v2] nvme-rdma: add a module parameter to disable T10-PI offload
@ 2026-11-12 21:01 Stephane Thiell
  2026-11-16  0:59 ` Max Gurtovoy
  0 siblings, 1 reply; 3+ messages in thread
From: Stephane Thiell @ 2026-11-12 21:01 UTC (permalink / raw)
  To: linux-nvme; +Cc: kbusch, hch, sagi, axboe, mgurtovoy, Stephane Thiell

When the RDMA device advertises IBK_INTEGRITY_HANDOVER, nvme-rdma
enables T10-PI offload for the controller before any namespace is
known, and every I/O queue then allocates an integrity MR pool of
queue_size entries next to its plain MR pool. Hosts whose namespaces
are not formatted with protection information never use these MRs but
pay for them on every connect, reconnect and disconnect.

On mlx5 an integrity MR is three mkeys and a PSV pair, so each request
slot costs five synchronous firmware commands (four CREATE_MKEY, one
CREATE_PSV) instead of one, the same objects are destroyed again at
teardown, and the HCA context memory per slot grows about fivefold.

Measured before the parallel I/O queue setup of v7.3, with 48
controllers of 24 I/O queues and a queue size of 128, connect went from
172 s to 46 s and disconnect from ~150 s to 35 s.

Add a pi_enable module parameter. With pi_enable=0, pi_capable stays
false in nvme_rdma_configure_admin_queue() and the controller is set up
as on a device without integrity handover: max_integrity_segments is 0
and no integrity MR pool is allocated. Namespaces that work with PI
offload keep working, with the controller inserting and stripping the PI
(PRACT).

The default is true, so the behavior does not change unless the
parameter is set.
Signed-off-by: Stephane Thiell <sthiell@hudson-trading.com>
---
v2:
- Changelog: correct the PI namespace, teardown and timing statements
  (v1 wrongly said a PI namespace gets a capacity of 0); simplify the
  memory figures.
v1: https://lore.kernel.org/linux-nvme/20261112194838.928882-1-sthiell@hudson-trading.com/

Measured on a 6.12-based kernel; build-tested only on v7.4-rc2. If a
per-connection fabrics option, or creating the integrity pool only when
a PI namespace appears, is preferred to a module parameter, I can rework
it that way.

 drivers/nvme/host/rdma.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 32189fdfc0f5..267de27b743e 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -156,6 +156,11 @@ module_param(register_always, bool, 0444);
 MODULE_PARM_DESC(register_always,
 	 "Use memory registration even for contiguous memory regions");
 
+static bool pi_enable = true;
+module_param(pi_enable, bool, 0444);
+MODULE_PARM_DESC(pi_enable,
+	 "Use T10-PI offload if the RDMA device supports it (default: true)");
+
 static int nvme_rdma_cm_handler(struct rdma_cm_id *cm_id,
 		struct rdma_cm_event *event);
 static void nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc);
@@ -806,8 +811,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
 	ctrl->ctrl.numa_node = ibdev_to_node(ctrl->device->dev);
 
 	/* T10-PI support */
-	if (ctrl->device->dev->attrs.kernel_cap_flags &
-	    IBK_INTEGRITY_HANDOVER)
+	if (pi_enable && (ctrl->device->dev->attrs.kernel_cap_flags &
+			  IBK_INTEGRITY_HANDOVER))
 		pi_capable = true;
 
 	ctrl->max_fr_pages = nvme_rdma_get_max_fr_pages(ctrl->device->dev,

base-commit: 9d66f26fefacebfeed15de31c1123451a17ff119
-- 
2.39.1



^ permalink raw reply related	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2026-11-20 17:58 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-12 21:01 [PATCH v2] nvme-rdma: add a module parameter to disable T10-PI offload Stephane Thiell
2026-11-16  0:59 ` Max Gurtovoy
2026-11-20 17:58   ` Christoph Hellwig

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.