[PATCH] nvme-rdma: add a module parameter to disable T10-PI offload

[PATCH] nvme-rdma: add a module parameter to disable T10-PI offload

[Stephane Thiell]

When the RDMA device advertises IBK_INTEGRITY_HANDOVER, nvme-rdma
enables T10-PI offload for the controller before any namespace is
known, and every I/O queue then allocates an integrity MR pool of
queue_size entries next to its plain MR pool. Hosts whose namespaces
are not formatted with protection information never use these MRs but
pay for them on every connect, reconnect and disconnect.

On mlx5 an integrity MR is three mkeys and a PSV pair, so each request
slot costs five firmware commands (four CREATE_MKEY, one CREATE_PSV)
instead of one. These commands are synchronous and take about 0.2 ms
each, and the same number of destroy commands is issued at teardown.
The HCA context memory given to the firmware grows the same way: about
10.4 pages per I/O request slot, against 2.1 without the integrity
pool.

With 48 controllers of 24 I/O queues and a queue size of 128, connect
goes from 172 s to 46 s and disconnect from ~150 s to 35 s.

Add a pi_enable module parameter. With pi_enable=0, pi_capable stays
false in nvme_rdma_configure_admin_queue() and the controller is set up
as on a device without integrity handover: max_integrity_segments is 0
and no integrity MR pool is allocated. As on such a device, a namespace
formatted with protection information is not usable (the core gives it
a capacity of 0).

The default is true, so the behavior does not change unless the
parameter is set.
Signed-off-by: Stephane Thiell <sthiell@hudson-trading.com>
---
Developed and measured before the parallel I/O queue setup of v7.3;
build-tested on v7.4-rc2. If a per-connection fabrics option is preferred
to a module parameter, I can rework it that way.

 drivers/nvme/host/rdma.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 32189fdfc0f5..267de27b743e 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -156,6 +156,11 @@ module_param(register_always, bool, 0444);
 MODULE_PARM_DESC(register_always,
 	 "Use memory registration even for contiguous memory regions");
 
+static bool pi_enable = true;
+module_param(pi_enable, bool, 0444);
+MODULE_PARM_DESC(pi_enable,
+	 "Use T10-PI offload if the RDMA device supports it (default: true)");
+
 static int nvme_rdma_cm_handler(struct rdma_cm_id *cm_id,
 		struct rdma_cm_event *event);
 static void nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc);
@@ -806,8 +811,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
 	ctrl->ctrl.numa_node = ibdev_to_node(ctrl->device->dev);
 
 	/* T10-PI support */
-	if (ctrl->device->dev->attrs.kernel_cap_flags &
-	    IBK_INTEGRITY_HANDOVER)
+	if (pi_enable && (ctrl->device->dev->attrs.kernel_cap_flags &
+			  IBK_INTEGRITY_HANDOVER))
 		pi_capable = true;
 
 	ctrl->max_fr_pages = nvme_rdma_get_max_fr_pages(ctrl->device->dev,

base-commit: 9d66f26fefacebfeed15de31c1123451a17ff119
-- 
2.39.1

Re: [PATCH] nvme-rdma: add a module parameter to disable T10-PI offload

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] nvme-rdma: inaccurate commit message claim on PI namespace
  capacity
--

commit e90985fd114ef929dfa083aabec197dab7d0860a
Author: Stephane Thiell <sthiell@hudson-trading.com>

nvme-rdma: add a module parameter to disable T10-PI offload

This commit adds a pi_enable module parameter to nvme-rdma to allow
disabling T10-PI offload and avoiding integrity MR pool allocations on
RDMA devices that advertise IBK_INTEGRITY_HANDOVER.

>     Add a pi_enable module parameter. With pi_enable=0, pi_capable stays
>     false in nvme_rdma_configure_admin_queue() and the controller is set up
>     as on a device without integrity handover: max_integrity_segments is 0
>     and no integrity MR pool is allocated. As on such a device, a namespace
>     formatted with protection information is not usable (the core gives it
>     a capacity of 0).

[Severity: Low]
Is the commit message accurate about PI-formatted namespaces getting a
capacity of 0 when pi_enable=0?

nvme_rdma_ctrl_ops.flags sets NVME_F_METADATA_SUPPORTED unconditionally,
so nvme_configure_metadata() still populates head->pi_type and
head->pi_size when ctrl->max_integrity_segments is 0, and only omits
NVME_NS_METADATA_SUPPORTED from head->features:

drivers/nvme/host/core.c:nvme_init_integrity() {
	if (!IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) ||
	    !(head->features & NVME_NS_METADATA_SUPPORTED))
		return nvme_ns_has_pi(head);

For a PI-formatted namespace where ms == pi_size, nvme_ns_has_pi(head) is
true, so nvme_init_integrity() returns true and nvme_update_ns_info_block()
keeps the full namespace capacity (using NVME_RW_PRINFO_PRACT for
controller-side PI insert/strip), only zeroing capacity for metadata
namespaces where !nvme_ns_has_pi(head).

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261112194838.928882-1-sthiell@hudson-trading.com?part=1