* [PATCH v2] nvme-rdma: add a module parameter to disable T10-PI offload
@ 2026-11-12 21:01 Stephane Thiell
2026-11-16 0:59 ` Max Gurtovoy
0 siblings, 1 reply; 3+ messages in thread
From: Stephane Thiell @ 2026-11-12 21:01 UTC (permalink / raw)
To: linux-nvme; +Cc: kbusch, hch, sagi, axboe, mgurtovoy, Stephane Thiell
When the RDMA device advertises IBK_INTEGRITY_HANDOVER, nvme-rdma
enables T10-PI offload for the controller before any namespace is
known, and every I/O queue then allocates an integrity MR pool of
queue_size entries next to its plain MR pool. Hosts whose namespaces
are not formatted with protection information never use these MRs but
pay for them on every connect, reconnect and disconnect.
On mlx5 an integrity MR is three mkeys and a PSV pair, so each request
slot costs five synchronous firmware commands (four CREATE_MKEY, one
CREATE_PSV) instead of one, the same objects are destroyed again at
teardown, and the HCA context memory per slot grows about fivefold.
Measured before the parallel I/O queue setup of v7.3, with 48
controllers of 24 I/O queues and a queue size of 128, connect went from
172 s to 46 s and disconnect from ~150 s to 35 s.
Add a pi_enable module parameter. With pi_enable=0, pi_capable stays
false in nvme_rdma_configure_admin_queue() and the controller is set up
as on a device without integrity handover: max_integrity_segments is 0
and no integrity MR pool is allocated. Namespaces that work with PI
offload keep working, with the controller inserting and stripping the PI
(PRACT).
The default is true, so the behavior does not change unless the
parameter is set.
Signed-off-by: Stephane Thiell <sthiell@hudson-trading.com>
---
v2:
- Changelog: correct the PI namespace, teardown and timing statements
(v1 wrongly said a PI namespace gets a capacity of 0); simplify the
memory figures.
v1: https://lore.kernel.org/linux-nvme/20261112194838.928882-1-sthiell@hudson-trading.com/
Measured on a 6.12-based kernel; build-tested only on v7.4-rc2. If a
per-connection fabrics option, or creating the integrity pool only when
a PI namespace appears, is preferred to a module parameter, I can rework
it that way.
drivers/nvme/host/rdma.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 32189fdfc0f5..267de27b743e 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -156,6 +156,11 @@ module_param(register_always, bool, 0444);
MODULE_PARM_DESC(register_always,
"Use memory registration even for contiguous memory regions");
+static bool pi_enable = true;
+module_param(pi_enable, bool, 0444);
+MODULE_PARM_DESC(pi_enable,
+ "Use T10-PI offload if the RDMA device supports it (default: true)");
+
static int nvme_rdma_cm_handler(struct rdma_cm_id *cm_id,
struct rdma_cm_event *event);
static void nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc);
@@ -806,8 +811,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
ctrl->ctrl.numa_node = ibdev_to_node(ctrl->device->dev);
/* T10-PI support */
- if (ctrl->device->dev->attrs.kernel_cap_flags &
- IBK_INTEGRITY_HANDOVER)
+ if (pi_enable && (ctrl->device->dev->attrs.kernel_cap_flags &
+ IBK_INTEGRITY_HANDOVER))
pi_capable = true;
ctrl->max_fr_pages = nvme_rdma_get_max_fr_pages(ctrl->device->dev,
base-commit: 9d66f26fefacebfeed15de31c1123451a17ff119
--
2.39.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH v2] nvme-rdma: add a module parameter to disable T10-PI offload
2026-11-12 21:01 [PATCH v2] nvme-rdma: add a module parameter to disable T10-PI offload Stephane Thiell
@ 2026-11-16 0:59 ` Max Gurtovoy
2026-11-20 17:58 ` Christoph Hellwig
0 siblings, 1 reply; 3+ messages in thread
From: Max Gurtovoy @ 2026-11-16 0:59 UTC (permalink / raw)
To: Stephane Thiell, linux-nvme; +Cc: kbusch, hch, sagi, axboe
On 12/11/2026 23:01, Stephane Thiell wrote:
> When the RDMA device advertises IBK_INTEGRITY_HANDOVER, nvme-rdma
> enables T10-PI offload for the controller before any namespace is
> known, and every I/O queue then allocates an integrity MR pool of
> queue_size entries next to its plain MR pool. Hosts whose namespaces
> are not formatted with protection information never use these MRs but
> pay for them on every connect, reconnect and disconnect.
>
> On mlx5 an integrity MR is three mkeys and a PSV pair, so each request
> slot costs five synchronous firmware commands (four CREATE_MKEY, one
> CREATE_PSV) instead of one, the same objects are destroyed again at
> teardown, and the HCA context memory per slot grows about fivefold.
>
> Measured before the parallel I/O queue setup of v7.3, with 48
> controllers of 24 I/O queues and a queue size of 128, connect went from
> 172 s to 46 s and disconnect from ~150 s to 35 s.
>
> Add a pi_enable module parameter. With pi_enable=0, pi_capable stays
> false in nvme_rdma_configure_admin_queue() and the controller is set up
> as on a device without integrity handover: max_integrity_segments is 0
> and no integrity MR pool is allocated. Namespaces that work with PI
> offload keep working, with the controller inserting and stripping the PI
> (PRACT).
>
> The default is true, so the behavior does not change unless the
> parameter is set.
> Signed-off-by: Stephane Thiell <sthiell@hudson-trading.com>
> ---
> v2:
> - Changelog: correct the PI namespace, teardown and timing statements
> (v1 wrongly said a PI namespace gets a capacity of 0); simplify the
> memory figures.
> v1: https://lore.kernel.org/linux-nvme/20261112194838.928882-1-sthiell@hudson-trading.com/
>
> Measured on a 6.12-based kernel; build-tested only on v7.4-rc2. If a
> per-connection fabrics option, or creating the integrity pool only when
> a PI namespace appears, is preferred to a module parameter, I can rework
> it that way.
>
> drivers/nvme/host/rdma.c | 9 +++++++--
> 1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 32189fdfc0f5..267de27b743e 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -156,6 +156,11 @@ module_param(register_always, bool, 0444);
> MODULE_PARM_DESC(register_always,
> "Use memory registration even for contiguous memory regions");
>
> +static bool pi_enable = true;
> +module_param(pi_enable, bool, 0444);
> +MODULE_PARM_DESC(pi_enable,
> + "Use T10-PI offload if the RDMA device supports it (default: true)");
IIRC, when we developed this feature, a connect param to control PI
offload was part of the initial patchset, but the community didn’t ack it.
I think that connect param is a better option than the module param.
Later (Jan 2024) I proposed a per-connect option, nvme connect
--disallow_pi, together with uniform metadata settings for the ns head.
Some of the code for NS settings was merged in a separate patcheset that
didn't include the connect param.
I believe we can also merge the connect cmdline param (better call it
enable_pi and set default to 1 as suggested by the community).
I'll check the status of the old patches internally and update.
> +
> static int nvme_rdma_cm_handler(struct rdma_cm_id *cm_id,
> struct rdma_cm_event *event);
> static void nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc);
> @@ -806,8 +811,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
> ctrl->ctrl.numa_node = ibdev_to_node(ctrl->device->dev);
>
> /* T10-PI support */
> - if (ctrl->device->dev->attrs.kernel_cap_flags &
> - IBK_INTEGRITY_HANDOVER)
> + if (pi_enable && (ctrl->device->dev->attrs.kernel_cap_flags &
> + IBK_INTEGRITY_HANDOVER))
> pi_capable = true;
>
> ctrl->max_fr_pages = nvme_rdma_get_max_fr_pages(ctrl->device->dev,
>
> base-commit: 9d66f26fefacebfeed15de31c1123451a17ff119
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH v2] nvme-rdma: add a module parameter to disable T10-PI offload
2026-11-16 0:59 ` Max Gurtovoy
@ 2026-11-20 17:58 ` Christoph Hellwig
0 siblings, 0 replies; 3+ messages in thread
From: Christoph Hellwig @ 2026-11-20 17:58 UTC (permalink / raw)
To: Max Gurtovoy; +Cc: Stephane Thiell, linux-nvme, kbusch, hch, sagi, axboe
On Mon, Nov 16, 2026 at 02:59:10AM +0200, Max Gurtovoy wrote:
> IIRC, when we developed this feature, a connect param to control PI offload
> was part of the initial patchset, but the community didn’t ack it.
> I think that connect param is a better option than the module param.
Yes.
>
> Later (Jan 2024) I proposed a per-connect option, nvme connect
> --disallow_pi, together with uniform metadata settings for the ns head.
> Some of the code for NS settings was merged in a separate patcheset that
> didn't include the connect param.
> I believe we can also merge the connect cmdline param (better call it
> enable_pi and set default to 1 as suggested by the community).
>
> I'll check the status of the old patches internally and update.
Yeah, let's get the rest of this merged.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-20 17:58 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-12 21:01 [PATCH v2] nvme-rdma: add a module parameter to disable T10-PI offload Stephane Thiell
2026-11-16 0:59 ` Max Gurtovoy
2026-11-20 17:58 ` Christoph Hellwig
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox