From: Max Gurtovoy <mgurtovoy@nvidia.com>
To: Stephane Thiell <sthiell@hudson-trading.com>,
linux-nvme@lists.infradead.org
Cc: kbusch@kernel.org, hch@lst.de, sagi@grimberg.me, axboe@kernel.dk
Subject: Re: [PATCH v2] nvme-rdma: add a module parameter to disable T10-PI offload
Date: Mon, 16 Nov 2026 02:59:10 +0200 [thread overview]
Message-ID: <63f323c5-53bf-475f-808a-b6819b187bb0@nvidia.com> (raw)
In-Reply-To: <20261112210105.992764-1-sthiell@hudson-trading.com>
On 12/11/2026 23:01, Stephane Thiell wrote:
> When the RDMA device advertises IBK_INTEGRITY_HANDOVER, nvme-rdma
> enables T10-PI offload for the controller before any namespace is
> known, and every I/O queue then allocates an integrity MR pool of
> queue_size entries next to its plain MR pool. Hosts whose namespaces
> are not formatted with protection information never use these MRs but
> pay for them on every connect, reconnect and disconnect.
>
> On mlx5 an integrity MR is three mkeys and a PSV pair, so each request
> slot costs five synchronous firmware commands (four CREATE_MKEY, one
> CREATE_PSV) instead of one, the same objects are destroyed again at
> teardown, and the HCA context memory per slot grows about fivefold.
>
> Measured before the parallel I/O queue setup of v7.3, with 48
> controllers of 24 I/O queues and a queue size of 128, connect went from
> 172 s to 46 s and disconnect from ~150 s to 35 s.
>
> Add a pi_enable module parameter. With pi_enable=0, pi_capable stays
> false in nvme_rdma_configure_admin_queue() and the controller is set up
> as on a device without integrity handover: max_integrity_segments is 0
> and no integrity MR pool is allocated. Namespaces that work with PI
> offload keep working, with the controller inserting and stripping the PI
> (PRACT).
>
> The default is true, so the behavior does not change unless the
> parameter is set.
> Signed-off-by: Stephane Thiell <sthiell@hudson-trading.com>
> ---
> v2:
> - Changelog: correct the PI namespace, teardown and timing statements
> (v1 wrongly said a PI namespace gets a capacity of 0); simplify the
> memory figures.
> v1: https://lore.kernel.org/linux-nvme/20261112194838.928882-1-sthiell@hudson-trading.com/
>
> Measured on a 6.12-based kernel; build-tested only on v7.4-rc2. If a
> per-connection fabrics option, or creating the integrity pool only when
> a PI namespace appears, is preferred to a module parameter, I can rework
> it that way.
>
> drivers/nvme/host/rdma.c | 9 +++++++--
> 1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 32189fdfc0f5..267de27b743e 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -156,6 +156,11 @@ module_param(register_always, bool, 0444);
> MODULE_PARM_DESC(register_always,
> "Use memory registration even for contiguous memory regions");
>
> +static bool pi_enable = true;
> +module_param(pi_enable, bool, 0444);
> +MODULE_PARM_DESC(pi_enable,
> + "Use T10-PI offload if the RDMA device supports it (default: true)");
IIRC, when we developed this feature, a connect param to control PI
offload was part of the initial patchset, but the community didn’t ack it.
I think that connect param is a better option than the module param.
Later (Jan 2024) I proposed a per-connect option, nvme connect
--disallow_pi, together with uniform metadata settings for the ns head.
Some of the code for NS settings was merged in a separate patcheset that
didn't include the connect param.
I believe we can also merge the connect cmdline param (better call it
enable_pi and set default to 1 as suggested by the community).
I'll check the status of the old patches internally and update.
> +
> static int nvme_rdma_cm_handler(struct rdma_cm_id *cm_id,
> struct rdma_cm_event *event);
> static void nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc);
> @@ -806,8 +811,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
> ctrl->ctrl.numa_node = ibdev_to_node(ctrl->device->dev);
>
> /* T10-PI support */
> - if (ctrl->device->dev->attrs.kernel_cap_flags &
> - IBK_INTEGRITY_HANDOVER)
> + if (pi_enable && (ctrl->device->dev->attrs.kernel_cap_flags &
> + IBK_INTEGRITY_HANDOVER))
> pi_capable = true;
>
> ctrl->max_fr_pages = nvme_rdma_get_max_fr_pages(ctrl->device->dev,
>
> base-commit: 9d66f26fefacebfeed15de31c1123451a17ff119
next prev parent reply other threads:[~2026-11-16 0:59 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-12 21:01 [PATCH v2] nvme-rdma: add a module parameter to disable T10-PI offload Stephane Thiell
2026-11-16 0:59 ` Max Gurtovoy [this message]
2026-11-20 17:58 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=63f323c5-53bf-475f-808a-b6819b187bb0@nvidia.com \
--to=mgurtovoy@nvidia.com \
--cc=axboe@kernel.dk \
--cc=hch@lst.de \
--cc=kbusch@kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=sagi@grimberg.me \
--cc=sthiell@hudson-trading.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.