All of lore.kernel.org
 help / color / mirror / Atom feed
From: Max Gurtovoy <mgurtovoy@nvidia.com>
To: Stephane Thiell <sthiell@hudson-trading.com>,
	linux-nvme@lists.infradead.org
Cc: kbusch@kernel.org, hch@lst.de, sagi@grimberg.me, axboe@kernel.dk
Subject: Re: [PATCH v2] nvme-rdma: add a module parameter to disable T10-PI offload
Date: Mon, 16 Nov 2026 02:59:10 +0200	[thread overview]
Message-ID: <63f323c5-53bf-475f-808a-b6819b187bb0@nvidia.com> (raw)
In-Reply-To: <20261112210105.992764-1-sthiell@hudson-trading.com>


On 12/11/2026 23:01, Stephane Thiell wrote:
> When the RDMA device advertises IBK_INTEGRITY_HANDOVER, nvme-rdma
> enables T10-PI offload for the controller before any namespace is
> known, and every I/O queue then allocates an integrity MR pool of
> queue_size entries next to its plain MR pool. Hosts whose namespaces
> are not formatted with protection information never use these MRs but
> pay for them on every connect, reconnect and disconnect.
>
> On mlx5 an integrity MR is three mkeys and a PSV pair, so each request
> slot costs five synchronous firmware commands (four CREATE_MKEY, one
> CREATE_PSV) instead of one, the same objects are destroyed again at
> teardown, and the HCA context memory per slot grows about fivefold.
>
> Measured before the parallel I/O queue setup of v7.3, with 48
> controllers of 24 I/O queues and a queue size of 128, connect went from
> 172 s to 46 s and disconnect from ~150 s to 35 s.
>
> Add a pi_enable module parameter. With pi_enable=0, pi_capable stays
> false in nvme_rdma_configure_admin_queue() and the controller is set up
> as on a device without integrity handover: max_integrity_segments is 0
> and no integrity MR pool is allocated. Namespaces that work with PI
> offload keep working, with the controller inserting and stripping the PI
> (PRACT).
>
> The default is true, so the behavior does not change unless the
> parameter is set.
> Signed-off-by: Stephane Thiell <sthiell@hudson-trading.com>
> ---
> v2:
> - Changelog: correct the PI namespace, teardown and timing statements
>    (v1 wrongly said a PI namespace gets a capacity of 0); simplify the
>    memory figures.
> v1: https://lore.kernel.org/linux-nvme/20261112194838.928882-1-sthiell@hudson-trading.com/
>
> Measured on a 6.12-based kernel; build-tested only on v7.4-rc2. If a
> per-connection fabrics option, or creating the integrity pool only when
> a PI namespace appears, is preferred to a module parameter, I can rework
> it that way.
>
>   drivers/nvme/host/rdma.c | 9 +++++++--
>   1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 32189fdfc0f5..267de27b743e 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -156,6 +156,11 @@ module_param(register_always, bool, 0444);
>   MODULE_PARM_DESC(register_always,
>   	 "Use memory registration even for contiguous memory regions");
>   
> +static bool pi_enable = true;
> +module_param(pi_enable, bool, 0444);
> +MODULE_PARM_DESC(pi_enable,
> +	 "Use T10-PI offload if the RDMA device supports it (default: true)");
IIRC, when we developed this feature, a connect param to control PI 
offload was part of the initial patchset, but the community didn’t ack it.
I think that connect param is a better option than the module param.

Later (Jan 2024) I proposed a per-connect option, nvme connect 
--disallow_pi, together with uniform metadata settings for the ns head.
Some of the code for NS settings was merged in a separate patcheset that 
didn't include the connect param.
I believe we can also merge the connect cmdline param (better call it 
enable_pi and set default to 1 as suggested by the community).

I'll check the status of the old patches internally and update.


> +
>   static int nvme_rdma_cm_handler(struct rdma_cm_id *cm_id,
>   		struct rdma_cm_event *event);
>   static void nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc);
> @@ -806,8 +811,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
>   	ctrl->ctrl.numa_node = ibdev_to_node(ctrl->device->dev);
>   
>   	/* T10-PI support */
> -	if (ctrl->device->dev->attrs.kernel_cap_flags &
> -	    IBK_INTEGRITY_HANDOVER)
> +	if (pi_enable && (ctrl->device->dev->attrs.kernel_cap_flags &
> +			  IBK_INTEGRITY_HANDOVER))
>   		pi_capable = true;
>   
>   	ctrl->max_fr_pages = nvme_rdma_get_max_fr_pages(ctrl->device->dev,
>
> base-commit: 9d66f26fefacebfeed15de31c1123451a17ff119


  reply	other threads:[~2026-11-16  0:59 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12 21:01 [PATCH v2] nvme-rdma: add a module parameter to disable T10-PI offload Stephane Thiell
2026-11-16  0:59 ` Max Gurtovoy [this message]
2026-11-20 17:58   ` Christoph Hellwig

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=63f323c5-53bf-475f-808a-b6819b187bb0@nvidia.com \
    --to=mgurtovoy@nvidia.com \
    --cc=axboe@kernel.dk \
    --cc=hch@lst.de \
    --cc=kbusch@kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=sagi@grimberg.me \
    --cc=sthiell@hudson-trading.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.