All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Vladimir Petrzhikovskii <vladimir@dfb.sh>
Cc: "Dragos Tatulea" <dtatulea@nvidia.com>,
	"Jason Wang" <jasowangio@gmail.com>,
	"Eugenio Pérez" <eperezma@redhat.com>,
	"Xuan Zhuo" <xuanzhuo@linux.alibaba.com>,
	virtualization@lists.linux.dev, linux-kernel@vger.kernel.org,
	stable@vger.kernel.org, "Nikolai Kalinin" <nikolai@kalinidze.me>
Subject: Re: [PATCH] vdpa/mlx5: Bound the KLM count of the user MR
Date: Thu, 19 Nov 2026 10:15:32 -0500	[thread overview]
Message-ID: <20261119101523-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <20261109-b4-mlx5-klm-v1-1-787b45145719@dfb.sh>

On Mon, Nov 09, 2026 at 08:21:28PM +0100, Vladimir Petrzhikovskii wrote:
> create_user_mr() builds one indirect MKey that covers the whole vhost
> IOTLB. It fills each gap between two mappings with null KLMs, one per
> GiB, and never checks the total number of KLMs against the device
> limit, MLX5_CAP_GEN(log_max_klm_list_size).
> 
> With a virtio-IOMMU the guest chooses the IOVAs: the VMM (QEMU)
> forwards each map the guest makes to the vhost-vDPA device. If the
> guest maps one page at IOVA 0 and one at IOVA 2^58, the driver needs
> more than 2^28 KLMs. create_indirect_key() computes the buffer size in
> an int, which overflows: the driver allocates 336 bytes, and
> fill_indir() then writes 16 bytes per KLM, 4 GiB in total, far past
> the end of the buffer.
> 
> nnuls is an int, so a gap of 2^61 bytes or more is miscounted on every
> architecture, and on 32-bit the unsigned long num_klms wraps. The gap
> is rounded up to whole GiB as (size + 2^30 - 1) >> 30. For a gap
> larger than 2^64 - 2^30, for example below a page mapped in the top
> GiB, that addition wraps and the driver counts no null KLMs for the
> gap, while fill_indir() still writes one for every GiB of it.
> 
> Count KLMs in u64 and round the gap up without the overflowing
> addition. Before creating any direct key, return -E2BIG with a
> rate-limited warning if the MR needs more KLMs than the device limit,
> or more than fit in a CREATE_MKEY command whose length is an int.
> Allocate the command with kvzalloc(), because a list within the device
> limit can be too large for kzalloc(). A normal guest is unaffected:
> memory at 0-3 GiB and 4-68 GiB needs 68 KLMs.
> 
> Found by an LLM code review agent auditing vhost-vDPA backends. We have
> no mlx5 hardware, so the fix is untested on it; it builds with W=1 on
> x86_64 and i386.
> 
> Fixes: 94abbccdf291 ("vdpa/mlx5: Add shared memory registration code")
> Cc: stable@vger.kernel.org
> Co-developed-by: Nikolai Kalinin <nikolai@kalinidze.me>
> Signed-off-by: Nikolai Kalinin <nikolai@kalinidze.me>
> Signed-off-by: Vladimir Petrzhikovskii <vladimir@dfb.sh>
> ---
> Dragos, one question on the limit: the patch takes
> MLX5_CAP_GEN(log_max_klm_list_size) as the largest KLM list an indirect
> CREATE_MKEY accepts, as mlx5_ib does for fast-reg lists. If firmware
> applies a different limit to CREATE_MKEY, please say so and I will
> change the bound.
> 
> The guest kernel change used for the reachability check is available
> on request.


Dragos, care to answer?

> ---
>  drivers/vdpa/mlx5/core/mlx5_vdpa.h |  2 +-
>  drivers/vdpa/mlx5/core/mr.c        | 30 +++++++++++++++++++++++++-----
>  2 files changed, 26 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
> index 42f2f44b383c..1d84824ca429 100644
> --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h
> +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
> @@ -32,7 +32,7 @@ struct mlx5_vdpa_mr {
>  	/* list of direct MRs descendants of this indirect mr */
>  	struct list_head head;
>  	unsigned long num_directs;
> -	unsigned long num_klms;
> +	u64 num_klms;
>  
>  	struct vhost_iotlb *iotlb;
>  
> diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c
> index b0c5ff23d022..03db8345c1bb 100644
> --- a/drivers/vdpa/mlx5/core/mr.c
> +++ b/drivers/vdpa/mlx5/core/mr.c
> @@ -206,6 +206,16 @@ static int klm_byte_size(int nklms)
>  	return 16 * ALIGN(nklms, 4);
>  }
>  
> +/* Largest KLM count whose CREATE_MKEY input length still fits in an int. */
> +#define MLX5_VDPA_MAX_KLMS_INLEN \
> +	ALIGN_DOWN((INT_MAX - MLX5_ST_SZ_BYTES(create_mkey_in)) / 16, 4)
> +
> +static u64 max_klms(struct mlx5_vdpa_dev *mvdev)
> +{
> +	return min_t(u64, BIT_ULL(MLX5_CAP_GEN(mvdev->mdev, log_max_klm_list_size)),
> +		     MLX5_VDPA_MAX_KLMS_INLEN);
> +}
> +
>  #define MLX5_VDPA_MTT_ALIGN 16
>  
>  static int create_direct_keys(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr)
> @@ -338,7 +348,7 @@ static int create_indirect_key(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr
>  		return -EINVAL;
>  
>  	inlen = MLX5_ST_SZ_BYTES(create_mkey_in) + klm_byte_size(mr->num_klms);
> -	in = kzalloc(inlen, GFP_KERNEL);
> +	in = kvzalloc(inlen, GFP_KERNEL);
>  	if (!in)
>  		return -ENOMEM;
>  
> @@ -355,7 +365,7 @@ static int create_indirect_key(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr
>  	MLX5_SET(create_mkey_in, in, translations_octword_actual_size, mr->num_klms);
>  	fill_indir(mvdev, mr, in);
>  	err = mlx5_vdpa_create_mkey(mvdev, &mr->mkey, in, inlen);
> -	kfree(in);
> +	kvfree(in);
>  	return err;
>  }
>  
> @@ -501,6 +511,7 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev,
>  {
>  	struct mlx5_vdpa_direct_mr *dmr;
>  	struct mlx5_vdpa_direct_mr *n;
> +	u64 klm_limit = max_klms(mvdev);
>  	struct vhost_iotlb_map *map;
>  	u32 pperm = U16_MAX;
>  	u64 last = U64_MAX;
> @@ -508,7 +519,7 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev,
>  	u64 pe = U64_MAX;
>  	u64 start = 0;
>  	int err = 0;
> -	int nnuls;
> +	u64 nnuls;
>  
>  	INIT_LIST_HEAD(&mr->head);
>  	for (map = vhost_iotlb_itree_first(iotlb, start, last); map;
> @@ -521,9 +532,10 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev,
>  				if (pe < map->start) {
>  					/* We have a hole in the map. Check how
>  					 * many null keys are required to fill it.
> +					 * The hole can be nearly 2^64 bytes, so
> +					 * round up without adding to its size.
>  					 */
> -					nnuls = MLX5_DIV_ROUND_UP_POW2(map->start - pe,
> -								       LOG_MAX_KLM_SIZE);
> +					nnuls = ((map->start - pe - 1) >> LOG_MAX_KLM_SIZE) + 1;
>  					mr->num_klms += nnuls;
>  				}
>  				err = add_direct_chain(mvdev, mr, ps, pe - ps, pperm, iotlb);
> @@ -539,6 +551,14 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev,
>  	if (err)
>  		goto err_chain;
>  
> +	if (mr->num_klms > klm_limit) {
> +		dev_warn_ratelimited(mvdev->mdev->device,
> +				     "map needs %llu KLMs, limit is %llu\n",
> +				     mr->num_klms, klm_limit);
> +		err = -E2BIG;
> +		goto err_chain;
> +	}
> +
>  	err = create_direct_keys(mvdev, mr);
>  	if (err)
>  		goto err_chain;
> 
> ---
> base-commit: d48723711bae9b8cdc56719c448ba42fef28cbee
> change-id: 20261109-b4-mlx5-klm-798a398a159e
> 
> Best regards,
> --  
> Vladimir Petrzhikovskii <vladimir@dfb.sh>
> 


  parent reply	other threads:[~2026-11-19 15:15 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-09 19:21 [PATCH] vdpa/mlx5: Bound the KLM count of the user MR Vladimir Petrzhikovskii
2026-11-09 19:31 ` sashiko-bot
2026-11-19 15:15 ` Michael S. Tsirkin [this message]
2026-11-19 18:12   ` Dragos Tatulea
2026-11-20 13:20     ` Vladimir Petrzhikovskii

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261119101523-mutt-send-email-mst@kernel.org \
    --to=mst@redhat.com \
    --cc=dtatulea@nvidia.com \
    --cc=eperezma@redhat.com \
    --cc=jasowangio@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=nikolai@kalinidze.me \
    --cc=stable@vger.kernel.org \
    --cc=virtualization@lists.linux.dev \
    --cc=vladimir@dfb.sh \
    --cc=xuanzhuo@linux.alibaba.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.