[PATCH rdma-next 0/2] RDMA: Fix DEVX UMEM mailbox size overflow

[PATCH rdma-next 0/2] RDMA: Fix DEVX UMEM mailbox size overflow

[Edward Srouji]

Registering a DEVX UMEM builds a CREATE_UMEM mailbox whose size grows
with the number of MTTs, i.e. with the user-supplied length and
pgsz_bitmap. The size is stored in a u32, so a UMEM of about 2TB with
4KB pages wraps it, the mailbox is under-allocated and
mlx5_ib_populate_pas() writes past its end.

Patch 1 computes the size with overflow checks into an int, the in_size
type of mlx5_cmd_exec(), and rejects UMEMs whose mailbox does not fit.

Patch 2 handles the remaining corner: _uverbs_alloc() adds a small
header to the request before calling kvmalloc(), so a mailbox just
under INT_MAX can still push kvmalloc() over its INT_MAX limit and
trigger a WARN. Since these sizes come from userspace, the allocation
now uses __GFP_NOWARN and fails with -ENOMEM.

Sizes just below INT_MAX also overflow the block count in
mlx5_calc_cmd_blocks(). That predates this series and fails safely
with -ENOMEM before the command reaches firmware, so it is left for a
separate mlx5_core change.

The overflow was found with an AI-assisted static analysis tool and
confirmed by code review.

Patch 1 relies on size_mul() and mixed-type check_add_overflow(), so
stable trees older than 6.1 will need a manual backport.

Signed-off-by: Edward Srouji <edwards@nvidia.com>
---
Uri Dar (2):
      RDMA/mlx5: Fix DEVX UMEM mailbox size truncation
      RDMA/core: Don't warn on oversized bundle allocations

 drivers/infiniband/core/ib_core_uverbs.c |  2 +-
 drivers/infiniband/hw/mlx5/devx.c        | 15 +++++++++------
 2 files changed, 10 insertions(+), 7 deletions(-)
---
base-commit: e50525a486300e5ff8014c9985b8477b2fca0368
change-id: 20261109-devx-umem-mailbox-overflow-c741bc73deb4

Best regards,
-- 
Edward Srouji <edwards@nvidia.com>

[PATCH rdma-next 1/2] RDMA/mlx5: Fix DEVX UMEM mailbox size truncation

[Edward Srouji]

From: Uri Dar <udar@nvidia.com>

devx_umem_reg_cmd_alloc() stores the CREATE_UMEM mailbox size, a header
plus 8 bytes per MTT, in the u32 cmd->inlen. The number of MTTs is
derived from the user-supplied length and pgsz_bitmap, so a UMEM of about
2TB registered with 4KB pages wraps the size, the mailbox is
under-allocated and mlx5_ib_populate_pas() writes past its end.

Store the size in an int, the in_size type of mlx5_cmd_exec(), where a
larger value would turn negative, and compute it with overflow checks so
a UMEM whose mailbox does not fit is rejected.

Mailbox sizes within 512 bytes of INT_MAX still overflow the block count
in mlx5_calc_cmd_blocks(), but the command then fails with -ENOMEM
before reaching firmware. This predates the change.

Fixes: aeae94579caf ("IB/mlx5: Add DEVX support for memory registration")
Cc: stable@vger.kernel.org
Signed-off-by: Uri Dar <udar@nvidia.com>
Reviewed-by: Michael Gur <michaelgur@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
---
 drivers/infiniband/hw/mlx5/devx.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/devx.c b/drivers/infiniband/hw/mlx5/devx.c
index fc6f793dcc6584e6a22b40c57e2df8f24579eb14..1da6c0e440d6e19d8791b012a148f479cfdddad7 100644
--- a/drivers/infiniband/hw/mlx5/devx.c
+++ b/drivers/infiniband/hw/mlx5/devx.c
@@ -113,7 +113,7 @@ struct devx_umem {
 
 struct devx_umem_reg_cmd {
 	void				*in;
-	u32				inlen;
+	int				inlen;
 	u32				out[MLX5_ST_SZ_DW(create_umem_out)];
 };
 
@@ -2334,6 +2334,7 @@ static int devx_umem_reg_cmd_alloc(struct mlx5_ib_dev *dev,
 {
 	unsigned long pgsz_bitmap;
 	unsigned int page_size;
+	size_t num_mtt;
 	__be64 *mtt;
 	void *umem;
 	int ret;
@@ -2361,9 +2362,12 @@ static int devx_umem_reg_cmd_alloc(struct mlx5_ib_dev *dev,
 	if (!page_size)
 		return -EINVAL;
 
-	cmd->inlen = MLX5_ST_SZ_BYTES(create_umem_in) +
-		     (MLX5_ST_SZ_BYTES(mtt) *
-		      ib_umem_num_dma_blocks(obj->umem, page_size));
+	num_mtt = ib_umem_num_dma_blocks(obj->umem, page_size);
+	if (check_add_overflow(MLX5_ST_SZ_BYTES(create_umem_in),
+			       size_mul(MLX5_ST_SZ_BYTES(mtt), num_mtt),
+			       &cmd->inlen))
+		return -EINVAL;
+
 	cmd->in = uverbs_zalloc(attrs, cmd->inlen);
 	if (IS_ERR(cmd->in))
 		return PTR_ERR(cmd->in);
@@ -2372,8 +2376,7 @@ static int devx_umem_reg_cmd_alloc(struct mlx5_ib_dev *dev,
 	mtt = (__be64 *)MLX5_ADDR_OF(umem, umem, mtt);
 
 	MLX5_SET(create_umem_in, cmd->in, opcode, MLX5_CMD_OP_CREATE_UMEM);
-	MLX5_SET64(umem, umem, num_of_mtt,
-		   ib_umem_num_dma_blocks(obj->umem, page_size));
+	MLX5_SET64(umem, umem, num_of_mtt, num_mtt);
 	MLX5_SET(umem, umem, log_page_size,
 		 order_base_2(page_size) - MLX5_ADAPTER_PAGE_SHIFT);
 	MLX5_SET(umem, umem, page_offset,

-- 
2.49.0

Re: [PATCH rdma-next 1/2] RDMA/mlx5: Fix DEVX UMEM mailbox size truncation

[Jason Gunthorpe]

On Mon, Nov 09, 2026 at 04:42:32PM +0200, Edward Srouji wrote:
> From: Uri Dar <udar@nvidia.com>
> 
> devx_umem_reg_cmd_alloc() stores the CREATE_UMEM mailbox size, a header
> plus 8 bytes per MTT, in the u32 cmd->inlen. The number of MTTs is
> derived from the user-supplied length and pgsz_bitmap, so a UMEM of about
> 2TB registered with 4KB pages wraps the size, the mailbox is
> under-allocated and mlx5_ib_populate_pas() writes past its end.
> 
> Store the size in an int, the in_size type of mlx5_cmd_exec(), where a
> larger value would turn negative, and compute it with overflow checks so
> a UMEM whose mailbox does not fit is rejected.
> 
> Mailbox sizes within 512 bytes of INT_MAX still overflow the block count
> in mlx5_calc_cmd_blocks(), but the command then fails with -ENOMEM
> before reaching firmware. This predates the change.
> 
> Fixes: aeae94579caf ("IB/mlx5: Add DEVX support for memory registration")
> Cc: stable@vger.kernel.org
> Signed-off-by: Uri Dar <udar@nvidia.com>
> Reviewed-by: Michael Gur <michaelgur@nvidia.com>
> Signed-off-by: Edward Srouji <edwards@nvidia.com>
> ---
>  drivers/infiniband/hw/mlx5/devx.c | 15 +++++++++------
>  1 file changed, 9 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/infiniband/hw/mlx5/devx.c b/drivers/infiniband/hw/mlx5/devx.c
> index fc6f793dcc6584e6a22b40c57e2df8f24579eb14..1da6c0e440d6e19d8791b012a148f479cfdddad7 100644
> --- a/drivers/infiniband/hw/mlx5/devx.c
> +++ b/drivers/infiniband/hw/mlx5/devx.c
> @@ -113,7 +113,7 @@ struct devx_umem {
>  
>  struct devx_umem_reg_cmd {
>  	void				*in;
> -	u32				inlen;
> +	int				inlen;

Do not change things which must be unsigned into signed values.

Jason

Re: [PATCH rdma-next 1/2] RDMA/mlx5: Fix DEVX UMEM mailbox size truncation

[Edward Srouji]

On 11/9/2026 4:48 PM, Jason Gunthorpe wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Mon, Nov 09, 2026 at 04:42:32PM +0200, Edward Srouji wrote:
>> From: Uri Dar <udar@nvidia.com>
>>
>> devx_umem_reg_cmd_alloc() stores the CREATE_UMEM mailbox size, a header
>> plus 8 bytes per MTT, in the u32 cmd->inlen. The number of MTTs is
>> derived from the user-supplied length and pgsz_bitmap, so a UMEM of about
>> 2TB registered with 4KB pages wraps the size, the mailbox is
>> under-allocated and mlx5_ib_populate_pas() writes past its end.
>>
>> Store the size in an int, the in_size type of mlx5_cmd_exec(), where a
>> larger value would turn negative, and compute it with overflow checks so
>> a UMEM whose mailbox does not fit is rejected.
>>
>> Mailbox sizes within 512 bytes of INT_MAX still overflow the block count
>> in mlx5_calc_cmd_blocks(), but the command then fails with -ENOMEM
>> before reaching firmware. This predates the change.
>>
>> Fixes: aeae94579caf ("IB/mlx5: Add DEVX support for memory registration")
>> Cc: stable@vger.kernel.org
>> Signed-off-by: Uri Dar <udar@nvidia.com>
>> Reviewed-by: Michael Gur <michaelgur@nvidia.com>
>> Signed-off-by: Edward Srouji <edwards@nvidia.com>
>> ---
>>   drivers/infiniband/hw/mlx5/devx.c | 15 +++++++++------
>>   1 file changed, 9 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/infiniband/hw/mlx5/devx.c b/drivers/infiniband/hw/mlx5/devx.c
>> index fc6f793dcc6584e6a22b40c57e2df8f24579eb14..1da6c0e440d6e19d8791b012a148f479cfdddad7 100644
>> --- a/drivers/infiniband/hw/mlx5/devx.c
>> +++ b/drivers/infiniband/hw/mlx5/devx.c
>> @@ -113,7 +113,7 @@ struct devx_umem {
>>
>>   struct devx_umem_reg_cmd {
>>        void                            *in;
>> -     u32                             inlen;
>> +     int                             inlen;
> 
> Do not change things which must be unsigned into signed values.

I asked to use check_add_overflow() but it was dumb to ignore this from 
my side. Will fix in V2.

> 
> Jason

Re: [PATCH rdma-next 1/2] RDMA/mlx5: Fix DEVX UMEM mailbox size truncation

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found no new issues
in this patch, but noticed 1 related pre-existing issue it does not
introduce:
- [Medium] RDMA/mlx5: devx_umem_find_best_pgsize() ignores pgsz_bitmap
  after halving page_size [1]
--

commit bd5b83068caa4612b47dcf836a1a42f89cbb5cc5
Author: Uri Dar <udar@nvidia.com>

RDMA/mlx5: Fix DEVX UMEM mailbox size truncation

This commit changes cmd->inlen in struct devx_umem_reg_cmd to int and adds
overflow checks when computing the CREATE_UMEM mailbox size in
devx_umem_reg_cmd_alloc().

> diff --git a/drivers/infiniband/hw/mlx5/devx.c b/drivers/infiniband/hw/mlx5/devx.c
> index fc6f793dcc658..1da6c0e440d6e 100644
> --- a/drivers/infiniband/hw/mlx5/devx.c
> +++ b/drivers/infiniband/hw/mlx5/devx.c

[ ... ]

> @@ -2361,9 +2362,12 @@ static int devx_umem_reg_cmd_alloc(struct mlx5_ib_dev *dev,
>  	if (!page_size)
>  		return -EINVAL;
>  
> -	cmd->inlen = MLX5_ST_SZ_BYTES(create_umem_in) +
> -		     (MLX5_ST_SZ_BYTES(mtt) *
> -		      ib_umem_num_dma_blocks(obj->umem, page_size));
> +	num_mtt = ib_umem_num_dma_blocks(obj->umem, page_size);
> +	if (check_add_overflow(MLX5_ST_SZ_BYTES(create_umem_in),
> +			       size_mul(MLX5_ST_SZ_BYTES(mtt), num_mtt),
> +			       &cmd->inlen))
> +		return -EINVAL;

[Severity: Medium]
This problem wasn't introduced by this patch, but can
devx_umem_find_best_pgsize() return a page_size that is not set in
pgsz_bitmap [1]?

drivers/infiniband/hw/mlx5/devx.c:devx_umem_find_best_pgsize() {
	...
	page_size = ib_umem_find_best_pgoff(umem, pgsz_bitmap, U64_MAX);
	if (!page_size)
		return 0;
	...
	while ((ib_umem_dma_offset(umem, page_size) != 0 ||
		(umem->length % page_size) != 0) &&
		page_size > PAGE_SIZE)
		page_size /= 2;

	return page_size;
}

When a physically contiguous umem is registered with a restricted
MLX5_IB_ATTR_DEVX_UMEM_REG_PGSZ_BITMAP, ib_umem_find_best_pgoff() selects
the largest matching bit regardless of DMA offset or total length, and the
loop then halves page_size without checking that the reduced size is still
present in pgsz_bitmap.  devx_umem_reg_cmd_alloc() then registers the umem
with a page size that userspace excluded instead of returning -EINVAL.

[1] https://sashiko.dev/#/bug/linux-6cd20db6-58c6-4b3d-8dd9-c9ee893d4041

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261109-devx-umem-mailbox-overflow-v1-0-4a63baab25dc@nvidia.com?part=1

Re: [PATCH rdma-next 1/2] RDMA/mlx5: Fix DEVX UMEM mailbox size truncation

[Edward Srouji]

On 11/9/2026 4:52 PM, sashiko-bot@kernel.org wrote:
> External email: Use caution opening links or attachments
> 
> 
> Thank you for your contribution! Sashiko AI review found no new issues
> in this patch, but noticed 1 related pre-existing issue it does not
> introduce:
> - [Medium] RDMA/mlx5: devx_umem_find_best_pgsize() ignores pgsz_bitmap
>    after halving page_size [1]
> --
> 
> commit bd5b83068caa4612b47dcf836a1a42f89cbb5cc5
> Author: Uri Dar <udar@nvidia.com>
> 
> RDMA/mlx5: Fix DEVX UMEM mailbox size truncation
> 
> This commit changes cmd->inlen in struct devx_umem_reg_cmd to int and adds
> overflow checks when computing the CREATE_UMEM mailbox size in
> devx_umem_reg_cmd_alloc().
> 
>> diff --git a/drivers/infiniband/hw/mlx5/devx.c b/drivers/infiniband/hw/mlx5/devx.c
>> index fc6f793dcc658..1da6c0e440d6e 100644
>> --- a/drivers/infiniband/hw/mlx5/devx.c
>> +++ b/drivers/infiniband/hw/mlx5/devx.c
> 
> [ ... ]
> 
>> @@ -2361,9 +2362,12 @@ static int devx_umem_reg_cmd_alloc(struct mlx5_ib_dev *dev,
>>        if (!page_size)
>>                return -EINVAL;
>>
>> -     cmd->inlen = MLX5_ST_SZ_BYTES(create_umem_in) +
>> -                  (MLX5_ST_SZ_BYTES(mtt) *
>> -                   ib_umem_num_dma_blocks(obj->umem, page_size));
>> +     num_mtt = ib_umem_num_dma_blocks(obj->umem, page_size);
>> +     if (check_add_overflow(MLX5_ST_SZ_BYTES(create_umem_in),
>> +                            size_mul(MLX5_ST_SZ_BYTES(mtt), num_mtt),
>> +                            &cmd->inlen))
>> +             return -EINVAL;
> 
> [Severity: Medium]
> This problem wasn't introduced by this patch, but can
> devx_umem_find_best_pgsize() return a page_size that is not set in
> pgsz_bitmap [1]?
> 
> drivers/infiniband/hw/mlx5/devx.c:devx_umem_find_best_pgsize() {
>          ...
>          page_size = ib_umem_find_best_pgoff(umem, pgsz_bitmap, U64_MAX);
>          if (!page_size)
>                  return 0;
>          ...
>          while ((ib_umem_dma_offset(umem, page_size) != 0 ||
>                  (umem->length % page_size) != 0) &&
>                  page_size > PAGE_SIZE)
>                  page_size /= 2;
> 
>          return page_size;
> }
> 
> When a physically contiguous umem is registered with a restricted
> MLX5_IB_ATTR_DEVX_UMEM_REG_PGSZ_BITMAP, ib_umem_find_best_pgoff() selects
> the largest matching bit regardless of DMA offset or total length, and the
> loop then halves page_size without checking that the reduced size is still
> present in pgsz_bitmap.  devx_umem_reg_cmd_alloc() then registers the umem
> with a page size that userspace excluded instead of returning -EINVAL.
> 
> [1] https://sashiko.dev/#/bug/linux-6cd20db6-58c6-4b3d-8dd9-c9ee893d4041
> 

Sashiko is correct. Since it's a small local fix, I'll patch it in V2.

> --
> Sashiko AI review · https://sashiko.dev/#/patchset/20261109-devx-umem-mailbox-overflow-v1-0-4a63baab25dc@nvidia.com?part=1

[PATCH rdma-next 2/2] RDMA/core: Don't warn on oversized bundle allocations

[Edward Srouji]

From: Uri Dar <udar@nvidia.com>

_uverbs_alloc() serves requests that do not fit in the bundle's internal
buffer from kvmalloc(), prepending a small list header. The requested
size can be derived from userspace input, and since commit 7661809d493b
("mm: don't allow oversized kvmalloc() calls") kvmalloc() WARNs on any
request above INT_MAX unless __GFP_NOWARN is given.

mlx5 DEVX UMEM registration is one such caller: its CREATE_UMEM mailbox
scales with the size of the user's UMEM, and a mailbox that passes the
driver's INT_MAX check can still exceed INT_MAX once the header is
added.

As with memdup_user(), an allocation sized by userspace should fail
quietly rather than warn. Pass __GFP_NOWARN so such requests fail with
-ENOMEM.

Fixes: 461bb2eee4e1 ("IB/uverbs: Add a simple allocator to uverbs_attr_bundle")
Signed-off-by: Uri Dar <udar@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
---
 drivers/infiniband/core/ib_core_uverbs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/infiniband/core/ib_core_uverbs.c b/drivers/infiniband/core/ib_core_uverbs.c
index 74d9c006bad5aa5d8366b1b75843c9873c62c344..c41b77d7084034613678e8b4b25bc95f72c8e241 100644
--- a/drivers/infiniband/core/ib_core_uverbs.c
+++ b/drivers/infiniband/core/ib_core_uverbs.c
@@ -554,7 +554,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size,
 	if (new_used > pbundle->internal_avail) {
 		struct bundle_alloc_head *buf;
 
-		buf = kvmalloc_flex(*buf, data, size, flags);
+		buf = kvmalloc_flex(*buf, data, size, flags | __GFP_NOWARN);
 		if (!buf)
 			return ERR_PTR(-ENOMEM);
 		buf->next = pbundle->allocated_mem;

-- 
2.49.0

Re: [PATCH rdma-next 2/2] RDMA/core: Don't warn on oversized bundle allocations

[sashiko-bot]

> From: Uri Dar <udar@nvidia.com>
> 
> _uverbs_alloc() serves requests that do not fit in the bundle's internal
> buffer from kvmalloc(), prepending a small list header. The requested
> size can be derived from userspace input, and since commit 7661809d493b
> ("mm: don't allow oversized kvmalloc() calls") kvmalloc() WARNs on any
> request above INT_MAX unless __GFP_NOWARN is given.
> 
> mlx5 DEVX UMEM registration is one such caller: its CREATE_UMEM mailbox
> scales with the size of the user's UMEM, and a mailbox that passes the
> driver's INT_MAX check can still exceed INT_MAX once the header is
> added.
> 
> As with memdup_user(), an allocation sized by userspace should fail
> quietly rather than warn. Pass __GFP_NOWARN so such requests fail with
> -ENOMEM.
> 
> Fixes: 461bb2eee4e1 ("IB/uverbs: Add a simple allocator to uverbs_attr_bundle")
> Signed-off-by: Uri Dar <udar@nvidia.com>
> Signed-off-by: Edward Srouji <edwards@nvidia.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261109-devx-umem-mailbox-overflow-v1-0-4a63baab25dc@nvidia.com?part=2