[PATCH rdma-next 0/2] RDMA/mlx5: Fix user-triggerable overflows in QP creation

[PATCH rdma-next 0/2] RDMA/mlx5: Fix user-triggerable overflows in QP creation

[Edward Srouji]

This short series fixes two undefined behavior bugs in user QP creation
path.

Signed-off-by: Edward Srouji <edwards@nvidia.com>
---
Maher Sanalla (2):
      RDMA/mlx5: Fix undefined shift of user RQ WQE size
      RDMA/mlx5: Fix integer overflow of user QP buffer size

 drivers/infiniband/hw/mlx5/qp.c | 54 ++++++++++++++++++++++++++++++++---------
 1 file changed, 42 insertions(+), 12 deletions(-)
---
base-commit: 20ff9350862468af21b46cae2c22d17d6ec637f9
change-id: 20260611-maher-sec-fixes-4cd89b9fe4bd

Best regards,
-- 
Edward Srouji <edwards@nvidia.com>

[PATCH rdma-next 1/2] RDMA/mlx5: Fix undefined shift of user RQ WQE size

[Edward Srouji]

From: Maher Sanalla <msanalla@nvidia.com>

set_rq_size() computes the RQ WQE size as "1 << rq_wqe_shift" based on
the user-provided rq_wqe_shift, which is only checked to be greater than
32, so shifts of 32 are still accepted. A shift of 31 also overflows a
signed integer, leading to undefined behavior.

Use check_shl_overflow() to compute the RQ WQE size and reject any
invalid values.

Fixes: e126ba97dba9 ("mlx5: Add driver for Mellanox Connect-IB adapters")
Signed-off-by: Maher Sanalla <msanalla@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
---
 drivers/infiniband/hw/mlx5/qp.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index e8d34d54b43527e0595ec9e2fb93dc7e9bedba92..7674290d0afaf466a6b98cbed86d247ee550bd8d 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -454,16 +454,13 @@ static int set_rq_size(struct mlx5_ib_dev *dev, struct ib_qp_cap *cap,
 
 		if (ucmd) {
 			qp->rq.wqe_cnt = ucmd->rq_wqe_count;
-			if (ucmd->rq_wqe_shift > BITS_PER_BYTE * sizeof(ucmd->rq_wqe_shift))
-				return -EINVAL;
 			qp->rq.wqe_shift = ucmd->rq_wqe_shift;
-			if ((1 << qp->rq.wqe_shift) /
-				    sizeof(struct mlx5_wqe_data_seg) <
-			    wq_sig)
+			if (check_shl_overflow(1, qp->rq.wqe_shift, &wqe_size))
+				return -EINVAL;
+			if (wqe_size / sizeof(struct mlx5_wqe_data_seg) < wq_sig)
 				return -EINVAL;
 			qp->rq.max_gs =
-				(1 << qp->rq.wqe_shift) /
-					sizeof(struct mlx5_wqe_data_seg) -
+				wqe_size / sizeof(struct mlx5_wqe_data_seg) -
 				wq_sig;
 			qp->rq.max_post = qp->rq.wqe_cnt;
 		} else {

-- 
2.49.0

Re: [PATCH rdma-next 1/2] RDMA/mlx5: Fix undefined shift of user RQ WQE size

[Jason Gunthorpe]

On Thu, Jun 11, 2026 at 03:50:42PM +0300, Edward Srouji wrote:
> From: Maher Sanalla <msanalla@nvidia.com>
> 
> set_rq_size() computes the RQ WQE size as "1 << rq_wqe_shift" based on
> the user-provided rq_wqe_shift, which is only checked to be greater than
> 32, so shifts of 32 are still accepted. A shift of 31 also overflows a
> signed integer, leading to undefined behavior.
> 
> Use check_shl_overflow() to compute the RQ WQE size and reject any
> invalid values.
> 
> Fixes: e126ba97dba9 ("mlx5: Add driver for Mellanox Connect-IB adapters")
> Signed-off-by: Maher Sanalla <msanalla@nvidia.com>
> Signed-off-by: Edward Srouji <edwards@nvidia.com>
> ---
>  drivers/infiniband/hw/mlx5/qp.c | 11 ++++-------
>  1 file changed, 4 insertions(+), 7 deletions(-)

Applied just this one, please resend the other one

Thanks,
Jason

[PATCH rdma-next 2/2] RDMA/mlx5: Fix integer overflow of user QP buffer size

[Edward Srouji]

From: Maher Sanalla <msanalla@nvidia.com>

set_user_buf_size() calculates the QP buffer size by left‑shifting the
user‑provided rq.wqe_cnt and rq.wqe_shift as signed integers. A large
rq.wqe_cnt can trigger a signed integer overflow, which is undefined
behavior and may yield a small or even negative buf_size. This can lead
ib_umem_get() to map a buffer smaller than what the hardware will write.

Replace the existing shifts and additions with check_shl_overflow() and
check_add_overflow(), and reject invalid user inputs.

Apply the same checks to the calculation used for qp->sq.offset in
_create_user_qp(). Even though set_user_buf_size() validates this again,
the guard protects us against future changes in the internal
implementation.

Fixes: e126ba97dba9 ("mlx5: Add driver for Mellanox Connect-IB adapters")
Signed-off-by: Maher Sanalla <msanalla@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
---
 drivers/infiniband/hw/mlx5/qp.c | 43 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 38 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 7674290d0afaf466a6b98cbed86d247ee550bd8d..6ecdbda2b471f6c102bceba5d02eb12af8d8e1b1 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -640,6 +640,8 @@ static int set_user_buf_size(struct mlx5_ib_dev *dev,
 			    struct ib_qp_init_attr *attr)
 {
 	int desc_sz = 1 << qp->sq.wqe_shift;
+	int rq_buf_size;
+	int sq_buf_size;
 
 	if (desc_sz > MLX5_CAP_GEN(dev->mdev, max_wqe_sz_sq)) {
 		mlx5_ib_warn(dev, "desc_sz %d, max_sq_desc_sz %d\n",
@@ -664,11 +666,36 @@ static int set_user_buf_size(struct mlx5_ib_dev *dev,
 
 	if (attr->qp_type == IB_QPT_RAW_PACKET ||
 	    qp->flags & IB_QP_CREATE_SOURCE_QPN) {
-		base->ubuffer.buf_size = qp->rq.wqe_cnt << qp->rq.wqe_shift;
-		qp->raw_packet_qp.sq.ubuffer.buf_size = qp->sq.wqe_cnt << 6;
+		if (check_shl_overflow(qp->rq.wqe_cnt, qp->rq.wqe_shift,
+				       &base->ubuffer.buf_size)) {
+			mlx5_ib_warn(dev, "rq buf size overflow: wqe_cnt %d wqe_shift %d\n",
+				     qp->rq.wqe_cnt, qp->rq.wqe_shift);
+			return -EINVAL;
+		}
+		if (check_shl_overflow(qp->sq.wqe_cnt, 6,
+				       &qp->raw_packet_qp.sq.ubuffer.buf_size)) {
+			mlx5_ib_warn(dev, "sq buf size overflow: wqe_cnt %d\n",
+				     qp->sq.wqe_cnt);
+			return -EINVAL;
+		}
 	} else {
-		base->ubuffer.buf_size = (qp->rq.wqe_cnt << qp->rq.wqe_shift) +
-					 (qp->sq.wqe_cnt << 6);
+		if (check_shl_overflow(qp->rq.wqe_cnt, qp->rq.wqe_shift,
+				       &rq_buf_size)) {
+			mlx5_ib_warn(dev, "rq buf size overflow: wqe_cnt %d wqe_shift %d\n",
+				     qp->rq.wqe_cnt, qp->rq.wqe_shift);
+			return -EINVAL;
+		}
+		if (check_shl_overflow(qp->sq.wqe_cnt, 6, &sq_buf_size)) {
+			mlx5_ib_warn(dev, "sq buf size overflow: wqe_cnt %d\n",
+				     qp->sq.wqe_cnt);
+			return -EINVAL;
+		}
+		if (check_add_overflow(rq_buf_size, sq_buf_size,
+				       &base->ubuffer.buf_size)) {
+			mlx5_ib_warn(dev, "qp buf size overflow: rq %d sq %d\n",
+				     rq_buf_size, sq_buf_size);
+			return -EINVAL;
+		}
 	}
 
 	return 0;
@@ -997,7 +1024,13 @@ static int _create_user_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd,
 
 	qp->rq.offset = 0;
 	qp->sq.wqe_shift = ilog2(MLX5_SEND_WQE_BB);
-	qp->sq.offset = qp->rq.wqe_cnt << qp->rq.wqe_shift;
+	if (check_shl_overflow(qp->rq.wqe_cnt, qp->rq.wqe_shift,
+			       &qp->sq.offset)) {
+		mlx5_ib_warn(dev, "sq offset overflow: wqe_cnt %d wqe_shift %d\n",
+			     qp->rq.wqe_cnt, qp->rq.wqe_shift);
+		err = -EINVAL;
+		goto err_bfreg;
+	}
 
 	err = set_user_buf_size(dev, qp, ucmd, base, attr);
 	if (err)

-- 
2.49.0

Re: [PATCH rdma-next 2/2] RDMA/mlx5: Fix integer overflow of user QP buffer size

[Jason Gunthorpe]

On Thu, Jun 11, 2026 at 03:50:43PM +0300, Edward Srouji wrote:
> @@ -664,11 +666,36 @@ static int set_user_buf_size(struct mlx5_ib_dev *dev,
>  
>  	if (attr->qp_type == IB_QPT_RAW_PACKET ||
>  	    qp->flags & IB_QP_CREATE_SOURCE_QPN) {
> -		base->ubuffer.buf_size = qp->rq.wqe_cnt << qp->rq.wqe_shift;
> -		qp->raw_packet_qp.sq.ubuffer.buf_size = qp->sq.wqe_cnt << 6;
> +		if (check_shl_overflow(qp->rq.wqe_cnt, qp->rq.wqe_shift,
> +				       &base->ubuffer.buf_size)) {
> +			mlx5_ib_warn(dev, "rq buf size overflow: wqe_cnt %d wqe_shift %d\n",
> +				     qp->rq.wqe_cnt, qp->rq.wqe_shift);
> +			return -EINVAL;

No prints triggerable by uapi.

Jason

Re: [PATCH rdma-next 2/2] RDMA/mlx5: Fix integer overflow of user QP buffer size

[Edward Srouji]

On 6/11/2026 10:17 PM, Jason Gunthorpe wrote:
> On Thu, Jun 11, 2026 at 03:50:43PM +0300, Edward Srouji wrote:
>> @@ -664,11 +666,36 @@ static int set_user_buf_size(struct mlx5_ib_dev *dev,
>>   
>>   	if (attr->qp_type == IB_QPT_RAW_PACKET ||
>>   	    qp->flags & IB_QP_CREATE_SOURCE_QPN) {
>> -		base->ubuffer.buf_size = qp->rq.wqe_cnt << qp->rq.wqe_shift;
>> -		qp->raw_packet_qp.sq.ubuffer.buf_size = qp->sq.wqe_cnt << 6;
>> +		if (check_shl_overflow(qp->rq.wqe_cnt, qp->rq.wqe_shift,
>> +				       &base->ubuffer.buf_size)) {
>> +			mlx5_ib_warn(dev, "rq buf size overflow: wqe_cnt %d wqe_shift %d\n",
>> +				     qp->rq.wqe_cnt, qp->rq.wqe_shift);
>> +			return -EINVAL;
> 
> No prints triggerable by uapi.
> 
Right, will drop them.
Note that set_user_buf_size() already has a pre-existing mlx5_ib_warn() 
prints, which is equally uapi-triggerable.
Should we clean that up in a separate patch? Should we drop such prints 
entirely? or convert them to mlx5_ib_dbg()?

> Jason

Re: [PATCH rdma-next 2/2] RDMA/mlx5: Fix integer overflow of user QP buffer size

[Jason Gunthorpe]

On Sun, Jun 14, 2026 at 02:23:00PM +0300, Edward Srouji wrote:
> 
> 
> On 6/11/2026 10:17 PM, Jason Gunthorpe wrote:
> > On Thu, Jun 11, 2026 at 03:50:43PM +0300, Edward Srouji wrote:
> > > @@ -664,11 +666,36 @@ static int set_user_buf_size(struct mlx5_ib_dev *dev,
> > >   	if (attr->qp_type == IB_QPT_RAW_PACKET ||
> > >   	    qp->flags & IB_QP_CREATE_SOURCE_QPN) {
> > > -		base->ubuffer.buf_size = qp->rq.wqe_cnt << qp->rq.wqe_shift;
> > > -		qp->raw_packet_qp.sq.ubuffer.buf_size = qp->sq.wqe_cnt << 6;
> > > +		if (check_shl_overflow(qp->rq.wqe_cnt, qp->rq.wqe_shift,
> > > +				       &base->ubuffer.buf_size)) {
> > > +			mlx5_ib_warn(dev, "rq buf size overflow: wqe_cnt %d wqe_shift %d\n",
> > > +				     qp->rq.wqe_cnt, qp->rq.wqe_shift);
> > > +			return -EINVAL;
> > 
> > No prints triggerable by uapi.
> > 
> Right, will drop them.
> Note that set_user_buf_size() already has a pre-existing mlx5_ib_warn()
> prints, which is equally uapi-triggerable.
> Should we clean that up in a separate patch? Should we drop such prints
> entirely? or convert them to mlx5_ib_dbg()?

Yes clean them up, up to you if you want dbg versions

Jason