[PATCH rdma-next 00/15] DMA direction and mlx5 driver correctness fixes

[PATCH rdma-next 00/15] DMA direction and mlx5 driver correctness fixes

[Yishai Hadas]

This series contains two independent groups of fixes for the RDMA
subsystem.

Group 1: DMA direction correctness (patches 1-7).

RDMA drivers have historically mapped all pinned user buffers
DMA_BIDIRECTIONAL regardless of how the device actually accesses them.
This group tightens the DMA mapping direction to match the real access
pattern for each buffer type:

  - CQ ring buffers: the device writes CQEs, the CPU only reads.
    Mapped DMA_FROM_DEVICE.

  - WQE rings, SRQ rings, doorbell records: the CPU writes, the device
    only reads.  Mapped DMA_TO_DEVICE (derived from the absence of any
    write access flag).

  - Writable MRs and other buffers with write access flags: unchanged,
    DMA_BIDIRECTIONAL.

On platforms with a write-enforcing IOMMU this prevents the device from
accessing memory beyond what the mapping semantically declares.  On
standard deployments without direction enforcement the change declares
the correct semantic at zero runtime cost.

The group includes preparatory driver fixes (patches 1-3) that correct
existing access=0 pins on writable buffers in erdma, hns, and
vmw_pvrdma, and a refactoring patch (4) that consolidates all CQ buffer
pinning through ib_umem_get_cq_buf_or_va() so the directional change in
patch 6 covers every CQ call site automatically.  Full details are in
the individual commit logs.

Group 2: mlx5 driver correctness fixes (patches 8-15).

Patch 8 fixes driver probe failure on XRC-less devices.
mlx5_ib_dev_res_init() was returning -EOPNOTSUPP when the firmware XRC
capability is absent. XRC is optional; the patch makes the devr
resource path XRC-optional throughout.

Patches 9-14 fix a cluster of related bugs in the mlx5 firmware event
dispatch path. The common root cause is that resources were made
visible to the event notifier (via radix tree or xarray insertion)
before their event callback, container pointer, and refcount were fully
initialized, leaving windows where firmware EQEs could call through NULL
function pointers or corrupt refcounts:

- mlx5_ib_wq_event() never released the reference taken before
    dispatch, leaking it on every WQ event.
  - WQ, SRQ, QP, and raw-packet QP SQ/RQ event callbacks were
    assigned after the resource became visible to the notifier.
    For SRQ and QP paths this directly causes a kernel crash on any
    matching firmware event.
  - create_resource_common() initialized the refcount and completion
    after the radix tree insert, allowing refcount_inc() on a
    still-zero refcount_t, which saturates the counter and permanently
    corrupts the resource's lifecycle tracking.

Patch 15 fixes undefined behavior in the EQE type field shift: the u8
type is left-shifted by 24 without a cast, promoting it to signed int
and causing UB for values >= 128.  All currently used type codes are
below 128, but a cast to u32 makes the shift well-defined.

Yishai

Yishai Hadas (15):
  RDMA/erdma: Pin CQ buffer writable to match device DMA write access
  RDMA/hns: Pin CQ buffer writable to match device DMA write access
  RDMA/vmw_pvrdma: Pin QP and SRQ rings writable to match device DMA
    write access
  RDMA/umem: Reuse ib_umem_get_cq_buf_or_va() for VA-only CQ pinning
  RDMA/umem: Support an explicit DMA direction other than
    DMA_BIDIRECTIONAL
  RDMA/umem: Map CQ buffers DMA_FROM_DEVICE
  RDMA/umem: Derive DMA direction from IB access flags
  RDMA/mlx5: Fix mlx5_ib_dev_res_init() failure when XRC cap is absent
  RDMA/mlx5: Put resource reference in mlx5_ib_wq_event()
  RDMA/mlx5: Set WQ event handler before firmware RQ insertion
  RDMA/mlx5: Set SRQ event handler before xarray insertion
  RDMA/mlx5: Set RQ event handler for raw-packet QP
  RDMA/mlx5: Set QP event handler before firmware QPC insertion
  RDMA/mlx5: Initialize QP/RQ/SQ resource refcount before publishing it
  RDMA/mlx5: Fix signed integer overflow in EQE qp_srq type shift

 drivers/infiniband/core/umem.c                | 87 ++++++++++++++-----
 drivers/infiniband/hw/bnxt_re/ib_verbs.c      |  9 +-
 drivers/infiniband/hw/erdma/erdma_verbs.c     | 19 ++--
 drivers/infiniband/hw/hns/hns_roce_cq.c       |  3 +-
 drivers/infiniband/hw/hns/hns_roce_device.h   |  2 +-
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c    |  2 +-
 drivers/infiniband/hw/hns/hns_roce_mr.c       | 22 +++--
 drivers/infiniband/hw/hns/hns_roce_qp.c       |  2 +-
 drivers/infiniband/hw/hns/hns_roce_srq.c      |  4 +-
 .../infiniband/hw/ionic/ionic_controlpath.c   |  5 +-
 drivers/infiniband/hw/mana/cq.c               |  2 +-
 drivers/infiniband/hw/mana/main.c             |  9 +-
 drivers/infiniband/hw/mana/mana_ib.h          |  2 +-
 drivers/infiniband/hw/mana/qp.c               |  9 +-
 drivers/infiniband/hw/mana/wq.c               |  3 +-
 drivers/infiniband/hw/mlx4/cq.c               | 14 +--
 drivers/infiniband/hw/mlx5/cq.c               |  6 +-
 drivers/infiniband/hw/mlx5/main.c             | 56 +++++++-----
 drivers/infiniband/hw/mlx5/qp.c               | 29 ++++---
 drivers/infiniband/hw/mlx5/qpc.c              | 20 +++--
 drivers/infiniband/hw/mlx5/srq.c              |  3 +-
 drivers/infiniband/hw/qedr/verbs.c            | 23 +++--
 drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c  |  6 +-
 drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c |  3 +-
 include/rdma/ib_umem.h                        |  2 +
 25 files changed, 223 insertions(+), 119 deletions(-)

-- 
2.18.1

[PATCH rdma-next 01/15] RDMA/erdma: Pin CQ buffer writable to match device DMA write access

[Yishai Hadas]

erdma_init_user_cq() pins the CQ ring with access=0. The device writes
CQEs into this buffer via DMA, but access=0 means ib_access_writable()
is false, so ib_umem_get_va() pins the pages without FOLL_WRITE and does
not mark them dirty on unpin.

Without FOLL_WRITE, pin_user_pages_fast() can hand back a page this
process does not exclusively own (e.g. the shared zero page for an
untouched anonymous mapping) instead of forcing a private copy. The
device is then free to DMA-write CQEs into that shared physical page,
corrupting memory visible to every other mapper of it. Missing the dirty
mark on unpin also risks the device's writes being silently discarded on
reclaim.

Pass IB_ACCESS_LOCAL_WRITE so the CQ buffer is pinned consistently with
how the device actually uses it.

Fixes: 155055771704 ("RDMA/erdma: Add verbs implementation")
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/hw/erdma/erdma_verbs.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c
index 65b1af1e6623..fca2553e47ad 100644
--- a/drivers/infiniband/hw/erdma/erdma_verbs.c
+++ b/drivers/infiniband/hw/erdma/erdma_verbs.c
@@ -1930,8 +1930,8 @@ static int erdma_init_user_cq(struct erdma_ucontext *ctx, struct erdma_cq *cq,
 	struct erdma_dev *dev = to_edev(cq->ibcq.device);
 
 	ret = get_mtt_entries(dev, &cq->user_cq.qbuf_mem, ureq->qbuf_va,
-			      ureq->qbuf_len, 0, ureq->qbuf_va, SZ_64M - SZ_4K,
-			      true);
+			      ureq->qbuf_len, IB_ACCESS_LOCAL_WRITE,
+			      ureq->qbuf_va, SZ_64M - SZ_4K, true);
 	if (ret)
 		return ret;
 
-- 
2.18.1

[PATCH rdma-next 02/15] RDMA/hns: Pin CQ buffer writable to match device DMA write access

[Yishai Hadas]

alloc_cq_buf() builds its hns_roce_buf_attr with user_access left at its
zero-initialized default. The device writes CQEs into this buffer via
DMA, but with user_access=0, ib_access_writable() is false, so the umem
is pinned without FOLL_WRITE and is not marked dirty on unpin.

Without FOLL_WRITE, pin_user_pages_fast() can hand back a page this
process does not exclusively own (e.g. the shared zero page for an
untouched anonymous mapping) instead of forcing a private copy. The
device is then free to DMA-write CQEs into that shared physical page,
corrupting memory visible to every other mapper of it. Missing the dirty
mark on unpin also risks the device's writes being silently discarded on
reclaim.

Set user_access to IB_ACCESS_LOCAL_WRITE so the CQ buffer is pinned
consistently with how the device actually uses it.

Fixes: 744b7bdfa79e ("RDMA/hns: Support 0 hop addressing for CQE buffer")
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/hw/hns/hns_roce_cq.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/infiniband/hw/hns/hns_roce_cq.c b/drivers/infiniband/hw/hns/hns_roce_cq.c
index 1dd0efb5620d..ae314f136732 100644
--- a/drivers/infiniband/hw/hns/hns_roce_cq.c
+++ b/drivers/infiniband/hw/hns/hns_roce_cq.c
@@ -261,6 +261,7 @@ static int alloc_cq_buf(struct hns_roce_dev *hr_dev, struct hns_roce_cq *hr_cq,
 	buf_attr.region[0].size = hr_cq->cq_depth * hr_cq->cqe_size;
 	buf_attr.region[0].hopnum = hr_dev->caps.cqe_hop_num;
 	buf_attr.region_count = 1;
+	buf_attr.user_access = IB_ACCESS_LOCAL_WRITE;
 
 	ret = hns_roce_mtr_create(hr_dev, &hr_cq->mtr, &buf_attr,
 				  hr_dev->caps.cqe_ba_pg_sz + PAGE_SHIFT,
-- 
2.18.1

[PATCH rdma-next 03/15] RDMA/vmw_pvrdma: Pin QP and SRQ rings writable to match device DMA write access

[Yishai Hadas]

pvrdma_create_qp() and pvrdma_create_srq() pin their ring buffers with
access=0. PVRDMA embeds a pvrdma_ring_state header directly inside these
buffers:

  qp->sq.ring = qp->pdir.pages[0];
  qp->rq.ring = is_srq ? NULL : &qp->sq.ring[1];

and the hypervisor writes cons_head into that header -- a field the
driver never writes itself outside of QP reset. With access=0,
ib_access_writable() is false, so ib_umem_get_va() pins the pages
without FOLL_WRITE and does not mark them dirty on unpin.

Without FOLL_WRITE, pin_user_pages_fast() can hand back a page this
process does not exclusively own (e.g. the shared zero page for an
untouched anonymous mapping) instead of forcing a private copy. The
hypervisor is then free to write cons_head into that shared physical
page, corrupting memory visible to every other mapper of it. Missing the
dirty mark on unpin also risks the hypervisor's writes being silently
discarded on reclaim.

This is not gated by any IOMMU permission check: PVRDMA is a fully
emulated device, not a real PCIe device sitting behind a guest-facing
IOMMU, so the DMA mapping direction has no enforcement effect here.  The
hypervisor already owns the guest's entire physical address space and
writes to it directly, without going through this process's page tables
or any virtual-address permission check -- the CPU's page protection
bits only gate CPU-issued load/store instructions via
virtual-to-physical translation, not a hypervisor writing to guest RAM
it already controls. FOLL_WRITE is a one-time decision made at pin time
about which physical page GUP hands back for this VA; once that page is
chosen (or the wrong one is chosen, as happens today), nothing stops a
later write to it.

PVRDMA's SRQ ring does not currently wire up its ring_state field (no
post_srq_recv is implemented), so nothing depends on it being written by
the hypervisor today. Pin it writable anyway, both for symmetry with the
QP rings this same driver embeds ring state in, and because the umem
pinning here has no way to know if that will still be true tomorrow.

Pass IB_ACCESS_LOCAL_WRITE for all three rings so they are pinned
consistently with how the hypervisor actually uses them.

Fixes: 29c8d9eba550 ("IB: Add vmw_pvrdma driver")
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c  | 6 ++++--
 drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c | 3 ++-
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c
index e939cd5ce40b..53cc49f2b8c9 100644
--- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c
+++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c
@@ -270,7 +270,8 @@ int pvrdma_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init_attr,
 				/* set qp->sq.wqe_cnt, shift, buf_size.. */
 				qp->rumem = ib_umem_get_va(ibqp->device,
 							   ucmd.rbuf_addr,
-							   ucmd.rbuf_size, 0);
+							   ucmd.rbuf_size,
+							   IB_ACCESS_LOCAL_WRITE);
 				if (IS_ERR(qp->rumem)) {
 					ret = PTR_ERR(qp->rumem);
 					goto err_qp;
@@ -282,7 +283,8 @@ int pvrdma_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init_attr,
 			}
 
 			qp->sumem = ib_umem_get_va(ibqp->device, ucmd.sbuf_addr,
-						   ucmd.sbuf_size, 0);
+						   ucmd.sbuf_size,
+						   IB_ACCESS_LOCAL_WRITE);
 			if (IS_ERR(qp->sumem)) {
 				if (!is_srq)
 					ib_umem_release(qp->rumem);
diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c
index 345ec486a223..3252c2eb405a 100644
--- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c
+++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c
@@ -146,7 +146,8 @@ int pvrdma_create_srq(struct ib_srq *ibsrq, struct ib_srq_init_attr *init_attr,
 	if (ret)
 		goto err_srq;
 
-	srq->umem = ib_umem_get_va(ibsrq->device, ucmd.buf_addr, ucmd.buf_size, 0);
+	srq->umem = ib_umem_get_va(ibsrq->device, ucmd.buf_addr, ucmd.buf_size,
+				   IB_ACCESS_LOCAL_WRITE);
 	if (IS_ERR(srq->umem)) {
 		ret = PTR_ERR(srq->umem);
 		goto err_srq;
-- 
2.18.1

[PATCH rdma-next 04/15] RDMA/umem: Reuse ib_umem_get_cq_buf_or_va() for VA-only CQ pinning

[Yishai Hadas]

Several drivers pin a CQ ring buffer directly from a plain UHW VA
instead of going through the per-command UMEM attribute machinery,
calling the generic ib_umem_get_va() with IB_ACCESS_LOCAL_WRITE:

  - mlx5's resize_user() (CQ resize)
  - mlx4's legacy fallback in mlx4_ib_create_user_cq() (no UMEM
    attribute source available) and mlx4_alloc_resize_umem() (CQ
    resize)
  - bnxt_re_resize_cq() (CQ resize)
  - ionic's ionic_create_cq_common()
  - qedr_create_cq(), via the shared qedr_init_user_queue() helper
  - mana_ib_create_cq(), via the shared mana_ib_create_queue() helper
  - erdma_init_user_cq(), via the shared get_mtt_entries() helper
  - hns_roce_cq.c's alloc_cq_buf(), via the shared
    hns_roce_mtr_create() helper

ib_umem_get_cq_buf_or_va() already falls back to a plain VA pin whenever
its attrs argument is NULL (ib_umem_resolve_desc() returns -ENOENT
immediately for attrs == NULL, before ever touching
attr_id/legacy_filler) -- the same attrs=NULL idiom ib_umem_get_va()
itself already uses via ib_umem_get_attr_or_va(device, NULL, 0, addr,
size, access). Switch these call sites to
ib_umem_get_cq_buf_or_va(device, NULL, addr, size, access) instead of
ib_umem_get_va(), so all CQ buffer pinning -- attribute-based or plain
VA, initial creation or resize -- goes through the same CQ-specific
helper.

qedr_init_user_queue(), mana_ib_create_queue(), get_mtt_entries()
(erdma), and hns_roce_mtr_create() are shared between CQ and other
callers (QP, SRQ, and for qedr/erdma/hns also MR), so each gains a new
is_cq parameter: true from the CQ call site, false from the others,
which must keep deriving direction from access flags rather than always
mapping DMA_FROM_DEVICE.

Also update bnxt_re_resize_cq()'s and qedr_init_user_queue()'s error
messages to name the new call instead of the old ib_umem_get_va().

vmw_pvrdma's pvrdma_create_cq() is deliberately NOT converted here.
Unlike every other driver checked, PVRDMA embeds a pvrdma_ring_state
producer/consumer header directly inside the same buffer as the CQEs
(cq->ring_state = cq->pdir.pages[0]), and the driver itself writes
cons_head back into it after polling (pvrdma_idx_ring_inc(&cq->ring_
state->rx.cons_head, ...)). That makes the buffer genuinely
bidirectional, not device-write/CPU-read-only like every other audited
driver's CQ. Forcing it onto ib_umem_get_cq_buf_or_va()'s hardcoded
DMA_FROM_DEVICE (added in the next commit) would break that CPU write.
PVRDMA's CQ stays on the generic ib_umem_get_va() path, where it already
passes IB_ACCESS_LOCAL_WRITE and so continues to correctly derive
DMA_BIDIRECTIONAL. See the dedicated vmw_pvrdma fix later in this series
for its QP and SRQ rings, which have the same embedded ring-state issue.

This is a pure refactor with no functional change: today
ib_umem_get_cq_buf_or_va() maps every buffer DMA_BIDIRECTIONAL, same as
ib_umem_get_va(). It prepares for the next commit, which changes
ib_umem_get_cq_buf_or_va() (and ib_umem_get_cq_buf()) to map CQ buffers
DMA_FROM_DEVICE -- once these call sites are already routed through it,
that change picks them all up automatically, with no follow-up gap to
close.

Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/hw/bnxt_re/ib_verbs.c      |  9 ++++----
 drivers/infiniband/hw/erdma/erdma_verbs.c     | 17 +++++++++-----
 drivers/infiniband/hw/hns/hns_roce_cq.c       |  2 +-
 drivers/infiniband/hw/hns/hns_roce_device.h   |  2 +-
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c    |  2 +-
 drivers/infiniband/hw/hns/hns_roce_mr.c       | 22 +++++++++++++-----
 drivers/infiniband/hw/hns/hns_roce_qp.c       |  2 +-
 drivers/infiniband/hw/hns/hns_roce_srq.c      |  4 ++--
 .../infiniband/hw/ionic/ionic_controlpath.c   |  5 ++--
 drivers/infiniband/hw/mana/cq.c               |  2 +-
 drivers/infiniband/hw/mana/main.c             |  9 ++++++--
 drivers/infiniband/hw/mana/mana_ib.h          |  2 +-
 drivers/infiniband/hw/mana/qp.c               |  9 ++++----
 drivers/infiniband/hw/mana/wq.c               |  3 ++-
 drivers/infiniband/hw/mlx4/cq.c               | 14 ++++++-----
 drivers/infiniband/hw/mlx5/cq.c               |  6 ++---
 drivers/infiniband/hw/qedr/verbs.c            | 23 +++++++++++++------
 17 files changed, 84 insertions(+), 49 deletions(-)

diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
index ccd2702db78b..e1197868c7a9 100644
--- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c
+++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
@@ -3749,12 +3749,13 @@ int bnxt_re_resize_cq(struct ib_cq *ibcq, unsigned int cqe,
 	if (rc)
 		goto fail;
 
-	cq->resize_umem = ib_umem_get_va(&rdev->ibdev, req.cq_va,
-					 entries * sizeof(struct cq_base),
-					 IB_ACCESS_LOCAL_WRITE);
+	cq->resize_umem = ib_umem_get_cq_buf_or_va(&rdev->ibdev, NULL,
+						   req.cq_va,
+						   entries * sizeof(struct cq_base),
+						   IB_ACCESS_LOCAL_WRITE);
 	if (IS_ERR(cq->resize_umem)) {
 		rc = PTR_ERR(cq->resize_umem);
-		ibdev_err(&rdev->ibdev, "%s: ib_umem_get_va failed! rc = %pe\n",
+		ibdev_err(&rdev->ibdev, "%s: ib_umem_get_cq_buf_or_va failed! rc = %pe\n",
 			  __func__, cq->resize_umem);
 		cq->resize_umem = NULL;
 		goto fail;
diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c
index fca2553e47ad..3519b5044e8f 100644
--- a/drivers/infiniband/hw/erdma/erdma_verbs.c
+++ b/drivers/infiniband/hw/erdma/erdma_verbs.c
@@ -828,11 +828,16 @@ static void erdma_destroy_mtt(struct erdma_dev *dev, struct erdma_mtt *mtt)
 
 static int get_mtt_entries(struct erdma_dev *dev, struct erdma_mem *mem,
 			   u64 start, u64 len, int access, u64 virt,
-			   unsigned long req_page_size, bool force_continuous)
+			   unsigned long req_page_size, bool force_continuous,
+			   bool is_cq)
 {
 	int ret = 0;
 
-	mem->umem = ib_umem_get_va(&dev->ibdev, start, len, access);
+	if (is_cq)
+		mem->umem = ib_umem_get_cq_buf_or_va(&dev->ibdev, NULL, start,
+						     len, access);
+	else
+		mem->umem = ib_umem_get_va(&dev->ibdev, start, len, access);
 	if (IS_ERR(mem->umem)) {
 		ret = PTR_ERR(mem->umem);
 		mem->umem = NULL;
@@ -951,7 +956,7 @@ static int init_user_qp(struct erdma_qp *qp, struct erdma_ucontext *uctx,
 
 	ret = get_mtt_entries(qp->dev, &qp->user_qp.sq_mem, va,
 			      qp->attrs.sq_size << SQEBB_SHIFT, 0, va,
-			      (SZ_1M - SZ_4K), true);
+			      (SZ_1M - SZ_4K), true, false);
 	if (ret)
 		return ret;
 
@@ -960,7 +965,7 @@ static int init_user_qp(struct erdma_qp *qp, struct erdma_ucontext *uctx,
 
 	ret = get_mtt_entries(qp->dev, &qp->user_qp.rq_mem, va + rq_offset,
 			      qp->attrs.rq_size << RQE_SHIFT, 0, va + rq_offset,
-			      (SZ_1M - SZ_4K), true);
+			      (SZ_1M - SZ_4K), true, false);
 	if (ret)
 		goto put_sq_mtt;
 
@@ -1250,7 +1255,7 @@ struct ib_mr *erdma_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 len,
 		return ERR_PTR(-ENOMEM);
 
 	ret = get_mtt_entries(dev, &mr->mem, start, len, access, virt,
-			      SZ_2G - SZ_4K, false);
+			      SZ_2G - SZ_4K, false, false);
 	if (ret)
 		goto err_out_free;
 
@@ -1931,7 +1936,7 @@ static int erdma_init_user_cq(struct erdma_ucontext *ctx, struct erdma_cq *cq,
 
 	ret = get_mtt_entries(dev, &cq->user_cq.qbuf_mem, ureq->qbuf_va,
 			      ureq->qbuf_len, IB_ACCESS_LOCAL_WRITE,
-			      ureq->qbuf_va, SZ_64M - SZ_4K, true);
+			      ureq->qbuf_va, SZ_64M - SZ_4K, true, true);
 	if (ret)
 		return ret;
 
diff --git a/drivers/infiniband/hw/hns/hns_roce_cq.c b/drivers/infiniband/hw/hns/hns_roce_cq.c
index ae314f136732..7dfaa01cda67 100644
--- a/drivers/infiniband/hw/hns/hns_roce_cq.c
+++ b/drivers/infiniband/hw/hns/hns_roce_cq.c
@@ -265,7 +265,7 @@ static int alloc_cq_buf(struct hns_roce_dev *hr_dev, struct hns_roce_cq *hr_cq,
 
 	ret = hns_roce_mtr_create(hr_dev, &hr_cq->mtr, &buf_attr,
 				  hr_dev->caps.cqe_ba_pg_sz + PAGE_SHIFT,
-				  udata, addr);
+				  udata, addr, true);
 	if (ret)
 		ibdev_err(ibdev, "failed to alloc CQ mtr, ret = %d.\n", ret);
 
diff --git a/drivers/infiniband/hw/hns/hns_roce_device.h b/drivers/infiniband/hw/hns/hns_roce_device.h
index f4f899e87ea6..3accd77341ae 100644
--- a/drivers/infiniband/hw/hns/hns_roce_device.h
+++ b/drivers/infiniband/hw/hns/hns_roce_device.h
@@ -1232,7 +1232,7 @@ int hns_roce_mtr_find(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr,
 int hns_roce_mtr_create(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr,
 			struct hns_roce_buf_attr *buf_attr,
 			unsigned int page_shift, struct ib_udata *udata,
-			unsigned long user_addr);
+			unsigned long user_addr, bool is_cq);
 void hns_roce_mtr_destroy(struct hns_roce_dev *hr_dev,
 			  struct hns_roce_mtr *mtr);
 int hns_roce_mtr_map(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr,
diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
index 368e1d74c283..b8177d054a31 100644
--- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
+++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
@@ -6919,7 +6919,7 @@ static int alloc_eq_buf(struct hns_roce_dev *hr_dev, struct hns_roce_eq *eq)
 
 	err = hns_roce_mtr_create(hr_dev, &eq->mtr, &buf_attr,
 				  hr_dev->caps.eqe_ba_pg_sz + PAGE_SHIFT, NULL,
-				  0);
+				  0, false);
 	if (err)
 		dev_err(hr_dev->dev, "failed to alloc EQE mtr, err %d\n", err);
 
diff --git a/drivers/infiniband/hw/hns/hns_roce_mr.c b/drivers/infiniband/hw/hns/hns_roce_mr.c
index 7d41ae897458..4799f667eeda 100644
--- a/drivers/infiniband/hw/hns/hns_roce_mr.c
+++ b/drivers/infiniband/hw/hns/hns_roce_mr.c
@@ -112,7 +112,7 @@ static int alloc_mr_pbl(struct hns_roce_dev *hr_dev, struct hns_roce_mr *mr,
 
 	err = hns_roce_mtr_create(hr_dev, &mr->pbl_mtr, &buf_attr,
 				  hr_dev->caps.pbl_ba_pg_sz + PAGE_SHIFT,
-				  udata, start);
+				  udata, start, false);
 	if (err) {
 		ibdev_err(ibdev, "failed to alloc pbl mtr, ret = %d.\n", err);
 		return err;
@@ -586,7 +586,8 @@ static void mtr_free_bufs(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr)
 
 static int mtr_alloc_bufs(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr,
 			  struct hns_roce_buf_attr *buf_attr,
-			  struct ib_udata *udata, unsigned long user_addr)
+			  struct ib_udata *udata, unsigned long user_addr,
+			  bool is_cq)
 {
 	struct ib_device *ibdev = &hr_dev->ib_dev;
 	size_t total_size;
@@ -595,8 +596,14 @@ static int mtr_alloc_bufs(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr,
 
 	if (udata) {
 		mtr->kmem = NULL;
-		mtr->umem = ib_umem_get_va(ibdev, user_addr, total_size,
-					   buf_attr->user_access);
+		if (is_cq)
+			mtr->umem = ib_umem_get_cq_buf_or_va(ibdev, NULL,
+							     user_addr,
+							     total_size,
+							     buf_attr->user_access);
+		else
+			mtr->umem = ib_umem_get_va(ibdev, user_addr, total_size,
+						   buf_attr->user_access);
 		if (IS_ERR(mtr->umem)) {
 			ibdev_err(ibdev, "failed to get umem, ret = %pe.\n",
 				  mtr->umem);
@@ -1035,11 +1042,13 @@ static void mtr_free_mtt(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr)
  * @ba_page_shift: page shift for multi-hop base address table
  * @udata: user space context, if it's NULL, means kernel space
  * @user_addr: userspace virtual address to start at
+ * @is_cq: true when @mtr backs a CQ buffer, which the device writes
+ *         completion entries into
  */
 int hns_roce_mtr_create(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr,
 			struct hns_roce_buf_attr *buf_attr,
 			unsigned int ba_page_shift, struct ib_udata *udata,
-			unsigned long user_addr)
+			unsigned long user_addr, bool is_cq)
 {
 	struct ib_device *ibdev = &hr_dev->ib_dev;
 	int ret;
@@ -1052,7 +1061,8 @@ int hns_roce_mtr_create(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr,
 		mtr->umem = NULL;
 		mtr->kmem = NULL;
 	} else {
-		ret = mtr_alloc_bufs(hr_dev, mtr, buf_attr, udata, user_addr);
+		ret = mtr_alloc_bufs(hr_dev, mtr, buf_attr, udata, user_addr,
+				     is_cq);
 		if (ret) {
 			ibdev_err(ibdev,
 				  "failed to alloc mtr bufs, ret = %d.\n", ret);
diff --git a/drivers/infiniband/hw/hns/hns_roce_qp.c b/drivers/infiniband/hw/hns/hns_roce_qp.c
index e333a8c4acb5..c6c0dfdbdbc3 100644
--- a/drivers/infiniband/hw/hns/hns_roce_qp.c
+++ b/drivers/infiniband/hw/hns/hns_roce_qp.c
@@ -808,7 +808,7 @@ static int alloc_qp_buf(struct hns_roce_dev *hr_dev, struct hns_roce_qp *hr_qp,
 	}
 	ret = hns_roce_mtr_create(hr_dev, &hr_qp->mtr, &buf_attr,
 				  PAGE_SHIFT + hr_dev->caps.mtt_ba_pg_sz,
-				  udata, addr);
+				  udata, addr, false);
 	if (ret) {
 		ibdev_err(ibdev, "failed to create WQE mtr, ret = %d.\n", ret);
 		goto err_inline;
diff --git a/drivers/infiniband/hw/hns/hns_roce_srq.c b/drivers/infiniband/hw/hns/hns_roce_srq.c
index 4a54394f96be..b5e9ac9cfd59 100644
--- a/drivers/infiniband/hw/hns/hns_roce_srq.c
+++ b/drivers/infiniband/hw/hns/hns_roce_srq.c
@@ -180,7 +180,7 @@ static int alloc_srq_idx(struct hns_roce_dev *hr_dev, struct hns_roce_srq *srq,
 
 	ret = hns_roce_mtr_create(hr_dev, &idx_que->mtr, &buf_attr,
 				  hr_dev->caps.idx_ba_pg_sz + PAGE_SHIFT,
-				  udata, addr);
+				  udata, addr, false);
 	if (ret) {
 		ibdev_err(ibdev,
 			  "failed to alloc SRQ idx mtr, ret = %d.\n", ret);
@@ -235,7 +235,7 @@ static int alloc_srq_wqe_buf(struct hns_roce_dev *hr_dev,
 
 	ret = hns_roce_mtr_create(hr_dev, &srq->buf_mtr, &buf_attr,
 				  hr_dev->caps.srqwqe_ba_pg_sz + PAGE_SHIFT,
-				  udata, addr);
+				  udata, addr, false);
 	if (ret)
 		ibdev_err(ibdev,
 			  "failed to alloc SRQ buf mtr, ret = %d.\n", ret);
diff --git a/drivers/infiniband/hw/ionic/ionic_controlpath.c b/drivers/infiniband/hw/ionic/ionic_controlpath.c
index 37f71fb43811..f392c635338b 100644
--- a/drivers/infiniband/hw/ionic/ionic_controlpath.c
+++ b/drivers/infiniband/hw/ionic/ionic_controlpath.c
@@ -110,8 +110,9 @@ int ionic_create_cq_common(struct ionic_vcq *vcq,
 		if (rc)
 			goto err_qdesc;
 
-		cq->umem = ib_umem_get_va(&dev->ibdev, req_cq->addr,
-					  req_cq->size, IB_ACCESS_LOCAL_WRITE);
+		cq->umem = ib_umem_get_cq_buf_or_va(&dev->ibdev, NULL,
+						    req_cq->addr, req_cq->size,
+						    IB_ACCESS_LOCAL_WRITE);
 		if (IS_ERR(cq->umem)) {
 			rc = PTR_ERR(cq->umem);
 			goto err_qdesc;
diff --git a/drivers/infiniband/hw/mana/cq.c b/drivers/infiniband/hw/mana/cq.c
index 6f9ac8b4aac8..de81a4b994e0 100644
--- a/drivers/infiniband/hw/mana/cq.c
+++ b/drivers/infiniband/hw/mana/cq.c
@@ -40,7 +40,7 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
 
 		cq->cqe = attr->cqe;
 		err = mana_ib_create_queue(mdev, ucmd.buf_addr, cq->cqe * COMP_ENTRY_SIZE,
-					   &cq->queue);
+					   &cq->queue, true);
 		if (err) {
 			ibdev_dbg(ibdev, "Failed to create queue for create cq, %d\n", err);
 			return err;
diff --git a/drivers/infiniband/hw/mana/main.c b/drivers/infiniband/hw/mana/main.c
index 83a97f1c5caa..46cb51ea08b9 100644
--- a/drivers/infiniband/hw/mana/main.c
+++ b/drivers/infiniband/hw/mana/main.c
@@ -293,7 +293,7 @@ int mana_ib_create_kernel_queue(struct mana_ib_dev *mdev, u32 size, enum gdma_qu
 }
 
 int mana_ib_create_queue(struct mana_ib_dev *mdev, u64 addr, u32 size,
-			 struct mana_ib_queue *queue)
+			 struct mana_ib_queue *queue, bool is_cq)
 {
 	struct ib_umem *umem;
 	int err;
@@ -302,7 +302,12 @@ int mana_ib_create_queue(struct mana_ib_dev *mdev, u64 addr, u32 size,
 	queue->id = INVALID_QUEUE_ID;
 	queue->gdma_region = GDMA_INVALID_DMA_REGION;
 
-	umem = ib_umem_get_va(&mdev->ib_dev, addr, size, IB_ACCESS_LOCAL_WRITE);
+	if (is_cq)
+		umem = ib_umem_get_cq_buf_or_va(&mdev->ib_dev, NULL, addr,
+						size, IB_ACCESS_LOCAL_WRITE);
+	else
+		umem = ib_umem_get_va(&mdev->ib_dev, addr, size,
+				      IB_ACCESS_LOCAL_WRITE);
 	if (IS_ERR(umem)) {
 		ibdev_dbg(&mdev->ib_dev, "Failed to get umem, %pe\n", umem);
 		return PTR_ERR(umem);
diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h
index 1be33ed8bd3b..fc71d0b7d57e 100644
--- a/drivers/infiniband/hw/mana/mana_ib.h
+++ b/drivers/infiniband/hw/mana/mana_ib.h
@@ -730,7 +730,7 @@ int mana_ib_gd_destroy_dma_region(struct mana_ib_dev *dev,
 int mana_ib_create_kernel_queue(struct mana_ib_dev *mdev, u32 size, enum gdma_queue_type type,
 				struct mana_ib_queue *queue);
 int mana_ib_create_queue(struct mana_ib_dev *mdev, u64 addr, u32 size,
-			 struct mana_ib_queue *queue);
+			 struct mana_ib_queue *queue, bool is_cq);
 void mana_ib_destroy_queue(struct mana_ib_dev *mdev, struct mana_ib_queue *queue);
 
 struct ib_wq *mana_ib_create_wq(struct ib_pd *pd,
diff --git a/drivers/infiniband/hw/mana/qp.c b/drivers/infiniband/hw/mana/qp.c
index fac43b3a5eb7..88e792cb0f85 100644
--- a/drivers/infiniband/hw/mana/qp.c
+++ b/drivers/infiniband/hw/mana/qp.c
@@ -340,7 +340,8 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
 	ibdev_dbg(&mdev->ib_dev, "ucmd sq_buf_addr 0x%llx port %u\n",
 		  ucmd.sq_buf_addr, ucmd.port);
 
-	err = mana_ib_create_queue(mdev, ucmd.sq_buf_addr, ucmd.sq_buf_size, &qp->raw_sq);
+	err = mana_ib_create_queue(mdev, ucmd.sq_buf_addr, ucmd.sq_buf_size, &qp->raw_sq,
+				   false);
 	if (err) {
 		ibdev_dbg(&mdev->ib_dev,
 			  "Failed to create queue for create qp-raw, err %d\n", err);
@@ -579,7 +580,7 @@ static int mana_ib_create_rc_qp(struct ib_qp *ibqp, struct ib_pd *ibpd,
 			if (ucmd.comp_mask & MANA_IB_RC_MMQ_CREATE) {
 				flags &= ~MANA_RC_FLAG_NO_MMQ;
 				err = mana_ib_create_queue(mdev, ucmd.mmq_buf, ucmd.mmq_size,
-							   &qp->rc_qp.queues[i]);
+							   &qp->rc_qp.queues[i], false);
 				if (err)
 					goto destroy_queues;
 			} else {
@@ -589,7 +590,7 @@ static int mana_ib_create_rc_qp(struct ib_qp *ibqp, struct ib_pd *ibpd,
 			continue;
 		}
 		err = mana_ib_create_queue(mdev, ucmd.queue_buf[j], ucmd.queue_size[j],
-					   &qp->rc_qp.queues[i]);
+					   &qp->rc_qp.queues[i], false);
 		if (err)
 			goto destroy_queues;
 		j++;
@@ -663,7 +664,7 @@ static int mana_ib_create_uc_qp(struct ib_qp *ibqp, struct ib_pd *ibpd,
 
 	for (i = 0; i < MANA_UC_QUEUE_TYPE_MAX; ++i) {
 		err = mana_ib_create_queue(mdev, ucmd.queue_buf[i], ucmd.queue_size[i],
-					   &qp->uc_qp.queues[i]);
+					   &qp->uc_qp.queues[i], false);
 		if (err)
 			goto destroy_queues;
 	}
diff --git a/drivers/infiniband/hw/mana/wq.c b/drivers/infiniband/hw/mana/wq.c
index 6b066d605dcb..4042f26632c0 100644
--- a/drivers/infiniband/hw/mana/wq.c
+++ b/drivers/infiniband/hw/mana/wq.c
@@ -25,7 +25,8 @@ struct ib_wq *mana_ib_create_wq(struct ib_pd *pd,
 
 	ibdev_dbg(&mdev->ib_dev, "ucmd wq_buf_addr 0x%llx\n", ucmd.wq_buf_addr);
 
-	err = mana_ib_create_queue(mdev, ucmd.wq_buf_addr, ucmd.wq_buf_size, &wq->queue);
+	err = mana_ib_create_queue(mdev, ucmd.wq_buf_addr, ucmd.wq_buf_size, &wq->queue,
+				   false);
 	if (err) {
 		ibdev_dbg(&mdev->ib_dev,
 			  "Failed to create queue for create wq, %d\n", err);
diff --git a/drivers/infiniband/hw/mlx4/cq.c b/drivers/infiniband/hw/mlx4/cq.c
index 887912469742..74d1b07f9cc1 100644
--- a/drivers/infiniband/hw/mlx4/cq.c
+++ b/drivers/infiniband/hw/mlx4/cq.c
@@ -185,9 +185,10 @@ int mlx4_ib_create_user_cq(struct ib_cq *ibcq,
 			goto err_umem;
 		}
 	} else {
-		cq->umem = ib_umem_get_va(&dev->ib_dev, ucmd.buf_addr,
-					  entries * cqe_size,
-					  IB_ACCESS_LOCAL_WRITE);
+		cq->umem = ib_umem_get_cq_buf_or_va(&dev->ib_dev, NULL,
+						    ucmd.buf_addr,
+						    entries * cqe_size,
+						    IB_ACCESS_LOCAL_WRITE);
 		if (IS_ERR(cq->umem)) {
 			err = PTR_ERR(cq->umem);
 			goto err_cq;
@@ -354,9 +355,10 @@ static int mlx4_alloc_resize_umem(struct mlx4_ib_dev *dev, struct mlx4_ib_cq *cq
 	if (!cq->resize_buf)
 		return -ENOMEM;
 
-	cq->resize_umem = ib_umem_get_va(&dev->ib_dev, ucmd.buf_addr,
-					 entries * cqe_size,
-					 IB_ACCESS_LOCAL_WRITE);
+	cq->resize_umem = ib_umem_get_cq_buf_or_va(&dev->ib_dev, NULL,
+						   ucmd.buf_addr,
+						   entries * cqe_size,
+						   IB_ACCESS_LOCAL_WRITE);
 	if (IS_ERR(cq->resize_umem)) {
 		err = PTR_ERR(cq->resize_umem);
 		goto err_buf;
diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
index 49b4bf148a4a..ec4833c59c65 100644
--- a/drivers/infiniband/hw/mlx5/cq.c
+++ b/drivers/infiniband/hw/mlx5/cq.c
@@ -1245,9 +1245,9 @@ static int resize_user(struct mlx5_ib_dev *dev, struct mlx5_ib_cq *cq,
 	if (ucmd.cqe_size && SIZE_MAX / ucmd.cqe_size <= entries - 1)
 		return -EINVAL;
 
-	umem = ib_umem_get_va(&dev->ib_dev, ucmd.buf_addr,
-			      (size_t)ucmd.cqe_size * entries,
-			      IB_ACCESS_LOCAL_WRITE);
+	umem = ib_umem_get_cq_buf_or_va(&dev->ib_dev, NULL, ucmd.buf_addr,
+					(size_t)ucmd.cqe_size * entries,
+					IB_ACCESS_LOCAL_WRITE);
 	if (IS_ERR(umem)) {
 		err = PTR_ERR(umem);
 		return err;
diff --git a/drivers/infiniband/hw/qedr/verbs.c b/drivers/infiniband/hw/qedr/verbs.c
index 012a0ab98d6b..4fecb90b1b8b 100644
--- a/drivers/infiniband/hw/qedr/verbs.c
+++ b/drivers/infiniband/hw/qedr/verbs.c
@@ -775,16 +775,23 @@ static inline int qedr_init_user_queue(struct ib_udata *udata,
 				       struct qedr_userq *q, u64 buf_addr,
 				       size_t buf_len, bool requires_db_rec,
 				       int access,
-				       int alloc_and_init)
+				       int alloc_and_init, bool is_cq)
 {
 	u32 fw_pages;
 	int rc;
 
 	q->buf_addr = buf_addr;
 	q->buf_len = buf_len;
-	q->umem = ib_umem_get_va(&dev->ibdev, q->buf_addr, q->buf_len, access);
+	if (is_cq)
+		q->umem = ib_umem_get_cq_buf_or_va(&dev->ibdev, NULL,
+						   q->buf_addr, q->buf_len,
+						   access);
+	else
+		q->umem = ib_umem_get_va(&dev->ibdev, q->buf_addr, q->buf_len,
+					 access);
 	if (IS_ERR(q->umem)) {
-		DP_ERR(dev, "create user queue: failed ib_umem_get_va, got %ld\n",
+		DP_ERR(dev, "create user queue: failed %s, got %ld\n",
+		       is_cq ? "ib_umem_get_cq_buf_or_va" : "ib_umem_get_va",
 		       PTR_ERR(q->umem));
 		return PTR_ERR(q->umem);
 	}
@@ -946,7 +953,7 @@ int qedr_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
 
 		rc = qedr_init_user_queue(udata, dev, &cq->q, ureq.addr,
 					  ureq.len, true, IB_ACCESS_LOCAL_WRITE,
-					  1);
+					  1, true);
 		if (rc)
 			goto err0;
 
@@ -1438,7 +1445,7 @@ static int qedr_init_srq_user_params(struct ib_udata *udata,
 	int rc;
 
 	rc = qedr_init_user_queue(udata, srq->dev, &srq->usrq, ureq->srq_addr,
-				  ureq->srq_len, false, access, 1);
+				  ureq->srq_len, false, access, 1, false);
 	if (rc)
 		return rc;
 
@@ -1831,7 +1838,8 @@ static int qedr_create_user_qp(struct qedr_dev *dev,
 	if (qedr_qp_has_sq(qp)) {
 		/* SQ - read access only (0) */
 		rc = qedr_init_user_queue(udata, dev, &qp->usq, ureq.sq_addr,
-					  ureq.sq_len, true, 0, alloc_and_init);
+					  ureq.sq_len, true, 0, alloc_and_init,
+					  false);
 		if (rc)
 			return rc;
 	}
@@ -1839,7 +1847,8 @@ static int qedr_create_user_qp(struct qedr_dev *dev,
 	if (qedr_qp_has_rq(qp)) {
 		/* RQ - read access only (0) */
 		rc = qedr_init_user_queue(udata, dev, &qp->urq, ureq.rq_addr,
-					  ureq.rq_len, true, 0, alloc_and_init);
+					  ureq.rq_len, true, 0, alloc_and_init,
+					  false);
 		if (rc) {
 			ib_umem_release(qp->usq.umem);
 			qp->usq.umem = NULL;
-- 
2.18.1

[PATCH rdma-next 05/15] RDMA/umem: Support an explicit DMA direction other than DMA_BIDIRECTIONAL

[Yishai Hadas]

User memory pinned via ib_umem_get_desc() (and the CQ buffer variant)
has always been mapped DMA_BIDIRECTIONAL, regardless of the IB access
flags or buffer type. This is unnecessarily broad for buffers where the
device only reads or only writes host memory, and is the first step
toward deriving the DMA direction from how each buffer type is actually
used.

Add a dma_dir field to struct ib_umem so the direction chosen at map
time is recorded and reused consistently at unmap time in
__ib_umem_release(). Thread an explicit dma_data_direction parameter
through the internal pinning helpers (__ib_umem_get_va(),
__ib_umem_get_desc_dir(), ib_umem_get_desc_check(),
ib_umem_get_from_attrs(), ib_umem_get_from_attrs_or_va()) so each caller
can supply the direction that matches how the device actually accesses
its buffer. Every existing caller still passes DMA_BIDIRECTIONAL
explicitly, so this patch is a pure mechanism addition with no
behavioural change.

Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/core/umem.c | 64 ++++++++++++++++++++++------------
 include/rdma/ib_umem.h         |  2 ++
 2 files changed, 43 insertions(+), 23 deletions(-)

diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c
index 88110b9661f5..c2f277ada042 100644
--- a/drivers/infiniband/core/umem.c
+++ b/drivers/infiniband/core/umem.c
@@ -55,7 +55,7 @@ static void __ib_umem_release(struct ib_device *dev, struct ib_umem *umem, int d
 
 	if (dirty)
 		ib_dma_unmap_sgtable_attrs(dev, &umem->sgt_append.sgt,
-					   DMA_BIDIRECTIONAL, umem->dma_attrs);
+					   umem->dma_dir, umem->dma_attrs);
 
 	for_each_sgtable_sg(&umem->sgt_append.sgt, sg, i) {
 		unpin_user_page_range_dirty_lock(sg_page(sg),
@@ -161,7 +161,8 @@ EXPORT_SYMBOL(ib_umem_find_best_pgsz);
 
 static struct ib_umem *__ib_umem_get_va(struct ib_device *device,
 					unsigned long addr, size_t size,
-					int access)
+					int access,
+					enum dma_data_direction dir)
 {
 	struct ib_umem *umem;
 	struct page **page_list;
@@ -202,6 +203,7 @@ static struct ib_umem *__ib_umem_get_va(struct ib_device *device,
 	 */
 	umem->iova = addr;
 	umem->writable   = ib_access_writable(access);
+	umem->dma_dir    = dir;
 	umem->owning_mm = mm = current->mm;
 	umem->dma_attrs = DMA_ATTR_REQUIRE_COHERENT;
 	if (access & IB_ACCESS_RELAXED_ORDERING)
@@ -261,7 +263,7 @@ static struct ib_umem *__ib_umem_get_va(struct ib_device *device,
 	}
 
 	ret = ib_dma_map_sgtable_attrs(device, &umem->sgt_append.sgt,
-				       DMA_BIDIRECTIONAL, umem->dma_attrs);
+				       dir, umem->dma_attrs);
 	if (ret)
 		goto umem_release;
 	goto out;
@@ -279,17 +281,16 @@ static struct ib_umem *__ib_umem_get_va(struct ib_device *device,
 	return ret ? ERR_PTR(ret) : umem;
 }
 
-/**
- * ib_umem_get_desc - Pin a umem from a buffer descriptor.
- * @device: IB device.
- * @desc:   buffer descriptor (VA or DMABUF).
- * @access: IB access flags.
+/*
+ * __ib_umem_get_desc_dir - core implementation for ib_umem_get_desc().
  *
- * Return: caller-owned umem on success, ERR_PTR(...) on error.
+ * @dir applies to VA buffers only; dmabuf direction is managed by the
+ * dmabuf subsystem and this argument is ignored for that type.
  */
-struct ib_umem *ib_umem_get_desc(struct ib_device *device,
-				 const struct ib_uverbs_buffer_desc *desc,
-				 int access)
+static struct ib_umem *
+__ib_umem_get_desc_dir(struct ib_device *device,
+		       const struct ib_uverbs_buffer_desc *desc,
+		       int access, enum dma_data_direction dir)
 {
 	struct ib_umem_dmabuf *umem_dmabuf;
 
@@ -310,11 +311,26 @@ struct ib_umem *ib_umem_get_desc(struct ib_device *device,
 		return &umem_dmabuf->umem;
 	case IB_UVERBS_BUFFER_TYPE_VA:
 		return __ib_umem_get_va(device, desc->addr, desc->length,
-					access);
+					access, dir);
 	default:
 		return ERR_PTR(-EINVAL);
 	}
 }
+
+/**
+ * ib_umem_get_desc - Pin a umem from a buffer descriptor.
+ * @device: IB device.
+ * @desc:   buffer descriptor (VA or DMABUF).
+ * @access: IB access flags.
+ *
+ * Return: caller-owned umem on success, ERR_PTR(...) on error.
+ */
+struct ib_umem *ib_umem_get_desc(struct ib_device *device,
+				 const struct ib_uverbs_buffer_desc *desc,
+				 int access)
+{
+	return __ib_umem_get_desc_dir(device, desc, access, DMA_BIDIRECTIONAL);
+}
 EXPORT_SYMBOL(ib_umem_get_desc);
 
 /*
@@ -376,11 +392,12 @@ static int ib_umem_resolve_desc(const struct uverbs_attr_bundle *attrs,
 static struct ib_umem *
 ib_umem_get_desc_check(struct ib_device *device,
 		       const struct ib_uverbs_buffer_desc *desc,
-		       size_t min_size, int access)
+		       size_t min_size, int access,
+		       enum dma_data_direction dir)
 {
 	struct ib_umem *umem;
 
-	umem = ib_umem_get_desc(device, desc, access);
+	umem = __ib_umem_get_desc_dir(device, desc, access, dir);
 	if (IS_ERR(umem))
 		return umem;
 	if (umem->length < min_size) {
@@ -401,7 +418,7 @@ static struct ib_umem *
 ib_umem_get_from_attrs(struct ib_device *device,
 		       const struct uverbs_attr_bundle *attrs,
 		       u16 attr_id, ib_umem_buf_desc_filler_t legacy_filler,
-		       size_t size, int access)
+		       size_t size, int access, enum dma_data_direction dir)
 {
 	struct ib_uverbs_buffer_desc desc = {};
 	int ret;
@@ -411,7 +428,7 @@ ib_umem_get_from_attrs(struct ib_device *device,
 		return NULL;
 	if (ret)
 		return ERR_PTR(ret);
-	return ib_umem_get_desc_check(device, &desc, size, access);
+	return ib_umem_get_desc_check(device, &desc, size, access, dir);
 }
 
 /*
@@ -431,7 +448,8 @@ ib_umem_get_from_attrs_or_va(struct ib_device *device,
 			     const struct uverbs_attr_bundle *attrs,
 			     u16 attr_id,
 			     ib_umem_buf_desc_filler_t legacy_filler,
-			     u64 addr, size_t size, int access)
+			     u64 addr, size_t size, int access,
+			     enum dma_data_direction dir)
 {
 	struct ib_uverbs_buffer_desc desc = {};
 	int ret;
@@ -445,7 +463,7 @@ ib_umem_get_from_attrs_or_va(struct ib_device *device,
 		};
 	else if (ret)
 		return ERR_PTR(ret);
-	return ib_umem_get_desc_check(device, &desc, size, access);
+	return ib_umem_get_desc_check(device, &desc, size, access, dir);
 }
 
 /**
@@ -464,7 +482,7 @@ struct ib_umem *ib_umem_get_attr(struct ib_device *device,
 				 u16 attr_id, size_t size, int access)
 {
 	return ib_umem_get_from_attrs(device, attrs, attr_id, NULL, size,
-				      access);
+				      access, DMA_BIDIRECTIONAL);
 }
 EXPORT_SYMBOL(ib_umem_get_attr);
 
@@ -503,7 +521,7 @@ struct ib_umem *ib_umem_get_attr_or_va(struct ib_device *device,
 				       int access)
 {
 	return ib_umem_get_from_attrs_or_va(device, attrs, attr_id, NULL, addr,
-					    size, access);
+					    size, access, DMA_BIDIRECTIONAL);
 }
 EXPORT_SYMBOL(ib_umem_get_attr_or_va);
 
@@ -579,7 +597,7 @@ struct ib_umem *ib_umem_get_cq_buf(struct ib_device *device,
 	return ib_umem_get_from_attrs(device, attrs,
 				      UVERBS_ATTR_CREATE_CQ_BUF_UMEM,
 				      uverbs_create_cq_get_buffer_desc,
-				      size, access);
+				      size, access, DMA_BIDIRECTIONAL);
 }
 EXPORT_SYMBOL(ib_umem_get_cq_buf);
 
@@ -608,7 +626,7 @@ struct ib_umem *ib_umem_get_cq_buf_or_va(struct ib_device *device,
 	return ib_umem_get_from_attrs_or_va(device, attrs,
 					    UVERBS_ATTR_CREATE_CQ_BUF_UMEM,
 					    uverbs_create_cq_get_buffer_desc,
-					    addr, size, access);
+					    addr, size, access, DMA_BIDIRECTIONAL);
 }
 EXPORT_SYMBOL(ib_umem_get_cq_buf_or_va);
 
diff --git a/include/rdma/ib_umem.h b/include/rdma/ib_umem.h
index 1fe87fd1d769..dcb645419692 100644
--- a/include/rdma/ib_umem.h
+++ b/include/rdma/ib_umem.h
@@ -7,6 +7,7 @@
 #ifndef IB_UMEM_H
 #define IB_UMEM_H
 
+#include <linux/dma-direction.h>
 #include <linux/scatterlist.h>
 
 struct ib_device;
@@ -19,6 +20,7 @@ struct ib_umem {
 	size_t			length;
 	unsigned long		address;
 	unsigned long		dma_attrs;
+	enum dma_data_direction dma_dir;
 	u32 writable : 1;
 	u32 is_odp : 1;
 	u32 is_dmabuf : 1;
-- 
2.18.1

[PATCH rdma-next 06/15] RDMA/umem: Map CQ buffers DMA_FROM_DEVICE

[Yishai Hadas]

ib_umem_get_cq_buf() / ib_umem_get_cq_buf_or_va() pass DMA_FROM_DEVICE
explicitly, overriding the default DMA_BIDIRECTIONAL mapping. CQ callers
pass IB_ACCESS_LOCAL_WRITE, which only means the umem must be pinned
writable for GUP purposes; it says nothing about which side of the PCIe
link actually touches the memory. For a CQ ring the device writes CQEs
and the CPU only reads, the opposite of what IB_ACCESS_LOCAL_WRITE
conventionally implies for MR/WQE buffers.  Deriving the direction from
access flags the way the following patch does for other buffer types
would give DMA_BIDIRECTIONAL here (ib_access_writable() treats
IB_ACCESS_LOCAL_WRITE as writable), identical to today's unconditional
behaviour, and CQ buffers would get none of this hardening -- hence the
explicit override instead.

For dmabuf buffers the direction is managed by the dmabuf subsystem and
ib_umem_get_desc() routes them unchanged, ignoring the derived
direction.

All drivers that call the CQ pinning functions benefit automatically:
mlx5, mlx4, efa, bnxt_re, ionic, qedr, mana, erdma, and hns -- including
CQ resize and legacy VA-only creation paths, since an earlier commit
already routed all of them through ib_umem_get_cq_buf_or_va() instead of
the generic ib_umem_get_va(). vmw_pvrdma's CQ was deliberately left on
the generic path by that same commit for its own embedded ring-state
reason, and continues to correctly derive DMA_BIDIRECTIONAL there.

Security gain by platform:
  - Platforms with a write-enforcing IOMMU (e.g. CoCo guests backed by
    ARM SMMU or Intel VT-d in strict mode): a hostile device is
    hardware-prevented from reading write-only buffers (CQ ring),
    limiting information leakage.
  - Standard deployments without IOMMU direction enforcement: no
    practical security effect today, but the correct semantic
    declaration and zero runtime cost.

Note: the CPU does not write to the CQ ring buffer; the CQ
consumer-index doorbell update goes through the UAR (MMIO), not through
this DMA-mapped buffer.

Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/core/umem.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c
index c2f277ada042..f1da70c4e352 100644
--- a/drivers/infiniband/core/umem.c
+++ b/drivers/infiniband/core/umem.c
@@ -587,6 +587,9 @@ static int uverbs_create_cq_get_buffer_desc(const struct uverbs_attr_bundle *att
  * must arrange its own backing (typically an in-kernel allocation)
  * when no source is available.
  *
+ * The buffer is mapped DMA_FROM_DEVICE: the NIC writes CQEs into it
+ * and the CPU only reads.
+ *
  * Return: caller-owned umem on success; NULL when no source supplied
  * a buffer; ERR_PTR(...) on error.
  */
@@ -597,7 +600,7 @@ struct ib_umem *ib_umem_get_cq_buf(struct ib_device *device,
 	return ib_umem_get_from_attrs(device, attrs,
 				      UVERBS_ATTR_CREATE_CQ_BUF_UMEM,
 				      uverbs_create_cq_get_buffer_desc,
-				      size, access, DMA_BIDIRECTIONAL);
+				      size, access, DMA_FROM_DEVICE);
 }
 EXPORT_SYMBOL(ib_umem_get_cq_buf);
 
@@ -613,6 +616,9 @@ EXPORT_SYMBOL(ib_umem_get_cq_buf);
  * Like ib_umem_get_cq_buf(), but pins @addr/@size when neither the
  * UMEM attribute nor the legacy CQ buffer attributes are supplied.
  *
+ * The buffer is mapped DMA_FROM_DEVICE: the NIC writes CQEs into it
+ * and the CPU only reads.
+ *
  * See ib_umem_get_attr_or_va() for the note on @size's dual role and
  * the migration path for drivers that would distinguish a user-supplied
  * length from a driver-computed minimum.
@@ -626,7 +632,7 @@ struct ib_umem *ib_umem_get_cq_buf_or_va(struct ib_device *device,
 	return ib_umem_get_from_attrs_or_va(device, attrs,
 					    UVERBS_ATTR_CREATE_CQ_BUF_UMEM,
 					    uverbs_create_cq_get_buffer_desc,
-					    addr, size, access, DMA_BIDIRECTIONAL);
+					    addr, size, access, DMA_FROM_DEVICE);
 }
 EXPORT_SYMBOL(ib_umem_get_cq_buf_or_va);
 
-- 
2.18.1

[PATCH rdma-next 07/15] RDMA/umem: Derive DMA direction from IB access flags

[Yishai Hadas]

ib_umem_get_desc() / ib_umem_get_attr() / ib_umem_get_attr_or_va() can
now derive the DMA direction from the IB access flags instead of always
mapping DMA_BIDIRECTIONAL:

  - No write access granted (as determined by ib_access_writable():
    IB_ACCESS_LOCAL_WRITE, IB_ACCESS_REMOTE_WRITE,
    IB_ACCESS_REMOTE_ATOMIC, or IB_ACCESS_MW_BIND): the NIC may only
    read the pages, so DMA_TO_DEVICE. This covers all callers that
    pass access=0 (SRQ WQE rings, QP SQ/RQ WQE rings, doorbell
    records) and MRs registered without write access
    (IB_ACCESS_REMOTE_READ only).

  - Any such flag set: DMA_BIDIRECTIONAL, unchanged behaviour.

This is not just a theoretical default: it has been verified, driver by
driver, against every QP SQ/RQ, SRQ, and doorbell-record buffer in the
tree pinned this way -- mlx5, mlx4, bnxt_re, efa, ionic, qedr, mana,
erdma, hns, and irdma all externalize their producer/consumer
bookkeeping through a separate doorbell/UAR MMIO write or a distinct
completion queue, never by the device writing back into the same buffer
the CPU posts WQEs into. vmw_pvrdma is the sole exception, already
pinning its QP/SRQ rings with IB_ACCESS_LOCAL_WRITE (see the earlier
fix), so it continues to correctly derive DMA_BIDIRECTIONAL here without
any special-casing.

For dmabuf buffers the direction is managed by the dmabuf subsystem and
ib_umem_get_desc() routes them unchanged, ignoring the derived
direction. CQ buffers already get their own tighter, unconditional
DMA_FROM_DEVICE mapping from the previous patch and are unaffected by
this rule.

Security gain by platform:
  - Platforms with a write-enforcing IOMMU (e.g. CoCo guests backed by
    ARM SMMU or Intel VT-d in strict mode): a hostile device is
    hardware-prevented from writing to read-only buffers (WQEs,
    doorbell records), limiting corruption.
  - Standard deployments without IOMMU direction enforcement: no
    practical security effect today, but the correct semantic
    declaration and zero runtime cost.

Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/core/umem.c | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c
index f1da70c4e352..490ce660193e 100644
--- a/drivers/infiniband/core/umem.c
+++ b/drivers/infiniband/core/umem.c
@@ -329,7 +329,20 @@ struct ib_umem *ib_umem_get_desc(struct ib_device *device,
 				 const struct ib_uverbs_buffer_desc *desc,
 				 int access)
 {
-	return __ib_umem_get_desc_dir(device, desc, access, DMA_BIDIRECTIONAL);
+	/*
+	 * Derive the DMA direction from the IB access flags.  If the caller
+	 * grants no write access (local or remote), the NIC may only read the
+	 * pages - use DMA_TO_DEVICE so that platforms with a write-enforcing
+	 * IOMMU (e.g. CoCo / SMMU with SMMU_CB_ARC_FAULT_ENABLE) can enforce
+	 * the restriction.
+	 *
+	 * Where the IOMMU does not enforce direction (most x86 bare-metal
+	 * deployments today) this has no security effect, but is still the
+	 * correct semantic declaration and costs nothing.
+	 */
+	return __ib_umem_get_desc_dir(device, desc, access,
+				      ib_access_writable(access) ?
+				      DMA_BIDIRECTIONAL : DMA_TO_DEVICE);
 }
 EXPORT_SYMBOL(ib_umem_get_desc);
 
@@ -482,7 +495,9 @@ struct ib_umem *ib_umem_get_attr(struct ib_device *device,
 				 u16 attr_id, size_t size, int access)
 {
 	return ib_umem_get_from_attrs(device, attrs, attr_id, NULL, size,
-				      access, DMA_BIDIRECTIONAL);
+				      access,
+				      ib_access_writable(access) ?
+				      DMA_BIDIRECTIONAL : DMA_TO_DEVICE);
 }
 EXPORT_SYMBOL(ib_umem_get_attr);
 
@@ -521,7 +536,9 @@ struct ib_umem *ib_umem_get_attr_or_va(struct ib_device *device,
 				       int access)
 {
 	return ib_umem_get_from_attrs_or_va(device, attrs, attr_id, NULL, addr,
-					    size, access, DMA_BIDIRECTIONAL);
+					    size, access,
+					    ib_access_writable(access) ?
+					    DMA_BIDIRECTIONAL : DMA_TO_DEVICE);
 }
 EXPORT_SYMBOL(ib_umem_get_attr_or_va);
 
-- 
2.18.1

[PATCH rdma-next 08/15] RDMA/mlx5: Fix mlx5_ib_dev_res_init() failure when XRC cap is absent

[Yishai Hadas]

mlx5_ib_dev_res_init() returned -EOPNOTSUPP when the firmware XRC
capability is zero, causing the driver probe to fail on any device where
XRC is not available.

XRC is an optional feature.  Make it optional throughout the devr
resource path:

- Initialize cq_lock and srq_lock unconditionally so that
  mlx5_ib_dev_res_srq_init() (called on every QP creation) can safely
  lock srq_lock regardless of whether XRC is present.

- Skip the mlx5_cmd_xrcd_alloc() calls when xrc=0 and guard the
  matching mlx5_cmd_xrcd_dealloc() calls in the cleanup path.

- In mlx5_ib_dev_res_srq_init(), skip creating the XRC-type placeholder
  SRQ (s0) when xrc=0.  All devr->s0 accesses in qp.c are inside XRC
  QP paths that the verbs layer rejects before reaching this driver when
  xrc=0, so devr->s0=NULL is safe.  Guard the s0 destruction in cleanup
  accordingly.

- The BASIC-type placeholder SRQ (s1) does not require XRC and is still
  created normally.  devr->s1 accesses in qp.c are not locally
  null-checked, but mlx5_ib_create_qp() always calls
  mlx5_ib_dev_res_srq_init() before reaching them, so devr->s1 is
  guaranteed non-NULL by the time they run.

Fixes: f4375443b786 ("RDMA/mlx5: Get XRCD number directly for the internal use")
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/hw/mlx5/main.c | 56 +++++++++++++++++++------------
 1 file changed, 34 insertions(+), 22 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 373ee1f42d4a..8b8a0f26cf19 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -3373,7 +3373,7 @@ int mlx5_ib_dev_res_srq_init(struct mlx5_ib_dev *dev)
 {
 	struct mlx5_ib_resources *devr = &dev->devr;
 	struct ib_srq_init_attr attr;
-	struct ib_srq *s0, *s1;
+	struct ib_srq *s0 = NULL, *s1;
 	int ret = 0;
 
 	/*
@@ -3391,19 +3391,27 @@ int mlx5_ib_dev_res_srq_init(struct mlx5_ib_dev *dev)
 	if (ret)
 		goto unlock;
 
-	memset(&attr, 0, sizeof(attr));
-	attr.attr.max_sge = 1;
-	attr.attr.max_wr = 1;
-	attr.srq_type = IB_SRQT_XRC;
-	attr.ext.cq = devr->c0;
-
-	s0 = ib_create_srq(devr->p0, &attr);
-	if (IS_ERR(s0)) {
-		ret = PTR_ERR(s0);
-		mlx5_ib_err(dev,
-			    "Couldn't create SRQ 0 for res init, err=%pe\n",
-			    s0);
-		goto unlock;
+	/*
+	 * s0 is an XRC-type placeholder SRQ used as the default XRQN for
+	 * XRC QPs.  Skip it when XRC is absent; all devr->s0 accesses in
+	 * qp.c are inside XRC QP paths that the verbs layer blocks before
+	 * reaching this driver when xrc=0, so NULL is safe.
+	 */
+	if (MLX5_CAP_GEN(dev->mdev, xrc)) {
+		memset(&attr, 0, sizeof(attr));
+		attr.attr.max_sge = 1;
+		attr.attr.max_wr = 1;
+		attr.srq_type = IB_SRQT_XRC;
+		attr.ext.cq = devr->c0;
+
+		s0 = ib_create_srq(devr->p0, &attr);
+		if (IS_ERR(s0)) {
+			ret = PTR_ERR(s0);
+			mlx5_ib_err(dev,
+				    "Couldn't create SRQ 0 for res init, err=%pe\n",
+				    s0);
+			goto unlock;
+		}
 	}
 
 	memset(&attr, 0, sizeof(attr));
@@ -3417,7 +3425,8 @@ int mlx5_ib_dev_res_srq_init(struct mlx5_ib_dev *dev)
 		mlx5_ib_err(dev,
 			    "Couldn't create SRQ 1 for res init, err=%pe\n",
 			    s1);
-		ib_destroy_srq(s0);
+		if (s0)
+			ib_destroy_srq(s0);
 		goto unlock;
 	}
 
@@ -3434,8 +3443,11 @@ static int mlx5_ib_dev_res_init(struct mlx5_ib_dev *dev)
 	struct mlx5_ib_resources *devr = &dev->devr;
 	int ret;
 
+	mutex_init(&devr->cq_lock);
+	mutex_init(&devr->srq_lock);
+
 	if (!MLX5_CAP_GEN(dev->mdev, xrc))
-		return -EOPNOTSUPP;
+		return 0;
 
 	ret = mlx5_cmd_xrcd_alloc(dev->mdev, &devr->xrcdn0, 0);
 	if (ret)
@@ -3447,9 +3459,6 @@ static int mlx5_ib_dev_res_init(struct mlx5_ib_dev *dev)
 		return ret;
 	}
 
-	mutex_init(&devr->cq_lock);
-	mutex_init(&devr->srq_lock);
-
 	return 0;
 }
 
@@ -3460,10 +3469,13 @@ static void mlx5_ib_dev_res_cleanup(struct mlx5_ib_dev *dev)
 	/* After s0/s1 init, they are not unset during the device lifetime. */
 	if (devr->s1) {
 		ib_destroy_srq(devr->s1);
-		ib_destroy_srq(devr->s0);
+		if (devr->s0)
+			ib_destroy_srq(devr->s0);
+	}
+	if (MLX5_CAP_GEN(dev->mdev, xrc)) {
+		mlx5_cmd_xrcd_dealloc(dev->mdev, devr->xrcdn1, 0);
+		mlx5_cmd_xrcd_dealloc(dev->mdev, devr->xrcdn0, 0);
 	}
-	mlx5_cmd_xrcd_dealloc(dev->mdev, devr->xrcdn1, 0);
-	mlx5_cmd_xrcd_dealloc(dev->mdev, devr->xrcdn0, 0);
 	/* After p0/c0 init, they are not unset during the device lifetime. */
 	if (devr->c0) {
 		ib_destroy_cq(devr->c0);
-- 
2.18.1

[PATCH rdma-next 09/15] RDMA/mlx5: Put resource reference in mlx5_ib_wq_event()

[Yishai Hadas]

rsc_event_notifier() takes a reference on the resource via
mlx5_get_rsc() before dispatching to the registered .event callback, and
documents that the callback owns releasing it:

  qp->event(qp, event_type);
  /* Need to put resource in event handler */
  return NOTIFY_OK;

mlx5_ib_qp_event() honors this, either releasing the reference
immediately when it has no work to do or handing it off to its deferred
workqueue callback. mlx5_ib_wq_event(), registered as the .event
callback for standalone RQ/WQ resources, never released it on any path.

Every MLX5_EVENT_TYPE_WQ_CATAS_ERROR (or any other event reaching this
callback) therefore leaked one reference on dev->qp_table's tracked
resource. Once leaked enough times, or even just once, the resource's
refcount can never reach zero, so destroy_resource_common()'s
wait_for_completion(&qp->common.free) hangs forever when the WQ is later
destroyed.

Add the missing mlx5_core_res_put() on every exit path.

Fixes: 350d0e4c7e4b ("IB/mlx5: Track asynchronous events on a receive work queue")
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/hw/mlx5/qp.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 0d4f8b109ad2..d4d439ebb595 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -5344,11 +5344,13 @@ static void mlx5_ib_wq_event(struct mlx5_core_qp *core_qp, int type)
 			break;
 		default:
 			mlx5_ib_warn(dev, "Unexpected event type %d on WQ %06x\n", type, core_qp->qpn);
-			return;
+			goto out;
 		}
 
 		rwq->ibwq.event_handler(&event, rwq->ibwq.wq_context);
 	}
+out:
+	mlx5_core_res_put(&core_qp->common);
 }
 
 static int set_delay_drop(struct mlx5_ib_dev *dev)
-- 
2.18.1

[PATCH rdma-next 10/15] RDMA/mlx5: Set WQ event handler before firmware RQ insertion

[Yishai Hadas]

mlx5_ib_create_wq() calls create_rq(), which inserts the RQ's
mlx5_core_qp into dev->qp_table.tree via mlx5_core_create_rq_tracked()
-> create_resource_common(), making it immediately visible to
rsc_event_notifier(). rwq->core_qp.event was assigned only after
create_rq() returned, and after several more steps (wq_num/state setup,
ib_respond_udata()), leaving a wide window where a firmware
WQ_CATAS_ERROR EQE would find rwq->core_qp.event still NULL and hit the
WARN_ON_ONCE(!qp->event) guard in rsc_event_notifier(), dropping the
event instead of delivering it.

Move the assignment right after the mlx5_ib_rwq allocation, before
create_rq() is called. mlx5_ib_wq_event() already null-checks
rwq->ibwq.event_handler before touching it, so setting core_qp.event
early is safe even though ibwq.event_handler is still assigned later.

Fixes: 350d0e4c7e4b ("IB/mlx5: Track asynchronous events on a receive work queue")
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/hw/mlx5/qp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index d4d439ebb595..d8e8a84ff3f1 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -5618,6 +5618,7 @@ struct ib_wq *mlx5_ib_create_wq(struct ib_pd *pd,
 		rwq = kzalloc_obj(*rwq);
 		if (!rwq)
 			return ERR_PTR(-ENOMEM);
+		rwq->core_qp.event = mlx5_ib_wq_event;
 		err = prepare_user_rq(pd, init_attr, udata, rwq);
 		if (err)
 			goto err;
@@ -5641,7 +5642,6 @@ struct ib_wq *mlx5_ib_create_wq(struct ib_pd *pd,
 			goto err_copy;
 	}
 
-	rwq->core_qp.event = mlx5_ib_wq_event;
 	rwq->ibwq.event_handler = init_attr->event_handler;
 	return &rwq->ibwq;
 
-- 
2.18.1

[PATCH rdma-next 11/15] RDMA/mlx5: Set SRQ event handler before xarray insertion

[Yishai Hadas]

mlx5_cmd_create_srq() stores the SRQ into the xarray via xa_store_irq(),
making it immediately visible to srq_event_notifier(). The caller was
assigning srq->msrq.event after that store, leaving a window where a
firmware SRQ event EQE could arrive and call through a NULL function
pointer (rdma_zalloc_drv_obj zero-fills the struct).

Move the assignment before mlx5_cmd_create_srq() so the event callback
is always valid by the time the SRQ is reachable by event handling.

Fixes: e126ba97dba9 ("mlx5: Add driver for Mellanox Connect-IB adapters")
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/hw/mlx5/srq.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/srq.c b/drivers/infiniband/hw/mlx5/srq.c
index a973c1b7515f..6a4a937d97ce 100644
--- a/drivers/infiniband/hw/mlx5/srq.c
+++ b/drivers/infiniband/hw/mlx5/srq.c
@@ -284,6 +284,7 @@ int mlx5_ib_create_srq(struct ib_srq *ib_srq,
 
 	in.pd = to_mpd(ib_srq->pd)->pdn;
 	in.db_record = srq->db.dma;
+	srq->msrq.event = mlx5_ib_srq_event;
 	err = mlx5_cmd_create_srq(dev, &srq->msrq, &in);
 	kvfree(in.pas);
 	if (err) {
@@ -292,8 +293,6 @@ int mlx5_ib_create_srq(struct ib_srq *ib_srq,
 	}
 
 	mlx5_ib_dbg(dev, "create SRQ with srqn 0x%x\n", srq->msrq.srqn);
-
-	srq->msrq.event = mlx5_ib_srq_event;
 	srq->ibsrq.ext.xrc.srq_num = srq->msrq.srqn;
 
 	if (udata) {
-- 
2.18.1

[PATCH rdma-next 12/15] RDMA/mlx5: Set RQ event handler for raw-packet QP

[Yishai Hadas]

create_raw_packet_qp() creates a separate RQ firmware object for
IB_QPT_RAW_PACKET QPs. mlx5_core_create_rq_tracked() inserts it into
dev->qp_table.tree via create_resource_common(), making it reachable by
rsc_event_notifier(), which dispatches firmware events by calling
qp->event().

rq->base.mqp.event was never assigned anywhere, for the RQ's entire
lifetime, not just around creation. Any firmware event on a raw-packet
QP's RQ resource (e.g. MLX5_EVENT_TYPE_WQ_CATAS_ERROR,
MLX5_EVENT_TYPE_WQ_ACCESS_ERROR) unconditionally calls through a NULL
function pointer, crashing the kernel. This is the same class of bug
originally reported and fixed for the sibling SQ object in commit
1d31e9c09f41 ("IB/mlx5: Fix Raw Packet QP event handler assignment"),
which never covered RQ.

Set rq->base.mqp.event before create_raw_packet_qp_rq() is called,
alongside the existing rq->base.container_mibqp assignment which was
already correctly ordered before the resource becomes visible to events.

Fixes: 0fb2ed66a14c ("IB/mlx5: Add create and destroy functionality for Raw Packet QP")
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/hw/mlx5/qp.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index d8e8a84ff3f1..7f18ff364e62 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -1647,6 +1647,7 @@ static int create_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 		}
 
 		rq->base.container_mibqp = qp;
+		rq->base.mqp.event = mlx5_ib_qp_event;
 
 		if (qp->flags & IB_QP_CREATE_CVLAN_STRIPPING)
 			rq->flags |= MLX5_IB_RQ_CVLAN_STRIPPING;
-- 
2.18.1

[PATCH rdma-next 13/15] RDMA/mlx5: Set QP event handler before firmware QPC insertion

[Yishai Hadas]

Four QP create paths (create_xrc_tgt_qp, create_dci, create_qp's default
path for create_user_qp/create_kernel_qp, and the raw-packet QP SQ
object in create_raw_packet_qp) assigned base->container_mibqp and
base->mqp.event only after mlx5_qpc_create_qp() /
create_raw_packet_qp_sq() returned. Those functions insert the QPC into
dev->qp_table.tree, making it immediately visible to
rsc_event_notifier(). A hostile NIC can fire a QP error EQE for the new
QPN/SQN in that window, reaching qp->event() with a NULL function
pointer and causing a kernel oops. This is the same class of bug already
fixed for the raw-packet QP's RQ object in a previous commit.

Move both assignments before the firmware create call in each path so
the event handler is always valid by the time the resource is reachable
by events. to_mibqp() indirects through container_mibqp, so it must also
be set before insertion.

Add a WARN_ON_ONCE(!qp->event) guard in rsc_event_notifier() as a
belt-and-suspenders defense against any future code paths that may
re-introduce the same ordering hazard.

Also set ibqp.qp_num in mlx5_qpc_create_qp() before
create_resource_common() inserts the QP into the radix tree, so an EQE
arriving in that window does not observe qp_num == 0. create_qp() still
overrides this with 0/1 for QP0/QP1 afterwards, and DCT sets its own
identifier independently since it does not go through this function.

Fixes: e126ba97dba9 ("mlx5: Add driver for Mellanox Connect-IB adapters")
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/hw/mlx5/qp.c  | 22 ++++++++++------------
 drivers/infiniband/hw/mlx5/qpc.c | 10 ++++++++++
 2 files changed, 20 insertions(+), 12 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 7f18ff364e62..2af599b81886 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -1624,6 +1624,8 @@ static int create_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 		if (err)
 			return err;
 
+		sq->base.container_mibqp = qp;
+		sq->base.mqp.event = mlx5_ib_qp_event;
 		err = create_raw_packet_qp_sq(dev, udata, attrs, sq, in, pd,
 					      to_mcq(init_attr->send_cq));
 		if (err)
@@ -1635,9 +1637,6 @@ static int create_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 			resp->sqn = sq->base.mqp.qpn;
 			resp->comp_mask |= MLX5_IB_CREATE_QP_RESP_MASK_SQN;
 		}
-
-		sq->base.container_mibqp = qp;
-		sq->base.mqp.event = mlx5_ib_qp_event;
 	}
 
 	if (qp->rq.wqe_cnt) {
@@ -2093,13 +2092,13 @@ static int create_xrc_tgt_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 	}
 
 	base = &qp->trans_qp.base;
+	base->container_mibqp = qp;
+	base->mqp.event = mlx5_ib_qp_event;
 	err = mlx5_qpc_create_qp(dev, &base->mqp, in, inlen, out);
 	kvfree(in);
 	if (err)
 		return err;
 
-	base->container_mibqp = qp;
-	base->mqp.event = mlx5_ib_qp_event;
 	if (MLX5_CAP_GEN(mdev, ece_support))
 		params->resp.ece_options = MLX5_GET(create_qp_out, out, ece);
 
@@ -2237,14 +2236,14 @@ static int create_dci(struct mlx5_ib_dev *dev, struct ib_pd *pd,
 		qp->flags &= ~IB_QP_CREATE_PCI_WRITE_END_PADDING;
 	}
 
+	base->container_mibqp = qp;
+	base->mqp.event = mlx5_ib_qp_event;
 	err = mlx5_qpc_create_qp(dev, &base->mqp, in, inlen, out);
 
 	kvfree(in);
 	if (err)
 		goto err_create;
 
-	base->container_mibqp = qp;
-	base->mqp.event = mlx5_ib_qp_event;
 	if (MLX5_CAP_GEN(mdev, ece_support))
 		params->resp.ece_options = MLX5_GET(create_qp_out, out, ece);
 
@@ -2433,6 +2432,8 @@ static int create_user_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd,
 		qp->flags &= ~IB_QP_CREATE_PCI_WRITE_END_PADDING;
 	}
 
+	base->container_mibqp = qp;
+	base->mqp.event = mlx5_ib_qp_event;
 	if (init_attr->qp_type == IB_QPT_RAW_PACKET ||
 	    qp->flags & IB_QP_CREATE_SOURCE_QPN) {
 		qp->raw_packet_qp.sq.ubuffer.buf_addr = ucmd->sq_buf_addr;
@@ -2447,8 +2448,6 @@ static int create_user_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd,
 	if (err)
 		goto err_create;
 
-	base->container_mibqp = qp;
-	base->mqp.event = mlx5_ib_qp_event;
 	if (MLX5_CAP_GEN(mdev, ece_support))
 		params->resp.ece_options = MLX5_GET(create_qp_out, out, ece);
 
@@ -2578,14 +2577,13 @@ static int create_kernel_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd,
 	    MLX5_CAP_GEN(mdev, go_back_n))
 		MLX5_SET(qpc, qpc, retry_mode, MLX5_QP_RM_GO_BACK_N);
 
+	base->container_mibqp = qp;
+	base->mqp.event = mlx5_ib_qp_event;
 	err = mlx5_qpc_create_qp(dev, &base->mqp, in, inlen, out);
 	kvfree(in);
 	if (err)
 		goto err_create;
 
-	base->container_mibqp = qp;
-	base->mqp.event = mlx5_ib_qp_event;
-
 	get_cqs(qp->type, attr->send_cq, attr->recv_cq,
 		&send_cq, &recv_cq);
 	spin_lock_irqsave(&dev->reset_flow_resource_lock, flags);
diff --git a/drivers/infiniband/hw/mlx5/qpc.c b/drivers/infiniband/hw/mlx5/qpc.c
index a7a4f9420271..77ddab666ecd 100644
--- a/drivers/infiniband/hw/mlx5/qpc.c
+++ b/drivers/infiniband/hw/mlx5/qpc.c
@@ -146,6 +146,8 @@ static int rsc_event_notifier(struct notifier_block *nb,
 	case MLX5_RES_RQ:
 	case MLX5_RES_SQ:
 		qp = (struct mlx5_core_qp *)common;
+		if (WARN_ON_ONCE(!qp->event))
+			goto out;
 		qp->event(qp, event_type);
 		/* Need to put resource in event handler */
 		return NOTIFY_OK;
@@ -258,6 +260,14 @@ int mlx5_qpc_create_qp(struct mlx5_ib_dev *dev, struct mlx5_core_qp *qp,
 
 	qp->uid = MLX5_GET(create_qp_in, in, uid);
 	qp->qpn = MLX5_GET(create_qp_out, out, qpn);
+	/* Set ibqp.qp_num before create_resource_common() inserts the QP into
+	 * the radix tree and makes it visible to EQE processing.  The
+	 * assignment in create_qp() happens only after this function returns,
+	 * leaving a window where an arriving EQE would observe qp_num == 0.
+	 * create_qp() still overrides this with 0/1 for QP0/QP1, and sets it
+	 * independently for DCT (which does not go through this function).
+	 */
+	to_mibqp(qp)->ibqp.qp_num = qp->qpn;
 
 	err = create_resource_common(dev, qp, MLX5_RES_QP);
 	if (err)
-- 
2.18.1

[PATCH rdma-next 14/15] RDMA/mlx5: Initialize QP/RQ/SQ resource refcount before publishing it

[Yishai Hadas]

create_resource_common() inserted the resource into dev->qp_table.tree
before initializing its refcount and completion:

  spin_lock_irq(&table->lock);
  err = radix_tree_insert(&table->tree, ..., qp);
  spin_unlock_irq(&table->lock);
  ...
  refcount_set(&qp->common.refcount, 1);
  init_completion(&qp->common.free);

Once inserted, the resource is reachable by mlx5_get_rsc(), called from
rsc_event_notifier() on any matching firmware event, which does:

  common = radix_tree_lookup(&table->tree, rsn);
  if (common && !common->invalid)
	  refcount_inc(&common->refcount);

qp->common.refcount is zero-initialized (the object is allocated with
kzalloc_obj()) until refcount_set() runs. An EQE arriving in the window
between insertion and refcount_set() calls refcount_inc() on a
still-zero refcount_t, which the refcount API treats as a use-after-free
class bug: it saturates the counter and emits a WARN, permanently
wedging that resource's refcounting for the rest of its lifetime.

Move refcount_set(), init_completion(), and the qp->pid assignment
before the radix_tree_insert() call, so the resource is always fully
initialized by the time it becomes visible to lookups.

Fixes: e126ba97dba9 ("mlx5: Add driver for Mellanox Connect-IB adapters")
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/hw/mlx5/qpc.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/qpc.c b/drivers/infiniband/hw/mlx5/qpc.c
index 77ddab666ecd..8b5bd33a7f0f 100644
--- a/drivers/infiniband/hw/mlx5/qpc.c
+++ b/drivers/infiniband/hw/mlx5/qpc.c
@@ -167,6 +167,10 @@ static int create_resource_common(struct mlx5_ib_dev *dev,
 	int err;
 
 	qp->common.res = rsc_type;
+	refcount_set(&qp->common.refcount, 1);
+	init_completion(&qp->common.free);
+	qp->pid = current->pid;
+
 	spin_lock_irq(&table->lock);
 	err = radix_tree_insert(&table->tree,
 				qp->qpn | (rsc_type << MLX5_USER_INDEX_LEN),
@@ -175,10 +179,6 @@ static int create_resource_common(struct mlx5_ib_dev *dev,
 	if (err)
 		return err;
 
-	refcount_set(&qp->common.refcount, 1);
-	init_completion(&qp->common.free);
-	qp->pid = current->pid;
-
 	return 0;
 }
 
-- 
2.18.1

[PATCH rdma-next 15/15] RDMA/mlx5: Fix signed integer overflow in EQE qp_srq type shift

[Yishai Hadas]

eqe->data.qp_srq.type is a u8 field (values 0-255). Shifting it left by
MLX5_USER_INDEX_LEN (24) promotes it to signed int before the shift,
causing undefined behavior when type >= 128: the result overflows into
the sign bit of int, which a signed-overflow sanitizer would flag.

Cast to u32 before the shift so the operation is well-defined across the
full 0-255 range of the EQE-supplied type field.

All currently defined and used type values (QP=0, RQ=1, SQ=2, SRQ=3,
XSRQ=4, XRQ=5, DCT=6) are well below 128, so this is not a bug that
manifests with current hardware. The cast is a preventive fix to make
the code well-defined against any future or firmware-forged type value.

Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
---
 drivers/infiniband/hw/mlx5/qpc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/infiniband/hw/mlx5/qpc.c b/drivers/infiniband/hw/mlx5/qpc.c
index 8b5bd33a7f0f..ce2ef8828c09 100644
--- a/drivers/infiniband/hw/mlx5/qpc.c
+++ b/drivers/infiniband/hw/mlx5/qpc.c
@@ -128,7 +128,7 @@ static int rsc_event_notifier(struct notifier_block *nb,
 	case MLX5_EVENT_TYPE_WQ_INVAL_REQ_ERROR:
 	case MLX5_EVENT_TYPE_WQ_ACCESS_ERROR:
 		rsn = be32_to_cpu(eqe->data.qp_srq.qp_srq_n) & 0xffffff;
-		rsn |= (eqe->data.qp_srq.type << MLX5_USER_INDEX_LEN);
+		rsn |= ((u32)eqe->data.qp_srq.type << MLX5_USER_INDEX_LEN);
 		break;
 	default:
 		return NOTIFY_DONE;
-- 
2.18.1