[PATCH rdma-next v2 00/15] RDMA: Support HW requiring relaxed ordering and add Unordered access flag

[Michael Gur <michaelgur@nvidia.com>]

This series prepares the mlx5 stack for hardware that restricts strong-
ordered memory accesses, i.e. HW whose fabric doesn't guarantee memory
operation ordering. The HW may choose to reject creation of an mkey that
doesn't request relaxed ordering, so every mkey the driver creates -
internal and user-facing - has to be handled accordingly.

The previously added 'ConnectX-10 NVLink-C2C' PCI device is such a device,
as will be any future PCIe Gen7 UIO devices.

The series does four things:

1. Allow optional access flags on all memory registration
   The uverbs ABI defines the optional access flags range as best-effort:
   drivers must accept any flag in this range and treat support as
   opportunistic, never failing registration. Several registration paths
   (DM, DEVX UMEM, dmabuf ioctl, reg_mr ioctl) did not allow this range
   and would reject such requests. These are fixed to allow newly added
   optional access flag added later in the series and any future optional
   access flag.

2. Groundwork and internal mkeys
   Rename the relaxed-ordering IFC fields to the new naming, add the new
   ordering capability and mkey-context bits, and consolidate RO-setting
   to a single helper in mlx5_core so all consumers share one cap-gated
   implementation. Then enable relaxed ordering on the internal mkeys that
   don't require strong ordering (resource dump, vfio live-migration data,
   ODP null mkey and FW tracer), so they keep working on strict-RO HW.

3. New Unordered access flag and enforcement
   Add IB_ACCESS_UNORDERED, an optional access flag that additionally
   allows read-after-write reordering, and wire up mlx5 support for it.
   The rdma-core series adding the flag, with its full definition in the
   man pages: https://github.com/linux-rdma/rdma-core/pull/1780

4. Enforce no strong-ordering in user MR registration on HW that requires
   it.

Changes in v2:
- Dropped the patch that enabled relaxed ordering on Memory Window mkey on
  strict-RO HW. Following Jason's comment, RO is not enforced over indirect
  mkeys, as it takes no effect.
- Added enabling of RO on FW tracer mkey, setting RO only on HW that
  restricts strong ordering, to work around the mkey creation limitation
  (patch 9).
- Following Jason's comment, extended the new RO setter to support setting
  the new Unordered access as well (patch 7).
- Added RO enforcement on FRMR 'set pinned' commands (patch 15).
- Link to v1: https://lore.kernel.org/r/20260726092943.2880176-1-michaelgur@nvidia.com

Michael Gur (15):
  RDMA/mlx5: Allow optional access flags in DM registration
  RDMA/mlx5: Allow optional access flags in DEVX UMEM registration
  RDMA/core: Allow optional access flags in dmabuf reg ioctl
  RDMA/core: Allow optional access flags in reg mr ioctl
  mlx5: Rename IFC bits of relaxed ordering fields
  net/mlx5: Add IFC bits for new ordering caps
  mlx5: Consolidate mkey ordering setting into a single core helper
  net/mlx5: Enable relaxed ordering on resource dump mkey
  net/mlx5: Enable relaxed ordering on FW tracer mkey
  vfio/mlx5: Enable relaxed ordering on the live migration data mkey
  RDMA/mlx5: Enable relaxed ordering on ODP null mkey
  RDMA/mlx5: Converge UMR access-flag cap checks
  RDMA/uverbs: Add new Unordered MR access flag
  RDMA/mlx5: Support new Unordered access flag
  RDMA/mlx5: Enforce relaxed ordering when HW requires it

 drivers/infiniband/core/umem.c                |  2 +-
 drivers/infiniband/core/uverbs_std_types_mr.c |  4 +-
 drivers/infiniband/hw/mlx5/devx.c             |  2 +-
 drivers/infiniband/hw/mlx5/mlx5_ib.h          | 13 +++-
 drivers/infiniband/hw/mlx5/mr.c               | 76 +++++++++----------
 drivers/infiniband/hw/mlx5/odp.c              |  1 +
 drivers/infiniband/hw/mlx5/umr.c              | 18 ++---
 drivers/infiniband/hw/mlx5/umr.h              | 45 ++++++-----
 .../ethernet/mellanox/mlx5/core/data_direct.c |  8 +-
 .../mellanox/mlx5/core/diag/fw_tracer.c       |  2 +
 .../mellanox/mlx5/core/diag/rsc_dump.c        |  1 +
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |  1 -
 .../ethernet/mellanox/mlx5/core/en/params.c   |  2 +-
 .../ethernet/mellanox/mlx5/core/en_common.c   | 13 +---
 .../net/ethernet/mellanox/mlx5/core/en_main.c |  4 +-
 drivers/net/ethernet/mellanox/mlx5/core/mr.c  | 32 ++++++++
 drivers/vfio/pci/mlx5/cmd.c                   |  7 +-
 include/linux/mlx5/driver.h                   |  8 ++
 include/linux/mlx5/mlx5_ifc.h                 | 24 ++++--
 include/rdma/ib_verbs.h                       |  1 +
 include/uapi/rdma/ib_user_ioctl_verbs.h       |  1 +
 21 files changed, 160 insertions(+), 105 deletions(-)

-- 
2.52.0