[PATCH rdma-next 00/15] RDMA: Support HW requiring relaxed ordering and add Unordered access flag

[Michael Gur <michaelgur@nvidia.com>]

This series prepares the mlx5 stack for hardware that restricts strong-
ordered memory accesses, i.e. HW whose fabric doesn't guarantee memory
operation ordering. The HW may choose to reject creation of an mkey that
doesn't request relaxed ordering, so every mkey the driver creates -
internal and user-facing - has to be handled accordingly.

The previously added 'ConnectX-10 NVLink-C2C' PCI device is such a device,
as will be any future PCIe Gen7 UIO devices. 

The series does four things:

1. Allow optional access flags on all memory registration
   The uverbs ABI defines the optional access flags range as best-effort:
   drivers must accept any flag in this range and treat support as
   opportunistic, never failing registration. Several registration paths
   (DM, DEVX UMEM, dmabuf ioctl, reg_mr ioctl) did not allow this range
   and would reject such requests. These are fixed to allow newly added
   optional access flag added later in the series and any future optional
   access flag.

2. Groundwork and internal mkeys
   Rename the relaxed-ordering IFC fields to the new naming, add the new
   ordering capability and mkey-context bits, and consolidate RO-setting
   to a single helper in mlx5_core so all consumers share one cap-gated
   implementation. Then enable relaxed ordering on the internal mkeys that
   don't require strong ordering (resource dump, vfio live-migration data,
   ODP null mkey, memory window mkey), so they keep working on strict-RO
   HW.

3. New Unordered access flag and enforcement
   Add IB_ACCESS_UNORDERED, an optional access flag that additionally
   allows read-after-write reordering, and wire up mlx5 support for it.

4. Enforce no-strong-ordering in user MR registration on HW that requires
   it.

Michael Gur (15):
  RDMA/mlx5: Allow optional access flags in DM registration
  RDMA/mlx5: Allow optional access flags in DEVX UMEM registration
  RDMA/core: Allow optional access flags in dmabuf reg ioctl
  RDMA/core: Allow optional access flags in reg mr ioctl
  mlx5: Rename IFC bits of relaxed ordering fields
  net/mlx5: Add IFC bits for new ordering caps
  mlx5: Move RO setting helper to core and consolidate mlx5 consumers
  net/mlx5: Enable relaxed ordering on resource dump mkey
  vfio/mlx5: Enable relaxed ordering on the live migration data mkey
  RDMA/mlx5: Enable relaxed ordering on ODP null mkey
  RDMA/mlx5: Enable relaxed ordering on Memory Window mkey on strict-RO
    HW
  RDMA/mlx5: Converge UMR access-flag cap checks
  RDMA/uverbs: Add new Unordered MR access flag
  RDMA/mlx5: Support new Unordered access flag
  RDMA/mlx5: Enforce relaxed ordering when HW requires it

 drivers/infiniband/core/umem.c                |  2 +-
 drivers/infiniband/core/uverbs_std_types_mr.c |  4 +-
 drivers/infiniband/hw/mlx5/devx.c             |  2 +-
 drivers/infiniband/hw/mlx5/main.c             |  8 +-
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |  4 +-
 drivers/infiniband/hw/mlx5/mr.c               | 79 ++++++++++---------
 drivers/infiniband/hw/mlx5/odp.c              |  1 +
 drivers/infiniband/hw/mlx5/umr.c              | 18 ++---
 drivers/infiniband/hw/mlx5/umr.h              | 45 ++++++-----
 .../mellanox/mlx5/core/diag/rsc_dump.c        |  1 +
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |  1 -
 .../ethernet/mellanox/mlx5/core/en/params.c   |  2 +-
 .../ethernet/mellanox/mlx5/core/en_common.c   | 13 +--
 .../net/ethernet/mellanox/mlx5/core/en_main.c |  4 +-
 drivers/net/ethernet/mellanox/mlx5/core/mr.c  | 21 +++++
 drivers/vfio/pci/mlx5/cmd.c                   |  7 +-
 include/linux/mlx5/driver.h                   |  1 +
 include/linux/mlx5/mlx5_ifc.h                 | 24 ++++--
 include/rdma/ib_verbs.h                       |  1 +
 include/uapi/rdma/ib_user_ioctl_verbs.h       |  1 +
 20 files changed, 134 insertions(+), 105 deletions(-)

-- 
2.52.0