Re: [PATCH rdma-next 00/15] RDMA: Support HW requiring relaxed ordering and add Unordered access flag

[Michael Gur <michaelgur@nvidia.com>]

On 8/10/2026 4:05 PM, Leon Romanovsky wrote:
> External email: Use caution opening links or attachments
>
>
> On Sun, Jul 26, 2026 at 12:29:14PM +0300, Michael Gur wrote:
>> This series prepares the mlx5 stack for hardware that restricts strong-
>> ordered memory accesses, i.e. HW whose fabric doesn't guarantee memory
>> operation ordering. The HW may choose to reject creation of an mkey that
>> doesn't request relaxed ordering, so every mkey the driver creates -
>> internal and user-facing - has to be handled accordingly.
>>
>> The previously added 'ConnectX-10 NVLink-C2C' PCI device is such a device,
>> as will be any future PCIe Gen7 UIO devices.
>>
>> The series does four things:
>>
>> 1. Allow optional access flags on all memory registration
>>     The uverbs ABI defines the optional access flags range as best-effort:
>>     drivers must accept any flag in this range and treat support as
>>     opportunistic, never failing registration. Several registration paths
>>     (DM, DEVX UMEM, dmabuf ioctl, reg_mr ioctl) did not allow this range
>>     and would reject such requests. These are fixed to allow newly added
>>     optional access flag added later in the series and any future optional
>>     access flag.
>>
>> 2. Groundwork and internal mkeys
>>     Rename the relaxed-ordering IFC fields to the new naming, add the new
>>     ordering capability and mkey-context bits, and consolidate RO-setting
>>     to a single helper in mlx5_core so all consumers share one cap-gated
>>     implementation. Then enable relaxed ordering on the internal mkeys that
>>     don't require strong ordering (resource dump, vfio live-migration data,
>>     ODP null mkey, memory window mkey), so they keep working on strict-RO
>>     HW.
>>
>> 3. New Unordered access flag and enforcement
>>     Add IB_ACCESS_UNORDERED, an optional access flag that additionally
>>     allows read-after-write reordering, and wire up mlx5 support for it.
>>
>> 4. Enforce no-strong-ordering in user MR registration on HW that requires
>>     it.
>>
>> Michael Gur (15):
>>    RDMA/mlx5: Allow optional access flags in DM registration
>>    RDMA/mlx5: Allow optional access flags in DEVX UMEM registration
>>    RDMA/core: Allow optional access flags in dmabuf reg ioctl
>>    RDMA/core: Allow optional access flags in reg mr ioctl
>>    mlx5: Rename IFC bits of relaxed ordering fields
>>    net/mlx5: Add IFC bits for new ordering caps
>>    mlx5: Move RO setting helper to core and consolidate mlx5 consumers
>>    net/mlx5: Enable relaxed ordering on resource dump mkey
>>    vfio/mlx5: Enable relaxed ordering on the live migration data mkey
>>    RDMA/mlx5: Enable relaxed ordering on ODP null mkey
>>    RDMA/mlx5: Enable relaxed ordering on Memory Window mkey on strict-RO
>>      HW
>>    RDMA/mlx5: Converge UMR access-flag cap checks
>>    RDMA/uverbs: Add new Unordered MR access flag
>>    RDMA/mlx5: Support new Unordered access flag
>>    RDMA/mlx5: Enforce relaxed ordering when HW requires it
>>
>>   drivers/infiniband/core/umem.c                |  2 +-
>>   drivers/infiniband/core/uverbs_std_types_mr.c |  4 +-
>>   drivers/infiniband/hw/mlx5/devx.c             |  2 +-
>>   drivers/infiniband/hw/mlx5/main.c             |  8 +-
>>   drivers/infiniband/hw/mlx5/mlx5_ib.h          |  4 +-
>>   drivers/infiniband/hw/mlx5/mr.c               | 79 ++++++++++---------
>>   drivers/infiniband/hw/mlx5/odp.c              |  1 +
>>   drivers/infiniband/hw/mlx5/umr.c              | 18 ++---
>>   drivers/infiniband/hw/mlx5/umr.h              | 45 ++++++-----
>>   .../mellanox/mlx5/core/diag/rsc_dump.c        |  1 +
>>   drivers/net/ethernet/mellanox/mlx5/core/en.h  |  1 -
>>   .../ethernet/mellanox/mlx5/core/en/params.c   |  2 +-
>>   .../ethernet/mellanox/mlx5/core/en_common.c   | 13 +--
>>   .../net/ethernet/mellanox/mlx5/core/en_main.c |  4 +-
>>   drivers/net/ethernet/mellanox/mlx5/core/mr.c  | 21 +++++
>>   drivers/vfio/pci/mlx5/cmd.c                   |  7 +-
>>   include/linux/mlx5/driver.h                   |  1 +
>>   include/linux/mlx5/mlx5_ifc.h                 | 24 ++++--
>>   include/rdma/ib_verbs.h                       |  1 +
>>   include/uapi/rdma/ib_user_ioctl_verbs.h       |  1 +
>>   20 files changed, 134 insertions(+), 105 deletions(-)
> Sashiko is not happy about this series.
>
> Thanks
Sashiko has 4 comments on this series:

2 comments (on patch 14 and 15) concerning data-direct flows not 
handling the Unordered access flag or RO enforcement:
These are irrelevant as we don't have HW that will support both caps. 
That is currently an invalid configuration.

Comment on patch 3: suggesting we need a fix that is actually added in 
patch 4.

Comment on patch 15: Sashiko misreading the meaning of the cap.
'mkc_order_write_after_write_ro_only' means we must set RO in the 
write_after_write field of every mkey, regardless of actual write access 
flags at registration.

Thanks

>> --
>> 2.52.0
>>