[PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

Leon Romanovsky
2021-04-05T05:24:14Z

From: Leon Romanovsky <leonro@nvidia.com>

From Avihai,

Relaxed Ordering is a PCIe mechanism that relaxes the strict ordering
imposed on PCI transactions, and thus, can improve performance.

Until now, relaxed ordering could be set only by user space applications
for user MRs. The following patch series enables relaxed ordering for the
kernel ULPs as well. Relaxed ordering is an optional capability, and as
such, it is ignored by vendors that don't support it.

The following test results show the performance improvement achieved
with relaxed ordering. The test was performed on a NVIDIA A100 in order
to check performance of storage infrastructure over xprtrdma:

Without Relaxed Ordering:
READ: bw=16.5GiB/s (17.7GB/s), 16.5GiB/s-16.5GiB/s (17.7GB/s-17.7GB/s),
io=1987GiB (2133GB), run=120422-120422msec

With relaxed ordering:
READ: bw=72.9GiB/s (78.2GB/s), 72.9GiB/s-72.9GiB/s (78.2GB/s-78.2GB/s),
io=2367GiB (2542GB), run=32492-32492msec

Thanks

Avihai Horon (10):
RDMA: Add access flags to ib_alloc_mr() and ib_mr_pool_init()
RDMA/core: Enable Relaxed Ordering in __ib_alloc_pd()
RDMA/iser: Enable Relaxed Ordering
RDMA/rtrs: Enable Relaxed Ordering
RDMA/srp: Enable Relaxed Ordering
nvme-rdma: Enable Relaxed Ordering
cifs: smbd: Enable Relaxed Ordering
net/rds: Enable Relaxed Ordering
net/smc: Enable Relaxed Ordering
xprtrdma: Enable Relaxed Ordering

drivers/infiniband/core/mr_pool.c | 7 +-
drivers/infiniband/core/rw.c | 12 ++--
drivers/infiniband/core/verbs.c | 26 +++++--
drivers/infiniband/hw/bnxt_re/ib_verbs.c | 2 +-
drivers/infiniband/hw/bnxt_re/ib_verbs.h | 2 +-
drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 2 +-
drivers/infiniband/hw/cxgb4/mem.c | 2 +-
drivers/infiniband/hw/hns/hns_roce_device.h | 2 +-
drivers/infiniband/hw/hns/hns_roce_mr.c | 2 +-
drivers/infiniband/hw/i40iw/i40iw_verbs.c | 3 +-
drivers/infiniband/hw/mlx4/mlx4_ib.h | 2 +-
drivers/infiniband/hw/mlx4/mr.c | 2 +-
drivers/infiniband/hw/mlx5/mlx5_ib.h | 12 ++--
drivers/infiniband/hw/mlx5/mr.c | 61 ++++++++--------
drivers/infiniband/hw/mlx5/wr.c | 69 ++++++++++++++-----
drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 2 +-
drivers/infiniband/hw/ocrdma/ocrdma_verbs.h | 2 +-
drivers/infiniband/hw/qedr/verbs.c | 2 +-
drivers/infiniband/hw/qedr/verbs.h | 2 +-
drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c | 4 +-
.../infiniband/hw/vmw_pvrdma/pvrdma_verbs.h | 2 +-
drivers/infiniband/sw/rdmavt/mr.c | 3 +-
drivers/infiniband/sw/rdmavt/mr.h | 2 +-
drivers/infiniband/sw/rxe/rxe_verbs.c | 2 +-
drivers/infiniband/sw/siw/siw_verbs.c | 2 +-
drivers/infiniband/sw/siw/siw_verbs.h | 2 +-
drivers/infiniband/ulp/iser/iser_memory.c | 10 ++-
drivers/infiniband/ulp/iser/iser_verbs.c | 6 +-
drivers/infiniband/ulp/rtrs/rtrs-clt.c | 6 +-
drivers/infiniband/ulp/rtrs/rtrs-srv.c | 15 ++--
drivers/infiniband/ulp/srp/ib_srp.c | 8 +--
drivers/nvme/host/rdma.c | 19 +++--
fs/cifs/smbdirect.c | 17 +++--
include/rdma/ib_verbs.h | 11 ++-
include/rdma/mr_pool.h | 3 +-
net/rds/ib_frmr.c | 7 +-
net/smc/smc_ib.c | 3 +-
net/smc/smc_wr.c | 3 +-
net/sunrpc/xprtrdma/frwr_ops.c | 10 +--
39 files changed, 209 insertions(+), 140 deletions(-)

--
2.30.2