[PATCH for-next v9 0/5] On-Demand Paging on SoftRoCE

[PATCH for-next v9 0/5] On-Demand Paging on SoftRoCE

[Daisuke Matsuda]

This patch series implements the On-Demand Paging feature on SoftRoCE(rxe)
driver, which has been available only in mlx5 driver[1] so far.

This series has been blocked because of the hang issue of srp 002 test[2],
which was believed to be caused after applying the commit 9b4b7c1f9f54
("RDMA/rxe: Add workqueue support for rxe tasks"). My patches are dependent
on the commit because the ODP feature requires sleeping in kernel space,
and it is impossible with the former tasklet implementation.

According to the original reporter[3], the hang issue is already gone in
v6.10. Additionally, tasklet is marked deprecated[4]. I think the rxe
driver is ready to accept this series since there is no longer any reason
to consider reverting back to the old tasklet.

I omitted some contents like the motive behind this series from the cover-
letter. Please see the cover letter of v3 for more details[5].

[Overview]
When applications register a memory region(MR), RDMA drivers normally pin
pages in the MR so that physical addresses are never changed during RDMA
communication. This requires the MR to fit in physical memory and
inevitably leads to memory pressure. On the other hand, On-Demand Paging
(ODP) allows applications to register MRs without pinning pages. They are
paged-in when the driver requires and paged-out when the OS reclaims. As a
result, it is possible to register a large MR that does not fit in physical
memory without taking up so much physical memory.

[How does ODP work?]
"struct ib_umem_odp" is used to manage pages. It is created for each
ODP-enabled MR on its registration. This struct holds a pair of arrays
(dma_list/pfn_list) that serve as a driver page table. DMA addresses and
PFNs are stored in the driver page table. They are updated on page-in and
page-out, both of which use the common interfaces in the ib_uverbs layer.

Page-in can occur when requester, responder or completer access an MR in
order to process RDMA operations. If they find that the pages being
accessed are not present on physical memory or requisite permissions are
not set on the pages, they provoke page fault to make the pages present
with proper permissions and at the same time update the driver page table.
After confirming the presence of the pages, they execute memory access such
as read, write or atomic operations.

Page-out is triggered by page reclaim or filesystem events (e.g. metadata
update of a file that is being used as an MR). When creating an ODP-enabled
MR, the driver registers an MMU notifier callback. When the kernel issues a
page invalidation notification, the callback is provoked to unmap DMA
addresses and update the driver page table. After that, the kernel releases
the pages.

[Supported operations]
All traditional operations are supported on RC connection. The new Atomic
write[6] and RDMA Flush[7] operations are not included in this patchset. I
will post them later after this patchset is merged. On UD connection, Send,
Recv, and SRQ-Recv are supported.

[How to test ODP?]
There are only a few resources available for testing. pyverbs testcases in
rdma-core and perftest[8] are recommendable ones. Other than them, the
ibv_rc_pingpong command can also be used for testing. Note that you may
have to build perftest from upstream because old versions do not handle ODP
capabilities correctly.

The latest ODP tree is available from github:
https://github.com/ddmatsu/linux/tree/odp_v9

[Future work]
My next work is to enable the new Atomic write[6] and RDMA Flush[7]
operations with ODP. After that, I am going to implement the prefetch
feature. It allows applications to trigger page fault using
ibv_advise_mr(3) to optimize performance. Some existing software like
librpma[9] use this feature. Additionally, I think we can also add the
implicit ODP feature in the future.

[1] Understanding On Demand Paging (ODP)
https://enterprise-support.nvidia.com/s/article/understanding-on-demand-paging--odp-x

[2] [bug report] blktests srp/002 hang
https://lore.kernel.org/linux-rdma/dsg6rd66tyiei32zaxs6ddv5ebefr5vtxjwz6d2ewqrcwisogl@ge7jzan7dg5u/T/

[3] blktests failures with v6.10-rc1 kernel
https://lore.kernel.org/linux-block/wnucs5oboi4flje5yvtea7puvn6zzztcnlrfz3lpzlwgblrxgw@7wvqdzioejgl/

[4] [00/15] ethernet: Convert from tasklet to BH workqueue
https://patchwork.kernel.org/project/linux-rdma/cover/20240621050525.3720069-1-allen.lkml@gmail.com/

[5] [PATCH for-next v3 0/7] On-Demand Paging on SoftRoCE
https://lore.kernel.org/lkml/cover.1671772917.git.matsuda-daisuke@fujitsu.com/

[6] [PATCH v7 0/8] RDMA/rxe: Add atomic write operation
https://lore.kernel.org/linux-rdma/1669905432-14-1-git-send-email-yangx.jy@fujitsu.com/

[7] [for-next PATCH 00/10] RDMA/rxe: Add RDMA FLUSH operation
https://lore.kernel.org/lkml/20221206130201.30986-1-lizhijian@fujitsu.com/

[8] linux-rdma/perftest: Infiniband Verbs Performance Tests
https://github.com/linux-rdma/perftest

[9] librpma: Remote Persistent Memory Access Library
https://github.com/pmem/rpma

v8->v9:
 1) Addressed comments from Jason Gunthorpe
 cf. https://lore.kernel.org/lkml/20241209192152.GC2368570@nvidia.com/
 cf. https://lore.kernel.org/lkml/20241209193102.GD2368570@nvidia.com/
 2) rebased to 6.13.0-rc1

v7->v8:
 1) Dropped the first patch because the same change was made by Bob Pearson.
 cf. https://github.com/torvalds/linux/commit/23bc06af547f2ca3b7d345e09fd8d04575406274
 2) Rebased to 6.12.1-rc2

v6->v7:
 1) Rebased to 6.6.0
 2) Disabled using hugepages with ODP
 3) Addressed comments on v6 from Jason and Zhu
   cf. https://lore.kernel.org/lkml/cover.1694153251.git.matsuda-daisuke@fujitsu.com/

v5->v6:
 Fixed the implementation according to Jason's suggestions
   cf. https://lore.kernel.org/all/ZIdFXfDu4IMKE+BQ@nvidia.com/
   cf. https://lore.kernel.org/all/ZIdGU709e1h5h4JJ@nvidia.com/

v4->v5:
 1) Rebased to 6.4.0-rc2+
 2) Changed to schedule all works on responder and completer to workqueue

v3->v4:
 1) Re-designed functions that access MRs to use the MR xarray.
 2) Rebased onto the latest jgg-for-next tree.

v2->v3:
 1) Removed a patch that changes the common ib_uverbs layer.
 2) Re-implemented patches for conversion to workqueue.
 3) Fixed compile errors (happened when CONFIG_INFINIBAND_ON_DEMAND_PAGING=n).
 4) Fixed some functions that returned incorrect errors.
 5) Temporarily disabled ODP for RDMA Flush and Atomic Write.

v1->v2:
 1) Fixed a crash issue reported by Haris Iqbal.
 2) Tried to make lock patters clearer as pointed out by Romanovsky.
 3) Minor clean ups and fixes.

Daisuke Matsuda (5):
  RDMA/rxe: Move some code to rxe_loc.h in preparation for ODP
  RDMA/rxe: Add page invalidation support
  RDMA/rxe: Allow registering MRs for On-Demand Paging
  RDMA/rxe: Add support for Send/Recv/Write/Read with ODP
  RDMA/rxe: Add support for the traditional Atomic operations with ODP

 drivers/infiniband/sw/rxe/Makefile    |   2 +
 drivers/infiniband/sw/rxe/rxe.c       |  18 ++
 drivers/infiniband/sw/rxe/rxe.h       |  37 ---
 drivers/infiniband/sw/rxe/rxe_loc.h   |  34 +++
 drivers/infiniband/sw/rxe/rxe_mr.c    |  13 +-
 drivers/infiniband/sw/rxe/rxe_odp.c   | 324 ++++++++++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_resp.c  |  18 +-
 drivers/infiniband/sw/rxe/rxe_verbs.c |   5 +-
 drivers/infiniband/sw/rxe/rxe_verbs.h |  37 +++
 9 files changed, 443 insertions(+), 45 deletions(-)
 create mode 100644 drivers/infiniband/sw/rxe/rxe_odp.c

-- 
2.43.0

[PATCH for-next v9 1/5] RDMA/rxe: Move some code to rxe_loc.h in preparation for ODP

[Daisuke Matsuda]

rxe_mr_init() and resp_states are going to be used in rxe_odp.c, which is
to be created in the subsequent patch.

Signed-off-by: Daisuke Matsuda <matsuda-daisuke@fujitsu.com>
---
 drivers/infiniband/sw/rxe/rxe.h       | 37 ---------------------------
 drivers/infiniband/sw/rxe/rxe_loc.h   |  1 +
 drivers/infiniband/sw/rxe/rxe_mr.c    |  2 +-
 drivers/infiniband/sw/rxe/rxe_verbs.h | 37 +++++++++++++++++++++++++++
 4 files changed, 39 insertions(+), 38 deletions(-)

diff --git a/drivers/infiniband/sw/rxe/rxe.h b/drivers/infiniband/sw/rxe/rxe.h
index d8fb2c7af30a..193f7caffaf2 100644
--- a/drivers/infiniband/sw/rxe/rxe.h
+++ b/drivers/infiniband/sw/rxe/rxe.h
@@ -100,43 +100,6 @@
 #define rxe_info_mw(mw, fmt, ...) ibdev_info_ratelimited((mw)->ibmw.device, \
 		"mw#%d %s:  " fmt, (mw)->elem.index, __func__, ##__VA_ARGS__)
 
-/* responder states */
-enum resp_states {
-	RESPST_NONE,
-	RESPST_GET_REQ,
-	RESPST_CHK_PSN,
-	RESPST_CHK_OP_SEQ,
-	RESPST_CHK_OP_VALID,
-	RESPST_CHK_RESOURCE,
-	RESPST_CHK_LENGTH,
-	RESPST_CHK_RKEY,
-	RESPST_EXECUTE,
-	RESPST_READ_REPLY,
-	RESPST_ATOMIC_REPLY,
-	RESPST_ATOMIC_WRITE_REPLY,
-	RESPST_PROCESS_FLUSH,
-	RESPST_COMPLETE,
-	RESPST_ACKNOWLEDGE,
-	RESPST_CLEANUP,
-	RESPST_DUPLICATE_REQUEST,
-	RESPST_ERR_MALFORMED_WQE,
-	RESPST_ERR_UNSUPPORTED_OPCODE,
-	RESPST_ERR_MISALIGNED_ATOMIC,
-	RESPST_ERR_PSN_OUT_OF_SEQ,
-	RESPST_ERR_MISSING_OPCODE_FIRST,
-	RESPST_ERR_MISSING_OPCODE_LAST_C,
-	RESPST_ERR_MISSING_OPCODE_LAST_D1E,
-	RESPST_ERR_TOO_MANY_RDMA_ATM_REQ,
-	RESPST_ERR_RNR,
-	RESPST_ERR_RKEY_VIOLATION,
-	RESPST_ERR_INVALIDATE_RKEY,
-	RESPST_ERR_LENGTH,
-	RESPST_ERR_CQ_OVERFLOW,
-	RESPST_ERROR,
-	RESPST_DONE,
-	RESPST_EXIT,
-};
-
 void rxe_set_mtu(struct rxe_dev *rxe, unsigned int dev_mtu);
 
 int rxe_add(struct rxe_dev *rxe, unsigned int mtu, const char *ibdev_name);
diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h
index ded46119151b..1c78b9d36da4 100644
--- a/drivers/infiniband/sw/rxe/rxe_loc.h
+++ b/drivers/infiniband/sw/rxe/rxe_loc.h
@@ -58,6 +58,7 @@ int rxe_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
 
 /* rxe_mr.c */
 u8 rxe_get_next_key(u32 last_key);
+void rxe_mr_init(int access, struct rxe_mr *mr);
 void rxe_mr_init_dma(int access, struct rxe_mr *mr);
 int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length,
 		     int access, struct rxe_mr *mr);
diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
index da3dee520876..12cacc4f60f2 100644
--- a/drivers/infiniband/sw/rxe/rxe_mr.c
+++ b/drivers/infiniband/sw/rxe/rxe_mr.c
@@ -45,7 +45,7 @@ int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length)
 	}
 }
 
-static void rxe_mr_init(int access, struct rxe_mr *mr)
+void rxe_mr_init(int access, struct rxe_mr *mr)
 {
 	u32 key = mr->elem.index << 8 | rxe_get_next_key(-1);
 
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h
index 3c1354f82283..e4656c7640f0 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.h
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.h
@@ -126,6 +126,43 @@ struct rxe_comp_info {
 	u32			rnr_retry;
 };
 
+/* responder states */
+enum resp_states {
+	RESPST_NONE,
+	RESPST_GET_REQ,
+	RESPST_CHK_PSN,
+	RESPST_CHK_OP_SEQ,
+	RESPST_CHK_OP_VALID,
+	RESPST_CHK_RESOURCE,
+	RESPST_CHK_LENGTH,
+	RESPST_CHK_RKEY,
+	RESPST_EXECUTE,
+	RESPST_READ_REPLY,
+	RESPST_ATOMIC_REPLY,
+	RESPST_ATOMIC_WRITE_REPLY,
+	RESPST_PROCESS_FLUSH,
+	RESPST_COMPLETE,
+	RESPST_ACKNOWLEDGE,
+	RESPST_CLEANUP,
+	RESPST_DUPLICATE_REQUEST,
+	RESPST_ERR_MALFORMED_WQE,
+	RESPST_ERR_UNSUPPORTED_OPCODE,
+	RESPST_ERR_MISALIGNED_ATOMIC,
+	RESPST_ERR_PSN_OUT_OF_SEQ,
+	RESPST_ERR_MISSING_OPCODE_FIRST,
+	RESPST_ERR_MISSING_OPCODE_LAST_C,
+	RESPST_ERR_MISSING_OPCODE_LAST_D1E,
+	RESPST_ERR_TOO_MANY_RDMA_ATM_REQ,
+	RESPST_ERR_RNR,
+	RESPST_ERR_RKEY_VIOLATION,
+	RESPST_ERR_INVALIDATE_RKEY,
+	RESPST_ERR_LENGTH,
+	RESPST_ERR_CQ_OVERFLOW,
+	RESPST_ERROR,
+	RESPST_DONE,
+	RESPST_EXIT,
+};
+
 enum rdatm_res_state {
 	rdatm_res_state_next,
 	rdatm_res_state_new,
-- 
2.43.0

[PATCH for-next v9 2/5] RDMA/rxe: Add page invalidation support

[Daisuke Matsuda]

On page invalidation, an MMU notifier callback is invoked to unmap DMA
addresses and update the driver page table(umem_odp->dma_list). The
callback is registered when an ODP-enabled MR is created.

Signed-off-by: Daisuke Matsuda <matsuda-daisuke@fujitsu.com>
---
 drivers/infiniband/sw/rxe/Makefile  |  2 ++
 drivers/infiniband/sw/rxe/rxe_loc.h |  3 +++
 drivers/infiniband/sw/rxe/rxe_odp.c | 38 +++++++++++++++++++++++++++++
 3 files changed, 43 insertions(+)
 create mode 100644 drivers/infiniband/sw/rxe/rxe_odp.c

diff --git a/drivers/infiniband/sw/rxe/Makefile b/drivers/infiniband/sw/rxe/Makefile
index 5395a581f4bb..93134f1d1d0c 100644
--- a/drivers/infiniband/sw/rxe/Makefile
+++ b/drivers/infiniband/sw/rxe/Makefile
@@ -23,3 +23,5 @@ rdma_rxe-y := \
 	rxe_task.o \
 	rxe_net.o \
 	rxe_hw_counters.o
+
+rdma_rxe-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += rxe_odp.o
diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h
index 1c78b9d36da4..0162ac9431c1 100644
--- a/drivers/infiniband/sw/rxe/rxe_loc.h
+++ b/drivers/infiniband/sw/rxe/rxe_loc.h
@@ -182,4 +182,7 @@ static inline unsigned int wr_opcode_mask(int opcode, struct rxe_qp *qp)
 	return rxe_wr_opcode_info[opcode].mask[qp->ibqp.qp_type];
 }
 
+/* rxe_odp.c */
+extern const struct mmu_interval_notifier_ops rxe_mn_ops;
+
 #endif /* RXE_LOC_H */
diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c
new file mode 100644
index 000000000000..2be8066db012
--- /dev/null
+++ b/drivers/infiniband/sw/rxe/rxe_odp.c
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/*
+ * Copyright (c) 2022-2023 Fujitsu Ltd. All rights reserved.
+ */
+
+#include <linux/hmm.h>
+
+#include <rdma/ib_umem_odp.h>
+
+#include "rxe.h"
+
+static bool rxe_ib_invalidate_range(struct mmu_interval_notifier *mni,
+				    const struct mmu_notifier_range *range,
+				    unsigned long cur_seq)
+{
+	struct ib_umem_odp *umem_odp =
+		container_of(mni, struct ib_umem_odp, notifier);
+	unsigned long start, end;
+
+	if (!mmu_notifier_range_blockable(range))
+		return false;
+
+	mutex_lock(&umem_odp->umem_mutex);
+	mmu_interval_set_seq(mni, cur_seq);
+
+	start = max_t(u64, ib_umem_start(umem_odp), range->start);
+	end = min_t(u64, ib_umem_end(umem_odp), range->end);
+
+	/* update umem_odp->dma_list */
+	ib_umem_odp_unmap_dma_pages(umem_odp, start, end);
+
+	mutex_unlock(&umem_odp->umem_mutex);
+	return true;
+}
+
+const struct mmu_interval_notifier_ops rxe_mn_ops = {
+	.invalidate = rxe_ib_invalidate_range,
+};
-- 
2.43.0

[PATCH for-next v9 3/5] RDMA/rxe: Allow registering MRs for On-Demand Paging

[Daisuke Matsuda]

Allow userspace to register an ODP-enabled MR, in which case the flag
IB_ACCESS_ON_DEMAND is passed to rxe_reg_user_mr(). However, there is no
RDMA operation enabled right now. They will be supported later in the
subsequent two patches.

rxe_odp_do_pagefault() is called to initialize an ODP-enabled MR. It syncs
process address space from the CPU page table to the driver page table
(dma_list/pfn_list in umem_odp) when called with RXE_PAGEFAULT_SNAPSHOT
flag. Additionally, It can be used to trigger page fault when pages being
accessed are not present or do not have proper read/write permissions, and
possibly to prefetch pages in the future.

Signed-off-by: Daisuke Matsuda <matsuda-daisuke@fujitsu.com>
---
 drivers/infiniband/sw/rxe/rxe.c       |  7 +++
 drivers/infiniband/sw/rxe/rxe_loc.h   | 12 ++++
 drivers/infiniband/sw/rxe/rxe_mr.c    |  9 ++-
 drivers/infiniband/sw/rxe/rxe_odp.c   | 86 +++++++++++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_resp.c  | 15 +++--
 drivers/infiniband/sw/rxe/rxe_verbs.c |  5 +-
 6 files changed, 128 insertions(+), 6 deletions(-)

diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c
index 255677bc12b2..3ca73f8d96cc 100644
--- a/drivers/infiniband/sw/rxe/rxe.c
+++ b/drivers/infiniband/sw/rxe/rxe.c
@@ -75,6 +75,13 @@ static void rxe_init_device_param(struct rxe_dev *rxe)
 			rxe->ndev->dev_addr);
 
 	rxe->max_ucontext			= RXE_MAX_UCONTEXT;
+
+	if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
+		rxe->attr.kernel_cap_flags |= IBK_ON_DEMAND_PAGING;
+
+		/* IB_ODP_SUPPORT_IMPLICIT is not supported right now. */
+		rxe->attr.odp_caps.general_caps |= IB_ODP_SUPPORT;
+	}
 }
 
 /* initialize port attributes */
diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h
index 0162ac9431c1..1043240daa97 100644
--- a/drivers/infiniband/sw/rxe/rxe_loc.h
+++ b/drivers/infiniband/sw/rxe/rxe_loc.h
@@ -185,4 +185,16 @@ static inline unsigned int wr_opcode_mask(int opcode, struct rxe_qp *qp)
 /* rxe_odp.c */
 extern const struct mmu_interval_notifier_ops rxe_mn_ops;
 
+#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
+int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length,
+			 u64 iova, int access_flags, struct rxe_mr *mr);
+#else /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
+static inline int
+rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
+		     int access_flags, struct rxe_mr *mr)
+{
+	return -EOPNOTSUPP;
+}
+#endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
+
 #endif /* RXE_LOC_H */
diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
index 12cacc4f60f2..190225a34139 100644
--- a/drivers/infiniband/sw/rxe/rxe_mr.c
+++ b/drivers/infiniband/sw/rxe/rxe_mr.c
@@ -323,7 +323,10 @@ int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr,
 		return err;
 	}
 
-	return rxe_mr_copy_xarray(mr, iova, addr, length, dir);
+	if (mr->umem->is_odp)
+		return -EOPNOTSUPP;
+	else
+		return rxe_mr_copy_xarray(mr, iova, addr, length, dir);
 }
 
 /* copy data in or out of a wqe, i.e. sg list
@@ -532,6 +535,10 @@ int rxe_mr_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value)
 	struct page *page;
 	u64 *va;
 
+	/* ODP is not supported right now. WIP. */
+	if (mr->umem->is_odp)
+		return RESPST_ERR_UNSUPPORTED_OPCODE;
+
 	/* See IBA oA19-28 */
 	if (unlikely(mr->state != RXE_MR_STATE_VALID)) {
 		rxe_dbg_mr(mr, "mr not in valid state\n");
diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c
index 2be8066db012..141290078754 100644
--- a/drivers/infiniband/sw/rxe/rxe_odp.c
+++ b/drivers/infiniband/sw/rxe/rxe_odp.c
@@ -36,3 +36,89 @@ static bool rxe_ib_invalidate_range(struct mmu_interval_notifier *mni,
 const struct mmu_interval_notifier_ops rxe_mn_ops = {
 	.invalidate = rxe_ib_invalidate_range,
 };
+
+#define RXE_PAGEFAULT_RDONLY BIT(1)
+#define RXE_PAGEFAULT_SNAPSHOT BIT(2)
+static int rxe_odp_do_pagefault_and_lock(struct rxe_mr *mr, u64 user_va, int bcnt, u32 flags)
+{
+	struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem);
+	bool fault = !(flags & RXE_PAGEFAULT_SNAPSHOT);
+	u64 access_mask;
+	int np;
+
+	access_mask = ODP_READ_ALLOWED_BIT;
+	if (umem_odp->umem.writable && !(flags & RXE_PAGEFAULT_RDONLY))
+		access_mask |= ODP_WRITE_ALLOWED_BIT;
+
+	/*
+	 * ib_umem_odp_map_dma_and_lock() locks umem_mutex on success.
+	 * Callers must release the lock later to let invalidation handler
+	 * do its work again.
+	 */
+	np = ib_umem_odp_map_dma_and_lock(umem_odp, user_va, bcnt,
+					  access_mask, fault);
+	return np;
+}
+
+static int rxe_odp_init_pages(struct rxe_mr *mr)
+{
+	struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem);
+	int ret;
+
+	ret = rxe_odp_do_pagefault_and_lock(mr, mr->umem->address,
+					    mr->umem->length,
+					    RXE_PAGEFAULT_SNAPSHOT);
+
+	if (ret >= 0)
+		mutex_unlock(&umem_odp->umem_mutex);
+
+	return ret >= 0 ? 0 : ret;
+}
+
+int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length,
+			 u64 iova, int access_flags, struct rxe_mr *mr)
+{
+	struct ib_umem_odp *umem_odp;
+	int err;
+
+	if (!IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING))
+		return -EOPNOTSUPP;
+
+	rxe_mr_init(access_flags, mr);
+
+	if (!start && length == U64_MAX) {
+		if (iova != 0)
+			return -EINVAL;
+		if (!(rxe->attr.odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT))
+			return -EINVAL;
+
+		/* Never reach here, for implicit ODP is not implemented. */
+	}
+
+	umem_odp = ib_umem_odp_get(&rxe->ib_dev, start, length, access_flags,
+				   &rxe_mn_ops);
+	if (IS_ERR(umem_odp)) {
+		rxe_dbg_mr(mr, "Unable to create umem_odp err = %d\n",
+			   (int)PTR_ERR(umem_odp));
+		return PTR_ERR(umem_odp);
+	}
+
+	umem_odp->private = mr;
+
+	mr->umem = &umem_odp->umem;
+	mr->access = access_flags;
+	mr->ibmr.length = length;
+	mr->ibmr.iova = iova;
+	mr->page_offset = ib_umem_offset(&umem_odp->umem);
+
+	err = rxe_odp_init_pages(mr);
+	if (err) {
+		ib_umem_odp_release(umem_odp);
+		return err;
+	}
+
+	mr->state = RXE_MR_STATE_VALID;
+	mr->ibmr.type = IB_MR_TYPE_USER;
+
+	return err;
+}
diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
index c11ab280551a..e703a3ab82d4 100644
--- a/drivers/infiniband/sw/rxe/rxe_resp.c
+++ b/drivers/infiniband/sw/rxe/rxe_resp.c
@@ -649,6 +649,10 @@ static enum resp_states process_flush(struct rxe_qp *qp,
 	struct rxe_mr *mr = qp->resp.mr;
 	struct resp_res *res = qp->resp.res;
 
+	/* ODP is not supported right now. WIP. */
+	if (mr->umem->is_odp)
+		return RESPST_ERR_UNSUPPORTED_OPCODE;
+
 	/* oA19-14, oA19-15 */
 	if (res && res->replay)
 		return RESPST_ACKNOWLEDGE;
@@ -702,10 +706,13 @@ static enum resp_states atomic_reply(struct rxe_qp *qp,
 	if (!res->replay) {
 		u64 iova = qp->resp.va + qp->resp.offset;
 
-		err = rxe_mr_do_atomic_op(mr, iova, pkt->opcode,
-					  atmeth_comp(pkt),
-					  atmeth_swap_add(pkt),
-					  &res->atomic.orig_val);
+		if (mr->umem->is_odp)
+			err = RESPST_ERR_UNSUPPORTED_OPCODE;
+		else
+			err = rxe_mr_do_atomic_op(mr, iova, pkt->opcode,
+						  atmeth_comp(pkt),
+						  atmeth_swap_add(pkt),
+						  &res->atomic.orig_val);
 		if (err)
 			return err;
 
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
index 5c18f7e342f2..13064302d766 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
@@ -1278,7 +1278,10 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, u64 start,
 	mr->ibmr.pd = ibpd;
 	mr->ibmr.device = ibpd->device;
 
-	err = rxe_mr_init_user(rxe, start, length, access, mr);
+	if (access & IB_ACCESS_ON_DEMAND)
+		err = rxe_odp_mr_init_user(rxe, start, length, iova, access, mr);
+	else
+		err = rxe_mr_init_user(rxe, start, length, access, mr);
 	if (err) {
 		rxe_dbg_mr(mr, "reg_user_mr failed, err = %d\n", err);
 		goto err_cleanup;
-- 
2.43.0

[PATCH for-next v9 4/5] RDMA/rxe: Add support for Send/Recv/Write/Read with ODP

[Daisuke Matsuda]

rxe_mr_copy() is used widely to copy data to/from a user MR. requester uses
it to load payloads of requesting packets; responder uses it to process
Send, Write, and Read operaetions; completer uses it to copy data from
response packets of Read and Atomic operations to a user MR.

Allow these operations to be used with ODP by adding a subordinate function
rxe_odp_mr_copy(). It is comprised of the following steps:
 1. Check the driver page table(umem_odp->dma_list) to see if pages being
    accessed are present with appropriate permission.
 2. If necessary, trigger page fault to map the pages.
 3. Convert their user space addresses to kernel logical addresses using
    PFNs in the driver page table(umem_odp->pfn_list).
 4. Execute data copy to/from the pages.

Signed-off-by: Daisuke Matsuda <matsuda-daisuke@fujitsu.com>
---
 drivers/infiniband/sw/rxe/rxe.c     |  10 +++
 drivers/infiniband/sw/rxe/rxe_loc.h |   7 ++
 drivers/infiniband/sw/rxe/rxe_mr.c  |   2 +-
 drivers/infiniband/sw/rxe/rxe_odp.c | 131 ++++++++++++++++++++++++++++
 4 files changed, 149 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c
index 3ca73f8d96cc..ea643ebf9667 100644
--- a/drivers/infiniband/sw/rxe/rxe.c
+++ b/drivers/infiniband/sw/rxe/rxe.c
@@ -81,6 +81,16 @@ static void rxe_init_device_param(struct rxe_dev *rxe)
 
 		/* IB_ODP_SUPPORT_IMPLICIT is not supported right now. */
 		rxe->attr.odp_caps.general_caps |= IB_ODP_SUPPORT;
+
+		rxe->attr.odp_caps.per_transport_caps.ud_odp_caps |= IB_ODP_SUPPORT_SEND;
+		rxe->attr.odp_caps.per_transport_caps.ud_odp_caps |= IB_ODP_SUPPORT_RECV;
+		rxe->attr.odp_caps.per_transport_caps.ud_odp_caps |= IB_ODP_SUPPORT_SRQ_RECV;
+
+		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_SEND;
+		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_RECV;
+		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_WRITE;
+		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_READ;
+		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_SRQ_RECV;
 	}
 }
 
diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h
index 1043240daa97..7a735108d475 100644
--- a/drivers/infiniband/sw/rxe/rxe_loc.h
+++ b/drivers/infiniband/sw/rxe/rxe_loc.h
@@ -188,6 +188,8 @@ extern const struct mmu_interval_notifier_ops rxe_mn_ops;
 #ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
 int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length,
 			 u64 iova, int access_flags, struct rxe_mr *mr);
+int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length,
+		    enum rxe_mr_copy_dir dir);
 #else /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
 static inline int
 rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
@@ -195,6 +197,11 @@ rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
 {
 	return -EOPNOTSUPP;
 }
+static inline int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr,
+				  int length, enum rxe_mr_copy_dir dir)
+{
+	return -EOPNOTSUPP;
+}
 #endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
 
 #endif /* RXE_LOC_H */
diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
index 190225a34139..6cd668a8dfb2 100644
--- a/drivers/infiniband/sw/rxe/rxe_mr.c
+++ b/drivers/infiniband/sw/rxe/rxe_mr.c
@@ -324,7 +324,7 @@ int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr,
 	}
 
 	if (mr->umem->is_odp)
-		return -EOPNOTSUPP;
+		return rxe_odp_mr_copy(mr, iova, addr, length, dir);
 	else
 		return rxe_mr_copy_xarray(mr, iova, addr, length, dir);
 }
diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c
index 141290078754..d3c67d18c173 100644
--- a/drivers/infiniband/sw/rxe/rxe_odp.c
+++ b/drivers/infiniband/sw/rxe/rxe_odp.c
@@ -122,3 +122,134 @@ int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length,
 
 	return err;
 }
+
+static inline bool rxe_check_pagefault(struct ib_umem_odp *umem_odp,
+				       u64 iova, int length, u32 perm)
+{
+	bool need_fault = false;
+	u64 addr;
+	int idx;
+
+	addr = iova & (~(BIT(umem_odp->page_shift) - 1));
+
+	/* Skim through all pages that are to be accessed. */
+	while (addr < iova + length) {
+		idx = (addr - ib_umem_start(umem_odp)) >> umem_odp->page_shift;
+
+		if (!(umem_odp->dma_list[idx] & perm)) {
+			need_fault = true;
+			break;
+		}
+
+		addr += BIT(umem_odp->page_shift);
+	}
+	return need_fault;
+}
+
+static int rxe_odp_map_range_and_lock(struct rxe_mr *mr, u64 iova, int length, u32 flags)
+{
+	struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem);
+	bool need_fault;
+	u64 perm;
+	int err;
+
+	if (unlikely(length < 1))
+		return -EINVAL;
+
+	perm = ODP_READ_ALLOWED_BIT;
+	if (!(flags & RXE_PAGEFAULT_RDONLY))
+		perm |= ODP_WRITE_ALLOWED_BIT;
+
+	mutex_lock(&umem_odp->umem_mutex);
+
+	need_fault = rxe_check_pagefault(umem_odp, iova, length, perm);
+	if (need_fault) {
+		mutex_unlock(&umem_odp->umem_mutex);
+
+		/* umem_mutex is locked on success. */
+		err = rxe_odp_do_pagefault_and_lock(mr, iova, length,
+						    flags);
+		if (err < 0)
+			return err;
+
+		need_fault = rxe_check_pagefault(umem_odp, iova, length, perm);
+		if (need_fault)
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+static int __rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr,
+			     int length, enum rxe_mr_copy_dir dir)
+{
+	struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem);
+	struct page *page;
+	int idx, bytes;
+	size_t offset;
+	u8 *user_va;
+
+	idx = (iova - ib_umem_start(umem_odp)) >> umem_odp->page_shift;
+	offset = iova & (BIT(umem_odp->page_shift) - 1);
+
+	while (length > 0) {
+		u8 *src, *dest;
+
+		page = hmm_pfn_to_page(umem_odp->pfn_list[idx]);
+		user_va = kmap_local_page(page);
+		if (!user_va)
+			return -EFAULT;
+
+		src = (dir == RXE_TO_MR_OBJ) ? addr : user_va;
+		dest = (dir == RXE_TO_MR_OBJ) ? user_va : addr;
+
+		bytes = BIT(umem_odp->page_shift) - offset;
+		if (bytes > length)
+			bytes = length;
+
+		memcpy(dest, src, bytes);
+		kunmap_local(user_va);
+
+		length  -= bytes;
+		idx++;
+		offset = 0;
+	}
+
+	return 0;
+}
+
+int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length,
+		    enum rxe_mr_copy_dir dir)
+{
+	struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem);
+	u32 flags = 0;
+	int err;
+
+	if (length == 0)
+		return 0;
+
+	if (unlikely(!mr->umem->is_odp))
+		return -EOPNOTSUPP;
+
+	switch (dir) {
+	case RXE_TO_MR_OBJ:
+		break;
+
+	case RXE_FROM_MR_OBJ:
+		flags = RXE_PAGEFAULT_RDONLY;
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	err = rxe_odp_map_range_and_lock(mr, iova, length, flags);
+	if (err)
+		return err;
+
+	err =  __rxe_odp_mr_copy(mr, iova, addr, length, dir);
+
+	mutex_unlock(&umem_odp->umem_mutex);
+
+	return err;
+}
-- 
2.43.0

[PATCH for-next v9 5/5] RDMA/rxe: Add support for the traditional Atomic operations with ODP

[Daisuke Matsuda]

Enable 'fetch and add' and 'compare and swap' operations to be used with
ODP. This is comprised of the following steps:
 1. Check the driver page table(umem_odp->dma_list) to see if the target
    page is both readable and writable.
 2. If not, then trigger page fault to map the page.
 3. Convert its user space address to a kernel logical address using PFNs
    in the driver page table(umem_odp->pfn_list).
 4. Execute the operation.

Signed-off-by: Daisuke Matsuda <matsuda-daisuke@fujitsu.com>
---
 drivers/infiniband/sw/rxe/rxe.c      |  1 +
 drivers/infiniband/sw/rxe/rxe_loc.h  | 11 +++++
 drivers/infiniband/sw/rxe/rxe_mr.c   |  2 +-
 drivers/infiniband/sw/rxe/rxe_odp.c  | 69 ++++++++++++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_resp.c |  5 +-
 5 files changed, 86 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c
index ea643ebf9667..08c69c637663 100644
--- a/drivers/infiniband/sw/rxe/rxe.c
+++ b/drivers/infiniband/sw/rxe/rxe.c
@@ -90,6 +90,7 @@ static void rxe_init_device_param(struct rxe_dev *rxe)
 		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_RECV;
 		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_WRITE;
 		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_READ;
+		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_ATOMIC;
 		rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_SRQ_RECV;
 	}
 }
diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h
index 7a735108d475..a45ff6236613 100644
--- a/drivers/infiniband/sw/rxe/rxe_loc.h
+++ b/drivers/infiniband/sw/rxe/rxe_loc.h
@@ -81,6 +81,9 @@ int rxe_invalidate_mr(struct rxe_qp *qp, u32 key);
 int rxe_reg_fast_mr(struct rxe_qp *qp, struct rxe_send_wqe *wqe);
 void rxe_mr_cleanup(struct rxe_pool_elem *elem);
 
+/* defined in rxe_mr.c; used in rxe_mr.c and rxe_odp.c */
+extern spinlock_t atomic_ops_lock;
+
 /* rxe_mw.c */
 int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata);
 int rxe_dealloc_mw(struct ib_mw *ibmw);
@@ -190,6 +193,8 @@ int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length,
 			 u64 iova, int access_flags, struct rxe_mr *mr);
 int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length,
 		    enum rxe_mr_copy_dir dir);
+int rxe_odp_atomic_op(struct rxe_mr *mr, u64 iova, int opcode,
+			 u64 compare, u64 swap_add, u64 *orig_val);
 #else /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
 static inline int
 rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova,
@@ -202,6 +207,12 @@ static inline int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr,
 {
 	return -EOPNOTSUPP;
 }
+static inline int
+rxe_odp_atomic_op(struct rxe_mr *mr, u64 iova, int opcode,
+		     u64 compare, u64 swap_add, u64 *orig_val)
+{
+	return RESPST_ERR_UNSUPPORTED_OPCODE;
+}
 #endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
 
 #endif /* RXE_LOC_H */
diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
index 6cd668a8dfb2..868d2f0b74e9 100644
--- a/drivers/infiniband/sw/rxe/rxe_mr.c
+++ b/drivers/infiniband/sw/rxe/rxe_mr.c
@@ -469,7 +469,7 @@ int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length)
 }
 
 /* Guarantee atomicity of atomic operations at the machine level. */
-static DEFINE_SPINLOCK(atomic_ops_lock);
+DEFINE_SPINLOCK(atomic_ops_lock);
 
 int rxe_mr_do_atomic_op(struct rxe_mr *mr, u64 iova, int opcode,
 			u64 compare, u64 swap_add, u64 *orig_val)
diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c
index d3c67d18c173..a82e5011360c 100644
--- a/drivers/infiniband/sw/rxe/rxe_odp.c
+++ b/drivers/infiniband/sw/rxe/rxe_odp.c
@@ -253,3 +253,72 @@ int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length,
 
 	return err;
 }
+
+static int rxe_odp_do_atomic_op(struct rxe_mr *mr, u64 iova, int opcode,
+				u64 compare, u64 swap_add, u64 *orig_val)
+{
+	struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem);
+	unsigned int page_offset;
+	struct page *page;
+	unsigned int idx;
+	u64 value;
+	u64 *va;
+	int err;
+
+	if (unlikely(mr->state != RXE_MR_STATE_VALID)) {
+		rxe_dbg_mr(mr, "mr not in valid state\n");
+		return RESPST_ERR_RKEY_VIOLATION;
+	}
+
+	err = mr_check_range(mr, iova, sizeof(value));
+	if (err) {
+		rxe_dbg_mr(mr, "iova out of range\n");
+		return RESPST_ERR_RKEY_VIOLATION;
+	}
+
+	idx = (iova - ib_umem_start(umem_odp)) >> umem_odp->page_shift;
+	page_offset = iova & (BIT(umem_odp->page_shift) - 1);
+	page = hmm_pfn_to_page(umem_odp->pfn_list[idx]);
+	if (!page)
+		return RESPST_ERR_RKEY_VIOLATION;
+
+	if (unlikely(page_offset & 0x7)) {
+		rxe_dbg_mr(mr, "iova not aligned\n");
+		return RESPST_ERR_MISALIGNED_ATOMIC;
+	}
+
+	va = kmap_local_page(page);
+
+	spin_lock_bh(&atomic_ops_lock);
+	value = *orig_val = va[page_offset >> 3];
+
+	if (opcode == IB_OPCODE_RC_COMPARE_SWAP) {
+		if (value == compare)
+			va[page_offset >> 3] = swap_add;
+	} else {
+		value += swap_add;
+		va[page_offset >> 3] = value;
+	}
+	spin_unlock_bh(&atomic_ops_lock);
+
+	kunmap_local(va);
+
+	return 0;
+}
+
+int rxe_odp_atomic_op(struct rxe_mr *mr, u64 iova, int opcode,
+			 u64 compare, u64 swap_add, u64 *orig_val)
+{
+	struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem);
+	int err;
+
+	err = rxe_odp_map_range_and_lock(mr, iova, sizeof(char), 0);
+	if (err < 0)
+		return err;
+
+	err = rxe_odp_do_atomic_op(mr, iova, opcode, compare, swap_add,
+				   orig_val);
+	mutex_unlock(&umem_odp->umem_mutex);
+
+	return err;
+}
diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
index e703a3ab82d4..54ba9ee1acc5 100644
--- a/drivers/infiniband/sw/rxe/rxe_resp.c
+++ b/drivers/infiniband/sw/rxe/rxe_resp.c
@@ -707,7 +707,10 @@ static enum resp_states atomic_reply(struct rxe_qp *qp,
 		u64 iova = qp->resp.va + qp->resp.offset;
 
 		if (mr->umem->is_odp)
-			err = RESPST_ERR_UNSUPPORTED_OPCODE;
+			err = rxe_odp_atomic_op(mr, iova, pkt->opcode,
+						atmeth_comp(pkt),
+						atmeth_swap_add(pkt),
+						&res->atomic.orig_val);
 		else
 			err = rxe_mr_do_atomic_op(mr, iova, pkt->opcode,
 						  atmeth_comp(pkt),
-- 
2.43.0

RE: [PATCH for-next v9 0/5] On-Demand Paging on SoftRoCE

[Daisuke Matsuda (Fujitsu)]

On Mon, Dec 23, 2024 2:25 AM Joe Klein <joe.klein812@gmail.com> wrote:
> We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to enable/disable this patchset?

Hi Joe,

Can you clarify the test and the problems you observed?
I wonder if you tried the test with the latest tree WITHOUT my patches.

As far as I know, there is something wrong with the upstream right now.
It does not complete the rdma-core testcases, and 'segmentation fault' is observed
in the middle of the full test run, which did not happen before October 2024.

Here are the details of the issue:
===== test log =====
ubuntu@rdma-dev:~$ sudo rdma link add rxe_ens3 type rxe netdev ens3
ubuntu@rdma-dev:~$ cd rdma-core
ubuntu@rdma-dev:~/rdma-core$ uname -r
6.13.0-rc1+
ubuntu@rdma-dev:~/rdma-core$ pwd
/home/ubuntu/rdma-core
ubuntu@rdma-dev:~/rdma-core$ ./build/bin/run_tests.py
..........ss.../usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpe7nsitov' mode='rb+' closefd=True>
  def _remove(item, selfref=ref(self)):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpid85cbou' mode='rb+' closefd=True>
  def _remove(item, selfref=ref(self)):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
.......ssssss/usr/lib/python3.12/contextlib.py:141: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmp9pgb7zo8' mode='rb+' closefd=True>
  def __exit__(self, typ, value, traceback):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssssssssssssssss/usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpate1loci' mode='rb+' closefd=True>
  def _remove(item, selfref=ref(self)):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
ssssTraceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
Traceback (most recent call last):
  File "pd.pyx", line 120, in pyverbs.pd.PD.close
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
s....ssSegmentation fault (core dumped)
===========

=====dmesg=====
[  147.464243] rxe_ens3: qp#21 make_send_cqe: non-flush error status = 4
[  147.473843] rxe_ens3: qp#23 make_send_cqe: non-flush error status = 10
[  147.484540] rxe_ens3: qp#25 make_send_cqe: non-flush error status = 9
[  147.494541] rxe_ens3: qp#27 make_send_cqe: non-flush error status = 10
[  147.524080] rxe_ens3: rxe_create_cq: returned err = -22
[  147.574197] rxe_ens3: cq#26 rxe_resize_cq: returned err = -22
[  147.605719] rxe_ens3: rxe_create_cq: returned err = -95
[  147.606454] rxe_ens3: rxe_create_cq: returned err = -22
[  148.803131] rxe_ens3: qp#51 make_send_cqe: non-flush error status = 10
[  148.831587] rxe_ens3: qp#57 make_send_cqe: non-flush error status = 10
[  148.841627] rxe_ens3: qp#59 make_send_cqe: non-flush error status = 10
[  148.851719] rxe_ens3: qp#61 make_send_cqe: non-flush error status = 10
[  149.104223] python3[1702]: segfault at d0 ip 00007ff95ced16c7 sp 00007fff5e775de0 error 4 in libibverbs.so.1.14.56.0[e6c7,7ff95ceca000+14000] likely on CPU 2 (core 0, socket 2)
[  149.104235] Code: 00 00 c1 e0 04 8b bf 08 01 00 00 48 8d 53 20 48 c7 43 28 00 00 00 00 83 c0 18 c7 43 34 00 00 00 00 be 01 1b 18 c0 66 89 43 20 <49> 8b 80 d0 00 00 00 8b 40 10 89 43 30 31 c0 e8 05 99 ff ff 41 89
=====

If you encounter any problems that surely comes from my ODP patches, please let me know what symptoms you are seeing.
I would also appreciate any help you can offer in fixing the upstream issue.

Thanks,
Daisuke

RE: [PATCH for-next v9 0/5] On-Demand Paging on SoftRoCE

[Daisuke Matsuda (Fujitsu)]

On Mon, Dec 23, 2024 10:55 AM Daisuke Matsuda (Fujitsu) <matsuda-daisuke@fujitsu.com> wrote:
> On Mon, Dec 23, 2024 2:25 AM Joe Klein <joe.klein812@gmail.com> wrote:
> > We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if
> others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to
> enable/disable this patchset?
> 
> Hi Joe,
> 
> Can you clarify the test and the problems you observed?
> I wonder if you tried the test with the latest tree WITHOUT my patches.
> 
> As far as I know, there is something wrong with the upstream right now.
> It does not complete the rdma-core testcases, and 'segmentation fault' is observed
> in the middle of the full test run, which did not happen before October 2024.

It appears that the root cause of this issue lies within the userspace components.
My report yesterday was based on experiments conducted on Ubuntu 24.04.1 LTS (x86_64).
It seems to me that rxe is somehow broken regardless of kernel version
as long as userspace components are provided by Ubuntu 24.04.1 LTS.
I built and tried linux-6.11, linux-6.10, and linux-6.8, and they all failed as I reported.

I switched to Ubuntu 22.04.5 LTS (aarch64) to test with the older libraries.
All tests available passed using the rdma for-next tree without any problem.
Then, I applied my ODP patches onto it, and everything is still fine.
####################
ubuntu@rdma-aarch64:~/rdma-core$ git branch -v
* master fb965e2d0 Merge pull request #1531 from selvintxavier/pbuf_optimization
ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py
..........ss..........ssssssssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssss..ss....s.sssssss....ss....ss..............s......................ss.............sss...ssss
----------------------------------------------------------------------
Ran 321 tests in 3.599s

OK (skipped=211)
ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py -k odp
sssssssss..ss....s.s
----------------------------------------------------------------------
Ran 20 tests in 0.269s

OK (skipped=13)
####################

Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2.
We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.

In conclusion, I believe there is nothing in my ODP patches that could cause
the rxe driver to fail. I would appreciate any feedback on potential improvements.

Thanks,
Daisuke

> 
> Here are the details of the issue:
> ===== test log =====
> ubuntu@rdma-dev:~$ sudo rdma link add rxe_ens3 type rxe netdev ens3
> ubuntu@rdma-dev:~$ cd rdma-core
> ubuntu@rdma-dev:~/rdma-core$ uname -r
> 6.13.0-rc1+
> ubuntu@rdma-dev:~/rdma-core$ pwd
> /home/ubuntu/rdma-core
> ubuntu@rdma-dev:~/rdma-core$ ./build/bin/run_tests.py
> ..........ss.../usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpe7nsitov'
> mode='rb+' closefd=True>
>   def _remove(item, selfref=ref(self)):
> ResourceWarning: Enable tracemalloc to get the object allocation traceback
> /usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpid85cbou'
> mode='rb+' closefd=True>
>   def _remove(item, selfref=ref(self)):
> ResourceWarning: Enable tracemalloc to get the object allocation traceback
> .......ssssss/usr/lib/python3.12/contextlib.py:141: ResourceWarning: unclosed file <_io.FileIO
> name='/tmp/tmp9pgb7zo8' mode='rb+' closefd=True>
>   def __exit__(self, typ, value, traceback):
> ResourceWarning: Enable tracemalloc to get the object allocation traceback
> ssss..............ssssssssssssssssssssssssss.ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss
> sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssssssssssssssss/
> usr/lib/python3.12/_weakrefset.py:39: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpate1loci'
> mode='rb+' closefd=True>
>   def _remove(item, selfref=ref(self)):
> ResourceWarning: Enable tracemalloc to get the object allocation traceback
> Traceback (most recent call last):
>   File "pd.pyx", line 120, in pyverbs.pd.PD.close
> pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
> Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
> Traceback (most recent call last):
>   File "pd.pyx", line 120, in pyverbs.pd.PD.close
> pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
> ssssTraceback (most recent call last):
>   File "pd.pyx", line 120, in pyverbs.pd.PD.close
> pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
> Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
> Traceback (most recent call last):
>   File "pd.pyx", line 120, in pyverbs.pd.PD.close
> pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
> Traceback (most recent call last):
>   File "pd.pyx", line 120, in pyverbs.pd.PD.close
> pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
> Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
> Traceback (most recent call last):
>   File "pd.pyx", line 120, in pyverbs.pd.PD.close
> pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
> Traceback (most recent call last):
>   File "pd.pyx", line 120, in pyverbs.pd.PD.close
> pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
> Exception ignored in: 'pyverbs.pd.PD.__dealloc__'
> Traceback (most recent call last):
>   File "pd.pyx", line 120, in pyverbs.pd.PD.close
> pyverbs.pyverbs_error.PyverbsRDMAError: Failed to dealloc PD. Errno: 9, Bad file descriptor
> s....ssSegmentation fault (core dumped)
> ===========
> 
> =====dmesg=====
> [  147.464243] rxe_ens3: qp#21 make_send_cqe: non-flush error status = 4
> [  147.473843] rxe_ens3: qp#23 make_send_cqe: non-flush error status = 10
> [  147.484540] rxe_ens3: qp#25 make_send_cqe: non-flush error status = 9
> [  147.494541] rxe_ens3: qp#27 make_send_cqe: non-flush error status = 10
> [  147.524080] rxe_ens3: rxe_create_cq: returned err = -22
> [  147.574197] rxe_ens3: cq#26 rxe_resize_cq: returned err = -22
> [  147.605719] rxe_ens3: rxe_create_cq: returned err = -95
> [  147.606454] rxe_ens3: rxe_create_cq: returned err = -22
> [  148.803131] rxe_ens3: qp#51 make_send_cqe: non-flush error status = 10
> [  148.831587] rxe_ens3: qp#57 make_send_cqe: non-flush error status = 10
> [  148.841627] rxe_ens3: qp#59 make_send_cqe: non-flush error status = 10
> [  148.851719] rxe_ens3: qp#61 make_send_cqe: non-flush error status = 10
> [  149.104223] python3[1702]: segfault at d0 ip 00007ff95ced16c7 sp 00007fff5e775de0 error 4 in
> libibverbs.so.1.14.56.0[e6c7,7ff95ceca000+14000] likely on CPU 2 (core 0, socket 2)
> [  149.104235] Code: 00 00 c1 e0 04 8b bf 08 01 00 00 48 8d 53 20 48 c7 43 28 00 00 00 00 83 c0 18 c7 43 34 00 00 00 00 be
> 01 1b 18 c0 66 89 43 20 <49> 8b 80 d0 00 00 00 8b 40 10 89 43 30 31 c0 e8 05 99 ff ff 41 89
> =====
> 
> If you encounter any problems that surely comes from my ODP patches, please let me know what symptoms you are
> seeing.
> I would also appreciate any help you can offer in fixing the upstream issue.
> 
> Thanks,
> Daisuke

Re: [PATCH for-next v9 0/5] On-Demand Paging on SoftRoCE

[Jason Gunthorpe]

On Tue, Dec 24, 2024 at 08:52:24AM +0000, Daisuke Matsuda (Fujitsu) wrote:
> On Mon, Dec 23, 2024 10:55 AM Daisuke Matsuda (Fujitsu) <matsuda-daisuke@fujitsu.com> wrote:
> > On Mon, Dec 23, 2024 2:25 AM Joe Klein <joe.klein812@gmail.com> wrote:
> > > We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if
> > others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to
> > enable/disable this patchset?
> > 
> > Hi Joe,
> > 
> > Can you clarify the test and the problems you observed?
> > I wonder if you tried the test with the latest tree WITHOUT my patches.
> > 
> > As far as I know, there is something wrong with the upstream right now.
> > It does not complete the rdma-core testcases, and 'segmentation fault' is observed
> > in the middle of the full test run, which did not happen before October 2024.
> 
> It appears that the root cause of this issue lies within the userspace components.
> My report yesterday was based on experiments conducted on Ubuntu 24.04.1 LTS (x86_64).
> It seems to me that rxe is somehow broken regardless of kernel version
> as long as userspace components are provided by Ubuntu 24.04.1 LTS.
> I built and tried linux-6.11, linux-6.10, and linux-6.8, and they all failed as I reported.
> 
> I switched to Ubuntu 22.04.5 LTS (aarch64) to test with the older libraries.
> All tests available passed using the rdma for-next tree without any problem.
> Then, I applied my ODP patches onto it, and everything is still fine.
> ####################
> ubuntu@rdma-aarch64:~/rdma-core$ git branch -v
> * master fb965e2d0 Merge pull request #1531 from selvintxavier/pbuf_optimization
> ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py
> ..........ss..........ssssssssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssss..ss....s.sssssss....ss....ss..............s......................ss.............sss...ssss
> ----------------------------------------------------------------------
> Ran 321 tests in 3.599s
> 
> OK (skipped=211)
> ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py -k odp
> sssssssss..ss....s.s
> ----------------------------------------------------------------------
> Ran 20 tests in 0.269s
> 
> OK (skipped=13)
> ####################
> 
> Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2.
> We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.

That's distressing.

> In conclusion, I believe there is nothing in my ODP patches that could cause
> the rxe driver to fail. I would appreciate any feedback on potential improvements.

What am I supposed to do with this though?

Joe, can you please answer Daisuke's questions on what problems you
observed and if you observe them without the ODP patches?

Jason

Re: [PATCH for-next v9 0/5] On-Demand Paging on SoftRoCE

[Joe Klein]

On Fri, Jan 3, 2025 at 4:05 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Tue, Dec 24, 2024 at 08:52:24AM +0000, Daisuke Matsuda (Fujitsu) wrote:
> > On Mon, Dec 23, 2024 10:55 AM Daisuke Matsuda (Fujitsu) <matsuda-daisuke@fujitsu.com> wrote:
> > > On Mon, Dec 23, 2024 2:25 AM Joe Klein <joe.klein812@gmail.com> wrote:
> > > > We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if
> > > others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to
> > > enable/disable this patchset?
> > >
> > > Hi Joe,
> > >
> > > Can you clarify the test and the problems you observed?
> > > I wonder if you tried the test with the latest tree WITHOUT my patches.
> > >
> > > As far as I know, there is something wrong with the upstream right now.
> > > It does not complete the rdma-core testcases, and 'segmentation fault' is observed
> > > in the middle of the full test run, which did not happen before October 2024.
> >
> > It appears that the root cause of this issue lies within the userspace components.
> > My report yesterday was based on experiments conducted on Ubuntu 24.04.1 LTS (x86_64).
> > It seems to me that rxe is somehow broken regardless of kernel version
> > as long as userspace components are provided by Ubuntu 24.04.1 LTS.
> > I built and tried linux-6.11, linux-6.10, and linux-6.8, and they all failed as I reported.
> >
> > I switched to Ubuntu 22.04.5 LTS (aarch64) to test with the older libraries.
> > All tests available passed using the rdma for-next tree without any problem.
> > Then, I applied my ODP patches onto it, and everything is still fine.
> > ####################
> > ubuntu@rdma-aarch64:~/rdma-core$ git branch -v
> > * master fb965e2d0 Merge pull request #1531 from selvintxavier/pbuf_optimization
> > ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py
> > ..........ss..........ssssssssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssss..ss....s.sssssss....ss....ss..............s......................ss.............sss...ssss
> > ----------------------------------------------------------------------
> > Ran 321 tests in 3.599s
> >
> > OK (skipped=211)
> > ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py -k odp
> > sssssssss..ss....s.s
> > ----------------------------------------------------------------------
> > Ran 20 tests in 0.269s
> >
> > OK (skipped=13)
> > ####################
> >
> > Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2.
> > We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.
>
> That's distressing.
>
> > In conclusion, I believe there is nothing in my ODP patches that could cause
> > the rxe driver to fail. I would appreciate any feedback on potential improvements.
>
> What am I supposed to do with this though?
>
> Joe, can you please answer Daisuke's questions on what problems you
> observed and if you observe them without the ODP patches?

Will make tests and let you know the result very soon.

>
> Jason

Re: [PATCH for-next v9 0/5] On-Demand Paging on SoftRoCE

[Joe Klein]

On Fri, Jan 3, 2025 at 6:40 PM Joe Klein <joe.klein812@gmail.com> wrote:
>
> On Fri, Jan 3, 2025 at 4:05 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >
> > On Tue, Dec 24, 2024 at 08:52:24AM +0000, Daisuke Matsuda (Fujitsu) wrote:
> > > On Mon, Dec 23, 2024 10:55 AM Daisuke Matsuda (Fujitsu) <matsuda-daisuke@fujitsu.com> wrote:
> > > > On Mon, Dec 23, 2024 2:25 AM Joe Klein <joe.klein812@gmail.com> wrote:
> > > > > We have tested this patcheset and had a lot of problems, even without using the ODP option in softroce. I don't know if
> > > > others have done similar tests. If we have to merge this patchset into upstream, is it > possible to add a kernel option to
> > > > enable/disable this patchset?
> > > >
> > > > Hi Joe,
> > > >
> > > > Can you clarify the test and the problems you observed?
> > > > I wonder if you tried the test with the latest tree WITHOUT my patches.
> > > >
> > > > As far as I know, there is something wrong with the upstream right now.
> > > > It does not complete the rdma-core testcases, and 'segmentation fault' is observed
> > > > in the middle of the full test run, which did not happen before October 2024.
> > >
> > > It appears that the root cause of this issue lies within the userspace components.
> > > My report yesterday was based on experiments conducted on Ubuntu 24.04.1 LTS (x86_64).
> > > It seems to me that rxe is somehow broken regardless of kernel version
> > > as long as userspace components are provided by Ubuntu 24.04.1 LTS.
> > > I built and tried linux-6.11, linux-6.10, and linux-6.8, and they all failed as I reported.
> > >
> > > I switched to Ubuntu 22.04.5 LTS (aarch64) to test with the older libraries.
> > > All tests available passed using the rdma for-next tree without any problem.
> > > Then, I applied my ODP patches onto it, and everything is still fine.
> > > ####################
> > > ubuntu@rdma-aarch64:~/rdma-core$ git branch -v
> > > * master fb965e2d0 Merge pull request #1531 from selvintxavier/pbuf_optimization
> > > ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py
> > > ..........ss..........ssssssssss..............ssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........ssssss..ss....s.sssssss....ss....ss..............s......................ss.............sss...ssss
> > > ----------------------------------------------------------------------
> > > Ran 321 tests in 3.599s
> > >
> > > OK (skipped=211)
> > > ubuntu@rdma-aarch64:~/rdma-core$ ./build/bin/run_tests.py -k odp
> > > sssssssss..ss....s.s
> > > ----------------------------------------------------------------------
> > > Ran 20 tests in 0.269s
> > >
> > > OK (skipped=13)
> > > ####################
> > >
> > > Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2.
> > > We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.
> >
> > That's distressing.
> >
> > > In conclusion, I believe there is nothing in my ODP patches that could cause
> > > the rxe driver to fail. I would appreciate any feedback on potential improvements.
> >
> > What am I supposed to do with this though?
> >
> > Joe, can you please answer Daisuke's questions on what problems you
> > observed and if you observe them without the ODP patches?
>
> Will make tests and let you know the result very soon.

Need time to complete the test scenario configurations.

We made tests again. Some memory errors are from RXE ODP.
The whole tests can not be complete with this patch set.
Without this patch set, all the tests can pass.

>
> >
> > Jason

Re: [PATCH for-next v9 0/5] On-Demand Paging on SoftRoCE

[Jason Gunthorpe]

On Mon, Jan 13, 2025 at 02:15:27PM +0100, Joe Klein wrote:

> > > > Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2.
> > > > We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.
> > >
> > > That's distressing.
> > >
> > > > In conclusion, I believe there is nothing in my ODP patches that could cause
> > > > the rxe driver to fail. I would appreciate any feedback on potential improvements.
> > >
> > > What am I supposed to do with this though?
> > >
> > > Joe, can you please answer Daisuke's questions on what problems you
> > > observed and if you observe them without the ODP patches?
> >
> > Will make tests and let you know the result very soon.
> 
> Need time to complete the test scenario configurations.
> 
> We made tests again. Some memory errors are from RXE ODP.
> The whole tests can not be complete with this patch set.
> Without this patch set, all the tests can pass.

Can you share your logs so that Daisuke can resolve whatever the
problem is?

Jason

RE: [PATCH for-next v9 0/5] On-Demand Paging on SoftRoCE

[Daisuke Matsuda (Fujitsu)]

On Tue, Jan 14, 2025 5:16 AM Jason Gunthorpe:
> On Mon, Jan 13, 2025 at 02:15:27PM +0100, Joe Klein wrote:
> 
> > > > > Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2.
> > > > > We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.
> > > >
> > > > That's distressing.

I am going to start bisecting the root cause to fix it.
It may take a while, so please stay patient.

> > > >
> > > > > In conclusion, I believe there is nothing in my ODP patches that could cause
> > > > > the rxe driver to fail. I would appreciate any feedback on potential improvements.
> > > >
> > > > What am I supposed to do with this though?
> > > >
> > > > Joe, can you please answer Daisuke's questions on what problems you
> > > > observed and if you observe them without the ODP patches?
> > >
> > > Will make tests and let you know the result very soon.
> >
> > Need time to complete the test scenario configurations.
> >
> > We made tests again. Some memory errors are from RXE ODP.
> > The whole tests can not be complete with this patch set.
> > Without this patch set, all the tests can pass.
> 
> Can you share your logs so that Daisuke can resolve whatever the
> problem is?

Since Joe does not provide any specific information, it would be nice
if someone familiar with rxe like Zhu Yanjun, Li Zhijian or Bob Pearson
could find the time to test and review the patch.
Cf. https://github.com/ddmatsu/linux/tree/odp_v9

However, to be honest, I do not think Joe's report is valid. It is not normal to
complain other's patches without providing any details. It makes difficult to 
assess the validity of the claim, and blocks any further discussions forever.

The testcases in libibverbs pass as I attached the log. Other than that, I verified
my patches work with perftest (with ODP enabled/disabled). I believe I did the
test properly and the patches are ready to be merged.

Daisuke

> 
> Jason

Re: [PATCH for-next v9 0/5] On-Demand Paging on SoftRoCE

[Jason Gunthorpe]

On Thu, Jan 30, 2025 at 10:51:58AM +0000, Daisuke Matsuda (Fujitsu) wrote:

> However, to be honest, I do not think Joe's report is valid. It is not normal to
> complain other's patches without providing any details. It makes difficult to 
> assess the validity of the claim, and blocks any further discussions forever.

Yes, I agree, let's move past anything without concrete actionable
details.

Jason

RE: [PATCH for-next v9 0/5] On-Demand Paging on SoftRoCE

[Daisuke Matsuda (Fujitsu)]

On Thu, Jan 30, 2025 7:52 PM Daisuke Matsuda (Fujitsu):
> On Tue, Jan 14, 2025 5:16 AM Jason Gunthorpe:
> > On Mon, Jan 13, 2025 at 02:15:27PM +0100, Joe Klein wrote:
> >
> > > > > > Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2.
> > > > > > We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.
> > > > >
> > > > > That's distressing.
> 
> I am going to start bisecting the root cause to fix it.
> It may take a while, so please stay patient.

On Ubuntu 22.04.5, both v50.0 branch and master branch pass the pyverbs testcases,
so it is not a regression of libibverbs. However, on Ubuntu 24.04.1, the test causes
segmentation fault with both branches. The issue looks specific to Ubuntu 24.04.

Could it be possible the update of python version leads to the failure? I saw strange
warnings while running build.sh, which did not appear on Ubuntu 22.04.5. I would like
to hear from those who have tried building rdma-core on Ubuntu 24.04.1 or with Python 3.12.3.

[422/607] Cythonizing /home/ubuntu/rdma-core/pyverbs/pd.pyx
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:102:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:124:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:142:30: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:158:61: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:180:46: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:272:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/pd.pyx:279:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/wr.pxd:12:36: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/wr.pxd:21:47: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:16:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:17:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:22:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:27:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cmid.pxd:32:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/mr.pxd:21:36: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/srq.pxd:28:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/srq.pxd:29:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/addr.pxd:22:50: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/addr.pxd:27:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:11:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:13:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:18:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:20:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:35:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/cq.pxd:37:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/qp.pxd:37:21: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/qp.pxd:38:36: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/qp.pxd:45:27: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/wq.pxd:27:28: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
warning: /home/ubuntu/rdma-core/pyverbs/wq.pxd:38:28: Implicit noexcept declaration is deprecated. Function declaration should contain 'noexcept' keyword.
[607/607] Linking C shared library python/pyverbs/providers/mlx5/mlx5dv.cpython-312-x86_64-linux-gnu.so

Daisuke

Re: [PATCH for-next v9 0/5] On-Demand Paging on SoftRoCE

[Jason Gunthorpe]

On Wed, Feb 19, 2025 at 10:48:25AM +0000, Daisuke Matsuda (Fujitsu) wrote:
> On Thu, Jan 30, 2025 7:52 PM Daisuke Matsuda (Fujitsu):
> > On Tue, Jan 14, 2025 5:16 AM Jason Gunthorpe:
> > > On Mon, Jan 13, 2025 at 02:15:27PM +0100, Joe Klein wrote:
> > >
> > > > > > > Possibly, there was a regression in libibverbs between v39.0-1 and v50.0-2build2.
> > > > > > > We need to take a closer look to resolve the malfunction of rxe on Ubuntu 24.04.
> > > > > >
> > > > > > That's distressing.
> > 
> > I am going to start bisecting the root cause to fix it.
> > It may take a while, so please stay patient.
> 
> On Ubuntu 22.04.5, both v50.0 branch and master branch pass the pyverbs testcases,
> so it is not a regression of libibverbs. However, on Ubuntu 24.04.1, the test causes
> segmentation fault with both branches. The issue looks specific to Ubuntu 24.04.
> 
> Could it be possible the update of python version leads to the
> failure? 

Or a cython update? It certainly could..

So maybe there is nothing to worry about kernel side if the test case
passes on older OS userspace

Jason

Re: [PATCH for-next v9 0/5] On-Demand Paging on SoftRoCE

[Jason Gunthorpe]

On Fri, Dec 20, 2024 at 07:09:31PM +0900, Daisuke Matsuda wrote:
> Daisuke Matsuda (5):
>   RDMA/rxe: Move some code to rxe_loc.h in preparation for ODP
>   RDMA/rxe: Add page invalidation support
>   RDMA/rxe: Allow registering MRs for On-Demand Paging
>   RDMA/rxe: Add support for Send/Recv/Write/Read with ODP
>   RDMA/rxe: Add support for the traditional Atomic operations with ODP
> 
>  drivers/infiniband/sw/rxe/Makefile    |   2 +
>  drivers/infiniband/sw/rxe/rxe.c       |  18 ++
>  drivers/infiniband/sw/rxe/rxe.h       |  37 ---
>  drivers/infiniband/sw/rxe/rxe_loc.h   |  34 +++
>  drivers/infiniband/sw/rxe/rxe_mr.c    |  13 +-
>  drivers/infiniband/sw/rxe/rxe_odp.c   | 324 ++++++++++++++++++++++++++
>  drivers/infiniband/sw/rxe/rxe_resp.c  |  18 +-
>  drivers/infiniband/sw/rxe/rxe_verbs.c |   5 +-
>  drivers/infiniband/sw/rxe/rxe_verbs.h |  37 +++
>  9 files changed, 443 insertions(+), 45 deletions(-)
>  create mode 100644 drivers/infiniband/sw/rxe/rxe_odp.c

Applied to for-next, thanks

Jason