* [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP
@ 2025-03-24 7:56 Daisuke Matsuda
2025-03-24 7:56 ` [PATCH for-next v3 1/2] RDMA/rxe: Enable ODP in RDMA FLUSH operation Daisuke Matsuda
` (3 more replies)
0 siblings, 4 replies; 13+ messages in thread
From: Daisuke Matsuda @ 2025-03-24 7:56 UTC (permalink / raw)
To: linux-rdma, leon, jgg, zyjzyj2000; +Cc: lizhijian, Daisuke Matsuda
RDMA FLUSH[1] and ATOMIC WRITE[2] were added to rxe, but they cannot run
in the ODP mode as of now. This series is for the kernel-side enablement.
There are also minor changes in libibverbs and pyverbs. The rdma-core tests
are also added so that people can test the features.
PR: https://github.com/linux-rdma/rdma-core/pull/1580
You can try the patches with the tree below:
https://github.com/ddmatsu/linux/tree/odp-extension3
Note that the tree is a bit old (6.13-rc1), because there was an issue[3]
in the for-next tree that disabled ibv_query_device_ex(), which is used to
query ODP capabilities. However, there is already a fix[4], and it is to be
resolved in the next release. I will update the tree once it is ready.
[1] [for-next PATCH 00/10] RDMA/rxe: Add RDMA FLUSH operation
https://lore.kernel.org/lkml/20221206130201.30986-1-lizhijian@fujitsu.com/
[2] [PATCH v7 0/8] RDMA/rxe: Add atomic write operation
https://lore.kernel.org/linux-rdma/1669905432-14-1-git-send-email-yangx.jy@fujitsu.com/
[3] [bug report] RDMA/rxe: Failure of ibv_query_device() and ibv_query_device_ex() tests in rdma-core
https://lore.kernel.org/all/1b9d6286-62fc-4b42-b304-0054c4ebee02@linux.dev/T/
[4] [PATCH rdma-rc 1/1] RDMA/rxe: Fix the failure of ibv_query_device() and ibv_query_device_ex() tests
https://lore.kernel.org/linux-rdma/174102882930.42565.11864314726635251412.b4-ty@kernel.org/T/#t
History:
v1->v2: Removed some code duplications
v2->v3: Addressed comments from Leon Romanovsky
cf. https://lore.kernel.org/linux-rdma/20250319085825.GH1322339@unreal/
Daisuke Matsuda (2):
RDMA/rxe: Enable ODP in RDMA FLUSH operation
RDMA/rxe: Enable ODP in ATOMIC WRITE operation
drivers/infiniband/sw/rxe/rxe.c | 2 +
drivers/infiniband/sw/rxe/rxe_loc.h | 18 ++++-
drivers/infiniband/sw/rxe/rxe_mr.c | 48 ++++++------
drivers/infiniband/sw/rxe/rxe_odp.c | 108 ++++++++++++++++++++++++++-
drivers/infiniband/sw/rxe/rxe_resp.c | 15 ++--
include/rdma/ib_verbs.h | 2 +
6 files changed, 159 insertions(+), 34 deletions(-)
--
2.43.0
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH for-next v3 1/2] RDMA/rxe: Enable ODP in RDMA FLUSH operation 2025-03-24 7:56 [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP Daisuke Matsuda @ 2025-03-24 7:56 ` Daisuke Matsuda 2025-03-24 7:56 ` [PATCH for-next v3 2/2] RDMA/rxe: Enable ODP in ATOMIC WRITE operation Daisuke Matsuda ` (2 subsequent siblings) 3 siblings, 0 replies; 13+ messages in thread From: Daisuke Matsuda @ 2025-03-24 7:56 UTC (permalink / raw) To: linux-rdma, leon, jgg, zyjzyj2000; +Cc: lizhijian, Daisuke Matsuda For persistent memories, add rxe_odp_flush_pmem_iova() so that ODP specific steps are executed. Otherwise, no additional consideration is required. Signed-off-by: Daisuke Matsuda <matsuda-daisuke@fujitsu.com> Reviewed-by: Li Zhijian <lizhijian@fujitsu.com> --- drivers/infiniband/sw/rxe/rxe.c | 1 + drivers/infiniband/sw/rxe/rxe_loc.h | 7 ++++ drivers/infiniband/sw/rxe/rxe_mr.c | 36 ++++++++++------ drivers/infiniband/sw/rxe/rxe_odp.c | 62 ++++++++++++++++++++++++++-- drivers/infiniband/sw/rxe/rxe_resp.c | 4 -- include/rdma/ib_verbs.h | 1 + 6 files changed, 91 insertions(+), 20 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index 4e56a371deb5..df66f8f9efa1 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -109,6 +109,7 @@ static void rxe_init_device_param(struct rxe_dev *rxe) rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_READ; rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_ATOMIC; rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_SRQ_RECV; + rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_FLUSH; } } diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h index feb386d98d1d..0012bebe96ef 100644 --- a/drivers/infiniband/sw/rxe/rxe_loc.h +++ b/drivers/infiniband/sw/rxe/rxe_loc.h @@ -194,6 +194,8 @@ int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length, enum rxe_mr_copy_dir dir); int rxe_odp_atomic_op(struct rxe_mr *mr, u64 iova, int opcode, u64 compare, u64 swap_add, u64 *orig_val); +int rxe_odp_flush_pmem_iova(struct rxe_mr *mr, u64 iova, + unsigned int length); #else /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */ static inline int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova, @@ -212,6 +214,11 @@ rxe_odp_atomic_op(struct rxe_mr *mr, u64 iova, int opcode, { return RESPST_ERR_UNSUPPORTED_OPCODE; } +static inline int rxe_odp_flush_pmem_iova(struct rxe_mr *mr, u64 iova, + unsigned int length) +{ + return -EOPNOTSUPP; +} #endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */ #endif /* RXE_LOC_H */ diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index 868d2f0b74e9..93e4b5acd3ac 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -424,7 +424,7 @@ int copy_data( return err; } -int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length) +static int rxe_mr_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length) { unsigned int page_offset; unsigned long index; @@ -433,16 +433,6 @@ int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length) int err; u8 *va; - /* mr must be valid even if length is zero */ - if (WARN_ON(!mr)) - return -EINVAL; - - if (length == 0) - return 0; - - if (mr->ibmr.type == IB_MR_TYPE_DMA) - return -EFAULT; - err = mr_check_range(mr, iova, length); if (err) return err; @@ -454,7 +444,7 @@ int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length) if (!page) return -EFAULT; bytes = min_t(unsigned int, length, - mr_page_size(mr) - page_offset); + mr_page_size(mr) - page_offset); va = kmap_local_page(page); arch_wb_cache_pmem(va + page_offset, bytes); @@ -468,6 +458,28 @@ int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length) return 0; } +int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 start, unsigned int length) +{ + int err; + + /* mr must be valid even if length is zero */ + if (WARN_ON(!mr)) + return -EINVAL; + + if (length == 0) + return 0; + + if (mr->ibmr.type == IB_MR_TYPE_DMA) + return -EFAULT; + + if (mr->umem->is_odp) + err = rxe_odp_flush_pmem_iova(mr, start, length); + else + err = rxe_mr_flush_pmem_iova(mr, start, length); + + return err; +} + /* Guarantee atomicity of atomic operations at the machine level. */ DEFINE_SPINLOCK(atomic_ops_lock); diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c index 9f6e2bb2a269..9a9aae967486 100644 --- a/drivers/infiniband/sw/rxe/rxe_odp.c +++ b/drivers/infiniband/sw/rxe/rxe_odp.c @@ -4,6 +4,7 @@ */ #include <linux/hmm.h> +#include <linux/libnvdimm.h> #include <rdma/ib_umem_odp.h> @@ -147,6 +148,16 @@ static inline bool rxe_check_pagefault(struct ib_umem_odp *umem_odp, return need_fault; } +static unsigned long rxe_odp_iova_to_index(struct ib_umem_odp *umem_odp, u64 iova) +{ + return (iova - ib_umem_start(umem_odp)) >> umem_odp->page_shift; +} + +static unsigned long rxe_odp_iova_to_page_offset(struct ib_umem_odp *umem_odp, u64 iova) +{ + return iova & (BIT(umem_odp->page_shift) - 1); +} + static int rxe_odp_map_range_and_lock(struct rxe_mr *mr, u64 iova, int length, u32 flags) { struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem); @@ -190,8 +201,8 @@ static int __rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, size_t offset; u8 *user_va; - idx = (iova - ib_umem_start(umem_odp)) >> umem_odp->page_shift; - offset = iova & (BIT(umem_odp->page_shift) - 1); + idx = rxe_odp_iova_to_index(umem_odp, iova); + offset = rxe_odp_iova_to_page_offset(umem_odp, iova); while (length > 0) { u8 *src, *dest; @@ -277,8 +288,8 @@ static int rxe_odp_do_atomic_op(struct rxe_mr *mr, u64 iova, int opcode, return RESPST_ERR_RKEY_VIOLATION; } - idx = (iova - ib_umem_start(umem_odp)) >> umem_odp->page_shift; - page_offset = iova & (BIT(umem_odp->page_shift) - 1); + idx = rxe_odp_iova_to_index(umem_odp, iova); + page_offset = rxe_odp_iova_to_page_offset(umem_odp, iova); page = hmm_pfn_to_page(umem_odp->pfn_list[idx]); if (!page) return RESPST_ERR_RKEY_VIOLATION; @@ -324,3 +335,46 @@ int rxe_odp_atomic_op(struct rxe_mr *mr, u64 iova, int opcode, return err; } + +int rxe_odp_flush_pmem_iova(struct rxe_mr *mr, u64 iova, + unsigned int length) +{ + struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem); + unsigned int page_offset; + unsigned long index; + struct page *page; + unsigned int bytes; + int err; + u8 *va; + + err = rxe_odp_map_range_and_lock(mr, iova, length, + RXE_PAGEFAULT_DEFAULT); + if (err) + return err; + + while (length > 0) { + index = rxe_odp_iova_to_index(umem_odp, iova); + page_offset = rxe_odp_iova_to_page_offset(umem_odp, iova); + + page = hmm_pfn_to_page(umem_odp->pfn_list[index]); + if (!page) { + mutex_unlock(&umem_odp->umem_mutex); + return -EFAULT; + } + + bytes = min_t(unsigned int, length, + mr_page_size(mr) - page_offset); + + va = kmap_local_page(page); + arch_wb_cache_pmem(va + page_offset, bytes); + kunmap_local(va); + + length -= bytes; + iova += bytes; + page_offset = 0; + } + + mutex_unlock(&umem_odp->umem_mutex); + + return 0; +} diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index 54ba9ee1acc5..304e3de740ad 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -649,10 +649,6 @@ static enum resp_states process_flush(struct rxe_qp *qp, struct rxe_mr *mr = qp->resp.mr; struct resp_res *res = qp->resp.res; - /* ODP is not supported right now. WIP. */ - if (mr->umem->is_odp) - return RESPST_ERR_UNSUPPORTED_OPCODE; - /* oA19-14, oA19-15 */ if (res && res->replay) return RESPST_ACKNOWLEDGE; diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index d42eae69d9a8..41bf98ccb275 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -325,6 +325,7 @@ enum ib_odp_transport_cap_bits { IB_ODP_SUPPORT_READ = 1 << 3, IB_ODP_SUPPORT_ATOMIC = 1 << 4, IB_ODP_SUPPORT_SRQ_RECV = 1 << 5, + IB_ODP_SUPPORT_FLUSH = 1 << 6, }; struct ib_odp_caps { -- 2.43.0 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH for-next v3 2/2] RDMA/rxe: Enable ODP in ATOMIC WRITE operation 2025-03-24 7:56 [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP Daisuke Matsuda 2025-03-24 7:56 ` [PATCH for-next v3 1/2] RDMA/rxe: Enable ODP in RDMA FLUSH operation Daisuke Matsuda @ 2025-03-24 7:56 ` Daisuke Matsuda 2025-04-08 11:11 ` [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP Leon Romanovsky 2025-04-08 11:11 ` Leon Romanovsky 3 siblings, 0 replies; 13+ messages in thread From: Daisuke Matsuda @ 2025-03-24 7:56 UTC (permalink / raw) To: linux-rdma, leon, jgg, zyjzyj2000; +Cc: lizhijian, Daisuke Matsuda Add rxe_odp_do_atomic_write() so that ODP specific steps are applied to ATOMIC WRITE requests. Signed-off-by: Daisuke Matsuda <matsuda-daisuke@fujitsu.com> --- drivers/infiniband/sw/rxe/rxe.c | 1 + drivers/infiniband/sw/rxe/rxe_loc.h | 11 ++++++- drivers/infiniband/sw/rxe/rxe_mr.c | 12 -------- drivers/infiniband/sw/rxe/rxe_odp.c | 46 ++++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_resp.c | 11 ++++++- include/rdma/ib_verbs.h | 1 + 6 files changed, 68 insertions(+), 14 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index df66f8f9efa1..21ce2d876b42 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -110,6 +110,7 @@ static void rxe_init_device_param(struct rxe_dev *rxe) rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_ATOMIC; rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_SRQ_RECV; rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_FLUSH; + rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_ATOMIC_WRITE; } } diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h index 0012bebe96ef..d27bd05ed21f 100644 --- a/drivers/infiniband/sw/rxe/rxe_loc.h +++ b/drivers/infiniband/sw/rxe/rxe_loc.h @@ -187,7 +187,7 @@ static inline unsigned int wr_opcode_mask(int opcode, struct rxe_qp *qp) /* rxe_odp.c */ extern const struct mmu_interval_notifier_ops rxe_mn_ops; -#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING +#if defined CONFIG_INFINIBAND_ON_DEMAND_PAGING int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova, int access_flags, struct rxe_mr *mr); int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length, @@ -221,4 +221,13 @@ static inline int rxe_odp_flush_pmem_iova(struct rxe_mr *mr, u64 iova, } #endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */ +#if defined CONFIG_INFINIBAND_ON_DEMAND_PAGING && defined CONFIG_64BIT +enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value); +#else /* CONFIG_INFINIBAND_ON_DEMAND_PAGING && CONFIG_64BIT */ +static inline enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) +{ + return RESPST_ERR_UNSUPPORTED_OPCODE; +} +#endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING && CONFIG_64BIT */ + #endif /* RXE_LOC_H */ diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index 93e4b5acd3ac..d40fbe10633f 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -547,16 +547,6 @@ int rxe_mr_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) struct page *page; u64 *va; - /* ODP is not supported right now. WIP. */ - if (mr->umem->is_odp) - return RESPST_ERR_UNSUPPORTED_OPCODE; - - /* See IBA oA19-28 */ - if (unlikely(mr->state != RXE_MR_STATE_VALID)) { - rxe_dbg_mr(mr, "mr not in valid state\n"); - return RESPST_ERR_RKEY_VIOLATION; - } - if (mr->ibmr.type == IB_MR_TYPE_DMA) { page_offset = iova & (PAGE_SIZE - 1); page = ib_virt_dma_to_page(iova); @@ -584,10 +574,8 @@ int rxe_mr_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) } va = kmap_local_page(page); - /* Do atomic write after all prior operations have completed */ smp_store_release(&va[page_offset >> 3], value); - kunmap_local(va); return 0; diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c index 9a9aae967486..4bd7284e066a 100644 --- a/drivers/infiniband/sw/rxe/rxe_odp.c +++ b/drivers/infiniband/sw/rxe/rxe_odp.c @@ -378,3 +378,49 @@ int rxe_odp_flush_pmem_iova(struct rxe_mr *mr, u64 iova, return 0; } + +/* CONFIG_64BIT=y */ +enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) +{ + struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem); + unsigned int page_offset; + unsigned long index; + struct page *page; + int err; + u64 *va; + + /* See IBA oA19-28 */ + err = mr_check_range(mr, iova, sizeof(value)); + if (unlikely(err)) { + rxe_dbg_mr(mr, "iova out of range\n"); + return RESPST_ERR_RKEY_VIOLATION; + } + + err = rxe_odp_map_range_and_lock(mr, iova, sizeof(value), + RXE_PAGEFAULT_DEFAULT); + if (err) + return RESPST_ERR_RKEY_VIOLATION; + + page_offset = rxe_odp_iova_to_page_offset(umem_odp, iova); + index = rxe_odp_iova_to_index(umem_odp, iova); + page = hmm_pfn_to_page(umem_odp->pfn_list[index]); + if (!page) { + mutex_unlock(&umem_odp->umem_mutex); + return RESPST_ERR_RKEY_VIOLATION; + } + /* See IBA A19.4.2 */ + if (unlikely(page_offset & 0x7)) { + mutex_unlock(&umem_odp->umem_mutex); + rxe_dbg_mr(mr, "misaligned address\n"); + return RESPST_ERR_MISALIGNED_ATOMIC; + } + + va = kmap_local_page(page); + /* Do atomic write after all prior operations have completed */ + smp_store_release(&va[page_offset >> 3], value); + kunmap_local(va); + + mutex_unlock(&umem_odp->umem_mutex); + + return RESPST_NONE; +} diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index 304e3de740ad..fd7bac5bce18 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -749,7 +749,16 @@ static enum resp_states atomic_write_reply(struct rxe_qp *qp, value = *(u64 *)payload_addr(pkt); iova = qp->resp.va + qp->resp.offset; - err = rxe_mr_do_atomic_write(mr, iova, value); + /* See IBA oA19-28 */ + if (unlikely(mr->state != RXE_MR_STATE_VALID)) { + rxe_dbg_mr(mr, "mr not in valid state\n"); + return RESPST_ERR_RKEY_VIOLATION; + } + + if (mr->umem->is_odp) + err = rxe_odp_do_atomic_write(mr, iova, value); + else + err = rxe_mr_do_atomic_write(mr, iova, value); if (err) return err; diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 41bf98ccb275..0a7ccd08b365 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -326,6 +326,7 @@ enum ib_odp_transport_cap_bits { IB_ODP_SUPPORT_ATOMIC = 1 << 4, IB_ODP_SUPPORT_SRQ_RECV = 1 << 5, IB_ODP_SUPPORT_FLUSH = 1 << 6, + IB_ODP_SUPPORT_ATOMIC_WRITE = 1 << 7, }; struct ib_odp_caps { -- 2.43.0 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP 2025-03-24 7:56 [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP Daisuke Matsuda 2025-03-24 7:56 ` [PATCH for-next v3 1/2] RDMA/rxe: Enable ODP in RDMA FLUSH operation Daisuke Matsuda 2025-03-24 7:56 ` [PATCH for-next v3 2/2] RDMA/rxe: Enable ODP in ATOMIC WRITE operation Daisuke Matsuda @ 2025-04-08 11:11 ` Leon Romanovsky 2025-04-08 11:11 ` Leon Romanovsky 3 siblings, 0 replies; 13+ messages in thread From: Leon Romanovsky @ 2025-04-08 11:11 UTC (permalink / raw) To: Daisuke Matsuda; +Cc: linux-rdma, jgg, zyjzyj2000, lizhijian On Mon, Mar 24, 2025 at 04:56:47PM +0900, Daisuke Matsuda wrote: > RDMA FLUSH[1] and ATOMIC WRITE[2] were added to rxe, but they cannot run > in the ODP mode as of now. This series is for the kernel-side enablement. <...> > Daisuke Matsuda (2): > RDMA/rxe: Enable ODP in RDMA FLUSH operation > RDMA/rxe: Enable ODP in ATOMIC WRITE operation > > drivers/infiniband/sw/rxe/rxe.c | 2 + > drivers/infiniband/sw/rxe/rxe_loc.h | 18 ++++- > drivers/infiniband/sw/rxe/rxe_mr.c | 48 ++++++------ > drivers/infiniband/sw/rxe/rxe_odp.c | 108 ++++++++++++++++++++++++++- > drivers/infiniband/sw/rxe/rxe_resp.c | 15 ++-- > include/rdma/ib_verbs.h | 2 + I applied the series, but not sure that extending cap_bits for RXE only is justified. --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -325,6 +325,7 @@ enum ib_odp_transport_cap_bits { IB_ODP_SUPPORT_READ = 1 << 3, IB_ODP_SUPPORT_ATOMIC = 1 << 4, IB_ODP_SUPPORT_SRQ_RECV = 1 << 5, + IB_ODP_SUPPORT_FLUSH = 1 << 6, }; Thanks > 6 files changed, 159 insertions(+), 34 deletions(-) > > -- > 2.43.0 > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP 2025-03-24 7:56 [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP Daisuke Matsuda ` (2 preceding siblings ...) 2025-04-08 11:11 ` [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP Leon Romanovsky @ 2025-04-08 11:11 ` Leon Romanovsky 2025-04-11 8:13 ` Daisuke Matsuda (Fujitsu) 3 siblings, 1 reply; 13+ messages in thread From: Leon Romanovsky @ 2025-04-08 11:11 UTC (permalink / raw) To: linux-rdma, jgg, zyjzyj2000, Daisuke Matsuda; +Cc: lizhijian On Mon, 24 Mar 2025 16:56:47 +0900, Daisuke Matsuda wrote: > RDMA FLUSH[1] and ATOMIC WRITE[2] were added to rxe, but they cannot run > in the ODP mode as of now. This series is for the kernel-side enablement. > > There are also minor changes in libibverbs and pyverbs. The rdma-core tests > are also added so that people can test the features. > PR: https://github.com/linux-rdma/rdma-core/pull/1580 > > [...] Applied, thanks! [1/2] RDMA/rxe: Enable ODP in RDMA FLUSH operation https://git.kernel.org/rdma/rdma/c/32cad6aab9a699 [2/2] RDMA/rxe: Enable ODP in ATOMIC WRITE operation https://git.kernel.org/rdma/rdma/c/3e2746e0863f48 Best regards, -- Leon Romanovsky <leon@kernel.org> ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP 2025-04-08 11:11 ` Leon Romanovsky @ 2025-04-11 8:13 ` Daisuke Matsuda (Fujitsu) 2025-04-11 17:55 ` Leon Romanovsky 0 siblings, 1 reply; 13+ messages in thread From: Daisuke Matsuda (Fujitsu) @ 2025-04-11 8:13 UTC (permalink / raw) To: 'Leon Romanovsky', linux-rdma@vger.kernel.org Hi Leon, I have noticed the 2nd patch caused "kernel test robot" error, and you kindly amended the patch. However, another error has been detected by "the bot" because of the remaining fundamental problem that ATOMIC WRITE cannot be executed on non-64-bit architectures (at least on rxe). I think applying the change below to the original patch(*1) will resolve the issue. (*1) https://lore.kernel.org/linux-rdma/20250324075649.3313968-3-matsuda-daisuke@fujitsu.com/ ``` diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c index 02de05d759c6..ac3b3039db22 100644 --- a/drivers/infiniband/sw/rxe/rxe_odp.c +++ b/drivers/infiniband/sw/rxe/rxe_odp.c @@ -380,6 +380,7 @@ int rxe_odp_flush_pmem_iova(struct rxe_mr *mr, u64 iova, } /* CONFIG_64BIT=y */ +#ifdef CONFIG_64BIT enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) { struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem); @@ -424,3 +425,4 @@ enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) return RESPST_NONE; } +#endif ``` The definition of rxe_odp_do_atomic_write() should have been guarded in #ifdef CONFIG_64BIT. I believe this fix can address both: - the first error "error: redefinition of 'rxe_odp_do_atomic_write' " that could be caused when CONFIG_INFINIBAND_ON_DEMAND_PAGING=y && CONFIG_64BIT=n. - the second error caused by trying to compile 64-bit codes on 32-bit architectures. I am very sorry to bother you, but is it possible to make the modification? If I should provide a replacement patch, I will do so. Thanks, Daisuke On Tue, April 8, 2025 8:12 PM Leon Romanovsky wrote: > On Mon, 24 Mar 2025 16:56:47 +0900, Daisuke Matsuda wrote: > > RDMA FLUSH[1] and ATOMIC WRITE[2] were added to rxe, but they cannot run > > in the ODP mode as of now. This series is for the kernel-side enablement. > > > > There are also minor changes in libibverbs and pyverbs. The rdma-core tests > > are also added so that people can test the features. > > PR: https://github.com/linux-rdma/rdma-core/pull/1580 > > > > [...] > > Applied, thanks! > > [1/2] RDMA/rxe: Enable ODP in RDMA FLUSH operation > https://git.kernel.org/rdma/rdma/c/32cad6aab9a699 > [2/2] RDMA/rxe: Enable ODP in ATOMIC WRITE operation > https://git.kernel.org/rdma/rdma/c/3e2746e0863f48 > > Best regards, > -- > Leon Romanovsky <leon@kernel.org> ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP 2025-04-11 8:13 ` Daisuke Matsuda (Fujitsu) @ 2025-04-11 17:55 ` Leon Romanovsky 2025-04-14 10:16 ` Daisuke Matsuda (Fujitsu) 0 siblings, 1 reply; 13+ messages in thread From: Leon Romanovsky @ 2025-04-11 17:55 UTC (permalink / raw) To: Daisuke Matsuda (Fujitsu); +Cc: linux-rdma@vger.kernel.org On Fri, Apr 11, 2025 at 08:13:59AM +0000, Daisuke Matsuda (Fujitsu) wrote: > Hi Leon, > > I have noticed the 2nd patch caused "kernel test robot" error, and you > kindly amended the patch. However, another error has been detected by "the bot" > because of the remaining fundamental problem that ATOMIC WRITE cannot > be executed on non-64-bit architectures (at least on rxe). > > I think applying the change below to the original patch(*1) will resolve the issue. > (*1) https://lore.kernel.org/linux-rdma/20250324075649.3313968-3-matsuda-daisuke@fujitsu.com/ > ``` > diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c > index 02de05d759c6..ac3b3039db22 100644 > --- a/drivers/infiniband/sw/rxe/rxe_odp.c > +++ b/drivers/infiniband/sw/rxe/rxe_odp.c > @@ -380,6 +380,7 @@ int rxe_odp_flush_pmem_iova(struct rxe_mr *mr, u64 iova, > } > > /* CONFIG_64BIT=y */ > +#ifdef CONFIG_64BIT > enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) > { > struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem); > @@ -424,3 +425,4 @@ enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) > > return RESPST_NONE; > } > +#endif > ``` > The definition of rxe_odp_do_atomic_write() should have been guarded in #ifdef CONFIG_64BIT. > I believe this fix can address both: > - the first error "error: redefinition of 'rxe_odp_do_atomic_write' " that could be caused when > CONFIG_INFINIBAND_ON_DEMAND_PAGING=y && CONFIG_64BIT=n. > - the second error caused by trying to compile 64-bit codes on 32-bit architectures. > > I am very sorry to bother you, but is it possible to make the modification? > If I should provide a replacement patch, I will do so. I think that better will be simply make sure that RXE is dependent on 64bits. diff --git a/drivers/infiniband/sw/rxe/Kconfig b/drivers/infiniband/sw/rxe/Kconfig index c180e7ebcfc5..1ed5b63f8afc 100644 --- a/drivers/infiniband/sw/rxe/Kconfig +++ b/drivers/infiniband/sw/rxe/Kconfig @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only config RDMA_RXE tristate "Software RDMA over Ethernet (RoCE) driver" - depends on INET && PCI && INFINIBAND + depends on INET && PCI && INFINIBAND && 64BIT depends on INFINIBAND_VIRT_DMA select NET_UDP_TUNNEL select CRC32 WDYT? Thanks > > Thanks, > Daisuke > > On Tue, April 8, 2025 8:12 PM Leon Romanovsky wrote: > > On Mon, 24 Mar 2025 16:56:47 +0900, Daisuke Matsuda wrote: > > > RDMA FLUSH[1] and ATOMIC WRITE[2] were added to rxe, but they cannot run > > > in the ODP mode as of now. This series is for the kernel-side enablement. > > > > > > There are also minor changes in libibverbs and pyverbs. The rdma-core tests > > > are also added so that people can test the features. > > > PR: https://github.com/linux-rdma/rdma-core/pull/1580 > > > > > > [...] > > > > Applied, thanks! > > > > [1/2] RDMA/rxe: Enable ODP in RDMA FLUSH operation > > https://git.kernel.org/rdma/rdma/c/32cad6aab9a699 > > [2/2] RDMA/rxe: Enable ODP in ATOMIC WRITE operation > > https://git.kernel.org/rdma/rdma/c/3e2746e0863f48 > > > > Best regards, > > -- > > Leon Romanovsky <leon@kernel.org> > ^ permalink raw reply related [flat|nested] 13+ messages in thread
* RE: [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP 2025-04-11 17:55 ` Leon Romanovsky @ 2025-04-14 10:16 ` Daisuke Matsuda (Fujitsu) 2025-04-14 12:56 ` Zhu Yanjun 0 siblings, 1 reply; 13+ messages in thread From: Daisuke Matsuda (Fujitsu) @ 2025-04-14 10:16 UTC (permalink / raw) To: 'Leon Romanovsky'; +Cc: linux-rdma@vger.kernel.org On Sat, April 12, 2025 2:55 AM Leon Romanovsky wrote: > > Hi Leon, > > > > I have noticed the 2nd patch caused "kernel test robot" error, and you > > kindly amended the patch. However, another error has been detected by "the bot" > > because of the remaining fundamental problem that ATOMIC WRITE cannot > > be executed on non-64-bit architectures (at least on rxe). > > > > I think applying the change below to the original patch(*1) will resolve the issue. > > (*1) https://lore.kernel.org/linux-rdma/20250324075649.3313968-3-matsuda-daisuke@fujitsu.com/ > > ``` > > diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c > > index 02de05d759c6..ac3b3039db22 100644 > > --- a/drivers/infiniband/sw/rxe/rxe_odp.c > > +++ b/drivers/infiniband/sw/rxe/rxe_odp.c > > @@ -380,6 +380,7 @@ int rxe_odp_flush_pmem_iova(struct rxe_mr *mr, u64 iova, > > } > > > > /* CONFIG_64BIT=y */ > > +#ifdef CONFIG_64BIT > > enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) > > { > > struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem); > > @@ -424,3 +425,4 @@ enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) > > > > return RESPST_NONE; > > } > > +#endif > > ``` > > The definition of rxe_odp_do_atomic_write() should have been guarded in #ifdef CONFIG_64BIT. > > I believe this fix can address both: > > - the first error "error: redefinition of 'rxe_odp_do_atomic_write' " that could be caused when > > CONFIG_INFINIBAND_ON_DEMAND_PAGING=y && CONFIG_64BIT=n. > > - the second error caused by trying to compile 64-bit codes on 32-bit architectures. > > > > I am very sorry to bother you, but is it possible to make the modification? > > If I should provide a replacement patch, I will do so. > > I think that better will be simply make sure that RXE is dependent on 64bits. > > diff --git a/drivers/infiniband/sw/rxe/Kconfig b/drivers/infiniband/sw/rxe/Kconfig > index c180e7ebcfc5..1ed5b63f8afc 100644 > --- a/drivers/infiniband/sw/rxe/Kconfig > +++ b/drivers/infiniband/sw/rxe/Kconfig > @@ -1,7 +1,7 @@ > # SPDX-License-Identifier: GPL-2.0-only > config RDMA_RXE > tristate "Software RDMA over Ethernet (RoCE) driver" > - depends on INET && PCI && INFINIBAND > + depends on INET && PCI && INFINIBAND && 64BIT > depends on INFINIBAND_VIRT_DMA > select NET_UDP_TUNNEL > select CRC32 > > WDYT? It seems the driver is designed to be runnable on 32-bit nodes, so it may be overkill to disable 32-bit mode only for "ATOMIC WRITE" functionality. However, I do not have strong objection to making this change if you think it is better in terms of maintainability. Before making the change, I'd like to get an ACK or NACK from Zhu Yanjun. As far as I am aware, no one is actively maintaining or testing RXE on 32-bit, so it may be acceptable to drop 32-bit support, but it's best to confirm before proceeding. Thanks, Daisuke > > Thanks > > > > > Thanks, > > Daisuke > > > > On Tue, April 8, 2025 8:12 PM Leon Romanovsky wrote: > > > On Mon, 24 Mar 2025 16:56:47 +0900, Daisuke Matsuda wrote: > > > > RDMA FLUSH[1] and ATOMIC WRITE[2] were added to rxe, but they cannot run > > > > in the ODP mode as of now. This series is for the kernel-side enablement. > > > > > > > > There are also minor changes in libibverbs and pyverbs. The rdma-core tests > > > > are also added so that people can test the features. > > > > PR: https://github.com/linux-rdma/rdma-core/pull/1580 > > > > > > > > [...] > > > > > > Applied, thanks! > > > > > > [1/2] RDMA/rxe: Enable ODP in RDMA FLUSH operation > > > https://git.kernel.org/rdma/rdma/c/32cad6aab9a699 > > > [2/2] RDMA/rxe: Enable ODP in ATOMIC WRITE operation > > > https://git.kernel.org/rdma/rdma/c/3e2746e0863f48 > > > > > > Best regards, > > > -- > > > Leon Romanovsky <leon@kernel.org> > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP 2025-04-14 10:16 ` Daisuke Matsuda (Fujitsu) @ 2025-04-14 12:56 ` Zhu Yanjun 2025-04-16 16:58 ` Leon Romanovsky 0 siblings, 1 reply; 13+ messages in thread From: Zhu Yanjun @ 2025-04-14 12:56 UTC (permalink / raw) To: Daisuke Matsuda (Fujitsu), 'Leon Romanovsky' Cc: linux-rdma@vger.kernel.org On 14.04.25 12:16, Daisuke Matsuda (Fujitsu) wrote: > On Sat, April 12, 2025 2:55 AM Leon Romanovsky wrote: >>> Hi Leon, >>> >>> I have noticed the 2nd patch caused "kernel test robot" error, and you >>> kindly amended the patch. However, another error has been detected by "the bot" >>> because of the remaining fundamental problem that ATOMIC WRITE cannot >>> be executed on non-64-bit architectures (at least on rxe). >>> >>> I think applying the change below to the original patch(*1) will resolve the issue. >>> (*1) https://lore.kernel.org/linux-rdma/20250324075649.3313968-3-matsuda-daisuke@fujitsu.com/ >>> ``` >>> diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c >>> index 02de05d759c6..ac3b3039db22 100644 >>> --- a/drivers/infiniband/sw/rxe/rxe_odp.c >>> +++ b/drivers/infiniband/sw/rxe/rxe_odp.c >>> @@ -380,6 +380,7 @@ int rxe_odp_flush_pmem_iova(struct rxe_mr *mr, u64 iova, >>> } >>> >>> /* CONFIG_64BIT=y */ >>> +#ifdef CONFIG_64BIT >>> enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) >>> { >>> struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem); >>> @@ -424,3 +425,4 @@ enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) >>> >>> return RESPST_NONE; >>> } >>> +#endif >>> ``` >>> The definition of rxe_odp_do_atomic_write() should have been guarded in #ifdef CONFIG_64BIT. >>> I believe this fix can address both: >>> - the first error "error: redefinition of 'rxe_odp_do_atomic_write' " that could be caused when >>> CONFIG_INFINIBAND_ON_DEMAND_PAGING=y && CONFIG_64BIT=n. >>> - the second error caused by trying to compile 64-bit codes on 32-bit architectures. >>> >>> I am very sorry to bother you, but is it possible to make the modification? >>> If I should provide a replacement patch, I will do so. >> >> I think that better will be simply make sure that RXE is dependent on 64bits. >> >> diff --git a/drivers/infiniband/sw/rxe/Kconfig b/drivers/infiniband/sw/rxe/Kconfig >> index c180e7ebcfc5..1ed5b63f8afc 100644 >> --- a/drivers/infiniband/sw/rxe/Kconfig >> +++ b/drivers/infiniband/sw/rxe/Kconfig >> @@ -1,7 +1,7 @@ >> # SPDX-License-Identifier: GPL-2.0-only >> config RDMA_RXE >> tristate "Software RDMA over Ethernet (RoCE) driver" >> - depends on INET && PCI && INFINIBAND >> + depends on INET && PCI && INFINIBAND && 64BIT >> depends on INFINIBAND_VIRT_DMA >> select NET_UDP_TUNNEL >> select CRC32 >> >> WDYT? > > It seems the driver is designed to be runnable on 32-bit nodes, so it may be > overkill to disable 32-bit mode only for "ATOMIC WRITE" functionality. > However, I do not have strong objection to making this change if you > think it is better in terms of maintainability. > > Before making the change, I'd like to get an ACK or NACK from Zhu Yanjun. > As far as I am aware, no one is actively maintaining or testing RXE on 32-bit, > so it may be acceptable to drop 32-bit support, but it's best to confirm before proceeding. Hi, Daisuke Matsuda Thanks a lot for your efforts. There are some problems with 32-bit architectures, such as Year 2038 problem ( many 32-bit systems will stop working in the year 2038 when the 32-bit time_t overflows). And many binary distributions, like Fedora, Ubuntu, and openSUSE Leap, have dropped support for all 32-bit architectures other than Armv7 and are likely to drop that as well before they would consider rebuilding against a new glibc. In the kernel 6.15, support for larger 32-bit x86 systems (those with more than eight CPUs or more than 4GB of RAM) has been removed. Those hardware configurations have been unavailable for a long time, and any workloads needing such resources should have long since moved to 64-bit systems. Thus, it seems that it is a trend to not support 32-bit architecture in Linux kernel. In rxe, we will also follow this trend. If some user-space applications still use 32-bit architecture currently, we can apply your commit. But from Linux kernel community, sooner or later, the support of 32-bit architecture will be dropped. Finally if some user-space applications still need 32-bit architecture in rxe, we can keep it. Or else, we will follow Leon's advice. It is just my 2-cent advice.^_^ Please Jason Gunthorpe or Leon Romanovsky comments on this. Best Regards, Zhu Yanjun > > Thanks, > Daisuke > >> >> Thanks >> >>> >>> Thanks, >>> Daisuke >>> >>> On Tue, April 8, 2025 8:12 PM Leon Romanovsky wrote: >>>> On Mon, 24 Mar 2025 16:56:47 +0900, Daisuke Matsuda wrote: >>>>> RDMA FLUSH[1] and ATOMIC WRITE[2] were added to rxe, but they cannot run >>>>> in the ODP mode as of now. This series is for the kernel-side enablement. >>>>> >>>>> There are also minor changes in libibverbs and pyverbs. The rdma-core tests >>>>> are also added so that people can test the features. >>>>> PR: https://github.com/linux-rdma/rdma-core/pull/1580 >>>>> >>>>> [...] >>>> >>>> Applied, thanks! >>>> >>>> [1/2] RDMA/rxe: Enable ODP in RDMA FLUSH operation >>>> https://git.kernel.org/rdma/rdma/c/32cad6aab9a699 >>>> [2/2] RDMA/rxe: Enable ODP in ATOMIC WRITE operation >>>> https://git.kernel.org/rdma/rdma/c/3e2746e0863f48 >>>> >>>> Best regards, >>>> -- >>>> Leon Romanovsky <leon@kernel.org> >>> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP 2025-04-14 12:56 ` Zhu Yanjun @ 2025-04-16 16:58 ` Leon Romanovsky 2025-04-18 2:07 ` Daisuke Matsuda (Fujitsu) 0 siblings, 1 reply; 13+ messages in thread From: Leon Romanovsky @ 2025-04-16 16:58 UTC (permalink / raw) To: Zhu Yanjun; +Cc: Daisuke Matsuda (Fujitsu), linux-rdma@vger.kernel.org On Mon, Apr 14, 2025 at 02:56:51PM +0200, Zhu Yanjun wrote: > On 14.04.25 12:16, Daisuke Matsuda (Fujitsu) wrote: > > On Sat, April 12, 2025 2:55 AM Leon Romanovsky wrote: > > > > Hi Leon, > > > > > > > > I have noticed the 2nd patch caused "kernel test robot" error, and you > > > > kindly amended the patch. However, another error has been detected by "the bot" > > > > because of the remaining fundamental problem that ATOMIC WRITE cannot > > > > be executed on non-64-bit architectures (at least on rxe). > > > > > > > > I think applying the change below to the original patch(*1) will resolve the issue. > > > > (*1) https://lore.kernel.org/linux-rdma/20250324075649.3313968-3-matsuda-daisuke@fujitsu.com/ > > > > ``` > > > > diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c > > > > index 02de05d759c6..ac3b3039db22 100644 > > > > --- a/drivers/infiniband/sw/rxe/rxe_odp.c > > > > +++ b/drivers/infiniband/sw/rxe/rxe_odp.c > > > > @@ -380,6 +380,7 @@ int rxe_odp_flush_pmem_iova(struct rxe_mr *mr, u64 iova, > > > > } > > > > > > > > /* CONFIG_64BIT=y */ > > > > +#ifdef CONFIG_64BIT > > > > enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) > > > > { > > > > struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem); > > > > @@ -424,3 +425,4 @@ enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) > > > > > > > > return RESPST_NONE; > > > > } > > > > +#endif > > > > ``` > > > > The definition of rxe_odp_do_atomic_write() should have been guarded in #ifdef CONFIG_64BIT. > > > > I believe this fix can address both: > > > > - the first error "error: redefinition of 'rxe_odp_do_atomic_write' " that could be caused when > > > > CONFIG_INFINIBAND_ON_DEMAND_PAGING=y && CONFIG_64BIT=n. > > > > - the second error caused by trying to compile 64-bit codes on 32-bit architectures. > > > > > > > > I am very sorry to bother you, but is it possible to make the modification? > > > > If I should provide a replacement patch, I will do so. > > > > > > I think that better will be simply make sure that RXE is dependent on 64bits. > > > > > > diff --git a/drivers/infiniband/sw/rxe/Kconfig b/drivers/infiniband/sw/rxe/Kconfig > > > index c180e7ebcfc5..1ed5b63f8afc 100644 > > > --- a/drivers/infiniband/sw/rxe/Kconfig > > > +++ b/drivers/infiniband/sw/rxe/Kconfig > > > @@ -1,7 +1,7 @@ > > > # SPDX-License-Identifier: GPL-2.0-only > > > config RDMA_RXE > > > tristate "Software RDMA over Ethernet (RoCE) driver" > > > - depends on INET && PCI && INFINIBAND > > > + depends on INET && PCI && INFINIBAND && 64BIT > > > depends on INFINIBAND_VIRT_DMA > > > select NET_UDP_TUNNEL > > > select CRC32 > > > > > > WDYT? > > > > It seems the driver is designed to be runnable on 32-bit nodes, so it may be > > overkill to disable 32-bit mode only for "ATOMIC WRITE" functionality. > > However, I do not have strong objection to making this change if you > > think it is better in terms of maintainability. > > > > Before making the change, I'd like to get an ACK or NACK from Zhu Yanjun. > > As far as I am aware, no one is actively maintaining or testing RXE on 32-bit, > > so it may be acceptable to drop 32-bit support, but it's best to confirm before proceeding. > > Hi, Daisuke Matsuda > > Thanks a lot for your efforts. > > There are some problems with 32-bit architectures, such as Year 2038 problem > ( many 32-bit systems will stop working in the year 2038 when the 32-bit > time_t overflows). > > And many binary distributions, like Fedora, Ubuntu, and openSUSE Leap, have > dropped support for all 32-bit architectures other than Armv7 and are likely > to drop that as well before they would consider rebuilding against a new > glibc. > > In the kernel 6.15, support for larger 32-bit x86 systems (those with more > than eight CPUs or more than 4GB of RAM) has been removed. Those hardware > configurations have been unavailable for a long time, and any workloads > needing such resources should have long since moved to 64-bit systems. > > Thus, it seems that it is a trend to not support 32-bit architecture in > Linux kernel. In rxe, we will also follow this trend. > > If some user-space applications still use 32-bit architecture currently, we > can apply your commit. But from Linux kernel community, sooner or later, the > support of 32-bit architecture will be dropped. > > Finally if some user-space applications still need 32-bit architecture in > rxe, we can keep it. Or else, we will follow Leon's advice. > > > It is just my 2-cent advice.^_^ > > Please Jason Gunthorpe or Leon Romanovsky comments on this. At the end RXE is for development, testing and early prototyping. I can't believe that we have developers who are using 32bits machines for such type of work in RDMA domain. Thanks > > Best Regards, > Zhu Yanjun > > > > > > Thanks, > > Daisuke > > > > > > > > Thanks > > > > > > > > > > > Thanks, > > > > Daisuke > > > > > > > > On Tue, April 8, 2025 8:12 PM Leon Romanovsky wrote: > > > > > On Mon, 24 Mar 2025 16:56:47 +0900, Daisuke Matsuda wrote: > > > > > > RDMA FLUSH[1] and ATOMIC WRITE[2] were added to rxe, but they cannot run > > > > > > in the ODP mode as of now. This series is for the kernel-side enablement. > > > > > > > > > > > > There are also minor changes in libibverbs and pyverbs. The rdma-core tests > > > > > > are also added so that people can test the features. > > > > > > PR: https://github.com/linux-rdma/rdma-core/pull/1580 > > > > > > > > > > > > [...] > > > > > > > > > > Applied, thanks! > > > > > > > > > > [1/2] RDMA/rxe: Enable ODP in RDMA FLUSH operation > > > > > https://git.kernel.org/rdma/rdma/c/32cad6aab9a699 > > > > > [2/2] RDMA/rxe: Enable ODP in ATOMIC WRITE operation > > > > > https://git.kernel.org/rdma/rdma/c/3e2746e0863f48 > > > > > > > > > > Best regards, > > > > > -- > > > > > Leon Romanovsky <leon@kernel.org> > > > > > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP 2025-04-16 16:58 ` Leon Romanovsky @ 2025-04-18 2:07 ` Daisuke Matsuda (Fujitsu) 2025-04-18 11:15 ` Leon Romanovsky 0 siblings, 1 reply; 13+ messages in thread From: Daisuke Matsuda (Fujitsu) @ 2025-04-18 2:07 UTC (permalink / raw) To: 'Leon Romanovsky', Zhu Yanjun; +Cc: linux-rdma@vger.kernel.org On Thu, April 17, 2025 1:59 AM Leon Romanovsky wrote: > On Mon, Apr 14, 2025 at 02:56:51PM +0200, Zhu Yanjun wrote: > > On 14.04.25 12:16, Daisuke Matsuda (Fujitsu) wrote: > > > On Sat, April 12, 2025 2:55 AM Leon Romanovsky wrote: > > > > > Hi Leon, > > > > > > > > > > I have noticed the 2nd patch caused "kernel test robot" error, and you > > > > > kindly amended the patch. However, another error has been detected by "the bot" > > > > > because of the remaining fundamental problem that ATOMIC WRITE cannot > > > > > be executed on non-64-bit architectures (at least on rxe). > > > > > > > > > > I think applying the change below to the original patch(*1) will resolve the issue. > > > > > (*1) https://lore.kernel.org/linux-rdma/20250324075649.3313968-3-matsuda-daisuke@fujitsu.com/ > > > > > ``` > > > > > diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c > > > > > index 02de05d759c6..ac3b3039db22 100644 > > > > > --- a/drivers/infiniband/sw/rxe/rxe_odp.c > > > > > +++ b/drivers/infiniband/sw/rxe/rxe_odp.c > > > > > @@ -380,6 +380,7 @@ int rxe_odp_flush_pmem_iova(struct rxe_mr *mr, u64 iova, > > > > > } > > > > > > > > > > /* CONFIG_64BIT=y */ > > > > > +#ifdef CONFIG_64BIT > > > > > enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) > > > > > { > > > > > struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem); > > > > > @@ -424,3 +425,4 @@ enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) > > > > > > > > > > return RESPST_NONE; > > > > > } > > > > > +#endif > > > > > ``` > > > > > The definition of rxe_odp_do_atomic_write() should have been guarded in #ifdef CONFIG_64BIT. > > > > > I believe this fix can address both: > > > > > - the first error "error: redefinition of 'rxe_odp_do_atomic_write' " that could be caused when > > > > > CONFIG_INFINIBAND_ON_DEMAND_PAGING=y && CONFIG_64BIT=n. > > > > > - the second error caused by trying to compile 64-bit codes on 32-bit architectures. > > > > > > > > > > I am very sorry to bother you, but is it possible to make the modification? > > > > > If I should provide a replacement patch, I will do so. > > > > > > > > I think that better will be simply make sure that RXE is dependent on 64bits. > > > > > > > > diff --git a/drivers/infiniband/sw/rxe/Kconfig b/drivers/infiniband/sw/rxe/Kconfig > > > > index c180e7ebcfc5..1ed5b63f8afc 100644 > > > > --- a/drivers/infiniband/sw/rxe/Kconfig > > > > +++ b/drivers/infiniband/sw/rxe/Kconfig > > > > @@ -1,7 +1,7 @@ > > > > # SPDX-License-Identifier: GPL-2.0-only > > > > config RDMA_RXE > > > > tristate "Software RDMA over Ethernet (RoCE) driver" > > > > - depends on INET && PCI && INFINIBAND > > > > + depends on INET && PCI && INFINIBAND && 64BIT > > > > depends on INFINIBAND_VIRT_DMA > > > > select NET_UDP_TUNNEL > > > > select CRC32 > > > > > > > > WDYT? > > > > > > It seems the driver is designed to be runnable on 32-bit nodes, so it may be > > > overkill to disable 32-bit mode only for "ATOMIC WRITE" functionality. > > > However, I do not have strong objection to making this change if you > > > think it is better in terms of maintainability. > > > > > > Before making the change, I'd like to get an ACK or NACK from Zhu Yanjun. > > > As far as I am aware, no one is actively maintaining or testing RXE on 32-bit, > > > so it may be acceptable to drop 32-bit support, but it's best to confirm before proceeding. > > > > Hi, Daisuke Matsuda > > > > Thanks a lot for your efforts. > > > > There are some problems with 32-bit architectures, such as Year 2038 problem > > ( many 32-bit systems will stop working in the year 2038 when the 32-bit > > time_t overflows). > > > > And many binary distributions, like Fedora, Ubuntu, and openSUSE Leap, have > > dropped support for all 32-bit architectures other than Armv7 and are likely > > to drop that as well before they would consider rebuilding against a new > > glibc. > > > > In the kernel 6.15, support for larger 32-bit x86 systems (those with more > > than eight CPUs or more than 4GB of RAM) has been removed. Those hardware > > configurations have been unavailable for a long time, and any workloads > > needing such resources should have long since moved to 64-bit systems. > > > > Thus, it seems that it is a trend to not support 32-bit architecture in > > Linux kernel. In rxe, we will also follow this trend. > > > > If some user-space applications still use 32-bit architecture currently, we > > can apply your commit. But from Linux kernel community, sooner or later, the > > support of 32-bit architecture will be dropped. > > > > Finally if some user-space applications still need 32-bit architecture in > > rxe, we can keep it. Or else, we will follow Leon's advice. > > > > > > It is just my 2-cent advice.^_^ > > > > Please Jason Gunthorpe or Leon Romanovsky comments on this. > > At the end RXE is for development, testing and early prototyping. I can't > believe that we have developers who are using 32bits machines for such type > of work in RDMA domain. Agreed. Could you modify "RDMA/rxe: Enable ODP in ATOMIC WRITE operation" in the tree and change Kconfig as you suggested? Thanks, Daisuke ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP 2025-04-18 2:07 ` Daisuke Matsuda (Fujitsu) @ 2025-04-18 11:15 ` Leon Romanovsky 2025-04-20 23:31 ` Daisuke Matsuda (Fujitsu) 0 siblings, 1 reply; 13+ messages in thread From: Leon Romanovsky @ 2025-04-18 11:15 UTC (permalink / raw) To: Daisuke Matsuda (Fujitsu); +Cc: Zhu Yanjun, linux-rdma@vger.kernel.org On Fri, Apr 18, 2025 at 02:07:49AM +0000, Daisuke Matsuda (Fujitsu) wrote: > On Thu, April 17, 2025 1:59 AM Leon Romanovsky wrote: > > On Mon, Apr 14, 2025 at 02:56:51PM +0200, Zhu Yanjun wrote: > > > On 14.04.25 12:16, Daisuke Matsuda (Fujitsu) wrote: > > > > On Sat, April 12, 2025 2:55 AM Leon Romanovsky wrote: > > > > > > Hi Leon, > > > > > > > > > > > > I have noticed the 2nd patch caused "kernel test robot" error, and you > > > > > > kindly amended the patch. However, another error has been detected by "the bot" > > > > > > because of the remaining fundamental problem that ATOMIC WRITE cannot > > > > > > be executed on non-64-bit architectures (at least on rxe). > > > > > > > > > > > > I think applying the change below to the original patch(*1) will resolve the issue. > > > > > > (*1) https://lore.kernel.org/linux-rdma/20250324075649.3313968-3-matsuda-daisuke@fujitsu.com/ > > > > > > ``` > > > > > > diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c > > > > > > index 02de05d759c6..ac3b3039db22 100644 > > > > > > --- a/drivers/infiniband/sw/rxe/rxe_odp.c > > > > > > +++ b/drivers/infiniband/sw/rxe/rxe_odp.c > > > > > > @@ -380,6 +380,7 @@ int rxe_odp_flush_pmem_iova(struct rxe_mr *mr, u64 iova, > > > > > > } > > > > > > > > > > > > /* CONFIG_64BIT=y */ > > > > > > +#ifdef CONFIG_64BIT > > > > > > enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) > > > > > > { > > > > > > struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem); > > > > > > @@ -424,3 +425,4 @@ enum resp_states rxe_odp_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) > > > > > > > > > > > > return RESPST_NONE; > > > > > > } > > > > > > +#endif > > > > > > ``` > > > > > > The definition of rxe_odp_do_atomic_write() should have been guarded in #ifdef CONFIG_64BIT. > > > > > > I believe this fix can address both: > > > > > > - the first error "error: redefinition of 'rxe_odp_do_atomic_write' " that could be caused when > > > > > > CONFIG_INFINIBAND_ON_DEMAND_PAGING=y && CONFIG_64BIT=n. > > > > > > - the second error caused by trying to compile 64-bit codes on 32-bit architectures. > > > > > > > > > > > > I am very sorry to bother you, but is it possible to make the modification? > > > > > > If I should provide a replacement patch, I will do so. > > > > > > > > > > I think that better will be simply make sure that RXE is dependent on 64bits. > > > > > > > > > > diff --git a/drivers/infiniband/sw/rxe/Kconfig b/drivers/infiniband/sw/rxe/Kconfig > > > > > index c180e7ebcfc5..1ed5b63f8afc 100644 > > > > > --- a/drivers/infiniband/sw/rxe/Kconfig > > > > > +++ b/drivers/infiniband/sw/rxe/Kconfig > > > > > @@ -1,7 +1,7 @@ > > > > > # SPDX-License-Identifier: GPL-2.0-only > > > > > config RDMA_RXE > > > > > tristate "Software RDMA over Ethernet (RoCE) driver" > > > > > - depends on INET && PCI && INFINIBAND > > > > > + depends on INET && PCI && INFINIBAND && 64BIT > > > > > depends on INFINIBAND_VIRT_DMA > > > > > select NET_UDP_TUNNEL > > > > > select CRC32 > > > > > > > > > > WDYT? > > > > > > > > It seems the driver is designed to be runnable on 32-bit nodes, so it may be > > > > overkill to disable 32-bit mode only for "ATOMIC WRITE" functionality. > > > > However, I do not have strong objection to making this change if you > > > > think it is better in terms of maintainability. > > > > > > > > Before making the change, I'd like to get an ACK or NACK from Zhu Yanjun. > > > > As far as I am aware, no one is actively maintaining or testing RXE on 32-bit, > > > > so it may be acceptable to drop 32-bit support, but it's best to confirm before proceeding. > > > > > > Hi, Daisuke Matsuda > > > > > > Thanks a lot for your efforts. > > > > > > There are some problems with 32-bit architectures, such as Year 2038 problem > > > ( many 32-bit systems will stop working in the year 2038 when the 32-bit > > > time_t overflows). > > > > > > And many binary distributions, like Fedora, Ubuntu, and openSUSE Leap, have > > > dropped support for all 32-bit architectures other than Armv7 and are likely > > > to drop that as well before they would consider rebuilding against a new > > > glibc. > > > > > > In the kernel 6.15, support for larger 32-bit x86 systems (those with more > > > than eight CPUs or more than 4GB of RAM) has been removed. Those hardware > > > configurations have been unavailable for a long time, and any workloads > > > needing such resources should have long since moved to 64-bit systems. > > > > > > Thus, it seems that it is a trend to not support 32-bit architecture in > > > Linux kernel. In rxe, we will also follow this trend. > > > > > > If some user-space applications still use 32-bit architecture currently, we > > > can apply your commit. But from Linux kernel community, sooner or later, the > > > support of 32-bit architecture will be dropped. > > > > > > Finally if some user-space applications still need 32-bit architecture in > > > rxe, we can keep it. Or else, we will follow Leon's advice. > > > > > > > > > It is just my 2-cent advice.^_^ > > > > > > Please Jason Gunthorpe or Leon Romanovsky comments on this. > > > > At the end RXE is for development, testing and early prototyping. I can't > > believe that we have developers who are using 32bits machines for such type > > of work in RDMA domain. > > Agreed. > Could you modify "RDMA/rxe: Enable ODP in ATOMIC WRITE operation" > in the tree and change Kconfig as you suggested? Just send separate patch, please. > > Thanks, > Daisuke > ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP 2025-04-18 11:15 ` Leon Romanovsky @ 2025-04-20 23:31 ` Daisuke Matsuda (Fujitsu) 0 siblings, 0 replies; 13+ messages in thread From: Daisuke Matsuda (Fujitsu) @ 2025-04-20 23:31 UTC (permalink / raw) To: 'Leon Romanovsky'; +Cc: Zhu Yanjun, linux-rdma@vger.kernel.org On Fri, April 18, 2025 8:16 PM Leon Romanovsky wrote: > On Fri, Apr 18, 2025 at 02:07:49AM +0000, Daisuke Matsuda (Fujitsu) wrote: > > On Thu, April 17, 2025 1:59 AM Leon Romanovsky wrote: > > > On Mon, Apr 14, 2025 at 02:56:51PM +0200, Zhu Yanjun wrote: > > > > On 14.04.25 12:16, Daisuke Matsuda (Fujitsu) wrote: > > > > > On Sat, April 12, 2025 2:55 AM Leon Romanovsky wrote: <...> > > > At the end RXE is for development, testing and early prototyping. I can't > > > believe that we have developers who are using 32bits machines for such type > > > of work in RDMA domain. > > > > Agreed. > > Could you modify "RDMA/rxe: Enable ODP in ATOMIC WRITE operation" > > in the tree and change Kconfig as you suggested? > > Just send separate patch, please. Roger that. ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2025-04-20 23:32 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-03-24 7:56 [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP Daisuke Matsuda 2025-03-24 7:56 ` [PATCH for-next v3 1/2] RDMA/rxe: Enable ODP in RDMA FLUSH operation Daisuke Matsuda 2025-03-24 7:56 ` [PATCH for-next v3 2/2] RDMA/rxe: Enable ODP in ATOMIC WRITE operation Daisuke Matsuda 2025-04-08 11:11 ` [PATCH for-next v3 0/2] RDMA/rxe: RDMA FLUSH and ATOMIC WRITE with ODP Leon Romanovsky 2025-04-08 11:11 ` Leon Romanovsky 2025-04-11 8:13 ` Daisuke Matsuda (Fujitsu) 2025-04-11 17:55 ` Leon Romanovsky 2025-04-14 10:16 ` Daisuke Matsuda (Fujitsu) 2025-04-14 12:56 ` Zhu Yanjun 2025-04-16 16:58 ` Leon Romanovsky 2025-04-18 2:07 ` Daisuke Matsuda (Fujitsu) 2025-04-18 11:15 ` Leon Romanovsky 2025-04-20 23:31 ` Daisuke Matsuda (Fujitsu)
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).