* [PATCH for-next v3 1/2] RDMA/efa: Use device ABI MR permissions instead of verbs flags
2026-09-22 16:01 [PATCH for-next v3 0/2] RDMA/efa: Add support for MR relaxed ordering Michael Margolin
@ 2026-09-22 16:01 ` Michael Margolin
2026-09-22 16:07 ` sashiko-bot
2026-09-22 16:01 ` [PATCH for-next v3 2/2] RDMA/efa: Pass relaxed ordering flag to device Michael Margolin
2026-09-29 18:07 ` [PATCH for-next v3 0/2] RDMA/efa: Add support for MR relaxed ordering Leon Romanovsky
2 siblings, 1 reply; 7+ messages in thread
From: Michael Margolin @ 2026-09-22 16:01 UTC (permalink / raw)
To: jgg, leon, linux-rdma
Cc: sleybo, matua, gal.pressman, Dana Malachi, Chen Brasch,
Yonatan Nachum
From: Dana Malachi <danamala@amazon.com>
Set MR permissions explicitly using device interface
definitions rather than copying raw verbs access flags.
This is needed for the next commit, to allow access flag bits
that are not in the permissions field.
Reviewed-by: Chen Brasch <cbrasch@amazon.com>
Reviewed-by: Yonatan Nachum <ynachum@amazon.com>
Signed-off-by: Dana Malachi <danamala@amazon.com>
Signed-off-by: Michael Margolin <mrgolin@amazon.com>
---
drivers/infiniband/hw/efa/efa_com_cmd.c | 7 ++++++-
drivers/infiniband/hw/efa/efa_com_cmd.h | 7 +++++--
drivers/infiniband/hw/efa/efa_verbs.c | 4 +++-
3 files changed, 14 insertions(+), 4 deletions(-)
diff --git a/drivers/infiniband/hw/efa/efa_com_cmd.c b/drivers/infiniband/hw/efa/efa_com_cmd.c
index 1b00f16b8ea8..aac0e936601a 100644
--- a/drivers/infiniband/hw/efa/efa_com_cmd.c
+++ b/drivers/infiniband/hw/efa/efa_com_cmd.c
@@ -237,7 +237,12 @@ int efa_com_register_mr(struct efa_com_dev *edev,
EFA_SET(&mr_cmd.flags, EFA_ADMIN_REG_MR_CMD_PHYS_PAGE_SIZE_SHIFT,
params->page_shift);
mr_cmd.iova = params->iova;
- mr_cmd.permissions = params->permissions;
+ if (params->permissions.local_write)
+ EFA_SET(&mr_cmd.permissions, EFA_ADMIN_REG_MR_CMD_LOCAL_WRITE_ENABLE, 1);
+ if (params->permissions.remote_write)
+ EFA_SET(&mr_cmd.permissions, EFA_ADMIN_REG_MR_CMD_REMOTE_WRITE_ENABLE, 1);
+ if (params->permissions.remote_read)
+ EFA_SET(&mr_cmd.permissions, EFA_ADMIN_REG_MR_CMD_REMOTE_READ_ENABLE, 1);
if (params->inline_pbl) {
memcpy(mr_cmd.pbl.inline_pbl_array,
diff --git a/drivers/infiniband/hw/efa/efa_com_cmd.h b/drivers/infiniband/hw/efa/efa_com_cmd.h
index bca722f021ea..7b0399811cfd 100644
--- a/drivers/infiniband/hw/efa/efa_com_cmd.h
+++ b/drivers/infiniband/hw/efa/efa_com_cmd.h
@@ -204,8 +204,11 @@ struct efa_com_reg_mr_params {
* address mapping
*/
u8 page_shift;
- /* see permissions field of struct efa_admin_reg_mr_cmd */
- u8 permissions;
+ struct {
+ u8 local_write : 1;
+ u8 remote_write : 1;
+ u8 remote_read : 1;
+ } permissions;
u8 inline_pbl;
u8 indirect;
};
diff --git a/drivers/infiniband/hw/efa/efa_verbs.c b/drivers/infiniband/hw/efa/efa_verbs.c
index 2d28d68efe77..635687275350 100644
--- a/drivers/infiniband/hw/efa/efa_verbs.c
+++ b/drivers/infiniband/hw/efa/efa_verbs.c
@@ -1676,7 +1676,9 @@ static int efa_register_mr(struct ib_pd *ibpd, struct efa_mr *mr, u64 start,
params.pd = to_epd(ibpd)->pdn;
params.iova = virt_addr;
params.mr_length_in_bytes = length;
- params.permissions = access_flags;
+ params.permissions.local_write = !!(access_flags & IB_ACCESS_LOCAL_WRITE);
+ params.permissions.remote_write = !!(access_flags & IB_ACCESS_REMOTE_WRITE);
+ params.permissions.remote_read = !!(access_flags & IB_ACCESS_REMOTE_READ);
pg_sz = ib_umem_find_best_pgsz(mr->umem,
dev->dev_attr.page_size_cap,
--
2.47.3
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH for-next v3 2/2] RDMA/efa: Pass relaxed ordering flag to device
2026-09-22 16:01 [PATCH for-next v3 0/2] RDMA/efa: Add support for MR relaxed ordering Michael Margolin
2026-09-22 16:01 ` [PATCH for-next v3 1/2] RDMA/efa: Use device ABI MR permissions instead of verbs flags Michael Margolin
@ 2026-09-22 16:01 ` Michael Margolin
2026-09-22 16:14 ` sashiko-bot
2026-09-29 18:07 ` [PATCH for-next v3 0/2] RDMA/efa: Add support for MR relaxed ordering Leon Romanovsky
2 siblings, 1 reply; 7+ messages in thread
From: Michael Margolin @ 2026-09-22 16:01 UTC (permalink / raw)
To: jgg, leon, linux-rdma
Cc: sleybo, matua, gal.pressman, Dana Malachi, Chen Brasch,
Yonatan Nachum
From: Dana Malachi <danamala@amazon.com>
Accept the relaxed ordering access flag during memory region
registration and forward it to the device firmware via the
admin command path.
Reviewed-by: Chen Brasch <cbrasch@amazon.com>
Reviewed-by: Yonatan Nachum <ynachum@amazon.com>
Signed-off-by: Dana Malachi <danamala@amazon.com>
Signed-off-by: Michael Margolin <mrgolin@amazon.com>
---
drivers/infiniband/hw/efa/efa_admin_cmds_defs.h | 5 +++--
drivers/infiniband/hw/efa/efa_com_cmd.c | 2 ++
drivers/infiniband/hw/efa/efa_com_cmd.h | 1 +
drivers/infiniband/hw/efa/efa_verbs.c | 1 +
4 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/hw/efa/efa_admin_cmds_defs.h b/drivers/infiniband/hw/efa/efa_admin_cmds_defs.h
index ab830764e3b4..30a38e633bfd 100644
--- a/drivers/infiniband/hw/efa/efa_admin_cmds_defs.h
+++ b/drivers/infiniband/hw/efa/efa_admin_cmds_defs.h
@@ -358,7 +358,8 @@ struct efa_admin_reg_mr_cmd {
* 5:0 : phys_page_size_shift - page size is (1 <<
* phys_page_size_shift). Page size is used for
* building the Virtual to Physical address mapping
- * 6 : reserved - MBZ
+ * 6 : relaxed_ordering_enable - Enable PCIe relaxed
+ * ordering for this memory region
* 7 : mem_addr_phy_mode_en - Enable bit for physical
* memory registration (no translation), can be used
* only by privileged clients. If set, PBL must
@@ -1203,6 +1204,7 @@ struct efa_admin_modify_event_counter_resp {
/* reg_mr_cmd */
#define EFA_ADMIN_REG_MR_CMD_PHYS_PAGE_SIZE_SHIFT_MASK GENMASK(5, 0)
+#define EFA_ADMIN_REG_MR_CMD_RELAXED_ORDERING_ENABLE_MASK BIT(6)
#define EFA_ADMIN_REG_MR_CMD_MEM_ADDR_PHY_MODE_EN_MASK BIT(7)
#define EFA_ADMIN_REG_MR_CMD_LOCAL_WRITE_ENABLE_MASK BIT(0)
#define EFA_ADMIN_REG_MR_CMD_REMOTE_WRITE_ENABLE_MASK BIT(1)
@@ -1231,7 +1233,6 @@ struct efa_admin_modify_event_counter_resp {
#define EFA_ADMIN_FEATURE_DEVICE_ATTR_DESC_RDMA_WRITE_MASK BIT(3)
#define EFA_ADMIN_FEATURE_DEVICE_ATTR_DESC_UNSOLICITED_WRITE_RECV_MASK BIT(4)
#define EFA_ADMIN_FEATURE_DEVICE_ATTR_DESC_EVENT_COUNTERS_MASK BIT(5)
-#define EFA_ADMIN_FEATURE_DEVICE_ATTR_DESC_SQ_64_BIT_REQ_ID_SHIFT 10
#define EFA_ADMIN_FEATURE_DEVICE_ATTR_DESC_SQ_64_BIT_REQ_ID_MASK BIT(10)
/* feature_queue_attr_desc_2 */
diff --git a/drivers/infiniband/hw/efa/efa_com_cmd.c b/drivers/infiniband/hw/efa/efa_com_cmd.c
index aac0e936601a..07bfcb093df8 100644
--- a/drivers/infiniband/hw/efa/efa_com_cmd.c
+++ b/drivers/infiniband/hw/efa/efa_com_cmd.c
@@ -236,6 +236,8 @@ int efa_com_register_mr(struct efa_com_dev *edev,
mr_cmd.mr_length = params->mr_length_in_bytes;
EFA_SET(&mr_cmd.flags, EFA_ADMIN_REG_MR_CMD_PHYS_PAGE_SIZE_SHIFT,
params->page_shift);
+ if (params->relaxed_ordering)
+ EFA_SET(&mr_cmd.flags, EFA_ADMIN_REG_MR_CMD_RELAXED_ORDERING_ENABLE, 1);
mr_cmd.iova = params->iova;
if (params->permissions.local_write)
EFA_SET(&mr_cmd.permissions, EFA_ADMIN_REG_MR_CMD_LOCAL_WRITE_ENABLE, 1);
diff --git a/drivers/infiniband/hw/efa/efa_com_cmd.h b/drivers/infiniband/hw/efa/efa_com_cmd.h
index 7b0399811cfd..524407f36d43 100644
--- a/drivers/infiniband/hw/efa/efa_com_cmd.h
+++ b/drivers/infiniband/hw/efa/efa_com_cmd.h
@@ -209,6 +209,7 @@ struct efa_com_reg_mr_params {
u8 remote_write : 1;
u8 remote_read : 1;
} permissions;
+ u8 relaxed_ordering;
u8 inline_pbl;
u8 indirect;
};
diff --git a/drivers/infiniband/hw/efa/efa_verbs.c b/drivers/infiniband/hw/efa/efa_verbs.c
index 635687275350..9e4435401a4c 100644
--- a/drivers/infiniband/hw/efa/efa_verbs.c
+++ b/drivers/infiniband/hw/efa/efa_verbs.c
@@ -1679,6 +1679,7 @@ static int efa_register_mr(struct ib_pd *ibpd, struct efa_mr *mr, u64 start,
params.permissions.local_write = !!(access_flags & IB_ACCESS_LOCAL_WRITE);
params.permissions.remote_write = !!(access_flags & IB_ACCESS_REMOTE_WRITE);
params.permissions.remote_read = !!(access_flags & IB_ACCESS_REMOTE_READ);
+ params.relaxed_ordering = !!(access_flags & IB_ACCESS_RELAXED_ORDERING);
pg_sz = ib_umem_find_best_pgsz(mr->umem,
dev->dev_attr.page_size_cap,
--
2.47.3
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH for-next v3 2/2] RDMA/efa: Pass relaxed ordering flag to device
2026-09-22 16:01 ` [PATCH for-next v3 2/2] RDMA/efa: Pass relaxed ordering flag to device Michael Margolin
@ 2026-09-22 16:14 ` sashiko-bot
2026-09-22 16:28 ` Michael Margolin
0 siblings, 1 reply; 7+ messages in thread
From: sashiko-bot @ 2026-09-22 16:14 UTC (permalink / raw)
To: Michael Margolin; +Cc: linux-rdma
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] efa: missing firmware capability check for relaxed ordering
--
commit 8cde16309300638ef5ed5570f8cb4ae347e5c6e6
Author: Dana Malachi <danamala@amazon.com>
RDMA/efa: Pass relaxed ordering flag to device
This commit accepts the relaxed ordering access flag during memory region
registration and forwards it to the device firmware via the admin command
path.
> diff --git a/drivers/infiniband/hw/efa/efa_com_cmd.c b/drivers/infiniband/hw/efa/efa_com_cmd.c
> index aac0e936601a1..07bfcb093df85 100644
> --- a/drivers/infiniband/hw/efa/efa_com_cmd.c
> +++ b/drivers/infiniband/hw/efa/efa_com_cmd.c
[ ... ]
> @@ -236,6 +236,8 @@ int efa_com_register_mr(struct efa_com_dev *edev,
> mr_cmd.mr_length = params->mr_length_in_bytes;
> EFA_SET(&mr_cmd.flags, EFA_ADMIN_REG_MR_CMD_PHYS_PAGE_SIZE_SHIFT,
> params->page_shift);
> + if (params->relaxed_ordering)
> + EFA_SET(&mr_cmd.flags, EFA_ADMIN_REG_MR_CMD_RELAXED_ORDERING_ENABLE, 1);
[Severity: Medium]
Could setting this bit cause memory region registration to fail on older
firmware versions?
If older firmware strictly enforces the MBZ (Must Be Zero) rule for bit 6,
it might reject the command when userspace requests relaxed ordering. Since
there doesn't appear to be a check against device attributes or firmware
capabilities here, does the driver need to verify firmware support before
passing this flag down in efa_com_register_mr()?
[ ... ]
> diff --git a/drivers/infiniband/hw/efa/efa_verbs.c b/drivers/infiniband/hw/efa/efa_verbs.c
> index 6356872753507..9e4435401a4c7 100644
> --- a/drivers/infiniband/hw/efa/efa_verbs.c
> +++ b/drivers/infiniband/hw/efa/efa_verbs.c
[ ... ]
> @@ -1679,6 +1679,7 @@ static int efa_register_mr(struct ib_pd *ibpd, struct efa_mr *mr, u64 start,
> params.permissions.local_write = !!(access_flags & IB_ACCESS_LOCAL_WRITE);
> params.permissions.remote_write = !!(access_flags & IB_ACCESS_REMOTE_WRITE);
> params.permissions.remote_read = !!(access_flags & IB_ACCESS_REMOTE_READ);
> + params.relaxed_ordering = !!(access_flags & IB_ACCESS_RELAXED_ORDERING);
[Severity: Medium]
This passes the userspace IB_ACCESS_RELAXED_ORDERING flag directly into
the command parameters in efa_register_mr().
Should this be gated by a capability check so we don't inadvertently break
MR registration for older devices if a userspace application unconditionally
sets this access flag?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260922160118.990-1-mrgolin@amazon.com?part=2
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH for-next v3 2/2] RDMA/efa: Pass relaxed ordering flag to device
2026-09-22 16:14 ` sashiko-bot
@ 2026-09-22 16:28 ` Michael Margolin
0 siblings, 0 replies; 7+ messages in thread
From: Michael Margolin @ 2026-09-22 16:28 UTC (permalink / raw)
To: sashiko-reviews; +Cc: linux-rdma
On Tue, Sep 22, 2026 at 04:14:43PM +0000, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [Medium] efa: missing firmware capability check for relaxed ordering
> --
>
> commit 8cde16309300638ef5ed5570f8cb4ae347e5c6e6
> Author: Dana Malachi <danamala@amazon.com>
>
> RDMA/efa: Pass relaxed ordering flag to device
>
> This commit accepts the relaxed ordering access flag during memory region
> registration and forwards it to the device firmware via the admin command
> path.
>
> > diff --git a/drivers/infiniband/hw/efa/efa_com_cmd.c b/drivers/infiniband/hw/efa/efa_com_cmd.c
> > index aac0e936601a1..07bfcb093df85 100644
> > --- a/drivers/infiniband/hw/efa/efa_com_cmd.c
> > +++ b/drivers/infiniband/hw/efa/efa_com_cmd.c
> [ ... ]
> > @@ -236,6 +236,8 @@ int efa_com_register_mr(struct efa_com_dev *edev,
> > mr_cmd.mr_length = params->mr_length_in_bytes;
> > EFA_SET(&mr_cmd.flags, EFA_ADMIN_REG_MR_CMD_PHYS_PAGE_SIZE_SHIFT,
> > params->page_shift);
> > + if (params->relaxed_ordering)
> > + EFA_SET(&mr_cmd.flags, EFA_ADMIN_REG_MR_CMD_RELAXED_ORDERING_ENABLE, 1);
>
> [Severity: Medium]
> Could setting this bit cause memory region registration to fail on older
> firmware versions?
>
> If older firmware strictly enforces the MBZ (Must Be Zero) rule for bit 6,
> it might reject the command when userspace requests relaxed ordering. Since
> there doesn't appear to be a check against device attributes or firmware
> capabilities here, does the driver need to verify firmware support before
> passing this flag down in efa_com_register_mr()?
>
> [ ... ]
> > diff --git a/drivers/infiniband/hw/efa/efa_verbs.c b/drivers/infiniband/hw/efa/efa_verbs.c
> > index 6356872753507..9e4435401a4c7 100644
> > --- a/drivers/infiniband/hw/efa/efa_verbs.c
> > +++ b/drivers/infiniband/hw/efa/efa_verbs.c
> [ ... ]
> > @@ -1679,6 +1679,7 @@ static int efa_register_mr(struct ib_pd *ibpd, struct efa_mr *mr, u64 start,
> > params.permissions.local_write = !!(access_flags & IB_ACCESS_LOCAL_WRITE);
> > params.permissions.remote_write = !!(access_flags & IB_ACCESS_REMOTE_WRITE);
> > params.permissions.remote_read = !!(access_flags & IB_ACCESS_REMOTE_READ);
> > + params.relaxed_ordering = !!(access_flags & IB_ACCESS_RELAXED_ORDERING);
>
> [Severity: Medium]
> This passes the userspace IB_ACCESS_RELAXED_ORDERING flag directly into
> the command parameters in efa_register_mr().
>
> Should this be gated by a capability check so we don't inadvertently break
> MR registration for older devices if a userspace application unconditionally
> sets this access flag?
>
For both comments (that are basically same), that's ok in this case -
devices ignore this bit when RO is not supported.
Michael
> --
> Sashiko AI review · https://sashiko.dev/#/patchset/20260922160118.990-1-mrgolin@amazon.com?part=2
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH for-next v3 0/2] RDMA/efa: Add support for MR relaxed ordering
2026-09-22 16:01 [PATCH for-next v3 0/2] RDMA/efa: Add support for MR relaxed ordering Michael Margolin
2026-09-22 16:01 ` [PATCH for-next v3 1/2] RDMA/efa: Use device ABI MR permissions instead of verbs flags Michael Margolin
2026-09-22 16:01 ` [PATCH for-next v3 2/2] RDMA/efa: Pass relaxed ordering flag to device Michael Margolin
@ 2026-09-29 18:07 ` Leon Romanovsky
2 siblings, 0 replies; 7+ messages in thread
From: Leon Romanovsky @ 2026-09-29 18:07 UTC (permalink / raw)
To: linux-rdma, Jason Gunthorpe, Michael Margolin; +Cc: sleybo, matua, gal.pressman
On Tue, 22 Sep 2026 16:01:16 +0000, Michael Margolin wrote:
> Add support for MR relaxed ordering access flag, allowing userspace to
> create PCIe relaxed ordering MRs.
>
> -------------------------------------------------------------------------
> Changelog:
> v3:
> * No changes, just resend
> * Link to v2: https://lore.kernel.org/all/20260824073057.1046630-1-ynachum@amazon.com/
> v2:
> * Remove the relaxed ordering device capability.
> * Link to v1: https://lore.kernel.org/all/20260806075758.3476667-1-ynachum@amazon.com/
>
> [...]
Applied, thanks!
[1/2] RDMA/efa: Use device ABI MR permissions instead of verbs flags
https://git.kernel.org/rdma/rdma/c/b840f0890ad06f
[2/2] RDMA/efa: Pass relaxed ordering flag to device
https://git.kernel.org/rdma/rdma/c/eb6aed55cc0152
Best regards,
--
Leon Romanovsky <leon@kernel.org>
^ permalink raw reply [flat|nested] 7+ messages in thread