* [PATCH rdma-next v2 01/13] RDMA/core: widen the uverbs object key
2026-10-20 3:03 [PATCH rdma-next v2 00/13] RDMA/core: Ultra Ethernet transport verbs Eric Davis via B4 Relay
@ 2026-10-20 3:03 ` Eric Davis via B4 Relay
2026-10-20 8:10 ` sashiko-bot
2026-10-20 3:03 ` [PATCH rdma-next v2 02/13] RDMA/core: add the UET GID types Eric Davis via B4 Relay
` (11 subsequent siblings)
12 siblings, 1 reply; 27+ messages in thread
From: Eric Davis via B4 Relay @ 2026-10-20 3:03 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, linux-kernel, Eric Davis
From: Eric Davis <eric.davis@broadcom.com>
Three more core objects would leave drivers nine object keys, and mlx5
alone uses ten. Add a bit to the object key.
Signed-off-by: Eric Davis <eric.davis@broadcom.com>
---
include/rdma/uverbs_ioctl.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/include/rdma/uverbs_ioctl.h b/include/rdma/uverbs_ioctl.h
index 1aaaadb8daf6..f4d93bc3625a 100644
--- a/include/rdma/uverbs_ioctl.h
+++ b/include/rdma/uverbs_ioctl.h
@@ -144,7 +144,7 @@ enum uapi_radix_data {
UVERBS_API_METHOD_KEY_BITS + UVERBS_API_METHOD_KEY_SHIFT - 1,
UVERBS_API_METHOD_KEY_SHIFT),
- UVERBS_API_OBJ_KEY_BITS = 5,
+ UVERBS_API_OBJ_KEY_BITS = 6,
UVERBS_API_OBJ_KEY_SHIFT =
UVERBS_API_METHOD_KEY_BITS + UVERBS_API_METHOD_KEY_SHIFT,
UVERBS_API_OBJ_KEY_NUM_CORE = 20,
--
2.56.0
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH rdma-next v2 02/13] RDMA/core: add the UET GID types
2026-10-20 3:03 [PATCH rdma-next v2 00/13] RDMA/core: Ultra Ethernet transport verbs Eric Davis via B4 Relay
2026-10-20 3:03 ` [PATCH rdma-next v2 01/13] RDMA/core: widen the uverbs object key Eric Davis via B4 Relay
@ 2026-10-20 3:03 ` Eric Davis via B4 Relay
2026-10-20 8:16 ` sashiko-bot
2026-10-20 3:03 ` [PATCH rdma-next v2 03/13] RDMA/core: report the reliable unconnected device capabilities Eric Davis via B4 Relay
` (10 subsequent siblings)
12 siblings, 1 reply; 27+ messages in thread
From: Eric Davis via B4 Relay @ 2026-10-20 3:03 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, linux-kernel, Eric Davis
From: Eric Davis <eric.davis@broadcom.com>
Ultra Ethernet runs over two encapsulations: UDP and IP. Each becomes
a new GID type. A port can then hold both UET and RoCE GIDs, with the
transport following from the GID a queue pair is given, the driver
enforces that.
These UET ports build their GID table from a netdev, as RoCE does. Added
rdma_port_gids_from_netdev() and use it where the code tested for RoCE
alone.
Signed-off-by: Eric Davis <eric.davis@broadcom.com>
---
drivers/infiniband/core/cache.c | 12 +++++----
drivers/infiniband/core/cma.c | 4 +++
drivers/infiniband/core/device.c | 2 +-
drivers/infiniband/core/nldev.c | 2 ++
drivers/infiniband/core/roce_gid_mgmt.c | 6 +++--
drivers/infiniband/core/verbs.c | 4 +++
include/rdma/ib_verbs.h | 48 ++++++++++++++++++++++++++++++---
include/uapi/rdma/ib_user_ioctl_verbs.h | 2 ++
8 files changed, 69 insertions(+), 11 deletions(-)
diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index 647a547e2d7f..d5e17a36f186 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -139,6 +139,8 @@ static const char * const gid_type_str[] = {
[IB_GID_TYPE_IB] = "IB/RoCE v1",
[IB_GID_TYPE_ROCE] = "IB/RoCE v1",
[IB_GID_TYPE_ROCE_UDP_ENCAP] = "RoCE v2",
+ [IB_GID_TYPE_UET_UDP] = "UET UDP",
+ [IB_GID_TYPE_UET_IP] = "UET IP",
};
const char *ib_cache_gid_type_str(enum ib_gid_type gid_type)
@@ -392,10 +394,10 @@ static void del_gid(struct ib_device *ib_dev, u32 port,
write_lock_irq(&table->rwlock);
entry = table->data_vec[ix];
entry->state = GID_TABLE_ENTRY_PENDING_DEL;
- /*
- * For non RoCE protocol, GID entry slot is ready to use.
+ /* Where the GID table is not derived from a netdev, the entry slot is
+ * ready to use.
*/
- if (!rdma_protocol_roce(ib_dev, port))
+ if (!rdma_port_gids_from_netdev(ib_dev, port))
table->data_vec[ix] = NULL;
write_unlock_irq(&table->rwlock);
@@ -447,7 +449,7 @@ static int add_modify_gid(struct ib_gid_table *table,
if (!entry)
return -ENOMEM;
- if (rdma_protocol_roce(attr->device, attr->port_num)) {
+ if (rdma_port_gids_from_netdev(attr->device, attr->port_num)) {
ret = add_roce_gid(entry);
if (ret)
goto done;
@@ -1459,7 +1461,7 @@ ib_cache_update(struct ib_device *device, u32 port, bool update_gids,
goto err;
}
- if (!rdma_protocol_roce(device, port) && update_gids) {
+ if (!rdma_port_gids_from_netdev(device, port) && update_gids) {
ret = config_non_roce_gid_cache(device, port,
tprops);
if (ret)
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 337a49d1acf7..7dedf93ac465 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -293,6 +293,10 @@ int cma_set_default_gid_type(struct cma_device *cma_dev,
if (!rdma_is_port_valid(cma_dev->device, port))
return -EINVAL;
+ if (default_gid_type == IB_GID_TYPE_UET_UDP ||
+ default_gid_type == IB_GID_TYPE_UET_IP)
+ return -EINVAL;
+
if (default_gid_type == IB_GID_TYPE_IB &&
rdma_protocol_roce_eth_encap(cma_dev->device, port))
default_gid_type = IB_GID_TYPE_ROCE;
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index 29af9fe860e2..7c65a9373701 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2567,7 +2567,7 @@ void ib_enum_roce_netdev(struct ib_device *ib_dev,
u32 port;
rdma_for_each_port (ib_dev, port)
- if (rdma_protocol_roce(ib_dev, port)) {
+ if (rdma_port_gids_from_netdev(ib_dev, port)) {
struct net_device *idev =
ib_device_get_netdev(ib_dev, port);
diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c
index 849a5db54e1c..91b7be84b15f 100644
--- a/drivers/infiniband/core/nldev.c
+++ b/drivers/infiniband/core/nldev.c
@@ -350,6 +350,8 @@ static int fill_dev_info(struct sk_buff *msg, struct ib_device *device)
ret = nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_PROTOCOL, "iw");
else if (rdma_protocol_roce(device, port))
ret = nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_PROTOCOL, "roce");
+ else if (rdma_protocol_uet(device, port))
+ ret = nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_PROTOCOL, "uet");
else if (rdma_protocol_usnic(device, port))
ret = nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_PROTOCOL,
"usnic");
diff --git a/drivers/infiniband/core/roce_gid_mgmt.c b/drivers/infiniband/core/roce_gid_mgmt.c
index 4a2a4638b476..214af921341a 100644
--- a/drivers/infiniband/core/roce_gid_mgmt.c
+++ b/drivers/infiniband/core/roce_gid_mgmt.c
@@ -75,6 +75,8 @@ static const struct {
} PORT_CAP_TO_GID_TYPE[] = {
{rdma_protocol_roce_eth_encap, IB_GID_TYPE_ROCE},
{rdma_protocol_roce_udp_encap, IB_GID_TYPE_ROCE_UDP_ENCAP},
+ {rdma_protocol_uet_udp, IB_GID_TYPE_UET_UDP},
+ {rdma_protocol_uet_ip, IB_GID_TYPE_UET_IP},
};
#define CAP_TO_GID_TABLE_SIZE ARRAY_SIZE(PORT_CAP_TO_GID_TYPE)
@@ -84,7 +86,7 @@ unsigned long roce_gid_type_mask_support(struct ib_device *ib_dev, u32 port)
int i;
unsigned int ret_flags = 0;
- if (!rdma_protocol_roce(ib_dev, port))
+ if (!rdma_port_gids_from_netdev(ib_dev, port))
return 1UL << IB_GID_TYPE_IB;
for (i = 0; i < CAP_TO_GID_TABLE_SIZE; i++)
@@ -526,7 +528,7 @@ void rdma_roce_rescan_port(struct ib_device *ib_dev, u32 port)
{
struct net_device *ndev = NULL;
- if (rdma_protocol_roce(ib_dev, port)) {
+ if (rdma_port_gids_from_netdev(ib_dev, port)) {
ndev = ib_device_get_netdev(ib_dev, port);
if (!ndev)
return;
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 7d52d3036208..1299125752d8 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -862,6 +862,10 @@ int ib_init_ah_attr_from_wc(struct ib_device *device, u32 port_num,
might_sleep();
+ if (rdma_protocol_uet(device, port_num) &&
+ !rdma_protocol_roce(device, port_num))
+ return -EPROTONOSUPPORT;
+
memset(ah_attr, 0, sizeof *ah_attr);
ah_attr->type = rdma_ah_find_type(device, port_num);
if (rdma_cap_eth_ah(device, port_num)) {
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index d6eaa3fee66e..4e3a5c38cbe8 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -145,6 +145,8 @@ enum ib_gid_type {
IB_GID_TYPE_IB = IB_UVERBS_GID_TYPE_IB,
IB_GID_TYPE_ROCE = IB_UVERBS_GID_TYPE_ROCE_V1,
IB_GID_TYPE_ROCE_UDP_ENCAP = IB_UVERBS_GID_TYPE_ROCE_V2,
+ IB_GID_TYPE_UET_UDP = IB_UVERBS_GID_TYPE_UET_UDP,
+ IB_GID_TYPE_UET_IP = IB_UVERBS_GID_TYPE_UET_IP,
IB_GID_TYPE_SIZE
};
@@ -664,10 +666,16 @@ void rdma_free_hw_stats_struct(struct rdma_hw_stats *stats);
#define RDMA_CORE_CAP_PROT_ROCE_UDP_ENCAP 0x00800000
#define RDMA_CORE_CAP_PROT_RAW_PACKET 0x01000000
#define RDMA_CORE_CAP_PROT_USNIC 0x02000000
+#define RDMA_CORE_CAP_PROT_UET_UDP 0x04000000
+#define RDMA_CORE_CAP_PROT_UET_IP 0x08000000
+
+#define RDMA_CORE_CAP_PROT_UET (RDMA_CORE_CAP_PROT_UET_UDP \
+ | RDMA_CORE_CAP_PROT_UET_IP)
#define RDMA_CORE_PORT_IB_GRH_REQUIRED (RDMA_CORE_CAP_IB_GRH_REQUIRED \
| RDMA_CORE_CAP_PROT_ROCE \
- | RDMA_CORE_CAP_PROT_ROCE_UDP_ENCAP)
+ | RDMA_CORE_CAP_PROT_ROCE_UDP_ENCAP \
+ | RDMA_CORE_CAP_PROT_UET)
#define RDMA_CORE_PORT_IBA_IB (RDMA_CORE_CAP_PROT_IB \
| RDMA_CORE_CAP_IB_MAD \
@@ -691,6 +699,11 @@ void rdma_free_hw_stats_struct(struct rdma_hw_stats *stats);
#define RDMA_CORE_PORT_INTEL_OPA (RDMA_CORE_PORT_IBA_IB \
| RDMA_CORE_CAP_OPA_MAD)
+#define RDMA_CORE_PORT_UET_UDP (RDMA_CORE_CAP_PROT_UET_UDP \
+ | RDMA_CORE_CAP_ETH_AH)
+#define RDMA_CORE_PORT_UET_IP (RDMA_CORE_CAP_PROT_UET_IP \
+ | RDMA_CORE_CAP_ETH_AH)
+
#define RDMA_CORE_PORT_RAW_PACKET (RDMA_CORE_CAP_PROT_RAW_PACKET)
#define RDMA_CORE_PORT_USNIC (RDMA_CORE_CAP_PROT_USNIC)
@@ -3407,6 +3420,34 @@ static inline bool rdma_protocol_roce_eth_encap(const struct ib_device *device,
RDMA_CORE_CAP_PROT_ROCE;
}
+static inline bool rdma_protocol_uet(const struct ib_device *device,
+ u32 port_num)
+{
+ return device->port_data[port_num].immutable.core_cap_flags &
+ RDMA_CORE_CAP_PROT_UET;
+}
+
+static inline bool rdma_protocol_uet_udp(const struct ib_device *device,
+ u32 port_num)
+{
+ return device->port_data[port_num].immutable.core_cap_flags &
+ RDMA_CORE_CAP_PROT_UET_UDP;
+}
+
+static inline bool rdma_protocol_uet_ip(const struct ib_device *device,
+ u32 port_num)
+{
+ return device->port_data[port_num].immutable.core_cap_flags &
+ RDMA_CORE_CAP_PROT_UET_IP;
+}
+
+static inline bool rdma_port_gids_from_netdev(const struct ib_device *device,
+ u32 port_num)
+{
+ return rdma_protocol_roce(device, port_num) ||
+ rdma_protocol_uet(device, port_num);
+}
+
static inline bool rdma_protocol_iwarp(const struct ib_device *device,
u32 port_num)
{
@@ -3677,7 +3718,7 @@ static inline size_t rdma_max_mad_size(const struct ib_device *device,
static inline bool rdma_cap_roce_gid_table(const struct ib_device *device,
u32 port_num)
{
- return rdma_protocol_roce(device, port_num) &&
+ return rdma_port_gids_from_netdev(device, port_num) &&
device->ops.add_gid && device->ops.del_gid;
}
@@ -4984,7 +5025,8 @@ void rdma_move_ah_attr(struct rdma_ah_attr *dest, struct rdma_ah_attr *src);
static inline enum rdma_ah_attr_type rdma_ah_find_type(struct ib_device *dev,
u32 port_num)
{
- if (rdma_protocol_roce(dev, port_num))
+ if (rdma_protocol_roce(dev, port_num) ||
+ rdma_protocol_uet(dev, port_num))
return RDMA_AH_ATTR_TYPE_ROCE;
if (rdma_protocol_ib(dev, port_num)) {
if (rdma_cap_opa_ah(dev, port_num))
diff --git a/include/uapi/rdma/ib_user_ioctl_verbs.h b/include/uapi/rdma/ib_user_ioctl_verbs.h
index 21f86cc7bb1f..ed72f48960f8 100644
--- a/include/uapi/rdma/ib_user_ioctl_verbs.h
+++ b/include/uapi/rdma/ib_user_ioctl_verbs.h
@@ -263,6 +263,8 @@ enum ib_uverbs_gid_type {
IB_UVERBS_GID_TYPE_IB,
IB_UVERBS_GID_TYPE_ROCE_V1,
IB_UVERBS_GID_TYPE_ROCE_V2,
+ IB_UVERBS_GID_TYPE_UET_UDP,
+ IB_UVERBS_GID_TYPE_UET_IP,
};
struct ib_uverbs_gid_entry {
--
2.56.0
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH rdma-next v2 03/13] RDMA/core: report the reliable unconnected device capabilities
2026-10-20 3:03 [PATCH rdma-next v2 00/13] RDMA/core: Ultra Ethernet transport verbs Eric Davis via B4 Relay
2026-10-20 3:03 ` [PATCH rdma-next v2 01/13] RDMA/core: widen the uverbs object key Eric Davis via B4 Relay
2026-10-20 3:03 ` [PATCH rdma-next v2 02/13] RDMA/core: add the UET GID types Eric Davis via B4 Relay
@ 2026-10-20 3:03 ` Eric Davis via B4 Relay
2026-10-20 8:19 ` sashiko-bot
2026-10-20 3:03 ` [PATCH rdma-next v2 04/13] RDMA/core: add the reliable unconnected queue pair type Eric Davis via B4 Relay
` (9 subsequent siblings)
12 siblings, 1 reply; 27+ messages in thread
From: Eric Davis via B4 Relay @ 2026-10-20 3:03 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, linux-kernel, Eric Davis
From: Eric Davis <eric.davis@broadcom.com>
Add the capability bits a device uses to advertise reliable unconnected
support alongside additional features. The transport itself, 64-bit
immediate data, 64-bit memory keys, caller-chosen keys, and the four
memory region access restrictions.
Report the job, job key, and address table limits through the extended
query_device response so userspace can size against them.
Signed-off-by: Eric Davis <eric.davis@broadcom.com>
---
drivers/infiniband/core/uverbs_cmd.c | 3 +++
include/rdma/ib_verbs.h | 14 ++++++++++++++
include/uapi/rdma/ib_user_verbs.h | 25 +++++++++++++++++++++++++
3 files changed, 42 insertions(+)
diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index aca7c6ab55cd..ed55fbbdfc18 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -3597,6 +3597,9 @@ static int ib_uverbs_ex_query_device(struct uverbs_attr_bundle *attrs)
resp.cq_moderation_caps.max_cq_moderation_period =
attr.cq_caps.max_cq_moderation_period;
resp.max_dm_size = attr.max_dm_size;
+ resp.max_job_ids = attr.max_job_ids;
+ resp.max_job_keys = attr.max_job_keys;
+ resp.max_addr_entries = attr.max_addr_entries;
resp.response_length = uverbs_response_length(attrs, sizeof(resp));
return uverbs_response(attrs, &resp, sizeof(resp));
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 4e3a5c38cbe8..02f3312c7cf2 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -278,6 +278,14 @@ enum ib_device_cap_flags {
IB_DEVICE_FLUSH_PERSISTENT = IB_UVERBS_DEVICE_FLUSH_PERSISTENT,
IB_DEVICE_ATOMIC_WRITE = IB_UVERBS_DEVICE_ATOMIC_WRITE,
IB_DEVICE_CC_DMA_BOUNCE = IB_UVERBS_DEVICE_CC_DMA_BOUNCE,
+ IB_DEVICE_RU = IB_UVERBS_DEVICE_RU,
+ IB_DEVICE_IMM64 = IB_UVERBS_DEVICE_IMM64,
+ IB_DEVICE_KEY64 = IB_UVERBS_DEVICE_KEY64,
+ IB_DEVICE_USER_RKEY = IB_UVERBS_DEVICE_USER_RKEY,
+ IB_DEVICE_MR_UNRESTRICTED = IB_UVERBS_DEVICE_MR_UNRESTRICTED,
+ IB_DEVICE_MR_JOB_RESTRICTED = IB_UVERBS_DEVICE_MR_JOB_RESTRICTED,
+ IB_DEVICE_MR_QP_RESTRICTED = IB_UVERBS_DEVICE_MR_QP_RESTRICTED,
+ IB_DEVICE_MR_QP_JOB_RESTRICTED = IB_UVERBS_DEVICE_MR_QP_JOB_RESTRICTED,
};
enum ib_kernel_cap_flags {
@@ -456,6 +464,12 @@ struct ib_device_attr {
u64 max_dm_size;
/* Max entries for sgl for optimized performance per READ */
u32 max_sgl_rd;
+ /* Jobs allocated at once, not the largest JobID */
+ u32 max_job_ids;
+ /* Job keys allocated at once */
+ u32 max_job_keys;
+ /* Largest address table a single job may have */
+ u32 max_addr_entries;
};
enum ib_mtu {
diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_verbs.h
index 565301594634..bae6fcfa01dd 100644
--- a/include/uapi/rdma/ib_user_verbs.h
+++ b/include/uapi/rdma/ib_user_verbs.h
@@ -300,6 +300,10 @@ struct ib_uverbs_ex_query_device_resp {
__aligned_u64 max_dm_size;
__u32 xrc_odp_caps;
__u32 reserved;
+ __u32 max_job_ids; /* jobs allocated at once, not a JobID */
+ __u32 max_job_keys; /* job keys allocated at once */
+ __u32 max_addr_entries; /* largest address table of one job */
+ __u32 reserved2;
};
struct ib_uverbs_query_port {
@@ -1370,6 +1374,27 @@ enum ib_uverbs_device_cap_flags {
IB_UVERBS_DEVICE_ATOMIC_WRITE = 1ULL << 40,
/* CoCo guest with DMA bounce buffering required */
IB_UVERBS_DEVICE_CC_DMA_BOUNCE = 1ULL << 41,
+ /* Reliable unconnected QPs, jobs, job keys and address tables */
+ IB_UVERBS_DEVICE_RU = 1ULL << 42,
+ /* 64-bit immediate data */
+ IB_UVERBS_DEVICE_IMM64 = 1ULL << 43,
+ /* An MR may carry 64-bit keys as a second name beside its 32-bit ones */
+ IB_UVERBS_DEVICE_KEY64 = 1ULL << 44,
+ /* MR registration with a caller-chosen rkey */
+ IB_UVERBS_DEVICE_USER_RKEY = 1ULL << 45,
+ /*
+ * MR access restrictions, meaningful only with IB_UVERBS_DEVICE_RU.
+ * Each names what an incoming request must match, and each is its own
+ * match: supporting two does not imply their combination.
+ */
+ /* PD: any QP */
+ IB_UVERBS_DEVICE_MR_UNRESTRICTED = 1ULL << 46,
+ /* PD and JobID: any QP, one job */
+ IB_UVERBS_DEVICE_MR_JOB_RESTRICTED = 1ULL << 47,
+ /* PD and QP: the QP the MR is attached to */
+ IB_UVERBS_DEVICE_MR_QP_RESTRICTED = 1ULL << 48,
+ /* PD, QP and JobID: the attached QP, one job */
+ IB_UVERBS_DEVICE_MR_QP_JOB_RESTRICTED = 1ULL << 49,
};
enum ib_uverbs_raw_packet_caps {
--
2.56.0
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH rdma-next v2 04/13] RDMA/core: add the reliable unconnected queue pair type
2026-10-20 3:03 [PATCH rdma-next v2 00/13] RDMA/core: Ultra Ethernet transport verbs Eric Davis via B4 Relay
` (2 preceding siblings ...)
2026-10-20 3:03 ` [PATCH rdma-next v2 03/13] RDMA/core: report the reliable unconnected device capabilities Eric Davis via B4 Relay
@ 2026-10-20 3:03 ` Eric Davis via B4 Relay
2026-10-20 8:28 ` sashiko-bot
2026-10-20 3:03 ` [PATCH rdma-next v2 05/13] RDMA/core: extended address handle attributes Eric Davis via B4 Relay
` (8 subsequent siblings)
12 siblings, 1 reply; 27+ messages in thread
From: Eric Davis via B4 Relay @ 2026-10-20 3:03 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, linux-kernel, Eric Davis
From: Eric Davis <eric.davis@broadcom.com>
Add IB_QPT_RU and its userspace counterpart, together with the field
layout of a reliable unconnected queue pair number used by UET.
Such a number uses all 32 bits, unlike the 24-bit InfiniBand and RoCE
one, so exempt it from the restrack assertion that no queue pair number
sets the top byte.
Signed-off-by: Eric Davis <eric.davis@broadcom.com>
---
drivers/infiniband/core/restrack.c | 13 ++++++++++++-
drivers/infiniband/core/uverbs_cmd.c | 14 ++++++++++++++
include/rdma/ib_verbs.h | 1 +
include/uapi/rdma/ib_user_ioctl_verbs.h | 16 ++++++++++++++++
4 files changed, 43 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/core/restrack.c b/drivers/infiniband/core/restrack.c
index a8b838fdc7b5..ddb09b17b733 100644
--- a/drivers/infiniband/core/restrack.c
+++ b/drivers/infiniband/core/restrack.c
@@ -241,7 +241,18 @@ void rdma_restrack_add(struct rdma_restrack_entry *res)
/* Special case to ensure that LQPN points to right QP */
struct ib_qp *qp = container_of(res, struct ib_qp, res);
- WARN_ONCE(qp->qp_num >> 24 || qp->port >> 8,
+ /* An InfiniBand/RoCE QP number is 24 bits and the top byte is
+ * where the port is folded in for SMI and GSI. A reliable
+ * unconnected pair is an exception as its QP number is a
+ * full 32b wide, though of the top byte only the absolute bit
+ * may be set. An absolute number whose low bits are 0 or 1
+ * still conflicts SMI/GSI on port 128, which takes a device
+ * with both RU and MADs on that many ports.
+ */
+ WARN_ONCE(qp->port >> 8 ||
+ (qp->qp_type != IB_QPT_RU && qp->qp_num >> 24) ||
+ (qp->qp_type == IB_QPT_RU &&
+ qp->qp_num & IB_UVERBS_QPN_UET_RESERVED_MASK),
"QP number 0x%0X and port 0x%0X", qp->qp_num,
qp->port);
res->id = qp->qp_num;
diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index ed55fbbdfc18..e4fdfa0d9548 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -2058,6 +2058,12 @@ static int ib_uverbs_post_send(struct uverbs_attr_bundle *attrs)
goto out;
}
+ /* reliable unconnected work requests are posted by the provider */
+ if (qp->qp_type == IB_QPT_RU) {
+ ret = -EOPNOTSUPP;
+ goto out_put;
+ }
+
is_ud = qp->qp_type == IB_QPT_UD;
sg_ind = 0;
last = NULL;
@@ -2335,6 +2341,14 @@ static int ib_uverbs_post_recv(struct uverbs_attr_bundle *attrs)
goto out;
}
+ /* reliable unconnected work requests are posted by the provider */
+ if (qp->qp_type == IB_QPT_RU) {
+ rdma_lookup_put_uobject(&qp->uobject->uevent.uobject,
+ UVERBS_LOOKUP_READ);
+ ret = -EOPNOTSUPP;
+ goto out;
+ }
+
resp.bad_wr = 0;
ret = qp->device->ops.post_recv(qp->real_qp, wr, &bad_wr);
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 02f3312c7cf2..8d74ed4fc581 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1188,6 +1188,7 @@ enum ib_qp_type {
IB_QPT_RAW_PACKET = IB_UVERBS_QPT_RAW_PACKET,
IB_QPT_XRC_INI = IB_UVERBS_QPT_XRC_INI,
IB_QPT_XRC_TGT = IB_UVERBS_QPT_XRC_TGT,
+ IB_QPT_RU = IB_UVERBS_QPT_RU,
IB_QPT_MAX,
IB_QPT_DRIVER = IB_UVERBS_QPT_DRIVER,
/* Reserve a range for qp types internal to the low level driver.
diff --git a/include/uapi/rdma/ib_user_ioctl_verbs.h b/include/uapi/rdma/ib_user_ioctl_verbs.h
index ed72f48960f8..b6d46115f799 100644
--- a/include/uapi/rdma/ib_user_ioctl_verbs.h
+++ b/include/uapi/rdma/ib_user_ioctl_verbs.h
@@ -91,6 +91,7 @@ enum ib_uverbs_qp_type {
IB_UVERBS_QPT_RAW_PACKET = 8,
IB_UVERBS_QPT_XRC_INI,
IB_UVERBS_QPT_XRC_TGT,
+ IB_UVERBS_QPT_RU,
IB_UVERBS_QPT_DRIVER = 0xFF,
};
@@ -302,6 +303,21 @@ struct ib_uverbs_buffer_desc {
__aligned_u64 length;
};
+/* A UET reliable unconnected queue pair number uses all 32 bits:
+ *
+ * [ absolute 1b | reserved 7b | resource index 12b | PIDonFEP 12b ]
+ *
+ * An absolute pair is named by its number alone. A relative one is named
+ * within a job, so it must carry a job key when it is created. The reserved
+ * bits are zero.
+ */
+#define IB_UVERBS_QPN_UET_PID_ON_FEP_MASK 0x00000fff
+#define IB_UVERBS_QPN_UET_PID_ON_FEP_SHIFT 0
+#define IB_UVERBS_QPN_UET_RI_MASK 0x00fff000
+#define IB_UVERBS_QPN_UET_RI_SHIFT 12
+#define IB_UVERBS_QPN_UET_RESERVED_MASK 0x7f000000
+#define IB_UVERBS_QPN_UET_ABSOLUTE_ADDR_BIT 0x80000000
+
enum ib_uverbs_comp_cntr_entry {
IB_UVERBS_COMP_CNTR_ENTRY_COMP,
IB_UVERBS_COMP_CNTR_ENTRY_ERR,
--
2.56.0
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH rdma-next v2 05/13] RDMA/core: extended address handle attributes
2026-10-20 3:03 [PATCH rdma-next v2 00/13] RDMA/core: Ultra Ethernet transport verbs Eric Davis via B4 Relay
` (3 preceding siblings ...)
2026-10-20 3:03 ` [PATCH rdma-next v2 04/13] RDMA/core: add the reliable unconnected queue pair type Eric Davis via B4 Relay
@ 2026-10-20 3:03 ` Eric Davis via B4 Relay
2026-10-20 8:40 ` sashiko-bot
2026-10-20 3:03 ` [PATCH rdma-next v2 06/13] RDMA/core: query the semantics a queue pair type guarantees Eric Davis via B4 Relay
` (7 subsequent siblings)
12 siblings, 1 reply; 27+ messages in thread
From: Eric Davis via B4 Relay @ 2026-10-20 3:03 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, linux-kernel, Eric Davis
From: Eric Davis <eric.davis@broadcom.com>
An unconnected transport over UET resolves a peer by both its network
address and its queue pair number. The write() create-AH command has no
room for additional attributes. Add rdma_ah_attr_ex to carry new
attributes, reach it through rdma_ah_init_attr, and add an ioctl method
that supports it.
rdma_create_user_ah() now takes the init attributes rather than the
address alone, so callers pass the wider set.
Signed-off-by: Eric Davis <eric.davis@broadcom.com>
---
drivers/infiniband/core/user_mad.c | 4 +-
drivers/infiniband/core/uverbs.h | 3 ++
drivers/infiniband/core/uverbs_cmd.c | 60 +++++++++++++++----------
drivers/infiniband/core/uverbs_std_types.c | 70 +++++++++++++++++++++++++++++-
drivers/infiniband/core/verbs.c | 43 ++++++++++--------
include/rdma/ib_verbs.h | 20 +++++++--
include/uapi/rdma/ib_user_ioctl_cmds.h | 7 +++
include/uapi/rdma/ib_user_ioctl_verbs.h | 9 ++++
8 files changed, 169 insertions(+), 47 deletions(-)
diff --git a/drivers/infiniband/core/user_mad.c b/drivers/infiniband/core/user_mad.c
index c3ef9f85c7eb..05e2898f0a6a 100644
--- a/drivers/infiniband/core/user_mad.c
+++ b/drivers/infiniband/core/user_mad.c
@@ -512,6 +512,7 @@ static ssize_t ib_umad_write(struct file *filp, const char __user *buf,
struct ib_umad_packet *packet;
struct ib_mad_agent *agent;
struct rdma_ah_attr ah_attr;
+ struct rdma_ah_init_attr init_attr = {};
struct ib_ah *ah;
__be64 *tid;
int ret, hdr_len, copy_offset, rmpp_active;
@@ -569,7 +570,8 @@ static ssize_t ib_umad_write(struct file *filp, const char __user *buf,
rdma_ah_set_dgid_raw(&ah_attr, packet->mad.hdr.gid);
}
- ah = rdma_create_user_ah(agent->qp->pd, &ah_attr, NULL);
+ init_attr.ah_attr = &ah_attr;
+ ah = rdma_create_user_ah(agent->qp->pd, &init_attr, NULL);
if (IS_ERR(ah)) {
ret = PTR_ERR(ah);
goto err_up;
diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h
index 728e09b41992..6aaac1c309f0 100644
--- a/drivers/infiniband/core/uverbs.h
+++ b/drivers/infiniband/core/uverbs.h
@@ -236,6 +236,9 @@ int ib_uverbs_dealloc_xrcd(struct ib_uobject *uobject, struct ib_xrcd *xrcd,
int uverbs_dealloc_mw(struct ib_mw *mw);
void ib_uverbs_detach_umcast(struct ib_qp *qp,
struct ib_uqp_object *uobj);
+int copy_ah_attr_from_uverbs_ah(struct ib_device *dev,
+ struct rdma_ah_attr *rdma_attr,
+ const struct ib_uverbs_ah_attr *uverb_attr);
struct bundle_alloc_head {
struct_group_tagged(bundle_alloc_head_hdr, hdr,
diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index e4fdfa0d9548..cca386b2c3f0 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -1746,6 +1746,37 @@ static int modify_qp_mask(enum ib_qp_type qp_type, int mask)
}
}
+/* The ib_uverbs_ah_attr counterpart of copy_ah_attr_from_uverbs() below,
+ * which takes the modify QP layout. This one also rejects an invalid port.
+ */
+int copy_ah_attr_from_uverbs_ah(struct ib_device *dev,
+ struct rdma_ah_attr *rdma_attr,
+ const struct ib_uverbs_ah_attr *uverb_attr)
+{
+ if (!rdma_is_port_valid(dev, uverb_attr->port_num))
+ return -EINVAL;
+
+ rdma_attr->type = rdma_ah_find_type(dev, uverb_attr->port_num);
+ rdma_ah_set_make_grd(rdma_attr, false);
+ rdma_ah_set_dlid(rdma_attr, uverb_attr->dlid);
+ rdma_ah_set_sl(rdma_attr, uverb_attr->sl);
+ rdma_ah_set_path_bits(rdma_attr, uverb_attr->src_path_bits);
+ rdma_ah_set_static_rate(rdma_attr, uverb_attr->static_rate);
+ rdma_ah_set_port_num(rdma_attr, uverb_attr->port_num);
+
+ if (uverb_attr->is_global) {
+ rdma_ah_set_grh(rdma_attr, NULL, uverb_attr->grh.flow_label,
+ uverb_attr->grh.sgid_index,
+ uverb_attr->grh.hop_limit,
+ uverb_attr->grh.traffic_class);
+ rdma_ah_set_dgid_raw(rdma_attr, uverb_attr->grh.dgid);
+ } else {
+ rdma_ah_set_ah_flags(rdma_attr, 0);
+ }
+
+ return 0;
+}
+
static void copy_ah_attr_from_uverbs(struct ib_device *dev,
struct rdma_ah_attr *rdma_attr,
struct ib_uverbs_qp_dest *uverb_attr)
@@ -2435,6 +2466,7 @@ static int ib_uverbs_create_ah(struct uverbs_attr_bundle *attrs)
struct ib_pd *pd;
struct ib_ah *ah;
struct rdma_ah_attr attr = {};
+ struct rdma_ah_init_attr init_attr = {};
int ret;
struct ib_device *ib_dev;
@@ -2446,36 +2478,18 @@ static int ib_uverbs_create_ah(struct uverbs_attr_bundle *attrs)
if (IS_ERR(uobj))
return PTR_ERR(uobj);
- if (!rdma_is_port_valid(ib_dev, cmd.attr.port_num)) {
- ret = -EINVAL;
- goto err;
- }
-
pd = uobj_get_obj_read(pd, UVERBS_OBJECT_PD, cmd.pd_handle, attrs);
if (IS_ERR(pd)) {
ret = PTR_ERR(pd);
goto err;
}
- attr.type = rdma_ah_find_type(ib_dev, cmd.attr.port_num);
- rdma_ah_set_make_grd(&attr, false);
- rdma_ah_set_dlid(&attr, cmd.attr.dlid);
- rdma_ah_set_sl(&attr, cmd.attr.sl);
- rdma_ah_set_path_bits(&attr, cmd.attr.src_path_bits);
- rdma_ah_set_static_rate(&attr, cmd.attr.static_rate);
- rdma_ah_set_port_num(&attr, cmd.attr.port_num);
-
- if (cmd.attr.is_global) {
- rdma_ah_set_grh(&attr, NULL, cmd.attr.grh.flow_label,
- cmd.attr.grh.sgid_index,
- cmd.attr.grh.hop_limit,
- cmd.attr.grh.traffic_class);
- rdma_ah_set_dgid_raw(&attr, cmd.attr.grh.dgid);
- } else {
- rdma_ah_set_ah_flags(&attr, 0);
- }
+ ret = copy_ah_attr_from_uverbs_ah(ib_dev, &attr, &cmd.attr);
+ if (ret)
+ goto err_put;
- ah = rdma_create_user_ah(pd, &attr, &attrs->driver_udata);
+ init_attr.ah_attr = &attr;
+ ah = rdma_create_user_ah(pd, &init_attr, &attrs->driver_udata);
if (IS_ERR(ah)) {
ret = PTR_ERR(ah);
goto err_put;
diff --git a/drivers/infiniband/core/uverbs_std_types.c b/drivers/infiniband/core/uverbs_std_types.c
index e160786e1df1..27cc1d162293 100644
--- a/drivers/infiniband/core/uverbs_std_types.c
+++ b/drivers/infiniband/core/uverbs_std_types.c
@@ -184,6 +184,73 @@ DECLARE_UVERBS_NAMED_OBJECT(UVERBS_OBJECT_MW,
UVERBS_TYPE_ALLOC_IDR(uverbs_free_mw),
&UVERBS_METHOD(UVERBS_METHOD_MW_DESTROY));
+/* Creating an address handle is still a write() command whose request
+ * struct cannot grow. A transport (reliable unconnected) that resolves a
+ * peer using both transport and network addresses requires additional
+ * fields in the address handle. Here the address handle create command is
+ * made available as an ioctl method, where additional attributes can be
+ * carried.
+ */
+static int UVERBS_HANDLER(UVERBS_METHOD_AH_EX_CREATE)(
+ struct uverbs_attr_bundle *attrs)
+{
+ struct ib_uobject *uobj = uverbs_attr_get_uobject(
+ attrs, UVERBS_ATTR_CREATE_AH_EX_HANDLE);
+ struct ib_pd *pd =
+ uverbs_attr_get_obj(attrs, UVERBS_ATTR_CREATE_AH_EX_PD_HANDLE);
+ struct ib_device *ib_dev = attrs->context->device;
+ struct ib_uverbs_ah_attr_ex uah;
+ struct rdma_ah_init_attr init_attr = {};
+ struct rdma_ah_attr_ex attr = {};
+ struct ib_ah *ah;
+ int ret;
+
+ /* Only a reliable unconnected device has a use for the extended
+ * attributes.
+ */
+ if (!(ib_dev->attrs.device_cap_flags & IB_DEVICE_RU))
+ return -EOPNOTSUPP;
+
+ ret = uverbs_copy_from_or_zero(&uah, attrs,
+ UVERBS_ATTR_CREATE_AH_EX_AH_ATTR_EX);
+ if (ret)
+ return ret;
+
+ ret = copy_ah_attr_from_uverbs_ah(ib_dev, &attr.ah_attr, &uah.ah_attr);
+ if (ret)
+ return ret;
+
+ attr.remote_qpn = uah.remote_qpn;
+
+ init_attr.ah_attr = &attr.ah_attr;
+ init_attr.ah_attr_ex = &attr;
+
+ ah = rdma_create_user_ah(pd, &init_attr, &attrs->driver_udata);
+ if (IS_ERR(ah))
+ return PTR_ERR(ah);
+
+ ah->uobject = uobj;
+ uobj->object = ah;
+
+ return 0;
+}
+
+DECLARE_UVERBS_NAMED_METHOD(
+ UVERBS_METHOD_AH_EX_CREATE,
+ UVERBS_ATTR_IDR(UVERBS_ATTR_CREATE_AH_EX_HANDLE,
+ UVERBS_OBJECT_AH,
+ UVERBS_ACCESS_NEW,
+ UA_MANDATORY),
+ UVERBS_ATTR_IDR(UVERBS_ATTR_CREATE_AH_EX_PD_HANDLE,
+ UVERBS_OBJECT_PD,
+ UVERBS_ACCESS_READ,
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_IN(UVERBS_ATTR_CREATE_AH_EX_AH_ATTR_EX,
+ UVERBS_ATTR_STRUCT(struct ib_uverbs_ah_attr_ex,
+ remote_qpn),
+ UA_MANDATORY),
+ UVERBS_ATTR_UHW());
+
DECLARE_UVERBS_NAMED_METHOD_DESTROY(
UVERBS_METHOD_AH_DESTROY,
UVERBS_ATTR_IDR(UVERBS_ATTR_DESTROY_AH_HANDLE,
@@ -193,7 +260,8 @@ DECLARE_UVERBS_NAMED_METHOD_DESTROY(
DECLARE_UVERBS_NAMED_OBJECT(UVERBS_OBJECT_AH,
UVERBS_TYPE_ALLOC_IDR(uverbs_free_ah),
- &UVERBS_METHOD(UVERBS_METHOD_AH_DESTROY));
+ &UVERBS_METHOD(UVERBS_METHOD_AH_DESTROY),
+ &UVERBS_METHOD(UVERBS_METHOD_AH_EX_CREATE));
DECLARE_UVERBS_NAMED_METHOD_DESTROY(
UVERBS_METHOD_FLOW_DESTROY,
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 1299125752d8..2d04ffd13ad3 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -561,24 +561,23 @@ rdma_update_sgid_attr(struct rdma_ah_attr *ah_attr,
}
static struct ib_ah *_rdma_create_ah(struct ib_pd *pd,
- struct rdma_ah_attr *ah_attr,
- u32 flags,
- struct ib_udata *udata,
- struct net_device *xmit_slave)
+ struct rdma_ah_init_attr *init_attr,
+ struct ib_udata *udata)
{
- struct rdma_ah_init_attr init_attr = {};
+ struct rdma_ah_attr *ah_attr = init_attr->ah_attr;
struct ib_device *device = pd->device;
struct ib_ah *ah;
int ret;
- might_sleep_if(flags & RDMA_CREATE_AH_SLEEPABLE);
+ might_sleep_if(init_attr->flags & RDMA_CREATE_AH_SLEEPABLE);
if (!udata && !device->ops.create_ah)
return ERR_PTR(-EOPNOTSUPP);
ah = rdma_zalloc_drv_obj_gfp(
device, ib_ah,
- (flags & RDMA_CREATE_AH_SLEEPABLE) ? GFP_KERNEL : GFP_ATOMIC);
+ (init_attr->flags & RDMA_CREATE_AH_SLEEPABLE) ? GFP_KERNEL :
+ GFP_ATOMIC);
if (!ah)
return ERR_PTR(-ENOMEM);
@@ -586,14 +585,12 @@ static struct ib_ah *_rdma_create_ah(struct ib_pd *pd,
ah->pd = pd;
ah->type = ah_attr->type;
ah->sgid_attr = rdma_update_sgid_attr(ah_attr, NULL);
- init_attr.ah_attr = ah_attr;
- init_attr.flags = flags;
- init_attr.xmit_slave = xmit_slave;
if (udata)
- ret = device->ops.create_user_ah(ah, &init_attr, udata);
+ ret = device->ops.create_user_ah(ah, init_attr, udata);
else
- ret = device->ops.create_ah(ah, &init_attr, NULL);
+ ret = device->ops.create_ah(ah, init_attr, NULL);
+
if (ret) {
if (ah->sgid_attr)
rdma_put_gid_attr(ah->sgid_attr);
@@ -619,6 +616,7 @@ static struct ib_ah *_rdma_create_ah(struct ib_pd *pd,
struct ib_ah *rdma_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr,
u32 flags)
{
+ struct rdma_ah_init_attr init_attr = {};
const struct ib_gid_attr *old_sgid_attr;
struct net_device *slave;
struct ib_ah *ah;
@@ -634,7 +632,12 @@ struct ib_ah *rdma_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr,
rdma_unfill_sgid_attr(ah_attr, old_sgid_attr);
return ERR_CAST(slave);
}
- ah = _rdma_create_ah(pd, ah_attr, flags, NULL, slave);
+
+ init_attr.ah_attr = ah_attr;
+ init_attr.flags = flags;
+ init_attr.xmit_slave = slave;
+
+ ah = _rdma_create_ah(pd, &init_attr, NULL);
rdma_lag_put_ah_roce_slave(slave);
rdma_unfill_sgid_attr(ah_attr, old_sgid_attr);
return ah;
@@ -646,18 +649,21 @@ EXPORT_SYMBOL(rdma_create_ah);
* given address vector.
* It resolves destination mac address for ah attribute of RoCE type.
* @pd: The protection domain associated with the address handle.
- * @ah_attr: The attributes of the address vector.
+ * @init_attr: The address vector, and for an extended handle the
+ * attributes beyond it.
* @udata: pointer to user's input output buffer information need by
* provider driver.
*
- * It returns 0 on success and returns appropriate error code on error.
* The address handle is used to reference a local or global destination
* in all UD QP post sends.
+ *
+ * Return: the address handle, or an ERR_PTR on error.
*/
struct ib_ah *rdma_create_user_ah(struct ib_pd *pd,
- struct rdma_ah_attr *ah_attr,
+ struct rdma_ah_init_attr *init_attr,
struct ib_udata *udata)
{
+ struct rdma_ah_attr *ah_attr = init_attr->ah_attr;
const struct ib_gid_attr *old_sgid_attr;
struct ib_ah *ah;
int err;
@@ -674,8 +680,9 @@ struct ib_ah *rdma_create_user_ah(struct ib_pd *pd,
}
}
- ah = _rdma_create_ah(pd, ah_attr, RDMA_CREATE_AH_SLEEPABLE,
- udata, NULL);
+ init_attr->flags |= RDMA_CREATE_AH_SLEEPABLE;
+
+ ah = _rdma_create_ah(pd, init_attr, udata);
out:
rdma_unfill_sgid_attr(ah_attr, old_sgid_attr);
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 8d74ed4fc581..68f7a271cf7a 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -974,10 +974,14 @@ struct ib_mr_status {
*/
__attribute_const__ enum ib_rate mult_to_ib_rate(int mult);
+struct rdma_ah_attr_ex;
+
struct rdma_ah_init_attr {
struct rdma_ah_attr *ah_attr;
u32 flags;
struct net_device *xmit_slave;
+ /* set to name a peer by both transport+network addresses */
+ struct rdma_ah_attr_ex *ah_attr_ex;
};
enum rdma_ah_attr_type {
@@ -1016,6 +1020,11 @@ struct rdma_ah_attr {
};
};
+struct rdma_ah_attr_ex {
+ struct rdma_ah_attr ah_attr;
+ u32 remote_qpn;
+};
+
enum ib_wc_status {
IB_WC_SUCCESS,
IB_WC_LOC_LEN_ERR,
@@ -3893,16 +3902,18 @@ struct ib_ah *rdma_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr,
* rdma_create_user_ah - Creates an address handle for the given address vector.
* It resolves destination mac address for ah attribute of RoCE type.
* @pd: The protection domain associated with the address handle.
- * @ah_attr: The attributes of the address vector.
+ * @init_attr: The address vector, and for an extended handle the
+ * attributes beyond it.
* @udata: pointer to user's input output buffer information need by
* provider driver.
*
- * It returns 0 on success and returns appropriate error code on error.
* The address handle is used to reference a local or global destination
* in all UD QP post sends.
+ *
+ * Return: the address handle, or an ERR_PTR on error.
*/
struct ib_ah *rdma_create_user_ah(struct ib_pd *pd,
- struct rdma_ah_attr *ah_attr,
+ struct rdma_ah_init_attr *init_attr,
struct ib_udata *udata);
/**
* ib_get_gids_from_rdma_hdr - Get sgid and dgid from GRH or IPv4 header
@@ -4981,7 +4992,8 @@ static inline struct ib_global_route
return &attr->grh;
}
-static inline void rdma_ah_set_dgid_raw(struct rdma_ah_attr *attr, void *dgid)
+static inline void rdma_ah_set_dgid_raw(struct rdma_ah_attr *attr,
+ const void *dgid)
{
struct ib_global_route *grh = rdma_ah_retrieve_grh(attr);
diff --git a/include/uapi/rdma/ib_user_ioctl_cmds.h b/include/uapi/rdma/ib_user_ioctl_cmds.h
index 816b3107464f..40fcd4bd1b5e 100644
--- a/include/uapi/rdma/ib_user_ioctl_cmds.h
+++ b/include/uapi/rdma/ib_user_ioctl_cmds.h
@@ -410,6 +410,13 @@ enum uverbs_attrs_xrcd_destroy_ids {
enum uverbs_methods_ah {
UVERBS_METHOD_AH_DESTROY,
+ UVERBS_METHOD_AH_EX_CREATE,
+};
+
+enum uverbs_attrs_create_ah_ex_cmd_attr_ids {
+ UVERBS_ATTR_CREATE_AH_EX_HANDLE,
+ UVERBS_ATTR_CREATE_AH_EX_PD_HANDLE,
+ UVERBS_ATTR_CREATE_AH_EX_AH_ATTR_EX,
};
enum uverbs_attrs_ah_destroy_ids {
diff --git a/include/uapi/rdma/ib_user_ioctl_verbs.h b/include/uapi/rdma/ib_user_ioctl_verbs.h
index b6d46115f799..210f3f945bd7 100644
--- a/include/uapi/rdma/ib_user_ioctl_verbs.h
+++ b/include/uapi/rdma/ib_user_ioctl_verbs.h
@@ -318,6 +318,15 @@ struct ib_uverbs_buffer_desc {
#define IB_UVERBS_QPN_UET_RESERVED_MASK 0x7f000000
#define IB_UVERBS_QPN_UET_ABSOLUTE_ADDR_BIT 0x80000000
+/* An address handle that names a peer using both transport+network addresses.
+ * Used for reliable unconnected QPs in both work request address handles and
+ * a Job address table.
+ */
+struct ib_uverbs_ah_attr_ex {
+ struct ib_uverbs_ah_attr ah_attr;
+ __u32 remote_qpn;
+};
+
enum ib_uverbs_comp_cntr_entry {
IB_UVERBS_COMP_CNTR_ENTRY_COMP,
IB_UVERBS_COMP_CNTR_ENTRY_ERR,
--
2.56.0
^ permalink raw reply related [flat|nested] 27+ messages in thread* Re: [PATCH rdma-next v2 05/13] RDMA/core: extended address handle attributes
2026-10-20 3:03 ` [PATCH rdma-next v2 05/13] RDMA/core: extended address handle attributes Eric Davis via B4 Relay
@ 2026-10-20 8:40 ` sashiko-bot
0 siblings, 0 replies; 27+ messages in thread
From: sashiko-bot @ 2026-10-20 8:40 UTC (permalink / raw)
To: Eric Davis; +Cc: linux-rdma
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] RDMA/uverbs: missing uverbs_finalize_uobj_create() in AH_EX_CREATE handler
--
commit 4cd859f734837412850e0706a926ea891c66f8a6
Author: Eric Davis <eric.davis@broadcom.com>
RDMA/core: extended address handle attributes
This commit introduces struct rdma_ah_attr_ex to carry extended address
handle attributes such as remote_qpn for UET transports, updates
rdma_create_user_ah() to accept struct rdma_ah_init_attr, and adds the
UVERBS_METHOD_AH_EX_CREATE ioctl method.
> diff --git a/drivers/infiniband/core/uverbs_std_types.c b/drivers/infiniband/core/uverbs_std_types.c
> index e160786e1df16..27cc1d1622939 100644
> --- a/drivers/infiniband/core/uverbs_std_types.c
> +++ b/drivers/infiniband/core/uverbs_std_types.c
> @@ -184,6 +184,73 @@ DECLARE_UVERBS_NAMED_OBJECT(UVERBS_OBJECT_MW,
> UVERBS_TYPE_ALLOC_IDR(uverbs_free_mw),
> &UVERBS_METHOD(UVERBS_METHOD_MW_DESTROY));
>
> +/* Creating an address handle is still a write() command whose request
> + * struct cannot grow. A transport (reliable unconnected) that resolves a
> + * peer using both transport and network addresses requires additional
> + * fields in the address handle. Here the address handle create command is
> + * made available as an ioctl method, where additional attributes can be
> + * carried.
> + */
> +static int UVERBS_HANDLER(UVERBS_METHOD_AH_EX_CREATE)(
> + struct uverbs_attr_bundle *attrs)
> +{
[ ... ]
> + ah = rdma_create_user_ah(pd, &init_attr, &attrs->driver_udata);
> + if (IS_ERR(ah))
> + return PTR_ERR(ah);
> +
> + ah->uobject = uobj;
> + uobj->object = ah;
> +
> + return 0;
> +}
[Severity: High]
Should UVERBS_HANDLER(UVERBS_METHOD_AH_EX_CREATE) call
uverbs_finalize_uobj_create(attrs, UVERBS_ATTR_CREATE_AH_EX_HANDLE) before
returning 0?
Because UVERBS_METHOD_AH_EX_CREATE includes UVERBS_ATTR_UHW(),
ib_uverbs_run_method() can call uverbs_set_output() for UVERBS_ATTR_UHW_OUT
after the handler succeeds:
drivers/infiniband/core/uverbs_ioctl.c:ib_uverbs_run_method() {
ret = handler(bundle);
if (ret)
return ret;
if (!ret && bundle->method_elm->has_udata) {
const struct uverbs_attr *attr =
uverbs_attr_get(bundle, UVERBS_ATTR_UHW_OUT);
if (!IS_ERR(attr))
ret = uverbs_set_output(bundle, attr);
}
}
If uverbs_set_output() fails with -EFAULT, ib_uverbs_cmd_verbs() aborts the
command via bundle_destroy(pbundle, false) -> uverbs_finalize_object() ->
rdma_alloc_abort_uobject():
drivers/infiniband/core/rdma_core.c:rdma_alloc_abort_uobject() {
if (hw_obj_valid) {
uverbs_get_cleared_udata(attrs);
ret = uobj->uapi_object->type_class->destroy_hw(
uobj, RDMA_REMOVE_ABORT, attrs);
...
}
}
Without uverbs_finalize_uobj_create() setting the bit in
pbundle->uobj_hw_obj_valid, hw_obj_valid is false on abort, so destroy_hw
(uverbs_free_ah() -> rdma_destroy_ah_user()) is skipped. Does this leak the
newly created struct ib_ah, its ah->sgid_attr reference, and its pd->usecnt
reference?
[ ... ]
> +DECLARE_UVERBS_NAMED_METHOD(
> + UVERBS_METHOD_AH_EX_CREATE,
> + UVERBS_ATTR_IDR(UVERBS_ATTR_CREATE_AH_EX_HANDLE,
> + UVERBS_OBJECT_AH,
> + UVERBS_ACCESS_NEW,
> + UA_MANDATORY),
> + UVERBS_ATTR_IDR(UVERBS_ATTR_CREATE_AH_EX_PD_HANDLE,
> + UVERBS_OBJECT_PD,
> + UVERBS_ACCESS_READ,
> + UA_MANDATORY),
> + UVERBS_ATTR_PTR_IN(UVERBS_ATTR_CREATE_AH_EX_AH_ATTR_EX,
> + UVERBS_ATTR_STRUCT(struct ib_uverbs_ah_attr_ex,
> + remote_qpn),
> + UA_MANDATORY),
> + UVERBS_ATTR_UHW());
[ ... ]
> --
> 2.51.2
>
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019-uet-ib-core-v2-0-8bd654f25370@broadcom.com?part=5
^ permalink raw reply [flat|nested] 27+ messages in thread
* [PATCH rdma-next v2 06/13] RDMA/core: query the semantics a queue pair type guarantees
2026-10-20 3:03 [PATCH rdma-next v2 00/13] RDMA/core: Ultra Ethernet transport verbs Eric Davis via B4 Relay
` (4 preceding siblings ...)
2026-10-20 3:03 ` [PATCH rdma-next v2 05/13] RDMA/core: extended address handle attributes Eric Davis via B4 Relay
@ 2026-10-20 3:03 ` Eric Davis via B4 Relay
2026-10-20 8:45 ` sashiko-bot
2026-10-20 3:03 ` [PATCH rdma-next v2 07/13] RDMA/core: job and job key objects Eric Davis via B4 Relay
` (6 subsequent siblings)
12 siblings, 1 reply; 27+ messages in thread
From: Eric Davis via B4 Relay @ 2026-10-20 3:03 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, linux-kernel, Eric Davis
From: Eric Davis <eric.davis@broadcom.com>
An unconnected transport orders far less than a connected one, and how
much varies by device. Add a device method that reports, for a queue pair
type, which operations it orders against which, the largest packet it
puts on the wire, and how wide an immediate it carries.
Signed-off-by: Eric Davis <eric.davis@broadcom.com>
---
drivers/infiniband/core/device.c | 1 +
drivers/infiniband/core/uverbs_std_types_device.c | 73 ++++++++++++++++++++++-
include/rdma/ib_verbs.h | 15 +++++
include/uapi/rdma/ib_user_ioctl_cmds.h | 6 ++
include/uapi/rdma/ib_user_ioctl_verbs.h | 63 +++++++++++++++++++
5 files changed, 157 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index 7c65a9373701..8b1563de4726 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -3011,6 +3011,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
SET_DEVICE_OP(dev_ops, query_port);
SET_DEVICE_OP(dev_ops, query_port_speed);
SET_DEVICE_OP(dev_ops, query_qp);
+ SET_DEVICE_OP(dev_ops, query_qp_semantics);
SET_DEVICE_OP(dev_ops, query_srq);
SET_DEVICE_OP(dev_ops, query_ucontext);
SET_DEVICE_OP(dev_ops, rdma_netdev_get_params);
diff --git a/drivers/infiniband/core/uverbs_std_types_device.c b/drivers/infiniband/core/uverbs_std_types_device.c
index db65032e80b4..997fc82f9010 100644
--- a/drivers/infiniband/core/uverbs_std_types_device.c
+++ b/drivers/infiniband/core/uverbs_std_types_device.c
@@ -591,6 +591,76 @@ DECLARE_UVERBS_NAMED_METHOD(
UVERBS_ATTR_TYPE(u32),
UA_OPTIONAL));
+/* What a QP type guarantees on this device. Which operations it orders
+ * against, how large a packet it puts on the wire, how wide an immediate it
+ * carries.
+ */
+static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_QP_SEMANTICS)(
+ struct uverbs_attr_bundle *attrs)
+{
+ struct ib_uverbs_qp_semantics resp = {};
+ struct ib_qp_semantics sem = {};
+ struct ib_ucontext *ucontext;
+ struct ib_device *ib_dev;
+ enum ib_qp_type qp_type;
+ int ret;
+
+ ucontext = ib_uverbs_get_ucontext(attrs);
+ if (IS_ERR(ucontext))
+ return PTR_ERR(ucontext);
+ ib_dev = ucontext->device;
+
+ if (!ib_dev->ops.query_qp_semantics)
+ return -EOPNOTSUPP;
+
+ ret = uverbs_get_const(&qp_type, attrs,
+ UVERBS_ATTR_QUERY_QP_SEMANTICS_QP_TYPE);
+ if (ret)
+ return ret;
+
+ /* only the types userspace can create */
+ switch (qp_type) {
+ case IB_UVERBS_QPT_RC:
+ case IB_UVERBS_QPT_UC:
+ case IB_UVERBS_QPT_UD:
+ case IB_UVERBS_QPT_RAW_PACKET:
+ case IB_UVERBS_QPT_XRC_INI:
+ case IB_UVERBS_QPT_XRC_TGT:
+ case IB_UVERBS_QPT_RU:
+ case IB_UVERBS_QPT_DRIVER:
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ ret = ib_dev->ops.query_qp_semantics(ib_dev, qp_type, &sem);
+ if (ret)
+ return ret;
+
+ resp.comp_mask = sem.comp_mask;
+ resp.msg_order = sem.msg_order;
+ resp.max_rdma_raw_size = sem.max_rdma_raw_size;
+ resp.max_rdma_war_size = sem.max_rdma_war_size;
+ resp.max_rdma_waw_size = sem.max_rdma_waw_size;
+ resp.max_pdu = sem.max_pdu;
+ resp.usage_flags = sem.usage_flags;
+ resp.imm_data_size = sem.imm_data_size;
+
+ return uverbs_copy_to_struct_or_zero(
+ attrs, UVERBS_ATTR_QUERY_QP_SEMANTICS_RESP, &resp,
+ sizeof(resp));
+}
+
+DECLARE_UVERBS_NAMED_METHOD(
+ UVERBS_METHOD_QUERY_QP_SEMANTICS,
+ UVERBS_ATTR_CONST_IN(UVERBS_ATTR_QUERY_QP_SEMANTICS_QP_TYPE,
+ enum ib_uverbs_qp_type,
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_QUERY_QP_SEMANTICS_RESP,
+ UVERBS_ATTR_STRUCT(struct ib_uverbs_qp_semantics,
+ imm_data_size),
+ UA_MANDATORY));
+
DECLARE_UVERBS_GLOBAL_METHODS(UVERBS_OBJECT_DEVICE,
&UVERBS_METHOD(UVERBS_METHOD_GET_CONTEXT),
&UVERBS_METHOD(UVERBS_METHOD_INVOKE_WRITE),
@@ -600,7 +670,8 @@ DECLARE_UVERBS_GLOBAL_METHODS(UVERBS_OBJECT_DEVICE,
&UVERBS_METHOD(UVERBS_METHOD_QUERY_CONTEXT),
&UVERBS_METHOD(UVERBS_METHOD_QUERY_GID_TABLE),
&UVERBS_METHOD(UVERBS_METHOD_QUERY_GID_ENTRY),
- &UVERBS_METHOD(UVERBS_METHOD_QUERY_COMP_CNTR_CAPS));
+ &UVERBS_METHOD(UVERBS_METHOD_QUERY_COMP_CNTR_CAPS),
+ &UVERBS_METHOD(UVERBS_METHOD_QUERY_QP_SEMANTICS));
const struct uapi_definition uverbs_def_obj_device[] = {
UAPI_DEF_CHAIN_OBJ_TREE_NAMED(UVERBS_OBJECT_DEVICE),
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 68f7a271cf7a..003b70424429 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2496,6 +2496,18 @@ struct ib_counters {
atomic_t usecnt;
};
+/* what a queue pair of a given type guarantees (query_qp_semantics) */
+struct ib_qp_semantics {
+ u32 comp_mask; /* enum ib_uverbs_qp_semantics_mask */
+ u32 msg_order; /* enum ib_uverbs_qp_msg_order */
+ u32 max_rdma_raw_size;
+ u32 max_rdma_war_size;
+ u32 max_rdma_waw_size;
+ u32 max_pdu;
+ u32 usage_flags; /* enum ib_uverbs_qp_usage_flags */
+ u8 imm_data_size; /* bits */
+};
+
struct ib_counters_read_attr {
u64 *counters_buff;
u32 ncounters;
@@ -2751,6 +2763,9 @@ struct ib_device_ops {
int mr_access_flags, struct ib_pd *pd,
struct ib_udata *udata);
int (*dereg_mr)(struct ib_mr *mr, struct ib_udata *udata);
+ int (*query_qp_semantics)(struct ib_device *device,
+ enum ib_qp_type qp_type,
+ struct ib_qp_semantics *sem);
struct ib_mr *(*alloc_mr)(struct ib_pd *pd, enum ib_mr_type mr_type,
u32 max_num_sg);
struct ib_mr *(*alloc_mr_integrity)(struct ib_pd *pd,
diff --git a/include/uapi/rdma/ib_user_ioctl_cmds.h b/include/uapi/rdma/ib_user_ioctl_cmds.h
index 40fcd4bd1b5e..fb746257eb2c 100644
--- a/include/uapi/rdma/ib_user_ioctl_cmds.h
+++ b/include/uapi/rdma/ib_user_ioctl_cmds.h
@@ -77,6 +77,12 @@ enum uverbs_methods_device {
UVERBS_METHOD_QUERY_GID_ENTRY,
UVERBS_METHOD_QUERY_PORT_SPEED,
UVERBS_METHOD_QUERY_COMP_CNTR_CAPS,
+ UVERBS_METHOD_QUERY_QP_SEMANTICS,
+};
+
+enum uverbs_attrs_query_qp_semantics_attr_ids {
+ UVERBS_ATTR_QUERY_QP_SEMANTICS_QP_TYPE,
+ UVERBS_ATTR_QUERY_QP_SEMANTICS_RESP,
};
enum uverbs_attrs_invoke_write_cmd_attr_ids {
diff --git a/include/uapi/rdma/ib_user_ioctl_verbs.h b/include/uapi/rdma/ib_user_ioctl_verbs.h
index 210f3f945bd7..3d05a6c7ca8f 100644
--- a/include/uapi/rdma/ib_user_ioctl_verbs.h
+++ b/include/uapi/rdma/ib_user_ioctl_verbs.h
@@ -318,6 +318,50 @@ struct ib_uverbs_buffer_desc {
#define IB_UVERBS_QPN_UET_RESERVED_MASK 0x7f000000
#define IB_UVERBS_QPN_UET_ABSOLUTE_ADDR_BIT 0x80000000
+enum ib_uverbs_qp_semantics_mask {
+ IB_UVERBS_QP_SEMANTICS_MASK_MSG_ORDER = 1 << 0,
+ IB_UVERBS_QP_SEMANTICS_MASK_RAW = 1 << 1,
+ IB_UVERBS_QP_SEMANTICS_MASK_WAR = 1 << 2,
+ IB_UVERBS_QP_SEMANTICS_MASK_WAW = 1 << 3,
+ IB_UVERBS_QP_SEMANTICS_MASK_PDU = 1 << 4,
+ IB_UVERBS_QP_SEMANTICS_MASK_IMM = 1 << 5,
+ IB_UVERBS_QP_SEMANTICS_MASK_USAGE = 1 << 6,
+};
+
+/* Ordering guarantees between operations on one QP. "X after Y" means an
+ * X may not pass a Y posted before it. A bit that is clear means the pair
+ * does not order that combination, and an application that needs it must
+ * wait for the earlier completion.
+ */
+enum ib_uverbs_qp_msg_order {
+ /* atomic after atomic */
+ IB_UVERBS_QP_ORDER_ATOMIC_RAR = 1 << 0,
+ IB_UVERBS_QP_ORDER_ATOMIC_RAW = 1 << 1,
+ IB_UVERBS_QP_ORDER_ATOMIC_WAR = 1 << 2,
+ IB_UVERBS_QP_ORDER_ATOMIC_WAW = 1 << 3,
+ /* rma after rma */
+ IB_UVERBS_QP_ORDER_RDMA_RAR = 1 << 4,
+ IB_UVERBS_QP_ORDER_RDMA_RAW = 1 << 5,
+ IB_UVERBS_QP_ORDER_RDMA_WAR = 1 << 6,
+ IB_UVERBS_QP_ORDER_RDMA_WAW = 1 << 7,
+ /* send against atomic and rma */
+ IB_UVERBS_QP_ORDER_RAS = 1 << 8,
+ IB_UVERBS_QP_ORDER_SAR = 1 << 9,
+ IB_UVERBS_QP_ORDER_SAS = 1 << 10,
+ IB_UVERBS_QP_ORDER_SAW = 1 << 11,
+ IB_UVERBS_QP_ORDER_WAS = 1 << 12,
+ /* rma and atomic, either kind, against each other */
+ IB_UVERBS_QP_ORDER_RAR = 1 << 13,
+ IB_UVERBS_QP_ORDER_RAW = 1 << 14,
+ IB_UVERBS_QP_ORDER_WAR = 1 << 15,
+ IB_UVERBS_QP_ORDER_WAW = 1 << 16,
+};
+
+enum ib_uverbs_qp_usage_flags {
+ /* immediate data consumes a posted receive */
+ IB_UVERBS_QP_USAGE_IMM_DATA_RQ = 1 << 0,
+};
+
/* An address handle that names a peer using both transport+network addresses.
* Used for reliable unconnected QPs in both work request address handles and
* a Job address table.
@@ -327,6 +371,25 @@ struct ib_uverbs_ah_attr_ex {
__u32 remote_qpn;
};
+struct ib_uverbs_qp_semantics {
+ __u32 comp_mask; /* enum ib_uverbs_qp_semantics_mask */
+ __u32 msg_order; /* enum ib_uverbs_qp_msg_order */
+ /*
+ * Data ordering: the largest transfer, in bytes, for which the target
+ * memory reflects the order of the pair under its RDMA_ and plain
+ * msg_order bits. 0 if not provided, and those bits are then clear.
+ * There is no read-after-read size, as two reads leave memory
+ * unchanged.
+ */
+ __u32 max_rdma_raw_size; /* read after write */
+ __u32 max_rdma_war_size; /* write after read */
+ __u32 max_rdma_waw_size; /* write after write */
+ __u32 max_pdu; /* largest packet, bytes */
+ __u32 usage_flags; /* enum ib_uverbs_qp_usage_flags */
+ __u8 imm_data_size; /* widest immediate, in bits */
+ __u8 reserved[3];
+};
+
enum ib_uverbs_comp_cntr_entry {
IB_UVERBS_COMP_CNTR_ENTRY_COMP,
IB_UVERBS_COMP_CNTR_ENTRY_ERR,
--
2.56.0
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH rdma-next v2 07/13] RDMA/core: job and job key objects
2026-10-20 3:03 [PATCH rdma-next v2 00/13] RDMA/core: Ultra Ethernet transport verbs Eric Davis via B4 Relay
` (5 preceding siblings ...)
2026-10-20 3:03 ` [PATCH rdma-next v2 06/13] RDMA/core: query the semantics a queue pair type guarantees Eric Davis via B4 Relay
@ 2026-10-20 3:03 ` Eric Davis via B4 Relay
2026-10-20 8:59 ` sashiko-bot
2026-10-20 3:03 ` [PATCH rdma-next v2 08/13] RDMA/uverbs: factor out the queue pair attribute copy Eric Davis via B4 Relay
` (5 subsequent siblings)
12 siblings, 1 reply; 27+ messages in thread
From: Eric Davis via B4 Relay @ 2026-10-20 3:03 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, linux-kernel, Eric Davis
From: Eric Davis <eric.davis@broadcom.com>
An unconnected transport demultiplexes incoming traffic on a JobID rather
than on a connection. Add two objects for it: a job, which owns that ID
and a table of peer addresses, and a job key, which ties a job to a
protection domain and authorizes a queue pair to act within it.
A job can be exported to a file descriptor and imported elsewhere, so an
unprivileged process can join a job that was set up for it.
Signed-off-by: Eric Davis <eric.davis@broadcom.com>
---
drivers/infiniband/core/Makefile | 3 +-
drivers/infiniband/core/device.c | 10 +
drivers/infiniband/core/nldev.c | 2 +
drivers/infiniband/core/rdma_core.h | 2 +
drivers/infiniband/core/restrack.c | 4 +
drivers/infiniband/core/uverbs.h | 6 +
drivers/infiniband/core/uverbs_std_types_job.c | 628 +++++++++++++++++++++++++
drivers/infiniband/core/uverbs_uapi.c | 2 +
drivers/infiniband/core/verbs.c | 37 ++
include/rdma/ib_verbs.h | 67 +++
include/rdma/restrack.h | 8 +
include/rdma/uverbs_ioctl.h | 2 +-
include/uapi/rdma/ib_user_ioctl_cmds.h | 77 +++
13 files changed, 846 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/core/Makefile b/drivers/infiniband/core/Makefile
index 47ef6b0afd29..f169c1bfa8b8 100644
--- a/drivers/infiniband/core/Makefile
+++ b/drivers/infiniband/core/Makefile
@@ -47,7 +47,8 @@ ib_uverbs-y := uverbs_main.o uverbs_cmd.o uverbs_marshall.o \
uverbs_std_types_async_fd.o \
uverbs_std_types_srq.o \
uverbs_std_types_wq.o \
- uverbs_std_types_qp.o
+ uverbs_std_types_qp.o \
+ uverbs_std_types_job.o
ib_uverbs_support-y := rdma_core.o \
ucaps.o \
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index 8b1563de4726..cf2a59afcf76 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2915,6 +2915,14 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
SET_DEVICE_OP(dev_ops, counter_unbind_qp);
SET_DEVICE_OP(dev_ops, counter_update_stats);
SET_DEVICE_OP(dev_ops, create_ah);
+ SET_DEVICE_OP(dev_ops, alloc_job);
+ SET_DEVICE_OP(dev_ops, dealloc_job);
+ SET_DEVICE_OP(dev_ops, query_job);
+ SET_DEVICE_OP(dev_ops, job_addr_insert);
+ SET_DEVICE_OP(dev_ops, job_addr_remove);
+ SET_DEVICE_OP(dev_ops, job_addr_query);
+ SET_DEVICE_OP(dev_ops, create_jkey);
+ SET_DEVICE_OP(dev_ops, destroy_jkey);
SET_DEVICE_OP(dev_ops, create_counters);
SET_DEVICE_OP(dev_ops, create_cq);
SET_DEVICE_OP(dev_ops, create_comp_cntr);
@@ -3031,6 +3039,8 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
SET_OBJ_SIZE(dev_ops, ib_ah);
SET_OBJ_SIZE(dev_ops, ib_counters);
+ SET_OBJ_SIZE(dev_ops, ib_job);
+ SET_OBJ_SIZE(dev_ops, ib_jkey);
SET_OBJ_SIZE(dev_ops, ib_cq);
SET_OBJ_SIZE(dev_ops, ib_comp_cntr);
SET_OBJ_SIZE(dev_ops, ib_dmah);
diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c
index 91b7be84b15f..5313eb4849c7 100644
--- a/drivers/infiniband/core/nldev.c
+++ b/drivers/infiniband/core/nldev.c
@@ -454,6 +454,8 @@ static int fill_res_info(struct sk_buff *msg, struct ib_device *device,
[RDMA_RESTRACK_CTX] = "ctx",
[RDMA_RESTRACK_SRQ] = "srq",
[RDMA_RESTRACK_COMP_CNTR] = "comp_cntr",
+ [RDMA_RESTRACK_JOB] = "job",
+ [RDMA_RESTRACK_JKEY] = "jkey",
};
struct ib_comp_cntr_caps comp_cntr_caps = {};
diff --git a/drivers/infiniband/core/rdma_core.h b/drivers/infiniband/core/rdma_core.h
index 9de14e625dd3..523a6f7d4439 100644
--- a/drivers/infiniband/core/rdma_core.h
+++ b/drivers/infiniband/core/rdma_core.h
@@ -164,6 +164,8 @@ void uverbs_user_mmap_disassociate(struct ib_uverbs_file *ufile);
extern const struct uapi_definition uverbs_def_obj_async_fd[];
extern const struct uapi_definition uverbs_def_obj_counters[];
+extern const struct uapi_definition uverbs_def_obj_job[];
+extern const struct uapi_definition uverbs_def_obj_jkey[];
extern const struct uapi_definition uverbs_def_obj_comp_cntr[];
extern const struct uapi_definition uverbs_def_obj_cq[];
extern const struct uapi_definition uverbs_def_obj_device[];
diff --git a/drivers/infiniband/core/restrack.c b/drivers/infiniband/core/restrack.c
index ddb09b17b733..53217f0ad4c8 100644
--- a/drivers/infiniband/core/restrack.c
+++ b/drivers/infiniband/core/restrack.c
@@ -106,6 +106,10 @@ static struct ib_device *res_to_dev(struct rdma_restrack_entry *res)
return container_of(res, struct ib_dmah, res)->device;
case RDMA_RESTRACK_COMP_CNTR:
return container_of(res, struct ib_comp_cntr, res)->device;
+ case RDMA_RESTRACK_JOB:
+ return container_of(res, struct ib_job, res)->device;
+ case RDMA_RESTRACK_JKEY:
+ return container_of(res, struct ib_jkey, res)->device;
default:
WARN_ONCE(true, "Wrong resource tracking type %u\n", res->type);
return NULL;
diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h
index 6aaac1c309f0..6bc7cc6621cb 100644
--- a/drivers/infiniband/core/uverbs.h
+++ b/drivers/infiniband/core/uverbs.h
@@ -183,6 +183,12 @@ struct ib_uxrcd_object {
atomic_t refcnt;
};
+struct ib_ujob_object {
+ struct ib_uobject uobject;
+ /* job keys created through this handle */
+ atomic_t refcnt;
+};
+
struct ib_usrq_object {
struct ib_uevent_object uevent;
struct ib_uxrcd_object *uxrcd;
diff --git a/drivers/infiniband/core/uverbs_std_types_job.c b/drivers/infiniband/core/uverbs_std_types_job.c
new file mode 100644
index 000000000000..20acb2aa9871
--- /dev/null
+++ b/drivers/infiniband/core/uverbs_std_types_job.c
@@ -0,0 +1,628 @@
+// SPDX-License-Identifier: GPL-2.0-only OR Linux-OpenIB
+
+/*
+ * Copyright (c) 2026 Broadcom. All rights reserved. The term Broadcom
+ * refers to Broadcom Inc. and/or its subsidiaries.
+ */
+
+/*
+ * The job object of a reliable unordered transport.
+ */
+
+#include <linux/file.h>
+
+#include <rdma/ib_marshall.h>
+#include <rdma/uverbs_std_types.h>
+
+#include "rdma_core.h"
+#include "restrack.h"
+#include "uverbs.h"
+
+static void ib_job_release(struct kref *ref)
+{
+ struct ib_job *job = container_of(ref, struct ib_job, ref);
+
+ rdma_restrack_del(&job->res);
+ job->device->ops.dealloc_job(job);
+ kfree(job);
+}
+
+/* An exported job. The descriptor belongs to the context that exported it,
+ * so it stops naming the job once that context is closed or the device is
+ * removed. A context that imported the job holds its own reference.
+ */
+struct ib_ujob_fd_object {
+ struct ib_uobject uobject;
+ /* serializes an import against the descriptor's destroy */
+ spinlock_t lock;
+ struct ib_job *job;
+};
+
+static const struct file_operations ib_job_fops = {
+ .owner = THIS_MODULE,
+ .release = uverbs_uobject_fd_release,
+};
+
+static void uverbs_job_fd_destroy_uobj(struct ib_uobject *uobj,
+ enum rdma_remove_reason why)
+{
+ struct ib_ujob_fd_object *ujf =
+ container_of(uobj, struct ib_ujob_fd_object, uobject);
+ struct ib_job *job;
+
+ spin_lock(&ujf->lock);
+ job = ujf->job;
+ ujf->job = NULL;
+ spin_unlock(&ujf->lock);
+
+ if (job)
+ kref_put(&job->ref, ib_job_release);
+}
+
+/* Take a reference on the job an exported descriptor names */
+static int ib_job_get_from_fd(int fd, struct ib_device *ib_dev,
+ struct ib_job **job)
+{
+ CLASS(fd, f)(fd);
+ struct ib_ujob_fd_object *ujf;
+ int ret = 0;
+
+ if (fd_empty(f))
+ return -EBADF;
+
+ /* refuse anything that is not an exported job */
+ if (fd_file(f)->f_op != &ib_job_fops)
+ return -EINVAL;
+
+ ujf = container_of((struct ib_uobject *)fd_file(f)->private_data,
+ struct ib_ujob_fd_object, uobject);
+
+ spin_lock(&ujf->lock);
+ *job = ujf->job;
+ if (!*job)
+ ret = -EIO;
+ /* a handle names a job on the device it was allocated on */
+ else if ((*job)->device != ib_dev)
+ ret = -EINVAL;
+ /* an imported job holds one reference */
+ else
+ kref_get(&(*job)->ref);
+ spin_unlock(&ujf->lock);
+
+ return ret;
+}
+
+static int UVERBS_HANDLER(UVERBS_METHOD_JOB_EXPORT)(
+ struct uverbs_attr_bundle *attrs)
+{
+ struct ib_job *job =
+ uverbs_attr_get_obj(attrs, UVERBS_ATTR_EXPORT_JOB_HANDLE);
+ struct ib_uobject *uobj =
+ uverbs_attr_get_uobject(attrs, UVERBS_ATTR_EXPORT_JOB_FD);
+ struct ib_ujob_fd_object *ujf =
+ container_of(uobj, struct ib_ujob_fd_object, uobject);
+
+ spin_lock_init(&ujf->lock);
+
+ /* an exported job is a descriptor holding one reference */
+ kref_get(&job->ref);
+ ujf->job = job;
+ uverbs_finalize_uobj_create(attrs, UVERBS_ATTR_EXPORT_JOB_FD);
+
+ return 0;
+}
+
+static int UVERBS_HANDLER(UVERBS_METHOD_JOB_IMPORT)(
+ struct uverbs_attr_bundle *attrs)
+{
+ struct ib_uobject *uobj =
+ uverbs_attr_get_uobject(attrs, UVERBS_ATTR_IMPORT_JOB_HANDLE);
+ struct ib_device *ib_dev = attrs->context->device;
+ struct ib_job *job;
+ int fd, ret;
+
+ ret = uverbs_get_raw_fd(&fd, attrs, UVERBS_ATTR_IMPORT_JOB_FD);
+ if (ret)
+ return ret;
+
+ ret = ib_job_get_from_fd(fd, ib_dev, &job);
+ if (ret)
+ return ret;
+
+ uobj->object = job;
+
+ return 0;
+}
+
+static int uverbs_free_job(struct ib_uobject *uobject,
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
+{
+ struct ib_ujob_object *ujob =
+ container_of(uobject, struct ib_ujob_object, uobject);
+ struct ib_job *job = uobject->object;
+
+ /* don't free if there are still references to this job */
+ if (atomic_read(&ujob->refcnt))
+ return -EBUSY;
+
+ kref_put(&job->ref, ib_job_release);
+
+ return 0;
+}
+
+static int UVERBS_HANDLER(UVERBS_METHOD_JOB_ALLOC)(
+ struct uverbs_attr_bundle *attrs)
+{
+ struct ib_uobject *uobj =
+ uverbs_attr_get_uobject(attrs, UVERBS_ATTR_ALLOC_JOB_HANDLE);
+ struct ib_device *ib_dev = attrs->context->device;
+ struct ib_job_attr attr = {};
+ struct ib_job *job;
+ int ret;
+
+ if (!ib_dev->ops.alloc_job)
+ return -EOPNOTSUPP;
+
+ /* A JobID names a domain across the fabric rather than anything
+ * local to this process, so claiming one is an administrative act.
+ */
+ if (!rdma_uattrs_has_raw_cap(attrs))
+ return -EPERM;
+
+ ret = uverbs_copy_from(&attr.job_id, attrs, UVERBS_ATTR_ALLOC_JOB_ID);
+ if (ret)
+ return ret;
+
+ ret = uverbs_copy_from(&attr.max_addr_entries, attrs,
+ UVERBS_ATTR_ALLOC_JOB_MAX_ADDR_ENTRIES);
+ if (ret)
+ return ret;
+
+ /* The rest are optional and an absent attribute reads as -ENOENT
+ * which IS_UVERBS_COPY_ERR() checks.
+ */
+ ret = uverbs_copy_from(&attr.port_num, attrs,
+ UVERBS_ATTR_ALLOC_JOB_PORT_NUM);
+ if (IS_UVERBS_COPY_ERR(ret))
+ return ret;
+
+ ret = uverbs_copy_from(&attr.sgid_index, attrs,
+ UVERBS_ATTR_ALLOC_JOB_SGID_INDEX);
+ if (IS_UVERBS_COPY_ERR(ret))
+ return ret;
+
+ /* the table is sized once, every index is checked against it */
+ if (!attr.max_addr_entries ||
+ attr.max_addr_entries > ib_dev->attrs.max_addr_entries)
+ return -EINVAL;
+
+ /* the port carries the job's traffic, so it has to be one we have */
+ if (!attr.port_num)
+ attr.port_num = rdma_start_port(ib_dev);
+ if (!rdma_is_port_valid(ib_dev, attr.port_num))
+ return -EINVAL;
+
+ if (attr.sgid_index >=
+ ib_dev->port_data[attr.port_num].immutable.gid_tbl_len)
+ return -EINVAL;
+
+ job = rdma_zalloc_drv_obj(ib_dev, ib_job);
+ if (!job)
+ return -ENOMEM;
+
+ job->device = ib_dev;
+ job->job_id = attr.job_id;
+ job->max_addr_entries = attr.max_addr_entries;
+ kref_init(&job->ref);
+ uobj->object = job;
+
+ rdma_restrack_new(&job->res, RDMA_RESTRACK_JOB);
+ rdma_restrack_set_name(&job->res, NULL);
+
+ ret = ib_dev->ops.alloc_job(job, &attr, attrs);
+ if (ret)
+ goto err_free;
+
+ rdma_restrack_add(&job->res);
+
+ return 0;
+
+err_free:
+ rdma_restrack_put(&job->res);
+ kfree(job);
+ return ret;
+}
+
+static int UVERBS_HANDLER(UVERBS_METHOD_JOB_QUERY)(
+ struct uverbs_attr_bundle *attrs)
+{
+ struct ib_job *job =
+ uverbs_attr_get_obj(attrs, UVERBS_ATTR_QUERY_JOB_HANDLE);
+ struct ib_job_attr attr = {};
+ int ret;
+
+ if (!job->device->ops.query_job)
+ return -EOPNOTSUPP;
+
+ ret = job->device->ops.query_job(job, &attr);
+ if (ret)
+ return ret;
+
+ ret = uverbs_copy_to(attrs, UVERBS_ATTR_QUERY_JOB_ID, &attr.job_id,
+ sizeof(attr.job_id));
+ if (ret)
+ return ret;
+
+ ret = uverbs_copy_to(attrs, UVERBS_ATTR_QUERY_JOB_MAX_ADDR_ENTRIES,
+ &attr.max_addr_entries,
+ sizeof(attr.max_addr_entries));
+ if (ret)
+ return ret;
+
+ ret = uverbs_copy_to(attrs, UVERBS_ATTR_QUERY_JOB_PORT_NUM,
+ &attr.port_num, sizeof(attr.port_num));
+ if (ret)
+ return ret;
+
+ return uverbs_copy_to(attrs, UVERBS_ATTR_QUERY_JOB_SGID_INDEX,
+ &attr.sgid_index, sizeof(attr.sgid_index));
+}
+
+static int UVERBS_HANDLER(UVERBS_METHOD_JOB_ADDR_INSERT)(
+ struct uverbs_attr_bundle *attrs)
+{
+ struct ib_job *job = uverbs_attr_get_obj(
+ attrs, UVERBS_ATTR_JOB_ADDR_INSERT_HANDLE);
+ struct ib_uverbs_ah_attr_ex uah;
+ struct rdma_ah_attr_ex ah_attr = {};
+ u32 index;
+ int ret;
+
+ if (!job->device->ops.job_addr_insert)
+ return -EOPNOTSUPP;
+
+ /* An address table entry names a peer on the fabric for every process
+ * in the job, so writing one redirects their traffic. An address
+ * handle, by contrast, serves only the process that made it.
+ */
+ if (!rdma_uattrs_has_raw_cap(attrs))
+ return -EPERM;
+
+ ret = uverbs_copy_from(&index, attrs,
+ UVERBS_ATTR_JOB_ADDR_INSERT_INDEX);
+ if (ret)
+ return ret;
+
+ ret = uverbs_copy_from_or_zero(&uah, attrs,
+ UVERBS_ATTR_JOB_ADDR_INSERT_AH_ATTR_EX);
+ if (ret)
+ return ret;
+
+ if (index >= job->max_addr_entries)
+ return -EINVAL;
+
+ ret = copy_ah_attr_from_uverbs_ah(job->device, &ah_attr.ah_attr,
+ &uah.ah_attr);
+ if (ret)
+ return ret;
+
+ ah_attr.remote_qpn = uah.remote_qpn;
+
+ return rdma_job_addr_insert(job, index, &ah_attr);
+}
+
+static int UVERBS_HANDLER(UVERBS_METHOD_JOB_ADDR_REMOVE)(
+ struct uverbs_attr_bundle *attrs)
+{
+ struct ib_job *job = uverbs_attr_get_obj(
+ attrs, UVERBS_ATTR_JOB_ADDR_REMOVE_HANDLE);
+ u32 index;
+ int ret;
+
+ if (!job->device->ops.job_addr_remove)
+ return -EOPNOTSUPP;
+
+ if (!rdma_uattrs_has_raw_cap(attrs))
+ return -EPERM;
+
+ ret = uverbs_copy_from(&index, attrs,
+ UVERBS_ATTR_JOB_ADDR_REMOVE_INDEX);
+ if (ret)
+ return ret;
+
+ if (index >= job->max_addr_entries)
+ return -EINVAL;
+
+ return job->device->ops.job_addr_remove(job, index);
+}
+
+static int UVERBS_HANDLER(UVERBS_METHOD_JOB_ADDR_QUERY)(
+ struct uverbs_attr_bundle *attrs)
+{
+ struct ib_job *job = uverbs_attr_get_obj(
+ attrs, UVERBS_ATTR_JOB_ADDR_QUERY_HANDLE);
+ struct ib_uverbs_ah_attr_ex uah = {};
+ struct rdma_ah_attr_ex ah_attr = {};
+ u32 index;
+ int ret;
+
+ if (!job->device->ops.job_addr_query)
+ return -EOPNOTSUPP;
+
+ ret = uverbs_copy_from(&index, attrs,
+ UVERBS_ATTR_JOB_ADDR_QUERY_INDEX);
+ if (ret)
+ return ret;
+
+ if (index >= job->max_addr_entries)
+ return -EINVAL;
+
+ ret = job->device->ops.job_addr_query(job, index, &ah_attr);
+ if (ret)
+ return ret;
+
+ ib_copy_ah_attr_to_user(job->device, &uah.ah_attr, &ah_attr.ah_attr);
+ uah.remote_qpn = ah_attr.remote_qpn;
+
+ return uverbs_copy_to_struct_or_zero(
+ attrs, UVERBS_ATTR_JOB_ADDR_QUERY_AH_ATTR_EX, &uah, sizeof(uah));
+}
+
+DECLARE_UVERBS_NAMED_METHOD(
+ UVERBS_METHOD_JOB_ADDR_INSERT,
+ UVERBS_ATTR_IDR(UVERBS_ATTR_JOB_ADDR_INSERT_HANDLE,
+ UVERBS_OBJECT_JOB,
+ UVERBS_ACCESS_READ,
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_IN(UVERBS_ATTR_JOB_ADDR_INSERT_INDEX,
+ UVERBS_ATTR_TYPE(u32),
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_IN(UVERBS_ATTR_JOB_ADDR_INSERT_AH_ATTR_EX,
+ UVERBS_ATTR_STRUCT(struct ib_uverbs_ah_attr_ex,
+ remote_qpn),
+ UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD(
+ UVERBS_METHOD_JOB_ADDR_REMOVE,
+ UVERBS_ATTR_IDR(UVERBS_ATTR_JOB_ADDR_REMOVE_HANDLE,
+ UVERBS_OBJECT_JOB,
+ UVERBS_ACCESS_READ,
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_IN(UVERBS_ATTR_JOB_ADDR_REMOVE_INDEX,
+ UVERBS_ATTR_TYPE(u32),
+ UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD(
+ UVERBS_METHOD_JOB_ADDR_QUERY,
+ UVERBS_ATTR_IDR(UVERBS_ATTR_JOB_ADDR_QUERY_HANDLE,
+ UVERBS_OBJECT_JOB,
+ UVERBS_ACCESS_READ,
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_IN(UVERBS_ATTR_JOB_ADDR_QUERY_INDEX,
+ UVERBS_ATTR_TYPE(u32),
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_JOB_ADDR_QUERY_AH_ATTR_EX,
+ UVERBS_ATTR_STRUCT(struct ib_uverbs_ah_attr_ex,
+ remote_qpn),
+ UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD(
+ UVERBS_METHOD_JOB_ALLOC,
+ UVERBS_ATTR_IDR(UVERBS_ATTR_ALLOC_JOB_HANDLE,
+ UVERBS_OBJECT_JOB,
+ UVERBS_ACCESS_NEW,
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_IN(UVERBS_ATTR_ALLOC_JOB_ID,
+ UVERBS_ATTR_TYPE(u32),
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_IN(UVERBS_ATTR_ALLOC_JOB_MAX_ADDR_ENTRIES,
+ UVERBS_ATTR_TYPE(u32),
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_IN(UVERBS_ATTR_ALLOC_JOB_PORT_NUM,
+ UVERBS_ATTR_TYPE(u32),
+ UA_OPTIONAL),
+ UVERBS_ATTR_PTR_IN(UVERBS_ATTR_ALLOC_JOB_SGID_INDEX,
+ UVERBS_ATTR_TYPE(u8),
+ UA_OPTIONAL),
+ UVERBS_ATTR_UHW());
+
+DECLARE_UVERBS_NAMED_METHOD(
+ UVERBS_METHOD_JOB_EXPORT,
+ UVERBS_ATTR_IDR(UVERBS_ATTR_EXPORT_JOB_HANDLE,
+ UVERBS_OBJECT_JOB,
+ UVERBS_ACCESS_READ,
+ UA_MANDATORY),
+ UVERBS_ATTR_FD(UVERBS_ATTR_EXPORT_JOB_FD,
+ UVERBS_OBJECT_JOB_FD,
+ UVERBS_ACCESS_NEW,
+ UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD(
+ UVERBS_METHOD_JOB_IMPORT,
+ UVERBS_ATTR_IDR(UVERBS_ATTR_IMPORT_JOB_HANDLE,
+ UVERBS_OBJECT_JOB,
+ UVERBS_ACCESS_NEW,
+ UA_MANDATORY),
+ UVERBS_ATTR_RAW_FD(UVERBS_ATTR_IMPORT_JOB_FD,
+ UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD_DESTROY(
+ UVERBS_METHOD_JOB_DEALLOC,
+ UVERBS_ATTR_IDR(UVERBS_ATTR_DEALLOC_JOB_HANDLE,
+ UVERBS_OBJECT_JOB,
+ UVERBS_ACCESS_DESTROY,
+ UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD(
+ UVERBS_METHOD_JOB_QUERY,
+ UVERBS_ATTR_IDR(UVERBS_ATTR_QUERY_JOB_HANDLE,
+ UVERBS_OBJECT_JOB,
+ UVERBS_ACCESS_READ,
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_QUERY_JOB_ID,
+ UVERBS_ATTR_TYPE(u32),
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_QUERY_JOB_MAX_ADDR_ENTRIES,
+ UVERBS_ATTR_TYPE(u32),
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_QUERY_JOB_PORT_NUM,
+ UVERBS_ATTR_TYPE(u32),
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_QUERY_JOB_SGID_INDEX,
+ UVERBS_ATTR_TYPE(u8),
+ UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_OBJECT(UVERBS_OBJECT_JOB,
+ UVERBS_TYPE_ALLOC_IDR_SZ(
+ sizeof(struct ib_ujob_object),
+ uverbs_free_job),
+ &UVERBS_METHOD(UVERBS_METHOD_JOB_ALLOC),
+ &UVERBS_METHOD(UVERBS_METHOD_JOB_DEALLOC),
+ &UVERBS_METHOD(UVERBS_METHOD_JOB_QUERY),
+ &UVERBS_METHOD(UVERBS_METHOD_JOB_ADDR_INSERT),
+ &UVERBS_METHOD(UVERBS_METHOD_JOB_ADDR_REMOVE),
+ &UVERBS_METHOD(UVERBS_METHOD_JOB_ADDR_QUERY),
+ &UVERBS_METHOD(UVERBS_METHOD_JOB_EXPORT),
+ &UVERBS_METHOD(UVERBS_METHOD_JOB_IMPORT));
+
+DECLARE_UVERBS_NAMED_OBJECT(UVERBS_OBJECT_JOB_FD,
+ UVERBS_TYPE_ALLOC_FD(
+ sizeof(struct ib_ujob_fd_object),
+ uverbs_job_fd_destroy_uobj,
+ &ib_job_fops, "[ib_job]", O_RDWR));
+
+const struct uapi_definition uverbs_def_obj_job[] = {
+ UAPI_DEF_CHAIN_OBJ_TREE_NAMED(UVERBS_OBJECT_JOB,
+ UAPI_DEF_OBJ_NEEDS_FN(dealloc_job)),
+ UAPI_DEF_CHAIN_OBJ_TREE_NAMED(UVERBS_OBJECT_JOB_FD,
+ UAPI_DEF_OBJ_NEEDS_FN(dealloc_job)),
+ {}
+};
+
+/* drop what JKEY_CREATE took on the key's job and PD */
+static void ib_jkey_put_refs(struct ib_jkey *jkey)
+{
+ atomic_dec(&jkey->pd->usecnt);
+ atomic_dec(&container_of(jkey->job_uobject, struct ib_ujob_object,
+ uobject)->refcnt);
+ kref_put(&jkey->job->ref, ib_job_release);
+}
+
+static int uverbs_free_jkey(struct ib_uobject *uobject,
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
+{
+ struct ib_jkey *jkey = uobject->object;
+ int ret;
+
+ /* queue pairs created with this key still hold it */
+ if (atomic_read(&jkey->usecnt))
+ return -EBUSY;
+
+ rdma_restrack_begin_del(&jkey->res);
+
+ ret = jkey->device->ops.destroy_jkey(jkey, attrs);
+ if (ret) {
+ rdma_restrack_abort_del(&jkey->res);
+ return ret;
+ }
+
+ rdma_restrack_commit_del(&jkey->res);
+
+ ib_jkey_put_refs(jkey);
+ kfree(jkey);
+
+ return 0;
+}
+
+static int UVERBS_HANDLER(UVERBS_METHOD_JKEY_CREATE)(
+ struct uverbs_attr_bundle *attrs)
+{
+ struct ib_uobject *uobj =
+ uverbs_attr_get_uobject(attrs, UVERBS_ATTR_CREATE_JKEY_HANDLE);
+ struct ib_uobject *job_uobj = uverbs_attr_get_uobject(
+ attrs, UVERBS_ATTR_CREATE_JKEY_JOB_HANDLE);
+ struct ib_pd *pd =
+ uverbs_attr_get_obj(attrs, UVERBS_ATTR_CREATE_JKEY_PD_HANDLE);
+ struct ib_job *job;
+ struct ib_device *ib_dev = attrs->context->device;
+ struct ib_jkey *jkey;
+ int ret;
+
+ if (!ib_dev->ops.create_jkey)
+ return -EOPNOTSUPP;
+
+ job = job_uobj->object;
+
+ jkey = rdma_zalloc_drv_obj(ib_dev, ib_jkey);
+ if (!jkey)
+ return -ENOMEM;
+
+ jkey->device = ib_dev;
+ jkey->uobject = uobj;
+ jkey->job = job;
+ jkey->job_uobject = job_uobj;
+ jkey->pd = pd;
+ atomic_set(&jkey->usecnt, 0);
+ uobj->object = jkey;
+
+ rdma_restrack_new(&jkey->res, RDMA_RESTRACK_JKEY);
+ rdma_restrack_set_name(&jkey->res, NULL);
+
+ ret = ib_dev->ops.create_jkey(jkey, attrs);
+ if (ret) {
+ rdma_restrack_put(&jkey->res);
+ kfree(jkey);
+ return ret;
+ }
+
+ rdma_restrack_add(&jkey->res);
+
+ /* the job and PD cannot be freed while this key references them */
+ kref_get(&job->ref);
+ atomic_inc(&container_of(job_uobj, struct ib_ujob_object,
+ uobject)->refcnt);
+ atomic_inc(&pd->usecnt);
+ uverbs_finalize_uobj_create(attrs, UVERBS_ATTR_CREATE_JKEY_HANDLE);
+
+ return uverbs_copy_to(attrs, UVERBS_ATTR_CREATE_JKEY_JKEY, &jkey->jkey,
+ sizeof(jkey->jkey));
+}
+
+DECLARE_UVERBS_NAMED_METHOD(
+ UVERBS_METHOD_JKEY_CREATE,
+ UVERBS_ATTR_IDR(UVERBS_ATTR_CREATE_JKEY_HANDLE,
+ UVERBS_OBJECT_JKEY,
+ UVERBS_ACCESS_NEW,
+ UA_MANDATORY),
+ UVERBS_ATTR_IDR(UVERBS_ATTR_CREATE_JKEY_JOB_HANDLE,
+ UVERBS_OBJECT_JOB,
+ UVERBS_ACCESS_READ,
+ UA_MANDATORY),
+ UVERBS_ATTR_IDR(UVERBS_ATTR_CREATE_JKEY_PD_HANDLE,
+ UVERBS_OBJECT_PD,
+ UVERBS_ACCESS_READ,
+ UA_MANDATORY),
+ UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_CREATE_JKEY_JKEY,
+ UVERBS_ATTR_TYPE(u32),
+ UA_MANDATORY),
+ UVERBS_ATTR_UHW());
+
+DECLARE_UVERBS_NAMED_METHOD_DESTROY(
+ UVERBS_METHOD_JKEY_DESTROY,
+ UVERBS_ATTR_IDR(UVERBS_ATTR_DESTROY_JKEY_HANDLE,
+ UVERBS_OBJECT_JKEY,
+ UVERBS_ACCESS_DESTROY,
+ UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_OBJECT(UVERBS_OBJECT_JKEY,
+ UVERBS_TYPE_ALLOC_IDR(uverbs_free_jkey),
+ &UVERBS_METHOD(UVERBS_METHOD_JKEY_CREATE),
+ &UVERBS_METHOD(UVERBS_METHOD_JKEY_DESTROY));
+
+const struct uapi_definition uverbs_def_obj_jkey[] = {
+ UAPI_DEF_CHAIN_OBJ_TREE_NAMED(UVERBS_OBJECT_JKEY,
+ UAPI_DEF_OBJ_NEEDS_FN(destroy_jkey)),
+ {}
+};
diff --git a/drivers/infiniband/core/uverbs_uapi.c b/drivers/infiniband/core/uverbs_uapi.c
index b67db3790f03..ee84be3061ea 100644
--- a/drivers/infiniband/core/uverbs_uapi.c
+++ b/drivers/infiniband/core/uverbs_uapi.c
@@ -636,6 +636,8 @@ static const struct uapi_definition uverbs_core_api[] = {
UAPI_DEF_CHAIN(uverbs_def_obj_dmah),
UAPI_DEF_CHAIN(uverbs_def_obj_flow_action),
UAPI_DEF_CHAIN(uverbs_def_obj_intf),
+ UAPI_DEF_CHAIN(uverbs_def_obj_job),
+ UAPI_DEF_CHAIN(uverbs_def_obj_jkey),
UAPI_DEF_CHAIN(uverbs_def_obj_mr),
UAPI_DEF_CHAIN(uverbs_def_obj_qp),
UAPI_DEF_CHAIN(uverbs_def_obj_srq),
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 2d04ffd13ad3..467a872ca395 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -690,6 +690,43 @@ struct ib_ah *rdma_create_user_ah(struct ib_pd *pd,
}
EXPORT_SYMBOL(rdma_create_user_ah);
+/**
+ * rdma_job_addr_insert - Insert a peer into a job's address table
+ * @job: The job.
+ * @index: The entry in the job's address table.
+ * @ah_attr: The peer's address and queue pair.
+ *
+ * The address is resolved as rdma_create_user_ah() resolves an address
+ * handle's: the source GID attribute is filled in, and for an Ethernet
+ * address so is the destination MAC.
+ *
+ * Return: 0 on success, or a negative errno.
+ */
+int rdma_job_addr_insert(struct ib_job *job, u32 index,
+ struct rdma_ah_attr_ex *ah_attr)
+{
+ struct rdma_ah_attr *attr = &ah_attr->ah_attr;
+ const struct ib_gid_attr *old_sgid_attr;
+ int ret;
+
+ ret = rdma_fill_sgid_attr(job->device, attr, &old_sgid_attr);
+ if (ret)
+ return ret;
+
+ if (attr->type == RDMA_AH_ATTR_TYPE_ROCE) {
+ ret = ib_resolve_eth_dmac(job->device, attr);
+ if (ret)
+ goto out;
+ }
+
+ ret = job->device->ops.job_addr_insert(job, index, ah_attr);
+
+out:
+ rdma_unfill_sgid_attr(attr, old_sgid_attr);
+ return ret;
+}
+EXPORT_SYMBOL(rdma_job_addr_insert);
+
int ib_get_rdma_header_version(const union rdma_network_hdr *hdr)
{
const struct iphdr *ip4h = (struct iphdr *)&hdr->roce4grh;
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 003b70424429..61a7ab60da65 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2496,6 +2496,41 @@ struct ib_counters {
atomic_t usecnt;
};
+struct ib_job_attr {
+ u32 job_id;
+ u32 max_addr_entries;
+ u32 port_num;
+ u8 sgid_index;
+};
+
+/* A job is a receive-side demultiplexing domain of an unconnected transport.
+ * A JobID names it on the wire, memory regions bind to it, and job keys
+ * authorize queue pairs within it.
+ */
+struct ib_job {
+ struct ib_device *device;
+ /* exports, job keys, regions, and queue pairs referencing this job */
+ struct kref ref;
+ u32 job_id;
+ /* entries in the address table, every index is below it */
+ u32 max_addr_entries;
+ struct rdma_restrack_entry res;
+};
+
+/* A job key ties a job to a PD and authorizes a queue pair to act within a
+ * job. It holds a reference to its job and PD.
+ */
+struct ib_jkey {
+ struct ib_device *device;
+ struct ib_uobject *uobject;
+ struct ib_job *job;
+ struct ib_uobject *job_uobject;
+ struct ib_pd *pd;
+ u32 jkey;
+ atomic_t usecnt;
+ struct rdma_restrack_entry res;
+};
+
/* what a queue pair of a given type guarantees (query_qp_semantics) */
struct ib_qp_semantics {
u32 comp_mask; /* enum ib_uverbs_qp_semantics_mask */
@@ -2830,6 +2865,20 @@ struct ib_device_ops {
struct ib_mr *(*reg_dm_mr)(struct ib_pd *pd, struct ib_dm *dm,
struct ib_dm_mr_attr *attr,
struct uverbs_attr_bundle *attrs);
+ /* job_addr_* may run concurrently on one job; the driver serializes */
+ int (*alloc_job)(struct ib_job *job, struct ib_job_attr *attr,
+ struct uverbs_attr_bundle *attrs);
+ void (*dealloc_job)(struct ib_job *job);
+ int (*query_job)(struct ib_job *job, struct ib_job_attr *attr);
+ int (*job_addr_insert)(struct ib_job *job, u32 index,
+ struct rdma_ah_attr_ex *ah_attr);
+ int (*job_addr_remove)(struct ib_job *job, u32 index);
+ int (*job_addr_query)(struct ib_job *job, u32 index,
+ struct rdma_ah_attr_ex *ah_attr);
+ int (*create_jkey)(struct ib_jkey *jkey,
+ struct uverbs_attr_bundle *attrs);
+ int (*destroy_jkey)(struct ib_jkey *jkey,
+ struct uverbs_attr_bundle *attrs);
int (*create_counters)(struct ib_counters *counters,
struct uverbs_attr_bundle *attrs);
int (*destroy_counters)(struct ib_counters *counters);
@@ -2983,6 +3032,8 @@ struct ib_device_ops {
DECLARE_RDMA_OBJ_SIZE(ib_ah);
DECLARE_RDMA_OBJ_SIZE(ib_counters);
+ DECLARE_RDMA_OBJ_SIZE(ib_job);
+ DECLARE_RDMA_OBJ_SIZE(ib_jkey);
DECLARE_RDMA_OBJ_SIZE(ib_cq);
DECLARE_RDMA_OBJ_SIZE(ib_comp_cntr);
DECLARE_RDMA_OBJ_SIZE(ib_dmah);
@@ -3930,6 +3981,22 @@ struct ib_ah *rdma_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr,
struct ib_ah *rdma_create_user_ah(struct ib_pd *pd,
struct rdma_ah_init_attr *init_attr,
struct ib_udata *udata);
+
+/**
+ * rdma_job_addr_insert - Insert a peer into a job's address table
+ * @job: The job.
+ * @index: The entry in the job's address table.
+ * @ah_attr: The peer's address and queue pair.
+ *
+ * The address is resolved as rdma_create_user_ah() resolves an address
+ * handle's: the source GID attribute is filled in, and for an Ethernet
+ * address so is the destination MAC.
+ *
+ * Return: 0 on success, or a negative errno.
+ */
+int rdma_job_addr_insert(struct ib_job *job, u32 index,
+ struct rdma_ah_attr_ex *ah_attr);
+
/**
* ib_get_gids_from_rdma_hdr - Get sgid and dgid from GRH or IPv4 header
* work completion.
diff --git a/include/rdma/restrack.h b/include/rdma/restrack.h
index 47fec3083d28..a7d53540da67 100644
--- a/include/rdma/restrack.h
+++ b/include/rdma/restrack.h
@@ -64,6 +64,14 @@ enum rdma_restrack_type {
* @RDMA_RESTRACK_COMP_CNTR: Completion Counter
*/
RDMA_RESTRACK_COMP_CNTR,
+ /**
+ * @RDMA_RESTRACK_JOB: Transport job
+ */
+ RDMA_RESTRACK_JOB,
+ /**
+ * @RDMA_RESTRACK_JKEY: Job key
+ */
+ RDMA_RESTRACK_JKEY,
/**
* @RDMA_RESTRACK_MAX: Last entry, used for array dclarations
*/
diff --git a/include/rdma/uverbs_ioctl.h b/include/rdma/uverbs_ioctl.h
index f4d93bc3625a..96cfe0ba7bd1 100644
--- a/include/rdma/uverbs_ioctl.h
+++ b/include/rdma/uverbs_ioctl.h
@@ -147,7 +147,7 @@ enum uapi_radix_data {
UVERBS_API_OBJ_KEY_BITS = 6,
UVERBS_API_OBJ_KEY_SHIFT =
UVERBS_API_METHOD_KEY_BITS + UVERBS_API_METHOD_KEY_SHIFT,
- UVERBS_API_OBJ_KEY_NUM_CORE = 20,
+ UVERBS_API_OBJ_KEY_NUM_CORE = 23,
UVERBS_API_OBJ_KEY_NUM_DRIVER =
(1 << UVERBS_API_OBJ_KEY_BITS) - UVERBS_API_OBJ_KEY_NUM_CORE,
UVERBS_API_OBJ_KEY_MASK = GENMASK(31, UVERBS_API_OBJ_KEY_SHIFT),
diff --git a/include/uapi/rdma/ib_user_ioctl_cmds.h b/include/uapi/rdma/ib_user_ioctl_cmds.h
index fb746257eb2c..70de1e4001c3 100644
--- a/include/uapi/rdma/ib_user_ioctl_cmds.h
+++ b/include/uapi/rdma/ib_user_ioctl_cmds.h
@@ -58,6 +58,9 @@ enum uverbs_default_objects {
UVERBS_OBJECT_DMAH,
UVERBS_OBJECT_DMABUF,
UVERBS_OBJECT_COMP_CNTR,
+ UVERBS_OBJECT_JOB,
+ UVERBS_OBJECT_JKEY,
+ UVERBS_OBJECT_JOB_FD,
};
enum {
@@ -495,4 +498,78 @@ enum uverbs_attrs_read_comp_cntr_cmd_attr_ids {
UVERBS_ATTR_READ_COMP_CNTR_RESP_VALUE,
};
+enum uverbs_methods_job {
+ UVERBS_METHOD_JOB_ALLOC,
+ UVERBS_METHOD_JOB_DEALLOC,
+ UVERBS_METHOD_JOB_QUERY,
+ UVERBS_METHOD_JOB_ADDR_INSERT,
+ UVERBS_METHOD_JOB_ADDR_REMOVE,
+ UVERBS_METHOD_JOB_ADDR_QUERY,
+ UVERBS_METHOD_JOB_EXPORT,
+ UVERBS_METHOD_JOB_IMPORT,
+};
+
+enum uverbs_attrs_alloc_job_cmd_attr_ids {
+ UVERBS_ATTR_ALLOC_JOB_HANDLE,
+ UVERBS_ATTR_ALLOC_JOB_ID,
+ UVERBS_ATTR_ALLOC_JOB_MAX_ADDR_ENTRIES,
+ UVERBS_ATTR_ALLOC_JOB_PORT_NUM,
+ UVERBS_ATTR_ALLOC_JOB_SGID_INDEX,
+};
+
+enum uverbs_attrs_dealloc_job_cmd_attr_ids {
+ UVERBS_ATTR_DEALLOC_JOB_HANDLE,
+};
+
+enum uverbs_attrs_query_job_cmd_attr_ids {
+ UVERBS_ATTR_QUERY_JOB_HANDLE,
+ UVERBS_ATTR_QUERY_JOB_ID,
+ UVERBS_ATTR_QUERY_JOB_MAX_ADDR_ENTRIES,
+ UVERBS_ATTR_QUERY_JOB_PORT_NUM,
+ UVERBS_ATTR_QUERY_JOB_SGID_INDEX,
+};
+
+enum uverbs_attrs_job_addr_insert_cmd_attr_ids {
+ UVERBS_ATTR_JOB_ADDR_INSERT_HANDLE,
+ UVERBS_ATTR_JOB_ADDR_INSERT_INDEX,
+ UVERBS_ATTR_JOB_ADDR_INSERT_AH_ATTR_EX,
+};
+
+enum uverbs_attrs_job_addr_remove_cmd_attr_ids {
+ UVERBS_ATTR_JOB_ADDR_REMOVE_HANDLE,
+ UVERBS_ATTR_JOB_ADDR_REMOVE_INDEX,
+};
+
+enum uverbs_attrs_job_addr_query_cmd_attr_ids {
+ UVERBS_ATTR_JOB_ADDR_QUERY_HANDLE,
+ UVERBS_ATTR_JOB_ADDR_QUERY_INDEX,
+ UVERBS_ATTR_JOB_ADDR_QUERY_AH_ATTR_EX,
+};
+
+enum uverbs_attrs_export_job_cmd_attr_ids {
+ UVERBS_ATTR_EXPORT_JOB_HANDLE,
+ UVERBS_ATTR_EXPORT_JOB_FD,
+};
+
+enum uverbs_attrs_import_job_cmd_attr_ids {
+ UVERBS_ATTR_IMPORT_JOB_HANDLE,
+ UVERBS_ATTR_IMPORT_JOB_FD,
+};
+
+enum uverbs_methods_jkey {
+ UVERBS_METHOD_JKEY_CREATE,
+ UVERBS_METHOD_JKEY_DESTROY,
+};
+
+enum uverbs_attrs_create_jkey_cmd_attr_ids {
+ UVERBS_ATTR_CREATE_JKEY_HANDLE,
+ UVERBS_ATTR_CREATE_JKEY_JOB_HANDLE,
+ UVERBS_ATTR_CREATE_JKEY_PD_HANDLE,
+ UVERBS_ATTR_CREATE_JKEY_JKEY,
+};
+
+enum uverbs_attrs_destroy_jkey_cmd_attr_ids {
+ UVERBS_ATTR_DESTROY_JKEY_HANDLE,
+};
+
#endif
--
2.56.0
^ permalink raw reply related [flat|nested] 27+ messages in thread* Re: [PATCH rdma-next v2 07/13] RDMA/core: job and job key objects
2026-10-20 3:03 ` [PATCH rdma-next v2 07/13] RDMA/core: job and job key objects Eric Davis via B4 Relay
@ 2026-10-20 8:59 ` sashiko-bot
0 siblings, 0 replies; 27+ messages in thread
From: sashiko-bot @ 2026-10-20 8:59 UTC (permalink / raw)
To: Eric Davis; +Cc: linux-rdma
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] RDMA/uverbs: missing uverbs_finalize_uobj_create() in job alloc and import
--
commit 5f0d9f9de261cb667b880d34e0c2d952d5a61944
Author: Eric Davis <eric.davis@broadcom.com>
RDMA/core: job and job key objects
This commit introduces job and job key objects to RDMA core to support
unconnected transports that demultiplex incoming traffic on a JobID. It
also allows jobs to be exported to a file descriptor and imported into
another context.
> diff --git a/drivers/infiniband/core/uverbs_std_types_job.c b/drivers/infiniband/core/uverbs_std_types_job.c
> new file mode 100644
> index 0000000000000..20acb2aa9871f
> --- /dev/null
> +++ b/drivers/infiniband/core/uverbs_std_types_job.c
[ ... ]
> +static int UVERBS_HANDLER(UVERBS_METHOD_JOB_IMPORT)(
> + struct uverbs_attr_bundle *attrs)
> +{
> + struct ib_uobject *uobj =
> + uverbs_attr_get_uobject(attrs, UVERBS_ATTR_IMPORT_JOB_HANDLE);
> + struct ib_device *ib_dev = attrs->context->device;
> + struct ib_job *job;
> + int fd, ret;
> +
> + ret = uverbs_get_raw_fd(&fd, attrs, UVERBS_ATTR_IMPORT_JOB_FD);
> + if (ret)
> + return ret;
> +
> + ret = ib_job_get_from_fd(fd, ib_dev, &job);
> + if (ret)
> + return ret;
> +
> + uobj->object = job;
> +
> + return 0;
> +}
> +
> +static int uverbs_free_job(struct ib_uobject *uobject,
> + enum rdma_remove_reason why,
> + struct uverbs_attr_bundle *attrs)
> +{
> + struct ib_ujob_object *ujob =
> + container_of(uobject, struct ib_ujob_object, uobject);
> + struct ib_job *job = uobject->object;
> +
> + /* don't free if there are still references to this job */
> + if (atomic_read(&ujob->refcnt))
> + return -EBUSY;
> +
> + kref_put(&job->ref, ib_job_release);
> +
> + return 0;
> +}
> +
> +static int UVERBS_HANDLER(UVERBS_METHOD_JOB_ALLOC)(
> + struct uverbs_attr_bundle *attrs)
> +{
> + struct ib_uobject *uobj =
> + uverbs_attr_get_uobject(attrs, UVERBS_ATTR_ALLOC_JOB_HANDLE);
> + struct ib_device *ib_dev = attrs->context->device;
> + struct ib_job_attr attr = {};
> + struct ib_job *job;
> + int ret;
[ ... ]
> + job = rdma_zalloc_drv_obj(ib_dev, ib_job);
> + if (!job)
> + return -ENOMEM;
> +
> + job->device = ib_dev;
> + job->job_id = attr.job_id;
> + job->max_addr_entries = attr.max_addr_entries;
> + kref_init(&job->ref);
> + uobj->object = job;
> +
> + rdma_restrack_new(&job->res, RDMA_RESTRACK_JOB);
> + rdma_restrack_set_name(&job->res, NULL);
> +
> + ret = ib_dev->ops.alloc_job(job, &attr, attrs);
> + if (ret)
> + goto err_free;
> +
> + rdma_restrack_add(&job->res);
> +
> + return 0;
[Severity: High]
Should UVERBS_HANDLER(UVERBS_METHOD_JOB_ALLOC) and
UVERBS_HANDLER(UVERBS_METHOD_JOB_IMPORT) call uverbs_finalize_uobj_create()
before returning 0?
UVERBS_METHOD_JOB_ALLOC declares UVERBS_ATTR_UHW(), which sets
bundle->method_elm->has_udata = true. After the handler returns 0,
ib_uverbs_run_method() updates UVERBS_ATTR_UHW_OUT when present:
drivers/infiniband/core/uverbs_ioctl.c:ib_uverbs_run_method() {
...
ret = handler(&pbundle->bundle);
if (!ret && bundle->method_elm->has_udata) {
const struct uverbs_attr *attr =
uverbs_attr_get(bundle, UVERBS_ATTR_UHW_OUT);
if (!IS_ERR(attr))
ret = uverbs_set_output(bundle, attr);
}
...
}
If uverbs_set_output() fails with -EFAULT when writing to user_attrs,
ib_uverbs_cmd_verbs() calls bundle_destroy(pbundle, false). Because
uverbs_finalize_uobj_create() was not called to mark uobj_hw_obj_valid,
uverbs_finalize_object() passes hw_obj_valid = false to
rdma_alloc_abort_uobject():
drivers/infiniband/core/rdma_core.c:rdma_alloc_abort_uobject() {
...
if (hw_obj_valid) {
uverbs_get_cleared_udata(attrs);
ret = uobj->uapi_object->type_class->destroy_hw(
uobj, RDMA_REMOVE_ABORT, attrs);
...
}
Without hw_obj_valid set, destroy_hw (uverbs_free_job()) is skipped on the
abort path. Does this leak the struct ib_job allocation, the driver job
resource from alloc_job(), and leave job->res in the RDMA_RESTRACK_JOB
table?
[ ... ]
> +DECLARE_UVERBS_NAMED_METHOD(
> + UVERBS_METHOD_JOB_ALLOC,
> + UVERBS_ATTR_IDR(UVERBS_ATTR_ALLOC_JOB_HANDLE,
> + UVERBS_OBJECT_JOB,
> + UVERBS_ACCESS_NEW,
> + UA_MANDATORY),
> + UVERBS_ATTR_PTR_IN(UVERBS_ATTR_ALLOC_JOB_ID,
> + UVERBS_ATTR_TYPE(u32),
> + UA_MANDATORY),
> + UVERBS_ATTR_PTR_IN(UVERBS_ATTR_ALLOC_JOB_MAX_ADDR_ENTRIES,
> + UVERBS_ATTR_TYPE(u32),
> + UA_MANDATORY),
> + UVERBS_ATTR_PTR_IN(UVERBS_ATTR_ALLOC_JOB_PORT_NUM,
> + UVERBS_ATTR_TYPE(u32),
> + UA_OPTIONAL),
> + UVERBS_ATTR_PTR_IN(UVERBS_ATTR_ALLOC_JOB_SGID_INDEX,
> + UVERBS_ATTR_TYPE(u8),
> + UA_OPTIONAL),
> + UVERBS_ATTR_UHW());
[ ... ]
完美结束，确保最后有一个空行。
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019-uet-ib-core-v2-0-8bd654f25370@broadcom.com?part=7
^ permalink raw reply [flat|nested] 27+ messages in thread
* [PATCH rdma-next v2 08/13] RDMA/uverbs: factor out the queue pair attribute copy
2026-10-20 3:03 [PATCH rdma-next v2 00/13] RDMA/core: Ultra Ethernet transport verbs Eric Davis via B4 Relay
` (6 preceding siblings ...)
2026-10-20 3:03 ` [PATCH rdma-next v2 07/13] RDMA/core: job and job key objects Eric Davis via B4 Relay
@ 2026-10-20 3:03 ` Eric Davis via B4 Relay
2026-10-20 9:02 ` sashiko-bot
2026-10-20 3:03 ` [PATCH rdma-next v2 09/13] RDMA/core: create a queue pair ready to send Eric Davis via B4 Relay
` (4 subsequent siblings)
12 siblings, 1 reply; 27+ messages in thread
From: Eric Davis via B4 Relay @ 2026-10-20 3:03 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, linux-kernel, Eric Davis
From: Eric Davis <eric.davis@broadcom.com>
modify_qp() validates a set of attributes that came from userspace and
copies the ones its mask names into their kernel form. Queue pair
creation needs the same work for a transport that is ready to send as
soon as the pair exists, so move it into ib_uverbs_copy_qp_attr().
No functional change.
Signed-off-by: Eric Davis <eric.davis@broadcom.com>
---
drivers/infiniband/core/uverbs.h | 6 ++
drivers/infiniband/core/uverbs_cmd.c | 191 ++++++++++++++++++-----------------
2 files changed, 105 insertions(+), 92 deletions(-)
diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h
index 6bc7cc6621cb..8c461a172217 100644
--- a/drivers/infiniband/core/uverbs.h
+++ b/drivers/infiniband/core/uverbs.h
@@ -319,6 +319,12 @@ struct ib_uverbs_flow_spec {
};
};
+int ib_uverbs_copy_qp_attr(struct uverbs_attr_bundle *attrs,
+ struct ib_device *device,
+ struct ib_qp_attr *attr,
+ const struct ib_uverbs_ex_modify_qp *src,
+ u32 mask);
+
int ib_uverbs_kern_spec_to_ib_spec_filter(enum ib_flow_spec_type type,
const void *kern_spec_mask,
const void *kern_spec_val,
diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index cca386b2c3f0..a35b743ab9bd 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -1779,7 +1779,7 @@ int copy_ah_attr_from_uverbs_ah(struct ib_device *dev,
static void copy_ah_attr_from_uverbs(struct ib_device *dev,
struct rdma_ah_attr *rdma_attr,
- struct ib_uverbs_qp_dest *uverb_attr)
+ const struct ib_uverbs_qp_dest *uverb_attr)
{
rdma_attr->type = rdma_ah_find_type(dev, uverb_attr->port_num);
if (uverb_attr->is_global) {
@@ -1800,6 +1800,101 @@ static void copy_ah_attr_from_uverbs(struct ib_device *dev,
rdma_ah_set_make_grd(rdma_attr, false);
}
+/* Validate a set of QP attributes that came from userspace and copy the ones
+ * the mask names into their kernel form. Shared by the modify command, and by
+ * queue pair creation which takes the same set of attributes for a transport
+ * that is ready to send as soon as the QP is created.
+ */
+int ib_uverbs_copy_qp_attr(struct uverbs_attr_bundle *attrs,
+ struct ib_device *device,
+ struct ib_qp_attr *attr,
+ const struct ib_uverbs_ex_modify_qp *src,
+ u32 mask)
+{
+ /* Last bit is reserved for extending the attr_mask by using
+ * another field.
+ */
+ if (mask & ~(IB_QP_ATTR_STANDARD_BITS | IB_QP_RATE_LIMIT))
+ return -EOPNOTSUPP;
+
+ if ((mask & IB_QP_PORT) &&
+ !rdma_is_port_valid(device, src->base.port_num))
+ return -EINVAL;
+
+ if ((mask & IB_QP_AV) &&
+ !rdma_is_port_valid(device, src->base.dest.port_num))
+ return -EINVAL;
+
+ if ((mask & IB_QP_ALT_PATH) &&
+ (!rdma_is_port_valid(device, src->base.alt_port_num) ||
+ !rdma_is_port_valid(device, src->base.alt_dest.port_num) ||
+ src->base.alt_port_num != src->base.alt_dest.port_num))
+ return -EINVAL;
+
+ if (((mask & IB_QP_CUR_STATE) && src->base.cur_qp_state > IB_QPS_ERR) ||
+ ((mask & IB_QP_STATE) && src->base.qp_state > IB_QPS_ERR))
+ return -EINVAL;
+
+ if (mask & IB_QP_STATE)
+ attr->qp_state = src->base.qp_state;
+ if (mask & IB_QP_CUR_STATE)
+ attr->cur_qp_state = src->base.cur_qp_state;
+ if (mask & IB_QP_PATH_MTU)
+ attr->path_mtu = src->base.path_mtu;
+ if (mask & IB_QP_PATH_MIG_STATE)
+ attr->path_mig_state = src->base.path_mig_state;
+ if (mask & IB_QP_QKEY) {
+ if (src->base.qkey & IB_QP_SET_QKEY &&
+ !(rdma_nl_get_privileged_qkey() ||
+ rdma_uattrs_has_raw_cap(attrs)))
+ return -EPERM;
+ attr->qkey = src->base.qkey;
+ }
+ if (mask & IB_QP_RQ_PSN)
+ attr->rq_psn = src->base.rq_psn;
+ if (mask & IB_QP_SQ_PSN)
+ attr->sq_psn = src->base.sq_psn;
+ if (mask & IB_QP_DEST_QPN)
+ attr->dest_qp_num = src->base.dest_qp_num;
+ if (mask & IB_QP_ACCESS_FLAGS)
+ attr->qp_access_flags = src->base.qp_access_flags;
+ if (mask & IB_QP_PKEY_INDEX)
+ attr->pkey_index = src->base.pkey_index;
+ if (mask & IB_QP_EN_SQD_ASYNC_NOTIFY)
+ attr->en_sqd_async_notify = src->base.en_sqd_async_notify;
+ if (mask & IB_QP_MAX_QP_RD_ATOMIC)
+ attr->max_rd_atomic = src->base.max_rd_atomic;
+ if (mask & IB_QP_MAX_DEST_RD_ATOMIC)
+ attr->max_dest_rd_atomic = src->base.max_dest_rd_atomic;
+ if (mask & IB_QP_MIN_RNR_TIMER)
+ attr->min_rnr_timer = src->base.min_rnr_timer;
+ if (mask & IB_QP_PORT)
+ attr->port_num = src->base.port_num;
+ if (mask & IB_QP_TIMEOUT)
+ attr->timeout = src->base.timeout;
+ if (mask & IB_QP_RETRY_CNT)
+ attr->retry_cnt = src->base.retry_cnt;
+ if (mask & IB_QP_RNR_RETRY)
+ attr->rnr_retry = src->base.rnr_retry;
+ if (mask & IB_QP_ALT_PATH) {
+ attr->alt_port_num = src->base.alt_port_num;
+ attr->alt_timeout = src->base.alt_timeout;
+ attr->alt_pkey_index = src->base.alt_pkey_index;
+ }
+ if (mask & IB_QP_RATE_LIMIT)
+ attr->rate_limit = src->rate_limit;
+
+ if (mask & IB_QP_AV)
+ copy_ah_attr_from_uverbs(device, &attr->ah_attr,
+ &src->base.dest);
+
+ if (mask & IB_QP_ALT_PATH)
+ copy_ah_attr_from_uverbs(device, &attr->alt_ah_attr,
+ &src->base.alt_dest);
+
+ return 0;
+}
+
static int modify_qp(struct uverbs_attr_bundle *attrs,
struct ib_uverbs_ex_modify_qp *cmd)
{
@@ -1818,18 +1913,7 @@ static int modify_qp(struct uverbs_attr_bundle *attrs,
goto out;
}
- if ((cmd->base.attr_mask & IB_QP_PORT) &&
- !rdma_is_port_valid(qp->device, cmd->base.port_num)) {
- ret = -EINVAL;
- goto release_qp;
- }
-
if ((cmd->base.attr_mask & IB_QP_AV)) {
- if (!rdma_is_port_valid(qp->device, cmd->base.dest.port_num)) {
- ret = -EINVAL;
- goto release_qp;
- }
-
if (cmd->base.attr_mask & IB_QP_STATE &&
cmd->base.qp_state == IB_QPS_RTR) {
/* We are in INIT->RTR TRANSITION (if we are not,
@@ -1878,80 +1962,10 @@ static int modify_qp(struct uverbs_attr_bundle *attrs,
}
}
- if ((cmd->base.attr_mask & IB_QP_ALT_PATH) &&
- (!rdma_is_port_valid(qp->device, cmd->base.alt_port_num) ||
- !rdma_is_port_valid(qp->device, cmd->base.alt_dest.port_num) ||
- cmd->base.alt_port_num != cmd->base.alt_dest.port_num)) {
- ret = -EINVAL;
- goto release_qp;
- }
-
- if ((cmd->base.attr_mask & IB_QP_CUR_STATE &&
- cmd->base.cur_qp_state > IB_QPS_ERR) ||
- (cmd->base.attr_mask & IB_QP_STATE &&
- cmd->base.qp_state > IB_QPS_ERR)) {
- ret = -EINVAL;
+ ret = ib_uverbs_copy_qp_attr(attrs, qp->device, attr, cmd,
+ cmd->base.attr_mask);
+ if (ret)
goto release_qp;
- }
-
- if (cmd->base.attr_mask & IB_QP_STATE)
- attr->qp_state = cmd->base.qp_state;
- if (cmd->base.attr_mask & IB_QP_CUR_STATE)
- attr->cur_qp_state = cmd->base.cur_qp_state;
- if (cmd->base.attr_mask & IB_QP_PATH_MTU)
- attr->path_mtu = cmd->base.path_mtu;
- if (cmd->base.attr_mask & IB_QP_PATH_MIG_STATE)
- attr->path_mig_state = cmd->base.path_mig_state;
- if (cmd->base.attr_mask & IB_QP_QKEY) {
- if (cmd->base.qkey & IB_QP_SET_QKEY &&
- !(rdma_nl_get_privileged_qkey() ||
- rdma_uattrs_has_raw_cap(attrs))) {
- ret = -EPERM;
- goto release_qp;
- }
- attr->qkey = cmd->base.qkey;
- }
- if (cmd->base.attr_mask & IB_QP_RQ_PSN)
- attr->rq_psn = cmd->base.rq_psn;
- if (cmd->base.attr_mask & IB_QP_SQ_PSN)
- attr->sq_psn = cmd->base.sq_psn;
- if (cmd->base.attr_mask & IB_QP_DEST_QPN)
- attr->dest_qp_num = cmd->base.dest_qp_num;
- if (cmd->base.attr_mask & IB_QP_ACCESS_FLAGS)
- attr->qp_access_flags = cmd->base.qp_access_flags;
- if (cmd->base.attr_mask & IB_QP_PKEY_INDEX)
- attr->pkey_index = cmd->base.pkey_index;
- if (cmd->base.attr_mask & IB_QP_EN_SQD_ASYNC_NOTIFY)
- attr->en_sqd_async_notify = cmd->base.en_sqd_async_notify;
- if (cmd->base.attr_mask & IB_QP_MAX_QP_RD_ATOMIC)
- attr->max_rd_atomic = cmd->base.max_rd_atomic;
- if (cmd->base.attr_mask & IB_QP_MAX_DEST_RD_ATOMIC)
- attr->max_dest_rd_atomic = cmd->base.max_dest_rd_atomic;
- if (cmd->base.attr_mask & IB_QP_MIN_RNR_TIMER)
- attr->min_rnr_timer = cmd->base.min_rnr_timer;
- if (cmd->base.attr_mask & IB_QP_PORT)
- attr->port_num = cmd->base.port_num;
- if (cmd->base.attr_mask & IB_QP_TIMEOUT)
- attr->timeout = cmd->base.timeout;
- if (cmd->base.attr_mask & IB_QP_RETRY_CNT)
- attr->retry_cnt = cmd->base.retry_cnt;
- if (cmd->base.attr_mask & IB_QP_RNR_RETRY)
- attr->rnr_retry = cmd->base.rnr_retry;
- if (cmd->base.attr_mask & IB_QP_ALT_PATH) {
- attr->alt_port_num = cmd->base.alt_port_num;
- attr->alt_timeout = cmd->base.alt_timeout;
- attr->alt_pkey_index = cmd->base.alt_pkey_index;
- }
- if (cmd->base.attr_mask & IB_QP_RATE_LIMIT)
- attr->rate_limit = cmd->rate_limit;
-
- if (cmd->base.attr_mask & IB_QP_AV)
- copy_ah_attr_from_uverbs(qp->device, &attr->ah_attr,
- &cmd->base.dest);
-
- if (cmd->base.attr_mask & IB_QP_ALT_PATH)
- copy_ah_attr_from_uverbs(qp->device, &attr->alt_ah_attr,
- &cmd->base.alt_dest);
ret = ib_modify_qp_with_udata(qp, attr,
modify_qp_mask(qp->qp_type,
@@ -1994,13 +2008,6 @@ static int ib_uverbs_ex_modify_qp(struct uverbs_attr_bundle *attrs)
if (ret)
return ret;
- /*
- * Last bit is reserved for extending the attr_mask by
- * using another field.
- */
- if (cmd.base.attr_mask & ~(IB_QP_ATTR_STANDARD_BITS | IB_QP_RATE_LIMIT))
- return -EOPNOTSUPP;
-
ret = modify_qp(attrs, &cmd);
if (ret)
return ret;
--
2.56.0
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH rdma-next v2 09/13] RDMA/core: create a queue pair ready to send
2026-10-20 3:03 [PATCH rdma-next v2 00/13] RDMA/core: Ultra Ethernet transport verbs Eric Davis via B4 Relay
` (7 preceding siblings ...)
2026-10-20 3:03 ` [PATCH rdma-next v2 08/13] RDMA/uverbs: factor out the queue pair attribute copy Eric Davis via B4 Relay
@ 2026-10-20 3:03 ` Eric Davis via B4 Relay
2026-10-20 9:25 ` sashiko-bot
2026-10-20 3:03 ` [PATCH rdma-next v2 10/13] RDMA/core: queue pair attributes for unconnected transports Eric Davis via B4 Relay
` (3 subsequent siblings)
12 siblings, 1 reply; 27+ messages in thread
From: Eric Davis via B4 Relay @ 2026-10-20 3:03 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, linux-kernel, Eric Davis
From: Eric Davis <eric.davis@broadcom.com>
Some transports have nothing to negotiate, so a queue pair can be ready
to send the moment it exists and the modify sequence carries no useful
information. Let a reliable unconnected queue pair be created with the
attributes that sequence would have supplied, in the same form the
modify command uses, and apply them as that modify to RTS once the
queue pair exists, so its checks run. A queue pair whose modify fails
is destroyed.
Note that the QP handle field is refused, since the create is what makes
the handle.
Signed-off-by: Eric Davis <eric.davis@broadcom.com>
---
drivers/infiniband/core/uverbs_std_types_qp.c | 80 +++++++++++++++++++++++++++
include/uapi/rdma/ib_user_ioctl_cmds.h | 1 +
2 files changed, 81 insertions(+)
diff --git a/drivers/infiniband/core/uverbs_std_types_qp.c b/drivers/infiniband/core/uverbs_std_types_qp.c
index 30fc20fb251f..c6225156fe30 100644
--- a/drivers/infiniband/core/uverbs_std_types_qp.c
+++ b/drivers/infiniband/core/uverbs_std_types_qp.c
@@ -79,6 +79,61 @@ static void set_caps(struct ib_qp_init_attr *attr,
}
}
+/* what a reliable unconnected QP may be created ready to send with */
+#define CREATE_QP_ATTR_RU_MASK (IB_QP_STATE | IB_QP_PKEY_INDEX | \
+ IB_QP_PORT | IB_QP_ACCESS_FLAGS | \
+ IB_QP_PATH_MTU)
+
+/* The attributes the QP would be modified with via modify_qp calls. Used for
+ * implementations that support having QPs be in RTS immediately after QP
+ * creation. These attributes arrive in the same format the modify command
+ * uses so that converter serves both, and they are applied as that modify
+ * to RTS, once the QP exists.
+ */
+static int create_qp_attr_set(struct uverbs_attr_bundle *attrs,
+ struct ib_device *device,
+ struct ib_qp_init_attr *attr,
+ struct ib_qp_attr *qp_attr, u32 *mask)
+{
+ struct ib_uverbs_ex_modify_qp uqpa = {};
+ int ret;
+
+ *mask = 0;
+
+ if (!uverbs_attr_is_valid(attrs, UVERBS_ATTR_CREATE_QP_ATTR))
+ return 0;
+
+ if (attr->qp_type != IB_QPT_RU ||
+ !(device->attrs.device_cap_flags & IB_DEVICE_RU))
+ return -EOPNOTSUPP;
+
+ ret = uverbs_copy_from_or_zero(&uqpa, attrs,
+ UVERBS_ATTR_CREATE_QP_ATTR);
+ if (ret)
+ return ret;
+
+ /* the create is what makes the handle, can't be one specified */
+ if (uqpa.base.qp_handle)
+ return -EINVAL;
+
+ if (uqpa.reserved || uqpa.base.reserved[0] || uqpa.base.reserved[1])
+ return -EINVAL;
+
+ ret = ib_uverbs_copy_qp_attr(attrs, device, qp_attr, &uqpa,
+ uqpa.base.attr_mask);
+ if (ret)
+ return ret;
+
+ if ((uqpa.base.attr_mask & ~CREATE_QP_ATTR_RU_MASK) ||
+ ((uqpa.base.attr_mask & IB_QP_STATE) &&
+ uqpa.base.qp_state != IB_QPS_RTS))
+ return -EINVAL;
+
+ *mask = uqpa.base.attr_mask;
+
+ return 0;
+}
+
static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
struct uverbs_attr_bundle *attrs)
{
@@ -86,6 +141,8 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
uverbs_attr_get_uobject(attrs, UVERBS_ATTR_CREATE_QP_HANDLE),
typeof(*obj), uevent.uobject);
struct ib_qp_init_attr attr = {};
+ struct ib_qp_attr qp_attr = {};
+ u32 qp_attr_mask;
struct ib_uverbs_qp_cap cap = {};
struct ib_rwq_ind_table *rwq_ind_tbl = NULL;
struct ib_qp *qp;
@@ -214,6 +271,11 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
attr.create_flags |= IB_QP_CREATE_SOURCE_QPN;
}
+ ret = create_qp_attr_set(attrs, device, &attr, &qp_attr,
+ &qp_attr_mask);
+ if (ret)
+ return ret;
+
srq = uverbs_attr_get_obj(attrs,
UVERBS_ATTR_CREATE_QP_SRQ_HANDLE);
if (!IS_ERR(srq)) {
@@ -264,6 +326,20 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
obj->uevent.uobject.object = qp;
uverbs_finalize_uobj_create(attrs, UVERBS_ATTR_CREATE_QP_HANDLE);
+ /* A reliable unconnected QP is ready to send once created. What the
+ * caller passed is applied as the modify to RTS it stands in for, so
+ * the modify's checks run. A failure here destroys the QP.
+ */
+ if (attr.qp_type == IB_QPT_RU) {
+ qp_attr.qp_state = IB_QPS_RTS;
+ /* the driver's data was for create, there is none for this */
+ ret = ib_modify_qp_with_udata(qp, &qp_attr,
+ qp_attr_mask | IB_QP_STATE,
+ uverbs_get_cleared_udata(attrs));
+ if (ret)
+ return ret;
+ }
+
set_caps(&attr, &cap, false);
ret = uverbs_copy_to_struct_or_zero(attrs,
UVERBS_ATTR_CREATE_QP_RESP_CAP, &cap,
@@ -325,6 +401,10 @@ DECLARE_UVERBS_NAMED_METHOD(
UVERBS_ATTR_FLAGS_IN(UVERBS_ATTR_CREATE_QP_FLAGS,
enum ib_uverbs_qp_create_flags,
UA_OPTIONAL),
+ UVERBS_ATTR_PTR_IN(UVERBS_ATTR_CREATE_QP_ATTR,
+ UVERBS_ATTR_STRUCT(struct ib_uverbs_ex_modify_qp,
+ reserved),
+ UA_OPTIONAL),
UVERBS_ATTR_PTR_IN(UVERBS_ATTR_CREATE_QP_SOURCE_QPN,
UVERBS_ATTR_TYPE(u32),
UA_OPTIONAL),
diff --git a/include/uapi/rdma/ib_user_ioctl_cmds.h b/include/uapi/rdma/ib_user_ioctl_cmds.h
index 70de1e4001c3..b2bf8ca92397 100644
--- a/include/uapi/rdma/ib_user_ioctl_cmds.h
+++ b/include/uapi/rdma/ib_user_ioctl_cmds.h
@@ -179,6 +179,7 @@ enum uverbs_attrs_create_qp_cmd_attr_ids {
UVERBS_ATTR_CREATE_QP_BUF_UMEM,
UVERBS_ATTR_CREATE_QP_RQ_BUF_UMEM,
UVERBS_ATTR_CREATE_QP_SQ_BUF_UMEM,
+ UVERBS_ATTR_CREATE_QP_ATTR,
};
enum uverbs_attrs_destroy_qp_cmd_attr_ids {
--
2.56.0
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH rdma-next v2 10/13] RDMA/core: queue pair attributes for unconnected transports
2026-10-20 3:03 [PATCH rdma-next v2 00/13] RDMA/core: Ultra Ethernet transport verbs Eric Davis via B4 Relay
` (8 preceding siblings ...)
2026-10-20 3:03 ` [PATCH rdma-next v2 09/13] RDMA/core: create a queue pair ready to send Eric Davis via B4 Relay
@ 2026-10-20 3:03 ` Eric Davis via B4 Relay
2026-10-20 9:53 ` sashiko-bot
2026-10-20 3:03 ` [PATCH rdma-next v2 11/13] RDMA/core: extended memory registration Eric Davis via B4 Relay
` (2 subsequent siblings)
12 siblings, 1 reply; 27+ messages in thread
From: Eric Davis via B4 Relay @ 2026-10-20 3:03 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, linux-kernel, Eric Davis
From: Eric Davis <eric.davis@broadcom.com>
A queue pair on an unconnected transport is created with more than the
connected attributes. A job key authorizing it to act within a job, the
initiator identity it sends under, and the ordering guarantees the caller
wants it to keep.
The QP holds a reference to its job key for as long as it lives, so the
job cannot go away underneath it.
Signed-off-by: Eric Davis <eric.davis@broadcom.com>
---
drivers/infiniband/core/uverbs_std_types_qp.c | 150 ++++++++++++++++++++++++++
drivers/infiniband/core/verbs.c | 5 +
include/rdma/ib_verbs.h | 14 +++
include/uapi/rdma/ib_user_ioctl_cmds.h | 3 +
4 files changed, 172 insertions(+)
diff --git a/drivers/infiniband/core/uverbs_std_types_qp.c b/drivers/infiniband/core/uverbs_std_types_qp.c
index c6225156fe30..47ca994f9ac9 100644
--- a/drivers/infiniband/core/uverbs_std_types_qp.c
+++ b/drivers/infiniband/core/uverbs_std_types_qp.c
@@ -79,6 +79,139 @@ static void set_caps(struct ib_qp_init_attr *attr,
}
}
+/* the semantics fields, ordering bits and usage flags uverbs knows of */
+#define QP_SEMANTICS_MASK_ALL ((IB_UVERBS_QP_SEMANTICS_MASK_USAGE << 1) - 1)
+#define QP_ORDER_ALL ((IB_UVERBS_QP_ORDER_WAW << 1) - 1)
+#define QP_USAGE_ALL IB_UVERBS_QP_USAGE_IMM_DATA_RQ
+
+/* A QP may relax what its type guarantees on this device, but not ask for
+ * more than the device advertises for it.
+ */
+static int create_qp_semantics_check(struct ib_device *device,
+ const struct ib_qp_semantics *req)
+{
+ struct ib_qp_semantics adv = {};
+ u32 m = req->comp_mask;
+ int ret;
+
+ if (!device->ops.query_qp_semantics)
+ return -EOPNOTSUPP;
+
+ ret = device->ops.query_qp_semantics(device, IB_QPT_RU, &adv);
+ if (ret)
+ return ret;
+
+ if (m & ~adv.comp_mask)
+ return -EINVAL;
+
+ if (((m & IB_UVERBS_QP_SEMANTICS_MASK_MSG_ORDER) &&
+ (req->msg_order & ~adv.msg_order)) ||
+ ((m & IB_UVERBS_QP_SEMANTICS_MASK_RAW) &&
+ req->max_rdma_raw_size > adv.max_rdma_raw_size) ||
+ ((m & IB_UVERBS_QP_SEMANTICS_MASK_WAR) &&
+ req->max_rdma_war_size > adv.max_rdma_war_size) ||
+ ((m & IB_UVERBS_QP_SEMANTICS_MASK_WAW) &&
+ req->max_rdma_waw_size > adv.max_rdma_waw_size) ||
+ ((m & IB_UVERBS_QP_SEMANTICS_MASK_PDU) &&
+ req->max_pdu > adv.max_pdu) ||
+ ((m & IB_UVERBS_QP_SEMANTICS_MASK_IMM) &&
+ req->imm_data_size > adv.imm_data_size) ||
+ ((m & IB_UVERBS_QP_SEMANTICS_MASK_USAGE) &&
+ (req->usage_flags & ~adv.usage_flags)))
+ return -EINVAL;
+
+ return 0;
+}
+
+/* Process the optional creation attributes. A job to act within, a source
+ * identity, a set of ordering guarantees to ask for. They and the checks on
+ * a requested QP number belong to a reliable unconnected QP.
+ */
+static int create_qp_opt_attrs(struct uverbs_attr_bundle *attrs,
+ struct ib_device *device, struct ib_pd *pd,
+ struct ib_qp_init_attr *attr,
+ struct ib_qp_semantics *sem)
+{
+ struct ib_uverbs_qp_semantics usem = {};
+ struct ib_jkey *jkey;
+ bool has_semantics;
+ int ret;
+
+ has_semantics = uverbs_attr_is_valid(attrs,
+ UVERBS_ATTR_CREATE_QP_SEMANTICS);
+
+ if (attr->qp_type != IB_QPT_RU ||
+ !(device->attrs.device_cap_flags & IB_DEVICE_RU)) {
+ if (has_semantics ||
+ uverbs_attr_is_valid(attrs, UVERBS_ATTR_CREATE_QP_SRC_ID) ||
+ uverbs_attr_is_valid(attrs, UVERBS_ATTR_CREATE_QP_JOB_KEY))
+ return -EOPNOTSUPP;
+
+ return 0;
+ }
+
+ if (uverbs_attr_is_valid(attrs, UVERBS_ATTR_CREATE_QP_SRC_ID)) {
+ ret = uverbs_copy_from(&attr->src_id, attrs,
+ UVERBS_ATTR_CREATE_QP_SRC_ID);
+ if (ret)
+ return ret;
+ }
+
+ if (uverbs_attr_is_valid(attrs, UVERBS_ATTR_CREATE_QP_JOB_KEY)) {
+ jkey = uverbs_attr_get_obj(attrs,
+ UVERBS_ATTR_CREATE_QP_JOB_KEY);
+ if (IS_ERR(jkey))
+ return PTR_ERR(jkey);
+
+ /* a key acts within the PD it was made on */
+ if (jkey->pd != pd)
+ return -EINVAL;
+
+ attr->jkey = jkey;
+ }
+
+ if (attr->create_flags & IB_QP_CREATE_SOURCE_QPN) {
+ if (attr->source_qpn & IB_UVERBS_QPN_UET_RESERVED_MASK)
+ return -EINVAL;
+
+ /* a relative QP is named within a job, so it needs a key */
+ if (!(attr->source_qpn &
+ IB_UVERBS_QPN_UET_ABSOLUTE_ADDR_BIT) &&
+ !attr->jkey)
+ return -EINVAL;
+ }
+
+ if (has_semantics) {
+ ret = uverbs_copy_from_or_zero(&usem, attrs,
+ UVERBS_ATTR_CREATE_QP_SEMANTICS);
+ if (ret)
+ return ret;
+
+ if ((usem.comp_mask & ~QP_SEMANTICS_MASK_ALL) ||
+ (usem.msg_order & ~QP_ORDER_ALL) ||
+ (usem.usage_flags & ~QP_USAGE_ALL) ||
+ memchr_inv(usem.reserved, 0, sizeof(usem.reserved)))
+ return -EINVAL;
+
+ sem->comp_mask = usem.comp_mask;
+ sem->msg_order = usem.msg_order;
+ sem->max_rdma_raw_size = usem.max_rdma_raw_size;
+ sem->max_rdma_war_size = usem.max_rdma_war_size;
+ sem->max_rdma_waw_size = usem.max_rdma_waw_size;
+ sem->max_pdu = usem.max_pdu;
+ sem->usage_flags = usem.usage_flags;
+ sem->imm_data_size = usem.imm_data_size;
+
+ ret = create_qp_semantics_check(device, sem);
+ if (ret)
+ return ret;
+
+ attr->qp_semantics = sem;
+ }
+
+ return 0;
+}
+
/* what a reliable unconnected QP may be created ready to send with */
#define CREATE_QP_ATTR_RU_MASK (IB_QP_STATE | IB_QP_PKEY_INDEX | \
IB_QP_PORT | IB_QP_ACCESS_FLAGS | \
@@ -143,6 +276,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
struct ib_qp_init_attr attr = {};
struct ib_qp_attr qp_attr = {};
u32 qp_attr_mask;
+ struct ib_qp_semantics qp_semantics = {};
struct ib_uverbs_qp_cap cap = {};
struct ib_rwq_ind_table *rwq_ind_tbl = NULL;
struct ib_qp *qp;
@@ -197,6 +331,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
case IB_UVERBS_QPT_UC:
case IB_UVERBS_QPT_UD:
case IB_UVERBS_QPT_XRC_INI:
+ case IB_UVERBS_QPT_RU:
case IB_UVERBS_QPT_DRIVER:
if (uverbs_attr_is_valid(attrs,
UVERBS_ATTR_CREATE_QP_XRCD_HANDLE) ||
@@ -271,6 +406,10 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
attr.create_flags |= IB_QP_CREATE_SOURCE_QPN;
}
+ ret = create_qp_opt_attrs(attrs, device, pd, &attr, &qp_semantics);
+ if (ret)
+ return ret;
+
ret = create_qp_attr_set(attrs, device, &attr, &qp_attr,
&qp_attr_mask);
if (ret)
@@ -401,6 +540,17 @@ DECLARE_UVERBS_NAMED_METHOD(
UVERBS_ATTR_FLAGS_IN(UVERBS_ATTR_CREATE_QP_FLAGS,
enum ib_uverbs_qp_create_flags,
UA_OPTIONAL),
+ UVERBS_ATTR_PTR_IN(UVERBS_ATTR_CREATE_QP_SRC_ID,
+ UVERBS_ATTR_TYPE(u32),
+ UA_OPTIONAL),
+ UVERBS_ATTR_PTR_IN(UVERBS_ATTR_CREATE_QP_SEMANTICS,
+ UVERBS_ATTR_STRUCT(struct ib_uverbs_qp_semantics,
+ imm_data_size),
+ UA_OPTIONAL),
+ UVERBS_ATTR_IDR(UVERBS_ATTR_CREATE_QP_JOB_KEY,
+ UVERBS_OBJECT_JKEY,
+ UVERBS_ACCESS_READ,
+ UA_OPTIONAL),
UVERBS_ATTR_PTR_IN(UVERBS_ATTR_CREATE_QP_ATTR,
UVERBS_ATTR_STRUCT(struct ib_uverbs_ex_modify_qp,
reserved),
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 467a872ca395..ad281e565499 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -1343,6 +1343,7 @@ static struct ib_qp *create_qp(struct ib_device *dev, struct ib_pd *pd,
qp->qp_type = attr->qp_type;
qp->rwq_ind_tbl = attr->rwq_ind_tbl;
qp->srq = attr->srq;
+ qp->jkey = attr->jkey;
qp->event_handler = __ib_qp_event_handler;
qp->registered_event_handler = attr->event_handler;
qp->port = attr->port_num;
@@ -1438,11 +1439,15 @@ void ib_qp_usecnt_inc(struct ib_qp *qp)
atomic_inc(&qp->srq->usecnt);
if (qp->rwq_ind_tbl)
atomic_inc(&qp->rwq_ind_tbl->usecnt);
+ if (qp->jkey)
+ atomic_inc(&qp->jkey->usecnt);
}
EXPORT_SYMBOL(ib_qp_usecnt_inc);
void ib_qp_usecnt_dec(struct ib_qp *qp)
{
+ if (qp->jkey)
+ atomic_dec(&qp->jkey->usecnt);
if (qp->rwq_ind_tbl)
atomic_dec(&qp->rwq_ind_tbl->usecnt);
if (qp->srq)
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 61a7ab60da65..83fa94d579c3 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1243,6 +1243,9 @@ enum ib_qp_create_flags {
* callback to destroy the passed in QP.
*/
+struct ib_jkey;
+struct ib_qp_semantics;
+
struct ib_qp_init_attr {
/* This callback occurs in workqueue context */
void (*event_handler)(struct ib_event *, void *);
@@ -1263,6 +1266,15 @@ struct ib_qp_init_attr {
u32 port_num;
struct ib_rwq_ind_table *rwq_ind_tbl;
u32 source_qpn;
+
+ /* The initiator ID this QP sends with, the caller's rank in its job.
+ * UET lets the caller choose it; the JobID is what is trusted.
+ */
+ u32 src_id;
+ /* the job this QP belongs to */
+ struct ib_jkey *jkey;
+ /* what the caller asked the QP to guarantee, NULL if it didn't ask */
+ struct ib_qp_semantics *qp_semantics;
};
struct ib_qp_open_attr {
@@ -2009,6 +2021,8 @@ struct ib_qp {
enum ib_qp_type qp_type;
struct ib_rwq_ind_table *rwq_ind_tbl;
struct ib_qp_security *qp_sec;
+ /* the job key this pair was created within, NULL if none */
+ struct ib_jkey *jkey;
u32 port;
bool integrity_en;
diff --git a/include/uapi/rdma/ib_user_ioctl_cmds.h b/include/uapi/rdma/ib_user_ioctl_cmds.h
index b2bf8ca92397..1de6a1ffd92c 100644
--- a/include/uapi/rdma/ib_user_ioctl_cmds.h
+++ b/include/uapi/rdma/ib_user_ioctl_cmds.h
@@ -179,6 +179,9 @@ enum uverbs_attrs_create_qp_cmd_attr_ids {
UVERBS_ATTR_CREATE_QP_BUF_UMEM,
UVERBS_ATTR_CREATE_QP_RQ_BUF_UMEM,
UVERBS_ATTR_CREATE_QP_SQ_BUF_UMEM,
+ UVERBS_ATTR_CREATE_QP_SRC_ID,
+ UVERBS_ATTR_CREATE_QP_SEMANTICS,
+ UVERBS_ATTR_CREATE_QP_JOB_KEY,
UVERBS_ATTR_CREATE_QP_ATTR,
};
--
2.56.0
^ permalink raw reply related [flat|nested] 27+ messages in thread* Re: [PATCH rdma-next v2 10/13] RDMA/core: queue pair attributes for unconnected transports
2026-10-20 3:03 ` [PATCH rdma-next v2 10/13] RDMA/core: queue pair attributes for unconnected transports Eric Davis via B4 Relay
@ 2026-10-20 9:53 ` sashiko-bot
0 siblings, 0 replies; 27+ messages in thread
From: sashiko-bot @ 2026-10-20 9:53 UTC (permalink / raw)
To: Eric Davis; +Cc: linux-rdma
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] RDMA/core: unmasked semantics fields used and copied in create_qp_opt_attrs()
- [Medium] RDMA/core: missing IB_DEVICE_RU check for IB_QPT_RU in create_qp_opt_attrs()
--
commit 814ed695a9e745ffe7fb729da61fd57b0d746109
Author: Eric Davis <eric.davis@broadcom.com>
RDMA/core: queue pair attributes for unconnected transports
This commit adds support for reliable unconnected (IB_QPT_RU) queue pair
creation attributes in uverbs and ib_verbs, including source ID, job key
(jkey), and ordering/usage semantics. It also tracks the jkey use count
over the lifetime of the QP.
> diff --git a/drivers/infiniband/core/uverbs_std_types_qp.c b/drivers/infiniband/core/uverbs_std_types_qp.c
> index c6225156fe30e..47ca994f9ac96 100644
> --- a/drivers/infiniband/core/uverbs_std_types_qp.c
> +++ b/drivers/infiniband/core/uverbs_std_types_qp.c
> @@ -79,6 +79,139 @@ static void set_caps(struct ib_qp_init_attr *attr,
> }
> }
>
> +/* the semantics fields, ordering bits and usage flags uverbs knows of */
> +#define QP_SEMANTICS_MASK_ALL ((IB_UVERBS_QP_SEMANTICS_MASK_USAGE << 1) - 1)
> +#define QP_ORDER_ALL ((IB_UVERBS_QP_ORDER_WAW << 1) - 1)
> +#define QP_USAGE_ALL IB_UVERBS_QP_USAGE_IMM_DATA_RQ
> +
> +/* A QP may relax what its type guarantees on this device, but not ask for
> + * more than the device advertises for it.
> + */
> +static int create_qp_semantics_check(struct ib_device *device,
> + const struct ib_qp_semantics *req)
> +{
> + struct ib_qp_semantics adv = {};
> + u32 m = req->comp_mask;
> + int ret;
> +
> + if (!device->ops.query_qp_semantics)
> + return -EOPNOTSUPP;
> +
> + ret = device->ops.query_qp_semantics(device, IB_QPT_RU, &adv);
> + if (ret)
> + return ret;
> +
> + if (m & ~adv.comp_mask)
> + return -EINVAL;
> +
> + if (((m & IB_UVERBS_QP_SEMANTICS_MASK_MSG_ORDER) &&
> + (req->msg_order & ~adv.msg_order)) ||
> + ((m & IB_UVERBS_QP_SEMANTICS_MASK_RAW) &&
> + req->max_rdma_raw_size > adv.max_rdma_raw_size) ||
> + ((m & IB_UVERBS_QP_SEMANTICS_MASK_WAR) &&
> + req->max_rdma_war_size > adv.max_rdma_war_size) ||
> + ((m & IB_UVERBS_QP_SEMANTICS_MASK_WAW) &&
> + req->max_rdma_waw_size > adv.max_rdma_waw_size) ||
> + ((m & IB_UVERBS_QP_SEMANTICS_MASK_PDU) &&
> + req->max_pdu > adv.max_pdu) ||
> + ((m & IB_UVERBS_QP_SEMANTICS_MASK_IMM) &&
> + req->imm_data_size > adv.imm_data_size) ||
> + ((m & IB_UVERBS_QP_SEMANTICS_MASK_USAGE) &&
> + (req->usage_flags & ~adv.usage_flags)))
> + return -EINVAL;
> +
> + return 0;
> +}
> +
> +/* Process the optional creation attributes. A job to act within, a source
> + * identity, a set of ordering guarantees to ask for. They and the checks on
> + * a requested QP number belong to a reliable unconnected QP.
> + */
> +static int create_qp_opt_attrs(struct uverbs_attr_bundle *attrs,
> + struct ib_device *device, struct ib_pd *pd,
> + struct ib_qp_init_attr *attr,
> + struct ib_qp_semantics *sem)
> +{
> + struct ib_uverbs_qp_semantics usem = {};
> + struct ib_jkey *jkey;
> + bool has_semantics;
> + int ret;
> +
> + has_semantics = uverbs_attr_is_valid(attrs,
> + UVERBS_ATTR_CREATE_QP_SEMANTICS);
> +
> + if (attr->qp_type != IB_QPT_RU ||
> + !(device->attrs.device_cap_flags & IB_DEVICE_RU)) {
> + if (has_semantics ||
> + uverbs_attr_is_valid(attrs, UVERBS_ATTR_CREATE_QP_SRC_ID) ||
> + uverbs_attr_is_valid(attrs, UVERBS_ATTR_CREATE_QP_JOB_KEY))
> + return -EOPNOTSUPP;
> +
> + return 0;
> + }
[Severity: Medium]
What happens in UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE) ->
create_qp_opt_attrs() when userspace requests attr.qp_type == IB_QPT_RU on
a device that does not advertise IB_DEVICE_RU, and omits the optional
UVERBS_ATTR_CREATE_QP_SEMANTICS, UVERBS_ATTR_CREATE_QP_SRC_ID, and
UVERBS_ATTR_CREATE_QP_JOB_KEY attributes?
Because UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE) adds IB_UVERBS_QPT_RU to
the switch statement without checking IB_DEVICE_RU, and this check combines
attr->qp_type != IB_QPT_RU with !(device->attrs.device_cap_flags &
IB_DEVICE_RU), create_qp_opt_attrs() returns 0 instead of -EOPNOTSUPP.
Returning 0 early here also skips the IB_QP_CREATE_SOURCE_QPN validation
below (checking attr->source_qpn against IB_UVERBS_QPN_UET_RESERVED_MASK
and requiring attr->jkey for relative QPNs) and allows QP creation to
proceed to ib_create_qp_user() on a device without IB_DEVICE_RU.
Should create_qp_opt_attrs() return -EOPNOTSUPP whenever attr->qp_type ==
IB_QPT_RU and !(device->attrs.device_cap_flags & IB_DEVICE_RU)?
> +
> + if (uverbs_attr_is_valid(attrs, UVERBS_ATTR_CREATE_QP_SRC_ID)) {
> + ret = uverbs_copy_from(&attr->src_id, attrs,
> + UVERBS_ATTR_CREATE_QP_SRC_ID);
> + if (ret)
> + return ret;
> + }
> +
> + if (uverbs_attr_is_valid(attrs, UVERBS_ATTR_CREATE_QP_JOB_KEY)) {
> + jkey = uverbs_attr_get_obj(attrs,
> + UVERBS_ATTR_CREATE_QP_JOB_KEY);
> + if (IS_ERR(jkey))
> + return PTR_ERR(jkey);
> +
> + /* a key acts within the PD it was made on */
> + if (jkey->pd != pd)
> + return -EINVAL;
> +
> + attr->jkey = jkey;
> + }
> +
> + if (attr->create_flags & IB_QP_CREATE_SOURCE_QPN) {
> + if (attr->source_qpn & IB_UVERBS_QPN_UET_RESERVED_MASK)
> + return -EINVAL;
> +
> + /* a relative QP is named within a job, so it needs a key */
> + if (!(attr->source_qpn &
> + IB_UVERBS_QPN_UET_ABSOLUTE_ADDR_BIT) &&
> + !attr->jkey)
> + return -EINVAL;
> + }
> +
> + if (has_semantics) {
> + ret = uverbs_copy_from_or_zero(&usem, attrs,
> + UVERBS_ATTR_CREATE_QP_SEMANTICS);
> + if (ret)
> + return ret;
> +
> + if ((usem.comp_mask & ~QP_SEMANTICS_MASK_ALL) ||
> + (usem.msg_order & ~QP_ORDER_ALL) ||
> + (usem.usage_flags & ~QP_USAGE_ALL) ||
> + memchr_inv(usem.reserved, 0, sizeof(usem.reserved)))
> + return -EINVAL;
> +
> + sem->comp_mask = usem.comp_mask;
> + sem->msg_order = usem.msg_order;
> + sem->max_rdma_raw_size = usem.max_rdma_raw_size;
> + sem->max_rdma_war_size = usem.max_rdma_war_size;
> + sem->max_rdma_waw_size = usem.max_rdma_waw_size;
> + sem->max_pdu = usem.max_pdu;
> + sem->usage_flags = usem.usage_flags;
> + sem->imm_data_size = usem.imm_data_size;
> +
> + ret = create_qp_semantics_check(device, sem);
> + if (ret)
> + return ret;
> +
> + attr->qp_semantics = sem;
> + }
[Severity: Medium]
Should the checks and assignments on the fields of usem in
create_qp_opt_attrs() be gated by their corresponding bits in
usem.comp_mask?
When UVERBS_ATTR_CREATE_QP_SEMANTICS is provided, individual fields in
struct ib_uverbs_qp_semantics are gated by IB_UVERBS_QP_SEMANTICS_MASK_*
bits in usem.comp_mask.
However, create_qp_opt_attrs() checks (usem.msg_order & ~QP_ORDER_ALL) and
(usem.usage_flags & ~QP_USAGE_ALL) unconditionally even when
IB_UVERBS_QP_SEMANTICS_MASK_MSG_ORDER or IB_UVERBS_QP_SEMANTICS_MASK_USAGE
is not set in usem.comp_mask, which can reject valid requests with -EINVAL
if unused fields in the user structure were not zeroed.
In addition, create_qp_opt_attrs() copies all fields from usem into sem
regardless of usem.comp_mask, whereas create_qp_semantics_check() only
validates the fields in sem whose bits are set in req->comp_mask. As a
result, any field whose bit is clear in usem.comp_mask bypasses
create_qp_semantics_check() and is passed with unvalidated user values in
attr->qp_semantics to dev->ops.create_qp().
[ ... ]
> @@ -197,6 +331,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
> case IB_UVERBS_QPT_UC:
> case IB_UVERBS_QPT_UD:
> case IB_UVERBS_QPT_XRC_INI:
> + case IB_UVERBS_QPT_RU:
> case IB_UVERBS_QPT_DRIVER:
> if (uverbs_attr_is_valid(attrs,
> UVERBS_ATTR_CREATE_QP_XRCD_HANDLE) ||
> @@ -271,6 +406,10 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
> attr.create_flags |= IB_QP_CREATE_SOURCE_QPN;
> }
>
> + ret = create_qp_opt_attrs(attrs, device, pd, &attr, &qp_semantics);
> + if (ret)
> + return ret;
> +
> ret = create_qp_attr_set(attrs, device, &attr, &qp_attr,
> &qp_attr_mask);
> if (ret)
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019-uet-ib-core-v2-0-8bd654f25370@broadcom.com?part=10
^ permalink raw reply [flat|nested] 27+ messages in thread
* [PATCH rdma-next v2 11/13] RDMA/core: extended memory registration
2026-10-20 3:03 [PATCH rdma-next v2 00/13] RDMA/core: Ultra Ethernet transport verbs Eric Davis via B4 Relay
` (9 preceding siblings ...)
2026-10-20 3:03 ` [PATCH rdma-next v2 10/13] RDMA/core: queue pair attributes for unconnected transports Eric Davis via B4 Relay
@ 2026-10-20 3:03 ` Eric Davis via B4 Relay
2026-10-20 10:04 ` sashiko-bot
2026-10-20 3:03 ` [PATCH rdma-next v2 12/13] RDMA/core: 64-bit memory keys Eric Davis via B4 Relay
2026-10-20 3:03 ` [PATCH rdma-next v2 13/13] RDMA/core: attach a memory region to a queue pair Eric Davis via B4 Relay
12 siblings, 1 reply; 27+ messages in thread
From: Eric Davis via B4 Relay @ 2026-10-20 3:03 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, linux-kernel, Eric Davis
From: Eric Davis <eric.davis@broadcom.com>
Memory registration for an unconnected transport may take more than an
address and an access mask. A job to bind the region to, a key the caller
chose, or a region carved out of another. None of it fits the write()
command, so add a device method that takes the wider set of attributes
and reach it when the ioctl carries any of these new fields.
A region bound to a job holds a reference to its job key until it is
deregistered.
Signed-off-by: Eric Davis <eric.davis@broadcom.com>
---
drivers/infiniband/core/device.c | 1 +
drivers/infiniband/core/uverbs_cmd.c | 7 +-
drivers/infiniband/core/uverbs_std_types_mr.c | 179 +++++++++++++++++++++++++-
drivers/infiniband/core/verbs.c | 9 ++
include/rdma/ib_verbs.h | 43 +++++++
include/uapi/rdma/ib_user_ioctl_cmds.h | 5 +
include/uapi/rdma/ib_user_ioctl_verbs.h | 10 ++
7 files changed, 249 insertions(+), 5 deletions(-)
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index cf2a59afcf76..fd45decf9c4f 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -3028,6 +3028,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
SET_DEVICE_OP(dev_ops, reg_dm_mr);
SET_DEVICE_OP(dev_ops, reg_user_mr);
SET_DEVICE_OP(dev_ops, reg_user_mr_dmabuf);
+ SET_DEVICE_OP(dev_ops, reg_user_mr_ex);
SET_DEVICE_OP(dev_ops, req_notify_cq);
SET_DEVICE_OP(dev_ops, rereg_user_mr);
SET_DEVICE_OP(dev_ops, resize_user_cq);
diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index a35b743ab9bd..a4793c5d59bf 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -731,6 +731,7 @@ static int ib_uverbs_reg_mr(struct uverbs_attr_bundle *attrs)
mr->type = IB_MR_TYPE_USER;
mr->dm = NULL;
mr->sig_attrs = NULL;
+ ib_mr_init_core_refs(mr);
mr->uobject = uobj;
atomic_inc(&pd->usecnt);
mr->iova = cmd.hca_va;
@@ -790,7 +791,10 @@ static int ib_uverbs_rereg_mr(struct uverbs_attr_bundle *attrs)
mr = uobj->object;
- if (mr->dm) {
+ /* A region bound to a job or to another region is re-created rather
+ * than re-registered, so those bindings can't be lost or bypassed.
+ */
+ if (mr->dm || mr->jkey || mr->parent || atomic_read(&mr->usecnt)) {
ret = -EINVAL;
goto put_uobjs;
}
@@ -838,6 +842,7 @@ static int ib_uverbs_rereg_mr(struct uverbs_attr_bundle *attrs)
new_mr->device = new_pd->device;
new_mr->pd = new_pd;
new_mr->type = IB_MR_TYPE_USER;
+ ib_mr_init_core_refs(new_mr);
new_mr->uobject = uobj;
atomic_inc(&new_pd->usecnt);
new_uobj->object = new_mr;
diff --git a/drivers/infiniband/core/uverbs_std_types_mr.c b/drivers/infiniband/core/uverbs_std_types_mr.c
index 0c72f801e0d3..e595c53fa34c 100644
--- a/drivers/infiniband/core/uverbs_std_types_mr.c
+++ b/drivers/infiniband/core/uverbs_std_types_mr.c
@@ -40,8 +40,13 @@ static int uverbs_free_mr(struct ib_uobject *uobject,
enum rdma_remove_reason why,
struct uverbs_attr_bundle *attrs)
{
- return ib_dereg_mr_user((struct ib_mr *)uobject->object,
- &attrs->driver_udata);
+ struct ib_mr *mr = uobject->object;
+
+ /* regions derived from this one still use it */
+ if (atomic_read(&mr->usecnt))
+ return -EBUSY;
+
+ return ib_dereg_mr_user(mr, &attrs->driver_udata);
}
static int UVERBS_HANDLER(UVERBS_METHOD_ADVISE_MR)(
@@ -132,6 +137,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_DM_MR_REG)(
mr->pd = pd;
mr->type = IB_MR_TYPE_DM;
mr->dm = dm;
+ ib_mr_init_core_refs(mr);
mr->uobject = uobj;
atomic_inc(&pd->usecnt);
atomic_inc(&dm->usecnt);
@@ -246,6 +252,9 @@ static int UVERBS_HANDLER(UVERBS_METHOD_REG_DMABUF_MR)(
mr->device = pd->device;
mr->pd = pd;
mr->type = IB_MR_TYPE_USER;
+ mr->iova = iova;
+ mr->length = length;
+ ib_mr_init_core_refs(mr);
mr->uobject = uobj;
atomic_inc(&pd->usecnt);
@@ -266,6 +275,113 @@ static int UVERBS_HANDLER(UVERBS_METHOD_REG_DMABUF_MR)(
return ret;
}
+/* Attributes that an extended MR registration uses. The 64b wide keys are
+ * not in the list as they are not a property of how the MR was registered
+ * and instead a device property which could be used by an ordinary register.
+ */
+static const u16 reg_mr_ext_attrs[] = {
+ UVERBS_ATTR_REG_MR_FLAGS,
+ UVERBS_ATTR_REG_MR_JKEY_HANDLE,
+ UVERBS_ATTR_REG_MR_PARENT_MR_HANDLE,
+ UVERBS_ATTR_REG_MR_DERIVE_CNT,
+ UVERBS_ATTR_REG_MR_REQ_RKEY64,
+};
+
+static bool reg_mr_has_ext_attrs(struct uverbs_attr_bundle *attrs)
+{
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(reg_mr_ext_attrs); i++)
+ if (uverbs_attr_is_valid(attrs, reg_mr_ext_attrs[i]))
+ return true;
+
+ return false;
+}
+
+static int reg_mr_ex_attr(struct uverbs_attr_bundle *attrs, struct ib_pd *pd,
+ struct ib_mr_ex_attr *ex)
+{
+ u64 caps = pd->device->attrs.device_cap_flags;
+ int ret;
+
+ ret = uverbs_get_flags32(&ex->flags, attrs, UVERBS_ATTR_REG_MR_FLAGS,
+ (IB_UVERBS_REG_MR_USER_RKEY |
+ IB_UVERBS_REG_MR_IDEMPOTENT_SAFE));
+ if (ret)
+ return ret;
+
+ if (uverbs_attr_is_valid(attrs, UVERBS_ATTR_REG_MR_JKEY_HANDLE)) {
+ struct ib_jkey *jkey = uverbs_attr_get_obj(
+ attrs, UVERBS_ATTR_REG_MR_JKEY_HANDLE);
+
+ if (IS_ERR(jkey))
+ return PTR_ERR(jkey);
+
+ /* a key acts within the PD it was made on */
+ if (jkey->pd != pd)
+ return -EINVAL;
+
+ if (!(caps & (IB_DEVICE_MR_JOB_RESTRICTED |
+ IB_DEVICE_MR_QP_JOB_RESTRICTED)))
+ return -EOPNOTSUPP;
+
+ ex->jkey = jkey;
+ }
+
+ /* a region carved out of another, execute the range check */
+ if (uverbs_attr_is_valid(attrs, UVERBS_ATTR_REG_MR_PARENT_MR_HANDLE)) {
+ struct ib_mr *parent = uverbs_attr_get_obj(
+ attrs, UVERBS_ATTR_REG_MR_PARENT_MR_HANDLE);
+
+ if (IS_ERR(parent))
+ return PTR_ERR(parent);
+
+ if (parent->pd != pd || parent->type != IB_MR_TYPE_USER)
+ return -EINVAL;
+
+ if ((ex->iova < parent->iova) ||
+ (ex->length > parent->length) ||
+ ((ex->iova - parent->iova) >
+ (parent->length - ex->length)))
+ return -EINVAL;
+
+ ex->parent_mr = parent;
+ }
+
+ if (uverbs_attr_is_valid(attrs, UVERBS_ATTR_REG_MR_DERIVE_CNT)) {
+ ret = uverbs_copy_from(&ex->derive_cnt, attrs,
+ UVERBS_ATTR_REG_MR_DERIVE_CNT);
+ if (ret)
+ return ret;
+ }
+
+ if (ex->flags & IB_UVERBS_REG_MR_USER_RKEY) {
+ if (!(caps & IB_DEVICE_USER_RKEY))
+ return -EOPNOTSUPP;
+
+ ret = uverbs_copy_from(&ex->req_rkey64, attrs,
+ UVERBS_ATTR_REG_MR_REQ_RKEY64);
+ if (ret)
+ return ret;
+
+ if (ex->req_rkey64 > U32_MAX && !(caps & IB_DEVICE_KEY64))
+ return -EOPNOTSUPP;
+ } else if (uverbs_attr_is_valid(attrs,
+ UVERBS_ATTR_REG_MR_REQ_RKEY64)) {
+ /* a key with no flag to say it is the caller's */
+ return -EINVAL;
+ }
+
+ /* Idempotent-safe is a property of the key so it can only be asked
+ * for on a key the caller chose.
+ */
+ if ((ex->flags & IB_UVERBS_REG_MR_IDEMPOTENT_SAFE) &&
+ !(ex->flags & IB_UVERBS_REG_MR_USER_RKEY))
+ return -EINVAL;
+
+ return 0;
+}
+
static int UVERBS_HANDLER(UVERBS_METHOD_REG_MR)(
struct uverbs_attr_bundle *attrs)
{
@@ -276,10 +392,12 @@ static int UVERBS_HANDLER(UVERBS_METHOD_REG_MR)(
u32 valid_access_flags = IB_ACCESS_SUPPORTED;
u64 length, iova, fd_offset = 0, addr = 0;
struct ib_device *ib_dev = pd->device;
+ struct ib_mr_ex_attr ex = { .fd = -1 };
struct ib_dmah *dmah = NULL;
bool has_fd_offset = false;
bool has_addr = false;
bool has_fd = false;
+ bool has_ext;
u32 access_flags;
struct ib_mr *mr;
int fd;
@@ -293,6 +411,10 @@ static int UVERBS_HANDLER(UVERBS_METHOD_REG_MR)(
if (ret)
return ret;
+ has_ext = reg_mr_has_ext_attrs(attrs);
+ if (has_ext && !ib_dev->ops.reg_user_mr_ex)
+ return -EOPNOTSUPP;
+
if (uverbs_attr_is_valid(attrs, UVERBS_ATTR_REG_MR_ADDR)) {
ret = uverbs_copy_from(&addr, attrs,
UVERBS_ATTR_REG_MR_ADDR);
@@ -318,7 +440,8 @@ static int UVERBS_HANDLER(UVERBS_METHOD_REG_MR)(
}
if (has_fd) {
- if (!ib_dev->ops.reg_user_mr_dmabuf)
+ /* an extended registration hands the fd to reg_user_mr_ex */
+ if (!has_ext && !ib_dev->ops.reg_user_mr_dmabuf)
return -EOPNOTSUPP;
/* FD requires offset and can't come with addr */
@@ -358,7 +481,24 @@ static int UVERBS_HANDLER(UVERBS_METHOD_REG_MR)(
if (ret)
return ret;
- if (has_fd)
+ if (has_ext) {
+ ex.addr = addr;
+ ex.length = length;
+ ex.iova = iova;
+ ex.access_flags = access_flags;
+ ex.dmah = dmah;
+ if (has_fd) {
+ ex.fd = fd;
+ ex.fd_offset = fd_offset;
+ }
+
+ ret = reg_mr_ex_attr(attrs, pd, &ex);
+ if (ret)
+ return ret;
+
+ mr = pd->device->ops.reg_user_mr_ex(pd, &ex,
+ &attrs->driver_udata);
+ } else if (has_fd)
mr = pd->device->ops.reg_user_mr_dmabuf(pd, fd_offset, length,
iova, fd, access_flags,
dmah, attrs);
@@ -373,12 +513,26 @@ static int UVERBS_HANDLER(UVERBS_METHOD_REG_MR)(
mr->device = pd->device;
mr->pd = pd;
mr->type = IB_MR_TYPE_USER;
+ mr->iova = iova;
+ mr->length = length;
+ ib_mr_init_core_refs(mr);
mr->uobject = uobj;
atomic_inc(&pd->usecnt);
if (dmah) {
mr->dmah = dmah;
atomic_inc(&dmah->usecnt);
}
+
+ /* hold a reference to the job key, deregistration will drop it */
+ mr->jkey = ex.jkey;
+ if (ex.jkey)
+ atomic_inc(&ex.jkey->usecnt);
+
+ /* and to the region this one was derived from */
+ mr->parent = ex.parent_mr;
+ if (ex.parent_mr)
+ atomic_inc(&ex.parent_mr->usecnt);
+
rdma_restrack_new(&mr->res, RDMA_RESTRACK_MR);
rdma_restrack_set_name(&mr->res, NULL);
rdma_restrack_add(&mr->res);
@@ -529,6 +683,23 @@ DECLARE_UVERBS_NAMED_METHOD(
UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_REG_MR_RESP_RKEY,
UVERBS_ATTR_TYPE(u32),
UA_MANDATORY),
+ UVERBS_ATTR_FLAGS_IN(UVERBS_ATTR_REG_MR_FLAGS,
+ enum ib_uverbs_reg_mr_flags,
+ UA_OPTIONAL),
+ UVERBS_ATTR_IDR(UVERBS_ATTR_REG_MR_JKEY_HANDLE,
+ UVERBS_OBJECT_JKEY,
+ UVERBS_ACCESS_READ,
+ UA_OPTIONAL),
+ UVERBS_ATTR_IDR(UVERBS_ATTR_REG_MR_PARENT_MR_HANDLE,
+ UVERBS_OBJECT_MR,
+ UVERBS_ACCESS_READ,
+ UA_OPTIONAL),
+ UVERBS_ATTR_PTR_IN(UVERBS_ATTR_REG_MR_DERIVE_CNT,
+ UVERBS_ATTR_TYPE(u32),
+ UA_OPTIONAL),
+ UVERBS_ATTR_PTR_IN(UVERBS_ATTR_REG_MR_REQ_RKEY64,
+ UVERBS_ATTR_TYPE(u64),
+ UA_OPTIONAL),
UVERBS_ATTR_UHW());
DECLARE_UVERBS_NAMED_METHOD_DESTROY(
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index ad281e565499..ca5eac4a96b3 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -2358,6 +2358,7 @@ struct ib_mr *ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
mr->type = IB_MR_TYPE_USER;
mr->pd = pd;
mr->dm = NULL;
+ ib_mr_init_core_refs(mr);
atomic_inc(&pd->usecnt);
mr->iova = virt_addr;
mr->length = length;
@@ -2389,6 +2390,8 @@ int ib_dereg_mr_user(struct ib_mr *mr, struct ib_udata *udata)
struct ib_pd *pd = mr->pd;
struct ib_dm *dm = mr->dm;
struct ib_dmah *dmah = mr->dmah;
+ struct ib_jkey *jkey = mr->jkey;
+ struct ib_mr *parent = mr->parent;
struct ib_sig_attrs *sig_attrs = mr->sig_attrs;
int ret;
@@ -2401,6 +2404,10 @@ int ib_dereg_mr_user(struct ib_mr *mr, struct ib_udata *udata)
atomic_dec(&dm->usecnt);
if (dmah)
atomic_dec(&dmah->usecnt);
+ if (jkey)
+ atomic_dec(&jkey->usecnt);
+ if (parent)
+ atomic_dec(&parent->usecnt);
kfree(sig_attrs);
}
@@ -2448,6 +2455,7 @@ struct ib_mr *ib_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
mr->need_inval = false;
mr->type = mr_type;
mr->sig_attrs = NULL;
+ ib_mr_init_core_refs(mr);
rdma_restrack_new(&mr->res, RDMA_RESTRACK_MR);
rdma_restrack_parent_name(&mr->res, &pd->res);
@@ -2509,6 +2517,7 @@ struct ib_mr *ib_alloc_mr_integrity(struct ib_pd *pd,
mr->need_inval = false;
mr->type = IB_MR_TYPE_INTEGRITY;
mr->sig_attrs = sig_attrs;
+ ib_mr_init_core_refs(mr);
rdma_restrack_new(&mr->res, RDMA_RESTRACK_MR);
rdma_restrack_parent_name(&mr->res, &pd->res);
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 83fa94d579c3..b86ac63fab36 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2084,6 +2084,13 @@ struct ib_mr {
struct list_head qp_entry; /* FR */
};
+ /* the job this region is restricted to, NULL if it is unrestricted */
+ struct ib_jkey *jkey;
+ /* the region this one was derived from, NULL if registered fresh */
+ struct ib_mr *parent;
+ /* regions derived from this one that are still registered */
+ atomic_t usecnt;
+
struct ib_dm *dm;
struct ib_sig_attrs *sig_attrs; /* only for IB_MR_TYPE_INTEGRITY MRs */
struct ib_dmah *dmah;
@@ -2098,6 +2105,25 @@ struct ib_mr {
struct rdma_restrack_entry res;
};
+struct ib_mr_ex_attr {
+ u64 addr;
+ u64 length;
+ u64 iova;
+ u32 access_flags; /* enum ib_access_flags */
+ u32 flags; /* enum ib_uverbs_reg_mr_flags */
+ int fd; /* -1 = host memory addr */
+ u64 fd_offset;
+ struct ib_dmah *dmah;
+ /* bind the region to a job, NULL leaves it unrestricted */
+ struct ib_jkey *jkey;
+ /* carve/derive the region out of this one, NULL registers fresh */
+ struct ib_mr *parent_mr;
+ /* how many regions the user expects to derive from this one */
+ u32 derive_cnt;
+ /* the key the user chose, valid with IB_UVERBS_REG_MR_USER_RKEY */
+ u64 req_rkey64;
+};
+
struct ib_mw {
struct ib_device *device;
struct ib_pd *pd;
@@ -2811,6 +2837,9 @@ struct ib_device_ops {
u64 length, u64 virt_addr,
int mr_access_flags, struct ib_pd *pd,
struct ib_udata *udata);
+ struct ib_mr *(*reg_user_mr_ex)(struct ib_pd *pd,
+ struct ib_mr_ex_attr *attr,
+ struct ib_udata *udata);
int (*dereg_mr)(struct ib_mr *mr, struct ib_udata *udata);
int (*query_qp_semantics)(struct ib_device *device,
enum ib_qp_type qp_type,
@@ -4820,6 +4849,20 @@ static inline u32 ib_inc_rkey(u32 rkey)
return ((rkey + 1) & mask) | (rkey & ~mask);
}
+/**
+ * ib_mr_init_core_refs - clears what core reads back when an MR is
+ * deregistered.
+ * @mr: struct ib_mr pointer to initialize.
+ *
+ * A driver's MR need not be zeroed, so every path that makes an MR calls this.
+ */
+static inline void ib_mr_init_core_refs(struct ib_mr *mr)
+{
+ mr->jkey = NULL;
+ mr->parent = NULL;
+ atomic_set(&mr->usecnt, 0);
+}
+
/**
* ib_attach_mcast - Attaches the specified QP to a multicast group.
* @qp: QP to attach to the multicast group. The QP must be type
diff --git a/include/uapi/rdma/ib_user_ioctl_cmds.h b/include/uapi/rdma/ib_user_ioctl_cmds.h
index 1de6a1ffd92c..fd8372c0c550 100644
--- a/include/uapi/rdma/ib_user_ioctl_cmds.h
+++ b/include/uapi/rdma/ib_user_ioctl_cmds.h
@@ -369,6 +369,11 @@ enum uverbs_attrs_reg_mr_cmd_attr_ids {
UVERBS_ATTR_REG_MR_FD_OFFSET,
UVERBS_ATTR_REG_MR_RESP_LKEY,
UVERBS_ATTR_REG_MR_RESP_RKEY,
+ UVERBS_ATTR_REG_MR_FLAGS,
+ UVERBS_ATTR_REG_MR_JKEY_HANDLE,
+ UVERBS_ATTR_REG_MR_PARENT_MR_HANDLE,
+ UVERBS_ATTR_REG_MR_DERIVE_CNT,
+ UVERBS_ATTR_REG_MR_REQ_RKEY64,
};
enum uverbs_attrs_create_counters_cmd_attr_ids {
diff --git a/include/uapi/rdma/ib_user_ioctl_verbs.h b/include/uapi/rdma/ib_user_ioctl_verbs.h
index 3d05a6c7ca8f..75cdfdbdb2f7 100644
--- a/include/uapi/rdma/ib_user_ioctl_verbs.h
+++ b/include/uapi/rdma/ib_user_ioctl_verbs.h
@@ -303,6 +303,16 @@ struct ib_uverbs_buffer_desc {
__aligned_u64 length;
};
+/* Extended memory registration, for devices whose memory keys are 64 bits
+ * wide and may be bound to a job. See UVERBS_ATTR_REG_MR_FLAGS.
+ */
+enum ib_uverbs_reg_mr_flags {
+ /* REQ_RKEY64 carries the key, rather than the device assigning one */
+ IB_UVERBS_REG_MR_USER_RKEY = 1 << 0,
+ /* the region tolerates a replayed idempotent operation */
+ IB_UVERBS_REG_MR_IDEMPOTENT_SAFE = 1 << 1,
+};
+
/* A UET reliable unconnected queue pair number uses all 32 bits:
*
* [ absolute 1b | reserved 7b | resource index 12b | PIDonFEP 12b ]
--
2.56.0
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH rdma-next v2 12/13] RDMA/core: 64-bit memory keys
2026-10-20 3:03 [PATCH rdma-next v2 00/13] RDMA/core: Ultra Ethernet transport verbs Eric Davis via B4 Relay
` (10 preceding siblings ...)
2026-10-20 3:03 ` [PATCH rdma-next v2 11/13] RDMA/core: extended memory registration Eric Davis via B4 Relay
@ 2026-10-20 3:03 ` Eric Davis via B4 Relay
2026-10-20 10:09 ` sashiko-bot
2026-10-20 3:03 ` [PATCH rdma-next v2 13/13] RDMA/core: attach a memory region to a queue pair Eric Davis via B4 Relay
12 siblings, 1 reply; 27+ messages in thread
From: Eric Davis via B4 Relay @ 2026-10-20 3:03 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, linux-kernel, Eric Davis
From: Eric Davis <eric.davis@broadcom.com>
A device whose keys are wider than 32 bits has no way to name a region
through lkey and rkey. Carry the wide pair on the region and return it
from registration when userspace asks for it.
The attributes are optional in both directions, so a kernel that does not
know them ignores the request, and a region without a wide key answers
zero.
Signed-off-by: Eric Davis <eric.davis@broadcom.com>
---
drivers/infiniband/core/uverbs_std_types_mr.c | 39 ++++++++++++++++++++++++++-
include/rdma/ib_verbs.h | 12 +++++++--
include/uapi/rdma/ib_user_ioctl_cmds.h | 4 +++
3 files changed, 52 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/core/uverbs_std_types_mr.c b/drivers/infiniband/core/uverbs_std_types_mr.c
index e595c53fa34c..1ff45556e0b2 100644
--- a/drivers/infiniband/core/uverbs_std_types_mr.c
+++ b/drivers/infiniband/core/uverbs_std_types_mr.c
@@ -185,6 +185,16 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_MR)(
ret = uverbs_copy_to(attrs, UVERBS_ATTR_QUERY_MR_RESP_IOVA,
&mr->iova, sizeof(mr->iova));
+ if (IS_UVERBS_COPY_ERR(ret))
+ return ret;
+
+ ret = uverbs_copy_to(attrs, UVERBS_ATTR_QUERY_MR_RESP_LKEY64,
+ &mr->lkey64, sizeof(mr->lkey64));
+ if (IS_UVERBS_COPY_ERR(ret))
+ return ret;
+
+ ret = uverbs_copy_to(attrs, UVERBS_ATTR_QUERY_MR_RESP_RKEY64,
+ &mr->rkey64, sizeof(mr->rkey64));
return IS_UVERBS_COPY_ERR(ret) ? ret : 0;
}
@@ -364,6 +374,10 @@ static int reg_mr_ex_attr(struct uverbs_attr_bundle *attrs, struct ib_pd *pd,
if (ret)
return ret;
+ /* zero names no wide key, so it can't be one */
+ if (!ex->req_rkey64)
+ return -EINVAL;
+
if (ex->req_rkey64 > U32_MAX && !(caps & IB_DEVICE_KEY64))
return -EOPNOTSUPP;
} else if (uverbs_attr_is_valid(attrs,
@@ -547,7 +561,18 @@ static int UVERBS_HANDLER(UVERBS_METHOD_REG_MR)(
ret = uverbs_copy_to(attrs, UVERBS_ATTR_REG_MR_RESP_RKEY,
&mr->rkey, sizeof(mr->rkey));
- return ret;
+ if (ret)
+ return ret;
+
+ ret = uverbs_copy_to(attrs, UVERBS_ATTR_REG_MR_RESP_LKEY64,
+ &mr->lkey64, sizeof(mr->lkey64));
+ if (IS_UVERBS_COPY_ERR(ret))
+ return ret;
+
+ ret = uverbs_copy_to(attrs, UVERBS_ATTR_REG_MR_RESP_RKEY64,
+ &mr->rkey64, sizeof(mr->rkey64));
+
+ return IS_UVERBS_COPY_ERR(ret) ? ret : 0;
}
DECLARE_UVERBS_NAMED_METHOD(
@@ -583,6 +608,12 @@ DECLARE_UVERBS_NAMED_METHOD(
UVERBS_ATTR_TYPE(u64),
UA_MANDATORY),
UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_QUERY_MR_RESP_IOVA,
+ UVERBS_ATTR_TYPE(u64),
+ UA_OPTIONAL),
+ UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_QUERY_MR_RESP_LKEY64,
+ UVERBS_ATTR_TYPE(u64),
+ UA_OPTIONAL),
+ UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_QUERY_MR_RESP_RKEY64,
UVERBS_ATTR_TYPE(u64),
UA_OPTIONAL));
@@ -700,6 +731,12 @@ DECLARE_UVERBS_NAMED_METHOD(
UVERBS_ATTR_PTR_IN(UVERBS_ATTR_REG_MR_REQ_RKEY64,
UVERBS_ATTR_TYPE(u64),
UA_OPTIONAL),
+ UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_REG_MR_RESP_LKEY64,
+ UVERBS_ATTR_TYPE(u64),
+ UA_OPTIONAL),
+ UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_REG_MR_RESP_RKEY64,
+ UVERBS_ATTR_TYPE(u64),
+ UA_OPTIONAL),
UVERBS_ATTR_UHW());
DECLARE_UVERBS_NAMED_METHOD_DESTROY(
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index b86ac63fab36..725b630a3b90 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2084,6 +2084,9 @@ struct ib_mr {
struct list_head qp_entry; /* FR */
};
+ /* 64b keys, set by a driver reporting IB_DEVICE_KEY64, else zero */
+ u64 lkey64;
+ u64 rkey64;
/* the job this region is restricted to, NULL if it is unrestricted */
struct ib_jkey *jkey;
/* the region this one was derived from, NULL if registered fresh */
@@ -4851,8 +4854,8 @@ static inline u32 ib_inc_rkey(u32 rkey)
/**
* ib_mr_init_core_refs - clears what core reads back when an MR is
- * deregistered.
- * @mr: struct ib_mr pointer to initialize.
+ * deregistered, and the wide keys of an MR on a device without them.
+ * @mr: struct ib_mr pointer to initialize, with its device set.
*
* A driver's MR need not be zeroed, so every path that makes an MR calls this.
*/
@@ -4861,6 +4864,11 @@ static inline void ib_mr_init_core_refs(struct ib_mr *mr)
mr->jkey = NULL;
mr->parent = NULL;
atomic_set(&mr->usecnt, 0);
+
+ if (!(mr->device->attrs.device_cap_flags & IB_DEVICE_KEY64)) {
+ mr->lkey64 = 0;
+ mr->rkey64 = 0;
+ }
}
/**
diff --git a/include/uapi/rdma/ib_user_ioctl_cmds.h b/include/uapi/rdma/ib_user_ioctl_cmds.h
index fd8372c0c550..bc2797fc37fe 100644
--- a/include/uapi/rdma/ib_user_ioctl_cmds.h
+++ b/include/uapi/rdma/ib_user_ioctl_cmds.h
@@ -343,6 +343,8 @@ enum uverbs_attrs_query_mr_cmd_attr_ids {
UVERBS_ATTR_QUERY_MR_RESP_RKEY,
UVERBS_ATTR_QUERY_MR_RESP_LENGTH,
UVERBS_ATTR_QUERY_MR_RESP_IOVA,
+ UVERBS_ATTR_QUERY_MR_RESP_LKEY64,
+ UVERBS_ATTR_QUERY_MR_RESP_RKEY64,
};
enum uverbs_attrs_reg_dmabuf_mr_cmd_attr_ids {
@@ -374,6 +376,8 @@ enum uverbs_attrs_reg_mr_cmd_attr_ids {
UVERBS_ATTR_REG_MR_PARENT_MR_HANDLE,
UVERBS_ATTR_REG_MR_DERIVE_CNT,
UVERBS_ATTR_REG_MR_REQ_RKEY64,
+ UVERBS_ATTR_REG_MR_RESP_LKEY64,
+ UVERBS_ATTR_REG_MR_RESP_RKEY64,
};
enum uverbs_attrs_create_counters_cmd_attr_ids {
--
2.56.0
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH rdma-next v2 13/13] RDMA/core: attach a memory region to a queue pair
2026-10-20 3:03 [PATCH rdma-next v2 00/13] RDMA/core: Ultra Ethernet transport verbs Eric Davis via B4 Relay
` (11 preceding siblings ...)
2026-10-20 3:03 ` [PATCH rdma-next v2 12/13] RDMA/core: 64-bit memory keys Eric Davis via B4 Relay
@ 2026-10-20 3:03 ` Eric Davis via B4 Relay
2026-10-20 10:21 ` sashiko-bot
12 siblings, 1 reply; 27+ messages in thread
From: Eric Davis via B4 Relay @ 2026-10-20 3:03 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky; +Cc: linux-rdma, linux-kernel, Eric Davis
From: Eric Davis <eric.davis@broadcom.com>
A region registered against a protection domain is reachable from every QP
in it. When a device can narrow that, let a region be attached to one QP
and detached again, so the network only reaches the MR over that QP.
Which restrictions a device supports is reported through the memory region
access capability bits.
Signed-off-by: Eric Davis <eric.davis@broadcom.com>
---
drivers/infiniband/core/device.c | 2 +
drivers/infiniband/core/uverbs_std_types_mr.c | 2 +-
drivers/infiniband/core/uverbs_std_types_qp.c | 110 +++++++++++++++++++++++++-
drivers/infiniband/core/verbs.c | 10 +++
include/rdma/ib_verbs.h | 11 ++-
include/uapi/rdma/ib_user_ioctl_cmds.h | 12 +++
6 files changed, 144 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index fd45decf9c4f..1b838bf3600d 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -3018,6 +3018,8 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
SET_DEVICE_OP(dev_ops, query_pkey);
SET_DEVICE_OP(dev_ops, query_port);
SET_DEVICE_OP(dev_ops, query_port_speed);
+ SET_DEVICE_OP(dev_ops, qp_attach_mr);
+ SET_DEVICE_OP(dev_ops, qp_detach_mr);
SET_DEVICE_OP(dev_ops, query_qp);
SET_DEVICE_OP(dev_ops, query_qp_semantics);
SET_DEVICE_OP(dev_ops, query_srq);
diff --git a/drivers/infiniband/core/uverbs_std_types_mr.c b/drivers/infiniband/core/uverbs_std_types_mr.c
index 1ff45556e0b2..cb33d4e66507 100644
--- a/drivers/infiniband/core/uverbs_std_types_mr.c
+++ b/drivers/infiniband/core/uverbs_std_types_mr.c
@@ -42,7 +42,7 @@ static int uverbs_free_mr(struct ib_uobject *uobject,
{
struct ib_mr *mr = uobject->object;
- /* regions derived from this one still use it */
+ /* regions derived from this one, or the QP it is attached to, use it */
if (atomic_read(&mr->usecnt))
return -EBUSY;
diff --git a/drivers/infiniband/core/uverbs_std_types_qp.c b/drivers/infiniband/core/uverbs_std_types_qp.c
index 47ca994f9ac9..4593965069f1 100644
--- a/drivers/infiniband/core/uverbs_std_types_qp.c
+++ b/drivers/infiniband/core/uverbs_std_types_qp.c
@@ -666,12 +666,120 @@ DECLARE_UVERBS_NAMED_METHOD(
enum ib_uverbs_qp_attach_comp_cntr_op,
UA_MANDATORY));
+/* bind a region to the QP that will use it */
+static int UVERBS_HANDLER(UVERBS_METHOD_QP_ATTACH_MR)(
+ struct uverbs_attr_bundle *attrs)
+{
+ struct ib_qp *qp =
+ uverbs_attr_get_obj(attrs, UVERBS_ATTR_QP_ATTACH_MR_QP_HANDLE);
+ struct ib_mr *mr =
+ uverbs_attr_get_obj(attrs, UVERBS_ATTR_QP_ATTACH_MR_MR_HANDLE);
+ u64 caps = qp->device->attrs.device_cap_flags;
+ u32 id;
+ int ret;
+
+ if (!qp->device->ops.qp_attach_mr)
+ return -EOPNOTSUPP;
+
+ if (qp->real_qp != qp || qp->qp_type != IB_QPT_RU)
+ return -EINVAL;
+
+ if (mr->pd != qp->pd || mr->type != IB_MR_TYPE_USER)
+ return -EINVAL;
+
+ /* the device has to implement the access class the region asks for */
+ if (!(caps & (mr->jkey ? IB_DEVICE_MR_QP_JOB_RESTRICTED :
+ IB_DEVICE_MR_QP_RESTRICTED)))
+ return -EOPNOTSUPP;
+
+ /* a region restricted to a job serves a QP in that job */
+ if (mr->jkey && (!qp->jkey || qp->jkey->job != mr->jkey->job))
+ return -EINVAL;
+
+ /* one QP at a time */
+ if (cmpxchg(&mr->attached_qp, NULL, qp))
+ return -EBUSY;
+
+ ret = xa_alloc(&qp->attached_mrs, &id, mr, xa_limit_32b, GFP_KERNEL);
+ if (ret)
+ goto err_claim;
+
+ ret = qp->device->ops.qp_attach_mr(qp, mr);
+ if (ret)
+ goto err_xa;
+
+ /* the QP holds the region until detach or its own destroy */
+ mr->attach_id = id;
+ atomic_inc(&mr->usecnt);
+
+ return 0;
+
+err_xa:
+ xa_erase(&qp->attached_mrs, id);
+err_claim:
+ WRITE_ONCE(mr->attached_qp, NULL);
+ return ret;
+}
+
+static int UVERBS_HANDLER(UVERBS_METHOD_QP_DETACH_MR)(
+ struct uverbs_attr_bundle *attrs)
+{
+ struct ib_qp *qp =
+ uverbs_attr_get_obj(attrs, UVERBS_ATTR_QP_DETACH_MR_QP_HANDLE);
+ struct ib_mr *mr =
+ uverbs_attr_get_obj(attrs, UVERBS_ATTR_QP_DETACH_MR_MR_HANDLE);
+ int ret;
+
+ if (!qp->device->ops.qp_detach_mr)
+ return -EOPNOTSUPP;
+
+ if (qp->real_qp != qp)
+ return -EINVAL;
+
+ if (READ_ONCE(mr->attached_qp) != qp)
+ return -EINVAL;
+
+ ret = qp->device->ops.qp_detach_mr(qp, mr);
+ if (ret)
+ return ret;
+
+ xa_erase(&qp->attached_mrs, mr->attach_id);
+ WRITE_ONCE(mr->attached_qp, NULL);
+ atomic_dec(&mr->usecnt);
+
+ return 0;
+}
+
+DECLARE_UVERBS_NAMED_METHOD(
+ UVERBS_METHOD_QP_ATTACH_MR,
+ UVERBS_ATTR_IDR(UVERBS_ATTR_QP_ATTACH_MR_QP_HANDLE,
+ UVERBS_OBJECT_QP,
+ UVERBS_ACCESS_WRITE,
+ UA_MANDATORY),
+ UVERBS_ATTR_IDR(UVERBS_ATTR_QP_ATTACH_MR_MR_HANDLE,
+ UVERBS_OBJECT_MR,
+ UVERBS_ACCESS_READ,
+ UA_MANDATORY));
+
+DECLARE_UVERBS_NAMED_METHOD(
+ UVERBS_METHOD_QP_DETACH_MR,
+ UVERBS_ATTR_IDR(UVERBS_ATTR_QP_DETACH_MR_QP_HANDLE,
+ UVERBS_OBJECT_QP,
+ UVERBS_ACCESS_WRITE,
+ UA_MANDATORY),
+ UVERBS_ATTR_IDR(UVERBS_ATTR_QP_DETACH_MR_MR_HANDLE,
+ UVERBS_OBJECT_MR,
+ UVERBS_ACCESS_READ,
+ UA_MANDATORY));
+
DECLARE_UVERBS_NAMED_OBJECT(
UVERBS_OBJECT_QP,
UVERBS_TYPE_ALLOC_IDR_SZ(sizeof(struct ib_uqp_object), uverbs_free_qp),
&UVERBS_METHOD(UVERBS_METHOD_QP_CREATE),
&UVERBS_METHOD(UVERBS_METHOD_QP_DESTROY),
- &UVERBS_METHOD(UVERBS_METHOD_QP_ATTACH_COMP_CNTR));
+ &UVERBS_METHOD(UVERBS_METHOD_QP_ATTACH_COMP_CNTR),
+ &UVERBS_METHOD(UVERBS_METHOD_QP_ATTACH_MR),
+ &UVERBS_METHOD(UVERBS_METHOD_QP_DETACH_MR));
const struct uapi_definition uverbs_def_obj_qp[] = {
UAPI_DEF_CHAIN_OBJ_TREE_NAMED(UVERBS_OBJECT_QP,
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index ca5eac4a96b3..adcda33b04f2 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -1351,6 +1351,7 @@ static struct ib_qp *create_qp(struct ib_device *dev, struct ib_pd *pd,
spin_lock_init(&qp->mr_lock);
xa_init(&qp->comp_cntrs);
+ xa_init_flags(&qp->attached_mrs, XA_FLAGS_ALLOC);
INIT_LIST_HEAD(&qp->rdma_mrs);
INIT_LIST_HEAD(&qp->sig_mrs);
init_completion(&qp->srq_completion);
@@ -1385,6 +1386,7 @@ static struct ib_qp *create_qp(struct ib_device *dev, struct ib_pd *pd,
err_create:
rdma_restrack_put(&qp->res);
xa_destroy(&qp->comp_cntrs);
+ xa_destroy(&qp->attached_mrs);
kfree(qp);
return ERR_PTR(ret);
@@ -2212,6 +2214,7 @@ int ib_destroy_qp_user(struct ib_qp *qp, struct ib_udata *udata)
struct ib_qp_security *sec;
struct ib_comp_cntr *cc;
unsigned long index;
+ struct ib_mr *mr;
int ret;
WARN_ON_ONCE(qp->mrs_used > 0);
@@ -2249,6 +2252,13 @@ int ib_destroy_qp_user(struct ib_qp *qp, struct ib_udata *udata)
atomic_dec(&cc->usecnt);
xa_destroy(&qp->comp_cntrs);
+ /* the driver ended the bindings with the QP, release the regions */
+ xa_for_each(&qp->attached_mrs, index, mr) {
+ WRITE_ONCE(mr->attached_qp, NULL);
+ atomic_dec(&mr->usecnt);
+ }
+ xa_destroy(&qp->attached_mrs);
+
ib_qp_usecnt_dec(qp);
if (sec)
ib_destroy_qp_security_end(sec);
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 725b630a3b90..a93715323ad5 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2003,6 +2003,8 @@ struct ib_qp {
struct list_head xrcd_list;
struct xarray comp_cntrs; /* op_mask -> comp_cntr */
u32 comp_cntr_op_mask;
+ /* regions attached to this QP, each held until detach or destroy */
+ struct xarray attached_mrs;
/* count times opened, mcast attaches, flow attaches */
atomic_t usecnt;
@@ -2091,8 +2093,11 @@ struct ib_mr {
struct ib_jkey *jkey;
/* the region this one was derived from, NULL if registered fresh */
struct ib_mr *parent;
- /* regions derived from this one that are still registered */
+ /* regions derived from this one, and an attachment, that hold it */
atomic_t usecnt;
+ /* the QP this region is attached to, NULL if none */
+ struct ib_qp *attached_qp;
+ u32 attach_id; /* index in attached_qp->attached_mrs */
struct ib_dm *dm;
struct ib_sig_attrs *sig_attrs; /* only for IB_MR_TYPE_INTEGRITY MRs */
@@ -2844,6 +2849,9 @@ struct ib_device_ops {
struct ib_mr_ex_attr *attr,
struct ib_udata *udata);
int (*dereg_mr)(struct ib_mr *mr, struct ib_udata *udata);
+ /* attach/detach a region so it is bound to single QP (restricted) */
+ int (*qp_attach_mr)(struct ib_qp *qp, struct ib_mr *mr);
+ int (*qp_detach_mr)(struct ib_qp *qp, struct ib_mr *mr);
int (*query_qp_semantics)(struct ib_device *device,
enum ib_qp_type qp_type,
struct ib_qp_semantics *sem);
@@ -4863,6 +4871,7 @@ static inline void ib_mr_init_core_refs(struct ib_mr *mr)
{
mr->jkey = NULL;
mr->parent = NULL;
+ mr->attached_qp = NULL;
atomic_set(&mr->usecnt, 0);
if (!(mr->device->attrs.device_cap_flags & IB_DEVICE_KEY64)) {
diff --git a/include/uapi/rdma/ib_user_ioctl_cmds.h b/include/uapi/rdma/ib_user_ioctl_cmds.h
index bc2797fc37fe..8de110b8db34 100644
--- a/include/uapi/rdma/ib_user_ioctl_cmds.h
+++ b/include/uapi/rdma/ib_user_ioctl_cmds.h
@@ -200,6 +200,18 @@ enum uverbs_methods_qp {
UVERBS_METHOD_QP_CREATE,
UVERBS_METHOD_QP_DESTROY,
UVERBS_METHOD_QP_ATTACH_COMP_CNTR,
+ UVERBS_METHOD_QP_ATTACH_MR,
+ UVERBS_METHOD_QP_DETACH_MR,
+};
+
+enum uverbs_attrs_qp_attach_mr_cmd_attr_ids {
+ UVERBS_ATTR_QP_ATTACH_MR_QP_HANDLE,
+ UVERBS_ATTR_QP_ATTACH_MR_MR_HANDLE,
+};
+
+enum uverbs_attrs_qp_detach_mr_cmd_attr_ids {
+ UVERBS_ATTR_QP_DETACH_MR_QP_HANDLE,
+ UVERBS_ATTR_QP_DETACH_MR_MR_HANDLE,
};
enum uverbs_attrs_create_srq_cmd_attr_ids {
--
2.56.0
^ permalink raw reply related [flat|nested] 27+ messages in thread* Re: [PATCH rdma-next v2 13/13] RDMA/core: attach a memory region to a queue pair
2026-10-20 3:03 ` [PATCH rdma-next v2 13/13] RDMA/core: attach a memory region to a queue pair Eric Davis via B4 Relay
@ 2026-10-20 10:21 ` sashiko-bot
0 siblings, 0 replies; 27+ messages in thread
From: sashiko-bot @ 2026-10-20 10:21 UTC (permalink / raw)
To: Eric Davis; +Cc: linux-rdma
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] RDMA/core: UAF and race releasing mr->attached_qp in ib_destroy_qp_user()
--
commit 42577e79697ce5d46b89cf9d6635da521b34d381
Author: Eric Davis <eric.davis@broadcom.com>
RDMA/core: attach a memory region to a queue pair
This commit adds uverbs methods and driver callbacks to attach and detach a
user memory region to a single queue pair, restricting access to that
region to the bound queue pair.
> diff --git a/drivers/infiniband/core/uverbs_std_types_mr.c b/drivers/infiniband/core/uverbs_std_types_mr.c
> index 1ff45556e0b21..cb33d4e665072 100644
> --- a/drivers/infiniband/core/uverbs_std_types_mr.c
> +++ b/drivers/infiniband/core/uverbs_std_types_mr.c
> @@ -42,7 +42,7 @@ static int uverbs_free_mr(struct ib_uobject *uobject,
> {
> struct ib_mr *mr = uobject->object;
>
> - /* regions derived from this one still use it */
> + /* regions derived from this one, or the QP it is attached to, use it */
> if (atomic_read(&mr->usecnt))
> return -EBUSY;
>
> diff --git a/drivers/infiniband/core/uverbs_std_types_qp.c b/drivers/infiniband/core/uverbs_std_types_qp.c
> index 47ca994f9ac96..4593965069f19 100644
> --- a/drivers/infiniband/core/uverbs_std_types_qp.c
> +++ b/drivers/infiniband/core/uverbs_std_types_qp.c
> @@ -666,12 +666,120 @@ DECLARE_UVERBS_NAMED_METHOD(
> enum ib_uverbs_qp_attach_comp_cntr_op,
> UA_MANDATORY));
>
> +/* bind a region to the QP that will use it */
> +static int UVERBS_HANDLER(UVERBS_METHOD_QP_ATTACH_MR)(
> + struct uverbs_attr_bundle *attrs)
> +{
> + struct ib_qp *qp =
> + uverbs_attr_get_obj(attrs, UVERBS_ATTR_QP_ATTACH_MR_QP_HANDLE);
> + struct ib_mr *mr =
> + uverbs_attr_get_obj(attrs, UVERBS_ATTR_QP_ATTACH_MR_MR_HANDLE);
> + u64 caps = qp->device->attrs.device_cap_flags;
> + u32 id;
> + int ret;
> +
> + if (!qp->device->ops.qp_attach_mr)
> + return -EOPNOTSUPP;
> +
> + if (qp->real_qp != qp || qp->qp_type != IB_QPT_RU)
> + return -EINVAL;
> +
> + if (mr->pd != qp->pd || mr->type != IB_MR_TYPE_USER)
> + return -EINVAL;
> +
> + /* the device has to implement the access class the region asks for */
> + if (!(caps & (mr->jkey ? IB_DEVICE_MR_QP_JOB_RESTRICTED :
> + IB_DEVICE_MR_QP_RESTRICTED)))
> + return -EOPNOTSUPP;
> +
> + /* a region restricted to a job serves a QP in that job */
> + if (mr->jkey && (!qp->jkey || qp->jkey->job != mr->jkey->job))
> + return -EINVAL;
> +
> + /* one QP at a time */
> + if (cmpxchg(&mr->attached_qp, NULL, qp))
> + return -EBUSY;
> +
> + ret = xa_alloc(&qp->attached_mrs, &id, mr, xa_limit_32b, GFP_KERNEL);
> + if (ret)
> + goto err_claim;
> +
> + ret = qp->device->ops.qp_attach_mr(qp, mr);
> + if (ret)
> + goto err_xa;
> +
> + /* the QP holds the region until detach or its own destroy */
> + mr->attach_id = id;
> + atomic_inc(&mr->usecnt);
> +
> + return 0;
> +
> +err_xa:
> + xa_erase(&qp->attached_mrs, id);
> +err_claim:
> + WRITE_ONCE(mr->attached_qp, NULL);
> + return ret;
> +}
> +
> +static int UVERBS_HANDLER(UVERBS_METHOD_QP_DETACH_MR)(
> + struct uverbs_attr_bundle *attrs)
> +{
> + struct ib_qp *qp =
> + uverbs_attr_get_obj(attrs, UVERBS_ATTR_QP_DETACH_MR_QP_HANDLE);
> + struct ib_mr *mr =
> + uverbs_attr_get_obj(attrs, UVERBS_ATTR_QP_DETACH_MR_MR_HANDLE);
> + int ret;
> +
> + if (!qp->device->ops.qp_detach_mr)
> + return -EOPNOTSUPP;
> +
> + if (qp->real_qp != qp)
> + return -EINVAL;
> +
> + if (READ_ONCE(mr->attached_qp) != qp)
> + return -EINVAL;
> +
> + ret = qp->device->ops.qp_detach_mr(qp, mr);
> + if (ret)
> + return ret;
> +
> + xa_erase(&qp->attached_mrs, mr->attach_id);
> + WRITE_ONCE(mr->attached_qp, NULL);
> + atomic_dec(&mr->usecnt);
> +
> + return 0;
> +}
[ ... ]
> diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
> index ca5eac4a96b3c..adcda33b04f2b 100644
> --- a/drivers/infiniband/core/verbs.c
> +++ b/drivers/infiniband/core/verbs.c
[ ... ]
> @@ -2249,6 +2252,13 @@ int ib_destroy_qp_user(struct ib_qp *qp, struct ib_udata *udata)
> atomic_dec(&cc->usecnt);
> xa_destroy(&qp->comp_cntrs);
>
> + /* the driver ended the bindings with the QP, release the regions */
> + xa_for_each(&qp->attached_mrs, index, mr) {
> + WRITE_ONCE(mr->attached_qp, NULL);
> + atomic_dec(&mr->usecnt);
> + }
> + xa_destroy(&qp->attached_mrs);
[Severity: Critical]
Can this sequence in ib_destroy_qp_user() cause a use-after-free write to
mr->attached_qp or a race on mr->usecnt?
In ib_destroy_qp_user(), only the QP's ib_uobject is locked, and no lock is
held on the attached MRs. If one thread destroys the QP while another thread
concurrently destroys the MR via uverbs_free_mr():
CPU0 CPU1
ib_destroy_qp_user() uverbs_free_mr()
WRITE_ONCE(mr->attached_qp, NULL);
atomic_dec(&mr->usecnt);
if (atomic_read(&mr->usecnt))
return -EBUSY;
ib_dereg_mr_user(mr, ...)
-> dereg_mr(mr, ...)
-> kfree(mr)
Because non-value-returning atomic_dec() and atomic_read() are relaxed
operations without release or acquire ordering, on weakly-ordered
architectures atomic_dec(&mr->usecnt) can be reordered before
WRITE_ONCE(mr->attached_qp, NULL) becomes visible. CPU1 can observe
atomic_read(&mr->usecnt) == 0 and free mr before CPU0's store to
mr->attached_qp completes.
Additionally, both ib_destroy_qp_user() and
UVERBS_HANDLER(UVERBS_METHOD_QP_DETACH_MR) clear mr->attached_qp before
calling atomic_dec(&mr->usecnt), while UVERBS_METHOD_QP_ATTACH_MR and
UVERBS_METHOD_QP_DETACH_MR only hold a shared UVERBS_ACCESS_READ lock on the
MR's uobject. Once mr->attached_qp is cleared to NULL, another thread can
attach and detach the MR on a second QP via cmpxchg(&mr->attached_qp, NULL,
qp2) and then call uverbs_free_mr() or ib_uverbs_rereg_mr() before the first
thread runs atomic_dec(&mr->usecnt), causing a spurious -EBUSY or -EINVAL
failure while mr->usecnt is still elevated.
Should mr->usecnt be decremented before clearing mr->attached_qp (or with
proper ordering / exclusive uobject locking on the MR) so that mr cannot be
freed or re-attached while the prior detach or QP teardown is still touching
it?
> +
> ib_qp_usecnt_dec(qp);
> if (sec)
> ib_destroy_qp_security_end(sec);
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019-uet-ib-core-v2-0-8bd654f25370@broadcom.com?part=13
^ permalink raw reply [flat|nested] 27+ messages in thread