[RFC RESEND iproute2-next 0/2] Supports to add/delete IB devices with type SMI

[RFC RESEND iproute2-next 0/2] Supports to add/delete IB devices with type SMI

[Mark Zhang]

This series supports to add/delete an IB device with type SMI. This is
complimentary to the kernel patches that support to IB sub device
and mlx5 implementation.

https://lore.kernel.org/all/cover.1718553901.git.leon@kernel.org/

Thanks

Mark Zhang (2):
  rdma: update uapi header
  rdma: Supports to add/delete a device with type SMI

 man/man8/rdma-dev.8                   |  40 +++++++++
 rdma/dev.c                            | 120 ++++++++++++++++++++++++++
 rdma/include/uapi/rdma/rdma_netlink.h |  13 +++
 rdma/rdma.h                           |   2 +
 rdma/utils.c                          |   2 +
 5 files changed, 177 insertions(+)

-- 
2.26.3

[RFC RESEND iproute2-next 1/2] rdma: update uapi header

[Mark Zhang]

Update rdma_netlink.h file upto kernel commit 294424839b5e
("RDMA/nldev: Add support to dump device type and parent device if exists")

Signed-off-by: Mark Zhang <markzhang@nvidia.com>
---
 rdma/include/uapi/rdma/rdma_netlink.h | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/rdma/include/uapi/rdma/rdma_netlink.h b/rdma/include/uapi/rdma/rdma_netlink.h
index a6c8a52d..aea1d5ef 100644
--- a/rdma/include/uapi/rdma/rdma_netlink.h
+++ b/rdma/include/uapi/rdma/rdma_netlink.h
@@ -301,6 +301,10 @@ enum rdma_nldev_command {
 
 	RDMA_NLDEV_CMD_RES_SRQ_GET_RAW,
 
+	RDMA_NLDEV_CMD_NEWDEV,
+
+	RDMA_NLDEV_CMD_DELDEV,
+
 	RDMA_NLDEV_NUM_OPS
 };
 
@@ -564,6 +568,10 @@ enum rdma_nldev_attr {
 	 */
 	RDMA_NLDEV_ATTR_RES_SUBTYPE,		/* string */
 
+	RDMA_NLDEV_ATTR_DEV_TYPE,		/* u8 */
+
+	RDMA_NLDEV_ATTR_PARENT_NAME,		/* string */
+
 	/*
 	 * Always the end
 	 */
@@ -602,4 +610,9 @@ enum rdma_nl_counter_mask {
 	RDMA_COUNTER_MASK_QP_TYPE = 1,
 	RDMA_COUNTER_MASK_PID = 1 << 1,
 };
+
+/* Supported rdma device types. */
+enum rdma_nl_dev_type {
+	RDMA_DEVICE_TYPE_SMI = 1,
+};
 #endif /* _RDMA_NETLINK_H */
-- 
2.26.3

Re: [RFC RESEND iproute2-next 1/2] rdma: update uapi header

[Leon Romanovsky]

On Thu, Jul 04, 2024 at 09:29:00AM +0300, Mark Zhang wrote:
> Update rdma_netlink.h file upto kernel commit 294424839b5e
> ("RDMA/nldev: Add support to dump device type and parent device if exists")
> 
> Signed-off-by: Mark Zhang <markzhang@nvidia.com>
> ---
>  rdma/include/uapi/rdma/rdma_netlink.h | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 

Thanks,
Acked-by: Leon Romanovsky <leon@kernel.org>

[RFC RESEND iproute2-next 2/2] rdma: Supports to add/delete a device with type SMI

[Mark Zhang]

This patch adds a new device attribute "type", as well as supports to
add and delete a rdma device with a specific type. This new device
provides a subset of functionalists defined in IBTA spec.

Currently only type "SMI" is supported: A SMI device provides SMI (QP0)
interface; This device and it's parent associates with the same HCA port
and shares the physical link, so when the parent doesn't support SMI,
It allows the subnet manager to configure the link.

This patch also supports to print device type and parent if any.

Examples:
$ rdma dev add smi1 type SMI parent ibp8s0f1
$ rdma dev show smi1
2: smi1: node_type ca fw 20.38.1002 node_guid 9803:9b03:009f:d5ef sys_image_guid 9803:9b03:009f:d5ee type smi parent ibp8s0f1
$ rdma dev del smi1

Signed-off-by: Mark Zhang <markzhang@nvidia.com>
---
 man/man8/rdma-dev.8 |  40 +++++++++++++++
 rdma/dev.c          | 120 ++++++++++++++++++++++++++++++++++++++++++++
 rdma/rdma.h         |   2 +
 rdma/utils.c        |   2 +
 4 files changed, 164 insertions(+)

diff --git a/man/man8/rdma-dev.8 b/man/man8/rdma-dev.8
index 368cdc7c..e26f738c 100644
--- a/man/man8/rdma-dev.8
+++ b/man/man8/rdma-dev.8
@@ -40,6 +40,18 @@ rdma-dev \- RDMA device configuration
 .BR adaptive-moderation
 .BR [on/off]
 
+.ti -8
+.B rdma dev add
+.RI "[ " NAME " ]"
+.B type
+.RI "[ " TYPE " ]"
+.B parent
+.RI "[ " PARENT_DEV " ]"
+
+.ti -8
+.B rdma dev delete
+.RI "[ " NAME " ]"
+
 .ti -8
 .B rdma dev help
 
@@ -53,6 +65,22 @@ rdma-dev \- RDMA device configuration
 - specifies the RDMA device to show.
 If this argument is omitted all devices are listed.
 
+.SS rdma dev add - Add a RDMA device with a specific type and parent IB device; This new device provides subset of functionalities defined in IBTA spec.
+.SS rdma dev delete - Delete a RDMA device which is created with the "add" command.
+.I NAME
+- The name of the device being added/deleted.
+
+.I TYPE
+- The type of the device being added; It specifies which functionalities it provides. Supported device type:
+.sp
+.in +8
+.B SMI
+- A device with this type provides SMI (QP0) interface. This device and it's parent associates with the same HCA port and shares the physical link, so when the parent doesn't support SMI, then this type of device can be created to allow the subnet manager to configure the link.
+.in -8
+
+.I PARENT_DEV
+- The name of its parent IB device.
+
 .SH "EXAMPLES"
 .PP
 rdma dev
@@ -84,6 +112,16 @@ Sets the state of adaptive interrupt moderation for the RDMA device.
 This is a global setting for the RDMA device but the value is printed for each CQ individually because the state is constant from CQ allocation.
 .RE
 .PP
+rdma dev add smi1 type SMI parent ibp8s0f1
+.RS 4
+Add a new IB device with name "smi1", type "SMI", and "ibp8s0f1" as its parent device.
+.RE
+.PP
+rdma dev delete smi1
+.RS 4
+Delete the IB device "smi1".
+.RE
+.PP
 
 .SH SEE ALSO
 .BR ip (8),
@@ -96,3 +134,5 @@ This is a global setting for the RDMA device but the value is printed for each C
 
 .SH AUTHOR
 Leon Romanovsky <leonro@mellanox.com>
+.br
+Mark Zhang <markzhang@nvidia.com>
diff --git a/rdma/dev.c b/rdma/dev.c
index f495b713..fd60c1a0 100644
--- a/rdma/dev.c
+++ b/rdma/dev.c
@@ -10,6 +10,8 @@
 static int dev_help(struct rd *rd)
 {
 	pr_out("Usage: %s dev show [DEV]\n", rd->filename);
+	pr_out("       %s dev add DEVNAME type TYPE parent PARENT_DEVNAME\n", rd->filename);
+	pr_out("       %s dev delete DEVNAME\n", rd->filename);
 	pr_out("       %s dev set [DEV] name DEVNAME\n", rd->filename);
 	pr_out("       %s dev set [DEV] netns NSNAME\n", rd->filename);
 	pr_out("       %s dev set [DEV] adaptive-moderation [on|off]\n", rd->filename);
@@ -148,6 +150,36 @@ static void dev_print_sys_image_guid(struct nlattr **tb)
 	print_string(PRINT_ANY, "sys_image_guid", "sys_image_guid %s ", str);
 }
 
+static const char *dev_type2str(uint32_t type)
+{
+	if (type == RDMA_DEVICE_TYPE_SMI)
+		return "smi";
+
+	return "unknown";
+}
+
+static void dev_print_type(struct nlattr **tb)
+{
+	uint32_t type;
+
+	if (!tb[RDMA_NLDEV_ATTR_DEV_TYPE])
+		return;
+
+	type = mnl_attr_get_u8(tb[RDMA_NLDEV_ATTR_DEV_TYPE]);
+	print_string(PRINT_ANY, "type", "type %s ", dev_type2str(type));
+}
+
+static void dev_print_parent(struct nlattr **tb)
+{
+	const char *parent;
+
+	if (!tb[RDMA_NLDEV_ATTR_PARENT_NAME])
+		return;
+
+	parent = mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_PARENT_NAME]);
+	print_string(PRINT_ANY, "parent", "parent %s ", parent);
+}
+
 static void dev_print_dim_setting(struct nlattr **tb)
 {
 	uint8_t dim_setting;
@@ -221,6 +253,8 @@ static int dev_parse_cb(const struct nlmsghdr *nlh, void *data)
 	dev_print_fw(tb);
 	dev_print_node_guid(tb);
 	dev_print_sys_image_guid(tb);
+	dev_print_type(tb);
+	dev_print_parent(tb);
 	if (rd->show_details) {
 		dev_print_dim_setting(tb);
 		dev_print_caps(tb);
@@ -366,10 +400,96 @@ static int dev_set(struct rd *rd)
 	return rd_exec_require_dev(rd, dev_one_set);
 }
 
+static int _dev_add_type_parent(struct rd *rd)
+{
+	uint32_t seq;
+
+	rd_prepare_msg(rd, RDMA_NLDEV_CMD_NEWDEV, &seq,
+		       (NLM_F_REQUEST | NLM_F_ACK));
+	mnl_attr_put_u32(rd->nlh, RDMA_NLDEV_ATTR_DEV_INDEX, rd->dev_idx);
+	mnl_attr_put_strz(rd->nlh, RDMA_NLDEV_ATTR_DEV_NAME, rd->dev_name);
+	mnl_attr_put_u8(rd->nlh, RDMA_NLDEV_ATTR_DEV_TYPE, rd->dev_type);
+
+	return rd_sendrecv_msg(rd, seq);
+}
+
+static int dev_add_type_parent(struct rd *rd)
+{
+	return rd_exec_require_dev(rd, _dev_add_type_parent);
+}
+
+static int dev_get_type(const char *stype)
+{
+	if (strcasecmp(stype, "smi") == 0)
+		return RDMA_DEVICE_TYPE_SMI;
+
+	return -1;
+}
+
+static int dev_add_type(struct rd *rd)
+{
+	const struct rd_cmd cmds[] = {
+		{ NULL,		dev_help },
+		{ "parent",	dev_add_type_parent },
+		{ 0 }
+	};
+
+	if (rd_no_arg(rd)) {
+		pr_err("Please provide a device type name.\n");
+		return -EINVAL;
+	}
+	rd->dev_type = dev_get_type(rd_argv(rd));
+	if (rd->dev_type <= 0) {
+		pr_err("Invalid device type %s\n", rd_argv(rd));
+		return -EINVAL;
+	}
+	rd_arg_inc(rd);
+	return rd_exec_cmd(rd, cmds, "parameter");
+}
+
+static int dev_add(struct rd *rd)
+{
+	const struct rd_cmd cmds[] = {
+		{ NULL,		dev_help },
+		{ "type",	dev_add_type },
+		{ 0 }
+	};
+
+	if (rd_no_arg(rd)) {
+		pr_err("Please provide a device name to add.\n");
+		return -EINVAL;
+	}
+	rd->dev_name = rd_argv(rd);
+	rd_arg_inc(rd);
+
+	return rd_exec_cmd(rd, cmds, "parameter");
+}
+
+static int _dev_del(struct rd *rd)
+{
+	uint32_t seq;
+
+	if (!rd_no_arg(rd)) {
+		pr_err("Unknown parameter %s\n", rd_argv(rd));
+		return -EINVAL;
+	}
+	rd_prepare_msg(rd, RDMA_NLDEV_CMD_DELDEV, &seq,
+		       (NLM_F_REQUEST | NLM_F_ACK));
+	mnl_attr_put_u32(rd->nlh, RDMA_NLDEV_ATTR_DEV_INDEX, rd->dev_idx);
+	return rd_sendrecv_msg(rd, seq);
+}
+
+static int dev_del(struct rd *rd)
+{
+	return rd_exec_require_dev(rd, _dev_del);
+}
+
 int cmd_dev(struct rd *rd)
 {
 	const struct rd_cmd cmds[] = {
 		{ NULL,		dev_show },
+		{ "add",	dev_add },
+		{ "delete",	dev_del },
 		{ "show",	dev_show },
 		{ "list",	dev_show },
 		{ "set",	dev_set },
diff --git a/rdma/rdma.h b/rdma/rdma.h
index df1852db..d224ec57 100644
--- a/rdma/rdma.h
+++ b/rdma/rdma.h
@@ -72,6 +72,8 @@ struct rd {
 	struct list_head filter_list;
 	char *link_name;
 	char *link_type;
+	char *dev_name;
+	int dev_type;
 };
 
 struct rd_cmd {
diff --git a/rdma/utils.c b/rdma/utils.c
index 0f41013a..4d3803b5 100644
--- a/rdma/utils.c
+++ b/rdma/utils.c
@@ -475,6 +475,8 @@ static const enum mnl_attr_data_type nldev_policy[RDMA_NLDEV_ATTR_MAX] = {
 	[RDMA_NLDEV_ATTR_DEV_DIM] = MNL_TYPE_U8,
 	[RDMA_NLDEV_ATTR_RES_RAW] = MNL_TYPE_BINARY,
 	[RDMA_NLDEV_SYS_ATTR_PRIVILEGED_QKEY_MODE] = MNL_TYPE_U8,
+	[RDMA_NLDEV_ATTR_DEV_TYPE] = MNL_TYPE_U8,
+	[RDMA_NLDEV_ATTR_PARENT_NAME] = MNL_TYPE_STRING,
 };
 
 static int rd_attr_check(const struct nlattr *attr, int *typep)
-- 
2.26.3

Re: [RFC RESEND iproute2-next 2/2] rdma: Supports to add/delete a device with type SMI

[Leon Romanovsky]

On Thu, Jul 04, 2024 at 09:29:01AM +0300, Mark Zhang wrote:
> This patch adds a new device attribute "type", as well as supports to
> add and delete a rdma device with a specific type. This new device
> provides a subset of functionalists defined in IBTA spec.
> 
> Currently only type "SMI" is supported: A SMI device provides SMI (QP0)
> interface; This device and it's parent associates with the same HCA port
> and shares the physical link, so when the parent doesn't support SMI,
> It allows the subnet manager to configure the link.
> 
> This patch also supports to print device type and parent if any.
> 
> Examples:
> $ rdma dev add smi1 type SMI parent ibp8s0f1
> $ rdma dev show smi1
> 2: smi1: node_type ca fw 20.38.1002 node_guid 9803:9b03:009f:d5ef sys_image_guid 9803:9b03:009f:d5ee type smi parent ibp8s0f1
> $ rdma dev del smi1
> 
> Signed-off-by: Mark Zhang <markzhang@nvidia.com>
> ---
>  man/man8/rdma-dev.8 |  40 +++++++++++++++
>  rdma/dev.c          | 120 ++++++++++++++++++++++++++++++++++++++++++++
>  rdma/rdma.h         |   2 +
>  rdma/utils.c        |   2 +
>  4 files changed, 164 insertions(+)
> 

Thanks,
Acked-by: Leon Romanovsky <leon@kernel.org>

Re: [RFC RESEND iproute2-next 0/2] Supports to add/delete IB devices with type SMI

[Mark Zhang]

On 7/4/2024 2:28 PM, Mark Zhang wrote:
> This series supports to add/delete an IB device with type SMI. This is
> complimentary to the kernel patches that support to IB sub device
> and mlx5 implementation.
> 
> https://lore.kernel.org/all/cover.1718553901.git.leon@kernel.org/
> 
> Thanks
> 
> Mark Zhang (2):
>    rdma: update uapi header
>    rdma: Supports to add/delete a device with type SMI
> 
>   man/man8/rdma-dev.8                   |  40 +++++++++
>   rdma/dev.c                            | 120 ++++++++++++++++++++++++++
>   rdma/include/uapi/rdma/rdma_netlink.h |  13 +++
>   rdma/rdma.h                           |   2 +
>   rdma/utils.c                          |   2 +
>   5 files changed, 177 insertions(+)
> 
Actually these are the patches, no RFC. Sorry for confusion.

Re: [RFC RESEND iproute2-next 0/2] Supports to add/delete IB devices with type SMI

[David Ahern]

On 7/4/24 12:28 AM, Mark Zhang wrote:
> This series supports to add/delete an IB device with type SMI. This is
> complimentary to the kernel patches that support to IB sub device
> and mlx5 implementation.
> 
> https://lore.kernel.org/all/cover.1718553901.git.leon@kernel.org/
> 
> Thanks
> 
> Mark Zhang (2):
>   rdma: update uapi header
>   rdma: Supports to add/delete a device with type SMI
> 
>  man/man8/rdma-dev.8                   |  40 +++++++++
>  rdma/dev.c                            | 120 ++++++++++++++++++++++++++
>  rdma/include/uapi/rdma/rdma_netlink.h |  13 +++
>  rdma/rdma.h                           |   2 +
>  rdma/utils.c                          |   2 +
>  5 files changed, 177 insertions(+)
> 

applied to iproute2-next