[PATCH net-next 00/14] devlink, mlx5: Add new parameters for link management and SRIOV/eSwitch configurations

[PATCH net-next 00/14] devlink, mlx5: Add new parameters for link management and SRIOV/eSwitch configurations

[Saeed Mahameed]

From: Saeed Mahameed <saeedm@nvidia.com>

This patch series introduces several devlink parameters improving device
configuration capabilities, link management, and SRIOV/eSwitch, by adding
NV config boot time parameters.

Below is a summary of the key changes:

1) Enable support for devlink port parameters

2) Implement multi attribute devlink param value data, for u32 array
   type parameters

3) Implement the following parameters:

   3.a) total_vfs Parameter:
   -------------------------

Adds support for managing the number of VFs (total_vfs) and enabling
SR-IOV (enable_sriov for mlx5) through devlink. These additions enhance
user control over virtualization features directly from standard kernel
interfaces without relying on additional external tools. total_vfs
functionality is critical for environments that require flexible num VF
configuration.

   3.b) devlink keep_link_up Parameter:
   ------------------------------------

Introduces a new devlink parameter 'keep_link_up', allowing devices to
keep the link active even when the driver is not loaded. This
functionality is especially useful for maintaining link stability during
driver upgrades or reboots without dropping connectivity.

   3.c) eSwitch Hairpin per Priority Buffers:
   ------------------------------------------

Implements new devlink parameters to configure eSwitch hairpin per
priority buffers. These parameters provide granular control over how
packets are buffered for IEEE802.1p priorities, offering improved traffic
management and efficiency for specific priority levels.

   3.d) CQE Compression Type:
   --------------------------

Introduces a new devlink parameter, cqe_compress_type, to configure the
rate of CQE compression based on PCIe bus conditions. This setting
provides a balance between compression efficiency and overall NIC
performance under different traffic loads.

Detailed examples of usage for each parameter have been included in the
respective commits.


Jiri Pirko (2):
  devlink: define enum for attr types of dynamic attributes
  devlink: pass struct devlink_port * as arg to devlink_nl_param_fill()

Saeed Mahameed (9):
  net/mlx5: Implement cqe_compress_type via devlink params
  devlink: Implement port params registration
  devlink: Implement get/dump netlink commands for port params
  devlink: Implement set netlink command for port params
  devlink: Add 'keep_link_up' generic devlink device param
  net/mlx5: Implement devlink keep_link_up port parameter
  devlink: Throw extack messages on param value validation error
  devlink: Implement devlink param multi attribute nested data values
  net/mlx5: Implement eSwitch hairpin per prio buffers devlink params

Vlad Dumitrescu (3):
  devlink: Add 'total_vfs' generic device param
  net/mlx5: Implement devlink enable_sriov parameter
  net/mlx5: Implement devlink total_vfs parameter

 Documentation/netlink/specs/devlink.yaml      |  45 +-
 .../networking/devlink/devlink-params.rst     |   7 +
 Documentation/networking/devlink/mlx5.rst     |  65 +-
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +-
 .../net/ethernet/mellanox/mlx5/core/devlink.c |   9 +
 .../net/ethernet/mellanox/mlx5/core/devlink.h |   3 +
 .../ethernet/mellanox/mlx5/core/en/devlink.c  |  16 +-
 .../ethernet/mellanox/mlx5/core/en/devlink.h  |   3 +-
 .../net/ethernet/mellanox/mlx5/core/en_main.c |   4 +-
 .../mellanox/mlx5/core/lib/nv_param.c         | 944 ++++++++++++++++++
 .../mellanox/mlx5/core/lib/nv_param.h         |  16 +
 include/linux/mlx5/driver.h                   |   1 +
 include/net/devlink.h                         |  30 +
 include/uapi/linux/devlink.h                  |  18 +
 net/devlink/health.c                          |  17 +-
 net/devlink/netlink_gen.c                     |  23 +-
 net/devlink/param.c                           | 437 ++++++--
 net/devlink/port.c                            |   3 +
 18 files changed, 1536 insertions(+), 107 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.h

-- 
2.48.1

[PATCH net-next 01/14] devlink: define enum for attr types of dynamic attributes

[Saeed Mahameed]

From: Jiri Pirko <jiri@nvidia.com>

Devlink param and health reporter fmsg use attributes with dynamic type
which is determined according to a different type. Currently used values
are NLA_*. The problem is, they are not part of UAPI. They may change
which would cause a break.

To make this future safe, introduce a enum that shadows NLA_* values in
it and is part of UAPI.

Also, this allows to possibly carry types that are unrelated to NLA_*
values.

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
---
 Documentation/netlink/specs/devlink.yaml | 23 ++++++++++++++++++
 include/uapi/linux/devlink.h             | 17 ++++++++++++++
 net/devlink/health.c                     | 17 ++++++++++++--
 net/devlink/param.c                      | 30 ++++++++++++------------
 4 files changed, 70 insertions(+), 17 deletions(-)

diff --git a/Documentation/netlink/specs/devlink.yaml b/Documentation/netlink/specs/devlink.yaml
index 09fbb4c03fc8..e99fc51856c5 100644
--- a/Documentation/netlink/specs/devlink.yaml
+++ b/Documentation/netlink/specs/devlink.yaml
@@ -202,6 +202,29 @@ definitions:
         name: exception
       -
         name: control
+  -
+    type: enum
+    name: dyn_attr_type
+    entries:
+      -
+        name: u8
+        value: 1
+      -
+        name: u16
+      -
+        name: u32
+      -
+        name: u64
+      -
+        name: string
+      -
+        name: flag
+      -
+        name: nul_string
+        value: 10
+      -
+        name: binary
+  -
 
 attribute-sets:
   -
diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
index 9401aa343673..8cdd60eb3c43 100644
--- a/include/uapi/linux/devlink.h
+++ b/include/uapi/linux/devlink.h
@@ -385,6 +385,23 @@ enum devlink_linecard_state {
 	DEVLINK_LINECARD_STATE_MAX = __DEVLINK_LINECARD_STATE_MAX - 1
 };
 
+/**
+ * enum devlink_dyn_attr_type - Dynamic attribute type type.
+ */
+enum devlink_dyn_attr_type {
+	/* Following values relate to the internal NLA_* values */
+	DEVLINK_DYN_ATTR_TYPE_U8 = 1,
+	DEVLINK_DYN_ATTR_TYPE_U16,
+	DEVLINK_DYN_ATTR_TYPE_U32,
+	DEVLINK_DYN_ATTR_TYPE_U64,
+	DEVLINK_DYN_ATTR_TYPE_STRING,
+	DEVLINK_DYN_ATTR_TYPE_FLAG,
+	DEVLINK_DYN_ATTR_TYPE_NUL_STRING = 10,
+	DEVLINK_DYN_ATTR_TYPE_BINARY,
+	__DEVLINK_DYN_ATTR_TYPE_CUSTOM_BASE = 0x80,
+	/* Any possible custom types, unrelated to NLA_* values go below */
+};
+
 enum devlink_attr {
 	/* don't change the order or add anything between, this is ABI! */
 	DEVLINK_ATTR_UNSPEC,
diff --git a/net/devlink/health.c b/net/devlink/health.c
index 57db6799722a..accb80cd3348 100644
--- a/net/devlink/health.c
+++ b/net/devlink/health.c
@@ -930,18 +930,31 @@ EXPORT_SYMBOL_GPL(devlink_fmsg_binary_pair_put);
 static int
 devlink_fmsg_item_fill_type(struct devlink_fmsg_item *msg, struct sk_buff *skb)
 {
+	enum devlink_dyn_attr_type dyn_attr_type;
+
 	switch (msg->nla_type) {
 	case NLA_FLAG:
+		dyn_attr_type = DEVLINK_DYN_ATTR_TYPE_FLAG;
+		break;
 	case NLA_U8:
+		dyn_attr_type = DEVLINK_DYN_ATTR_TYPE_U8;
+		break;
 	case NLA_U32:
+		dyn_attr_type = DEVLINK_DYN_ATTR_TYPE_U32;
+		break;
 	case NLA_U64:
+		dyn_attr_type = DEVLINK_DYN_ATTR_TYPE_U64;
+		break;
 	case NLA_NUL_STRING:
+		dyn_attr_type = DEVLINK_DYN_ATTR_TYPE_NUL_STRING;
+		break;
 	case NLA_BINARY:
-		return nla_put_u8(skb, DEVLINK_ATTR_FMSG_OBJ_VALUE_TYPE,
-				  msg->nla_type);
+		dyn_attr_type = DEVLINK_DYN_ATTR_TYPE_BINARY;
+		break;
 	default:
 		return -EINVAL;
 	}
+	return nla_put_u8(skb, DEVLINK_ATTR_FMSG_OBJ_VALUE_TYPE, dyn_attr_type);
 }
 
 static int
diff --git a/net/devlink/param.c b/net/devlink/param.c
index dcf0d1ccebba..e19d978dffa6 100644
--- a/net/devlink/param.c
+++ b/net/devlink/param.c
@@ -167,19 +167,19 @@ static int devlink_param_set(struct devlink *devlink,
 }
 
 static int
-devlink_param_type_to_nla_type(enum devlink_param_type param_type)
+devlink_param_type_to_dyn_attr_type(enum devlink_param_type param_type)
 {
 	switch (param_type) {
 	case DEVLINK_PARAM_TYPE_U8:
-		return NLA_U8;
+		return DEVLINK_DYN_ATTR_TYPE_U8;
 	case DEVLINK_PARAM_TYPE_U16:
-		return NLA_U16;
+		return DEVLINK_DYN_ATTR_TYPE_U16;
 	case DEVLINK_PARAM_TYPE_U32:
-		return NLA_U32;
+		return DEVLINK_DYN_ATTR_TYPE_U32;
 	case DEVLINK_PARAM_TYPE_STRING:
-		return NLA_STRING;
+		return DEVLINK_DYN_ATTR_TYPE_STRING;
 	case DEVLINK_PARAM_TYPE_BOOL:
-		return NLA_FLAG;
+		return DEVLINK_DYN_ATTR_TYPE_FLAG;
 	default:
 		return -EINVAL;
 	}
@@ -247,7 +247,7 @@ static int devlink_nl_param_fill(struct sk_buff *msg, struct devlink *devlink,
 	struct devlink_param_gset_ctx ctx;
 	struct nlattr *param_values_list;
 	struct nlattr *param_attr;
-	int nla_type;
+	int dyn_attr_type;
 	void *hdr;
 	int err;
 	int i;
@@ -294,10 +294,10 @@ static int devlink_nl_param_fill(struct sk_buff *msg, struct devlink *devlink,
 	if (param->generic && nla_put_flag(msg, DEVLINK_ATTR_PARAM_GENERIC))
 		goto param_nest_cancel;
 
-	nla_type = devlink_param_type_to_nla_type(param->type);
-	if (nla_type < 0)
+	dyn_attr_type = devlink_param_type_to_dyn_attr_type(param->type);
+	if (dyn_attr_type < 0)
 		goto param_nest_cancel;
-	if (nla_put_u8(msg, DEVLINK_ATTR_PARAM_TYPE, nla_type))
+	if (nla_put_u8(msg, DEVLINK_ATTR_PARAM_TYPE, dyn_attr_type))
 		goto param_nest_cancel;
 
 	param_values_list = nla_nest_start_noflag(msg,
@@ -420,19 +420,19 @@ devlink_param_type_get_from_info(struct genl_info *info,
 		return -EINVAL;
 
 	switch (nla_get_u8(info->attrs[DEVLINK_ATTR_PARAM_TYPE])) {
-	case NLA_U8:
+	case DEVLINK_DYN_ATTR_TYPE_U8:
 		*param_type = DEVLINK_PARAM_TYPE_U8;
 		break;
-	case NLA_U16:
+	case DEVLINK_DYN_ATTR_TYPE_U16:
 		*param_type = DEVLINK_PARAM_TYPE_U16;
 		break;
-	case NLA_U32:
+	case DEVLINK_DYN_ATTR_TYPE_U32:
 		*param_type = DEVLINK_PARAM_TYPE_U32;
 		break;
-	case NLA_STRING:
+	case DEVLINK_DYN_ATTR_TYPE_STRING:
 		*param_type = DEVLINK_PARAM_TYPE_STRING;
 		break;
-	case NLA_FLAG:
+	case DEVLINK_DYN_ATTR_TYPE_FLAG:
 		*param_type = DEVLINK_PARAM_TYPE_BOOL;
 		break;
 	default:
-- 
2.48.1

[PATCH net-next 02/14] devlink: Add 'total_vfs' generic device param

[Saeed Mahameed]

From: Vlad Dumitrescu <vdumitrescu@nvidia.com>

NICs are typically configured with total_vfs=0, forcing users to rely
on external tools to enable SR-IOV (a widely used and essential feature).

Add total_vfs parameter to devlink for SR-IOV max VF configurability.
Enables standard kernel tools to manage SR-IOV, addressing the need for
flexible VF configuration.

Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
---
 Documentation/networking/devlink/devlink-params.rst | 3 +++
 include/net/devlink.h                               | 4 ++++
 net/devlink/param.c                                 | 5 +++++
 3 files changed, 12 insertions(+)

diff --git a/Documentation/networking/devlink/devlink-params.rst b/Documentation/networking/devlink/devlink-params.rst
index 4e01dc32bc08..f266da05ab0d 100644
--- a/Documentation/networking/devlink/devlink-params.rst
+++ b/Documentation/networking/devlink/devlink-params.rst
@@ -137,3 +137,6 @@ own name.
    * - ``event_eq_size``
      - u32
      - Control the size of asynchronous control events EQ.
+   * - ``total_vfs``
+     - u32
+     - The total number of Virtual Functions (VFs) supported by the PF.
diff --git a/include/net/devlink.h b/include/net/devlink.h
index b8783126c1ed..eed1e4507d17 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -520,6 +520,7 @@ enum devlink_param_generic_id {
 	DEVLINK_PARAM_GENERIC_ID_ENABLE_IWARP,
 	DEVLINK_PARAM_GENERIC_ID_IO_EQ_SIZE,
 	DEVLINK_PARAM_GENERIC_ID_EVENT_EQ_SIZE,
+	DEVLINK_PARAM_GENERIC_ID_TOTAL_VFS,
 
 	/* add new param generic ids above here*/
 	__DEVLINK_PARAM_GENERIC_ID_MAX,
@@ -578,6 +579,9 @@ enum devlink_param_generic_id {
 #define DEVLINK_PARAM_GENERIC_EVENT_EQ_SIZE_NAME "event_eq_size"
 #define DEVLINK_PARAM_GENERIC_EVENT_EQ_SIZE_TYPE DEVLINK_PARAM_TYPE_U32
 
+#define DEVLINK_PARAM_GENERIC_TOTAL_VFS_NAME "total_vfs"
+#define DEVLINK_PARAM_GENERIC_TOTAL_VFS_TYPE DEVLINK_PARAM_TYPE_U32
+
 #define DEVLINK_PARAM_GENERIC(_id, _cmodes, _get, _set, _validate)	\
 {									\
 	.id = DEVLINK_PARAM_GENERIC_ID_##_id,				\
diff --git a/net/devlink/param.c b/net/devlink/param.c
index e19d978dffa6..d163afbadab9 100644
--- a/net/devlink/param.c
+++ b/net/devlink/param.c
@@ -92,6 +92,11 @@ static const struct devlink_param devlink_param_generic[] = {
 		.name = DEVLINK_PARAM_GENERIC_EVENT_EQ_SIZE_NAME,
 		.type = DEVLINK_PARAM_GENERIC_EVENT_EQ_SIZE_TYPE,
 	},
+	{
+		.id = DEVLINK_PARAM_GENERIC_ID_TOTAL_VFS,
+		.name = DEVLINK_PARAM_GENERIC_TOTAL_VFS_NAME,
+		.type = DEVLINK_PARAM_GENERIC_TOTAL_VFS_TYPE,
+	},
 };
 
 static int devlink_param_generic_verify(const struct devlink_param *param)
-- 
2.48.1

[PATCH net-next 03/14] net/mlx5: Implement cqe_compress_type via devlink params

[Saeed Mahameed]

From: Saeed Mahameed <saeedm@nvidia.com>

Selects which algorithm should be used by the NIC in order to decide rate of
CQE compression dependeng on PCIe bus conditions.

Supported values:

1) balanced, merges fewer CQEs, resulting in a moderate compression ratio
   but maintaining a balance between bandwidth savings and performance
2) aggressive, merges more CQEs into a single entry, achieving a higher
   compression rate and maximizing performance, particularly under high
   traffic loads.

Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
---
 Documentation/networking/devlink/mlx5.rst     |   9 +
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +-
 .../net/ethernet/mellanox/mlx5/core/devlink.c |   8 +
 .../net/ethernet/mellanox/mlx5/core/devlink.h |   1 +
 .../mellanox/mlx5/core/lib/nv_param.c         | 234 ++++++++++++++++++
 .../mellanox/mlx5/core/lib/nv_param.h         |  14 ++
 include/linux/mlx5/driver.h                   |   1 +
 7 files changed, 268 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.h

diff --git a/Documentation/networking/devlink/mlx5.rst b/Documentation/networking/devlink/mlx5.rst
index 7febe0aecd53..417e5cdcd35d 100644
--- a/Documentation/networking/devlink/mlx5.rst
+++ b/Documentation/networking/devlink/mlx5.rst
@@ -117,6 +117,15 @@ parameters.
      - driverinit
      - Control the size (in packets) of the hairpin queues.
 
+   * - ``cqe_compress_type``
+     - string
+     - permanent
+     - Configure which algorithm should be used by the NIC in order to decide
+       rate of CQE compression dependeng on PCIe bus conditions.
+
+       * ``balanced`` : Merges fewer CQEs, resulting in a moderate compression ratio but maintaining a balance between bandwidth savings and performance
+       * ``aggressive`` : Merges more CQEs into a single entry, achieving a higher compression rate and maximizing performance, particularly under high traffic loads
+
 The ``mlx5`` driver supports reloading via ``DEVLINK_CMD_RELOAD``
 
 Info versions
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index 568bbe5f83f5..31a6406a4814 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -17,7 +17,7 @@ mlx5_core-y :=	main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \
 		fs_counters.o fs_ft_pool.o rl.o lag/debugfs.o lag/lag.o dev.o events.o wq.o lib/gid.o \
 		lib/devcom.o lib/pci_vsc.o lib/dm.o lib/fs_ttc.o diag/fs_tracepoint.o \
 		diag/fw_tracer.o diag/crdump.o devlink.o diag/rsc_dump.o diag/reporter_vnic.o \
-		fw_reset.o qos.o lib/tout.o lib/aso.o wc.o fs_pool.o
+		fw_reset.o qos.o lib/tout.o lib/aso.o wc.o fs_pool.o lib/nv_param.o
 
 #
 # Netdev basic
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
index 98d4306929f3..1f764ae4f4aa 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
@@ -10,6 +10,7 @@
 #include "esw/qos.h"
 #include "sf/dev/dev.h"
 #include "sf/sf.h"
+#include "lib/nv_param.h"
 
 static int mlx5_devlink_flash_update(struct devlink *devlink,
 				     struct devlink_flash_update_params *params,
@@ -836,8 +837,14 @@ int mlx5_devlink_params_register(struct devlink *devlink)
 	if (err)
 		goto max_uc_list_err;
 
+	err = mlx5_nv_param_register_dl_params(devlink);
+	if (err)
+		goto nv_param_err;
+
 	return 0;
 
+nv_param_err:
+	mlx5_devlink_max_uc_list_params_unregister(devlink);
 max_uc_list_err:
 	mlx5_devlink_auxdev_params_unregister(devlink);
 auxdev_reg_err:
@@ -848,6 +855,7 @@ int mlx5_devlink_params_register(struct devlink *devlink)
 
 void mlx5_devlink_params_unregister(struct devlink *devlink)
 {
+	mlx5_nv_param_unregister_dl_params(devlink);
 	mlx5_devlink_max_uc_list_params_unregister(devlink);
 	mlx5_devlink_auxdev_params_unregister(devlink);
 	devl_params_unregister(devlink, mlx5_devlink_params,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h
index 961f75da6227..74bcdfa70361 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h
@@ -22,6 +22,7 @@ enum mlx5_devlink_param_id {
 	MLX5_DEVLINK_PARAM_ID_ESW_MULTIPORT,
 	MLX5_DEVLINK_PARAM_ID_HAIRPIN_NUM_QUEUES,
 	MLX5_DEVLINK_PARAM_ID_HAIRPIN_QUEUE_SIZE,
+	MLX5_DEVLINK_PARAM_ID_CQE_COMPRESSION_TYPE
 };
 
 struct mlx5_trap_ctx {
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
new file mode 100644
index 000000000000..5ab37a88c260
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
@@ -0,0 +1,234 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
+
+#include "nv_param.h"
+#include "mlx5_core.h"
+
+enum {
+	MLX5_CLASS_0_CTRL_ID_NV_SW_OFFLOAD_CONFIG             = 0x10a,
+};
+
+struct mlx5_ifc_configuration_item_type_class_global_bits {
+	u8         type_class[0x8];
+	u8         parameter_index[0x18];
+};
+
+union mlx5_ifc_config_item_type_auto_bits {
+	struct mlx5_ifc_configuration_item_type_class_global_bits
+				configuration_item_type_class_global;
+	u8 reserved_at_0[0x20];
+};
+
+struct mlx5_ifc_config_item_bits {
+	u8         valid[0x2];
+	u8         priority[0x2];
+	u8         header_type[0x2];
+	u8         ovr_en[0x1];
+	u8         rd_en[0x1];
+	u8         access_mode[0x2];
+	u8         reserved_at_a[0x1];
+	u8         writer_id[0x5];
+	u8         version[0x4];
+	u8         reserved_at_14[0x2];
+	u8         host_id_valid[0x1];
+	u8         length[0x9];
+
+	union mlx5_ifc_config_item_type_auto_bits type;
+
+	u8         reserved_at_40[0x10];
+	u8         crc16[0x10];
+};
+
+struct mlx5_ifc_mnvda_reg_bits {
+	struct mlx5_ifc_config_item_bits configuration_item_header;
+
+	u8         configuration_item_data[64][0x20];
+};
+
+struct mlx5_ifc_nv_sw_offload_conf_bits {
+	u8         ip_over_vxlan_port[0x10];
+	u8         tunnel_ecn_copy_offload_disable[0x1];
+	u8         pci_atomic_mode[0x3];
+	u8         sr_enable[0x1];
+	u8         ptp_cyc2realtime[0x1];
+	u8         vector_calc_disable[0x1];
+	u8         uctx_en[0x1];
+	u8         prio_tag_required_en[0x1];
+	u8         esw_fdb_ipv4_ttl_modify_enable[0x1];
+	u8         mkey_by_name[0x1];
+	u8         ip_over_vxlan_en[0x1];
+	u8         one_qp_per_recovery[0x1];
+	u8         cqe_compression[0x3];
+	u8         tunnel_udp_entropy_proto_disable[0x1];
+	u8         reserved_at_21[0x1];
+	u8         ar_enable[0x1];
+	u8         log_max_outstanding_wqe[0x5];
+	u8         vf_migration[0x2];
+	u8         log_tx_psn_win[0x6];
+	u8         lro_log_timeout3[0x4];
+	u8         lro_log_timeout2[0x4];
+	u8         lro_log_timeout1[0x4];
+	u8         lro_log_timeout0[0x4];
+};
+
+#define MNVDA_HDR_SZ \
+	(MLX5_ST_SZ_BYTES(mnvda_reg) - MLX5_BYTE_OFF(mnvda_reg, configuration_item_data))
+
+#define MLX5_SET_CONFIG_ITEM_TYPE(_cls_name, _mnvda_ptr, _field, _val) \
+	MLX5_SET(mnvda_reg, _mnvda_ptr, \
+		 configuration_item_header.type.configuration_item_type_class_##_cls_name._field, \
+		 _val)
+
+#define MLX5_SET_CONFIG_HDR_LEN(_mnvda_ptr, _cls_name) \
+	MLX5_SET(mnvda_reg, _mnvda_ptr, configuration_item_header.length, \
+		 MLX5_ST_SZ_BYTES(_cls_name))
+
+#define MLX5_GET_CONFIG_HDR_LEN(_mnvda_ptr) \
+	MLX5_GET(mnvda_reg, _mnvda_ptr, configuration_item_header.length)
+
+static int mlx5_nv_param_read(struct mlx5_core_dev *dev, void *mnvda, size_t len)
+{
+	u32 param_idx, type_class;
+	u32 header_len;
+	void *cls_ptr;
+	int err;
+
+	if (WARN_ON(len > MLX5_ST_SZ_BYTES(mnvda_reg)) || len < MNVDA_HDR_SZ)
+		return -EINVAL; /* A caller bug */
+
+	err = mlx5_core_access_reg(dev, mnvda, len, mnvda, len, MLX5_REG_MNVDA, 0, 0);
+	if (!err)
+		return 0;
+
+	cls_ptr = MLX5_ADDR_OF(mnvda_reg, mnvda,
+			       configuration_item_header.type.configuration_item_type_class_global);
+
+	type_class = MLX5_GET(configuration_item_type_class_global, cls_ptr, type_class);
+	param_idx = MLX5_GET(configuration_item_type_class_global, cls_ptr, parameter_index);
+	header_len = MLX5_GET_CONFIG_HDR_LEN(mnvda);
+
+	mlx5_core_warn(dev, "Failed to read mnvda reg: type_class 0x%x, param_idx 0x%x, header_len %u, err %d\n",
+		       type_class, param_idx, header_len, err);
+
+	/* Let devlink skip this one if it fails, kernel log will have the failure */
+	return -EOPNOTSUPP;
+}
+
+static int mlx5_nv_param_write(struct mlx5_core_dev *dev, void *mnvda, size_t len)
+{
+	if (WARN_ON(len > MLX5_ST_SZ_BYTES(mnvda_reg)) || len < MNVDA_HDR_SZ)
+		return -EINVAL;
+
+	if (WARN_ON(MLX5_GET_CONFIG_HDR_LEN(mnvda) == 0))
+		return -EINVAL;
+
+	return mlx5_core_access_reg(dev, mnvda, len, mnvda, len, MLX5_REG_MNVDA, 0, 1);
+}
+
+static int
+mlx5_nv_param_read_sw_offload_conf(struct mlx5_core_dev *dev, void *mnvda, size_t len)
+{
+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, type_class, 0);
+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, parameter_index,
+				  MLX5_CLASS_0_CTRL_ID_NV_SW_OFFLOAD_CONFIG);
+	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_sw_offload_conf);
+
+	return mlx5_nv_param_read(dev, mnvda, len);
+}
+
+static const char *const
+	cqe_compress_str[] = { "balanced", "aggressive" };
+
+static int mlx5_nv_param_devlink_cqe_compress_get(struct devlink *devlink, u32 id,
+						  struct devlink_param_gset_ctx *ctx)
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
+	u8 value = U8_MAX;
+	void *data;
+	int err;
+
+	err = mlx5_nv_param_read_sw_offload_conf(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+	value = MLX5_GET(nv_sw_offload_conf, data, cqe_compression);
+
+	if (value >= ARRAY_SIZE(cqe_compress_str))
+		return -EOPNOTSUPP;
+
+	strscpy(ctx->val.vstr, cqe_compress_str[value], sizeof(ctx->val.vstr));
+	return 0;
+}
+
+static int
+mlx5_nv_param_devlink_cqe_compress_validate(struct devlink *devlink, u32 id,
+					    union devlink_param_value val,
+					    struct netlink_ext_ack *extack)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(cqe_compress_str); i++) {
+		if (!strcmp(val.vstr, cqe_compress_str[i]))
+			return 0;
+	}
+
+	NL_SET_ERR_MSG_MOD(extack,
+			   "Invalid value, supported values are balanced/aggressive");
+	return -EOPNOTSUPP;
+}
+
+static int mlx5_nv_param_devlink_cqe_compress_set(struct devlink *devlink, u32 id,
+						  struct devlink_param_gset_ctx *ctx,
+						  struct netlink_ext_ack *extack)
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
+	int err = 0;
+	void *data;
+	u8 value;
+
+	if (!strcmp(ctx->val.vstr, "aggressive"))
+		value = 1;
+	else /* balanced: can't be anything else already validated above */
+		value = 0;
+
+	err = mlx5_nv_param_read_sw_offload_conf(dev, mnvda, sizeof(mnvda));
+	if (err) {
+		NL_SET_ERR_MSG_MOD(extack, "Failed to read sw_offload_conf mnvda reg");
+		return err;
+	}
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+	MLX5_SET(nv_sw_offload_conf, data, cqe_compression, value);
+
+	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
+}
+
+static const struct devlink_param mlx5_nv_param_devlink_params[] = {
+	DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_ID_CQE_COMPRESSION_TYPE,
+			     "cqe_compress_type", DEVLINK_PARAM_TYPE_STRING,
+			     BIT(DEVLINK_PARAM_CMODE_PERMANENT),
+			     mlx5_nv_param_devlink_cqe_compress_get,
+			     mlx5_nv_param_devlink_cqe_compress_set,
+			     mlx5_nv_param_devlink_cqe_compress_validate),
+};
+
+int mlx5_nv_param_register_dl_params(struct devlink *devlink)
+{
+	if (!mlx5_core_is_pf(devlink_priv(devlink)))
+		return 0;
+
+	return devl_params_register(devlink, mlx5_nv_param_devlink_params,
+				    ARRAY_SIZE(mlx5_nv_param_devlink_params));
+}
+
+void mlx5_nv_param_unregister_dl_params(struct devlink *devlink)
+{
+	if (!mlx5_core_is_pf(devlink_priv(devlink)))
+		return;
+
+	devl_params_unregister(devlink, mlx5_nv_param_devlink_params,
+			       ARRAY_SIZE(mlx5_nv_param_devlink_params));
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.h
new file mode 100644
index 000000000000..9f4922ff7745
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.h
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
+
+#ifndef __MLX5_NV_PARAM_H
+#define __MLX5_NV_PARAM_H
+
+#include <linux/mlx5/driver.h>
+#include "devlink.h"
+
+int mlx5_nv_param_register_dl_params(struct devlink *devlink);
+void mlx5_nv_param_unregister_dl_params(struct devlink *devlink);
+
+#endif
+
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 46bd7550adf8..2ce3c04fe237 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -135,6 +135,7 @@ enum {
 	MLX5_REG_MTCAP		 = 0x9009,
 	MLX5_REG_MTMP		 = 0x900A,
 	MLX5_REG_MCIA		 = 0x9014,
+	MLX5_REG_MNVDA		 = 0x9024,
 	MLX5_REG_MFRL		 = 0x9028,
 	MLX5_REG_MLCR		 = 0x902b,
 	MLX5_REG_MRTC		 = 0x902d,
-- 
2.48.1

[PATCH net-next 04/14] net/mlx5: Implement devlink enable_sriov parameter

[Saeed Mahameed]

From: Vlad Dumitrescu <vdumitrescu@nvidia.com>

Example usage:
  devlink dev param set pci/0000:01:00.0 name enable_sriov value {true, false} cmode permanent
  devlink dev reload pci/0000:01:00.0 action fw_activate
  echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove
  echo 1 >/sys/bus/pci/rescan
  grep ^ /sys/bus/pci/devices/0000:01:00.0/sriov_*

Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
---
 Documentation/networking/devlink/mlx5.rst     |  14 +-
 .../net/ethernet/mellanox/mlx5/core/devlink.c |   1 +
 .../mellanox/mlx5/core/lib/nv_param.c         | 184 ++++++++++++++++++
 3 files changed, 196 insertions(+), 3 deletions(-)

diff --git a/Documentation/networking/devlink/mlx5.rst b/Documentation/networking/devlink/mlx5.rst
index 417e5cdcd35d..587e0200c1cd 100644
--- a/Documentation/networking/devlink/mlx5.rst
+++ b/Documentation/networking/devlink/mlx5.rst
@@ -15,23 +15,31 @@ Parameters
    * - Name
      - Mode
      - Validation
+     - Notes
    * - ``enable_roce``
      - driverinit
-     - Type: Boolean
-
-       If the device supports RoCE disablement, RoCE enablement state controls
+     - Boolean
+     - If the device supports RoCE disablement, RoCE enablement state controls
        device support for RoCE capability. Otherwise, the control occurs in the
        driver stack. When RoCE is disabled at the driver level, only raw
        ethernet QPs are supported.
    * - ``io_eq_size``
      - driverinit
      - The range is between 64 and 4096.
+     -
    * - ``event_eq_size``
      - driverinit
      - The range is between 64 and 4096.
+     -
    * - ``max_macs``
      - driverinit
      - The range is between 1 and 2^31. Only power of 2 values are supported.
+     -
+   * - ``enable_sriov``
+     - permanent
+     - Boolean
+     - Applies to each physical function (PF) independently, if the device
+       supports it. Otherwise, it applies symmetrically to all PFs.
 
 The ``mlx5`` driver also implements the following driver-specific
 parameters.
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
index 1f764ae4f4aa..7a702d84f19a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
@@ -8,6 +8,7 @@
 #include "fs_core.h"
 #include "eswitch.h"
 #include "esw/qos.h"
+#include "lib/nv_param.h"
 #include "sf/dev/dev.h"
 #include "sf/sf.h"
 #include "lib/nv_param.h"
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
index 5ab37a88c260..6b63fc110e2d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
@@ -5,7 +5,11 @@
 #include "mlx5_core.h"
 
 enum {
+	MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CONF               = 0x80,
+	MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CAP                = 0x81,
 	MLX5_CLASS_0_CTRL_ID_NV_SW_OFFLOAD_CONFIG             = 0x10a,
+
+	MLX5_CLASS_3_CTRL_ID_NV_PF_PCI_CONF                   = 0x80,
 };
 
 struct mlx5_ifc_configuration_item_type_class_global_bits {
@@ -13,9 +17,18 @@ struct mlx5_ifc_configuration_item_type_class_global_bits {
 	u8         parameter_index[0x18];
 };
 
+struct mlx5_ifc_configuration_item_type_class_per_host_pf_bits {
+	u8         type_class[0x8];
+	u8         pf_index[0x6];
+	u8         pci_bus_index[0x8];
+	u8         parameter_index[0xa];
+};
+
 union mlx5_ifc_config_item_type_auto_bits {
 	struct mlx5_ifc_configuration_item_type_class_global_bits
 				configuration_item_type_class_global;
+	struct mlx5_ifc_configuration_item_type_class_per_host_pf_bits
+				configuration_item_type_class_per_host_pf;
 	u8 reserved_at_0[0x20];
 };
 
@@ -45,6 +58,45 @@ struct mlx5_ifc_mnvda_reg_bits {
 	u8         configuration_item_data[64][0x20];
 };
 
+struct mlx5_ifc_nv_global_pci_conf_bits {
+	u8         sriov_valid[0x1];
+	u8         reserved_at_1[0x10];
+	u8         per_pf_total_vf[0x1];
+	u8         reserved_at_12[0xe];
+
+	u8         sriov_en[0x1];
+	u8         reserved_at_21[0xf];
+	u8         total_vfs[0x10];
+
+	u8         reserved_at_40[0x20];
+};
+
+struct mlx5_ifc_nv_global_pci_cap_bits {
+	u8         max_vfs_per_pf_valid[0x1];
+	u8         reserved_at_1[0x13];
+	u8         per_pf_total_vf_supported[0x1];
+	u8         reserved_at_15[0xb];
+
+	u8         sriov_support[0x1];
+	u8         reserved_at_21[0xf];
+	u8         max_vfs_per_pf[0x10];
+
+	u8         reserved_at_40[0x60];
+};
+
+struct mlx5_ifc_nv_pf_pci_conf_bits {
+	u8         reserved_at_0[0x9];
+	u8         pf_total_vf_en[0x1];
+	u8         reserved_at_a[0x16];
+
+	u8         reserved_at_20[0x20];
+
+	u8         reserved_at_40[0x10];
+	u8         total_vf[0x10];
+
+	u8         reserved_at_60[0x20];
+};
+
 struct mlx5_ifc_nv_sw_offload_conf_bits {
 	u8         ip_over_vxlan_port[0x10];
 	u8         tunnel_ecn_copy_offload_disable[0x1];
@@ -206,7 +258,139 @@ static int mlx5_nv_param_devlink_cqe_compress_set(struct devlink *devlink, u32 i
 	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
 }
 
+static int
+mlx5_nv_param_read_global_pci_conf(struct mlx5_core_dev *dev, void *mnvda, size_t len)
+{
+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, type_class, 0);
+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, parameter_index,
+				  MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CONF);
+	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_global_pci_conf);
+
+	return mlx5_nv_param_read(dev, mnvda, len);
+}
+
+static int
+mlx5_nv_param_read_global_pci_cap(struct mlx5_core_dev *dev, void *mnvda, size_t len)
+{
+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, type_class, 0);
+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, parameter_index,
+				  MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CAP);
+	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_global_pci_cap);
+
+	return mlx5_nv_param_read(dev, mnvda, len);
+}
+
+static int
+mlx5_nv_param_read_per_host_pf_conf(struct mlx5_core_dev *dev, void *mnvda, size_t len)
+{
+	MLX5_SET_CONFIG_ITEM_TYPE(per_host_pf, mnvda, type_class, 3);
+	MLX5_SET_CONFIG_ITEM_TYPE(per_host_pf, mnvda, parameter_index,
+				  MLX5_CLASS_3_CTRL_ID_NV_PF_PCI_CONF);
+	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_pf_pci_conf);
+
+	return mlx5_nv_param_read(dev, mnvda, len);
+}
+
+static int mlx5_devlink_enable_sriov_get(struct devlink *devlink, u32 id,
+					 struct devlink_param_gset_ctx *ctx)
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
+	void *data;
+	int err;
+
+	err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+	if (!MLX5_GET(nv_global_pci_cap, data, sriov_support)) {
+		ctx->val.vbool = false;
+		return 0;
+	}
+
+	memset(mnvda, 0, sizeof(mnvda));
+	err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+	if (!MLX5_GET(nv_global_pci_conf, data, per_pf_total_vf)) {
+		ctx->val.vbool = MLX5_GET(nv_global_pci_conf, data, sriov_en);
+		return 0;
+	}
+
+	/* SRIOV is per PF */
+	memset(mnvda, 0, sizeof(mnvda));
+	err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+	ctx->val.vbool = MLX5_GET(nv_pf_pci_conf, data, pf_total_vf_en);
+	return 0;
+}
+
+static int mlx5_devlink_enable_sriov_set(struct devlink *devlink, u32 id,
+					 struct devlink_param_gset_ctx *ctx,
+					 struct netlink_ext_ack *extack)
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
+	bool per_pf_support;
+	void *cap, *data;
+	int err;
+
+	err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
+	if (err) {
+		NL_SET_ERR_MSG_MOD(extack, "Failed to read global PCI capability");
+		return err;
+	}
+
+	cap = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+	per_pf_support = MLX5_GET(nv_global_pci_cap, cap, per_pf_total_vf_supported);
+
+	if (!MLX5_GET(nv_global_pci_cap, cap, sriov_support)) {
+		NL_SET_ERR_MSG_MOD(extack, "Not configurable on this device");
+		return -EOPNOTSUPP;
+	}
+
+	memset(mnvda, 0, sizeof(mnvda));
+	err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
+	if (err) {
+		NL_SET_ERR_MSG_MOD(extack, "Unable to read global PCI configuration");
+		return err;
+	}
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+	MLX5_SET(nv_global_pci_conf, data, sriov_valid, 1);
+	MLX5_SET(nv_global_pci_conf, data, sriov_en, ctx->val.vbool);
+	MLX5_SET(nv_global_pci_conf, data, per_pf_total_vf, per_pf_support);
+
+	err = mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
+	if (err) {
+		NL_SET_ERR_MSG_MOD(extack, "Unable to write global PCI configuration");
+		return err;
+	}
+
+	if (!per_pf_support)
+		return 0;
+
+	/* SRIOV is per PF */
+	memset(mnvda, 0, sizeof(mnvda));
+	err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
+	if (err) {
+		NL_SET_ERR_MSG_MOD(extack, "Unable to read per host PF configuration");
+		return err;
+	}
+	MLX5_SET(nv_pf_pci_conf, data, pf_total_vf_en, ctx->val.vbool);
+	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
+}
+
 static const struct devlink_param mlx5_nv_param_devlink_params[] = {
+	DEVLINK_PARAM_GENERIC(ENABLE_SRIOV, BIT(DEVLINK_PARAM_CMODE_PERMANENT),
+			      mlx5_devlink_enable_sriov_get,
+			      mlx5_devlink_enable_sriov_set, NULL),
 	DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_ID_CQE_COMPRESSION_TYPE,
 			     "cqe_compress_type", DEVLINK_PARAM_TYPE_STRING,
 			     BIT(DEVLINK_PARAM_CMODE_PERMANENT),
-- 
2.48.1

[PATCH net-next 05/14] net/mlx5: Implement devlink total_vfs parameter

[Saeed Mahameed]

From: Vlad Dumitrescu <vdumitrescu@nvidia.com>

Some devices support both symmetric (same value for all PFs) and
asymmetric, while others only support symmetric configuration. This
implementation prefers asymmetric, since it is closer to the devlink
model (per function settings), but falls back to symmetric when needed.

Example usage:
  devlink dev param set pci/0000:01:00.0 name total_vfs value <u16> cmode permanent
  devlink dev reload pci/0000:01:00.0 action fw_activate
  echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove
  echo 1 >/sys/bus/pci/rescan
  cat /sys/bus/pci/devices/0000:01:00.0/sriov_totalvfs

Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
---
 Documentation/networking/devlink/mlx5.rst     |  22 +++
 .../mellanox/mlx5/core/lib/nv_param.c         | 125 ++++++++++++++++++
 2 files changed, 147 insertions(+)

diff --git a/Documentation/networking/devlink/mlx5.rst b/Documentation/networking/devlink/mlx5.rst
index 587e0200c1cd..00a43324dec2 100644
--- a/Documentation/networking/devlink/mlx5.rst
+++ b/Documentation/networking/devlink/mlx5.rst
@@ -40,6 +40,28 @@ Parameters
      - Boolean
      - Applies to each physical function (PF) independently, if the device
        supports it. Otherwise, it applies symmetrically to all PFs.
+   * - ``total_vfs``
+     - permanent
+     - The range is between 1 and a device-specific max.
+     - Applies to each physical function (PF) independently, if the device
+       supports it. Otherwise, it applies symmetrically to all PFs.
+
+Note: permanent parameters such as ``enable_sriov`` and ``total_vfs`` require FW reset to take effect
+
+.. code-block:: bash
+
+   # setup parameters
+   devlink dev param set pci/0000:01:00.0 name enable_sriov value true cmode permanent
+   devlink dev param set pci/0000:01:00.0 name total_vfs value 8 cmode permanent
+
+   # Fw reset
+   devlink dev reload pci/0000:01:00.0 action fw_activate
+
+   # for PCI related config such as sriov PCI reset/rescan is required:
+   echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove
+   echo 1 >/sys/bus/pci/rescan
+   grep ^ /sys/bus/pci/devices/0000:01:00.0/sriov_*
+
 
 The ``mlx5`` driver also implements the following driver-specific
 parameters.
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
index 6b63fc110e2d..97d74d890582 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
@@ -387,10 +387,135 @@ static int mlx5_devlink_enable_sriov_set(struct devlink *devlink, u32 id,
 	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
 }
 
+static int mlx5_devlink_total_vfs_get(struct devlink *devlink, u32 id,
+				      struct devlink_param_gset_ctx *ctx)
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
+	void *data;
+	int err;
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+
+	err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+
+	if (!MLX5_GET(nv_global_pci_cap, data, sriov_support)) {
+		ctx->val.vu32 = 0;
+		return 0;
+	}
+
+	memset(mnvda, 0, sizeof(mnvda));
+	err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+
+	if (!MLX5_GET(nv_global_pci_conf, data, per_pf_total_vf)) {
+		ctx->val.vu32 = MLX5_GET(nv_global_pci_conf, data, total_vfs);
+		return 0;
+	}
+
+	/* SRIOV is per PF */
+	memset(mnvda, 0, sizeof(mnvda));
+	err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+
+	ctx->val.vu32 = MLX5_GET(nv_pf_pci_conf, data, total_vf);
+
+	return 0;
+}
+
+static int mlx5_devlink_total_vfs_set(struct devlink *devlink, u32 id,
+				      struct devlink_param_gset_ctx *ctx,
+				      struct netlink_ext_ack *extack)
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)];
+	bool per_pf_support;
+	void *data;
+	int err;
+
+	err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
+	if (err) {
+		NL_SET_ERR_MSG_MOD(extack, "Failed to read global pci cap");
+		return err;
+	}
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+	if (!MLX5_GET(nv_global_pci_cap, data, sriov_support)) {
+		NL_SET_ERR_MSG_MOD(extack, "Not configurable on this device");
+		return -EOPNOTSUPP;
+	}
+
+	per_pf_support = MLX5_GET(nv_global_pci_cap, data, per_pf_total_vf_supported);
+	memset(mnvda, 0, sizeof(mnvda));
+
+	err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+
+	MLX5_SET(nv_global_pci_conf, data, sriov_valid, 1);
+	MLX5_SET(nv_global_pci_conf, data, per_pf_total_vf, per_pf_support);
+
+	if (!per_pf_support) {
+		MLX5_SET(nv_global_pci_conf, data, total_vfs, ctx->val.vu32);
+		return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
+	}
+
+	/* SRIOV is per PF */
+	err = mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+
+	memset(mnvda, 0, sizeof(mnvda));
+	err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+	MLX5_SET(nv_pf_pci_conf, data, pf_total_vf_en, 1);
+	MLX5_SET(nv_pf_pci_conf, data, total_vf, ctx->val.vu32);
+	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
+}
+
+static int mlx5_devlink_total_vfs_validate(struct devlink *devlink, u32 id,
+					   union devlink_param_value val,
+					   struct netlink_ext_ack *extack)
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	u32 cap[MLX5_ST_SZ_DW(mnvda_reg)];
+	void *data;
+	u16 max;
+	int err;
+
+	data = MLX5_ADDR_OF(mnvda_reg, cap, configuration_item_data);
+
+	err = mlx5_nv_param_read_global_pci_cap(dev, cap, sizeof(cap));
+	if (err)
+		return err;
+
+	if (!MLX5_GET(nv_global_pci_cap, data, max_vfs_per_pf_valid))
+		return 0; /* optimistic, but set might fail later */
+
+	max = MLX5_GET(nv_global_pci_cap, data, max_vfs_per_pf);
+	if (val.vu16 > max) {
+		NL_SET_ERR_MSG_FMT_MOD(extack,
+				       "Max allowed by device is %u", max);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
 static const struct devlink_param mlx5_nv_param_devlink_params[] = {
 	DEVLINK_PARAM_GENERIC(ENABLE_SRIOV, BIT(DEVLINK_PARAM_CMODE_PERMANENT),
 			      mlx5_devlink_enable_sriov_get,
 			      mlx5_devlink_enable_sriov_set, NULL),
+	DEVLINK_PARAM_GENERIC(TOTAL_VFS, BIT(DEVLINK_PARAM_CMODE_PERMANENT),
+			      mlx5_devlink_total_vfs_get, mlx5_devlink_total_vfs_set,
+			      mlx5_devlink_total_vfs_validate),
 	DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_ID_CQE_COMPRESSION_TYPE,
 			     "cqe_compress_type", DEVLINK_PARAM_TYPE_STRING,
 			     BIT(DEVLINK_PARAM_CMODE_PERMANENT),
-- 
2.48.1

[PATCH net-next 06/14] devlink: pass struct devlink_port * as arg to devlink_nl_param_fill()

[Saeed Mahameed]

From: Jiri Pirko <jiri@nvidia.com>

As the follow-up patch will need to get struct devlink_port *, avoid
unnecessary lookup and instead of port_index pass the struct
devlink_port * directly.

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
---
 net/devlink/param.c | 29 +++++++++++++++--------------
 1 file changed, 15 insertions(+), 14 deletions(-)

diff --git a/net/devlink/param.c b/net/devlink/param.c
index d163afbadab9..2263aba85a79 100644
--- a/net/devlink/param.c
+++ b/net/devlink/param.c
@@ -241,7 +241,7 @@ devlink_nl_param_value_fill_one(struct sk_buff *msg,
 }
 
 static int devlink_nl_param_fill(struct sk_buff *msg, struct devlink *devlink,
-				 unsigned int port_index,
+				 struct devlink_port *devlink_port,
 				 struct devlink_param_item *param_item,
 				 enum devlink_command cmd,
 				 u32 portid, u32 seq, int flags)
@@ -288,7 +288,8 @@ static int devlink_nl_param_fill(struct sk_buff *msg, struct devlink *devlink,
 	if (cmd == DEVLINK_CMD_PORT_PARAM_GET ||
 	    cmd == DEVLINK_CMD_PORT_PARAM_NEW ||
 	    cmd == DEVLINK_CMD_PORT_PARAM_DEL)
-		if (nla_put_u32(msg, DEVLINK_ATTR_PORT_INDEX, port_index))
+		if (nla_put_u32(msg, DEVLINK_ATTR_PORT_INDEX,
+				devlink_port->index))
 			goto genlmsg_cancel;
 
 	param_attr = nla_nest_start_noflag(msg, DEVLINK_ATTR_PARAM);
@@ -334,7 +335,7 @@ static int devlink_nl_param_fill(struct sk_buff *msg, struct devlink *devlink,
 }
 
 static void devlink_param_notify(struct devlink *devlink,
-				 unsigned int port_index,
+				 struct devlink_port *devlink_port,
 				 struct devlink_param_item *param_item,
 				 enum devlink_command cmd)
 {
@@ -355,7 +356,7 @@ static void devlink_param_notify(struct devlink *devlink,
 	msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
 	if (!msg)
 		return;
-	err = devlink_nl_param_fill(msg, devlink, port_index, param_item, cmd,
+	err = devlink_nl_param_fill(msg, devlink, devlink_port, param_item, cmd,
 				    0, 0, 0);
 	if (err) {
 		nlmsg_free(msg);
@@ -372,7 +373,7 @@ static void devlink_params_notify(struct devlink *devlink,
 	unsigned long param_id;
 
 	xa_for_each(&devlink->params, param_id, param_item)
-		devlink_param_notify(devlink, 0, param_item, cmd);
+		devlink_param_notify(devlink, NULL, param_item, cmd);
 }
 
 void devlink_params_notify_register(struct devlink *devlink)
@@ -396,7 +397,7 @@ static int devlink_nl_param_get_dump_one(struct sk_buff *msg,
 	int err = 0;
 
 	xa_for_each_start(&devlink->params, param_id, param_item, state->idx) {
-		err = devlink_nl_param_fill(msg, devlink, 0, param_item,
+		err = devlink_nl_param_fill(msg, devlink, NULL, param_item,
 					    DEVLINK_CMD_PARAM_GET,
 					    NETLINK_CB(cb->skb).portid,
 					    cb->nlh->nlmsg_seq, flags);
@@ -520,7 +521,7 @@ int devlink_nl_param_get_doit(struct sk_buff *skb,
 	if (!msg)
 		return -ENOMEM;
 
-	err = devlink_nl_param_fill(msg, devlink, 0, param_item,
+	err = devlink_nl_param_fill(msg, devlink, NULL, param_item,
 				    DEVLINK_CMD_PARAM_GET,
 				    info->snd_portid, info->snd_seq, 0);
 	if (err) {
@@ -532,7 +533,7 @@ int devlink_nl_param_get_doit(struct sk_buff *skb,
 }
 
 static int __devlink_nl_cmd_param_set_doit(struct devlink *devlink,
-					   unsigned int port_index,
+					   struct devlink_port *devlink_port,
 					   struct xarray *params,
 					   struct genl_info *info,
 					   enum devlink_command cmd)
@@ -582,7 +583,7 @@ static int __devlink_nl_cmd_param_set_doit(struct devlink *devlink,
 			return err;
 	}
 
-	devlink_param_notify(devlink, port_index, param_item, cmd);
+	devlink_param_notify(devlink, devlink_port, param_item, cmd);
 	return 0;
 }
 
@@ -590,7 +591,7 @@ int devlink_nl_param_set_doit(struct sk_buff *skb, struct genl_info *info)
 {
 	struct devlink *devlink = info->user_ptr[0];
 
-	return __devlink_nl_cmd_param_set_doit(devlink, 0, &devlink->params,
+	return __devlink_nl_cmd_param_set_doit(devlink, NULL, &devlink->params,
 					       info, DEVLINK_CMD_PARAM_NEW);
 }
 
@@ -649,7 +650,7 @@ static int devlink_param_register(struct devlink *devlink,
 	if (err)
 		goto err_xa_insert;
 
-	devlink_param_notify(devlink, 0, param_item, DEVLINK_CMD_PARAM_NEW);
+	devlink_param_notify(devlink, NULL, param_item, DEVLINK_CMD_PARAM_NEW);
 	return 0;
 
 err_xa_insert:
@@ -665,7 +666,7 @@ static void devlink_param_unregister(struct devlink *devlink,
 	param_item = devlink_param_find_by_id(&devlink->params, param->id);
 	if (WARN_ON(!param_item))
 		return;
-	devlink_param_notify(devlink, 0, param_item, DEVLINK_CMD_PARAM_DEL);
+	devlink_param_notify(devlink, NULL, param_item, DEVLINK_CMD_PARAM_DEL);
 	xa_erase(&devlink->params, param->id);
 	kfree(param_item);
 }
@@ -826,7 +827,7 @@ void devl_param_driverinit_value_set(struct devlink *devlink, u32 param_id,
 	param_item->driverinit_value = init_val;
 	param_item->driverinit_value_valid = true;
 
-	devlink_param_notify(devlink, 0, param_item, DEVLINK_CMD_PARAM_NEW);
+	devlink_param_notify(devlink, NULL, param_item, DEVLINK_CMD_PARAM_NEW);
 }
 EXPORT_SYMBOL_GPL(devl_param_driverinit_value_set);
 
@@ -865,6 +866,6 @@ void devl_param_value_changed(struct devlink *devlink, u32 param_id)
 	param_item = devlink_param_find_by_id(&devlink->params, param_id);
 	WARN_ON(!param_item);
 
-	devlink_param_notify(devlink, 0, param_item, DEVLINK_CMD_PARAM_NEW);
+	devlink_param_notify(devlink, NULL, param_item, DEVLINK_CMD_PARAM_NEW);
 }
 EXPORT_SYMBOL_GPL(devl_param_value_changed);
-- 
2.48.1

[PATCH net-next 07/14] devlink: Implement port params registration

[Saeed Mahameed]

From: Saeed Mahameed <saeedm@nvidia.com>

Port params infrastructure is incomplete and needs a bit of plumbing to
support port params commands from netlink.

Introduce port params registration API, very similar to current devlink
params API, add the params xarray to devlink_port structure and
decouple devlink params registration routines from the devlink
structure.

Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
---
 include/net/devlink.h |  14 ++++
 net/devlink/param.c   | 150 ++++++++++++++++++++++++++++++++++--------
 net/devlink/port.c    |   3 +
 3 files changed, 140 insertions(+), 27 deletions(-)

diff --git a/include/net/devlink.h b/include/net/devlink.h
index eed1e4507d17..11f98e3a750b 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -125,6 +125,7 @@ struct devlink_port {
 	struct list_head region_list;
 	struct devlink *devlink;
 	const struct devlink_port_ops *ops;
+	struct xarray params;
 	unsigned int index;
 	spinlock_t type_lock; /* Protects type and type_eth/ib
 			       * structures consistency.
@@ -1823,6 +1824,19 @@ void devl_params_unregister(struct devlink *devlink,
 void devlink_params_unregister(struct devlink *devlink,
 			       const struct devlink_param *params,
 			       size_t params_count);
+int devl_port_params_register(struct devlink_port *devlink_port,
+			      const struct devlink_param *params,
+			      size_t params_count);
+int devlink_port_params_register(struct devlink_port *devlink_port,
+				 const struct devlink_param *params,
+				 size_t params_count);
+void devl_port_params_unregister(struct devlink_port *devlink_port,
+				 const struct devlink_param *params,
+				 size_t params_count);
+void devlink_port_params_unregister(struct devlink_port *devlink_port,
+				    const struct devlink_param *params,
+				    size_t params_count);
+
 int devl_param_driverinit_value_get(struct devlink *devlink, u32 param_id,
 				    union devlink_param_value *val);
 void devl_param_driverinit_value_set(struct devlink *devlink, u32 param_id,
diff --git a/net/devlink/param.c b/net/devlink/param.c
index 2263aba85a79..719eeb5152c3 100644
--- a/net/devlink/param.c
+++ b/net/devlink/param.c
@@ -627,13 +627,16 @@ static int devlink_param_verify(const struct devlink_param *param)
 }
 
 static int devlink_param_register(struct devlink *devlink,
+				  struct devlink_port *devlink_port,
+				  struct xarray *params_arr,
 				  const struct devlink_param *param)
 {
 	struct devlink_param_item *param_item;
+	enum devlink_command cmd;
 	int err;
 
 	WARN_ON(devlink_param_verify(param));
-	WARN_ON(devlink_param_find_by_name(&devlink->params, param->name));
+	WARN_ON(devlink_param_find_by_name(params_arr, param->name));
 
 	if (param->supported_cmodes == BIT(DEVLINK_PARAM_CMODE_DRIVERINIT))
 		WARN_ON(param->get || param->set);
@@ -646,11 +649,13 @@ static int devlink_param_register(struct devlink *devlink,
 
 	param_item->param = param;
 
-	err = xa_insert(&devlink->params, param->id, param_item, GFP_KERNEL);
+	err = xa_insert(params_arr, param->id, param_item, GFP_KERNEL);
 	if (err)
 		goto err_xa_insert;
 
-	devlink_param_notify(devlink, NULL, param_item, DEVLINK_CMD_PARAM_NEW);
+	cmd = devlink_port ? DEVLINK_CMD_PORT_PARAM_NEW : DEVLINK_CMD_PARAM_NEW;
+	devlink_param_notify(devlink, devlink_port, param_item, cmd);
+
 	return 0;
 
 err_xa_insert:
@@ -659,30 +664,28 @@ static int devlink_param_register(struct devlink *devlink,
 }
 
 static void devlink_param_unregister(struct devlink *devlink,
+				     struct devlink_port *devlink_port,
+				     struct xarray *params_arr,
 				     const struct devlink_param *param)
 {
 	struct devlink_param_item *param_item;
+	enum devlink_command cmd;
 
-	param_item = devlink_param_find_by_id(&devlink->params, param->id);
+	param_item = devlink_param_find_by_id(params_arr, param->id);
 	if (WARN_ON(!param_item))
 		return;
-	devlink_param_notify(devlink, NULL, param_item, DEVLINK_CMD_PARAM_DEL);
-	xa_erase(&devlink->params, param->id);
+
+	cmd = devlink_port ? DEVLINK_CMD_PORT_PARAM_DEL : DEVLINK_CMD_PARAM_DEL;
+	devlink_param_notify(devlink, devlink_port, param_item, cmd);
+	xa_erase(params_arr, param->id);
 	kfree(param_item);
 }
 
-/**
- *	devl_params_register - register configuration parameters
- *
- *	@devlink: devlink
- *	@params: configuration parameters array
- *	@params_count: number of parameters provided
- *
- *	Register the configuration parameters supported by the driver.
- */
-int devl_params_register(struct devlink *devlink,
-			 const struct devlink_param *params,
-			 size_t params_count)
+static int __devlink_params_register(struct devlink *devlink,
+				     struct devlink_port *devlink_port,
+				     struct xarray *params_arr,
+				     const struct devlink_param *params,
+				     size_t params_count)
 {
 	const struct devlink_param *param = params;
 	int i, err;
@@ -690,10 +693,12 @@ int devl_params_register(struct devlink *devlink,
 	lockdep_assert_held(&devlink->lock);
 
 	for (i = 0; i < params_count; i++, param++) {
-		err = devlink_param_register(devlink, param);
+		err = devlink_param_register(devlink, devlink_port, params_arr,
+					     param);
 		if (err)
 			goto rollback;
 	}
+
 	return 0;
 
 rollback:
@@ -701,9 +706,28 @@ int devl_params_register(struct devlink *devlink,
 		return err;
 
 	for (param--; i > 0; i--, param--)
-		devlink_param_unregister(devlink, param);
+		devlink_param_unregister(devlink, devlink_port, params_arr,
+					 param);
+
 	return err;
 }
+
+/**
+ *	devl_params_register - register configuration parameters
+ *
+ *	@devlink: devlink
+ *	@params: configuration parameters array
+ *	@params_count: number of parameters provided
+ *
+ *	Register the configuration parameters supported by the driver.
+ */
+int devl_params_register(struct devlink *devlink,
+			 const struct devlink_param *params,
+			 size_t params_count)
+{
+	return __devlink_params_register(devlink, NULL, &devlink->params,
+					 params, params_count);
+}
 EXPORT_SYMBOL_GPL(devl_params_register);
 
 int devlink_params_register(struct devlink *devlink,
@@ -719,6 +743,22 @@ int devlink_params_register(struct devlink *devlink,
 }
 EXPORT_SYMBOL_GPL(devlink_params_register);
 
+static void __devlink_params_unregister(struct devlink *devlink,
+					struct devlink_port *devlink_port,
+					struct xarray *params_arr,
+					const struct devlink_param *params,
+					size_t params_count)
+{
+	const struct devlink_param *param = params;
+	int i;
+
+	lockdep_assert_held(&devlink->lock);
+
+	for (i = 0; i < params_count; i++, param++)
+		devlink_param_unregister(devlink, devlink_port, params_arr,
+					 param);
+}
+
 /**
  *	devl_params_unregister - unregister configuration parameters
  *	@devlink: devlink
@@ -729,13 +769,8 @@ void devl_params_unregister(struct devlink *devlink,
 			    const struct devlink_param *params,
 			    size_t params_count)
 {
-	const struct devlink_param *param = params;
-	int i;
-
-	lockdep_assert_held(&devlink->lock);
-
-	for (i = 0; i < params_count; i++, param++)
-		devlink_param_unregister(devlink, param);
+	__devlink_params_unregister(devlink, NULL, &devlink->params,
+				    params, params_count);
 }
 EXPORT_SYMBOL_GPL(devl_params_unregister);
 
@@ -749,6 +784,67 @@ void devlink_params_unregister(struct devlink *devlink,
 }
 EXPORT_SYMBOL_GPL(devlink_params_unregister);
 
+/**
+ *	devl_port_params_register - register configuration parameters for port
+ *
+ *	@devlink_port: devlink port
+ *	@params: configuration parameters array
+ *	@params_count: number of parameters provided
+ *
+ *	Register the configuration parameters supported by the driver for the
+ *	specific port.
+ */
+int devl_port_params_register(struct devlink_port *devlink_port,
+			      const struct devlink_param *params,
+			      size_t params_count)
+{
+	return __devlink_params_register(devlink_port->devlink,
+					 devlink_port,
+					 &devlink_port->params,
+					 params, params_count);
+}
+EXPORT_SYMBOL_GPL(devl_port_params_register);
+
+/**
+ *	devl_port_params_unregister - unregister configuration parameters for port
+ *
+ *	@devlink_port: devlink port
+ *	@params: configuration parameters to unregister
+ *	@params_count: number of parameters provided
+ */
+void devl_port_params_unregister(struct devlink_port *devlink_port,
+				 const struct devlink_param *params,
+				 size_t params_count)
+{
+	__devlink_params_unregister(devlink_port->devlink, devlink_port,
+				    &devlink_port->params,
+				    params, params_count);
+}
+EXPORT_SYMBOL_GPL(devl_port_params_unregister);
+
+int devlink_port_params_register(struct devlink_port *devlink_port,
+				 const struct devlink_param *params,
+				 size_t params_count)
+{
+	int err;
+
+	devl_lock(devlink_port->devlink);
+	err = devl_port_params_register(devlink_port, params, params_count);
+	devl_unlock(devlink_port->devlink);
+	return err;
+}
+EXPORT_SYMBOL_GPL(devlink_port_params_register);
+
+void devlink_port_params_unregister(struct devlink_port *devlink_port,
+				    const struct devlink_param *params,
+				    size_t params_count)
+{
+	devl_lock(devlink_port->devlink);
+	devl_port_params_unregister(devlink_port, params, params_count);
+	devl_unlock(devlink_port->devlink);
+}
+EXPORT_SYMBOL_GPL(devlink_port_params_unregister);
+
 /**
  *	devl_param_driverinit_value_get - get configuration parameter
  *					  value for driver initializing
diff --git a/net/devlink/port.c b/net/devlink/port.c
index 939081a0e615..39bba3f7a1f9 100644
--- a/net/devlink/port.c
+++ b/net/devlink/port.c
@@ -1075,6 +1075,8 @@ int devl_port_register_with_ops(struct devlink *devlink,
 	devlink_port->registered = true;
 	devlink_port->index = port_index;
 	devlink_port->ops = ops ? ops : &devlink_port_dummy_ops;
+	xa_init_flags(&devlink_port->params, XA_FLAGS_ALLOC);
+
 	spin_lock_init(&devlink_port->type_lock);
 	INIT_LIST_HEAD(&devlink_port->reporter_list);
 	err = xa_insert(&devlink->ports, port_index, devlink_port, GFP_KERNEL);
@@ -1134,6 +1136,7 @@ void devl_port_unregister(struct devlink_port *devlink_port)
 	devlink_port_type_warn_cancel(devlink_port);
 	devlink_port_notify(devlink_port, DEVLINK_CMD_PORT_DEL);
 	xa_erase(&devlink_port->devlink->ports, devlink_port->index);
+	xa_destroy(&devlink_port->params);
 	WARN_ON(!list_empty(&devlink_port->reporter_list));
 	devlink_port->registered = false;
 }
-- 
2.48.1

[PATCH net-next 08/14] devlink: Implement get/dump netlink commands for port params

[Saeed Mahameed]

From: Saeed Mahameed <saeedm@nvidia.com>

Add missing port-params netlink attributes and policies to devlink's
spec, reuse existing get_doit/dump_doit of the devlink params for port
params and implement the dump command for all devlink ports params.

This implements:
1)  devlink port param show
2)  devlink port param show <device>/<port>
3)  devlink port param show <device>/<port> name <param_name>

Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
---
 Documentation/netlink/specs/devlink.yaml | 13 ++--
 include/net/devlink.h                    |  1 +
 net/devlink/netlink_gen.c                | 16 ++++-
 net/devlink/param.c                      | 88 ++++++++++++++++++++----
 4 files changed, 99 insertions(+), 19 deletions(-)

diff --git a/Documentation/netlink/specs/devlink.yaml b/Documentation/netlink/specs/devlink.yaml
index e99fc51856c5..fb3b2bea0ac3 100644
--- a/Documentation/netlink/specs/devlink.yaml
+++ b/Documentation/netlink/specs/devlink.yaml
@@ -1851,12 +1851,17 @@ operations:
         pre: devlink-nl-pre-doit-port
         post: devlink-nl-post-doit
         request:
-          attributes: *port-id-attrs
-        reply:
-          attributes: *port-id-attrs
+          attributes: &port-param-id-attrs
+            - bus-name
+            - dev-name
+            - port-index
+            - param-name
+        reply: &port-param-get-reply
+          attributes: *port-param-id-attrs
       dump:
-        reply:
+        request:
           attributes: *port-id-attrs
+        reply: *port-param-get-reply
 
     -
       name: port-param-set
diff --git a/include/net/devlink.h b/include/net/devlink.h
index 11f98e3a750b..b409ccbcfd12 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -439,6 +439,7 @@ union devlink_param_value {
 struct devlink_param_gset_ctx {
 	union devlink_param_value val;
 	enum devlink_param_cmode cmode;
+	struct devlink_port *devlink_port;
 };
 
 /**
diff --git a/net/devlink/netlink_gen.c b/net/devlink/netlink_gen.c
index f9786d51f68f..d4876cf0f049 100644
--- a/net/devlink/netlink_gen.c
+++ b/net/devlink/netlink_gen.c
@@ -322,7 +322,15 @@ static const struct nla_policy devlink_region_read_nl_policy[DEVLINK_ATTR_REGION
 };
 
 /* DEVLINK_CMD_PORT_PARAM_GET - do */
-static const struct nla_policy devlink_port_param_get_nl_policy[DEVLINK_ATTR_PORT_INDEX + 1] = {
+static const struct nla_policy devlink_port_param_get_do_nl_policy[DEVLINK_ATTR_PARAM_NAME + 1] = {
+	[DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING, },
+	[DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING, },
+	[DEVLINK_ATTR_PORT_INDEX] = { .type = NLA_U32, },
+	[DEVLINK_ATTR_PARAM_NAME] = { .type = NLA_NUL_STRING, },
+};
+
+/* DEVLINK_CMD_PORT_PARAM_GET - dump */
+static const struct nla_policy devlink_port_param_get_dump_nl_policy[DEVLINK_ATTR_PORT_INDEX + 1] = {
 	[DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING, },
 	[DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING, },
 	[DEVLINK_ATTR_PORT_INDEX] = { .type = NLA_U32, },
@@ -938,14 +946,16 @@ const struct genl_split_ops devlink_nl_ops[74] = {
 		.pre_doit	= devlink_nl_pre_doit_port,
 		.doit		= devlink_nl_port_param_get_doit,
 		.post_doit	= devlink_nl_post_doit,
-		.policy		= devlink_port_param_get_nl_policy,
-		.maxattr	= DEVLINK_ATTR_PORT_INDEX,
+		.policy		= devlink_port_param_get_do_nl_policy,
+		.maxattr	= DEVLINK_ATTR_PARAM_NAME,
 		.flags		= GENL_CMD_CAP_DO,
 	},
 	{
 		.cmd		= DEVLINK_CMD_PORT_PARAM_GET,
 		.validate	= GENL_DONT_VALIDATE_DUMP_STRICT,
 		.dumpit		= devlink_nl_port_param_get_dumpit,
+		.policy		= devlink_port_param_get_dump_nl_policy,
+		.maxattr	= DEVLINK_ATTR_PORT_INDEX,
 		.flags		= GENL_CMD_CAP_DUMP,
 	},
 	{
diff --git a/net/devlink/param.c b/net/devlink/param.c
index 719eeb5152c3..45f1847f9435 100644
--- a/net/devlink/param.c
+++ b/net/devlink/param.c
@@ -153,11 +153,14 @@ devlink_param_cmode_is_supported(const struct devlink_param *param,
 }
 
 static int devlink_param_get(struct devlink *devlink,
+			     struct devlink_port *devlink_port,
 			     const struct devlink_param *param,
 			     struct devlink_param_gset_ctx *ctx)
 {
 	if (!param->get)
 		return -EOPNOTSUPP;
+
+	ctx->devlink_port = devlink_port;
 	return param->get(devlink, param->id, ctx);
 }
 
@@ -249,7 +252,7 @@ static int devlink_nl_param_fill(struct sk_buff *msg, struct devlink *devlink,
 	union devlink_param_value param_value[DEVLINK_PARAM_CMODE_MAX + 1];
 	bool param_value_set[DEVLINK_PARAM_CMODE_MAX + 1] = {};
 	const struct devlink_param *param = param_item->param;
-	struct devlink_param_gset_ctx ctx;
+	struct devlink_param_gset_ctx ctx = {};
 	struct nlattr *param_values_list;
 	struct nlattr *param_attr;
 	int dyn_attr_type;
@@ -270,7 +273,7 @@ static int devlink_nl_param_fill(struct sk_buff *msg, struct devlink *devlink,
 				return -EOPNOTSUPP;
 		} else {
 			ctx.cmode = i;
-			err = devlink_param_get(devlink, param, &ctx);
+			err = devlink_param_get(devlink, devlink_port, param, &ctx);
 			if (err)
 				return err;
 			param_value[i] = ctx.val;
@@ -505,15 +508,17 @@ devlink_param_get_from_info(struct xarray *params, struct genl_info *info)
 	return devlink_param_find_by_name(params, param_name);
 }
 
-int devlink_nl_param_get_doit(struct sk_buff *skb,
-			      struct genl_info *info)
+static int __devlink_nl_param_get_doit(struct devlink *devlink,
+				       struct devlink_port *devlink_port,
+				       struct xarray *params,
+				       struct genl_info *info,
+				       enum devlink_command cmd)
 {
-	struct devlink *devlink = info->user_ptr[0];
 	struct devlink_param_item *param_item;
 	struct sk_buff *msg;
 	int err;
 
-	param_item = devlink_param_get_from_info(&devlink->params, info);
+	param_item = devlink_param_get_from_info(params, info);
 	if (!param_item)
 		return -EINVAL;
 
@@ -521,8 +526,7 @@ int devlink_nl_param_get_doit(struct sk_buff *skb,
 	if (!msg)
 		return -ENOMEM;
 
-	err = devlink_nl_param_fill(msg, devlink, NULL, param_item,
-				    DEVLINK_CMD_PARAM_GET,
+	err = devlink_nl_param_fill(msg, devlink, devlink_port, param_item, cmd,
 				    info->snd_portid, info->snd_seq, 0);
 	if (err) {
 		nlmsg_free(msg);
@@ -532,6 +536,14 @@ int devlink_nl_param_get_doit(struct sk_buff *skb,
 	return genlmsg_reply(msg, info);
 }
 
+int devlink_nl_param_get_doit(struct sk_buff *skb, struct genl_info *info)
+{
+	struct devlink *devlink = info->user_ptr[0];
+
+	return __devlink_nl_param_get_doit(devlink, NULL, &devlink->params,
+					   info, DEVLINK_CMD_PARAM_GET);
+}
+
 static int __devlink_nl_cmd_param_set_doit(struct devlink *devlink,
 					   struct devlink_port *devlink_port,
 					   struct xarray *params,
@@ -595,18 +607,70 @@ int devlink_nl_param_set_doit(struct sk_buff *skb, struct genl_info *info)
 					       info, DEVLINK_CMD_PARAM_NEW);
 }
 
+static int
+devlink_nl_port_param_get_dump_one(struct sk_buff *msg,
+				   struct devlink *devlink,
+				   struct netlink_callback *cb,
+				   int flags)
+{
+	struct devlink_nl_dump_state *state = devlink_dump_state(cb);
+	const struct genl_info *info = genl_info_dump(cb);
+	unsigned long port_index_end = ULONG_MAX;
+	struct devlink_param_item *param_item;
+	struct nlattr **attrs = info->attrs;
+	unsigned long port_index_start = 0;
+	struct devlink_port *devlink_port;
+	unsigned long port_index;
+	unsigned long param_id;
+	int idx = 0;
+	int err = 0;
+
+	if (attrs && attrs[DEVLINK_ATTR_PORT_INDEX]) {
+		port_index_start = nla_get_u32(attrs[DEVLINK_ATTR_PORT_INDEX]);
+		port_index_end = port_index_start;
+		flags |= NLM_F_DUMP_FILTERED;
+	}
+
+	xa_for_each_range(&devlink->ports, port_index, devlink_port,
+			  port_index_start, port_index_end) {
+		xa_for_each_start(&devlink_port->params, param_id, param_item,
+				  state->idx) {
+			if (idx < state->idx) {
+				idx++;
+				continue;
+			}
+			err = devlink_nl_param_fill(msg, devlink, devlink_port,
+						    param_item,
+						    DEVLINK_CMD_PORT_PARAM_GET,
+						    NETLINK_CB(cb->skb).portid,
+						    cb->nlh->nlmsg_seq, flags);
+			if (err == -EOPNOTSUPP) {
+				err = 0;
+			} else if (err) {
+				state->idx = param_id;
+				break;
+			}
+		}
+	}
+
+	return err;
+}
+
 int devlink_nl_port_param_get_dumpit(struct sk_buff *msg,
 				     struct netlink_callback *cb)
 {
-	NL_SET_ERR_MSG(cb->extack, "Port params are not supported");
-	return msg->len;
+	return devlink_nl_dumpit(msg, cb, devlink_nl_port_param_get_dump_one);
 }
 
 int devlink_nl_port_param_get_doit(struct sk_buff *skb,
 				   struct genl_info *info)
 {
-	NL_SET_ERR_MSG(info->extack, "Port params are not supported");
-	return -EINVAL;
+	struct devlink_port *devlink_port = info->user_ptr[1];
+	struct devlink *devlink = info->user_ptr[0];
+
+	return __devlink_nl_param_get_doit(devlink, devlink_port,
+					   &devlink_port->params,
+					   info, DEVLINK_CMD_PORT_PARAM_GET);
 }
 
 int devlink_nl_port_param_set_doit(struct sk_buff *skb,
-- 
2.48.1

[PATCH net-next 09/14] devlink: Implement set netlink command for port params

[Saeed Mahameed]

From: Saeed Mahameed <saeedm@nvidia.com>

Add missing port-params netlink attributes and policies to devlink's
spec, reuse existing set_doit of the devlink dev params.

This implements:
  devlink port param set <device>/<port> name <param_name> value <val> \
               cmode <cmode>

Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
---
 Documentation/netlink/specs/devlink.yaml |  9 ++++++++-
 net/devlink/netlink_gen.c                |  7 +++++--
 net/devlink/param.c                      | 16 ++++++++++++----
 3 files changed, 25 insertions(+), 7 deletions(-)

diff --git a/Documentation/netlink/specs/devlink.yaml b/Documentation/netlink/specs/devlink.yaml
index fb3b2bea0ac3..aca4d0557944 100644
--- a/Documentation/netlink/specs/devlink.yaml
+++ b/Documentation/netlink/specs/devlink.yaml
@@ -1873,7 +1873,14 @@ operations:
         pre: devlink-nl-pre-doit-port
         post: devlink-nl-post-doit
         request:
-          attributes: *port-id-attrs
+          attributes:
+            - bus-name
+            - dev-name
+            - port-index
+            - param-name
+            - param-type
+            # param-value-data is missing here as the type is variable
+            - param-value-cmode
 
     -
       name: info-get
diff --git a/net/devlink/netlink_gen.c b/net/devlink/netlink_gen.c
index d4876cf0f049..bb1a916c8764 100644
--- a/net/devlink/netlink_gen.c
+++ b/net/devlink/netlink_gen.c
@@ -337,10 +337,13 @@ static const struct nla_policy devlink_port_param_get_dump_nl_policy[DEVLINK_ATT
 };
 
 /* DEVLINK_CMD_PORT_PARAM_SET - do */
-static const struct nla_policy devlink_port_param_set_nl_policy[DEVLINK_ATTR_PORT_INDEX + 1] = {
+static const struct nla_policy devlink_port_param_set_nl_policy[DEVLINK_ATTR_PARAM_VALUE_CMODE + 1] = {
 	[DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING, },
 	[DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING, },
 	[DEVLINK_ATTR_PORT_INDEX] = { .type = NLA_U32, },
+	[DEVLINK_ATTR_PARAM_NAME] = { .type = NLA_NUL_STRING, },
+	[DEVLINK_ATTR_PARAM_TYPE] = { .type = NLA_U8, },
+	[DEVLINK_ATTR_PARAM_VALUE_CMODE] = NLA_POLICY_MAX(NLA_U8, 2),
 };
 
 /* DEVLINK_CMD_INFO_GET - do */
@@ -965,7 +968,7 @@ const struct genl_split_ops devlink_nl_ops[74] = {
 		.doit		= devlink_nl_port_param_set_doit,
 		.post_doit	= devlink_nl_post_doit,
 		.policy		= devlink_port_param_set_nl_policy,
-		.maxattr	= DEVLINK_ATTR_PORT_INDEX,
+		.maxattr	= DEVLINK_ATTR_PARAM_VALUE_CMODE,
 		.flags		= GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
 	},
 	{
diff --git a/net/devlink/param.c b/net/devlink/param.c
index 45f1847f9435..c3d39817a908 100644
--- a/net/devlink/param.c
+++ b/net/devlink/param.c
@@ -165,12 +165,15 @@ static int devlink_param_get(struct devlink *devlink,
 }
 
 static int devlink_param_set(struct devlink *devlink,
+			     struct devlink_port *devlink_port,
 			     const struct devlink_param *param,
 			     struct devlink_param_gset_ctx *ctx,
 			     struct netlink_ext_ack *extack)
 {
 	if (!param->set)
 		return -EOPNOTSUPP;
+
+	ctx->devlink_port = devlink_port;
 	return param->set(devlink, param->id, ctx, extack);
 }
 
@@ -550,8 +553,8 @@ static int __devlink_nl_cmd_param_set_doit(struct devlink *devlink,
 					   struct genl_info *info,
 					   enum devlink_command cmd)
 {
+	struct devlink_param_gset_ctx ctx = {};
 	enum devlink_param_type param_type;
-	struct devlink_param_gset_ctx ctx;
 	enum devlink_param_cmode cmode;
 	struct devlink_param_item *param_item;
 	const struct devlink_param *param;
@@ -590,7 +593,8 @@ static int __devlink_nl_cmd_param_set_doit(struct devlink *devlink,
 			return -EOPNOTSUPP;
 		ctx.val = value;
 		ctx.cmode = cmode;
-		err = devlink_param_set(devlink, param, &ctx, info->extack);
+		err = devlink_param_set(devlink, devlink_port, param,
+					&ctx, info->extack);
 		if (err)
 			return err;
 	}
@@ -676,8 +680,12 @@ int devlink_nl_port_param_get_doit(struct sk_buff *skb,
 int devlink_nl_port_param_set_doit(struct sk_buff *skb,
 				   struct genl_info *info)
 {
-	NL_SET_ERR_MSG(info->extack, "Port params are not supported");
-	return -EINVAL;
+	struct devlink_port *devlink_port = info->user_ptr[1];
+	struct devlink *devlink = info->user_ptr[0];
+
+	return __devlink_nl_cmd_param_set_doit(devlink, devlink_port,
+					       &devlink_port->params, info,
+					       DEVLINK_CMD_PORT_PARAM_NEW);
 }
 
 static int devlink_param_verify(const struct devlink_param *param)
-- 
2.48.1

[PATCH net-next 10/14] devlink: Add 'keep_link_up' generic devlink device param

[Saeed Mahameed]

From: Saeed Mahameed <saeedm@nvidia.com>

Devices that support this in permanent mode will be requested to keep the
port link up even when driver is not loaded, netdev carrier state won't
affect the physical port link state.

This is useful for when the link is needed to access onboard management
such as BMC, even if the host driver isn't loaded.

Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
---
 Documentation/networking/devlink/devlink-params.rst | 4 ++++
 include/net/devlink.h                               | 4 ++++
 net/devlink/param.c                                 | 4 ++++
 3 files changed, 12 insertions(+)

diff --git a/Documentation/networking/devlink/devlink-params.rst b/Documentation/networking/devlink/devlink-params.rst
index f266da05ab0d..2fbfab78030f 100644
--- a/Documentation/networking/devlink/devlink-params.rst
+++ b/Documentation/networking/devlink/devlink-params.rst
@@ -140,3 +140,7 @@ own name.
    * - ``total_vfs``
      - u32
      - The total number of Virtual Functions (VFs) supported by the PF.
+   * - ``keep_link_up``
+     - Boolean
+     - When enabled, the device will keep the port link up even if the driver is
+       not loaded.
diff --git a/include/net/devlink.h b/include/net/devlink.h
index b409ccbcfd12..ca32c61583cf 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -523,6 +523,7 @@ enum devlink_param_generic_id {
 	DEVLINK_PARAM_GENERIC_ID_IO_EQ_SIZE,
 	DEVLINK_PARAM_GENERIC_ID_EVENT_EQ_SIZE,
 	DEVLINK_PARAM_GENERIC_ID_TOTAL_VFS,
+	DEVLINK_PARAM_GENERIC_ID_KEEP_LINK_UP,
 
 	/* add new param generic ids above here*/
 	__DEVLINK_PARAM_GENERIC_ID_MAX,
@@ -584,6 +585,9 @@ enum devlink_param_generic_id {
 #define DEVLINK_PARAM_GENERIC_TOTAL_VFS_NAME "total_vfs"
 #define DEVLINK_PARAM_GENERIC_TOTAL_VFS_TYPE DEVLINK_PARAM_TYPE_U32
 
+#define DEVLINK_PARAM_GENERIC_KEEP_LINK_UP_NAME "keep_link_up"
+#define DEVLINK_PARAM_GENERIC_KEEP_LINK_UP_TYPE DEVLINK_PARAM_TYPE_BOOL
+
 #define DEVLINK_PARAM_GENERIC(_id, _cmodes, _get, _set, _validate)	\
 {									\
 	.id = DEVLINK_PARAM_GENERIC_ID_##_id,				\
diff --git a/net/devlink/param.c b/net/devlink/param.c
index c3d39817a908..03c65ccf2acf 100644
--- a/net/devlink/param.c
+++ b/net/devlink/param.c
@@ -97,6 +97,10 @@ static const struct devlink_param devlink_param_generic[] = {
 		.name = DEVLINK_PARAM_GENERIC_TOTAL_VFS_NAME,
 		.type = DEVLINK_PARAM_GENERIC_TOTAL_VFS_TYPE,
 	},
+	{	.id = DEVLINK_PARAM_GENERIC_ID_KEEP_LINK_UP,
+		.name = DEVLINK_PARAM_GENERIC_KEEP_LINK_UP_NAME,
+		.type = DEVLINK_PARAM_GENERIC_KEEP_LINK_UP_TYPE,
+	},
 };
 
 static int devlink_param_generic_verify(const struct devlink_param *param)
-- 
2.48.1

[PATCH net-next 11/14] net/mlx5: Implement devlink keep_link_up port parameter

[Saeed Mahameed]

From: Saeed Mahameed <saeedm@nvidia.com>

When set, the NIC keeps the link up as long as the host is not in standby
mode, even when the driver is not loaded.

When enabled, netdev carrier state won't affect the physical port link state.

This is useful for when the link is needed to access onboard management
such as BMC, even if the host driver isn't loaded.

Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
---
 Documentation/networking/devlink/mlx5.rst     |   5 +
 .../ethernet/mellanox/mlx5/core/en/devlink.c  |  16 ++-
 .../ethernet/mellanox/mlx5/core/en/devlink.h  |   3 +-
 .../net/ethernet/mellanox/mlx5/core/en_main.c |   4 +-
 .../mellanox/mlx5/core/lib/nv_param.c         | 129 ++++++++++++++++++
 .../mellanox/mlx5/core/lib/nv_param.h         |   2 +
 6 files changed, 153 insertions(+), 6 deletions(-)

diff --git a/Documentation/networking/devlink/mlx5.rst b/Documentation/networking/devlink/mlx5.rst
index 00a43324dec2..c9c064de4699 100644
--- a/Documentation/networking/devlink/mlx5.rst
+++ b/Documentation/networking/devlink/mlx5.rst
@@ -45,6 +45,11 @@ Parameters
      - The range is between 1 and a device-specific max.
      - Applies to each physical function (PF) independently, if the device
        supports it. Otherwise, it applies symmetrically to all PFs.
+   * - ``keep_link_up``
+     - permanent
+     - Boolean
+     - When set, the NIC keeps the link up as long as the host is not in standby
+       mode, even when the driver is not loaded.
 
 Note: permanent parameters such as ``enable_sriov`` and ``total_vfs`` require FW reset to take effect
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/en/devlink.c
index 0b1ac6e5c890..e9d073d1f532 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/devlink.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/devlink.c
@@ -3,6 +3,7 @@
 
 #include "en/devlink.h"
 #include "eswitch.h"
+#include "lib/nv_param.h"
 
 static const struct devlink_ops mlx5e_devlink_ops = {
 };
@@ -54,6 +55,7 @@ int mlx5e_devlink_port_register(struct mlx5e_dev *mlx5e_dev,
 	struct devlink_port_attrs attrs = {};
 	struct netdev_phys_item_id ppid = {};
 	unsigned int dl_port_index;
+	int err;
 
 	if (mlx5_core_is_pf(mdev)) {
 		attrs.flavour = DEVLINK_PORT_FLAVOUR_PHYSICAL;
@@ -72,11 +74,19 @@ int mlx5e_devlink_port_register(struct mlx5e_dev *mlx5e_dev,
 
 	devlink_port_attrs_set(&mlx5e_dev->dl_port, &attrs);
 
-	return devlink_port_register(devlink, &mlx5e_dev->dl_port,
-				     dl_port_index);
+	err = devlink_port_register(devlink, &mlx5e_dev->dl_port,
+				    dl_port_index);
+	if (err)
+		return err;
+
+	err = mlx5_nv_port_param_register(mdev, &mlx5e_dev->dl_port);
+	if (err)
+		mlx5_core_warn(mdev, "Failed to register eth port params\n");
+	return 0;
 }
 
-void mlx5e_devlink_port_unregister(struct mlx5e_dev *mlx5e_dev)
+void mlx5e_devlink_port_unregister(struct mlx5e_dev *mlx5e_dev, struct mlx5_core_dev *mdev)
 {
+	mlx5_nv_port_param_unregister(mdev, &mlx5e_dev->dl_port);
 	devlink_port_unregister(&mlx5e_dev->dl_port);
 }
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/devlink.h b/drivers/net/ethernet/mellanox/mlx5/core/en/devlink.h
index d5ec4461f300..049d82732f72 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/devlink.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/devlink.h
@@ -12,6 +12,7 @@ struct mlx5e_dev *mlx5e_create_devlink(struct device *dev,
 void mlx5e_destroy_devlink(struct mlx5e_dev *mlx5e_dev);
 int mlx5e_devlink_port_register(struct mlx5e_dev *mlx5e_dev,
 				struct mlx5_core_dev *mdev);
-void mlx5e_devlink_port_unregister(struct mlx5e_dev *mlx5e_dev);
+void mlx5e_devlink_port_unregister(struct mlx5e_dev *mlx5e_dev,
+				   struct mlx5_core_dev *mdev);
 
 #endif
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 5d5e7b19c396..e7828203e29b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -6484,7 +6484,7 @@ static int _mlx5e_probe(struct auxiliary_device *adev)
 err_destroy_netdev:
 	mlx5e_destroy_netdev(priv);
 err_devlink_port_unregister:
-	mlx5e_devlink_port_unregister(mlx5e_dev);
+	mlx5e_devlink_port_unregister(mlx5e_dev, mdev);
 err_devlink_unregister:
 	mlx5e_destroy_devlink(mlx5e_dev);
 	return err;
@@ -6538,7 +6538,7 @@ static void _mlx5e_remove(struct auxiliary_device *adev)
 	if (priv->profile)
 		priv->profile->cleanup(priv);
 	mlx5e_destroy_netdev(priv);
-	mlx5e_devlink_port_unregister(mlx5e_dev);
+	mlx5e_devlink_port_unregister(mlx5e_dev, mdev);
 	mlx5e_destroy_devlink(mlx5e_dev);
 }
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
index 97d74d890582..159d75967a48 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
@@ -3,12 +3,15 @@
 
 #include "nv_param.h"
 #include "mlx5_core.h"
+#include "en.h"
 
 enum {
 	MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CONF               = 0x80,
 	MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CAP                = 0x81,
 	MLX5_CLASS_0_CTRL_ID_NV_SW_OFFLOAD_CONFIG             = 0x10a,
 
+	MLX5_CLASS_1_CTRL_ID_NV_KEEP_LINK_UP                  = 0x190,
+
 	MLX5_CLASS_3_CTRL_ID_NV_PF_PCI_CONF                   = 0x80,
 };
 
@@ -17,6 +20,12 @@ struct mlx5_ifc_configuration_item_type_class_global_bits {
 	u8         parameter_index[0x18];
 };
 
+struct mlx5_ifc_configuration_item_type_class_physical_port_bits {
+	u8    type_class[0x8];
+	u8    port[0x8];
+	u8    parameter_index[0x10];
+};
+
 struct mlx5_ifc_configuration_item_type_class_per_host_pf_bits {
 	u8         type_class[0x8];
 	u8         pf_index[0x6];
@@ -29,6 +38,9 @@ union mlx5_ifc_config_item_type_auto_bits {
 				configuration_item_type_class_global;
 	struct mlx5_ifc_configuration_item_type_class_per_host_pf_bits
 				configuration_item_type_class_per_host_pf;
+	struct mlx5_ifc_configuration_item_type_class_physical_port_bits
+				configuration_item_type_class_physical_port;
+
 	u8 reserved_at_0[0x20];
 };
 
@@ -123,6 +135,16 @@ struct mlx5_ifc_nv_sw_offload_conf_bits {
 	u8         lro_log_timeout0[0x4];
 };
 
+struct mlx5_ifc_nv_keep_link_up_bits {
+	u8    reserved_at_0[0x1a];
+	u8    auto_power_save_link_down[0x1];
+	u8    do_not_clear_port_stats[0x1];
+	u8    keep_link_up_on_standby[0x1];
+	u8    keep_link_up_on_boot[0x1];
+	u8    keep_ib_link_up[0x1];
+	u8    keep_eth_link_up[0x1];
+};
+
 #define MNVDA_HDR_SZ \
 	(MLX5_ST_SZ_BYTES(mnvda_reg) - MLX5_BYTE_OFF(mnvda_reg, configuration_item_data))
 
@@ -541,3 +563,110 @@ void mlx5_nv_param_unregister_dl_params(struct devlink *devlink)
 	devl_params_unregister(devlink, mlx5_nv_param_devlink_params,
 			       ARRAY_SIZE(mlx5_nv_param_devlink_params));
 }
+
+/* mlx5e devlink port params */
+
+static int
+mlx5_nv_param_read_keep_link_up(struct mlx5_core_dev *dev, void *mnvda, size_t len)
+{
+	MLX5_SET_CONFIG_ITEM_TYPE(physical_port, mnvda, type_class, 1);
+	MLX5_SET_CONFIG_ITEM_TYPE(physical_port, mnvda, parameter_index,
+				  MLX5_CLASS_1_CTRL_ID_NV_KEEP_LINK_UP);
+	MLX5_SET_CONFIG_ITEM_TYPE(physical_port, mnvda, port, mlx5_get_dev_index(dev) + 1);
+	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_keep_link_up);
+
+	return mlx5_nv_param_read(dev, mnvda, len);
+}
+
+static int
+mlx5_nv_port_param_keep_link_up_get(struct devlink *devlink, u32 id,
+				    struct devlink_param_gset_ctx *ctx)
+{
+	struct mlx5e_dev *edev = devlink_priv(devlink);
+	struct mlx5_core_dev *dev = edev->priv->mdev;
+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
+	void *data;
+	int err;
+
+	err = mlx5_nv_param_read_keep_link_up(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+	if (MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_ETH)
+		ctx->val.vbool = !!MLX5_GET(nv_keep_link_up, data, keep_eth_link_up);
+	else if (MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_IB)
+		ctx->val.vbool = !!MLX5_GET(nv_keep_link_up, data, keep_ib_link_up);
+	else
+		ctx->val.vbool = false;
+
+	return 0;
+}
+
+static int
+mlx5_nv_port_param_keep_link_up_set(struct devlink *devlink, u32 id,
+				    struct devlink_param_gset_ctx *ctx,
+				    struct netlink_ext_ack *extack)
+{
+	struct mlx5e_dev *edev = devlink_priv(devlink);
+	struct mlx5_core_dev *dev = edev->priv->mdev;
+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
+	void *data;
+	int err;
+
+	err = mlx5_nv_param_read_keep_link_up(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+
+	if (MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_ETH)
+		MLX5_SET(nv_keep_link_up, data, keep_eth_link_up, ctx->val.vbool);
+	else if (MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_IB)
+		MLX5_SET(nv_keep_link_up, data, keep_ib_link_up, ctx->val.vbool);
+	else
+		return -EOPNOTSUPP;
+
+	return mlx5_nv_param_write(dev, mnvda,  sizeof(mnvda));
+}
+
+static int
+mlx5_nv_port_param_keep_link_up_validate(struct devlink *devlink, u32 id,
+					 union devlink_param_value val,
+					 struct netlink_ext_ack *extack)
+{
+	struct mlx5e_dev *edev = devlink_priv(devlink);
+	struct mlx5_core_dev *dev = edev->priv->mdev;
+
+	if (MLX5_CAP_GEN(dev, port_type) != MLX5_CAP_PORT_TYPE_ETH &&
+	    MLX5_CAP_GEN(dev, port_type) != MLX5_CAP_PORT_TYPE_IB) {
+		NL_SET_ERR_MSG_MOD(extack,
+				   "Not supported on this device link type");
+		return -EOPNOTSUPP;
+	}
+
+	return 0;
+}
+
+static const struct devlink_param mlx5_nv_param_devlink_port_params[] = {
+	DEVLINK_PARAM_GENERIC(KEEP_LINK_UP, BIT(DEVLINK_PARAM_CMODE_PERMANENT),
+			      mlx5_nv_port_param_keep_link_up_get,
+			      mlx5_nv_port_param_keep_link_up_set,
+			      mlx5_nv_port_param_keep_link_up_validate),
+};
+
+int mlx5_nv_port_param_register(struct mlx5_core_dev *dev, struct devlink_port *port)
+{
+	if (!mlx5_core_is_pf(dev))
+		return 0;
+	return devlink_port_params_register(port, mlx5_nv_param_devlink_port_params,
+					    ARRAY_SIZE(mlx5_nv_param_devlink_port_params));
+}
+
+void mlx5_nv_port_param_unregister(struct mlx5_core_dev *dev, struct devlink_port *port)
+{
+	if (!mlx5_core_is_pf(dev))
+		return;
+	devlink_port_params_unregister(port, mlx5_nv_param_devlink_port_params,
+				       ARRAY_SIZE(mlx5_nv_param_devlink_port_params));
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.h
index 9f4922ff7745..943933bdc246 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.h
@@ -9,6 +9,8 @@
 
 int mlx5_nv_param_register_dl_params(struct devlink *devlink);
 void mlx5_nv_param_unregister_dl_params(struct devlink *devlink);
+int mlx5_nv_port_param_register(struct mlx5_core_dev *dev, struct devlink_port *port);
+void mlx5_nv_port_param_unregister(struct mlx5_core_dev *dev, struct devlink_port *port);
 
 #endif
 
-- 
2.48.1

[PATCH net-next 12/14] devlink: Throw extack messages on param value validation error

[Saeed Mahameed]

From: Saeed Mahameed <saeedm@nvidia.com>

Centralize devlink param value data validation in one function and
fill corresponding extack error messages on validation error.

Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
---
 net/devlink/param.c | 76 +++++++++++++++++++++++++++++++++++----------
 1 file changed, 60 insertions(+), 16 deletions(-)

diff --git a/net/devlink/param.c b/net/devlink/param.c
index 03c65ccf2acf..1922ca5b9cbc 100644
--- a/net/devlink/param.c
+++ b/net/devlink/param.c
@@ -458,45 +458,89 @@ devlink_param_type_get_from_info(struct genl_info *info,
 	return 0;
 }
 
+static int
+devlink_param_value_validate(struct genl_info *info,
+			     enum devlink_param_type type)
+{
+	struct netlink_ext_ack *extack = info->extack;
+	struct nlattr *param_data;
+	int len = 0;
+
+	if (type != DEVLINK_PARAM_TYPE_BOOL &&
+	    GENL_REQ_ATTR_CHECK(info, DEVLINK_ATTR_PARAM_VALUE_DATA))
+		return -EINVAL;
+
+	param_data = info->attrs[DEVLINK_ATTR_PARAM_VALUE_DATA];
+
+	if (param_data)
+		len = nla_len(param_data);
+
+	switch (type) {
+	case DEVLINK_PARAM_TYPE_U8:
+		if (len == sizeof(u8))
+			return 0;
+		NL_SET_ERR_MSG_FMT_MOD(extack,
+				       "Expected uint8, got %d bytes", len);
+		break;
+	case DEVLINK_PARAM_TYPE_U16:
+		if (len == sizeof(u16))
+			return 0;
+		NL_SET_ERR_MSG_FMT_MOD(extack,
+				       "Expected uint16, got %d bytes", len);
+		break;
+	case DEVLINK_PARAM_TYPE_U32:
+		if (len == sizeof(u32))
+			return 0;
+		NL_SET_ERR_MSG_FMT_MOD(extack,
+				       "Expected uint32, got %d bytes", len);
+		break;
+	case DEVLINK_PARAM_TYPE_STRING:
+		len = strnlen(nla_data(param_data), nla_len(param_data));
+
+		if (len < nla_len(param_data) &&
+		    len < __DEVLINK_PARAM_MAX_STRING_VALUE)
+			return 0;
+		NL_SET_ERR_MSG_MOD(extack, "String too long");
+		break;
+	case DEVLINK_PARAM_TYPE_BOOL:
+		if (!len)
+			return 0;
+		NL_SET_ERR_MSG_MOD(extack, "Expected flag, got data");
+		break;
+	default:
+		NL_SET_ERR_MSG_FMT_MOD(extack,
+				       "Not supported value type %d", type);
+		break;
+	}
+	return -EINVAL;
+}
+
 static int
 devlink_param_value_get_from_info(const struct devlink_param *param,
 				  struct genl_info *info,
 				  union devlink_param_value *value)
 {
 	struct nlattr *param_data;
-	int len;
-
-	param_data = info->attrs[DEVLINK_ATTR_PARAM_VALUE_DATA];
 
-	if (param->type != DEVLINK_PARAM_TYPE_BOOL && !param_data)
+	if (devlink_param_value_validate(info, param->type))
 		return -EINVAL;
 
+	param_data = info->attrs[DEVLINK_ATTR_PARAM_VALUE_DATA];
+
 	switch (param->type) {
 	case DEVLINK_PARAM_TYPE_U8:
-		if (nla_len(param_data) != sizeof(u8))
-			return -EINVAL;
 		value->vu8 = nla_get_u8(param_data);
 		break;
 	case DEVLINK_PARAM_TYPE_U16:
-		if (nla_len(param_data) != sizeof(u16))
-			return -EINVAL;
 		value->vu16 = nla_get_u16(param_data);
 		break;
 	case DEVLINK_PARAM_TYPE_U32:
-		if (nla_len(param_data) != sizeof(u32))
-			return -EINVAL;
 		value->vu32 = nla_get_u32(param_data);
 		break;
 	case DEVLINK_PARAM_TYPE_STRING:
-		len = strnlen(nla_data(param_data), nla_len(param_data));
-		if (len == nla_len(param_data) ||
-		    len >= __DEVLINK_PARAM_MAX_STRING_VALUE)
-			return -EINVAL;
 		strcpy(value->vstr, nla_data(param_data));
 		break;
 	case DEVLINK_PARAM_TYPE_BOOL:
-		if (param_data && nla_len(param_data))
-			return -EINVAL;
 		value->vbool = nla_get_flag(param_data);
 		break;
 	}
-- 
2.48.1

[PATCH net-next 13/14] devlink: Implement devlink param multi attribute nested data values

[Saeed Mahameed]

From: Saeed Mahameed <saeedm@nvidia.com>

Devlink param value attribute is not defined since devlink is handling
the value validating and parsing internally, this allows us to implement
multi attribute values without breaking any policies.

Devlink param multi-attribute values are considered to be dynamically
sized arrays of u32 values, by introducing a new devlink param type
DEVLINK_PARAM_TYPE_ARR_U32, driver and user space can set a variable
count of u32 values into the DEVLINK_ATTR_PARAM_VALUE_DATA attribute.

Implement get/set parsing and add to the internal value structure passed
to drivers.

This is useful for devices that need to configure a list of values for
a specific configuration.

example:
 $ devlink dev param show pci/... name multi-value-param
    name multi-value-param type driver-specific
    values:
      cmode permanent value: 0,1,2,3,4,5,6,7

 $ devlink dev param set pci/... name multi-value-param \
	value 4,5,6,7,0,1,2,3 cmode permanent

Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
---
 include/net/devlink.h        |  7 ++++++
 include/uapi/linux/devlink.h |  1 +
 net/devlink/param.c          | 47 +++++++++++++++++++++++++++++++++++-
 3 files changed, 54 insertions(+), 1 deletion(-)

diff --git a/include/net/devlink.h b/include/net/devlink.h
index ca32c61583cf..c8f14ea3604e 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -420,12 +420,15 @@ typedef u64 devlink_resource_occ_get_t(void *priv);
 #define DEVLINK_RESOURCE_GENERIC_NAME_PORTS "physical_ports"
 
 #define __DEVLINK_PARAM_MAX_STRING_VALUE 32
+#define __DEVLINK_PARAM_MAX_ARRAY_SIZE 32
+
 enum devlink_param_type {
 	DEVLINK_PARAM_TYPE_U8,
 	DEVLINK_PARAM_TYPE_U16,
 	DEVLINK_PARAM_TYPE_U32,
 	DEVLINK_PARAM_TYPE_STRING,
 	DEVLINK_PARAM_TYPE_BOOL,
+	DEVLINK_PARAM_TYPE_ARR_U32,
 };
 
 union devlink_param_value {
@@ -434,6 +437,10 @@ union devlink_param_value {
 	u32 vu32;
 	char vstr[__DEVLINK_PARAM_MAX_STRING_VALUE];
 	bool vbool;
+	struct {
+		u32 size;
+		u32 vu32[__DEVLINK_PARAM_MAX_ARRAY_SIZE];
+	} arr;
 };
 
 struct devlink_param_gset_ctx {
diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
index 8cdd60eb3c43..df7c29bbb7a7 100644
--- a/include/uapi/linux/devlink.h
+++ b/include/uapi/linux/devlink.h
@@ -400,6 +400,7 @@ enum devlink_dyn_attr_type {
 	DEVLINK_DYN_ATTR_TYPE_BINARY,
 	__DEVLINK_DYN_ATTR_TYPE_CUSTOM_BASE = 0x80,
 	/* Any possible custom types, unrelated to NLA_* values go below */
+	DEVLINK_DYN_ATTR_TYPE_U32_ARRAY,
 };
 
 enum devlink_attr {
diff --git a/net/devlink/param.c b/net/devlink/param.c
index 1922ca5b9cbc..31a8e61bae09 100644
--- a/net/devlink/param.c
+++ b/net/devlink/param.c
@@ -195,6 +195,8 @@ devlink_param_type_to_dyn_attr_type(enum devlink_param_type param_type)
 		return DEVLINK_DYN_ATTR_TYPE_STRING;
 	case DEVLINK_PARAM_TYPE_BOOL:
 		return DEVLINK_DYN_ATTR_TYPE_FLAG;
+	case DEVLINK_PARAM_TYPE_ARR_U32:
+		return DEVLINK_DYN_ATTR_TYPE_U32_ARRAY;
 	default:
 		return -EINVAL;
 	}
@@ -239,6 +241,13 @@ devlink_nl_param_value_fill_one(struct sk_buff *msg,
 		    nla_put_flag(msg, DEVLINK_ATTR_PARAM_VALUE_DATA))
 			goto value_nest_cancel;
 		break;
+	case DEVLINK_PARAM_TYPE_ARR_U32:
+		for (int i = 0; i < val.arr.size; i++) {
+			if (nla_put_u32(msg, DEVLINK_ATTR_PARAM_VALUE_DATA,
+					val.arr.vu32[i]))
+				goto value_nest_cancel;
+		}
+		break;
 	}
 
 	nla_nest_end(msg, param_value_attr);
@@ -451,6 +460,9 @@ devlink_param_type_get_from_info(struct genl_info *info,
 	case DEVLINK_DYN_ATTR_TYPE_FLAG:
 		*param_type = DEVLINK_PARAM_TYPE_BOOL;
 		break;
+	case DEVLINK_DYN_ATTR_TYPE_U32_ARRAY:
+		*param_type = DEVLINK_PARAM_TYPE_ARR_U32;
+		break;
 	default:
 		return -EINVAL;
 	}
@@ -464,7 +476,7 @@ devlink_param_value_validate(struct genl_info *info,
 {
 	struct netlink_ext_ack *extack = info->extack;
 	struct nlattr *param_data;
-	int len = 0;
+	int len = 0, rem;
 
 	if (type != DEVLINK_PARAM_TYPE_BOOL &&
 	    GENL_REQ_ATTR_CHECK(info, DEVLINK_ATTR_PARAM_VALUE_DATA))
@@ -507,6 +519,28 @@ devlink_param_value_validate(struct genl_info *info,
 			return 0;
 		NL_SET_ERR_MSG_MOD(extack, "Expected flag, got data");
 		break;
+	case DEVLINK_PARAM_TYPE_ARR_U32:
+		len = 0;
+		nla_for_each_attr_type(param_data,
+				       DEVLINK_ATTR_PARAM_VALUE_DATA,
+				       genlmsg_data(info->genlhdr),
+				       genlmsg_len(info->genlhdr), rem) {
+			if (nla_len(param_data) != sizeof(u32)) {
+				NL_SET_ERR_MSG_MOD(extack,
+						   "Array element size must be 4 bytes");
+				return -EINVAL;
+			}
+			if (++len > __DEVLINK_PARAM_MAX_ARRAY_SIZE) {
+				NL_SET_ERR_MSG_MOD(extack,
+						   "Array size exceeds maximum");
+				return -EINVAL;
+			}
+		}
+		if (len)
+			return 0;
+		NL_SET_ERR_MSG_MOD(extack,
+				   "Value array must have at least one entry");
+		break;
 	default:
 		NL_SET_ERR_MSG_FMT_MOD(extack,
 				       "Not supported value type %d", type);
@@ -521,6 +555,7 @@ devlink_param_value_get_from_info(const struct devlink_param *param,
 				  union devlink_param_value *value)
 {
 	struct nlattr *param_data;
+	int rem, i = 0;
 
 	if (devlink_param_value_validate(info, param->type))
 		return -EINVAL;
@@ -543,6 +578,16 @@ devlink_param_value_get_from_info(const struct devlink_param *param,
 	case DEVLINK_PARAM_TYPE_BOOL:
 		value->vbool = nla_get_flag(param_data);
 		break;
+	case DEVLINK_PARAM_TYPE_ARR_U32: {
+		nla_for_each_attr_type(param_data,
+				       DEVLINK_ATTR_PARAM_VALUE_DATA,
+				       genlmsg_data(info->genlhdr),
+				       genlmsg_len(info->genlhdr), rem)
+			value->arr.vu32[i++] = nla_get_u32(param_data);
+
+		value->arr.size = i;
+		break;
+		}
 	}
 	return 0;
 }
-- 
2.48.1

[PATCH net-next 14/14] net/mlx5: Implement eSwitch hairpin per prio buffers devlink params

[Saeed Mahameed]

From: Saeed Mahameed <saeedm@nvidia.com>

E-Switch hairpin per prio buffers are controlled and configurable by the
device, add two devlink params to control them.

esw_haripin_per_prio_log_queue_size: p0,p1,....,p7
  Log(base 2) of the number of packets descriptors allocated
  internally for hairpin for IEEE802.1p priorities.
  0 means that no descriptors are allocated for this priority
  and traffic with this priority will be dropped.

esw_hairpin_per_prio_log_buf_size: p0,p1,...,p7
  Log(base 2) of the buffer size (in bytes) allocated internally
  for hairpin for IEEE802.1p priorities.
  0 means no buffer for this priority and traffic with this
  priority will be dropped.

Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
---
 Documentation/networking/devlink/mlx5.rst     |  15 +
 .../net/ethernet/mellanox/mlx5/core/devlink.h |   4 +-
 .../mellanox/mlx5/core/lib/nv_param.c         | 272 ++++++++++++++++++
 3 files changed, 290 insertions(+), 1 deletion(-)

diff --git a/Documentation/networking/devlink/mlx5.rst b/Documentation/networking/devlink/mlx5.rst
index c9c064de4699..053060de6126 100644
--- a/Documentation/networking/devlink/mlx5.rst
+++ b/Documentation/networking/devlink/mlx5.rst
@@ -161,6 +161,21 @@ parameters.
        * ``balanced`` : Merges fewer CQEs, resulting in a moderate compression ratio but maintaining a balance between bandwidth savings and performance
        * ``aggressive`` : Merges more CQEs into a single entry, achieving a higher compression rate and maximizing performance, particularly under high traffic loads
 
+   * - ``esw_hairpin_per_prio_log_queue_size``
+     - u32 array[8]
+     - permanent
+     - each item is log(base 2) of the number of packet descriptors allocated
+       internally for hairpin for IEEE802.1p priorities.
+       0 means that no descriptors are allocated for this priority
+       and traffic with this priority will be dropped.
+
+   * - ``esw_hairpin_per_prio_log_buf_size``
+     - u32 array[8]
+     - permanent
+     - each item is log(base 2) of the buffer size (in bytes) allocated internally
+       for hairpin for IEEE802.1p priorities.
+       0 means no buffer for this priority and traffic with this priority will be dropped.
+
 The ``mlx5`` driver supports reloading via ``DEVLINK_CMD_RELOAD``
 
 Info versions
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h
index 74bcdfa70361..b2c10ce1eac5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h
@@ -22,7 +22,9 @@ enum mlx5_devlink_param_id {
 	MLX5_DEVLINK_PARAM_ID_ESW_MULTIPORT,
 	MLX5_DEVLINK_PARAM_ID_HAIRPIN_NUM_QUEUES,
 	MLX5_DEVLINK_PARAM_ID_HAIRPIN_QUEUE_SIZE,
-	MLX5_DEVLINK_PARAM_ID_CQE_COMPRESSION_TYPE
+	MLX5_DEVLINK_PARAM_ID_CQE_COMPRESSION_TYPE,
+	MLX5_DEVLINK_PARAM_ID_ESW_HAIRPIN_DESCRIPTORS,
+	MLX5_DEVLINK_PARAM_ID_ESW_HAIRPIN_DATA_SIZE,
 };
 
 struct mlx5_trap_ctx {
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
index 159d75967a48..d9815c66ea58 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
@@ -1,11 +1,15 @@
 // SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
 /* Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
 
+#include <net/dcbnl.h>
+
 #include "nv_param.h"
 #include "mlx5_core.h"
 #include "en.h"
 
 enum {
+	MLX5_CLASS_0_CTRL_ID_NV_INTERNAL_HAIRPIN_CONF         = 0x13,
+	MLX5_CLASS_0_CTRL_ID_NV_INTERNAL_HAIRPIN_CAP          = 0x14,
 	MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CONF               = 0x80,
 	MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CAP                = 0x81,
 	MLX5_CLASS_0_CTRL_ID_NV_SW_OFFLOAD_CONFIG             = 0x10a,
@@ -145,6 +149,19 @@ struct mlx5_ifc_nv_keep_link_up_bits {
 	u8    keep_eth_link_up[0x1];
 };
 
+struct mlx5_ifc_nv_internal_hairpin_cap_bits {
+	u8    log_max_hpin_total_num_descriptors[0x8];
+	u8    log_max_hpin_total_data_size[0x8];
+	u8    log_max_hpin_num_descriptor_per_prio[0x8];
+	u8    log_max_hpin_data_size_per_prio[0x8];
+};
+
+struct mlx5_ifc_nv_internal_hairpin_conf_bits {
+	u8    log_hpin_num_descriptor[8][0x8];
+
+	u8    log_hpin_data_size[8][0x8];
+};
+
 #define MNVDA_HDR_SZ \
 	(MLX5_ST_SZ_BYTES(mnvda_reg) - MLX5_BYTE_OFF(mnvda_reg, configuration_item_data))
 
@@ -531,6 +548,247 @@ static int mlx5_devlink_total_vfs_validate(struct devlink *devlink, u32 id,
 	return 0;
 }
 
+static int
+mlx5_nv_param_read_internal_hairpin_conf(struct mlx5_core_dev *dev,
+					 void *mnvda, size_t len)
+{
+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, type_class, 0);
+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, parameter_index,
+				  MLX5_CLASS_0_CTRL_ID_NV_INTERNAL_HAIRPIN_CONF);
+	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_internal_hairpin_conf);
+
+	return mlx5_nv_param_read(dev, mnvda, len);
+}
+
+static int
+mlx5_nv_param_read_internal_hairpin_cap(struct mlx5_core_dev *dev,
+					void *mnvda, size_t len)
+{
+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, type_class, 0);
+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, parameter_index,
+				  MLX5_CLASS_0_CTRL_ID_NV_INTERNAL_HAIRPIN_CAP);
+
+	return mlx5_nv_param_read(dev, mnvda, len);
+}
+
+static int
+mlx5_nv_param_esw_hairpin_descriptors_get(struct devlink *devlink, u32 id,
+					  struct devlink_param_gset_ctx *ctx)
+
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
+	void *data;
+	int err, i;
+
+	BUILD_BUG_ON(IEEE_8021QAZ_MAX_TCS > __DEVLINK_PARAM_MAX_ARRAY_SIZE);
+
+	err = mlx5_nv_param_read_internal_hairpin_conf(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+
+	ctx->val.arr.size = IEEE_8021QAZ_MAX_TCS;
+	for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++)
+		ctx->val.arr.vu32[i] = MLX5_GET(nv_internal_hairpin_conf, data,
+						log_hpin_num_descriptor[i]);
+	return 0;
+}
+
+static int
+mlx5_nv_param_esw_hairpin_descriptors_set(struct devlink *devlink, u32 id,
+					  struct devlink_param_gset_ctx *ctx,
+					  struct netlink_ext_ack *extack)
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
+	void *data;
+	int err, i;
+
+	err = mlx5_nv_param_read_internal_hairpin_conf(dev, mnvda, sizeof(mnvda));
+	if (err) {
+		NL_SET_ERR_MSG_MOD(extack, "Unable to query internal hairpin conf");
+		return err;
+	}
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+	for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++)
+		MLX5_SET(nv_internal_hairpin_conf, data,
+			 log_hpin_num_descriptor[i], ctx->val.arr.vu32[i]);
+
+	return mlx5_nv_param_write(dev, mnvda,  sizeof(mnvda));
+}
+
+static int
+mlx5_nv_param_esw_hairpin_descriptors_validate(struct devlink *devlink, u32 id,
+					       union devlink_param_value val,
+					       struct netlink_ext_ack *extack)
+{
+	u8 log_max_num_descriptors, log_max_total_descriptors;
+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
+	u16 total = 0;
+	void *data;
+	int err, i;
+
+	if (val.arr.size != IEEE_8021QAZ_MAX_TCS) {
+		NL_SET_ERR_MSG_FMT_MOD(extack, "Array size must be %d",
+				       IEEE_8021QAZ_MAX_TCS);
+		return -EINVAL;
+	}
+	err = mlx5_nv_param_read_internal_hairpin_cap(devlink_priv(devlink),
+						      mnvda, sizeof(mnvda));
+	if (err) {
+		NL_SET_ERR_MSG_MOD(extack, "Unable to query internal hairpin cap");
+		return err;
+	}
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+	log_max_total_descriptors = MLX5_GET(nv_internal_hairpin_cap, data,
+					     log_max_hpin_total_num_descriptors);
+	log_max_num_descriptors = MLX5_GET(nv_internal_hairpin_cap, data,
+					   log_max_hpin_num_descriptor_per_prio);
+
+	for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++) {
+		if (val.arr.vu32[i] <= log_max_num_descriptors)
+			continue;
+
+		NL_SET_ERR_MSG_FMT_MOD(extack,
+				       "Max allowed value per prio is %d",
+				       log_max_num_descriptors);
+		return -ERANGE;
+	}
+
+	/* Validate total number of descriptors */
+	memset(mnvda, 0, sizeof(mnvda));
+	err = mlx5_nv_param_read_internal_hairpin_conf(devlink_priv(devlink),
+						       mnvda, sizeof(mnvda));
+	if (err) {
+		NL_SET_ERR_MSG_MOD(extack, "Unable to query internal hairpin conf");
+		return err;
+	}
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+
+	for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++)
+		total += 1 << val.arr.vu32[i];
+
+	if (total > (1 << log_max_total_descriptors)) {
+		NL_SET_ERR_MSG_FMT_MOD(extack,
+				       "Log max total value allowed is %d",
+				       log_max_total_descriptors);
+		return -ERANGE;
+	}
+
+	return 0;
+}
+
+static int
+mlx5_nv_param_esw_hairpin_data_size_get(struct devlink *devlink, u32 id,
+					struct devlink_param_gset_ctx *ctx)
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
+	void *data;
+	int err, i;
+
+	err = mlx5_nv_param_read_internal_hairpin_conf(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+	ctx->val.arr.size = IEEE_8021QAZ_MAX_TCS;
+	for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++)
+		ctx->val.arr.vu32[i] = MLX5_GET(nv_internal_hairpin_conf, data,
+						log_hpin_data_size[i]);
+	return 0;
+}
+
+static int
+mlx5_nv_param_esw_hairpin_data_size_set(struct devlink *devlink, u32 id,
+					struct devlink_param_gset_ctx *ctx,
+					struct netlink_ext_ack *extack)
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
+	int err, i;
+	void *data;
+
+	err = mlx5_nv_param_read_internal_hairpin_conf(dev, mnvda, sizeof(mnvda));
+	if (err)
+		return err;
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+
+	for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++)
+		MLX5_SET(nv_internal_hairpin_conf, data, log_hpin_data_size[i],
+			 ctx->val.arr.vu32[i]);
+
+	return mlx5_nv_param_write(dev, mnvda,  sizeof(mnvda));
+}
+
+static int
+mlx5_nv_param_esw_hairpin_data_size_validate(struct devlink *devlink, u32 id,
+					     union devlink_param_value val,
+					     struct netlink_ext_ack *extack)
+{
+	u8 log_max_data_size, log_max_total_data_size;
+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
+	unsigned long total = 0;
+	void *data;
+	int err, i;
+
+	if (val.arr.size != IEEE_8021QAZ_MAX_TCS) {
+		NL_SET_ERR_MSG_FMT_MOD(extack, "Array size must be %d",
+				       IEEE_8021QAZ_MAX_TCS);
+		return -EINVAL;
+	}
+
+	err = mlx5_nv_param_read_internal_hairpin_cap(devlink_priv(devlink),
+						      mnvda, sizeof(mnvda));
+	if (err) {
+		NL_SET_ERR_MSG_MOD(extack, "Unable to query internal hairpin cap");
+		return err;
+	}
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+	log_max_data_size = MLX5_GET(nv_internal_hairpin_cap, data,
+				     log_max_hpin_data_size_per_prio);
+	log_max_total_data_size = MLX5_GET(nv_internal_hairpin_cap, data,
+					   log_max_hpin_total_data_size);
+
+	for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++) {
+		if (val.arr.vu32[i] <= log_max_data_size)
+			continue;
+
+		NL_SET_ERR_MSG_FMT_MOD(extack,
+				       "Max allowed value per prio is %d",
+				       log_max_data_size);
+		return -ERANGE;
+	}
+
+	/* Validate total data size */
+	memset(mnvda, 0, sizeof(mnvda));
+	err = mlx5_nv_param_read_internal_hairpin_conf(devlink_priv(devlink),
+						       mnvda, sizeof(mnvda));
+	if (err) {
+		NL_SET_ERR_MSG_MOD(extack, "Unable to query internal hairpin conf");
+		return err;
+	}
+
+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+
+	for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++)
+		total += 1 << val.arr.vu32[i];
+
+	if (total > (1 << log_max_total_data_size)) {
+		NL_SET_ERR_MSG_FMT_MOD(extack,
+				       "Log max total value allowed is %d",
+				       log_max_total_data_size);
+		return -ERANGE;
+	}
+
+	return 0;
+}
+
 static const struct devlink_param mlx5_nv_param_devlink_params[] = {
 	DEVLINK_PARAM_GENERIC(ENABLE_SRIOV, BIT(DEVLINK_PARAM_CMODE_PERMANENT),
 			      mlx5_devlink_enable_sriov_get,
@@ -544,6 +802,20 @@ static const struct devlink_param mlx5_nv_param_devlink_params[] = {
 			     mlx5_nv_param_devlink_cqe_compress_get,
 			     mlx5_nv_param_devlink_cqe_compress_set,
 			     mlx5_nv_param_devlink_cqe_compress_validate),
+	DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_ID_ESW_HAIRPIN_DESCRIPTORS,
+			     "esw_hairpin_per_prio_log_queue_size",
+			     DEVLINK_PARAM_TYPE_ARR_U32,
+			     BIT(DEVLINK_PARAM_CMODE_PERMANENT),
+			     mlx5_nv_param_esw_hairpin_descriptors_get,
+			     mlx5_nv_param_esw_hairpin_descriptors_set,
+			     mlx5_nv_param_esw_hairpin_descriptors_validate),
+	DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_ID_ESW_HAIRPIN_DATA_SIZE,
+			     "esw_hairpin_per_prio_log_buf_size",
+			     DEVLINK_PARAM_TYPE_ARR_U32,
+			     BIT(DEVLINK_PARAM_CMODE_PERMANENT),
+			     mlx5_nv_param_esw_hairpin_data_size_get,
+			     mlx5_nv_param_esw_hairpin_data_size_set,
+			     mlx5_nv_param_esw_hairpin_data_size_validate),
 };
 
 int mlx5_nv_param_register_dl_params(struct devlink *devlink)
-- 
2.48.1

Re: [PATCH net-next 07/14] devlink: Implement port params registration

[Przemek Kitszel]

On 2/28/25 03:12, Saeed Mahameed wrote:
> From: Saeed Mahameed <saeedm@nvidia.com>
> 
> Port params infrastructure is incomplete and needs a bit of plumbing to
> support port params commands from netlink.
> 
> Introduce port params registration API, very similar to current devlink
> params API, add the params xarray to devlink_port structure and
> decouple devlink params registration routines from the devlink
> structure.
> 
> Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
> Reviewed-by: Jiri Pirko <jiri@nvidia.com>
> ---
>   include/net/devlink.h |  14 ++++
>   net/devlink/param.c   | 150 ++++++++++++++++++++++++++++++++++--------
>   net/devlink/port.c    |   3 +
>   3 files changed, 140 insertions(+), 27 deletions(-)
For me devlink and devlink-port should be really the same, to the point
that the only difference is `bool is_port` flag inside of the
struct devlink. Then you could put special logic if really desired (to
exclude something for port).
Then for ease of driver programming you could have also a flag
"for_port" in the struct devlink_param, so developers will fill that
out statically and call it on all their devlinks (incl port).

Multiplying the APIs instead of rethinking a problem is not a good long
term solution.

Re: [PATCH net-next 07/14] devlink: Implement port params registration

[Jiri Pirko]

Fri, Feb 28, 2025 at 12:58:38PM +0100, przemyslaw.kitszel@intel.com wrote:
>On 2/28/25 03:12, Saeed Mahameed wrote:
>> From: Saeed Mahameed <saeedm@nvidia.com>
>> 
>> Port params infrastructure is incomplete and needs a bit of plumbing to
>> support port params commands from netlink.
>> 
>> Introduce port params registration API, very similar to current devlink
>> params API, add the params xarray to devlink_port structure and
>> decouple devlink params registration routines from the devlink
>> structure.
>> 
>> Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
>> Reviewed-by: Jiri Pirko <jiri@nvidia.com>
>> ---
>>   include/net/devlink.h |  14 ++++
>>   net/devlink/param.c   | 150 ++++++++++++++++++++++++++++++++++--------
>>   net/devlink/port.c    |   3 +
>>   3 files changed, 140 insertions(+), 27 deletions(-)
>For me devlink and devlink-port should be really the same, to the point
>that the only difference is `bool is_port` flag inside of the
>struct devlink. Then you could put special logic if really desired (to
>exclude something for port).

Why? Why other devlink objects shouldn't be the same as well. Then we
can have one union. Does not make sense to me. The only think dev and
port is sharing would be params. What else? Totally different beast.


>Then for ease of driver programming you could have also a flag
>"for_port" in the struct devlink_param, so developers will fill that
>out statically and call it on all their devlinks (incl port).
>
>Multiplying the APIs instead of rethinking a problem is not a good long
>term solution.
>

Re: [PATCH net-next 02/14] devlink: Add 'total_vfs' generic device param

[Jiri Pirko]

Fri, Feb 28, 2025 at 03:12:15AM +0100, saeed@kernel.org wrote:
>From: Vlad Dumitrescu <vdumitrescu@nvidia.com>
>
>NICs are typically configured with total_vfs=0, forcing users to rely
>on external tools to enable SR-IOV (a widely used and essential feature).
>
>Add total_vfs parameter to devlink for SR-IOV max VF configurability.
>Enables standard kernel tools to manage SR-IOV, addressing the need for
>flexible VF configuration.
>
>Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com>
>Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

Reviewed-by: Jiri Pirko <jiri@nvidia.com>

Re: [PATCH net-next 04/14] net/mlx5: Implement devlink enable_sriov parameter

[Jiri Pirko]

Fri, Feb 28, 2025 at 03:12:17AM +0100, saeed@kernel.org wrote:
>From: Vlad Dumitrescu <vdumitrescu@nvidia.com>
>
>Example usage:
>  devlink dev param set pci/0000:01:00.0 name enable_sriov value {true, false} cmode permanent
>  devlink dev reload pci/0000:01:00.0 action fw_activate
>  echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove
>  echo 1 >/sys/bus/pci/rescan
>  grep ^ /sys/bus/pci/devices/0000:01:00.0/sriov_*
>
>Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com>
>Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
>---
> Documentation/networking/devlink/mlx5.rst     |  14 +-
> .../net/ethernet/mellanox/mlx5/core/devlink.c |   1 +
> .../mellanox/mlx5/core/lib/nv_param.c         | 184 ++++++++++++++++++
> 3 files changed, 196 insertions(+), 3 deletions(-)
>
>diff --git a/Documentation/networking/devlink/mlx5.rst b/Documentation/networking/devlink/mlx5.rst
>index 417e5cdcd35d..587e0200c1cd 100644
>--- a/Documentation/networking/devlink/mlx5.rst
>+++ b/Documentation/networking/devlink/mlx5.rst
>@@ -15,23 +15,31 @@ Parameters
>    * - Name
>      - Mode
>      - Validation
>+     - Notes
>    * - ``enable_roce``
>      - driverinit
>-     - Type: Boolean
>-
>-       If the device supports RoCE disablement, RoCE enablement state controls
>+     - Boolean
>+     - If the device supports RoCE disablement, RoCE enablement state controls
>        device support for RoCE capability. Otherwise, the control occurs in the
>        driver stack. When RoCE is disabled at the driver level, only raw
>        ethernet QPs are supported.
>    * - ``io_eq_size``
>      - driverinit
>      - The range is between 64 and 4096.
>+     -
>    * - ``event_eq_size``
>      - driverinit
>      - The range is between 64 and 4096.
>+     -
>    * - ``max_macs``
>      - driverinit
>      - The range is between 1 and 2^31. Only power of 2 values are supported.
>+     -
>+   * - ``enable_sriov``
>+     - permanent
>+     - Boolean
>+     - Applies to each physical function (PF) independently, if the device
>+       supports it. Otherwise, it applies symmetrically to all PFs.
> 
> The ``mlx5`` driver also implements the following driver-specific
> parameters.
>diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>index 1f764ae4f4aa..7a702d84f19a 100644
>--- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>+++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>@@ -8,6 +8,7 @@
> #include "fs_core.h"
> #include "eswitch.h"
> #include "esw/qos.h"
>+#include "lib/nv_param.h"
> #include "sf/dev/dev.h"
> #include "sf/sf.h"
> #include "lib/nv_param.h"
>diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
>index 5ab37a88c260..6b63fc110e2d 100644
>--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
>+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
>@@ -5,7 +5,11 @@
> #include "mlx5_core.h"
> 
> enum {
>+	MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CONF               = 0x80,
>+	MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CAP                = 0x81,
> 	MLX5_CLASS_0_CTRL_ID_NV_SW_OFFLOAD_CONFIG             = 0x10a,
>+
>+	MLX5_CLASS_3_CTRL_ID_NV_PF_PCI_CONF                   = 0x80,
> };
> 
> struct mlx5_ifc_configuration_item_type_class_global_bits {
>@@ -13,9 +17,18 @@ struct mlx5_ifc_configuration_item_type_class_global_bits {
> 	u8         parameter_index[0x18];
> };
> 
>+struct mlx5_ifc_configuration_item_type_class_per_host_pf_bits {
>+	u8         type_class[0x8];
>+	u8         pf_index[0x6];
>+	u8         pci_bus_index[0x8];
>+	u8         parameter_index[0xa];
>+};
>+
> union mlx5_ifc_config_item_type_auto_bits {
> 	struct mlx5_ifc_configuration_item_type_class_global_bits
> 				configuration_item_type_class_global;
>+	struct mlx5_ifc_configuration_item_type_class_per_host_pf_bits
>+				configuration_item_type_class_per_host_pf;
> 	u8 reserved_at_0[0x20];
> };
> 
>@@ -45,6 +58,45 @@ struct mlx5_ifc_mnvda_reg_bits {
> 	u8         configuration_item_data[64][0x20];
> };
> 
>+struct mlx5_ifc_nv_global_pci_conf_bits {
>+	u8         sriov_valid[0x1];
>+	u8         reserved_at_1[0x10];
>+	u8         per_pf_total_vf[0x1];
>+	u8         reserved_at_12[0xe];
>+
>+	u8         sriov_en[0x1];
>+	u8         reserved_at_21[0xf];
>+	u8         total_vfs[0x10];
>+
>+	u8         reserved_at_40[0x20];
>+};
>+
>+struct mlx5_ifc_nv_global_pci_cap_bits {
>+	u8         max_vfs_per_pf_valid[0x1];
>+	u8         reserved_at_1[0x13];
>+	u8         per_pf_total_vf_supported[0x1];
>+	u8         reserved_at_15[0xb];
>+
>+	u8         sriov_support[0x1];
>+	u8         reserved_at_21[0xf];
>+	u8         max_vfs_per_pf[0x10];
>+
>+	u8         reserved_at_40[0x60];
>+};
>+
>+struct mlx5_ifc_nv_pf_pci_conf_bits {
>+	u8         reserved_at_0[0x9];
>+	u8         pf_total_vf_en[0x1];
>+	u8         reserved_at_a[0x16];
>+
>+	u8         reserved_at_20[0x20];
>+
>+	u8         reserved_at_40[0x10];
>+	u8         total_vf[0x10];
>+
>+	u8         reserved_at_60[0x20];
>+};
>+
> struct mlx5_ifc_nv_sw_offload_conf_bits {
> 	u8         ip_over_vxlan_port[0x10];
> 	u8         tunnel_ecn_copy_offload_disable[0x1];
>@@ -206,7 +258,139 @@ static int mlx5_nv_param_devlink_cqe_compress_set(struct devlink *devlink, u32 i
> 	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
> }
> 
>+static int
>+mlx5_nv_param_read_global_pci_conf(struct mlx5_core_dev *dev, void *mnvda, size_t len)
>+{
>+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, type_class, 0);
>+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, parameter_index,
>+				  MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CONF);
>+	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_global_pci_conf);
>+
>+	return mlx5_nv_param_read(dev, mnvda, len);
>+}
>+
>+static int
>+mlx5_nv_param_read_global_pci_cap(struct mlx5_core_dev *dev, void *mnvda, size_t len)
>+{
>+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, type_class, 0);
>+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, parameter_index,
>+				  MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CAP);
>+	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_global_pci_cap);
>+
>+	return mlx5_nv_param_read(dev, mnvda, len);
>+}
>+
>+static int
>+mlx5_nv_param_read_per_host_pf_conf(struct mlx5_core_dev *dev, void *mnvda, size_t len)
>+{
>+	MLX5_SET_CONFIG_ITEM_TYPE(per_host_pf, mnvda, type_class, 3);
>+	MLX5_SET_CONFIG_ITEM_TYPE(per_host_pf, mnvda, parameter_index,
>+				  MLX5_CLASS_3_CTRL_ID_NV_PF_PCI_CONF);
>+	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_pf_pci_conf);
>+
>+	return mlx5_nv_param_read(dev, mnvda, len);
>+}
>+
>+static int mlx5_devlink_enable_sriov_get(struct devlink *devlink, u32 id,
>+					 struct devlink_param_gset_ctx *ctx)
>+{
>+	struct mlx5_core_dev *dev = devlink_priv(devlink);
>+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
>+	void *data;
>+	int err;
>+
>+	err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
>+	if (err)
>+		return err;
>+
>+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>+	if (!MLX5_GET(nv_global_pci_cap, data, sriov_support)) {
>+		ctx->val.vbool = false;
>+		return 0;
>+	}
>+
>+	memset(mnvda, 0, sizeof(mnvda));
>+	err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
>+	if (err)
>+		return err;
>+
>+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>+	if (!MLX5_GET(nv_global_pci_conf, data, per_pf_total_vf)) {
>+		ctx->val.vbool = MLX5_GET(nv_global_pci_conf, data, sriov_en);
>+		return 0;
>+	}
>+
>+	/* SRIOV is per PF */
>+	memset(mnvda, 0, sizeof(mnvda));
>+	err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
>+	if (err)
>+		return err;
>+
>+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>+	ctx->val.vbool = MLX5_GET(nv_pf_pci_conf, data, pf_total_vf_en);
>+	return 0;
>+}
>+
>+static int mlx5_devlink_enable_sriov_set(struct devlink *devlink, u32 id,
>+					 struct devlink_param_gset_ctx *ctx,
>+					 struct netlink_ext_ack *extack)
>+{
>+	struct mlx5_core_dev *dev = devlink_priv(devlink);
>+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
>+	bool per_pf_support;
>+	void *cap, *data;
>+	int err;
>+
>+	err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
>+	if (err) {
>+		NL_SET_ERR_MSG_MOD(extack, "Failed to read global PCI capability");
>+		return err;
>+	}
>+
>+	cap = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>+	per_pf_support = MLX5_GET(nv_global_pci_cap, cap, per_pf_total_vf_supported);
>+
>+	if (!MLX5_GET(nv_global_pci_cap, cap, sriov_support)) {
>+		NL_SET_ERR_MSG_MOD(extack, "Not configurable on this device");
>+		return -EOPNOTSUPP;
>+	}
>+
>+	memset(mnvda, 0, sizeof(mnvda));
>+	err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
>+	if (err) {
>+		NL_SET_ERR_MSG_MOD(extack, "Unable to read global PCI configuration");
>+		return err;
>+	}
>+
>+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>+	MLX5_SET(nv_global_pci_conf, data, sriov_valid, 1);
>+	MLX5_SET(nv_global_pci_conf, data, sriov_en, ctx->val.vbool);
>+	MLX5_SET(nv_global_pci_conf, data, per_pf_total_vf, per_pf_support);
>+
>+	err = mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
>+	if (err) {
>+		NL_SET_ERR_MSG_MOD(extack, "Unable to write global PCI configuration");
>+		return err;
>+	}
>+
>+	if (!per_pf_support)


Hmm, given the discussion we have in parallel about some shared-PF
devlink instance, perhaps it would be good idea to allow only per-pf
configuration here for now and let the "global" per-device configuration
knob to be attached on the shared-PF devlink, when/if it lands. What do
you think?


>+		return 0;
>+
>+	/* SRIOV is per PF */
>+	memset(mnvda, 0, sizeof(mnvda));
>+	err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
>+	if (err) {
>+		NL_SET_ERR_MSG_MOD(extack, "Unable to read per host PF configuration");
>+		return err;
>+	}
>+	MLX5_SET(nv_pf_pci_conf, data, pf_total_vf_en, ctx->val.vbool);
>+	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
>+}
>+
> static const struct devlink_param mlx5_nv_param_devlink_params[] = {
>+	DEVLINK_PARAM_GENERIC(ENABLE_SRIOV, BIT(DEVLINK_PARAM_CMODE_PERMANENT),
>+			      mlx5_devlink_enable_sriov_get,
>+			      mlx5_devlink_enable_sriov_set, NULL),
> 	DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_ID_CQE_COMPRESSION_TYPE,
> 			     "cqe_compress_type", DEVLINK_PARAM_TYPE_STRING,
> 			     BIT(DEVLINK_PARAM_CMODE_PERMANENT),
>-- 
>2.48.1
>
>

Re: [PATCH net-next 09/14] devlink: Implement set netlink command for port params

[Jiri Pirko]

Fri, Feb 28, 2025 at 03:12:22AM +0100, saeed@kernel.org wrote:
>From: Saeed Mahameed <saeedm@nvidia.com>
>
>Add missing port-params netlink attributes and policies to devlink's
>spec, reuse existing set_doit of the devlink dev params.
>
>This implements:
>  devlink port param set <device>/<port> name <param_name> value <val> \
>               cmode <cmode>
>
>Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

Reviewed-by: Jiri Pirko <jiri@nvidia.com>

Re: [PATCH net-next 10/14] devlink: Add 'keep_link_up' generic devlink device param

[Jiri Pirko]

Fri, Feb 28, 2025 at 03:12:23AM +0100, saeed@kernel.org wrote:
>From: Saeed Mahameed <saeedm@nvidia.com>
>
>Devices that support this in permanent mode will be requested to keep the
>port link up even when driver is not loaded, netdev carrier state won't
>affect the physical port link state.
>
>This is useful for when the link is needed to access onboard management
>such as BMC, even if the host driver isn't loaded.
>
>Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

Reviewed-by: Jiri Pirko <jiri@nvidia.com>

Re: [PATCH net-next 11/14] net/mlx5: Implement devlink keep_link_up port parameter

[Jiri Pirko]

Fri, Feb 28, 2025 at 03:12:24AM +0100, saeed@kernel.org wrote:
>From: Saeed Mahameed <saeedm@nvidia.com>
>
>When set, the NIC keeps the link up as long as the host is not in standby
>mode, even when the driver is not loaded.
>
>When enabled, netdev carrier state won't affect the physical port link state.
>
>This is useful for when the link is needed to access onboard management
>such as BMC, even if the host driver isn't loaded.
>
>Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

Reviewed-by: Jiri Pirko <jiri@nvidia.com>

Re: [PATCH net-next 12/14] devlink: Throw extack messages on param value validation error

[Jiri Pirko]

Fri, Feb 28, 2025 at 03:12:25AM +0100, saeed@kernel.org wrote:
>From: Saeed Mahameed <saeedm@nvidia.com>
>
>Centralize devlink param value data validation in one function and
>fill corresponding extack error messages on validation error.
>
>Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

Reviewed-by: Jiri Pirko <jiri@nvidia.com>

Re: [PATCH net-next 07/14] devlink: Implement port params registration

[Przemek Kitszel]

On 2/28/25 13:28, Jiri Pirko wrote:
> Fri, Feb 28, 2025 at 12:58:38PM +0100, przemyslaw.kitszel@intel.com wrote:
>> On 2/28/25 03:12, Saeed Mahameed wrote:
>>> From: Saeed Mahameed <saeedm@nvidia.com>
>>>
>>> Port params infrastructure is incomplete and needs a bit of plumbing to
>>> support port params commands from netlink.
>>>
>>> Introduce port params registration API, very similar to current devlink
>>> params API, add the params xarray to devlink_port structure and
>>> decouple devlink params registration routines from the devlink
>>> structure.
>>>
>>> Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
>>> Reviewed-by: Jiri Pirko <jiri@nvidia.com>
>>> ---
>>>    include/net/devlink.h |  14 ++++
>>>    net/devlink/param.c   | 150 ++++++++++++++++++++++++++++++++++--------
>>>    net/devlink/port.c    |   3 +
>>>    3 files changed, 140 insertions(+), 27 deletions(-)
>> For me devlink and devlink-port should be really the same, to the point
>> that the only difference is `bool is_port` flag inside of the
>> struct devlink. Then you could put special logic if really desired (to
>> exclude something for port).
> 
> Why? Why other devlink objects shouldn't be the same as well. Then we
> can have one union. Does not make sense to me. The only think dev and
> port is sharing would be params. What else? Totally different beast.

Instead of focusing on differences try to find similarities.

health reporters per port and "toplevel",
just by grepping:
devlink_nl_sb_pool_fill()
devlink_nl_sb_port_pool_fill(),

devlink_region_create()
devlink_port_region_create()

and there is no reason to assume that someone will not want to
extend ports to have devlink resources or other thing

Re: [PATCH net-next 07/14] devlink: Implement port params registration

[Jiri Pirko]

Fri, Feb 28, 2025 at 02:23:00PM +0100, przemyslaw.kitszel@intel.com wrote:
>On 2/28/25 13:28, Jiri Pirko wrote:
>> Fri, Feb 28, 2025 at 12:58:38PM +0100, przemyslaw.kitszel@intel.com wrote:
>> > On 2/28/25 03:12, Saeed Mahameed wrote:
>> > > From: Saeed Mahameed <saeedm@nvidia.com>
>> > > 
>> > > Port params infrastructure is incomplete and needs a bit of plumbing to
>> > > support port params commands from netlink.
>> > > 
>> > > Introduce port params registration API, very similar to current devlink
>> > > params API, add the params xarray to devlink_port structure and
>> > > decouple devlink params registration routines from the devlink
>> > > structure.
>> > > 
>> > > Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
>> > > Reviewed-by: Jiri Pirko <jiri@nvidia.com>
>> > > ---
>> > >    include/net/devlink.h |  14 ++++
>> > >    net/devlink/param.c   | 150 ++++++++++++++++++++++++++++++++++--------
>> > >    net/devlink/port.c    |   3 +
>> > >    3 files changed, 140 insertions(+), 27 deletions(-)
>> > For me devlink and devlink-port should be really the same, to the point
>> > that the only difference is `bool is_port` flag inside of the
>> > struct devlink. Then you could put special logic if really desired (to
>> > exclude something for port).
>> 
>> Why? Why other devlink objects shouldn't be the same as well. Then we
>> can have one union. Does not make sense to me. The only think dev and
>> port is sharing would be params. What else? Totally different beast.
>
>Instead of focusing on differences try to find similarities.
>
>health reporters per port and "toplevel",
>just by grepping:
>devlink_nl_sb_pool_fill()
>devlink_nl_sb_port_pool_fill(),

Sharedbuffer is separate story.

>
>devlink_region_create()
>devlink_port_region_create()

Okay, regions I missed.

>
>and there is no reason to assume that someone will not want to
>extend ports to have devlink resources or other thing

But looks at differences. They are huge.

But perhaps I'm missing the point. What you want to achieve? Just to
reduce API? That is always a tradeoff. I don't think the pros top the
cons here.

Re: [PATCH net-next 04/14] net/mlx5: Implement devlink enable_sriov parameter

[Saeed Mahameed]

On 28 Feb 13:46, Jiri Pirko wrote:
>Fri, Feb 28, 2025 at 03:12:17AM +0100, saeed@kernel.org wrote:
>>From: Vlad Dumitrescu <vdumitrescu@nvidia.com>
>>
>>Example usage:
>>  devlink dev param set pci/0000:01:00.0 name enable_sriov value {true, false} cmode permanent
>>  devlink dev reload pci/0000:01:00.0 action fw_activate
>>  echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove
>>  echo 1 >/sys/bus/pci/rescan
>>  grep ^ /sys/bus/pci/devices/0000:01:00.0/sriov_*
>>
>>Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com>
>>Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
>>---
>> Documentation/networking/devlink/mlx5.rst     |  14 +-
>> .../net/ethernet/mellanox/mlx5/core/devlink.c |   1 +
>> .../mellanox/mlx5/core/lib/nv_param.c         | 184 ++++++++++++++++++
>> 3 files changed, 196 insertions(+), 3 deletions(-)
>>
>>diff --git a/Documentation/networking/devlink/mlx5.rst b/Documentation/networking/devlink/mlx5.rst
>>index 417e5cdcd35d..587e0200c1cd 100644
>>--- a/Documentation/networking/devlink/mlx5.rst
>>+++ b/Documentation/networking/devlink/mlx5.rst
>>@@ -15,23 +15,31 @@ Parameters
>>    * - Name
>>      - Mode
>>      - Validation
>>+     - Notes
>>    * - ``enable_roce``
>>      - driverinit
>>-     - Type: Boolean
>>-
>>-       If the device supports RoCE disablement, RoCE enablement state controls
>>+     - Boolean
>>+     - If the device supports RoCE disablement, RoCE enablement state controls
>>        device support for RoCE capability. Otherwise, the control occurs in the
>>        driver stack. When RoCE is disabled at the driver level, only raw
>>        ethernet QPs are supported.
>>    * - ``io_eq_size``
>>      - driverinit
>>      - The range is between 64 and 4096.
>>+     -
>>    * - ``event_eq_size``
>>      - driverinit
>>      - The range is between 64 and 4096.
>>+     -
>>    * - ``max_macs``
>>      - driverinit
>>      - The range is between 1 and 2^31. Only power of 2 values are supported.
>>+     -
>>+   * - ``enable_sriov``
>>+     - permanent
>>+     - Boolean
>>+     - Applies to each physical function (PF) independently, if the device
>>+       supports it. Otherwise, it applies symmetrically to all PFs.
>>
>> The ``mlx5`` driver also implements the following driver-specific
>> parameters.
>>diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>>index 1f764ae4f4aa..7a702d84f19a 100644
>>--- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>>+++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>>@@ -8,6 +8,7 @@
>> #include "fs_core.h"
>> #include "eswitch.h"
>> #include "esw/qos.h"
>>+#include "lib/nv_param.h"
>> #include "sf/dev/dev.h"
>> #include "sf/sf.h"
>> #include "lib/nv_param.h"
>>diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
>>index 5ab37a88c260..6b63fc110e2d 100644
>>--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
>>+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
>>@@ -5,7 +5,11 @@
>> #include "mlx5_core.h"
>>
>> enum {
>>+	MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CONF               = 0x80,
>>+	MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CAP                = 0x81,
>> 	MLX5_CLASS_0_CTRL_ID_NV_SW_OFFLOAD_CONFIG             = 0x10a,
>>+
>>+	MLX5_CLASS_3_CTRL_ID_NV_PF_PCI_CONF                   = 0x80,
>> };
>>
>> struct mlx5_ifc_configuration_item_type_class_global_bits {
>>@@ -13,9 +17,18 @@ struct mlx5_ifc_configuration_item_type_class_global_bits {
>> 	u8         parameter_index[0x18];
>> };
>>
>>+struct mlx5_ifc_configuration_item_type_class_per_host_pf_bits {
>>+	u8         type_class[0x8];
>>+	u8         pf_index[0x6];
>>+	u8         pci_bus_index[0x8];
>>+	u8         parameter_index[0xa];
>>+};
>>+
>> union mlx5_ifc_config_item_type_auto_bits {
>> 	struct mlx5_ifc_configuration_item_type_class_global_bits
>> 				configuration_item_type_class_global;
>>+	struct mlx5_ifc_configuration_item_type_class_per_host_pf_bits
>>+				configuration_item_type_class_per_host_pf;
>> 	u8 reserved_at_0[0x20];
>> };
>>
>>@@ -45,6 +58,45 @@ struct mlx5_ifc_mnvda_reg_bits {
>> 	u8         configuration_item_data[64][0x20];
>> };
>>
>>+struct mlx5_ifc_nv_global_pci_conf_bits {
>>+	u8         sriov_valid[0x1];
>>+	u8         reserved_at_1[0x10];
>>+	u8         per_pf_total_vf[0x1];
>>+	u8         reserved_at_12[0xe];
>>+
>>+	u8         sriov_en[0x1];
>>+	u8         reserved_at_21[0xf];
>>+	u8         total_vfs[0x10];
>>+
>>+	u8         reserved_at_40[0x20];
>>+};
>>+
>>+struct mlx5_ifc_nv_global_pci_cap_bits {
>>+	u8         max_vfs_per_pf_valid[0x1];
>>+	u8         reserved_at_1[0x13];
>>+	u8         per_pf_total_vf_supported[0x1];
>>+	u8         reserved_at_15[0xb];
>>+
>>+	u8         sriov_support[0x1];
>>+	u8         reserved_at_21[0xf];
>>+	u8         max_vfs_per_pf[0x10];
>>+
>>+	u8         reserved_at_40[0x60];
>>+};
>>+
>>+struct mlx5_ifc_nv_pf_pci_conf_bits {
>>+	u8         reserved_at_0[0x9];
>>+	u8         pf_total_vf_en[0x1];
>>+	u8         reserved_at_a[0x16];
>>+
>>+	u8         reserved_at_20[0x20];
>>+
>>+	u8         reserved_at_40[0x10];
>>+	u8         total_vf[0x10];
>>+
>>+	u8         reserved_at_60[0x20];
>>+};
>>+
>> struct mlx5_ifc_nv_sw_offload_conf_bits {
>> 	u8         ip_over_vxlan_port[0x10];
>> 	u8         tunnel_ecn_copy_offload_disable[0x1];
>>@@ -206,7 +258,139 @@ static int mlx5_nv_param_devlink_cqe_compress_set(struct devlink *devlink, u32 i
>> 	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
>> }
>>
>>+static int
>>+mlx5_nv_param_read_global_pci_conf(struct mlx5_core_dev *dev, void *mnvda, size_t len)
>>+{
>>+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, type_class, 0);
>>+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, parameter_index,
>>+				  MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CONF);
>>+	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_global_pci_conf);
>>+
>>+	return mlx5_nv_param_read(dev, mnvda, len);
>>+}
>>+
>>+static int
>>+mlx5_nv_param_read_global_pci_cap(struct mlx5_core_dev *dev, void *mnvda, size_t len)
>>+{
>>+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, type_class, 0);
>>+	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, parameter_index,
>>+				  MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CAP);
>>+	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_global_pci_cap);
>>+
>>+	return mlx5_nv_param_read(dev, mnvda, len);
>>+}
>>+
>>+static int
>>+mlx5_nv_param_read_per_host_pf_conf(struct mlx5_core_dev *dev, void *mnvda, size_t len)
>>+{
>>+	MLX5_SET_CONFIG_ITEM_TYPE(per_host_pf, mnvda, type_class, 3);
>>+	MLX5_SET_CONFIG_ITEM_TYPE(per_host_pf, mnvda, parameter_index,
>>+				  MLX5_CLASS_3_CTRL_ID_NV_PF_PCI_CONF);
>>+	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_pf_pci_conf);
>>+
>>+	return mlx5_nv_param_read(dev, mnvda, len);
>>+}
>>+
>>+static int mlx5_devlink_enable_sriov_get(struct devlink *devlink, u32 id,
>>+					 struct devlink_param_gset_ctx *ctx)
>>+{
>>+	struct mlx5_core_dev *dev = devlink_priv(devlink);
>>+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
>>+	void *data;
>>+	int err;
>>+
>>+	err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
>>+	if (err)
>>+		return err;
>>+
>>+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>>+	if (!MLX5_GET(nv_global_pci_cap, data, sriov_support)) {
>>+		ctx->val.vbool = false;
>>+		return 0;
>>+	}
>>+
>>+	memset(mnvda, 0, sizeof(mnvda));
>>+	err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
>>+	if (err)
>>+		return err;
>>+
>>+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>>+	if (!MLX5_GET(nv_global_pci_conf, data, per_pf_total_vf)) {
>>+		ctx->val.vbool = MLX5_GET(nv_global_pci_conf, data, sriov_en);
>>+		return 0;
>>+	}
>>+
>>+	/* SRIOV is per PF */
>>+	memset(mnvda, 0, sizeof(mnvda));
>>+	err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
>>+	if (err)
>>+		return err;
>>+
>>+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>>+	ctx->val.vbool = MLX5_GET(nv_pf_pci_conf, data, pf_total_vf_en);
>>+	return 0;
>>+}
>>+
>>+static int mlx5_devlink_enable_sriov_set(struct devlink *devlink, u32 id,
>>+					 struct devlink_param_gset_ctx *ctx,
>>+					 struct netlink_ext_ack *extack)
>>+{
>>+	struct mlx5_core_dev *dev = devlink_priv(devlink);
>>+	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
>>+	bool per_pf_support;
>>+	void *cap, *data;
>>+	int err;
>>+
>>+	err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
>>+	if (err) {
>>+		NL_SET_ERR_MSG_MOD(extack, "Failed to read global PCI capability");
>>+		return err;
>>+	}
>>+
>>+	cap = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>>+	per_pf_support = MLX5_GET(nv_global_pci_cap, cap, per_pf_total_vf_supported);
>>+
>>+	if (!MLX5_GET(nv_global_pci_cap, cap, sriov_support)) {
>>+		NL_SET_ERR_MSG_MOD(extack, "Not configurable on this device");
>>+		return -EOPNOTSUPP;
>>+	}
>>+
>>+	memset(mnvda, 0, sizeof(mnvda));
>>+	err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
>>+	if (err) {
>>+		NL_SET_ERR_MSG_MOD(extack, "Unable to read global PCI configuration");
>>+		return err;
>>+	}
>>+
>>+	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>>+	MLX5_SET(nv_global_pci_conf, data, sriov_valid, 1);
>>+	MLX5_SET(nv_global_pci_conf, data, sriov_en, ctx->val.vbool);
>>+	MLX5_SET(nv_global_pci_conf, data, per_pf_total_vf, per_pf_support);
>>+
>>+	err = mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
>>+	if (err) {
>>+		NL_SET_ERR_MSG_MOD(extack, "Unable to write global PCI configuration");
>>+		return err;
>>+	}
>>+
>>+	if (!per_pf_support)
>
>
>Hmm, given the discussion we have in parallel about some shared-PF
>devlink instance, perhaps it would be good idea to allow only per-pf
>configuration here for now and let the "global" per-device configuration
>knob to be attached on the shared-PF devlink, when/if it lands. What do
>you think?

Do we have an RFC? can you point me to it? 

I am just worried about the conflicts between per-pf and global configs
this will introduce, currently it is driver best effort, after that we
might want to pick one direction, global vs per-pf if it will be separate
knobs, and we probably will go with per-pf. Most CX devices support both
modes and it is up to the driver to chose. So why do both global and
per-pf when you can almost always do per-pf?

>
>
>>+		return 0;
>>+
>>+	/* SRIOV is per PF */
>>+	memset(mnvda, 0, sizeof(mnvda));
>>+	err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
>>+	if (err) {
>>+		NL_SET_ERR_MSG_MOD(extack, "Unable to read per host PF configuration");
>>+		return err;
>>+	}
>>+	MLX5_SET(nv_pf_pci_conf, data, pf_total_vf_en, ctx->val.vbool);
>>+	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
>>+}
>>+
>> static const struct devlink_param mlx5_nv_param_devlink_params[] = {
>>+	DEVLINK_PARAM_GENERIC(ENABLE_SRIOV, BIT(DEVLINK_PARAM_CMODE_PERMANENT),
>>+			      mlx5_devlink_enable_sriov_get,
>>+			      mlx5_devlink_enable_sriov_set, NULL),
>> 	DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_ID_CQE_COMPRESSION_TYPE,
>> 			     "cqe_compress_type", DEVLINK_PARAM_TYPE_STRING,
>> 			     BIT(DEVLINK_PARAM_CMODE_PERMANENT),
>>--
>>2.48.1
>>
>>

Re: [PATCH net-next 04/14] net/mlx5: Implement devlink enable_sriov parameter

[kernel test robot]

Hi Saeed,

kernel test robot noticed the following build warnings:

[auto build test WARNING on net-next/main]

url:    https://github.com/intel-lab-lkp/linux/commits/Saeed-Mahameed/devlink-define-enum-for-attr-types-of-dynamic-attributes/20250228-101818
base:   net-next/main
patch link:    https://lore.kernel.org/r/20250228021227.871993-5-saeed%40kernel.org
patch subject: [PATCH net-next 04/14] net/mlx5: Implement devlink enable_sriov parameter
compiler: clang version 19.1.7 (https://github.com/llvm/llvm-project cd708029e0b2869e80abe31ddb175f7c35361f90)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202503030926.KNBxmdVW-lkp@intel.com/

includecheck warnings: (new ones prefixed by >>)
>> drivers/net/ethernet/mellanox/mlx5/core/devlink.c: lib/nv_param.h is included more than once.

vim +11 drivers/net/ethernet/mellanox/mlx5/core/devlink.c

     5	
     6	#include "mlx5_core.h"
     7	#include "fw_reset.h"
     8	#include "fs_core.h"
     9	#include "eswitch.h"
    10	#include "esw/qos.h"
  > 11	#include "lib/nv_param.h"
    12	#include "sf/dev/dev.h"
    13	#include "sf/sf.h"
  > 14	#include "lib/nv_param.h"
    15	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH net-next 12/14] devlink: Throw extack messages on param value validation error

[Dan Carpenter]

Hi Saeed,

kernel test robot noticed the following build warnings:

url:    https://github.com/intel-lab-lkp/linux/commits/Saeed-Mahameed/devlink-define-enum-for-attr-types-of-dynamic-attributes/20250228-101818
base:   net-next/main
patch link:    https://lore.kernel.org/r/20250228021227.871993-13-saeed%40kernel.org
patch subject: [PATCH net-next 12/14] devlink: Throw extack messages on param value validation error
config: sparc-randconfig-r071-20250302 (https://download.01.org/0day-ci/archive/20250302/202503022006.eOu80RMF-lkp@intel.com/config)
compiler: sparc-linux-gcc (GCC) 14.2.0

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Reported-by: Dan Carpenter <dan.carpenter@linaro.org>
| Closes: https://lore.kernel.org/r/202503022006.eOu80RMF-lkp@intel.com/

smatch warnings:
net/devlink/param.c:498 devlink_param_value_validate() error: we previously assumed 'param_data' could be null (see line 475)

vim +/param_data +498 net/devlink/param.c

ca557d9df319fed Saeed Mahameed 2025-02-27  461  static int
ca557d9df319fed Saeed Mahameed 2025-02-27  462  devlink_param_value_validate(struct genl_info *info,
ca557d9df319fed Saeed Mahameed 2025-02-27  463  			     enum devlink_param_type type)
ca557d9df319fed Saeed Mahameed 2025-02-27  464  {
ca557d9df319fed Saeed Mahameed 2025-02-27  465  	struct netlink_ext_ack *extack = info->extack;
ca557d9df319fed Saeed Mahameed 2025-02-27  466  	struct nlattr *param_data;
ca557d9df319fed Saeed Mahameed 2025-02-27  467  	int len = 0;
ca557d9df319fed Saeed Mahameed 2025-02-27  468  
ca557d9df319fed Saeed Mahameed 2025-02-27  469  	if (type != DEVLINK_PARAM_TYPE_BOOL &&
ca557d9df319fed Saeed Mahameed 2025-02-27  470  	    GENL_REQ_ATTR_CHECK(info, DEVLINK_ATTR_PARAM_VALUE_DATA))
ca557d9df319fed Saeed Mahameed 2025-02-27  471  		return -EINVAL;
ca557d9df319fed Saeed Mahameed 2025-02-27  472  
ca557d9df319fed Saeed Mahameed 2025-02-27  473  	param_data = info->attrs[DEVLINK_ATTR_PARAM_VALUE_DATA];
ca557d9df319fed Saeed Mahameed 2025-02-27  474  
ca557d9df319fed Saeed Mahameed 2025-02-27 @475  	if (param_data)

This assumes param_data can be NULL

ca557d9df319fed Saeed Mahameed 2025-02-27  476  		len = nla_len(param_data);
ca557d9df319fed Saeed Mahameed 2025-02-27  477  
ca557d9df319fed Saeed Mahameed 2025-02-27  478  	switch (type) {
ca557d9df319fed Saeed Mahameed 2025-02-27  479  	case DEVLINK_PARAM_TYPE_U8:
ca557d9df319fed Saeed Mahameed 2025-02-27  480  		if (len == sizeof(u8))
ca557d9df319fed Saeed Mahameed 2025-02-27  481  			return 0;
ca557d9df319fed Saeed Mahameed 2025-02-27  482  		NL_SET_ERR_MSG_FMT_MOD(extack,
ca557d9df319fed Saeed Mahameed 2025-02-27  483  				       "Expected uint8, got %d bytes", len);
ca557d9df319fed Saeed Mahameed 2025-02-27  484  		break;
ca557d9df319fed Saeed Mahameed 2025-02-27  485  	case DEVLINK_PARAM_TYPE_U16:
ca557d9df319fed Saeed Mahameed 2025-02-27  486  		if (len == sizeof(u16))
ca557d9df319fed Saeed Mahameed 2025-02-27  487  			return 0;
ca557d9df319fed Saeed Mahameed 2025-02-27  488  		NL_SET_ERR_MSG_FMT_MOD(extack,
ca557d9df319fed Saeed Mahameed 2025-02-27  489  				       "Expected uint16, got %d bytes", len);
ca557d9df319fed Saeed Mahameed 2025-02-27  490  		break;
ca557d9df319fed Saeed Mahameed 2025-02-27  491  	case DEVLINK_PARAM_TYPE_U32:
ca557d9df319fed Saeed Mahameed 2025-02-27  492  		if (len == sizeof(u32))
ca557d9df319fed Saeed Mahameed 2025-02-27  493  			return 0;
ca557d9df319fed Saeed Mahameed 2025-02-27  494  		NL_SET_ERR_MSG_FMT_MOD(extack,
ca557d9df319fed Saeed Mahameed 2025-02-27  495  				       "Expected uint32, got %d bytes", len);
ca557d9df319fed Saeed Mahameed 2025-02-27  496  		break;
ca557d9df319fed Saeed Mahameed 2025-02-27  497  	case DEVLINK_PARAM_TYPE_STRING:
ca557d9df319fed Saeed Mahameed 2025-02-27 @498  		len = strnlen(nla_data(param_data), nla_len(param_data));
                                                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Unchecked dereferences

ca557d9df319fed Saeed Mahameed 2025-02-27  499  
ca557d9df319fed Saeed Mahameed 2025-02-27  500  		if (len < nla_len(param_data) &&
ca557d9df319fed Saeed Mahameed 2025-02-27  501  		    len < __DEVLINK_PARAM_MAX_STRING_VALUE)
ca557d9df319fed Saeed Mahameed 2025-02-27  502  			return 0;
ca557d9df319fed Saeed Mahameed 2025-02-27  503  		NL_SET_ERR_MSG_MOD(extack, "String too long");
ca557d9df319fed Saeed Mahameed 2025-02-27  504  		break;
ca557d9df319fed Saeed Mahameed 2025-02-27  505  	case DEVLINK_PARAM_TYPE_BOOL:
ca557d9df319fed Saeed Mahameed 2025-02-27  506  		if (!len)
ca557d9df319fed Saeed Mahameed 2025-02-27  507  			return 0;
ca557d9df319fed Saeed Mahameed 2025-02-27  508  		NL_SET_ERR_MSG_MOD(extack, "Expected flag, got data");
ca557d9df319fed Saeed Mahameed 2025-02-27  509  		break;
ca557d9df319fed Saeed Mahameed 2025-02-27  510  	default:
ca557d9df319fed Saeed Mahameed 2025-02-27  511  		NL_SET_ERR_MSG_FMT_MOD(extack,
ca557d9df319fed Saeed Mahameed 2025-02-27  512  				       "Not supported value type %d", type);
ca557d9df319fed Saeed Mahameed 2025-02-27  513  		break;
ca557d9df319fed Saeed Mahameed 2025-02-27  514  	}
ca557d9df319fed Saeed Mahameed 2025-02-27  515  	return -EINVAL;
ca557d9df319fed Saeed Mahameed 2025-02-27  516  }

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH net-next 04/14] net/mlx5: Implement devlink enable_sriov parameter

[Jiri Pirko]

Fri, Feb 28, 2025 at 07:19:19PM +0100, saeedm@nvidia.com wrote:
>On 28 Feb 13:46, Jiri Pirko wrote:
>> Fri, Feb 28, 2025 at 03:12:17AM +0100, saeed@kernel.org wrote:
>> > From: Vlad Dumitrescu <vdumitrescu@nvidia.com>
>> > 
>> > Example usage:
>> >  devlink dev param set pci/0000:01:00.0 name enable_sriov value {true, false} cmode permanent
>> >  devlink dev reload pci/0000:01:00.0 action fw_activate
>> >  echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove
>> >  echo 1 >/sys/bus/pci/rescan
>> >  grep ^ /sys/bus/pci/devices/0000:01:00.0/sriov_*
>> > 
>> > Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com>
>> > Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
>> > ---
>> > Documentation/networking/devlink/mlx5.rst     |  14 +-
>> > .../net/ethernet/mellanox/mlx5/core/devlink.c |   1 +
>> > .../mellanox/mlx5/core/lib/nv_param.c         | 184 ++++++++++++++++++
>> > 3 files changed, 196 insertions(+), 3 deletions(-)
>> > 
>> > diff --git a/Documentation/networking/devlink/mlx5.rst b/Documentation/networking/devlink/mlx5.rst
>> > index 417e5cdcd35d..587e0200c1cd 100644
>> > --- a/Documentation/networking/devlink/mlx5.rst
>> > +++ b/Documentation/networking/devlink/mlx5.rst
>> > @@ -15,23 +15,31 @@ Parameters
>> >    * - Name
>> >      - Mode
>> >      - Validation
>> > +     - Notes
>> >    * - ``enable_roce``
>> >      - driverinit
>> > -     - Type: Boolean
>> > -
>> > -       If the device supports RoCE disablement, RoCE enablement state controls
>> > +     - Boolean
>> > +     - If the device supports RoCE disablement, RoCE enablement state controls
>> >        device support for RoCE capability. Otherwise, the control occurs in the
>> >        driver stack. When RoCE is disabled at the driver level, only raw
>> >        ethernet QPs are supported.
>> >    * - ``io_eq_size``
>> >      - driverinit
>> >      - The range is between 64 and 4096.
>> > +     -
>> >    * - ``event_eq_size``
>> >      - driverinit
>> >      - The range is between 64 and 4096.
>> > +     -
>> >    * - ``max_macs``
>> >      - driverinit
>> >      - The range is between 1 and 2^31. Only power of 2 values are supported.
>> > +     -
>> > +   * - ``enable_sriov``
>> > +     - permanent
>> > +     - Boolean
>> > +     - Applies to each physical function (PF) independently, if the device
>> > +       supports it. Otherwise, it applies symmetrically to all PFs.
>> > 
>> > The ``mlx5`` driver also implements the following driver-specific
>> > parameters.
>> > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>> > index 1f764ae4f4aa..7a702d84f19a 100644
>> > --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>> > +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>> > @@ -8,6 +8,7 @@
>> > #include "fs_core.h"
>> > #include "eswitch.h"
>> > #include "esw/qos.h"
>> > +#include "lib/nv_param.h"
>> > #include "sf/dev/dev.h"
>> > #include "sf/sf.h"
>> > #include "lib/nv_param.h"
>> > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
>> > index 5ab37a88c260..6b63fc110e2d 100644
>> > --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
>> > +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
>> > @@ -5,7 +5,11 @@
>> > #include "mlx5_core.h"
>> > 
>> > enum {
>> > +	MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CONF               = 0x80,
>> > +	MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CAP                = 0x81,
>> > 	MLX5_CLASS_0_CTRL_ID_NV_SW_OFFLOAD_CONFIG             = 0x10a,
>> > +
>> > +	MLX5_CLASS_3_CTRL_ID_NV_PF_PCI_CONF                   = 0x80,
>> > };
>> > 
>> > struct mlx5_ifc_configuration_item_type_class_global_bits {
>> > @@ -13,9 +17,18 @@ struct mlx5_ifc_configuration_item_type_class_global_bits {
>> > 	u8         parameter_index[0x18];
>> > };
>> > 
>> > +struct mlx5_ifc_configuration_item_type_class_per_host_pf_bits {
>> > +	u8         type_class[0x8];
>> > +	u8         pf_index[0x6];
>> > +	u8         pci_bus_index[0x8];
>> > +	u8         parameter_index[0xa];
>> > +};
>> > +
>> > union mlx5_ifc_config_item_type_auto_bits {
>> > 	struct mlx5_ifc_configuration_item_type_class_global_bits
>> > 				configuration_item_type_class_global;
>> > +	struct mlx5_ifc_configuration_item_type_class_per_host_pf_bits
>> > +				configuration_item_type_class_per_host_pf;
>> > 	u8 reserved_at_0[0x20];
>> > };
>> > 
>> > @@ -45,6 +58,45 @@ struct mlx5_ifc_mnvda_reg_bits {
>> > 	u8         configuration_item_data[64][0x20];
>> > };
>> > 
>> > +struct mlx5_ifc_nv_global_pci_conf_bits {
>> > +	u8         sriov_valid[0x1];
>> > +	u8         reserved_at_1[0x10];
>> > +	u8         per_pf_total_vf[0x1];
>> > +	u8         reserved_at_12[0xe];
>> > +
>> > +	u8         sriov_en[0x1];
>> > +	u8         reserved_at_21[0xf];
>> > +	u8         total_vfs[0x10];
>> > +
>> > +	u8         reserved_at_40[0x20];
>> > +};
>> > +
>> > +struct mlx5_ifc_nv_global_pci_cap_bits {
>> > +	u8         max_vfs_per_pf_valid[0x1];
>> > +	u8         reserved_at_1[0x13];
>> > +	u8         per_pf_total_vf_supported[0x1];
>> > +	u8         reserved_at_15[0xb];
>> > +
>> > +	u8         sriov_support[0x1];
>> > +	u8         reserved_at_21[0xf];
>> > +	u8         max_vfs_per_pf[0x10];
>> > +
>> > +	u8         reserved_at_40[0x60];
>> > +};
>> > +
>> > +struct mlx5_ifc_nv_pf_pci_conf_bits {
>> > +	u8         reserved_at_0[0x9];
>> > +	u8         pf_total_vf_en[0x1];
>> > +	u8         reserved_at_a[0x16];
>> > +
>> > +	u8         reserved_at_20[0x20];
>> > +
>> > +	u8         reserved_at_40[0x10];
>> > +	u8         total_vf[0x10];
>> > +
>> > +	u8         reserved_at_60[0x20];
>> > +};
>> > +
>> > struct mlx5_ifc_nv_sw_offload_conf_bits {
>> > 	u8         ip_over_vxlan_port[0x10];
>> > 	u8         tunnel_ecn_copy_offload_disable[0x1];
>> > @@ -206,7 +258,139 @@ static int mlx5_nv_param_devlink_cqe_compress_set(struct devlink *devlink, u32 i
>> > 	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
>> > }
>> > 
>> > +static int
>> > +mlx5_nv_param_read_global_pci_conf(struct mlx5_core_dev *dev, void *mnvda, size_t len)
>> > +{
>> > +	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, type_class, 0);
>> > +	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, parameter_index,
>> > +				  MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CONF);
>> > +	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_global_pci_conf);
>> > +
>> > +	return mlx5_nv_param_read(dev, mnvda, len);
>> > +}
>> > +
>> > +static int
>> > +mlx5_nv_param_read_global_pci_cap(struct mlx5_core_dev *dev, void *mnvda, size_t len)
>> > +{
>> > +	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, type_class, 0);
>> > +	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, parameter_index,
>> > +				  MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CAP);
>> > +	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_global_pci_cap);
>> > +
>> > +	return mlx5_nv_param_read(dev, mnvda, len);
>> > +}
>> > +
>> > +static int
>> > +mlx5_nv_param_read_per_host_pf_conf(struct mlx5_core_dev *dev, void *mnvda, size_t len)
>> > +{
>> > +	MLX5_SET_CONFIG_ITEM_TYPE(per_host_pf, mnvda, type_class, 3);
>> > +	MLX5_SET_CONFIG_ITEM_TYPE(per_host_pf, mnvda, parameter_index,
>> > +				  MLX5_CLASS_3_CTRL_ID_NV_PF_PCI_CONF);
>> > +	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_pf_pci_conf);
>> > +
>> > +	return mlx5_nv_param_read(dev, mnvda, len);
>> > +}
>> > +
>> > +static int mlx5_devlink_enable_sriov_get(struct devlink *devlink, u32 id,
>> > +					 struct devlink_param_gset_ctx *ctx)
>> > +{
>> > +	struct mlx5_core_dev *dev = devlink_priv(devlink);
>> > +	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
>> > +	void *data;
>> > +	int err;
>> > +
>> > +	err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
>> > +	if (err)
>> > +		return err;
>> > +
>> > +	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>> > +	if (!MLX5_GET(nv_global_pci_cap, data, sriov_support)) {
>> > +		ctx->val.vbool = false;
>> > +		return 0;
>> > +	}
>> > +
>> > +	memset(mnvda, 0, sizeof(mnvda));
>> > +	err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
>> > +	if (err)
>> > +		return err;
>> > +
>> > +	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>> > +	if (!MLX5_GET(nv_global_pci_conf, data, per_pf_total_vf)) {
>> > +		ctx->val.vbool = MLX5_GET(nv_global_pci_conf, data, sriov_en);
>> > +		return 0;
>> > +	}
>> > +
>> > +	/* SRIOV is per PF */
>> > +	memset(mnvda, 0, sizeof(mnvda));
>> > +	err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
>> > +	if (err)
>> > +		return err;
>> > +
>> > +	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>> > +	ctx->val.vbool = MLX5_GET(nv_pf_pci_conf, data, pf_total_vf_en);
>> > +	return 0;
>> > +}
>> > +
>> > +static int mlx5_devlink_enable_sriov_set(struct devlink *devlink, u32 id,
>> > +					 struct devlink_param_gset_ctx *ctx,
>> > +					 struct netlink_ext_ack *extack)
>> > +{
>> > +	struct mlx5_core_dev *dev = devlink_priv(devlink);
>> > +	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
>> > +	bool per_pf_support;
>> > +	void *cap, *data;
>> > +	int err;
>> > +
>> > +	err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
>> > +	if (err) {
>> > +		NL_SET_ERR_MSG_MOD(extack, "Failed to read global PCI capability");
>> > +		return err;
>> > +	}
>> > +
>> > +	cap = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>> > +	per_pf_support = MLX5_GET(nv_global_pci_cap, cap, per_pf_total_vf_supported);
>> > +
>> > +	if (!MLX5_GET(nv_global_pci_cap, cap, sriov_support)) {
>> > +		NL_SET_ERR_MSG_MOD(extack, "Not configurable on this device");
>> > +		return -EOPNOTSUPP;
>> > +	}
>> > +
>> > +	memset(mnvda, 0, sizeof(mnvda));
>> > +	err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
>> > +	if (err) {
>> > +		NL_SET_ERR_MSG_MOD(extack, "Unable to read global PCI configuration");
>> > +		return err;
>> > +	}
>> > +
>> > +	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
>> > +	MLX5_SET(nv_global_pci_conf, data, sriov_valid, 1);
>> > +	MLX5_SET(nv_global_pci_conf, data, sriov_en, ctx->val.vbool);
>> > +	MLX5_SET(nv_global_pci_conf, data, per_pf_total_vf, per_pf_support);
>> > +
>> > +	err = mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
>> > +	if (err) {
>> > +		NL_SET_ERR_MSG_MOD(extack, "Unable to write global PCI configuration");
>> > +		return err;
>> > +	}
>> > +
>> > +	if (!per_pf_support)
>> 
>> 
>> Hmm, given the discussion we have in parallel about some shared-PF
>> devlink instance, perhaps it would be good idea to allow only per-pf
>> configuration here for now and let the "global" per-device configuration
>> knob to be attached on the shared-PF devlink, when/if it lands. What do
>> you think?
>
>Do we have an RFC? can you point me to it?

https://lore.kernel.org/all/20250219164410.35665-2-przemyslaw.kitszel@intel.com/


>
>I am just worried about the conflicts between per-pf and global configs
>this will introduce, currently it is driver best effort, after that we
>might want to pick one direction, global vs per-pf if it will be separate
>knobs, and we probably will go with per-pf. Most CX devices support both
>modes and it is up to the driver to chose. So why do both global and
>per-pf when you can almost always do per-pf?

I was thinking that is device supports per-pf, only per-pf knob will be
present. And if not, only "global" know will be present on the shared
devlink instance.

Okay. So let's implement per-pf only now. And if the global is needed in
the future for older devices and we'll have a devlink instance to hang
in on, let's add it later. Makes sense?

>
>> 
>> 
>> > +		return 0;
>> > +
>> > +	/* SRIOV is per PF */
>> > +	memset(mnvda, 0, sizeof(mnvda));
>> > +	err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
>> > +	if (err) {
>> > +		NL_SET_ERR_MSG_MOD(extack, "Unable to read per host PF configuration");
>> > +		return err;
>> > +	}
>> > +	MLX5_SET(nv_pf_pci_conf, data, pf_total_vf_en, ctx->val.vbool);
>> > +	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
>> > +}
>> > +
>> > static const struct devlink_param mlx5_nv_param_devlink_params[] = {
>> > +	DEVLINK_PARAM_GENERIC(ENABLE_SRIOV, BIT(DEVLINK_PARAM_CMODE_PERMANENT),
>> > +			      mlx5_devlink_enable_sriov_get,
>> > +			      mlx5_devlink_enable_sriov_set, NULL),
>> > 	DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_ID_CQE_COMPRESSION_TYPE,
>> > 			     "cqe_compress_type", DEVLINK_PARAM_TYPE_STRING,
>> > 			     BIT(DEVLINK_PARAM_CMODE_PERMANENT),
>> > --
>> > 2.48.1
>> > 
>> > 

Re: [PATCH net-next 02/14] devlink: Add 'total_vfs' generic device param

[Kamal Heib]

On Thu, Feb 27, 2025 at 06:12:15PM -0800, Saeed Mahameed wrote:
> From: Vlad Dumitrescu <vdumitrescu@nvidia.com>
> 
> NICs are typically configured with total_vfs=0, forcing users to rely
> on external tools to enable SR-IOV (a widely used and essential feature).
> 
> Add total_vfs parameter to devlink for SR-IOV max VF configurability.
> Enables standard kernel tools to manage SR-IOV, addressing the need for
> flexible VF configuration.
> 
> Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com>
> Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

Tested-by: Kamal Heib <kheib@redhat.com>
> ---
>  Documentation/networking/devlink/devlink-params.rst | 3 +++
>  include/net/devlink.h                               | 4 ++++
>  net/devlink/param.c                                 | 5 +++++
>  3 files changed, 12 insertions(+)
> 
> diff --git a/Documentation/networking/devlink/devlink-params.rst b/Documentation/networking/devlink/devlink-params.rst
> index 4e01dc32bc08..f266da05ab0d 100644
> --- a/Documentation/networking/devlink/devlink-params.rst
> +++ b/Documentation/networking/devlink/devlink-params.rst
> @@ -137,3 +137,6 @@ own name.
>     * - ``event_eq_size``
>       - u32
>       - Control the size of asynchronous control events EQ.
> +   * - ``total_vfs``
> +     - u32
> +     - The total number of Virtual Functions (VFs) supported by the PF.
> diff --git a/include/net/devlink.h b/include/net/devlink.h
> index b8783126c1ed..eed1e4507d17 100644
> --- a/include/net/devlink.h
> +++ b/include/net/devlink.h
> @@ -520,6 +520,7 @@ enum devlink_param_generic_id {
>  	DEVLINK_PARAM_GENERIC_ID_ENABLE_IWARP,
>  	DEVLINK_PARAM_GENERIC_ID_IO_EQ_SIZE,
>  	DEVLINK_PARAM_GENERIC_ID_EVENT_EQ_SIZE,
> +	DEVLINK_PARAM_GENERIC_ID_TOTAL_VFS,
>  
>  	/* add new param generic ids above here*/
>  	__DEVLINK_PARAM_GENERIC_ID_MAX,
> @@ -578,6 +579,9 @@ enum devlink_param_generic_id {
>  #define DEVLINK_PARAM_GENERIC_EVENT_EQ_SIZE_NAME "event_eq_size"
>  #define DEVLINK_PARAM_GENERIC_EVENT_EQ_SIZE_TYPE DEVLINK_PARAM_TYPE_U32
>  
> +#define DEVLINK_PARAM_GENERIC_TOTAL_VFS_NAME "total_vfs"
> +#define DEVLINK_PARAM_GENERIC_TOTAL_VFS_TYPE DEVLINK_PARAM_TYPE_U32
> +
>  #define DEVLINK_PARAM_GENERIC(_id, _cmodes, _get, _set, _validate)	\
>  {									\
>  	.id = DEVLINK_PARAM_GENERIC_ID_##_id,				\
> diff --git a/net/devlink/param.c b/net/devlink/param.c
> index e19d978dffa6..d163afbadab9 100644
> --- a/net/devlink/param.c
> +++ b/net/devlink/param.c
> @@ -92,6 +92,11 @@ static const struct devlink_param devlink_param_generic[] = {
>  		.name = DEVLINK_PARAM_GENERIC_EVENT_EQ_SIZE_NAME,
>  		.type = DEVLINK_PARAM_GENERIC_EVENT_EQ_SIZE_TYPE,
>  	},
> +	{
> +		.id = DEVLINK_PARAM_GENERIC_ID_TOTAL_VFS,
> +		.name = DEVLINK_PARAM_GENERIC_TOTAL_VFS_NAME,
> +		.type = DEVLINK_PARAM_GENERIC_TOTAL_VFS_TYPE,
> +	},
>  };
>  
>  static int devlink_param_generic_verify(const struct devlink_param *param)
> -- 
> 2.48.1
> 
> 

Re: [PATCH net-next 04/14] net/mlx5: Implement devlink enable_sriov parameter

[Kamal Heib]

On Thu, Feb 27, 2025 at 06:12:17PM -0800, Saeed Mahameed wrote:
> From: Vlad Dumitrescu <vdumitrescu@nvidia.com>
> 
> Example usage:
>   devlink dev param set pci/0000:01:00.0 name enable_sriov value {true, false} cmode permanent
>   devlink dev reload pci/0000:01:00.0 action fw_activate
>   echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove
>   echo 1 >/sys/bus/pci/rescan
>   grep ^ /sys/bus/pci/devices/0000:01:00.0/sriov_*
> 
> Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com>
> Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

Tested-by: Kamal Heib <kheib@redhat.com>

> ---
>  Documentation/networking/devlink/mlx5.rst     |  14 +-
>  .../net/ethernet/mellanox/mlx5/core/devlink.c |   1 +
>  .../mellanox/mlx5/core/lib/nv_param.c         | 184 ++++++++++++++++++
>  3 files changed, 196 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/networking/devlink/mlx5.rst b/Documentation/networking/devlink/mlx5.rst
> index 417e5cdcd35d..587e0200c1cd 100644
> --- a/Documentation/networking/devlink/mlx5.rst
> +++ b/Documentation/networking/devlink/mlx5.rst
> @@ -15,23 +15,31 @@ Parameters
>     * - Name
>       - Mode
>       - Validation
> +     - Notes
>     * - ``enable_roce``
>       - driverinit
> -     - Type: Boolean
> -
> -       If the device supports RoCE disablement, RoCE enablement state controls
> +     - Boolean
> +     - If the device supports RoCE disablement, RoCE enablement state controls
>         device support for RoCE capability. Otherwise, the control occurs in the
>         driver stack. When RoCE is disabled at the driver level, only raw
>         ethernet QPs are supported.
>     * - ``io_eq_size``
>       - driverinit
>       - The range is between 64 and 4096.
> +     -
>     * - ``event_eq_size``
>       - driverinit
>       - The range is between 64 and 4096.
> +     -
>     * - ``max_macs``
>       - driverinit
>       - The range is between 1 and 2^31. Only power of 2 values are supported.
> +     -
> +   * - ``enable_sriov``
> +     - permanent
> +     - Boolean
> +     - Applies to each physical function (PF) independently, if the device
> +       supports it. Otherwise, it applies symmetrically to all PFs.
>  
>  The ``mlx5`` driver also implements the following driver-specific
>  parameters.
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
> index 1f764ae4f4aa..7a702d84f19a 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
> @@ -8,6 +8,7 @@
>  #include "fs_core.h"
>  #include "eswitch.h"
>  #include "esw/qos.h"
> +#include "lib/nv_param.h"
>  #include "sf/dev/dev.h"
>  #include "sf/sf.h"
>  #include "lib/nv_param.h"
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
> index 5ab37a88c260..6b63fc110e2d 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
> @@ -5,7 +5,11 @@
>  #include "mlx5_core.h"
>  
>  enum {
> +	MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CONF               = 0x80,
> +	MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CAP                = 0x81,
>  	MLX5_CLASS_0_CTRL_ID_NV_SW_OFFLOAD_CONFIG             = 0x10a,
> +
> +	MLX5_CLASS_3_CTRL_ID_NV_PF_PCI_CONF                   = 0x80,
>  };
>  
>  struct mlx5_ifc_configuration_item_type_class_global_bits {
> @@ -13,9 +17,18 @@ struct mlx5_ifc_configuration_item_type_class_global_bits {
>  	u8         parameter_index[0x18];
>  };
>  
> +struct mlx5_ifc_configuration_item_type_class_per_host_pf_bits {
> +	u8         type_class[0x8];
> +	u8         pf_index[0x6];
> +	u8         pci_bus_index[0x8];
> +	u8         parameter_index[0xa];
> +};
> +
>  union mlx5_ifc_config_item_type_auto_bits {
>  	struct mlx5_ifc_configuration_item_type_class_global_bits
>  				configuration_item_type_class_global;
> +	struct mlx5_ifc_configuration_item_type_class_per_host_pf_bits
> +				configuration_item_type_class_per_host_pf;
>  	u8 reserved_at_0[0x20];
>  };
>  
> @@ -45,6 +58,45 @@ struct mlx5_ifc_mnvda_reg_bits {
>  	u8         configuration_item_data[64][0x20];
>  };
>  
> +struct mlx5_ifc_nv_global_pci_conf_bits {
> +	u8         sriov_valid[0x1];
> +	u8         reserved_at_1[0x10];
> +	u8         per_pf_total_vf[0x1];
> +	u8         reserved_at_12[0xe];
> +
> +	u8         sriov_en[0x1];
> +	u8         reserved_at_21[0xf];
> +	u8         total_vfs[0x10];
> +
> +	u8         reserved_at_40[0x20];
> +};
> +
> +struct mlx5_ifc_nv_global_pci_cap_bits {
> +	u8         max_vfs_per_pf_valid[0x1];
> +	u8         reserved_at_1[0x13];
> +	u8         per_pf_total_vf_supported[0x1];
> +	u8         reserved_at_15[0xb];
> +
> +	u8         sriov_support[0x1];
> +	u8         reserved_at_21[0xf];
> +	u8         max_vfs_per_pf[0x10];
> +
> +	u8         reserved_at_40[0x60];
> +};
> +
> +struct mlx5_ifc_nv_pf_pci_conf_bits {
> +	u8         reserved_at_0[0x9];
> +	u8         pf_total_vf_en[0x1];
> +	u8         reserved_at_a[0x16];
> +
> +	u8         reserved_at_20[0x20];
> +
> +	u8         reserved_at_40[0x10];
> +	u8         total_vf[0x10];
> +
> +	u8         reserved_at_60[0x20];
> +};
> +
>  struct mlx5_ifc_nv_sw_offload_conf_bits {
>  	u8         ip_over_vxlan_port[0x10];
>  	u8         tunnel_ecn_copy_offload_disable[0x1];
> @@ -206,7 +258,139 @@ static int mlx5_nv_param_devlink_cqe_compress_set(struct devlink *devlink, u32 i
>  	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
>  }
>  
> +static int
> +mlx5_nv_param_read_global_pci_conf(struct mlx5_core_dev *dev, void *mnvda, size_t len)
> +{
> +	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, type_class, 0);
> +	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, parameter_index,
> +				  MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CONF);
> +	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_global_pci_conf);
> +
> +	return mlx5_nv_param_read(dev, mnvda, len);
> +}
> +
> +static int
> +mlx5_nv_param_read_global_pci_cap(struct mlx5_core_dev *dev, void *mnvda, size_t len)
> +{
> +	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, type_class, 0);
> +	MLX5_SET_CONFIG_ITEM_TYPE(global, mnvda, parameter_index,
> +				  MLX5_CLASS_0_CTRL_ID_NV_GLOBAL_PCI_CAP);
> +	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_global_pci_cap);
> +
> +	return mlx5_nv_param_read(dev, mnvda, len);
> +}
> +
> +static int
> +mlx5_nv_param_read_per_host_pf_conf(struct mlx5_core_dev *dev, void *mnvda, size_t len)
> +{
> +	MLX5_SET_CONFIG_ITEM_TYPE(per_host_pf, mnvda, type_class, 3);
> +	MLX5_SET_CONFIG_ITEM_TYPE(per_host_pf, mnvda, parameter_index,
> +				  MLX5_CLASS_3_CTRL_ID_NV_PF_PCI_CONF);
> +	MLX5_SET_CONFIG_HDR_LEN(mnvda, nv_pf_pci_conf);
> +
> +	return mlx5_nv_param_read(dev, mnvda, len);
> +}
> +
> +static int mlx5_devlink_enable_sriov_get(struct devlink *devlink, u32 id,
> +					 struct devlink_param_gset_ctx *ctx)
> +{
> +	struct mlx5_core_dev *dev = devlink_priv(devlink);
> +	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
> +	void *data;
> +	int err;
> +
> +	err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
> +	if (err)
> +		return err;
> +
> +	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
> +	if (!MLX5_GET(nv_global_pci_cap, data, sriov_support)) {
> +		ctx->val.vbool = false;
> +		return 0;
> +	}
> +
> +	memset(mnvda, 0, sizeof(mnvda));
> +	err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
> +	if (err)
> +		return err;
> +
> +	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
> +	if (!MLX5_GET(nv_global_pci_conf, data, per_pf_total_vf)) {
> +		ctx->val.vbool = MLX5_GET(nv_global_pci_conf, data, sriov_en);
> +		return 0;
> +	}
> +
> +	/* SRIOV is per PF */
> +	memset(mnvda, 0, sizeof(mnvda));
> +	err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
> +	if (err)
> +		return err;
> +
> +	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
> +	ctx->val.vbool = MLX5_GET(nv_pf_pci_conf, data, pf_total_vf_en);
> +	return 0;
> +}
> +
> +static int mlx5_devlink_enable_sriov_set(struct devlink *devlink, u32 id,
> +					 struct devlink_param_gset_ctx *ctx,
> +					 struct netlink_ext_ack *extack)
> +{
> +	struct mlx5_core_dev *dev = devlink_priv(devlink);
> +	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
> +	bool per_pf_support;
> +	void *cap, *data;
> +	int err;
> +
> +	err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
> +	if (err) {
> +		NL_SET_ERR_MSG_MOD(extack, "Failed to read global PCI capability");
> +		return err;
> +	}
> +
> +	cap = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
> +	per_pf_support = MLX5_GET(nv_global_pci_cap, cap, per_pf_total_vf_supported);
> +
> +	if (!MLX5_GET(nv_global_pci_cap, cap, sriov_support)) {
> +		NL_SET_ERR_MSG_MOD(extack, "Not configurable on this device");
> +		return -EOPNOTSUPP;
> +	}
> +
> +	memset(mnvda, 0, sizeof(mnvda));
> +	err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
> +	if (err) {
> +		NL_SET_ERR_MSG_MOD(extack, "Unable to read global PCI configuration");
> +		return err;
> +	}
> +
> +	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
> +	MLX5_SET(nv_global_pci_conf, data, sriov_valid, 1);
> +	MLX5_SET(nv_global_pci_conf, data, sriov_en, ctx->val.vbool);
> +	MLX5_SET(nv_global_pci_conf, data, per_pf_total_vf, per_pf_support);
> +
> +	err = mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
> +	if (err) {
> +		NL_SET_ERR_MSG_MOD(extack, "Unable to write global PCI configuration");
> +		return err;
> +	}
> +
> +	if (!per_pf_support)
> +		return 0;
> +
> +	/* SRIOV is per PF */
> +	memset(mnvda, 0, sizeof(mnvda));
> +	err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
> +	if (err) {
> +		NL_SET_ERR_MSG_MOD(extack, "Unable to read per host PF configuration");
> +		return err;
> +	}
> +	MLX5_SET(nv_pf_pci_conf, data, pf_total_vf_en, ctx->val.vbool);
> +	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
> +}
> +
>  static const struct devlink_param mlx5_nv_param_devlink_params[] = {
> +	DEVLINK_PARAM_GENERIC(ENABLE_SRIOV, BIT(DEVLINK_PARAM_CMODE_PERMANENT),
> +			      mlx5_devlink_enable_sriov_get,
> +			      mlx5_devlink_enable_sriov_set, NULL),
>  	DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_ID_CQE_COMPRESSION_TYPE,
>  			     "cqe_compress_type", DEVLINK_PARAM_TYPE_STRING,
>  			     BIT(DEVLINK_PARAM_CMODE_PERMANENT),
> -- 
> 2.48.1
> 
> 

Re: [PATCH net-next 05/14] net/mlx5: Implement devlink total_vfs parameter

[Kamal Heib]

On Thu, Feb 27, 2025 at 06:12:18PM -0800, Saeed Mahameed wrote:
> From: Vlad Dumitrescu <vdumitrescu@nvidia.com>
> 
> Some devices support both symmetric (same value for all PFs) and
> asymmetric, while others only support symmetric configuration. This
> implementation prefers asymmetric, since it is closer to the devlink
> model (per function settings), but falls back to symmetric when needed.
> 
> Example usage:
>   devlink dev param set pci/0000:01:00.0 name total_vfs value <u16> cmode permanent
>   devlink dev reload pci/0000:01:00.0 action fw_activate
>   echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove
>   echo 1 >/sys/bus/pci/rescan
>   cat /sys/bus/pci/devices/0000:01:00.0/sriov_totalvfs
> 
> Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com>
> Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
> Reviewed-by: Jiri Pirko <jiri@nvidia.com>

Tested-by: Kamal Heib <kheib@redhat.com>

> ---
>  Documentation/networking/devlink/mlx5.rst     |  22 +++
>  .../mellanox/mlx5/core/lib/nv_param.c         | 125 ++++++++++++++++++
>  2 files changed, 147 insertions(+)
> 
> diff --git a/Documentation/networking/devlink/mlx5.rst b/Documentation/networking/devlink/mlx5.rst
> index 587e0200c1cd..00a43324dec2 100644
> --- a/Documentation/networking/devlink/mlx5.rst
> +++ b/Documentation/networking/devlink/mlx5.rst
> @@ -40,6 +40,28 @@ Parameters
>       - Boolean
>       - Applies to each physical function (PF) independently, if the device
>         supports it. Otherwise, it applies symmetrically to all PFs.
> +   * - ``total_vfs``
> +     - permanent
> +     - The range is between 1 and a device-specific max.
> +     - Applies to each physical function (PF) independently, if the device
> +       supports it. Otherwise, it applies symmetrically to all PFs.
> +
> +Note: permanent parameters such as ``enable_sriov`` and ``total_vfs`` require FW reset to take effect
> +
> +.. code-block:: bash
> +
> +   # setup parameters
> +   devlink dev param set pci/0000:01:00.0 name enable_sriov value true cmode permanent
> +   devlink dev param set pci/0000:01:00.0 name total_vfs value 8 cmode permanent
> +
> +   # Fw reset
> +   devlink dev reload pci/0000:01:00.0 action fw_activate
> +
> +   # for PCI related config such as sriov PCI reset/rescan is required:
> +   echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove
> +   echo 1 >/sys/bus/pci/rescan
> +   grep ^ /sys/bus/pci/devices/0000:01:00.0/sriov_*
> +
>  
>  The ``mlx5`` driver also implements the following driver-specific
>  parameters.
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
> index 6b63fc110e2d..97d74d890582 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
> @@ -387,10 +387,135 @@ static int mlx5_devlink_enable_sriov_set(struct devlink *devlink, u32 id,
>  	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
>  }
>  
> +static int mlx5_devlink_total_vfs_get(struct devlink *devlink, u32 id,
> +				      struct devlink_param_gset_ctx *ctx)
> +{
> +	struct mlx5_core_dev *dev = devlink_priv(devlink);
> +	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
> +	void *data;
> +	int err;
> +
> +	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
> +
> +	err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
> +	if (err)
> +		return err;
> +
> +	if (!MLX5_GET(nv_global_pci_cap, data, sriov_support)) {
> +		ctx->val.vu32 = 0;
> +		return 0;
> +	}
> +
> +	memset(mnvda, 0, sizeof(mnvda));
> +	err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
> +	if (err)
> +		return err;
> +
> +	if (!MLX5_GET(nv_global_pci_conf, data, per_pf_total_vf)) {
> +		ctx->val.vu32 = MLX5_GET(nv_global_pci_conf, data, total_vfs);
> +		return 0;
> +	}
> +
> +	/* SRIOV is per PF */
> +	memset(mnvda, 0, sizeof(mnvda));
> +	err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
> +	if (err)
> +		return err;
> +
> +	ctx->val.vu32 = MLX5_GET(nv_pf_pci_conf, data, total_vf);
> +
> +	return 0;
> +}
> +
> +static int mlx5_devlink_total_vfs_set(struct devlink *devlink, u32 id,
> +				      struct devlink_param_gset_ctx *ctx,
> +				      struct netlink_ext_ack *extack)
> +{
> +	struct mlx5_core_dev *dev = devlink_priv(devlink);
> +	u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)];
> +	bool per_pf_support;
> +	void *data;
> +	int err;
> +
> +	err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
> +	if (err) {
> +		NL_SET_ERR_MSG_MOD(extack, "Failed to read global pci cap");
> +		return err;
> +	}
> +
> +	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
> +	if (!MLX5_GET(nv_global_pci_cap, data, sriov_support)) {
> +		NL_SET_ERR_MSG_MOD(extack, "Not configurable on this device");
> +		return -EOPNOTSUPP;
> +	}
> +
> +	per_pf_support = MLX5_GET(nv_global_pci_cap, data, per_pf_total_vf_supported);
> +	memset(mnvda, 0, sizeof(mnvda));
> +
> +	err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
> +	if (err)
> +		return err;
> +
> +	MLX5_SET(nv_global_pci_conf, data, sriov_valid, 1);
> +	MLX5_SET(nv_global_pci_conf, data, per_pf_total_vf, per_pf_support);
> +
> +	if (!per_pf_support) {
> +		MLX5_SET(nv_global_pci_conf, data, total_vfs, ctx->val.vu32);
> +		return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
> +	}
> +
> +	/* SRIOV is per PF */
> +	err = mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
> +	if (err)
> +		return err;
> +
> +	memset(mnvda, 0, sizeof(mnvda));
> +	err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
> +	if (err)
> +		return err;
> +
> +	data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
> +	MLX5_SET(nv_pf_pci_conf, data, pf_total_vf_en, 1);
> +	MLX5_SET(nv_pf_pci_conf, data, total_vf, ctx->val.vu32);
> +	return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
> +}
> +
> +static int mlx5_devlink_total_vfs_validate(struct devlink *devlink, u32 id,
> +					   union devlink_param_value val,
> +					   struct netlink_ext_ack *extack)
> +{
> +	struct mlx5_core_dev *dev = devlink_priv(devlink);
> +	u32 cap[MLX5_ST_SZ_DW(mnvda_reg)];
> +	void *data;
> +	u16 max;
> +	int err;
> +
> +	data = MLX5_ADDR_OF(mnvda_reg, cap, configuration_item_data);
> +
> +	err = mlx5_nv_param_read_global_pci_cap(dev, cap, sizeof(cap));
> +	if (err)
> +		return err;
> +
> +	if (!MLX5_GET(nv_global_pci_cap, data, max_vfs_per_pf_valid))
> +		return 0; /* optimistic, but set might fail later */
> +
> +	max = MLX5_GET(nv_global_pci_cap, data, max_vfs_per_pf);
> +	if (val.vu16 > max) {
> +		NL_SET_ERR_MSG_FMT_MOD(extack,
> +				       "Max allowed by device is %u", max);
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
>  static const struct devlink_param mlx5_nv_param_devlink_params[] = {
>  	DEVLINK_PARAM_GENERIC(ENABLE_SRIOV, BIT(DEVLINK_PARAM_CMODE_PERMANENT),
>  			      mlx5_devlink_enable_sriov_get,
>  			      mlx5_devlink_enable_sriov_set, NULL),
> +	DEVLINK_PARAM_GENERIC(TOTAL_VFS, BIT(DEVLINK_PARAM_CMODE_PERMANENT),
> +			      mlx5_devlink_total_vfs_get, mlx5_devlink_total_vfs_set,
> +			      mlx5_devlink_total_vfs_validate),
>  	DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_ID_CQE_COMPRESSION_TYPE,
>  			     "cqe_compress_type", DEVLINK_PARAM_TYPE_STRING,
>  			     BIT(DEVLINK_PARAM_CMODE_PERMANENT),
> -- 
> 2.48.1
> 
> 

Re: [PATCH net-next 01/14] devlink: define enum for attr types of dynamic attributes

[Simon Horman]

On Thu, Feb 27, 2025 at 06:12:14PM -0800, Saeed Mahameed wrote:
> From: Jiri Pirko <jiri@nvidia.com>
> 
> Devlink param and health reporter fmsg use attributes with dynamic type
> which is determined according to a different type. Currently used values
> are NLA_*. The problem is, they are not part of UAPI. They may change
> which would cause a break.
> 
> To make this future safe, introduce a enum that shadows NLA_* values in
> it and is part of UAPI.
> 
> Also, this allows to possibly carry types that are unrelated to NLA_*
> values.
> 
> Signed-off-by: Jiri Pirko <jiri@nvidia.com>
> Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
> ---
>  Documentation/netlink/specs/devlink.yaml | 23 ++++++++++++++++++
>  include/uapi/linux/devlink.h             | 17 ++++++++++++++
>  net/devlink/health.c                     | 17 ++++++++++++--
>  net/devlink/param.c                      | 30 ++++++++++++------------
>  4 files changed, 70 insertions(+), 17 deletions(-)
> 
> diff --git a/Documentation/netlink/specs/devlink.yaml b/Documentation/netlink/specs/devlink.yaml
> index 09fbb4c03fc8..e99fc51856c5 100644
> --- a/Documentation/netlink/specs/devlink.yaml
> +++ b/Documentation/netlink/specs/devlink.yaml
> @@ -202,6 +202,29 @@ definitions:
>          name: exception
>        -
>          name: control
> +  -
> +    type: enum
> +    name: dyn_attr_type
> +    entries:
> +      -
> +        name: u8
> +        value: 1
> +      -
> +        name: u16
> +      -
> +        name: u32
> +      -
> +        name: u64
> +      -
> +        name: string
> +      -
> +        name: flag
> +      -
> +        name: nul_string
> +        value: 10
> +      -
> +        name: binary
> +  -
>  
>  attribute-sets:
>    -

Hi Saeed,

Thanks for these patches.

Unfortunately the above seems to cause ynl-regen to blow up.
I don't know why, but I see:

  $ ./tools/net/ynl/ynl-regen.sh
  Traceback (most recent call last):
    File "/home/nipa/net-next/wt-1/tools/net/ynl/pyynl/ynl_gen_c.py", line 3074, in <module>
      main()
      ~~~~^^
    File "/home/nipa/net-next/wt-1/tools/net/ynl/pyynl/ynl_gen_c.py", line 2783, in main
      parsed = Family(args.spec, exclude_ops)
    File "/home/nipa/net-next/wt-1/tools/net/ynl/pyynl/ynl_gen_c.py", line 954, in __init__
      super().__init__(file_name, exclude_ops=exclude_ops)
      ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/nipa/net-next/wt-1/tools/net/ynl/pyynl/lib/nlspec.py", line 462, in __init__
      jsonschema.validate(self.yaml, schema)
      ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.13/site-packages/jsonschema/validators.py", line 1307, in validate
      raise error
  jsonschema.exceptions.ValidationError: None is not of type 'object'

  ...

Perhaps I need a newer version of jsonschema?


> diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
> index 9401aa343673..8cdd60eb3c43 100644
> --- a/include/uapi/linux/devlink.h
> +++ b/include/uapi/linux/devlink.h
> @@ -385,6 +385,23 @@ enum devlink_linecard_state {
>  	DEVLINK_LINECARD_STATE_MAX = __DEVLINK_LINECARD_STATE_MAX - 1
>  };
>  
> +/**
> + * enum devlink_dyn_attr_type - Dynamic attribute type type.

Perhaps this relates to auto-generation.
But I think the Kernel doc should also document the enum values:
DEVLINK_DYN_ATTR_TYPE_U8, ...

> + */
> +enum devlink_dyn_attr_type {
> +	/* Following values relate to the internal NLA_* values */
> +	DEVLINK_DYN_ATTR_TYPE_U8 = 1,
> +	DEVLINK_DYN_ATTR_TYPE_U16,
> +	DEVLINK_DYN_ATTR_TYPE_U32,
> +	DEVLINK_DYN_ATTR_TYPE_U64,
> +	DEVLINK_DYN_ATTR_TYPE_STRING,
> +	DEVLINK_DYN_ATTR_TYPE_FLAG,
> +	DEVLINK_DYN_ATTR_TYPE_NUL_STRING = 10,
> +	DEVLINK_DYN_ATTR_TYPE_BINARY,
> +	__DEVLINK_DYN_ATTR_TYPE_CUSTOM_BASE = 0x80,
> +	/* Any possible custom types, unrelated to NLA_* values go below */
> +};
> +
>  enum devlink_attr {
>  	/* don't change the order or add anything between, this is ABI! */
>  	DEVLINK_ATTR_UNSPEC,

...

Re: [PATCH net-next 01/14] devlink: define enum for attr types of dynamic attributes

[Saeed Mahameed]

On 06 Mar 12:05, Simon Horman wrote:
>On Thu, Feb 27, 2025 at 06:12:14PM -0800, Saeed Mahameed wrote:
>> From: Jiri Pirko <jiri@nvidia.com>
>>
>> Devlink param and health reporter fmsg use attributes with dynamic type
>> which is determined according to a different type. Currently used values
>> are NLA_*. The problem is, they are not part of UAPI. They may change
>> which would cause a break.
>>
>> To make this future safe, introduce a enum that shadows NLA_* values in
>> it and is part of UAPI.
>>
>> Also, this allows to possibly carry types that are unrelated to NLA_*
>> values.
>>
>> Signed-off-by: Jiri Pirko <jiri@nvidia.com>
>> Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
>> ---
>>  Documentation/netlink/specs/devlink.yaml | 23 ++++++++++++++++++
>>  include/uapi/linux/devlink.h             | 17 ++++++++++++++
>>  net/devlink/health.c                     | 17 ++++++++++++--
>>  net/devlink/param.c                      | 30 ++++++++++++------------
>>  4 files changed, 70 insertions(+), 17 deletions(-)
>>
>> diff --git a/Documentation/netlink/specs/devlink.yaml b/Documentation/netlink/specs/devlink.yaml
>> index 09fbb4c03fc8..e99fc51856c5 100644
>> --- a/Documentation/netlink/specs/devlink.yaml
>> +++ b/Documentation/netlink/specs/devlink.yaml
>> @@ -202,6 +202,29 @@ definitions:
>>          name: exception
>>        -
>>          name: control
>> +  -
>> +    type: enum
>> +    name: dyn_attr_type
>> +    entries:
>> +      -
>> +        name: u8
>> +        value: 1
>> +      -
>> +        name: u16
>> +      -
>> +        name: u32
>> +      -
>> +        name: u64
>> +      -
>> +        name: string
>> +      -
>> +        name: flag
>> +      -
>> +        name: nul_string
>> +        value: 10
>> +      -
>> +        name: binary
>> +  -
^^^^^^^ extra empty type.. 

>
>Hi Saeed,
>
>Thanks for these patches.
>
>Unfortunately the above seems to cause ynl-regen to blow up.

Thanks Simon, good catch, there was an extra empty type added by mistake,
see above, will remove in V2.

>I don't know why, but I see:
>
>  $ ./tools/net/ynl/ynl-regen.sh
>  Traceback (most recent call last):
>    File "/home/nipa/net-next/wt-1/tools/net/ynl/pyynl/ynl_gen_c.py", line 3074, in <module>
>      main()
>      ~~~~^^
>    File "/home/nipa/net-next/wt-1/tools/net/ynl/pyynl/ynl_gen_c.py", line 2783, in main
>      parsed = Family(args.spec, exclude_ops)
>    File "/home/nipa/net-next/wt-1/tools/net/ynl/pyynl/ynl_gen_c.py", line 954, in __init__
>      super().__init__(file_name, exclude_ops=exclude_ops)
>      ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>    File "/home/nipa/net-next/wt-1/tools/net/ynl/pyynl/lib/nlspec.py", line 462, in __init__
>      jsonschema.validate(self.yaml, schema)
>      ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
>    File "/usr/lib/python3.13/site-packages/jsonschema/validators.py", line 1307, in validate
>      raise error
>  jsonschema.exceptions.ValidationError: None is not of type 'object'
>
>  ...
>
>Perhaps I need a newer version of jsonschema?
>
>
>> diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
>> index 9401aa343673..8cdd60eb3c43 100644
>> --- a/include/uapi/linux/devlink.h
>> +++ b/include/uapi/linux/devlink.h
>> @@ -385,6 +385,23 @@ enum devlink_linecard_state {
>>  	DEVLINK_LINECARD_STATE_MAX = __DEVLINK_LINECARD_STATE_MAX - 1
>>  };
>>
>> +/**
>> + * enum devlink_dyn_attr_type - Dynamic attribute type type.
>
>Perhaps this relates to auto-generation.
>But I think the Kernel doc should also document the enum values:
>DEVLINK_DYN_ATTR_TYPE_U8, ...
>
>> + */
>> +enum devlink_dyn_attr_type {
>> +	/* Following values relate to the internal NLA_* values */
>> +	DEVLINK_DYN_ATTR_TYPE_U8 = 1,
>> +	DEVLINK_DYN_ATTR_TYPE_U16,
>> +	DEVLINK_DYN_ATTR_TYPE_U32,
>> +	DEVLINK_DYN_ATTR_TYPE_U64,
>> +	DEVLINK_DYN_ATTR_TYPE_STRING,
>> +	DEVLINK_DYN_ATTR_TYPE_FLAG,
>> +	DEVLINK_DYN_ATTR_TYPE_NUL_STRING = 10,
>> +	DEVLINK_DYN_ATTR_TYPE_BINARY,
>> +	__DEVLINK_DYN_ATTR_TYPE_CUSTOM_BASE = 0x80,
>> +	/* Any possible custom types, unrelated to NLA_* values go below */
>> +};
>> +
>>  enum devlink_attr {
>>  	/* don't change the order or add anything between, this is ABI! */
>>  	DEVLINK_ATTR_UNSPEC,
>
>...

Re: [PATCH net-next 07/14] devlink: Implement port params registration

[Przemek Kitszel]

On 2/28/25 16:21, Jiri Pirko wrote:
> Fri, Feb 28, 2025 at 02:23:00PM +0100, przemyslaw.kitszel@intel.com wrote:
>> On 2/28/25 13:28, Jiri Pirko wrote:
>>> Fri, Feb 28, 2025 at 12:58:38PM +0100, przemyslaw.kitszel@intel.com wrote:
>>>> On 2/28/25 03:12, Saeed Mahameed wrote:
>>>>> From: Saeed Mahameed <saeedm@nvidia.com>
>>>>>
>>>>> Port params infrastructure is incomplete and needs a bit of plumbing to
>>>>> support port params commands from netlink.
>>>>>
>>>>> Introduce port params registration API, very similar to current devlink
>>>>> params API, add the params xarray to devlink_port structure and
>>>>> decouple devlink params registration routines from the devlink
>>>>> structure.
>>>>>
>>>>> Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
>>>>> Reviewed-by: Jiri Pirko <jiri@nvidia.com>
>>>>> ---
>>>>>     include/net/devlink.h |  14 ++++
>>>>>     net/devlink/param.c   | 150 ++++++++++++++++++++++++++++++++++--------
>>>>>     net/devlink/port.c    |   3 +
>>>>>     3 files changed, 140 insertions(+), 27 deletions(-)
>>>> For me devlink and devlink-port should be really the same, to the point
>>>> that the only difference is `bool is_port` flag inside of the
>>>> struct devlink. Then you could put special logic if really desired (to
>>>> exclude something for port).
>>>
>>> Why? Why other devlink objects shouldn't be the same as well. Then we
>>> can have one union. Does not make sense to me. The only think dev and
>>> port is sharing would be params. What else? Totally different beast.
>>
>> Instead of focusing on differences try to find similarities.
>>
>> health reporters per port and "toplevel",
>> just by grepping:
>> devlink_nl_sb_pool_fill()
>> devlink_nl_sb_port_pool_fill(),
> 
> Sharedbuffer is separate story.
> 
>>
>> devlink_region_create()
>> devlink_port_region_create()
> 
> Okay, regions I missed.
> 
>>
>> and there is no reason to assume that someone will not want to
>> extend ports to have devlink resources or other thing
> 
> But looks at differences. They are huge.
> 
> But perhaps I'm missing the point. What you want to achieve? Just to
> reduce API? That is always a tradeoff. I don't think the pros top the
> cons here.

not necessarily "reduce API", rather reuse the impl
but it is not a request for this series, just a comment

by itself this series is an improvement, and I didn't found anything
wrong in the code