[PATCH net-next 0/5][pull request] ice: Support 5 layer Tx scheduler topology

[PATCH net-next 0/5][pull request] ice: Support 5 layer Tx scheduler topology

[Tony Nguyen]

Michal Wilczynski says:

For performance reasons there is a need to have support for selectable
Tx scheduler topology. Currently firmware supports only the default
9-layer and 5-layer topology. This patch series enables switch from
default to 5-layer topology, if user decides to opt-in.

The following are changes since commit b2e3406a38f0f48b1dfb81e5bb73d243ff6af179:
  octeontx2-pf: Add support for page pool
and are available in the git repository at:
  git://git.kernel.org/pub/scm/linux/kernel/git/tnguy/next-queue 100GbE

Lukasz Czapnik (1):
  ice: Add txbalancing devlink param

Michal Wilczynski (2):
  ice: Enable switching default tx scheduler topology
  ice: Document txbalancing parameter

Raj Victor (2):
  ice: Support 5 layer topology
  ice: Adjust the VSI/Aggregator layers

 Documentation/networking/devlink/ice.rst      |  17 ++
 .../net/ethernet/intel/ice/ice_adminq_cmd.h   |  31 +++
 drivers/net/ethernet/intel/ice/ice_common.c   |   6 +
 drivers/net/ethernet/intel/ice/ice_ddp.c      | 200 ++++++++++++++++++
 drivers/net/ethernet/intel/ice/ice_ddp.h      |   7 +-
 drivers/net/ethernet/intel/ice/ice_devlink.c  | 161 +++++++++++++-
 .../net/ethernet/intel/ice/ice_fw_update.c    |   2 +-
 .../net/ethernet/intel/ice/ice_fw_update.h    |   3 +
 drivers/net/ethernet/intel/ice/ice_main.c     | 105 +++++++--
 drivers/net/ethernet/intel/ice/ice_nvm.c      |   2 +-
 drivers/net/ethernet/intel/ice/ice_nvm.h      |   3 +
 drivers/net/ethernet/intel/ice/ice_sched.c    |  34 +--
 drivers/net/ethernet/intel/ice/ice_sched.h    |   3 +
 drivers/net/ethernet/intel/ice/ice_type.h     |   1 +
 14 files changed, 534 insertions(+), 41 deletions(-)

-- 
2.38.1

[PATCH net-next 1/5] ice: Support 5 layer topology

[Tony Nguyen]

From: Raj Victor <victor.raj@intel.com>

There is a performance issue reported when the number of VSIs are not
multiple of 8. This is caused due to the max children limitation per
node(8) in 9 layer topology. The BW credits are shared evenly among the
children by default. Assume one node has 8 children and the other has 1.
The parent of these nodes share the BW credit equally among them.
Apparently this causes a problem for the first node which has 8 children.
The 9th VM get more BW credits than the first 8 VMs.

Example:

1) With 8 VM's:
for x in 0 1 2 3 4 5 6 7;
do taskset -c ${x} netperf -P0 -H 172.68.169.125 &  sleep .1 ; done

tx_queue_0_packets: 23283027
tx_queue_1_packets: 23292289
tx_queue_2_packets: 23276136
tx_queue_3_packets: 23279828
tx_queue_4_packets: 23279828
tx_queue_5_packets: 23279333
tx_queue_6_packets: 23277745
tx_queue_7_packets: 23279950
tx_queue_8_packets: 0

2) With 9 VM's:
for x in 0 1 2 3 4 5 6 7 8;
do taskset -c ${x} netperf -P0 -H 172.68.169.125 &  sleep .1 ; done

tx_queue_0_packets: 24163396
tx_queue_1_packets: 24164623
tx_queue_2_packets: 24163188
tx_queue_3_packets: 24163701
tx_queue_4_packets: 24163683
tx_queue_5_packets: 24164668
tx_queue_6_packets: 23327200
tx_queue_7_packets: 24163853
tx_queue_8_packets: 91101417

So on average queue 8 statistics show that 3.7 times more packets were
send there than to the other queues.

The FW starting with version 3.20, has increased the max number of
children per node by reducing the number of layers from 9 to 5. Reflect
this on driver side.

Signed-off-by: Raj Victor <victor.raj@intel.com>
Co-developed-by: Michal Wilczynski <michal.wilczynski@intel.com>
Signed-off-by: Michal Wilczynski <michal.wilczynski@intel.com>
Reviewed-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Tested-by: Pucha Himasekhar Reddy <himasekharx.reddy.pucha@intel.com> (A Contingent worker at Intel)
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
 .../net/ethernet/intel/ice/ice_adminq_cmd.h   |  22 ++
 drivers/net/ethernet/intel/ice/ice_common.c   |   6 +
 drivers/net/ethernet/intel/ice/ice_ddp.c      | 200 ++++++++++++++++++
 drivers/net/ethernet/intel/ice/ice_ddp.h      |   7 +-
 drivers/net/ethernet/intel/ice/ice_sched.h    |   3 +
 drivers/net/ethernet/intel/ice/ice_type.h     |   1 +
 6 files changed, 237 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h b/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h
index 63d3e1dcbba5..d1a449b07fdf 100644
--- a/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h
+++ b/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h
@@ -120,6 +120,7 @@ struct ice_aqc_list_caps_elem {
 #define ICE_AQC_CAPS_PCIE_RESET_AVOIDANCE		0x0076
 #define ICE_AQC_CAPS_POST_UPDATE_RESET_RESTRICT		0x0077
 #define ICE_AQC_CAPS_NVM_MGMT				0x0080
+#define ICE_AQC_CAPS_TX_SCHED_TOPO_COMP_MODE		0x0085
 
 	u8 major_ver;
 	u8 minor_ver;
@@ -798,6 +799,23 @@ struct ice_aqc_get_topo {
 	__le32 addr_low;
 };
 
+/* Get/Set Tx Topology (indirect 0x0418/0x0417) */
+struct ice_aqc_get_set_tx_topo {
+	u8 set_flags;
+#define ICE_AQC_TX_TOPO_FLAGS_CORRER		BIT(0)
+#define ICE_AQC_TX_TOPO_FLAGS_SRC_RAM		BIT(1)
+#define ICE_AQC_TX_TOPO_FLAGS_LOAD_NEW		BIT(4)
+#define ICE_AQC_TX_TOPO_FLAGS_ISSUED		BIT(5)
+
+	u8 get_flags;
+#define ICE_AQC_TX_TOPO_GET_RAM	2
+
+	__le16 reserved1;
+	__le32 reserved2;
+	__le32 addr_high;
+	__le32 addr_low;
+};
+
 /* Update TSE (indirect 0x0403)
  * Get TSE (indirect 0x0404)
  * Add TSE (indirect 0x0401)
@@ -2197,6 +2215,7 @@ struct ice_aq_desc {
 		struct ice_aqc_get_link_topo get_link_topo;
 		struct ice_aqc_i2c read_write_i2c;
 		struct ice_aqc_read_i2c_resp read_i2c_resp;
+		struct ice_aqc_get_set_tx_topo get_set_tx_topo;
 	} params;
 };
 
@@ -2302,6 +2321,9 @@ enum ice_adminq_opc {
 	ice_aqc_opc_query_sched_res			= 0x0412,
 	ice_aqc_opc_remove_rl_profiles			= 0x0415,
 
+	ice_aqc_opc_set_tx_topo				= 0x0417,
+	ice_aqc_opc_get_tx_topo				= 0x0418,
+
 	/* PHY commands */
 	ice_aqc_opc_get_phy_caps			= 0x0600,
 	ice_aqc_opc_set_phy_cfg				= 0x0601,
diff --git a/drivers/net/ethernet/intel/ice/ice_common.c b/drivers/net/ethernet/intel/ice/ice_common.c
index 0157f6e98d3e..a9f2e6bff806 100644
--- a/drivers/net/ethernet/intel/ice/ice_common.c
+++ b/drivers/net/ethernet/intel/ice/ice_common.c
@@ -1687,6 +1687,8 @@ ice_aq_send_cmd(struct ice_hw *hw, struct ice_aq_desc *desc, void *buf,
 	case ice_aqc_opc_set_port_params:
 	case ice_aqc_opc_get_vlan_mode_parameters:
 	case ice_aqc_opc_set_vlan_mode_parameters:
+	case ice_aqc_opc_set_tx_topo:
+	case ice_aqc_opc_get_tx_topo:
 	case ice_aqc_opc_add_recipe:
 	case ice_aqc_opc_recipe_to_profile:
 	case ice_aqc_opc_get_recipe:
@@ -2243,6 +2245,10 @@ ice_parse_common_caps(struct ice_hw *hw, struct ice_hw_common_caps *caps,
 			  "%s: reset_restrict_support = %d\n", prefix,
 			  caps->reset_restrict_support);
 		break;
+	case ICE_AQC_CAPS_TX_SCHED_TOPO_COMP_MODE:
+		caps->tx_sched_topo_comp_mode_en = (number == 1);
+		break;
+
 	default:
 		/* Not one of the recognized common capabilities */
 		found = false;
diff --git a/drivers/net/ethernet/intel/ice/ice_ddp.c b/drivers/net/ethernet/intel/ice/ice_ddp.c
index d71ed210f9c4..08e4f18ea59a 100644
--- a/drivers/net/ethernet/intel/ice/ice_ddp.c
+++ b/drivers/net/ethernet/intel/ice/ice_ddp.c
@@ -4,6 +4,7 @@
 #include "ice_common.h"
 #include "ice.h"
 #include "ice_ddp.h"
+#include "ice_sched.h"
 
 /* For supporting double VLAN mode, it is necessary to enable or disable certain
  * boost tcam entries. The metadata labels names that match the following
@@ -1895,3 +1896,202 @@ enum ice_ddp_state ice_copy_and_init_pkg(struct ice_hw *hw, const u8 *buf,
 
 	return state;
 }
+
+/**
+ * ice_get_set_tx_topo - get or set Tx topology
+ * @hw: pointer to the HW struct
+ * @buf: pointer to Tx topology buffer
+ * @buf_size: buffer size
+ * @cd: pointer to command details structure or NULL
+ * @flags: pointer to descriptor flags
+ * @set: 0-get, 1-set topology
+ *
+ * The function will get or set Tx topology
+ */
+static int
+ice_get_set_tx_topo(struct ice_hw *hw, u8 *buf, u16 buf_size,
+		    struct ice_sq_cd *cd, u8 *flags, bool set)
+{
+	struct ice_aqc_get_set_tx_topo *cmd;
+	struct ice_aq_desc desc;
+	int status;
+
+	cmd = &desc.params.get_set_tx_topo;
+	if (set) {
+		ice_fill_dflt_direct_cmd_desc(&desc, ice_aqc_opc_set_tx_topo);
+		cmd->set_flags = ICE_AQC_TX_TOPO_FLAGS_ISSUED;
+		/* requested to update a new topology, not a default topology */
+		if (buf)
+			cmd->set_flags |= ICE_AQC_TX_TOPO_FLAGS_SRC_RAM |
+					  ICE_AQC_TX_TOPO_FLAGS_LOAD_NEW;
+	} else {
+		ice_fill_dflt_direct_cmd_desc(&desc, ice_aqc_opc_get_tx_topo);
+		cmd->get_flags = ICE_AQC_TX_TOPO_GET_RAM;
+	}
+	desc.flags |= cpu_to_le16(ICE_AQ_FLAG_RD);
+	status = ice_aq_send_cmd(hw, &desc, buf, buf_size, cd);
+	if (status)
+		return status;
+	/* read the return flag values (first byte) for get operation */
+	if (!set && flags)
+		*flags = desc.params.get_set_tx_topo.set_flags;
+
+	return 0;
+}
+
+/**
+ * ice_cfg_tx_topo - Initialize new Tx topology if available
+ * @hw: pointer to the HW struct
+ * @buf: pointer to Tx topology buffer
+ * @len: buffer size
+ *
+ * The function will apply the new Tx topology from the package buffer
+ * if available.
+ */
+int ice_cfg_tx_topo(struct ice_hw *hw, u8 *buf, u32 len)
+{
+	u8 *current_topo, *new_topo = NULL;
+	struct ice_run_time_cfg_seg *seg;
+	struct ice_buf_hdr *section;
+	struct ice_pkg_hdr *pkg_hdr;
+	enum ice_ddp_state state;
+	u16 size = 0, offset;
+	u32 reg = 0;
+	int status;
+	u8 flags;
+
+	if (!buf || !len)
+		return -EINVAL;
+
+	/* Does FW support new Tx topology mode ? */
+	if (!hw->func_caps.common_cap.tx_sched_topo_comp_mode_en) {
+		ice_debug(hw, ICE_DBG_INIT, "FW doesn't support compatibility mode\n");
+		return -EOPNOTSUPP;
+	}
+
+	current_topo = kzalloc(ICE_AQ_MAX_BUF_LEN, GFP_KERNEL);
+	if (!current_topo)
+		return -ENOMEM;
+
+	/* get the current Tx topology */
+	status = ice_get_set_tx_topo(hw, current_topo, ICE_AQ_MAX_BUF_LEN, NULL,
+				     &flags, false);
+
+	kfree(current_topo);
+
+	if (status) {
+		ice_debug(hw, ICE_DBG_INIT, "Get current topology is failed\n");
+		return status;
+	}
+
+	/* Is default topology already applied ? */
+	if (!(flags & ICE_AQC_TX_TOPO_FLAGS_LOAD_NEW) &&
+	    hw->num_tx_sched_layers == ICE_SCHED_9_LAYERS) {
+		ice_debug(hw, ICE_DBG_INIT, "Loaded default topology\n");
+		/* Already default topology is loaded */
+		return -EEXIST;
+	}
+
+	/* Is new topology already applied ? */
+	if ((flags & ICE_AQC_TX_TOPO_FLAGS_LOAD_NEW) &&
+	    hw->num_tx_sched_layers == ICE_SCHED_5_LAYERS) {
+		ice_debug(hw, ICE_DBG_INIT, "Loaded new topology\n");
+		/* Already new topology is loaded */
+		return -EEXIST;
+	}
+
+	/* Is set topology issued already ? */
+	if (flags & ICE_AQC_TX_TOPO_FLAGS_ISSUED) {
+		ice_debug(hw, ICE_DBG_INIT, "Update Tx topology was done by another PF\n");
+		/* add a small delay before exiting */
+		msleep(2000);
+		return -EEXIST;
+	}
+
+	/* Change the topology from new to default (5 to 9) */
+	if (!(flags & ICE_AQC_TX_TOPO_FLAGS_LOAD_NEW) &&
+	    hw->num_tx_sched_layers == ICE_SCHED_5_LAYERS) {
+		ice_debug(hw, ICE_DBG_INIT, "Change topology from 5 to 9 layers\n");
+		goto update_topo;
+	}
+
+	pkg_hdr = (struct ice_pkg_hdr *)buf;
+	state = ice_verify_pkg(pkg_hdr, len);
+	if (state) {
+		ice_debug(hw, ICE_DBG_INIT, "failed to verify pkg (err: %d)\n",
+			  state);
+		return -EIO;
+	}
+
+	/* find run time configuration segment */
+	seg = (struct ice_run_time_cfg_seg *)
+		ice_find_seg_in_pkg(hw, SEGMENT_TYPE_ICE_RUN_TIME_CFG, pkg_hdr);
+	if (!seg) {
+		ice_debug(hw, ICE_DBG_INIT, "5 layer topology segment is missing\n");
+		return -EIO;
+	}
+
+	if (le32_to_cpu(seg->buf_table.buf_count) < ICE_MIN_S_COUNT) {
+		ice_debug(hw, ICE_DBG_INIT, "5 layer topology segment count(%d) is wrong\n",
+			  seg->buf_table.buf_count);
+		return -EIO;
+	}
+
+	section = ice_pkg_val_buf(seg->buf_table.buf_array);
+	if (!section || le32_to_cpu(section->section_entry[0].type) !=
+		ICE_SID_TX_5_LAYER_TOPO) {
+		ice_debug(hw, ICE_DBG_INIT, "5 layer topology section type is wrong\n");
+		return -EIO;
+	}
+
+	size = le16_to_cpu(section->section_entry[0].size);
+	offset = le16_to_cpu(section->section_entry[0].offset);
+	if (size < ICE_MIN_S_SZ || size > ICE_MAX_S_SZ) {
+		ice_debug(hw, ICE_DBG_INIT, "5 layer topology section size is wrong\n");
+		return -EIO;
+	}
+
+	/* make sure the section fits in the buffer */
+	if (offset + size > ICE_PKG_BUF_SIZE) {
+		ice_debug(hw, ICE_DBG_INIT, "5 layer topology buffer > 4K\n");
+		return -EIO;
+	}
+
+	/* Get the new topology buffer */
+	new_topo = ((u8 *)section) + offset;
+
+update_topo:
+	/* acquire global lock to make sure that set topology issued
+	 * by one PF
+	 */
+	status = ice_acquire_res(hw, ICE_GLOBAL_CFG_LOCK_RES_ID, ICE_RES_WRITE,
+				 ICE_GLOBAL_CFG_LOCK_TIMEOUT);
+	if (status) {
+		ice_debug(hw, ICE_DBG_INIT, "Failed to acquire global lock\n");
+		return status;
+	}
+
+	/* check reset was triggered already or not */
+	reg = rd32(hw, GLGEN_RSTAT);
+	if (reg & GLGEN_RSTAT_DEVSTATE_M) {
+		/* Reset is in progress, re-init the hw again */
+		ice_debug(hw, ICE_DBG_INIT, "Reset is in progress. layer topology might be applied already\n");
+		ice_check_reset(hw);
+		return 0;
+	}
+
+	/* set new topology */
+	status = ice_get_set_tx_topo(hw, new_topo, size, NULL, NULL, true);
+	if (status) {
+		ice_debug(hw, ICE_DBG_INIT, "Set Tx topology is failed\n");
+		return status;
+	}
+
+	/* new topology is updated, delay 1 second before issuing the CORRER */
+	msleep(1000);
+	ice_reset(hw, ICE_RESET_CORER);
+	/* CORER will clear the global lock, so no explicit call
+	 * required for release
+	 */
+	return 0;
+}
diff --git a/drivers/net/ethernet/intel/ice/ice_ddp.h b/drivers/net/ethernet/intel/ice/ice_ddp.h
index 37eadb3d27a8..2427b9db88e0 100644
--- a/drivers/net/ethernet/intel/ice/ice_ddp.h
+++ b/drivers/net/ethernet/intel/ice/ice_ddp.h
@@ -100,8 +100,9 @@ struct ice_pkg_hdr {
 
 /* generic segment */
 struct ice_generic_seg_hdr {
-#define SEGMENT_TYPE_METADATA 0x00000001
-#define SEGMENT_TYPE_ICE 0x00000010
+#define SEGMENT_TYPE_METADATA		0x00000001
+#define SEGMENT_TYPE_ICE		0x00000010
+#define SEGMENT_TYPE_ICE_RUN_TIME_CFG	0x00000020
 	__le32 seg_type;
 	struct ice_pkg_ver seg_format_ver;
 	__le32 seg_size;
@@ -442,4 +443,6 @@ void *ice_pkg_enum_section(struct ice_seg *ice_seg, struct ice_pkg_enum *state,
 
 struct ice_buf_hdr *ice_pkg_val_buf(struct ice_buf *buf);
 
+int ice_cfg_tx_topo(struct ice_hw *hw, u8 *buf, u32 len);
+
 #endif
diff --git a/drivers/net/ethernet/intel/ice/ice_sched.h b/drivers/net/ethernet/intel/ice/ice_sched.h
index 9c100747445a..1d01a1898b8b 100644
--- a/drivers/net/ethernet/intel/ice/ice_sched.h
+++ b/drivers/net/ethernet/intel/ice/ice_sched.h
@@ -6,6 +6,9 @@
 
 #include "ice_common.h"
 
+#define ICE_SCHED_5_LAYERS	5
+#define ICE_SCHED_9_LAYERS	9
+
 #define SCHED_NODE_NAME_MAX_LEN 32
 
 #define ICE_QGRP_LAYER_OFFSET	2
diff --git a/drivers/net/ethernet/intel/ice/ice_type.h b/drivers/net/ethernet/intel/ice/ice_type.h
index a09556e57803..5602695243a8 100644
--- a/drivers/net/ethernet/intel/ice/ice_type.h
+++ b/drivers/net/ethernet/intel/ice/ice_type.h
@@ -290,6 +290,7 @@ struct ice_hw_common_caps {
 	bool pcie_reset_avoidance;
 	/* Post update reset restriction */
 	bool reset_restrict_support;
+	bool tx_sched_topo_comp_mode_en;
 };
 
 /* IEEE 1588 TIME_SYNC specific info */
-- 
2.38.1

[PATCH net-next 2/5] ice: Adjust the VSI/Aggregator layers

[Tony Nguyen]

From: Raj Victor <victor.raj@intel.com>

Adjust the VSI/Aggregator layers based on the number of logical layers
supported by the FW. Currently the VSI and aggregator layers are
fixed based on the 9 layer scheduler tree layout. Due to performance
reasons the number of layers of the scheduler tree is changing from
9 to 5. It requires a readjustment of these VSI/Aggregator layer values.

Signed-off-by: Raj Victor <victor.raj@intel.com>
Co-developed-by: Michal Wilczynski <michal.wilczynski@intel.com>
Signed-off-by: Michal Wilczynski <michal.wilczynski@intel.com>
Tested-by: Pucha Himasekhar Reddy <himasekharx.reddy.pucha@intel.com> (A Contingent worker at Intel)
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
 drivers/net/ethernet/intel/ice/ice_sched.c | 34 +++++++++++-----------
 1 file changed, 17 insertions(+), 17 deletions(-)

diff --git a/drivers/net/ethernet/intel/ice/ice_sched.c b/drivers/net/ethernet/intel/ice/ice_sched.c
index b7682de0ae05..cb75b202694b 100644
--- a/drivers/net/ethernet/intel/ice/ice_sched.c
+++ b/drivers/net/ethernet/intel/ice/ice_sched.c
@@ -1132,12 +1132,11 @@ static u8 ice_sched_get_vsi_layer(struct ice_hw *hw)
 	 *     5 or less       sw_entry_point_layer
 	 */
 	/* calculate the VSI layer based on number of layers. */
-	if (hw->num_tx_sched_layers > ICE_VSI_LAYER_OFFSET + 1) {
-		u8 layer = hw->num_tx_sched_layers - ICE_VSI_LAYER_OFFSET;
-
-		if (layer > hw->sw_entry_point_layer)
-			return layer;
-	}
+	if (hw->num_tx_sched_layers == ICE_SCHED_9_LAYERS)
+		return hw->num_tx_sched_layers - ICE_VSI_LAYER_OFFSET;
+	if (hw->num_tx_sched_layers == ICE_SCHED_5_LAYERS)
+		/* qgroup and VSI layers are same */
+		return hw->num_tx_sched_layers - ICE_QGRP_LAYER_OFFSET;
 	return hw->sw_entry_point_layer;
 }
 
@@ -1154,12 +1153,8 @@ static u8 ice_sched_get_agg_layer(struct ice_hw *hw)
 	 *     7 or less       sw_entry_point_layer
 	 */
 	/* calculate the aggregator layer based on number of layers. */
-	if (hw->num_tx_sched_layers > ICE_AGG_LAYER_OFFSET + 1) {
-		u8 layer = hw->num_tx_sched_layers - ICE_AGG_LAYER_OFFSET;
-
-		if (layer > hw->sw_entry_point_layer)
-			return layer;
-	}
+	if (hw->num_tx_sched_layers == ICE_SCHED_9_LAYERS)
+		return hw->num_tx_sched_layers - ICE_AGG_LAYER_OFFSET;
 	return hw->sw_entry_point_layer;
 }
 
@@ -1515,10 +1510,11 @@ ice_sched_get_free_qparent(struct ice_port_info *pi, u16 vsi_handle, u8 tc,
 {
 	struct ice_sched_node *vsi_node, *qgrp_node;
 	struct ice_vsi_ctx *vsi_ctx;
+	u8 qgrp_layer, vsi_layer;
 	u16 max_children;
-	u8 qgrp_layer;
 
 	qgrp_layer = ice_sched_get_qgrp_layer(pi->hw);
+	vsi_layer = ice_sched_get_vsi_layer(pi->hw);
 	max_children = pi->hw->max_children[qgrp_layer];
 
 	vsi_ctx = ice_get_vsi_ctx(pi->hw, vsi_handle);
@@ -1529,6 +1525,12 @@ ice_sched_get_free_qparent(struct ice_port_info *pi, u16 vsi_handle, u8 tc,
 	if (!vsi_node)
 		return NULL;
 
+	/* If the queue group and VSI layer are same then queues
+	 * are all attached directly to VSI
+	 */
+	if (qgrp_layer == vsi_layer)
+		return vsi_node;
+
 	/* get the first queue group node from VSI sub-tree */
 	qgrp_node = ice_sched_get_first_node(pi, vsi_node, qgrp_layer);
 	while (qgrp_node) {
@@ -3212,7 +3214,7 @@ ice_sched_add_rl_profile(struct ice_port_info *pi,
 	u8 profile_type;
 	int status;
 
-	if (layer_num >= ICE_AQC_TOPO_MAX_LEVEL_NUM)
+	if (!pi || layer_num >= pi->hw->num_tx_sched_layers)
 		return NULL;
 	switch (rl_type) {
 	case ICE_MIN_BW:
@@ -3228,8 +3230,6 @@ ice_sched_add_rl_profile(struct ice_port_info *pi,
 		return NULL;
 	}
 
-	if (!pi)
-		return NULL;
 	hw = pi->hw;
 	list_for_each_entry(rl_prof_elem, &pi->rl_prof_list[layer_num],
 			    list_entry)
@@ -3459,7 +3459,7 @@ ice_sched_rm_rl_profile(struct ice_port_info *pi, u8 layer_num, u8 profile_type,
 	struct ice_aqc_rl_profile_info *rl_prof_elem;
 	int status = 0;
 
-	if (layer_num >= ICE_AQC_TOPO_MAX_LEVEL_NUM)
+	if (layer_num >= pi->hw->num_tx_sched_layers)
 		return -EINVAL;
 	/* Check the existing list for RL profile */
 	list_for_each_entry(rl_prof_elem, &pi->rl_prof_list[layer_num],
-- 
2.38.1

[PATCH net-next 3/5] ice: Enable switching default tx scheduler topology

[Tony Nguyen]

From: Michal Wilczynski <michal.wilczynski@intel.com>

Introduce support for tx scheduler topology change, based on user
selection, from default 9-layer to 5-layer. In order for switch to be
successful there is a new NVM(version 3.20 or older) and DDP package(OS
Package 1.3.29 or older).

Enable 5-layer topology switch in init path of the driver. To accomplish
that upload of the DDP package needs to be delayed, until change in Tx
topology is finished. To trigger the Tx change user selection should be
changed in NVM using devlink. Then the platform should be rebooted.

Signed-off-by: Michal Wilczynski <michal.wilczynski@intel.com>
Tested-by: Pucha Himasekhar Reddy <himasekharx.reddy.pucha@intel.com> (A Contingent worker at Intel)
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
 drivers/net/ethernet/intel/ice/ice_main.c | 105 ++++++++++++++++++----
 1 file changed, 86 insertions(+), 19 deletions(-)

diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
index 62e91512aeab..4367da4eb352 100644
--- a/drivers/net/ethernet/intel/ice/ice_main.c
+++ b/drivers/net/ethernet/intel/ice/ice_main.c
@@ -4264,11 +4264,11 @@ static char *ice_get_opt_fw_name(struct ice_pf *pf)
 /**
  * ice_request_fw - Device initialization routine
  * @pf: pointer to the PF instance
+ * @firmware: double pointer to firmware struct
  */
-static void ice_request_fw(struct ice_pf *pf)
+static int ice_request_fw(struct ice_pf *pf, const struct firmware **firmware)
 {
 	char *opt_fw_filename = ice_get_opt_fw_name(pf);
-	const struct firmware *firmware = NULL;
 	struct device *dev = ice_pf_to_dev(pf);
 	int err = 0;
 
@@ -4277,29 +4277,94 @@ static void ice_request_fw(struct ice_pf *pf)
 	 * and warning messages for other errors.
 	 */
 	if (opt_fw_filename) {
-		err = firmware_request_nowarn(&firmware, opt_fw_filename, dev);
-		if (err) {
-			kfree(opt_fw_filename);
-			goto dflt_pkg_load;
-		}
-
-		/* request for firmware was successful. Download to device */
-		ice_load_pkg(firmware, pf);
+		err = firmware_request_nowarn(firmware, opt_fw_filename, dev);
 		kfree(opt_fw_filename);
-		release_firmware(firmware);
-		return;
+		if (!err)
+			return err;
 	}
+	err = request_firmware(firmware, ICE_DDP_PKG_FILE, dev);
+	if (err)
+		dev_err(dev, "The DDP package file was not found or could not be read. Entering Safe Mode\n");
+
+	return err;
+}
+
+/**
+ * ice_init_tx_topology - performs Tx topology initialization
+ * @hw: pointer to the hardware structure
+ * @firmware: pointer to firmware structure
+ */
+static int
+ice_init_tx_topology(struct ice_hw *hw, const struct firmware *firmware)
+{
+	u8 num_tx_sched_layers = hw->num_tx_sched_layers;
+	struct ice_pf *pf = hw->back;
+	struct device *dev;
+	u8 *buf_copy;
+	int err;
+
+	dev = ice_pf_to_dev(pf);
+	/* ice_cfg_tx_topo buf argument is not a constant,
+	 * so we have to make a copy
+	 */
+	buf_copy = kmemdup(firmware->data, firmware->size, GFP_KERNEL);
+
+	err = ice_cfg_tx_topo(hw, buf_copy, firmware->size);
+	if (!err) {
+		if (hw->num_tx_sched_layers > num_tx_sched_layers)
+			dev_info(dev, "Transmit balancing feature disabled\n");
+		else
+			dev_info(dev, "Transmit balancing feature enabled\n");
+		/* if there was a change in topology ice_cfg_tx_topo triggered
+		 * a CORER and we need to re-init hw
+		 */
+		ice_deinit_hw(hw);
+		err = ice_init_hw(hw);
+
+		/* in this case we're not allowing safe mode */
+		kfree(buf_copy);
 
-dflt_pkg_load:
-	err = request_firmware(&firmware, ICE_DDP_PKG_FILE, dev);
+		return err;
+	} else if (err == -EIO) {
+		dev_info(dev, "DDP package does not support transmit balancing feature - please update to the latest DDP package and try again\n");
+	}
+
+	kfree(buf_copy);
+
+	return 0;
+}
+
+/**
+ * ice_init_ddp_config - DDP related configuration
+ * @hw: pointer to the hardware structure
+ * @pf: pointer to pf structure
+ *
+ * This function loads DDP file from the disk, then initializes Tx
+ * topology. At the end DDP package is loaded on the card.
+ */
+static int ice_init_ddp_config(struct ice_hw *hw, struct ice_pf *pf)
+{
+	struct device *dev = ice_pf_to_dev(pf);
+	const struct firmware *firmware = NULL;
+	int err;
+
+	err = ice_request_fw(pf, &firmware);
+	if (err)
+		return 0;
+
+	err = ice_init_tx_topology(hw, firmware);
 	if (err) {
-		dev_err(dev, "The DDP package file was not found or could not be read. Entering Safe Mode\n");
-		return;
+		dev_err(dev, "ice_init_hw during change of tx topology failed: %d\n",
+			err);
+		release_firmware(firmware);
+		return err;
 	}
 
-	/* request for firmware was successful. Download to device */
+	/* Download firmware to device */
 	ice_load_pkg(firmware, pf);
 	release_firmware(firmware);
+
+	return 0;
 }
 
 /**
@@ -4506,9 +4571,11 @@ static int ice_init_dev(struct ice_pf *pf)
 
 	ice_init_feature_support(pf);
 
-	ice_request_fw(pf);
+	err = ice_init_ddp_config(hw, pf);
+	if (err)
+		return err;
 
-	/* if ice_request_fw fails, ICE_FLAG_ADV_FEATURES bit won't be
+	/* if ice_init_ddp_config fails, ICE_FLAG_ADV_FEATURES bit won't be
 	 * set in pf->state, which will cause ice_is_safe_mode to return
 	 * true
 	 */
-- 
2.38.1

[PATCH net-next 4/5] ice: Add txbalancing devlink param

[Tony Nguyen]

From: Lukasz Czapnik <lukasz.czapnik@intel.com>

It was observed that Tx performance was inconsistent across all queues
and/or VSIs and that it was directly connected to existing 9-layer
topology of the Tx scheduler.

Introduce new private devlink param - txbalance. This parameter gives user
flexibility to choose the 5-layer transmit scheduler topology which helps
to smooth out the transmit performance.

Allowed parameter values are true for enabled and false for disabled.

Example usage:

Show:
devlink dev param show pci/0000:4b:00.0 name txbalancing
pci/0000:4b:00.0:
  name txbalancing type driver-specific
    values:
      cmode permanent value true

Set:
devlink dev param set pci/0000:4b:00.0 name txbalancing value true cmode
permanent

Signed-off-by: Lukasz Czapnik <lukasz.czapnik@intel.com>
Signed-off-by: Michal Wilczynski <michal.wilczynski@intel.com>
Tested-by: Pucha Himasekhar Reddy <himasekharx.reddy.pucha@intel.com> (A Contingent worker at Intel)
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
 .../net/ethernet/intel/ice/ice_adminq_cmd.h   |   9 +
 drivers/net/ethernet/intel/ice/ice_devlink.c  | 161 +++++++++++++++++-
 .../net/ethernet/intel/ice/ice_fw_update.c    |   2 +-
 .../net/ethernet/intel/ice/ice_fw_update.h    |   3 +
 drivers/net/ethernet/intel/ice/ice_nvm.c      |   2 +-
 drivers/net/ethernet/intel/ice/ice_nvm.h      |   3 +
 6 files changed, 177 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h b/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h
index d1a449b07fdf..8a624cef0eb0 100644
--- a/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h
+++ b/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h
@@ -1571,6 +1571,15 @@ struct ice_aqc_nvm {
 };
 
 #define ICE_AQC_NVM_START_POINT			0
+#define ICE_AQC_NVM_TX_TOPO_MOD_ID		0x14B
+
+struct ice_aqc_nvm_tx_topo_user_sel {
+	__le16 length;
+	u8 data;
+#define ICE_AQC_NVM_TX_TOPO_USER_SEL		BIT(4)
+
+	u8 reserved;
+};
 
 /* NVM Checksum Command (direct, 0x0706) */
 struct ice_aqc_nvm_checksum {
diff --git a/drivers/net/ethernet/intel/ice/ice_devlink.c b/drivers/net/ethernet/intel/ice/ice_devlink.c
index bc44cc220818..c716b1442d17 100644
--- a/drivers/net/ethernet/intel/ice/ice_devlink.c
+++ b/drivers/net/ethernet/intel/ice/ice_devlink.c
@@ -736,6 +736,159 @@ ice_devlink_port_unsplit(struct devlink *devlink, struct devlink_port *port,
 	return ice_devlink_port_split(devlink, port, 1, extack);
 }
 
+enum ice_devlink_param_id {
+	ICE_DEVLINK_PARAM_ID_BASE = DEVLINK_PARAM_GENERIC_ID_MAX,
+	ICE_DEVLINK_PARAM_ID_TX_BALANCE,
+};
+
+/**
+ * ice_get_tx_topo_user_sel - Read user's choice from flash
+ * @pf: pointer to pf structure
+ * @txbalance_ena: value read from flash will be saved here
+ *
+ * Reads user's preference for Tx Scheduler Topology Tree from PFA TLV.
+ *
+ * Returns zero when read was successful, negative values otherwise.
+ */
+static int ice_get_tx_topo_user_sel(struct ice_pf *pf, bool *txbalance_ena)
+{
+	struct ice_aqc_nvm_tx_topo_user_sel usr_sel = {};
+	struct ice_hw *hw = &pf->hw;
+	int status;
+
+	status = ice_acquire_nvm(hw, ICE_RES_READ);
+	if (status)
+		return status;
+
+	status = ice_aq_read_nvm(hw, ICE_AQC_NVM_TX_TOPO_MOD_ID, 0,
+				 sizeof(usr_sel), &usr_sel, true, true, NULL);
+	ice_release_nvm(hw);
+
+	*txbalance_ena = usr_sel.data & ICE_AQC_NVM_TX_TOPO_USER_SEL;
+
+	return status;
+}
+
+/**
+ * ice_update_tx_topo_user_sel - Save user's preference in flash
+ * @pf: pointer to pf structure
+ * @txbalance_ena: value to be saved in flash
+ *
+ * When txbalance_ena is set to true it means user's preference is to use
+ * five layer Tx Scheduler Topology Tree, when it is set to false then it is
+ * nine layer. This choice should be stored in PFA TLV field and should be
+ * picked up by driver, next time during init.
+ *
+ * Returns zero when save was successful, negative values otherwise.
+ */
+static int ice_update_tx_topo_user_sel(struct ice_pf *pf, bool txbalance_ena)
+{
+	struct ice_aqc_nvm_tx_topo_user_sel usr_sel = {};
+	struct ice_hw *hw = &pf->hw;
+	int err;
+
+	err = ice_acquire_nvm(hw, ICE_RES_WRITE);
+	if (err)
+		return err;
+
+	err = ice_aq_read_nvm(hw, ICE_AQC_NVM_TX_TOPO_MOD_ID, 0,
+			      sizeof(usr_sel), &usr_sel, true, true, NULL);
+	if (err)
+		goto exit_release_res;
+
+	if (txbalance_ena)
+		usr_sel.data |= ICE_AQC_NVM_TX_TOPO_USER_SEL;
+	else
+		usr_sel.data &= ~ICE_AQC_NVM_TX_TOPO_USER_SEL;
+
+	err = ice_write_one_nvm_block(pf, ICE_AQC_NVM_TX_TOPO_MOD_ID, 2,
+				      sizeof(usr_sel.data), &usr_sel.data,
+				      true, NULL, NULL);
+	if (err)
+		err = -EIO;
+
+exit_release_res:
+	ice_release_nvm(hw);
+
+	return err;
+}
+
+/**
+ * ice_devlink_txbalance_get - Get txbalance parameter
+ * @devlink: pointer to the devlink instance
+ * @id: the parameter ID to set
+ * @ctx: context to store the parameter value
+ *
+ * Returns zero on success and negative value on failure.
+ */
+static int ice_devlink_txbalance_get(struct devlink *devlink, u32 id,
+				     struct devlink_param_gset_ctx *ctx)
+{
+	struct ice_pf *pf = devlink_priv(devlink);
+	struct device *dev = ice_pf_to_dev(pf);
+	int status;
+
+	status = ice_get_tx_topo_user_sel(pf, &ctx->val.vbool);
+	if (status) {
+		dev_warn(dev, "Failed to read Tx Scheduler Tree - User Selection data from flash\n");
+		return -EIO;
+	}
+
+	return 0;
+}
+
+/**
+ * ice_devlink_txbalance_set - Set txbalance parameter
+ * @devlink: pointer to the devlink instance
+ * @id: the parameter ID to set
+ * @ctx: context to get the parameter value
+ *
+ * Returns zero on success and negative value on failure.
+ */
+static int ice_devlink_txbalance_set(struct devlink *devlink, u32 id,
+				     struct devlink_param_gset_ctx *ctx)
+{
+	struct ice_pf *pf = devlink_priv(devlink);
+	struct device *dev = ice_pf_to_dev(pf);
+	int status;
+
+	status = ice_update_tx_topo_user_sel(pf, ctx->val.vbool);
+	if (status)
+		return -EIO;
+
+	dev_warn(dev, "Transmit balancing setting has been changed on this device. You must reboot the system for the change to take effect");
+
+	return 0;
+}
+
+/**
+ * ice_devlink_txbalance_validate - Validate passed txbalance parameter value
+ * @devlink: unused pointer to devlink instance
+ * @id: the parameter ID to validate
+ * @val: value to validate
+ * @extack: netlink extended ACK structure
+ *
+ * Supported values are:
+ * true - five layer, false - nine layer Tx Scheduler Topology Tree
+ *
+ * Returns zero when passed parameter value is supported. Negative value on
+ * error.
+ */
+static int ice_devlink_txbalance_validate(struct devlink *devlink, u32 id,
+					  union devlink_param_value val,
+					  struct netlink_ext_ack *extack)
+{
+	struct ice_pf *pf = devlink_priv(devlink);
+	struct ice_hw *hw = &pf->hw;
+
+	if (!hw->func_caps.common_cap.tx_sched_topo_comp_mode_en) {
+		NL_SET_ERR_MSG_MOD(extack, "Error: Requested feature is not supported by the FW on this device. Update the FW and run this command again.");
+		return -EOPNOTSUPP;
+	}
+
+	return 0;
+}
+
 /**
  * ice_tear_down_devlink_rate_tree - removes devlink-rate exported tree
  * @pf: pf struct
@@ -1391,7 +1544,13 @@ static const struct devlink_param ice_devlink_params[] = {
 			      ice_devlink_enable_iw_get,
 			      ice_devlink_enable_iw_set,
 			      ice_devlink_enable_iw_validate),
-
+	DEVLINK_PARAM_DRIVER(ICE_DEVLINK_PARAM_ID_TX_BALANCE,
+			     "txbalancing",
+			     DEVLINK_PARAM_TYPE_BOOL,
+			     BIT(DEVLINK_PARAM_CMODE_PERMANENT),
+			     ice_devlink_txbalance_get,
+			     ice_devlink_txbalance_set,
+			     ice_devlink_txbalance_validate),
 };
 
 static void ice_devlink_free(void *devlink_ptr)
diff --git a/drivers/net/ethernet/intel/ice/ice_fw_update.c b/drivers/net/ethernet/intel/ice/ice_fw_update.c
index 3dc5662d62a6..dc5b49104674 100644
--- a/drivers/net/ethernet/intel/ice/ice_fw_update.c
+++ b/drivers/net/ethernet/intel/ice/ice_fw_update.c
@@ -286,7 +286,7 @@ ice_send_component_table(struct pldmfw *context, struct pldmfw_component *compon
  *
  * Returns: zero on success, or a negative error code on failure.
  */
-static int
+int
 ice_write_one_nvm_block(struct ice_pf *pf, u16 module, u32 offset,
 			u16 block_size, u8 *block, bool last_cmd,
 			u8 *reset_level, struct netlink_ext_ack *extack)
diff --git a/drivers/net/ethernet/intel/ice/ice_fw_update.h b/drivers/net/ethernet/intel/ice/ice_fw_update.h
index 750574885716..04b200462757 100644
--- a/drivers/net/ethernet/intel/ice/ice_fw_update.h
+++ b/drivers/net/ethernet/intel/ice/ice_fw_update.h
@@ -9,5 +9,8 @@ int ice_devlink_flash_update(struct devlink *devlink,
 			     struct netlink_ext_ack *extack);
 int ice_get_pending_updates(struct ice_pf *pf, u8 *pending,
 			    struct netlink_ext_ack *extack);
+int ice_write_one_nvm_block(struct ice_pf *pf, u16 module, u32 offset,
+			    u16 block_size, u8 *block, bool last_cmd,
+			    u8 *reset_level, struct netlink_ext_ack *extack);
 
 #endif
diff --git a/drivers/net/ethernet/intel/ice/ice_nvm.c b/drivers/net/ethernet/intel/ice/ice_nvm.c
index f6f52a248066..745f2459943f 100644
--- a/drivers/net/ethernet/intel/ice/ice_nvm.c
+++ b/drivers/net/ethernet/intel/ice/ice_nvm.c
@@ -18,7 +18,7 @@
  *
  * Read the NVM using the admin queue commands (0x0701)
  */
-static int
+int
 ice_aq_read_nvm(struct ice_hw *hw, u16 module_typeid, u32 offset, u16 length,
 		void *data, bool last_command, bool read_shadow_ram,
 		struct ice_sq_cd *cd)
diff --git a/drivers/net/ethernet/intel/ice/ice_nvm.h b/drivers/net/ethernet/intel/ice/ice_nvm.h
index 774c2317967d..88978b9a95b1 100644
--- a/drivers/net/ethernet/intel/ice/ice_nvm.h
+++ b/drivers/net/ethernet/intel/ice/ice_nvm.h
@@ -14,6 +14,9 @@ struct ice_orom_civd_info {
 
 int ice_acquire_nvm(struct ice_hw *hw, enum ice_aq_res_access_type access);
 void ice_release_nvm(struct ice_hw *hw);
+int ice_aq_read_nvm(struct ice_hw *hw, u16 module_typeid, u32 offset, u16 length,
+		    void *data, bool last_command, bool read_shadow_ram,
+		    struct ice_sq_cd *cd);
 int
 ice_read_flat_nvm(struct ice_hw *hw, u32 offset, u32 *length, u8 *data,
 		  bool read_shadow_ram);
-- 
2.38.1

Re: [PATCH net-next 4/5] ice: Add txbalancing devlink param

[Jiri Pirko]

Tue, May 23, 2023 at 07:40:07PM CEST, anthony.l.nguyen@intel.com wrote:
>From: Lukasz Czapnik <lukasz.czapnik@intel.com>
>
>It was observed that Tx performance was inconsistent across all queues
>and/or VSIs and that it was directly connected to existing 9-layer
>topology of the Tx scheduler.
>
>Introduce new private devlink param - txbalance. This parameter gives user
>flexibility to choose the 5-layer transmit scheduler topology which helps
>to smooth out the transmit performance.
>
>Allowed parameter values are true for enabled and false for disabled.
>
>Example usage:
>
>Show:
>devlink dev param show pci/0000:4b:00.0 name txbalancing
>pci/0000:4b:00.0:
>  name txbalancing type driver-specific
>    values:
>      cmode permanent value true

"TXbalancing" sounds quite generic. Also, TXbalancing==false might sound
there is no tx balancing. That is confusing.

Re: [PATCH net-next 4/5] ice: Add txbalancing devlink param

[Lukasz Czapnik]

On 5/24/2023 1:57 PM, Jiri Pirko wrote:
> Tue, May 23, 2023 at 07:40:07PM CEST, anthony.l.nguyen@intel.com wrote:
>> From: Lukasz Czapnik <lukasz.czapnik@intel.com>
>>
>> It was observed that Tx performance was inconsistent across all queues
>> and/or VSIs and that it was directly connected to existing 9-layer
>> topology of the Tx scheduler.
>>
>> Introduce new private devlink param - txbalance. This parameter gives user
>> flexibility to choose the 5-layer transmit scheduler topology which helps
>> to smooth out the transmit performance.
>>
>> Allowed parameter values are true for enabled and false for disabled.
>>
>> Example usage:
>>
>> Show:
>> devlink dev param show pci/0000:4b:00.0 name txbalancing
>> pci/0000:4b:00.0:
>>   name txbalancing type driver-specific
>>     values:
>>       cmode permanent value true
> 
> "TXbalancing" sounds quite generic. Also, TXbalancing==false might sound
> there is no tx balancing. That is confusing.
> 

Right, agree this can be confusing. Also discussion under other patch
suggest txbalancing is not the best name for this feature.

We'll rename it. Open to suggestions.

Thank you!

[PATCH net-next 5/5] ice: Document txbalancing parameter

[Tony Nguyen]

From: Michal Wilczynski <michal.wilczynski@intel.com>

New driver specific parameter 'txbalancing' was introduced.
Describe parameter in the documentation.

Signed-off-by: Michal Wilczynski <michal.wilczynski@intel.com>
Tested-by: Pucha Himasekhar Reddy <himasekharx.reddy.pucha@intel.com> (A Contingent worker at Intel)
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
 Documentation/networking/devlink/ice.rst | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/Documentation/networking/devlink/ice.rst b/Documentation/networking/devlink/ice.rst
index 2f60e34ab926..298df8ec1930 100644
--- a/Documentation/networking/devlink/ice.rst
+++ b/Documentation/networking/devlink/ice.rst
@@ -22,6 +22,23 @@ Parameters
      - runtime
      - mutually exclusive with ``enable_roce``
 
+.. list-table:: Driver-specific parameters implemented
+   :widths: 5 5 5 85
+
+   * - Name
+     - Type
+     - Mode
+     - Description
+   * - ``txbalancing``
+     - Boolean
+     - permanent
+     - This parameter gives user flexibility to choose the 5-layer
+       transmit scheduler topology, which helps to smooth out the transmit
+       performance. The default topology is 9-layer. Each layer represents
+       a physical junction in the network. Decreased number of layers
+       improves performance, but at the same time number of network junctions
+       is reduced, which might not be desirable depending on the use case.
+
 Info versions
 =============
 
-- 
2.38.1

Re: [PATCH net-next 0/5][pull request] ice: Support 5 layer Tx scheduler topology

[Jiri Pirko]

Tue, May 23, 2023 at 07:40:03PM CEST, anthony.l.nguyen@intel.com wrote:
>Michal Wilczynski says:
>
>For performance reasons there is a need to have support for selectable
>Tx scheduler topology. Currently firmware supports only the default
>9-layer and 5-layer topology. This patch series enables switch from
>default to 5-layer topology, if user decides to opt-in.

Why exactly the user cares which FW implementation you use. From what I
see, there is a FW but causing unequal queue distribution in some cases,
you fox this. Why would the user want to alter the behaviour between
fixed and unfixed?


>
>The following are changes since commit b2e3406a38f0f48b1dfb81e5bb73d243ff6af179:
>  octeontx2-pf: Add support for page pool
>and are available in the git repository at:
>  git://git.kernel.org/pub/scm/linux/kernel/git/tnguy/next-queue 100GbE
>
>Lukasz Czapnik (1):
>  ice: Add txbalancing devlink param
>
>Michal Wilczynski (2):
>  ice: Enable switching default tx scheduler topology
>  ice: Document txbalancing parameter
>
>Raj Victor (2):
>  ice: Support 5 layer topology
>  ice: Adjust the VSI/Aggregator layers
>
> Documentation/networking/devlink/ice.rst      |  17 ++
> .../net/ethernet/intel/ice/ice_adminq_cmd.h   |  31 +++
> drivers/net/ethernet/intel/ice/ice_common.c   |   6 +
> drivers/net/ethernet/intel/ice/ice_ddp.c      | 200 ++++++++++++++++++
> drivers/net/ethernet/intel/ice/ice_ddp.h      |   7 +-
> drivers/net/ethernet/intel/ice/ice_devlink.c  | 161 +++++++++++++-
> .../net/ethernet/intel/ice/ice_fw_update.c    |   2 +-
> .../net/ethernet/intel/ice/ice_fw_update.h    |   3 +
> drivers/net/ethernet/intel/ice/ice_main.c     | 105 +++++++--
> drivers/net/ethernet/intel/ice/ice_nvm.c      |   2 +-
> drivers/net/ethernet/intel/ice/ice_nvm.h      |   3 +
> drivers/net/ethernet/intel/ice/ice_sched.c    |  34 +--
> drivers/net/ethernet/intel/ice/ice_sched.h    |   3 +
> drivers/net/ethernet/intel/ice/ice_type.h     |   1 +
> 14 files changed, 534 insertions(+), 41 deletions(-)
>
>-- 
>2.38.1
>

Re: [PATCH net-next 0/5][pull request] ice: Support 5 layer Tx scheduler topology

[Wilczynski, Michal]

On 5/24/2023 1:54 PM, Jiri Pirko wrote:
> Tue, May 23, 2023 at 07:40:03PM CEST, anthony.l.nguyen@intel.com wrote:
>> Michal Wilczynski says:
>>
>> For performance reasons there is a need to have support for selectable
>> Tx scheduler topology. Currently firmware supports only the default
>> 9-layer and 5-layer topology. This patch series enables switch from
>> default to 5-layer topology, if user decides to opt-in.
> Why exactly the user cares which FW implementation you use. From what I
> see, there is a FW but causing unequal queue distribution in some cases,
> you fox this. Why would the user want to alter the behaviour between
> fixed and unfixed?

I wouldn't say it's a FW bug. Both approaches - 9-layer and 5-layer have their own pros
and cons, and in some cases 5-layer is preferable, especially if the user desires the performance
to be better. But at the same time the user gives up the layers in a tree that are actually useful in
some cases (especially if using DCB, but also recently added devlink-rate implementation).

Regards,
Michał

>
>
>> The following are changes since commit b2e3406a38f0f48b1dfb81e5bb73d243ff6af179:
>>  octeontx2-pf: Add support for page pool
>> and are available in the git repository at:
>>  git://git.kernel.org/pub/scm/linux/kernel/git/tnguy/next-queue 100GbE
>>
>> Lukasz Czapnik (1):
>>  ice: Add txbalancing devlink param
>>
>> Michal Wilczynski (2):
>>  ice: Enable switching default tx scheduler topology
>>  ice: Document txbalancing parameter
>>
>> Raj Victor (2):
>>  ice: Support 5 layer topology
>>  ice: Adjust the VSI/Aggregator layers
>>
>> Documentation/networking/devlink/ice.rst      |  17 ++
>> .../net/ethernet/intel/ice/ice_adminq_cmd.h   |  31 +++
>> drivers/net/ethernet/intel/ice/ice_common.c   |   6 +
>> drivers/net/ethernet/intel/ice/ice_ddp.c      | 200 ++++++++++++++++++
>> drivers/net/ethernet/intel/ice/ice_ddp.h      |   7 +-
>> drivers/net/ethernet/intel/ice/ice_devlink.c  | 161 +++++++++++++-
>> .../net/ethernet/intel/ice/ice_fw_update.c    |   2 +-
>> .../net/ethernet/intel/ice/ice_fw_update.h    |   3 +
>> drivers/net/ethernet/intel/ice/ice_main.c     | 105 +++++++--
>> drivers/net/ethernet/intel/ice/ice_nvm.c      |   2 +-
>> drivers/net/ethernet/intel/ice/ice_nvm.h      |   3 +
>> drivers/net/ethernet/intel/ice/ice_sched.c    |  34 +--
>> drivers/net/ethernet/intel/ice/ice_sched.h    |   3 +
>> drivers/net/ethernet/intel/ice/ice_type.h     |   1 +
>> 14 files changed, 534 insertions(+), 41 deletions(-)
>>
>> -- 
>> 2.38.1
>>

Re: [PATCH net-next 0/5][pull request] ice: Support 5 layer Tx scheduler topology

[Jakub Kicinski]

On Wed, 24 May 2023 15:25:22 +0200 Wilczynski, Michal wrote:
> >> For performance reasons there is a need to have support for selectable
> >> Tx scheduler topology. Currently firmware supports only the default
> >> 9-layer and 5-layer topology. This patch series enables switch from
> >> default to 5-layer topology, if user decides to opt-in.  
> > Why exactly the user cares which FW implementation you use. From what I
> > see, there is a FW but causing unequal queue distribution in some cases,
> > you fox this. Why would the user want to alter the behaviour between
> > fixed and unfixed?  
> 
> I wouldn't say it's a FW bug. Both approaches - 9-layer and 5-layer
> have their own pros and cons, and in some cases 5-layer is
> preferable, especially if the user desires the performance to be
> better. But at the same time the user gives up the layers in a tree
> that are actually useful in some cases (especially if using DCB, but
> also recently added devlink-rate implementation).

I didn't notice mentions of DCB and devlink-rate in the series.
The whole thing is really poorly explained.

Re: [PATCH net-next 0/5][pull request] ice: Support 5 layer Tx scheduler topology

[Wilczynski, Michal]

On 5/24/2023 6:26 PM, Jakub Kicinski wrote:
> On Wed, 24 May 2023 15:25:22 +0200 Wilczynski, Michal wrote:
>>>> For performance reasons there is a need to have support for selectable
>>>> Tx scheduler topology. Currently firmware supports only the default
>>>> 9-layer and 5-layer topology. This patch series enables switch from
>>>> default to 5-layer topology, if user decides to opt-in.  
>>> Why exactly the user cares which FW implementation you use. From what I
>>> see, there is a FW but causing unequal queue distribution in some cases,
>>> you fox this. Why would the user want to alter the behaviour between
>>> fixed and unfixed?  
>> I wouldn't say it's a FW bug. Both approaches - 9-layer and 5-layer
>> have their own pros and cons, and in some cases 5-layer is
>> preferable, especially if the user desires the performance to be
>> better. But at the same time the user gives up the layers in a tree
>> that are actually useful in some cases (especially if using DCB, but
>> also recently added devlink-rate implementation).
> I didn't notice mentions of DCB and devlink-rate in the series.
> The whole thing is really poorly explained.

Sorry about that, I gave examples from the top of my head, since those are the
features that potentially could modify the scheduler tree, seemed obvious to me
at the time. Lowering number of layers in the scheduling tree increases performance,
but only allows you to create a much simpler scheduling tree. I agree that mentioning the
features that actually modify the scheduling tree could be helpful to the reviewer.

Regards,
Michał

Re: [PATCH net-next 0/5][pull request] ice: Support 5 layer Tx scheduler topology

[Jakub Kicinski]

On Wed, 24 May 2023 18:59:20 +0200 Wilczynski, Michal wrote:
>  [...]  
> >> I wouldn't say it's a FW bug. Both approaches - 9-layer and 5-layer
> >> have their own pros and cons, and in some cases 5-layer is
> >> preferable, especially if the user desires the performance to be
> >> better. But at the same time the user gives up the layers in a tree
> >> that are actually useful in some cases (especially if using DCB, but
> >> also recently added devlink-rate implementation).  
> > I didn't notice mentions of DCB and devlink-rate in the series.
> > The whole thing is really poorly explained.  
> 
> Sorry about that, I gave examples from the top of my head, since those are the
> features that potentially could modify the scheduler tree, seemed obvious to me
> at the time. Lowering number of layers in the scheduling tree increases performance,
> but only allows you to create a much simpler scheduling tree. I agree that mentioning the
> features that actually modify the scheduling tree could be helpful to the reviewer.

Reviewer is one thing, but also the user. The documentation needs to be
clear enough for the user to be able to confidently make a choice one
way or the other. I'm not sure 5- vs 9-layer is meaningful to the user
at all. In fact, the entire configuration would be better defined as
a choice of features user wants to be available and the FW || driver
makes the decision on how to implement that most efficiently.

Re: [PATCH net-next 0/5][pull request] ice: Support 5 layer Tx scheduler topology

[Wilczynski, Michal]

On 5/24/2023 10:02 PM, Jakub Kicinski wrote:
> On Wed, 24 May 2023 18:59:20 +0200 Wilczynski, Michal wrote:
>>  [...]  
>>>> I wouldn't say it's a FW bug. Both approaches - 9-layer and 5-layer
>>>> have their own pros and cons, and in some cases 5-layer is
>>>> preferable, especially if the user desires the performance to be
>>>> better. But at the same time the user gives up the layers in a tree
>>>> that are actually useful in some cases (especially if using DCB, but
>>>> also recently added devlink-rate implementation).  
>>> I didn't notice mentions of DCB and devlink-rate in the series.
>>> The whole thing is really poorly explained.  
>> Sorry about that, I gave examples from the top of my head, since those are the
>> features that potentially could modify the scheduler tree, seemed obvious to me
>> at the time. Lowering number of layers in the scheduling tree increases performance,
>> but only allows you to create a much simpler scheduling tree. I agree that mentioning the
>> features that actually modify the scheduling tree could be helpful to the reviewer.
> Reviewer is one thing, but also the user. The documentation needs to be
> clear enough for the user to be able to confidently make a choice one
> way or the other. I'm not sure 5- vs 9-layer is meaningful to the user
> at all.

It is relevant especially if the number of VF's/queues is not a multiply of 8, as described
in the first commit of this series - that's the real-world user problem. Performance was
not consistent among queues if you had 9 queues for example.

But I was also trying to provide some background on why we don't want to make 5-layer
topology the default in the answers above.


>  In fact, the entire configuration would be better defined as
> a choice of features user wants to be available and the FW || driver
> makes the decision on how to implement that most efficiently.

User can change number of queues/VF's 'on the fly' , but change in topology
requires a reboot basically, since the contents of the NVM are changed.

So to accomplish that we would need to perform topology change after each
change to number of queues to adapt, and it's not feasible to reboot every time
user changes number of queues.

Additionally 5-layer topology doesn't disable any of the features mentioned
(i.e. DCB/devlink-rate) it just makes them work a bit differently, but they still
should work.

To summarize: I would say that this series address specific performance problem
user might have if their queue count is not a power of 8. I can't see how this can
be solved by a choice of features, as the decision regarding number of queues can
be made 'on-the-fly'.

Regards,
Michał

Re: [PATCH net-next 0/5][pull request] ice: Support 5 layer Tx scheduler topology

[Jakub Kicinski]

On Thu, 25 May 2023 09:49:53 +0200 Wilczynski, Michal wrote:
> On 5/24/2023 10:02 PM, Jakub Kicinski wrote:
> > On Wed, 24 May 2023 18:59:20 +0200 Wilczynski, Michal wrote:  
> >> Sorry about that, I gave examples from the top of my head, since those are the
> >> features that potentially could modify the scheduler tree, seemed obvious to me
> >> at the time. Lowering number of layers in the scheduling tree increases performance,
> >> but only allows you to create a much simpler scheduling tree. I agree that mentioning the
> >> features that actually modify the scheduling tree could be helpful to the reviewer.  
> > Reviewer is one thing, but also the user. The documentation needs to be
> > clear enough for the user to be able to confidently make a choice one
> > way or the other. I'm not sure 5- vs 9-layer is meaningful to the user
> > at all.  
> 
> It is relevant especially if the number of VF's/queues is not a multiply of 8, as described
> in the first commit of this series - that's the real-world user problem. Performance was
> not consistent among queues if you had 9 queues for example.
> 
> But I was also trying to provide some background on why we don't want to make 5-layer
> topology the default in the answers above.

What I'm saying is that 5- vs 9-layer is not meaningful as 
a description. The user has to (somehow?!) know that the number 
of layers in the hierarchy implies the grouping problem.
The documentation doesn't mention the grouping problem!

+     - This parameter gives user flexibility to choose the 5-layer
+       transmit scheduler topology, which helps to smooth out the transmit
+       performance. The default topology is 9-layer. Each layer represents
+       a physical junction in the network. Decreased number of layers
+       improves performance, but at the same time number of network junctions
+       is reduced, which might not be desirable depending on the use case.

> >  In fact, the entire configuration would be better defined as
> > a choice of features user wants to be available and the FW || driver
> > makes the decision on how to implement that most efficiently.  
> 
> User can change number of queues/VF's 'on the fly' , but change in topology
> requires a reboot basically, since the contents of the NVM are changed.
> 
> So to accomplish that we would need to perform topology change after each
> change to number of queues to adapt, and it's not feasible to reboot every time
> user changes number of queues.
> 
> Additionally 5-layer topology doesn't disable any of the features mentioned
> (i.e. DCB/devlink-rate) it just makes them work a bit differently, but they still
> should work.
> 
> To summarize: I would say that this series address specific performance problem
> user might have if their queue count is not a power of 8. I can't see how this can
> be solved by a choice of features, as the decision regarding number of queues can
> be made 'on-the-fly'.

Well, think among yourselves. "txbalancing" and a enigmatic
documentation talking about topology and junctions is a no go.

Re: [PATCH net-next 0/5][pull request] ice: Support 5 layer Tx scheduler topology

[Wilczynski, Michal]

On 5/25/2023 5:41 PM, Jakub Kicinski wrote:
> On Thu, 25 May 2023 09:49:53 +0200 Wilczynski, Michal wrote:
>> On 5/24/2023 10:02 PM, Jakub Kicinski wrote:
>>> On Wed, 24 May 2023 18:59:20 +0200 Wilczynski, Michal wrote:  
>>>> Sorry about that, I gave examples from the top of my head, since those are the
>>>> features that potentially could modify the scheduler tree, seemed obvious to me
>>>> at the time. Lowering number of layers in the scheduling tree increases performance,
>>>> but only allows you to create a much simpler scheduling tree. I agree that mentioning the
>>>> features that actually modify the scheduling tree could be helpful to the reviewer.  
>>> Reviewer is one thing, but also the user. The documentation needs to be
>>> clear enough for the user to be able to confidently make a choice one
>>> way or the other. I'm not sure 5- vs 9-layer is meaningful to the user
>>> at all.  
>> It is relevant especially if the number of VF's/queues is not a multiply of 8, as described
>> in the first commit of this series - that's the real-world user problem. Performance was
>> not consistent among queues if you had 9 queues for example.
>>
>> But I was also trying to provide some background on why we don't want to make 5-layer
>> topology the default in the answers above.
> What I'm saying is that 5- vs 9-layer is not meaningful as 
> a description. The user has to (somehow?!) know that the number 
> of layers in the hierarchy implies the grouping problem.
> The documentation doesn't mention the grouping problem!
>
> +     - This parameter gives user flexibility to choose the 5-layer
> +       transmit scheduler topology, which helps to smooth out the transmit
> +       performance. The default topology is 9-layer. Each layer represents
> +       a physical junction in the network. Decreased number of layers
> +       improves performance, but at the same time number of network junctions
> +       is reduced, which might not be desirable depending on the use case.
>
>>>  In fact, the entire configuration would be better defined as
>>> a choice of features user wants to be available and the FW || driver
>>> makes the decision on how to implement that most efficiently.  
>> User can change number of queues/VF's 'on the fly' , but change in topology
>> requires a reboot basically, since the contents of the NVM are changed.
>>
>> So to accomplish that we would need to perform topology change after each
>> change to number of queues to adapt, and it's not feasible to reboot every time
>> user changes number of queues.
>>
>> Additionally 5-layer topology doesn't disable any of the features mentioned
>> (i.e. DCB/devlink-rate) it just makes them work a bit differently, but they still
>> should work.
>>
>> To summarize: I would say that this series address specific performance problem
>> user might have if their queue count is not a power of 8. I can't see how this can
>> be solved by a choice of features, as the decision regarding number of queues can
>> be made 'on-the-fly'.
> Well, think among yourselves. "txbalancing" and a enigmatic
> documentation talking about topology and junctions is a no go.


Sure, thank you for your feedback, we'll fix the documentation and figure out a better
name instead of "txbalancing"