[PATCH net 0/2] ixgbe: Multiple RSS bugfixes

[PATCH net 0/2] ixgbe: Multiple RSS bugfixes

[Joe Damato]

Greetings:

This series fixes two bugs I stumbled on with ixgbe:

1. The flow hash cannot be set manually with ethool at all. Patch 1/2
addresses this by fixing what appears to be a small bug in set_rxfh in
ixgbe. See the commit message for more details.

2. Once the above patch is applied and the flow hash can be set,
resetting the flow hash to default fails if the number of queues is
greater than the number of queues supported by RSS. Other drivers (like
i40e) will reset the flowhash to use the maximum number of queues
supported by RSS even if the queue count configured is larger. In other
words: some queues will not have packets distributed to them by the RSS
hash if the queue count is too large. Patch 2/2 allows the user to reset
ixgbe to default and the flowhash is set correctly to either the
maximum number of queues supported by RSS or the configured queue count,
whichever is smaller.

I believe this is correct and it mimics the behavior of i40e;
`ethtool -X $iface default` should probably always succeed even if all the
queues cannot be utilized. See the commit message for more details and
examples.

I tested these on an ixgbe system I have access to and they appear to
work as intended, but I would appreciate a review by the experts on this
list :)

Thanks,
Joe

Joe Damato (2):
  ixgbe: Allow flow hash to be set via ethtool
  ixgbe: Allow ixgbe to reset default flow hash

 .../net/ethernet/intel/ixgbe/ixgbe_ethtool.c  | 23 ++++++++++---------
 1 file changed, 12 insertions(+), 11 deletions(-)

-- 
2.25.1

[PATCH net 1/2] ixgbe: Allow flow hash to be set via ethtool

[Joe Damato]

ixgbe currently returns `EINVAL` whenever the flowhash it set by ethtool
because the ethtool code in the kernel passes a non-zero value for hfunc
that ixgbe should allow.

When ethtool is called with `ETHTOOL_SRXFHINDIR`,
`ethtool_set_rxfh_indir` will call ixgbe's set_rxfh function
with `ETH_RSS_HASH_NO_CHANGE`. This value should be accepted.

When ethtool is called with `ETHTOOL_SRSSH`, `ethtool_set_rxfh` will
call ixgbe's set_rxfh function with `rxfh.hfunc`, which appears to be
hardcoded in ixgbe to always be `ETH_RSS_HASH_TOP`. This value should
also be accepted.

Before this patch:

$ sudo ethtool -L eth1 combined 10
$ sudo ethtool -X eth1 default
Cannot set RX flow hash configuration: Invalid argument

After this patch:

$ sudo ethtool -L eth1 combined 10
$ sudo ethtool -X eth1 default
$ sudo ethtool -x eth1
RX flow hash indirection table for eth1 with 10 RX ring(s):
    0:      0     1     2     3     4     5     6     7
    8:      8     9     0     1     2     3     4     5
   16:      6     7     8     9     0     1     2     3
   24:      4     5     6     7     8     9     0     1
   ...

Signed-off-by: Joe Damato <jdamato@fastly.com>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
index 6cfc9dc16537..821dfd323fa9 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
@@ -3131,8 +3131,8 @@ static int ixgbe_set_rxfh(struct net_device *netdev, const u32 *indir,
 	int i;
 	u32 reta_entries = ixgbe_rss_indir_tbl_entries(adapter);
 
-	if (hfunc)
-		return -EINVAL;
+	if (hfunc != ETH_RSS_HASH_NO_CHANGE && hfunc != ETH_RSS_HASH_TOP)
+		return -EOPNOTSUPP;
 
 	/* Fill out the redirection table */
 	if (indir) {
-- 
2.25.1

Re: [PATCH net 1/2] ixgbe: Allow flow hash to be set via ethtool

[Samudrala, Sridhar]

On 4/15/2023 12:48 AM, Joe Damato wrote:
> ixgbe currently returns `EINVAL` whenever the flowhash it set by ethtool
> because the ethtool code in the kernel passes a non-zero value for hfunc
> that ixgbe should allow.
> 
> When ethtool is called with `ETHTOOL_SRXFHINDIR`,
> `ethtool_set_rxfh_indir` will call ixgbe's set_rxfh function
> with `ETH_RSS_HASH_NO_CHANGE`. This value should be accepted.
> 
> When ethtool is called with `ETHTOOL_SRSSH`, `ethtool_set_rxfh` will
> call ixgbe's set_rxfh function with `rxfh.hfunc`, which appears to be
> hardcoded in ixgbe to always be `ETH_RSS_HASH_TOP`. This value should
> also be accepted.
> 
> Before this patch:
> 
> $ sudo ethtool -L eth1 combined 10
> $ sudo ethtool -X eth1 default
> Cannot set RX flow hash configuration: Invalid argument
> 
> After this patch:
> 
> $ sudo ethtool -L eth1 combined 10
> $ sudo ethtool -X eth1 default
> $ sudo ethtool -x eth1
> RX flow hash indirection table for eth1 with 10 RX ring(s):
>      0:      0     1     2     3     4     5     6     7
>      8:      8     9     0     1     2     3     4     5
>     16:      6     7     8     9     0     1     2     3
>     24:      4     5     6     7     8     9     0     1
>     ...
> 
> Signed-off-by: Joe Damato <jdamato@fastly.com>

Reviewed-by: Sridhar Samudrala <sridhar.samudrala@intel.com>

> ---
>   drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
> index 6cfc9dc16537..821dfd323fa9 100644
> --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
> +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
> @@ -3131,8 +3131,8 @@ static int ixgbe_set_rxfh(struct net_device *netdev, const u32 *indir,
>   	int i;
>   	u32 reta_entries = ixgbe_rss_indir_tbl_entries(adapter);
>   
> -	if (hfunc)
> -		return -EINVAL;
> +	if (hfunc != ETH_RSS_HASH_NO_CHANGE && hfunc != ETH_RSS_HASH_TOP)
> +		return -EOPNOTSUPP;
>   
>   	/* Fill out the redirection table */
>   	if (indir) {

[PATCH net 2/2] ixgbe: Allow ixgbe to reset default flow hash

[Joe Damato]

ethtool uses `ETHTOOL_GRXRINGS` to compute how many queues are supported
by RSS. The driver should return the smaller of either:
  - The maximum number of RSS queues the device supports, OR
  - The number of RX queues configured

Prior to this change, running `ethtool -X $iface default` fails if the
number of queues configured is larger than the number supported by RSS,
even though changing the queue count correctly resets the flowhash to
use all supported queues.

Other drivers (for example, i40e) will succeed but the flow hash will
reset to support the maximum number of queues supported by RSS, even if
that amount is smaller than the configured amount.

Prior to this change:

$ sudo ethtool -L eth1 combined 20
$ sudo ethtool -x eth1
RX flow hash indirection table for eth1 with 20 RX ring(s):
    0:      0     1     2     3     4     5     6     7
    8:      8     9    10    11    12    13    14    15
   16:      0     1     2     3     4     5     6     7
   24:      8     9    10    11    12    13    14    15
   32:      0     1     2     3     4     5     6     7
...

You can see that the flowhash was correctly set to use the maximum
number of queues supported by the driver (16).

However, asking the NIC to reset to "default" fails:

$ sudo ethtool -X eth1 default
Cannot set RX flow hash configuration: Invalid argument

After this change, the flowhash can be reset to default which will use
all of the available RSS queues (16) or the configured queue count,
whichever is smaller.

Starting with eth1 which has 10 queues and a flowhash distributing to
all 10 queues:

$ sudo ethtool -x eth1
RX flow hash indirection table for eth1 with 10 RX ring(s):
    0:      0     1     2     3     4     5     6     7
    8:      8     9     0     1     2     3     4     5
   16:      6     7     8     9     0     1     2     3
...

Increasing the queue count to 48 resets the flowhash to distribute to 16
queues, as it did before this patch:

$ sudo ethtool -L eth1 combined 48
$ sudo ethtool -x eth1
RX flow hash indirection table for eth1 with 16 RX ring(s):
    0:      0     1     2     3     4     5     6     7
    8:      8     9    10    11    12    13    14    15
   16:      0     1     2     3     4     5     6     7
...

Due to the other bugfix in this series, the flowhash can be set to use
queues 0-5:

$ sudo ethtool -X eth1 equal 5
$ sudo ethtool -x eth1
RX flow hash indirection table for eth1 with 16 RX ring(s):
    0:      0     1     2     3     4     0     1     2
    8:      3     4     0     1     2     3     4     0
   16:      1     2     3     4     0     1     2     3
...

Due to this bugfix, the flowhash can be reset to default and use 16
queues:

$ sudo ethtool -X eth1 default
$ sudo ethtool -x eth1
RX flow hash indirection table for eth1 with 16 RX ring(s):
    0:      0     1     2     3     4     5     6     7
    8:      8     9    10    11    12    13    14    15
   16:      0     1     2     3     4     5     6     7
...

Signed-off-by: Joe Damato <jdamato@fastly.com>
---
 .../net/ethernet/intel/ixgbe/ixgbe_ethtool.c  | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
index 821dfd323fa9..0bbad4a5cc2f 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
@@ -2665,6 +2665,14 @@ static int ixgbe_get_rss_hash_opts(struct ixgbe_adapter *adapter,
 	return 0;
 }
 
+static int ixgbe_rss_indir_tbl_max(struct ixgbe_adapter *adapter)
+{
+	if (adapter->hw.mac.type < ixgbe_mac_X550)
+		return 16;
+	else
+		return 64;
+}
+
 static int ixgbe_get_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd,
 			   u32 *rule_locs)
 {
@@ -2673,7 +2681,8 @@ static int ixgbe_get_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd,
 
 	switch (cmd->cmd) {
 	case ETHTOOL_GRXRINGS:
-		cmd->data = adapter->num_rx_queues;
+		cmd->data = min_t(int, adapter->num_rx_queues,
+				  ixgbe_rss_indir_tbl_max(adapter));
 		ret = 0;
 		break;
 	case ETHTOOL_GRXCLSRLCNT:
@@ -3075,14 +3084,6 @@ static int ixgbe_set_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd)
 	return ret;
 }
 
-static int ixgbe_rss_indir_tbl_max(struct ixgbe_adapter *adapter)
-{
-	if (adapter->hw.mac.type < ixgbe_mac_X550)
-		return 16;
-	else
-		return 64;
-}
-
 static u32 ixgbe_get_rxfh_key_size(struct net_device *netdev)
 {
 	return IXGBE_RSS_KEY_SIZE;
-- 
2.25.1

Re: [PATCH net 2/2] ixgbe: Allow ixgbe to reset default flow hash

[Samudrala, Sridhar]

On 4/15/2023 12:48 AM, Joe Damato wrote:
> ethtool uses `ETHTOOL_GRXRINGS` to compute how many queues are supported
> by RSS. The driver should return the smaller of either:
>    - The maximum number of RSS queues the device supports, OR
>    - The number of RX queues configured
> 
> Prior to this change, running `ethtool -X $iface default` fails if the
> number of queues configured is larger than the number supported by RSS,
> even though changing the queue count correctly resets the flowhash to
> use all supported queues.
> 
> Other drivers (for example, i40e) will succeed but the flow hash will
> reset to support the maximum number of queues supported by RSS, even if
> that amount is smaller than the configured amount.
> 
> Prior to this change:
> 
> $ sudo ethtool -L eth1 combined 20
> $ sudo ethtool -x eth1
> RX flow hash indirection table for eth1 with 20 RX ring(s):
>      0:      0     1     2     3     4     5     6     7
>      8:      8     9    10    11    12    13    14    15
>     16:      0     1     2     3     4     5     6     7
>     24:      8     9    10    11    12    13    14    15
>     32:      0     1     2     3     4     5     6     7
> ...
> 
> You can see that the flowhash was correctly set to use the maximum
> number of queues supported by the driver (16).
> 
> However, asking the NIC to reset to "default" fails:
> 
> $ sudo ethtool -X eth1 default
> Cannot set RX flow hash configuration: Invalid argument
> 
> After this change, the flowhash can be reset to default which will use
> all of the available RSS queues (16) or the configured queue count,
> whichever is smaller.
> 
> Starting with eth1 which has 10 queues and a flowhash distributing to
> all 10 queues:
> 
> $ sudo ethtool -x eth1
> RX flow hash indirection table for eth1 with 10 RX ring(s):
>      0:      0     1     2     3     4     5     6     7
>      8:      8     9     0     1     2     3     4     5
>     16:      6     7     8     9     0     1     2     3
> ...
> 
> Increasing the queue count to 48 resets the flowhash to distribute to 16
> queues, as it did before this patch:
> 
> $ sudo ethtool -L eth1 combined 48
> $ sudo ethtool -x eth1
> RX flow hash indirection table for eth1 with 16 RX ring(s):
>      0:      0     1     2     3     4     5     6     7
>      8:      8     9    10    11    12    13    14    15
>     16:      0     1     2     3     4     5     6     7
> ...
> 
> Due to the other bugfix in this series, the flowhash can be set to use
> queues 0-5:
> 
> $ sudo ethtool -X eth1 equal 5
> $ sudo ethtool -x eth1
> RX flow hash indirection table for eth1 with 16 RX ring(s):
>      0:      0     1     2     3     4     0     1     2
>      8:      3     4     0     1     2     3     4     0
>     16:      1     2     3     4     0     1     2     3
> ...
> 
> Due to this bugfix, the flowhash can be reset to default and use 16
> queues:
> 
> $ sudo ethtool -X eth1 default
> $ sudo ethtool -x eth1
> RX flow hash indirection table for eth1 with 16 RX ring(s):
>      0:      0     1     2     3     4     5     6     7
>      8:      8     9    10    11    12    13    14    15
>     16:      0     1     2     3     4     5     6     7
> ...
> 
> Signed-off-by: Joe Damato <jdamato@fastly.com>

Thanks for the detailed commit message and steps to reproduce
and validate the issue.

Would suggest changing the title to indicate that this fix is enabling
setting the RSS indirection table to default value.

Reviewed-by: Sridhar Samudrala <sridhar.samudrala@intel.com>

> ---
>   .../net/ethernet/intel/ixgbe/ixgbe_ethtool.c  | 19 ++++++++++---------
>   1 file changed, 10 insertions(+), 9 deletions(-)
> 
> diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
> index 821dfd323fa9..0bbad4a5cc2f 100644
> --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
> +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
> @@ -2665,6 +2665,14 @@ static int ixgbe_get_rss_hash_opts(struct ixgbe_adapter *adapter,
>   	return 0;
>   }
>   
> +static int ixgbe_rss_indir_tbl_max(struct ixgbe_adapter *adapter)
> +{
> +	if (adapter->hw.mac.type < ixgbe_mac_X550)
> +		return 16;
> +	else
> +		return 64;
> +}
> +
>   static int ixgbe_get_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd,
>   			   u32 *rule_locs)
>   {
> @@ -2673,7 +2681,8 @@ static int ixgbe_get_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd,
>   
>   	switch (cmd->cmd) {
>   	case ETHTOOL_GRXRINGS:
> -		cmd->data = adapter->num_rx_queues;
> +		cmd->data = min_t(int, adapter->num_rx_queues,
> +				  ixgbe_rss_indir_tbl_max(adapter));
>   		ret = 0;
>   		break;
>   	case ETHTOOL_GRXCLSRLCNT:
> @@ -3075,14 +3084,6 @@ static int ixgbe_set_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd)
>   	return ret;
>   }
>   
> -static int ixgbe_rss_indir_tbl_max(struct ixgbe_adapter *adapter)
> -{
> -	if (adapter->hw.mac.type < ixgbe_mac_X550)
> -		return 16;
> -	else
> -		return 64;
> -}
> -
>   static u32 ixgbe_get_rxfh_key_size(struct net_device *netdev)
>   {
>   	return IXGBE_RSS_KEY_SIZE;

Re: [PATCH net 2/2] ixgbe: Allow ixgbe to reset default flow hash

[Joe Damato]

On Sun, Apr 16, 2023 at 10:29:36AM -0500, Samudrala, Sridhar wrote:
> 
> 
> On 4/15/2023 12:48 AM, Joe Damato wrote:
> >ethtool uses `ETHTOOL_GRXRINGS` to compute how many queues are supported
> >by RSS. The driver should return the smaller of either:
> >   - The maximum number of RSS queues the device supports, OR
> >   - The number of RX queues configured
> >
> >Prior to this change, running `ethtool -X $iface default` fails if the
> >number of queues configured is larger than the number supported by RSS,
> >even though changing the queue count correctly resets the flowhash to
> >use all supported queues.
> >
> >Other drivers (for example, i40e) will succeed but the flow hash will
> >reset to support the maximum number of queues supported by RSS, even if
> >that amount is smaller than the configured amount.
> >
> >Prior to this change:
> >
> >$ sudo ethtool -L eth1 combined 20
> >$ sudo ethtool -x eth1
> >RX flow hash indirection table for eth1 with 20 RX ring(s):
> >     0:      0     1     2     3     4     5     6     7
> >     8:      8     9    10    11    12    13    14    15
> >    16:      0     1     2     3     4     5     6     7
> >    24:      8     9    10    11    12    13    14    15
> >    32:      0     1     2     3     4     5     6     7
> >...
> >
> >You can see that the flowhash was correctly set to use the maximum
> >number of queues supported by the driver (16).
> >
> >However, asking the NIC to reset to "default" fails:
> >
> >$ sudo ethtool -X eth1 default
> >Cannot set RX flow hash configuration: Invalid argument
> >
> >After this change, the flowhash can be reset to default which will use
> >all of the available RSS queues (16) or the configured queue count,
> >whichever is smaller.
> >
> >Starting with eth1 which has 10 queues and a flowhash distributing to
> >all 10 queues:
> >
> >$ sudo ethtool -x eth1
> >RX flow hash indirection table for eth1 with 10 RX ring(s):
> >     0:      0     1     2     3     4     5     6     7
> >     8:      8     9     0     1     2     3     4     5
> >    16:      6     7     8     9     0     1     2     3
> >...
> >
> >Increasing the queue count to 48 resets the flowhash to distribute to 16
> >queues, as it did before this patch:
> >
> >$ sudo ethtool -L eth1 combined 48
> >$ sudo ethtool -x eth1
> >RX flow hash indirection table for eth1 with 16 RX ring(s):
> >     0:      0     1     2     3     4     5     6     7
> >     8:      8     9    10    11    12    13    14    15
> >    16:      0     1     2     3     4     5     6     7
> >...
> >
> >Due to the other bugfix in this series, the flowhash can be set to use
> >queues 0-5:
> >
> >$ sudo ethtool -X eth1 equal 5
> >$ sudo ethtool -x eth1
> >RX flow hash indirection table for eth1 with 16 RX ring(s):
> >     0:      0     1     2     3     4     0     1     2
> >     8:      3     4     0     1     2     3     4     0
> >    16:      1     2     3     4     0     1     2     3
> >...
> >
> >Due to this bugfix, the flowhash can be reset to default and use 16
> >queues:
> >
> >$ sudo ethtool -X eth1 default
> >$ sudo ethtool -x eth1
> >RX flow hash indirection table for eth1 with 16 RX ring(s):
> >     0:      0     1     2     3     4     5     6     7
> >     8:      8     9    10    11    12    13    14    15
> >    16:      0     1     2     3     4     5     6     7
> >...
> >
> >Signed-off-by: Joe Damato <jdamato@fastly.com>
> 
> Thanks for the detailed commit message and steps to reproduce
> and validate the issue.

No worries. Thanks for the review.

> Would suggest changing the title to indicate that this fix is enabling
> setting the RSS indirection table to default value.

OK, sure. I can send a v2 that includes your reviewed tags and changes the
title of this commit (but makes no other changes).

I'll make the new title of this commit ixgbe: Enable setting RSS table to
default values.

Hope that's ok; will send out the v2 shortly.

> Reviewed-by: Sridhar Samudrala <sridhar.samudrala@intel.com>
> 
> >---
> >  .../net/ethernet/intel/ixgbe/ixgbe_ethtool.c  | 19 ++++++++++---------
> >  1 file changed, 10 insertions(+), 9 deletions(-)
> >
> >diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
> >index 821dfd323fa9..0bbad4a5cc2f 100644
> >--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
> >+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
> >@@ -2665,6 +2665,14 @@ static int ixgbe_get_rss_hash_opts(struct ixgbe_adapter *adapter,
> >  	return 0;
> >  }
> >+static int ixgbe_rss_indir_tbl_max(struct ixgbe_adapter *adapter)
> >+{
> >+	if (adapter->hw.mac.type < ixgbe_mac_X550)
> >+		return 16;
> >+	else
> >+		return 64;
> >+}
> >+
> >  static int ixgbe_get_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd,
> >  			   u32 *rule_locs)
> >  {
> >@@ -2673,7 +2681,8 @@ static int ixgbe_get_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd,
> >  	switch (cmd->cmd) {
> >  	case ETHTOOL_GRXRINGS:
> >-		cmd->data = adapter->num_rx_queues;
> >+		cmd->data = min_t(int, adapter->num_rx_queues,
> >+				  ixgbe_rss_indir_tbl_max(adapter));
> >  		ret = 0;
> >  		break;
> >  	case ETHTOOL_GRXCLSRLCNT:
> >@@ -3075,14 +3084,6 @@ static int ixgbe_set_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd)
> >  	return ret;
> >  }
> >-static int ixgbe_rss_indir_tbl_max(struct ixgbe_adapter *adapter)
> >-{
> >-	if (adapter->hw.mac.type < ixgbe_mac_X550)
> >-		return 16;
> >-	else
> >-		return 64;
> >-}
> >-
> >  static u32 ixgbe_get_rxfh_key_size(struct net_device *netdev)
> >  {
> >  	return IXGBE_RSS_KEY_SIZE;