[net-next v4 0/2] tg3: Link IRQs, NAPIs, and queues

[net-next v4 0/2] tg3: Link IRQs, NAPIs, and queues

[Joe Damato]

Greetings:

Welcome to v4, now an actual submission instead of an RFC.

This follows from a previous RFC (wherein I botched the subject lines of
all the messages) [1].

I've taken Michael Chan's suggestion on modifying patch 2 and I've
updated the commit messages of both patches to test and show the output
for the default 1 TX 4 RX queues and the 4 TX and 4 RX queues cases.

Reviewers: please check the commit messages carefully to ensure the
output is correct (or on your own systems to verify, if you like). I am
not a tg3 expert and it's possible that I got something wrong.

Thanks,
Joe

[1]: https://lore.kernel.org/all/20241005145717.302575-3-jdamato@fastly.com/T/

v4:
  - Upgraded from RFC to official submission
  - Patch 1:
    - Updated commit message to test more cases, no code or functional
      changes, so I retained the Reviewed-by
  - Patch 2:
    - Switched the if ... else if ... to two ifs as per Michael Chan's
      suggestion when tg3 uses combined tx and rx queues
    - Updated commit message to test more cases, including combined tx
      and rx queues

rfcv3:
  - Patch 1:
    - Line wrap to 80 characters, no functional changes
    - Added Pavan Chebbi's Reviewed-by
  - Patch 2:
    - changed tg3_napi_enable and tg3_napi_disable to use running
      counters to number the queues as there is no explicit queue index
      assigned in tg3

Joe Damato (2):
  tg3: Link IRQs to NAPI instances
  tg3: Link queues to NAPIs

 drivers/net/ethernet/broadcom/tg3.c | 47 ++++++++++++++++++++++++-----
 1 file changed, 40 insertions(+), 7 deletions(-)

-- 
2.25.1

[net-next v4 1/2] tg3: Link IRQs to NAPI instances

[Joe Damato]

Link IRQs to NAPI instances with netif_napi_set_irq. This information
can be queried with the netdev-genl API.

Begin by testing my tg3 device in its default state: 1 TX queue and 4 RX
queues.

Compare the output of /proc/interrupts for my tg3 device with the output of
netdev-genl after applying this patch:

$ cat /proc/interrupts | grep eth0
343: [...] eth0-tx-0
344: [...] eth0-rx-1
345: [...] eth0-rx-2
346: [...] eth0-rx-3
347: [...] eth0-rx-4

As you can see above, tg3 has named the IRQs such that there is a
dedicated tx IRQ and 4 dedicated rx IRQs, for a total of 5 IRQs.

$ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \
			 --dump napi-get --json='{"ifindex": 2}'

[{'id': 8197, 'ifindex': 2, 'irq': 347},
 {'id': 8196, 'ifindex': 2, 'irq': 346},
 {'id': 8195, 'ifindex': 2, 'irq': 345},
 {'id': 8194, 'ifindex': 2, 'irq': 344},
 {'id': 8193, 'ifindex': 2, 'irq': 343}]

Netlink displays the same IRQs as above, noting that each is mapped to a
unique NAPI instance.

Now, reconfigure the NIC to have 4 TX queues and 4 RX queues:

$ sudo ethtool -L eth0 rx 4 tx 4
$ sudo ethtool -l eth0 | tail -5
Current hardware settings:
RX:		4
TX:		4
Other:		n/a
Combined:	n/a

Examine /proc/interrupts once again, noting that tg3 will now rename the
IRQs to suggest that they are combined tx and rx without allocating
additional IRQs, so the total IRQ count in /proc/interrupts is
unchanged:

343: [...] eth0-0
344: [...] eth0-txrx-1
345: [...] eth0-txrx-2
346: [...] eth0-txrx-3
347: [...] eth0-txrx-4

Check the output from netlink again:

$ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \
                         --dump napi-get --json='{"ifindex": 2}'
[{'id': 8973, 'ifindex': 2, 'irq': 347},
 {'id': 8972, 'ifindex': 2, 'irq': 346},
 {'id': 8971, 'ifindex': 2, 'irq': 345},
 {'id': 8970, 'ifindex': 2, 'irq': 344},
 {'id': 8969, 'ifindex': 2, 'irq': 343}]

Signed-off-by: Joe Damato <jdamato@fastly.com>
Reviewed-by: Pavan Chebbi <pavan.chebbi@broadcom.com>
---
 v4:
   - Updated commit message

 rfcv3:
   - wrapped the netif_napi_add call to 80 characters

 drivers/net/ethernet/broadcom/tg3.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/broadcom/tg3.c b/drivers/net/ethernet/broadcom/tg3.c
index 378815917741..6564072b47ba 100644
--- a/drivers/net/ethernet/broadcom/tg3.c
+++ b/drivers/net/ethernet/broadcom/tg3.c
@@ -7413,9 +7413,11 @@ static void tg3_napi_init(struct tg3 *tp)
 {
 	int i;
 
-	netif_napi_add(tp->dev, &tp->napi[0].napi, tg3_poll);
-	for (i = 1; i < tp->irq_cnt; i++)
-		netif_napi_add(tp->dev, &tp->napi[i].napi, tg3_poll_msix);
+	for (i = 0; i < tp->irq_cnt; i++) {
+		netif_napi_add(tp->dev, &tp->napi[i].napi,
+			       i ? tg3_poll_msix : tg3_poll);
+		netif_napi_set_irq(&tp->napi[i].napi, tp->napi[i].irq_vec);
+	}
 }
 
 static void tg3_napi_fini(struct tg3 *tp)
-- 
2.25.1

[net-next v4 2/2] tg3: Link queues to NAPIs

[Joe Damato]

Link queues to NAPIs using the netdev-genl API so this information is
queryable.

First, test with the default setting on my tg3 NIC at boot with 1 TX
queue:

$ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \
                         --dump queue-get --json='{"ifindex": 2}'

[{'id': 0, 'ifindex': 2, 'napi-id': 8194, 'type': 'rx'},
 {'id': 1, 'ifindex': 2, 'napi-id': 8195, 'type': 'rx'},
 {'id': 2, 'ifindex': 2, 'napi-id': 8196, 'type': 'rx'},
 {'id': 3, 'ifindex': 2, 'napi-id': 8197, 'type': 'rx'},
 {'id': 0, 'ifindex': 2, 'napi-id': 8193, 'type': 'tx'}]

Now, adjust the number of TX queues to be 4 via ethtool:

$ sudo ethtool -L eth0 tx 4
$ sudo ethtool -l eth0 | tail -5
Current hardware settings:
RX:		4
TX:		4
Other:		n/a
Combined:	n/a

Despite "Combined: n/a" in the ethtool output, /proc/interrupts shows
the tg3 has renamed the IRQs to be combined:

343: [...] eth0-0
344: [...] eth0-txrx-1
345: [...] eth0-txrx-2
346: [...] eth0-txrx-3
347: [...] eth0-txrx-4

Now query this via netlink to ensure the queues are linked properly to
their NAPIs:

$ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \
                         --dump queue-get --json='{"ifindex": 2}'
[{'id': 0, 'ifindex': 2, 'napi-id': 8960, 'type': 'rx'},
 {'id': 1, 'ifindex': 2, 'napi-id': 8961, 'type': 'rx'},
 {'id': 2, 'ifindex': 2, 'napi-id': 8962, 'type': 'rx'},
 {'id': 3, 'ifindex': 2, 'napi-id': 8963, 'type': 'rx'},
 {'id': 0, 'ifindex': 2, 'napi-id': 8960, 'type': 'tx'},
 {'id': 1, 'ifindex': 2, 'napi-id': 8961, 'type': 'tx'},
 {'id': 2, 'ifindex': 2, 'napi-id': 8962, 'type': 'tx'},
 {'id': 3, 'ifindex': 2, 'napi-id': 8963, 'type': 'tx'}]

As you can see above, id 0 for both TX and RX share a NAPI, NAPI ID
8960, and so on for each queue index up to 3.

Signed-off-by: Joe Damato <jdamato@fastly.com>
---
 v4:
   - Switch the if ... else if to two ifs as suggested by Michael Chan
     to handle the case where tg3 might use combined queues
   - Updated the commit message to test both the default and combined
     queue cases to ensure correctness

 rfcv3:
   - Added running counters for numbering the rx and tx queue IDs to
     tg3_napi_enable and tg3_napi_disable

 drivers/net/ethernet/broadcom/tg3.c | 39 ++++++++++++++++++++++++++---
 1 file changed, 35 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/broadcom/tg3.c b/drivers/net/ethernet/broadcom/tg3.c
index 6564072b47ba..c4bce6493afa 100644
--- a/drivers/net/ethernet/broadcom/tg3.c
+++ b/drivers/net/ethernet/broadcom/tg3.c
@@ -7395,18 +7395,49 @@ static int tg3_poll(struct napi_struct *napi, int budget)
 
 static void tg3_napi_disable(struct tg3 *tp)
 {
+	int txq_idx = tp->txq_cnt - 1;
+	int rxq_idx = tp->rxq_cnt - 1;
+	struct tg3_napi *tnapi;
 	int i;
 
-	for (i = tp->irq_cnt - 1; i >= 0; i--)
-		napi_disable(&tp->napi[i].napi);
+	for (i = tp->irq_cnt - 1; i >= 0; i--) {
+		tnapi = &tp->napi[i];
+		if (tnapi->tx_buffers) {
+			netif_queue_set_napi(tp->dev, txq_idx,
+					     NETDEV_QUEUE_TYPE_TX, NULL);
+			txq_idx--;
+		}
+		if (tnapi->rx_rcb) {
+			netif_queue_set_napi(tp->dev, rxq_idx,
+					     NETDEV_QUEUE_TYPE_RX, NULL);
+			rxq_idx--;
+		}
+		napi_disable(&tnapi->napi);
+	}
 }
 
 static void tg3_napi_enable(struct tg3 *tp)
 {
+	int txq_idx = 0, rxq_idx = 0;
+	struct tg3_napi *tnapi;
 	int i;
 
-	for (i = 0; i < tp->irq_cnt; i++)
-		napi_enable(&tp->napi[i].napi);
+	for (i = 0; i < tp->irq_cnt; i++) {
+		tnapi = &tp->napi[i];
+		napi_enable(&tnapi->napi);
+		if (tnapi->tx_buffers) {
+			netif_queue_set_napi(tp->dev, txq_idx,
+					     NETDEV_QUEUE_TYPE_TX,
+					     &tnapi->napi);
+			txq_idx++;
+		}
+		if (tnapi->rx_rcb) {
+			netif_queue_set_napi(tp->dev, rxq_idx,
+					     NETDEV_QUEUE_TYPE_RX,
+					     &tnapi->napi);
+			rxq_idx++;
+		}
+	}
 }
 
 static void tg3_napi_init(struct tg3 *tp)
-- 
2.25.1

Re: [net-next v4 2/2] tg3: Link queues to NAPIs

[Michael Chan]

[-- Attachment #1: Type: text/plain, Size: 2533 bytes --]

On Wed, Oct 9, 2024 at 10:55 AM Joe Damato <jdamato@fastly.com> wrote:
>
> Link queues to NAPIs using the netdev-genl API so this information is
> queryable.
>
> First, test with the default setting on my tg3 NIC at boot with 1 TX
> queue:
>
> $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \
>                          --dump queue-get --json='{"ifindex": 2}'
>
> [{'id': 0, 'ifindex': 2, 'napi-id': 8194, 'type': 'rx'},
>  {'id': 1, 'ifindex': 2, 'napi-id': 8195, 'type': 'rx'},
>  {'id': 2, 'ifindex': 2, 'napi-id': 8196, 'type': 'rx'},
>  {'id': 3, 'ifindex': 2, 'napi-id': 8197, 'type': 'rx'},
>  {'id': 0, 'ifindex': 2, 'napi-id': 8193, 'type': 'tx'}]
>

This is correct.  When TSS is not enabled (1 TX ring only), the TX
ring uses NAPI 0 and the RSS RX rings use NAPI 1, 2, 3, 4.

> Now, adjust the number of TX queues to be 4 via ethtool:
>
> $ sudo ethtool -L eth0 tx 4
> $ sudo ethtool -l eth0 | tail -5
> Current hardware settings:
> RX:             4
> TX:             4
> Other:          n/a
> Combined:       n/a
>
> Despite "Combined: n/a" in the ethtool output, /proc/interrupts shows
> the tg3 has renamed the IRQs to be combined:
>
> 343: [...] eth0-0
> 344: [...] eth0-txrx-1
> 345: [...] eth0-txrx-2
> 346: [...] eth0-txrx-3
> 347: [...] eth0-txrx-4
>
> Now query this via netlink to ensure the queues are linked properly to
> their NAPIs:
>
> $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \
>                          --dump queue-get --json='{"ifindex": 2}'
> [{'id': 0, 'ifindex': 2, 'napi-id': 8960, 'type': 'rx'},
>  {'id': 1, 'ifindex': 2, 'napi-id': 8961, 'type': 'rx'},
>  {'id': 2, 'ifindex': 2, 'napi-id': 8962, 'type': 'rx'},
>  {'id': 3, 'ifindex': 2, 'napi-id': 8963, 'type': 'rx'},
>  {'id': 0, 'ifindex': 2, 'napi-id': 8960, 'type': 'tx'},
>  {'id': 1, 'ifindex': 2, 'napi-id': 8961, 'type': 'tx'},
>  {'id': 2, 'ifindex': 2, 'napi-id': 8962, 'type': 'tx'},
>  {'id': 3, 'ifindex': 2, 'napi-id': 8963, 'type': 'tx'}]
>

This is also correct after reviewing the driver code.  When TSS is
enabled, NAPI 0 is no longer used for any TX ring.  All TSS and RSS
rings start from NAPI 1.  NAPI 0 is only used for link change and
other error interrupts.

> As you can see above, id 0 for both TX and RX share a NAPI, NAPI ID
> 8960, and so on for each queue index up to 3.
>
> Signed-off-by: Joe Damato <jdamato@fastly.com>

Thanks.
Reviewed-by: Michael Chan <michael.chan@broadcom.com>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4209 bytes --]

Re: [net-next v4 1/2] tg3: Link IRQs to NAPI instances

[Michael Chan]

[-- Attachment #1: Type: text/plain, Size: 370 bytes --]

On Wed, Oct 9, 2024 at 10:55 AM Joe Damato <jdamato@fastly.com> wrote:
>
> Link IRQs to NAPI instances with netif_napi_set_irq. This information
> can be queried with the netdev-genl API.
>

> Signed-off-by: Joe Damato <jdamato@fastly.com>
> Reviewed-by: Pavan Chebbi <pavan.chebbi@broadcom.com>
Thanks.
Reviewed-by: Michael Chan <michael.chan@broadcom.com>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4209 bytes --]

Re: [net-next v4 0/2] tg3: Link IRQs, NAPIs, and queues

[patchwork-bot+netdevbpf]

Hello:

This series was applied to netdev/net-next.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Wed,  9 Oct 2024 17:55:07 +0000 you wrote:
> Greetings:
> 
> Welcome to v4, now an actual submission instead of an RFC.
> 
> This follows from a previous RFC (wherein I botched the subject lines of
> all the messages) [1].
> 
> [...]

Here is the summary with links:
  - [net-next,v4,1/2] tg3: Link IRQs to NAPI instances
    https://git.kernel.org/netdev/net-next/c/25118cce6627
  - [net-next,v4,2/2] tg3: Link queues to NAPIs
    https://git.kernel.org/netdev/net-next/c/aec5514d739f

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html