[RFC net-next 0/2] gve: Link IRQs, queues, and NAPI instances

[RFC net-next 0/2] gve: Link IRQs, queues, and NAPI instances

[Joe Damato]

Greetings:

This RFC uses the netdev-genl API to link IRQs and queues to NAPI IDs so
that this information is queryable by user apps. This is particularly
useful for epoll-based busy polling apps which rely on having access to
the NAPI ID.

I've tested these commits on a GCP instance with a GVE NIC configured
and have included test output in the commit messages for each patch
showing how to query the information.

I hope to send this next week when net-next reopens as an official
submission, but wanted to send an RFC now in case there were any changes
requested from the GVE maintainers.

Thanks,
Joe

Joe Damato (2):
  gve: Map IRQs to NAPI instances
  gve: Map NAPI instances to queues

 drivers/net/ethernet/google/gve/gve_main.c  | 12 ++++++++++++
 drivers/net/ethernet/google/gve/gve_utils.c |  1 +
 2 files changed, 13 insertions(+)

-- 
2.43.0

[RFC net-next 1/2] gve: Map IRQs to NAPI instances

[Joe Damato]

Use netdev-genl interface to map IRQs to NAPI instances so that this
information is accesible by user apps via netlink.

$ cat /proc/interrupts | grep gve | grep -v mgmnt | cut -f1 --delimiter=':'
 34
 35
 36
 37
 38
 39
 40
[...]
 65

$ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \
                         --dump napi-get --json='{"ifindex": 2}'
[{'id': 8288, 'ifindex': 2, 'irq': 65},
  [...]
 {'id': 8263, 'ifindex': 2, 'irq': 40},
 {'id': 8262, 'ifindex': 2, 'irq': 39},
 {'id': 8261, 'ifindex': 2, 'irq': 38},
 {'id': 8260, 'ifindex': 2, 'irq': 37},
 {'id': 8259, 'ifindex': 2, 'irq': 36},
 {'id': 8258, 'ifindex': 2, 'irq': 35},
 {'id': 8257, 'ifindex': 2, 'irq': 34}]

Signed-off-by: Joe Damato <jdamato@fastly.com>
---
 drivers/net/ethernet/google/gve/gve_utils.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/net/ethernet/google/gve/gve_utils.c b/drivers/net/ethernet/google/gve/gve_utils.c
index 2349750075a5..30fef100257e 100644
--- a/drivers/net/ethernet/google/gve/gve_utils.c
+++ b/drivers/net/ethernet/google/gve/gve_utils.c
@@ -111,6 +111,7 @@ void gve_add_napi(struct gve_priv *priv, int ntfy_idx,
 	struct gve_notify_block *block = &priv->ntfy_blocks[ntfy_idx];
 
 	netif_napi_add(priv->dev, &block->napi, gve_poll);
+	netif_napi_set_irq(&block->napi, block->irq);
 }
 
 void gve_remove_napi(struct gve_priv *priv, int ntfy_idx)
-- 
2.43.0

[RFC net-next 2/2] gve: Map NAPI instances to queues

[Joe Damato]

Use the netdev-genl interface to map NAPI instances to queues so that
this information is accessible to user programs via netlink.

$ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \
			 --dump queue-get --json='{"ifindex": 2}'

[{'id': 0, 'ifindex': 2, 'napi-id': 8313, 'type': 'rx'},
 {'id': 1, 'ifindex': 2, 'napi-id': 8314, 'type': 'rx'},
 {'id': 2, 'ifindex': 2, 'napi-id': 8315, 'type': 'rx'},
 {'id': 3, 'ifindex': 2, 'napi-id': 8316, 'type': 'rx'},
 {'id': 4, 'ifindex': 2, 'napi-id': 8317, 'type': 'rx'},
[...]
 {'id': 0, 'ifindex': 2, 'napi-id': 8297, 'type': 'tx'},
 {'id': 1, 'ifindex': 2, 'napi-id': 8298, 'type': 'tx'},
 {'id': 2, 'ifindex': 2, 'napi-id': 8299, 'type': 'tx'},
 {'id': 3, 'ifindex': 2, 'napi-id': 8300, 'type': 'tx'},
 {'id': 4, 'ifindex': 2, 'napi-id': 8301, 'type': 'tx'},
[...]

Signed-off-by: Joe Damato <jdamato@fastly.com>
---
 drivers/net/ethernet/google/gve/gve_main.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c
index 661566db68c8..da811e90bdfa 100644
--- a/drivers/net/ethernet/google/gve/gve_main.c
+++ b/drivers/net/ethernet/google/gve/gve_main.c
@@ -1875,6 +1875,9 @@ static void gve_turndown(struct gve_priv *priv)
 
 		if (!gve_tx_was_added_to_block(priv, idx))
 			continue;
+
+		netif_queue_set_napi(priv->dev, idx, NETDEV_QUEUE_TYPE_TX,
+				     NULL);
 		napi_disable(&block->napi);
 	}
 	for (idx = 0; idx < priv->rx_cfg.num_queues; idx++) {
@@ -1883,6 +1886,9 @@ static void gve_turndown(struct gve_priv *priv)
 
 		if (!gve_rx_was_added_to_block(priv, idx))
 			continue;
+
+		netif_queue_set_napi(priv->dev, idx, NETDEV_QUEUE_TYPE_RX,
+				     NULL);
 		napi_disable(&block->napi);
 	}
 
@@ -1909,6 +1915,9 @@ static void gve_turnup(struct gve_priv *priv)
 			continue;
 
 		napi_enable(&block->napi);
+		netif_queue_set_napi(priv->dev, idx, NETDEV_QUEUE_TYPE_TX,
+				     &block->napi);
+
 		if (gve_is_gqi(priv)) {
 			iowrite32be(0, gve_irq_doorbell(priv, block));
 		} else {
@@ -1931,6 +1940,9 @@ static void gve_turnup(struct gve_priv *priv)
 			continue;
 
 		napi_enable(&block->napi);
+		netif_queue_set_napi(priv->dev, idx, NETDEV_QUEUE_TYPE_RX,
+				     &block->napi);
+
 		if (gve_is_gqi(priv)) {
 			iowrite32be(0, gve_irq_doorbell(priv, block));
 		} else {
-- 
2.43.0

Re: [RFC net-next 2/2] gve: Map NAPI instances to queues

[Praveen Kaligineedi]

> diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c
> index 661566db68c8..da811e90bdfa 100644
> --- a/drivers/net/ethernet/google/gve/gve_main.c
> +++ b/drivers/net/ethernet/google/gve/gve_main.c
> @@ -1875,6 +1875,9 @@ static void gve_turndown(struct gve_priv *priv)
>
>                 if (!gve_tx_was_added_to_block(priv, idx))
>                         continue;
> +
> +               netif_queue_set_napi(priv->dev, idx, NETDEV_QUEUE_TYPE_TX,
> +                                    NULL);
>                 napi_disable(&block->napi);
>         }
When XDP program is installed, the for loop iterates over both
configured TX queues (idx <  priv->tx_cfg.num_queues) as well as
dedicated XDP TX queues ( idx >= priv->tx_cfg.num_queues).
Should add if (idx <  priv->tx_cfg.num_queues) check here.

> @@ -1909,6 +1915,9 @@ static void gve_turnup(struct gve_priv *priv)
>                         continue;
>
>                 napi_enable(&block->napi);
> +               netif_queue_set_napi(priv->dev, idx, NETDEV_QUEUE_TYPE_TX,
> +                                    &block->napi);
> +
>                 if (gve_is_gqi(priv)) {
>                         iowrite32be(0, gve_irq_doorbell(priv, block));
>                 } else {

Same as above. When XDP program is installed, the for loop iterates
over both configured TX queues (idx <  priv->tx_cfg.num_queues) as
well as dedicated XDP TX queues ( idx >= priv->tx_cfg.num_queues)

Re: [RFC net-next 2/2] gve: Map NAPI instances to queues

[Joe Damato]

On Fri, Sep 27, 2024 at 11:15:39AM -0700, Praveen Kaligineedi wrote:
> > diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c
> > index 661566db68c8..da811e90bdfa 100644
> > --- a/drivers/net/ethernet/google/gve/gve_main.c
> > +++ b/drivers/net/ethernet/google/gve/gve_main.c
> > @@ -1875,6 +1875,9 @@ static void gve_turndown(struct gve_priv *priv)
> >
> >                 if (!gve_tx_was_added_to_block(priv, idx))
> >                         continue;
> > +
> > +               netif_queue_set_napi(priv->dev, idx, NETDEV_QUEUE_TYPE_TX,
> > +                                    NULL);
> >                 napi_disable(&block->napi);
> >         }
> When XDP program is installed, the for loop iterates over both
> configured TX queues (idx <  priv->tx_cfg.num_queues) as well as
> dedicated XDP TX queues ( idx >= priv->tx_cfg.num_queues).
> Should add if (idx <  priv->tx_cfg.num_queues) check here.
> 
> > @@ -1909,6 +1915,9 @@ static void gve_turnup(struct gve_priv *priv)
> >                         continue;
> >
> >                 napi_enable(&block->napi);
> > +               netif_queue_set_napi(priv->dev, idx, NETDEV_QUEUE_TYPE_TX,
> > +                                    &block->napi);
> > +
> >                 if (gve_is_gqi(priv)) {
> >                         iowrite32be(0, gve_irq_doorbell(priv, block));
> >                 } else {
> 
> Same as above. When XDP program is installed, the for loop iterates
> over both configured TX queues (idx <  priv->tx_cfg.num_queues) as
> well as dedicated XDP TX queues ( idx >= priv->tx_cfg.num_queues)

Ah, OK. Thanks for the review. I'll make that change and retest it
on my GCP instance.

Since net-next is (I think) likely to reopen soon, I'll include that
change in the submission I send next week and mention it in the
changelog.

Re: [RFC net-next 1/2] gve: Map IRQs to NAPI instances

[Simon Horman]

On Thu, Sep 26, 2024 at 03:00:21AM +0000, Joe Damato wrote:
> Use netdev-genl interface to map IRQs to NAPI instances so that this
> information is accesible by user apps via netlink.

nit: accessible

Flagged by checkpatch.pl --codespell

...

Re: [RFC net-next 1/2] gve: Map IRQs to NAPI instances

[Joe Damato]

On Fri, Sep 27, 2024 at 07:56:19PM +0100, Simon Horman wrote:
> On Thu, Sep 26, 2024 at 03:00:21AM +0000, Joe Damato wrote:
> > Use netdev-genl interface to map IRQs to NAPI instances so that this
> > information is accesible by user apps via netlink.
> 
> nit: accessible
> 
> Flagged by checkpatch.pl --codespell

Thanks, will fix that.