[net-next v2 0/2] ena: Link IRQs, queues, and NAPI instances

[net-next v2 0/2] ena: Link IRQs, queues, and NAPI instances

[Joe Damato]

Greetings:

Welcome to v2. This includes only cosmetic changes, see changelog below
and in each patch.

This series uses the netdev-genl API to link IRQs and queues to NAPI IDs
so that this information is queryable by user apps. This is particularly
useful for epoll-based busy polling apps which rely on having access to
the NAPI ID.

I've tested these commits on an EC2 instance with an ENA NIC configured
and have included test output in the commit messages for each patch
showing how to query the information.

I noted in the implementation that the driver requests an IRQ for
management purposes which does not have an associated NAPI. I tried to
take this into account in patch 1, but would appreciate if ENA
maintainers can verify I did this correctly.

Thanks,
Joe

v2:
  - Preserve reverse christmas tree ordering in patch 1
  - Add comment that the API is for non-XDP queues only to patch 2

v1:
  - https://lore.kernel.org/all/20240930195617.37369-1-jdamato@fastly.com/

Joe Damato (2):
  ena: Link IRQs to NAPI instances
  ena: Link queues to NAPIs

 drivers/net/ethernet/amazon/ena/ena_netdev.c | 40 +++++++++++++++++---
 1 file changed, 35 insertions(+), 5 deletions(-)

-- 
2.25.1

[net-next v2 1/2] ena: Link IRQs to NAPI instances

[Joe Damato]

Link IRQs to NAPI instances with netif_napi_set_irq. This information
can be queried with the netdev-genl API. Note that the ENA device
appears to allocate an IRQ for management purposes which does not have a
NAPI associated with it; this commit takes this into consideration to
accurately construct a map between IRQs and NAPI instances.

Compare the output of /proc/interrupts for my ena device with the output of
netdev-genl after applying this patch:

$ cat /proc/interrupts | grep enp55s0 | cut -f1 --delimiter=':'
 94
 95
 96
 97
 98
 99
100
101

$ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \
			 --dump napi-get --json='{"ifindex": 2}'

[{'id': 8208, 'ifindex': 2, 'irq': 101},
 {'id': 8207, 'ifindex': 2, 'irq': 100},
 {'id': 8206, 'ifindex': 2, 'irq': 99},
 {'id': 8205, 'ifindex': 2, 'irq': 98},
 {'id': 8204, 'ifindex': 2, 'irq': 97},
 {'id': 8203, 'ifindex': 2, 'irq': 96},
 {'id': 8202, 'ifindex': 2, 'irq': 95},
 {'id': 8201, 'ifindex': 2, 'irq': 94}]

Signed-off-by: Joe Damato <jdamato@fastly.com>
---
 v2:
   - Preserve reverse christmas tree order in ena_request_io_irq
   - No functional changes

 drivers/net/ethernet/amazon/ena/ena_netdev.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index c5b50cfa935a..74ce9fa45cf8 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -1677,9 +1677,9 @@ static int ena_request_mgmnt_irq(struct ena_adapter *adapter)
 static int ena_request_io_irq(struct ena_adapter *adapter)
 {
 	u32 io_queue_count = adapter->num_io_queues + adapter->xdp_num_queues;
+	int rc = 0, i, k, irq_idx;
 	unsigned long flags = 0;
 	struct ena_irq *irq;
-	int rc = 0, i, k;
 
 	if (!test_bit(ENA_FLAG_MSIX_ENABLED, &adapter->flags)) {
 		netif_err(adapter, ifup, adapter->netdev,
@@ -1705,6 +1705,16 @@ static int ena_request_io_irq(struct ena_adapter *adapter)
 		irq_set_affinity_hint(irq->vector, &irq->affinity_hint_mask);
 	}
 
+	/* Now that IO IRQs have been successfully allocated map them to the
+	 * corresponding IO NAPI instance. Note that the mgmnt IRQ does not
+	 * have a NAPI, so care must be taken to correctly map IRQs to NAPIs.
+	 */
+	for (i = 0; i < io_queue_count; i++) {
+		irq_idx = ENA_IO_IRQ_IDX(i);
+		irq = &adapter->irq_tbl[irq_idx];
+		netif_napi_set_irq(&adapter->ena_napi[i].napi, irq->vector);
+	}
+
 	return rc;
 
 err:
-- 
2.25.1

[net-next v2 2/2] ena: Link queues to NAPIs

[Joe Damato]

Link queues to NAPIs using the netdev-genl API so this information is
queryable.

$ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \
                         --dump queue-get --json='{"ifindex": 2}'

[{'id': 0, 'ifindex': 2, 'napi-id': 8201, 'type': 'rx'},
 {'id': 1, 'ifindex': 2, 'napi-id': 8202, 'type': 'rx'},
 {'id': 2, 'ifindex': 2, 'napi-id': 8203, 'type': 'rx'},
 {'id': 3, 'ifindex': 2, 'napi-id': 8204, 'type': 'rx'},
 {'id': 4, 'ifindex': 2, 'napi-id': 8205, 'type': 'rx'},
 {'id': 5, 'ifindex': 2, 'napi-id': 8206, 'type': 'rx'},
 {'id': 6, 'ifindex': 2, 'napi-id': 8207, 'type': 'rx'},
 {'id': 7, 'ifindex': 2, 'napi-id': 8208, 'type': 'rx'},
 {'id': 0, 'ifindex': 2, 'napi-id': 8201, 'type': 'tx'},
 {'id': 1, 'ifindex': 2, 'napi-id': 8202, 'type': 'tx'},
 {'id': 2, 'ifindex': 2, 'napi-id': 8203, 'type': 'tx'},
 {'id': 3, 'ifindex': 2, 'napi-id': 8204, 'type': 'tx'},
 {'id': 4, 'ifindex': 2, 'napi-id': 8205, 'type': 'tx'},
 {'id': 5, 'ifindex': 2, 'napi-id': 8206, 'type': 'tx'},
 {'id': 6, 'ifindex': 2, 'napi-id': 8207, 'type': 'tx'},
 {'id': 7, 'ifindex': 2, 'napi-id': 8208, 'type': 'tx'}]

Signed-off-by: Joe Damato <jdamato@fastly.com>
---
 v2:
   - Comments added to ena_napi_disable_in_range and
     ena_napi_enable_in_range
   - No functional changes

 drivers/net/ethernet/amazon/ena/ena_netdev.c | 28 +++++++++++++++++---
 1 file changed, 24 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index 74ce9fa45cf8..96df20854eb9 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -1821,20 +1821,40 @@ static void ena_napi_disable_in_range(struct ena_adapter *adapter,
 				      int first_index,
 				      int count)
 {
+	struct napi_struct *napi;
 	int i;
 
-	for (i = first_index; i < first_index + count; i++)
-		napi_disable(&adapter->ena_napi[i].napi);
+	for (i = first_index; i < first_index + count; i++) {
+		napi = &adapter->ena_napi[i].napi;
+		if (!ENA_IS_XDP_INDEX(adapter, i)) {
+			/* This API is supported for non-XDP queues only */
+			netif_queue_set_napi(adapter->netdev, i,
+					     NETDEV_QUEUE_TYPE_TX, NULL);
+			netif_queue_set_napi(adapter->netdev, i,
+					     NETDEV_QUEUE_TYPE_RX, NULL);
+		}
+		napi_disable(napi);
+	}
 }
 
 static void ena_napi_enable_in_range(struct ena_adapter *adapter,
 				     int first_index,
 				     int count)
 {
+	struct napi_struct *napi;
 	int i;
 
-	for (i = first_index; i < first_index + count; i++)
-		napi_enable(&adapter->ena_napi[i].napi);
+	for (i = first_index; i < first_index + count; i++) {
+		napi = &adapter->ena_napi[i].napi;
+		napi_enable(napi);
+		if (!ENA_IS_XDP_INDEX(adapter, i)) {
+			/* This API is supported for non-XDP queues only */
+			netif_queue_set_napi(adapter->netdev, i,
+					     NETDEV_QUEUE_TYPE_RX, napi);
+			netif_queue_set_napi(adapter->netdev, i,
+					     NETDEV_QUEUE_TYPE_TX, napi);
+		}
+	}
 }
 
 /* Configure the Rx forwarding */
-- 
2.25.1

RE: [net-next v2 1/2] ena: Link IRQs to NAPI instances

[Arinzon, David]

> Link IRQs to NAPI instances with netif_napi_set_irq. This information can be
> queried with the netdev-genl API. Note that the ENA device appears to
> allocate an IRQ for management purposes which does not have a NAPI
> associated with it; this commit takes this into consideration to accurately
> construct a map between IRQs and NAPI instances.
> 
> Compare the output of /proc/interrupts for my ena device with the output
> of netdev-genl after applying this patch:
> 
> $ cat /proc/interrupts | grep enp55s0 | cut -f1 --delimiter=':'
>  94
>  95
>  96
>  97
>  98
>  99
> 100
> 101
> 
> $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \
>                          --dump napi-get --json='{"ifindex": 2}'
> 
> [{'id': 8208, 'ifindex': 2, 'irq': 101},
>  {'id': 8207, 'ifindex': 2, 'irq': 100},
>  {'id': 8206, 'ifindex': 2, 'irq': 99},
>  {'id': 8205, 'ifindex': 2, 'irq': 98},
>  {'id': 8204, 'ifindex': 2, 'irq': 97},
>  {'id': 8203, 'ifindex': 2, 'irq': 96},
>  {'id': 8202, 'ifindex': 2, 'irq': 95},
>  {'id': 8201, 'ifindex': 2, 'irq': 94}]
> 
> Signed-off-by: Joe Damato <jdamato@fastly.com>
> ---
>  v2:
>    - Preserve reverse christmas tree order in ena_request_io_irq
>    - No functional changes
> 
>  drivers/net/ethernet/amazon/ena/ena_netdev.c | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c
> b/drivers/net/ethernet/amazon/ena/ena_netdev.c
> index c5b50cfa935a..74ce9fa45cf8 100644
> --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
> +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
> @@ -1677,9 +1677,9 @@ static int ena_request_mgmnt_irq(struct
> ena_adapter *adapter)  static int ena_request_io_irq(struct ena_adapter
> *adapter)  {
>         u32 io_queue_count = adapter->num_io_queues + adapter-
> >xdp_num_queues;
> +       int rc = 0, i, k, irq_idx;
>         unsigned long flags = 0;
>         struct ena_irq *irq;
> -       int rc = 0, i, k;
> 
>         if (!test_bit(ENA_FLAG_MSIX_ENABLED, &adapter->flags)) {
>                 netif_err(adapter, ifup, adapter->netdev, @@ -1705,6 +1705,16 @@
> static int ena_request_io_irq(struct ena_adapter *adapter)
>                 irq_set_affinity_hint(irq->vector, &irq->affinity_hint_mask);
>         }
> 
> +       /* Now that IO IRQs have been successfully allocated map them to the
> +        * corresponding IO NAPI instance. Note that the mgmnt IRQ does not
> +        * have a NAPI, so care must be taken to correctly map IRQs to NAPIs.
> +        */
> +       for (i = 0; i < io_queue_count; i++) {
> +               irq_idx = ENA_IO_IRQ_IDX(i);
> +               irq = &adapter->irq_tbl[irq_idx];
> +               netif_napi_set_irq(&adapter->ena_napi[i].napi, irq->vector);
> +       }
> +
>         return rc;
> 
>  err:
> --
> 2.25.1

LGTM.

Reviewed-by: David Arinzon <darinzon@amazon.com>

RE: [net-next v2 2/2] ena: Link queues to NAPIs

[Arinzon, David]

> Link queues to NAPIs using the netdev-genl API so this information is
> queryable.
> 
> $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \
>                          --dump queue-get --json='{"ifindex": 2}'
> 
> [{'id': 0, 'ifindex': 2, 'napi-id': 8201, 'type': 'rx'},
>  {'id': 1, 'ifindex': 2, 'napi-id': 8202, 'type': 'rx'},
>  {'id': 2, 'ifindex': 2, 'napi-id': 8203, 'type': 'rx'},
>  {'id': 3, 'ifindex': 2, 'napi-id': 8204, 'type': 'rx'},
>  {'id': 4, 'ifindex': 2, 'napi-id': 8205, 'type': 'rx'},
>  {'id': 5, 'ifindex': 2, 'napi-id': 8206, 'type': 'rx'},
>  {'id': 6, 'ifindex': 2, 'napi-id': 8207, 'type': 'rx'},
>  {'id': 7, 'ifindex': 2, 'napi-id': 8208, 'type': 'rx'},
>  {'id': 0, 'ifindex': 2, 'napi-id': 8201, 'type': 'tx'},
>  {'id': 1, 'ifindex': 2, 'napi-id': 8202, 'type': 'tx'},
>  {'id': 2, 'ifindex': 2, 'napi-id': 8203, 'type': 'tx'},
>  {'id': 3, 'ifindex': 2, 'napi-id': 8204, 'type': 'tx'},
>  {'id': 4, 'ifindex': 2, 'napi-id': 8205, 'type': 'tx'},
>  {'id': 5, 'ifindex': 2, 'napi-id': 8206, 'type': 'tx'},
>  {'id': 6, 'ifindex': 2, 'napi-id': 8207, 'type': 'tx'},
>  {'id': 7, 'ifindex': 2, 'napi-id': 8208, 'type': 'tx'}]
> 
> Signed-off-by: Joe Damato <jdamato@fastly.com>
> ---
>  v2:
>    - Comments added to ena_napi_disable_in_range and
>      ena_napi_enable_in_range
>    - No functional changes
> 
>  drivers/net/ethernet/amazon/ena/ena_netdev.c | 28
> +++++++++++++++++---
>  1 file changed, 24 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c
> b/drivers/net/ethernet/amazon/ena/ena_netdev.c
> index 74ce9fa45cf8..96df20854eb9 100644
> --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
> +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
> @@ -1821,20 +1821,40 @@ static void ena_napi_disable_in_range(struct
> ena_adapter *adapter,
>                                       int first_index,
>                                       int count)  {
> +       struct napi_struct *napi;
>         int i;
> 
> -       for (i = first_index; i < first_index + count; i++)
> -               napi_disable(&adapter->ena_napi[i].napi);
> +       for (i = first_index; i < first_index + count; i++) {
> +               napi = &adapter->ena_napi[i].napi;
> +               if (!ENA_IS_XDP_INDEX(adapter, i)) {
> +                       /* This API is supported for non-XDP queues only */
> +                       netif_queue_set_napi(adapter->netdev, i,
> +                                            NETDEV_QUEUE_TYPE_TX, NULL);
> +                       netif_queue_set_napi(adapter->netdev, i,
> +                                            NETDEV_QUEUE_TYPE_RX, NULL);
> +               }
> +               napi_disable(napi);
> +       }
>  }
> 
>  static void ena_napi_enable_in_range(struct ena_adapter *adapter,
>                                      int first_index,
>                                      int count)  {
> +       struct napi_struct *napi;
>         int i;
> 
> -       for (i = first_index; i < first_index + count; i++)
> -               napi_enable(&adapter->ena_napi[i].napi);
> +       for (i = first_index; i < first_index + count; i++) {
> +               napi = &adapter->ena_napi[i].napi;
> +               napi_enable(napi);
> +               if (!ENA_IS_XDP_INDEX(adapter, i)) {
> +                       /* This API is supported for non-XDP queues only */
> +                       netif_queue_set_napi(adapter->netdev, i,
> +                                            NETDEV_QUEUE_TYPE_RX, napi);
> +                       netif_queue_set_napi(adapter->netdev, i,
> +                                            NETDEV_QUEUE_TYPE_TX, napi);
> +               }
> +       }
>  }
> 
>  /* Configure the Rx forwarding */
> --
> 2.25.1

LGTM

Reviewed-by: David Arinzon <darinzon@amazon.com>

RE: [net-next v2 0/2] ena: Link IRQs, queues, and NAPI instances

[Arinzon, David]

> Greetings:
> 
> Welcome to v2. This includes only cosmetic changes, see changelog below
> and in each patch.
> 
> This series uses the netdev-genl API to link IRQs and queues to NAPI IDs so
> that this information is queryable by user apps. This is particularly useful for
> epoll-based busy polling apps which rely on having access to the NAPI ID.
> 
> I've tested these commits on an EC2 instance with an ENA NIC configured
> and have included test output in the commit messages for each patch
> showing how to query the information.
> 
> I noted in the implementation that the driver requests an IRQ for
> management purposes which does not have an associated NAPI. I tried to
> take this into account in patch 1, but would appreciate if ENA maintainers can
> verify I did this correctly.
> 
> Thanks,
> Joe
> 
> v2:
>   - Preserve reverse christmas tree ordering in patch 1
>   - Add comment that the API is for non-XDP queues only to patch 2
> 
> v1:
>   - https://lore.kernel.org/all/20240930195617.37369-1-jdamato@fastly.com/
> 
> Joe Damato (2):
>   ena: Link IRQs to NAPI instances
>   ena: Link queues to NAPIs
> 
>  drivers/net/ethernet/amazon/ena/ena_netdev.c | 40
> +++++++++++++++++---
>  1 file changed, 35 insertions(+), 5 deletions(-)
> 
> --
> 2.25.1

Thanks!

Reviewed-by: David Arinzon <darinzon@amazon.com>

Re: [net-next v2 0/2] ena: Link IRQs, queues, and NAPI instances

[patchwork-bot+netdevbpf]

Hello:

This series was applied to netdev/net-next.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Wed,  2 Oct 2024 00:13:26 +0000 you wrote:
> Greetings:
> 
> Welcome to v2. This includes only cosmetic changes, see changelog below
> and in each patch.
> 
> This series uses the netdev-genl API to link IRQs and queues to NAPI IDs
> so that this information is queryable by user apps. This is particularly
> useful for epoll-based busy polling apps which rely on having access to
> the NAPI ID.
> 
> [...]

Here is the summary with links:
  - [net-next,v2,1/2] ena: Link IRQs to NAPI instances
    https://git.kernel.org/netdev/net-next/c/989867846f7f
  - [net-next,v2,2/2] ena: Link queues to NAPIs
    https://git.kernel.org/netdev/net-next/c/888634377f8e

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html