[Question] Generic way to retrieve IRQ number of Tx/Rx queue

[Question] Generic way to retrieve IRQ number of Tx/Rx queue

[Daniel Sedlak]

Hello,
I am writing an affinity scheduler in the userspace for network cards's 
Tx/Rx queues. Is there a generic way to retrieve all IRQ numbers for 
those queues for each interface?

My goal is to get all Tx/Rx queues for a given interface, get the IRQ 
number of the individual queues, and set an affinity hint for each 
queue. I have tried to loop over /proc/interrupts to retrieve all queues 
for an interface in a hope that the last column would contain the 
interface name however this does not work since the naming is not 
unified across drivers. My second attempt was to retrieve all registered 
interrupts by network interface from 
/sys/class/net/{interface_name}/device/msi_irqs/, but this attempt was 
also without luck because some drivers request more IRQs than the number 
of queues (for example i40e driver).

Thank you for any help or advice

Daniel

Re: [Question] Generic way to retrieve IRQ number of Tx/Rx queue

[Stephen Hemminger]

On Fri, 10 Jan 2025 10:07:18 +0100
Daniel Sedlak <daniel@sedlak.dev> wrote:

> Hello,
> I am writing an affinity scheduler in the userspace for network cards's 
> Tx/Rx queues. Is there a generic way to retrieve all IRQ numbers for 
> those queues for each interface?
> 
> My goal is to get all Tx/Rx queues for a given interface, get the IRQ 
> number of the individual queues, and set an affinity hint for each 
> queue. I have tried to loop over /proc/interrupts to retrieve all queues 
> for an interface in a hope that the last column would contain the 
> interface name however this does not work since the naming is not 
> unified across drivers. My second attempt was to retrieve all registered 
> interrupts by network interface from 
> /sys/class/net/{interface_name}/device/msi_irqs/, but this attempt was 
> also without luck because some drivers request more IRQs than the number 
> of queues (for example i40e driver).
> 
> Thank you for any help or advice
> 
> Daniel
> 

Good luck reinventing irqbalance.

There can be multiple interrupts per queue, or one interrupt can serve multiple queues.

Re: [Question] Generic way to retrieve IRQ number of Tx/Rx queue

[Jakub Kicinski]

On Fri, 10 Jan 2025 10:07:18 +0100 Daniel Sedlak wrote:
> Hello,
> I am writing an affinity scheduler in the userspace for network cards's 
> Tx/Rx queues. Is there a generic way to retrieve all IRQ numbers for 
> those queues for each interface?
> 
> My goal is to get all Tx/Rx queues for a given interface, get the IRQ 
> number of the individual queues, and set an affinity hint for each 
> queue. I have tried to loop over /proc/interrupts to retrieve all queues 
> for an interface in a hope that the last column would contain the 
> interface name however this does not work since the naming is not 
> unified across drivers. My second attempt was to retrieve all registered 
> interrupts by network interface from 
> /sys/class/net/{interface_name}/device/msi_irqs/, but this attempt was 
> also without luck because some drivers request more IRQs than the number 
> of queues (for example i40e driver).

We do have an API for that
https://docs.kernel.org/next/networking/netlink_spec/netdev.html#napi
but unfortunately the driver needs to support it, and i40e currently
doesn't:

$ git grep --files-with-matches  netif_napi_set_irq 
drivers/net/ethernet/amazon/ena/ena_netdev.c
drivers/net/ethernet/broadcom/bnxt/bnxt.c
drivers/net/ethernet/broadcom/tg3.c
drivers/net/ethernet/google/gve/gve_utils.c
drivers/net/ethernet/intel/e1000/e1000_main.c
drivers/net/ethernet/intel/e1000e/netdev.c
drivers/net/ethernet/intel/ice/ice_lib.c
drivers/net/ethernet/intel/igc/igc_main.c
drivers/net/ethernet/mellanox/mlx4/en_cq.c
drivers/net/ethernet/mellanox/mlx5/core/en_main.c
drivers/net/ethernet/meta/fbnic/fbnic_txrx.c

Should be easy to add. Let me CC the Intel list in case they already
have a relevant change queued for i40e..

Re: [Question] Generic way to retrieve IRQ number of Tx/Rx queue

[Daniel Sedlak]

On 1/13/25 10:15 PM, Jakub Kicinski wrote:
> On Fri, 10 Jan 2025 10:07:18 +0100 Daniel Sedlak wrote:
>> Hello,
>> I am writing an affinity scheduler in the userspace for network cards's
>> Tx/Rx queues. Is there a generic way to retrieve all IRQ numbers for
>> those queues for each interface?
>>
>> My goal is to get all Tx/Rx queues for a given interface, get the IRQ
>> number of the individual queues, and set an affinity hint for each
>> queue. I have tried to loop over /proc/interrupts to retrieve all queues
>> for an interface in a hope that the last column would contain the
>> interface name however this does not work since the naming is not
>> unified across drivers. My second attempt was to retrieve all registered
>> interrupts by network interface from
>> /sys/class/net/{interface_name}/device/msi_irqs/, but this attempt was
>> also without luck because some drivers request more IRQs than the number
>> of queues (for example i40e driver).
> 
> We do have an API for that
> https://docs.kernel.org/next/networking/netlink_spec/netdev.html#napi
> but unfortunately the driver needs to support it, and i40e currently
> doesn't:

Thank you for the link, I somehow missed that part of netlink…

> $ git grep --files-with-matches  netif_napi_set_irq
> drivers/net/ethernet/amazon/ena/ena_netdev.c
> drivers/net/ethernet/broadcom/bnxt/bnxt.c
> drivers/net/ethernet/broadcom/tg3.c
> drivers/net/ethernet/google/gve/gve_utils.c
> drivers/net/ethernet/intel/e1000/e1000_main.c
> drivers/net/ethernet/intel/e1000e/netdev.c
> drivers/net/ethernet/intel/ice/ice_lib.c
> drivers/net/ethernet/intel/igc/igc_main.c
> drivers/net/ethernet/mellanox/mlx4/en_cq.c
> drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> drivers/net/ethernet/meta/fbnic/fbnic_txrx.c
> 
> Should be easy to add. Let me CC the Intel list in case they already
> have a relevant change queued for i40e..

Thank you for directions, will check Intel's mailing list and poke 
around with implementing that.

Daniel

Re: [Question] Generic way to retrieve IRQ number of Tx/Rx queue

[Joe Damato]

On Tue, Jan 14, 2025 at 09:32:26AM +0100, Daniel Sedlak wrote:
> 
> 
> On 1/13/25 10:15 PM, Jakub Kicinski wrote:
> > 
> > We do have an API for that
> > https://docs.kernel.org/next/networking/netlink_spec/netdev.html#napi
> > but unfortunately the driver needs to support it, and i40e currently
> > doesn't:
> 
> Thank you for the link, I somehow missed that part of netlink...
> 
> > $ git grep --files-with-matches  netif_napi_set_irq
> > drivers/net/ethernet/amazon/ena/ena_netdev.c
> > drivers/net/ethernet/broadcom/bnxt/bnxt.c
> > drivers/net/ethernet/broadcom/tg3.c
> > drivers/net/ethernet/google/gve/gve_utils.c
> > drivers/net/ethernet/intel/e1000/e1000_main.c
> > drivers/net/ethernet/intel/e1000e/netdev.c
> > drivers/net/ethernet/intel/ice/ice_lib.c
> > drivers/net/ethernet/intel/igc/igc_main.c
> > drivers/net/ethernet/mellanox/mlx4/en_cq.c
> > drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> > drivers/net/ethernet/meta/fbnic/fbnic_txrx.c
> > 
> > Should be easy to add. Let me CC the Intel list in case they already
> > have a relevant change queued for i40e..
> 
> Thank you for directions, will check Intel's mailing list and poke around
> with implementing that.

I previously tried to add support for this API to i40e [1], but got
pulled into other stuff and never picked it back up. Wanted to
mention it in case it is useful for you.

[1]: https://lore.kernel.org/lkml/20240410043936.206169-1-jdamato@fastly.com/