[Intel-wired-lan] ixgbe limits on core count

[Intel-wired-lan] ixgbe limits on core count

[Brian Topping]

Hello experts! 

I?m setting up a machine with two E5-2696 v3 CPUs using XDP on kernel 5.10.6-051006-generic and running into the constraint at https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c#n10136 <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c#n10136>. 

Any advice what the most practical long term solution is?

Short term, I can dial back the number of usable cores. I haven?t dug into the bits to understand if this constraint is based on hardware limits on the card or this was a choice that was made when processors had fewer cores.

Thanks kindly!

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20210118/64ba4e55/attachment.html>

[Intel-wired-lan] ixgbe limits on core count

[Maciej Fijalkowski]

On Mon, Jan 18, 2021 at 04:19:23AM -0700, Brian Topping wrote:
> Hello experts! 
> 
> I?m setting up a machine with two E5-2696 v3 CPUs using XDP on kernel
> 5.10.6-051006-generic and running into the constraint at
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c#n10136
> <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c#n10136>. 
> 

Hi Brian,
we're aware of this issue, thanks for bringing this up. We plan to address
this in some near future.

> Any advice what the most practical long term solution is?
> 
> Short term, I can dial back the number of usable cores. I haven?t dug
> into the bits to understand if this constraint is based on hardware
> limits on the card or this was a choice that was made when processors
> had fewer cores.

ixgbe has only 128 Tx HW queues AFAIK, so currently we have a check
against cpu count and do not load XDP for case when machine has more than
64 cpus as XDP needs to have a dedicated HW Tx queue so that it is fully
operational.

I think that solution (or workaround, not sure how to relate to that) is
picking what is left in terms of HW Tx resources and assigning it to XDP.

If sending happens on a cpu nr 70:

cpu_nr = 70
cpu_cnt = 72;
xdp_tx_cnt = 56;

then we pick (cpu_nr % xdp_tx_cnt) + cpu_cnt = (70 % 56) + 72 = 86 HW Tx
queue. At the same time, same queue could be accessed from cpu 14 if I am
not mistaken, so this brings the need for locking. We would be detecting
such configuration at the time when XDP prog is being loaded onto
interface and picking that 'fallback' mode. For < 64 cpu machines, current
scheme will be picked so we wouldn't be hurting performance via unneeded
lock acquiring for such systems.

Maciej

> 
> Thanks kindly!
> 
> Brian

> _______________________________________________
> Intel-wired-lan mailing list
> Intel-wired-lan at osuosl.org
> https://lists.osuosl.org/mailman/listinfo/intel-wired-lan