NIC rx/tx ring size vs XSK rx/tx ring size

NIC rx/tx ring size vs XSK rx/tx ring size

[Daniel Salzman]

Hello,

Please what is the relation between NIC ring sizes and XDP socket ring sizes?
If my app sets XDP_RX_RING and XDP_TX_RING, what are the optimal values for ethtool -G rx/tx?

Also I don't understand why changing TX ring size changes some XDP ring size for the ice driver (Kernel 5.13):
[  424.542118] ice 0000:41:00.0 enp65s0f0: Changing Tx descriptor count from 256 to 2048
[  424.542176] ice 0000:41:00.0 enp65s0f0: Changing XDP descriptor count from 256 to 2048   <- What is that?

In the case of the i40e driver, there is no XDP ring logged:
[847243.925269] i40e 0000:81:00.0 enp129s0f0: Changing Tx descriptor count from 256 to 2048.

Thank you,
Daniel

Re: NIC rx/tx ring size vs XSK rx/tx ring size

[Maciej Fijalkowski]

On Thu, Dec 30, 2021 at 01:21:24PM +0100, Daniel Salzman wrote:
> Hello,
> 
> Please what is the relation between NIC ring sizes and XDP socket ring sizes?
> If my app sets XDP_RX_RING and XDP_TX_RING, what are the optimal values for ethtool -G rx/tx?

I'm working on 1k Tx/Rx HW rings and standard size (2k) of AF_XDP rings.

> 
> Also I don't understand why changing TX ring size changes some XDP ring size for the ice driver (Kernel 5.13):
> [  424.542118] ice 0000:41:00.0 enp65s0f0: Changing Tx descriptor count from 256 to 2048
> [  424.542176] ice 0000:41:00.0 enp65s0f0: Changing XDP descriptor count from 256 to 2048   <- What is that?

I think you're mixing AF_XDP rings with HW rings. Above log comes from the
changing the HW descriptor count - drivers in order to support XDP_TX and
XDP_REDIRECT actions create a set of HW Tx queues that serve XDP and they
are usually referred to as XDP rings.

XDP_{R,T}X_RING are AF_XDP rings and with ethtool -G you're not changing
their size.

> 
> In the case of the i40e driver, there is no XDP ring logged:
> [847243.925269] i40e 0000:81:00.0 enp129s0f0: Changing Tx descriptor count from 256 to 2048.

Behavior is the same for both i40e and ice, it's just that i40e doesn't
log the desc count change when XDP resources are up.

> 
> Thank you,
> Daniel
> 

Re: NIC rx/tx ring size vs XSK rx/tx ring size

[Daniel Salzman]

On 12/30/21 2:04 PM, Maciej Fijalkowski wrote:
> On Thu, Dec 30, 2021 at 01:21:24PM +0100, Daniel Salzman wrote:
>> Hello,
>>
>> Please what is the relation between NIC ring sizes and XDP socket ring sizes?
>> If my app sets XDP_RX_RING and XDP_TX_RING, what are the optimal values for ethtool -G rx/tx?
> 
> I'm working on 1k Tx/Rx HW rings and standard size (2k) of AF_XDP rings.

Okay. When finding the optimal HW ring sizes for a setup with our application (DNS server using 4k AF_XDP rings)
the OS sometimes collapsed and I'm not sure if the HW and XSK ring sizes are fully independent or if there
is some restriction or recommendation on their value combination.

> 
>>
>> Also I don't understand why changing TX ring size changes some XDP ring size for the ice driver (Kernel 5.13):
>> [  424.542118] ice 0000:41:00.0 enp65s0f0: Changing Tx descriptor count from 256 to 2048
>> [  424.542176] ice 0000:41:00.0 enp65s0f0: Changing XDP descriptor count from 256 to 2048   <- What is that?
> 
> I think you're mixing AF_XDP rings with HW rings. Above log comes from the
> changing the HW descriptor count - drivers in order to support XDP_TX and
> XDP_REDIRECT actions create a set of HW Tx queues that serve XDP and they
> are usually referred to as XDP rings.
> 
> XDP_{R,T}X_RING are AF_XDP rings and with ethtool -G you're not changing
> their size

Yes, this is clear. I just didn't know about the additional XDP rings.

> 
>>
>> In the case of the i40e driver, there is no XDP ring logged:
>> [847243.925269] i40e 0000:81:00.0 enp129s0f0: Changing Tx descriptor count from 256 to 2048.
> 
> Behavior is the same for both i40e and ice, it's just that i40e doesn't
> log the desc count change when XDP resources are up.

Thanks!

> 
>>
>> Thank you,
>> Daniel
>>