AF_XDP busy poll receives packets in batches of 8 on i40e

AF_XDP busy poll receives packets in batches of 8 on i40e

[Federico Parola]

Hello everybody,
I'm experiencing a strange problem when running an AF_XDP application 
with busy poll enabled on a Intel XL710 NIC (i40e driver).
The problem can be replicated running the xdpsock kernel sample in rx or 
l2fwd mode.
The first packet I send to the machine is correctly received by the 
application. After this, packets are only received in batches of 8.
If I send 7 packets the application sees nothing, while the 8th one 
triggers the reception of all 8 packets.
Disabling the busy poll mode everything works fine and packets are 
immediately received as they are sent.

I tried changing kernel (5.12, 5.14 and 5.16) but all present the same 
problem.
I also tried using another NIC, an Intel X540 with ixgbe driver and the 
problem isn't there, so I guess is NIC/driver related.

I tried monitoring ethtool statistics. When sending packets between 1 
and 7 these counters are increased:
stat:           64 (             64) <= port.rx_bytes /sec
stat:            1 (              1) <= port.rx_size_64 /sec
stat:            1 (              1) <= port.rx_unicast /sec
stat:            1 (              1) <= rx_unicast /sec

While the 8th one triggers this updates:
stat:           64 (             64) <= port.rx_bytes /sec
stat:            1 (              1) <= port.rx_size_64 /sec
stat:            1 (              1) <= port.rx_unicast /sec
stat:          477 (            477) <= rx-0.bytes /sec
stat:            8 (              8) <= rx-0.packets /sec
stat:          477 (            477) <= rx_bytes /sec
stat:            8 (              8) <= rx_packets /sec
stat:            1 (              1) <= rx_unicast /sec

As far as I understand the first set of counters are hardware counters, 
so it makes me think that packets are kept in the NIC and not even sent 
to memory.

Does anyone have any suggestion on what could be causing this problem? 
Does enabling busy poll set some flag on the NIC?

Best regards,
Federico Parola

Re: AF_XDP busy poll receives packets in batches of 8 on i40e

[Jay Vosburgh]

Federico Parola <federico.parola@polito.it> wrote:

>Hello everybody,
>I'm experiencing a strange problem when running an AF_XDP application with
>busy poll enabled on a Intel XL710 NIC (i40e driver).
>The problem can be replicated running the xdpsock kernel sample in rx or
>l2fwd mode.
>The first packet I send to the machine is correctly received by the
>application. After this, packets are only received in batches of 8.
>If I send 7 packets the application sees nothing, while the 8th one
>triggers the reception of all 8 packets.
>Disabling the busy poll mode everything works fine and packets are
>immediately received as they are sent.
>
>I tried changing kernel (5.12, 5.14 and 5.16) but all present the same
>problem.
>I also tried using another NIC, an Intel X540 with ixgbe driver and the
>problem isn't there, so I guess is NIC/driver related.
>
>I tried monitoring ethtool statistics. When sending packets between 1 and
>7 these counters are increased:
>stat:           64 (             64) <= port.rx_bytes /sec
>stat:            1 (              1) <= port.rx_size_64 /sec
>stat:            1 (              1) <= port.rx_unicast /sec
>stat:            1 (              1) <= rx_unicast /sec
>
>While the 8th one triggers this updates:
>stat:           64 (             64) <= port.rx_bytes /sec
>stat:            1 (              1) <= port.rx_size_64 /sec
>stat:            1 (              1) <= port.rx_unicast /sec
>stat:          477 (            477) <= rx-0.bytes /sec
>stat:            8 (              8) <= rx-0.packets /sec
>stat:          477 (            477) <= rx_bytes /sec
>stat:            8 (              8) <= rx_packets /sec
>stat:            1 (              1) <= rx_unicast /sec
>
>As far as I understand the first set of counters are hardware counters, so
>it makes me think that packets are kept in the NIC and not even sent to
>memory.
>
>Does anyone have any suggestion on what could be causing this problem?
>Does enabling busy poll set some flag on the NIC?

	We observed similar "batching" behavior on i40e devices late
last year in ordinary use (not XDP, but using SR-IOV VFs).  We
instrumented the drivers at the send and receive sides, and determined
that it appeared to be a behavior of the receiving device itself, i.e.,
packets 1 - 7 would be held indefinitely (as I recall, no interrupt or
update of the RX ring pointers) until packet 8 arrives, at which point
all 8 were delivered simultaneously.

	The issue was evidently in the firmware, and was resolved after
a firmware upgrade.

	-J

>
>Best regards,
>Federico Parola

---
	-Jay Vosburgh, jay.vosburgh@canonical.com

Re: AF_XDP busy poll receives packets in batches of 8 on i40e

[Federico Parola]

On 17/03/22 15:59, Jay Vosburgh wrote:
> Federico Parola <federico.parola@polito.it> wrote:
> 
>> Hello everybody,
>> I'm experiencing a strange problem when running an AF_XDP application with
>> busy poll enabled on a Intel XL710 NIC (i40e driver).
>> The problem can be replicated running the xdpsock kernel sample in rx or
>> l2fwd mode.
>> The first packet I send to the machine is correctly received by the
>> application. After this, packets are only received in batches of 8.
>> If I send 7 packets the application sees nothing, while the 8th one
>> triggers the reception of all 8 packets.
>> Disabling the busy poll mode everything works fine and packets are
>> immediately received as they are sent.
>>
>> I tried changing kernel (5.12, 5.14 and 5.16) but all present the same
>> problem.
>> I also tried using another NIC, an Intel X540 with ixgbe driver and the
>> problem isn't there, so I guess is NIC/driver related.
>>
>> I tried monitoring ethtool statistics. When sending packets between 1 and
>> 7 these counters are increased:
>> stat:           64 (             64) <= port.rx_bytes /sec
>> stat:            1 (              1) <= port.rx_size_64 /sec
>> stat:            1 (              1) <= port.rx_unicast /sec
>> stat:            1 (              1) <= rx_unicast /sec
>>
>> While the 8th one triggers this updates:
>> stat:           64 (             64) <= port.rx_bytes /sec
>> stat:            1 (              1) <= port.rx_size_64 /sec
>> stat:            1 (              1) <= port.rx_unicast /sec
>> stat:          477 (            477) <= rx-0.bytes /sec
>> stat:            8 (              8) <= rx-0.packets /sec
>> stat:          477 (            477) <= rx_bytes /sec
>> stat:            8 (              8) <= rx_packets /sec
>> stat:            1 (              1) <= rx_unicast /sec
>>
>> As far as I understand the first set of counters are hardware counters, so
>> it makes me think that packets are kept in the NIC and not even sent to
>> memory.
>>
>> Does anyone have any suggestion on what could be causing this problem?
>> Does enabling busy poll set some flag on the NIC?
> 
> 	We observed similar "batching" behavior on i40e devices late
> last year in ordinary use (not XDP, but using SR-IOV VFs).  We
> instrumented the drivers at the send and receive sides, and determined
> that it appeared to be a behavior of the receiving device itself, i.e.,
> packets 1 - 7 would be held indefinitely (as I recall, no interrupt or
> update of the RX ring pointers) until packet 8 arrives, at which point
> all 8 were delivered simultaneously.
> 
> 	The issue was evidently in the firmware, and was resolved after
> a firmware upgrade.

Hi Jay,
I just updated the firmware to the latest version (v8.50 from v8.30) but 
unfortunately the problem is still there.
However I'm experiencing the problem only when using AF_XDP in busy poll 
mode, all other modes (standard AF_XDP and normal packet reception) work 
just fine.
Maybe the two problems are correlated in some way.

> 
> 	-J
> 
>>
>> Best regards,
>> Federico Parola
> 
> ---
> 	-Jay Vosburgh, jay.vosburgh@canonical.com

Re: AF_XDP busy poll receives packets in batches of 8 on i40e

[Jesse Brandeburg]

On 3/17/2022 8:34 AM, Federico Parola wrote:
>>     We observed similar "batching" behavior on i40e devices late
>> last year in ordinary use (not XDP, but using SR-IOV VFs).  We
>> instrumented the drivers at the send and receive sides, and determined
>> that it appeared to be a behavior of the receiving device itself, i.e.,
>> packets 1 - 7 would be held indefinitely (as I recall, no interrupt or
>> update of the RX ring pointers) until packet 8 arrives, at which point
>> all 8 were delivered simultaneously.
>>
>>     The issue was evidently in the firmware, and was resolved after
>> a firmware upgrade.
> 
> Hi Jay,
> I just updated the firmware to the latest version (v8.50 from v8.30) but 
> unfortunately the problem is still there.
> However I'm experiencing the problem only when using AF_XDP in busy poll 
> mode, all other modes (standard AF_XDP and normal packet reception) work 
> just fine.
> Maybe the two problems are correlated in some way.

This sounds related to the WB_ON_ITR feature in our hardware. If 
interrupts are disabled the driver needs to set that bit (and an ITR 
value) so that packets get written back in a timely manner and don't 
just wait for a cache line edge (I bet your Cache Line Size value in 
PCIe space (lspci) is set to 128?)

Maybe something to chase there? We may also be able to look into it if 
it wasn't fixed already upstream.

Re: AF_XDP busy poll receives packets in batches of 8 on i40e

[Jay Vosburgh]

Jesse Brandeburg <jesse.brandeburg@intel.com> wrote:

>On 3/17/2022 8:34 AM, Federico Parola wrote:
>>>     We observed similar "batching" behavior on i40e devices late
>>> last year in ordinary use (not XDP, but using SR-IOV VFs).  We
>>> instrumented the drivers at the send and receive sides, and determined
>>> that it appeared to be a behavior of the receiving device itself, i.e.,
>>> packets 1 - 7 would be held indefinitely (as I recall, no interrupt or
>>> update of the RX ring pointers) until packet 8 arrives, at which point
>>> all 8 were delivered simultaneously.
>>>
>>>     The issue was evidently in the firmware, and was resolved after
>>> a firmware upgrade.
>> Hi Jay,
>> I just updated the firmware to the latest version (v8.50 from v8.30) but
>> unfortunately the problem is still there.
>> However I'm experiencing the problem only when using AF_XDP in busy poll
>> mode, all other modes (standard AF_XDP and normal packet reception) work
>> just fine.
>> Maybe the two problems are correlated in some way.
>
>This sounds related to the WB_ON_ITR feature in our hardware. If
>interrupts are disabled the driver needs to set that bit (and an ITR
>value) so that packets get written back in a timely manner and don't just
>wait for a cache line edge (I bet your Cache Line Size value in PCIe space
>(lspci) is set to 128?)

	FWIW, in the case we had, driver changes (5.14 upstream, the
then-current Intel out of tree driver) didn't change the behavior.  The
PCI Cache Line Size was 64 bytes; the device was

13:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 562SFP+ Adapter

	From an lspci -vvv perspective, failing and non-failing cards
differed only in the serial numbers.

>Maybe something to chase there? We may also be able to look into it if it
>wasn't fixed already upstream.

	-J

---
	-Jay Vosburgh, jay.vosburgh@canonical.com

Re: AF_XDP busy poll receives packets in batches of 8 on i40e

[Federico Parola]

On 18/03/22 00:26, Jay Vosburgh wrote:
> Jesse Brandeburg <jesse.brandeburg@intel.com> wrote:
> 
>> On 3/17/2022 8:34 AM, Federico Parola wrote:
>>>>      We observed similar "batching" behavior on i40e devices late
>>>> last year in ordinary use (not XDP, but using SR-IOV VFs).  We
>>>> instrumented the drivers at the send and receive sides, and determined
>>>> that it appeared to be a behavior of the receiving device itself, i.e.,
>>>> packets 1 - 7 would be held indefinitely (as I recall, no interrupt or
>>>> update of the RX ring pointers) until packet 8 arrives, at which point
>>>> all 8 were delivered simultaneously.
>>>>
>>>>      The issue was evidently in the firmware, and was resolved after
>>>> a firmware upgrade.
>>> Hi Jay,
>>> I just updated the firmware to the latest version (v8.50 from v8.30) but
>>> unfortunately the problem is still there.
>>> However I'm experiencing the problem only when using AF_XDP in busy poll
>>> mode, all other modes (standard AF_XDP and normal packet reception) work
>>> just fine.
>>> Maybe the two problems are correlated in some way.
>>
>> This sounds related to the WB_ON_ITR feature in our hardware. If
>> interrupts are disabled the driver needs to set that bit (and an ITR
>> value) so that packets get written back in a timely manner and don't just
>> wait for a cache line edge (I bet your Cache Line Size value in PCIe space
>> (lspci) is set to 128?)
> 
> 	FWIW, in the case we had, driver changes (5.14 upstream, the
> then-current Intel out of tree driver) didn't change the behavior.  The
> PCI Cache Line Size was 64 bytes; the device was
> 
> 13:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
>          Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 562SFP+ Adapter
> 
> 	From an lspci -vvv perspective, failing and non-failing cards
> differed only in the serial numbers.
> 

I tested it on 3 different XL710 NICs, all updated to the latest 8.50 
firmware, and they all present the same 8 packets "batching" behavior.
How do I check the PCI Cache Line Size (I didn't find such entry in 
lspci -vvv)? Normal cache line size is 64B on my system, but I guess 
they are two different things.

>> Maybe something to chase there? We may also be able to look into it if it
>> wasn't fixed already upstream.
> 
> 	-J
> 
> ---
> 	-Jay Vosburgh, jay.vosburgh@canonical.com

Re: AF_XDP busy poll receives packets in batches of 8 on i40e

[Jay Vosburgh]

Federico Parola <federico.parola@polito.it> wrote:

>On 18/03/22 00:26, Jay Vosburgh wrote:
>> Jesse Brandeburg <jesse.brandeburg@intel.com> wrote:
>> 
>>> On 3/17/2022 8:34 AM, Federico Parola wrote:
>>>>>      We observed similar "batching" behavior on i40e devices late
>>>>> last year in ordinary use (not XDP, but using SR-IOV VFs).  We
>>>>> instrumented the drivers at the send and receive sides, and determined
>>>>> that it appeared to be a behavior of the receiving device itself, i.e.,
>>>>> packets 1 - 7 would be held indefinitely (as I recall, no interrupt or
>>>>> update of the RX ring pointers) until packet 8 arrives, at which point
>>>>> all 8 were delivered simultaneously.
>>>>>
>>>>>      The issue was evidently in the firmware, and was resolved after
>>>>> a firmware upgrade.
>>>> Hi Jay,
>>>> I just updated the firmware to the latest version (v8.50 from v8.30) but
>>>> unfortunately the problem is still there.
>>>> However I'm experiencing the problem only when using AF_XDP in busy poll
>>>> mode, all other modes (standard AF_XDP and normal packet reception) work
>>>> just fine.
>>>> Maybe the two problems are correlated in some way.
>>>
>>> This sounds related to the WB_ON_ITR feature in our hardware. If
>>> interrupts are disabled the driver needs to set that bit (and an ITR
>>> value) so that packets get written back in a timely manner and don't just
>>> wait for a cache line edge (I bet your Cache Line Size value in PCIe space
>>> (lspci) is set to 128?)
>> 	FWIW, in the case we had, driver changes (5.14 upstream, the
>> then-current Intel out of tree driver) didn't change the behavior.  The
>> PCI Cache Line Size was 64 bytes; the device was
>> 13:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710
>> for 10GbE SFP+ (rev 02)
>>          Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 562SFP+ Adapter
>> 	From an lspci -vvv perspective, failing and non-failing cards
>> differed only in the serial numbers.
>> 
>
>I tested it on 3 different XL710 NICs, all updated to the latest 8.50
>firmware, and they all present the same 8 packets "batching" behavior.
>How do I check the PCI Cache Line Size (I didn't find such entry in lspci
>-vvv)? Normal cache line size is 64B on my system, but I guess they are
>two different things.

	It's in the lspci -vvv, near the top, e.g.,

08:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
        Subsystem: Intel Corporation Ethernet Converged Network Adapter XL710-Q2
        Physical Slot: 1
        Control: [...bunch of stuff...]
        Status: [...bunch of stuff...]
        Latency: 0, Cache Line Size: 64 bytes

	Checking the PCIe spec, though, I'm not sure why this would
matter, as PCIe 4.0 7.5.1.3 Cache Line Size Register says it "has no
effect on any PCI Express device behavior."

	-J

---
	-Jay Vosburgh, jay.vosburgh@canonical.com

Re: AF_XDP busy poll receives packets in batches of 8 on i40e

[Jesse Brandeburg]

On 3/18/2022 11:33 AM, Jay Vosburgh wrote:
>> I tested it on 3 different XL710 NICs, all updated to the latest 8.50
>> firmware, and they all present the same 8 packets "batching" behavior.
>> How do I check the PCI Cache Line Size (I didn't find such entry in lspci
>> -vvv)? Normal cache line size is 64B on my system, but I guess they are
>> two different things.
> 
> 	It's in the lspci -vvv, near the top, e.g.,
> 
> 08:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
>          Subsystem: Intel Corporation Ethernet Converged Network Adapter XL710-Q2
>          Physical Slot: 1
>          Control: [...bunch of stuff...]
>          Status: [...bunch of stuff...]
>          Latency: 0, Cache Line Size: 64 bytes
> 
> 	Checking the PCIe spec, though, I'm not sure why this would
> matter, as PCIe 4.0 7.5.1.3 Cache Line Size Register says it "has no
> effect on any PCI Express device behavior."

Thanks Jay, that might just be the sound of my brain short circuiting, 
as I thought that held some relevance for some reason. Our receive 
descriptors are 16 bytes (with options for 32 bytes sometimes), and the 
transmit are 16. 8*16=128, so I was suspecting something related to 
cache line size in the PCIe is triggering a writeback when we get to 
that boundary, it's probably some other trigger I don't remember.

I know the XL710 has a documented erratum where while interrupts are 
disabled (like while polling) that hardware won't write back until it 
hits a writeback criteria, or unless the interrupt is re-enabled. We 
have a bunch of workarounds in the driver trying to address this, I was 
thinking maybe one of those workarounds isn't hitting in the XDP code.

Oh, and WB_ON_ITR didn't appear until the X722 generation, so it's not 
there on XL710. Hope this helps!

Jesse

Re: AF_XDP busy poll receives packets in batches of 8 on i40e

[Jay Vosburgh]

Federico Parola <federico.parola@polito.it> wrote:

>On 17/03/22 15:59, Jay Vosburgh wrote:
>> Federico Parola <federico.parola@polito.it> wrote:
>> 
>>> Hello everybody,
>>> I'm experiencing a strange problem when running an AF_XDP application with
>>> busy poll enabled on a Intel XL710 NIC (i40e driver).
>>> The problem can be replicated running the xdpsock kernel sample in rx or
>>> l2fwd mode.
>>> The first packet I send to the machine is correctly received by the
>>> application. After this, packets are only received in batches of 8.
>>> If I send 7 packets the application sees nothing, while the 8th one
>>> triggers the reception of all 8 packets.
>>> Disabling the busy poll mode everything works fine and packets are
>>> immediately received as they are sent.
>>>
>>> I tried changing kernel (5.12, 5.14 and 5.16) but all present the same
>>> problem.
>>> I also tried using another NIC, an Intel X540 with ixgbe driver and the
>>> problem isn't there, so I guess is NIC/driver related.
>>>
>>> I tried monitoring ethtool statistics. When sending packets between 1 and
>>> 7 these counters are increased:
>>> stat:           64 (             64) <= port.rx_bytes /sec
>>> stat:            1 (              1) <= port.rx_size_64 /sec
>>> stat:            1 (              1) <= port.rx_unicast /sec
>>> stat:            1 (              1) <= rx_unicast /sec
>>>
>>> While the 8th one triggers this updates:
>>> stat:           64 (             64) <= port.rx_bytes /sec
>>> stat:            1 (              1) <= port.rx_size_64 /sec
>>> stat:            1 (              1) <= port.rx_unicast /sec
>>> stat:          477 (            477) <= rx-0.bytes /sec
>>> stat:            8 (              8) <= rx-0.packets /sec
>>> stat:          477 (            477) <= rx_bytes /sec
>>> stat:            8 (              8) <= rx_packets /sec
>>> stat:            1 (              1) <= rx_unicast /sec
>>>
>>> As far as I understand the first set of counters are hardware counters, so
>>> it makes me think that packets are kept in the NIC and not even sent to
>>> memory.
>>>
>>> Does anyone have any suggestion on what could be causing this problem?
>>> Does enabling busy poll set some flag on the NIC?
>> 	We observed similar "batching" behavior on i40e devices late
>> last year in ordinary use (not XDP, but using SR-IOV VFs).  We
>> instrumented the drivers at the send and receive sides, and determined
>> that it appeared to be a behavior of the receiving device itself, i.e.,
>> packets 1 - 7 would be held indefinitely (as I recall, no interrupt or
>> update of the RX ring pointers) until packet 8 arrives, at which point
>> all 8 were delivered simultaneously.
>> 	The issue was evidently in the firmware, and was resolved after
>> a firmware upgrade.
>
>Hi Jay,
>I just updated the firmware to the latest version (v8.50 from v8.30) but
>unfortunately the problem is still there.
>However I'm experiencing the problem only when using AF_XDP in busy poll
>mode, all other modes (standard AF_XDP and normal packet reception) work
>just fine.
>Maybe the two problems are correlated in some way.

	I don't have the firmware release versions for the issue we
looked into (it was at a customer of ours), but the fixed firmware was
provided by HPE/Intel via a support ticket.  Not all devices exhibited
the problem, even with identical firmware; the end solution was to
either apply the vendor's new firmware or replace the cards showing the
problem.  I also don't recall if the fix was the firmware, the NVM
image, or both (at probe time, the i40e driver prints firmware, api and
nvm versions into the kernel dmesg log).

	We saw packet "batches" of both 4 and 8.  Nevertheless, given
the very specific "batching packets into groups of N" behavior, it seems
unlikely to be an unrelated problem.

	-J

---
	-Jay Vosburgh, jay.vosburgh@canonical.com