Re: AF_XDP busy poll receives packets in batches of 8 on i40e

[Jay Vosburgh <jay.vosburgh@canonical.com>]

Federico Parola <federico.parola@polito.it> wrote:

>On 18/03/22 00:26, Jay Vosburgh wrote:
>> Jesse Brandeburg <jesse.brandeburg@intel.com> wrote:
>> 
>>> On 3/17/2022 8:34 AM, Federico Parola wrote:
>>>>>      We observed similar "batching" behavior on i40e devices late
>>>>> last year in ordinary use (not XDP, but using SR-IOV VFs).  We
>>>>> instrumented the drivers at the send and receive sides, and determined
>>>>> that it appeared to be a behavior of the receiving device itself, i.e.,
>>>>> packets 1 - 7 would be held indefinitely (as I recall, no interrupt or
>>>>> update of the RX ring pointers) until packet 8 arrives, at which point
>>>>> all 8 were delivered simultaneously.
>>>>>
>>>>>      The issue was evidently in the firmware, and was resolved after
>>>>> a firmware upgrade.
>>>> Hi Jay,
>>>> I just updated the firmware to the latest version (v8.50 from v8.30) but
>>>> unfortunately the problem is still there.
>>>> However I'm experiencing the problem only when using AF_XDP in busy poll
>>>> mode, all other modes (standard AF_XDP and normal packet reception) work
>>>> just fine.
>>>> Maybe the two problems are correlated in some way.
>>>
>>> This sounds related to the WB_ON_ITR feature in our hardware. If
>>> interrupts are disabled the driver needs to set that bit (and an ITR
>>> value) so that packets get written back in a timely manner and don't just
>>> wait for a cache line edge (I bet your Cache Line Size value in PCIe space
>>> (lspci) is set to 128?)
>> 	FWIW, in the case we had, driver changes (5.14 upstream, the
>> then-current Intel out of tree driver) didn't change the behavior.  The
>> PCI Cache Line Size was 64 bytes; the device was
>> 13:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710
>> for 10GbE SFP+ (rev 02)
>>          Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 562SFP+ Adapter
>> 	From an lspci -vvv perspective, failing and non-failing cards
>> differed only in the serial numbers.
>> 
>
>I tested it on 3 different XL710 NICs, all updated to the latest 8.50
>firmware, and they all present the same 8 packets "batching" behavior.
>How do I check the PCI Cache Line Size (I didn't find such entry in lspci
>-vvv)? Normal cache line size is 64B on my system, but I guess they are
>two different things.

	It's in the lspci -vvv, near the top, e.g.,

08:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
        Subsystem: Intel Corporation Ethernet Converged Network Adapter XL710-Q2
        Physical Slot: 1
        Control: [...bunch of stuff...]
        Status: [...bunch of stuff...]
        Latency: 0, Cache Line Size: 64 bytes

	Checking the PCIe spec, though, I'm not sure why this would
matter, as PCIe 4.0 7.5.1.3 Cache Line Size Register says it "has no
effect on any PCI Express device behavior."

	-J

---
	-Jay Vosburgh, jay.vosburgh@canonical.com