Re: XDP Performance Regression in recent kernel versions

[Daniel Borkmann <daniel@iogearbox.net>]

On 6/19/24 9:17 PM, Toke Høiland-Jørgensen wrote:
> Jesper Dangaard Brouer <hawk@kernel.org> writes:
>> On 18/06/2024 17.28, Sebastiano Miano wrote:
>>> I have been conducting some basic experiments with XDP and have
>>> observed a significant performance regression in recent kernel
>>> versions compared to v5.15.
>>>
>>> My setup is the following:
>>> - Hardware: Two machines connected back-to-back with 100G Mellanox
>>> ConnectX-6 Dx.
>>> - DUT: 2x16 core Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz.
>>> - Software: xdp-bench program from [1] running on the DUT in both DROP
>>> and TX modes.
>>> - Traffic generator: Pktgen-DPDK sending traffic with a single 64B UDP
>>> flow at ~130Mpps.
>>> - Tests: Single core, HT disabled
>>>
>>> Results:
>>>
>>> Kernel version |-------| XDP_DROP |--------|   XDP_TX  |
>>> 5.15                      30Mpps               16.1Mpps
>>> 6.2                       21.3Mpps             14.1Mpps
>>> 6.5                       19.9Mpps              8.6Mpps
>>> bpf-next (6.10-rc2)       22.1Mpps              9.2Mpps
>>>
>>
>> Around when I left Red Hat there were a project with [LNST] that used
>> xdp-bench for tracking and finding regressions like this.
>>
>> Perhaps Toke can enlighten us, if that project have caught similar
>> regressions?
>>
>> [LNST] https://github.com/LNST-project/lnst
> 
> Yes, actually, we have! Here's the bugzilla for it:
> https://bugzilla.redhat.com/show_bug.cgi?id=2270408

 > We compared performance of ELN and RHEL9 candidate kernels and noticed significant
 > drop in XDP drop [1] on mlx5 (25G).
 >
 > On any rhel9 candidate kernel we are able to drop 19-20M pkts/sec but on an ELN
 > kernels, we are reaching just 15M pkts/sec (CPU utillization remains the same -
 > around 100%).
 >
 > We don't see such regression on ixgbe or i40e.

It looks like this is known since March, was this ever reported to Nvidia back
then? :/

Given XDP is in the critical path for many in production, we should think about
regular performance reporting for the different vendors for each released kernel,
similar to here [0].

Out of curiosity, @Saeed: Is Nvidia internally regularly assessing XDP perf for mlx5
as part of QA? (Probably not, but I thought I'd ask.)

Thanks,
Daniel

   [0] http://core.dpdk.org/perf-reports/