Re: XDP Performance Regression in recent kernel versions

[Tariq Toukan <tariqt@nvidia.com>]

On 21/06/2024 15:35, Samuel Dobron wrote:
> Hey all,
> 
> Yeah, we do tests for ELN kernels [1] on a regular basis. Since
> ~January of this year.
> 
> As already mentioned, mlx5 is the only driver affected by this regression.
> Unfortunately, I think Jesper is actually hitting 2 regressions we noticed,
> the one already mentioned by Toke, another one [0] has been reported
> in early February.
> Btw. issue mentioned by Toke has been moved to Jira, see [5].
> 
> Not sure all of you are able to see the content of [0], Jira says it's
> RH-confidental.
> So, I am not sure how much I can share without being fired :D. Anyway,
> affected kernels have been released a while ago, so anyone can find it
> on its own.
> Basically, we detected 5% regression on XDP_DROP+mlx5 (currently, we
> don't have data for any other XDP mode) in kernel-5.14 compared to
> previous builds.
> 
>  From tests history, I can see (most likely) the same improvement
> on 6.10rc2 (from 15Mpps to 17-18Mpps), so I'd say 20% drop has been
> (partially) fixed?
> 
> For earlier 6.10. kernels we don't have data due to [3] (there is regression on
> XDP_DROP as well, but I believe it's turbo-boost issue, as I mentioned
> in issue).
> So if you want to run tests on 6.10. please see [3].
> 
> Summary XDP_DROP+mlx5@25G:
> kernel       pps
> <5.14        20.5M        baseline
>> =5.14      19M           [0]
> <6.4          19-20M      baseline for ELN kernels
>> =6.4        15M           [4 and 5] (mentioned by Toke)

+ @Dragos

That's about when we added several changes to the RX datapath.
Most relevant are:
- Fully removing the in-driver RX page-cache.
- Refactoring to support XDP multi-buffer.

We tested XDP performance before submission, I don't recall we noticed 
such a degradation.

I'll check with Dragos as he probably has these reports.

>> =6.10      ???            [3]
>> =6.10rc2 17M-18M
> 
> 
>> It looks like this is known since March, was this ever reported to Nvidia back
>> then? :/
> 
> Not sure if that's a question for me, I was told, filling an issue in
> Bugzilla/Jira is where
> our competences end. Who is supposed to report it to them?
> 
>> Given XDP is in the critical path for many in production, we should think about
>> regular performance reporting for the different vendors for each released kernel,
>> similar to here [0].
> 
> I think this might be the part of upstream kernel testing with LNST?
> Maybe Jesper
> knows more about that? Until then, I think, I can let you know about
> new regressions we catch.
> 
> Thanks,
> Sam.
> 
> [0] https://issues.redhat.com/browse/RHEL-24054
> [1] https://koji.fedoraproject.org/koji/search?terms=kernel-%5Cd.*eln*&type=build&match=regexp
> [2] https://koji.fedoraproject.org/koji/buildinfo?buildID=2469107
> [3] https://bugzilla.redhat.com/show_bug.cgi?id=2282969
> [4] https://bugzilla.redhat.com/show_bug.cgi?id=2270408
> [5] https://issues.redhat.com/browse/RHEL-24054
>