Re: Network performance regression in Linux kernel 6.6 for small socket size test cases

Re: Network performance regression in Linux kernel 6.6 for small socket size test cases

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 2095 bytes --]

[also Cc: regressions ML]

On Wed, Feb 28, 2024 at 12:13:27PM +0530, Abdul Anshad Azeez wrote:
> During performance regression workload execution of the Linux
> kernel we observed up to 30% performance decrease in a specific networking
> workload on the 6.6 kernel compared to 6.5 (details below). The regression is
> reproducible in both Linux VMs running on ESXi and bare metal Linux.
> 
> Workload details:
> 
> Benchmark - Netperf TCP_STREAM
> Socket buffer size - 8K
> Message size - 256B
> MTU - 1500B
> Socket option - TCP_NODELAY
> # of STREAMs - 32
> Direction - Uni-Directional Receive
> Duration - 60 Seconds
> NIC - Mellanox Technologies ConnectX-6 Dx EN 100G
> Server Config - Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz & 512G Memory
> 
> Bisect between 6.5 and 6.6 kernel concluded that this regression originated
> from the below commit:
> 
> commit - dfa2f0483360d4d6f2324405464c9f281156bd87 (tcp: get rid of
> sysctl_tcp_adv_win_scale)
> Author - Eric Dumazet <edumazet@google.com>
> Link -
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=
> dfa2f0483360d4d6f2324405464c9f281156bd87
> 
> Performance data for (Linux VM on ESXi):
> Test case - TCP_STREAM_RECV Throughput in Gbps
> (for different socket buffer sizes and with constant message size - 256B):
> 
> Socket buffer size - [LK6.5 vs LK6.6]
> 8K - [8.4 vs 5.9 Gbps]
> 16K - [13.4 vs 10.6 Gbps]
> 32K - [19.1 vs 16.3 Gbps]
> 64K - [19.6 vs 19.7 Gbps]
> Autotune - [19.7 vs 19.6 Gbps]
> 
> >From the above performance data, we can infer that:
> * Regression is specific to lower fixed socket buffer sizes (8K, 16K & 32K).
> * Increasing the socket buffer size gradually decreases the throughput impact.
> * Performance is equal for higher fixed socket size (64K) and Autotune socket
> tests.
> 
> We would like to know if there are any opportunities for optimization in
> the test cases with small socket sizes.
> 

Can you verify the regression on current mainline (v6.8-rc6)?

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Network performance regression in Linux kernel 6.6 for small socket size test cases

[Linux regression tracking (Thorsten Leemhuis)]

On 28.02.24 09:32, Bagas Sanjaya wrote:
> [also Cc: regressions ML]
> 
> On Wed, Feb 28, 2024 at 12:13:27PM +0530, Abdul Anshad Azeez wrote:
>> During performance regression workload execution of the Linux
>> kernel we observed up to 30% performance decrease in a specific networking
>> workload on the 6.6 kernel compared to 6.5 (details below). The regression is
>> reproducible in both Linux VMs running on ESXi and bare metal Linux.
>>
>> [...]
>>
>> We would like to know if there are any opportunities for optimization in
>> the test cases with small socket sizes.
> 
> Can you verify the regression on current mainline (v6.8-rc6)?

Bagas, I know that you are trying to help, but this is not helpful at
all (and indirectly puts regression tracking and the kernel development
community into a bad light).

Asking that question can be the right thing sometimes, for example in a
bugzilla ticket where the reporter is clearly reporting their first bug.
But the quoted report above clearly does not fall into that category for
various obvious reasons.

If you want to ensure that reports like that are acted upon, wait at
least two or three work days and see if there is a reply from a
developer. In case there is none (which happens, but I assume for a bug
report like this is likely rare) prodding a bit can be okay. But even
then you definitely want to use a more friendly tone. Maybe something
like "None of the developers reacted yet; maybe none of them bothered to
take a closer look because it's unclear if the problem still happens
with the latest code. You thus might want to verify and report back if
the problem happens with latest mainline, maybe then someone will take a
closer look".

Okay, that has way too many "maybe" in it, but I'm sure you'll get the
idea. :-D

Ciao, Thorsten

Re: Network performance regression in Linux kernel 6.6 for small socket size test cases

[Bagas Sanjaya]

On 2/28/24 16:09, Linux regression tracking (Thorsten Leemhuis) wrote:
> On 28.02.24 09:32, Bagas Sanjaya wrote:
>> [also Cc: regressions ML]
>>
>> On Wed, Feb 28, 2024 at 12:13:27PM +0530, Abdul Anshad Azeez wrote:
>>> During performance regression workload execution of the Linux
>>> kernel we observed up to 30% performance decrease in a specific networking
>>> workload on the 6.6 kernel compared to 6.5 (details below). The regression is
>>> reproducible in both Linux VMs running on ESXi and bare metal Linux.
>>>
>>> [...]
>>>
>>> We would like to know if there are any opportunities for optimization in
>>> the test cases with small socket sizes.
>>
>> Can you verify the regression on current mainline (v6.8-rc6)?
> 
> Bagas, I know that you are trying to help, but this is not helpful at
> all (and indirectly puts regression tracking and the kernel development
> community into a bad light).
> 
> Asking that question can be the right thing sometimes, for example in a
> bugzilla ticket where the reporter is clearly reporting their first bug.
> But the quoted report above clearly does not fall into that category for
> various obvious reasons.
> 
> If you want to ensure that reports like that are acted upon, wait at
> least two or three work days and see if there is a reply from a
> developer. In case there is none (which happens, but I assume for a bug
> report like this is likely rare) prodding a bit can be okay. But even
> then you definitely want to use a more friendly tone. Maybe something
> like "None of the developers reacted yet; maybe none of them bothered to
> take a closer look because it's unclear if the problem still happens
> with the latest code. You thus might want to verify and report back if
> the problem happens with latest mainline, maybe then someone will take a
> closer look".
> 
> Okay, that has way too many "maybe" in it, but I'm sure you'll get the
> idea. :-D
> 

Oops, I'm always impatient (and forgot to privately mail you) in this case.
Sorry for inconvenience.

-- 
An old man doll... just what I always wanted! - Clara