From: Cristian Prundeanu <cpru@amazon.com>
To: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Cristian Prundeanu <cpru@amazon.com>,
Hazem Mohamed Abuelfotoh <abuehaze@amazon.com>,
Ali Saidi <alisaidi@amazon.com>,
"Benjamin Herrenschmidt" <benh@kernel.crashing.org>,
Geoff Blake <blakgeof@amazon.com>, Borislav Petkov <bp@alien8.de>,
Csaba Csoma <csabac@amazon.com>,
"Dietmar Eggemann" <dietmar.eggemann@arm.com>,
Bjoern Doebel <doebel@amazon.de>,
Gautham Shenoy <gautham.shenoy@amd.com>,
Joseph Salisbury <joseph.salisbury@oracle.com>,
Chris Redpath <chris.redpath@arm.com>,
<linux-arm-kernel@lists.infradead.org>,
<linux-kernel@vger.kernel.org>,
<linux-tip-commits@vger.kernel.org>, <x86@kernel.org>,
Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Swapnil Sapkal <swapnil.sapkal@amd.com>,
Linus Torvalds <torvalds@linux-foundation.org>
Subject: Re: EEVDF regression still exists
Date: Tue, 20 May 2025 13:14:51 -0500 [thread overview]
Message-ID: <20250520181451.18994-1-cpru@amazon.com> (raw)
>> The only _scheduler_ change that looks relevant is commit bbce3de72be5
>> ("sched/eevdf: Fix se->slice being set to U64_MAX and resulting
>> crash"). Which does affect the slice calculation, although supposedly
>> only under special circumstances.>
>> Of course, it could be something else.
>
> Since it is the only !SCHED_EXT change in kernel/sched, Cristian can
> perhaps try reverting it on top of v6.15-rc4 and checking if the
> benchmark results jump back to v6.15-rc3 level to rule that single
> change out. Very likely it could be something else.
I have tested reverting this commit, and the performance indeed jumped back
to rc3 levels.
> The VU count should really be based on the SUT core count, and be at least
> 8 * SUT vCPUs to ensure a full load.
I've modified the reproducer to more accurately configure the VU count
based on the SUT's vCPU count, and use the above multiplier going forward.
Retesting the entire kernel range with the resulting 128 VUs shows a
slightly higher performance everywhere compared to the previous 256 VUs.
The regression is even more visible now, with a few notable points:
- There is a performance inversion from before (6.15-rc3 now underperforms
6.15-rc4). This may be useful data for characterizing the regression.
- Kernel 6.14.7 is about the same as 6.14.6 in default mode, but slower in
SCHED_BATCH mode (-7.1% vs -6.4%).
- Kernel 6.15-rc5 is faster than all other 6.15-rcX builds, especially in
default mode.
- Kernel 6.15-rc7 is worse than 6.15-rc6 everywhere except for the default
mode throughput.
- With either VU value, disabling PLACE_LAG and RUN_TO_PARITY no longer
improves performance significantly on up to date kernels 6.12 and above.
Summary below, full details in the reproducer repo [1].
* All without SCHED_BATCH:
Kernel | Throughput | P50 latency | NOPL+NORTP
aarm64 | (NOPM) | (larger is worse) | (NOPM)
=========+============+===================+============
6.5.13 | baseline | baseline | N/A
---------+------------+-------------------+------------
6.6.91 | -5.7% | +9.9% | -2.6%
---------+------------+-------------------+------------
6.8.12 | -6.0% | +10.7% | -3.4%
---------+------------+-------------------+------------
6.12.29 | -6.8% | +9.5% | -8.0%
---------+------------+-------------------+------------
6.13.12 | -7.6% | +10.5% | -8.5%
---------+------------+-------------------+------------
6.14.7 | -7.0% | +9.8% | -9.8%
---------+------------+-------------------+------------
6.15-rc3 | -8.5% | +11.7% |
---------+------------+-------------------+------------
6.15-rc4 | -7.5% | +10.2% |
---------+------------+-------------------+------------
6.15-rc5 | -6.4% | +8.6 % |
---------+------------+-------------------+------------
6.15-rc6 | -7.5% | +10.4% | -9.0%
---------+------------+-------------------+------------
6.15-rc7 | -7.8% | +11.1% | -8.5%
=========+============+===================+============
* All with SCHED_BATCH:
Kernel | Throughput | P50 latency
aarm64 | (NOPM) | (larger is worse)
=========+============+==================
6.5.13 | baseline | baseline
---------+------------+------------------
6.6.91 | -5.1% | +7.4%
---------+------------+------------------
6.8.12 | -6.0% | +8.6%
---------+------------+------------------
6.12.29 | -6.6% | +8.4%
---------+------------+------------------
6.13.12 | -6.9% | +8.9%
---------+------------+------------------
6.14.7 | -7.1% | +8.7%
---------+------------+------------------
6.15-rc3 | -9.6% | +11.8%
---------+------------+------------------
6.15-rc4 | -7.0% | +8.6%
---------+------------+------------------
6.15-rc5 | -6.6% | +7.9%
---------+------------+------------------
6.15-rc6 | -6.6% | +8.4%
---------+------------+------------------
6.15-rc7 | -7.7% | +9.7%
=========+============+==================
[1] https://github.com/aws/repro-collection/blob/main/repros/repro-mysql-EEVDF-regression/results/20250519/README.md
-Cristian
next reply other threads:[~2025-05-20 18:15 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-05-20 18:14 Cristian Prundeanu [this message]
-- strict thread matches above, loose matches on Subject: below --
2025-04-29 21:38 EEVDF regression still exists Cristian Prundeanu
2025-04-29 21:56 ` Peter Zijlstra
2025-04-29 22:06 ` Prundeanu, Cristian
2025-04-30 3:33 ` K Prateek Nayak
2025-05-01 16:16 ` Prundeanu, Cristian
2025-05-02 5:56 ` K Prateek Nayak
2025-05-02 6:33 ` K Prateek Nayak
2025-05-02 18:06 ` Prundeanu, Cristian
2025-05-02 8:48 ` Peter Zijlstra
2025-05-02 16:52 ` Prundeanu, Cristian
2025-05-14 21:26 ` Dietmar Eggemann
2025-04-30 10:02 ` Peter Zijlstra
2025-05-02 7:08 ` Sapkal, Swapnil
2025-05-02 17:25 ` Prundeanu, Cristian
2025-05-02 17:52 ` Linus Torvalds
2025-05-03 3:34 ` K Prateek Nayak
2025-05-21 13:30 ` Cristian Prundeanu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250520181451.18994-1-cpru@amazon.com \
--to=cpru@amazon.com \
--cc=abuehaze@amazon.com \
--cc=alisaidi@amazon.com \
--cc=benh@kernel.crashing.org \
--cc=blakgeof@amazon.com \
--cc=bp@alien8.de \
--cc=chris.redpath@arm.com \
--cc=csabac@amazon.com \
--cc=dietmar.eggemann@arm.com \
--cc=doebel@amazon.de \
--cc=gautham.shenoy@amd.com \
--cc=joseph.salisbury@oracle.com \
--cc=kprateek.nayak@amd.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-tip-commits@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=swapnil.sapkal@amd.com \
--cc=torvalds@linux-foundation.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox