From: Mihail Mihov <mihovmihailp@gmail.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: mingo@kernel.org, longman@redhat.com, chenridong@huaweicloud.com,
juri.lelli@redhat.com, vincent.guittot@linaro.org,
dietmar.eggemann@arm.com, rostedt@goodmis.org,
bsegall@google.com, mgorman@suse.de, vschneid@redhat.com,
tj@kernel.org, hannes@cmpxchg.org, mkoutny@suse.com,
cgroups@vger.kernel.org, linux-kernel@vger.kernel.org,
jstultz@google.com, kprateek.nayak@amd.com, qyousef@layalina.io,
regressions@lists.linux.dev
Subject: [REGRESSION] core scheduling latency with idle load
Date: Mon, 16 Nov 2026 23:22:07 +0100 [thread overview]
Message-ID: <avt_mdiPQ5gwAisa@lima-fedora> (raw)
In-Reply-To: <20260605124052.227463677@infradead.org>
Hi Peter,
I was running some experiments that co-locate a latency-sensitive
service with SCHED_IDLE batch work in a cpu.idle=1 cgroup, with the
service in its own core-scheduling cookie. After updating to 7.3 its
tail latency got much worse. With the help of an LLM I bisected it to
85570f10a4c6 ("sched/eevdf: Move to a single runqueue") and reduced it
to the script below. It is still present in v7.4-rc3.
Reproducer (root, SMT machine, CONFIG_SCHED_CORE, cyclictest):
#!/bin/sh
set -e
mkdir -p /sys/fs/cgroup/be
echo 1 > /sys/fs/cgroup/be/cpu.idle
trap 'echo 1 > /sys/fs/cgroup/be/cgroup.kill; sleep 1; rmdir /sys/fs/cgroup/be' EXIT
# one SCHED_IDLE spinner per CPU in the cpu.idle=1 cgroup
chrt -i 0 sh -c 'echo $$ > /sys/fs/cgroup/be/cgroup.procs; for i in $(seq $(nproc)); do while :; do :; done & done'
sleep 1
# p99 from cyclictest's 1 us histogram, overflows (> 1 s) included
p99() { awk '/^[0-9]/ { h[$1+0] = $2; n += $2 } /Overflows/ { o = $4 } END { n += o; for (i = 0; c < n * 0.99; i++) c += h[i]; print "p99", i - 1, "us" }'; }
printf 'no cookie: '; cyclictest -q -m -i 1000 -D 10 -h 1000000 | p99
printf 'cookie: '; coresched new -- cyclictest -q -m -i 1000 -D 10 -h 1000000 | p99
i7-1185G7 (4C/8T), vanilla kernels built with the same config (Gentoo's
distribution kernel config, HZ=300, PREEMPT_LAZY). p99 in us:
kernel no cookie cookie
85570f10a4c6^ 54 59
85570f10a4c6 53 3695
v7.3.4 54 3276
v7.4-rc3 53 3290
I also tried the two cpu.idle preemption fixes under discussion [1][2]
on v7.3.4; the cookie p99 stayed around 3.1 ms with both.
Changing cgroup_mode (up, smp, concur, max, tasks) on v7.4-rc3 did not
fix it.
On Fri, Jun 05, 2026 at 02:40:20PM +0200, Peter Zijlstra wrote:
> @@ -14893,30 +14803,8 @@ bool cfs_prio_less(const struct task_str
[...]
> - se_fi_update(sea, rq->core->core_forceidle_seq, in_fi);
> - se_fi_update(seb, rq->core->core_forceidle_seq, in_fi);
In case it helps: adding these two calls back on v7.3.4 brought the
cookie p99 back to 55 us. I don't know whether that is the right thing
to do, only that it changes this result.
Thanks,
Mihail
[1] https://lore.kernel.org/all/20261019182622.1584454-1-vincent.guittot@linaro.org/
[2] https://lore.kernel.org/all/20261105165525.011876858@infradead.org/
#regzbot introduced: 85570f10a4c6
next prev parent reply other threads:[~2026-11-16 22:22 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-06-05 12:40 [PATCH v3 0/7] sched: Flatten the pick Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 1/7] sched/fair: Add cgroup_mode switch Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 2/7] sched/fair: Add cgroup_mode: up Peter Zijlstra
2026-06-05 15:07 ` Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 3/7] sched/fair: Add cgroup_mode: max Peter Zijlstra
2026-06-10 15:09 ` Waiman Long
2026-06-10 15:42 ` Waiman Long
2026-06-11 13:49 ` Peter Zijlstra
2026-06-11 13:47 ` Peter Zijlstra
2026-06-11 20:57 ` Waiman Long
2026-06-05 12:40 ` [PATCH v3 4/7] sched/fair: Add cgroup_mode: concur Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 5/7] sched/fair: Add cgroup_mode: tasks Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 6/7] sched/fair: Change the default cgroup_mode to concur Peter Zijlstra
2026-06-05 12:40 ` [PATCH v3 7/7] sched/eevdf: Move to a single runqueue Peter Zijlstra
2026-06-20 3:54 ` Chen, Yu C
2026-06-26 11:40 ` Peter Zijlstra
2026-06-29 14:02 ` Vincent Guittot
2026-09-23 14:21 ` [REGRESSION] " Aishwarya Rambhadran
2026-09-23 16:48 ` Chen Yu
2026-09-24 7:05 ` K Prateek Nayak
2026-09-24 8:19 ` Mike Galbraith
2026-09-28 3:12 ` Chen Yu
2026-09-28 13:00 ` Mike Galbraith
2026-10-07 12:05 ` Aishwarya Rambhadran
2026-10-12 11:27 ` Mike Galbraith
2026-10-16 13:18 ` Jakub Okoński
2026-10-18 6:49 ` Mike Galbraith
2026-10-19 17:01 ` Vincent Guittot
2026-10-19 17:35 ` Jakub Okoński
2026-10-19 18:04 ` Vincent Guittot
2026-10-19 18:24 ` Jakub Okoński
2026-10-19 18:26 ` Vincent Guittot
2026-10-21 11:55 ` Vincent Guittot
2026-10-22 12:05 ` Jakub Okoński
2026-10-22 2:12 ` Mike Galbraith
2026-11-16 22:22 ` Mihail Mihov [this message]
2026-06-09 5:37 ` [PATCH v3 0/7] sched: Flatten the pick K Prateek Nayak
2026-06-12 2:29 ` Shubhang Kaushik
2026-08-17 16:05 ` Szabina Korbai
2026-08-17 16:35 ` K Prateek Nayak
2026-08-18 9:04 ` Szabina Korbai
2026-08-18 9:16 ` Peter Zijlstra
2026-08-21 10:37 ` Szabina Korbai
2026-08-24 14:51 ` Chen Yu
2026-08-24 13:53 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=avt_mdiPQ5gwAisa@lima-fedora \
--to=mihovmihailp@gmail.com \
--cc=bsegall@google.com \
--cc=cgroups@vger.kernel.org \
--cc=chenridong@huaweicloud.com \
--cc=dietmar.eggemann@arm.com \
--cc=hannes@cmpxchg.org \
--cc=jstultz@google.com \
--cc=juri.lelli@redhat.com \
--cc=kprateek.nayak@amd.com \
--cc=linux-kernel@vger.kernel.org \
--cc=longman@redhat.com \
--cc=mgorman@suse.de \
--cc=mingo@kernel.org \
--cc=mkoutny@suse.com \
--cc=peterz@infradead.org \
--cc=qyousef@layalina.io \
--cc=regressions@lists.linux.dev \
--cc=rostedt@goodmis.org \
--cc=tj@kernel.org \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox