From: Matt Fleming <matt@readmodwrite.com>
To: Tejun Heo <tj@kernel.org>, Andrea Righi <arighi@nvidia.com>
Cc: "Paul E . McKenney" <paulmck@kernel.org>,
sched-ext@lists.linux.dev, linux-kernel@vger.kernel.org,
kernel-team@cloudflare.com
Subject: sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path
Date: Tue, 9 Jun 2026 11:47:33 +0100 [thread overview]
Message-ID: <20260609104733.1184001-1-mfleming@cloudflare.com> (raw)
Hi there,
We're investigating a hard lockup on a 6.18.33-based kernel with scx_lavd
running.
The vmcore shows CPU#67 stuck in:
native_queued_spin_lock_slowpath
_raw_spin_lock
task_rq_lock
sched_ext_free
__put_task_struct
rcu_core
handle_softirqs
irq_exit_rcu
sysvec_apic_timer_interrupt
The rq lock being waited on is for CPU#66.
Another CPU in the same dump is in:
sched_ext_free
scx_exit_task
bpf_task_storage_delete
call_rcu_tasks_generic+547
_printk
console_unlock
wake_up_q
try_to_wake_up
__task_rq_lock
The `call_rcu_tasks_generic+547` site is the old needadjust path that
prints:
Switching RCU Tasks Trace to per-CPU callback queuing.
So the current theory is that a task teardown under rq lock enters
bpf_task_storage_delete(), hits the callback-queue adjustment printk,
then console wakeup tries to acquire the same rq lock again.
We found related upstream changes:
3063b33a347c ("rcu-tasks: Avoid raw-spinlocked wakeups from call_rcu_tasks_generic()")
d245698d727a ("cgroup: Defer task cgroup unlink until after the task is done switching out")
7900aa699c34 ("sched_ext: Fix cgroup exit ordering by moving sched_ext_free() to finish_task_switch()")
7c405fb3279b ("rcu: Use an intermediate irq_work to start process_srcu()")
But none appears to directly fix the old 6.18 needadjust printk path.
Would backporting d245698d727a and 7900aa699c34 be useful, or should
the needadjust printk path itself be deferred away from rq-locked
callers?
Thanks,
Matt
next reply other threads:[~2026-06-09 10:47 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-06-09 10:47 Matt Fleming [this message]
2026-06-09 11:23 ` sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path Paul E. McKenney
2026-06-11 13:02 ` Matt Fleming
2026-06-11 13:45 ` Paul E. McKenney
2026-06-12 10:55 ` Matt Fleming
2026-06-12 14:00 ` Paul E. McKenney
2026-06-16 10:56 ` Matt Fleming
2026-06-17 3:24 ` Paul E. McKenney
2026-06-18 6:56 ` Matt Fleming
2026-06-18 15:55 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260609104733.1184001-1-mfleming@cloudflare.com \
--to=matt@readmodwrite.com \
--cc=arighi@nvidia.com \
--cc=kernel-team@cloudflare.com \
--cc=linux-kernel@vger.kernel.org \
--cc=paulmck@kernel.org \
--cc=sched-ext@lists.linux.dev \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox