The Linux Kernel Mailing List
 help / color / mirror / Atom feed
From: Matt Fleming <matt@readmodwrite.com>
To: Tejun Heo <tj@kernel.org>, Andrea Righi <arighi@nvidia.com>
Cc: "Paul E . McKenney" <paulmck@kernel.org>,
	sched-ext@lists.linux.dev, linux-kernel@vger.kernel.org,
	kernel-team@cloudflare.com
Subject: sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path
Date: Tue,  9 Jun 2026 11:47:33 +0100	[thread overview]
Message-ID: <20260609104733.1184001-1-mfleming@cloudflare.com> (raw)

Hi there,

We're investigating a hard lockup on a 6.18.33-based kernel with scx_lavd
running.

The vmcore shows CPU#67 stuck in:

  native_queued_spin_lock_slowpath
  _raw_spin_lock
  task_rq_lock
  sched_ext_free
  __put_task_struct
  rcu_core
  handle_softirqs
  irq_exit_rcu
  sysvec_apic_timer_interrupt

The rq lock being waited on is for CPU#66.

Another CPU in the same dump is in:

  sched_ext_free
  scx_exit_task
  bpf_task_storage_delete
  call_rcu_tasks_generic+547
  _printk
  console_unlock
  wake_up_q
  try_to_wake_up
  __task_rq_lock

The `call_rcu_tasks_generic+547` site is the old needadjust path that
prints:

  Switching RCU Tasks Trace to per-CPU callback queuing.

So the current theory is that a task teardown under rq lock enters
bpf_task_storage_delete(), hits the callback-queue adjustment printk,
then console wakeup tries to acquire the same rq lock again.

We found related upstream changes:

  3063b33a347c ("rcu-tasks: Avoid raw-spinlocked wakeups from call_rcu_tasks_generic()")
  d245698d727a ("cgroup: Defer task cgroup unlink until after the task is done switching out")
  7900aa699c34 ("sched_ext: Fix cgroup exit ordering by moving sched_ext_free() to finish_task_switch()")
  7c405fb3279b ("rcu: Use an intermediate irq_work to start process_srcu()")

But none appears to directly fix the old 6.18 needadjust printk path.

Would backporting d245698d727a and 7900aa699c34 be useful, or should
the needadjust printk path itself be deferred away from rq-locked
callers?

Thanks,
Matt

             reply	other threads:[~2026-06-09 10:47 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-06-09 10:47 Matt Fleming [this message]
2026-06-09 11:23 ` sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path Paul E. McKenney
2026-06-11 13:02   ` Matt Fleming
2026-06-11 13:45     ` Paul E. McKenney
2026-06-12 10:55       ` Matt Fleming
2026-06-12 14:00         ` Paul E. McKenney
2026-06-16 10:56           ` Matt Fleming
2026-06-17  3:24             ` Paul E. McKenney
2026-06-18  6:56               ` Matt Fleming
2026-06-18 15:55                 ` Paul E. McKenney

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260609104733.1184001-1-mfleming@cloudflare.com \
    --to=matt@readmodwrite.com \
    --cc=arighi@nvidia.com \
    --cc=kernel-team@cloudflare.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=paulmck@kernel.org \
    --cc=sched-ext@lists.linux.dev \
    --cc=tj@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox