From: Boqun Feng <boqun@kernel.org>
To: Emil Renner Berthing <emil.renner.berthing@canonical.com>
Cc: Joel Fernandes <joelagnelf@nvidia.com>,
"Paul E. McKenney" <paulmck@kernel.org>,
Kumar Kartikeya Dwivedi <memxor@gmail.com>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
frederic@kernel.org, neeraj.iitr10@gmail.com, urezki@gmail.com,
boqun.feng@gmail.com, rcu@vger.kernel.org,
Tejun Heo <tj@kernel.org>,
bpf@vger.kernel.org, Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
John Fastabend <john.fastabend@gmail.com>,
Andrea Righi <arighi@nvidia.com>, Zqiang <qiang.zhang@linux.dev>,
Samuel Holland <samuel.holland@sifive.com>,
linux-riscv@lists.infradead.org,
Sarah Emery <sarah.emery@canonical.com>
Subject: Re: [PATCH v2] rcu: Use an intermediate irq_work to start process_srcu()
Date: Mon, 22 Jun 2026 08:38:10 -0700 [thread overview]
Message-ID: <ajlW4oSzEqav6gxc@MacBook-0RXW5> (raw)
In-Reply-To: <CAJM55Z_15U7GssTOims1=J34CYSj=8CDE_di1kJYs5hE3=J53w@mail.gmail.com>
On Sat, Jun 20, 2026 at 05:25:25AM -0400, Emil Renner Berthing wrote:
> Quoting Boqun Feng (2026-06-18 17:30:10)
> > On Thu, Jun 18, 2026 at 08:11:36AM -0500, Emil Renner Berthing wrote:
> > > Quoting Boqun Feng (2026-03-20 23:29:16)
> > > > Since commit c27cea4416a3 ("rcu: Re-implement RCU Tasks Trace in terms
> > > > of SRCU-fast") we switched to SRCU in BPF. However as BPF instrument can
> > > > happen basically everywhere (including where a scheduler lock is held),
> > > > call_srcu() now needs to avoid acquiring scheduler lock because
> > > > otherwise it could cause deadlock [1]. Fix this by following what the
> > > > previous RCU Tasks Trace did: using an irq_work to delay the queuing of
> > > > the work to start process_srcu().
> > > >
> > > > [boqun: Apply Joel's feedback]
> > > > [boqun: Apply Andrea's test feedback]
> > > >
> > > > Reported-by: Andrea Righi <arighi@nvidia.com>
> > > > Closes: https://lore.kernel.org/all/abjzvz_tL_siV17s@gpd4/
> > > > Fixes: commit c27cea4416a3 ("rcu: Re-implement RCU Tasks Trace in terms of SRCU-fast")
> > > > Link: https://lore.kernel.org/rcu/3c4c5a29-24ea-492d-aeee-e0d9605b4183@nvidia.com/ [1]
> > > > Suggested-by: Zqiang <qiang.zhang@linux.dev>
> > > > Tested-by: Andrea Righi <arighi@nvidia.com>
> > > > Signed-off-by: Boqun Feng <boqun@kernel.org>
> > >
> > > Thanks for the patch, which is now upstream as
> > > 7c405fb3279b ("rcu: Use an intermediate irq_work to start process_srcu()")
> > >
> > > Unfortunately it seems to halt booting on the Allwinner D1 single-core RISC-V
> > > SoC both on the 7.0 and 7.1 kernels:
> > >
> >
> > By "single-core", is it only one CPU in that system?
> >
> > > https://esmil.dk/d1-7.0.txt
> > > https://esmil.dk/d1-7.1.txt
> > >
>
> Yes, the SoC only has one CPU core capable of running Linux at least.
>
> > From the callstack,
> >
> > [ 243.002818] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 243.010681] task:kworker/0:0 state:D stack:0 pid:9 tgid:9 ppid:2 task_flags:0x4208060 flags:0x00000000
> > [ 243.021858] Workqueue: device_link_wq device_link_release_fn
> > [ 243.027587] Call Trace:
> > [ 243.030079] [<ffffffff80745b66>] __schedule+0x232/0x530
> > [ 243.035365] [<ffffffff80745e82>] schedule+0x1e/0x94
> > [ 243.040298] [<ffffffff8074aa7a>] schedule_timeout+0x6e/0xa8
> > [ 243.045929] [<ffffffff8004d6a0>] wakeup_preempt+0x8c/0x98
> > [ 243.051380] [<ffffffff80746aee>] wait_for_completion+0x3a/0xb0
> > [ 243.057267] [<ffffffff80087226>] __synchronize_srcu.part.0+0x4e/0x60
> > [ 243.063712] [<ffffffff80084414>] rcu_tasks_get_gp_data+0xc/0x10
> > [ 243.069699] [<ffffffff80490898>] device_link_release_fn+0x14/0x80
> > [ 243.075850] [<ffffffff8003dbd0>] process_one_work+0xf8/0x1c8
> > [ 243.081583] [<ffffffff8003e282>] worker_thread+0x11a/0x254
> > [ 243.087127] [<ffffffff8003e164>] rescuer_thread+0x3a0/0x3a4
> > [ 243.092755] [<ffffffff8003e164>] rescuer_thread+0x3a0/0x3a4
> > [ 243.098381] [<ffffffff8004499e>] kthread+0xbe/0xe4
> > [ 243.103231] [<ffffffff8001349e>] ret_from_fork_kernel+0x6/0xd0
> > [ 243.109127] [<ffffffff800517a8>] schedule_tail+0x8/0xac
> > [ 243.114410] [<ffffffff800448dc>] kthreads_online_cpu+0x0/0x4
> > [ 243.120121] [<ffffffff8074be0a>] ret_from_fork_kernel_asm+0x12/0x18
> >
> > I will guess for some reasons the irq_work was missed or the IPI never
> > happened. Is it possible for you to enable trace points ipi_send_cpu and
> > ipi_entry to get more information?
>
> I have to admit I've not used tracepoints before, but it seems like only
> ipi_send_cpu and ipi_send_cpumask is available on RISC-V.
> I've now added trace_event=ipi:* and tp_printk to the command line, and this
Thank you! Yeah, this is what I meant ;-)
> is what I get:
> https://esmil.dk/d1-7.0-2.txt
>
[ 0.071132] rcu: Max phase no-delay instances is 1000.
[ 0.077837] ipi_send_cpu: cpu=0 callsite=irq_work_queue+0x22/0x4c callback=srcu_irq_work+0x0/0x58
Is this the last line you can see from your (serial?) console, or we are
missing a few more lines? Maybe it'll should the same hung task
messages?
The ipi_send_cpu message means RCU was able to send an IPI, now if there
is no ipi_handler between the send and the hung task, that would mean
the IPI was missed.
Regards,
Boqun
> Let me know if there is something I'm missing or something else I can do to help
> find the bug.
>
> /Emil
WARNING: multiple messages have this Message-ID (diff)
From: Boqun Feng <boqun@kernel.org>
To: Emil Renner Berthing <emil.renner.berthing@canonical.com>
Cc: Joel Fernandes <joelagnelf@nvidia.com>,
"Paul E. McKenney" <paulmck@kernel.org>,
Kumar Kartikeya Dwivedi <memxor@gmail.com>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
frederic@kernel.org, neeraj.iitr10@gmail.com, urezki@gmail.com,
boqun.feng@gmail.com, rcu@vger.kernel.org,
Tejun Heo <tj@kernel.org>,
bpf@vger.kernel.org, Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
John Fastabend <john.fastabend@gmail.com>,
Andrea Righi <arighi@nvidia.com>, Zqiang <qiang.zhang@linux.dev>,
Samuel Holland <samuel.holland@sifive.com>,
linux-riscv@lists.infradead.org,
Sarah Emery <sarah.emery@canonical.com>
Subject: Re: [PATCH v2] rcu: Use an intermediate irq_work to start process_srcu()
Date: Mon, 22 Jun 2026 08:38:10 -0700 [thread overview]
Message-ID: <ajlW4oSzEqav6gxc@MacBook-0RXW5> (raw)
In-Reply-To: <CAJM55Z_15U7GssTOims1=J34CYSj=8CDE_di1kJYs5hE3=J53w@mail.gmail.com>
On Sat, Jun 20, 2026 at 05:25:25AM -0400, Emil Renner Berthing wrote:
> Quoting Boqun Feng (2026-06-18 17:30:10)
> > On Thu, Jun 18, 2026 at 08:11:36AM -0500, Emil Renner Berthing wrote:
> > > Quoting Boqun Feng (2026-03-20 23:29:16)
> > > > Since commit c27cea4416a3 ("rcu: Re-implement RCU Tasks Trace in terms
> > > > of SRCU-fast") we switched to SRCU in BPF. However as BPF instrument can
> > > > happen basically everywhere (including where a scheduler lock is held),
> > > > call_srcu() now needs to avoid acquiring scheduler lock because
> > > > otherwise it could cause deadlock [1]. Fix this by following what the
> > > > previous RCU Tasks Trace did: using an irq_work to delay the queuing of
> > > > the work to start process_srcu().
> > > >
> > > > [boqun: Apply Joel's feedback]
> > > > [boqun: Apply Andrea's test feedback]
> > > >
> > > > Reported-by: Andrea Righi <arighi@nvidia.com>
> > > > Closes: https://lore.kernel.org/all/abjzvz_tL_siV17s@gpd4/
> > > > Fixes: commit c27cea4416a3 ("rcu: Re-implement RCU Tasks Trace in terms of SRCU-fast")
> > > > Link: https://lore.kernel.org/rcu/3c4c5a29-24ea-492d-aeee-e0d9605b4183@nvidia.com/ [1]
> > > > Suggested-by: Zqiang <qiang.zhang@linux.dev>
> > > > Tested-by: Andrea Righi <arighi@nvidia.com>
> > > > Signed-off-by: Boqun Feng <boqun@kernel.org>
> > >
> > > Thanks for the patch, which is now upstream as
> > > 7c405fb3279b ("rcu: Use an intermediate irq_work to start process_srcu()")
> > >
> > > Unfortunately it seems to halt booting on the Allwinner D1 single-core RISC-V
> > > SoC both on the 7.0 and 7.1 kernels:
> > >
> >
> > By "single-core", is it only one CPU in that system?
> >
> > > https://esmil.dk/d1-7.0.txt
> > > https://esmil.dk/d1-7.1.txt
> > >
>
> Yes, the SoC only has one CPU core capable of running Linux at least.
>
> > From the callstack,
> >
> > [ 243.002818] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 243.010681] task:kworker/0:0 state:D stack:0 pid:9 tgid:9 ppid:2 task_flags:0x4208060 flags:0x00000000
> > [ 243.021858] Workqueue: device_link_wq device_link_release_fn
> > [ 243.027587] Call Trace:
> > [ 243.030079] [<ffffffff80745b66>] __schedule+0x232/0x530
> > [ 243.035365] [<ffffffff80745e82>] schedule+0x1e/0x94
> > [ 243.040298] [<ffffffff8074aa7a>] schedule_timeout+0x6e/0xa8
> > [ 243.045929] [<ffffffff8004d6a0>] wakeup_preempt+0x8c/0x98
> > [ 243.051380] [<ffffffff80746aee>] wait_for_completion+0x3a/0xb0
> > [ 243.057267] [<ffffffff80087226>] __synchronize_srcu.part.0+0x4e/0x60
> > [ 243.063712] [<ffffffff80084414>] rcu_tasks_get_gp_data+0xc/0x10
> > [ 243.069699] [<ffffffff80490898>] device_link_release_fn+0x14/0x80
> > [ 243.075850] [<ffffffff8003dbd0>] process_one_work+0xf8/0x1c8
> > [ 243.081583] [<ffffffff8003e282>] worker_thread+0x11a/0x254
> > [ 243.087127] [<ffffffff8003e164>] rescuer_thread+0x3a0/0x3a4
> > [ 243.092755] [<ffffffff8003e164>] rescuer_thread+0x3a0/0x3a4
> > [ 243.098381] [<ffffffff8004499e>] kthread+0xbe/0xe4
> > [ 243.103231] [<ffffffff8001349e>] ret_from_fork_kernel+0x6/0xd0
> > [ 243.109127] [<ffffffff800517a8>] schedule_tail+0x8/0xac
> > [ 243.114410] [<ffffffff800448dc>] kthreads_online_cpu+0x0/0x4
> > [ 243.120121] [<ffffffff8074be0a>] ret_from_fork_kernel_asm+0x12/0x18
> >
> > I will guess for some reasons the irq_work was missed or the IPI never
> > happened. Is it possible for you to enable trace points ipi_send_cpu and
> > ipi_entry to get more information?
>
> I have to admit I've not used tracepoints before, but it seems like only
> ipi_send_cpu and ipi_send_cpumask is available on RISC-V.
> I've now added trace_event=ipi:* and tp_printk to the command line, and this
Thank you! Yeah, this is what I meant ;-)
> is what I get:
> https://esmil.dk/d1-7.0-2.txt
>
[ 0.071132] rcu: Max phase no-delay instances is 1000.
[ 0.077837] ipi_send_cpu: cpu=0 callsite=irq_work_queue+0x22/0x4c callback=srcu_irq_work+0x0/0x58
Is this the last line you can see from your (serial?) console, or we are
missing a few more lines? Maybe it'll should the same hung task
messages?
The ipi_send_cpu message means RCU was able to send an IPI, now if there
is no ipi_handler between the send and the hung task, that would mean
the IPI was missed.
Regards,
Boqun
> Let me know if there is something I'm missing or something else I can do to help
> find the bug.
>
> /Emil
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
next prev parent reply other threads:[~2026-06-22 15:38 UTC|newest]
Thread overview: 120+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-03-17 13:34 Next-level bug in SRCU implementation of RCU Tasks Trace + PREEMPT_RT Paul E. McKenney
2026-03-18 10:50 ` Sebastian Andrzej Siewior
2026-03-18 11:49 ` Paul E. McKenney
2026-03-18 14:43 ` Sebastian Andrzej Siewior
2026-03-18 15:43 ` Paul E. McKenney
2026-03-18 16:04 ` Sebastian Andrzej Siewior
2026-03-18 16:32 ` Paul E. McKenney
2026-03-18 16:42 ` Boqun Feng
2026-03-18 18:45 ` Paul E. McKenney
2026-03-18 16:47 ` Sebastian Andrzej Siewior
2026-03-18 18:48 ` Paul E. McKenney
2026-03-19 8:55 ` Sebastian Andrzej Siewior
2026-03-19 10:05 ` Paul E. McKenney
2026-03-19 10:43 ` Paul E. McKenney
2026-03-19 10:51 ` Sebastian Andrzej Siewior
2026-03-18 15:51 ` Boqun Feng
2026-03-18 18:42 ` Paul E. McKenney
2026-03-18 20:04 ` Joel Fernandes
2026-03-18 20:11 ` Kumar Kartikeya Dwivedi
2026-03-18 20:25 ` Joel Fernandes
2026-03-18 21:52 ` Boqun Feng
2026-03-18 21:55 ` Boqun Feng
2026-03-18 22:15 ` Boqun Feng
2026-03-18 22:52 ` Joel Fernandes
2026-03-18 23:27 ` Boqun Feng
2026-03-19 1:08 ` Boqun Feng
2026-03-19 9:03 ` Sebastian Andrzej Siewior
2026-03-19 16:27 ` Boqun Feng
2026-03-19 16:33 ` Sebastian Andrzej Siewior
2026-03-19 16:48 ` Boqun Feng
2026-03-19 16:59 ` Kumar Kartikeya Dwivedi
2026-03-19 17:27 ` Boqun Feng
2026-03-19 18:41 ` Kumar Kartikeya Dwivedi
2026-03-19 20:14 ` Boqun Feng
2026-03-19 20:21 ` Joel Fernandes
2026-03-19 20:39 ` Boqun Feng
2026-03-20 15:34 ` Paul E. McKenney
2026-03-20 15:59 ` Boqun Feng
2026-03-20 16:24 ` Paul E. McKenney
2026-03-20 16:57 ` Boqun Feng
2026-03-20 17:54 ` Joel Fernandes
2026-03-20 18:14 ` [PATCH] rcu: Use an intermediate irq_work to start process_srcu() Boqun Feng
2026-03-20 19:18 ` Joel Fernandes
2026-03-20 20:47 ` Andrea Righi
2026-03-20 20:54 ` Boqun Feng
2026-03-20 21:00 ` Andrea Righi
2026-03-20 21:02 ` Andrea Righi
2026-03-20 21:06 ` Boqun Feng
2026-03-20 22:29 ` [PATCH v2] " Boqun Feng
2026-03-23 21:09 ` Joel Fernandes
2026-03-23 22:18 ` Boqun Feng
2026-03-23 22:50 ` Joel Fernandes
2026-03-24 11:27 ` Frederic Weisbecker
2026-03-24 14:56 ` Joel Fernandes
2026-03-24 14:56 ` Alexei Starovoitov
2026-03-24 17:36 ` Boqun Feng
2026-03-24 18:40 ` Joel Fernandes
2026-03-24 19:23 ` Paul E. McKenney
2026-03-26 19:12 ` patchwork-bot+netdevbpf
2026-06-18 13:11 ` Emil Renner Berthing
2026-06-18 13:11 ` Emil Renner Berthing
2026-06-18 15:30 ` Boqun Feng
2026-06-18 15:30 ` Boqun Feng
2026-06-20 9:25 ` Emil Renner Berthing
2026-06-20 9:25 ` Emil Renner Berthing
2026-06-21 21:22 ` Indrek Kruusa
2026-06-22 15:38 ` Boqun Feng [this message]
2026-06-22 15:38 ` Boqun Feng
2026-06-23 9:34 ` Emil Renner Berthing
2026-06-23 9:34 ` Emil Renner Berthing
2026-06-23 15:53 ` Samuel Holland
2026-06-23 15:53 ` Samuel Holland
2026-06-24 8:55 ` Emil Renner Berthing
2026-06-24 8:55 ` Emil Renner Berthing
2026-07-06 11:32 ` Thorsten Leemhuis
2026-07-06 11:32 ` Thorsten Leemhuis
2026-07-08 6:58 ` Indrek Kruusa
2026-07-08 6:58 ` Indrek Kruusa
2026-03-21 4:27 ` [PATCH] " Zqiang
2026-03-21 18:15 ` Boqun Feng
2026-03-21 10:10 ` Paul E. McKenney
2026-03-21 17:15 ` Boqun Feng
2026-03-21 17:41 ` Paul E. McKenney
2026-03-21 18:06 ` Boqun Feng
2026-03-21 19:31 ` Paul E. McKenney
2026-03-21 19:45 ` Boqun Feng
2026-03-21 20:07 ` Paul E. McKenney
2026-03-21 20:08 ` Boqun Feng
2026-03-22 10:09 ` Paul E. McKenney
2026-03-22 16:16 ` Boqun Feng
2026-03-22 17:09 ` Paul E. McKenney
2026-03-22 17:31 ` Boqun Feng
2026-03-22 17:44 ` Paul E. McKenney
2026-03-22 18:17 ` Boqun Feng
2026-03-22 19:47 ` Paul E. McKenney
2026-03-22 20:26 ` Boqun Feng
2026-03-23 7:50 ` Paul E. McKenney
2026-03-20 18:20 ` Next-level bug in SRCU implementation of RCU Tasks Trace + PREEMPT_RT Boqun Feng
2026-03-20 23:11 ` Paul E. McKenney
2026-03-21 3:29 ` Paul E. McKenney
2026-03-21 17:03 ` [RFC PATCH] rcu-tasks: Avoid using mod_timer() in call_rcu_tasks_generic() Boqun Feng
2026-03-23 15:17 ` Boqun Feng
2026-03-23 20:37 ` Joel Fernandes
2026-03-23 21:50 ` Kumar Kartikeya Dwivedi
2026-03-23 22:13 ` Boqun Feng
2026-03-20 16:15 ` Next-level bug in SRCU implementation of RCU Tasks Trace + PREEMPT_RT Boqun Feng
2026-03-20 16:24 ` Paul E. McKenney
2026-03-19 17:02 ` Sebastian Andrzej Siewior
2026-03-19 17:44 ` Boqun Feng
2026-03-19 18:42 ` Joel Fernandes
2026-03-19 20:20 ` Boqun Feng
2026-03-19 20:26 ` Joel Fernandes
2026-03-19 20:45 ` Joel Fernandes
2026-03-19 10:02 ` Paul E. McKenney
2026-03-19 14:34 ` Boqun Feng
2026-03-19 16:10 ` Paul E. McKenney
2026-03-18 23:56 ` Kumar Kartikeya Dwivedi
2026-03-19 0:26 ` Zqiang
2026-03-19 1:13 ` Boqun Feng
2026-03-19 2:47 ` Joel Fernandes
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ajlW4oSzEqav6gxc@MacBook-0RXW5 \
--to=boqun@kernel.org \
--cc=arighi@nvidia.com \
--cc=ast@kernel.org \
--cc=bigeasy@linutronix.de \
--cc=boqun.feng@gmail.com \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=emil.renner.berthing@canonical.com \
--cc=frederic@kernel.org \
--cc=joelagnelf@nvidia.com \
--cc=john.fastabend@gmail.com \
--cc=linux-riscv@lists.infradead.org \
--cc=memxor@gmail.com \
--cc=neeraj.iitr10@gmail.com \
--cc=paulmck@kernel.org \
--cc=qiang.zhang@linux.dev \
--cc=rcu@vger.kernel.org \
--cc=samuel.holland@sifive.com \
--cc=sarah.emery@canonical.com \
--cc=tj@kernel.org \
--cc=urezki@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.