public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
From: Boqun Feng <boqun.feng@gmail.com>
To: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Joel Fernandes <joelagnelf@nvidia.com>,
	linux-kernel@vger.kernel.org,
	Frederic Weisbecker <frederic@kernel.org>,
	Neeraj Upadhyay <neeraj.upadhyay@kernel.org>,
	Joel Fernandes <joel@joelfernandes.org>,
	Josh Triplett <josh@joshtriplett.org>,
	Uladzislau Rezki <urezki@gmail.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
	Lai Jiangshan <jiangshanlai@gmail.com>,
	Zqiang <qiang.zhang1211@gmail.com>,
	Xiongfeng Wang <wangxiongfeng2@huawei.com>,
	rcu@vger.kernel.org, bpf@vger.kernel.org
Subject: Re: [PATCH 2/2] rcu: Fix lockup when RCU reader used while IRQ exiting
Date: Wed, 11 Jun 2025 09:21:42 -0700	[thread overview]
Message-ID: <aEmtFr3sHCgLpWoT@tardis.local> (raw)
In-Reply-To: <2a0902a7-852b-4868-b0c5-2a6962f273ed@paulmck-laptop>

On Wed, Jun 11, 2025 at 09:16:05AM -0700, Paul E. McKenney wrote:
> On Wed, Jun 11, 2025 at 09:05:06AM -0700, Boqun Feng wrote:
> > On Mon, Jun 09, 2025 at 02:01:24PM -0400, Joel Fernandes wrote:
> > > During rcu_read_unlock_special(), if this happens during irq_exit(), we
> > > can lockup if an IPI is issued. This is because the IPI itself triggers
> > > the irq_exit() path causing a recursive lock up.
> > > 
> > > This is precisely what Xiongfeng found when invoking a BPF program on
> > > the trace_tick_stop() tracepoint As shown in the trace below. Fix by
> > > using context-tracking to tell us if we're still in an IRQ.
> > > context-tracking keeps track of the IRQ until after the tracepoint, so
> > > it cures the issues.
> > > 
> > > irq_exit()
> > >   __irq_exit_rcu()
> > >     /* in_hardirq() returns false after this */
> > >     preempt_count_sub(HARDIRQ_OFFSET)
> > >     tick_irq_exit()
> > 
> > @Frederic, while we are at it, what's the purpose of in_hardirq() in
> > tick_irq_exit()? For nested interrupt detection?
> 
> If you are talking about the comment, these sorts of comments help
> people reading the code, the point being that some common-code function
> that invokes in_hardirq() after that point will get the wrong answer
> from it.  The context-tracking code does the same for whether or not

The thing is that tick_irq_exit() is supposed to be only called in
irq_exit() IIUC (given its name), and so without nested interrupts,
in_hardirq() will also give the wrong answer.

Regards,
Boqun

> RCU is watching.
> 
> 							Thanx, Paul
> 
> > Regards,
> > Boqun
> > 
> > >       tick_nohz_irq_exit()
> > > 	    tick_nohz_stop_sched_tick()
> > > 	      trace_tick_stop()  /* a bpf prog is hooked on this trace point */
> > > 		   __bpf_trace_tick_stop()
> > > 		      bpf_trace_run2()
> > > 			    rcu_read_unlock_special()
> > >                               /* will send a IPI to itself */
> > > 			      irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu);
> > > 
> > > A simple reproducer can also be obtained by doing the following in
> > > tick_irq_exit(). It will hang on boot without the patch:
> > > 
> > >   static inline void tick_irq_exit(void)
> > >   {
> > >  +	rcu_read_lock();
> > >  +	WRITE_ONCE(current->rcu_read_unlock_special.b.need_qs, true);
> > >  +	rcu_read_unlock();
> > >  +
> > > 
> > > While at it, add some comments to this code.
> > > 
> > > Reported-by: Xiongfeng Wang <wangxiongfeng2@huawei.com>
> > > Closes: https://lore.kernel.org/all/9acd5f9f-6732-7701-6880-4b51190aa070@huawei.com/
> > > Tested-by: Xiongfeng Wang <wangxiongfeng2@huawei.com>
> > > Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
> > [...]
> 

  reply	other threads:[~2025-06-11 16:21 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-06-09 18:01 [PATCH 1/2] context_tracking: Provide helper to determine if we're in IRQ Joel Fernandes
2025-06-09 18:01 ` [PATCH 2/2] rcu: Fix lockup when RCU reader used while IRQ exiting Joel Fernandes
2025-06-09 19:49   ` Boqun Feng
2025-06-09 23:26     ` Frederic Weisbecker
2025-06-10  0:49       ` Boqun Feng
2025-06-10 12:23   ` Frederic Weisbecker
2025-06-10 15:47     ` Joel Fernandes
2025-06-12  3:06     ` Xiongfeng Wang
2025-06-12 11:37       ` Frederic Weisbecker
2025-06-11 16:05   ` Boqun Feng
2025-06-11 16:16     ` Paul E. McKenney
2025-06-11 16:21       ` Boqun Feng [this message]
2025-06-09 18:05 ` [PATCH 1/2] context_tracking: Provide helper to determine if we're in IRQ Joel Fernandes
2025-06-11 16:25 ` Frederic Weisbecker

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=aEmtFr3sHCgLpWoT@tardis.local \
    --to=boqun.feng@gmail.com \
    --cc=bpf@vger.kernel.org \
    --cc=frederic@kernel.org \
    --cc=jiangshanlai@gmail.com \
    --cc=joel@joelfernandes.org \
    --cc=joelagnelf@nvidia.com \
    --cc=josh@joshtriplett.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mathieu.desnoyers@efficios.com \
    --cc=neeraj.upadhyay@kernel.org \
    --cc=paulmck@kernel.org \
    --cc=qiang.zhang1211@gmail.com \
    --cc=rcu@vger.kernel.org \
    --cc=rostedt@goodmis.org \
    --cc=urezki@gmail.com \
    --cc=wangxiongfeng2@huawei.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox