* [BUG] NOHZ tick-stop error: local softirq work is pending
@ 2023-09-10 2:28 Joel Fernandes
2023-09-10 10:21 ` Paul E. McKenney
0 siblings, 1 reply; 6+ messages in thread
From: Joel Fernandes @ 2023-09-10 2:28 UTC (permalink / raw)
To: Frederic Weisbecker, Paul E. McKenney; +Cc: rcu
Hi Paul, Frederic, ever since this recent patch got merged into all
recent stables:
https://lore.kernel.org/all/20230818200757.1808398-1-paul.gortmaker@windriver.com/
..
.. I am seeing this happen a lot now causing rcutorture failures:
"NOHZ tick-stop error: local softirq work is pending"
It looks like the "tick-stop error" as local_bh_blocked always returns
false on !PREEMPT_RT, so that patch itself seems correct.
However, it appears the string in console-badness.sh is different:
console-badness.sh:grep -v 'NOHZ tick-stop error: Non-RCU local
softirq work is pending, handler'
So the string in the script needs an update?
Also, considering that it is being grep -v'ed out, any background on
why this message is not bad and any history? Why is softirq work
pending when the tick is stopped?
thanks,
- Joel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [BUG] NOHZ tick-stop error: local softirq work is pending
2023-09-10 2:28 [BUG] NOHZ tick-stop error: local softirq work is pending Joel Fernandes
@ 2023-09-10 10:21 ` Paul E. McKenney
2023-09-10 20:25 ` Frederic Weisbecker
0 siblings, 1 reply; 6+ messages in thread
From: Paul E. McKenney @ 2023-09-10 10:21 UTC (permalink / raw)
To: Joel Fernandes; +Cc: Frederic Weisbecker, rcu
On Sat, Sep 09, 2023 at 10:28:36PM -0400, Joel Fernandes wrote:
> Hi Paul, Frederic, ever since this recent patch got merged into all
> recent stables:
> https://lore.kernel.org/all/20230818200757.1808398-1-paul.gortmaker@windriver.com/
> ..
>
> .. I am seeing this happen a lot now causing rcutorture failures:
> "NOHZ tick-stop error: local softirq work is pending"
>
> It looks like the "tick-stop error" as local_bh_blocked always returns
> false on !PREEMPT_RT, so that patch itself seems correct.
>
> However, it appears the string in console-badness.sh is different:
>
> console-badness.sh:grep -v 'NOHZ tick-stop error: Non-RCU local
> softirq work is pending, handler'
>
> So the string in the script needs an update?
Looks like it, though I believe that there is a fix in the offing,
at which point that "grep -v" should just be removed. Depending on
timing, of course.
> Also, considering that it is being grep -v'ed out, any background on
> why this message is not bad and any history? Why is softirq work
> pending when the tick is stopped?
That was so long back that it is pretty hazy. I vaguely recall that
this was considered to be in need of fixing, but a false positive from
a torture-test perspective.
Thanx, Paul
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [BUG] NOHZ tick-stop error: local softirq work is pending
2023-09-10 10:21 ` Paul E. McKenney
@ 2023-09-10 20:25 ` Frederic Weisbecker
2023-09-10 21:48 ` Paul E. McKenney
2023-09-10 23:02 ` Joel Fernandes
0 siblings, 2 replies; 6+ messages in thread
From: Frederic Weisbecker @ 2023-09-10 20:25 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: Joel Fernandes, rcu
Le Sun, Sep 10, 2023 at 03:21:33AM -0700, Paul E. McKenney a écrit :
> On Sat, Sep 09, 2023 at 10:28:36PM -0400, Joel Fernandes wrote:
> > Hi Paul, Frederic, ever since this recent patch got merged into all
> > recent stables:
> > https://lore.kernel.org/all/20230818200757.1808398-1-paul.gortmaker@windriver.com/
> > ..
> >
> > .. I am seeing this happen a lot now causing rcutorture failures:
> > "NOHZ tick-stop error: local softirq work is pending"
> >
> > It looks like the "tick-stop error" as local_bh_blocked always returns
> > false on !PREEMPT_RT, so that patch itself seems correct.
> >
> > However, it appears the string in console-badness.sh is different:
> >
> > console-badness.sh:grep -v 'NOHZ tick-stop error: Non-RCU local
> > softirq work is pending, handler'
> >
> > So the string in the script needs an update?
>
> Looks like it, though I believe that there is a fix in the offing,
> at which point that "grep -v" should just be removed. Depending on
> timing, of course.
Yep I have a pending fix for that, will post soonish. The grep will still
make sense afterward though.
Thanks.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [BUG] NOHZ tick-stop error: local softirq work is pending
2023-09-10 20:25 ` Frederic Weisbecker
@ 2023-09-10 21:48 ` Paul E. McKenney
2023-09-10 23:02 ` Joel Fernandes
1 sibling, 0 replies; 6+ messages in thread
From: Paul E. McKenney @ 2023-09-10 21:48 UTC (permalink / raw)
To: Frederic Weisbecker; +Cc: Joel Fernandes, rcu
On Sun, Sep 10, 2023 at 10:25:43PM +0200, Frederic Weisbecker wrote:
> Le Sun, Sep 10, 2023 at 03:21:33AM -0700, Paul E. McKenney a écrit :
> > On Sat, Sep 09, 2023 at 10:28:36PM -0400, Joel Fernandes wrote:
> > > Hi Paul, Frederic, ever since this recent patch got merged into all
> > > recent stables:
> > > https://lore.kernel.org/all/20230818200757.1808398-1-paul.gortmaker@windriver.com/
> > > ..
> > >
> > > .. I am seeing this happen a lot now causing rcutorture failures:
> > > "NOHZ tick-stop error: local softirq work is pending"
> > >
> > > It looks like the "tick-stop error" as local_bh_blocked always returns
> > > false on !PREEMPT_RT, so that patch itself seems correct.
> > >
> > > However, it appears the string in console-badness.sh is different:
> > >
> > > console-badness.sh:grep -v 'NOHZ tick-stop error: Non-RCU local
> > > softirq work is pending, handler'
> > >
> > > So the string in the script needs an update?
> >
> > Looks like it, though I believe that there is a fix in the offing,
> > at which point that "grep -v" should just be removed. Depending on
> > timing, of course.
>
> Yep I have a pending fix for that, will post soonish. The grep will still
> make sense afterward though.
So I do nothing? I can do that! ;-)
Thanx, Paul
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [BUG] NOHZ tick-stop error: local softirq work is pending
2023-09-10 20:25 ` Frederic Weisbecker
2023-09-10 21:48 ` Paul E. McKenney
@ 2023-09-10 23:02 ` Joel Fernandes
2023-09-11 21:23 ` Frederic Weisbecker
1 sibling, 1 reply; 6+ messages in thread
From: Joel Fernandes @ 2023-09-10 23:02 UTC (permalink / raw)
To: Frederic Weisbecker; +Cc: Paul E. McKenney, rcu
On Sun, Sep 10, 2023 at 10:25:43PM +0200, Frederic Weisbecker wrote:
> Le Sun, Sep 10, 2023 at 03:21:33AM -0700, Paul E. McKenney a écrit :
> > On Sat, Sep 09, 2023 at 10:28:36PM -0400, Joel Fernandes wrote:
> > > Hi Paul, Frederic, ever since this recent patch got merged into all
> > > recent stables:
> > > https://lore.kernel.org/all/20230818200757.1808398-1-paul.gortmaker@windriver.com/
> > > ..
> > >
> > > .. I am seeing this happen a lot now causing rcutorture failures:
> > > "NOHZ tick-stop error: local softirq work is pending"
> > >
> > > It looks like the "tick-stop error" as local_bh_blocked always returns
> > > false on !PREEMPT_RT, so that patch itself seems correct.
> > >
> > > However, it appears the string in console-badness.sh is different:
> > >
> > > console-badness.sh:grep -v 'NOHZ tick-stop error: Non-RCU local
> > > softirq work is pending, handler'
> > >
> > > So the string in the script needs an update?
> >
> > Looks like it, though I believe that there is a fix in the offing,
> > at which point that "grep -v" should just be removed. Depending on
> > timing, of course.
>
> Yep I have a pending fix for that, will post soonish. The grep will still
> make sense afterward though.
I was wondering why the grep -v will still make sense if the pending fix
works. Or is there another legitimate reason for the string to appear?
Either way, looking forward to the patch!
thanks,
- Joel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [BUG] NOHZ tick-stop error: local softirq work is pending
2023-09-10 23:02 ` Joel Fernandes
@ 2023-09-11 21:23 ` Frederic Weisbecker
0 siblings, 0 replies; 6+ messages in thread
From: Frederic Weisbecker @ 2023-09-11 21:23 UTC (permalink / raw)
To: Joel Fernandes; +Cc: Paul E. McKenney, rcu
Le Sun, Sep 10, 2023 at 11:02:09PM +0000, Joel Fernandes a écrit :
> On Sun, Sep 10, 2023 at 10:25:43PM +0200, Frederic Weisbecker wrote:
> > Le Sun, Sep 10, 2023 at 03:21:33AM -0700, Paul E. McKenney a écrit :
> > > On Sat, Sep 09, 2023 at 10:28:36PM -0400, Joel Fernandes wrote:
> > > > Hi Paul, Frederic, ever since this recent patch got merged into all
> > > > recent stables:
> > > > https://lore.kernel.org/all/20230818200757.1808398-1-paul.gortmaker@windriver.com/
> > > > ..
> > > >
> > > > .. I am seeing this happen a lot now causing rcutorture failures:
> > > > "NOHZ tick-stop error: local softirq work is pending"
> > > >
> > > > It looks like the "tick-stop error" as local_bh_blocked always returns
> > > > false on !PREEMPT_RT, so that patch itself seems correct.
> > > >
> > > > However, it appears the string in console-badness.sh is different:
> > > >
> > > > console-badness.sh:grep -v 'NOHZ tick-stop error: Non-RCU local
> > > > softirq work is pending, handler'
> > > >
> > > > So the string in the script needs an update?
> > >
> > > Looks like it, though I believe that there is a fix in the offing,
> > > at which point that "grep -v" should just be removed. Depending on
> > > timing, of course.
> >
> > Yep I have a pending fix for that, will post soonish. The grep will still
> > make sense afterward though.
>
> I was wondering why the grep -v will still make sense if the pending fix
> works. Or is there another legitimate reason for the string to appear?
Because my fix will only take out (hr)timer softirqs out of the report. The
others will still be considered buggy if pending while the CPU goes down.
Anyway, cooking the timer patchset to post.
Thanks.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-09-12 4:17 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-09-10 2:28 [BUG] NOHZ tick-stop error: local softirq work is pending Joel Fernandes
2023-09-10 10:21 ` Paul E. McKenney
2023-09-10 20:25 ` Frederic Weisbecker
2023-09-10 21:48 ` Paul E. McKenney
2023-09-10 23:02 ` Joel Fernandes
2023-09-11 21:23 ` Frederic Weisbecker
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.