[BUG] NOHZ tick-stop error: local softirq work is pending

[BUG] NOHZ tick-stop error: local softirq work is pending

[Michal Pecio]

Hi,

I'm getting these warnings on current mainline. Quite reliable trigger
is I/O load (like md5sum /dev/sda) on an otherwise idle system:

[   86.082138] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  117.959975] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  121.054544] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  121.988208] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  143.932445] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  156.083120] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  156.685222] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  156.789911] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
[  158.986070] NOHZ tick-stop error: local softirq work is pending, handler #40!!!

I see it consistently across several reboots. I also did two v7.2
boots with the same workload and these ran cleanly 10 minutes each,
so it looks like a merge window regression.

The system is ordinary x86-64 with CONFIG_PREEMPT.

grep HZ .config: 

CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ_FULL is not set
# CONFIG_NO_HZ is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000

It seems bisectable, but maybe somebody has better ideas?

Regards,
Michal

Re: [BUG] NOHZ tick-stop error: local softirq work is pending

[Thomas Gleixner]

On Sat, Aug 29 2026 at 20:14, Michal Pecio wrote:
> I'm getting these warnings on current mainline. Quite reliable trigger
> is I/O load (like md5sum /dev/sda) on an otherwise idle system:
>
> [   86.082138] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  117.959975] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  121.054544] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  121.988208] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  143.932445] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  156.083120] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  156.685222] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  156.789911] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> [  158.986070] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
>
> I see it consistently across several reboots. I also did two v7.2
> boots with the same workload and these ran cleanly 10 minutes each,
> so it looks like a merge window regression.
>
>
> It seems bisectable, but maybe somebody has better ideas?

It's likely the issue being discussed here:

   https://lore.kernel.org/lkml/20260824105523.GA4121620@noisy.programming.kicks-ass.net/

There is a revert queued which probably will cure it for you unless you
are using Rust drivers. Plus a fix for the remaining Rust specific issue
is on the way.

Thanks,

        tglx

Re: [BUG] NOHZ tick-stop error: local softirq work is pending

[Michal Pecio]

On Sun, 30 Aug 2026 07:56:49 +0200, Thomas Gleixner wrote:
> On Sat, Aug 29 2026 at 20:14, Michal Pecio wrote:
> > I'm getting these warnings on current mainline. Quite reliable trigger
> > is I/O load (like md5sum /dev/sda) on an otherwise idle system:
> >
> > [   86.082138] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  117.959975] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  121.054544] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  121.988208] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  143.932445] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  156.083120] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  156.685222] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  156.789911] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> > [  158.986070] NOHZ tick-stop error: local softirq work is pending, handler #40!!!
> >
> > I see it consistently across several reboots. I also did two v7.2
> > boots with the same workload and these ran cleanly 10 minutes each,
> > so it looks like a merge window regression.
> >
> >
> > It seems bisectable, but maybe somebody has better ideas?  
> 
> It's likely the issue being discussed here:
> 
>    https://lore.kernel.org/lkml/20260824105523.GA4121620@noisy.programming.kicks-ass.net/
> 
> There is a revert queued which probably will cure it for you unless
> you are using Rust drivers. Plus a fix for the remaining Rust
> specific issue is on the way.

Thank you, cherry picking 46094a7708b7 from tip solved this.

And no, I absolutely loathe this cargo cult and it saddens me to see
people's time and CPU cycles being wasted on converting the kernel to
idiomatic C++ with its arbitrary limitations.

Surely all those irq_disable counters in Linux data centers around the
world will have a carbon footprint, even if (hypothetically) they are
"free" on superscalar CPUs with branch prediction?

And all of that for the sake of potential bugs which may or may not
exist, and which look like they could be found (or proven absence of)
by static analysis if required, and because of an experiment 15 years
which showed that static analysis beats reference counting?

Could only be more ironic if Moore's law were approaching a wall and
energy costs were rising everywhere with no end in sight...

Have a nice Sunday everyone,

Michal

[BUG] NOHZ tick-stop error: local softirq work is pending

[Joel Fernandes]

Hi Paul, Frederic, ever since this recent patch got merged into all
recent stables:
https://lore.kernel.org/all/20230818200757.1808398-1-paul.gortmaker@windriver.com/
 ..

.. I am seeing this happen a lot now causing rcutorture failures:
"NOHZ tick-stop error: local softirq work is pending"

It looks like the "tick-stop error" as local_bh_blocked always returns
false on !PREEMPT_RT, so that patch itself seems correct.

However, it appears the string in console-badness.sh is different:

console-badness.sh:grep -v 'NOHZ tick-stop error: Non-RCU local
softirq work is pending, handler'

So the string in the script needs an update?

Also, considering that it is being grep -v'ed out, any background on
why this message is not bad and any history? Why is softirq work
pending when the tick is stopped?

thanks,

 - Joel

Re: [BUG] NOHZ tick-stop error: local softirq work is pending

[Paul E. McKenney]

On Sat, Sep 09, 2023 at 10:28:36PM -0400, Joel Fernandes wrote:
> Hi Paul, Frederic, ever since this recent patch got merged into all
> recent stables:
> https://lore.kernel.org/all/20230818200757.1808398-1-paul.gortmaker@windriver.com/
>  ..
> 
> .. I am seeing this happen a lot now causing rcutorture failures:
> "NOHZ tick-stop error: local softirq work is pending"
> 
> It looks like the "tick-stop error" as local_bh_blocked always returns
> false on !PREEMPT_RT, so that patch itself seems correct.
> 
> However, it appears the string in console-badness.sh is different:
> 
> console-badness.sh:grep -v 'NOHZ tick-stop error: Non-RCU local
> softirq work is pending, handler'
> 
> So the string in the script needs an update?

Looks like it, though I believe that there is a fix in the offing,
at which point that "grep -v" should just be removed.  Depending on
timing, of course.

> Also, considering that it is being grep -v'ed out, any background on
> why this message is not bad and any history? Why is softirq work
> pending when the tick is stopped?

That was so long back that it is pretty hazy.  I vaguely recall that
this was considered to be in need of fixing, but a false positive from
a torture-test perspective.

							Thanx, Paul

Re: [BUG] NOHZ tick-stop error: local softirq work is pending

[Frederic Weisbecker]

Le Sun, Sep 10, 2023 at 03:21:33AM -0700, Paul E. McKenney a écrit :
> On Sat, Sep 09, 2023 at 10:28:36PM -0400, Joel Fernandes wrote:
> > Hi Paul, Frederic, ever since this recent patch got merged into all
> > recent stables:
> > https://lore.kernel.org/all/20230818200757.1808398-1-paul.gortmaker@windriver.com/
> >  ..
> > 
> > .. I am seeing this happen a lot now causing rcutorture failures:
> > "NOHZ tick-stop error: local softirq work is pending"
> > 
> > It looks like the "tick-stop error" as local_bh_blocked always returns
> > false on !PREEMPT_RT, so that patch itself seems correct.
> > 
> > However, it appears the string in console-badness.sh is different:
> > 
> > console-badness.sh:grep -v 'NOHZ tick-stop error: Non-RCU local
> > softirq work is pending, handler'
> > 
> > So the string in the script needs an update?
> 
> Looks like it, though I believe that there is a fix in the offing,
> at which point that "grep -v" should just be removed.  Depending on
> timing, of course.

Yep I have a pending fix for that, will post soonish. The grep will still
make sense afterward though.

Thanks.

Re: [BUG] NOHZ tick-stop error: local softirq work is pending

[Paul E. McKenney]

On Sun, Sep 10, 2023 at 10:25:43PM +0200, Frederic Weisbecker wrote:
> Le Sun, Sep 10, 2023 at 03:21:33AM -0700, Paul E. McKenney a écrit :
> > On Sat, Sep 09, 2023 at 10:28:36PM -0400, Joel Fernandes wrote:
> > > Hi Paul, Frederic, ever since this recent patch got merged into all
> > > recent stables:
> > > https://lore.kernel.org/all/20230818200757.1808398-1-paul.gortmaker@windriver.com/
> > >  ..
> > > 
> > > .. I am seeing this happen a lot now causing rcutorture failures:
> > > "NOHZ tick-stop error: local softirq work is pending"
> > > 
> > > It looks like the "tick-stop error" as local_bh_blocked always returns
> > > false on !PREEMPT_RT, so that patch itself seems correct.
> > > 
> > > However, it appears the string in console-badness.sh is different:
> > > 
> > > console-badness.sh:grep -v 'NOHZ tick-stop error: Non-RCU local
> > > softirq work is pending, handler'
> > > 
> > > So the string in the script needs an update?
> > 
> > Looks like it, though I believe that there is a fix in the offing,
> > at which point that "grep -v" should just be removed.  Depending on
> > timing, of course.
> 
> Yep I have a pending fix for that, will post soonish. The grep will still
> make sense afterward though.

So I do nothing?  I can do that!  ;-)

							Thanx, Paul

Re: [BUG] NOHZ tick-stop error: local softirq work is pending

[Joel Fernandes]

On Sun, Sep 10, 2023 at 10:25:43PM +0200, Frederic Weisbecker wrote:
> Le Sun, Sep 10, 2023 at 03:21:33AM -0700, Paul E. McKenney a écrit :
> > On Sat, Sep 09, 2023 at 10:28:36PM -0400, Joel Fernandes wrote:
> > > Hi Paul, Frederic, ever since this recent patch got merged into all
> > > recent stables:
> > > https://lore.kernel.org/all/20230818200757.1808398-1-paul.gortmaker@windriver.com/
> > >  ..
> > > 
> > > .. I am seeing this happen a lot now causing rcutorture failures:
> > > "NOHZ tick-stop error: local softirq work is pending"
> > > 
> > > It looks like the "tick-stop error" as local_bh_blocked always returns
> > > false on !PREEMPT_RT, so that patch itself seems correct.
> > > 
> > > However, it appears the string in console-badness.sh is different:
> > > 
> > > console-badness.sh:grep -v 'NOHZ tick-stop error: Non-RCU local
> > > softirq work is pending, handler'
> > > 
> > > So the string in the script needs an update?
> > 
> > Looks like it, though I believe that there is a fix in the offing,
> > at which point that "grep -v" should just be removed.  Depending on
> > timing, of course.
> 
> Yep I have a pending fix for that, will post soonish. The grep will still
> make sense afterward though.

I was wondering why the grep -v will still make sense if the pending fix
works. Or is there another legitimate reason for the string to appear?

Either way, looking forward to the patch!

thanks,

 - Joel

Re: [BUG] NOHZ tick-stop error: local softirq work is pending

[Frederic Weisbecker]

Le Sun, Sep 10, 2023 at 11:02:09PM +0000, Joel Fernandes a écrit :
> On Sun, Sep 10, 2023 at 10:25:43PM +0200, Frederic Weisbecker wrote:
> > Le Sun, Sep 10, 2023 at 03:21:33AM -0700, Paul E. McKenney a écrit :
> > > On Sat, Sep 09, 2023 at 10:28:36PM -0400, Joel Fernandes wrote:
> > > > Hi Paul, Frederic, ever since this recent patch got merged into all
> > > > recent stables:
> > > > https://lore.kernel.org/all/20230818200757.1808398-1-paul.gortmaker@windriver.com/
> > > >  ..
> > > > 
> > > > .. I am seeing this happen a lot now causing rcutorture failures:
> > > > "NOHZ tick-stop error: local softirq work is pending"
> > > > 
> > > > It looks like the "tick-stop error" as local_bh_blocked always returns
> > > > false on !PREEMPT_RT, so that patch itself seems correct.
> > > > 
> > > > However, it appears the string in console-badness.sh is different:
> > > > 
> > > > console-badness.sh:grep -v 'NOHZ tick-stop error: Non-RCU local
> > > > softirq work is pending, handler'
> > > > 
> > > > So the string in the script needs an update?
> > > 
> > > Looks like it, though I believe that there is a fix in the offing,
> > > at which point that "grep -v" should just be removed.  Depending on
> > > timing, of course.
> > 
> > Yep I have a pending fix for that, will post soonish. The grep will still
> > make sense afterward though.
> 
> I was wondering why the grep -v will still make sense if the pending fix
> works. Or is there another legitimate reason for the string to appear?

Because my fix will only take out (hr)timer softirqs out of the report. The
others will still be considered buggy if pending while the CPU goes down.
Anyway, cooking the timer patchset to post.

Thanks.