Backport request: Revert "softirq: Let ksoftirqd do its job" (d15121be7485)

Backport request: Revert "softirq: Let ksoftirqd do its job" (d15121be7485)

[Gyutae Bae]

Hi,

Please backport the following upstream commit to 6.1.y and 5.15.y:

  d15121be7485655129101f3960ae6add40204463
  Revert "softirq: Let ksoftirqd do its job"

On a production 5.15 PREEMPT_NONE node, long-running overlayfs
getdents64() calls prevented ksoftirqd from running.  RPS IPIs
continued to arrive, but NET_RX processing on that CPU stalled for
up to about 3 seconds.  Over 15 days, the node logged about 1,000
bonding ARP monitor failovers, with no link-state changes recorded
during that period.  The failovers stopped after the workload was
removed.

On a 5.15.27 PREEMPT_NONE test node, we used a livepatch to make
ksoftirqd_running() always return false.  This removes the same
softirq deferral checks as the revert on this build.  Under the same
load, we measured three 10-minute windows: before patching, with the
livepatch active, and after removing it.  The number of HTTP requests
taking more than 200 ms was 17, 0 and 11, respectively.  TCPTimeouts
increased by 25, 1 and 21.

We verified with patch --dry-run that the revert applies cleanly
to v6.1.190 and v5.15.223.  Both already contain the related RCU
fix 1dd1eff161bd.  6.6.y and later already contain the revert.

The original commit was intended to prevent user-task starvation
under a softirq flood.  The revert's changelog points to threaded
NAPI as an alternative, but it is disabled by default.

The revert was merged in v6.5 without a Cc: stable tag.
Syzbot has also reported bisections to this revert [1].
Paolo, Thomas, Eric: I would appreciate your thoughts on whether
this revert is suitable for 6.1.y and 5.15.y.

[1] https://lore.kernel.org/all/000000000000a249880603717c15@google.com/

Thanks,
Gyutae Bae

Re: Backport request: Revert "softirq: Let ksoftirqd do its job" (d15121be7485)

[Paolo Abeni]

On 10/29/26 06:42, Gyutae Bae wrote:
> Please backport the following upstream commit to 6.1.y and 5.15.y:
> 
>    d15121be7485655129101f3960ae6add40204463
>    Revert "softirq: Let ksoftirqd do its job"
> 
> On a production 5.15 PREEMPT_NONE node, long-running overlayfs
> getdents64() calls prevented ksoftirqd from running.  RPS IPIs
> continued to arrive, but NET_RX processing on that CPU stalled for
> up to about 3 seconds.  Over 15 days, the node logged about 1,000
> bonding ARP monitor failovers, with no link-state changes recorded
> during that period.  The failovers stopped after the workload was
> removed.
> 
> On a 5.15.27 PREEMPT_NONE test node, we used a livepatch to make
> ksoftirqd_running() always return false.  This removes the same
> softirq deferral checks as the revert on this build.  Under the same
> load, we measured three 10-minute windows: before patching, with the
> livepatch active, and after removing it.  The number of HTTP requests
> taking more than 200 ms was 17, 0 and 11, respectively.  TCPTimeouts
> increased by 25, 1 and 21.
> 
> We verified with patch --dry-run that the revert applies cleanly
> to v6.1.190 and v5.15.223.  Both already contain the related RCU
> fix 1dd1eff161bd.  6.6.y and later already contain the revert.
> 
> The original commit was intended to prevent user-task starvation
> under a softirq flood.  The revert's changelog points to threaded
> NAPI as an alternative, but it is disabled by default.
> 
> The revert was merged in v6.5 without a Cc: stable tag.
> Syzbot has also reported bisections to this revert [1].
> Paolo, Thomas, Eric: I would appreciate your thoughts on whether
> this revert is suitable for 6.1.y and 5.15.y.
> 
> [1] https://lore.kernel.org/all/000000000000a249880603717c15@google.com/
IMHO such change should not land to stable lightly. It's a behavioural
change with subtle side effects. The threaded napi feature is disabled
by default for the same reason. Also the revert has some soft functional
dependencies in napi threaded cleanup and improvements landed into v6.4,
i.e. 6bb218b5a6e3f3c5e2e76faf8fc976f3f6f72496, likely more.

/P

Re: Backport request: Revert "softirq: Let ksoftirqd do its job" (d15121be7485)

[Gyutae Bae]

On 10/29/26 09:33, Paolo Abeni wrote:
> IMHO such change should not land to stable lightly. It's a behavioural
> change with subtle side effects. The threaded napi feature is disabled
> by default for the same reason. Also the revert has some soft functional
> dependencies in napi threaded cleanup and improvements landed into v6.4,
> i.e. 6bb218b5a6e3f3c5e2e76faf8fc976f3f6f72496, likely more.

Thanks for taking a look. I understand the concern about changing
behavior in stable kernels.

Gyutae Bae

Re: Backport request: Revert "softirq: Let ksoftirqd do its job" (d15121be7485)

[Jason Xing]

Hi Paolo,

On Thu, Oct 29, 2026 at 4:33 PM Paolo Abeni <pabeni@redhat.com> wrote:
>
> On 10/29/26 06:42, Gyutae Bae wrote:
> > Please backport the following upstream commit to 6.1.y and 5.15.y:
> >
> >    d15121be7485655129101f3960ae6add40204463
> >    Revert "softirq: Let ksoftirqd do its job"
> >
> > On a production 5.15 PREEMPT_NONE node, long-running overlayfs
> > getdents64() calls prevented ksoftirqd from running.  RPS IPIs
> > continued to arrive, but NET_RX processing on that CPU stalled for
> > up to about 3 seconds.  Over 15 days, the node logged about 1,000
> > bonding ARP monitor failovers, with no link-state changes recorded
> > during that period.  The failovers stopped after the workload was
> > removed.
> >
> > On a 5.15.27 PREEMPT_NONE test node, we used a livepatch to make
> > ksoftirqd_running() always return false.  This removes the same
> > softirq deferral checks as the revert on this build.  Under the same
> > load, we measured three 10-minute windows: before patching, with the
> > livepatch active, and after removing it.  The number of HTTP requests
> > taking more than 200 ms was 17, 0 and 11, respectively.  TCPTimeouts
> > increased by 25, 1 and 21.
> >
> > We verified with patch --dry-run that the revert applies cleanly
> > to v6.1.190 and v5.15.223.  Both already contain the related RCU
> > fix 1dd1eff161bd.  6.6.y and later already contain the revert.
> >
> > The original commit was intended to prevent user-task starvation
> > under a softirq flood.  The revert's changelog points to threaded
> > NAPI as an alternative, but it is disabled by default.
> >
> > The revert was merged in v6.5 without a Cc: stable tag.
> > Syzbot has also reported bisections to this revert [1].
> > Paolo, Thomas, Eric: I would appreciate your thoughts on whether
> > this revert is suitable for 6.1.y and 5.15.y.
> >
> > [1] https://lore.kernel.org/all/000000000000a249880603717c15@google.com/
> IMHO such change should not land to stable lightly. It's a behavioural
> change with subtle side effects. The threaded napi feature is disabled

Technically speaking, yes, it's actually not a bug but an optimization.

> by default for the same reason. Also the revert has some soft functional
> dependencies in napi threaded cleanup and improvements landed into v6.4,
> i.e. 6bb218b5a6e3f3c5e2e76faf8fc976f3f6f72496, likely more.

Maybe I have a slightly different thought on this: since I really
received/noticed there are not a few reports showing the 'issue' in
the older kernels. It's not easy for some people or company to upgrade
the stable kernel. I'm wondering if it's still possible to land it
with more careful consideration in the patch if Gyutae volunteers to
do so? Please feel free to reject the proposal :)

Thanks,
Jason

Re: Backport request: Revert "softirq: Let ksoftirqd do its job" (d15121be7485)

[Greg KH]

On Fri, Oct 30, 2026 at 02:16:57PM +0800, Jason Xing wrote:
> Hi Paolo,
> 
> On Thu, Oct 29, 2026 at 4:33 PM Paolo Abeni <pabeni@redhat.com> wrote:
> >
> > On 10/29/26 06:42, Gyutae Bae wrote:
> > > Please backport the following upstream commit to 6.1.y and 5.15.y:
> > >
> > >    d15121be7485655129101f3960ae6add40204463
> > >    Revert "softirq: Let ksoftirqd do its job"
> > >
> > > On a production 5.15 PREEMPT_NONE node, long-running overlayfs
> > > getdents64() calls prevented ksoftirqd from running.  RPS IPIs
> > > continued to arrive, but NET_RX processing on that CPU stalled for
> > > up to about 3 seconds.  Over 15 days, the node logged about 1,000
> > > bonding ARP monitor failovers, with no link-state changes recorded
> > > during that period.  The failovers stopped after the workload was
> > > removed.
> > >
> > > On a 5.15.27 PREEMPT_NONE test node, we used a livepatch to make
> > > ksoftirqd_running() always return false.  This removes the same
> > > softirq deferral checks as the revert on this build.  Under the same
> > > load, we measured three 10-minute windows: before patching, with the
> > > livepatch active, and after removing it.  The number of HTTP requests
> > > taking more than 200 ms was 17, 0 and 11, respectively.  TCPTimeouts
> > > increased by 25, 1 and 21.
> > >
> > > We verified with patch --dry-run that the revert applies cleanly
> > > to v6.1.190 and v5.15.223.  Both already contain the related RCU
> > > fix 1dd1eff161bd.  6.6.y and later already contain the revert.
> > >
> > > The original commit was intended to prevent user-task starvation
> > > under a softirq flood.  The revert's changelog points to threaded
> > > NAPI as an alternative, but it is disabled by default.
> > >
> > > The revert was merged in v6.5 without a Cc: stable tag.
> > > Syzbot has also reported bisections to this revert [1].
> > > Paolo, Thomas, Eric: I would appreciate your thoughts on whether
> > > this revert is suitable for 6.1.y and 5.15.y.
> > >
> > > [1] https://lore.kernel.org/all/000000000000a249880603717c15@google.com/
> > IMHO such change should not land to stable lightly. It's a behavioural
> > change with subtle side effects. The threaded napi feature is disabled
> 
> Technically speaking, yes, it's actually not a bug but an optimization.
> 
> > by default for the same reason. Also the revert has some soft functional
> > dependencies in napi threaded cleanup and improvements landed into v6.4,
> > i.e. 6bb218b5a6e3f3c5e2e76faf8fc976f3f6f72496, likely more.
> 
> Maybe I have a slightly different thought on this: since I really
> received/noticed there are not a few reports showing the 'issue' in
> the older kernels. It's not easy for some people or company to upgrade
> the stable kernel. I'm wondering if it's still possible to land it
> with more careful consideration in the patch if Gyutae volunteers to
> do so? Please feel free to reject the proposal :)

Given that 5.15.y is about to go end-of-life in a month, that's a
non-starter.

For 6.1.y, you better be planning how to move off of that _now_ given
that you only have 1 more year before that is gone.  Anyone hitting this
issue should be moving now as that gives you an incentive to do so :)

thanks,

greg k-h