From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
To: Jakub Kicinski <kuba@kernel.org>, Petr Mladek <pmladek@suse.com>,
John Ogness <john.ogness@linutronix.de>,
Sergey Senozhatsky <senozhatsky@chromium.org>,
Peter Zijlstra <peterz@infradead.org>
Cc: Vlad Poenaru <vlad.wing@gmail.com>,
Thomas Gleixner <tglx@kernel.org>,
netdev@vger.kernel.org, "David S . Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Paolo Abeni <pabeni@redhat.com>, Simon Horman <horms@kernel.org>,
Breno Leitao <leitao@debian.org>,
Clark Williams <clrkwllms@kernel.org>,
Steven Rostedt <rostedt@goodmis.org>,
linux-rt-devel@lists.linux.dev, linux-kernel@vger.kernel.org,
stable@vger.kernel.org, Frederic Weisbecker <frederic@kernel.org>,
Ingo Molnar <mingo@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
K Prateek Nayak <kprateek.nayak@amd.com>
Subject: Re: [PATCH net] netpoll: run NAPI poll in softirq context to avoid rq->lock self-deadlock
Date: Tue, 16 Jun 2026 12:35:29 +0200 [thread overview]
Message-ID: <20260616103529.Yh9Dxsjp@linutronix.de> (raw)
In-Reply-To: <20260611191114.5bc43a59@kernel.org>
On 2026-06-11 19:11:14 [-0700], Jakub Kicinski wrote:
> On Wed, 10 Jun 2026 11:36:21 -0700 Vlad Poenaru wrote:
> > @@ -194,11 +194,56 @@ void netpoll_poll_dev(struct net_device *dev)
> > + local_bh_disable();
> > + poll_napi(dev);
> > + _local_bh_enable();
>
> tglx, Sebastian, are you okay with using _local_bh_enable() to trick
> softirq into not waking ksoftirqd? The problematic path is:
>
> scheduler -> printk -> netconsole -> raise softirq -> scheduler (deadlock)
>
> so the softirq may never get serviced.
>
> In netcons we try to avoid touching the network driver if the Tx path
> locks are already held. Ideally we'd do something similar with the
> scheduler. Try to do bare minimum if we may be in the scheduler.
> Failing that - don't poll the driver if we were called with irqs
> already disabled.
>
> Or maybe we only poll from console->write_thread ?
So this is not an issue since commit 7eab73b18630e ("netconsole: convert
to NBCON console infrastructure"). Because from here now on writes are
deferred to the nbcon thread. So this purely about -stable in this case.
Looking at the patch and the amount of comments vs code changes look
somehow hackish. That ifdef for PREEMPT_RT is not needed because on
PREEMPT_RT we have either nbcon or the legacy console (including
netconsole before the mentioned commit) wrapped in a dedicated thread
(via force_legacy_kthread()).
That means in both cases the flow never ends there and the problem is
limited to !PREEMPT_RT.
Now. The scheduler usually does printk_deferred() because of the rq lock
so it does not deadlock for various reasons. It is kind of a pity that
the various WARN macros don't do that.
I don't think that patch is enough. It works around the problem in this
scenario but should the NIC driver invoke schedule_work() then we are
back here again.
Should the network driver acquire a lock then lockdep might observe
rq -> driver-lock and then driver-lock -> rq and yell dead lock (CPU1
doing AB and CPU2 doing BA). This includes also other console driver so
it is not limited to netconsole.
Point being made is that we should avoid the callchain:
| console_unlock
| vprintk_emit
| __warn
| __enqueue_entity // WARN_ON_ONCE() here -- rq->lock held
| put_prev_entity
| put_prev_task_fair
| __schedule
basically a printk under the rq lock.
We could add printk_deferred_enter/exit() to all the rq_lock() variants.
I think PeterZ loves this the most. And Greg will appreciate it too
while backporting because of all the context changes.
We could also introduce WARN_ON_DEFERRED +variants which do the
printk_deferred_enter/exit() thingy should around the printk and replace
all the WARNs in kernel/sched/.
I *think* the tty/console layer has also a deadlock problem where it
holds locks and then the WARN(), that never triggers, asks for the same
locks again so we might have a second user…
Adding sched and printk folks for opinions while eyeballing
WARN_ON_DEFERRED().
Sebastian
next prev parent reply other threads:[~2026-06-16 10:35 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-06-10 18:36 [PATCH net] netpoll: run NAPI poll in softirq context to avoid rq->lock self-deadlock Vlad Poenaru
[not found] ` <20260611191114.5bc43a59@kernel.org>
2026-06-15 13:56 ` Sebastian Andrzej Siewior
2026-06-16 10:35 ` Sebastian Andrzej Siewior [this message]
2026-06-16 15:11 ` Jakub Kicinski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260616103529.Yh9Dxsjp@linutronix.de \
--to=bigeasy@linutronix.de \
--cc=clrkwllms@kernel.org \
--cc=davem@davemloft.net \
--cc=dietmar.eggemann@arm.com \
--cc=edumazet@google.com \
--cc=frederic@kernel.org \
--cc=horms@kernel.org \
--cc=john.ogness@linutronix.de \
--cc=kprateek.nayak@amd.com \
--cc=kuba@kernel.org \
--cc=leitao@debian.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rt-devel@lists.linux.dev \
--cc=mingo@redhat.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=peterz@infradead.org \
--cc=pmladek@suse.com \
--cc=rostedt@goodmis.org \
--cc=senozhatsky@chromium.org \
--cc=stable@vger.kernel.org \
--cc=tglx@kernel.org \
--cc=vincent.guittot@linaro.org \
--cc=vlad.wing@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox