[BUG] virtio_net: RX stall caused by masking callbacks after NAPI ownership release

[BUG] virtio_net: RX stall caused by masking callbacks after NAPI ownership release

[Congjie Zhou]

Hello,

I observed a persistent virtio-net RX stall on Linux 6.18.58 during a
multi-threaded TCP receive workload. After the stall, the RX virtqueue
contained unconsumed used entries, but its NAPI instance had no owner,
the NAPI timer was inactive and virtqueue notifications were disabled.
The receiving TCP threads remained blocked waiting for data.

Completion sequence
------------------------

The completion sequence can produce this interleaving:

  old poll                         successor poll
  enable_cb_prepare()
  napi_complete_done()
    clear SCHED
    arm watchdog; result=false
                                   acquire NAPI
                                   poll, enable callbacks, complete
  return false to driver
  disable_cb()                     callbacks are now masked again

Root cause
-----------

virtqueue_napi_complete() calls virtqueue_enable_cb_prepare() before
napi_complete_done(), then calls virtqueue_disable_cb() if completion
returns false.

A false return does not guarantee that the old caller still owns NAPI.
The hard IRQ deferral path can clear SCHED, arm the watchdog and return
false. Another CPU can acquire NAPI, poll, rearm callbacks and complete
before the old caller reaches its final virtqueue_disable_cb(). That
stale mask overwrites the notification state established by the new
poll.

The watchdog is one-shot. If it has already run, there need not be any
remaining owner or timer to recover after the old caller masks the
queue. This leaves later used entries without a notification-driven
poll.

Current source inspection and local validation
----------------------------------------------

We reproduced the notification-loss condition using a deterministic
KUnit fault-injection test on the latest mainline snapshot fetched on
November 10, 2026 (7.4-rc2, commit b3dfbde62cf1). The candidate fix
passes the same test.

Candidate fix
--------------

I have prepared a candidate change that masks notifications on the
relevant completion paths before NAPI ownership can be handed off,
rather than masking after a false return. It adds an ordered,
non-sleeping, idempotent mask callback to a new completion entry point
and uses it in virtio-net. Existing completion callers retain the old
API. virtqueue_enable_cb_prepare() remains before ownership release,
preserving the ordering required by the earlier RX virtqueue race fix.

[PATCH net 1/2] net: napi: add completion with an ordered IRQ mask callback

[Congjie Zhou]

napi_complete_done() can clear SCHED, arm the watchdog and return false
when deferring hard IRQs. A false return therefore does not guarantee
that the caller still owns NAPI. Masking interrupts after this return
can overwrite the interrupt state established by a successor poll.

Add napi_complete_done_with_irq_mask() with a mask callback that runs on
false-return paths before ownership can be handed off. The callback
must be non-sleeping, idempotent and leave NAPI state unchanged, since
a state-update retry may invoke it more than once.

Keep napi_complete_done() as a wrapper without a callback, and inline
the shared implementation to eliminate the callback path for existing
callers.

Signed-off-by: Congjie Zhou <zcjie0802@qq.com>
---
 include/linux/netdevice.h | 18 ++++++++++++++++++
 net/core/dev.c            | 29 +++++++++++++++++++++++++++--
 2 files changed, 45 insertions(+), 2 deletions(-)

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 435be7880..810527cd8 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -601,6 +601,24 @@ static inline void napi_schedule_irqoff(struct napi_struct *n)
  */
 bool napi_complete_done(struct napi_struct *n, int work_done);
 
+/**
+ * napi_complete_done_with_irq_mask - complete with an ordered IRQ mask callback
+ * @n: NAPI context
+ * @work_done: number of packets processed
+ * @mask: non-sleeping, idempotent device interrupt mask callback
+ * @data: callback argument
+ *
+ * Like napi_complete_done(), but call @mask on false-return paths before
+ * releasing NAPI ownership. Drivers which prepare interrupt rearming before
+ * completion must not mask interrupts after ownership has been released.
+ * The callback must not modify NAPI state and may run more than once if the
+ * state update is retried.
+ *
+ * Return: false if device interrupts have been masked by the callback.
+ */
+bool napi_complete_done_with_irq_mask(struct napi_struct *n, int work_done,
+				      void (*mask)(void *data), void *data);
+
 static inline bool napi_complete(struct napi_struct *n)
 {
 	return napi_complete_done(n, 0);
diff --git a/net/core/dev.c b/net/core/dev.c
index df1a840b0..4ba9e4c1b 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -7001,7 +7001,10 @@ void __napi_schedule_irqoff(struct napi_struct *n)
 }
 EXPORT_SYMBOL(__napi_schedule_irqoff);
 
-bool napi_complete_done(struct napi_struct *n, int work_done)
+/* Inline so the legacy API eliminates the optional mask callback path. */
+static __always_inline bool
+__napi_complete_done(struct napi_struct *n, int work_done,
+		     void (*mask)(void *data), void *data)
 {
 	unsigned long flags, val, new, timeout = 0;
 	bool ret = true;
@@ -7013,8 +7016,11 @@ bool napi_complete_done(struct napi_struct *n, int work_done)
 	 *    the guarantee we will be called later.
 	 */
 	if (unlikely(n->state & (NAPIF_STATE_NPSVC |
-				 NAPIF_STATE_IN_BUSY_POLL)))
+				 NAPIF_STATE_IN_BUSY_POLL))) {
+		if (mask)
+			mask(data);
 		return false;
+	}
 
 	if (work_done) {
 		if (n->gro.bitmask)
@@ -7047,6 +7053,13 @@ bool napi_complete_done(struct napi_struct *n, int work_done)
 	do {
 		WARN_ON_ONCE(!(val & NAPIF_STATE_SCHED));
 
+		/*
+		 * A false result is not proof of retained ownership. Mask
+		 * before the state transition lets another CPU enter poll.
+		 */
+		if (mask && (!ret || (val & NAPIF_STATE_MISSED)))
+			mask(data);
+
 		new = val & ~(NAPIF_STATE_MISSED | NAPIF_STATE_SCHED |
 			      NAPIF_STATE_SCHED_THREADED |
 			      NAPIF_STATE_PREFER_BUSY_POLL);
@@ -7069,8 +7082,20 @@ bool napi_complete_done(struct napi_struct *n, int work_done)
 			      HRTIMER_MODE_REL_PINNED);
 	return ret;
 }
+
+bool napi_complete_done(struct napi_struct *n, int work_done)
+{
+	return __napi_complete_done(n, work_done, NULL, NULL);
+}
 EXPORT_SYMBOL(napi_complete_done);
 
+bool napi_complete_done_with_irq_mask(struct napi_struct *n, int work_done,
+				      void (*mask)(void *data), void *data)
+{
+	return __napi_complete_done(n, work_done, mask, data);
+}
+EXPORT_SYMBOL(napi_complete_done_with_irq_mask);
+
 static void __skb_defer_free_flush(struct skb_defer_node *sdn, int budget)
 {
 	struct llist_node *free_list;

base-commit: b3dfbde62cf1981c4638939c159dfd087b7a72e9
-- 
2.43.0

Re: [PATCH net 1/2] net: napi: add completion with an ordered IRQ mask callback

[netdev-bot+sinfo]

Hi!

This is an automated message. This series looks like a fix, but its
commit messages seem to be missing some information:

 - How the issue was discovered, e.g. hit in production, hit during
   development, syzbot report, manual code inspection, LLM or static
   analysis tool scan.

 - Whether the issue was actually triggered, or is only theoretical
   (e.g. found by LLM and not proven at runtime). If it was triggered
   please include the symptoms, like the stack trace or kernel messages.

Please do not repost the series just to address the above. Instead,
reply to this email with the missing information, so that reviewers
can take it into account. If the series needs another revision for
other reasons, please include the information in the commit messages
then.

The evaluation is done by an LLM so it may be wrong, if you think
that is the case please reply and explain.

Re: [PATCH net 1/2] net: napi: add completion with an ordered IRQ mask callback

[zcj]

Hi,

I first encountered this issue on Linux 6.18.58 during a multi-threaded
TCP receive workload with virtio-net in KVM guests, and then investigated
the NAPI/virtqueue completion ordering.

The observed symptom was a persistent RX stall: the RX virtqueue had
unconsumed used entries, but notifications were disabled, NAPI had no
owner and its watchdog timer was inactive. The receiving TCP threads
remained blocked waiting for data.

A deterministic KUnit fault-injection test also reproduced the
notification-loss ordering using the production NAPI completion APIs
and watchdog. The test deliberately delays the old caller until a
successor poll has rearmed callbacks and completed. The proposed fix
passes the same test. 

The original bug report describes the symptoms and completion race:
https://lore.kernel.org/netdev/tencent_AA4F39DA7ADE330C721C03AAA832B92DD608@qq.com/

Thanks,
Congjie Zhou

Re: [PATCH net 1/2] net: napi: add completion with an ordered IRQ mask callback

[sashiko-bot]

> napi_complete_done() can clear SCHED, arm the watchdog and return false
> when deferring hard IRQs. A false return therefore does not guarantee
> that the caller still owns NAPI. Masking interrupts after this return
> can overwrite the interrupt state established by a successor poll.
> 
> Add napi_complete_done_with_irq_mask() with a mask callback that runs on
> false-return paths before ownership can be handed off. The callback
> must be non-sleeping, idempotent and leave NAPI state unchanged, since
> a state-update retry may invoke it more than once.
> 
> Keep napi_complete_done() as a wrapper without a callback, and inline
> the shared implementation to eliminate the callback path for existing
> callers.
> 
> Signed-off-by: Congjie Zhou <zcjie0802@qq.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/tencent_74EA7A2E8BF5758E9C5EF18C9B1CED401D06@qq.com?part=1

[PATCH net 2/2] virtio_net: mask callbacks before releasing NAPI ownership

[Congjie Zhou]

virtqueue_napi_complete() prepares callbacks before calling
napi_complete_done(), then disables them if completion returns false.
On the hard IRQ deferral path, completion can release NAPI ownership
and arm the watchdog before returning false.

A successor poll can rearm callbacks and complete before the old caller
executes virtqueue_disable_cb(). The stale mask then disables
notifications again. If the one-shot watchdog has already run, no NAPI
owner or timer remains to consume later used entries, causing a
persistent RX stall.

Use napi_complete_done_with_irq_mask() to disable callbacks before
completion can hand NAPI to a successor, and remove the stale mask after
a false return. Keep virtqueue_enable_cb_prepare() before completion:
moving it after ownership release would reintroduce the RX virtqueue
race fixed by commit cbdadbbf0c79 ("virtio_net: fix race in RX VQ
processing").

Signed-off-by: Congjie Zhou <zcjie0802@qq.com>
---
 drivers/net/virtio_net.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index dd2c41ef6..0de6bd95a 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -727,19 +727,23 @@ static void virtqueue_napi_schedule(struct napi_struct *napi,
 	}
 }
 
+static void virtqueue_napi_mask(void *data)
+{
+	virtqueue_disable_cb(data);
+}
+
 static bool virtqueue_napi_complete(struct napi_struct *napi,
 				    struct virtqueue *vq, int processed)
 {
 	int opaque;
 
 	opaque = virtqueue_enable_cb_prepare(vq);
-	if (napi_complete_done(napi, processed)) {
+	if (napi_complete_done_with_irq_mask(napi, processed,
+					     virtqueue_napi_mask, vq)) {
 		if (unlikely(virtqueue_poll(vq, opaque)))
 			virtqueue_napi_schedule(napi, vq);
 		else
 			return true;
-	} else {
-		virtqueue_disable_cb(vq);
 	}
 
 	return false;
-- 
2.43.0

Re: [PATCH net 2/2] virtio_net: mask callbacks before releasing NAPI ownership

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] virtio_net: missing Fixes trailer for virtqueue_napi_complete()
  race fix
--

commit c4014f837fc88f44df9c4728b3bea8a252b5c2fb
Author: Congjie Zhou <zcjie0802@qq.com>

> virtio_net: mask callbacks before releasing NAPI ownership
>
> virtqueue_napi_complete() prepares callbacks before calling
> napi_complete_done(), then disables them if completion returns false.
> On the hard IRQ deferral path, completion can release NAPI ownership
> and arm the watchdog before returning false.
>
> A successor poll can rearm callbacks and complete before the old caller
> executes virtqueue_disable_cb(). The stale mask then disables
> notifications again. If the one-shot watchdog has already run, no NAPI
> owner or timer remains to consume later used entries, causing a
> persistent RX stall.
>
> Use napi_complete_done_with_irq_mask() to disable callbacks before
> completion can hand NAPI to a successor, and remove the stale mask after
> a false return. Keep virtqueue_enable_cb_prepare() before completion:
> moving it after ownership release would reintroduce the RX virtqueue
> race fixed by commit cbdadbbf0c79 ("virtio_net: fix race in RX VQ
> processing").
>
> Signed-off-by: Congjie Zhou <zcjie0802@qq.com>

[Severity: Low]
This fixes a persistent RX stall race in virtqueue_napi_complete().
Should it include a Fixes tag for the post-completion virtqueue_disable_cb()
call?

  Fixes: fdaa767aefc1 ("virtio_net: Disable interrupts if napi_complete_done rescheduled napi")

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/tencent_74EA7A2E8BF5758E9C5EF18C9B1CED401D06@qq.com?part=2