The Linux Kernel Mailing List
 help / color / mirror / Atom feed
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Congjie Zhou <zcjie0802@qq.com>
Cc: netdev@vger.kernel.org, jasowangio@gmail.com,
	eperezma@redhat.com, xuanzhuo@linux.alibaba.com,
	andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@kernel.org,
	kuba@kernel.org, pabeni@redhat.com, horms@kernel.org,
	virtualization@lists.linux.dev, linux-kernel@vger.kernel.org
Subject: Re: [BUG] virtio_net: RX stall caused by masking callbacks after NAPI ownership release
Date: Sun, 15 Nov 2026 15:09:52 -0500	[thread overview]
Message-ID: <20261115145652-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <tencent_AA4F39DA7ADE330C721C03AAA832B92DD608@qq.com>

On Wed, Nov 11, 2026 at 09:36:10PM +0800, Congjie Zhou wrote:
> Hello,
> 
> I observed a persistent virtio-net RX stall on Linux 6.18.58 during a
> multi-threaded TCP receive workload. After the stall, the RX virtqueue
> contained unconsumed used entries, but its NAPI instance had no owner,
> the NAPI timer was inactive and virtqueue notifications were disabled.
> The receiving TCP threads remained blocked waiting for data.
> 
> Completion sequence
> ------------------------
> 
> The completion sequence can produce this interleaving:
> 
>   old poll                         successor poll
>   enable_cb_prepare()
>   napi_complete_done()
>     clear SCHED
>     arm watchdog; result=false
>                                    acquire NAPI
>                                    poll, enable callbacks, complete
>   return false to driver
>   disable_cb()                     callbacks are now masked again
> 
> Root cause
> -----------
> 
> virtqueue_napi_complete() calls virtqueue_enable_cb_prepare() before
> napi_complete_done(), then calls virtqueue_disable_cb() if completion
> returns false.
> 
> A false return does not guarantee that the old caller still owns NAPI.
> The hard IRQ deferral path can clear SCHED, arm the watchdog and return
> false. Another CPU can acquire NAPI, poll, rearm callbacks and complete
> before the old caller reaches its final virtqueue_disable_cb(). That
> stale mask overwrites the notification state established by the new
> poll.
> 
> The watchdog is one-shot. If it has already run, there need not be any
> remaining owner or timer to recover after the old caller masks the
> queue. This leaves later used entries without a notification-driven
> poll.
> 
> Current source inspection and local validation
> ----------------------------------------------
> 
> We reproduced the notification-loss condition using a deterministic
> KUnit fault-injection test on the latest mainline snapshot fetched on
> November 10, 2026 (7.4-rc2, commit b3dfbde62cf1). The candidate fix
> passes the same test.
> 
> Candidate fix
> --------------
> 
> I have prepared a candidate change that masks notifications on the
> relevant completion paths before NAPI ownership can be handed off,
> rather than masking after a false return. It adds an ordered,
> non-sleeping, idempotent mask callback to a new completion entry point
> and uses it in virtio-net. Existing completion callers retain the old
> API. virtqueue_enable_cb_prepare() remains before ownership release,
> preserving the ordering required by the earlier RX virtqueue race fix.

so... can we just check for this case? Like this?

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index dd2c41ef6fbe7..41778d727a89a 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -740,6 +740,8 @@ static bool virtqueue_napi_complete(struct napi_struct *napi,
 			return true;
 	} else {
 		virtqueue_disable_cb(vq);
+		if (!hrtimer_active(&napi->timer))
+			napi_schedule(napi);
 	}
 
 	return false;


      parent reply	other threads:[~2026-11-15 20:10 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-11 13:36 [BUG] virtio_net: RX stall caused by masking callbacks after NAPI ownership release Congjie Zhou
2026-11-11 13:50 ` [PATCH net 1/2] net: napi: add completion with an ordered IRQ mask callback Congjie Zhou
2026-11-11 13:56   ` netdev-bot+sinfo
2026-11-11 15:08     ` zcj
2026-11-11 13:50 ` [PATCH net 2/2] virtio_net: mask callbacks before releasing NAPI ownership Congjie Zhou
2026-11-15  7:12 ` [PATCH net v2 1/2] net: napi: add completion with an ordered IRQ mask callback Congjie Zhou
2026-11-15 18:26   ` Jakub Kicinski
2026-11-15  7:12 ` [PATCH net v2 2/2] virtio_net: mask callbacks before releasing NAPI ownership Congjie Zhou
2026-11-15 20:09 ` Michael S. Tsirkin [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261115145652-mutt-send-email-mst@kernel.org \
    --to=mst@redhat.com \
    --cc=andrew+netdev@lunn.ch \
    --cc=davem@davemloft.net \
    --cc=edumazet@kernel.org \
    --cc=eperezma@redhat.com \
    --cc=horms@kernel.org \
    --cc=jasowangio@gmail.com \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=virtualization@lists.linux.dev \
    --cc=xuanzhuo@linux.alibaba.com \
    --cc=zcjie0802@qq.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox