All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Congjie Zhou <zcjie0802@qq.com>
Cc: netdev@vger.kernel.org, jasowangio@gmail.com,
	eperezma@redhat.com, xuanzhuo@linux.alibaba.com,
	andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@kernel.org,
	kuba@kernel.org, pabeni@redhat.com, horms@kernel.org,
	virtualization@lists.linux.dev, linux-kernel@vger.kernel.org
Subject: Re: [BUG] virtio_net: RX stall caused by masking callbacks after NAPI ownership release
Date: Sun, 15 Nov 2026 15:09:52 -0500	[thread overview]
Message-ID: <20261115145652-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <tencent_AA4F39DA7ADE330C721C03AAA832B92DD608@qq.com>

On Wed, Nov 11, 2026 at 09:36:10PM +0800, Congjie Zhou wrote:
> Hello,
> 
> I observed a persistent virtio-net RX stall on Linux 6.18.58 during a
> multi-threaded TCP receive workload. After the stall, the RX virtqueue
> contained unconsumed used entries, but its NAPI instance had no owner,
> the NAPI timer was inactive and virtqueue notifications were disabled.
> The receiving TCP threads remained blocked waiting for data.
> 
> Completion sequence
> ------------------------
> 
> The completion sequence can produce this interleaving:
> 
>   old poll                         successor poll
>   enable_cb_prepare()
>   napi_complete_done()
>     clear SCHED
>     arm watchdog; result=false
>                                    acquire NAPI
>                                    poll, enable callbacks, complete
>   return false to driver
>   disable_cb()                     callbacks are now masked again
> 
> Root cause
> -----------
> 
> virtqueue_napi_complete() calls virtqueue_enable_cb_prepare() before
> napi_complete_done(), then calls virtqueue_disable_cb() if completion
> returns false.
> 
> A false return does not guarantee that the old caller still owns NAPI.
> The hard IRQ deferral path can clear SCHED, arm the watchdog and return
> false. Another CPU can acquire NAPI, poll, rearm callbacks and complete
> before the old caller reaches its final virtqueue_disable_cb(). That
> stale mask overwrites the notification state established by the new
> poll.
> 
> The watchdog is one-shot. If it has already run, there need not be any
> remaining owner or timer to recover after the old caller masks the
> queue. This leaves later used entries without a notification-driven
> poll.
> 
> Current source inspection and local validation
> ----------------------------------------------
> 
> We reproduced the notification-loss condition using a deterministic
> KUnit fault-injection test on the latest mainline snapshot fetched on
> November 10, 2026 (7.4-rc2, commit b3dfbde62cf1). The candidate fix
> passes the same test.
> 
> Candidate fix
> --------------
> 
> I have prepared a candidate change that masks notifications on the
> relevant completion paths before NAPI ownership can be handed off,
> rather than masking after a false return. It adds an ordered,
> non-sleeping, idempotent mask callback to a new completion entry point
> and uses it in virtio-net. Existing completion callers retain the old
> API. virtqueue_enable_cb_prepare() remains before ownership release,
> preserving the ordering required by the earlier RX virtqueue race fix.

so... can we just check for this case? Like this?

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index dd2c41ef6fbe7..41778d727a89a 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -740,6 +740,8 @@ static bool virtqueue_napi_complete(struct napi_struct *napi,
 			return true;
 	} else {
 		virtqueue_disable_cb(vq);
+		if (!hrtimer_active(&napi->timer))
+			napi_schedule(napi);
 	}
 
 	return false;


  parent reply	other threads:[~2026-11-15 20:10 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-11 13:36 [BUG] virtio_net: RX stall caused by masking callbacks after NAPI ownership release Congjie Zhou
2026-11-11 13:50 ` [PATCH net 1/2] net: napi: add completion with an ordered IRQ mask callback Congjie Zhou
2026-11-11 13:56   ` netdev-bot+sinfo
2026-11-11 15:08     ` zcj
2026-11-12 13:52   ` sashiko-bot
2026-11-11 13:50 ` [PATCH net 2/2] virtio_net: mask callbacks before releasing NAPI ownership Congjie Zhou
2026-11-12 13:52   ` sashiko-bot
2026-11-13 10:38     ` zcj
2026-11-15  7:12 ` [PATCH net v2 1/2] net: napi: add completion with an ordered IRQ mask callback Congjie Zhou
2026-11-15  7:21   ` sashiko-bot
2026-11-15 18:26   ` Jakub Kicinski
2026-11-15  7:12 ` [PATCH net v2 2/2] virtio_net: mask callbacks before releasing NAPI ownership Congjie Zhou
2026-11-15  7:21   ` sashiko-bot
2026-11-15 20:09 ` Michael S. Tsirkin [this message]
2026-11-15 23:07 ` [BUG] virtio_net: RX stall caused by masking callbacks after NAPI ownership release Michael S. Tsirkin
2026-11-16  8:47   ` Congjie Zhou
2026-11-16  9:52     ` Michael S. Tsirkin
2026-11-16 16:05       ` Congjie Zhou

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261115145652-mutt-send-email-mst@kernel.org \
    --to=mst@redhat.com \
    --cc=andrew+netdev@lunn.ch \
    --cc=davem@davemloft.net \
    --cc=edumazet@kernel.org \
    --cc=eperezma@redhat.com \
    --cc=horms@kernel.org \
    --cc=jasowangio@gmail.com \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=virtualization@lists.linux.dev \
    --cc=xuanzhuo@linux.alibaba.com \
    --cc=zcjie0802@qq.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.