Linux kernel -stable discussions
 help / color / mirror / Atom feed
From: Sasha Levin <sashal@kernel.org>
To: patches@lists.linux.dev, stable@vger.kernel.org
Cc: Junnan Wu <junnan01.wu@samsung.com>,
	Ying Xu <ying123.xu@samsung.com>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	Sasha Levin <sashal@kernel.org>,
	jasowang@redhat.com, andrew+netdev@lunn.ch, davem@davemloft.net,
	edumazet@google.com, kuba@kernel.org, pabeni@redhat.com,
	virtualization@lists.linux.dev, netdev@vger.kernel.org
Subject: [PATCH AUTOSEL 6.16] virtio_net: adjust the execution order of function `virtnet_close` during freeze
Date: Tue,  2 Sep 2025 08:08:28 -0400	[thread overview]
Message-ID: <20250902120833.1342615-17-sashal@kernel.org> (raw)
In-Reply-To: <20250902120833.1342615-1-sashal@kernel.org>

From: Junnan Wu <junnan01.wu@samsung.com>

[ Upstream commit 45d8ef6322b8a828d3b1e2cfb8893e2ff882cb23 ]

"Use after free" issue appears in suspend once race occurs when
napi poll scheduls after `netif_device_detach` and before napi disables.

For details, during suspend flow of virtio-net,
the tx queue state is set to "__QUEUE_STATE_DRV_XOFF" by CPU-A.

And at some coincidental times, if a TCP connection is still working,
CPU-B does `virtnet_poll` before napi disable.
In this flow, the state "__QUEUE_STATE_DRV_XOFF"
of tx queue will be cleared. This is not the normal process it expects.

After that, CPU-A continues to close driver then virtqueue is removed.

Sequence likes below:
--------------------------------------------------------------------------
              CPU-A                            CPU-B
              -----                            -----
         suspend is called                  A TCP based on
                                        virtio-net still work
 virtnet_freeze
 |- virtnet_freeze_down
 | |- netif_device_detach
 | | |- netif_tx_stop_all_queues
 | |  |- netif_tx_stop_queue
 | |   |- set_bit
 | |     (__QUEUE_STATE_DRV_XOFF,...)
 | |                                     softirq rasied
 | |                                    |- net_rx_action
 | |                                     |- napi_poll
 | |                                      |- virtnet_poll
 | |                                       |- virtnet_poll_cleantx
 | |                                        |- netif_tx_wake_queue
 | |                                         |- test_and_clear_bit
 | |                                          (__QUEUE_STATE_DRV_XOFF,...)
 | |- virtnet_close
 |  |- virtnet_disable_queue_pair
 |   |- virtnet_napi_tx_disable
 |- remove_vq_common
--------------------------------------------------------------------------

When TCP delayack timer is up, a cpu gets softirq and irq handler
`tcp_delack_timer_handler` will be called, which will finally call
`start_xmit` in virtio net driver.
Then the access to tx virtq will cause panic.

The root cause of this issue is that napi tx
is not disable before `netif_tx_stop_queue`,
once `virnet_poll` schedules in such coincidental time,
the tx queue state will be cleared.

To solve this issue, adjusts the order of
function `virtnet_close` in `virtnet_freeze_down`.

Co-developed-by: Ying Xu <ying123.xu@samsung.com>
Signed-off-by: Ying Xu <ying123.xu@samsung.com>
Signed-off-by: Junnan Wu <junnan01.wu@samsung.com>
Message-Id: <20250812090817.3463403-1-junnan01.wu@samsung.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---

LLM Generated explanations, may be completely bogus:

**Backport Status: YES**

This commit should be backported to stable kernel trees. Here's my
extensive analysis:

## Critical Bug Fix
This commit fixes a **use-after-free vulnerability** in the virtio_net
driver that can cause kernel panics during system suspend. The issue is
a race condition between the suspend process and active network traffic,
specifically:

1. **Race Condition Details** (from commit message):
   - CPU-A calls `netif_device_detach()` which sets
     `__QUEUE_STATE_DRV_XOFF` on TX queues
   - CPU-B running `virtnet_poll()` can clear this flag via
     `netif_tx_wake_queue()`
   - CPU-A then calls `virtnet_close()` which disables NAPI and removes
     virtqueues
   - Later, when TCP delayed ACK timer fires, accessing the removed TX
     virtqueue causes a panic

## Code Analysis

The fix is **minimal and contained** - it simply reorders two operations
in `virtnet_freeze_down()`:

**Before (buggy order):**
```c
netif_tx_lock_bh(vi->dev);
netif_device_detach(vi->dev);  // Sets __QUEUE_STATE_DRV_XOFF
netif_tx_unlock_bh(vi->dev);
if (netif_running(vi->dev)) {
    virtnet_close(vi->dev);     // Disables NAPI - TOO LATE!
}
```

**After (fixed order):**
```c
if (netif_running(vi->dev)) {
    virtnet_close(vi->dev);     // Disables NAPI FIRST
}
netif_tx_lock_bh(vi->dev);
netif_device_detach(vi->dev);  // Then detach device
netif_tx_unlock_bh(vi->dev);
```

## Backport Suitability Criteria

1. **Fixes a real bug**: YES - Use-after-free causing kernel panics
2. **Small and contained**: YES - Only 8 lines moved, no logic changes
3. **No side effects**: YES - Just reordering existing operations
4. **No architectural changes**: YES - Simple ordering fix
5. **Critical subsystem**: YES - Network driver suspend/resume path
6. **Clear bug fix**: YES - Not a feature addition
7. **Minimal regression risk**: YES - The new order is logically correct
   (disable NAPI before marking queues stopped)

## Historical Context

From git blame analysis:
- The problematic code structure has existed since 2017 (commit
  9fe7bfce8b3e11)
- Recent changes in 2025 (commit e7231f49d52682) modified the locking
  but didn't fix the race
- This indicates the bug has been latent for **years** and affects many
  kernel versions

## Impact Assessment

- **Severity**: HIGH - Causes kernel panics during suspend with active
  network traffic
- **Likelihood**: MODERATE - Requires specific timing but can happen
  with normal TCP traffic
- **Affected systems**: All systems using virtio_net driver (VMs,
  containers)

The commit already has "[ Upstream commit
45d8ef6322b8a828d3b1e2cfb8893e2ff882cb23 ]" indicating it's been
identified for stable backporting, which further confirms its
importance.

 drivers/net/virtio_net.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 82b4a2a2b8c42..c9b18b13cd940 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -5728,14 +5728,15 @@ static void virtnet_freeze_down(struct virtio_device *vdev)
 	disable_rx_mode_work(vi);
 	flush_work(&vi->rx_mode_work);
 
-	netif_tx_lock_bh(vi->dev);
-	netif_device_detach(vi->dev);
-	netif_tx_unlock_bh(vi->dev);
 	if (netif_running(vi->dev)) {
 		rtnl_lock();
 		virtnet_close(vi->dev);
 		rtnl_unlock();
 	}
+
+	netif_tx_lock_bh(vi->dev);
+	netif_device_detach(vi->dev);
+	netif_tx_unlock_bh(vi->dev);
 }
 
 static int init_vqs(struct virtnet_info *vi);
-- 
2.50.1


      parent reply	other threads:[~2025-09-02 12:08 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-09-02 12:08 [PATCH AUTOSEL 6.16] gpiolib: acpi: Add quirk for ASUS ProArt PX13 Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16] pinctrl: meson: Fix typo in device table macro Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16-5.15] HID: intel-ish-hid: Increase ISHTP resume ack timeout to 300ms Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16-5.4] gpio: timberdale: fix off-by-one in IRQ type boundary check Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16-6.12] drm/msm: Fix a7xx debugbus read Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16-6.1] ata: ahci_xgene: Use int type for 'rc' to store error codes Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16] HID: elecom: add support for ELECOM M-DT2DRBK Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16-6.12] hid: fix I2C read buffer overflow in raw_event() for mcp2221 Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16-6.12] drm/msm: Fix a7xx TPL1 cluster snapshot Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16-6.1] virtio_input: Improve freeze handling Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16-6.12] regulator: pm8008: fix probe failure due to negative voltage selector Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16-6.12] drm/msm: Fix debugbus snapshot Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16-6.6] HID: quirks: add support for Legion Go dual dinput modes Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16-6.6] HID: logitech: Add ids for G PRO 2 LIGHTSPEED Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16-6.12] drm/msm: Fix order of selector programming in cluster snapshot Sasha Levin
2025-09-02 12:08 ` [PATCH AUTOSEL 6.16-5.4] HID: hid-ntrig: fix unable to handle page fault in ntrig_report_version() Sasha Levin
2025-09-02 12:08 ` Sasha Levin [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20250902120833.1342615-17-sashal@kernel.org \
    --to=sashal@kernel.org \
    --cc=andrew+netdev@lunn.ch \
    --cc=davem@davemloft.net \
    --cc=edumazet@google.com \
    --cc=jasowang@redhat.com \
    --cc=junnan01.wu@samsung.com \
    --cc=kuba@kernel.org \
    --cc=mst@redhat.com \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=patches@lists.linux.dev \
    --cc=stable@vger.kernel.org \
    --cc=virtualization@lists.linux.dev \
    --cc=ying123.xu@samsung.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox