[PATCH RFC] PCI: Don't restore state of a device whose config space is inaccessible

[PATCH RFC] PCI: Don't restore state of a device whose config space is inaccessible

[Patryk Janik]

[-- Attachment #1: Type: text/plain, Size: 2762 bytes --]

After a failed reset, vfio_pci_core_disable() unconditionally calls
pci_restore_state(). When the device did not come back (FLR timed out,
config reads return ~0), restoring state walks the extended capability
list and rewrites the whole config space. Every one of those accesses
stalls for a completion timeout, so the host spends minutes in soft
lockups instead of failing fast.

Seen with an NVIDIA RTX PRO 6000 Blackwell Max-Q (10de:2bb4) on vfio-pci
after the guest was terminated (Ubuntu 6.8.0-147, code path unchanged in
v7.3):

  vfio-pci 0000:81:00.0: not ready 65535ms after FLR; giving up
  watchdog: BUG: soft lockup - CPU#9 stuck for 28s! [worker:361063]
  RIP: 0010:pci_mmcfg_read+0xba/0x110
  Call Trace:
   pci_read_config_dword+0x27/0x50
   pci_find_next_ext_capability+0x83/0xe0
   pci_find_ext_capability+0x12/0x20
   pci_restore_ltr_state+0x29/0x60
   pci_restore_state+0x16/0x270
   vfio_pci_core_disable+0x339/0x440 [vfio_pci_core]
   vfio_pci_core_close_device+0x68/0xe0 [vfio_pci_core]

191 soft lockups followed over ~2.7 hours, until the host was power
cycled. The device itself is a separate (firmware) problem and stays
dead either way; this patch only stops the host from hammering it.

pci_save_state() and pcie_flr() already bail out through
pci_dev_config_accessible(); do the same in pci_restore_state().

Signed-off-by: Patryk Janik <patrykjanik1710@gmail.com>
---
RFC: written against v7.3 (master 21a7f44fd076); not tested on hardware
yet. Happy to test a revised version on the affected machine.

Full report, timeline, nvidia-bug-report.log and the complete kernel log
of the failing boot are attached to the NVIDIA forum topic:
https://forums.developer.nvidia.com/t/386790

Related:
- "PCI: Avoid FLR for NVIDIA 5090 GPU" (linux-pci, 2026-04-16), where the
  underlying FLR failure was described as a known device firmware defect.
- nova-core "run unload sequence" series (merged v7.2), which mentions a
  Blackwell VBIOS bug that sometimes breaks the GPU upon PCI reset.

 drivers/pci/pci.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -1883,6 +1883,16 @@
  */
 void pci_restore_state(struct pci_dev *dev)
 {
+ /*
+ * A device that did not come back from reset (e.g. an FLR that
+ * timed out) returns ~0 for every config read. Restoring state
+ * would walk the extended capability list and write the whole
+ * config space, each access stalling for a completion timeout,
+ * which shows up as minutes of soft lockups. Bail out instead.
+ */
+ if (!pci_dev_config_accessible(dev, "restore state"))
+ return;
+
  pci_restore_pcie_state(dev);
  pci_restore_pasid_state(dev);
  pci_restore_pri_state(dev);
--
2.47.0

[-- Attachment #2: 0001-PCI-dont-restore-inaccessible-device.patch --]
[-- Type: application/octet-stream, Size: 2022 bytes --]

From: <your name> <<your email>>
Subject: [PATCH RFC] PCI: Don't restore state of a device whose config space is inaccessible

After a failed reset, vfio_pci_core_disable() unconditionally calls
pci_restore_state().  When the device did not come back (FLR timed out,
config reads return ~0), restoring state walks the extended capability
list and rewrites the whole config space.  Every one of those accesses
stalls for a completion timeout, so the host spends minutes in soft
lockups instead of failing fast.

Seen with an NVIDIA RTX PRO 6000 Blackwell Max-Q (10de:2bb4) on vfio-pci
after the guest was killed:

  vfio-pci 0000:81:00.0: not ready 65535ms after FLR; giving up
  watchdog: BUG: soft lockup - CPU#9 stuck for 28s! [worker:361063]
  RIP: 0010:pci_mmcfg_read+0xba/0x110
   pci_find_ext_capability+0x12/0x20
   pci_restore_ltr_state+0x29/0x60
   pci_restore_state+0x16/0x270
   vfio_pci_core_disable+0x339/0x440 [vfio_pci_core]

191 soft lockups followed over ~2.7 hours, until the host was power
cycled.  The device itself is a separate (firmware) problem and stays
dead either way; this patch only stops the host from hammering it.

pci_save_state() and pcie_flr() already bail out through
pci_dev_config_accessible(); do the same in pci_restore_state().

Signed-off-by: <your name> <<your email>>
---
RFC: written against v7.3 master (21a7f44fd076); not tested on hardware yet.

--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -1883,6 +1883,16 @@
  */
 void pci_restore_state(struct pci_dev *dev)
 {
+	/*
+	 * A device that did not come back from reset (e.g. an FLR that
+	 * timed out) returns ~0 for every config read.  Restoring state
+	 * would walk the extended capability list and write the whole
+	 * config space, each access stalling for a completion timeout,
+	 * which shows up as minutes of soft lockups.  Bail out instead.
+	 */
+	if (!pci_dev_config_accessible(dev, "restore state"))
+		return;
+
 	pci_restore_pcie_state(dev);
 	pci_restore_pasid_state(dev);
 	pci_restore_pri_state(dev);

Re: [PATCH RFC] PCI: Don't restore state of a device whose config space is inaccessible

[Lukas Wunner]

On Sun, Oct 25, 2026 at 05:37:39PM +0100, Patryk Janik wrote:
> After a failed reset, vfio_pci_core_disable() unconditionally calls
> pci_restore_state(). When the device did not come back (FLR timed out,
> config reads return ~0), restoring state walks the extended capability
> list and rewrites the whole config space. Every one of those accesses
> stalls for a completion timeout, so the host spends minutes in soft
> lockups instead of failing fast.

Config Space Writes are Posted Requests.  There are no Completions
sent in return and consequently no Completion Timeout is observed.
So the above paragraph doesn't really make sense.

> - "PCI: Avoid FLR for NVIDIA 5090 GPU" (linux-pci, 2026-04-16), where the
>   underlying FLR failure was described as a known device firmware defect.

I assume you're referring to:

https://lore.kernel.org/r/20260416070707.3242381-1-yuan.gao@ucloud.cn/

The outcome was that there's a firmware fix available and a quirk
should at least differentiate between good and bad firmware.
Or the fix is to update the firmware and then we don't need a quirk.

Thanks,

Lukas

Re: [PATCH RFC] PCI: Don't restore state of a device whose config space is inaccessible

[Patryk Janik]

On Sun, Oct 25, 2026 at 05:58PM +0100, Lukas Wunner wrote:
> Config Space Writes are Posted Requests. There are no Completions
> sent in return and consequently no Completion Timeout is observed.
> So the above paragraph doesn't really make sense.

You're right that the writes aren't where the time goes; the
paragraph is worded badly. (Strictly, Configuration Writes are
Non-Posted on the link and do get a Completion, but with ECAM the CPU
doesn't wait for it, so they don't stall either way.)

What stalls are the config *reads*. The trace is in pci_mmcfg_read()
with RCX = R13 = 0xffc: when every header reads ~0,
PCI_EXT_CAP_NEXT(0xffffffff) is 0xffc, and
pci_find_next_ext_capability() re-reads that offset until its ttl of
480 expires. pci_restore_state() does about ten such walks, so a
single restore issues several thousand config reads to a device that
no longer answers, each one stalling until the Root Port's Completion
Timeout. On this host that was ~56 minutes in pci_restore_state()
(and another ~33 minutes in the subsequent save for the bus reset).

I'll fix the commit message in a v2.

> The outcome was that there's a firmware fix available and a quirk
> should at least differentiate between good and bad firmware.
> Or the fix is to update the firmware and then we don't need a quirk.

Agreed on the FLR failure itself, which I'm pursuing with NVIDIA
separately (for this SKU, 10de:2bb4, no newer VBIOS is published yet).

This patch is not a quirk, though, and not NVIDIA-specific: it doesn't
touch the reset, it only makes pci_restore_state() bail out on a
device whose config space reads ~0, the same way pci_save_state() and
pcie_flr() already do via pci_dev_config_accessible(). Whatever
firmware the device runs, once it has fallen off after a failed reset
the kernel shouldn't spend an hour in soft lockups restoring state
into it. The same applies to any device that dies during reset.

I left dev->state_saved untouched on the early return so a later
restore can still apply the saved state if the device comes back.

Thanks,
Patryk Janik


On Sun, 25 Oct 2026 at 17:58, Lukas Wunner <lukas@wunner.de> wrote:
>
> On Sun, Oct 25, 2026 at 05:37:39PM +0100, Patryk Janik wrote:
> > After a failed reset, vfio_pci_core_disable() unconditionally calls
> > pci_restore_state(). When the device did not come back (FLR timed out,
> > config reads return ~0), restoring state walks the extended capability
> > list and rewrites the whole config space. Every one of those accesses
> > stalls for a completion timeout, so the host spends minutes in soft
> > lockups instead of failing fast.
>
> Config Space Writes are Posted Requests.  There are no Completions
> sent in return and consequently no Completion Timeout is observed.
> So the above paragraph doesn't really make sense.
>
> > - "PCI: Avoid FLR for NVIDIA 5090 GPU" (linux-pci, 2026-04-16), where the
> >   underlying FLR failure was described as a known device firmware defect.
>
> I assume you're referring to:
>
> https://lore.kernel.org/r/20260416070707.3242381-1-yuan.gao@ucloud.cn/
>
> The outcome was that there's a firmware fix available and a quirk
> should at least differentiate between good and bad firmware.
> Or the fix is to update the firmware and then we don't need a quirk.
>
> Thanks,
>
> Lukas