[RFC PATCH 0/1] vfio/pci: Wait for GB202 firmware after reset

[RFC PATCH 0/1] vfio/pci: Wait for GB202 firmware after reset

[Gustavo A.]

On the affected system, GPU passthrough worked with the CachyOS 7.2.10
host kernel. After upgrading the host to 7.3, the guest still booted but
its physical display stayed grey. The exact triggering change is unknown,
so this patch has no Fixes tag.

PCI reset returns successfully while the Firmware Security Processor (FSP)
is still initializing. Restricting BAR1 to 4 KiB mappings masks the problem.
Waiting for FSP completion after reset restores normal output with stock
KVM and verified 1 GiB BAR1 mappings.

The RFC question is where this wait belongs. It currently follows
successful resets in the VFIO enable, reset ioctl, PCIe/AF FLR and hot-reset
paths. Would a PCI-core readiness check after state restoration be more
appropriate? A device-specific reset method runs before BAR and state
restoration and does not cover all bus resets. The existing reset_done
callback cannot propagate errors.

Validation:
- Local host and Linux guest: 7.3.4-1-cachyos, QEMU 11.1.3, GPU 10de:2b85.
  The physical display works with startup tracing disabled. The submitted
  readiness code adds only an IORESOURCE_UNSET guard to this tested code.
  A focused check fails without the guard and passes with it.
- Rebased onto mainline 97d73c825122 (7.4-rc2). Full W=1 builds pass with
  GCC 16.2.1 (VFIO modules, PM enabled) and Clang 23.1.5 (VFIO built in,
  PM disabled). No VFIO compiler warnings. Unrelated warnings remain.
- 79 helper checks and 51 reset-path checks pass using extracted source
  with modeled PCI, time and locking. The unpatched reset-path baseline
  fails 22 checks. The hot-reset model covers completion and unwind for
  three devices, excluding ownership validation and initial locking.
- Strict checkpatch and kernel-doc pass. Sparse 0.6.4 reports the same 17
  diagnostics on patched and unpatched code, with no added findings.

The rebased mainline kernel has not been booted. Runtime lockdep, D3/resume
and repeated hardware reset cycles remain untested. Hardware testing covers
only the device listed above.

Related FSP-ordering work:
https://lists.openwall.net/linux-kernel/2026/06/15/1402

Gustavo A. (1):
  vfio/pci: Wait for GB202 firmware initialization after reset

[RFC PATCH] vfio/pci: Wait for GB202 firmware initialization after reset

[Gustavo A.]

From: "Gustavo A." <gustavogyn@gmail.com>

On the affected system, GPU passthrough worked with the CachyOS 7.2.10
host kernel. After upgrading the host to 7.3, the guest still booted but
its physical display stayed grey. The exact triggering change is unknown.

GB202 GPUs can complete a PCI reset before their Firmware Security
Processor (FSP) finishes initialization. Wait for the BAR firewall to open,
then poll FSP's boot-complete status before reporting reset success. This
restores normal display output with huge BAR1 mappings.

Use a shared five-second timeout. Reject unassigned BAR0 resources,
temporarily enable memory decoding to read FSP status, and restore the
original decoding state while preserving concurrent command-register
updates.

Apply the check to physical GB202 GPU functions, covering both GeForce
and RTX PRO variants.

Link: https://github.com/NVIDIA/open-gpu-kernel-modules/blob/e6a2e01579fbbd1ca2ed43fef2ee2589e9bd8bd9/src/nvidia/src/kernel/gpu/fsp/arch/blackwell/kern_fsp_gb202.c
Link: https://github.com/NVIDIA/open-gpu-kernel-modules/blob/e6a2e01579fbbd1ca2ed43fef2ee2589e9bd8bd9/src/nvidia/src/kernel/gpu/bif/arch/blackwell/kernel_bif_gb202.c
Signed-off-by: Gustavo A. <gustavogyn@gmail.com>
---
 drivers/vfio/pci/vfio_pci_config.c |  16 +++-
 drivers/vfio/pci/vfio_pci_core.c   | 141 +++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_priv.h   |   2 +
 3 files changed, 157 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_config.c
b/drivers/vfio/pci/vfio_pci_config.c
index 9914f3ac6..94d49f2e2 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -908,12 +908,18 @@ static int vfio_exp_config_write(struct
vfio_pci_core_device *vdev, int pos,
 						 &cap);

 		if (!ret && (cap & PCI_EXP_DEVCAP_FLR)) {
+			int ready_ret = 0;
+
 			vfio_pci_zap_and_down_write_memory_lock(vdev);
 			vfio_pci_dma_buf_move(vdev, true);
-			pci_try_reset_function(vdev->pdev);
+			ret = pci_try_reset_function(vdev->pdev);
+			if (!ret)
+				ready_ret = vfio_pci_gb202_wait_ready(vdev);
 			if (__vfio_pci_memory_enabled(vdev))
 				vfio_pci_dma_buf_move(vdev, false);
 			up_write(&vdev->memory_lock);
+			if (ready_ret)
+				return ready_ret;
 		}
 	}

@@ -993,12 +999,18 @@ static int vfio_af_config_write(struct
vfio_pci_core_device *vdev, int pos,
 						&cap);

 		if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP)) {
+			int ready_ret = 0;
+
 			vfio_pci_zap_and_down_write_memory_lock(vdev);
 			vfio_pci_dma_buf_move(vdev, true);
-			pci_try_reset_function(vdev->pdev);
+			ret = pci_try_reset_function(vdev->pdev);
+			if (!ret)
+				ready_ret = vfio_pci_gb202_wait_ready(vdev);
 			if (__vfio_pci_memory_enabled(vdev))
 				vfio_pci_dma_buf_move(vdev, false);
 			up_write(&vdev->memory_lock);
+			if (ready_ret)
+				return ready_ret;
 		}
 	}

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index fcbb2da86..c77de8868 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -17,6 +17,8 @@
 #include <linux/file.h>
 #include <linux/interrupt.h>
 #include <linux/iommu.h>
+#include <linux/iopoll.h>
+#include <linux/ktime.h>
 #include <linux/module.h>
 #include <linux/mutex.h>
 #include <linux/notifier.h>
@@ -24,6 +26,7 @@
 #include <linux/pci.h>
 #include <linux/pm_runtime.h>
 #include <linux/slab.h>
+#include <linux/time64.h>
 #include <linux/types.h>
 #include <linux/uaccess.h>
 #include <linux/vgaarb.h>
@@ -575,6 +578,129 @@ static void vfio_pci_core_map_bars(struct
vfio_pci_core_device *vdev)
 	}
 }

+#define GB202_BAR_FIREWALL	0x2b8
+#define GB202_BAR_BLOCKED		BIT(24)
+#define GB202_FSP_STATUS		0xad00bc
+#define GB202_FSP_READY		0xff
+#define GB202_TIMEOUT_US		(5 * USEC_PER_SEC)
+
+/* Serialize with INTx; byte writes preserve concurrent COMMAND+1 updates. */
+static int vfio_pci_set_memory_decode(struct vfio_pci_core_device *vdev,
+				      bool enable, bool *was_enabled)
+{
+	unsigned long flags;
+	u16 command, readback;
+	u8 value;
+	int ret;
+
+	spin_lock_irqsave(&vdev->irqlock, flags);
+	ret = pci_read_config_word(vdev->pdev, PCI_COMMAND, &command);
+	if (ret)
+		goto out_unlock;
+	if (PCI_POSSIBLE_ERROR(command)) {
+		ret = -EIO;
+		goto out_unlock;
+	}
+
+	if (was_enabled)
+		*was_enabled = command & PCI_COMMAND_MEMORY;
+	value = enable ? command | PCI_COMMAND_MEMORY :
+			 command & ~PCI_COMMAND_MEMORY;
+	if (value == (u8)command)
+		goto out_unlock;
+
+	ret = pci_write_config_byte(vdev->pdev, PCI_COMMAND, value);
+	if (ret)
+		goto out_unlock;
+	ret = pci_read_config_word(vdev->pdev, PCI_COMMAND, &readback);
+	if (!ret && (PCI_POSSIBLE_ERROR(readback) || (u8)readback != value))
+		ret = -EIO;
+
+out_unlock:
+	spin_unlock_irqrestore(&vdev->irqlock, flags);
+	return pcibios_err_to_errno(ret);
+}
+
+/**
+ * vfio_pci_gb202_wait_ready - wait for GB202 firmware after reset
+ * @vdev: VFIO PCI device
+ *
+ * GB202 releases PCI reset before FSP initialization completes. Wait for BAR
+ * access first, then firmware readiness.
+ *
+ * Context: Sleepable. Called during initial device enable or with
+ * memory_lock held for write.
+ * Return: 0 for unaffected or ready devices, negative errno on failure.
+ */
+int vfio_pci_gb202_wait_ready(struct vfio_pci_core_device *vdev)
+{
+	struct pci_dev *pdev = vdev->pdev;
+	u32 firewall = ~0U, fsp_status = ~0U;
+	bool memory_was_enabled = true;
+	void __iomem *fsp_status_reg;
+	ktime_t deadline;
+	s64 timeout_us;
+	int ret, config_ret, restore_ret;
+
+	/* GB202 GeForce and RTX PRO GPUs use device IDs 0x2b80-0x2bbf. */
+	if (pdev->vendor != PCI_VENDOR_ID_NVIDIA || pdev->is_virtfn ||
+	    (pdev->class >> 16) != PCI_BASE_CLASS_DISPLAY ||
+	    (pdev->device & 0xffc0) != 0x2b80)
+		return 0;
+	if (pdev->current_state != PCI_D0 ||
+	    !(pci_resource_flags(pdev, 0) & IORESOURCE_MEM) ||
+	    (pci_resource_flags(pdev, 0) & IORESOURCE_UNSET) ||
+	    pci_resource_len(pdev, 0) < GB202_FSP_STATUS + sizeof(u32))
+		return -ENODEV;
+
+	deadline = ktime_add_us(ktime_get(), GB202_TIMEOUT_US);
+	ret = read_poll_timeout(pci_read_config_dword, config_ret,
+				config_ret || PCI_POSSIBLE_ERROR(firewall) ||
+				!(firewall & GB202_BAR_BLOCKED), USEC_PER_MSEC,
+				GB202_TIMEOUT_US, false, pdev,
+				GB202_BAR_FIREWALL, &firewall);
+	if (!ret && config_ret)
+		ret = pcibios_err_to_errno(config_ret);
+	if (!ret && PCI_POSSIBLE_ERROR(firewall))
+		ret = -EIO;
+	if (ret) {
+		pci_err(pdev, "BAR firewall not ready: %#x (%d)\n", firewall, ret);
+		return ret;
+	}
+
+	fsp_status_reg = pci_iomap_range(pdev, 0, GB202_FSP_STATUS, sizeof(u32));
+	if (!fsp_status_reg)
+		return -ENOMEM;
+
+	ret = vfio_pci_set_memory_decode(vdev, true, &memory_was_enabled);
+	if (ret) {
+		pci_err(pdev, "Cannot enable BAR0 access (%d)\n", ret);
+		goto out_restore;
+	}
+
+	/* A zero timeout would disable the polling deadline. */
+	timeout_us = max_t(s64, 1, ktime_us_delta(deadline, ktime_get()));
+	ret = readl_poll_timeout(fsp_status_reg, fsp_status,
+				 fsp_status == GB202_FSP_READY ||
+				 PCI_POSSIBLE_ERROR(fsp_status),
+				 USEC_PER_MSEC, timeout_us);
+	if (!ret && PCI_POSSIBLE_ERROR(fsp_status))
+		ret = -EIO;
+	if (ret)
+		pci_err(pdev, "FSP not ready: %#x (%d)\n", fsp_status, ret);
+
+out_restore:
+	if (!memory_was_enabled) {
+		restore_ret = vfio_pci_set_memory_decode(vdev, false, NULL);
+		if (restore_ret) {
+			pci_err(pdev, "Cannot restore memory decoding (%d)\n", restore_ret);
+			ret = restore_ret;
+		}
+	}
+	pci_iounmap(pdev, fsp_status_reg);
+	return ret;
+}
+
 int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 {
 	struct pci_dev *pdev = vdev->pdev;
@@ -601,6 +727,12 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 		goto out_disable_device;

 	vdev->reset_works = !ret;
+	if (!ret) {
+		ret = vfio_pci_gb202_wait_ready(vdev);
+		if (ret)
+			goto out_disable_device;
+	}
+
 	pci_save_state(pdev);
 	vdev->pci_saved_state = pci_store_saved_state(pdev);
 	if (!vdev->pci_saved_state)
@@ -1322,6 +1454,8 @@ static int vfio_pci_ioctl_reset(struct
vfio_pci_core_device *vdev,

 	vfio_pci_dma_buf_move(vdev, true);
 	ret = pci_try_reset_function(vdev->pdev);
+	if (!ret)
+		ret = vfio_pci_gb202_wait_ready(vdev);
 	if (__vfio_pci_memory_enabled(vdev))
 		vfio_pci_dma_buf_move(vdev, false);
 	up_write(&vdev->memory_lock);
@@ -2574,6 +2708,13 @@ static int vfio_pci_dev_set_hot_reset(struct
vfio_device_set *dev_set,
 		vfio_pci_set_power_state(vdev, PCI_D0);

 	ret = pci_reset_bus(pdev);
+	if (!ret) {
+		list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) {
+			ret = vfio_pci_gb202_wait_ready(vdev);
+			if (ret)
+				break;
+		}
+	}

 	vdev = list_last_entry(&dev_set->device_list,
 			       struct vfio_pci_core_device, vdev.dev_set_list);
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index 4e7162234..fcc7d1b53 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -68,6 +68,8 @@ void vfio_config_free(struct vfio_pci_core_device *vdev);
 int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev,
 			     pci_power_t state);

+int vfio_pci_gb202_wait_ready(struct vfio_pci_core_device *vdev);
+
 void vfio_pci_zap_and_down_write_memory_lock(struct
vfio_pci_core_device *vdev);
 u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev);
 void vfio_pci_memory_unlock_and_restore(struct vfio_pci_core_device *vdev,

base-commit: 97d73c8251226cf27cc6656b4c542869bb8c69fa
-- 
2.56.1



On Sat, 14 Nov 2026 02:48:53 +0000, "Gustavo A." <gustavogyn@gmail.com> wrote:
> On the affected system, GPU passthrough worked with the CachyOS 7.2.10
> host kernel. After upgrading the host to 7.3, the guest still booted but
> its physical display stayed grey. The exact triggering change is unknown,
> so this patch has no Fixes tag.
>
> PCI reset returns successfully while the Firmware Security Processor (FSP)
> is still initializing. Restricting BAR1 to 4 KiB mappings masks the problem.
> Waiting for FSP completion after reset restores normal output with stock
> KVM and verified 1 GiB BAR1 mappings.
>
> The RFC question is where this wait belongs. It currently follows
> successful resets in the VFIO enable, reset ioctl, PCIe/AF FLR and hot-reset
> paths. Would a PCI-core readiness check after state restoration be more
> appropriate? A device-specific reset method runs before BAR and state
> restoration and does not cover all bus resets. The existing reset_done
> callback cannot propagate errors.
>
> Validation:
> - Local host and Linux guest: 7.3.4-1-cachyos, QEMU 11.1.3, GPU 10de:2b85.
>   The physical display works with startup tracing disabled. The submitted
>   readiness code adds only an IORESOURCE_UNSET guard to this tested code.
>   A focused check fails without the guard and passes with it.
> - Rebased onto mainline 97d73c825122 (7.4-rc2). Full W=1 builds pass with
>   GCC 16.2.1 (VFIO modules, PM enabled) and Clang 23.1.5 (VFIO built in,
>   PM disabled). No VFIO compiler warnings. Unrelated warnings remain.
> - 79 helper checks and 51 reset-path checks pass using extracted source
>   with modeled PCI, time and locking. The unpatched reset-path baseline
>   fails 22 checks. The hot-reset model covers completion and unwind for
>   three devices, excluding ownership validation and initial locking.
> - Strict checkpatch and kernel-doc pass. Sparse 0.6.4 reports the same 17
>   diagnostics on patched and unpatched code, with no added findings.
>
> The rebased mainline kernel has not been booted. Runtime lockdep, D3/resume
> and repeated hardware reset cycles remain untested. Hardware testing covers
> only the device listed above.
>
> Related FSP-ordering work:
> https://lists.openwall.net/linux-kernel/2026/06/15/1402
>
> Gustavo A. (1):
>   vfio/pci: Wait for GB202 firmware initialization after reset

Re: [RFC PATCH 0/1] vfio/pci: Wait for GB202 firmware after reset

[Jason Gunthorpe]

On Sat, Nov 14, 2026 at 02:48:53AM +0000, Gustavo A. wrote:

> PCI reset returns successfully while the Firmware Security Processor (FSP)
> is still initializing. Restricting BAR1 to 4 KiB mappings masks the problem.
> Waiting for FSP completion after reset restores normal output with stock
> KVM and verified 1 GiB BAR1 mappings.

That doesn't really make any sense, you are probably hitting a race
condition somewhere since 1G/4k changes the timing. And possibly
because your new kernel versions boot VFIO faster than older ones.

IIRC there was some VBIOS bug in old versions that caused issues with
vfio.

VFIO should definately not be doing things like this.

Jason