[PATCH] PCI: Avoid bus reset on NVIDIA GB202 and GB203 GPUs

[PATCH] PCI: Avoid bus reset on NVIDIA GB202 and GB203 GPUs

[mark-ship-it]

NVIDIA GB202 (GeForce RTX 5090, RTX PRO 6000) and GB203 (GeForce RTX
5080/5070 Ti/5070) GPUs train their link at x16 at boot, but after a
Secondary Bus Reset the link comes back at x8 and stays there until the
next cold boot.

The GPU's HD Audio function (e.g. 10de:22e9 on GB203) has no reset method,
so when both functions are assigned to a VM, vfio-pci falls back to a bus
reset at VM start and again at release, and the GPU then runs at half
width. Assigning only function 0 avoids the bus reset because the GPU
itself supports FLR, which is why the problem is usually reported as
"passing the audio function breaks x16".

Observed on an RTX 5080 (10de:2c02) behind an AMD Root Port, kernel
7.0.14, Proxmox VE host with a Windows 11 Pro guest:

  boot:                         LnkSta: Speed 32GT/s, Width x16
  echo 1 > /sys/bus/pci/devices/<root port>/reset_subordinate
                                LnkSta: Speed 32GT/s, Width x8 (downgraded)

No VM is involved in that reproduction. Stepping the link speed through
bwctrl (Gen5 -> Gen4 -> Gen5) retrains the link and the speed follows,
but the width stays at x8. With a VM started frozen (-S), the link is
already at x8 before any guest code has run, so the guest driver is not
involved.

With this patch applied, reset_method for the GPU is "flr" only, both
functions can be assigned, and the link stays at x16 Gen5 across VM
start, shutdown and restart, with the Windows guest driver working.

The same symptom has been reported with an RTX 5090 (10de:2b85) on an
AMD X870E board, where passing only function 0 was the only thing that
kept x16.

Link: https://forum.proxmox.com/threads/asus-x870e-pcie-x8-bifurcation-bug-with-rtx-5090-vfio-passthrough.178461/

Signed-off-by: mark-ship-it <williamwoodstock79@gmail.com>
---
 drivers/pci/quirks.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 1aa87a3ef..a4adcc3e7 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -3769,10 +3769,17 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, 0x22D0, quirk_no_bus_reset);
 /*
  * Some NVIDIA GPU devices do not work with bus reset, SBR needs to be
  * prevented for those affected devices.
+ *
+ * GB202 (0x2b80-0x2bbf) and GB203 (0x2c00-0x2c7f) GPUs train at x16 at
+ * boot, but after a Secondary Bus Reset the link comes back at x8 and
+ * stays there until the next cold boot; a link retrain restores the speed
+ * but not the width.  Function Level Reset does not have this problem.
  */
 static void quirk_nvidia_no_bus_reset(struct pci_dev *dev)
 {
-	if ((dev->device & 0xffc0) == 0x2340)
+	if ((dev->device & 0xffc0) == 0x2340 ||
+	    (dev->device & 0xffc0) == 0x2b80 ||
+	    (dev->device & 0xff80) == 0x2c00)
 		quirk_no_bus_reset(dev);
 }
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
-- 
2.55.0

Re: [PATCH] PCI: Avoid bus reset on NVIDIA GB202 and GB203 GPUs

[sashiko-bot]

> NVIDIA GB202 (GeForce RTX 5090, RTX PRO 6000) and GB203 (GeForce RTX
> 5080/5070 Ti/5070) GPUs train their link at x16 at boot, but after a
> Secondary Bus Reset the link comes back at x8 and stays there until the
> next cold boot.
> 
> The GPU's HD Audio function (e.g. 10de:22e9 on GB203) has no reset method,
> so when both functions are assigned to a VM, vfio-pci falls back to a bus
> reset at VM start and again at release, and the GPU then runs at half
> width. Assigning only function 0 avoids the bus reset because the GPU
> itself supports FLR, which is why the problem is usually reported as
> "passing the audio function breaks x16".
> 
> Observed on an RTX 5080 (10de:2c02) behind an AMD Root Port, kernel
> 7.0.14, Proxmox VE host with a Windows 11 Pro guest:
> 
> [ ... ]
> kept x16.
> 
> Link: https://forum.proxmox.com/threads/asus-x870e-pcie-x8-bifurcation-bug-with-rtx-5090-vfio-passthrough.178461/
> 
> Signed-off-by: mark-ship-it <williamwoodstock79@gmail.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261018201727.96924-1-williamwoodstock79@gmail.com?part=1