[PATCH] PCI: Avoid FLR for NVIDIA 5090 GPU

[PATCH] PCI: Avoid FLR for NVIDIA 5090 GPU

[yuan.gao]

When passing through the NVIDIA 5090 GPU to a vm, there is a certain
probability of encountering an flr timeout during vm shutdown, which
subsequently leads to a soft lock of the host cpu.

As described in this post
(https://forum.level1techs.com/t/do-your-rtx-5090-or-general-rtx-50-series-has-reset-bug-in-vm-passthrough/228549).

And in dmesg:

 [401106.011979] vfio-pci 0000:d8:00.0: not ready 1023ms after FLR; waiting
 [401108.700074] vfio-pci 0000:d8:00.0: not ready 2047ms after FLR; waiting
 [401112.412204] vfio-pci 0000:d8:00.0: not ready 4095ms after FLR; waiting
 [401118.620399] vfio-pci 0000:d8:00.0: not ready 8191ms after FLR; waiting
 [401128.860788] vfio-pci 0000:d8:00.0: not ready 16383ms after FLR; waiting
 [401147.293518] vfio-pci 0000:d8:00.0: not ready 32767ms after FLR; waiting
 [401185.694859] vfio-pci 0000:d8:00.0: not ready 65535ms after FLR; giving up
 [401195.372583] vfio-pci 0000:38:00.2: Relaying device request to user (#0)

 [401208.274941] watchdog: BUG: soft lockup - CPU#11 stuck for 21s! [CPU 22/KVM:30337]

 [401209.887848] CPU: 11 PID: 30337 Comm: CPU 22/KVM Kdump: loaded Not tainted
 [401209.887854] RIP: 0010:pci_mmcfg_read+0xaa/0xd0

 [401209.887866] Call Trace:
 [401209.887872]  pci_bus_read_config_dword+0x43/0x70
 [401209.b887876]  pci_find_next_ext_capability.part.20+0x65/0xc0
 [401209.887879]  pci_restore_state.part.39+0x6d/0x3f0
 [401209.887883]  vfio_pci_disable+0x22b/0x4d0 [vfio_pci]
 [401209.887886]  ? __dentry_kill+0x118/0x160
 [401209.887888]  vfio_pci_release+0x5a/0xb0 [vfio_pci]
 [401209.887891]  vfio_device_fops_release+0x18/0x30 [vfio]
 [401209.887894]  __fput+0x98/0x240
 [401209.887897]  task_work_run+0x6a/0xa0
 [401209.887899]  do_exit+0x375/0xb10
 [401209.887900]  do_group_exit+0x3a/0xa0
 [401209.887902]  get_signal+0x140/0x7d0
 [401209.887906]  arch_do_signal+0x2c/0x260
 [401209.887909]  exit_to_user_mode_prepare+0xc0/0x120
 [401209.887912]  syscall_exit_to_user_mode+0x27/0x180
 [401209.887915]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

The flr seems to have some issues on the NVIDIA 5090 GPU,
so I’ve added flr-related quirks for these devices.

And with this patch in place, the host kernel doesn't exhibit these
problems. The vm starts up and works as expected with the passed-through
NVIDIA 5090 GPU.

Signed-off-by: yuan.gao <yuan.gao@ucloud.cn>
---
 drivers/pci/quirks.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 48946cca4be72..71f833f3e2d84 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -5618,6 +5618,9 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x7901, quirk_no_flr);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1502, quirk_no_flr);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1503, quirk_no_flr);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_MEDIATEK, 0x0616, quirk_no_flr);
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x2b85, quirk_no_flr);
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x2b87, quirk_no_flr);
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x2b8c, quirk_no_flr);
 
 /* FLR may cause the SolidRun SNET DPU (rev 0x1) to hang */
 static void quirk_no_flr_snet(struct pci_dev *dev)
-- 
2.32.0

Re: [PATCH] PCI: Avoid FLR for NVIDIA 5090 GPU

[Bjorn Helgaas]

[+cc Alex, Jason]

On Thu, Apr 16, 2026 at 03:07:06PM +0800, yuan.gao wrote:
> When passing through the NVIDIA 5090 GPU to a vm, there is a certain
> probability of encountering an flr timeout during vm shutdown, which
> subsequently leads to a soft lock of the host cpu.

If possible, would like confirmation of device erratum from Nvidia.
If there's no known erratum, there might be something wrong in the
Linux FLR and wait.

> As described in this post
> (https://forum.level1techs.com/t/do-your-rtx-5090-or-general-rtx-50-series-has-reset-bug-in-vm-passthrough/228549).
> 
> And in dmesg:
> 
>  [401106.011979] vfio-pci 0000:d8:00.0: not ready 1023ms after FLR; waiting
>  [401108.700074] vfio-pci 0000:d8:00.0: not ready 2047ms after FLR; waiting
>  [401112.412204] vfio-pci 0000:d8:00.0: not ready 4095ms after FLR; waiting
>  [401118.620399] vfio-pci 0000:d8:00.0: not ready 8191ms after FLR; waiting
>  [401128.860788] vfio-pci 0000:d8:00.0: not ready 16383ms after FLR; waiting
>  [401147.293518] vfio-pci 0000:d8:00.0: not ready 32767ms after FLR; waiting
>  [401185.694859] vfio-pci 0000:d8:00.0: not ready 65535ms after FLR; giving up
>  [401195.372583] vfio-pci 0000:38:00.2: Relaying device request to user (#0)
> 
>  [401208.274941] watchdog: BUG: soft lockup - CPU#11 stuck for 21s! [CPU 22/KVM:30337]
> 
>  [401209.887848] CPU: 11 PID: 30337 Comm: CPU 22/KVM Kdump: loaded Not tainted
>  [401209.887854] RIP: 0010:pci_mmcfg_read+0xaa/0xd0
> 
>  [401209.887866] Call Trace:
>  [401209.887872]  pci_bus_read_config_dword+0x43/0x70
>  [401209.b887876]  pci_find_next_ext_capability.part.20+0x65/0xc0
>  [401209.887879]  pci_restore_state.part.39+0x6d/0x3f0
>  [401209.887883]  vfio_pci_disable+0x22b/0x4d0 [vfio_pci]
>  [401209.887886]  ? __dentry_kill+0x118/0x160
>  [401209.887888]  vfio_pci_release+0x5a/0xb0 [vfio_pci]
>  [401209.887891]  vfio_device_fops_release+0x18/0x30 [vfio]
>  [401209.887894]  __fput+0x98/0x240
>  [401209.887897]  task_work_run+0x6a/0xa0
>  [401209.887899]  do_exit+0x375/0xb10
>  [401209.887900]  do_group_exit+0x3a/0xa0
>  [401209.887902]  get_signal+0x140/0x7d0
>  [401209.887906]  arch_do_signal+0x2c/0x260
>  [401209.887909]  exit_to_user_mode_prepare+0xc0/0x120
>  [401209.887912]  syscall_exit_to_user_mode+0x27/0x180
>  [401209.887915]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> 
> The flr seems to have some issues on the NVIDIA 5090 GPU,
> so I’ve added flr-related quirks for these devices.
> 
> And with this patch in place, the host kernel doesn't exhibit these
> problems. The vm starts up and works as expected with the passed-through
> NVIDIA 5090 GPU.
> 
> Signed-off-by: yuan.gao <yuan.gao@ucloud.cn>
> ---
>  drivers/pci/quirks.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index 48946cca4be72..71f833f3e2d84 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -5618,6 +5618,9 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x7901, quirk_no_flr);
>  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1502, quirk_no_flr);
>  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1503, quirk_no_flr);
>  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_MEDIATEK, 0x0616, quirk_no_flr);
> +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x2b85, quirk_no_flr);
> +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x2b87, quirk_no_flr);
> +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x2b8c, quirk_no_flr);
>  
>  /* FLR may cause the SolidRun SNET DPU (rev 0x1) to hang */
>  static void quirk_no_flr_snet(struct pci_dev *dev)
> -- 
> 2.32.0
> 

Re: [PATCH] PCI: Avoid FLR for NVIDIA 5090 GPU

[Jason Gunthorpe]

On Thu, Apr 16, 2026 at 12:31:00PM -0500, Bjorn Helgaas wrote:
> On Thu, Apr 16, 2026 at 03:07:06PM +0800, yuan.gao wrote:
> > When passing through the NVIDIA 5090 GPU to a vm, there is a certain
> > probability of encountering an flr timeout during vm shutdown, which
> > subsequently leads to a soft lock of the host cpu.
> 
> If possible, would like confirmation of device erratum from Nvidia.
> If there's no known erratum, there might be something wrong in the
> Linux FLR and wait.

I asked and was told there is a known device firmware defect that
causes this.

So blanket disabling FLR without detecting good and bad FW is not a
good idea.

I suggest Yuan try to use an NVIDIA support channel to try to resolve
the issue with their card..

Jason

Re: [PATCH] PCI: Avoid FLR for NVIDIA 5090 GPU

[yuan.gao]

On Thu, Apr 16, 2026 at 07:37:54PM -0300, Jason Gunthorpe wrote:
> On Thu, Apr 16, 2026 at 12:31:00PM -0500, Bjorn Helgaas wrote:
> > On Thu, Apr 16, 2026 at 03:07:06PM +0800, yuan.gao wrote:
> > > When passing through the NVIDIA 5090 GPU to a vm, there is a certain
> > > probability of encountering an flr timeout during vm shutdown, which
> > > subsequently leads to a soft lock of the host cpu.
> > 
> > If possible, would like confirmation of device erratum from Nvidia.
> > If there's no known erratum, there might be something wrong in the
> > Linux FLR and wait.
> 
> I asked and was told there is a known device firmware defect that
> causes this.
> 
> So blanket disabling FLR without detecting good and bad FW is not a
> good idea.
> 
> I suggest Yuan try to use an NVIDIA support channel to try to resolve
> the issue with their card..
> 
> Jason
Got it, thanks.

Cheers,
Yuan Gao