Re: [PATCH] PCI: Mark NVIDIA T4 GPUs to avoid bus reset

[Wu Zongyong <wuzongyong@linux.alibaba.com>]

On Wed, Aug 09, 2023 at 06:05:18PM -0500, Bjorn Helgaas wrote:
> On Mon, Apr 10, 2023 at 08:34:11PM +0800, Wu Zongyong wrote:
> > NVIDIA T4 GPUs do not work with SBR. This problem is found when the T4
> > card is direct attached to a Root Port only. So avoid bus reset by
> > marking T4 GPUs PCI_DEV_FLAGS_NO_BUS_RESET.
> > 
> > Fixes: 4c207e7121fa ("PCI: Mark some NVIDIA GPUs to avoid bus reset")
> > Signed-off-by: Wu Zongyong <wuzongyong@linux.alibaba.com>
> 
> Applied to pci/virtualization for v6.6, thanks!

I talk about the issue with NVIDIA, and they think the issue is probably related
the pci link instead of the T4 GPU card.

I will try to describe the issue I met in detail.

The T4 card which is direct attached to a Root Port and I rebind it to
vfio-pci driver. Then I try to use to call some vfio-related api and the
ioctl VFIO_GROUP_GET_DEVICE_FD failed.

The stack is (base on kernel v5.10):
    vfio_group_fops_unl_ioctl
         vfio_group_get_device_fd
            vfio_pci_open
                vfio_pci_enable // return value is -19
                    pci_try_reset_function
                        __pci_reset_function_locked

After the __pci_reset_function_locked(), the dmesg shows:
   [12207494.508467] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Link Down
   [12207494.508535] vfio-pci 0000:40:00.0: No device request channel registered, blocked until released by user
   [12207494.518426] pci 0000:40:00.0: Removing from iommu group 84
   [12207495.532365] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Card present
   [12207495.532367] pcieport 0000:3f:00.0: pciehp: Slot(5-1): Link Up

NVIDIA people thinks this root port is not going through this reset logic and getting the
link down/hot plug interrupts[1].

Can you revert the patch I sent and maybe we should dig it deeply.

> 
> > ---
> >  drivers/pci/quirks.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > index 44cab813bf95..be86b93b9e38 100644
> > --- a/drivers/pci/quirks.c
> > +++ b/drivers/pci/quirks.c
> > @@ -3618,7 +3618,7 @@ static void quirk_no_bus_reset(struct pci_dev *dev)
> >   */
> >  static void quirk_nvidia_no_bus_reset(struct pci_dev *dev)
> >  {
> > -	if ((dev->device & 0xffc0) == 0x2340)
> > +	if ((dev->device & 0xffc0) == 0x2340 || dev->device == 0x1eb8)
> >  		quirk_no_bus_reset(dev);
> >  }
> >  DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
> > -- 
> > 2.34.3
> >