Re: [PATCH] pci: Avoid FLR for AMD FCH AHCI adapters

[Bjorn Helgaas <helgaas@kernel.org>]

On Mon, Jan 30, 2023 at 03:46:06PM +0000, Niklas Cassel wrote:
> On Mon, Jan 30, 2023 at 09:21:11AM -0600, Bjorn Helgaas wrote:
> > On Sat, Jan 28, 2023 at 10:39:51AM +0900, Damien Le Moal wrote:
> > > PCI passthrough to VMs does not work with AMD FCH AHCI adapters: the
> > > guest OS fails to correctly probe devices attached to the controller due
> > > to FIS communication failures. 
> > 
> > What does a FIS communication failure look like?  Can we include a
> > line or two of dmesg output here to help users find this fix?
> 
> It looks like this:
> 
> [   22.402368] ata4: softreset failed (1st FIS failed)
> [   32.417855] ata4: softreset failed (1st FIS failed)
> [   67.441641] ata4: softreset failed (1st FIS failed)
> [   67.453227] ata4: limiting SATA link speed to 3.0 Gbps
> [   72.661738] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
> [   78.121263] ata4.00: qc timeout after 5000 msecs (cmd 0xec)
> [   78.134413] ata4.00: failed to IDENTIFY (I/O error, err_mask=0x4)
> 
> Basically, we can read and write MMIO registers in the AHCI HBA,
> but the communication between the AHCI HBA and the ATA device does
> not work properly.
> 
> (Because the AHCI HBA did not get reset when binding/unbinding the
> device.)
> 
> The exact same kernel, using the same generic AHCI driver within the VM,
> can communicate perfectly fine when using e.g. an Intel AHCI HBA.
> 
> (With both the AMD and Intel AHCI HBAs being bound to the vfio-pci driver
> in the host.)
> 
> We can send a v2 with the above dmesg output.

Don't bother, I added the above and applied this to pci/virtualization
for v6.2, thanks!

> > AMD folks: Can you confirm/deny that this is a hardware erratum in
> > this device?  Do you know of any other devices that need a similar
> > workaround?
> > 
> > > Forcing the "bus" reset method before
> > > unbinding & binding the adapter to the vfio-pci driver solves this
> > > issue. I.e.:
> > > 
> > > echo "bus" > /sys/bus/pci/devices/<ID>/reset_method
> > > 
> > > gives a working guest OS, thus indicating that the default flr reset
> > > method is defective, resulting in the adapter not being reset correctly.
> > > 
> > > This patch applies the no_flr quirk to AMD FCH AHCI devices to
> > > permanently solve this issue.
> > > 
> > > Reported-by: Niklas Cassel <niklas.cassel@wdc.com>
> > > Cc: stable@vger.kernel.org
> > > Signed-off-by: Damien Le Moal <damien.lemoal@opensource.wdc.com>
> > > ---
> > >  drivers/pci/quirks.c | 1 +
> > >  1 file changed, 1 insertion(+)
> > > 
> > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > > index 285acc4aaccc..20ac67d59034 100644
> > > --- a/drivers/pci/quirks.c
> > > +++ b/drivers/pci/quirks.c
> > > @@ -5340,6 +5340,7 @@ static void quirk_no_flr(struct pci_dev *dev)
> > >  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1487, quirk_no_flr);
> > >  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x148c, quirk_no_flr);
> > >  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x149c, quirk_no_flr);
> > > +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x7901, quirk_no_flr);
> > >  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1502, quirk_no_flr);
> > >  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1503, quirk_no_flr);
> > >  
> > > -- 
> > > 2.39.1
> > >