Re: PCI resource allocation mismatch with BIOS

[Alex Williamson <alex.williamson@redhat.com>]

On Mon, 28 Nov 2022 14:39:32 -0600
Bjorn Helgaas <helgaas@kernel.org> wrote:

> [+cc Alex]
> 
> Hi Mika,
> 
> On Mon, Nov 28, 2022 at 01:14:14PM +0200, Mika Westerberg wrote:
> > Hi Bjorn,
> > 
> > There is another PCI resource allocation issue with some Intel GPUs but
> > probably applies to other similar devices as well. This is something
> > encountered in data centers where they trigger reset (secondary bus
> > reset) to the GPUs if there is hang or similar detected. Basically they
> > do something like:
> > 
> >   1. Unbind the graphics driver(s) through sysfs.
> >   2. Remove the PCIe devices under the root port or the PCIe switch
> >      upstream port through sysfs (echo 1 > ../remove).
> >   3. Trigger reset through config space or use the sysfs reset attribute.
> >   4. Run rescan on the root bus (echo 1 > /sys/bus/pci/rescan) 
> > 
> > Expectation is to see the devices come back in the same way prior the
> > reset but what actually happens is that the Linux PCI resource
> > allocation fails to allocate space for some of the resources. In this
> > case it is the IOV BARs.
> > 
> > BIOS allocates resources for all these at boot time but after the rescan
> > Linux tries to re-allocate them but since the allocation algorithm is
> > more "consuming" some of the BARs do not fit to the available resource
> > space.  
> 
> Thanks for the report!  Definitely sounds like an issue.  I doubt that
> I'll have time to work on it myself in the near future.
> 
> Is the "remove" before the reset actually necessary?  If we could
> avoid the removal, maybe the config space save/restore we already do
> around reset would avoid the issue?

Agreed.  Is this convoluted removal process being used to force a SBR,
versus a FLR or PM reset that might otherwise be used by twiddling the
reset attribute of the GPU directly?  If so, the reset_method attribute
can be used to force a bus reset and perform all the state save/restore
handling to avoid reallocating BARs.  A reset from the upstream switch
port would only be necessary if you have some reason to also reset the
switch downstream ports.  Thanks,

Alex