Re: [PATCH] PCI: Release unused bridge resources during resize

["Ilpo Järvinen" <ilpo.jarvinen@linux.intel.com>]

[-- Attachment #1: Type: text/plain, Size: 7813 bytes --]

On Thu, 16 May 2024, Alex Williamson wrote:

> On Mon, 13 May 2024 16:46:09 +0300 (EEST)
> Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> wrote:
> 
> > On Tue, 7 May 2024, Alex Williamson wrote:
> > 
> > > Resizing BARs can be blocked when a device in the bridge hierarchy
> > > itself consumes resources from the resized range.  This scenario is
> > > common with Intel Arc DG2 GPUs where the following is a typical
> > > topology:
> > > 
> > >  +-[0000:5d]-+-00.0-[5e-61]----00.0-[5f-61]--+-01.0-[60]----00.0  Intel Corporation DG2 [Arc A380]
> > >                                              \-04.0-[61]----00.0  Intel Corporation DG2 Audio Controller
> > > 
> > > Here the system BIOS has provided a large 64bit, prefetchable window:
> > > 
> > > pci_bus 0000:5d: root bus resource [mem 0xb000000000-0xbfffffffff window]
> > > 
> > > But only a small portion is programmed into the root port aperture:
> > > 
> > > pci 0000:5d:00.0:   bridge window [mem 0xbfe0000000-0xbff07fffff 64bit pref]
> > > 
> > > The upstream port then provides the following aperture:
> > > 
> > > pci 0000:5e:00.0:   bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
> > > 
> > > With the missing range found to be consumed by the switch port itself:
> > > 
> > > pci 0000:5e:00.0: BAR 0 [mem 0xbff0000000-0xbff07fffff 64bit pref]
> > > 
> > > The downstream port above the GPU provides the same aperture as upstream:
> > > 
> > > pci 0000:5f:01.0:   bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
> > > 
> > > Which is entirely consumed by the GPU:
> > > 
> > > pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]
> > > 
> > > In summary, iomem reports the following:
> > > 
> > > b000000000-bfffffffff : PCI Bus 0000:5d
> > >   bfe0000000-bff07fffff : PCI Bus 0000:5e
> > >     bfe0000000-bfefffffff : PCI Bus 0000:5f
> > >       bfe0000000-bfefffffff : PCI Bus 0000:60
> > >         bfe0000000-bfefffffff : 0000:60:00.0
> > >     bff0000000-bff07fffff : 0000:5e:00.0
> > > 
> > > The GPU at 0000:60:00.0 supports a Resizable BAR:
> > > 
> > > 	Capabilities: [420 v1] Physical Resizable BAR
> > > 		BAR 2: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB
> > > 
> > > However when attempting a resize we get -ENOSPC:
> > > 
> > > pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> > > pcieport 0000:5f:01.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> > > pcieport 0000:5e:00.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> > > pcieport 0000:5e:00.0: bridge window [mem size 0x200000000 64bit pref]: can't assign; no space
> > > pcieport 0000:5e:00.0: bridge window [mem size 0x200000000 64bit pref]: failed to assign
> > > pcieport 0000:5f:01.0: bridge window [mem size 0x200000000 64bit pref]: can't assign; no space
> > > pcieport 0000:5f:01.0: bridge window [mem size 0x200000000 64bit pref]: failed to assign
> > > pci 0000:60:00.0: BAR 2 [mem size 0x200000000 64bit pref]: can't assign; no space
> > > pci 0000:60:00.0: BAR 2 [mem size 0x200000000 64bit pref]: failed to assign
> > > pcieport 0000:5d:00.0: PCI bridge to [bus 5e-61]
> > > pcieport 0000:5d:00.0:   bridge window [mem 0xb9000000-0xba0fffff]
> > > pcieport 0000:5d:00.0:   bridge window [mem 0xbfe0000000-0xbff07fffff 64bit pref]
> > > pcieport 0000:5e:00.0: PCI bridge to [bus 5f-61]
> > > pcieport 0000:5e:00.0:   bridge window [mem 0xb9000000-0xba0fffff]
> > > pcieport 0000:5e:00.0:   bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
> > > pcieport 0000:5f:01.0: PCI bridge to [bus 60]
> > > pcieport 0000:5f:01.0:   bridge window [mem 0xb9000000-0xb9ffffff]
> > > pcieport 0000:5f:01.0:   bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
> > > pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]: assigned
> > > 
> > > In this example we need to resize all the way up to the root port
> > > aperture, but we refuse to change the root port aperture while resources
> > > are allocated for the upstream port BAR.
> > > 
> > > The solution proposed here builds on the idea in commit 91fa127794ac
> > > ("PCI: Expose PCIe Resizable BAR support via sysfs") where the BAR can
> > > be resized while there is no driver attached.  In this case, when there
> > > is no driver bound to the upstream switch port we'll release resources
> > > of the bridge which match the reallocation.  Therefore we can achieve
> > > the below successful resize operation by unbinding 0000:5e:00.0 from the
> > > pcieport driver before invoking the resource2_resize interface on the
> > > GPU at 0000:60:00.0.
> > > 
> > > pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> > > pcieport 0000:5f:01.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> > > pci 0000:5e:00.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> > > pci 0000:5e:00.0: BAR 0 [mem 0xbff0000000-0xbff07fffff 64bit pref]: releasing
> > > pcieport 0000:5d:00.0: bridge window [mem 0xbfe0000000-0xbff07fffff 64bit pref]: releasing
> > > pcieport 0000:5d:00.0: bridge window [mem 0xb000000000-0xb2ffffffff 64bit pref]: assigned
> > > pci 0000:5e:00.0: bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]: assigned
> > > pci 0000:5e:00.0: BAR 0 [mem 0xb200000000-0xb2007fffff 64bit pref]: assigned
> > > pcieport 0000:5f:01.0: bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]: assigned
> > > pci 0000:60:00.0: BAR 2 [mem 0xb000000000-0xb1ffffffff 64bit pref]: assigned
> > > pci 0000:5e:00.0: PCI bridge to [bus 5f-61]
> > > pci 0000:5e:00.0:   bridge window [mem 0xb9000000-0xba0fffff]
> > > pci 0000:5e:00.0:   bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]
> > > pcieport 0000:5d:00.0: PCI bridge to [bus 5e-61]
> > > pcieport 0000:5d:00.0:   bridge window [mem 0xb9000000-0xba0fffff]
> > > pcieport 0000:5d:00.0:   bridge window [mem 0xb000000000-0xb2ffffffff 64bit pref]
> > > pci 0000:5e:00.0: PCI bridge to [bus 5f-61]
> > > pci 0000:5e:00.0:   bridge window [mem 0xb9000000-0xba0fffff]
> > > pci 0000:5e:00.0:   bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]
> > > pcieport 0000:5f:01.0: PCI bridge to [bus 60]
> > > pcieport 0000:5f:01.0:   bridge window [mem 0xb9000000-0xb9ffffff]
> > > pcieport 0000:5f:01.0:   bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]
> > > 
> > > Signed-off-by: Alex Williamson <alex.williamson@redhat.com>  
> > 
> > Yes. Looks another case where an already assigned resource prevents some 
> > operation from succeeding.
> > 
> > > diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> > > index 909e6a7c3cc3..15fc8e4e84c9 100644
> > > --- a/drivers/pci/setup-bus.c
> > > +++ b/drivers/pci/setup-bus.c
> > > @@ -2226,6 +2226,26 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
> > >  }
> > >  EXPORT_SYMBOL_GPL(pci_assign_unassigned_bridge_resources);
> > >  
> > > +static void pci_release_resource_type(struct pci_dev *pdev, unsigned long type)
> > > +{
> > > +	int i;
> > > +
> > > +	if (!device_trylock(&pdev->dev))
> > > +		return;
> > > +
> > > +	if (pdev->dev.driver)  
> > 
> > Isn't portdrv bound to bridges so how does this ends up working?
> 
> The user will need to unbind the bridge from the driver, just like
> they'd need to unbind the endpoint from a driver to resize a BAR
> through sysfs.  I'm not sure how else to avoid races with drivers
> requesting resources other than to assert that there is no driver for
> the device.  Do you have an alternative suggestion?  Thanks,

Okay, understood. It just wasn't immediately obvious there was 
this additional requirement related to unbinding the portdrv.

-- 
 i.