Re: [PATCH v2 1/2] PCI: Take multifunction devices into account when distributing resources

[Jonathan Cameron <Jonathan.Cameron@Huawei.com>]

On Tue, 22 Nov 2022 08:42:58 +0200
Mika Westerberg <mika.westerberg@linux.intel.com> wrote:

> Hi,
> 
> On Mon, Nov 21, 2022 at 04:45:48PM -0600, Bjorn Helgaas wrote:
> > IIUC, the summary is this:
> > 
> >   00:02.0 bridge window [mem 0x10000000-0x102fffff] to [bus 01-02]
> >   01:02.0 bridge window [mem 0x10000000-0x100fffff] to [bus 02]
> >   01:03.0 NIC BAR [mem 0x10200000-0x1021ffff]
> >   01:04.0 NIC BAR [mem 0x10220000-0x1023ffff]
> >   02:05.0 NIC BAR [mem 0x10080000-0x1009ffff]
> > 
> > and it's the same with and without the current patch.
> > 
> > Are all these assignments done by BIOS, or did Linux update them?  
> 
> > Did we exercise the same "distribute available resources" path as in
> > the PCIe case?  I expect we *should*, because there really shouldn't
> > be any PCI vs PCIe differences in how resources are handled.  This is
> > why I'm not comfortable with assumptions here that depend on PCIe.
> > 
> > I can't tell from Jonathan's PCIe case whether we got a working config
> > from BIOS or not because our logging of bridge windows is kind of
> > poor.  
> 
> This is ARM64 so there is no "BIOS" involved (something similar though).

It's EDK2 in my tests  - so very similar to other arch.
Possible to boot without though and rely on DT, but various things don't
work yet...

> 
> It is the same "system" that Jonathan used where the regression happened
> with the multifunction PCIe configuration with the exception that I'm
> now using PCI devices instead of PCIe as you asked.
> 
> I'm not 100% sure if the all the same code paths are used here, though.
> 

I wondered if it was possibly to do with fairly minimal handling of pci-pxb
(the weird root bridge) in EDK2, so tried the obvious of hanging your PCI
test below one of those rather than directly below a normal bridge.
Despite shuffling things around into configurations
I thought might trigger the problem, it all seems fine.

Note that I can't currently test the pxb-pcie configurations without EDK2
as arm-virt doesn't provide the relevant DT for those root bridges yet
(it's on my todo list as it's a prereq for getting the QEMU CXL ARM64
emulation upstream)

So no guarantees as I don't understand fully why PCI is ending up
with different handling.

From liberal distribution of printk()s it seems that for PCI bridges
pci_bridge_resources_not_assigned() is false, but for PCI express
example it is true.  My first instinct is quirk of the EDK2 handling? 
I'll have a dig, but I'm not an expert in EDK2 at all, so may not get
to the bottom of this.

Ultimately it seems that when the OS takes over the prefetchable memory
resources are not configured for the PCIe case, but are for the PCI case.
So we aren't currently walking the new code for PCI.

Jonathan