Re: Are back to back PCIe BAR allocations supported by Linux?

[Bjorn Helgaas <helgaas@kernel.org>]

On Thu, Apr 01, 2021 at 02:03:34PM +0200, Christian König wrote:
> Hello everyone,
> 
> we recently had a bug report of a system which works fine when a PCIe
> hotplug device is connected on boot, but fails to initialize if those device
> are disconnected and then reconnected again.
> 
> During investigation I've found that Linux isn't able to assign the BARs of
> the device correctly while reconnecting. The problem seems to be that the
> Linux PCI code doesn't seem to use back to back BAR allocations.
> 
> Now what's back to back BAR allocation? Let's assume you have two devices
> with a 256MiB BAR and a 2MiB BAR each behind a common upstream bridge.
> 
> The configuration Linux seems to use is the following:
> Device A - 256MiB BAR
> Device A -     2MiB BAR
> Padding     254MiB
> Device B - 256MiB BAR
> Device B -     2MIB BAR
> 
> With padding this results in at least 770MiB address space requirement for
> the common upstream bridge, with alignment this is probably more like 1GiB.
> 
> The BIOS on the other hand seems to be capable of configuring the BARs like
> this:
> 
> Device A - 256MiB BAR
> Device A -     2MiB BAR
> Padding     252MiB
> Device B -     2MIB BAR
> Device B - 256MiB BAR
> 
> The result is that you only need 768MiB address space for the upstream
> bridge which then perfectly fits into what is assigned for hotplug here.
> 
> Is that already supported by the Linux PCIe code? If yes then how? I've
> tried to read a bit into the BAR allocation code, but it is kind of hard to
> understand.

I'm sorry that I don't have a ready answer.  The Linux resource
allocation code is indeed very difficult to understand.  I'm quite
sure it is incapable of doing optimal assignments.

If the BIOS has already assigned working values, Linux generally
doesn't change them.  But when you hot-remove and then reconnect a
device, the BIOS assignments are lost, of course, and it's not
surprising that Linux doesn't recreate them exactly.  I certainly
agree that if it worked before, Linux *should* be able to make it work
again.

Bjorn