Re: [PATCH] PCI: Reserve prefetchable window headroom for Resizable BARs

["Ilpo Järvinen" <ilpo.jarvinen@linux.intel.com>]

[-- Attachment #1: Type: text/plain, Size: 4006 bytes --]

On Mon, 31 Aug 2026, Geramy Loveless wrote:
> On 8/31/26 9:58 AM, Ilpo Järvinen wrote:
> > On Mon, 31 Aug 2026, Christian König wrote:
> >
> >> On 8/28/26 23:37, Geramy Loveless wrote:
> >>> Firmware typically sizes prefetchable bridge windows for the boot-time
> >>> BAR size. Behind a fixed (non-hotplug) PCIe switch fabric, there is then
> >>> no room for a driver to grow a Resizable BAR afterwards: every window
> >>> from the leaf up to the root is sized for the small BAR, so
> >>> pci_resize_resource() fails with -ENOSPC. 
> >>>
> >>> Furthermore, a small prefetchable BAR (e.g., a 2 MiB doorbell) sharing a 
> >>> bridge's single prefetchable window with a much larger one (e.g., a 32 GiB 
> >>> VRAM BAR) pushes the required window size past the large BAR's alignment. 
> >>> Because bridge windows round up to a power of two, this forces a massive 
> >>> alignment waste (e.g., 32 GiB + 2 MiB rounds up to a 64 GiB window, 
> >>> wasting ~32 GiB per GPU).
> >>>
> >>> This patch solves both issues to enable ReBAR on cascaded switch fabrics:
> >>>
> >>> 1. Reserve Headroom: 
> >>> Reserve prefetchable window headroom for the maximum size of each 
> >>> downstream Resizable BAR during the bridge sizing pass. The device BAR 
> >>> and hardware ReBAR are left at their boot size to prevent tearing down 
> >>> firmware-loaded state (e.g., AMD R9700 PSP) before the driver binds.
> >> Yeah that was suggested before but that is clearly not something you can 
> >> do in common code.
> >>
> >> The ReBAR fields often doesn't reflect the actual needed space but 
> >> rather the maximum the HW address logic can resolve.
> >>
> >> So what you end up with is allocating multiple TiB for a window which 
> >> just needs few GiB, sometimes even completely overflowing the 64bit 
> >> address space made available by the root complex.
> 
> Yeah I could imagine that would be bad, hence I tried to compact the 
> 64-bit space as well. As far as I understand PCI/PCIe standards its 
> making this patch difficult. 

No compacting will save you in some cases. There are setups with many GPUs 
and with VF BARs, the space requirement to fit full-sized ReBARs is way 
beyond what the root bus resource can hold. Where's your fallback for such 
cases? What are the failure modes of that fallback?

> Christian I'm not sure if you could share these reference documents but 
> that probably would be a better start for me to look at before I update 
> the patch or make changes, also I need to wait for a review on the last 
> patch I submitted too, so that leaves me with some time to review some 
> standards if you know specifically where to look, if not thats fine too.
> 
> > Yes. A naive approach to (only) go to the max ReBAR allows just doesn't 
> > work well enough to be usable in general case.
>
> I did not know the max would report above the amount is actually needed 
> because of vendors decisions in the cards, that's interesting. 

Not just that, using too large size has ripple effects on other resources 
in the system, and those failures are hard to correct (your code doesn't 
even try to do that).

> > A general algorithm to pick the largests fitting sizes for ReBARs is still 
> > under works, a few steps away still, and more if I discover more 
> > challenges I need address before I can take that final step.
>
> So then I would assume instead of doing the "hack" i'm having to use I 
> would change it to your new algorithm?

Assuming you meant we apply this kind of naive approach first and later 
changing to the new algorithm, the problem with that approach is this
"hack" must not break working setups either, but it surely would. If it 
does break things, the change will just end up reverted. And that end 
result is same as not doing the "hack" at all.

It's not enough it doesn't break your setup. It generally must not break 
any setup, which is unfortunately very very high bar.


-- 
 i.