On Mon, 31 Aug 2026, Geramy Loveless wrote:

> 
> On 8/31/26 9:31 AM, Ilpo Järvinen wrote:
> > On Fri, 28 Aug 2026, Geramy Loveless wrote:
> >
> >> Firmware typically sizes prefetchable bridge windows for the boot-time
> >> BAR size. Behind a fixed (non-hotplug) PCIe switch fabric, there is then
> >> no room for a driver to grow a Resizable BAR afterwards: every window
> >> from the leaf up to the root is sized for the small BAR, so
> >> pci_resize_resource() fails with -ENOSPC. 
> >>
> >> Furthermore, a small prefetchable BAR (e.g., a 2 MiB doorbell) sharing a 
> >> bridge's single prefetchable window with a much larger one (e.g., a 32 GiB 
> >> VRAM BAR) pushes the required window size past the large BAR's alignment. 
> >> Because bridge windows round up to a power of two,
> > Do they??? In which kernel version I must ask??
> I'm sorry this was in reference to a prior patch I made, **ignore**
> >
> >> this forces a massive 
> >> alignment waste (e.g., 32 GiB + 2 MiB rounds up to a 64 GiB window, 
> >> wasting ~32 GiB per GPU).
> > The largest I've seen is 32GB + half of that. And that's not with the 
> > latest kernel.
> 
> This is in reference to when there are two R9700s on one switch basically.
> As far as my understanding currently how it works is that when there are two GPUs they both have a BAR0, BAR2, and BAR5 the BAR5 goes onto the 32-bit window.
> Bar 0 and 2 don't go onto the second window by default as they could overfill it of course, bar 0 would not fit period, and 2 might which is the case I took. 
> This allows BAR 0 on (GPU 0) in leaf 0 on window 0 to align as a multiple of 32G like it I guess has to? This is how I understand it, so if R9700 has 32G on, bar0, leaf 0, window 0, and share the same window as GPU 1 then we have to pad the space between GPU0 BARs on window 0 by 32G, meaning BAR2 really is also using 32G, its not possible or at least right now its not possible to move BAR2 to the end of all the allocations it doesnt fit the standards? Keep in mind this is not my specialty here I am trying my best to understand and solve the issue at hand.

With the empty space in between, your 64GB calculation would indeed be 
true, but that wasn't what you said because you made an inaccurate claim 
about window sizes being power of two in case of 32G+2M.

(I suppose we're generally in agreement here but just ended up discussion 
on finer details and the interpretation of those details.)

> >> This patch solves both issues to enable ReBAR on cascaded switch fabrics:
> >>
> >> 1. Reserve Headroom: 
> >> Reserve prefetchable window headroom for the maximum size of each 
> >> downstream Resizable BAR during the bridge sizing pass. The device BAR 
> >> and hardware ReBAR are left at their boot size to prevent tearing down 
> >> firmware-loaded state (e.g., AMD R9700 PSP) before the driver binds.
> > ... reorganizing ...
> >
> >> Fallback / Alignment logic:
> >> If the 32-bit space is exhausted, small BARs remain in the prefetchable 
> >> window. The alignment logic gracefully handles this by rounding the bridge 
> >> window up to the next power of two (e.g., growing to 64 GiB to fit a 32 GiB 
> >> BAR + BAR2) to maintain PCIe compatibility, ensuring sibling GPUs land in 
> >> properly aligned slots.
> >>
> >> Reserving is safe by default: __assign_resources_sorted() satisfies all
> >> required resources first. Use pci=no_rebar_reserve and pci=no_bar_demote 
> >> to restore previous behaviours.
> >
> > And how does this fallback, if the largest size doesn't fit? What are the 
> > failure modes?
>
> So if the smaller bars under 16M don't fit in the 32-bit addressable 
> space the fallback mode is to do as I said above to your prior comment, 
> basically put. 
> Don't move the 2M BAR2 over to the 32-bit addressable space / window and 
> instead fallback to aligning the prior window by the size of the GPUs 
> memory space as I understand it, when you have gpu0 and gpu1 both 
> requiring 32GB their address has to be a multiple of their allocation 
> size? meaning we get 32G for BAR0 and 32G for BAR2 even though BAR2 only 
> needs 2M because its not aligned to allow GPU1 to be on a aligned space?

I'm more referring to the case where the big ReBARs cannot be assigned as 
space runs out or some other BAR fails to assign because ReBAR took "too 
much", and the numerous ripple effects from that. When that happens in 
some system for something important breaking something, you'd have to 
solve that problem too or this change gets reverted...

> > It's correct that the sizing decision should be done in pbus_size_mem() 
> > but it must not result in breaking things which a naive approach like 
> > this will surely do.
> >
> > Without proper fallbacks done, there will be too much breakage it's a 
> > showstopper for a naive patch like this.
> >
> > Basically, what we'd want to do is try with the largest size, then reduce 
> > it one step at a time (because it's 2^n -> 2^(n-1) even one step saves 
> > quite much so one might not need to reduce that many steps). But also, in 
> > some IOV enabled cases the oversubscription of the space is very 
> > substantial and you'd still want to maximize the BAR sizes to what is 
> > allowed by the available space.
> >
> > Fallback is hard to implement, because sizing and assignment are quite far 
> > from each other codewise and there are no structures to carry information 
> > over to the next retry phase.

...And that's why there should be proper fallback so that failing one 
assignment somewhere middle will not result in failure in the end.

Basically, my own plan is for the algorithm to try smaller sizes until all 
things fit. And all that retry logic is far from simple.

-- 
 i.