Re: [PATCH] PCI: Fix resizable bar fails due to bridge memory region

Maarten Lankhorst
2026-04-24T17:58:54Z

Hey,

Den 2026-04-24 kl. 19:42, skrev Ilpo Järvinen:
> On Fri, 24 Apr 2026, Maarten Lankhorst wrote:
>> Den 2026-04-24 kl. 17:04, skrev Ilpo Järvinen:
>>> On Fri, 24 Apr 2026, Maarten Lankhorst wrote:
>>>
>>>> I encountered a problem that I have on my system, where I cannot resize
>>>> the bar because one of the bridges has a
>>>
>>> You're missing words from here. But I can guess you've that extra BAR on
>>> in-card the bridge.
>>>
>>>> If I take a look at the topology, the GPU shares the memory region with a bridge,
>>>>
>>>> +-[0000:64]-+-00.0-[65-68]----00.0-[66-68]--+-01.0-[67]----00.0
>>>>
>>>> The specific bridge likely causing a failure is:
>>>>
>>>> 65:00.0 PCI bridge: Intel Corporation Device e2ff (rev 01) (prog-if 00 [Normal decode])
>>>> Flags: bus master, fast devsel, latency 0, IRQ 32, IOMMU group 1
>>>> Memory at 382400000000 (64-bit, prefetchable) [size=8M]
>>>> ....
>>>>
>>>> Which causes upstream bridge 64:00.0 initially to allocate the region
>>>> [38fe0000000-38ff0000000) for the GPU, and [382ff0000000..382ff07fffff]
>>>> for the bridge device.
>>>>
>>>> Bridge 64 is big enough for 1 BMG with a 32 GB bar and the second 8 MB allocation:
>>>> pci_bus 0000:64: resource 9 [mem 0x382000000000-0x382fffffffff window] (64GB window)
>>>>
>>>> The reason for failure is that bridge 65 has a 8 MB memory region assigned,
>>>
>>>> and previously it was ignored when reallocating.
>>>
>>> What does this mean? I don't think it was ever ignored while
>>> reallocating??
>>>
>>> Note that kernel has become much more verbose in explaining why things
>>> fail so perhaps the added message is confusing you (they won't appear in
>>> the old kernels).
>>>
>>>> Failing case:
>>>> xe 0000:67:00.0: [drm] Attempting to resize bar from 256MiB -> 16384MiB
>>>> xe 0000:67:00.0: BAR 2 [mem 0x382fe0000000-0x382fefffffff 64bit pref]: releasing
>>>> pcieport 0000:66:01.0: bridge window [mem 0x382fe0000000-0x382fefffffff 64bit pref]: releasing
>>>> pcieport 0000:65:00.0: bridge window [mem 0x382fe0000000-0x382fefffffff 64bit pref]: releasing
>>>> pcieport 0000:64:00.0: bridge window [mem 0x382fe0000000-0x382ff07fffff 64bit pref]: was not released (still contains assigned resources)
>>>> pcieport 0000:65:00.0: bridge window [mem size 0x400000000 64bit pref]: can't assign; no space
>>>> pcieport 0000:65:00.0: bridge window [mem size 0x400000000 64bit pref]: failed to assign
>>>> pcieport 0000:65:00.0: bridge window [mem size 0x400000000 64bit pref]: can't assign; no space
>>>> pcieport 0000:65:00.0: bridge window [mem size 0x400000000 64bit pref]: failed to assign
>>>> pcieport 0000:66:01.0: bridge window [mem size 0x400000000 64bit pref]: can't assign; no space
>>>> pcieport 0000:66:01.0: bridge window [mem size 0x400000000 64bit pref]: failed to assign
>>>> pcieport 0000:66:01.0: bridge window [mem size 0x400000000 64bit pref]: can't assign; no space
>>>> pcieport 0000:66:01.0: bridge window [mem size 0x400000000 64bit pref]: failed to assign
>>>> xe 0000:67:00.0: BAR 2 [mem size 0x400000000 64bit pref]: can't assign; no space
>>>> xe 0000:67:00.0: BAR 2 [mem size 0x400000000 64bit pref]: failed to assign
>>>> xe 0000:67:00.0: BAR 2 [mem size 0x400000000 64bit pref]: can't assign; no space
>>>> xe 0000:67:00.0: BAR 2 [mem size 0x400000000 64bit pref]: failed to assign
>>>> pcieport 0000:64:00.0: PCI bridge to [bus 65-68]
>>>> pcieport 0000:64:00.0: bridge window [mem 0xd7000000-0xd83fffff]
>>>> pcieport 0000:64:00.0: bridge window [mem 0x382fe0000000-0x382ff07fffff 64bit pref]
>>>> pcieport 0000:65:00.0: PCI bridge to [bus 66-68]
>>>> pcieport 0000:65:00.0: bridge window [mem 0xd7000000-0xd83fffff]
>>>> pcieport 0000:65:00.0: bridge window [mem 0x382fe0000000-0x382fefffffff 64bit pref]
>>>> pcieport 0000:66:01.0: PCI bridge to [bus 67]
>>>> pcieport 0000:66:01.0: bridge window [mem 0xd7000000-0xd81fffff]
>>>> pcieport 0000:66:01.0: bridge window [mem 0x382fe0000000-0x382fefffffff 64bit pref]
>>>>
>>>> Working with the patch below:
>>>> xe 0000:67:00.0: [drm] Attempting to resize bar from 256MiB -> 16384MiB
>>>> xe 0000:67:00.0: BAR 2 [mem 0x382fe0000000-0x382fefffffff 64bit pref]: releasing
>>>> pcieport 0000:66:01.0: bridge window [mem 0x382fe0000000-0x382fefffffff 64bit pref]: releasing
>>>> pcieport 0000:65:00.0: bridge window [mem 0x382fe0000000-0x382fefffffff 64bit pref]: releasing
>>>> pcieport 0000:65:00.0: BAR 0 [mem 0x382ff0000000-0x382ff07fffff 64bit pref]: releasing
>>>> pcieport 0000:64:00.0: bridge window [mem 0x382fe0000000-0x382ff07fffff 64bit pref]: releasing
>>>> pcieport 0000:64:00.0: bridge window [mem 0x382000000000-0x3824007fffff 64bit pref]: assigned
>>>> pcieport 0000:65:00.0: bridge window [mem 0x382000000000-0x3823ffffffff 64bit pref]: assigned
>>>> pcieport 0000:65:00.0: BAR 0 [mem 0x382400000000-0x3824007fffff 64bit pref]: assigned
>>>> pcieport 0000:66:01.0: bridge window [mem 0x382000000000-0x3823ffffffff 64bit pref]: assigned
>>>> xe 0000:67:00.0: BAR 2 [mem 0x382000000000-0x3823ffffffff 64bit pref]: assigned
>>>> pcieport 0000:64:00.0: PCI bridge to [bus 65-68]
>>>> pcieport 0000:64:00.0: bridge window [mem 0xd7000000-0xd83fffff]
>>>> pcieport 0000:64:00.0: bridge window [mem 0x382000000000-0x3824007fffff 64bit pref]
>>>> pcieport 0000:65:00.0: PCI bridge to [bus 66-68]
>>>> pcieport 0000:65:00.0: bridge window [mem 0xd7000000-0xd83fffff]
>>>> pcieport 0000:65:00.0: bridge window [mem 0x382000000000-0x3823ffffffff 64bit pref]
>>>> pcieport 0000:65:00.0: PCI bridge to [bus 66-68]
>>>> pcieport 0000:65:00.0: bridge window [mem 0xd7000000-0xd83fffff]
>>>> pcieport 0000:65:00.0: bridge window [mem 0x382000000000-0x3823ffffffff 64bit pref]
>>>> pcieport 0000:66:01.0: PCI bridge to [bus 67]
>>>> pcieport 0000:66:01.0: bridge window [mem 0xd7000000-0xd81fffff]
>>>> pcieport 0000:66:01.0: bridge window [mem 0x382000000000-0x3823ffffffff 64bit pref]
>>>> xe 0000:67:00.0: [drm] BAR2 resized to 16384MiB
>>>>
>>>>
>>>> Signed-off-by: Maarten Lankhorst <dev@lankhorst.se>
>>>> ---
>>>> I'm not 100% this is the correct fix, I don't know why the bridge itself has
>>>> a memory region, why the kernel allocates it and when it's supposed to
>>>> be used. Not a PCI expert. :-)
>>>
>>> I don't know why it is there either. Nothing in the portdrv really uses it
>>> for anything. There is patchset somewhere lying around which adds a quirk
>>> that releases the "extra" BAR.
>>
>> Thank you, I've taken a look at that patch series. It looks like patch 1/2
>> would help a lot.
>
> I should one day find time to finish that series, Bjorn didn't like how
> the code didn't "disable" BAR.
>
> I did some (unset) work towards placing the BARs outside the bridge
> window but my algorithm was quite basic (just found min & max window
> addresses and put BAR outside of that range but then I thought maybe I
> should add the possiblity for placing the BAR into the gap in the middle to
> allow placement of a large BAR in cases where 32-bit mmio blocks low-end +
> 64-bit mmio range blocking the top of the address space.
>
>> I'm wondering if it will completely fix the issue, it is still possible
>> that in the same config I have 2 identical cards, where resizing might
>> work, but when GPU1's VRAM BAR is already bound, it might be unable
>> to resize GPU2's VRAM BAR for the same reason as this patch.
>
> It is always possible to find cases where a sibling pins the shared bridge
> window in place.
>
>> What would be the best way to handle this case?
>
> The best way is to size them into the largest possible sizes right from
> the beginning. But when doing that, nothing must break (introducing
> regressions is not okay) so it kernel must be able to graciously fallback
> to smaller sizes when there isn't enough space available for the largest
> size.
>
> What makes it complicated is that sizing and assignment are done very much
> separately, so retrying with a smaller size is complicated. I'm working
> toward this kind of solution but there are various things that have to be
> fixed first.

How feasible would it be to add a quirk to resize the bar (or reserve enough
space for max) very early before any allocations are done?

I doubt any user of resizable bar is going to hotplug their GPU's.

Kind regards,
~Maarten Lankhorst