Re: ReBAR over Thunderbolt

["Ilpo Järvinen" <ilpo.jarvinen@linux.intel.com>]

[-- Attachment #1: Type: text/plain, Size: 4484 bytes --]

On Mon, 30 Mar 2026, Cristian Cocos wrote:
> On Mon, 2026-03-30 at 16:23 +0300, Ilpo Järvinen wrote:
> > > [  292.204936] pcieport 0000:02:00.0: bridge window [mem
> > > 0x6000000000-
> > > 0x60f04fffff 64bit pref]: was not released (still contains assigned
> > > resources)
> > > [  292.204937] pcieport 0000:00:07.0: bridge window [mem
> > > 0x6000000000-
> > > 0x60ffffffff 64bit pref]: was not released (still contains assigned
> > > resources)
> > 
> > These 2 lines indicate where the reason likely lies. BAR resize
> > attempted to release these windows but there are some sibling
> > resources that prevent the resize from succeeding. I cannot see what
> > those resources are from these excerpts.
> > 
> > Basically, the resize walks upwards in the PCI hierarchy and tries to
> > free any bridge window it encounters in order to allow them to be
> > sized freely (enlarged). The device whose BAR is being resized has
> > its resources released prior to the resize commencing, but any other
> > device under any of those bridge windows are not and they pin their
> > bridge windows in-place (it won't be possible to release them in the
> > general case as some of those devices may be in use at this point).
> > 
> > It may be possible to workaround this problem by manually removing
> > the sibling devices that pin the relevant bridge windows, performing
> > resize through sysfs manually, and then rescanning (but with GPUs it
> > may be problematic if it's the primary GPU).
> 
> Thank you. Your suggestion does indeed work! The resize succeeds after
> removing the three empty TB downstream ports that were pinning the
> bridge windows. The BAR went from 256 MB to 16 GB.

Great.

> However, it appears
> that the re-probe after rebinding hits an amdgpu driver bug: Failed to
> create device file mem_info_preempt_used (EEXIST from stale sysfs),
> which cascades to sw_init of IP block <gmc_v12_0> failed -17 → NULL
> deref in amdgpu_discovery_fini.

This problem is beyond PCI so I'm not of much help with it.

I suggest reporting this to amdgpu people (see the MAINTAINERS file) 
with logs, they may be able to help.

> Be that as it may, I do see a workaround happening via a udev rule or
> systemd service that removes the three empty ports immediately after TB
> hotplug and before amdgpu binds. However, I can't help but feel that it
> is the kernel that should handle this sort of situations without any
> user side involvement. The problem (and please take this as a
> suggestion) seems to be that the kernel's PCI resource allocator
> speculatively assigns those three empty (see topology excerpt below)
> ports large prefetchable bridge windows "just in case" a device appears
> behind them later. That's what eats up the 4 GB root port window. If
> the allocator didn't hand out speculative windows to empty ports — or,
> even better, if it checked ReBAR capabilities on the one port that does
> have a device and gave it priority — the resize would work without
> removing anything.
> 
> Laptop (Raptor Lake-P SoC)
>   └─ 00:07.0  Integrated TB4 root port          ← laptop
>        └─ 02:00.0  8086:5786 Switch Upstream     ← Razer Core X V2
>             ├─ 03:00.0  Switch Downstream → GPU path (04→05→06:00.0)
>             ├─ 03:01.0  Switch Downstream → empty
>             ├─ 03:02.0  Switch Downstream → empty
>             └─ 03:03.0  Switch Downstream → empty
> 
> Again, please take this suggestion for what it's worth (i.e. my $.02),
> and many thanks for your attention to this matter.

I am already looking into resizable BAR aware resource fitting, hopefully 
we'll get there in this year. However, it's a very complex change due to 
fallbacks that have to be put into place and has various prerequisites 
that I've been slowly upstreaming (and fixing the issues they've caused).

I've also considered not assigning bridge windows that do not have any 
real resources underneath them (but still consider them in the sizing 
decisions). It's not trivial to realize though, the assignment logic 
heavily depends on those placeholders resources to satisfy alignment 
constraints (so I've been thinking of walking through the assigned 
resources post-assignment algorithm and releasing to empty windows 
at that point to avoid violating the alignment constraints without extra 
state tracking logic).


-- 
 i.