Re: USB4v2 BAR resizing problems - hotplug gpu via razor tb5 dock - Intel Barlow Ridge

["Ilpo Järvinen" <ilpo.jarvinen@linux.intel.com>]

[-- Attachment #1: Type: text/plain, Size: 13106 bytes --]

On Tue, 7 Apr 2026, Geramy Loveless wrote:

> I am hoping someone here can help me; I'm currently having some issues
> with a Razor thunderbolt 5 / USB4v2 Dock connected to a Halo Strix or
> AMD Ryzen AI 395+ Max machine.
> Specifically a Minisforum MS-S1 with a Host controller: Intel
> 8086:5780 — Barlow Ridge Host 80G (rev 84) and the Dock hub: Intel
> 8086:5786 — Barlow Ridge Hub 80G (rev 85) These are the JHL9580 (host)
> and JHL9586 (hub) — Intel's Barlow Ridge TB5 80Gbps silicon.
> The specific problem I'm having, which I think is the largest issue
> besides power management causing havoc which i can fix with kernel
> parameters but the one I cannot "fix" granted not kernel params are
> not a fix is that on thunderbolt driver it will not expand its bar
> size to the parent size.
> so when setting hpmmiopresize to 64G it gets split between the two
> usb4v2 ports giving me 32G on each port and then the memory space is
> not aligned so when amdgpu tries to resize it cant. I would think the
> ideal situation at least in my head and not being a linux kernel
> expert is that the thunderbolt driver should request the pcie device
> max memory size and then create the bar at the correct sizes or at the
> least let me resize it which might be a difficult and bad path to let
> the "user" resize it.
> 
> Current kernel image is: 6.18.7-061807-generic

Hi,

There have been significant improvements to the kernel's resource fitting 
strategy since that kernel, some of the most recent improvements are not 
yet even in Linus' tree but only in pci/resource branch slated for 7.1.

There's one fix in particular related to how hotplug memory distribution 
is handled that might be relevant here but I cannot say for sure because 
it only print a log line with dynamic debugging enabled 
(CONFIG_DYNAMIC_DEBUG=y + dyndbg="file drivers/pci/*.c +p" on kernel 
command line).

> Custom Parameters:
>  pcie_port_pm=off pcie_aspm=off
> pci=hpmemsize=2G,realloc,hpmmioprefsize=64G amdgpu.dc=0

The effect of these size parameterd may get nullified because of that bug
I mentioned.

> This was required to get the kernel to stop powering down the
> thunderbolt link on the amd gfx1201 device because it goes into a low
> power state when there is no display connected.
> For my application thats okay, i'm using it for compute based workloads anyways.
> 
> Thank you for taking a look at this here are some log snippets have a great day!

> Here is my amdgpu issue with full logs
> https://gitlab.freedesktop.org/drm/amd/-/work_items/4978#note_3339645

This is not a full log but starts in the middle.

You should pass log_buf_len=16M to kernel command line capture it from 
start.

It would also help if you capture /proc/iomem dump.

> Here are more recent logs after running the kernel parameters.
> 
> System: MinisForum MS-S1 (AMD Ryzen AI 395+ Max / Strix Halo) + Razer
> Core X V2 via Intel Barlow Ridge TB5 (JHL9580/JHL9586)
> GPU: AMD Radeon AI PRO R9700 [1002:7551] (RDNA 4 / gfx1201, 32GB GDDR6)
> Kernel: 6.18.7 (Ubuntu mainline)
> Cmdline: pcie_port_pm=off pcie_aspm=off
> pci=hpmemsize=2G,realloc,hpmmioprefsize=64G amdgpu.dc=0
> 
> --- Boot-time TB5 bridge layout (host-side, before tunnel) ---
> pci 0000:65:00.0: [8086:5780] type 01 class 0x060400 PCIe Switch Upstream Port
> pci 0000:65:00.0: PCI bridge to [bus 66-bc]
> pci 0000:65:00.0:   bridge window [mem 0x8880000000-0x98800fffff 64bit pref]
> pci 0000:66:01.0: [8086:5780] type 01 class 0x060400 PCIe Switch Downstream Port
> pci 0000:66:01.0: PCI bridge to [bus 68-91]
> pci 0000:66:01.0:   bridge window [mem 0x9080000000-0x987fffffff 64bit pref]
> pci 0000:66:03.0: [8086:5780] type 01 class 0x060400 PCIe Switch Downstream Port
> pci 0000:66:03.0: PCI bridge to [bus 93-bc]
> pci 0000:66:03.0:   bridge window [mem 0x8880000000-0x907fffffff 64bit pref]
> 
> --- Dock hot-plug enumeration (after TB tunnel up + PCI rescan) ---
> pci 0000:93:00.0: [8086:5786] type 01 class 0x060400 PCIe Switch Upstream Port
> pci 0000:93:00.0: PCI bridge to [bus 00]
> pci 0000:93:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
> pci 0000:93:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> pci 0000:94:00.0: [8086:5786] type 01 class 0x060400 PCIe Switch Downstream Port
> pci 0000:94:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
> 
> --- Kernel resource allocation: cannot fit rebar + hotplug reserves ---
> pcieport 0000:66:03.0: Assigned bridge window [mem
> 0x8880000000-0x907fffffff 64bit pref] to [bus 93-bc] cannot fit
> 0x1000000000 required for 0000:94:01.0 bridging to [bus 98-a4]
> pcieport 0000:66:03.0: Assigned bridge window [mem
> 0x8880000000-0x907fffffff 64bit pref] to [bus 93-bc] cannot fit
> 0x1000000000 required for 0000:94:02.0 bridging to [bus a5-b1]
> pcieport 0000:66:03.0: Assigned bridge window [mem
> 0x8880000000-0x907fffffff 64bit pref] to [bus 93-bc] cannot fit
> 0x1000000000 required for 0000:94:03.0 bridging to [bus b2-bc]
> pcieport 0000:66:03.0: Assigned bridge window [mem
> 0x8880000000-0x907fffffff 64bit pref] to [bus 93-bc] cannot fit
> 0x3018000000 required for 0000:93:00.0 bridging to [bus 94-bc]
> pci 0000:93:00.0: bridge window [mem 0x04000000-0x1fffffff 64bit pref]
> to [bus 94-bc] requires relaxed alignment rules
> pci 0000:93:00.0: bridge window [mem size 0x3018000000 64bit pref]:
> can't assign; no space
> pci 0000:93:00.0: bridge window [mem size 0x3018000000 64bit pref]:
> failed to assign
> pci 0000:93:00.0: bridge window [mem 0x8880000000-0x889bffffff 64bit
> pref]: assigned
> 
> --- GPU BAR enumeration ---
> pci 0000:97:00.0: [1002:7551] type 00 class 0x030000 PCIe Legacy Endpoint
> pci 0000:97:00.0: BAR 0 [mem 0x00000000-0x0fffffff 64bit pref]
> pci 0000:97:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 2.5
> GT/s PCIe x4 link at 0000:66:03.0 (capable of 504.112 Gb/s with 32.0
> GT/s PCIe x16 link)
> pci 0000:97:00.0: BAR 0 [mem 0x8880000000-0x888fffffff 64bit pref]: assigned
> pci 0000:97:00.0: BAR 4 [io  size 0x0100]: can't assign; no space
> 
> --- amdgpu rebar attempt and failure ---
> amdgpu 0000:97:00.0: amdgpu: PCIE atomic ops is not supported
> amdgpu 0000:97:00.0: BAR 2 [mem 0x8890000000-0x88901fffff 64bit pref]: releasing
> amdgpu 0000:97:00.0: BAR 0 [mem 0x8880000000-0x888fffffff 64bit pref]: releasing

Here you filtered the explanation which immediately follows:

[   58.188911] pcieport 0000:96:00.0: bridge window [mem 0x8880000000-0x8897ffffff 64bit pref]: releasing
[   58.188913] pcieport 0000:95:00.0: bridge window [mem 0x8880000000-0x8897ffffff 64bit pref]: releasing
[   58.188913] pcieport 0000:94:00.0: bridge window [mem 0x8880000000-0x8897ffffff 64bit pref]: releasing
[   58.188914] pcieport 0000:93:00.0: bridge window [mem 0x8880000000-0x889bffffff 64bit pref]: was not released (still contains assigned resources)
[   58.188915] pcieport 0000:66:03.0: bridge window [mem 0x8880000000-0x907fffffff 64bit pref]: was not released (still contains assigned resources)
[   58.188916] pcieport 0000:65:00.0: bridge window [mem 0x8880000000-0x98800fffff 64bit pref]: was not released (still contains assigned resources)
[   58.188916] pcieport 0000:00:03.2: bridge window [mem 0x8880000000-0x98800fffff 64bit pref]: was not released (still contains assigned resources)

The last 4 lines tell those upstream bridge windows are pinned by other 
(sibling) resources (effectively, it means it's too late to resize them at 
this point). When attempting BAR resize, kernel release only the resource 
of the device and then attemptes to release all bridge windows upstream 
of the device to allow them to be changed (enlarged). If there are sibling 
devices, their resources will prevent the shared bridge window from being 
released, effectively preventing enlarging the bridge window.

The resize might succeedeed, if you manually remove those sibling devices 
first, and then attempt the resize through sysfs (and rescan the bring 
back the removed devices).

I cannot say for sure what the pinning devices/resources are as your log 
is partial but you can check them from /proc/iomem yourself.

-- 
 i.

> amdgpu 0000:97:00.0: BAR 0 [mem size 0x800000000 64bit pref]: can't
> assign; no space
> amdgpu 0000:97:00.0: BAR 0 [mem size 0x800000000 64bit pref]: failed to assign
> amdgpu 0000:97:00.0: BAR 2 [mem size 0x00200000 64bit pref]: can't
> assign; no space
> amdgpu 0000:97:00.0: BAR 2 [mem size 0x00200000 64bit pref]: failed to assign
> amdgpu 0000:97:00.0: amdgpu: Not enough PCI address space for a large BAR.
> amdgpu 0000:97:00.0: BAR 0 [mem 0x8880000000-0x888fffffff 64bit pref]: assigned
> amdgpu 0000:97:00.0: amdgpu: VRAM: 32624M 0x0000008000000000 -
> 0x00000087F6FFFFFF (32624M used)
> 
> System: MinisForum MS-S1 (AMD Ryzen AI 395+ Max / Strix Halo) + Razer
> Core X V2 via Intel Barlow Ridge TB5 (JHL9580/JHL9586)
> GPU: AMD Radeon AI PRO R9700 [1002:7551] (RDNA 4 / gfx1201, 32GB GDDR6)
> Kernel: 6.18.7 (Ubuntu mainline)
> Cmdline: pcie_port_pm=off pcie_aspm=off
> pci=hpmemsize=2G,realloc,hpmmioprefsize=64G amdgpu.dc=0
> 
> --- Boot-time TB5 bridge layout (host-side, before tunnel) ---
> pci 0000:65:00.0: [8086:5780] type 01 class 0x060400 PCIe Switch Upstream Port
> pci 0000:65:00.0: PCI bridge to [bus 66-bc]
> pci 0000:65:00.0:   bridge window [mem 0x8880000000-0x98800fffff 64bit pref]
> pci 0000:66:01.0: [8086:5780] type 01 class 0x060400 PCIe Switch Downstream Port
> pci 0000:66:01.0: PCI bridge to [bus 68-91]
> pci 0000:66:01.0:   bridge window [mem 0x9080000000-0x987fffffff 64bit pref]
> pci 0000:66:03.0: [8086:5780] type 01 class 0x060400 PCIe Switch Downstream Port
> pci 0000:66:03.0: PCI bridge to [bus 93-bc]
> pci 0000:66:03.0:   bridge window [mem 0x8880000000-0x907fffffff 64bit pref]
> 
> --- Dock hot-plug enumeration (after TB tunnel up + PCI rescan) ---
> pci 0000:93:00.0: [8086:5786] type 01 class 0x060400 PCIe Switch Upstream Port
> pci 0000:93:00.0: PCI bridge to [bus 00]
> pci 0000:93:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
> pci 0000:93:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> pci 0000:94:00.0: [8086:5786] type 01 class 0x060400 PCIe Switch Downstream Port
> pci 0000:94:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
> 
> --- Kernel resource allocation: cannot fit rebar + hotplug reserves ---
> pcieport 0000:66:03.0: Assigned bridge window [mem
> 0x8880000000-0x907fffffff 64bit pref] to [bus 93-bc] cannot fit
> 0x1000000000 required for 0000:94:01.0 bridging to [bus 98-a4]
> pcieport 0000:66:03.0: Assigned bridge window [mem
> 0x8880000000-0x907fffffff 64bit pref] to [bus 93-bc] cannot fit
> 0x1000000000 required for 0000:94:02.0 bridging to [bus a5-b1]
> pcieport 0000:66:03.0: Assigned bridge window [mem
> 0x8880000000-0x907fffffff 64bit pref] to [bus 93-bc] cannot fit
> 0x1000000000 required for 0000:94:03.0 bridging to [bus b2-bc]
> pcieport 0000:66:03.0: Assigned bridge window [mem
> 0x8880000000-0x907fffffff 64bit pref] to [bus 93-bc] cannot fit
> 0x3018000000 required for 0000:93:00.0 bridging to [bus 94-bc]
> pci 0000:93:00.0: bridge window [mem 0x04000000-0x1fffffff 64bit pref]
> to [bus 94-bc] requires relaxed alignment rules
> pci 0000:93:00.0: bridge window [mem size 0x3018000000 64bit pref]:
> can't assign; no space
> pci 0000:93:00.0: bridge window [mem size 0x3018000000 64bit pref]:
> failed to assign
> pci 0000:93:00.0: bridge window [mem 0x8880000000-0x889bffffff 64bit
> pref]: assigned
> 
> --- GPU BAR enumeration ---
> pci 0000:97:00.0: [1002:7551] type 00 class 0x030000 PCIe Legacy Endpoint
> pci 0000:97:00.0: BAR 0 [mem 0x00000000-0x0fffffff 64bit pref]
> pci 0000:97:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 2.5
> GT/s PCIe x4 link at 0000:66:03.0 (capable of 504.112 Gb/s with 32.0
> GT/s PCIe x16 link)
> pci 0000:97:00.0: BAR 0 [mem 0x8880000000-0x888fffffff 64bit pref]: assigned
> pci 0000:97:00.0: BAR 4 [io  size 0x0100]: can't assign; no space
> 
> --- amdgpu rebar attempt and failure ---
> amdgpu 0000:97:00.0: amdgpu: PCIE atomic ops is not supported
> amdgpu 0000:97:00.0: BAR 2 [mem 0x8890000000-0x88901fffff 64bit pref]: releasing
> amdgpu 0000:97:00.0: BAR 0 [mem 0x8880000000-0x888fffffff 64bit pref]: releasing
> amdgpu 0000:97:00.0: BAR 0 [mem size 0x800000000 64bit pref]: can't
> assign; no space
> amdgpu 0000:97:00.0: BAR 0 [mem size 0x800000000 64bit pref]: failed to assign
> amdgpu 0000:97:00.0: BAR 2 [mem size 0x00200000 64bit pref]: can't
> assign; no space
> amdgpu 0000:97:00.0: BAR 2 [mem size 0x00200000 64bit pref]: failed to assign
> amdgpu 0000:97:00.0: amdgpu: Not enough PCI address space for a large BAR.
> amdgpu 0000:97:00.0: BAR 0 [mem 0x8880000000-0x888fffffff 64bit pref]: assigned
> amdgpu 0000:97:00.0: amdgpu: VRAM: 32624M 0x0000008000000000 -
> 0x00000087F6FFFFFF (32624M used)
> 
> 
> 
> Geramy L. Loveless
>