Re: [BUG] Thunderbolt eGPU PCI BARs incorrectly assigned, fails to assign memory

["Ilpo Järvinen" <ilpo.jarvinen@linux.intel.com>]

[-- Attachment #1: Type: text/plain, Size: 6226 bytes --]

On Mon, 1 Sep 2025, Steve Oswald wrote:

> I've added the dmesg output fordyndbg="file drivers/pci/*. I wasn't
> sure if I added it with the escaped quotes correctly.
> https://gist.github.com/stepeos/cd060c7d66ab195f51ab4d5675b4e4af/raw/9cf5fc3a8c4f13588a33d61865f804f85e50470a/dmesg_linux_6.11.0_dyndbg.log

Thanks. It was correct and caught the reason for the problem:

[    9.090106] pci 0000:03:00.0: bridge window [mem 0x800000000-0x10003fffff 64bit pref] shrunken by 0x00000007e4400000

I'll get back to this once I've tried to figure why it does what it does.

-- 
 i.

> Am Mo., 1. Sept. 2025 um 19:07 Uhr schrieb Ilpo Järvinen
> <ilpo.jarvinen@linux.intel.com>:
> >
> > On Mon, 1 Sep 2025, Ilpo Järvinen wrote:
> >
> > > On Mon, 1 Sep 2025, Ilpo Järvinen wrote:
> > > > On Sun, 31 Aug 2025, Steve Oswald wrote:
> > > >
> > > > > I’ve encountered an issue with Thunderbolt eGPU (externally connected
> > > > > gpu via thunderbolt 4). The change from kernel 6.10.14 to 6.11.0 broke
> > > > > the pci memory assignment of the external pcie device. I figured out
> > > > > which version broke it by using ubuntu 25.04 and downgrading the
> > > > > kernel (https://raw.githubusercontent.com/pimlie/ubuntu-mainline-kernel.sh/master/ubuntu-mainline-kernel.sh).
> > > > >
> > > > > >From the dmesg output, on the broken 6.11.0 I see 'failed to assign'.
> > > > > The issue occurs (almost never) on previous kernel version 6.10.14.
> > > > > Using pci=realloc did not change the behavior (I can produce the dmesg
> > > > > output if necessary).
> > > > >
> > > > > The issue was tested with 2 egpus (Radeon Instinct MI50 32GB, NVIDIA
> > > > > 3080 10GB). Both the amd and the nvidia driver fail to initialize the
> > > > > device because they cannot write the pcie messages.
> > > > >
> > > > > System details:
> > > > > - Kernel: Linux 6.10.14-061014-generic (Ubuntu build) > 6.11.0-061100
> > > > > - Laptop: TUXEDO InfinityBook Pro 16 - Gen8 with Thunderbolt 4
> > > > > - eGPU: Radeon Instinct MI50 32GB, NVIDIA 3080 10GB
> > > > >
> > > > > Steps to reproduce:
> > > > > 1. Boot the system with the eGPU.
> > > > > 2. Observe PCI BAR message in `dmesg`.
> > > > >
> > > > > Logs:
> > > > > both kernel messages, lspci can be found here:
> > > > > https://gist.github.com/stepeos/cd060c7d66ab195f51ab4d5675b4e4af
> > > > > raw files:
> > > > > - dmesg_linux_6.11.0.log
> > > > > https://gist.githubusercontent.com/stepeos/cd060c7d66ab195f51ab4d5675b4e4af/raw/f9470a06ff929d386c50ec6b5d07e0ff3f053dcf/dmesg_linux_6.11.0.log
> > > > > - dmesg_linux_6.10.14.log
> > > > > https://gist.githubusercontent.com/stepeos/cd060c7d66ab195f51ab4d5675b4e4af/raw/f9470a06ff929d386c50ec6b5d07e0ff3f053dcf/dmesg_linux_6.10.14.log
> > > > >
> > > > > If additional info is needed, I'm happy to help.
> > > >
> > > > Hi Steve,
> > > >
> > > > Thanks for the report.
> > > >
> > > > My analysis is that the problem boils down to lack of this line with 6.11:
> > > >
> > > > pcieport 0000:00:07.0: resource 15 [mem 0x6000000000-0x601bffffff 64bit pref] released
> > > >
> > > > It means one of the upstream bridge windows could not be released for
> > > > resize as it is printed from pci_reassign_bridge_resources() which likely
> > > > occurs inside pci_resize_resource() call from amdgpu(?).
> > > >
> > > > The very likely cause is this check:
> > > >
> > > >                         /* Ignore BARs which are still in use */
> > > >                         if (res->child)
> > > >                                 continue;
> > > >
> > > > ...which (until very recently) is entirely silent so there's no warning
> > > > whatsover what is the root cause.
> > >
> > > Hi again,
> > >
> > > Actually, scratch most of that. It's not during resize as the log should
> > > say "releasing" (I don't know how I got this confused). "released" is from
> > > pci_bridge_release_resources() which is called from
> > > pci_bus_release_bridge_resources() doesn't even try to walk upwards.
> > >
> > > But that begs question, why didn't also the bridge windows fail their
> > > assignments.
> > >
> > > Resource fitting calculates size for the bridge window:
> > >
> > > pci 0000:03:00.0: bridge window [mem 0x800000000-0x10003fffff 64bit pref] to [bus 04-2c] add_size 100000 add_align 100000
> > >
> > > ...but I cannot see assignment for that even being attempted as almost
> > > immediately, this occurs:
> > >
> > > pci 0000:03:00.0: bridge window [mem 0x6000000000-0x601bffffff 64bit pref]: assigned
> > >
> > > ...which is much less than 0x10003fffff-0x800000000. I cannot think of
> > > anything what could make it shrink like that.
> > >
> > > I'll have to think this more, it might require a debug patch but I'll
> > > think until tomorrow to see if I can understand it from the code alone.
> >
> > Only thing I can think of is something going wrong in
> > adjust_bridge_window().
> >
> > Could you please provide a dmesg with dyndbg="file drivers/pci/* +p" on
> > the kernel cmdline from 6.11.
> >
> > > > What this means, is that there's some assigned resource underneath
> > > > 0000:00:07.0 with 6.11 that wasn't there with 6.10. And it is because 6.11
> > > > tried harder to get your resources assigned and was successful here and
> > > > there resulting in pinning the bridge window in its place, whereas 6.10
> > > > failed to assign the same resource.
> > > >
> > > > Could you provide /proc/iomem (it's enough to do that for 6.11 for now)?
> > > >
> > > >
> > > > You could try to use hpmmioprefsize= on kernel's command line to reserve
> > > > more space for the bridge windows, the default is only 2M and these GPUs
> > > > need a magnitude more (gigabytes), you can check from 6.10 what the sizes
> > > > of the BARs on the GPU are, and round the sum upwards to the next power of
> > > > two multiple.
> > > >
> > > > I'd also be interested to see why pci=realloc failed to solve this problem
> > > > as it should reconfigure the entire resource tree so if you could provide
> > > > the logs with that. Please take lspci with -vvv.
> > > >
> > > >
> > > >
> > >
> > >
> >
> > --
> >  i.
>