Re: [BUG] Thunderbolt eGPU PCI BARs incorrectly assigned, fails to assign memory

["Ilpo Järvinen" <ilpo.jarvinen@linux.intel.com>]

[-- Attachment #1: Type: text/plain, Size: 5117 bytes --]

On Mon, 1 Sep 2025, Ilpo Järvinen wrote:

> On Mon, 1 Sep 2025, Ilpo Järvinen wrote:
> > On Sun, 31 Aug 2025, Steve Oswald wrote:
> > 
> > > I’ve encountered an issue with Thunderbolt eGPU (externally connected
> > > gpu via thunderbolt 4). The change from kernel 6.10.14 to 6.11.0 broke
> > > the pci memory assignment of the external pcie device. I figured out
> > > which version broke it by using ubuntu 25.04 and downgrading the
> > > kernel (https://raw.githubusercontent.com/pimlie/ubuntu-mainline-kernel.sh/master/ubuntu-mainline-kernel.sh).
> > > 
> > > >From the dmesg output, on the broken 6.11.0 I see 'failed to assign'.
> > > The issue occurs (almost never) on previous kernel version 6.10.14.
> > > Using pci=realloc did not change the behavior (I can produce the dmesg
> > > output if necessary).
> > > 
> > > The issue was tested with 2 egpus (Radeon Instinct MI50 32GB, NVIDIA
> > > 3080 10GB). Both the amd and the nvidia driver fail to initialize the
> > > device because they cannot write the pcie messages.
> > > 
> > > System details:
> > > - Kernel: Linux 6.10.14-061014-generic (Ubuntu build) > 6.11.0-061100
> > > - Laptop: TUXEDO InfinityBook Pro 16 - Gen8 with Thunderbolt 4
> > > - eGPU: Radeon Instinct MI50 32GB, NVIDIA 3080 10GB
> > > 
> > > Steps to reproduce:
> > > 1. Boot the system with the eGPU.
> > > 2. Observe PCI BAR message in `dmesg`.
> > > 
> > > Logs:
> > > both kernel messages, lspci can be found here:
> > > https://gist.github.com/stepeos/cd060c7d66ab195f51ab4d5675b4e4af
> > > raw files:
> > > - dmesg_linux_6.11.0.log
> > > https://gist.githubusercontent.com/stepeos/cd060c7d66ab195f51ab4d5675b4e4af/raw/f9470a06ff929d386c50ec6b5d07e0ff3f053dcf/dmesg_linux_6.11.0.log
> > > - dmesg_linux_6.10.14.log
> > > https://gist.githubusercontent.com/stepeos/cd060c7d66ab195f51ab4d5675b4e4af/raw/f9470a06ff929d386c50ec6b5d07e0ff3f053dcf/dmesg_linux_6.10.14.log
> > > 
> > > If additional info is needed, I'm happy to help.
> > 
> > Hi Steve,
> > 
> > Thanks for the report.
> > 
> > My analysis is that the problem boils down to lack of this line with 6.11:
> > 
> > pcieport 0000:00:07.0: resource 15 [mem 0x6000000000-0x601bffffff 64bit pref] released
> > 
> > It means one of the upstream bridge windows could not be released for 
> > resize as it is printed from pci_reassign_bridge_resources() which likely 
> > occurs inside pci_resize_resource() call from amdgpu(?).
> > 
> > The very likely cause is this check:
> > 
> >                         /* Ignore BARs which are still in use */
> >                         if (res->child)
> >                                 continue;
> > 
> > ...which (until very recently) is entirely silent so there's no warning 
> > whatsover what is the root cause.
> 
> Hi again,
> 
> Actually, scratch most of that. It's not during resize as the log should 
> say "releasing" (I don't know how I got this confused). "released" is from 
> pci_bridge_release_resources() which is called from 
> pci_bus_release_bridge_resources() doesn't even try to walk upwards.
> 
> But that begs question, why didn't also the bridge windows fail their 
> assignments.
> 
> Resource fitting calculates size for the bridge window:
> 
> pci 0000:03:00.0: bridge window [mem 0x800000000-0x10003fffff 64bit pref] to [bus 04-2c] add_size 100000 add_align 100000
> 
> ...but I cannot see assignment for that even being attempted as almost 
> immediately, this occurs:
> 
> pci 0000:03:00.0: bridge window [mem 0x6000000000-0x601bffffff 64bit pref]: assigned
> 
> ...which is much less than 0x10003fffff-0x800000000. I cannot think of 
> anything what could make it shrink like that.
> 
> I'll have to think this more, it might require a debug patch but I'll 
> think until tomorrow to see if I can understand it from the code alone.

Only thing I can think of is something going wrong in 
adjust_bridge_window().

Could you please provide a dmesg with dyndbg="file drivers/pci/* +p" on 
the kernel cmdline from 6.11.

> > What this means, is that there's some assigned resource underneath 
> > 0000:00:07.0 with 6.11 that wasn't there with 6.10. And it is because 6.11 
> > tried harder to get your resources assigned and was successful here and 
> > there resulting in pinning the bridge window in its place, whereas 6.10 
> > failed to assign the same resource.
> > 
> > Could you provide /proc/iomem (it's enough to do that for 6.11 for now)?
> > 
> > 
> > You could try to use hpmmioprefsize= on kernel's command line to reserve 
> > more space for the bridge windows, the default is only 2M and these GPUs 
> > need a magnitude more (gigabytes), you can check from 6.10 what the sizes 
> > of the BARs on the GPU are, and round the sum upwards to the next power of 
> > two multiple.
> >
> > I'd also be interested to see why pci=realloc failed to solve this problem 
> > as it should reconfigure the entire resource tree so if you could provide 
> > the logs with that. Please take lspci with -vvv.
> > 
> > 
> > 
> 
> 

-- 
 i.