Re: Connecting multiple TB3 eGPUs via USB4 hub?'

[Mika Westerberg <mika.westerberg@linux.intel.com>]

On Thu, Jun 11, 2026 at 06:41:14AM +0200, Mika Westerberg wrote:
> Hi,
> 
> On Thu, Jun 11, 2026 at 12:48:40AM +0200, Dag B wrote:
> > 
> > On 6/9/26 14:08, Mika Westerberg wrote:
> > > Hi,
> > > 
> > > On Mon, Jun 08, 2026 at 10:26:40AM -0500, Bjorn Helgaas wrote:
> > > > [+cc linux-usb (Thunderbolt maintainers), Ilpo, Lukas]
> > > > 
> > > > On Sun, May 24, 2026 at 04:56:56PM +0200, Dag B wrote:
> > > > > I am attempting to connect two TB3 enclosures to the same USB4 port via a
> > > > > USB4 hub.
> > > > > 
> > > > > I cannot figure out of this is in violation of the TB3, TB4 or USB4 spec.
> > > > I'm not aware of a spec issue here.
> > Thank you very much Bjorn, slightly more interesting trying to get it to
> > work then.
> > > It should be fine but typically PCIe resources for one tunneled PCIe root
> > > port may not be enough for multiple GPUs. Therefore I suggest to connect
> > > them directly to the host USB4 ports without a hub.
> > 
> > That makes perfect sense. Until you run out of physical ports. The entire
> > point of this exercise started with trying to connect a 5th 3090 to my fw13
> > motherboard. :-D
> > 
> > Hence the USB4 hub. But even with just the hub and two connected GPUs, it
> > fails.
> 
> Yes most likely because a single GPU takes lots of resources and there is
> quite small window reserved per root port for example:
> 
>   Memory behind bridge: 60000000-6c1fffff [size=194M] [32-bit]
>   Prefetchable memory behind bridge: 6040000000-605bffffff [size=448M] [32-bit]
> 
> Looking at your logs you have all 4 PCIe root ports for tunneling listed
> there so I would expect you have 4 USB4/TB ports available as well
> (typically the root ports not available for tunneling will not be visible
> to the OS).
> 
> > For now, I have limited my testing to just the USB4 hub and either one or
> > two GPUs.
> > 
> > 
> > > Second thing is that you have bunch of PCIe related command line parameters
> > > that may affect. I suggest removing all of them, and retry. There is hardly
> > > any reason to add these - the kernel should be able to handle this by
> > > default.
> > 
> > Mhm. I have stripped the command line parameters for some of my tests. It
> > absolutely works with nothing extra for a single GPU. And it absolutely did
> > not work without command-line params and with 4 GPUs  when I started on this
> > adventure. But that was around 6.16-6.17.
> > 
> > 
> > > 
> > > Then provide full dmesg along with output of 'sudo lspci -vv' so we can
> > > look a the resource allocation.
> > 
> > This very quickly becomes a lot of data as I test various things. I put a
> > repo with various data up on github.
> > 
> > https://github.com/dagbdagb/usb4_to_tb3_troubleshooting/
> 
> There seems to be no dmesg + lspci without the command line parameters? Can
> you do this:
> 
>   1. Remove all the non-default command line parameters.
>   2. Boot the system up, nothing connected.
>   3. Plug in the USB4 dock, enable PCIe tunnel (this probably is automatic)
>   4. Plug in the first eGPU to the dock, enable PCIe tunnel
>   5. Plug in the second eGPU to the dock, enable PCIe tunnel
> 
> From this take full dmesg and output of 'sudo lspci -vv'. At least we can
> see how the "default" kernel resource allocation fails.

Actually there is dmesg.6+hotplug_egpu_2 and lspci-vv.6+hotplug_egpu_2.
Sorry missed that.

The first GPU seems to fit fine:

[   24.538295] nvidia 0000:66:00.0: enabling device (0000 -> 0003)
[   24.539803] nvidia 0000:66:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
[   24.590905] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  610.43.02  Release Build  (portage@localhost)  Wed Jun 10 19:38:57 -00 2026

But then the second one has not enough resources left, the 55:03.0
downstream port that is used to connect the second GPU has only these left:

   I/O behind bridge: b000-bfff [size=4K] [16-bit]
   Memory behind bridge: 66e00000-6a2fffff [size=53M] [32-bit]
   Prefetchable memory behind bridge: [disabled] [64-bit]

Some BIOSes has an option that you can increase the amount of resources for
the root ports but some vendors hide it.