[Bug 221822] New: tb-retry.patch (against drivers/thunderbolt/tb.c): makes USB4 link setup resilient to transient sideband failures on AMD hosts with slow-to-ready device routers.

[Bug 221822] New: tb-retry.patch (against drivers/thunderbolt/tb.c): makes USB4 link setup resilient to transient sideband failures on AMD hosts with slow-to-ready device routers.

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=221822

            Bug ID: 221822
           Summary: tb-retry.patch (against drivers/thunderbolt/tb.c):
                    makes USB4 link setup resilient to transient sideband
                    failures on AMD hosts with slow-to-ready device
                    routers.
           Product: Drivers
           Version: 2.5
    Kernel Version: 7.2-rc5
          Hardware: AMD
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P3
         Component: USB
          Assignee: drivers_usb@kernel-bugs.kernel.org
          Reporter: j.jamesjohnson@icloud.com
        Regression: No

Created attachment 310555
  --> https://bugzilla.kernel.org/attachment.cgi?id=310555&action=edit
tb-retry.patch (against drivers/thunderbolt/tb.c): makes USB4 link setup
resilient to transient sideband failures on AMD hosts with slow-to-ready device
routers.

## Summary

CalDigit Element 5 Hub (USB4 v2 / TB5) on AMD Strix Point USB4 host: link
training fails CL-states/TMU/tunnel creation on ~5 of 6 plug cycles, then one
cycle lands and is fully functional — link never lane-bonds (single lane 20
Gb/s)

## Description

An AMD Strix Point mini PC (Meigao F8BAC, Ryzen AI 9 HX 370) cannot reliably
establish a link with a CalDigit Element 5 Hub (vendor 0x3d, device 0x28,
TB5/USB4 v2, firmware 64.1 — current per CalDigit). The hub works correctly on
a Mac at USB4 40 Gb/s with the same cables.

Most plug cycles fail identically:

```
thunderbolt 1-2: new device found, vendor=0x3d device=0x28
thunderbolt 1-2: CalDigit, Inc. Element 5 Hub
thunderbolt 0000:c8:00.6: 2: failed to enable CL states
thunderbolt 0000:c8:00.6: 2: failed to enable TMU
thunderbolt 0000:c8:00.6: 2: USB3 tunnel creation failed
thunderbolt 0000:c8:00.6: 2:1: hop deactivation failed for hop 1, index 8
thunderbolt 0000:c8:00.6: PCIe Down path activation failed: -107
thunderbolt 0000:c8:00.6: 2:9: PCIe tunnel activation failed, aborting
thunderbolt 1-2: device disconnected
```

The host router autosuspends and re-scans every ~18 s, so cycles repeat
indefinitely — **and after ~5 failed cycles, one attempt succeeds**. Once
linked, the device is fully functional and stable:

- `thunderbolt 0-2: Element 5 Hub`, authorized=1, **20.0 Gb/s rx/tx** (link
only ever trains **single lane** — bonding never occurs, even on successful
links)
- USB3 tunnel up: Intel USB3 HUB (8087:5787, JHL9480 internal) + CalDigit
USB3.2 10G hub enumerate through the tunnel
- PCIe tunnel up: full JHL9480 bridge hierarchy (01:00.0 → 02:00.0/01.0/02.0)
- 5 USB HDDs, card readers, and an RTL8153 GbE adapter all enumerate and pass
traffic through the tunnel; sustained reads 208–215 MB/s (platter-limited) with
zero disconnects or errors after the link lands

So the failure is not a hard incompatibility — the link *can* come up, but lane
training/CL-state enablement is unreliable and only succeeds sporadically (and
never with lane bonding).

### Hardware

- Host: Micro Computer (HK) Tech Limited "AI Series", board **F8BAC**
(Meigao/miniscloud mini PC), AMD Ryzen AI 9 HX 370 (Strix Point), AMI BIOS 1.04
(2025-04-11, latest for this board)
- USB4 host routers: `0000:c8:00.5` [1022:151c] (domain0 / Port A) and
`0000:c8:00.6` [1022:151d] (domain1 / Port B). **Both ports train the device**
and both fail tunnel setup on most cycles; the working 40 Gb/s bonded link
currently in use is on Port A.
- Device: CalDigit Element 5 Hub, TB5/USB4 v2, FW 64.1
- Cables: the CalDigit-supplied TB5 cable

### Kernels tested (all byte-identical failure)

- Proxmox VE kernels: 6.14.8-2-pve, 6.14.11-9-pve, 7.0.6-2-pve, 7.0.14-8-pve
- **Mainline 7.2-rc5** (Ubuntu mainline build) with `thunderbolt.dyndbg=+p`
- thunderbolt module params: `clx=0`, `clx=0 bw_alloc_mode=0` (link trains,
tunnels still fail)
- Kernel cmdlines: `pcie_port_pm=off`, `pcie_aspm=off iommu=pt` — no effect
- Boot-attached vs hotplug — no effect
- Both USB4 ports and direct attach — no effect on the tunnel failure.
**Daisy-chaining the hub downstream of an Elgato TB3 Pro Dock was the only
working topology** (all drives restored, stable) — it was rolled back as
non-viable for this deployment, but it corroborates that the failure is
specific to native USB4-v1↔USB4-v2 link negotiation (the TB3 path skips
CL-state negotiation)

### Code analysis + retry patch experiment (drivers/thunderbolt @ v7.2-rc5)

- `-107` = **-ENOTCONN** ("transport endpoint not connected") — same errno lane
bonding returns (silently) when the second lane never comes up. PCIe path
activation dies on a port-state race.
- The printed `failed to enable CL states` is **not** a support check
(`-EOPNOTSUPP` is swallowed without warning); it is a real **sideband
config-space transaction failure** against the device router in
`tb_port_pm_secondary_set()`/`tb_port_clx_enable()` (clx.c). TMU and
USB3-tunnel failures are the same class.
- **No-fallback aggravator in `tb_enable_clx()`:** CL2 is attempted first for
v2 routers (JHL9480 is v2); only `-EOPNOTSUPP` falls back to CL0s/CL1. A
*transient* CL2 transaction error aborts CL enablement with no fallback and no
retry. Same no-retry pattern in `tb_enable_tmu()` and `tb_tunnel_usb3()`.
- Retimer NVM reads report `vendor=0x0 device=0x0` even on successful links —
further sideband unreliability.
- Gil Fine's 7.2-rc1 router-readiness patches (Router-Ready verify `062023c`,
Config-Ready timeout `ba2cc38`, notification timeout 255 ms `e24f3c0`) are
present in the test kernel and do not cure it.

**Tested a 3-site retry patch** (CL2→CL0s/CL1 fallback on ANY error + 3× retry
with 100/200/300 ms backoff on
`tb_enable_clx`/`tb_enable_tmu`/`tb_tunnel_usb3`/`tb_tunnel_activate`), built
as `7.2.0-rc5-tbfix1`:

- **Boot-attached:** CL-states and TMU failures **eliminated** (fallback +
retries fixed both). USB3 tunnel creation still failed every cycle — the 600 ms
retry window isn't wide enough for USB3 specifically.
- **Hotplug (unplug 10 s, replug): full success on first cycle** —
CL/TMU/USB3/PCIe all succeed, zero errors, `PCIe Down/Up path activation
complete`, 4 JHL9480 bridges enumerated, all USB devices + storage via tunnel,
sustained reads 207–222 MB/s (platter-limited), stable.
- **Conclusion:** the CL/TMU failures are provably transient and recoverable
in-kernel (retries work). USB3 tunnel creation is the remaining hard point — it
needs either a device reset (hotplug) or a substantially longer
backoff/readiness wait. This narrows the fix target for upstream considerably.
- **Lane bonding is also timing-flaky:** the host's second-lane port sits in
`TB_PORT_CONNECTING` (state 1) for the full 1 s `tb_wait_for_port` window on
most cycles, then "failed to reach state TB_PORT_UP. Ignoring port." When lane
1 does train late, `tb_switch_set_link_width(DUAL)` is never re-attempted, so
the session stays single-lane (20 G) even though both lanes are up. Both a TB3
dock and this TB5 hub DO bond at 40 Gb/s aggregate on cycles where lane 1
trains in time. Suggested hardening: re-attempt lane bonding when the second
lane reaches `TB_PORT_UP` after the initial attempt fails.

### Notes

- A Thunderbolt 3 dock (Elgato TB3 Pro Dock) links and tunnels reliably on the
same host/port — the TB3 path skips USB4 CL-state negotiation, so the host
hardware is functional for TB3.
- Blanket auto-authorization udev rule is in place; failure precedes
authorization anyway.
- Possibly related: bug 221319 (AMD USB4 host + TB5 peripherals), and the May
2026 linux-usb thread "USB4 v2 TBGAA tunnel creation crash in TMU enhanced
uni-directional mode" (AMD Strix Halo + TB5 dock) — same class, but that thread
ended unresolved with a suspected device PD-firmware cause. This report adds a
distinct data point: here the link *does* come up (single lane) and the failure
is specifically at device-switch CL-state enablement and tunnel creation.

### Attachments

1. `dmesg-tb-plug.txt` — standard dmesg of one plug cycle on 7.0.14-8-pve
2. `dmesg-tb-dyndbg.txt` — full `thunderbolt.dyndbg=+p` trace of plug cycles on
mainline 7.2-rc5
3. `tb-retry.patch` - against drivers/thunderbolt/tb.c: makes USB4 link setup
resilient to transient sideband failures on AMD hosts with slow-to-ready device
routers.

---

## Attachment 1: dmesg-tb-plug.txt (7.0.14-8-pve)

```
[  342.119755] thunderbolt 1-2: new device found, vendor=0x3d device=0x28
[  342.119951] thunderbolt 1-2: CalDigit, Inc. Element 5 Hub
[  343.058197] thunderbolt 0000:c8:00.6: 2: failed to enable CL states
[  343.058526] thunderbolt 0000:c8:00.6: 2: failed to enable TMU
[  343.059968] thunderbolt 0000:c8:00.6: 2: USB3 tunnel creation failed
[  343.061282] thunderbolt 0000:c8:00.6: 2:1: hop deactivation failed for hop
1, index 8
[  343.061386] thunderbolt 0000:c8:00.6: PCIe Down path activation failed: -107
[  343.061749] thunderbolt 0000:c8:00.6: 2:9: PCIe tunnel activation failed,
aborting
[  343.062408] thunderbolt 1-2: device disconnected
```

## Attachment 2: dmesg-tb-dyndbg.txt (mainline 7.2-rc5, thunderbolt.dyndbg=+p)

Key excerpt (full trace on request — multi-hundred-line cycles):

```
[  267.142724] thunderbolt 0000:c8:00.6: 2: current link width symmetric,
single lane
[  267.145051] thunderbolt 1-2: new device found, vendor=0x3d device=0x28
[  267.145265] thunderbolt 1-2: CalDigit, Inc. Element 5 Hub
[  268.493956] thunderbolt 0000:c8:00.6: 2: failed to enable CL states
[  268.494796] thunderbolt 0000:c8:00.6: 2:20: available bandwidth for new USB3
tunnel 18000/18000 Mb/s
[  268.495368] thunderbolt 0000:c8:00.6: 2: USB3 tunnel creation failed
[  268.496009] thunderbolt 0000:c8:00.6: 0:5 <-> 2:9 (PCI): activating
[  268.496102] thunderbolt 0000:c8:00.6: 2:9: PCIe tunnel activation failed,
aborting
[  268.496176] thunderbolt 0000:c8:00.6: 0:2: switch unplugged
[  268.496231] thunderbolt 0000:c8:00.6: 2: link width set to symmetric, single
lane
[  268.496469] thunderbolt 1-2: device disconnected
```

Host-router TMU succeeds each resume: `0: TMU: mode set to: bi-directional,
HiFi`.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 221822] tb-retry.patch (against drivers/thunderbolt/tb.c): makes USB4 link setup resilient to transient sideband failures on AMD hosts with slow-to-ready device routers.

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=221822

--- Comment #1 from j. james johnson (j.jamesjohnson@icloud.com) ---
The 3-site retry patch (CL2→CL0s/CL1 fallback on *any* CL-enable error; 3×
retry with 100/200/300 ms backoff on
`tb_enable_clx`/`tb_enable_tmu`/`tb_tunnel_usb3`/`tb_tunnel_activate`) was
ported to the **shipping distro kernel `7.0.14-8-pve`** (pve-kernel build with
patch applied as `patches/kernel/0052-...`) and is now the daily driver on this
host. Results correct the original report:

- **Boot-attached hub self-recovers after ~2 failed cycles (~252 s) — no
hotplug.** This invalidates the report's claim that boot-attached USB3 tunnel
creation is a hard failure point requiring a device reset. 

- **CL-state and TMU failures eliminated** — the CL2→CL0s/CL1 fallback absorbs
them.
- **The link lane-bonds at 40 Gb/s** on the successful cycle (`link width set
to symmetric, dual lanes`) — the original Summary's "never bonds" is wrong;
bonding occurs whenever lane 1 trains in time, and the retries keep the session
alive long enough for that to happen.
- **New errno data:** the winning cycle logged `PCIe Down path activation
failed: -13` (EACCES) twice before succeeding, alongside the `-107` (ENOTCONN)
seen on failing cycles. Both are transient sideband/port-state races during
device-router enumeration; retry recovers both.
- **BIOS excluded:** flashed board BIOS 1.06 (which adds a retimer GPIO-detect
change). Retimer sideband reads unchanged — lane-0 NVM still all-zeros, lane-1
retimer still never responds. Firmware is not the active variable; tracked
separately with the board vendor.
- 5 HDDs + PCIe/USB3 devices stable under sustained load (I/O at platter speed)

Open, load-tested asks to restate to upstream when the thread is next touched:
(a) retry-with-backoff on transient CL/TMU/USB3/PCIe-path-activation failures;
(b) CL2→CL0s/CL1 fallback on any CL-enable error, not just `-EOPNOTSUPP`; (c)
re-attempt `tb_switch_set_link_width(DUAL)` when the second lane reaches
`TB_PORT_UP` late — single-lane sessions still occur on cycles where lane 1
trains after the initial bonding attempt.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.