[Bug 222091] New: Regression 7.1.10→7.2.x: poweroff hangs after Lenovo USB4 dock (40BF) connected during session; state persists after unplug

[Bug 222091] New: Regression 7.1.10→7.2.x: poweroff hangs after Lenovo USB4 dock (40BF) connected during session; state persists after unplug

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=222091

            Bug ID: 222091
           Summary: Regression 7.1.10→7.2.x: poweroff hangs after Lenovo
                    USB4 dock (40BF) connected during session; state
                    persists after unplug
           Product: Drivers
           Version: 2.5
          Hardware: Intel
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P3
         Component: USB
          Assignee: drivers_usb@kernel-bugs.kernel.org
          Reporter: abadragan@gmail.com
        Regression: No

[Filed under Drivers/USB because there is no Thunderbolt component; this is a
drivers/thunderbolt (NHI) regression. Downstream report: see Red Hat Bugzilla,
Fedora component "kernel" -- URL to be added in a follow-up comment.]

REGRESSION: System fails to power off once a Lenovo ThinkPad USB4 Dock 5000
(40BF) has been connected during the session. Userspace shutdown completes
cleanly (journal shows poweroff.target reached, filesystems synced, journald
stopped), then the machine hangs at the firmware splash screen and requires a
hard power-off. The state is LATCHED: unplugging the dock afterwards does not
help; only sessions where the dock was never connected power off normally.

BISECTION (all verified on this machine):
  6.19.10-300.fc44  GOOD  (install-era kernel)
  6.19.12-300.fc44  GOOD
  7.1.10-200.fc44   GOOD  (also verified on a parallel Bluefin install with
                           identical kernel config and cmdline)
  7.2.5-200.fc44    BAD   (first 7.x kernel this machine booted; install
                           jumped 6.19.10 -> 7.2.5)
  7.2.7-200.fc44    BAD
  7.2.8-200.fc44    BAD

ADDITIONAL FINDING: "modprobe -r thunderbolt" while docked deadlocks
permanently (process in D state). The dock router (thunderbolt 0-3) and
retimer (0-0:3.1) disconnect, then the NHI teardown blocks:

  [<0>] nhi_pci_remove+0x53/0x60 [thunderbolt]
  [<0>] pci_device_remove+0x4a/0xb0
  [<0>] device_release_driver_internal+0x19e/0x200
  [<0>] driver_detach+0x48/0x90
  [<0>] bus_remove_driver+0x78/0x110
  [<0>] pci_unregister_driver+0x36/0xe0
  [<0>] nhi_unload+0x10/0x540 [thunderbolt]

However, blacklisting the thunderbolt module entirely does NOT fix the
poweroff hang, and neither does blacklisting ucsi_acpi -- suggesting the
latched state is below these drivers (ACPI power resources / EC interaction).
The BIOS exposes \_SB_.PC00.TBT0/TBT1 power resources and hands USB4 _OSC
control to the OS at boot; ACPI tables also show pre-existing
AE_ALREADY_EXISTS errors (\_SB.BGNV, \_SB.PC00.PTMA, ...).

HARDWARE:
  Laptop: PC Specialist FusionVI 14 (Clevo/Kapok 1558:2680), Insyde BIOS
          1.07.03dTPCS (2024-10-24), Intel Core Ultra 200V (Lunar Lake)
  TB4 controller: Intel [8086:a833], driver thunderbolt
  Dock: Lenovo ThinkPad USB4 Dock 5000 (40BF), "Lenovo Dock - USB4 Router"
        (17ef:118d), Intel retimer (8086:0d9c), dock DMC fw 1.2.04 (latest)

REPRODUCER: boot 7.2.x on this hardware, attach dock at any point, then
poweroff (or "modprobe -r thunderbolt" for the unload deadlock).

WORKAROUND: pin kernel to 7.1.10-200.fc44.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 222091] Regression 7.1.10→7.2.x: poweroff hangs after Lenovo USB4 dock (40BF) connected during session; state persists after unplug

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=222091

Artem S. Tashkinov (aros@gmx.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |NEEDINFO

--- Comment #1 from Artem S. Tashkinov (aros@gmx.com) ---
Could you please bisect?

Any LLM can guide you through it.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 222091] Regression 7.1.10→7.2.x: poweroff hangs after Lenovo USB4 dock (40BF) connected during session; state persists after unplug

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=222091

--- Comment #2 from AlexTLDR (abadragan@gmail.com) ---
(In reply to Artem S. Tashkinov from comment #1)
> Could you please bisect?
> 
> Any LLM can guide you through it.

Follow-up: I found that the regression is resolved by the kernel
parameter thunderbolt.host_reset=0. With it, the machine powers off
cleanly even after the dock has been connected during the session.

Looking at what changed in drivers/thunderbolt between the last good
(7.1.x) and first bad (7.2.x) kernels, the suspect is commit
5f40cba7d4 "thunderbolt: Assert downstream port reset on shutdown",
merged for v7.2-rc1. That commit forces the host_reset path in the
PCI ->shutdown callback and asserts PORT_CS_19.DPR on each connected
downstream port before tearing down the router tree. This matches all
observations on this machine:

- the hang only occurs once the dock (router 17ef:118d) has been
  connected in the session, i.e. once a connected downstream port
  exists to assert the reset on
- the fault is latched (later unplugging does not help)
- host_reset=0 skips exactly this new shutdown-time reset sequence
  and the poweroff hang disappears
- "modprobe -r thunderbolt" deadlocks in the same teardown path

So the regression appears to be in the new downstream-port-reset
(SBTX-low) sequence: on this hardware (Insyde BIOS 1.07.03dTPCS,
Intel Core Ultra 200V, NHI 8086:a833) the assertion against the
Lenovo 40BF dock router never completes and the machine hangs at
firmware handoff.

Since that commit fixed a real issue elsewhere (60 s polling of a
dead link / warm reset on some platforms), a plain revert is probably
not the answer - perhaps the DPR assertion needs a timeout or a
quirk. Happy to test a patch or a scratch build with the revert to
confirm the identification.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 222091] Regression 7.1.10→7.2.x: poweroff hangs after Lenovo USB4 dock (40BF) connected during session; state persists after unplug

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=222091

--- Comment #3 from Artem S. Tashkinov (aros@gmx.com) ---
Guesses aren't really accepted here.

Fetch 7.2.9, revert the patch, confirm everything's working fine.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 222091] Regression 7.1.10→7.2.x: poweroff hangs after Lenovo USB4 dock (40BF) connected during session; state persists after unplug

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=222091

--- Comment #4 from AlexTLDR (abadragan@gmail.com) ---
(In reply to Artem S. Tashkinov from comment #3)
> Guesses aren't really accepted here.
> 
> Fetch 7.2.9, revert the patch, confirm everything's working fine.

Follow up on the test. Fresh install, 7.2.9-200.fc44.

First boot, clean command line, no thunderbolt.host_reset. Dock connected
during the session, router 0-3 (17ef:118d) and the retimer enumerated
fine. Poweroff hung again, stuck on the firmware splash, needed a hard
power off. The journal from that boot ends at

systemd-shutdown[1]: Syncing filesystems and block devices.
systemd-shutdown[1]: Sending SIGTERM to remaining processes...
systemd-journald[789]: Journal stopped

so userspace finished cleanly (poweroff.target reached, filesystems
synced) and the hang happens after journald is gone. pstore caught
nothing (Fedora has CONFIG_PSTORE_CONSOLE off, and a hang is not a panic
anyway), so the last kernel messages are not recoverable from disk. I
kept the full kernel log of that session, can attach it if useful.

Then booted the same 7.2.9 kernel with thunderbolt.host_reset=0, dock
connected for the whole session, and poweroff works cleanly again.

Summary: stock 7.2.9 hangs on poweroff once the 40BF dock has been
connected, same 7.2.9 with thunderbolt.host_reset=0 powers off fine. So
the parameter still fully works around the issue on 7.2.9, same behavior
as 7.2.5 to 7.2.8.

If it helps I can set up netconsole to a second machine and capture the
console during the hanging poweroff, that should show exactly where the
shutdown path stops. Also happy to test any patch or scratch build.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.