[Bug 216766] New: USB enumeration lockup

[Bug 216766] New: USB enumeration lockup

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216766

            Bug ID: 216766
           Summary: USB enumeration lockup
           Product: Drivers
           Version: 2.5
    Kernel Version: 5.19.16 and 6.0.10
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: USB
          Assignee: drivers_usb@kernel-bugs.kernel.org
          Reporter: jan.burgmeier@gmx.de
        Regression: No

Hi,

Environment:
> Lenovo ThinkPad T14s Gen 1, model 20UJS00K00
> Fedora 37

after resuming from suspend the gnome-shell/gdm hangs because of fprintd and I
was able to track this down to usb device enumeration hanging on reading the
"descriptors" sysfs file. Calls to lsusb also hang on the same file.

The error can be reproduced with all recend kernels, tested with various Fedora
Kernels, from:
> Linux jbu-laptop 5.19.16-301.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Oct 21
> 15:55:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
to:
> Linux jbu-laptop 6.0.10-300.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Nov 26
> 16:55:13 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux


Here is the backtrace from the hanging usb read:
> (gdb) bt
> #0  0x00007ffff7e77fac in read () from /lib64/libc.so.6
> #1  0x00007ffff7f90885 in read (__nbytes=256, __buf=<optimized out>, __fd=8)
> at /usr/include/bits/unistd.h:38
> #2  initialize_device (dev=dev@entry=0x5555555a87c0, busnum=busnum@entry=4
> '\004', devaddr=devaddr@entry=1 '\001',
> sysfs_dir=sysfs_dir@entry=0x5555555a87a0 "usb4",
> wrapped_fd=wrapped_fd@entry=-1) at os/linux_usbfs.c:967
> #3  0x00007ffff7f92989 in linux_enumerate_device (ctx=0x55555559f610,
> busnum=<optimized out>, devaddr=<optimized out>, sysfs_dir=0x5555555a87a0
> "usb4") at os/linux_usbfs.c:1117
> #4  0x00007ffff7f93115 in linux_udev_scan_devices (ctx=0x55555559f610) at
> os/linux_udev.c:299
> #5  linux_scan_devices (ctx=0x55555559f610) at os/linux_usbfs.c:458
> #6  op_init (ctx=ctx@entry=0x55555559f610) at os/linux_usbfs.c:410
> #7  0x00007ffff7f93e79 in libusb_init (ctx=ctx@entry=0x7fffffffe138) at
> /usr/src/debug/libusb1-1.0.25-9.fc37.x86_64/libusb/core.c:2353
> #8  0x0000555555565e23 in main (argc=<optimized out>, argv=0x7fffffffe398) at
> /usr/src/debug/usbutils-014-3.fc37.x86_64/lsusb.c:3895

USB4 seems to be the integrated card reader:
> [root@jbu-laptop ~]# lsusb
> Bus 007 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
> Bus 006 Device 003: ID 8087:0029 Intel Corp. AX200 Bluetooth
> Bus 006 Device 002: ID 06cb:00bd Synaptics, Inc. Prometheus MIS Touch
> Fingerprint Reader
> Bus 006 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
> Bus 005 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
> Bus 004 Device 002: ID 058f:9540 Alcor Micro Corp. AU9540 Smartcard Reader
> Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
> Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
> Bus 002 Device 002: ID 13d3:5405 IMC Networks Integrated Camera
> Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
> Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The repoduction is not that easy and I'm not able to reliably reproduce the
issue mostly it works if I do the following:
 1. Connect to docking station with Monitors, Headset and webcam attached
 2. Suspend while connected to the docking station
 3. Disconnect from the docking station
 4. Resume while disconnected from the docking station

Please let me know if you need any further logs I can gather them if the next
hang occures.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 216766] USB enumeration lockup

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216766

--- Comment #1 from Alan Stern (stern@rowland.harvard.edu) ---
Please attach a copy of the dmesg log following one of these hangs.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 216766] USB enumeration lockup

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216766

--- Comment #2 from Jan Burgmeier (jan.burgmeier@gmx.de) ---
Created attachment 303343
  --> https://bugzilla.kernel.org/attachment.cgi?id=303343&action=edit
dmesg output after hang

Attached dmesg output

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 216766] USB enumeration lockup

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216766

--- Comment #3 from Alan Stern (stern@rowland.harvard.edu) ---
Judging from the log, it looks like the problem is related to the xHCI
controller at 0000:06:00.3 (although another possibility is PCI, since the log
includes several PCI error messages).  The problem seems to begin when all the
USB devices connected to that controller disappear during the second system
resume and never show up again.

I suggest you send an email describing the problem (include a pointer to this
Bugzilla entry) to the linux-usb@vger.kernel.org mailing list and CC: the xHCI
maintainer: Mathias Nyman <mathias.nyman@intel.com>.  He'll probably have some
requests for more tests you can run.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 216766] USB enumeration lockup

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216766

--- Comment #4 from Jan Burgmeier (jan.burgmeier@gmx.de) ---
Thanks for the fast response and analyzing the dmesg output, I will contact the
mailing list directly. I assume the xHCI controller is the one where the
docking station is connected via USB-C.

The used Docking station is:
> Lenovo ThinkPad Universal USB-C Dockingstation

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 216766] USB enumeration lockup

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216766

Mathias Nyman (mathias.nyman@linux.intel.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mathias.nyman@linux.intel.c
                   |                            |om

--- Comment #5 from Mathias Nyman (mathias.nyman@linux.intel.com) ---
This could also be related to USB Type-C port setting in resume.

I'm guessing this dock uses DP Alt-mode, with both Display port
and USB 3 over Type-C.

In both resume cases there are some error related to the display:

[drm:dm_late_init [amdgpu]] *ERROR* DM_MST: Failed to start MST
[drm:amdgpu_device_ip_late_init [amdgpu]] *ERROR* late_init of IP block <dm>
failed -5

After this all usb devices on usb4 and usb5 buses disconnect (xHCI
0000:06:00.3:)
There are no xhci errors in the log, just disconnects, as if Type-C port was
muxed away from xHCI.

A bit later we see DP Alt mode entry after first resume, and usb devices are
enumerated again.
[  136.881649] [drm] DP Alt mode state on HPD: 1
[  137.153415] [drm] DM_MST: starting TM on aconnector: 00000000018c53ab [id:
89]

After second resume we don't se this DP Alt mode entry in the log, and usb
devices remain disconnected.

Not that familiar with Type-C settings, could be AMD specific

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 216766] USB enumeration lockup

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216766

--- Comment #6 from Mathias Nyman (mathias.nyman@linux.intel.com) ---
But reading usbfs shouldn't block forever either, so there's something else
wrong as well.
Maybe we fails to give back an URB, or retry reading all the disconnected
devices forever.

Adding xhci and usb core dynamic debug could show more info about this.
before suspeding could you as root add:

mount -t debugfs none /sys/kernel/debug
echo 'module xhci_hcd =p' >/sys/kernel/debug/dynamic_debug/control
echo 'module usbcore =p' >/sys/kernel/debug/dynamic_debug/control

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 216766] USB enumeration lockup

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216766

--- Comment #7 from Jan Burgmeier (jan.burgmeier@gmx.de) ---
Created attachment 303360
  --> https://bugzilla.kernel.org/attachment.cgi?id=303360&action=edit
dmesg usb debug weird state

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 216766] USB enumeration lockup

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216766

--- Comment #8 from Jan Burgmeier (jan.burgmeier@gmx.de) ---
Created attachment 303361
  --> https://bugzilla.kernel.org/attachment.cgi?id=303361&action=edit
dmesg with debug multiple suspend resume

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 216766] USB enumeration lockup

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=216766

--- Comment #9 from Jan Burgmeier (jan.burgmeier@gmx.de) ---
Hi Matthias,

thanks for your feedback. You are right I have also 2 Monitors connected to the
docking station.

I enabled the logging and tried to reproduce the error. The behavior seems to
be a bit different, the error seems to happen a bit less but after resuming
when reconnecting the docking station I need to connect it twice on the first
connect the monitors are not recognized.

After some tries I was able to reproduce the error, multiple suspend and
resumes where needed, it seems that a longer suspend time increases the chance
to get the error:
>> https://bugzilla.kernel.org/attachment.cgi?id=303361

I had also one try which was even weirder, where the system seemed not to go
into suspend because the backlight stayed on but after ?resume? the error
occurred:
>> https://bugzilla.kernel.org/attachment.cgi?id=303360

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.