btrtl/btusb: RTL8761BU enters unrecoverable USB reset storm after autosuspend

[Barak Pearlmutter <barak@cs.nuim.ie>]

Reporting a bug where a Realtek RTL8761BU Bluetooth USB adapter enters an
infinite reset loop (~every 2 seconds) after coming out of USB autosuspend,
from which it cannot recover without a physical power cycle (unplugging
and replugging the dongle).

Device:
  Realtek Bluetooth 5.1 Radio
  USB ID: 0bda:a725 (RTL8761BU)
  Firmware: rtl_bt/rtl8761bu_fw.bin, version 0xdfc6d922
  Driver: btusb/btrtl (in-tree)
  Kernel: 7.2.15+deb14-amd64 (Debian 13 / linux-image-amd64)
  Note: this is a Debian-patched kernel; Debian-specific patches may be
  relevant, though the btusb/btrtl driver is unlikely to be heavily patched.
  USB autosuspend delay at time of failure: 2 seconds (default)

Failure sequence:

The adapter is idle (nothing paired/connected).  After ~2 seconds the
kernel autosuspends the USB device.  On the next activity, the firmware
fails to respond to register reads (-ETIMEDOUT / error -110):

  Bluetooth: hci0: command 0xfc61 tx timeout
  Bluetooth: hci0: RTL: Failed to generate devcoredump
  Bluetooth: hci0: Resetting usb device.
  Bluetooth: hci0: RTL: RTL: Read reg16 failed (-110)
  usb 3-3: reset full-speed USB device number 2 using xhci_hcd

This repeats every ~2 seconds indefinitely.  The driver's own USB reset
does not clear the firmware hang state.  Workarounds attempted without
success:

  - sudo usbreset 0bda:a725       (USB reset, same as driver does)
  - driver unbind/rebind           (echo "3-3" to unbind then bind)
  - deauthorize/reauthorize        (echo 0/1 to authorized)

None of these recover the device.  Only a physical unplug/replug works.

Secondary effect:

The BT reset storm generates substantial USB bus traffic on a shared
xhci_hcd controller.  On this machine a companion RTL8822BU WiFi adapter
(0bda:b812) on a different USB port begins emitting "failed to get tx
report from firmware" kernel messages within 21 seconds of the BT storm
starting, and the WiFi interface eventually deauthenticates (~20 minutes
later).  Fixing the BT autosuspend issue therefore also fixes spurious
WiFi failures.

Workaround:

Disabling USB autosuspend for the device via udev prevents the failure
from occurring in the first place:

  ACTION=="add", SUBSYSTEM=="usb", ATTRS{idVendor}=="0bda",
    ATTRS{idProduct}=="a725", ATTR{power/autosuspend}="-1"

With this rule in place and the device freshly power-cycled the adapter
initialises cleanly and has remained stable.

Possibly related: Red Hat bug #2521504 describes an RTL8761BU (TP-Link
UB500, 2357:0604) entering an infinite USB reset loop, but with a
different error (0x200c timeout rather than "Read reg16 failed (-110)")
and reportedly fixed in kernel 7.1.13.  Our kernel is 7.2.15 and the
bug is present, suggesting this may be a distinct failure mode, or the
fix was not complete.

Log excerpt (storm onset):

  Nov 06 06:49:50 kernel: Bluetooth: hci0: command 0xfc61 tx timeout
  Nov 06 06:49:50 kernel: Bluetooth: hci0: RTL: Failed to generate devcoredump
  Nov 06 06:49:50 kernel: Bluetooth: hci0: Resetting usb device.
  Nov 06 06:49:50 kernel: Bluetooth: hci0: RTL: RTL: Read reg16 failed (-110)
  Nov 06 06:49:51 kernel: usb 3-3: reset full-speed USB device number 2 using xhci_hcd
  Nov 06 06:49:53 kernel: Bluetooth: hci0: command 0xfc61 tx timeout
  Nov 06 06:49:53 kernel: Bluetooth: hci0: RTL: Failed to generate devcoredump
  Nov 06 06:49:53 kernel: Bluetooth: hci0: Resetting usb device.
  Nov 06 06:49:53 kernel: Bluetooth: hci0: RTL: RTL: Read reg16 failed (-110)
  Nov 06 06:49:53 kernel: usb 3-3: reset full-speed USB device number 2 using xhci_hcd
  [continues every ~2 seconds for 20+ minutes until manual intervention]

Log excerpt (clean initialisation after power cycle, with autosuspend fix applied):

  kernel: usb 3-3: New USB device found, idVendor=0bda, idProduct=a725, bcdDevice= 2.00
  kernel: usb 3-3: Product: Bluetooth 5.1 Radio
  kernel: usb 3-3: Manufacturer: Realtek
  kernel: Bluetooth: hci0: RTL: loading rtl_bt/rtl8761bu_fw.bin
  kernel: Bluetooth: hci0: RTL: loading rtl_bt/rtl8761bu_config.bin
  kernel: Bluetooth: hci0: RTL: fw version 0xdfc6d922
  kernel: Bluetooth: MGMT ver 1.23
  [no further errors]

Happy to test a patched kernel or provide further information.

Barak Pearlmutter <barak@cs.nuim.ie>