(no subject)

[Mathieu Fluhr]

Hi all,

I'm reporting a regression that causes a hard platform reset on my workstation,
But, before digging into the technical details, I would like to first re-trace
how I came to this particular commit.

As an AOSP developer I am compiling daily different versions of AOSP on it,
mostly building an Android (Automotive) emulator for quick "code-build-test"
runs. I use Ubuntu 24.04 LTS for that, updating to the latest HWE kernels that
Ubuntu provide there (6.8, 6.11, 6.14 etc. until 7.0 recently), with no manual
modifications applied.

3 weeks ago, I needed to analyze and issue happening inside Android 11,
building an emulator for a simple Android phone [1]. But when I started using
this emulator, first with the latest 7.0 kernel I noticed several hard crashes,
sometimes just freezing my workstation (with fans full on, but sometimes fans
full off), but also sometimes automatically rebooting it.

After a few days of deep investigations (To be honest, I first suspected an
issue with the nivida driver), I found out that a pattern to reproduce this
quickly was to let the computer idle with the emulator running. The crash was
always occurring under 20/25 minutes, most of the time letting it idle for less
than 10 was even sufficient.

This made me a bit curious, and looking a bit deeper (and with a little help
of AI) I was able to find a workaround. forbidding my CPU to enter C2 state,
using "processor.max_cstate=1" argument. Using this, I was not able to
reproduce the crash for more than an hour, but I did not pursue there very
much: As a developer I hate workarounds :)

I instead started to get "back in time" on a clean and up-to-date fresh Ubuntu
24.04 install, reverting back to "good old" kernel versions, since I could not
convince myself that first my CPU was dying and second that the issue has always
been there. I confirmed that 6.8 and 6.11 (again Ubuntu HWE packages) work
fine, but the crash was reproducible using 6.14 and above.

I then got my hands dirty, and started to test different mainline kernel
prebuilt packages up to a point where I confirmed that 6.12.35 worked fine, but
6.12.40 not. I then bisected both versions, ensuring a good case meant the
emulator was idling without any crash for 1 hour minimum. This lead at the end
to the following commit:

  aec11e5f9c452ef64e2c113637ab89a67a5ceb62
  ("usb: hub: fix detection of high tier USB3 devices behind suspended hubs")
  [mainline: 8f5b7e2bec1c36578fdaa74a6951833541103e27]

Being quite astonished that something related to C2 state was triggered by an
USB patch, I then tested the latest 7.0 kernel, this time using the
"usbcore.autosuspend=-1" argument instead. To my surprise, I could not
reproduce the crash, even with the exact same emulator idling for 2 hours.

Also, something very astonishing, that I still cannot fully understand today:
 1. "priming" my system with a 30 seconds (!) run of a modern Android
     emulator [2] cleared the issue: After closing the 15 emulator and starting
     the 11, I could let it idle for again more than an hour. It seems even not
     be related to the 'kvm' kernel modules, since removing the module and
     re-inserting it between both emulator did not change a thing.
 2. A few times (I did not really invest debugging this TBH), the crash even
    occurred shortly (2-3 minutes) after I closed the Android 11 emulator.

Now, the technical details of my setup...

Hardware / software
===================

CPU:        AMD Ryzen Threadripper 7970X (Storm Peak, family 19h)
Microcode:  0x0a10810c
Board:      ASUS Pro WS TRX50-SAGE WIFI, BIOS 1203 and 1317 (both affected)
Memory:     128 GB DDR5 ECC RDIMM @ 4800 MT/s (JEDEC, no XMP/EXPO)
GPU:        NVIDIA RTX 3080 Ti (also reproduced with RTX 5080)
Distro:     Ubuntu 24.04 and 26.04 with 595-open NVIDIA driver
USB:        8 onboard xHCI controllers; only a USB keyboard and mouse
            attached

Error signature
===============

On the boot following each crash I could always see the following lines in
the dmesg logs:
---8<-------------------------------------------------------------------------
  x86/amd: Previous system reset reason [0x88000800]: an uncorrected
           error caused a data fabric sync flood event
  x86/amd: Previous system reset reason [0x88000800]: a software sync
           flood event occurred
---8<-------------------------------------------------------------------------

When I was fortunate enough and had an automatic reboot, this was also inside:
---8<-------------------------------------------------------------------------
  [Hardware Error]: event severity: fatal
  [Hardware Error]:   section_type: IA32/X64 processor error
  [Hardware Error]:    Error Structure Type: cache error
  [Hardware Error]:    Check Information: 0x000000000602001f
  [Hardware Error]:     Transaction Type: 2, Generic
  [Hardware Error]:     Level: 0
  [Hardware Error]:     Processor Context Corrupt: true
  [Hardware Error]:     Uncorrected: true
  mce: [Hardware Error]: CPU 34: Machine Check: 0 Bank 5: aea0000000000108
  mce: [Hardware Error]: TSC 0 ADDR 1ffffff98e8ee38 MISC d0150fff00000000
       SYND 4d000000 IPID 500b020049b00
---8<-------------------------------------------------------------------------

The signature is bit-identical across every occurrence except for the
reporting CPU, which varies (observed on CPUs 1, 3, 7, 16, 32, 34, 35).


Reproducer
==========

1. Boot an affected kernel with default idle settings (C2 available,
   no max_cstate restriction).
2. Start an old and/or 32/64-bit Android emulator (QEMU/KVM guest) and leave it
   idle on the launcher screen. Nothing else running.
3. System hard-resets within 20 minutes.

Under sustained CPU load the fault never occurs; it requires the system to be
idle. turbostat confirms ~99% C2 residency across all cores in the crashing
condition.


Bisection
=========

---8<-------------------------------------------------------------------------
git bisect start
# status: waiting for both good and bad commits
# good: [783cd2c3dca8b6c434e955b84c20c8940588dc68] Linux 6.12.35
git bisect good 783cd2c3dca8b6c434e955b84c20c8940588dc68
# status: waiting for bad commit, 1 good commit known
# bad: [d90ecb2b1308b3e362ec4c21ff7cf0a051b445df] Linux 6.12.40
git bisect bad d90ecb2b1308b3e362ec4c21ff7cf0a051b445df
# good: [3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc] ata: libata-acpi:
Do not assume 40 wire cable if no devices are enabled
git bisect good 3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc
# good: [bbd385b65f9e56cab1243e753510a99a59110083] net/mlx5e: Add new
prio for promiscuous mode
git bisect good bbd385b65f9e56cab1243e753510a99a59110083
# good: [2a76bc2b24ed889a689fb1c9015307bf16aafb5b] smb: client: fix
use-after-free in crypt_message when using async crypto
git bisect good 2a76bc2b24ed889a689fb1c9015307bf16aafb5b
# good: [95a13b0a6b042ac5333b67f59af1dffe7b71137b] riscv:
traps_misaligned: properly sign extend value in misaligned load
handler
git bisect good 95a13b0a6b042ac5333b67f59af1dffe7b71137b
# good: [40b5b4ba8ed87c0bfb6268c10589777652ebde4c] drm/mediatek: Add
wait_event_timeout when disabling plane
git bisect good 40b5b4ba8ed87c0bfb6268c10589777652ebde4c
# bad: [affb46db59f908474a211f23953c3b9109f0d647] net: libwx: fix
multicast packets received count
git bisect bad affb46db59f908474a211f23953c3b9109f0d647
# good: [ee56da95f8962b86fec4ef93f866e64c8d025a58] btrfs: fix block
group refcount race in btrfs_create_pending_block_groups()
git bisect good ee56da95f8962b86fec4ef93f866e64c8d025a58
# bad: [bf71baa3cfe754c6e99ad62711b9fb3e3da33517] usb: hub: Fix
flushing of delayed work used for post resume purposes
git bisect bad bf71baa3cfe754c6e99ad62711b9fb3e3da33517
# bad: [e11359640090ab26e4153cce583f50e0b8979f0d] usb: hub: Fix
flushing and scheduling of delayed work that tunes runtime pm
git bisect bad e11359640090ab26e4153cce583f50e0b8979f0d
# bad: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb: hub: fix
detection of high tier USB3 devices behind suspended hubs
git bisect bad aec11e5f9c452ef64e2c113637ab89a67a5ceb62
# first bad commit: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb:
hub: fix detection of high tier USB3 devices behind suspended hubs
---8<-------------------------------------------------------------------------

(Please note that I did not perform a full "revert test", since the aec11e5f9c45
commit could not be cleanly reverted on both 6.12.40 and 7.0.0)


Workarounds
===========

Either of these prevents the crash on an affected kernel:
  usbcore.autosuspend=-1     (disables USB runtime PM)
  processor.max_cstate=1     (prevents C2 entry)


Ruled out
=========

- GPU/driver: reproduced with RTX 5080 (595-open module) and RTX 3080 Ti
              (both 595-open and 595 proprietary modules)
- ASUS USB4 PCIe add-in card: reproduces in ~5 min with the card physically
                              removed
- AVIC: kvm_amd avic=N on both good and bad kernels
- TSA mitigation: tsa=off verified applied
  (/sys/devices/system/cpu/vulnerabilities/tsa = "Vulnerable"), still crashes
- Memory: stock JEDEC 4800, no XMP/EXPO, no ECC errors logged
  (ras-mc-ctl reports zero CE/UE)
- Thermal/power: ~47-50 C at time of crash, system idle; 1600W Seasonic PSU
- Hardware health: multi-hour AOSP compiles (all 64 threads, heavy cache
  load) and repeated GPU stress runs complete without error


I hope this gives you enough details to start looking at what could cause this
weird behavior. Just FYI, when asked about hardware damage, AI suggested more
something like "a CPU-level microcode erratum in the deep-idle path on this
platform", but being old-school, I tend to always triple-check what AI tells
before claiming it myself :)

Thanks and Kind Regards,
Mathieu

------
[1] using android-11.0.0_r46 release tag and "sdk_phone_x86_64" as lunch target
[3] using android-15.0.0_r32 release tag and "sdk_phone64_x86_64" as lunch
    target this time, since Google only releases 64-bit only today

Re:

[Mathieu Fluhr]

[REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers
uncorrected MCE / data fabric sync flood on Threadripper 7970X
(bisected to aec11e5f9c45)

.... this time with the correct subject too.
(apologies for that stupid mistake)

On Sun, Aug 23, 2026 at 11:55 AM Mathieu Fluhr <mathieu.fluhr@gmail.com> wrote:
>
> Hi all,
>
> I'm reporting a regression that causes a hard platform reset on my workstation,
> But, before digging into the technical details, I would like to first re-trace
> how I came to this particular commit.
>
> As an AOSP developer I am compiling daily different versions of AOSP on it,
> mostly building an Android (Automotive) emulator for quick "code-build-test"
> runs. I use Ubuntu 24.04 LTS for that, updating to the latest HWE kernels that
> Ubuntu provide there (6.8, 6.11, 6.14 etc. until 7.0 recently), with no manual
> modifications applied.
>
> 3 weeks ago, I needed to analyze and issue happening inside Android 11,
> building an emulator for a simple Android phone [1]. But when I started using
> this emulator, first with the latest 7.0 kernel I noticed several hard crashes,
> sometimes just freezing my workstation (with fans full on, but sometimes fans
> full off), but also sometimes automatically rebooting it.
>
> After a few days of deep investigations (To be honest, I first suspected an
> issue with the nivida driver), I found out that a pattern to reproduce this
> quickly was to let the computer idle with the emulator running. The crash was
> always occurring under 20/25 minutes, most of the time letting it idle for less
> than 10 was even sufficient.
>
> This made me a bit curious, and looking a bit deeper (and with a little help
> of AI) I was able to find a workaround. forbidding my CPU to enter C2 state,
> using "processor.max_cstate=1" argument. Using this, I was not able to
> reproduce the crash for more than an hour, but I did not pursue there very
> much: As a developer I hate workarounds :)
>
> I instead started to get "back in time" on a clean and up-to-date fresh Ubuntu
> 24.04 install, reverting back to "good old" kernel versions, since I could not
> convince myself that first my CPU was dying and second that the issue has always
> been there. I confirmed that 6.8 and 6.11 (again Ubuntu HWE packages) work
> fine, but the crash was reproducible using 6.14 and above.
>
> I then got my hands dirty, and started to test different mainline kernel
> prebuilt packages up to a point where I confirmed that 6.12.35 worked fine, but
> 6.12.40 not. I then bisected both versions, ensuring a good case meant the
> emulator was idling without any crash for 1 hour minimum. This lead at the end
> to the following commit:
>
>   aec11e5f9c452ef64e2c113637ab89a67a5ceb62
>   ("usb: hub: fix detection of high tier USB3 devices behind suspended hubs")
>   [mainline: 8f5b7e2bec1c36578fdaa74a6951833541103e27]
>
> Being quite astonished that something related to C2 state was triggered by an
> USB patch, I then tested the latest 7.0 kernel, this time using the
> "usbcore.autosuspend=-1" argument instead. To my surprise, I could not
> reproduce the crash, even with the exact same emulator idling for 2 hours.
>
> Also, something very astonishing, that I still cannot fully understand today:
>  1. "priming" my system with a 30 seconds (!) run of a modern Android
>      emulator [2] cleared the issue: After closing the 15 emulator and starting
>      the 11, I could let it idle for again more than an hour. It seems even not
>      be related to the 'kvm' kernel modules, since removing the module and
>      re-inserting it between both emulator did not change a thing.
>  2. A few times (I did not really invest debugging this TBH), the crash even
>     occurred shortly (2-3 minutes) after I closed the Android 11 emulator.
>
> Now, the technical details of my setup...
>
> Hardware / software
> ===================
>
> CPU:        AMD Ryzen Threadripper 7970X (Storm Peak, family 19h)
> Microcode:  0x0a10810c
> Board:      ASUS Pro WS TRX50-SAGE WIFI, BIOS 1203 and 1317 (both affected)
> Memory:     128 GB DDR5 ECC RDIMM @ 4800 MT/s (JEDEC, no XMP/EXPO)
> GPU:        NVIDIA RTX 3080 Ti (also reproduced with RTX 5080)
> Distro:     Ubuntu 24.04 and 26.04 with 595-open NVIDIA driver
> USB:        8 onboard xHCI controllers; only a USB keyboard and mouse
>             attached
>
> Error signature
> ===============
>
> On the boot following each crash I could always see the following lines in
> the dmesg logs:
> ---8<-------------------------------------------------------------------------
>   x86/amd: Previous system reset reason [0x88000800]: an uncorrected
>            error caused a data fabric sync flood event
>   x86/amd: Previous system reset reason [0x88000800]: a software sync
>            flood event occurred
> ---8<-------------------------------------------------------------------------
>
> When I was fortunate enough and had an automatic reboot, this was also inside:
> ---8<-------------------------------------------------------------------------
>   [Hardware Error]: event severity: fatal
>   [Hardware Error]:   section_type: IA32/X64 processor error
>   [Hardware Error]:    Error Structure Type: cache error
>   [Hardware Error]:    Check Information: 0x000000000602001f
>   [Hardware Error]:     Transaction Type: 2, Generic
>   [Hardware Error]:     Level: 0
>   [Hardware Error]:     Processor Context Corrupt: true
>   [Hardware Error]:     Uncorrected: true
>   mce: [Hardware Error]: CPU 34: Machine Check: 0 Bank 5: aea0000000000108
>   mce: [Hardware Error]: TSC 0 ADDR 1ffffff98e8ee38 MISC d0150fff00000000
>        SYND 4d000000 IPID 500b020049b00
> ---8<-------------------------------------------------------------------------
>
> The signature is bit-identical across every occurrence except for the
> reporting CPU, which varies (observed on CPUs 1, 3, 7, 16, 32, 34, 35).
>
>
> Reproducer
> ==========
>
> 1. Boot an affected kernel with default idle settings (C2 available,
>    no max_cstate restriction).
> 2. Start an old and/or 32/64-bit Android emulator (QEMU/KVM guest) and leave it
>    idle on the launcher screen. Nothing else running.
> 3. System hard-resets within 20 minutes.
>
> Under sustained CPU load the fault never occurs; it requires the system to be
> idle. turbostat confirms ~99% C2 residency across all cores in the crashing
> condition.
>
>
> Bisection
> =========
>
> ---8<-------------------------------------------------------------------------
> git bisect start
> # status: waiting for both good and bad commits
> # good: [783cd2c3dca8b6c434e955b84c20c8940588dc68] Linux 6.12.35
> git bisect good 783cd2c3dca8b6c434e955b84c20c8940588dc68
> # status: waiting for bad commit, 1 good commit known
> # bad: [d90ecb2b1308b3e362ec4c21ff7cf0a051b445df] Linux 6.12.40
> git bisect bad d90ecb2b1308b3e362ec4c21ff7cf0a051b445df
> # good: [3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc] ata: libata-acpi:
> Do not assume 40 wire cable if no devices are enabled
> git bisect good 3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc
> # good: [bbd385b65f9e56cab1243e753510a99a59110083] net/mlx5e: Add new
> prio for promiscuous mode
> git bisect good bbd385b65f9e56cab1243e753510a99a59110083
> # good: [2a76bc2b24ed889a689fb1c9015307bf16aafb5b] smb: client: fix
> use-after-free in crypt_message when using async crypto
> git bisect good 2a76bc2b24ed889a689fb1c9015307bf16aafb5b
> # good: [95a13b0a6b042ac5333b67f59af1dffe7b71137b] riscv:
> traps_misaligned: properly sign extend value in misaligned load
> handler
> git bisect good 95a13b0a6b042ac5333b67f59af1dffe7b71137b
> # good: [40b5b4ba8ed87c0bfb6268c10589777652ebde4c] drm/mediatek: Add
> wait_event_timeout when disabling plane
> git bisect good 40b5b4ba8ed87c0bfb6268c10589777652ebde4c
> # bad: [affb46db59f908474a211f23953c3b9109f0d647] net: libwx: fix
> multicast packets received count
> git bisect bad affb46db59f908474a211f23953c3b9109f0d647
> # good: [ee56da95f8962b86fec4ef93f866e64c8d025a58] btrfs: fix block
> group refcount race in btrfs_create_pending_block_groups()
> git bisect good ee56da95f8962b86fec4ef93f866e64c8d025a58
> # bad: [bf71baa3cfe754c6e99ad62711b9fb3e3da33517] usb: hub: Fix
> flushing of delayed work used for post resume purposes
> git bisect bad bf71baa3cfe754c6e99ad62711b9fb3e3da33517
> # bad: [e11359640090ab26e4153cce583f50e0b8979f0d] usb: hub: Fix
> flushing and scheduling of delayed work that tunes runtime pm
> git bisect bad e11359640090ab26e4153cce583f50e0b8979f0d
> # bad: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb: hub: fix
> detection of high tier USB3 devices behind suspended hubs
> git bisect bad aec11e5f9c452ef64e2c113637ab89a67a5ceb62
> # first bad commit: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb:
> hub: fix detection of high tier USB3 devices behind suspended hubs
> ---8<-------------------------------------------------------------------------
>
> (Please note that I did not perform a full "revert test", since the aec11e5f9c45
> commit could not be cleanly reverted on both 6.12.40 and 7.0.0)
>
>
> Workarounds
> ===========
>
> Either of these prevents the crash on an affected kernel:
>   usbcore.autosuspend=-1     (disables USB runtime PM)
>   processor.max_cstate=1     (prevents C2 entry)
>
>
> Ruled out
> =========
>
> - GPU/driver: reproduced with RTX 5080 (595-open module) and RTX 3080 Ti
>               (both 595-open and 595 proprietary modules)
> - ASUS USB4 PCIe add-in card: reproduces in ~5 min with the card physically
>                               removed
> - AVIC: kvm_amd avic=N on both good and bad kernels
> - TSA mitigation: tsa=off verified applied
>   (/sys/devices/system/cpu/vulnerabilities/tsa = "Vulnerable"), still crashes
> - Memory: stock JEDEC 4800, no XMP/EXPO, no ECC errors logged
>   (ras-mc-ctl reports zero CE/UE)
> - Thermal/power: ~47-50 C at time of crash, system idle; 1600W Seasonic PSU
> - Hardware health: multi-hour AOSP compiles (all 64 threads, heavy cache
>   load) and repeated GPU stress runs complete without error
>
>
> I hope this gives you enough details to start looking at what could cause this
> weird behavior. Just FYI, when asked about hardware damage, AI suggested more
> something like "a CPU-level microcode erratum in the deep-idle path on this
> platform", but being old-school, I tend to always triple-check what AI tells
> before claiming it myself :)
>
> Thanks and Kind Regards,
> Mathieu
>
> ------
> [1] using android-11.0.0_r46 release tag and "sdk_phone_x86_64" as lunch target
> [3] using android-15.0.0_r32 release tag and "sdk_phone64_x86_64" as lunch
>     target this time, since Google only releases 64-bit only today

[REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Mathieu Fluhr]

On Sun, Aug 23, 2026 at 12:12 PM Mathieu Fluhr <mathieu.fluhr@gmail.com> wrote:
>
> [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers
> uncorrected MCE / data fabric sync flood on Threadripper 7970X
> (bisected to aec11e5f9c45)
>
> .... this time with the correct subject too.
> (apologies for that stupid mistake)
>
> On Sun, Aug 23, 2026 at 11:55 AM Mathieu Fluhr <mathieu.fluhr@gmail.com> wrote:
> >
> > Hi all,
> >
> > I'm reporting a regression that causes a hard platform reset on my workstation,
> > But, before digging into the technical details, I would like to first re-trace
> > how I came to this particular commit.
> >
> > As an AOSP developer I am compiling daily different versions of AOSP on it,
> > mostly building an Android (Automotive) emulator for quick "code-build-test"
> > runs. I use Ubuntu 24.04 LTS for that, updating to the latest HWE kernels that
> > Ubuntu provide there (6.8, 6.11, 6.14 etc. until 7.0 recently), with no manual
> > modifications applied.
> >
> > 3 weeks ago, I needed to analyze and issue happening inside Android 11,
> > building an emulator for a simple Android phone [1]. But when I started using
> > this emulator, first with the latest 7.0 kernel I noticed several hard crashes,
> > sometimes just freezing my workstation (with fans full on, but sometimes fans
> > full off), but also sometimes automatically rebooting it.
> >
> > After a few days of deep investigations (To be honest, I first suspected an
> > issue with the nivida driver), I found out that a pattern to reproduce this
> > quickly was to let the computer idle with the emulator running. The crash was
> > always occurring under 20/25 minutes, most of the time letting it idle for less
> > than 10 was even sufficient.
> >
> > This made me a bit curious, and looking a bit deeper (and with a little help
> > of AI) I was able to find a workaround. forbidding my CPU to enter C2 state,
> > using "processor.max_cstate=1" argument. Using this, I was not able to
> > reproduce the crash for more than an hour, but I did not pursue there very
> > much: As a developer I hate workarounds :)
> >
> > I instead started to get "back in time" on a clean and up-to-date fresh Ubuntu
> > 24.04 install, reverting back to "good old" kernel versions, since I could not
> > convince myself that first my CPU was dying and second that the issue has always
> > been there. I confirmed that 6.8 and 6.11 (again Ubuntu HWE packages) work
> > fine, but the crash was reproducible using 6.14 and above.
> >
> > I then got my hands dirty, and started to test different mainline kernel
> > prebuilt packages up to a point where I confirmed that 6.12.35 worked fine, but
> > 6.12.40 not. I then bisected both versions, ensuring a good case meant the
> > emulator was idling without any crash for 1 hour minimum. This lead at the end
> > to the following commit:
> >
> >   aec11e5f9c452ef64e2c113637ab89a67a5ceb62
> >   ("usb: hub: fix detection of high tier USB3 devices behind suspended hubs")
> >   [mainline: 8f5b7e2bec1c36578fdaa74a6951833541103e27]
> >
> > Being quite astonished that something related to C2 state was triggered by an
> > USB patch, I then tested the latest 7.0 kernel, this time using the
> > "usbcore.autosuspend=-1" argument instead. To my surprise, I could not
> > reproduce the crash, even with the exact same emulator idling for 2 hours.
> >
> > Also, something very astonishing, that I still cannot fully understand today:
> >  1. "priming" my system with a 30 seconds (!) run of a modern Android
> >      emulator [2] cleared the issue: After closing the 15 emulator and starting
> >      the 11, I could let it idle for again more than an hour. It seems even not
> >      be related to the 'kvm' kernel modules, since removing the module and
> >      re-inserting it between both emulator did not change a thing.
> >  2. A few times (I did not really invest debugging this TBH), the crash even
> >     occurred shortly (2-3 minutes) after I closed the Android 11 emulator.
> >
> > Now, the technical details of my setup...
> >
> > Hardware / software
> > ===================
> >
> > CPU:        AMD Ryzen Threadripper 7970X (Storm Peak, family 19h)
> > Microcode:  0x0a10810c
> > Board:      ASUS Pro WS TRX50-SAGE WIFI, BIOS 1203 and 1317 (both affected)
> > Memory:     128 GB DDR5 ECC RDIMM @ 4800 MT/s (JEDEC, no XMP/EXPO)
> > GPU:        NVIDIA RTX 3080 Ti (also reproduced with RTX 5080)
> > Distro:     Ubuntu 24.04 and 26.04 with 595-open NVIDIA driver
> > USB:        8 onboard xHCI controllers; only a USB keyboard and mouse
> >             attached
> >
> > Error signature
> > ===============
> >
> > On the boot following each crash I could always see the following lines in
> > the dmesg logs:
> > ---8<-------------------------------------------------------------------------
> >   x86/amd: Previous system reset reason [0x88000800]: an uncorrected
> >            error caused a data fabric sync flood event
> >   x86/amd: Previous system reset reason [0x88000800]: a software sync
> >            flood event occurred
> > ---8<-------------------------------------------------------------------------
> >
> > When I was fortunate enough and had an automatic reboot, this was also inside:
> > ---8<-------------------------------------------------------------------------
> >   [Hardware Error]: event severity: fatal
> >   [Hardware Error]:   section_type: IA32/X64 processor error
> >   [Hardware Error]:    Error Structure Type: cache error
> >   [Hardware Error]:    Check Information: 0x000000000602001f
> >   [Hardware Error]:     Transaction Type: 2, Generic
> >   [Hardware Error]:     Level: 0
> >   [Hardware Error]:     Processor Context Corrupt: true
> >   [Hardware Error]:     Uncorrected: true
> >   mce: [Hardware Error]: CPU 34: Machine Check: 0 Bank 5: aea0000000000108
> >   mce: [Hardware Error]: TSC 0 ADDR 1ffffff98e8ee38 MISC d0150fff00000000
> >        SYND 4d000000 IPID 500b020049b00
> > ---8<-------------------------------------------------------------------------
> >
> > The signature is bit-identical across every occurrence except for the
> > reporting CPU, which varies (observed on CPUs 1, 3, 7, 16, 32, 34, 35).
> >
> >
> > Reproducer
> > ==========
> >
> > 1. Boot an affected kernel with default idle settings (C2 available,
> >    no max_cstate restriction).
> > 2. Start an old and/or 32/64-bit Android emulator (QEMU/KVM guest) and leave it
> >    idle on the launcher screen. Nothing else running.
> > 3. System hard-resets within 20 minutes.
> >
> > Under sustained CPU load the fault never occurs; it requires the system to be
> > idle. turbostat confirms ~99% C2 residency across all cores in the crashing
> > condition.
> >
> >
> > Bisection
> > =========
> >
> > ---8<-------------------------------------------------------------------------
> > git bisect start
> > # status: waiting for both good and bad commits
> > # good: [783cd2c3dca8b6c434e955b84c20c8940588dc68] Linux 6.12.35
> > git bisect good 783cd2c3dca8b6c434e955b84c20c8940588dc68
> > # status: waiting for bad commit, 1 good commit known
> > # bad: [d90ecb2b1308b3e362ec4c21ff7cf0a051b445df] Linux 6.12.40
> > git bisect bad d90ecb2b1308b3e362ec4c21ff7cf0a051b445df
> > # good: [3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc] ata: libata-acpi:
> > Do not assume 40 wire cable if no devices are enabled
> > git bisect good 3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc
> > # good: [bbd385b65f9e56cab1243e753510a99a59110083] net/mlx5e: Add new
> > prio for promiscuous mode
> > git bisect good bbd385b65f9e56cab1243e753510a99a59110083
> > # good: [2a76bc2b24ed889a689fb1c9015307bf16aafb5b] smb: client: fix
> > use-after-free in crypt_message when using async crypto
> > git bisect good 2a76bc2b24ed889a689fb1c9015307bf16aafb5b
> > # good: [95a13b0a6b042ac5333b67f59af1dffe7b71137b] riscv:
> > traps_misaligned: properly sign extend value in misaligned load
> > handler
> > git bisect good 95a13b0a6b042ac5333b67f59af1dffe7b71137b
> > # good: [40b5b4ba8ed87c0bfb6268c10589777652ebde4c] drm/mediatek: Add
> > wait_event_timeout when disabling plane
> > git bisect good 40b5b4ba8ed87c0bfb6268c10589777652ebde4c
> > # bad: [affb46db59f908474a211f23953c3b9109f0d647] net: libwx: fix
> > multicast packets received count
> > git bisect bad affb46db59f908474a211f23953c3b9109f0d647
> > # good: [ee56da95f8962b86fec4ef93f866e64c8d025a58] btrfs: fix block
> > group refcount race in btrfs_create_pending_block_groups()
> > git bisect good ee56da95f8962b86fec4ef93f866e64c8d025a58
> > # bad: [bf71baa3cfe754c6e99ad62711b9fb3e3da33517] usb: hub: Fix
> > flushing of delayed work used for post resume purposes
> > git bisect bad bf71baa3cfe754c6e99ad62711b9fb3e3da33517
> > # bad: [e11359640090ab26e4153cce583f50e0b8979f0d] usb: hub: Fix
> > flushing and scheduling of delayed work that tunes runtime pm
> > git bisect bad e11359640090ab26e4153cce583f50e0b8979f0d
> > # bad: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb: hub: fix
> > detection of high tier USB3 devices behind suspended hubs
> > git bisect bad aec11e5f9c452ef64e2c113637ab89a67a5ceb62
> > # first bad commit: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb:
> > hub: fix detection of high tier USB3 devices behind suspended hubs
> > ---8<-------------------------------------------------------------------------
> >
> > (Please note that I did not perform a full "revert test", since the aec11e5f9c45
> > commit could not be cleanly reverted on both 6.12.40 and 7.0.0)
> >
> >
> > Workarounds
> > ===========
> >
> > Either of these prevents the crash on an affected kernel:
> >   usbcore.autosuspend=-1     (disables USB runtime PM)
> >   processor.max_cstate=1     (prevents C2 entry)
> >
> >
> > Ruled out
> > =========
> >
> > - GPU/driver: reproduced with RTX 5080 (595-open module) and RTX 3080 Ti
> >               (both 595-open and 595 proprietary modules)
> > - ASUS USB4 PCIe add-in card: reproduces in ~5 min with the card physically
> >                               removed
> > - AVIC: kvm_amd avic=N on both good and bad kernels
> > - TSA mitigation: tsa=off verified applied
> >   (/sys/devices/system/cpu/vulnerabilities/tsa = "Vulnerable"), still crashes
> > - Memory: stock JEDEC 4800, no XMP/EXPO, no ECC errors logged
> >   (ras-mc-ctl reports zero CE/UE)
> > - Thermal/power: ~47-50 C at time of crash, system idle; 1600W Seasonic PSU
> > - Hardware health: multi-hour AOSP compiles (all 64 threads, heavy cache
> >   load) and repeated GPU stress runs complete without error
> >
> >
> > I hope this gives you enough details to start looking at what could cause this
> > weird behavior. Just FYI, when asked about hardware damage, AI suggested more
> > something like "a CPU-level microcode erratum in the deep-idle path on this
> > platform", but being old-school, I tend to always triple-check what AI tells
> > before claiming it myself :)
> >
> > Thanks and Kind Regards,
> > Mathieu
> >
> > ------
> > [1] using android-11.0.0_r46 release tag and "sdk_phone_x86_64" as lunch target
> > [3] using android-15.0.0_r32 release tag and "sdk_phone64_x86_64" as lunch
> >     target this time, since Google only releases 64-bit only today

Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Mathieu Fluhr]

On Sun, Aug 23, 2026 at 12:37 PM Thorsten Leemhuis
<regressions@leemhuis.info> wrote:
>
> On 8/23/26 12:15, Mathieu Fluhr wrote:
> Thx for the report. Side note: mentioning the essence at the top before
> explaining backstory and details is usually the way better approach to
> get people interested in a bug report, especially if it's more that one
> or two paras.

Point taken :)

> Anyway, as you mentioned later and quoted below, your problems are
> caused by a 6.12.y backport of 8f5b7e2bec1c36 ("usb: hub: fix detection
> of high tier USB3 devices behind suspended hubs") [v6.16-rc5 (2025-07-06
> 23:10:26)], authored by Mathias Nyman (now CCed).
>
> There were at least three earlier reports about problems caused by that
> commit, all mentioning adb (which I assume you use, too):

Yes, looking at the reports, this looks definitely related.
What is happening when the emulator starts is that it also starts adbd
if not running on the Linux side, to communicate with the emulator
later on.

So, this could perfectly explain the persistent state after the
"Android 15 priming": Since the Android 15 repo contains a more recent
ADB, where the issue has been eventually wored around, it is started
firs, and later, when 11 starts, it does not kill it, restarting its
older adbd, but instead keep the new one running.

So, long story short, on my side, I was still looking at the "wrong
side" of the problem, trying to find what QEMU was doing wrong, where
in the end, it was more the interactions that adb was doing with the
USB bus during C2 state that was causing the issue.

>
> https://lore.kernel.org/all/qc0nhk9c6l0a08bkfeplrm3qjssgrjkvkp@sonic.net/
> https://lore.kernel.org/all/CAGQo=TJXef-QzbkFEpTcJ138j0k6WYnV4=tNY8Q7s3U7DUOY+Q@mail.gmail.com/
> https://bugzilla.kernel.org/show_bug.cgi?id=221103
>
> FWIW, https://bugzilla.kernel.org/show_bug.cgi?id=221103#c16 mentions
> the workaround you apparently found, too. The patch Mario (now CCed,
> just like the authors of the two mailed reports linked above; and
> Michael, too, who seems to be involved) proposed in the last comment of
> that ticket apparently didn't help. And no other fix is in sight, or am
> I missing something?
>
> Mathias reverting the culprit in mainline a option to resolve this (I
> assume it is not, but I'm asking just to be sure)?

Ok, here you lost me a little: Looking to all the links provided, yes,
this looks very similar to what I am experiencing. But (and correct me
if I am wrong)
1. reverting the "bad commit" is not really an option, since it was
originally authored for a specific purpose (other than annoying AMD
owners :)
2. the patch that Mario proposed is very recent, and would address the
issue not on the USB but on the PCI level?

Thanks again for the swift answer here and all the informative links!
I must admit that I researched a lot debugging this issue, but I never
thought about looking to the USB side of the problem, focusing more on
the CPU side.

Kind Regards,
Mathieu

>
> Ciao, Thorsten
>
> >>> As an AOSP developer I am compiling daily different versions of AOSP on it,
> >>> mostly building an Android (Automotive) emulator for quick "code-build-test"
> >>> runs. I use Ubuntu 24.04 LTS for that, updating to the latest HWE kernels that
> >>> Ubuntu provide there (6.8, 6.11, 6.14 etc. until 7.0 recently), with no manual
> >>> modifications applied.
> >>>
> >>> 3 weeks ago, I needed to analyze and issue happening inside Android 11,
> >>> building an emulator for a simple Android phone [1]. But when I started using
> >>> this emulator, first with the latest 7.0 kernel I noticed several hard crashes,
> >>> sometimes just freezing my workstation (with fans full on, but sometimes fans
> >>> full off), but also sometimes automatically rebooting it.
> >>>
> >>> After a few days of deep investigations (To be honest, I first suspected an
> >>> issue with the nivida driver), I found out that a pattern to reproduce this
> >>> quickly was to let the computer idle with the emulator running. The crash was
> >>> always occurring under 20/25 minutes, most of the time letting it idle for less
> >>> than 10 was even sufficient.
> >>>
> >>> This made me a bit curious, and looking a bit deeper (and with a little help
> >>> of AI) I was able to find a workaround. forbidding my CPU to enter C2 state,
> >>> using "processor.max_cstate=1" argument. Using this, I was not able to
> >>> reproduce the crash for more than an hour, but I did not pursue there very
> >>> much: As a developer I hate workarounds :)
> >>>
> >>> I instead started to get "back in time" on a clean and up-to-date fresh Ubuntu
> >>> 24.04 install, reverting back to "good old" kernel versions, since I could not
> >>> convince myself that first my CPU was dying and second that the issue has always
> >>> been there. I confirmed that 6.8 and 6.11 (again Ubuntu HWE packages) work
> >>> fine, but the crash was reproducible using 6.14 and above.
> >>>
> >>> I then got my hands dirty, and started to test different mainline kernel
> >>> prebuilt packages up to a point where I confirmed that 6.12.35 worked fine, but
> >>> 6.12.40 not. I then bisected both versions, ensuring a good case meant the
> >>> emulator was idling without any crash for 1 hour minimum. This lead at the end
> >>> to the following commit:
> >>>
> >>>   aec11e5f9c452ef64e2c113637ab89a67a5ceb62
> >>>   ("usb: hub: fix detection of high tier USB3 devices behind suspended hubs")
> >>>   [mainline: 8f5b7e2bec1c36578fdaa74a6951833541103e27]
> >>>
> >>> Being quite astonished that something related to C2 state was triggered by an
> >>> USB patch, I then tested the latest 7.0 kernel, this time using the
> >>> "usbcore.autosuspend=-1" argument instead. To my surprise, I could not
> >>> reproduce the crash, even with the exact same emulator idling for 2 hours.
> >>>
> >>> Also, something very astonishing, that I still cannot fully understand today:
> >>>  1. "priming" my system with a 30 seconds (!) run of a modern Android
> >>>      emulator [2] cleared the issue: After closing the 15 emulator and starting
> >>>      the 11, I could let it idle for again more than an hour. It seems even not
> >>>      be related to the 'kvm' kernel modules, since removing the module and
> >>>      re-inserting it between both emulator did not change a thing.
> >>>  2. A few times (I did not really invest debugging this TBH), the crash even
> >>>     occurred shortly (2-3 minutes) after I closed the Android 11 emulator.
> >>>
> >>> Now, the technical details of my setup...
> >>>
> >>> Hardware / software
> >>> ===================
> >>>
> >>> CPU:        AMD Ryzen Threadripper 7970X (Storm Peak, family 19h)
> >>> Microcode:  0x0a10810c
> >>> Board:      ASUS Pro WS TRX50-SAGE WIFI, BIOS 1203 and 1317 (both affected)
> >>> Memory:     128 GB DDR5 ECC RDIMM @ 4800 MT/s (JEDEC, no XMP/EXPO)
> >>> GPU:        NVIDIA RTX 3080 Ti (also reproduced with RTX 5080)
> >>> Distro:     Ubuntu 24.04 and 26.04 with 595-open NVIDIA driver
> >>> USB:        8 onboard xHCI controllers; only a USB keyboard and mouse
> >>>             attached
> >>>
> >>> Error signature
> >>> ===============
> >>>
> >>> On the boot following each crash I could always see the following lines in
> >>> the dmesg logs:
> >>> ---8<-------------------------------------------------------------------------
> >>>   x86/amd: Previous system reset reason [0x88000800]: an uncorrected
> >>>            error caused a data fabric sync flood event
> >>>   x86/amd: Previous system reset reason [0x88000800]: a software sync
> >>>            flood event occurred
> >>> ---8<-------------------------------------------------------------------------
> >>>
> >>> When I was fortunate enough and had an automatic reboot, this was also inside:
> >>> ---8<-------------------------------------------------------------------------
> >>>   [Hardware Error]: event severity: fatal
> >>>   [Hardware Error]:   section_type: IA32/X64 processor error
> >>>   [Hardware Error]:    Error Structure Type: cache error
> >>>   [Hardware Error]:    Check Information: 0x000000000602001f
> >>>   [Hardware Error]:     Transaction Type: 2, Generic
> >>>   [Hardware Error]:     Level: 0
> >>>   [Hardware Error]:     Processor Context Corrupt: true
> >>>   [Hardware Error]:     Uncorrected: true
> >>>   mce: [Hardware Error]: CPU 34: Machine Check: 0 Bank 5: aea0000000000108
> >>>   mce: [Hardware Error]: TSC 0 ADDR 1ffffff98e8ee38 MISC d0150fff00000000
> >>>        SYND 4d000000 IPID 500b020049b00
> >>> ---8<-------------------------------------------------------------------------
> >>>
> >>> The signature is bit-identical across every occurrence except for the
> >>> reporting CPU, which varies (observed on CPUs 1, 3, 7, 16, 32, 34, 35).
> >>>
> >>>
> >>> Reproducer
> >>> ==========
> >>>
> >>> 1. Boot an affected kernel with default idle settings (C2 available,
> >>>    no max_cstate restriction).
> >>> 2. Start an old and/or 32/64-bit Android emulator (QEMU/KVM guest) and leave it
> >>>    idle on the launcher screen. Nothing else running.
> >>> 3. System hard-resets within 20 minutes.
> >>>
> >>> Under sustained CPU load the fault never occurs; it requires the system to be
> >>> idle. turbostat confirms ~99% C2 residency across all cores in the crashing
> >>> condition.
> >>>
> >>>
> >>> Bisection
> >>> =========
> >>>
> >>> ---8<-------------------------------------------------------------------------
> >>> git bisect start
> >>> # status: waiting for both good and bad commits
> >>> # good: [783cd2c3dca8b6c434e955b84c20c8940588dc68] Linux 6.12.35
> >>> git bisect good 783cd2c3dca8b6c434e955b84c20c8940588dc68
> >>> # status: waiting for bad commit, 1 good commit known
> >>> # bad: [d90ecb2b1308b3e362ec4c21ff7cf0a051b445df] Linux 6.12.40
> >>> git bisect bad d90ecb2b1308b3e362ec4c21ff7cf0a051b445df
> >>> # good: [3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc] ata: libata-acpi:
> >>> Do not assume 40 wire cable if no devices are enabled
> >>> git bisect good 3ce57d493dd81ecc27033c7a918cc0d0ddaa8edc
> >>> # good: [bbd385b65f9e56cab1243e753510a99a59110083] net/mlx5e: Add new
> >>> prio for promiscuous mode
> >>> git bisect good bbd385b65f9e56cab1243e753510a99a59110083
> >>> # good: [2a76bc2b24ed889a689fb1c9015307bf16aafb5b] smb: client: fix
> >>> use-after-free in crypt_message when using async crypto
> >>> git bisect good 2a76bc2b24ed889a689fb1c9015307bf16aafb5b
> >>> # good: [95a13b0a6b042ac5333b67f59af1dffe7b71137b] riscv:
> >>> traps_misaligned: properly sign extend value in misaligned load
> >>> handler
> >>> git bisect good 95a13b0a6b042ac5333b67f59af1dffe7b71137b
> >>> # good: [40b5b4ba8ed87c0bfb6268c10589777652ebde4c] drm/mediatek: Add
> >>> wait_event_timeout when disabling plane
> >>> git bisect good 40b5b4ba8ed87c0bfb6268c10589777652ebde4c
> >>> # bad: [affb46db59f908474a211f23953c3b9109f0d647] net: libwx: fix
> >>> multicast packets received count
> >>> git bisect bad affb46db59f908474a211f23953c3b9109f0d647
> >>> # good: [ee56da95f8962b86fec4ef93f866e64c8d025a58] btrfs: fix block
> >>> group refcount race in btrfs_create_pending_block_groups()
> >>> git bisect good ee56da95f8962b86fec4ef93f866e64c8d025a58
> >>> # bad: [bf71baa3cfe754c6e99ad62711b9fb3e3da33517] usb: hub: Fix
> >>> flushing of delayed work used for post resume purposes
> >>> git bisect bad bf71baa3cfe754c6e99ad62711b9fb3e3da33517
> >>> # bad: [e11359640090ab26e4153cce583f50e0b8979f0d] usb: hub: Fix
> >>> flushing and scheduling of delayed work that tunes runtime pm
> >>> git bisect bad e11359640090ab26e4153cce583f50e0b8979f0d
> >>> # bad: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb: hub: fix
> >>> detection of high tier USB3 devices behind suspended hubs
> >>> git bisect bad aec11e5f9c452ef64e2c113637ab89a67a5ceb62
> >>> # first bad commit: [aec11e5f9c452ef64e2c113637ab89a67a5ceb62] usb:
> >>> hub: fix detection of high tier USB3 devices behind suspended hubs
> >>> ---8<-------------------------------------------------------------------------
> >>>
> >>> (Please note that I did not perform a full "revert test", since the aec11e5f9c45
> >>> commit could not be cleanly reverted on both 6.12.40 and 7.0.0)
> >>>
> >>>
> >>> Workarounds
> >>> ===========
> >>>
> >>> Either of these prevents the crash on an affected kernel:
> >>>   usbcore.autosuspend=-1     (disables USB runtime PM)
> >>>   processor.max_cstate=1     (prevents C2 entry)
> >>>
> >>>
> >>> Ruled out
> >>> =========
> >>>
> >>> - GPU/driver: reproduced with RTX 5080 (595-open module) and RTX 3080 Ti
> >>>               (both 595-open and 595 proprietary modules)
> >>> - ASUS USB4 PCIe add-in card: reproduces in ~5 min with the card physically
> >>>                               removed
> >>> - AVIC: kvm_amd avic=N on both good and bad kernels
> >>> - TSA mitigation: tsa=off verified applied
> >>>   (/sys/devices/system/cpu/vulnerabilities/tsa = "Vulnerable"), still crashes
> >>> - Memory: stock JEDEC 4800, no XMP/EXPO, no ECC errors logged
> >>>   (ras-mc-ctl reports zero CE/UE)
> >>> - Thermal/power: ~47-50 C at time of crash, system idle; 1600W Seasonic PSU
> >>> - Hardware health: multi-hour AOSP compiles (all 64 threads, heavy cache
> >>>   load) and repeated GPU stress runs complete without error
> >>>
> >>>
> >>> I hope this gives you enough details to start looking at what could cause this
> >>> weird behavior. Just FYI, when asked about hardware damage, AI suggested more
> >>> something like "a CPU-level microcode erratum in the deep-idle path on this
> >>> platform", but being old-school, I tend to always triple-check what AI tells
> >>> before claiming it myself :)
> >>>
> >>> Thanks and Kind Regards,
> >>> Mathieu
> >>>
> >>> ------
> >>> [1] using android-11.0.0_r46 release tag and "sdk_phone_x86_64" as lunch target
> >>> [3] using android-15.0.0_r32 release tag and "sdk_phone64_x86_64" as lunch
> >>>     target this time, since Google only releases 64-bit only today
> >
>

Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Michal Pecio]

On Sun, 23 Aug 2026 13:44:53 +0200, Mathieu Fluhr wrote:
> Yes, looking at the reports, this looks definitely related.
> What is happening when the emulator starts is that it also starts adbd
> if not running on the Linux side, to communicate with the emulator
> later on.
> 
> [...]
>
> Ok, here you lost me a little: Looking to all the links provided, yes,
> this looks very similar to what I am experiencing. But (and correct me
> if I am wrong)
> 1. reverting the "bad commit" is not really an option, since it was
> originally authored for a specific purpose (other than annoying AMD
> owners :)

It does seem so, even if the specific issue report isn't known.

It also isn't clear (at least to me) whether the intended behavior of
the patch is causing this, or if the patch simply has some bug which
causes unintended side effects that can and should be fixed.

> 2. the patch that Mario proposed is very recent, and would address the
> issue not on the USB but on the PCI level?
> 
> Thanks again for the swift answer here and all the informative links!
> I must admit that I researched a lot debugging this issue, but I never
> thought about looking to the USB side of the problem, focusing more on
> the CPU side.

The issue is obviously a severe HW malfunction (you mentioned MCEs, the
Ryzen CPUs simply totally locked up), triggered by poking certain xHCI
controllers on the I/O die of these CPUs in some wrong way.

The minimum known trigger is this loop from bugzilla comment 8:

	while sleep .1 ; do true </dev/bus/usb/010/001 ; done

Opinions seem to vary on whether CPU load must be present or absent.

The minimum known workaround is disabling xHCI PCI function suspend,
see bugzilla comment 16. This knowledge has spread on distribution
forums and many users stopped caring about this bug, but clearly new
people are still running into it.

It seems USB devices and even the root hub can be suspended safely, as
long as the PCI function remains on full power. The influence of CPU
C-states was previously unknown. May be specific to Threadripper.

Regards,
Michal

Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Lovekesh Solanki]

On Sun, Aug 23, 2026 at 12:36:55PM +0200, Thorsten Leemhuis wrote:
> that ticket apparently didn't help. And no other fix is in sight, or am
> I missing something?
> 
> Mathias reverting the culprit in mainline a option to resolve this (I
> assume it is not, but I'm asking just to be sure)?
I don't think a revert is needed here, we're just trading one regression
for another.
Commit 8f5b7e2bec1c introduced the 200ms hold for all superspeed hubs,
but it's only useful for external ones. Root hubs are superspeed hubs
too so they go through the same code, but there's nothing useful for
the hold to do there:
A root hub has no upstream suspended hub whose wake propagation we need
to wait for and xhci already handles late USB3 link training itself.

Also hubs have a 0 second autosuspend delay (596d789a211d), so this hold
is the only thing stretching the awake window.

and anything opening or closing /dev/bus/usb nodes auto resumes and auto
suspends the whole host controller, adb's periodic enumeration does that
so on the reported affected systems every SS roothub cycle grows from
~30 to ~235ms (from the dynamic debug traces in
https://lore.kernel.org/all/qc0nhk9c6l0a08bkfeplrm3qjssgrjkvkp@sonic.net/)
which makes suspend move from close() call into delayed work and results
in the ~1Hz stress loop described upthread.

I think skipping the hold for hubs without a parent device, i.e. keep TB
dock behaviour everywhere it matters, should fix this.

Patch below could any of the reporters test it?

Thanks,
DrEggCake

diff --git a/drivers/usb/core/hub.c b/drivers/usb/core/hub.c
index d92bf887739d..f642ea0c31dc 100644
--- a/drivers/usb/core/hub.c
+++ b/drivers/usb/core/hub.c
@@ -1346,7 +1346,8 @@ static void hub_activate(struct usb_hub *hub, enum hub_activation_type type)
 		device_unlock(&hdev->dev);
 	}
 
-	if (type == HUB_RESUME && hub_is_superspeed(hub->hdev)) {
+	if (type == HUB_RESUME && hub->hdev->parent &&
+	    hub_is_superspeed(hub->hdev)) {
 		/* give usb3 downstream links training time after hub resume */
 		usb_autopm_get_interface_no_resume(
 			to_usb_interface(hub->intfdev));

Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Michal Pecio]

On Sun, 23 Aug 2026 20:47:55 +0530, Lovekesh Solanki wrote:
> I don't think a revert is needed here, we're just trading one
> regression for another.
> Commit 8f5b7e2bec1c introduced the 200ms hold for all superspeed hubs,
> but it's only useful for external ones. Root hubs are superspeed hubs
> too so they go through the same code, but there's nothing useful for
> the hold to do there:
> A root hub has no upstream suspended hub whose wake propagation we
> need to wait for and xhci already handles late USB3 link training
> itself.
> 
> Also hubs have a 0 second autosuspend delay (596d789a211d), so this
> hold is the only thing stretching the awake window.
> 
> and anything opening or closing /dev/bus/usb nodes auto resumes and
> auto suspends the whole host controller, adb's periodic enumeration
> does that so on the reported affected systems every SS roothub cycle
> grows from ~30 to ~235ms (from the dynamic debug traces in
> https://lore.kernel.org/all/qc0nhk9c6l0a08bkfeplrm3qjssgrjkvkp@sonic.net/)
> which makes suspend move from close() call into delayed work and
> results in the ~1Hz stress loop described upthread.
> 
> I think skipping the hold for hubs without a parent device, i.e. keep
> TB dock behaviour everywhere it matters, should fix this.

It likely will, as it effectively reverts the problematic commit for
those particular affected devices (root hubs).

But I'm not sure what you mean by "1Hz stress loop" and why is slowing
down the suspend/resume cycles or moving suspend from close() call into
a work supposed to create problems?

I would naively think that doing things *too fast* is more likely to
trigger races and break the HW. The whole issue smells like a HW bug,
or maybe one of those "undefined behaviors" that the xHCI spec warns
about if SW dares to do something out of spec.

So why was the culprit patch even a problem for those root hubs?
Can it not become a problem for external hubs, under other workloads?

Also, what if we connect a downstream SS hub to the root hub? Will this
not cause the root hub to stay awake for 200ms again? Problem is back?

Regards,
Michal

Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Lovekesh Solanki]

On Sun, Aug 23, 2026 at 05:40:59PM +0200, Michal Pecio wrote:
> But I'm not sure what you mean by "1Hz stress loop" and why is slowing
> down the suspend/resume cycles or moving suspend from close() call into
> a work supposed to create problems?
> 
> I would naively think that doing things *too fast* is more likely to
> trigger races and break the HW. The whole issue smells like a HW bug,
USB2 roothubs still do 30 ms cycles and are harmless, only stretched SS
ones kill the box. So if speed were the issue the faster ones would be
dying first and pre regression kernels ran fine.

About 1Hz, adb scans the bus once a second,
every open resumes the root hub and host controller out of D3 and close
puts it back this existed before regression too but with 30 ms of
close() and slept for rest of the second and now it keeps it
in D0 well after close(), the suspend itself happening later from the
delayed work, so this d3 - d0 - d3 trip of ~230ms repeats on
each scan. That's the loop I meant.

> Can it not become a problem for external hubs, under other workloads?
Possibly yes, but nobody reported that so I'd rather scope where the
harm is proven.

> Also, what if we connect a downstream SS hub to the root hub? Will this
> not cause the root hub to stay awake for 200ms again? Problem is back?
The root hub itself skips the hold since the check is on its own parent,
but it will be kept awake anyway while the downstream hub or devices are
in use which is normal activity based PM.
If heavy polling behind a real hub ever causes trouble that's probably a
xhci/platform level fix anyways since real hubs can't really just drop
the hold.

Regards,
Lovekesh

Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Mathieu Fluhr]

> The issue is obviously a severe HW malfunction (you mentioned MCEs, the
> Ryzen CPUs simply totally locked up), triggered by poking certain xHCI
> controllers on the I/O die of these CPUs in some wrong way.

Yes. As mentioned, I first thought that the emulator itself triggered that by
doing something that the CPU did not like.To be honest, I barely play with old
Android versions anymore, but seeing that I could reproduce it even with
Android 13 or 14 made me suspicious.

I _guess_ Google implemented a workaround inside adb for version Android
15 since using this version, it remains stable for more than 2 hours.

But, in the end, the situation is that from a simple user account having access
to some usb plugged in devices (I usually add my user account to the plugdev
group and use some known udev rules to access my Android tests devices),
you have a way to crash the complete system.

> Opinions seem to vary on whether CPU load must be present or absent.

On my side (and I am here only speaking about my TR. I don't know about
other Ryzen CPUs), I can't reproduce it under load, and one condition to
reproduce it is my CPU going in C2 state.
-> I did a 2:30 hour test using several youtube videos playing at the same
time on my desktop, also stressing the emulator with some 3D Mark runs
(as mentioned, I first suspected the nvidia driver to be faulty). As long as my
computer was busy everything went fine. But then I let it stand still for a few
minutes, and it just crashed.

If you need me to do some further tests or experiments, let me know. I will
be more than happy to play the guinea pig here.

On Sun, Aug 23, 2026 at 7:05 PM Lovekesh Solanki
<lovekeshsolanki00@gmail.com> wrote:
>
> On Sun, Aug 23, 2026 at 05:40:59PM +0200, Michal Pecio wrote:
> > But I'm not sure what you mean by "1Hz stress loop" and why is slowing
> > down the suspend/resume cycles or moving suspend from close() call into
> > a work supposed to create problems?
> >
> > I would naively think that doing things *too fast* is more likely to
> > trigger races and break the HW. The whole issue smells like a HW bug,
> USB2 roothubs still do 30 ms cycles and are harmless, only stretched SS
> ones kill the box. So if speed were the issue the faster ones would be
> dying first and pre regression kernels ran fine.
>
> About 1Hz, adb scans the bus once a second,
> every open resumes the root hub and host controller out of D3 and close
> puts it back this existed before regression too but with 30 ms of
> close() and slept for rest of the second and now it keeps it
> in D0 well after close(), the suspend itself happening later from the
> delayed work, so this d3 - d0 - d3 trip of ~230ms repeats on
> each scan. That's the loop I meant.
>
> > Can it not become a problem for external hubs, under other workloads?
> Possibly yes, but nobody reported that so I'd rather scope where the
> harm is proven.
>
> > Also, what if we connect a downstream SS hub to the root hub? Will this
> > not cause the root hub to stay awake for 200ms again? Problem is back?
> The root hub itself skips the hold since the check is on its own parent,
> but it will be kept awake anyway while the downstream hub or devices are
> in use which is normal activity based PM.
> If heavy polling behind a real hub ever causes trouble that's probably a
> xhci/platform level fix anyways since real hubs can't really just drop
> the hold.
>
> Regards,
> Lovekesh

Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Mario Limonciello]

On 8/24/26 14:37, Mathieu Fluhr wrote:
>> The issue is obviously a severe HW malfunction (you mentioned MCEs, the
>> Ryzen CPUs simply totally locked up), triggered by poking certain xHCI
>> controllers on the I/O die of these CPUs in some wrong way.
> 
> Yes. As mentioned, I first thought that the emulator itself triggered that by
> doing something that the CPU did not like.To be honest, I barely play with old
> Android versions anymore, but seeing that I could reproduce it even with
> Android 13 or 14 made me suspicious.
> 
> I _guess_ Google implemented a workaround inside adb for version Android
> 15 since using this version, it remains stable for more than 2 hours.
> 
> But, in the end, the situation is that from a simple user account having access
> to some usb plugged in devices (I usually add my user account to the plugdev
> group and use some known udev rules to access my Android tests devices),
> you have a way to crash the complete system.
> 
>> Opinions seem to vary on whether CPU load must be present or absent.
> 
> On my side (and I am here only speaking about my TR. I don't know about
> other Ryzen CPUs), I can't reproduce it under load, and one condition to
> reproduce it is my CPU going in C2 state.
> -> I did a 2:30 hour test using several youtube videos playing at the same
> time on my desktop, also stressing the emulator with some 3D Mark runs
> (as mentioned, I first suspected the nvidia driver to be faulty). As long as my
> computer was busy everything went fine. But then I let it stand still for a few
> minutes, and it just crashed.
> 
> If you need me to do some further tests or experiments, let me know. I will
> be more than happy to play the guinea pig here.
> 
The behavior that is described here sounds like a platform firmware bug 
to me.  Are you on the latest BIOS available from your OEM?

Can you please confirm:

1. Your CPU model number/codename
2. Your OEM (from /sys/class/dmi/id)
3. OEM BIOS version (from /sys/class/dmi/id)
4. AGESA version (see commit bc91133e260c8113c1119073c03b93c12aa41738 if 
the OEM didn't tear it out.  Otherwise look in BIOS menus)?

Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Mathieu Fluhr]

> The behavior that is described here sounds like a platform firmware bug
> Are you on the latest BIOS available from your OEM?

Ywa, And I even rolled back 2 BIOS before, All 3 are showing the issue...
...even the exact same one (with the exact same CMOS settings) that I
used in 2024 without any single issue.

> Can you please confirm:
> Your CPU model number/codename
AMD Ryzen Threadripper 7870X 32.Cores

mathieu@ubuntu:~$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 25
model : 24
model name : AMD Ryzen Threadripper 7970X 32-Cores
stepping : 1
microcode : 0xa10810c
cpu MHz : 2188.582
cache size : 1024 KB
physical id : 0
siblings : 64
core id : 0
cpu cores : 32
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 16
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt
pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology
nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor
ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx
f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a
misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core
perfctr_nb bpext perfctr_llc mwaitx cpuid_fault cpb cat_l3 cdp_l3
hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall
fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq
rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw
avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc
cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf
xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock
nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter
pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku
ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg
avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
debug_swap
bugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso
spectre_v2_user tsa vmscape
bogomips : 7988.27
TLB size : 3584 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 52 bits physical, 57 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
[..]

2. Your OEM (from /sys/class/dmi/id)
I am using an ASUS Pro WS TRX50-SAGE WIFI

root@ubuntu:/home/mathieu# cat /sys/class/dmi/id/board*
Default string
Pro WS TRX50-SAGE WIFI
231230340100705
ASUSTeK COMPUTER INC.
Rev 1.xx

3. OEM BIOS version (from /sys/class/dmi/id)
root@ubuntu:/home/mathieu# cat /sys/class/dmi/id/bios*
02/03/2026
13.17
American Megatrends Inc.
1317

4. AGESA version (see commit bc91133e260c8113c1119073c03b93c12aa41738 if
the OEM didn't tear it out.  Otherwise look in BIOS menus)?

The BOIS shoes this information (I also put the ones from previous
BIOS I tested):
BIOS 1317: ComboSP6PI 1.0.0.1e
BIOS 1203: ComboSP6PI 1.0.0.1a Patch A
BIOS 0803: StormPeakPI_1.1.0.0g

On Tue, Aug 25, 2026 at 12:09 AM Mario Limonciello
<mario.limonciello@amd.com> wrote:
>
>
>
> On 8/24/26 14:37, Mathieu Fluhr wrote:
> >> The issue is obviously a severe HW malfunction (you mentioned MCEs, the
> >> Ryzen CPUs simply totally locked up), triggered by poking certain xHCI
> >> controllers on the I/O die of these CPUs in some wrong way.
> >
> > Yes. As mentioned, I first thought that the emulator itself triggered that by
> > doing something that the CPU did not like.To be honest, I barely play with old
> > Android versions anymore, but seeing that I could reproduce it even with
> > Android 13 or 14 made me suspicious.
> >
> > I _guess_ Google implemented a workaround inside adb for version Android
> > 15 since using this version, it remains stable for more than 2 hours.
> >
> > But, in the end, the situation is that from a simple user account having access
> > to some usb plugged in devices (I usually add my user account to the plugdev
> > group and use some known udev rules to access my Android tests devices),
> > you have a way to crash the complete system.
> >
> >> Opinions seem to vary on whether CPU load must be present or absent.
> >
> > On my side (and I am here only speaking about my TR. I don't know about
> > other Ryzen CPUs), I can't reproduce it under load, and one condition to
> > reproduce it is my CPU going in C2 state.
> > -> I did a 2:30 hour test using several youtube videos playing at the same
> > time on my desktop, also stressing the emulator with some 3D Mark runs
> > (as mentioned, I first suspected the nvidia driver to be faulty). As long as my
> > computer was busy everything went fine. But then I let it stand still for a few
> > minutes, and it just crashed.
> >
> > If you need me to do some further tests or experiments, let me know. I will
> > be more than happy to play the guinea pig here.
> >
> The behavior that is described here sounds like a platform firmware bug
> to me.  Are you on the latest BIOS available from your OEM?
>
> Can you please confirm:
>
> 1. Your CPU model number/codename
> 2. Your OEM (from /sys/class/dmi/id)
> 3. OEM BIOS version (from /sys/class/dmi/id)
> 4. AGESA version (see commit bc91133e260c8113c1119073c03b93c12aa41738 if
> the OEM didn't tear it out.  Otherwise look in BIOS menus)?
>

Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Mario Limonciello]

On 8/25/26 03:57, Mathieu Fluhr wrote:
>> The behavior that is described here sounds like a platform firmware bug
>> Are you on the latest BIOS available from your OEM?
> 
> Ywa, And I even rolled back 2 BIOS before, All 3 are showing the issue...
> ...even the exact same one (with the exact same CMOS settings) that I
> used in 2024 without any single issue.
> 
>> Can you please confirm:
>> Your CPU model number/codename
> AMD Ryzen Threadripper 7870X 32.Cores
> 
> mathieu@ubuntu:~$ cat /proc/cpuinfo
> processor : 0
> vendor_id : AuthenticAMD
> cpu family : 25
> model : 24
> model name : AMD Ryzen Threadripper 7970X 32-Cores
> stepping : 1
> microcode : 0xa10810c
> cpu MHz : 2188.582
> cache size : 1024 KB
> physical id : 0
> siblings : 64
> core id : 0
> cpu cores : 32
> apicid : 0
> initial apicid : 0
> fpu : yes
> fpu_exception : yes
> cpuid level : 16
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
> pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt
> pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology
> nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor
> ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx
> f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a
> misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core
> perfctr_nb bpext perfctr_llc mwaitx cpuid_fault cpb cat_l3 cdp_l3
> hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall
> fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq
> rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw
> avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc
> cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf
> xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock
> nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter
> pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku
> ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg
> avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
> debug_swap
> bugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso
> spectre_v2_user tsa vmscape
> bogomips : 7988.27
> TLB size : 3584 4K pages
> clflush size : 64
> cache_alignment : 64
> address sizes : 52 bits physical, 57 bits virtual
> power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
> [..]
> 
> 2. Your OEM (from /sys/class/dmi/id)
> I am using an ASUS Pro WS TRX50-SAGE WIFI
> 
> root@ubuntu:/home/mathieu# cat /sys/class/dmi/id/board*
> Default string
> Pro WS TRX50-SAGE WIFI
> 231230340100705
> ASUSTeK COMPUTER INC.
> Rev 1.xx
> 
> 3. OEM BIOS version (from /sys/class/dmi/id)
> root@ubuntu:/home/mathieu# cat /sys/class/dmi/id/bios*
> 02/03/2026
> 13.17
> American Megatrends Inc.
> 1317
> 
> 4. AGESA version (see commit bc91133e260c8113c1119073c03b93c12aa41738 if
> the OEM didn't tear it out.  Otherwise look in BIOS menus)?
> 
> The BOIS shoes this information (I also put the ones from previous
> BIOS I tested):
> BIOS 1317: ComboSP6PI 1.0.0.1e
> BIOS 1203: ComboSP6PI 1.0.0.1a Patch A
> BIOS 0803: StormPeakPI_1.1.0.0g
> 
This isn't a BIOS regression, it's been around since the beginning on 
this threadripper model but it was harder to trigger.
The Linux kernel changes in the bisect exposed a platform issue.

The workaround you can do for now is to disable runtime PM on the XHCI 
controller to avoid triggering it.  But it's a platform issue.

For a proper fix you need to reach out to ASUS to get an AGESA upgrade.
You can ask them to reach out to AMD if they need assistance/guidance on 
the versions that contain the fix for this.  You can share this thread 
with them for reference.

Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Mathieu Fluhr]

> The workaround you can do for now is to disable runtime PM on the XHCI
> controller to avoid triggering it.  But it's a platform issue.

So, long story short, the kernel source code is correct, and the issue comes
from the BIOS/AGESA, correct?

> For a proper fix you need to reach out to ASUS to get an AGESA upgrade.
> You can ask them to reach out to AMD if they need assistance/guidance on
> the versions that contain the fix for this.  You can share this thread
> with them for reference.

I will. But this leaves me with a few questions here:
1. Do you already have a fixed AGESA on AMD side, that ASUS could use?
2. Does this issue also affect other threadripper generations, like the 9970x?
3. I guess not only ASUS is concerned but Also ASRock and Gigabyte which
also produces TRX50 mainboards, correct?

On Wed, Aug 26, 2026 at 5:14 AM Mario Limonciello
<mario.limonciello@amd.com> wrote:
>
>
>
> On 8/25/26 03:57, Mathieu Fluhr wrote:
> >> The behavior that is described here sounds like a platform firmware bug
> >> Are you on the latest BIOS available from your OEM?
> >
> > Ywa, And I even rolled back 2 BIOS before, All 3 are showing the issue...
> > ...even the exact same one (with the exact same CMOS settings) that I
> > used in 2024 without any single issue.
> >
> >> Can you please confirm:
> >> Your CPU model number/codename
> > AMD Ryzen Threadripper 7870X 32.Cores
> >
> > mathieu@ubuntu:~$ cat /proc/cpuinfo
> > processor : 0
> > vendor_id : AuthenticAMD
> > cpu family : 25
> > model : 24
> > model name : AMD Ryzen Threadripper 7970X 32-Cores
> > stepping : 1
> > microcode : 0xa10810c
> > cpu MHz : 2188.582
> > cache size : 1024 KB
> > physical id : 0
> > siblings : 64
> > core id : 0
> > cpu cores : 32
> > apicid : 0
> > initial apicid : 0
> > fpu : yes
> > fpu_exception : yes
> > cpuid level : 16
> > wp : yes
> > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
> > pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt
> > pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology
> > nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor
> > ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx
> > f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a
> > misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core
> > perfctr_nb bpext perfctr_llc mwaitx cpuid_fault cpb cat_l3 cdp_l3
> > hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall
> > fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq
> > rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw
> > avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc
> > cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf
> > xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock
> > nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter
> > pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku
> > ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg
> > avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
> > debug_swap
> > bugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso
> > spectre_v2_user tsa vmscape
> > bogomips : 7988.27
> > TLB size : 3584 4K pages
> > clflush size : 64
> > cache_alignment : 64
> > address sizes : 52 bits physical, 57 bits virtual
> > power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
> > [..]
> >
> > 2. Your OEM (from /sys/class/dmi/id)
> > I am using an ASUS Pro WS TRX50-SAGE WIFI
> >
> > root@ubuntu:/home/mathieu# cat /sys/class/dmi/id/board*
> > Default string
> > Pro WS TRX50-SAGE WIFI
> > 231230340100705
> > ASUSTeK COMPUTER INC.
> > Rev 1.xx
> >
> > 3. OEM BIOS version (from /sys/class/dmi/id)
> > root@ubuntu:/home/mathieu# cat /sys/class/dmi/id/bios*
> > 02/03/2026
> > 13.17
> > American Megatrends Inc.
> > 1317
> >
> > 4. AGESA version (see commit bc91133e260c8113c1119073c03b93c12aa41738 if
> > the OEM didn't tear it out.  Otherwise look in BIOS menus)?
> >
> > The BOIS shoes this information (I also put the ones from previous
> > BIOS I tested):
> > BIOS 1317: ComboSP6PI 1.0.0.1e
> > BIOS 1203: ComboSP6PI 1.0.0.1a Patch A
> > BIOS 0803: StormPeakPI_1.1.0.0g
> >
> This isn't a BIOS regression, it's been around since the beginning on
> this threadripper model but it was harder to trigger.
> The Linux kernel changes in the bisect exposed a platform issue.
>
> The workaround you can do for now is to disable runtime PM on the XHCI
> controller to avoid triggering it.  But it's a platform issue.
>
> For a proper fix you need to reach out to ASUS to get an AGESA upgrade.
> You can ask them to reach out to AMD if they need assistance/guidance on
> the versions that contain the fix for this.  You can share this thread
> with them for reference.

Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Michal Pecio]

On Tue, 25 Aug 2026 22:14:32 -0500, Mario Limonciello wrote:
> This isn't a BIOS regression, it's been around since the beginning on 
> this threadripper model but it was harder to trigger.
> The Linux kernel changes in the bisect exposed a platform issue.
> 
> The workaround you can do for now is to disable runtime PM on the
> XHCI controller to avoid triggering it.  But it's a platform issue.
> 
> For a proper fix you need to reach out to ASUS to get an AGESA
> upgrade. You can ask them to reach out to AMD if they need
> assistance/guidance on the versions that contain the fix for this.
> You can share this thread with them for reference.

This sounds like it's a known FW bug, so I suppose the exact failure
mechanism is known too. Is it triggered just by putting that PCI
function into D3 at a "wrong" time, or is there any USB/xHCI layer
involvement, particularly anything out of spec by software?

I ask because we know one case where dodgy SW actions cause some AMD
xHCI to malfunction, but the problem is contained in the USB layer and
the affected xHC doesn't bring down the whole SoC.

Any thoughts about the patch which reduces suspend delay for root hubs
specifically? It restores the original timing of PCI resume/suspend
cycles, at least as long as there are no external hubs connected, and
has just been reported to mitigate this problem.

Regards,
Michal

Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Lovekesh Solanki]

On Mon, Aug 24, 2026 at 05:09:02PM -0500, Mario Limonciello wrote:
> The behavior that is described here sounds like a platform firmware bug to
> me.  Are you on the latest BIOS available from your OEM?
> 
> Can you please confirm:
> 
> 1. Your CPU model number/codename
> 2. Your OEM (from /sys/class/dmi/id)
> 3. OEM BIOS version (from /sys/class/dmi/id)
> 4. AGESA version (see commit bc91133e260c8113c1119073c03b93c12aa41738 if the
> OEM didn't tear it out.  Otherwise look in BIOS menus)?
Though the issue's only reproducible in kernels including commmit
8f5b7e2bec1c36, which their bisection showed too.

Could the trigger not be the 200ms SS hold?

Regards,
Lovekesh

Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Lovekesh Solanki]

On Mon, Aug 24, 2026 at 09:37:55PM +0200, Mathieu Fluhr wrote:
> Yes. As mentioned, I first thought that the emulator itself triggered that by
> doing something that the CPU did not like.To be honest, I barely play with old
> Android versions anymore, but seeing that I could reproduce it even with
> Android 13 or 14 made me suspicious.
> 
> I _guess_ Google implemented a workaround inside adb for version Android
> 15 since using this version, it remains stable for more than 2 hours.
> 
> But, in the end, the situation is that from a simple user account having access
> to some usb plugged in devices (I usually add my user account to the plugdev
> group and use some known udev rules to access my Android tests devices),
> you have a way to crash the complete system.
> 
> > Opinions seem to vary on whether CPU load must be present or absent.
> 
> On my side (and I am here only speaking about my TR. I don't know about
> other Ryzen CPUs), I can't reproduce it under load, and one condition to
> reproduce it is my CPU going in C2 state.
> -> I did a 2:30 hour test using several youtube videos playing at the same
> time on my desktop, also stressing the emulator with some 3D Mark runs
> (as mentioned, I first suspected the nvidia driver to be faulty). As long as my
> computer was busy everything went fine. But then I let it stand still for a few
> minutes, and it just crashed.
> 
> If you need me to do some further tests or experiments, let me know. I will
> be more than happy to play the guinea pig here.
Was this result after applying the small patch I sent above?

Regards,
Lovekesh

Re: [REGRESSION] 6.12.36+: usb: hub: post-resume delayed work triggers > uncorrected MCE / data fabric sync flood on Threadripper 7970X > (bisected to aec11e5f9c45)

[Mathieu Fluhr]

>> If you need me to do some further tests or experiments, let me know. I will
>> be more than happy to play the guinea pig here.
> Was this result after applying the small patch I sent above?

The patch you sent makes the issue go away. I am now running the exact same
scenario for 2 hours, without any crash.

However, I am not sure if this can qualify for a proper fix, since I
do not have any
external hub connected to my motherboard: All my USB devices are directly
plugged into root hubs (and at the moment, there is only a keyboard and a
mouse).

On Tue, Aug 25, 2026 at 12:06 PM Lovekesh Solanki
<lovekeshsolanki00@gmail.com> wrote:
>
> On Mon, Aug 24, 2026 at 09:37:55PM +0200, Mathieu Fluhr wrote:
> > Yes. As mentioned, I first thought that the emulator itself triggered that by
> > doing something that the CPU did not like.To be honest, I barely play with old
> > Android versions anymore, but seeing that I could reproduce it even with
> > Android 13 or 14 made me suspicious.
> >
> > I _guess_ Google implemented a workaround inside adb for version Android
> > 15 since using this version, it remains stable for more than 2 hours.
> >
> > But, in the end, the situation is that from a simple user account having access
> > to some usb plugged in devices (I usually add my user account to the plugdev
> > group and use some known udev rules to access my Android tests devices),
> > you have a way to crash the complete system.
> >
> > > Opinions seem to vary on whether CPU load must be present or absent.
> >
> > On my side (and I am here only speaking about my TR. I don't know about
> > other Ryzen CPUs), I can't reproduce it under load, and one condition to
> > reproduce it is my CPU going in C2 state.
> > -> I did a 2:30 hour test using several youtube videos playing at the same
> > time on my desktop, also stressing the emulator with some 3D Mark runs
> > (as mentioned, I first suspected the nvidia driver to be faulty). As long as my
> > computer was busy everything went fine. But then I let it stand still for a few
> > minutes, and it just crashed.
> >
> > If you need me to do some further tests or experiments, let me know. I will
> > be more than happy to play the guinea pig here.
> Was this result after applying the small patch I sent above?
>
> Regards,
> Lovekesh