Re: [REGRESSION 00/04] Crash during resume of pcie bridge

Mario Limonciello
2025-10-07T21:33:55Z

On 10/6/25 7:09 AM, Bert Karwatzki wrote:
> Since linux version v6.15 I experience random crashes on my MSI Alpha 15 Laptop
> running debian trixie (amd64). The first such crash happened about in the midth
> of june, and as there were no useful log messages and even using netconsole
> gave no useful message I suspected faulty hardware. So I ran memtest86+ and
> found a faulty address line and replaced the memory (unfortunately 64G to 16G).
> But the crashes occured again and so I did a thorough investigation.
>
> The crashes occur after 30min to 33h (yes, hours) of uptime and consist of a
> sudden reboot after which the PCI bridge at 00:02.4 and the nvme device
> connected to it are missing. If there's sound running during the crash then the
> first sign of the crash is the sound looping like a broken record for about 2s,
> after which the reboot happens. With the missing nvme device the reboot drops to
> a rescue shell. Using "shutdown -h now" from that shell and starting the laptop
> with the power button restores the missing PCI bridge and nvme device.
>
> The hardware is the following (it's a dual GPU laptop where the GUI
> runs on the built-in GPU):
>
> $ cat /proc/cpuinfo
> processor : 0
> vendor_id : AuthenticAMD
> cpu family : 25
> model : 80
> model name : AMD Ryzen 7 5800H with Radeon Graphics
> stepping : 0
> microcode : 0xa50000c
> cpu MHz : 3394.238
> cache size : 512 KB
> physical id : 0
> siblings : 16
> core id : 0
> cpu cores : 8
> apicid : 0
> initial apicid : 0
> fpu : yes
> fpu_exception : yes
> cpuid level : 16
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm debug_swap
> bugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso ibpb_no_ret
> bogomips : 6388.57
> TLB size : 2560 4K pages
> clflush size : 64
> cache_alignment : 64
> address sizes : 48 bits physical, 48 bits virtual
> power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
>
> $ lspci -nn
> 00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne Root Complex [1022:1630]
> 00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne IOMMU [1022:1631]
> 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
> 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1633]
> 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
> 00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge [1022:1634]
> 00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge [1022:1634]
> 00:02.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge [1022:1634]
> 00:02.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge [1022:1634]
> 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
> 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
> 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51)
> 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
> 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 0 [1022:166a]
> 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 1 [1022:166b]
> 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 2 [1022:166c]
> 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 3 [1022:166d]
> 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 4 [1022:166e]
> 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 5 [1022:166f]
> 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 6 [1022:1670]
> 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 7 [1022:1671]
> 01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c3)
> 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
> 03:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] [1002:73ff] (rev c3)
> 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
> 04:00.0 Network controller [0280]: MEDIATEK Corp. MT7921K (RZ608) Wi-Fi 6E 80MHz [14c3:0608]
> 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
> 06:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. KC3000/FURY Renegade NVMe SSD [E18] [2646:5013] (rev 01)
> 07:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology P1 NVMe PCIe SSD[Frampton] [c0a9:2263] (rev 03)
> 08:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [Radeon Vega Series / Radeon Vega Mobile Series] [1002:1638] (rev c5)
> 08:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Renoir Radeon High Definition Audio Controller [1002:1637]
> 08:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
> 08:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1 [1022:1639]
> 08:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1 [1022:1639]
> 08:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] Audio Coprocessor [1022:15e2] (rev 01)
> 08:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h/1ah HD Audio Controller [1022:15e3]
> 08:00.7 Signal processing controller [1180]: Advanced Micro Devices, Inc. [AMD] Sensor Fusion Hub [1022:15e4]
>
> These devices are attached to the PCI bus like this:
>
> $ lspci -t
> -[0000:00]-+-00.0
> +-00.2
> +-01.0
> +-01.1-[01-03]----00.0-[02-03]----00.0-[03]--+-00.0 // This is the bridge which causes the crash
> | \-00.1
> +-02.0
> +-02.1-[04]----00.0
> +-02.2-[05]----00.0
> +-02.3-[06]----00.0
> +-02.4-[07]----00.0 // These are the bridge and nvme device which disappear after the crash.
> +-08.0
> +-08.1-[08]--+-00.0
> | +-00.1
> | +-00.2
> | +-00.3
> | +-00.4
> | +-00.5
> | +-00.6
> | \-00.7
> +-14.0
> +-14.3
> +-18.0
> +-18.1
> +-18.2
> +-18.3
> +-18.4
> +-18.5
> +-18.6
> \-18.7
>
> I tried to bisect this between v6.14 and v6.15 but due to the wildly varying time
> it takes to trigger the bug the bisections were not successful. Nevertheless they
> gave lots of data about affected and non-affected version of the linux kernel,
> and it's quite likely that version v6.14 is indeed free of the bug.
>
> Here's an almost complete list of tested versions:
> (Somewhat) sorted (by kernel version, 6.14.0-rc* kernels are from attempted bisections
> between v6.14 and v6.15)
> v6.14.0 no crash after 16h
> v6.14.11 no crash after 7.5h
> 6.14.0-rc1-bisect-00003-g541ddf31e300 booted 12:24, 22.8.2025, no crash after {48h, 17h}
> 6.14.0-rc1-mystery-00134-gcc28c0e5e725 booted 11:42, 5.8.2025, no crash after 10.5h
> 6.14.0-rc1-mystery-00198-gd7f6f07ecec9 booted 22:27, 5.8.2025, no crash after 12h
> 6.14.0-rc4-mystery-01022-gab498828fad7 booted 21:04, 3.8.2025, no crash after {14h, 24h}
> 6.14.0-rc4-mystery-01427-g7547510d4a91 booted 11:11, 4.8.2025, no crash after {13h, 23h}
> 6.14.0-rc6-mystery-01641-g0f04462874e1 booted 00:26, 5.8.2025, no crash after {11h, 24h}
> 6.14.0-mystery-00826-g327ecdbc0fda no crash after {16h, 17h, 6.5h}
> ############## here the crashes start (time to each crash, crashes do not always occur) ########
> 6.14.0-bisect-01053-gebfb94d87b35 booted 10:15, 20.8.2025 crash after ~33h
> 6.14.0-mystery-09584-g7d06015d936c crash 20.44 3.8.2025 after 7h
> 6.14.0-mystery-11703-geb0ece16027f crash 13.22 3.8.2025 after 1.75h
> 6.15.0 crashed around 15-17.6.2025, unknown uptime (This is the first crash!)
> 6.15.0-nort crash after 6.75h
> 6.16-rc4 (next-20250627) crash after ~4h
> 6.16-rc4 (next-20250630) crash after ~5h
> 6.16-rc4 (next-20250703) crash after ~2.5h (sound buffer repeated for ~1s before restarting)
> 6.16-rc6 (next-20250718) crash after {2h, 2h}
> 6.16-rc7 (next-20250721) crash after {~30min, 2h, 5.5h}
> 6.16.0-nortlockdep crash after 4h
> 6.17.0-rc4-next-20250902-master booted 8:36, 3.9.2025, crash after ~3.5h
> 6.17.0-rc5-next-20250908-master booted 10:25, 9.9.2025, crash after {~6.5h, 14h}
> 6.17.0-rc6-next-20250917-acpidebug booted 12:41, 20.9.2025, crash 15:22 20.8.2025 (~3h, 647 GPP notifies)
> The versions below contain additional debugging printk()s and dev_info()s.
> The details of these debugging statements are explained below.
> 6.17.0-rc6-next-20250917-gpudebug-00018-g7a38b625a003 booted 12:58, 26.9.2025, crash 12:01, 27.9.2025 (~23h, 1500 GPP notifies)
> 6.17.0-rc6-next-20250917-gpudebug-00021-gab98d880e3c8 booted 23:52, 28.9.2025, crash 2:25, 30.9.2025 (26.5h, 1504GPP0, 889GPP2)
> 6.17.0-rc6-next-20250917-gpudebug-00024-g5c6b49b810db booted 9:10, 2.10.2025, 60h 3093 GPP0 notifies without crash (too many printk()s?)
> 6.17.0-rc6-next-20250917-gpudebug-00028-gf99cf81b1da7 booted 21:21, 4.10.2025 first try stopped after 77min due to hung tasks
> 6.17.0-rc6-next-20250917-gpudebug-00028-gf99cf81b1da7 booted 23:37, 4.10.2025 crash 4:52, 6.10.2025 (~27.5h)
> 6.17.0-rc6-next-20250917-gpudebug-00029-ge797f42363d1 booted 13:00, 6.10.2025 currently testing
>
> As the bisections were not succesfull I tried to monitor the crash using
> netconsole and CONFIG_ACPI_DEBUG and "acpi.debug_layer=0xf acpi.debug_level=0x107"
> as command line parameters. With this the last message on netconsole before
> the crash is usually:
>
> [21465.639279] [ T251] evmisc-0132 ev_queue_notify_reques: Dispatching Notify on [GPP0] (Device) Value 0x00 (Bus Check) Node 00000000f81f36b8
>
> GPP0 is the ACPI name of this PCI bridge (at least that's my best guess):
>
> 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1633]
>
> to which the discrete GPU is connected
>
> 03:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] [1002:73ff] (rev c3)
>
> via the pci express switch
>
> 01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c3)
> 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
>
> While the GUI (xfce on xorg) on my laptop runs on the built-in GPU the discrete
> GPU usually wakes up quite often, e.g. when a window is opened or when scrolling down on youtube.
>
> A somewhat reliable method to generate GPP0 notifies is putting on a youtube
> video and the periodically starting evolution with this script:
>
> #!/bin/bash
> for i in {0..1000}
> do
> echo $i
> evolution &
> sleep 5
> killall evolution
> sleep 55
> done
>
> This is also the method I used to test the debug kernel in the following mails.
>
> Bert Karwatzki

Given the perpetrator and victim here don't share a common upstream root
port (the only common is the root complex) I wonder if this is actually
an issue with something non-obvious like the IOMMU.

Can you still reproduce with amd_iommu=off?