* [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot
@ 2026-10-23 7:32 Peter Deggendorfer
2026-10-23 13:14 ` Andrew Cooper
2026-10-23 23:51 ` Teddy Astie
0 siblings, 2 replies; 8+ messages in thread
From: Peter Deggendorfer @ 2026-10-23 7:32 UTC (permalink / raw)
To: xen-devel
[-- Attachment #1: Type: text/plain, Size: 3689 bytes --]
Hey Xen Team,
I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE
polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since
2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.
Environment
-----------
Xen: 4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still
present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1)
dom0: Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs
CPU: AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X,
microcode 0x0a201210
Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems
The panic
---------
(XEN) [2026-10-19 08:58:18.727] Xen BUG at arch/x86/cpu/mcheck/mce_amd.c:100
(XEN) CPU: 2
(XEN) RIP: e008:[<ffff82d04028a650>] mce_amd.c#mc_ec2type.part.0+0/0x10
(XEN) rbx: 0000000000000445 r12: 0000000000000011 r15: 0000000000000446
(XEN) Xen code around <ffff82d04028a650>:
(XEN) 90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90
(XEN) Xen call trace:
(XEN) [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10
(XEN) [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40
(XEN) [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840
(XEN) [<ffff82d04028a35c>] S amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
(XEN) [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90
(XEN) [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
(XEN) [<ffff82d040201915>] S common_interrupt+0x135/0x150
(XEN) [<ffff82d0402883ca>] S cpu_idle.c#acpi_idle_do_entry+0x8a/0x140
(XEN) [<ffff82d0402887cf>] S cpu_idle.c#acpi_processor_idle+0x1ef/0x650
(XEN) [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130
(XEN) Panic on CPU 2
The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is
the "unreachable" default branch of mc_ec2type() -- the extended error
code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.
Across the eight occurrences, r12 appears to be the bank index from the
loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS)
and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.
Not a hardware defect
---------------------
- Started within days of upgrading two hosts from Debian bookworm
(Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17 with no
incident.
- Three different banks, so not a single failing block.
- Updating to the current Debian package (4.20.5-pre, built 2026-10-16)
changed nothing.
Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks
updating").
Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
Filter bogus hardware errors on Zen3 clients"), where bogus MCA values
on Zen 3 are filtered rather than treated as errors.
https://ratatoskr.run/stable/2026/04/7143905
Diagnostics / workaround
------------------------
mce_verbosity=verbose yields nothing useful.
max_cstate=1 made no difference. mce=false is the only thing that stops
it, and it is what I'm running now.
One more thing: the host does not actually reboot after the panic.
"Reboot in five seconds" is printed, but the ring buffer continues with
normal activity minutes later and the remaining 31 CPUs keep running.
CPU 2 stays wedged, and "xl debug-keys t" then hangs. Afterwards dom0
and guest vCPUs start losing timer interrupts permanently (RCU stalls
with a frozen timer-softirq count while dynticks keeps advancing), so
VMs hang silently hours or days later.
I've attached the full xl dmesg log of the last crash, and I'm happy
to provide more if needed. I can also test patches.
Thanks for your work, and thanks in advance for taking a look.
Peter
[-- Attachment #2: xl_dmesg.log --]
[-- Type: text/plain, Size: 24397 bytes --]
(XEN) [00000031b97b608e] Xen version 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1) (pkg-xen-devel@lists.alioth.debian.org) (x86_64-linux-gnu-gcc (Debian 14.2.0-19) 14.2.0) debug=n Fri Oct 16 14:28:47 UTC 2026
(XEN) [00000031b9816cc0] build-id: f6b25888e42dd18e951f3097ba861c9996a5b74c
(XEN) [00000031b9838a7e] CPU Vendor: AMD, Family 25 (0x19), Model 33 (0x21), Stepping 2 (raw 00a20f12)
(XEN) [00000031b9866c88] BSP microcode revision: 0x0a201210
(XEN) [00000031b9881720] WARNING: Platform vulnerable to Entrysign (SB-7033, CVE-2024-36347) - firmware update required
(XEN) [00000031b98cf592] Bootloader: GRUB 2.12-9+deb13u2
(XEN) [00000031b98dcd20] Command line: placeholder dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems
(XEN) [00000031b9903230] Xen image load base address: 0xbe800000
(XEN) [00000031b992010a] Video information:
(XEN) [00000031b99337b0] VGA is text mode 80x25, font 8x16
(XEN) [00000031b994e26a] VBE/DDC methods: none; EDID transfer time: 1 seconds
(XEN) [00000031b9971678] EDID info not retrieved because no DDC retrieval method detected
(XEN) [00000031b999a162] Disc information:
(XEN) [00000031b99ad0fe] Found 2 MBR signatures
(XEN) [00000031b99c2c08] Found 2 EDD information structures
(XEN) [00000031b99dddcc] Enabling Supervisor Shadow Stacks
(XEN) [00000031b99f9700] Failed to convert E820 RAM e6cc0-e73bd to RESERVED
(XEN) [00000031b9a19f7e] Failed to convert E820 RAM e6cc0-e73bd to RESERVED
(XEN) [00000031b9a3e02a] Xen-e820 RAM map:
(XEN) [00000031b9fdef9e] [0000000000000000, 000000000009ebff] (usable)
(XEN) [00000031ba562202] [000000000009ec00, 000000000009ffff] (reserved)
(XEN) [00000031baaf4596] [00000000000e0000, 00000000000fffff] (reserved)
(XEN) [00000031bb0856d6] [0000000000100000, 0000000009bfefff] (usable)
(XEN) [00000031bb616e98] [0000000009bff000, 0000000009ffffff] (reserved)
(XEN) [00000031bbc0c7ac] [000000000a000000, 000000000a1fffff] (usable)
(XEN) [00000031bc19c500] [000000000a200000, 000000000a20dfff] (ACPI NVS)
(XEN) [00000031bc72ce04] [000000000a20e000, 000000000affffff] (usable)
(XEN) [00000031bccbca8c] [000000000b000000, 000000000b01ffff] (reserved)
(XEN) [00000031bd24c956] [000000000b020000, 00000000bb30bfff] (usable)
(XEN) [00000031bd7dcdf8] [00000000bb30c000, 00000000bcb0cfff] (reserved)
(XEN) [00000031bdd6c7b6] [00000000bcb0d000, 00000000bcb4bfff] (ACPI data)
(XEN) [00000031be2fe682] [00000000bcb4c000, 00000000bd203fff] (ACPI NVS)
(XEN) [00000031be88fbe0] [00000000bd204000, 00000000be1fefff] (reserved)
(XEN) [00000031bee85758] [00000000be1ff000, 00000000beffffff] (usable)
(XEN) [00000031bf4150b0] [00000000bf000000, 00000000bfffffff] (reserved)
(XEN) [00000031bf9a6e06] [00000000f0000000, 00000000f7ffffff] (reserved)
(XEN) [00000031bff39046] [00000000fd200000, 00000000fd2fffff] (reserved)
(XEN) [00000031c04ca8f6] [00000000fd600000, 00000000fd7fffff] (reserved)
(XEN) [00000031c0a5b728] [00000000fea00000, 00000000fea0ffff] (reserved)
(XEN) [00000031c0fed836] [00000000feb80000, 00000000fec01fff] (reserved)
(XEN) [00000031c157f218] [00000000fec10000, 00000000fec10fff] (reserved)
(XEN) [00000031c1b113d0] [00000000fec30000, 00000000fec30fff] (reserved)
(XEN) [00000031c20a505c] [00000000fed00000, 00000000fed00fff] (reserved)
(XEN) [00000031c269b102] [00000000fed40000, 00000000fed44fff] (reserved)
(XEN) [00000031c2c2c330] [00000000fed80000, 00000000fed8ffff] (reserved)
(XEN) [00000031c31bd294] [00000000fedc2000, 00000000fedcffff] (reserved)
(XEN) [00000031c374f24e] [00000000fedd4000, 00000000fedd5fff] (reserved)
(XEN) [00000031c3ce11c4] [00000000ff000000, 00000000ffffffff] (reserved)
(XEN) [00000031c4272ab8] [0000000100000000, 000000203f2fffff] (usable)
(XEN) [00000031c4802ba2] [000000203f300000, 000000203fffffff] (reserved)
(XEN) [00000031c82469bc] ACPI: RSDP 000F05B0, 0024 (r2 ALASKA)
(XEN) [00000031c87b4fb2] ACPI: XSDT BD1EC728, 00CC (r1 ALASKA A M I 1072009 AMI 1000013)
(XEN) [00000031c9325814] ACPI: FACP BCB3D000, 0114 (r6 ALASKA A M I 1072009 AMI 10013)
(XEN) [00000031c9e17910] ACPI: DSDT BCB37000, 5F6D (r2 ALASKA A M I 1072009 INTL 20120913)
(XEN) [00000031ca90a04a] ACPI: FACS BD1E7000, 0040
(XEN) [00000031cae76330] ACPI: SSDT BCB43000, 8CE9 (r2 AMD AmdTable 2 MSFT 4000000)
(XEN) [00000031cb976902] ACPI: SSDT BCB3F000, 3B8E (r2 AMD AMD AOD 1 INTL 20120913)
(XEN) [00000031cc469f1c] ACPI: SSDT BCB3E000, 03F1 (r2 ALASKA CPUSSDT 1072009 AMI 1072009)
(XEN) [00000031ccfc9aae] ACPI: FIDT BCB36000, 009C (r1 ALASKA A M I 1072009 AMI 10013)
(XEN) [00000031cdabd5d4] ACPI: MCFG BCB35000, 003C (r1 ALASKA A M I 1072009 MSFT 10013)
(XEN) [00000031ce5af978] ACPI: AAFT BCB34000, 0068 (r1 ALASKA OEMAAFT 1072009 MSFT 97)
(XEN) [00000031cf0a3526] ACPI: HPET BCB33000, 0038 (r1 ALASKA A M I 1072009 AMI 5)
(XEN) [00000031cfbfd954] ACPI: IVRS BCB32000, 00D0 (r2 AMD AmdTable 1 AMD 1)
(XEN) [00000031d06ee6a8] ACPI: FPDT BCB31000, 0044 (r1 ALASKA A M I 1072009 AMI 1000013)
(XEN) [00000031d11e1222] ACPI: PCCT BCB30000, 006E (r2 AMD AmdTable 1 AMD 1)
(XEN) [00000031d1cd3e02] ACPI: SSDT BCB27000, 8213 (r2 AMD AmdTable 1 AMD 1)
(XEN) [00000031d27c633e] ACPI: CRAT BCB25000, 1710 (r1 AMD AmdTable 1 AMD 1)
(XEN) [00000031d331f160] ACPI: CDIT BCB24000, 0029 (r1 AMD AmdTable 1 AMD 1)
(XEN) [00000031d3e11b86] ACPI: SSDT BCB23000, 052C (r2 AMD ArticNOI 1 INTL 20120913)
(XEN) [00000031d4906988] ACPI: SSDT BCB1F000, 37DC (r2 AMD ArticN 1 INTL 20120913)
(XEN) [00000031d53fbe0c] ACPI: WSMT BCB1E000, 0028 (r1 ALASKA A M I 1072009 AMI 10013)
(XEN) [00000031d5ef20e8] ACPI: APIC BCB1D000, 015E (r4 ALASKA A M I 1072009 AMI 10013)
(XEN) [00000031d6a49326] ACPI: SSDT BCB1C000, 007D (r2 AMD ArticDIS 1 INTL 20120913)
(XEN) [00000031d753dc1c] ACPI: SSDT BCB1A000, 10AF (r2 AMD ArticC 1 INTL 20120913)
(XEN) [00000031d802ff38] ACPI: SSDT BCB19000, 00BF (r1 AMD AmdTable 1000 INTL 20120913)
(XEN) [00000031d8b5f1d6] System RAM: 130991MB (134135280kB)
(XEN) [00000031e3558ae8] No NUMA configuration found
(XEN) [00000031e3ac153a] Faking a node at 0000000000000000-000000203f300000
(XEN) [00000031f622e910] Domain heap initialised
(XEN) [00000031fd1adc40] SMBIOS 3.3 present.
(XEN) [00000031fd725690] PCI: MCFG configuration 0: base f0000000 segment 0000 buses 00 - 7f
(XEN) [00000031fe2129fc] PCI: MCFG area at f0000000 reserved in E820
(XEN) [00000031fe78d1da] PCI: Using MCFG for segment 0000 bus 00-7f
(XEN) [00000031fedc7616] Using APIC driver default
(XEN) [00000031ff34856a] ACPI: PM-Timer IO Port: 0x808 (32 bits)
(XEN) [00000031ff8b7db4] ACPI: v5 SLEEP INFO: control[0:0], status[0:0]
(XEN) [00000031ffe460de] ACPI: SLEEP INFO: pm1x_cnt[1:804,1:0], pm1x_evt[1:800,1:0]
(XEN) [00000032009481a6] ACPI: 32/64X FACS address mismatch in FADT - bd1e7000/0000000000000000, using 32
(XEN) [000000320143c3d6] ACPI: wakeup_vec[bd1e700c], vec_size[20]
(XEN) [0000003201aadd00] Overriding APIC driver with bigsmp
(XEN) [000000320204020a] ACPI: IOAPIC (id[0x21] address[0xfec00000] gsi_base[0])
(XEN) [00000032026202de] IOAPIC[0]: apic_id 33, version 33, address 0xfec00000, GSI 0-23
(XEN) [0000003203127c80] ACPI: IOAPIC (id[0x22] address[0xfec01000] gsi_base[24])
(XEN) [0000003203c16b6a] IOAPIC[1]: apic_id 34, version 33, address 0xfec01000, GSI 24-55
(XEN) [0000003204729528] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) [0000003205211680] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
(XEN) [0000003205d869a8] ACPI: HPET id: 0x10228201 base: 0xfed00000
(XEN) [00000032062f609e] Using ACPI (MADT) for SMP configuration information
(XEN) [0000003206885286] SMP: Allowing 32 CPUs (0 hotplug CPUs)
(XEN) [0000003206e1dd2a] IRQ limits: 56 GSI, 6600 MSI/MSI-X
(XEN) [000000320759f632] AMD-Vi: IOMMU Extended Features:
(XEN) [0000003207b0b120] - Peripheral Page Service Request
(XEN) [000000320808cb7a] - NX bit
(XEN) [0000003208603530] - Invalidate All Command
(XEN) [0000003208b77a1c] - Performance Counters
(XEN) [0000003209156bee] - Host Address Translation Size: 0x2
(XEN) [00000032096d7c52] - Guest Address Translation Size: 0
(XEN) [0000003209c5cfc8] - Guest CR3 Root Table Level: 0x1
(XEN) [000000320a1e1eba] - Maximum PASID: 0xf
(XEN) [000000320a75db50] - SMI Filter Register: 0x1
(XEN) [000000320acd6d22] - SMI Filter Register Count: 0x2
(XEN) [000000320b2569bc] - Guest Virtual APIC Modes: 0x1
(XEN) [000000320b7d79dc] - Dual PPR Log: 0x2
(XEN) [000000320bd51384] - Dual Event Log: 0x2
(XEN) [000000320c32c530] - User / Supervisor Page Protection
(XEN) [000000320c8acd36] - Device Table Segmentation: 0x3
(XEN) [000000320ce317e8] - PPR Log Overflow Early Warning
(XEN) [000000320d3b4144] - PPR Automatic Response
(XEN) [000000320d932dcc] - Memory Access Routing and Control: 0x1
(XEN) [000000320deb6e88] - Block StopMark Message
(XEN) [000000320e43a85c] - Performance Optimization
(XEN) [000000320e9b74c0] - MSI Capability MMIO Access
(XEN) [000000320ef35554] - Guest I/O Protection
(XEN) [000000320f4b18b0] - Host Access
(XEN) [000000320fa8af22] - Enhanced PPR Handling
(XEN) [000000320fffee14] - Attribute Forward
(XEN) [0000003210576e1a] - Virtualized IOMMU
(XEN) [0000003210aeca88] - VMGuard I/O Support
(XEN) [0000003211063ede] - VM Table Size: 0x2
(XEN) [0000003212e3b0c0] Switched to APIC driver x2apic_mixed
(XEN) [000000321340d808] CPU0: 2200 ... 3400 MHz
(XEN) [000000321397381e] xstate: size: 0x988 and states: 0x207
(XEN) [0000003213f14ff0] CPU0: AMD Fam19h machine check reporting enabled
(XEN) [000000321448dd82] Speculative mitigation facilities:
(XEN) [0000003214a158f2] Hardware hints: STIBP_ALWAYS IBRS_FAST IBRS_SAME_MODE
(XEN) [0000003214fa625c] Hardware features: IBPB IBRS STIBP SSBD PSFD
(XEN) [0000003215538e4e] Compiled-in support: INDIRECT_THUNK RETURN_THUNK HARDEN_ARRAY HARDEN_BRANCH HARDEN_GUEST_ACCESS HARDEN_LOCK
(XEN) [0000003216059dae] Xen settings: BTI-Thunk: JMP, SPEC_CTRL: IBRS+ STIBP+ SSBD- PSFD-, Other: BRANCH_HARDEN
(XEN) [0000003216bd4faa] Support for HVM VMs: MSR_SPEC_CTRL MSR_VIRT_SPEC_CTRL RSB IBPB-entry
(XEN) [00000032176cfae8] Support for PV VMs: IBPB-entry
(XEN) [0000003217c41ca2] XPTI (64-bit PV only): Dom0 disabled, DomU disabled (without PCID)
(XEN) [0000003218746366] PV L1TF shadowing: Dom0 disabled, DomU disabled
(XEN) [0000003218cc07ae] Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) [000000321925249e] Initializing Credit2 scheduler
(XEN) [00000032197de012] load_precision_shift: 18
(XEN) [0000003219dbfc42] load_window_shift: 30
(XEN) [000000321a33a024] underload_balance_tolerance: 0
(XEN) [000000321a8b8030] overload_balance_tolerance: -3
(XEN) [000000321ae3a98c] runqueues arrangement: socket
(XEN) [000000321b3bc20a] cap enforcement granularity: 10ms
(XEN) [000000321b93ff30] load tracking window length 1073741824 ns
(XEN) [00000032260c4b72] Platform timer is 14.318MHz HPET
(XEN) [ 0.735054] Detected 3393.608 MHz processor.
(XEN) [ 0.739256] Freed 1024kB unused BSS memory
(XEN) [ 0.740930] alt table ffff82d0404533b8 -> ffff82d040462f64
(XEN) [ 0.754810] AMD-Vi: IOMMU 0 Enabled.
(XEN) [ 0.756476] I/O virtualisation enabled
(XEN) [ 0.758165] - Dom0 mode: Relaxed
(XEN) [ 0.759850] Interrupt remapping enabled
(XEN) [ 0.761541] Enabling APIC mode. Using 2 I/O APICs
(XEN) [ 0.763944] ENABLING IO-APIC IRQs
(XEN) [ 0.765606] -> Using new ACK method
(XEN) [ 0.767473] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) [ 0.919255] Wallclock source: CMOS RTC
(XEN) [2026-10-18 19:57:34.000] Defaulting to alternative key handling; send 'A' to switch to normal mode.
(XEN) [2026-10-18 19:57:34.003] Allocated console ring of 128 KiB.
(XEN) [2026-10-18 19:57:34.005] HVM: ASIDs enabled.
(XEN) [2026-10-18 19:57:34.006] SVM: Supported advanced features:
(XEN) [2026-10-18 19:57:34.008] - Nested Page Tables (NPT)
(XEN) [2026-10-18 19:57:34.010] - Last Branch Record (LBR) Virtualisation
(XEN) [2026-10-18 19:57:34.012] - Next-RIP Saved on #VMEXIT
(XEN) [2026-10-18 19:57:34.013] - VMCB Clean Bits
(XEN) [2026-10-18 19:57:34.015] - TLB flush by ASID
(XEN) [2026-10-18 19:57:34.017] - DecodeAssists
(XEN) [2026-10-18 19:57:34.018] - Virtual VMLOAD/VMSAVE
(XEN) [2026-10-18 19:57:34.020] - Virtual GIF
(XEN) [2026-10-18 19:57:34.022] - Pause-Intercept Filter
(XEN) [2026-10-18 19:57:34.024] - Pause-Intercept Filter Threshold
(XEN) [2026-10-18 19:57:34.025] - TSC Rate MSR
(XEN) [2026-10-18 19:57:34.027] - NPT Supervisor Shadow Stack
(XEN) [2026-10-18 19:57:34.029] - MSR_SPEC_CTRL virtualisation
(XEN) [2026-10-18 19:57:34.030] HVM: SVM enabled
(XEN) [2026-10-18 19:57:34.032] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [2026-10-18 19:57:34.034] HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) [2026-10-18 19:57:34.036] alt table ffff82d0404533b8 -> ffff82d040462f64
(XEN) [2026-10-18 19:57:34.042] Brought up 32 CPUs
(XEN) [2026-10-18 19:57:34.046] Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) [2026-10-18 19:57:34.050] Initializing Credit2 scheduler
(XEN) [2026-10-18 19:57:34.052] load_precision_shift: 18
(XEN) [2026-10-18 19:57:34.053] load_window_shift: 30
(XEN) [2026-10-18 19:57:34.055] underload_balance_tolerance: 0
(XEN) [2026-10-18 19:57:34.057] overload_balance_tolerance: -3
(XEN) [2026-10-18 19:57:34.059] runqueues arrangement: socket
(XEN) [2026-10-18 19:57:34.060] cap enforcement granularity: 10ms
(XEN) [2026-10-18 19:57:34.062] load tracking window length 1073741824 ns
(XEN) [2026-10-18 19:57:34.064] Adding cpu 0 to runqueue 0
(XEN) [2026-10-18 19:57:34.065] First cpu on runqueue, activating
(XEN) [2026-10-18 19:57:34.067] Adding cpu 1 to runqueue 0
(XEN) [2026-10-18 19:57:34.069] Adding cpu 2 to runqueue 0
(XEN) [2026-10-18 19:57:34.071] Adding cpu 3 to runqueue 0
(XEN) [2026-10-18 19:57:34.073] Adding cpu 4 to runqueue 0
(XEN) [2026-10-18 19:57:34.074] Adding cpu 5 to runqueue 0
(XEN) [2026-10-18 19:57:34.076] Adding cpu 6 to runqueue 0
(XEN) [2026-10-18 19:57:34.078] Adding cpu 7 to runqueue 0
(XEN) [2026-10-18 19:57:34.079] Adding cpu 8 to runqueue 0
(XEN) [2026-10-18 19:57:34.081] Adding cpu 9 to runqueue 0
(XEN) [2026-10-18 19:57:34.083] Adding cpu 10 to runqueue 0
(XEN) [2026-10-18 19:57:34.085] Adding cpu 11 to runqueue 0
(XEN) [2026-10-18 19:57:34.087] Adding cpu 12 to runqueue 0
(XEN) [2026-10-18 19:57:34.088] Adding cpu 13 to runqueue 0
(XEN) [2026-10-18 19:57:34.090] Adding cpu 14 to runqueue 0
(XEN) [2026-10-18 19:57:34.092] Adding cpu 15 to runqueue 0
(XEN) [2026-10-18 19:57:34.093] Adding cpu 16 to runqueue 1
(XEN) [2026-10-18 19:57:34.095] First cpu on runqueue, activating
(XEN) [2026-10-18 19:57:34.097] Adding cpu 17 to runqueue 1
(XEN) [2026-10-18 19:57:34.099] Adding cpu 18 to runqueue 1
(XEN) [2026-10-18 19:57:34.100] Adding cpu 19 to runqueue 1
(XEN) [2026-10-18 19:57:34.102] Adding cpu 20 to runqueue 1
(XEN) [2026-10-18 19:57:34.104] Adding cpu 21 to runqueue 1
(XEN) [2026-10-18 19:57:34.106] Adding cpu 22 to runqueue 1
(XEN) [2026-10-18 19:57:34.107] Adding cpu 23 to runqueue 1
(XEN) [2026-10-18 19:57:34.109] Adding cpu 24 to runqueue 1
(XEN) [2026-10-18 19:57:34.111] Adding cpu 25 to runqueue 1
(XEN) [2026-10-18 19:57:34.113] Adding cpu 26 to runqueue 1
(XEN) [2026-10-18 19:57:34.114] Adding cpu 27 to runqueue 1
(XEN) [2026-10-18 19:57:34.116] Adding cpu 28 to runqueue 1
(XEN) [2026-10-18 19:57:34.118] Adding cpu 29 to runqueue 1
(XEN) [2026-10-18 19:57:34.120] Adding cpu 30 to runqueue 1
(XEN) [2026-10-18 19:57:34.122] Adding cpu 31 to runqueue 1
(XEN) [2026-10-18 19:57:34.123] mcheck_poll: Machine check polling timer started.
(XEN) [2026-10-18 19:57:34.127] mtrr: your CPUs had inconsistent variable MTRR settings
(XEN) [2026-10-18 19:57:34.130] mtrr: probably your BIOS does not setup all CPUs.
(XEN) [2026-10-18 19:57:34.134] mtrr: corrected configuration.
(XEN) [2026-10-18 19:57:34.136] MTRR default type: uncachable
(XEN) [2026-10-18 19:57:34.137] MTRR fixed ranges enabled:
(XEN) [2026-10-18 19:57:34.139] 00000-9ffff write-back
(XEN) [2026-10-18 19:57:34.141] a0000-bffff write-through
(XEN) [2026-10-18 19:57:34.142] c0000-fffff write-protect
(XEN) [2026-10-18 19:57:34.144] MTRR variable ranges enabled:
(XEN) [2026-10-18 19:57:34.146] 0 base 00000000000 mask ffff80000000 write-back
(XEN) [2026-10-18 19:57:34.149] 1 base 00080000000 mask ffffc0000000 write-back
(XEN) [2026-10-18 19:57:34.153] 2 base 000bccb0000 mask ffffffff0000 uncachable
(XEN) [2026-10-18 19:57:34.156] 3 disabled
(XEN) [2026-10-18 19:57:34.158] 4 disabled
(XEN) [2026-10-18 19:57:34.160] 5 disabled
(XEN) [2026-10-18 19:57:34.161] 6 disabled
(XEN) [2026-10-18 19:57:34.163] 7 disabled
(XEN) [2026-10-18 19:57:34.165] TOM2: 002040000000 (WB)
(XEN) [2026-10-18 19:57:34.187] NX (Execute Disable) protection active
(XEN) [2026-10-18 19:57:34.189] d0 has maximum 6656 PIRQs
(XEN) [2026-10-18 19:57:34.190] *** Building a PV Dom0 ***
(XEN) [2026-10-18 19:57:34.261] Xen kernel: 64-bit, lsb
(XEN) [2026-10-18 19:57:34.262] Dom0 kernel: 64-bit, lsb, paddr 0x1000000 -> 0x4200000
(XEN) [2026-10-18 19:57:34.266] PHYSICAL MEMORY ARRANGEMENT:
(XEN) [2026-10-18 19:57:34.268] Dom0 alloc.: 0000001fd8000000->0000001fe0000000 (1001447 pages to be allocated)
(XEN) [2026-10-18 19:57:34.271] Init. ramdisk: 000000203b9e7000->000000203f1ff5fd
(XEN) [2026-10-18 19:57:34.275] VIRTUAL MEMORY ARRANGEMENT:
(XEN) [2026-10-18 19:57:34.276] Loaded kernel: ffffffff81000000->ffffffff84200000
(XEN) [2026-10-18 19:57:34.280] Phys-Mach map: 0000008000000000->0000008000800000
(XEN) [2026-10-18 19:57:34.283] Start info: ffffffff84200000->ffffffff842004b8
(XEN) [2026-10-18 19:57:34.287] Page tables: ffffffff84201000->ffffffff84226000
(XEN) [2026-10-18 19:57:34.290] Boot stack: ffffffff84226000->ffffffff84227000
(XEN) [2026-10-18 19:57:34.294] TOTAL: ffffffff80000000->ffffffff84400000
(XEN) [2026-10-18 19:57:34.297] ENTRY ADDRESS: ffffffff833657b0
(XEN) [2026-10-18 19:57:34.299] Dom0 has maximum 32 VCPUs
(XEN) [2026-10-18 19:57:34.317] Initial low memory virq threshold set at 0x4000 pages.
(XEN) [2026-10-18 19:57:34.321] Scrubbing Free RAM in background
(XEN) [2026-10-18 19:57:34.323] Std. Loglevel: Errors, warnings and info
(XEN) [2026-10-18 19:57:34.324] Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) [2026-10-18 19:57:34.328] Xen is relinquishing VGA console.
(XEN) [2026-10-18 19:57:34.330] *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) [2026-10-18 19:57:34.331] Freed 656kB init memory
(XEN) [2026-10-19 08:58:18.727] Xen BUG at arch/x86/cpu/mcheck/mce_amd.c:100
(XEN) [2026-10-19 08:58:18.727] ----[ Xen-4.20.5-pre x86_64 debug=n Tainted: S ]----
(XEN) [2026-10-19 08:58:18.727] CPU: 2
(XEN) [2026-10-19 08:58:18.727] RIP: e008:[<ffff82d04028a650>] arch/x86/cpu/mcheck/mce_amd.c#mc_ec2type.part.0+0/0x10
(XEN) [2026-10-19 08:58:18.727] RFLAGS: 0000000000010046 CONTEXT: hypervisor
(XEN) [2026-10-19 08:58:18.727] rax: 0000000000000001 rbx: 0000000000000445 rcx: 0000000000000000
(XEN) [2026-10-19 08:58:18.727] rdx: 0000000000000000 rsi: 0000000000000001 rdi: 0000000000000000
(XEN) [2026-10-19 08:58:18.727] rbp: 0000000000000001 rsp: ffff832036a87c38 r8: ffff82d04059af00
(XEN) [2026-10-19 08:58:18.727] r9: 0000000000000000 r10: ffff832036a8b000 r11: 0000000000000000
(XEN) [2026-10-19 08:58:18.727] r12: 0000000000000011 r13: ffff832036a33048 r14: ffff832036aebb70
(XEN) [2026-10-19 08:58:18.727] r15: 0000000000000446 cr0: 000000008005003b cr4: 0000000000f506e0
(XEN) [2026-10-19 08:58:18.727] cr3: 000000186e846000 cr2: 00007f1682f67a08
(XEN) [2026-10-19 08:58:18.727] fsb: 0000000000000000 gsb: ffff88817aa00000 gss: 0000000000000000
(XEN) [2026-10-19 08:58:18.727] ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) [2026-10-19 08:58:18.727] Xen code around <ffff82d04028a650> (arch/x86/cpu/mcheck/mce_amd.c#mc_ec2type.part.0):
(XEN) [2026-10-19 08:58:18.727] 90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90 90 90
(XEN) [2026-10-19 08:58:18.727] Xen stack trace from rsp=ffff832036a87c38:
(XEN) [2026-10-19 08:58:18.727] ffff82d04028a699 ffff82d04028cfdd ffff82d0405832d0 ffff832036ae1ea0
(XEN) [2026-10-19 08:58:18.727] 0000044500000001 00000001ee70c001 ffff832036aebd70 ffff832036ae1ea8
(XEN) [2026-10-19 08:58:18.727] 0001010100000000 0000000000000000 0000000000000000 ffff832036a87ce0
(XEN) [2026-10-19 08:58:18.727] 0000000800000001 ffff832036a8b000 ffff82d040588f80 0000000000000002
(XEN) [2026-10-19 08:58:18.727] 00000000000000fb 0000000000000000 ffff82d040588f80 0000000000000000
(XEN) [2026-10-19 08:58:18.727] ffff82d04028a35c ffff832036a87dd7 ffff82d0402bfc45 ffff82d040588f80
(XEN) [2026-10-19 08:58:18.727] ffff82d040240e8c 00000000000000fb 0000000000000000 ffff82d0403123d5
(XEN) [2026-10-19 08:58:18.727] ffff832036a87d40 0000000000000000 8000000000000005 ffff832036a87d98
(XEN) [2026-10-19 08:58:18.727] ffff831fe66bfaa0 000000000017aa2f 0000000000000001 0000000000000000
(XEN) [2026-10-19 08:58:18.727] 0000000000000000 0000000000000000 0000000000000000 ffff832036a87fff
(XEN) [2026-10-19 08:58:18.727] 0000000000000000 ffff82d040201915 ffff82d040466760 ffff82d040588f80
(XEN) [2026-10-19 08:58:18.727] 00002a9b609476e2 0000000000000002 ffff832036a88010 ffff832036a88068
(XEN) [2026-10-19 08:58:18.727] ffff8313740dc030 ffff82d040588f80 0000000000000002 0000000000000002
(XEN) [2026-10-19 08:58:18.727] 0000000000000000 0000000000000048 0000000000000000 ffff832036a87ef8
(XEN) [2026-10-19 08:58:18.727] 0000000000000048 000000fb00000000 ffff82d0402883ca 000000000000e008
(XEN) [2026-10-19 08:58:18.727] 0000000000000246 ffff832036a87e38 0000000000000000 00002a9b609476e2
(XEN) [2026-10-19 08:58:18.727] ffff82d0402887cf 0000000000000000 0000000000000000 0000000000000000
(XEN) [2026-10-19 08:58:18.727] ffff82d0000000ff 0000000000000002 ffff82d040466760 0000000000000002
(XEN) [2026-10-19 08:58:18.727] 0000000000000002 ffff82d04046bda0 0000000000007fff ffff82d040588f80
(XEN) [2026-10-19 08:58:18.727] ffff82d040466760 ffff82d040302cf1 ffff82d040589080 ffff82d040582210
(XEN) [2026-10-19 08:58:18.727] Xen call trace:
(XEN) [2026-10-19 08:58:18.727] [<ffff82d04028a650>] R arch/x86/cpu/mcheck/mce_amd.c#mc_ec2type.part.0+0/0x10
(XEN) [2026-10-19 08:58:18.727] [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40
(XEN) [2026-10-19 08:58:18.727] [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840
(XEN) [2026-10-19 08:58:18.727] [<ffff82d04028a35c>] S arch/x86/cpu/mcheck/amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
(XEN) [2026-10-19 08:58:18.727] [<ffff82d0402bfc45>] S arch/x86/hvm/hvm.c#__hvm_copy+0x1a5/0x470
(XEN) [2026-10-19 08:58:18.727] [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90
(XEN) [2026-10-19 08:58:18.727] [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
(XEN) [2026-10-19 08:58:18.727] [<ffff82d040201915>] S common_interrupt+0x135/0x150
(XEN) [2026-10-19 08:58:18.727] [<ffff82d0402883ca>] S arch/x86/acpi/cpu_idle.c#acpi_idle_do_entry+0x8a/0x140
(XEN) [2026-10-19 08:58:18.727] [<ffff82d0402887cf>] S arch/x86/acpi/cpu_idle.c#acpi_processor_idle+0x1ef/0x650
(XEN) [2026-10-19 08:58:18.727] [<ffff82d040302cf1>] S arch/x86/domain.c#idle_loop+0xc1/0x130
(XEN) [2026-10-19 08:58:18.727] [<ffff82d040302c30>] S arch/x86/domain.c#idle_loop+0/0x130
(XEN) [2026-10-19 08:58:18.727] [<ffff82d040304edc>] S context_switch+0x1ec/0x900
(XEN) [2026-10-19 08:58:18.727]
(XEN) [2026-10-19 08:58:18.727]
(XEN) [2026-10-19 08:58:18.727] ****************************************
(XEN) [2026-10-19 08:58:18.727] Panic on CPU 2:
(XEN) [2026-10-19 08:58:18.727] Xen BUG at arch/x86/cpu/mcheck/mce_amd.c:100
(XEN) [2026-10-19 08:58:18.727] ****************************************
(XEN) [2026-10-19 08:58:18.727]
(XEN) [2026-10-19 08:58:18.727] Reboot in five seconds...
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot
2026-10-23 7:32 [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot Peter Deggendorfer
@ 2026-10-23 13:14 ` Andrew Cooper
2026-10-23 21:07 ` Peter Deggendorfer
2026-10-23 23:51 ` Teddy Astie
1 sibling, 1 reply; 8+ messages in thread
From: Andrew Cooper @ 2026-10-23 13:14 UTC (permalink / raw)
To: Peter Deggendorfer, xen-devel
Cc: Andrew Cooper, Jan Beulich, Roger Pau Monné, Teddy Astie,
Jason Andryuk, Soham Dandapat
On 23/10/2026 8:32 am, Peter Deggendorfer wrote:
> Hey Xen Team,
>
> I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE
> polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since
> 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.
>
> Environment
> -----------
> Xen: 4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still
> present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1)
> dom0: Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs
> CPU: AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X,
> microcode 0x0a201210
> Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems
>
> The panic
> ---------
> (XEN) [2026-10-19 08:58:18.727] Xen BUG at
> arch/x86/cpu/mcheck/mce_amd.c:100
> (XEN) CPU: 2
> (XEN) RIP: e008:[<ffff82d04028a650>]
> mce_amd.c#mc_ec2type.part.0+0/0x10
> (XEN) rbx: 0000000000000445 r12: 0000000000000011 r15:
> 0000000000000446
> (XEN) Xen code around <ffff82d04028a650>:
> (XEN) 90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90
> 90 90
> (XEN) Xen call trace:
> (XEN) [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10
> (XEN) [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40
> (XEN) [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840
> (XEN) [<ffff82d04028a35c>] S
> amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
> (XEN) [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90
> (XEN) [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
> (XEN) [<ffff82d040201915>] S common_interrupt+0x135/0x150
> (XEN) [<ffff82d0402883ca>] S cpu_idle.c#acpi_idle_do_entry+0x8a/0x140
> (XEN) [<ffff82d0402887cf>] S
> cpu_idle.c#acpi_processor_idle+0x1ef/0x650
> (XEN) [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130
> (XEN) Panic on CPU 2
>
> The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is
> the "unreachable" default branch of mc_ec2type() -- the extended error
> code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.
I think it's safe to just drop the BUG().
That code is literally worse than useless, because if nothing else it's
just hiding the diagnostics.
I guess it means AMD have added new statuses.
>
> Across the eight occurrences, r12 appears to be the bank index from the
> loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS)
> and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.
>
> Not a hardware defect
> ---------------------
> - Started within days of upgrading two hosts from Debian bookworm
> (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17 with no
> incident.
> - Three different banks, so not a single failing block.
> - Updating to the current Debian package (4.20.5-pre, built 2026-10-16)
> changed nothing.
>
> Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks
> updating").
>
> Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
> Filter bogus hardware errors on Zen3 clients"), where bogus MCA values
> on Zen 3 are filtered rather than treated as errors.
> https://ratatoskr.run/stable/2026/04/7143905
From above, you're on a Vermeer, not a Cezanne, so I don't think that
Linux patch is relevant.
However, your observation about "fix mca_allbanks updating" quite
possibly is. That was backported into Xen 4.20.1 as 247b708b1400a, and
Debian has 4.20.3
Are you able to experiment with that patch reverted?
Alternatively, are you able to experiment with the BUG() removed?
>
> Diagnostics / workaround
> ------------------------
> mce_verbosity=verbose yields nothing useful.
> max_cstate=1 made no difference. mce=false is the only thing that stops
> it, and it is what I'm running now.
The fact that mce=false isn't causing machine resets means it's clearly
something the hardware thinks isn't fatal.
But, I am concerned by the fact it's always on CPU 2, and multiple
banks. Is this true for both systems?
>
> One more thing: the host does not actually reboot after the panic.
> "Reboot in five seconds" is printed, but the ring buffer continues with
> normal activity minutes later and the remaining 31 CPUs keep running.
> CPU 2 stays wedged, and "xl debug-keys t" then hangs. Afterwards dom0
> and guest vCPUs start losing timer interrupts permanently (RCU stalls
> with a frozen timer-softirq count while dynticks keeps advancing), so
> VMs hang silently hours or days later.
I've recently noticed that crashing with interrupts off doesn't reboot
any more. I thought it was something local to my development, but
apparently not if you're seeing the same.
>
>
> I've attached the full xl dmesg log of the last crash, and I'm happy
> to provide more if needed. I can also test patches.
>
> Thanks for your work, and thanks in advance for taking a look.
>
> Peter
Thankyou,
~Andrew
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot
2026-10-23 13:14 ` Andrew Cooper
@ 2026-10-23 21:07 ` Peter Deggendorfer
2026-10-25 12:32 ` Peter Deggendorfer
0 siblings, 1 reply; 8+ messages in thread
From: Peter Deggendorfer @ 2026-10-23 21:07 UTC (permalink / raw)
To: Andrew Cooper, xen-devel
Cc: Jan Beulich, Roger Pau Monné, Teddy Astie, Jason Andryuk,
Soham Dandapat
Hi Andrew,
Thanks a lot for the quick response.
> On 23/10/2026 8:32 am, Peter Deggendorfer wrote:
>> Hey Xen Team,
>>
>> I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE
>> polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since
>> 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.
>>
>> Environment
>> -----------
>> Xen: 4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still
>> present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1)
>> dom0: Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs
>> CPU: AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X,
>> microcode 0x0a201210
>> Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems
>>
>> The panic
>> ---------
>> (XEN) [2026-10-19 08:58:18.727] Xen BUG at
>> arch/x86/cpu/mcheck/mce_amd.c:100
>> (XEN) CPU: 2
>> (XEN) RIP: e008:[<ffff82d04028a650>]
>> mce_amd.c#mc_ec2type.part.0+0/0x10
>> (XEN) rbx: 0000000000000445 r12: 0000000000000011 r15:
>> 0000000000000446
>> (XEN) Xen code around <ffff82d04028a650>:
>> (XEN) 90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90
>> 90 90
>> (XEN) Xen call trace:
>> (XEN) [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10
>> (XEN) [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40
>> (XEN) [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840
>> (XEN) [<ffff82d04028a35c>] S
>> amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
>> (XEN) [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90
>> (XEN) [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
>> (XEN) [<ffff82d040201915>] S common_interrupt+0x135/0x150
>> (XEN) [<ffff82d0402883ca>] S cpu_idle.c#acpi_idle_do_entry+0x8a/0x140
>> (XEN) [<ffff82d0402887cf>] S
>> cpu_idle.c#acpi_processor_idle+0x1ef/0x650
>> (XEN) [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130
>> (XEN) Panic on CPU 2
>>
>> The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is
>> the "unreachable" default branch of mc_ec2type() -- the extended error
>> code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.
> I think it's safe to just drop the BUG().
>
> That code is literally worse than useless, because if nothing else it's
> just hiding the diagnostics.
>
> I guess it means AMD have added new statuses.
>
>> Across the eight occurrences, r12 appears to be the bank index from the
>> loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS)
>> and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.
>>
>> Not a hardware defect
>> ---------------------
>> - Started within days of upgrading two hosts from Debian bookworm
>> (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17 with no
>> incident.
>> - Three different banks, so not a single failing block.
>> - Updating to the current Debian package (4.20.5-pre, built 2026-10-16)
>> changed nothing.
>>
>> Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks
>> updating").
>>
>> Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
>> Filter bogus hardware errors on Zen3 clients"), where bogus MCA values
>> on Zen 3 are filtered rather than treated as errors.
>> https://ratatoskr.run/stable/2026/04/7143905
> From above, you're on a Vermeer, not a Cezanne, so I don't think that
> Linux patch is relevant.
>
> However, your observation about "fix mca_allbanks updating" quite
> possibly is. That was backported into Xen 4.20.1 as 247b708b1400a, and
> Debian has 4.20.3
>
> Are you able to experiment with that patch reverted?
>
> Alternatively, are you able to experiment with the BUG() removed?
I went with dropping the BUG() rather than the revert.
Reverting 247b708b1400a should stop any bank from being read at
all, which is effectively what mce=false already does here, so I don't
expect it to tell us anything new. I'll run it too if you'd like.
I'm not familiar with the Xen codebase, but I've built a package with
mc_ec2type() changed to:
if ( errorcode & MC_EC_TLB_TYPE )
return MC_EC_TLB_TYPE;
printk(XENLOG_WARNING "MCE: unknown AMD error code %#x\n", errorcode);
return 0;
Crashes have been 12 hours to 2 weeks apart, so it may take a while
before I can report back.
>> Diagnostics / workaround
>> ------------------------
>> mce_verbosity=verbose yields nothing useful.
>> max_cstate=1 made no difference. mce=false is the only thing that stops
>> it, and it is what I'm running now.
> The fact that mce=false isn't causing machine resets means it's clearly
> something the hardware thinks isn't fatal.
>
> But, I am concerned by the fact it's always on CPU 2, and multiple
> banks. Is this true for both systems?
To clarify one thing from my report: it is actually a single host whose
complete hardware (board, CPU, RAM) was replaced on
2026-10-07, keeping only the disks and thus the installation. The
crashes happened both before and after that swap. 10 more hosts
with same hardware still run on bookworm (xen 4.17) for years now
without any incidents.
So yes, always CPU 2. The bank index varies: on the current hardware
I have seen banks 23, 17 and 10 across four crashes, and banks 23 and
17 on the previous hardware.
>> One more thing: the host does not actually reboot after the panic.
>> "Reboot in five seconds" is printed, but the ring buffer continues with
>> normal activity minutes later and the remaining 31 CPUs keep running.
>> CPU 2 stays wedged, and "xl debug-keys t" then hangs. Afterwards dom0
>> and guest vCPUs start losing timer interrupts permanently (RCU stalls
>> with a frozen timer-softirq count while dynticks keeps advancing), so
>> VMs hang silently hours or days later.
> I've recently noticed that crashing with interrupts off doesn't reboot
> any more. I thought it was something local to my development, but
> apparently not if you're seeing the same.
>
>>
>> I've attached the full xl dmesg log of the last crash, and I'm happy
>> to provide more if needed. I can also test patches.
>>
>> Thanks for your work, and thanks in advance for taking a look.
>>
>> Peter
> Thankyou,
>
> ~Andrew
I have full xl dmesg logs for almost all crashes. Happy to send any of
them.
Peter
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot
2026-10-23 21:07 ` Peter Deggendorfer
@ 2026-10-25 12:32 ` Peter Deggendorfer
2026-10-26 10:53 ` Jan Beulich
0 siblings, 1 reply; 8+ messages in thread
From: Peter Deggendorfer @ 2026-10-25 12:32 UTC (permalink / raw)
To: Andrew Cooper, xen-devel
Cc: Jan Beulich, Roger Pau Monné, Teddy Astie, Jason Andryuk,
Soham Dandapat
> Hi Andrew,
>
> Thanks a lot for the quick response.
>
>
>> On 23/10/2026 8:32 am, Peter Deggendorfer wrote:
>>> Hey Xen Team,
>>>
>>> I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE
>>> polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since
>>> 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.
>>>
>>> Environment
>>> -----------
>>> Xen: 4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still
>>> present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1)
>>> dom0: Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs
>>> CPU: AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X,
>>> microcode 0x0a201210
>>> Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems
>>>
>>> The panic
>>> ---------
>>> (XEN) [2026-10-19 08:58:18.727] Xen BUG at
>>> arch/x86/cpu/mcheck/mce_amd.c:100
>>> (XEN) CPU: 2
>>> (XEN) RIP: e008:[<ffff82d04028a650>]
>>> mce_amd.c#mc_ec2type.part.0+0/0x10
>>> (XEN) rbx: 0000000000000445 r12: 0000000000000011 r15:
>>> 0000000000000446
>>> (XEN) Xen code around <ffff82d04028a650>:
>>> (XEN) 90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90
>>> 90 90
>>> (XEN) Xen call trace:
>>> (XEN) [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10
>>> (XEN) [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40
>>> (XEN) [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840
>>> (XEN) [<ffff82d04028a35c>] S
>>> amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
>>> (XEN) [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90
>>> (XEN) [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
>>> (XEN) [<ffff82d040201915>] S common_interrupt+0x135/0x150
>>> (XEN) [<ffff82d0402883ca>] S
>>> cpu_idle.c#acpi_idle_do_entry+0x8a/0x140
>>> (XEN) [<ffff82d0402887cf>] S
>>> cpu_idle.c#acpi_processor_idle+0x1ef/0x650
>>> (XEN) [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130
>>> (XEN) Panic on CPU 2
>>>
>>> The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is
>>> the "unreachable" default branch of mc_ec2type() -- the extended error
>>> code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.
>> I think it's safe to just drop the BUG().
>>
>> That code is literally worse than useless, because if nothing else it's
>> just hiding the diagnostics.
>>
>> I guess it means AMD have added new statuses.
>>
>>> Across the eight occurrences, r12 appears to be the bank index from the
>>> loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS)
>>> and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.
>>>
>>> Not a hardware defect
>>> ---------------------
>>> - Started within days of upgrading two hosts from Debian bookworm
>>> (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17
>>> with no
>>> incident.
>>> - Three different banks, so not a single failing block.
>>> - Updating to the current Debian package (4.20.5-pre, built 2026-10-16)
>>> changed nothing.
>>>
>>> Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks
>>> updating").
>>>
>>> Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
>>> Filter bogus hardware errors on Zen3 clients"), where bogus MCA values
>>> on Zen 3 are filtered rather than treated as errors.
>>> https://ratatoskr.run/stable/2026/04/7143905
>> From above, you're on a Vermeer, not a Cezanne, so I don't think that
>> Linux patch is relevant.
>>
>> However, your observation about "fix mca_allbanks updating" quite
>> possibly is. That was backported into Xen 4.20.1 as 247b708b1400a, and
>> Debian has 4.20.3
>>
>> Are you able to experiment with that patch reverted?
>>
>> Alternatively, are you able to experiment with the BUG() removed?
>
> I went with dropping the BUG() rather than the revert.
> Reverting 247b708b1400a should stop any bank from being read at
> all, which is effectively what mce=false already does here, so I don't
> expect it to tell us anything new. I'll run it too if you'd like.
>
> I'm not familiar with the Xen codebase, but I've built a package with
> mc_ec2type() changed to:
>
> if ( errorcode & MC_EC_TLB_TYPE )
> return MC_EC_TLB_TYPE;
>
> printk(XENLOG_WARNING "MCE: unknown AMD error code %#x\n",
> errorcode);
>
> return 0;
>
>
> Crashes have been 12 hours to 2 weeks apart, so it may take a while
> before I can report back.
After ~31 hours of uptime:
(XEN) [2026-10-25 04:06:27.260] MCE: unknown AMD error code 0
No panic, no wedged CPU, no RCU stalls, no hung guests. Host and all
VMs running normally.
I also dumped /dev/mcelog in dom0 and got the following records:
00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000030: 0000 0000 0000 0000 0200 0000 0000 0000 ................
00000040: 0010 0201 0200 0000 0000 0000 0200 0000 ................
00000050: 1c01 0000 0000 0000 0000 0000 0000 0000 ................
00000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000000a0: 0000 0000 0000 0000 0200 0000 0000 0000 ................
000000b0: 0017 0201 0200 0000 0000 0000 0200 0000 ................
000000c0: 1c01 0000 0000 0000 0000 0000 0000 0000 ................
000000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>
>
>>> Diagnostics / workaround
>>> ------------------------
>>> mce_verbosity=verbose yields nothing useful.
>>> max_cstate=1 made no difference. mce=false is the only thing that stops
>>> it, and it is what I'm running now.
>> The fact that mce=false isn't causing machine resets means it's clearly
>> something the hardware thinks isn't fatal.
>>
>> But, I am concerned by the fact it's always on CPU 2, and multiple
>> banks. Is this true for both systems?
>
> To clarify one thing from my report: it is actually a single host whose
> complete hardware (board, CPU, RAM) was replaced on
> 2026-10-07, keeping only the disks and thus the installation. The
> crashes happened both before and after that swap. 10 more hosts
> with same hardware still run on bookworm (xen 4.17) for years now
> without any incidents.
>
> So yes, always CPU 2. The bank index varies: on the current hardware
> I have seen banks 23, 17 and 10 across four crashes, and banks 23 and
> 17 on the previous hardware.
Regarding the output of /dev/mcelog: if my decoding is right, three
things stand out. MCi_STATUS is zero, i.e. these banks are reporting
nothing at all -- no error, no garbage value. Both records are again
from CPU 2, and bank 23 is one of the banks that appeared in the
panics, bank 16 is new.
>
>>> One more thing: the host does not actually reboot after the panic.
>>> "Reboot in five seconds" is printed, but the ring buffer continues with
>>> normal activity minutes later and the remaining 31 CPUs keep running.
>>> CPU 2 stays wedged, and "xl debug-keys t" then hangs. Afterwards dom0
>>> and guest vCPUs start losing timer interrupts permanently (RCU stalls
>>> with a frozen timer-softirq count while dynticks keeps advancing), so
>>> VMs hang silently hours or days later.
>> I've recently noticed that crashing with interrupts off doesn't reboot
>> any more. I thought it was something local to my development, but
>> apparently not if you're seeing the same.
>>
>>>
>>> I've attached the full xl dmesg log of the last crash, and I'm happy
>>> to provide more if needed. I can also test patches.
>>>
>>> Thanks for your work, and thanks in advance for taking a look.
>>>
>>> Peter
>> Thankyou,
>>
>> ~Andrew
>
> I have full xl dmesg logs for almost all crashes. Happy to send any of
> them.
>
> Peter
>
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot
2026-10-25 12:32 ` Peter Deggendorfer
@ 2026-10-26 10:53 ` Jan Beulich
2026-10-26 14:48 ` Peter Deggendorfer
0 siblings, 1 reply; 8+ messages in thread
From: Jan Beulich @ 2026-10-26 10:53 UTC (permalink / raw)
To: Peter Deggendorfer
Cc: Roger Pau Monné, Teddy Astie, Jason Andryuk, Soham Dandapat,
Andrew Cooper, xen-devel
On 25.10.2026 13:32, Peter Deggendorfer wrote:
>
>> Hi Andrew,
>>
>> Thanks a lot for the quick response.
>>
>>
>>> On 23/10/2026 8:32 am, Peter Deggendorfer wrote:
>>>> Hey Xen Team,
>>>>
>>>> I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE
>>>> polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since
>>>> 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.
>>>>
>>>> Environment
>>>> -----------
>>>> Xen: 4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still
>>>> present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1)
>>>> dom0: Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs
>>>> CPU: AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X,
>>>> microcode 0x0a201210
>>>> Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems
>>>>
>>>> The panic
>>>> ---------
>>>> (XEN) [2026-10-19 08:58:18.727] Xen BUG at
>>>> arch/x86/cpu/mcheck/mce_amd.c:100
>>>> (XEN) CPU: 2
>>>> (XEN) RIP: e008:[<ffff82d04028a650>]
>>>> mce_amd.c#mc_ec2type.part.0+0/0x10
>>>> (XEN) rbx: 0000000000000445 r12: 0000000000000011 r15:
>>>> 0000000000000446
>>>> (XEN) Xen code around <ffff82d04028a650>:
>>>> (XEN) 90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90
>>>> 90 90
>>>> (XEN) Xen call trace:
>>>> (XEN) [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10
>>>> (XEN) [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40
>>>> (XEN) [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840
>>>> (XEN) [<ffff82d04028a35c>] S
>>>> amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
>>>> (XEN) [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90
>>>> (XEN) [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
>>>> (XEN) [<ffff82d040201915>] S common_interrupt+0x135/0x150
>>>> (XEN) [<ffff82d0402883ca>] S
>>>> cpu_idle.c#acpi_idle_do_entry+0x8a/0x140
>>>> (XEN) [<ffff82d0402887cf>] S
>>>> cpu_idle.c#acpi_processor_idle+0x1ef/0x650
>>>> (XEN) [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130
>>>> (XEN) Panic on CPU 2
>>>>
>>>> The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is
>>>> the "unreachable" default branch of mc_ec2type() -- the extended error
>>>> code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.
>>> I think it's safe to just drop the BUG().
>>>
>>> That code is literally worse than useless, because if nothing else it's
>>> just hiding the diagnostics.
>>>
>>> I guess it means AMD have added new statuses.
>>>
>>>> Across the eight occurrences, r12 appears to be the bank index from the
>>>> loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS)
>>>> and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.
>>>>
>>>> Not a hardware defect
>>>> ---------------------
>>>> - Started within days of upgrading two hosts from Debian bookworm
>>>> (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17
>>>> with no
>>>> incident.
>>>> - Three different banks, so not a single failing block.
>>>> - Updating to the current Debian package (4.20.5-pre, built 2026-10-16)
>>>> changed nothing.
>>>>
>>>> Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks
>>>> updating").
>>>>
>>>> Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
>>>> Filter bogus hardware errors on Zen3 clients"), where bogus MCA values
>>>> on Zen 3 are filtered rather than treated as errors.
>>>> https://ratatoskr.run/stable/2026/04/7143905
>>> From above, you're on a Vermeer, not a Cezanne, so I don't think that
>>> Linux patch is relevant.
>>>
>>> However, your observation about "fix mca_allbanks updating" quite
>>> possibly is. That was backported into Xen 4.20.1 as 247b708b1400a, and
>>> Debian has 4.20.3
>>>
>>> Are you able to experiment with that patch reverted?
>>>
>>> Alternatively, are you able to experiment with the BUG() removed?
>>
>> I went with dropping the BUG() rather than the revert.
>> Reverting 247b708b1400a should stop any bank from being read at
>> all, which is effectively what mce=false already does here, so I don't
>> expect it to tell us anything new. I'll run it too if you'd like.
>>
>> I'm not familiar with the Xen codebase, but I've built a package with
>> mc_ec2type() changed to:
>>
>> if ( errorcode & MC_EC_TLB_TYPE )
>> return MC_EC_TLB_TYPE;
>>
>> printk(XENLOG_WARNING "MCE: unknown AMD error code %#x\n",
>> errorcode);
>>
>> return 0;
>>
>>
>> Crashes have been 12 hours to 2 weeks apart, so it may take a while
>> before I can report back.
>
> After ~31 hours of uptime:
>
> (XEN) [2026-10-25 04:06:27.260] MCE: unknown AMD error code 0
>
> No panic, no wedged CPU, no RCU stalls, no hung guests. Host and all
> VMs running normally.
>
>
> I also dumped /dev/mcelog in dom0 and got the following records:
>
> 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 00000030: 0000 0000 0000 0000 0200 0000 0000 0000 ................
> 00000040: 0010 0201 0200 0000 0000 0000 0200 0000 ................
> 00000050: 1c01 0000 0000 0000 0000 0000 0000 0000 ................
> 00000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 000000a0: 0000 0000 0000 0000 0200 0000 0000 0000 ................
> 000000b0: 0017 0201 0200 0000 0000 0000 0200 0000 ................
> 000000c0: 1c01 0000 0000 0000 0000 0000 0000 0000 ................
> 000000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>
>
>>
>>
>>>> Diagnostics / workaround
>>>> ------------------------
>>>> mce_verbosity=verbose yields nothing useful.
>>>> max_cstate=1 made no difference. mce=false is the only thing that stops
>>>> it, and it is what I'm running now.
>>> The fact that mce=false isn't causing machine resets means it's clearly
>>> something the hardware thinks isn't fatal.
>>>
>>> But, I am concerned by the fact it's always on CPU 2, and multiple
>>> banks. Is this true for both systems?
>>
>> To clarify one thing from my report: it is actually a single host whose
>> complete hardware (board, CPU, RAM) was replaced on
>> 2026-10-07, keeping only the disks and thus the installation. The
>> crashes happened both before and after that swap. 10 more hosts
>> with same hardware still run on bookworm (xen 4.17) for years now
>> without any incidents.
>>
>> So yes, always CPU 2. The bank index varies: on the current hardware
>> I have seen banks 23, 17 and 10 across four crashes, and banks 23 and
>> 17 on the previous hardware.
>
> Regarding the output of /dev/mcelog: if my decoding is right, three
> things stand out. MCi_STATUS is zero, i.e. these banks are reporting
> nothing at all -- no error, no garbage value.
Well, things can't quite be that way: If MCi_STATUS was zero, the first
check in mc_amd_recoverable_scan() would prevent mc_ec2type() from
being reached in the first place. Its caller also checks MCi_STATUS_VAL
to be set.
What I find puzzling is this:
errorcode = status & (MCi_STATUS_MCA | MCi_STATUS_MSEC);
ectype = mc_ec2type(errorcode);
With errorcode (and mc_ec2type()'s parameter type) being uint16_t, what
use is it to include MCi_STATUS_MSEC in the mask? (Yet that's likely
unrelated to the issue at hand.) Considering how mc_ec2type() works, I
see no reason why status couldn't be passed to it directly (with the
parameter type widened to uint32_t). Then the BUG() there could perhaps
become BUG_ON(errorcode).
> Both records are again
> from CPU 2, and bank 23 is one of the banks that appeared in the
> panics, bank 16 is new.
Isn't it possible that the lack of 532988dccc19 ("x86/mcheck: fix
mca_allbanks updating") simply hid the issue previously?
Jan
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot
2026-10-26 10:53 ` Jan Beulich
@ 2026-10-26 14:48 ` Peter Deggendorfer
2026-10-26 15:24 ` Jan Beulich
0 siblings, 1 reply; 8+ messages in thread
From: Peter Deggendorfer @ 2026-10-26 14:48 UTC (permalink / raw)
To: Jan Beulich
Cc: Roger Pau Monné, Teddy Astie, Jason Andryuk, Soham Dandapat,
Andrew Cooper, xen-devel
Am 26.10.2026 um 11:53 schrieb Jan Beulich:
> On 25.10.2026 13:32, Peter Deggendorfer wrote:
>>> Hi Andrew,
>>>
>>> Thanks a lot for the quick response.
>>>
>>>
>>>> On 23/10/2026 8:32 am, Peter Deggendorfer wrote:
>>>>> Hey Xen Team,
>>>>>
>>>>> I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE
>>>>> polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since
>>>>> 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.
>>>>>
>>>>> Environment
>>>>> -----------
>>>>> Xen: 4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still
>>>>> present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1)
>>>>> dom0: Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs
>>>>> CPU: AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X,
>>>>> microcode 0x0a201210
>>>>> Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems
>>>>>
>>>>> The panic
>>>>> ---------
>>>>> (XEN) [2026-10-19 08:58:18.727] Xen BUG at
>>>>> arch/x86/cpu/mcheck/mce_amd.c:100
>>>>> (XEN) CPU: 2
>>>>> (XEN) RIP: e008:[<ffff82d04028a650>]
>>>>> mce_amd.c#mc_ec2type.part.0+0/0x10
>>>>> (XEN) rbx: 0000000000000445 r12: 0000000000000011 r15:
>>>>> 0000000000000446
>>>>> (XEN) Xen code around <ffff82d04028a650>:
>>>>> (XEN) 90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90
>>>>> 90 90
>>>>> (XEN) Xen call trace:
>>>>> (XEN) [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10
>>>>> (XEN) [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40
>>>>> (XEN) [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840
>>>>> (XEN) [<ffff82d04028a35c>] S
>>>>> amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
>>>>> (XEN) [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90
>>>>> (XEN) [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
>>>>> (XEN) [<ffff82d040201915>] S common_interrupt+0x135/0x150
>>>>> (XEN) [<ffff82d0402883ca>] S
>>>>> cpu_idle.c#acpi_idle_do_entry+0x8a/0x140
>>>>> (XEN) [<ffff82d0402887cf>] S
>>>>> cpu_idle.c#acpi_processor_idle+0x1ef/0x650
>>>>> (XEN) [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130
>>>>> (XEN) Panic on CPU 2
>>>>>
>>>>> The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is
>>>>> the "unreachable" default branch of mc_ec2type() -- the extended error
>>>>> code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.
>>>> I think it's safe to just drop the BUG().
>>>>
>>>> That code is literally worse than useless, because if nothing else it's
>>>> just hiding the diagnostics.
>>>>
>>>> I guess it means AMD have added new statuses.
>>>>
>>>>> Across the eight occurrences, r12 appears to be the bank index from the
>>>>> loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS)
>>>>> and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.
>>>>>
>>>>> Not a hardware defect
>>>>> ---------------------
>>>>> - Started within days of upgrading two hosts from Debian bookworm
>>>>> (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17
>>>>> with no
>>>>> incident.
>>>>> - Three different banks, so not a single failing block.
>>>>> - Updating to the current Debian package (4.20.5-pre, built 2026-10-16)
>>>>> changed nothing.
>>>>>
>>>>> Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks
>>>>> updating").
>>>>>
>>>>> Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
>>>>> Filter bogus hardware errors on Zen3 clients"), where bogus MCA values
>>>>> on Zen 3 are filtered rather than treated as errors.
>>>>> https://ratatoskr.run/stable/2026/04/7143905
>>>> From above, you're on a Vermeer, not a Cezanne, so I don't think that
>>>> Linux patch is relevant.
>>>>
>>>> However, your observation about "fix mca_allbanks updating" quite
>>>> possibly is. That was backported into Xen 4.20.1 as 247b708b1400a, and
>>>> Debian has 4.20.3
>>>>
>>>> Are you able to experiment with that patch reverted?
>>>>
>>>> Alternatively, are you able to experiment with the BUG() removed?
>>> I went with dropping the BUG() rather than the revert.
>>> Reverting 247b708b1400a should stop any bank from being read at
>>> all, which is effectively what mce=false already does here, so I don't
>>> expect it to tell us anything new. I'll run it too if you'd like.
>>>
>>> I'm not familiar with the Xen codebase, but I've built a package with
>>> mc_ec2type() changed to:
>>>
>>> if ( errorcode & MC_EC_TLB_TYPE )
>>> return MC_EC_TLB_TYPE;
>>>
>>> printk(XENLOG_WARNING "MCE: unknown AMD error code %#x\n",
>>> errorcode);
>>>
>>> return 0;
>>>
>>>
>>> Crashes have been 12 hours to 2 weeks apart, so it may take a while
>>> before I can report back.
>> After ~31 hours of uptime:
>>
>> (XEN) [2026-10-25 04:06:27.260] MCE: unknown AMD error code 0
>>
>> No panic, no wedged CPU, no RCU stalls, no hung guests. Host and all
>> VMs running normally.
>>
>>
>> I also dumped /dev/mcelog in dom0 and got the following records:
>>
>> 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>> 00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>> 00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>> 00000030: 0000 0000 0000 0000 0200 0000 0000 0000 ................
>> 00000040: 0010 0201 0200 0000 0000 0000 0200 0000 ................
>> 00000050: 1c01 0000 0000 0000 0000 0000 0000 0000 ................
>> 00000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>> 00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>> 00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>> 00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>> 000000a0: 0000 0000 0000 0000 0200 0000 0000 0000 ................
>> 000000b0: 0017 0201 0200 0000 0000 0000 0200 0000 ................
>> 000000c0: 1c01 0000 0000 0000 0000 0000 0000 0000 ................
>> 000000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>>
>>
>>>
>>>>> Diagnostics / workaround
>>>>> ------------------------
>>>>> mce_verbosity=verbose yields nothing useful.
>>>>> max_cstate=1 made no difference. mce=false is the only thing that stops
>>>>> it, and it is what I'm running now.
>>>> The fact that mce=false isn't causing machine resets means it's clearly
>>>> something the hardware thinks isn't fatal.
>>>>
>>>> But, I am concerned by the fact it's always on CPU 2, and multiple
>>>> banks. Is this true for both systems?
>>> To clarify one thing from my report: it is actually a single host whose
>>> complete hardware (board, CPU, RAM) was replaced on
>>> 2026-10-07, keeping only the disks and thus the installation. The
>>> crashes happened both before and after that swap. 10 more hosts
>>> with same hardware still run on bookworm (xen 4.17) for years now
>>> without any incidents.
>>>
>>> So yes, always CPU 2. The bank index varies: on the current hardware
>>> I have seen banks 23, 17 and 10 across four crashes, and banks 23 and
>>> 17 on the previous hardware.
>> Regarding the output of /dev/mcelog: if my decoding is right, three
>> things stand out. MCi_STATUS is zero, i.e. these banks are reporting
>> nothing at all -- no error, no garbage value.
> Well, things can't quite be that way: If MCi_STATUS was zero, the first
> check in mc_amd_recoverable_scan() would prevent mc_ec2type() from
> being reached in the first place. Its caller also checks MCi_STATUS_VAL
> to be set.
>
> What I find puzzling is this:
>
> errorcode = status & (MCi_STATUS_MCA | MCi_STATUS_MSEC);
> ectype = mc_ec2type(errorcode);
>
> With errorcode (and mc_ec2type()'s parameter type) being uint16_t, what
> use is it to include MCi_STATUS_MSEC in the mask? (Yet that's likely
> unrelated to the issue at hand.) Considering how mc_ec2type() works, I
> see no reason why status couldn't be passed to it directly (with the
> parameter type widened to uint32_t). Then the BUG() there could perhaps
> become BUG_ON(errorcode).
You're right, my conclusion was wrong.
One more data point, from an hour ago:
(XEN) [2026-10-26 13:13:42.050] MCE: unknown AMD error code 0x1000
So it isn't always 0. The uint16_t truncation might explain the first
(MSEC bits dropped), but 0x1000 is a non-zero code that still matches
none of MC_EC_BUS_TYPE / MC_EC_MEM_TYPE / MC_EC_TLB_TYPE.
/dev/mcelog right afterwards showed three records, banks 17, 14 and 16,
all CPU 2 (status zero, as above). Banks seen so far across panics and
log lines: 10, 14, 16, 17, 23.
With the BUG() dropped, the host has now logged this twice with no
panic, no wedged CPU and no hung guests. Everything is running
normally.
Regarding passing status as uint32_t and BUG_ON(errorcode): with
today's 0x1000, bits 0-31 are non-zero but none of the type bits is
set, so that BUG_ON would still trigger. So unless 0x1000 is a known
type that mc_ec2type() should handle, I think the poller shouldn't BUG
on an unclassifiable code at all.
>> Both records are again
>> from CPU 2, and bank 23 is one of the banks that appeared in the
>> panics, bank 16 is new.
> Isn't it possible that the lack of 532988dccc19 ("x86/mcheck: fix
> mca_allbanks updating") simply hid the issue previously?
That is exactly what I suspect. This host, and 10 more with identical
hardware, ran for years on Xen 4.17 without a single incident. The
first panic came within days of upgrading this one to 4.20; the others
are still on 4.17 and still fine. If no bank was being collected
before, nothing could have reached mc_ec2type() -- so the condition
may well have been there the whole time, just never looked at.
Would it help if I logged the full MCi_STATUS of the first read, i.e.
in mcheck_mca_logout() right after the MCi_STATUS_VAL check, together
with CPU and bank number?
Peter
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot
2026-10-26 14:48 ` Peter Deggendorfer
@ 2026-10-26 15:24 ` Jan Beulich
0 siblings, 0 replies; 8+ messages in thread
From: Jan Beulich @ 2026-10-26 15:24 UTC (permalink / raw)
To: Peter Deggendorfer
Cc: Roger Pau Monné, Teddy Astie, Jason Andryuk, Soham Dandapat,
Andrew Cooper, xen-devel
On 26.10.2026 15:48, Peter Deggendorfer wrote:
> Am 26.10.2026 um 11:53 schrieb Jan Beulich:
>> On 25.10.2026 13:32, Peter Deggendorfer wrote:
>>> Regarding the output of /dev/mcelog: if my decoding is right, three
>>> things stand out. MCi_STATUS is zero, i.e. these banks are reporting
>>> nothing at all -- no error, no garbage value.
>> Well, things can't quite be that way: If MCi_STATUS was zero, the first
>> check in mc_amd_recoverable_scan() would prevent mc_ec2type() from
>> being reached in the first place. Its caller also checks MCi_STATUS_VAL
>> to be set.
>>
>> What I find puzzling is this:
>>
>> errorcode = status & (MCi_STATUS_MCA | MCi_STATUS_MSEC);
>> ectype = mc_ec2type(errorcode);
>>
>> With errorcode (and mc_ec2type()'s parameter type) being uint16_t, what
>> use is it to include MCi_STATUS_MSEC in the mask? (Yet that's likely
>> unrelated to the issue at hand.) Considering how mc_ec2type() works, I
>> see no reason why status couldn't be passed to it directly (with the
>> parameter type widened to uint32_t). Then the BUG() there could perhaps
>> become BUG_ON(errorcode).
>
> You're right, my conclusion was wrong.
>
> One more data point, from an hour ago:
>
> (XEN) [2026-10-26 13:13:42.050] MCE: unknown AMD error code 0x1000
>
> So it isn't always 0. The uint16_t truncation might explain the first
> (MSEC bits dropped), but 0x1000 is a non-zero code that still matches
> none of MC_EC_BUS_TYPE / MC_EC_MEM_TYPE / MC_EC_TLB_TYPE.
>
> /dev/mcelog right afterwards showed three records, banks 17, 14 and 16,
> all CPU 2 (status zero, as above). Banks seen so far across panics and
> log lines: 10, 14, 16, 17, 23.
>
> With the BUG() dropped, the host has now logged this twice with no
> panic, no wedged CPU and no hung guests. Everything is running
> normally.
>
> Regarding passing status as uint32_t and BUG_ON(errorcode): with
> today's 0x1000, bits 0-31 are non-zero but none of the type bits is
> set, so that BUG_ON would still trigger. So unless 0x1000 is a known
> type that mc_ec2type() should handle, I think the poller shouldn't BUG
> on an unclassifiable code at all.
I disagree. That code, aiui, is to decide whether something the hardware
reported as uncorrectable can still be recovered from. Seeing an error
code of unknown type therefore has to be treated as fatal. Whether that
is by keeping that BUG() or by some other means is secondary; continuing
to run Xen is putting all of the system's data at risk, even if in your
particular instance(s) you didn't observe further anomalies.
>>> Both records are again
>>> from CPU 2, and bank 23 is one of the banks that appeared in the
>>> panics, bank 16 is new.
>> Isn't it possible that the lack of 532988dccc19 ("x86/mcheck: fix
>> mca_allbanks updating") simply hid the issue previously?
>
> That is exactly what I suspect. This host, and 10 more with identical
> hardware, ran for years on Xen 4.17 without a single incident. The
> first panic came within days of upgrading this one to 4.20; the others
> are still on 4.17 and still fine. If no bank was being collected
> before, nothing could have reached mc_ec2type() -- so the condition
> may well have been there the whole time, just never looked at.
>
> Would it help if I logged the full MCi_STATUS of the first read, i.e.
> in mcheck_mca_logout() right after the MCi_STATUS_VAL check, together
> with CPU and bank number?
Yes please.
Jan
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot
2026-10-23 7:32 [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot Peter Deggendorfer
2026-10-23 13:14 ` Andrew Cooper
@ 2026-10-23 23:51 ` Teddy Astie
1 sibling, 0 replies; 8+ messages in thread
From: Teddy Astie @ 2026-10-23 23:51 UTC (permalink / raw)
To: Peter Deggendorfer, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 4711 bytes --]
Le 23/10/2026 à 12:20, Peter Deggendorfer a écrit :
> Hey Xen Team,
>
> I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE
> polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since
> 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.
>
> Environment
> -----------
> Xen: 4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still
> present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1)
> dom0: Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs
> CPU: AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X,
> microcode 0x0a201210
> Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems
>
> The panic
> ---------
> (XEN) [2026-10-19 08:58:18.727] Xen BUG at arch/x86/cpu/mcheck/
> mce_amd.c:100
> (XEN) CPU: 2
> (XEN) RIP: e008:[<ffff82d04028a650>] mce_amd.c#mc_ec2type.part.0+0/0x10
> (XEN) rbx: 0000000000000445 r12: 0000000000000011 r15: 0000000000000446
> (XEN) Xen code around <ffff82d04028a650>:
> (XEN) 90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90
> (XEN) Xen call trace:
> (XEN) [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10
> (XEN) [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40
> (XEN) [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840
> (XEN) [<ffff82d04028a35c>] S amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
> (XEN) [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90
> (XEN) [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
> (XEN) [<ffff82d040201915>] S common_interrupt+0x135/0x150
> (XEN) [<ffff82d0402883ca>] S cpu_idle.c#acpi_idle_do_entry+0x8a/0x140
> (XEN) [<ffff82d0402887cf>] S cpu_idle.c#acpi_processor_idle+0x1ef/0x650
> (XEN) [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130
> (XEN) Panic on CPU 2
>
> The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is
> the "unreachable" default branch of mc_ec2type() -- the extended error
> code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.
>
> Across the eight occurrences, r12 appears to be the bank index from the
> loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS)
> and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.
>
> Not a hardware defect
> ---------------------
> - Started within days of upgrading two hosts from Debian bookworm
> (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17 with no
> incident.
> - Three different banks, so not a single failing block.
> - Updating to the current Debian package (4.20.5-pre, built 2026-10-16)
> changed nothing.
>
> Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks
> updating").
>
> Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
> Filter bogus hardware errors on Zen3 clients"), where bogus MCA values
> on Zen 3 are filtered rather than treated as errors.
> https://ratatoskr.run/stable/2026/04/7143905
>
> Diagnostics / workaround
> ------------------------
> mce_verbosity=verbose yields nothing useful.
> max_cstate=1 made no difference. mce=false is the only thing that stops
> it, and it is what I'm running now.
>
> One more thing: the host does not actually reboot after the panic.
> "Reboot in five seconds" is printed, but the ring buffer continues with
> normal activity minutes later and the remaining 31 CPUs keep running.
> CPU 2 stays wedged, and "xl debug-keys t" then hangs. Afterwards dom0
> and guest vCPUs start losing timer interrupts permanently (RCU stalls
> with a frozen timer-softirq count while dynticks keeps advancing), so
> VMs hang silently hours or days later.
>
>
> I've attached the full xl dmesg log of the last crash, and I'm happy
> to provide more if needed. I can also test patches.
>
> Thanks for your work, and thanks in advance for taking a look.
>
> Peter
Could be related. I just observed a mysterious mce report on my Linux
system dmesg (Zen 3 CPU as well, Ryzen 7 5700X3D)
> [39230.481809] [Hardware Error]: System Fatal error.
> [39230.481813] [Hardware Error]: CPU:1 (19:21:2) MC12_STATUS[Over|UE|MiscV|AddrV|PCC|SyndV|UECC|Deferred|Poison|Scrub]: 0xffffffffb8616890
> [39230.481823] [Hardware Error]: Error Addr: 0x0000000000000000
> [39230.481825] [Hardware Error]: IPID: 0x0000000000000000, Syndrome: 0x0000000000000000
> [39230.481830] [Hardware Error]: Bank 12 is reserved.
> [39230.481832] [Hardware Error]: cache level: RESV, tx: INSN
It seems to be non-sense, and the system is still up as if nothing
happened. No #MC got emitted.
Teddy
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 665 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2026-10-26 15:25 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-23 7:32 [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot Peter Deggendorfer
2026-10-23 13:14 ` Andrew Cooper
2026-10-23 21:07 ` Peter Deggendorfer
2026-10-25 12:32 ` Peter Deggendorfer
2026-10-26 10:53 ` Jan Beulich
2026-10-26 14:48 ` Peter Deggendorfer
2026-10-26 15:24 ` Jan Beulich
2026-10-23 23:51 ` Teddy Astie
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.