Lunar Lake: silent fatal platform resets caused by idle-exit core hang

Lunar Lake: silent fatal platform resets caused by idle-exit core hang

[Polina Vishneva]

Hello.

So, I've got frustrated enough with my system cold-resetting every few
days to try to debug the issue, with some success.

TL;DR: cold reset on the cpuidle exit path.

It's *probably* not actually a kernel bug, but I don't know any better
place to submit it (hoping that the Intel folks will take a look).

System:
- Lenovo ThinkPad X1 Carbon Gen 13 (21NS001ACD), BIOS N4BET75W (1.45).
- Intel Core Ultra 7 258V (Lunar Lake), microcode 0x126.
- Kernel 7.1.1, intel_idle in ACPI _CST mode.
- Reproduces even with intel_idle.max_cstate=2, so the enabled
  states are only POLL, C1_ACPI and C2_ACPI.

On boot after each crash, the kernel logs:

  BERT: [Hardware Error]: Skipped 1 error records

I pulled /sys/firmware/acpi/tables/data/BERT by hand and decoded it with
https://github.com/intel/crashlog.

The region contains a full PMC crashlog (MCA, Punit, UNCORE, PCORE,
PMC, PMC_RST, PMC_TRACE records). iclg summarizes the failure as:

  CORE_TIMEOUT.SINGLE_STUCK_TRANSACTION.C898FH
  MCA.BANK3.INTERNAL_TIMER_ERROR.MSCOD_E184H
  CRASHLOG_REASON.PMC.10H
  CRASHLOG_REASON.PUNIT.20004H
  RESET_CAUSE.FIRMWARE_GLOBAL_RESET.FW_GBLRST_SCRATCH16
  RESET_CAUSE.GLOBAL_RESET.PMC_FW

The important registers (one P-core has captured state):

  mca.bank3.status = 0xbe000000e1840400
      VAL|UC|EN|MISCV|ADDRV|PCC, MCACOD 0x0400 (internal timer /
      retirement watchdog, "three-strike"), MSCOD 0xE184
  mca.bank3.addr = mca.bank3.misc = arch_state.lip
                 = 0xffffffff9471bb3c        (kernel text)
  super queue: exactly one stuck transaction (cacheline 0xC898F)
  pmc_rst: gblrst_cause_0.pmc_fw=1, fw_gblrst_cause_0.scratch16=1,
      gblrst_req_0 = pmc_fw|syspwr_flr|pchpwr_flr

The LIP: given that KASLR slide is 2MB-aligned, (lip mod 2M) is an
invariant. Solving over System.map text symbols gives 14 candidates, of
which only one can be related:

  cpuidle_enter_state+0xbc

Disassembly:

  +0x73  call __x86_indirect_thunk_rax   ; target_state->enter()
   ...   sched_clock_idle_wakeup_event / local_clock_noinstr /
         sched_idle_set_state
  +0xb6  call *pv_ops...irq_enable       ; patched to STI at boot
  +0xbc  test %r14d,%r14d                ; <- LIP

So (my best guess) the core had already returned from MWAIT, executed
STI on the idle-exit path, and then stopped retiring instructions. The
retirement watchdog then fires, Punit reports CORE_TIMEOUT, and PMC
firmware pulls a global platform reset.

The question is whether it is something known or rather something
platform-specific that should be reported to Lenovo instead.

I can provide the raw BERT dump, and also I'd be happy to debug/test if
requested.

Best regards,
Polina.