[BUG] Lunar Lake: 400 MHz after s2idle; PL1 Tau affects reproduction and recovery

[BUG] Lunar Lake: 400 MHz after s2idle; PL1 Tau affects reproduction and recovery

[Sandor Nagy]

Hello,

My ThinkPad X1 Carbon Gen 13 intermittently resumes from long s2idle 
suspends with all eight CPUs stuck around 400 MHz. The issue also occurs 
with
Fedora’s standard packaged kernel.

I have collected hardware telemetry, shortened reproduction of the 
power-limiting behavior by changing the MMIO PL1 averaging window (Tau), 
and observed
promising recovery with an early post-resume Tau pulse.

Could you help investigate this or direct the report to the appropriate 
Intel power-management engineers?

System:

- ThinkPad X1 Carbon Gen 13, model 21NSCTO1WW
- Intel Core Ultra 7 258V, family 6, model 189, stepping 1
- BIOS N4BET77W 1.47, EC 1.41, microcode 0x128
- Fedora 44 Workstation
- intel_pstate active, HWP enabled, s2idle
- Capture kernel: 7.1.13-200.btf.fc44.x86_64; recorded kernel taint: 0

The capture kernel is a local rebuild of Fedora’s kernel-7.1.13-200.fc44 
source RPM, with no additional kernel source patches, using 
dwarves/pahole 1.31
instead of 1.30 to correct BTF generation for sched_ext. The .btf suffix 
identifies this rebuild. The suspend issue occurs on standard Fedora 
kernels too
and predates the rebuild.

I have experienced it across multiple kernel versions but have no 
known-good version or bisected regression. I have not yet verified 
reproduction on
current vanilla mainline.

Normal reproduction and symptoms:

The issue occurs after long s2idle suspends. Short sleeps resume 
normally; longer sleeps produce the 400 MHz episode, and the lag 
generally lasts longer
after longer suspends. This matches my experience before instrumentation 
as well as the recorded experiments. Under the sleep-power conditions 
measured in
several captures, our energy-based model predicts onset at roughly three 
hours. The exact duration threshold can vary with energy consumption during
suspend. To reproduce at the normal Tau setting, leave the laptop 
suspended for several hours, preferably overnight, then resume.

During an episode, all eight CPUs can remain around 400 MHz despite 
active work. Package power is approximately 2–5 W and temperatures are low.
Performance recovers spontaneously after several seconds to tens of 
seconds. Shorter sleeps often resume normally.

Instrumentation:

I used custom diagnostic scripts to record a baseline before suspend and 
repeated snapshots after resume. They read CPU MSRs, powercap sysfs 
attributes,
and Intel PMT telemetry.

CPU frequency estimates use APERF/MPERF deltas over awake sampling 
intervals; they are not calculated across suspend. PMT fields are 
decoded using Intel’s
published Lunar Lake XML mappings, checking the product and 
crystal-frequency fields before converting residency counters. Residency 
figures below are
differences between counter readings.

The recorder is read-only. Separate intervention code performs and logs 
setting changes.

During captured 400 MHz episodes:

- PMT PL1_LIMITED_RESIDENCY advances at approximately one second per second.
- Power-limiting residency advances on the core groups, ring, and graphics.
- PROCHOT, VRHOT, thermal-limiting, and PSYS PL1 residency show no 
corresponding increases.
- MSR package PL1 remains 37 W; MMIO package PL1 remains 17 W.
- The normal MMIO PL1 time window remains 27,983,872 us.
- HWP requests remain unchanged.

A verified MSR PL1 write from 37 W to 36 W and back did not promptly 
release the fault. Cycling power profiles has also not helped.

Shortened reproduction:

The most informative experiments changed this setting before suspend:

/sys/class/powercap/intel-rapl-mmio:0/constraint_0_time_window_us

These trials used battery power, the balanced profile, and comparable 
wake procedures:

- Approximately 900 seconds suspended, Tau 27.984 seconds: no PL1 
residency increase.
- Approximately 900 seconds suspended, Tau 0.999 seconds: 1.316 seconds 
of PL1 residency, with corresponding core/ring/graphics power limiting.
- Approximately 60 seconds suspended, Tau 0.999 seconds: no PL1 
residency increase.

Shortening Tau therefore allowed reproduction of the limiter signature 
after 15 minutes. The short-Tau event ended before the first valid frequency
interval captured a 400 MHz lock, so this reproduced the power-limiting 
behavior rather than the complete user-visible symptom.

A separate experiment set Tau to approximately 2 seconds before a 
45-minute suspend, then restored approximately 28 seconds immediately 
after resume. It
captured all eight CPUs around 400 MHz for roughly 18 seconds and 20.118 
seconds of total PL1 residency. There is no matching untreated 
2-second-Tau/45-
minute control, so this does not establish how much the restore write 
prolonged the episode.

Automatic workaround experiment:

The workaround is launched asynchronously from a systemd post-resume 
hook. A small native helper saves the original MMIO Tau, writes 125,000 
us, and
verifies the readback, which is 124,928 us on this machine. It acts 
before starting the more expensive Python sampler, without waiting for fault
detection. The power-limit values themselves are unchanged.

The monitor restores the original Tau after two consecutive intervals 
with little PL1 limiting, or after an eight-second maximum hold. A 
recovery journal
and service cleanup path also attempt restoration after failure. 
Busy-clock recovery is assessed separately from PL1 residency.

In the latest natural suspend of 2 h 59 min:

- CPU0’s power-limit status was active before the write, and PL1 
residency had already advanced by 1.228 seconds since the pre-suspend 
baseline.
- The write landed approximately 0.254 seconds after the kernel’s 
journalled suspend-exit message.
- Total PL1 residency across the capture was 1.775 seconds.
- The first valid clock interval after the write measured 1.89–3.05 GHz; 
its sample completed 1.16 seconds after the write.
- Restoration of the original Tau was verified, with no additional PL1 
residency through approximately 12.8 seconds of subsequent recording.
- I noticed no resume lag.

Working hypothesis:

The observations suggest that package energy accumulated during sleep is 
mishandled in the PL1 averaging state or its timebase on resume. Historical
clean/fault captures are consistent with an energy threshold related to 
PL1 × Tau, and changing Tau changes the reproduction conditions.

However, the energy measurements span before-suspend and after-resume 
snapshots and include some awake activity. They do not expose the 
controller’s
internal state or establish whether the defect belongs to firmware, 
hardware, or Linux resume initialization.

A matching symptom has been reported on a ThinkPad X9-15 with the same 
processor:


https://github.com/intel/linux-intel-lts/issues/84

I can provide timestamped raw captures, kernel logs, diagnostic scripts, 
and intervention code with the corresponding revisions and command 
sequences.

LLM assistants helped develop the diagnostic and workaround code, 
analyze the captures, and draft this report. I ran the experiments on 
the affected
machine. The underlying code and raw data are available so the decoding 
and conclusions can be checked independently.

Is this a known Lunar Lake issue? What additional instrumentation would 
help distinguish incorrect energy-delta accounting from stale averaging 
state
across s2idle? Is there a supported way to resynchronize that state on 
resume?

Thanks,
Sandor