* [Bug 221909] amd-pstate: Cezanne (Ryzen 7 5800U) data fabric sync flood on DC gated by CPPC max_perf
2026-08-22 11:35 [Bug 221909] New: amd-pstate: Cezanne (Ryzen 7 5800U) data fabric sync flood on DC gated by CPPC max_perf bugzilla-daemon
` (2 preceding siblings ...)
2026-08-23 16:45 ` bugzilla-daemon
@ 2026-08-27 1:29 ` bugzilla-daemon
2026-08-28 1:39 ` bugzilla-daemon
4 siblings, 0 replies; 6+ messages in thread
From: bugzilla-daemon @ 2026-08-27 1:29 UTC (permalink / raw)
To: linux-pm
https://bugzilla.kernel.org/show_bug.cgi?id=221909
--- Comment #4 from David Smith (smithd98@gmail.com) ---
UPDATE 2026-08-26. SIGNIFICANT CORRECTIONS TO COMMENT #0, PLUS FOUR NEW
NEGATIVE RESULTS AND ONE NEW POSITIVE FINDING.
Several load-bearing claims in comment #0 are wrong and I am withdrawing them.
Anyone using that comment as a starting point should read this first. In
particular, one of the workarounds I published there has since been re-tested
properly and it does NOT work.
================================================================
1. WITHDRAWN: ROWS 3 AND 5 OF THE GATING TABLE, AND THE HEADLINE
================================================================
Re-deriving the ledger from raw logs rather than from my own prose, the
"surviving" runs at stock max_perf shared a disqualifying defect: the ~6.7 GB
model was not actually resident, so they were I/O-bound on page-ins instead of
running sustained all-core AVX2. They were not doing the same work as the runs
that died.
Row 5 (performance/performance, 4508086, "8 steps 0 deaths", 4374 MHz):
entered the window with 10 MiB MemAvailable, holding 5004 MiB of a
~6.7 GiB model, at 17.3-24.2 tok/s. A 3.8 GHz-capped run manages
30.7-31.2 tok/s. A run pinned to max_perf 166 that is 30% SLOWER than a
capped run was not stressing the part. Its quoted fmax 4374 MHz was a
transient peak in a starved run; mean fmax was 2953 MHz.
Row 3 (balance_performance/powersave, 4100000, "8 steps 0 deaths"):
same defect, worse - RSS 38 MiB.
**Therefore the headline claim of comment #0 - "the lethal configuration is the
MILDEST one measured", survives 4374 MHz/1462 mV/85.8 C and dies at
3617 MHz/1175 mV/63.8 C - is WITHDRAWN.** It rested entirely on row 5. So is
"Rows 1 and 5 are the controlling pair" and the conclusion "pinning perf HIGH
is safe".
Row 5's configuration has since been re-run correctly (RSS 6692 MiB,
MemAvailable 12.6 GiB, 27.9 tok/s warm). **It died.**
=========================================================
2. WITHDRAWN AND ACTIVELY WRONG: THE epp=performance ADVICE
=========================================================
Comment #0's WORKAROUND section says:
"epp=performance on DC also survived (8 steps) and preserves full boost"
**Do not use this. It has been re-tested validly and the machine dies.** That
"survival" was row 5. Deaths have now been observed at balance_power,
balance_performance and performance alike, and also with min_perf pinned equal
to max_perf. **EPP is not the gate.** The only workaround I still stand behind
is the DC frequency cap.
=====================================================
3. WITHDRAWN: THE "RANGE, NOT CEILING" INTERPRETATION
=====================================================
Comment #0 claimed the cap works by narrowing the CPPC range rather than
lowering the ceiling, "even though the workload never exceeds ~2.9 GHz
uncapped". **That was an instrument artifact.** The `fmax` field I derived it
from is unreliable on this platform.
Re-measured with per-core CPPC feedback counters (delivered/reference
counters, 43k+ core-samples per arm):
STOCK (4508086) CAPPED (3800000)
p99 3981 MHz 3752 MHz
p99.9 4171 MHz 3963 MHz
max 4473 MHz 4493 MHz
core-samples > 3.8 GHz 1271 (2.94%) 93 (0.21%)
**The workload does reach 4.2-4.5 GHz, and the cap cuts the above-3.8 GHz
population by 13.8x while leaving the median untouched (2336 -> 2331 MHz). The
cap acts entirely on the tail.** The excursions concentrate in LOW-THREAD
phases (cold load, page faults, memcpy, NVMe completion), not in the all-core
vector phase, and they track the CPPC preferred-core ranking exactly:
cpu 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
highest_perf 181 181 196 196 186 186 196 196 176 176 166 166 191 191 171 171
count >3.8G 46 50 162 119 81 61 146 122 63 66 46 48 88 90 46 37
Note also that the cap is **statistical, not absolute**: 0.21% of core-samples
still exceed it, peaking at 4493 MHz.
=====================================================
4. WITHDRAWN: "RULED OUT - THE BATTERY PACK"
=====================================================
Comment #0 lists the pack under RULED OUT, on the strength of "the same pack
survives 4374 MHz ... and dies at 3617 MHz". That is row 5 again. **With row 5
withdrawn, the exclusion has no valid arm behind it and is withdrawn.**
The pack is measurably bad. Measured today under load, at 82% charge:
voltage_min_design 11.400 V
voltage_now 10.981 V <- 419 mV BELOW its own design floor
current_now 2.234 A at only 2.2 A average
charge_full/design 76.1% 263 cycles, 5 yr 4 mo
I am not able to test a replacement pack, so I cannot close this. Treat the DC
power path as an open hypothesis, not an excluded one.
=====================================================
5. CORRECTED: "No MCE"
=====================================================
Comment #0 says "No MCE". That was wrong - nothing in my harness guaranteed the
log reached disk before the reset. With a per-line-fsync'd capture, one crash
produced, immediately before the flood:
CPU:10 MC0_STATUS[Over|UE|MiscV|AddrV|-|-|-|-|Poison|-]: 0xfc00080001010135
Error Addr: 0x0000000125006740 IPID: 0x001000b000000000
cache level: L1, tx: DATA, mem-tx: DRD
... and the identical bank on CPU:4, Error Addr 0x000000015e8c6740
IPID HWID 0xB0 / McaType 0x10 = SMCA_LS_V2, XEC 1 = "An ECC error or L2 poison
was detected on a data cache read by a load". Poison set, OVER set, PCC=0, two
cores, two addresses, same instant.
The LS unit is the CONSUMER of an already-poisoned line, not the source. OVER
on both banks means a burst. Two scattered addresses on two cores argues
against a single marginal DRAM cell and for transient corruption of in-flight
data. Only one MCE has been captured in ~21 crashes; two later attempts to
reproduce it failed.
================================================================
6. NEW - FOUR NEGATIVE RESULTS THE COHORT IN COMMENTS #2/#3 DOES NOT HAVE
================================================================
All run on battery, at stock max_perf, with guards on RSS >= 6000 MiB,
MemAvailable >= 1500 MiB and warm throughput >= 25 tok/s.
(a) **BIOS 01.25.00 (latest HP ships) DIES.** AMD asked me to test it. Flashed
and verified three ways; died 23 s into the ladder. This rules out AGESA
CezannePI FP6 1.0.1.1c and PSP 0.11.0.85 -> 0.11.0.88. It does NOT
exonerate the SMU: **SMU firmware is unchanged at 64.74.0 and microcode is
unchanged at 0xa500014 across 01.22 -> 01.25**, so the flash did not move
the component actually under suspicion.
(b) **CC6 disabled DIES, and this refutes the cohort's top workaround.**
With ACPI C3/CC6 disabled and verified at **0 CC6 entries/s** (normal arms
measure 3420-4403/s), the machine sync-flooded in **10 seconds**. So
`processor.max_cstate=2` - the most-recommended parameter in the gist
linked in comment #2 - **does not protect this machine.** C6 is also not
the gate by rate: an AC C6 matrix measured essentially identical entry
rates across lethal and safe configurations alike.
(c) **Kernel 6.12.75 DIES, which refutes the cohort's last-known-good.**
Comment #2 and the linked gist name 6.12.x as last-known-good against
6.18.x/7.1.x failing - exactly this machine's confirmed pair. I installed
6.12.75-1-lts and ran the lethal ladder. **It died in 20 seconds**, at the
same step as every other death.
This test was gated in advance so the result could not be explained away:
6.12.75 reaches a **harsher** operating point than 7.1.8 (3.67% vs 2.94% of
core-samples above 3.8 GHz), so "6.12 just boosts less" is unavailable.
The two kernels also derive different limits from the same hardware
(cpuinfo_min_freq 1113443 -> 400000), but the per-core highest_perf ladder
is identical.
(d) Deaths continue to occur across EPP and governor settings and with
min_perf pinned. Nothing but the frequency cap has ever moved the rate.
**Caveat I want to state plainly:** my failure fires in 10-50 s on demand and
is
gated by CPPC max_perf; the cohort in comments #2/#3 reports 1-14 days,
stochastic, with idle sufficient. Those differ by ~4 orders of magnitude in
rate and my gate has no counterpart in their reports. 0x08000800 is an
ENDPOINT, not a mechanism - it is what the DF does whenever it consumes a
poisoned line, whatever poisoned it - and the Framework case was already
root-caused to WD SN770 NVMe firmware. **I no longer assume we have the same
bug.** These negative results are offered as data, not as a merge.
=====================================================
7. NEW POSITIVE FINDING - IT CORRUPTS DATA BEFORE IT RESETS
=====================================================
This is the most useful thing I have found and it was not in the original
report.
The last kernel line before one reset was a userspace fault in the compute
workload, which the kernel resolved to a specific instruction:
traps: llama-server[34143] general protection fault ip:74fd9014dbd7
error:0 in libggml-cpu-haswell.so[e0bd7,74fd9007f000+d8000]
Disassembled, file offset 0xe0bd7 is inside `ggml_gemm_q4_K_8x8_q8_K`, the
hottest loop of the workload:
e0bc4: mov 0x1b8(%rsp),%rax ; base pointer from stack
e0bcc: add (%rsi),%rax ; += an 8-byte word from memory
e0bd3: add $0x8,%rsi
e0bd7: vmovdqu 0x10(%rax,%r12,1),%ymm0 ; <-- FAULTS HERE
**The same instruction faulted at an earlier crash** (identical ip low-20 bits)
- but as a SIGSEGV (error 4, page-not-present) on a canonical-but-unmapped
address, rather than a #GP on a non-canonical one.
`vmovdqu` is the UNALIGNED load; it does not raise #GP on misalignment. So both
faults are bad ADDRESSES, and %rax is built from two memory loads two
instructions earlier. **Same instruction, two different fault types, two
different bad addresses = a corrupted pointer operand, not a software bug.**
That is exactly what the MCE in section 5 describes: poison consumed on a data
cache read by a load.
Control: the workload has faulted twice, both times within seconds of a sync
flood, in ~14 valid stock-max_perf arms - and zero times across 43 capped
ladder steps and two capped soaks.
**Implication: this machine computes wrong answers before it resets.** Both
observed cases corrupted a pointer and were therefore loud. Corruption of a
value instead of a pointer would be silent.
Timing caveat, stated because it cuts against a simple di/dt story: in most
recent deaths the compute load collapses to idle **0.3-2.8 s BEFORE** the
reset, and in one case the machine sat at ~9 W for nearly three seconds after
the workload had already died. It is not universal - one arm died under full
load - but time-of-death is evidently not time-of-injection.
=====================================================
8. THE MITIGATION, AND WHAT IT IS NOW VALIDATED AGAINST
=====================================================
Unchanged in form - cap scaling_max_freq to 3800000 on DC only, driven from a
systemd unit at boot, a udev rule on SUBSYSTEM=="power_supply"
ATTR{type}=="Mains", and a system-sleep hook for resume.
At stock max_perf on DC: 13 deaths in 14 valid arms
At smax <= 3800000 on DC: 43 ladder steps + a 35-min mixed soak
+ a 20-min integrity soak, 0 deaths
New: I built a self-verifying canary (bit-exact AVX2 checksum over a 32 MiB
buffer, single-bit-flip selftest 60/60) and ran four instances pinned to the
CPPC preferred cores alongside the real workload, on battery under the cap, for
20 minutes: **~557,000 verified passes, ~17 TiB checked bit-exactly, zero
compute errors, zero memory corruption, zero faults, zero MCEs, no reset.**
So the cap now has support on two criteria - survival AND data integrity -
where before it had only survival.
**Limitation I want on the record:** those canaries held their cores at
SUSTAINED 100%, and a continuously busy core does not perform the idle->boost
transition that section 3 identifies as where the excursions live. So this
result licenses "no corruption under sustained load under the cap" and NOT "no
corruption during boost excursions under the cap". A burst-mode canary is the
correct instrument and I have not yet run it.
=====================================================
9. CORRECTED LEDGER AND STANDING
=====================================================
Reproduced on BIOS 01.21.00, 01.22.00 and 01.25.00, and on kernels 6.12.75-lts,
6.18.44-lts and 7.1.8. The machine-generated boot ledger begins 2026-08-21
(after the 01.22 flash) and records **20 data-fabric sync floods plus one hard
hang**; earlier crashes on 01.21 are described in comment #0 but predate the
ledger, so the true total is somewhat higher than the 21 I have been quoting.
The controlled figures are the arm-based ones above, and those I stand behind.
Where it stands: the failure is **mitigated but not root-caused**. Every cheap
software and firmware axis is now closed - BIOS (three versions), C6 (both
directions), EPP, governor, min_perf pinning, and three kernels. The remaining
candidates are the SMU's DC boost/voltage policy and the DC power path itself,
and I cannot separate them without a replacement pack, which I am not able to
test.
QUESTIONS, UPDATED
==================
1. Given SMU 64.74.0 and microcode 0xa500014 are constant across all three
BIOS versions tested, is there a known Cezanne (fam 25, model 80) erratum
covering fabric sync floods arising from autonomous CPPC boost transitions
on DC?
2. Which upstream unit sets poison on this part, so the MCE in section 5 can
be traced back from the consumer to the originator?
3. Is there any way to obtain a fabric-side error record with no APEI tables
(no BERT/HEST/ERST here, and no EDAC instance binds)?
4. Would a DMI-matched amd_pstate quirk clamping max_perf on DC be acceptable
upstream, or is this strictly an AGESA/PMFW fix to route through the OEM?
Happy to write and test it given a preferred shape.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 6+ messages in thread* [Bug 221909] amd-pstate: Cezanne (Ryzen 7 5800U) data fabric sync flood on DC gated by CPPC max_perf
2026-08-22 11:35 [Bug 221909] New: amd-pstate: Cezanne (Ryzen 7 5800U) data fabric sync flood on DC gated by CPPC max_perf bugzilla-daemon
` (3 preceding siblings ...)
2026-08-27 1:29 ` bugzilla-daemon
@ 2026-08-28 1:39 ` bugzilla-daemon
4 siblings, 0 replies; 6+ messages in thread
From: bugzilla-daemon @ 2026-08-28 1:39 UTC (permalink / raw)
To: linux-pm
https://bugzilla.kernel.org/show_bug.cgi?id=221909
--- Comment #5 from David Smith (smithd98@gmail.com) ---
UPDATE 2026-08-27. ONE CLEAN NEW RESULT, AND TWO MECHANISMS EXCLUDED -
INCLUDING THE ONE I WAS ABOUT TO PROPOSE.
This follows comment #4 and does not repeat it. Comment #4's withdrawals all
still stand. What is new here is a single-variable A/B pair that isolates the
frequency cap, and which also falsifies the voltage mechanism I had been
building toward. I am reporting the exclusions because a falsified mechanism
seems more useful than the confounded correlation I would otherwise have
published.
=====================================================
1. THE ONE CLEAN RESULT: A SINGLE-VARIABLE PAIR
=====================================================
Everything in comment #0, and most of comment #4, is confounded in some way.
This is not. It is the only experiment in this investigation in which exactly
one variable moves.
Both arms: same boot, same machine, same battery, same workload (llama.cpp
128-token prefill, ~6.7 GB model, verified resident, CPU only), 27 minutes
apart. Both with amdgpu power_dpm_force_performance_level=high, which pins
FCLK at its top DPM state. CPU VDD plane sampled at 492 Hz.
LETHAL SURVIVOR
scaling_max_freq 4508086 3800000
FCLK 1333 MHz, 100% 1333 MHz, 100%
FCLK transitions 0 in 23864 samples 0 in 66913 samples
VDDNB (SoC rail) 818 mV 84.5% 818 mV 94.0%
CPU VDD plane, peak 1250 mV 1281 mV
CPU VDD plane, % >=1300 mV 0.00% 0.00%
CPU VDD plane, % 1100-1300 25.6% 33.0%
exposure died at 20.9 s survived 318 s, 9/9 steps
outcome 0x08000800 clean
scaling_max_freq is the only thing that differs. Prior cap-efficacy evidence
in this bug was confounded: the compared runs differed in the cap AND in the
power source. This pair is not.
Note how harsh the survivor's configuration is. FCLK pinned at 1333 MHz is the
modifier that kills this machine FASTEST at stock max_perf - 11.6 s in one
arm, 21 s in another. Under the cap it ran 318 s clean.
=====================================================
2. EXCLUDED: CPU-PLANE VOLTAGE IS NOT THE MECHANISM
=====================================================
I had built a voltage story and was about to publish it: on DC at low thread
counts the plane sits at 1.37-1.39 V for ~99% of the time, and I believed the
cap protects by removing that operating point.
The pair above kills it, in the awkward direction:
the run that SURVIVED sat at a HIGHER peak plane voltage (1281 vs 1250 mV)
and spent MORE time in the 1100-1300 mV band (33.0% vs 25.6%), for 15x
longer exposure - and lived.
So CPU-plane voltage is not the discriminator: not as a peak, not as
residency, not as dose. Withdrawn before publication rather than after.
PROVENANCE CORRECTION, which anyone reading my hwmon numbers needs: the
"vddgfx" column in comment #0's gating table is NOT a GPU rail. On this part
it is the shared VDDCR_VDD plane and it tracks the CPU V/F point. Verified
directly: 1 thread -> cores 4574 MHz, plane 1393 mV; 16 threads -> cores
2761 MHz, plane 931 mV, i.e. the plane follows the CPU down while the iGPU
clock also falls. Any reading of that column as GPU behaviour, mine included,
was wrong.
SECOND INSTRUMENTATION CAVEAT: I sampled that plane at 4 Hz for most of this
investigation, while it updates about every 2.7 ms - roughly 1 value in 92.
Any voltage conclusion in comment #0 or comment #4 was drawn from aliased
data. The 492 Hz figures above are the first honest measurements I have of
that rail.
=====================================================
3. EXCLUDED: FABRIC DPM TRANSITIONS ARE NOT NECESSARY
=====================================================
Since the failure is a fabric sync flood, the obvious question is whether the
fabric's own DPM transitions are involved. They are not:
FCLK pinned at 1333 MHz, 100% residency, ZERO transitions across 23864
samples at 492 Hz - and the machine sync-flooded anyway, 21 s in.
I had also observed FCLK thrash (400<->1200, 13 transitions in 888 ms) in an
earlier death captured at 50 Hz, and wondered whether it was causal. In this
death the thrash is not merely downstream - it is entirely absent.
Combined with section 2, and with VDDNB identical at 818 mV on both sides of
the pair, that leaves:
EXCLUDED: fabric DPM transitions, SoC-rail level, CPU-plane voltage
REMAINING: peak core frequency itself, or core current / di-dt at the top
P-states
i.e. the fault lives in the top ~700 MHz, between 3.8 and 4.5 GHz, and is not
reachable through the plane voltage, the fabric clock, or the SoC rail.
I cannot narrow it further from here. There is no writable PPT interface on
this part - amdgpu hwmon exposes only power1_input, read-only - so I cannot
hold frequency at stock and lower the power budget independently.
The band 3.83-4.45 GHz is entirely untested. My coverage is <=3.80 GHz (0
deaths, 52 steps), 4.48 GHz (1/1 dead), and 4.51 GHz stock (dies reliably). A
cap at ~4.1 GHz bisects it. I will run that bisection if it is useful to
anyone.
=====================================================
4. INSTRUMENTATION WARNING: journald UNDER-REPORTS THE MOMENT OF DEATH
=====================================================
Relevant to anyone else triaging these resets from logs. In the death I
captured at 492 Hz:
journald's last write 09:42:52
my sampler's last write 09:43:00.907 (fsync'd every 0.25 s)
journald stops being flushed 8.0 seconds before the part actually resets. So
the last line in your journal is NOT the moment of death, and "the machine was
idle when it died, because the last log line is idle" is an unsafe inference.
=====================================================
5. CORRECTIONS TO COMMENT #4
=====================================================
(a) TIMING CLAIM WITHDRAWN. Comment #4 section 7 says the compute load
collapses to idle "0.3-2.8 s BEFORE" the reset, and treats that as cutting
against a simple di/dt story. Withdraw that. Those gaps were computed
against journald's last write, which section 4 above shows is up to 8.0 s
early, so they are not trustworthy. In the one death measured against a
reliable clock the collapse precedes the reset by under 0.4 s: last busy
sample 09:43:00.510 at 3065 MHz mean, first collapsed sample .758 at
498 MHz, reset at .907 - bounded between 0.15 and 0.40 s by the 250 ms
sampler. I am no longer claiming a long quiet interval before the flood.
(b) UPDATED TALLIES.
At stock max_perf on DC: 14 deaths in 15 valid arms
At smax <= 3800000 on DC: 52 ladder steps + a 35-min mixed soak
+ a 20-min integrity soak, 0 deaths
The 52 now includes the 9 steps from section 1, run with FCLK pinned at
1333 MHz. The ledger since 2026-08-21 now records 23 data-fabric sync
floods plus one hard hang.
(c) The userspace-fault control in comment #4 section 7 becomes: two faults in
~15 valid stock-max_perf arms, zero across 52 capped ladder steps and two
capped soaks.
=====================================================
6. WHERE IT STANDS
=====================================================
Still mitigated but not root-caused, and the mitigation is unchanged in form:
cap scaling_max_freq to 3800000 on DC only.
What is now closed: BIOS (three versions), C6 (both directions), EPP,
governor, min_perf pinning, fabric clock pinning, three kernels, and - new
here - CPU-plane voltage and SoC-rail level as mechanisms.
What remains is peak core frequency itself, or core current / di-dt at the top
P-states in the 3.8-4.5 GHz band, plus the DC power path, which I cannot close
without a replacement pack I am not able to test. I cannot separate those two
from userspace: there is no writable PPT interface here, and every voltage and
power number available to me from hwmon is an SMU self-report, while the one
independent instrument - the battery EC - refreshes about every 9 s against an
event lasting under a second.
ADDITIONAL QUESTIONS
====================
These are in addition to the four in comment #4.
5. Is there any Cezanne rail telemetry that is NOT an SMU self-report? If the
SMU is the suspect and also the only witness, I cannot distinguish "the SMU
commanded a bad V/F point" from "the rail sagged and the SMU did not see
it".
6. Is there a debug interface to set a PPT limit on this part? amdgpu exposes
only power1_input read-only here, so I cannot separate frequency from power
budget - which is exactly the separation section 3 now needs.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 6+ messages in thread