From: Peter Zijlstra <peterz@infradead.org>
To: Breno Leitao <leitao@debian.org>
Cc: Ingo Molnar <mingo@redhat.com>,
Arnaldo Carvalho de Melo <acme@kernel.org>,
Namhyung Kim <namhyung@kernel.org>,
Mark Rutland <mark.rutland@arm.com>,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Jiri Olsa <jolsa@kernel.org>, Ian Rogers <irogers@google.com>,
Adrian Hunter <adrian.hunter@intel.com>,
James Clark <james.clark@linaro.org>,
Thomas Gleixner <tglx@kernel.org>, Borislav Petkov <bp@alien8.de>,
Dave Hansen <dave.hansen@linux.intel.com>,
x86@kernel.org, "H. Peter Anvin" <hpa@zytor.com>,
Dapeng Mi <dapeng1.mi@linux.intel.com>,
linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org,
kernel-team@meta.com
Subject: Re: [PATCH] perf/x86: Restore event pointer setup in x86_pmu_start()
Date: Mon, 9 Mar 2026 17:38:47 +0100 [thread overview]
Message-ID: <20260309163847.GE2277644@noisy.programming.kicks-ass.net> (raw)
In-Reply-To: <20260309-perf-v1-1-601ffb531893@debian.org>
On Mon, Mar 09, 2026 at 07:40:56AM -0700, Breno Leitao wrote:
> A production AMD EPYC system crashed with a NULL pointer dereference
> in the PMU NMI handler:
>
> BUG: kernel NULL pointer dereference, address: 0000000000000198
> RIP: x86_perf_event_update+0xc/0xa0
> Call Trace:
> <NMI>
> amd_pmu_v2_handle_irq+0x1a6/0x390
> perf_event_nmi_handler+0x24/0x40
>
> The faulting instruction is `cmpq $0x0, 0x198(%rdi)` with RDI=0,
> corresponding to the `if (unlikely(!hwc->event_base))` check in
> x86_perf_event_update() where hwc = &event->hw and event is NULL.
>
> drgn inspection of the vmcore on CPU 106 showed a mismatch between
> cpuc->active_mask and cpuc->events[]:
>
> active_mask: 0x1e (bits 1, 2, 3, 4)
> events[1]: 0xff1100136cbd4f38 (valid)
> events[2]: 0x0 (NULL, but active_mask bit 2 set)
> events[3]: 0xff1100076fd2cf38 (valid)
> events[4]: 0xff1100079e990a90 (valid)
>
> The event that should occupy events[2] was found in event_list[2]
> with hw.idx=2 and hw.state=0x0, confirming x86_pmu_start() had run
> (which clears hw.state and sets active_mask) but events[2] was
> never populated.
>
> Another event (event_list[0]) had hw.state=0x7 (STOPPED|UPTODATE|ARCH),
> showing it was stopped when the PMU rescheduled events, confirming the
> throttle-then-reschedule sequence occurred.
>
> The root cause is commit 7e772a93eb61 ("perf/x86: Fix NULL event access
> and potential PEBS record loss") which moved the cpuc->events[idx]
> assignment out of x86_pmu_start() and into x86_pmu_enable(). This
> broke any path that calls pmu->start() without going through
> x86_pmu_enable() -- specifically the unthrottle path:
>
> perf_adjust_freq_unthr_events()
> -> perf_event_unthrottle_group()
> -> perf_event_unthrottle()
> -> event->pmu->start(event, 0)
> -> x86_pmu_start() // sets active_mask but not events[]
>
> The race sequence is:
>
> 1. A group of perf events overflows, triggering group throttle via
> perf_event_throttle_group(). All events are stopped: active_mask
> bits cleared, events[] preserved (x86_pmu_stop no longer clears
> events[] after commit 7e772a93eb61).
>
> 2. While still throttled (PERF_HES_STOPPED), x86_pmu_enable() runs
> due to other scheduling activity. Stopped events that need to
> move counters get PERF_HES_ARCH set and events[old_idx] cleared.
> In step 2 of x86_pmu_enable(), PERF_HES_ARCH causes these events
> to be skipped -- events[new_idx] is never set.
So why not just move this then? Having less sites that set that value is
more better, no?
---
diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index 03ce1bc7ef2e..54b4c315d927 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -1372,6 +1372,8 @@ static void x86_pmu_enable(struct pmu *pmu)
else if (i < n_running)
continue;
+ cpuc->events[hwc->idx] = event;
+
if (hwc->state & PERF_HES_ARCH)
continue;
@@ -1379,7 +1381,6 @@ static void x86_pmu_enable(struct pmu *pmu)
* if cpuc->enabled = 0, then no wrmsr as
* per x86_pmu_enable_event()
*/
- cpuc->events[hwc->idx] = event;
x86_pmu_start(event, PERF_EF_RELOAD);
}
cpuc->n_added = 0;
next prev parent reply other threads:[~2026-03-09 16:38 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-03-09 14:40 [PATCH] perf/x86: Restore event pointer setup in x86_pmu_start() Breno Leitao
2026-03-09 16:38 ` Peter Zijlstra [this message]
2026-03-09 17:00 ` Breno Leitao
2026-03-10 1:45 ` Mi, Dapeng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260309163847.GE2277644@noisy.programming.kicks-ass.net \
--to=peterz@infradead.org \
--cc=acme@kernel.org \
--cc=adrian.hunter@intel.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=bp@alien8.de \
--cc=dapeng1.mi@linux.intel.com \
--cc=dave.hansen@linux.intel.com \
--cc=hpa@zytor.com \
--cc=irogers@google.com \
--cc=james.clark@linaro.org \
--cc=jolsa@kernel.org \
--cc=kernel-team@meta.com \
--cc=leitao@debian.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=tglx@kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox