From: Peter Zijlstra <peterz@infradead.org>
To: Breno Leitao <leitao@debian.org>
Cc: Ingo Molnar <mingo@redhat.com>,
Arnaldo Carvalho de Melo <acme@kernel.org>,
Namhyung Kim <namhyung@kernel.org>,
Mark Rutland <mark.rutland@arm.com>,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Jiri Olsa <jolsa@kernel.org>, Ian Rogers <irogers@google.com>,
Adrian Hunter <adrian.hunter@intel.com>,
James Clark <james.clark@linaro.org>,
Thomas Gleixner <tglx@kernel.org>, Borislav Petkov <bp@alien8.de>,
Dave Hansen <dave.hansen@linux.intel.com>,
x86@kernel.org, "H. Peter Anvin" <hpa@zytor.com>,
Dapeng Mi <dapeng1.mi@linux.intel.com>,
linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org,
kernel-team@meta.com
Subject: Re: [PATCH] perf/x86: Restore event pointer setup in x86_pmu_start()
Date: Mon, 9 Mar 2026 17:38:47 +0100 [thread overview]
Message-ID: <20260309163847.GE2277644@noisy.programming.kicks-ass.net> (raw)
In-Reply-To: <20260309-perf-v1-1-601ffb531893@debian.org>
On Mon, Mar 09, 2026 at 07:40:56AM -0700, Breno Leitao wrote:
> A production AMD EPYC system crashed with a NULL pointer dereference
> in the PMU NMI handler:
>
> BUG: kernel NULL pointer dereference, address: 0000000000000198
> RIP: x86_perf_event_update+0xc/0xa0
> Call Trace:
> <NMI>
> amd_pmu_v2_handle_irq+0x1a6/0x390
> perf_event_nmi_handler+0x24/0x40
>
> The faulting instruction is `cmpq $0x0, 0x198(%rdi)` with RDI=0,
> corresponding to the `if (unlikely(!hwc->event_base))` check in
> x86_perf_event_update() where hwc = &event->hw and event is NULL.
>
> drgn inspection of the vmcore on CPU 106 showed a mismatch between
> cpuc->active_mask and cpuc->events[]:
>
> active_mask: 0x1e (bits 1, 2, 3, 4)
> events[1]: 0xff1100136cbd4f38 (valid)
> events[2]: 0x0 (NULL, but active_mask bit 2 set)
> events[3]: 0xff1100076fd2cf38 (valid)
> events[4]: 0xff1100079e990a90 (valid)
>
> The event that should occupy events[2] was found in event_list[2]
> with hw.idx=2 and hw.state=0x0, confirming x86_pmu_start() had run
> (which clears hw.state and sets active_mask) but events[2] was
> never populated.
>
> Another event (event_list[0]) had hw.state=0x7 (STOPPED|UPTODATE|ARCH),
> showing it was stopped when the PMU rescheduled events, confirming the
> throttle-then-reschedule sequence occurred.
>
> The root cause is commit 7e772a93eb61 ("perf/x86: Fix NULL event access
> and potential PEBS record loss") which moved the cpuc->events[idx]
> assignment out of x86_pmu_start() and into x86_pmu_enable(). This
> broke any path that calls pmu->start() without going through
> x86_pmu_enable() -- specifically the unthrottle path:
>
> perf_adjust_freq_unthr_events()
> -> perf_event_unthrottle_group()
> -> perf_event_unthrottle()
> -> event->pmu->start(event, 0)
> -> x86_pmu_start() // sets active_mask but not events[]
>
> The race sequence is:
>
> 1. A group of perf events overflows, triggering group throttle via
> perf_event_throttle_group(). All events are stopped: active_mask
> bits cleared, events[] preserved (x86_pmu_stop no longer clears
> events[] after commit 7e772a93eb61).
>
> 2. While still throttled (PERF_HES_STOPPED), x86_pmu_enable() runs
> due to other scheduling activity. Stopped events that need to
> move counters get PERF_HES_ARCH set and events[old_idx] cleared.
> In step 2 of x86_pmu_enable(), PERF_HES_ARCH causes these events
> to be skipped -- events[new_idx] is never set.
So why not just move this then? Having less sites that set that value is
more better, no?
---
diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index 03ce1bc7ef2e..54b4c315d927 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -1372,6 +1372,8 @@ static void x86_pmu_enable(struct pmu *pmu)
else if (i < n_running)
continue;
+ cpuc->events[hwc->idx] = event;
+
if (hwc->state & PERF_HES_ARCH)
continue;
@@ -1379,7 +1381,6 @@ static void x86_pmu_enable(struct pmu *pmu)
* if cpuc->enabled = 0, then no wrmsr as
* per x86_pmu_enable_event()
*/
- cpuc->events[hwc->idx] = event;
x86_pmu_start(event, PERF_EF_RELOAD);
}
cpuc->n_added = 0;
next prev parent reply other threads:[~2026-03-09 16:38 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-03-09 14:40 [PATCH] perf/x86: Restore event pointer setup in x86_pmu_start() Breno Leitao
2026-03-09 16:38 ` Peter Zijlstra [this message]
2026-03-09 17:00 ` Breno Leitao
2026-03-10 1:45 ` Mi, Dapeng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260309163847.GE2277644@noisy.programming.kicks-ass.net \
--to=peterz@infradead.org \
--cc=acme@kernel.org \
--cc=adrian.hunter@intel.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=bp@alien8.de \
--cc=dapeng1.mi@linux.intel.com \
--cc=dave.hansen@linux.intel.com \
--cc=hpa@zytor.com \
--cc=irogers@google.com \
--cc=james.clark@linaro.org \
--cc=jolsa@kernel.org \
--cc=kernel-team@meta.com \
--cc=leitao@debian.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=tglx@kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.