* [Patch v6 01/20] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:01 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 02/20] perf/x86/intel: Validate LBR XSAVE state before updating LBR stack Dapeng Mi
` (19 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
Disable LBR if x86_pmu.lbr_nr is larger than MAX_LBR_ENTRIES, preventing
out-of-bounds access in the cached LBR stack handling path.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/core.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index 4e344fb1ae3e..82d9b0e5ad1c 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -2645,6 +2645,12 @@ static int __init init_hw_perf_events(void)
if (!x86_pmu.config_mask)
x86_pmu.config_mask = X86_RAW_EVENT_MASK;
+ if (x86_pmu.lbr_nr > MAX_LBR_ENTRIES) {
+ pr_warn("LBR entries %d exceeds max %d, disable LBR.\n",
+ x86_pmu.lbr_nr, MAX_LBR_ENTRIES);
+ x86_pmu.lbr_nr = 0;
+ }
+
perf_events_lapic_init();
register_nmi_handler(NMI_LOCAL, perf_event_nmi_handler, 0, "PMI");
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [Patch v6 02/20] perf/x86/intel: Validate LBR XSAVE state before updating LBR stack
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
2026-11-06 4:32 ` [Patch v6 01/20] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:05 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 03/20] perf/x86/amd/lbr: Limit branch snapshots to valid entries Dapeng Mi
` (18 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
intel_pmu_arch_lbr_read_xsave() doesn't check if the LBR data is present
in the xsave area. If LBR data is in the INIT state, the LBR data may not
be really written into the xsave area.
Add an explicit LBR XSAVE state presence check before updating the cached
LBR stack. Also clear the cached LBR stack when intel_pmu_lbr_reset() is
called, in addition to resetting the hardware LBR register file, to prevent
stale LBR data from being reused after a context switch or similar
transition.
Fixes: c085fb877467 ("perf/x86/intel/lbr: Support XSAVES for arch LBR read")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/lbr.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 0b446d8912e4..a2e4ba7e1a4f 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -193,6 +193,8 @@ void intel_pmu_lbr_reset(void)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+ cpuc->lbr_stack.nr = 0;
+
if (!x86_pmu.lbr_nr)
return;
@@ -1010,9 +1012,15 @@ static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)
intel_pmu_store_lbr(cpuc, NULL);
return;
}
+ xsave->header.xfeatures = 0;
xsaves(&xsave->xsave, XFEATURE_MASK_LBR);
- intel_pmu_store_lbr(cpuc, xsave->lbr.entries);
+ if (xsave->header.xfeatures & XFEATURE_MASK_LBR) {
+ intel_pmu_store_lbr(cpuc, xsave->lbr.entries);
+ } else {
+ /* Clear the LBR stack to avoid leaking stale entries. */
+ cpuc->lbr_stack.nr = 0;
+ }
}
void intel_pmu_lbr_read(void)
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [Patch v6 03/20] perf/x86/amd/lbr: Limit branch snapshots to valid entries
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
2026-11-06 4:32 ` [Patch v6 01/20] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES Dapeng Mi
2026-11-06 4:32 ` [Patch v6 02/20] perf/x86/intel: Validate LBR XSAVE state before updating LBR stack Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 4:50 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 04/20] perf/x86/intel/lbr: " Dapeng Mi
` (17 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Sashiko, Dapeng Mi
From: Sandipan Das <sandipan.das@amd.com>
The branch snapshot callback used by bpf_get_branch_snapshot() copies
up to x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the
last LBR read kept fewer branches. The entries past lbr_stack.nr are
invalid, but they end up reaching BPF programs.
Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when
amd_pmu_lbr_read() skips reading the registers, so that a stale count
is never reused.
Fixes: a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-163130@kernel.org/
Signed-off-by: Sandipan Das <sandipan.das@amd.com>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/amd/core.c | 2 +-
arch/x86/events/amd/lbr.c | 4 +++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
index 49b6b8fce566..bf631f60757c 100644
--- a/arch/x86/events/amd/core.c
+++ b/arch/x86/events/amd/core.c
@@ -944,7 +944,7 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
cpuc = this_cpu_ptr(&cpu_hw_events);
amd_pmu_lbr_read();
- cnt = min(cnt, x86_pmu.lbr_nr);
+ cnt = min(cnt, cpuc->lbr_stack.nr);
memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
amd_pmu_v2_enable_all(0);
diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c
index a55646fcb846..6c93b8042c73 100644
--- a/arch/x86/events/amd/lbr.c
+++ b/arch/x86/events/amd/lbr.c
@@ -168,8 +168,10 @@ void amd_pmu_lbr_read(void)
struct branch_entry entry;
int out = 0, idx, i;
- if (!cpuc->lbr_users)
+ if (!cpuc->lbr_users) {
+ cpuc->lbr_stack.nr = 0;
return;
+ }
for (i = 0; i < x86_pmu.lbr_nr; i++) {
entry.from.full = amd_pmu_lbr_get_from(i);
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [Patch v6 04/20] perf/x86/intel/lbr: Limit branch snapshots to valid entries
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (2 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 03/20] perf/x86/amd/lbr: Limit branch snapshots to valid entries Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 4:52 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 05/20] perf/x86: Add lbr_enabled to track LBR state Dapeng Mi
` (16 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
The branch snapshot callback used by bpf_get_branch_snapshot() always
copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the
last LBR read kept fewer branches. The entries past lbr_stack.nr hold
invalid branches which end up reaching BPF programs.
Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when
no active LBR events or LBR hardware is owned by guest. So the stale or
guest LBR stack data can't be leaked.
For the case where all LBR events are PEBS events, hardware LBR reads are
currently skipped; that can leave stale LBR stack data from previous PEBS
sampling in place and may be inconsistent with the intended semantics of
bpf_get_branch_snapshot(). This will be addressed in a follow-up patch.
Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-163130@kernel.org/
Co-developed-by: Sandipan Das <sandipan.das@amd.com>
Signed-off-by: Sandipan Das <sandipan.das@amd.com>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 2 +-
arch/x86/events/intel/lbr.c | 9 +++++++--
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 0a34d674df59..5dfb9e9156e8 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2822,7 +2822,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
intel_pmu_lbr_read();
- cnt = min_t(unsigned int, cnt, x86_pmu.lbr_nr);
+ cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
intel_pmu_enable_all(0);
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index a2e4ba7e1a4f..1ae853f7ac1a 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -1027,14 +1027,19 @@ void intel_pmu_lbr_read(void)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+ if (!cpuc->lbr_users || vlbr_exclude_host()) {
+ /* Reset lbr_stack.nr to prevent stale data leakage. */
+ cpuc->lbr_stack.nr = 0;
+ return;
+ }
+
/*
* Don't read when all LBRs users are using adaptive PEBS.
*
* This could be smarter and actually check the event,
* but this simple approach seems to work for now.
*/
- if (!cpuc->lbr_users || vlbr_exclude_host() ||
- cpuc->lbr_users == cpuc->lbr_pebs_users)
+ if (cpuc->lbr_users == cpuc->lbr_pebs_users)
return;
x86_pmu.lbr_read(cpuc);
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* Re: [Patch v6 04/20] perf/x86/intel/lbr: Limit branch snapshots to valid entries
2026-11-06 4:32 ` [Patch v6 04/20] perf/x86/intel/lbr: " Dapeng Mi
@ 2026-11-06 4:52 ` sashiko-bot
0 siblings, 0 replies; 45+ messages in thread
From: sashiko-bot @ 2026-11-06 4:52 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users
> The branch snapshot callback used by bpf_get_branch_snapshot() always
> copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the
> last LBR read kept fewer branches. The entries past lbr_stack.nr hold
> invalid branches which end up reaching BPF programs.
>
> Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when
> no active LBR events or LBR hardware is owned by guest. So the stale or
> guest LBR stack data can't be leaked.
>
> For the case where all LBR events are PEBS events, hardware LBR reads are
> currently skipped; that can leave stale LBR stack data from previous PEBS
> sampling in place and may be inconsistent with the intended semantics of
> bpf_get_branch_snapshot(). This will be addressed in a follow-up patch.
>
> Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Closes: https://lore.kernel.org/all/sashiko-outbox-163130@kernel.org/
> Co-developed-by: Sandipan Das <sandipan.das@amd.com>
> Signed-off-by: Sandipan Das <sandipan.das@amd.com>
> Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261106043257.842864-1-dapeng1.mi@linux.intel.com?part=4
^ permalink raw reply [flat|nested] 45+ messages in thread
* [Patch v6 05/20] perf/x86: Add lbr_enabled to track LBR state
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (3 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 04/20] perf/x86/intel/lbr: " Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:10 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 06/20] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled Dapeng Mi
` (15 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
Add a lbr_enabled flag to track whether LBR is enabled. This will be
used by a subsequent change to avoid blindly re-enabling LBR in
perf_snapshot_branch_stack() without checking whether LBR has already
been disabled.
Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Fixes: a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/amd/lbr.c | 2 ++
arch/x86/events/intel/core.c | 5 +++++
arch/x86/events/intel/lbr.c | 2 ++
arch/x86/events/perf_event.h | 9 +++++++++
4 files changed, 18 insertions(+)
diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c
index 6c93b8042c73..400c94ecded0 100644
--- a/arch/x86/events/amd/lbr.c
+++ b/arch/x86/events/amd/lbr.c
@@ -399,6 +399,8 @@ void amd_pmu_lbr_enable_all(void)
if (!cpuc->lbr_users || !x86_pmu.lbr_nr)
return;
+ cpuc->lbr_enabled = 1;
+
/* Set hardware branch filter */
if (cpuc->lbr_select) {
lbr_select = cpuc->lbr_sel->config & LBR_SELECT_MASK;
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 5dfb9e9156e8..e99b311c7b8d 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -4001,6 +4001,11 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
* It needs to be restored when leaving the handler.
*/
pmu_enabled = cpuc->enabled;
+
+ /* If pmu version < 4, LBR freeze disables LBR. */
+ if (x86_pmu.version < 4)
+ cpuc->lbr_enabled = 0;
+
/*
* In general, the early ACK is only applied for old platforms.
* For the big core starts from Haswell, the late ACK should be
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 1ae853f7ac1a..ee724b28d90c 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -132,6 +132,8 @@ static void __intel_pmu_lbr_enable(bool pmi)
if (pmi && x86_pmu.version >= 4)
return;
+ cpuc->lbr_enabled = 1;
+
/*
* No need to reprogram LBR_SELECT in a PMI, as it
* did not change.
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index e274802ef062..a4aab8130e92 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -473,6 +473,7 @@ struct cpu_hw_events {
unsigned long active_mask[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
unsigned long dirty[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
int enabled;
+ int lbr_enabled;
int n_events; /* the # of events in the below arrays */
int n_added; /* the # last events in the below arrays;
@@ -1676,8 +1677,11 @@ int amd_pmu_lbr_hw_config(struct perf_event *event);
static __always_inline void __amd_pmu_lbr_disable(void)
{
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
u64 dbg_ctl, dbg_extn_cfg;
+ cpuc->lbr_enabled = 0;
+
rdmsrq(MSR_AMD_DBG_EXTN_CFG, dbg_extn_cfg);
wrmsrq(MSR_AMD_DBG_EXTN_CFG, dbg_extn_cfg & ~DBG_EXTN_CFG_LBRV2EN);
@@ -1829,13 +1833,18 @@ static __always_inline void __intel_pmu_pebs_disable_all(void)
static __always_inline void __intel_pmu_arch_lbr_disable(void)
{
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+
+ cpuc->lbr_enabled = 0;
wrmsrq(MSR_ARCH_LBR_CTL, 0);
}
static __always_inline void __intel_pmu_lbr_disable(void)
{
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
u64 debugctl;
+ cpuc->lbr_enabled = 0;
rdmsrq(MSR_IA32_DEBUGCTLMSR, debugctl);
debugctl &= ~(DEBUGCTLMSR_LBR | DEBUGCTLMSR_FREEZE_LBRS_ON_PMI);
wrmsrq(MSR_IA32_DEBUGCTLMSR, debugctl);
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [Patch v6 06/20] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (4 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 05/20] perf/x86: Add lbr_enabled to track LBR state Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:01 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 07/20] perf/x86/intel: Add helper to enable PMU without LBR Dapeng Mi
` (14 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
amd_pmu_v2_snapshot_branch_stack() currently re-enables the PMU and LBR
unconditionally even when they were already disabled in PMI context
This can cause events such as cycles and instructions to be counted
while the PMI handler is running, which is not intended.
Only restore the PMU and LBR on the BPF path if they were originally
enabled.
Cc: Sandipan Das <sandipan.das@amd.com>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-179823@kernel.org/
Fixes: a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/amd/core.c | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
index bf631f60757c..851d8a938f52 100644
--- a/arch/x86/events/amd/core.c
+++ b/arch/x86/events/amd/core.c
@@ -932,22 +932,28 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
{
struct cpu_hw_events *cpuc;
unsigned long flags;
+ int pmu_enabled;
+ int lbr_enabled;
/*
* The sequence of steps to freeze LBR should be completely inlined
* and contain no branches to minimize contamination of LBR snapshot
*/
local_irq_save(flags);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
+ pmu_enabled = cpuc->enabled;
+ lbr_enabled = cpuc->lbr_enabled;
amd_pmu_core_disable_all();
__amd_pmu_lbr_disable();
- cpuc = this_cpu_ptr(&cpu_hw_events);
-
amd_pmu_lbr_read();
cnt = min(cnt, cpuc->lbr_stack.nr);
memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
- amd_pmu_v2_enable_all(0);
+ if (lbr_enabled)
+ amd_pmu_lbr_enable_all();
+ if (pmu_enabled)
+ amd_pmu_core_enable_all();
local_irq_restore(flags);
return cnt;
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* Re: [Patch v6 06/20] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
2026-11-06 4:32 ` [Patch v6 06/20] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled Dapeng Mi
@ 2026-11-06 5:01 ` sashiko-bot
2026-11-06 7:07 ` Mi, Dapeng
0 siblings, 1 reply; 45+ messages in thread
From: sashiko-bot @ 2026-11-06 5:01 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users, bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/x86/amd: amd_pmu_v2_snapshot_branch_stack() leaves cpuc->enabled set
--
commit dbf85e8b6a5abeff651d80166ad0c339896c5b42
Author: Dapeng Mi <dapeng1.mi@linux.intel.com>
perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
This commit updates amd_pmu_v2_snapshot_branch_stack() to save cpuc->enabled
and cpuc->lbr_enabled before disabling the PMU and LBR, and only re-enable
them if they were previously enabled.
> diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
> index bf631f60757cc..851d8a938f522 100644
> --- a/arch/x86/events/amd/core.c
> +++ b/arch/x86/events/amd/core.c
> @@ -932,22 +932,28 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
> {
> struct cpu_hw_events *cpuc;
> unsigned long flags;
> + int pmu_enabled;
> + int lbr_enabled;
>
> /*
> * The sequence of steps to freeze LBR should be completely inlined
> * and contain no branches to minimize contamination of LBR snapshot
> */
> local_irq_save(flags);
> + cpuc = this_cpu_ptr(&cpu_hw_events);
> + pmu_enabled = cpuc->enabled;
> + lbr_enabled = cpuc->lbr_enabled;
> amd_pmu_core_disable_all();
> __amd_pmu_lbr_disable();
>
> - cpuc = this_cpu_ptr(&cpu_hw_events);
> -
> amd_pmu_lbr_read();
> cnt = min(cnt, cpuc->lbr_stack.nr);
> memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
>
> - amd_pmu_v2_enable_all(0);
> + if (lbr_enabled)
> + amd_pmu_lbr_enable_all();
> + if (pmu_enabled)
> + amd_pmu_core_enable_all();
[Severity: Medium]
As reported by Sashiko on v5 [1], while __amd_pmu_lbr_disable() clears
cpuc->lbr_enabled = 0, amd_pmu_core_disable_all() only clears the hardware
global control MSR and leaves cpuc->enabled set to 1.
If a BPF program in non-NMI context calls bpf_get_branch_snapshot() ->
amd_pmu_v2_snapshot_branch_stack() while cpuc->enabled is 1, local_irq_save()
does not mask NMIs. If an NMI arrives while amd_pmu_v2_snapshot_branch_stack()
is reading LBR MSRs or copying entries, perf_event_nmi_handler() ->
amd_pmu_v2_handle_irq() will see cpuc->enabled == 1 and re-enable the core PMU
counters on exit:
arch/x86/events/amd/core.c:amd_pmu_v2_handle_irq() {
pmu_enabled = cpuc->enabled;
cpuc->enabled = 0;
/* Stop counting but do not disable LBR */
amd_pmu_core_disable_all();
...
done:
cpuc->enabled = pmu_enabled;
/* Resume counting only if PMU is active */
if (pmu_enabled)
amd_pmu_core_enable_all();
}
Similarly, if an NMI triggers a nested BPF call to
amd_pmu_v2_snapshot_branch_stack(), the nested call will also observe
cpuc->enabled == 1 and call amd_pmu_core_enable_all() before returning to the
outer snapshot.
Should amd_pmu_v2_snapshot_branch_stack() clear cpuc->enabled = 0 after saving
pmu_enabled and restore cpuc->enabled = pmu_enabled before re-enabling the
core PMU, matching amd_pmu_v2_handle_irq()?
[1] https://lore.kernel.org/r/20261029043914.1395147-7-dapeng1.mi@linux.intel.com
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261106043257.842864-1-dapeng1.mi@linux.intel.com?part=6
^ permalink raw reply [flat|nested] 45+ messages in thread* Re: [Patch v6 06/20] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
2026-11-06 5:01 ` sashiko-bot
@ 2026-11-06 7:07 ` Mi, Dapeng
0 siblings, 0 replies; 45+ messages in thread
From: Mi, Dapeng @ 2026-11-06 7:07 UTC (permalink / raw)
To: sashiko-reviews; +Cc: linux-perf-users, bpf
On 11/6/2026 1:01 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [Medium] perf/x86/amd: amd_pmu_v2_snapshot_branch_stack() leaves cpuc->enabled set
> --
>
> commit dbf85e8b6a5abeff651d80166ad0c339896c5b42
> Author: Dapeng Mi <dapeng1.mi@linux.intel.com>
>
> perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
>
> This commit updates amd_pmu_v2_snapshot_branch_stack() to save cpuc->enabled
> and cpuc->lbr_enabled before disabling the PMU and LBR, and only re-enable
> them if they were previously enabled.
>
>> diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
>> index bf631f60757cc..851d8a938f522 100644
>> --- a/arch/x86/events/amd/core.c
>> +++ b/arch/x86/events/amd/core.c
>> @@ -932,22 +932,28 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
>> {
>> struct cpu_hw_events *cpuc;
>> unsigned long flags;
>> + int pmu_enabled;
>> + int lbr_enabled;
>>
>> /*
>> * The sequence of steps to freeze LBR should be completely inlined
>> * and contain no branches to minimize contamination of LBR snapshot
>> */
>> local_irq_save(flags);
>> + cpuc = this_cpu_ptr(&cpu_hw_events);
>> + pmu_enabled = cpuc->enabled;
>> + lbr_enabled = cpuc->lbr_enabled;
>> amd_pmu_core_disable_all();
>> __amd_pmu_lbr_disable();
>>
>> - cpuc = this_cpu_ptr(&cpu_hw_events);
>> -
>> amd_pmu_lbr_read();
>> cnt = min(cnt, cpuc->lbr_stack.nr);
>> memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
>>
>> - amd_pmu_v2_enable_all(0);
>> + if (lbr_enabled)
>> + amd_pmu_lbr_enable_all();
>> + if (pmu_enabled)
>> + amd_pmu_core_enable_all();
> [Severity: Medium]
> As reported by Sashiko on v5 [1], while __amd_pmu_lbr_disable() clears
> cpuc->lbr_enabled = 0, amd_pmu_core_disable_all() only clears the hardware
> global control MSR and leaves cpuc->enabled set to 1.
>
> If a BPF program in non-NMI context calls bpf_get_branch_snapshot() ->
> amd_pmu_v2_snapshot_branch_stack() while cpuc->enabled is 1, local_irq_save()
> does not mask NMIs. If an NMI arrives while amd_pmu_v2_snapshot_branch_stack()
> is reading LBR MSRs or copying entries, perf_event_nmi_handler() ->
> amd_pmu_v2_handle_irq() will see cpuc->enabled == 1 and re-enable the core PMU
> counters on exit:
This issue should never happen in reality since amd_pmu_core_disable_all()
would disable PMU, so no PMI can be interrupt the BPF LBR snapshot.
>
> arch/x86/events/amd/core.c:amd_pmu_v2_handle_irq() {
> pmu_enabled = cpuc->enabled;
> cpuc->enabled = 0;
>
> /* Stop counting but do not disable LBR */
> amd_pmu_core_disable_all();
> ...
> done:
> cpuc->enabled = pmu_enabled;
>
> /* Resume counting only if PMU is active */
> if (pmu_enabled)
> amd_pmu_core_enable_all();
> }
>
> Similarly, if an NMI triggers a nested BPF call to
> amd_pmu_v2_snapshot_branch_stack(), the nested call will also observe
> cpuc->enabled == 1 and call amd_pmu_core_enable_all() before returning to the
> outer snapshot.
>
> Should amd_pmu_v2_snapshot_branch_stack() clear cpuc->enabled = 0 after saving
> pmu_enabled and restore cpuc->enabled = pmu_enabled before re-enabling the
> core PMU, matching amd_pmu_v2_handle_irq()?
>
> [1] https://lore.kernel.org/r/20261029043914.1395147-7-dapeng1.mi@linux.intel.com
>
^ permalink raw reply [flat|nested] 45+ messages in thread
* [Patch v6 07/20] perf/x86/intel: Add helper to enable PMU without LBR
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (5 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 06/20] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 4:45 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 08/20] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled Dapeng Mi
` (13 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
Currently, __intel_pmu_enable_all() also re-enables LBR in addition to
the PMU. Split out the PMU-only path into a new helper,
___intel_pmu_enable_all(), so the PMU can be enabled independently. This
helper will be used by a subsequent patch.
Also remove the unused added argument from __intel_pmu_enable_all().
No functional change intended.
Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index e99b311c7b8d..9d8398da1989 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2783,13 +2783,11 @@ static __always_inline void intel_pmu_disable_all(void)
intel_pmu_lbr_disable_all();
}
-static void __intel_pmu_enable_all(int added, bool pmi)
+static void ___intel_pmu_enable_all(void)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
u64 intel_ctrl = hybrid(cpuc->pmu, intel_ctrl);
- intel_pmu_lbr_enable_all(pmi);
-
if (cpuc->fixed_ctrl_val != cpuc->active_fixed_ctrl_val) {
wrmsrq(MSR_ARCH_PERFMON_FIXED_CTR_CTRL, cpuc->fixed_ctrl_val);
cpuc->active_fixed_ctrl_val = cpuc->fixed_ctrl_val;
@@ -2809,10 +2807,16 @@ static void __intel_pmu_enable_all(int added, bool pmi)
}
}
+static void __intel_pmu_enable_all(bool pmi)
+{
+ intel_pmu_lbr_enable_all(pmi);
+ ___intel_pmu_enable_all();
+}
+
static void intel_pmu_enable_all(int added)
{
static_call_cond(x86_pmu_pebs_enable_all)();
- __intel_pmu_enable_all(added, false);
+ __intel_pmu_enable_all(false);
}
static noinline int
@@ -4056,7 +4060,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
/* Only restore PMU state when it's active. See x86_pmu_disable(). */
cpuc->enabled = pmu_enabled;
if (pmu_enabled)
- __intel_pmu_enable_all(0, true);
+ __intel_pmu_enable_all(true);
intel_bts_enable_local();
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [Patch v6 08/20] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (6 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 07/20] perf/x86/intel: Add helper to enable PMU without LBR Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:06 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 09/20] perf/x86/intel: Keep LBR frozen throughout PMI handling Dapeng Mi
` (12 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
__intel_pmu_snapshot_branch_stack() currently re-enables the PMU and LBR
unconditionally even when they were already disabled in PMI context.
This can cause events such as cycles and instructions to be counted
while the PMI handler is running, which is not intended.
Only restore the PMU and LBR on the BPF path if they were originally
enabled.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-179823@kernel.org/
Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 38 ++++++++++++++++++++++++++++++++----
1 file changed, 34 insertions(+), 4 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 9d8398da1989..739c454827ac 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2821,7 +2821,8 @@ static void intel_pmu_enable_all(int added)
static noinline int
__intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
- unsigned int cnt, unsigned long flags)
+ unsigned int cnt, unsigned long flags,
+ int pmu_enabled, int lbr_enabled)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
@@ -2829,7 +2830,22 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
- intel_pmu_enable_all(0);
+
+ /*
+ * Do not blindly call intel_pmu_enable_all() to re-enable LBR.
+ * This path may be entered from the PMI handler through
+ * bpf_overflow_handler().
+ *
+ * If PMU version < 4, LBR is disabled by the LBR freeze mechanism,
+ * LBR can't be re-enabled here.
+ * If PMU version >= 4, LBR is only frozen instead of disabled,
+ * LBR should be re-enabled here.
+ */
+ if (lbr_enabled)
+ intel_pmu_lbr_enable_all(false);
+ if (pmu_enabled)
+ ___intel_pmu_enable_all();
+
local_irq_restore(flags);
return cnt;
}
@@ -2837,27 +2853,41 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
static int
intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
{
+ struct cpu_hw_events *cpuc;
unsigned long flags;
+ int pmu_enabled;
+ int lbr_enabled;
/* must not have branches... */
local_irq_save(flags);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
+ pmu_enabled = cpuc->enabled;
+ lbr_enabled = cpuc->lbr_enabled;
__intel_pmu_disable_all(false); /* we don't care about BTS */
__intel_pmu_lbr_disable();
/* ... until here */
- return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
+ return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
+ pmu_enabled, lbr_enabled);
}
static int
intel_pmu_snapshot_arch_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
{
+ struct cpu_hw_events *cpuc;
unsigned long flags;
+ int pmu_enabled;
+ int lbr_enabled;
/* must not have branches... */
local_irq_save(flags);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
+ pmu_enabled = cpuc->enabled;
+ lbr_enabled = cpuc->lbr_enabled;
__intel_pmu_disable_all(false); /* we don't care about BTS */
__intel_pmu_arch_lbr_disable();
/* ... until here */
- return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
+ return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
+ pmu_enabled, lbr_enabled);
}
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* Re: [Patch v6 08/20] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
2026-11-06 4:32 ` [Patch v6 08/20] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled Dapeng Mi
@ 2026-11-06 5:06 ` sashiko-bot
2026-11-06 7:09 ` Mi, Dapeng
0 siblings, 1 reply; 45+ messages in thread
From: sashiko-bot @ 2026-11-06 5:06 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users, bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/x86/intel: cpuc->enabled not cleared in intel_pmu_snapshot_branch_stack()
--
commit f8f32e2d56da9a93e561d91b6c3fb89e787c5d9e
Author: Dapeng Mi <dapeng1.mi@linux.intel.com>
perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
This commit updates __intel_pmu_snapshot_branch_stack() and its callers
to save the PMU and LBR enabled state and only re-enable them after
taking a branch snapshot if they were originally enabled.
> diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
> index 9d8398da1989f..739c454827ac5 100644
> --- a/arch/x86/events/intel/core.c
> +++ b/arch/x86/events/intel/core.c
> @@ -2821,7 +2821,8 @@ static void intel_pmu_enable_all(int added)
>
> static noinline int
> __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
> - unsigned int cnt, unsigned long flags)
> + unsigned int cnt, unsigned long flags,
> + int pmu_enabled, int lbr_enabled)
> {
> struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
>
> @@ -2829,7 +2830,22 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
> cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
>
> memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
> - intel_pmu_enable_all(0);
> +
> + /*
> + * Do not blindly call intel_pmu_enable_all() to re-enable LBR.
> + * This path may be entered from the PMI handler through
> + * bpf_overflow_handler().
> + *
> + * If PMU version < 4, LBR is disabled by the LBR freeze mechanism,
> + * LBR can't be re-enabled here.
> + * If PMU version >= 4, LBR is only frozen instead of disabled,
> + * LBR should be re-enabled here.
> + */
> + if (lbr_enabled)
> + intel_pmu_lbr_enable_all(false);
> + if (pmu_enabled)
> + ___intel_pmu_enable_all();
> +
> local_irq_restore(flags);
> return cnt;
> }
> @@ -2837,27 +2853,41 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
> static int
> intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
> {
> + struct cpu_hw_events *cpuc;
> unsigned long flags;
> + int pmu_enabled;
> + int lbr_enabled;
>
> /* must not have branches... */
> local_irq_save(flags);
> + cpuc = this_cpu_ptr(&cpu_hw_events);
> + pmu_enabled = cpuc->enabled;
> + lbr_enabled = cpuc->lbr_enabled;
> __intel_pmu_disable_all(false); /* we don't care about BTS */
> __intel_pmu_lbr_disable();
> /* ... until here */
> - return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
> + return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
> + pmu_enabled, lbr_enabled);
> }
>
> static int
> intel_pmu_snapshot_arch_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
> {
> + struct cpu_hw_events *cpuc;
> unsigned long flags;
> + int pmu_enabled;
> + int lbr_enabled;
>
> /* must not have branches... */
> local_irq_save(flags);
> + cpuc = this_cpu_ptr(&cpu_hw_events);
> + pmu_enabled = cpuc->enabled;
> + lbr_enabled = cpuc->lbr_enabled;
> __intel_pmu_disable_all(false); /* we don't care about BTS */
> __intel_pmu_arch_lbr_disable();
> /* ... until here */
> - return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
> + return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
> + pmu_enabled, lbr_enabled);
> }
[Severity: Medium]
Should intel_pmu_snapshot_branch_stack() and
intel_pmu_snapshot_arch_branch_stack() also clear cpuc->enabled = 0
before calling __intel_pmu_disable_all(false), and restore
cpuc->enabled = pmu_enabled in __intel_pmu_snapshot_branch_stack()?
While __intel_pmu_lbr_disable() and __intel_pmu_arch_lbr_disable() clear
cpuc->lbr_enabled = 0, __intel_pmu_disable_all(false) only writes 0 to
MSR_CORE_PERF_GLOBAL_CTRL without updating cpuc->enabled.
When bpf_get_branch_snapshot() is invoked outside PMI context (where
cpuc->enabled is 1), local_irq_save() does not mask NMIs. If an in-flight
PMI or another NMI arrives while __intel_pmu_snapshot_branch_stack() is
reading the LBRs, perf_event_nmi_handler() -> intel_pmu_handle_irq()
observes cpuc->enabled == 1 and re-enables the PMU (and LBR on PMU
version < 4) before returning:
arch/x86/events/intel/core.c:intel_pmu_handle_irq() {
pmu_enabled = cpuc->enabled;
...
done:
if (mid_ack)
apic_write(APIC_LVTPC, APIC_DM_NMI);
/* Only restore PMU state when it's active. See x86_pmu_disable(). */
cpuc->enabled = pmu_enabled;
if (pmu_enabled)
__intel_pmu_enable_all(true);
}
Similarly, a nested snapshot from an NMI or fentry/kprobe BPF program
during __intel_pmu_snapshot_branch_stack() would see cpuc->enabled == 1
and prematurely call ___intel_pmu_enable_all() before the outer snapshot
finishes.
Setting cpuc->enabled = 0 is a plain store with no branches, so could it
be cleared alongside saving pmu_enabled and restored in
__intel_pmu_snapshot_branch_stack() to match x86_pmu_disable(),
__intel_pmu_quiesce(), and intel_pmu_handle_irq()?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261106043257.842864-1-dapeng1.mi@linux.intel.com?part=8
^ permalink raw reply [flat|nested] 45+ messages in thread* Re: [Patch v6 08/20] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
2026-11-06 5:06 ` sashiko-bot
@ 2026-11-06 7:09 ` Mi, Dapeng
0 siblings, 0 replies; 45+ messages in thread
From: Mi, Dapeng @ 2026-11-06 7:09 UTC (permalink / raw)
To: sashiko-reviews; +Cc: linux-perf-users, bpf
On 11/6/2026 1:06 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [Medium] perf/x86/intel: cpuc->enabled not cleared in intel_pmu_snapshot_branch_stack()
> --
>
> commit f8f32e2d56da9a93e561d91b6c3fb89e787c5d9e
> Author: Dapeng Mi <dapeng1.mi@linux.intel.com>
>
> perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
>
> This commit updates __intel_pmu_snapshot_branch_stack() and its callers
> to save the PMU and LBR enabled state and only re-enable them after
> taking a branch snapshot if they were originally enabled.
>
>> diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
>> index 9d8398da1989f..739c454827ac5 100644
>> --- a/arch/x86/events/intel/core.c
>> +++ b/arch/x86/events/intel/core.c
>> @@ -2821,7 +2821,8 @@ static void intel_pmu_enable_all(int added)
>>
>> static noinline int
>> __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
>> - unsigned int cnt, unsigned long flags)
>> + unsigned int cnt, unsigned long flags,
>> + int pmu_enabled, int lbr_enabled)
>> {
>> struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
>>
>> @@ -2829,7 +2830,22 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
>> cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
>>
>> memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
>> - intel_pmu_enable_all(0);
>> +
>> + /*
>> + * Do not blindly call intel_pmu_enable_all() to re-enable LBR.
>> + * This path may be entered from the PMI handler through
>> + * bpf_overflow_handler().
>> + *
>> + * If PMU version < 4, LBR is disabled by the LBR freeze mechanism,
>> + * LBR can't be re-enabled here.
>> + * If PMU version >= 4, LBR is only frozen instead of disabled,
>> + * LBR should be re-enabled here.
>> + */
>> + if (lbr_enabled)
>> + intel_pmu_lbr_enable_all(false);
>> + if (pmu_enabled)
>> + ___intel_pmu_enable_all();
>> +
>> local_irq_restore(flags);
>> return cnt;
>> }
>> @@ -2837,27 +2853,41 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
>> static int
>> intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
>> {
>> + struct cpu_hw_events *cpuc;
>> unsigned long flags;
>> + int pmu_enabled;
>> + int lbr_enabled;
>>
>> /* must not have branches... */
>> local_irq_save(flags);
>> + cpuc = this_cpu_ptr(&cpu_hw_events);
>> + pmu_enabled = cpuc->enabled;
>> + lbr_enabled = cpuc->lbr_enabled;
>> __intel_pmu_disable_all(false); /* we don't care about BTS */
>> __intel_pmu_lbr_disable();
>> /* ... until here */
>> - return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
>> + return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
>> + pmu_enabled, lbr_enabled);
>> }
>>
>> static int
>> intel_pmu_snapshot_arch_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
>> {
>> + struct cpu_hw_events *cpuc;
>> unsigned long flags;
>> + int pmu_enabled;
>> + int lbr_enabled;
>>
>> /* must not have branches... */
>> local_irq_save(flags);
>> + cpuc = this_cpu_ptr(&cpu_hw_events);
>> + pmu_enabled = cpuc->enabled;
>> + lbr_enabled = cpuc->lbr_enabled;
>> __intel_pmu_disable_all(false); /* we don't care about BTS */
>> __intel_pmu_arch_lbr_disable();
>> /* ... until here */
>> - return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
>> + return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
>> + pmu_enabled, lbr_enabled);
>> }
> [Severity: Medium]
> Should intel_pmu_snapshot_branch_stack() and
> intel_pmu_snapshot_arch_branch_stack() also clear cpuc->enabled = 0
> before calling __intel_pmu_disable_all(false), and restore
> cpuc->enabled = pmu_enabled in __intel_pmu_snapshot_branch_stack()?
>
> While __intel_pmu_lbr_disable() and __intel_pmu_arch_lbr_disable() clear
> cpuc->lbr_enabled = 0, __intel_pmu_disable_all(false) only writes 0 to
> MSR_CORE_PERF_GLOBAL_CTRL without updating cpuc->enabled.
>
> When bpf_get_branch_snapshot() is invoked outside PMI context (where
> cpuc->enabled is 1), local_irq_save() does not mask NMIs. If an in-flight
> PMI or another NMI arrives while __intel_pmu_snapshot_branch_stack() is
> reading the LBRs, perf_event_nmi_handler() -> intel_pmu_handle_irq()
> observes cpuc->enabled == 1 and re-enables the PMU (and LBR on PMU
> version < 4) before returning:
Same with AMD. This issue should never happen in reality since
__intel_pmu_disable_all() would disable PMU, so no PMI can be interrupt the
BPF LBR snapshot.
>
> arch/x86/events/intel/core.c:intel_pmu_handle_irq() {
> pmu_enabled = cpuc->enabled;
> ...
> done:
> if (mid_ack)
> apic_write(APIC_LVTPC, APIC_DM_NMI);
> /* Only restore PMU state when it's active. See x86_pmu_disable(). */
> cpuc->enabled = pmu_enabled;
> if (pmu_enabled)
> __intel_pmu_enable_all(true);
> }
>
> Similarly, a nested snapshot from an NMI or fentry/kprobe BPF program
> during __intel_pmu_snapshot_branch_stack() would see cpuc->enabled == 1
> and prematurely call ___intel_pmu_enable_all() before the outer snapshot
> finishes.
>
> Setting cpuc->enabled = 0 is a plain store with no branches, so could it
> be cleared alongside saving pmu_enabled and restored in
> __intel_pmu_snapshot_branch_stack() to match x86_pmu_disable(),
> __intel_pmu_quiesce(), and intel_pmu_handle_irq()?
>
^ permalink raw reply [flat|nested] 45+ messages in thread
* [Patch v6 09/20] perf/x86/intel: Keep LBR frozen throughout PMI handling
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (7 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 08/20] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 4:58 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 10/20] perf: Add counted-bounds tracking for perf_branch_stack Dapeng Mi
` (11 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
For the platforms which has 4+ PMU version, streamline LBR freeze is
supported. The PERF_GLOBAL_STATUS.LBR_FRZ bit is set to freeze LBR once
a PMU counter overflow. However, the current PMI handler clears the LBR
freeze bit too early, allowing branch operations that occur during PMI
handling to be recorded into the LBR stack.
This can pollute or overwrite the intended branch snapshot, especially
when bpf_get_branch_snapshot() is invoked from bpf_overflow_handler().
In that case, branches recorded while processing the PMI are mixed into
the original LBR data.
Defer unfreezing until after PMI handling completes so the LBR remains
frozen for the entire PMI path and no branch activity from the handler
is recorded.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-179823@kernel.org/
Fixes: 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 739c454827ac..ab6261ab54b6 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -4062,7 +4062,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
loops = 0;
again:
intel_pmu_lbr_read();
- intel_pmu_ack_status(status);
+
+ /*
+ * Do not clear the LBR frozen status here; keep LBR frozen
+ * throughout PMI handling.
+ */
+ intel_pmu_ack_status(status & ~GLOBAL_STATUS_LBRS_FROZEN);
+
if (++loops > 100) {
static bool warned;
@@ -4081,10 +4087,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
* Repeat if there is more work to be done:
*/
status = intel_pmu_get_status();
- if (status)
+ if (status & ~GLOBAL_STATUS_LBRS_FROZEN)
goto again;
done:
+ if (status & GLOBAL_STATUS_LBRS_FROZEN)
+ intel_pmu_ack_status(status & GLOBAL_STATUS_LBRS_FROZEN);
+
if (mid_ack)
apic_write(APIC_LVTPC, APIC_DM_NMI);
/* Only restore PMU state when it's active. See x86_pmu_disable(). */
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [Patch v6 10/20] perf: Add counted-bounds tracking for perf_branch_stack
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (8 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 09/20] perf/x86/intel: Keep LBR frozen throughout PMI handling Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:00 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 11/20] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks Dapeng Mi
` (10 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Madhavan Srinivasan,
Michael Ellerman, Nicholas Piggin, Christophe Leroy (CS GROUP),
Will Deacon, Mark Rutland, linux-arm-kernel
struct perf_branch_stack uses a trailing flexible array member,
entries[], whose valid length is tracked by nr. When this structure
is embedded as a value member in cpu_hw_events, the compiler can treat
the trailing flexible array as having a fixed zero-length layout under
-fstrict-flex-arrays=3 and UBSan, which can lead to strict array-bounds
diagnostics when entries[] is indexed.
Annotate entries[] with __counted_by(nr) so the compiler and sanitizer
track the array length from the nr field. Because a structure containing
a counted flexible array cannot be embedded as a value member, switch
the per-CPU LBR stack backing storage to a complete __lbr_stack[] array
and add a helper cpu_lbr_stack() to recover the struct perf_branch_stack*
view for x86 and powerpc.
Also initialize the branch-stack count before populating entries[] to
avoid false-positive bounds checks while the stack is being filled.
No functional change intended.
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Cc: Sandipan Das <sandipan.das@amd.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: "Christophe Leroy (CS GROUP)" <chleroy@kernel.org>
Cc: Will Deacon <will@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/powerpc/perf/core-book3s.c | 43 +++++++++++-----
arch/x86/events/amd/brs.c | 7 +--
arch/x86/events/amd/core.c | 6 +--
arch/x86/events/amd/lbr.c | 44 +++++++++-------
arch/x86/events/core.c | 2 +-
arch/x86/events/intel/core.c | 5 +-
arch/x86/events/intel/ds.c | 12 +++--
arch/x86/events/intel/lbr.c | 89 +++++++++++++++++----------------
arch/x86/events/perf_event.h | 19 ++++++-
drivers/perf/arm_brbe.c | 5 ++
include/linux/perf_event.h | 2 +-
11 files changed, 144 insertions(+), 90 deletions(-)
diff --git a/arch/powerpc/perf/core-book3s.c b/arch/powerpc/perf/core-book3s.c
index 10ded865583c..978fd39b586b 100644
--- a/arch/powerpc/perf/core-book3s.c
+++ b/arch/powerpc/perf/core-book3s.c
@@ -30,6 +30,8 @@
#define BHRB_PREDICTION 0x0000000000000001
#define BHRB_EA 0xFFFFFFFFFFFFFFFCUL
+#define LBR_STACK_SIZE struct_size_t(struct perf_branch_stack, entries, BHRB_MAX_ENTRIES)
+
struct cpu_hw_events {
int n_events;
int n_percpu;
@@ -54,8 +56,7 @@ struct cpu_hw_events {
u64 bhrb_filter; /* BHRB HW branch filter */
unsigned int bhrb_users;
void *bhrb_context;
- struct perf_branch_stack bhrb_stack;
- struct perf_branch_entry bhrb_entries[BHRB_MAX_ENTRIES];
+ u64 __lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
u64 ic_init;
/* Store the PMC values */
@@ -177,6 +178,20 @@ static bool regs_use_siar(struct pt_regs *regs)
return ((TRAP(regs) == INTERRUPT_PERFMON) && regs->result);
}
+static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_events *cpuc)
+{
+ return (struct perf_branch_stack *)cpuc->__lbr_stack;
+}
+
+static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr)
+{
+ /*
+ * Initialize the branch-stack count before populating entries[]
+ * to avoid UBSAN complaints from the __counted_by(nr) flexible array.
+ */
+ cpu_lbr_stack(cpuc)->nr = nr;
+}
+
/*
* Things that are specific to 64-bit implementations.
*/
@@ -493,6 +508,8 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events *
r_index = 0;
u_index = 0;
+
+ cpu_init_lbr_stack(cpuhw, ppmu->bhrb_nr);
while (r_index < ppmu->bhrb_nr) {
/* Assembly read function */
val = read_bhrb(r_index++);
@@ -541,9 +558,9 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events *
/* Target branches use two entries
* (ie. computed gotos/XL form)
*/
- cpuhw->bhrb_entries[u_index].to = addr;
- cpuhw->bhrb_entries[u_index].mispred = pred;
- cpuhw->bhrb_entries[u_index].predicted = ~pred;
+ cpu_lbr_stack(cpuhw)->entries[u_index].to = addr;
+ cpu_lbr_stack(cpuhw)->entries[u_index].mispred = pred;
+ cpu_lbr_stack(cpuhw)->entries[u_index].predicted = ~pred;
/* Get from address in next entry */
val = read_bhrb(r_index++);
@@ -554,22 +571,22 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events *
r_index--;
addr = 0;
}
- cpuhw->bhrb_entries[u_index].from = addr;
+ cpu_lbr_stack(cpuhw)->entries[u_index].from = addr;
} else {
/* Branches to immediate field
(ie I or B form) */
- cpuhw->bhrb_entries[u_index].from = addr;
- cpuhw->bhrb_entries[u_index].to =
+ cpu_lbr_stack(cpuhw)->entries[u_index].from = addr;
+ cpu_lbr_stack(cpuhw)->entries[u_index].to =
power_pmu_bhrb_to(addr);
- cpuhw->bhrb_entries[u_index].mispred = pred;
- cpuhw->bhrb_entries[u_index].predicted = ~pred;
+ cpu_lbr_stack(cpuhw)->entries[u_index].mispred = pred;
+ cpu_lbr_stack(cpuhw)->entries[u_index].predicted = ~pred;
}
u_index++;
}
}
- cpuhw->bhrb_stack.nr = u_index;
- cpuhw->bhrb_stack.hw_idx = -1ULL;
+ cpu_lbr_stack(cpuhw)->nr = u_index;
+ cpu_lbr_stack(cpuhw)->hw_idx = -1ULL;
return;
}
@@ -2332,7 +2349,7 @@ static void record_and_restart(struct perf_event *event, unsigned long val,
struct cpu_hw_events *cpuhw;
cpuhw = this_cpu_ptr(&cpu_hw_events);
power_pmu_bhrb_read(event, cpuhw);
- perf_sample_save_brstack(&data, event, &cpuhw->bhrb_stack, NULL);
+ perf_sample_save_brstack(&data, event, cpu_lbr_stack(cpuhw), NULL);
}
if (event->attr.sample_type & PERF_SAMPLE_DATA_SRC &&
diff --git a/arch/x86/events/amd/brs.c b/arch/x86/events/amd/brs.c
index 54b13faba116..1906d1257646 100644
--- a/arch/x86/events/amd/brs.c
+++ b/arch/x86/events/amd/brs.c
@@ -282,7 +282,6 @@ void amd_brs_drain(void)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct perf_event *event = cpuc->events[0];
- struct perf_branch_entry *br = cpuc->lbr_entries;
union amd_debug_extn_cfg cfg;
u32 i, nr = 0, num, tos, start;
u32 shift = 64 - boot_cpu_data.x86_virt_bits;
@@ -317,6 +316,8 @@ void amd_brs_drain(void)
num = tos - start + 1;
+ cpu_init_lbr_stack(cpuc, num);
+
/*
* BRS is only one pass (saturation) from MSROFF to depth-1
* MSROFF wraps to zero when buffer is full
@@ -343,7 +344,7 @@ void amd_brs_drain(void)
if (!amd_brs_match_plm(event, from, to))
continue;
- br[nr] = (struct perf_branch_entry){
+ cpu_lbr_stack(cpuc)->entries[nr] = (struct perf_branch_entry){
.from = from,
.to = to,
};
@@ -351,7 +352,7 @@ void amd_brs_drain(void)
}
empty:
/* Record number of sampled branches */
- cpuc->lbr_stack.nr = nr;
+ cpu_lbr_stack(cpuc)->nr = nr;
}
/*
diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
index 851d8a938f52..160102a46b18 100644
--- a/arch/x86/events/amd/core.c
+++ b/arch/x86/events/amd/core.c
@@ -947,8 +947,8 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
__amd_pmu_lbr_disable();
amd_pmu_lbr_read();
- cnt = min(cnt, cpuc->lbr_stack.nr);
- memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
+ cnt = min(cnt, cpu_lbr_stack(cpuc)->nr);
+ memcpy(entries, cpu_lbr_stack(cpuc)->entries, sizeof(struct perf_branch_entry) * cnt);
if (lbr_enabled)
amd_pmu_lbr_enable_all();
@@ -1020,7 +1020,7 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)
if (!x86_perf_event_set_period(event))
continue;
- perf_sample_save_brstack(&data, event, &cpuc->lbr_stack, NULL);
+ perf_sample_save_brstack(&data, event, cpu_lbr_stack(cpuc), NULL);
perf_event_overflow(event, &data, regs);
}
diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c
index 400c94ecded0..3e4e5e984649 100644
--- a/arch/x86/events/amd/lbr.c
+++ b/arch/x86/events/amd/lbr.c
@@ -104,15 +104,18 @@ static void amd_pmu_lbr_filter(void)
bool compress = false;
bool fused_only = false;
u64 from, to;
+ int lbr_nr;
/* If sampling all branches, there is nothing to filter */
if (((br_sel & X86_BR_ALL) == X86_BR_ALL) &&
((br_sel & X86_BR_TYPE_SAVE) != X86_BR_TYPE_SAVE))
fused_only = true;
- for (i = 0; i < cpuc->lbr_stack.nr; i++) {
- from = cpuc->lbr_entries[i].from;
- to = cpuc->lbr_entries[i].to;
+ for (i = 0; i < cpu_lbr_stack(cpuc)->nr; i++) {
+ struct perf_branch_entry *be = &cpu_lbr_stack(cpuc)->entries[i];
+
+ from = be->from;
+ to = be->to;
type = branch_type_fused(from, to, 0, &offset);
/*
@@ -121,37 +124,41 @@ static void amd_pmu_lbr_filter(void)
* actual branch
*/
if (offset) {
- cpuc->lbr_entries[i].from += offset;
+ be->from += offset;
if (fused_only)
continue;
}
/* If type does not correspond, then discard */
if (type == X86_BR_NONE || (br_sel & type) != type ||
- (!(br_sel & X86_BR_KERNEL) && kernel_ip(cpuc->lbr_entries[i].from))) {
- cpuc->lbr_entries[i].from = 0; /* mark invalid */
+ (!(br_sel & X86_BR_KERNEL) && kernel_ip(be->from))) {
+ be->from = 0; /* mark invalid */
compress = true;
}
if ((br_sel & X86_BR_TYPE_SAVE) == X86_BR_TYPE_SAVE)
- cpuc->lbr_entries[i].type = common_branch_type(type);
+ be->type = common_branch_type(type);
}
if (!compress)
return;
/* Remove all invalid entries */
- for (i = 0; i < cpuc->lbr_stack.nr; ) {
- if (!cpuc->lbr_entries[i].from) {
+ lbr_nr = cpu_lbr_stack(cpuc)->nr;
+ for (i = 0; i < lbr_nr; ) {
+ if (!cpu_lbr_stack(cpuc)->entries[i].from) {
j = i;
- while (++j < cpuc->lbr_stack.nr)
- cpuc->lbr_entries[j - 1] = cpuc->lbr_entries[j];
- cpuc->lbr_stack.nr--;
- if (!cpuc->lbr_entries[i].from)
+ while (++j < lbr_nr) {
+ cpu_lbr_stack(cpuc)->entries[j - 1] =
+ cpu_lbr_stack(cpuc)->entries[j];
+ }
+ lbr_nr--;
+ if (!cpu_lbr_stack(cpuc)->entries[i].from)
continue;
}
i++;
}
+ cpu_lbr_stack(cpuc)->nr = lbr_nr;
}
static const int lbr_spec_map[PERF_BR_SPEC_MAX] = {
@@ -164,15 +171,16 @@ static const int lbr_spec_map[PERF_BR_SPEC_MAX] = {
void amd_pmu_lbr_read(void)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
- struct perf_branch_entry *br = cpuc->lbr_entries;
struct branch_entry entry;
int out = 0, idx, i;
if (!cpuc->lbr_users) {
- cpuc->lbr_stack.nr = 0;
+ cpu_lbr_stack(cpuc)->nr = 0;
return;
}
+ cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
+
for (i = 0; i < x86_pmu.lbr_nr; i++) {
entry.from.full = amd_pmu_lbr_get_from(i);
entry.to.full = amd_pmu_lbr_get_to(i);
@@ -204,7 +212,7 @@ void amd_pmu_lbr_read(void)
*/
idx = (entry.to.split.valid << 1) | entry.to.split.spec;
- br[out] = (struct perf_branch_entry){
+ cpu_lbr_stack(cpuc)->entries[out] = (struct perf_branch_entry){
.from = sign_ext_branch_ip(entry.from.split.ip),
.to = sign_ext_branch_ip(entry.to.split.ip),
.mispred = entry.from.split.mispredict,
@@ -214,13 +222,13 @@ void amd_pmu_lbr_read(void)
out++;
}
- cpuc->lbr_stack.nr = out;
+ cpu_lbr_stack(cpuc)->nr = out;
/*
* Internal register renaming always ensures that LBR From[0] and
* LBR To[0] always represent the TOS
*/
- cpuc->lbr_stack.hw_idx = 0;
+ cpu_lbr_stack(cpuc)->hw_idx = 0;
/* Perform further software filtering */
amd_pmu_lbr_filter();
diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index 82d9b0e5ad1c..a7bdc6f370ec 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -2197,7 +2197,7 @@ int x86_pmu_handle_irq(struct pt_regs *regs)
perf_sample_data_init(&data, 0, last_period);
- perf_sample_save_brstack(&data, event, &cpuc->lbr_stack, NULL);
+ perf_sample_save_brstack(&data, event, cpu_lbr_stack(cpuc), NULL);
perf_event_overflow(event, &data, regs);
}
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index ab6261ab54b6..42d4d525a729 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2827,9 +2827,10 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
intel_pmu_lbr_read();
- cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
+ cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr);
- memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
+ memcpy(entries, cpu_lbr_stack(cpuc)->entries,
+ sizeof(struct perf_branch_entry) * cnt);
/*
* Do not blindly call intel_pmu_enable_all() to re-enable LBR.
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 8444670cee4a..332754e1da4f 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -2010,8 +2010,7 @@ void intel_pmu_pebs_disable_all(void)
static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
- unsigned long from = cpuc->lbr_entries[0].from;
- unsigned long old_to, to = cpuc->lbr_entries[0].to;
+ unsigned long from, old_to, to;
unsigned long ip = regs->ip;
int is_64bit = 0;
void *kaddr;
@@ -2026,7 +2025,12 @@ static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)
/*
* No LBR entry, no basic block, no rewinding
*/
- if (!cpuc->lbr_stack.nr || !from || !to)
+ if (!cpu_lbr_stack(cpuc)->nr)
+ return 0;
+
+ from = cpu_lbr_stack(cpuc)->entries[0].from;
+ to = cpu_lbr_stack(cpuc)->entries[0].to;
+ if (!from || !to)
return 0;
/*
@@ -2318,7 +2322,7 @@ static void setup_pebs_fixed_sample_data(struct perf_event *event,
if (x86_pmu.intel_cap.pebs_format >= 3)
setup_pebs_time(event, data, pebs->tsc);
- perf_sample_save_brstack(data, event, &cpuc->lbr_stack, NULL);
+ perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
}
static void adaptive_pebs_save_regs(struct pt_regs *regs,
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index ee724b28d90c..62f2f4b9cfde 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -195,7 +195,7 @@ void intel_pmu_lbr_reset(void)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
- cpuc->lbr_stack.nr = 0;
+ cpu_lbr_stack(cpuc)->nr = 0;
if (!x86_pmu.lbr_nr)
return;
@@ -744,10 +744,10 @@ void intel_pmu_lbr_disable_all(void)
void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
{
unsigned long mask = x86_pmu.lbr_nr - 1;
- struct perf_branch_entry *br = cpuc->lbr_entries;
u64 tos = intel_pmu_lbr_tos();
int i;
+ cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
for (i = 0; i < x86_pmu.lbr_nr; i++) {
unsigned long lbr_idx = (tos - i) & mask;
union {
@@ -760,14 +760,13 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
rdmsrq(x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr);
- *br = (struct perf_branch_entry){
+ cpu_lbr_stack(cpuc)->entries[i] = (struct perf_branch_entry){
.from = msr_lastbranch.from,
.to = msr_lastbranch.to,
};
- br++;
}
- cpuc->lbr_stack.nr = i;
- cpuc->lbr_stack.hw_idx = tos;
+ cpu_lbr_stack(cpuc)->nr = i;
+ cpu_lbr_stack(cpuc)->hw_idx = tos;
}
/*
@@ -779,7 +778,6 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
{
bool need_info = false, call_stack = false;
unsigned long mask = x86_pmu.lbr_nr - 1;
- struct perf_branch_entry *br = cpuc->lbr_entries;
u64 tos = intel_pmu_lbr_tos();
int i;
int out = 0;
@@ -791,6 +789,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
call_stack = true;
}
+ cpu_init_lbr_stack(cpuc, num);
for (i = 0; i < num; i++) {
unsigned long lbr_idx = (tos - i) & mask;
u64 from, to, mis = 0, pred = 0, in_tx = 0, abort = 0;
@@ -851,7 +850,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
if (abort && x86_pmu.lbr_double_abort && out > 0)
out--;
- br[out] = (struct perf_branch_entry){
+ cpu_lbr_stack(cpuc)->entries[out] = (struct perf_branch_entry){
.from = from,
.to = to,
.mispred = mis,
@@ -862,8 +861,8 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
};
out++;
}
- cpuc->lbr_stack.nr = out;
- cpuc->lbr_stack.hw_idx = tos;
+ cpu_lbr_stack(cpuc)->nr = out;
+ cpu_lbr_stack(cpuc)->hw_idx = tos;
}
static DEFINE_STATIC_KEY_FALSE(x86_lbr_mispred);
@@ -907,15 +906,14 @@ static_assert((64 - PERF_BRANCH_ENTRY_INFO_BITS_MAX) > LBR_INFO_BR_CNTR_NUM * LB
static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
struct lbr_entry *entries)
{
- struct perf_branch_entry *e;
struct lbr_entry *lbr;
u64 from, to, info;
bool mispred;
int i;
+ cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
for (i = 0; i < x86_pmu.lbr_nr; i++) {
lbr = entries ? &entries[i] : NULL;
- e = &cpuc->lbr_entries[i];
from = rdlbr_from(i, lbr);
/*
@@ -929,7 +927,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
mispred = get_lbr_mispred(info);
- *e = (struct perf_branch_entry){
+ cpu_lbr_stack(cpuc)->entries[i] = (struct perf_branch_entry){
.from = from,
.to = to,
.mispred = mispred,
@@ -939,18 +937,18 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
.cycles = get_lbr_cycles(info),
.type = get_lbr_br_type(info),
/*
- * Leverage the reserved field of
- * cpuc->lbr_entries[i] to temporarily store the
- * branch counters information. The later code will
- * decide what content can be disclosed to the perf
- * tool. Pleae see intel_pmu_lbr_counters_reorder().
+ * Leverage the reserved field of LBR entries to
+ * temporarily store the branch counters information.
+ * The later code will decide what content can be
+ * disclosed to the perf tool.
+ * Pleae see intel_pmu_lbr_counters_reorder().
*/
.reserved = (info >> LBR_INFO_BR_CNTR_OFFSET) &
LBR_INFO_BR_CNTR_FULL_MASK,
};
}
- cpuc->lbr_stack.nr = i;
+ cpu_lbr_stack(cpuc)->nr = i;
}
/*
@@ -976,15 +974,15 @@ static void intel_pmu_lbr_counters_reorder(struct cpu_hw_events *cpuc,
WARN_ON_ONCE(!pos);
- for (i = 0; i < cpuc->lbr_stack.nr; i++) {
- src = cpuc->lbr_entries[i].reserved;
+ for (i = 0; i < cpu_lbr_stack(cpuc)->nr; i++) {
+ src = cpu_lbr_stack(cpuc)->entries[i].reserved;
dst = 0;
for (j = 0; j < pos; j++) {
cnt = (src >> (order[j] * LBR_INFO_BR_CNTR_BITS)) & LBR_INFO_BR_CNTR_MASK;
dst |= cnt << j * LBR_INFO_BR_CNTR_BITS;
}
cpuc->lbr_counters[i] = dst;
- cpuc->lbr_entries[i].reserved = 0;
+ cpu_lbr_stack(cpuc)->entries[i].reserved = 0;
}
}
@@ -994,11 +992,11 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
{
if (is_branch_counters_group(event)) {
intel_pmu_lbr_counters_reorder(cpuc, event);
- perf_sample_save_brstack(data, event, &cpuc->lbr_stack, cpuc->lbr_counters);
+ perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), cpuc->lbr_counters);
return;
}
- perf_sample_save_brstack(data, event, &cpuc->lbr_stack, NULL);
+ perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
}
static void intel_pmu_arch_lbr_read(struct cpu_hw_events *cpuc)
@@ -1021,7 +1019,7 @@ static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)
intel_pmu_store_lbr(cpuc, xsave->lbr.entries);
} else {
/* Clear the LBR stack to avoid leaking stale entries. */
- cpuc->lbr_stack.nr = 0;
+ cpu_lbr_stack(cpuc)->nr = 0;
}
}
@@ -1031,7 +1029,7 @@ void intel_pmu_lbr_read(void)
if (!cpuc->lbr_users || vlbr_exclude_host()) {
/* Reset lbr_stack.nr to prevent stale data leakage. */
- cpuc->lbr_stack.nr = 0;
+ cpu_lbr_stack(cpuc)->nr = 0;
return;
}
@@ -1231,21 +1229,23 @@ static const int arch_lbr_br_type_map[ARCH_LBR_BR_TYPE_MAP_MAX] = {
static void
intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
{
- u64 from, to;
int br_sel = cpuc->br_sel;
int i, j, type, from_plm, to_plm;
bool compress = false;
+ u64 from, to;
+ int lbr_nr;
/* if sampling all branches, then nothing to filter */
if (((br_sel & X86_BR_ALL) == X86_BR_ALL) &&
((br_sel & X86_BR_TYPE_SAVE) != X86_BR_TYPE_SAVE))
return;
- for (i = 0; i < cpuc->lbr_stack.nr; i++) {
+ for (i = 0; i < cpu_lbr_stack(cpuc)->nr; i++) {
+ struct perf_branch_entry *be = &cpu_lbr_stack(cpuc)->entries[i];
- from = cpuc->lbr_entries[i].from;
- to = cpuc->lbr_entries[i].to;
- type = cpuc->lbr_entries[i].type;
+ from = be->from;
+ to = be->to;
+ type = be->type;
/*
* Parse the branch type recorded in LBR_x_INFO MSR.
@@ -1257,9 +1257,9 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
to_plm = kernel_ip(to) ? X86_BR_KERNEL : X86_BR_USER;
type = arch_lbr_br_type_map[type] | to_plm;
} else
- type = branch_type(from, to, cpuc->lbr_entries[i].abort);
+ type = branch_type(from, to, be->abort);
if (type != X86_BR_NONE && (br_sel & X86_BR_ANYTX)) {
- if (cpuc->lbr_entries[i].in_tx)
+ if (be->in_tx)
type |= X86_BR_IN_TX;
else
type |= X86_BR_NO_TX;
@@ -1273,31 +1273,34 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
*/
if (type == X86_BR_NONE || (br_sel & type) != type ||
(!(br_sel & X86_BR_KERNEL) && (from_plm & X86_BR_KERNEL))) {
- cpuc->lbr_entries[i].from = 0;
+ be->from = 0;
compress = true;
}
if ((br_sel & X86_BR_TYPE_SAVE) == X86_BR_TYPE_SAVE)
- cpuc->lbr_entries[i].type = common_branch_type(type);
+ be->type = common_branch_type(type);
}
if (!compress)
return;
/* remove all entries with from=0 */
- for (i = 0; i < cpuc->lbr_stack.nr; ) {
- if (!cpuc->lbr_entries[i].from) {
+ lbr_nr = cpu_lbr_stack(cpuc)->nr;
+ for (i = 0; i < lbr_nr; ) {
+ if (!cpu_lbr_stack(cpuc)->entries[i].from) {
j = i;
- while (++j < cpuc->lbr_stack.nr) {
- cpuc->lbr_entries[j-1] = cpuc->lbr_entries[j];
+ while (++j < lbr_nr) {
+ cpu_lbr_stack(cpuc)->entries[j-1] =
+ cpu_lbr_stack(cpuc)->entries[j];
cpuc->lbr_counters[j-1] = cpuc->lbr_counters[j];
}
- cpuc->lbr_stack.nr--;
- if (!cpuc->lbr_entries[i].from)
+ lbr_nr--;
+ if (!cpu_lbr_stack(cpuc)->entries[i].from)
continue;
}
i++;
}
+ cpu_lbr_stack(cpuc)->nr = lbr_nr;
}
void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
@@ -1307,9 +1310,9 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
/* Cannot get TOS for large PEBS and Arch LBR */
if (cpu_feature_enabled(X86_FEATURE_ARCH_LBR) ||
(cpuc->n_pebs == cpuc->n_large_pebs))
- cpuc->lbr_stack.hw_idx = -1ULL;
+ cpu_lbr_stack(cpuc)->hw_idx = -1ULL;
else
- cpuc->lbr_stack.hw_idx = intel_pmu_lbr_tos();
+ cpu_lbr_stack(cpuc)->hw_idx = intel_pmu_lbr_tos();
intel_pmu_store_lbr(cpuc, lbr);
intel_pmu_lbr_filter(cpuc);
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index a4aab8130e92..e3d625a6d7eb 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -447,6 +447,8 @@ struct intel_excl_cntrs {
struct x86_perf_task_context;
#define MAX_LBR_ENTRIES 32
+#define LBR_STACK_SIZE struct_size_t(struct perf_branch_stack, entries, MAX_LBR_ENTRIES)
+
enum {
LBR_FORMAT_32 = 0x00,
LBR_FORMAT_LIP = 0x01,
@@ -525,8 +527,7 @@ struct cpu_hw_events {
*/
int lbr_users;
int lbr_pebs_users;
- struct perf_branch_stack lbr_stack;
- struct perf_branch_entry lbr_entries[MAX_LBR_ENTRIES];
+ u64 __lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
u64 lbr_counters[MAX_LBR_ENTRIES]; /* branch stack extra */
union {
struct er_account *lbr_sel;
@@ -588,6 +589,20 @@ struct cpu_hw_events {
struct pmu *pmu;
};
+static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_events *cpuc)
+{
+ return (struct perf_branch_stack *)cpuc->__lbr_stack;
+}
+
+static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr)
+{
+ /*
+ * Initialize the branch-stack count before populating entries[]
+ * to avoid UBSAN complaints from the __counted_by(nr) flexible array.
+ */
+ cpu_lbr_stack(cpuc)->nr = nr;
+}
+
#define __EVENT_CONSTRAINT_RANGE(c, e, n, m, w, o, f) { \
{ .idxmsk64 = (n) }, \
.code = (c), \
diff --git a/drivers/perf/arm_brbe.c b/drivers/perf/arm_brbe.c
index 073d14584741..e2b32b239de1 100644
--- a/drivers/perf/arm_brbe.c
+++ b/drivers/perf/arm_brbe.c
@@ -763,6 +763,11 @@ void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack,
prepare_event_branch_type_mask(branch_sample_type, event_type_mask);
+ /*
+ * Initialize the branch_stack->nr before populating entries[]
+ * to avoid UBSAN complaints from the __counted_by(nr) flexible array.
+ */
+ branch_stack->nr = nr_hw;
for (int bank = 0; bank < nr_banks; bank++) {
int nr_remaining = nr_hw - (bank * BRBE_BANK_MAX_ENTRIES);
int nr_this_bank = min(nr_remaining, BRBE_BANK_MAX_ENTRIES);
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index eb3f7bab498c..19de3f1cbd6e 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -115,7 +115,7 @@ static __always_inline bool perf_raw_frag_last(const struct perf_raw_frag *frag)
struct perf_branch_stack {
u64 nr;
u64 hw_idx;
- struct perf_branch_entry entries[];
+ struct perf_branch_entry entries[] __counted_by(nr);
};
struct task_struct;
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* Re: [Patch v6 10/20] perf: Add counted-bounds tracking for perf_branch_stack
2026-11-06 4:32 ` [Patch v6 10/20] perf: Add counted-bounds tracking for perf_branch_stack Dapeng Mi
@ 2026-11-06 5:00 ` sashiko-bot
0 siblings, 0 replies; 45+ messages in thread
From: sashiko-bot @ 2026-11-06 5:00 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users
> struct perf_branch_stack uses a trailing flexible array member,
> entries[], whose valid length is tracked by nr. When this structure
> is embedded as a value member in cpu_hw_events, the compiler can treat
> the trailing flexible array as having a fixed zero-length layout under
> -fstrict-flex-arrays=3 and UBSan, which can lead to strict array-bounds
> diagnostics when entries[] is indexed.
>
> Annotate entries[] with __counted_by(nr) so the compiler and sanitizer
> track the array length from the nr field. Because a structure containing
> a counted flexible array cannot be embedded as a value member, switch
> the per-CPU LBR stack backing storage to a complete __lbr_stack[] array
> and add a helper cpu_lbr_stack() to recover the struct perf_branch_stack*
> view for x86 and powerpc.
>
> Also initialize the branch-stack count before populating entries[] to
> avoid false-positive bounds checks while the stack is being filled.
>
> No functional change intended.
>
> Suggested-by: Peter Zijlstra <peterz@infradead.org>
> Cc: Sandipan Das <sandipan.das@amd.com>
> Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Nicholas Piggin <npiggin@gmail.com>
> Cc: "Christophe Leroy (CS GROUP)" <chleroy@kernel.org>
> Cc: Will Deacon <will@kernel.org>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: linux-arm-kernel@lists.infradead.org
> Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261106043257.842864-1-dapeng1.mi@linux.intel.com?part=10
^ permalink raw reply [flat|nested] 45+ messages in thread
* [Patch v6 11/20] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (9 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 10/20] perf: Add counted-bounds tracking for perf_branch_stack Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:09 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 12/20] perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting Dapeng Mi
` (9 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
The LBR read helpers currently assume the default stack,
cpuc->lbr_stack, and therefore cannot be reused for other LBR
stacks that may be in use. Extend the *_lbr_read() helpers to take
an explicit LBR stack argument, so they can operate on any LBR stack
instead of only the default one.
Update the related callers, including intel_pmu_store_lbr() and
intel_pmu_lbr_filter(), to pass the appropriate stack explicitly.
No function change intended.
Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/amd/brs.c | 2 +-
arch/x86/events/amd/lbr.c | 2 +-
arch/x86/events/intel/core.c | 4 +-
arch/x86/events/intel/lbr.c | 89 ++++++++++++++++++++----------------
arch/x86/events/perf_event.h | 16 ++++---
5 files changed, 64 insertions(+), 49 deletions(-)
diff --git a/arch/x86/events/amd/brs.c b/arch/x86/events/amd/brs.c
index 1906d1257646..475e3699d3ca 100644
--- a/arch/x86/events/amd/brs.c
+++ b/arch/x86/events/amd/brs.c
@@ -316,7 +316,7 @@ void amd_brs_drain(void)
num = tos - start + 1;
- cpu_init_lbr_stack(cpuc, num);
+ cpu_init_lbr_stack(cpu_lbr_stack(cpuc), num);
/*
* BRS is only one pass (saturation) from MSROFF to depth-1
diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c
index 3e4e5e984649..95898a1d9901 100644
--- a/arch/x86/events/amd/lbr.c
+++ b/arch/x86/events/amd/lbr.c
@@ -179,7 +179,7 @@ void amd_pmu_lbr_read(void)
return;
}
- cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
+ cpu_init_lbr_stack(cpu_lbr_stack(cpuc), x86_pmu.lbr_nr);
for (i = 0; i < x86_pmu.lbr_nr; i++) {
entry.from.full = amd_pmu_lbr_get_from(i);
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 42d4d525a729..b7045c05b736 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2826,7 +2826,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
- intel_pmu_lbr_read();
+ intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr);
memcpy(entries, cpu_lbr_stack(cpuc)->entries,
@@ -4062,7 +4062,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
loops = 0;
again:
- intel_pmu_lbr_read();
+ intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
/*
* Do not clear the LBR frozen status here; keep LBR frozen
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 62f2f4b9cfde..979b094ae228 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -105,7 +105,9 @@
#define ARCH_LBR_CTL_MASK 0x7f000e
-static void intel_pmu_lbr_filter(struct cpu_hw_events *cpuc);
+static void intel_pmu_lbr_filter(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack,
+ u64 *lbr_counters);
static __always_inline bool is_lbr_call_stack_bit_set(u64 config)
{
@@ -741,13 +743,14 @@ void intel_pmu_lbr_disable_all(void)
}
}
-void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
+void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack)
{
unsigned long mask = x86_pmu.lbr_nr - 1;
u64 tos = intel_pmu_lbr_tos();
int i;
- cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
+ cpu_init_lbr_stack(lbr_stack, x86_pmu.lbr_nr);
for (i = 0; i < x86_pmu.lbr_nr; i++) {
unsigned long lbr_idx = (tos - i) & mask;
union {
@@ -760,13 +763,13 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
rdmsrq(x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr);
- cpu_lbr_stack(cpuc)->entries[i] = (struct perf_branch_entry){
+ lbr_stack->entries[i] = (struct perf_branch_entry){
.from = msr_lastbranch.from,
.to = msr_lastbranch.to,
};
}
- cpu_lbr_stack(cpuc)->nr = i;
- cpu_lbr_stack(cpuc)->hw_idx = tos;
+ lbr_stack->nr = i;
+ lbr_stack->hw_idx = tos;
}
/*
@@ -774,7 +777,8 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
* is the same as the linear address, allowing us to merge the LIP and EIP
* LBR formats.
*/
-void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
+void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack)
{
bool need_info = false, call_stack = false;
unsigned long mask = x86_pmu.lbr_nr - 1;
@@ -789,7 +793,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
call_stack = true;
}
- cpu_init_lbr_stack(cpuc, num);
+ cpu_init_lbr_stack(lbr_stack, num);
for (i = 0; i < num; i++) {
unsigned long lbr_idx = (tos - i) & mask;
u64 from, to, mis = 0, pred = 0, in_tx = 0, abort = 0;
@@ -850,7 +854,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
if (abort && x86_pmu.lbr_double_abort && out > 0)
out--;
- cpu_lbr_stack(cpuc)->entries[out] = (struct perf_branch_entry){
+ lbr_stack->entries[out] = (struct perf_branch_entry){
.from = from,
.to = to,
.mispred = mis,
@@ -861,8 +865,8 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
};
out++;
}
- cpu_lbr_stack(cpuc)->nr = out;
- cpu_lbr_stack(cpuc)->hw_idx = tos;
+ lbr_stack->nr = out;
+ lbr_stack->hw_idx = tos;
}
static DEFINE_STATIC_KEY_FALSE(x86_lbr_mispred);
@@ -903,17 +907,19 @@ static __always_inline u16 get_lbr_cycles(u64 info)
static_assert((64 - PERF_BRANCH_ENTRY_INFO_BITS_MAX) > LBR_INFO_BR_CNTR_NUM * LBR_INFO_BR_CNTR_BITS);
-static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
- struct lbr_entry *entries)
+static void intel_pmu_store_lbr(struct lbr_entry *entries,
+ struct perf_branch_stack *lbr_stack)
{
+ struct perf_branch_entry *e;
struct lbr_entry *lbr;
u64 from, to, info;
bool mispred;
int i;
- cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
+ cpu_init_lbr_stack(lbr_stack, x86_pmu.lbr_nr);
for (i = 0; i < x86_pmu.lbr_nr; i++) {
lbr = entries ? &entries[i] : NULL;
+ e = &lbr_stack->entries[i];
from = rdlbr_from(i, lbr);
/*
@@ -927,7 +933,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
mispred = get_lbr_mispred(info);
- cpu_lbr_stack(cpuc)->entries[i] = (struct perf_branch_entry){
+ *e = (struct perf_branch_entry){
.from = from,
.to = to,
.mispred = mispred,
@@ -948,7 +954,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
};
}
- cpu_lbr_stack(cpuc)->nr = i;
+ lbr_stack->nr = i;
}
/*
@@ -999,37 +1005,41 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
}
-static void intel_pmu_arch_lbr_read(struct cpu_hw_events *cpuc)
+static void intel_pmu_arch_lbr_read(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack)
{
- intel_pmu_store_lbr(cpuc, NULL);
+ intel_pmu_store_lbr(NULL, lbr_stack);
}
-static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)
+static void
+intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack)
{
struct x86_perf_task_context_arch_lbr_xsave *xsave = cpuc->lbr_xsave;
if (!xsave) {
- intel_pmu_store_lbr(cpuc, NULL);
+ intel_pmu_store_lbr(NULL, lbr_stack);
return;
}
xsave->header.xfeatures = 0;
xsaves(&xsave->xsave, XFEATURE_MASK_LBR);
if (xsave->header.xfeatures & XFEATURE_MASK_LBR) {
- intel_pmu_store_lbr(cpuc, xsave->lbr.entries);
+ intel_pmu_store_lbr(xsave->lbr.entries, lbr_stack);
} else {
/* Clear the LBR stack to avoid leaking stale entries. */
- cpu_lbr_stack(cpuc)->nr = 0;
+ lbr_stack->nr = 0;
}
}
-void intel_pmu_lbr_read(void)
+void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,
+ u64 *lbr_counters)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!cpuc->lbr_users || vlbr_exclude_host()) {
- /* Reset lbr_stack.nr to prevent stale data leakage. */
- cpu_lbr_stack(cpuc)->nr = 0;
+ /* Reset lbr_stack->nr to prevent stale data leakage. */
+ lbr_stack->nr = 0;
return;
}
@@ -1042,9 +1052,9 @@ void intel_pmu_lbr_read(void)
if (cpuc->lbr_users == cpuc->lbr_pebs_users)
return;
- x86_pmu.lbr_read(cpuc);
+ x86_pmu.lbr_read(cpuc, lbr_stack);
- intel_pmu_lbr_filter(cpuc);
+ intel_pmu_lbr_filter(cpuc, lbr_stack, lbr_counters);
}
/*
@@ -1227,7 +1237,9 @@ static const int arch_lbr_br_type_map[ARCH_LBR_BR_TYPE_MAP_MAX] = {
* in PERF_SAMPLE_BRANCH_STACK sample may vary.
*/
static void
-intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
+intel_pmu_lbr_filter(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack,
+ u64 *lbr_counters)
{
int br_sel = cpuc->br_sel;
int i, j, type, from_plm, to_plm;
@@ -1240,8 +1252,8 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
((br_sel & X86_BR_TYPE_SAVE) != X86_BR_TYPE_SAVE))
return;
- for (i = 0; i < cpu_lbr_stack(cpuc)->nr; i++) {
- struct perf_branch_entry *be = &cpu_lbr_stack(cpuc)->entries[i];
+ for (i = 0; i < lbr_stack->nr; i++) {
+ struct perf_branch_entry *be = &lbr_stack->entries[i];
from = be->from;
to = be->to;
@@ -1285,22 +1297,21 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
return;
/* remove all entries with from=0 */
- lbr_nr = cpu_lbr_stack(cpuc)->nr;
+ lbr_nr = lbr_stack->nr;
for (i = 0; i < lbr_nr; ) {
- if (!cpu_lbr_stack(cpuc)->entries[i].from) {
+ if (!lbr_stack->entries[i].from) {
j = i;
while (++j < lbr_nr) {
- cpu_lbr_stack(cpuc)->entries[j-1] =
- cpu_lbr_stack(cpuc)->entries[j];
- cpuc->lbr_counters[j-1] = cpuc->lbr_counters[j];
+ lbr_stack->entries[j-1] = lbr_stack->entries[j];
+ lbr_counters[j-1] = lbr_counters[j];
}
lbr_nr--;
- if (!cpu_lbr_stack(cpuc)->entries[i].from)
+ if (!lbr_stack->entries[i].from)
continue;
}
i++;
}
- cpu_lbr_stack(cpuc)->nr = lbr_nr;
+ lbr_stack->nr = lbr_nr;
}
void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
@@ -1314,8 +1325,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
else
cpu_lbr_stack(cpuc)->hw_idx = intel_pmu_lbr_tos();
- intel_pmu_store_lbr(cpuc, lbr);
- intel_pmu_lbr_filter(cpuc);
+ intel_pmu_store_lbr(lbr, cpu_lbr_stack(cpuc));
+ intel_pmu_lbr_filter(cpuc, cpu_lbr_stack(cpuc), cpuc->lbr_counters);
}
/*
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index e3d625a6d7eb..fa0af528cd1e 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -594,13 +594,13 @@ static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_eve
return (struct perf_branch_stack *)cpuc->__lbr_stack;
}
-static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr)
+static __always_inline void cpu_init_lbr_stack(struct perf_branch_stack *lbr_stack, int nr)
{
/*
* Initialize the branch-stack count before populating entries[]
* to avoid UBSAN complaints from the __counted_by(nr) flexible array.
*/
- cpu_lbr_stack(cpuc)->nr = nr;
+ lbr_stack->nr = nr;
}
#define __EVENT_CONSTRAINT_RANGE(c, e, n, m, w, o, f) { \
@@ -1207,7 +1207,8 @@ struct x86_pmu {
unsigned int lbr_counters:4;
void (*lbr_reset)(void);
- void (*lbr_read)(struct cpu_hw_events *cpuc);
+ void (*lbr_read)(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack);
void (*lbr_save)(void *ctx);
void (*lbr_restore)(void *ctx);
@@ -2007,11 +2008,14 @@ void intel_pmu_lbr_enable_all(bool pmi);
void intel_pmu_lbr_disable_all(void);
-void intel_pmu_lbr_read(void);
+void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,
+ u64 *lbr_counters);
-void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc);
+void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack);
-void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc);
+void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack);
void intel_pmu_lbr_save(void *ctx);
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [Patch v6 12/20] perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (10 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 11/20] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:17 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 13/20] perf/x86/intel: Always snapshot live hardware LBR state for BPF Dapeng Mi
` (8 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
BPF LBR snapshotting currently reuses the default LBR stack
(cpuc->__lbr_stack) when storing sampled data. This is unsafe when the
BPF helper runs in PMI context via bpf_overflow_handler(): it can
overwrite the cached LBR state for active PEBS LBR events, causing
incorrect LBR data to be reported to userspace. When the PEBS and BPF
stacks have different depths, this can also lead to out-of-bounds reads
when the LBR stack is later exported.
Use a dedicated LBR stack for BPF snapshotting so the BPF path no
longer clobbers the shared LBR state used by other perf events.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 16 ++++++++++++----
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index b7045c05b736..a3aaa98108cd 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2819,17 +2819,25 @@ static void intel_pmu_enable_all(int added)
__intel_pmu_enable_all(false);
}
+struct lbr_snapshot {
+ u64 __lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
+ u64 counters[MAX_LBR_ENTRIES];
+};
+static DEFINE_PER_CPU(struct lbr_snapshot, lbr_snapshot);
+
static noinline int
__intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
unsigned int cnt, unsigned long flags,
int pmu_enabled, int lbr_enabled)
{
- struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+ struct lbr_snapshot *bpf = this_cpu_ptr(&lbr_snapshot);
+ struct perf_branch_stack *lbr_stack =
+ (struct perf_branch_stack *)bpf->__lbr_stack;
- intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
- cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr);
+ intel_pmu_lbr_read(lbr_stack, bpf->counters);
+ cnt = min_t(unsigned int, cnt, lbr_stack->nr);
- memcpy(entries, cpu_lbr_stack(cpuc)->entries,
+ memcpy(entries, lbr_stack->entries,
sizeof(struct perf_branch_entry) * cnt);
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [Patch v6 13/20] perf/x86/intel: Always snapshot live hardware LBR state for BPF
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (11 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 12/20] perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:13 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 14/20] perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active Dapeng Mi
` (7 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
__intel_pmu_snapshot_branch_stack() is intended to capture the current
live hardware LBR state for the BPF path. However, that snapshot is
currently skipped when all LBR users are PEBS events.
BPF LBR snapshotting now uses its own dedicated LBR stack, so there is
no longer a risk of clobbering the shared default LBR stack. As a
result, always snapshot the live hardware LBR state for the BPF path.
Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 7 ++++++-
arch/x86/events/intel/lbr.c | 9 ---------
2 files changed, 6 insertions(+), 10 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index a3aaa98108cd..fbde874af618 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -4070,7 +4070,12 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
loops = 0;
again:
- intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
+ /*
+ * No need to read the LBR stack when all LBR users are PEBS events,
+ * since the PEBS handler updates it.
+ */
+ if (cpuc->lbr_users != cpuc->lbr_pebs_users)
+ intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
/*
* Do not clear the LBR frozen status here; keep LBR frozen
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 979b094ae228..27e4be839926 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -1043,15 +1043,6 @@ void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,
return;
}
- /*
- * Don't read when all LBRs users are using adaptive PEBS.
- *
- * This could be smarter and actually check the event,
- * but this simple approach seems to work for now.
- */
- if (cpuc->lbr_users == cpuc->lbr_pebs_users)
- return;
-
x86_pmu.lbr_read(cpuc, lbr_stack);
intel_pmu_lbr_filter(cpuc, lbr_stack, lbr_counters);
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [Patch v6 14/20] perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (12 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 13/20] perf/x86/intel: Always snapshot live hardware LBR state for BPF Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:09 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 15/20] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed Dapeng Mi
` (6 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
Refreshing the LBR stack in the PMI handler is unnecessary when no LBR
users are active. This adds avoidable overhead and hurts performance.
Limit the LBR stack refresh to the PMI path only when LBR users are
present.
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Cc: "Sandipan Das <sandipan.das@amd.com>"
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/amd/core.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
index 160102a46b18..f0425e369837 100644
--- a/arch/x86/events/amd/core.c
+++ b/arch/x86/events/amd/core.c
@@ -988,7 +988,8 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)
/* Read branch records */
if (x86_pmu.lbr_nr) {
- amd_pmu_lbr_read();
+ if (cpuc->lbr_users > 0)
+ amd_pmu_lbr_read();
status &= ~GLOBAL_STATUS_LBRS_FROZEN;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [Patch v6 15/20] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (13 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 14/20] perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:58 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 16/20] perf/x86/intel: Remove dead lbr_pebs_users Dapeng Mi
` (5 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
PEBS and non-PEBS events share the same LBR stack in cpuc->__lbr_stack.
In the PMI handler, the non-PEBS path refreshes the LBR state before the
PEBS buffer is drained.
When PEBS and non-PEBS LBR events overflow in the same PMI, draining the
PEBS buffer can overwrite the shared LBR stack with the PEBS snapshot.
This leaves stale LBR data from the PEBS record in place, so userspace
can see the wrong branch stack instead of the live LBR state at PMI
entry.
Track whether the LBR state has already been refreshed with a
lbr_refreshed flag. If PEBS events overwrite the shared LBR stack before
the non-PEBS LBR event is processed, clear the flag so the live LBR state
is refreshed again. Remove the unconditional LBR refresh at PMI entry and
initialize the flag as false so the stack is refreshed only when an active
non-PEBS LBR event requires it. Since the LBR state is frozen while the
PMI is being handled, it only needs to be read once for multiple non-PEBS
LBR events in the same interrupt.
Fixes: c22497f5838c ("perf/x86/intel: Support adaptive PEBS v4")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 40 +++++++++++++++++++++++++++---------
1 file changed, 30 insertions(+), 10 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index fbde874af618..b267c68f9467 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -3864,8 +3864,9 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
{
struct perf_sample_data data;
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
- int bit;
+ bool lbr_refreshed = false;
int handled = 0;
+ int bit;
inc_perf_irq_stat();
@@ -3906,6 +3907,15 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
if (__test_and_clear_bit(GLOBAL_STATUS_BUFFER_OVF_BIT, (unsigned long *)&status)) {
u64 pebs_enabled = cpuc->pebs_enabled;
+ /*
+ * Prior to adaptive PEBS, PEBS does not support LBR snapshotting.
+ * Refresh the LBR stack explicitly when LBR users are present.
+ */
+ if (cpuc->lbr_users > 0 && !x86_pmu.intel_cap.pebs_baseline) {
+ intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
+ lbr_refreshed = true;
+ }
+
handled += x86_pmu_handle_guest_pebs(regs, &data);
handled += static_call(x86_pmu_drain_pebs)(regs, &data);
/* Ensure no "suspicious NMI" warning for empty PEBS buffer. */
@@ -4003,8 +4013,19 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
*
* The PEBS buffer has to be drained before handling the A-PMI
*/
- if (is_pebs_counter_event_group(event))
+ if (is_pebs_counter_event_group(event)) {
+ /*
+ * PEBS counter sampling is only supported on adaptive PEBS
+ * and arch-PEBS, both of which snapshot the LBR stack.
+ * No explicit LBR refresh is needed here.
+ */
static_call(x86_pmu_drain_pebs)(regs, &data);
+ /*
+ * Draining the PEBS buffer can overwrite the shared
+ * LBR stack, so mark lbr_refreshed as false.
+ */
+ lbr_refreshed = false;
+ }
last_period = event->hw.last_period;
@@ -4013,8 +4034,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
perf_sample_data_init(&data, 0, last_period);
- if (has_branch_stack(event))
+ if (has_branch_stack(event)) {
+ if (!lbr_refreshed) {
+ intel_pmu_lbr_read(cpu_lbr_stack(cpuc),
+ cpuc->lbr_counters);
+ lbr_refreshed = true;
+ }
intel_pmu_lbr_save_brstack(&data, cpuc, event);
+ }
x86_pmu_clear_perf_regs(regs);
x86_pmu_update_perf_regs(event, &data, regs, false);
@@ -4070,13 +4097,6 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
loops = 0;
again:
- /*
- * No need to read the LBR stack when all LBR users are PEBS events,
- * since the PEBS handler updates it.
- */
- if (cpuc->lbr_users != cpuc->lbr_pebs_users)
- intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
-
/*
* Do not clear the LBR frozen status here; keep LBR frozen
* throughout PMI handling.
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* Re: [Patch v6 15/20] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
2026-11-06 4:32 ` [Patch v6 15/20] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed Dapeng Mi
@ 2026-11-06 5:58 ` sashiko-bot
2026-11-06 6:54 ` Mi, Dapeng
0 siblings, 1 reply; 45+ messages in thread
From: sashiko-bot @ 2026-11-06 5:58 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users, bpf
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] perf/x86/intel: unrefreshed or clobbered LBR stack on
indirect PEBS drain
--
commit f13e61130af8b47007c91f048b386359760f05ad
Author: Dapeng Mi <dapeng1.mi@linux.intel.com>
perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
This commit tracks whether the shared LBR stack has been refreshed during PMI
handling with a lbr_refreshed flag and removes the unconditional LBR read at
PMI entry so the live LBR state is read on demand for non-PEBS LBR events and
re-read if PEBS draining overwrites it.
> diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
> index fbde874af6182..b267c68f94676 100644
> --- a/arch/x86/events/intel/core.c
> +++ b/arch/x86/events/intel/core.c
[ ... ]
> @@ -4003,8 +4013,19 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
> *
> * The PEBS buffer has to be drained before handling the A-PMI
> */
> - if (is_pebs_counter_event_group(event))
> + if (is_pebs_counter_event_group(event)) {
> + /*
> + * PEBS counter sampling is only supported on adaptive PEBS
> + * and arch-PEBS, both of which snapshot the LBR stack.
> + * No explicit LBR refresh is needed here.
> + */
> static_call(x86_pmu_drain_pebs)(regs, &data);
> + /*
> + * Draining the PEBS buffer can overwrite the shared
> + * LBR stack, so mark lbr_refreshed as false.
> + */
> + lbr_refreshed = false;
> + }
>
> last_period = event->hw.last_period;
>
> @@ -4013,8 +4034,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
>
> perf_sample_data_init(&data, 0, last_period);
>
> - if (has_branch_stack(event))
> + if (has_branch_stack(event)) {
> + if (!lbr_refreshed) {
> + intel_pmu_lbr_read(cpu_lbr_stack(cpuc),
> + cpuc->lbr_counters);
> + lbr_refreshed = true;
> + }
> intel_pmu_lbr_save_brstack(&data, cpuc, event);
> + }
>
> x86_pmu_clear_perf_regs(regs);
> x86_pmu_update_perf_regs(event, &data, regs, false);
[Severity: Medium]
Can perf_event_overflow() still drain the PEBS buffer and clobber
cpu_lbr_stack(cpuc) when is_pebs_counter_event_group(event) is false?
When a non-PEBS event using PERF_SAMPLE_READ | PERF_FORMAT_GROUP is grouped
with a PERF_X86_EVENT_AUTO_RELOAD PEBS LBR sibling (without
PERF_X86_EVENT_PEBS_CNTR set on the leader), perf_event_overflow() ->
perf_output_sample() -> perf_output_read_group() -> intel_pmu_read_event()
calls intel_pmu_drain_pebs_buffer() before PERF_SAMPLE_BRANCH_STACK is
written. Draining pending PEBS records calls intel_pmu_store_pebs_lbrs(),
which overwrites cpu_lbr_stack(cpuc) before the current non-PEBS sample
copies data.br_stack and leaves lbr_refreshed set to true for any subsequent
non-PEBS LBR events in the same PMI.
In addition, because the unconditional intel_pmu_lbr_read() at PMI entry is
removed, an indirect PEBS drain on pre-adaptive PEBS
(!x86_pmu.intel_cap.pebs_baseline) when GLOBAL_STATUS_BUFFER_OVF_BIT is clear
will run intel_pmu_pebs_fixup_ip() against an unrefreshed
cpu_lbr_stack(cpuc).
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261106043257.842864-1-dapeng1.mi@linux.intel.com?part=15
^ permalink raw reply [flat|nested] 45+ messages in thread* Re: [Patch v6 15/20] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
2026-11-06 5:58 ` sashiko-bot
@ 2026-11-06 6:54 ` Mi, Dapeng
0 siblings, 0 replies; 45+ messages in thread
From: Mi, Dapeng @ 2026-11-06 6:54 UTC (permalink / raw)
To: sashiko-reviews; +Cc: linux-perf-users, bpf
On 11/6/2026 1:58 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential
> issue to consider:
> - [Medium] perf/x86/intel: unrefreshed or clobbered LBR stack on
> indirect PEBS drain
> --
>
> commit f13e61130af8b47007c91f048b386359760f05ad
> Author: Dapeng Mi <dapeng1.mi@linux.intel.com>
>
> perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
>
> This commit tracks whether the shared LBR stack has been refreshed during PMI
> handling with a lbr_refreshed flag and removes the unconditional LBR read at
> PMI entry so the live LBR state is read on demand for non-PEBS LBR events and
> re-read if PEBS draining overwrites it.
>
>> diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
>> index fbde874af6182..b267c68f94676 100644
>> --- a/arch/x86/events/intel/core.c
>> +++ b/arch/x86/events/intel/core.c
> [ ... ]
>
>> @@ -4003,8 +4013,19 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
>> *
>> * The PEBS buffer has to be drained before handling the A-PMI
>> */
>> - if (is_pebs_counter_event_group(event))
>> + if (is_pebs_counter_event_group(event)) {
>> + /*
>> + * PEBS counter sampling is only supported on adaptive PEBS
>> + * and arch-PEBS, both of which snapshot the LBR stack.
>> + * No explicit LBR refresh is needed here.
>> + */
>> static_call(x86_pmu_drain_pebs)(regs, &data);
>> + /*
>> + * Draining the PEBS buffer can overwrite the shared
>> + * LBR stack, so mark lbr_refreshed as false.
>> + */
>> + lbr_refreshed = false;
>> + }
>>
>> last_period = event->hw.last_period;
>>
>> @@ -4013,8 +4034,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
>>
>> perf_sample_data_init(&data, 0, last_period);
>>
>> - if (has_branch_stack(event))
>> + if (has_branch_stack(event)) {
>> + if (!lbr_refreshed) {
>> + intel_pmu_lbr_read(cpu_lbr_stack(cpuc),
>> + cpuc->lbr_counters);
>> + lbr_refreshed = true;
>> + }
>> intel_pmu_lbr_save_brstack(&data, cpuc, event);
>> + }
>>
>> x86_pmu_clear_perf_regs(regs);
>> x86_pmu_update_perf_regs(event, &data, regs, false);
> [Severity: Medium]
> Can perf_event_overflow() still drain the PEBS buffer and clobber
> cpu_lbr_stack(cpuc) when is_pebs_counter_event_group(event) is false?
>
> When a non-PEBS event using PERF_SAMPLE_READ | PERF_FORMAT_GROUP is grouped
> with a PERF_X86_EVENT_AUTO_RELOAD PEBS LBR sibling (without
> PERF_X86_EVENT_PEBS_CNTR set on the leader), perf_event_overflow() ->
> perf_output_sample() -> perf_output_read_group() -> intel_pmu_read_event()
> calls intel_pmu_drain_pebs_buffer() before PERF_SAMPLE_BRANCH_STACK is
> written. Draining pending PEBS records calls intel_pmu_store_pebs_lbrs(),
> which overwrites cpu_lbr_stack(cpuc) before the current non-PEBS sample
> copies data.br_stack and leaves lbr_refreshed set to true for any subsequent
> non-PEBS LBR events in the same PMI.
>
> In addition, because the unconditional intel_pmu_lbr_read() at PMI entry is
> removed, an indirect PEBS drain on pre-adaptive PEBS
> (!x86_pmu.intel_cap.pebs_baseline) when GLOBAL_STATUS_BUFFER_OVF_BIT is clear
> will run intel_pmu_pebs_fixup_ip() against an unrefreshed
> cpu_lbr_stack(cpuc).
Sigh, Sashiko can always give surprise. :)
It's a real issue. It looks a new version has to be posted to fix this issue.
>
^ permalink raw reply [flat|nested] 45+ messages in thread
* [Patch v6 16/20] perf/x86/intel: Remove dead lbr_pebs_users
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (14 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 15/20] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:08 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 17/20] perf/x86/intel: Clear raw LBR counters in the LBR stack Dapeng Mi
` (4 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
The lbr_pebs_users accounting is no longer used, so remove the dead
field and the related code paths that maintain it.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/lbr.c | 5 -----
arch/x86/events/perf_event.h | 1 -
2 files changed, 6 deletions(-)
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 27e4be839926..3500cf3518f4 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -617,8 +617,6 @@ void intel_pmu_lbr_add(struct perf_event *event)
* be 'new'. Conversely, a new event can get installed through the
* context switch path for the first time.
*/
- if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip > 0)
- cpuc->lbr_pebs_users++;
perf_sched_cb_inc(event->pmu);
if (!cpuc->lbr_users++ && !event->total_time_running)
intel_pmu_lbr_reset();
@@ -688,11 +686,8 @@ void intel_pmu_lbr_del(struct perf_event *event)
if (event->hw.flags & PERF_X86_EVENT_LBR_SELECT)
cpuc->lbr_select = 0;
- if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip > 0)
- cpuc->lbr_pebs_users--;
cpuc->lbr_users--;
WARN_ON_ONCE(cpuc->lbr_users < 0);
- WARN_ON_ONCE(cpuc->lbr_pebs_users < 0);
perf_sched_cb_dec(event->pmu);
/*
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index fa0af528cd1e..0213e2a8efff 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -526,7 +526,6 @@ struct cpu_hw_events {
* Intel LBR bits
*/
int lbr_users;
- int lbr_pebs_users;
u64 __lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
u64 lbr_counters[MAX_LBR_ENTRIES]; /* branch stack extra */
union {
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [Patch v6 17/20] perf/x86/intel: Clear raw LBR counters in the LBR stack
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (15 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 16/20] perf/x86/intel: Remove dead lbr_pebs_users Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:17 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 18/20] perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler Dapeng Mi
` (3 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
Raw LBR counters are currently stored in the reserved fields of the LBR
stack even when no LBR event needs them.
If an LBR counter event is present, it clears the raw LBR counters while
processing the stack, but those counters can be leaked to userspace when
no LBR event requires them. Clear the raw LBR counters when no LBR event
uses them.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: 33744916196b ("perf/x86/intel: Support branch counters logging")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 1 +
arch/x86/events/intel/lbr.c | 13 +++++++++++++
arch/x86/events/perf_event.h | 2 ++
3 files changed, 16 insertions(+)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index b267c68f9467..707bff285ed7 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2835,6 +2835,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
(struct perf_branch_stack *)bpf->__lbr_stack;
intel_pmu_lbr_read(lbr_stack, bpf->counters);
+ intel_pmu_lbr_clear_counters(lbr_stack);
cnt = min_t(unsigned int, cnt, lbr_stack->nr);
memcpy(entries, lbr_stack->entries,
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 3500cf3518f4..fbc053ebbeaf 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -987,6 +987,18 @@ static void intel_pmu_lbr_counters_reorder(struct cpu_hw_events *cpuc,
}
}
+void intel_pmu_lbr_clear_counters(struct perf_branch_stack *lbr_stack)
+{
+ int i;
+
+ if (!(x86_pmu.flags & PMU_FL_BR_CNTR))
+ return;
+
+ /* reserved field stores raw branch counters, clear it. */
+ for (i = 0; i < lbr_stack->nr; i++)
+ lbr_stack->entries[i].reserved = 0;
+}
+
void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
struct cpu_hw_events *cpuc,
struct perf_event *event)
@@ -997,6 +1009,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
return;
}
+ intel_pmu_lbr_clear_counters(cpu_lbr_stack(cpuc));
perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
}
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index 0213e2a8efff..a4e53c99eaf5 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -1984,6 +1984,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr);
void intel_pebs_init(void);
+void intel_pmu_lbr_clear_counters(struct perf_branch_stack *lbr_stack);
+
void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
struct cpu_hw_events *cpuc,
struct perf_event *event);
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [Patch v6 18/20] perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (16 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 17/20] perf/x86/intel: Clear raw LBR counters in the LBR stack Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:17 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 19/20] perf/x86/intel: Pass LBR stack size into store_lbr() Dapeng Mi
` (2 subsequent siblings)
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
When handling an LBR counter event, the raw LBR counters in the reserved
field of each LBR entry are cleared after the stack is processed and
reported to userspace. This prevents raw counter data from leaking.
However, if multiple events overflow in the same PMI, an LBR counter
event may not be the first one processed. In that case, the raw LBR
counters may already have been cleared by the previous LBR event, so the
later LBR counter event cannot read the correct values.
Refill the LBR stack before processing an LBR counter event when multiple
events are handled in a single PMI. This restores the correct raw LBR
counter data for the event in this corner case.
This situation is rare, the performance impact from extra refill should
be negligible.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: 33744916196b ("perf/x86/intel: Support branch counters logging")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 707bff285ed7..f4d4613fb9f7 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -3866,6 +3866,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
struct perf_sample_data data;
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
bool lbr_refreshed = false;
+ bool lbr_counter_cleared = false;
int handled = 0;
int bit;
@@ -4036,12 +4037,20 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
perf_sample_data_init(&data, 0, last_period);
if (has_branch_stack(event)) {
- if (!lbr_refreshed) {
+ /*
+ * The raw LBR counters stored in the reserved field of
+ * the LBR entries were cleared during processing of the
+ * previous LBR event, so refresh the LBR stack again if
+ * this event needs the LBR counters.
+ */
+ if (!lbr_refreshed ||
+ (lbr_counter_cleared && is_branch_counters_group(event))) {
intel_pmu_lbr_read(cpu_lbr_stack(cpuc),
cpuc->lbr_counters);
lbr_refreshed = true;
}
intel_pmu_lbr_save_brstack(&data, cpuc, event);
+ lbr_counter_cleared = true;
}
x86_pmu_clear_perf_regs(regs);
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* Re: [Patch v6 18/20] perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler
2026-11-06 4:32 ` [Patch v6 18/20] perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler Dapeng Mi
@ 2026-11-06 5:17 ` sashiko-bot
0 siblings, 0 replies; 45+ messages in thread
From: sashiko-bot @ 2026-11-06 5:17 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users
> When handling an LBR counter event, the raw LBR counters in the reserved
> field of each LBR entry are cleared after the stack is processed and
> reported to userspace. This prevents raw counter data from leaking.
>
> However, if multiple events overflow in the same PMI, an LBR counter
> event may not be the first one processed. In that case, the raw LBR
> counters may already have been cleared by the previous LBR event, so the
> later LBR counter event cannot read the correct values.
>
> Refill the LBR stack before processing an LBR counter event when multiple
> events are handled in a single PMI. This restores the correct raw LBR
> counter data for the event in this corner case.
>
> This situation is rare, the performance impact from extra refill should
> be negligible.
>
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Fixes: 33744916196b ("perf/x86/intel: Support branch counters logging")
> Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261106043257.842864-1-dapeng1.mi@linux.intel.com?part=18
^ permalink raw reply [flat|nested] 45+ messages in thread
* [Patch v6 19/20] perf/x86/intel: Pass LBR stack size into store_lbr()
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (17 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 18/20] perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:12 ` sashiko-bot
2026-11-06 4:32 ` [Patch v6 20/20] perf/core: Don't overwrite shared LBR stack depth Dapeng Mi
2026-11-06 7:23 ` [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Mi, Dapeng
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
intel_pmu_store_lbr() assumes the LBR stack always contains
x86_pmu.lbr_nr entries, but arch-PEBS can use a smaller configured depth.
This can result in out-of-bounds reads when the stack is shallower than
the hardware maximum.
Add a lbr_nr parameter to the helper so the code only processes the
available entries and avoids overrunning the LBR buffer.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: d21954c8a0ff ("perf/x86/intel: Process arch-PEBS records or record fragments")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/ds.c | 8 ++++----
arch/x86/events/intel/lbr.c | 18 ++++++++++--------
arch/x86/events/perf_event.h | 2 +-
3 files changed, 15 insertions(+), 13 deletions(-)
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 332754e1da4f..95a2166f5a47 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -2595,12 +2595,12 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event,
if (format_group & PEBS_DATACFG_LBRS) {
struct lbr_entry *lbr = next_record;
- int num_lbr = ((format_group >> PEBS_DATACFG_LBR_SHIFT)
+ unsigned int num_lbr = ((format_group >> PEBS_DATACFG_LBR_SHIFT)
& 0xff) + 1;
next_record = next_record + num_lbr * sizeof(struct lbr_entry);
if (has_branch_stack(event)) {
- intel_pmu_store_pebs_lbrs(lbr);
+ intel_pmu_store_pebs_lbrs(lbr, num_lbr);
intel_pmu_lbr_save_brstack(data, cpuc, event);
}
}
@@ -2781,7 +2781,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event,
if (header->lbr) {
struct arch_pebs_lbr_header *lbr_header = next_record;
struct lbr_entry *lbr;
- int num_lbr;
+ unsigned int num_lbr;
next_record = lbr_header + 1;
lbr = next_record;
@@ -2792,7 +2792,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event,
next_record += num_lbr * sizeof(struct lbr_entry);
if (has_branch_stack(event)) {
- intel_pmu_store_pebs_lbrs(lbr);
+ intel_pmu_store_pebs_lbrs(lbr, num_lbr);
intel_pmu_lbr_save_brstack(data, cpuc, event);
}
}
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index fbc053ebbeaf..ce6aa35e20fb 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -902,7 +902,7 @@ static __always_inline u16 get_lbr_cycles(u64 info)
static_assert((64 - PERF_BRANCH_ENTRY_INFO_BITS_MAX) > LBR_INFO_BR_CNTR_NUM * LBR_INFO_BR_CNTR_BITS);
-static void intel_pmu_store_lbr(struct lbr_entry *entries,
+static void intel_pmu_store_lbr(struct lbr_entry *entries, int lbr_nr,
struct perf_branch_stack *lbr_stack)
{
struct perf_branch_entry *e;
@@ -911,8 +911,8 @@ static void intel_pmu_store_lbr(struct lbr_entry *entries,
bool mispred;
int i;
- cpu_init_lbr_stack(lbr_stack, x86_pmu.lbr_nr);
- for (i = 0; i < x86_pmu.lbr_nr; i++) {
+ cpu_init_lbr_stack(lbr_stack, lbr_nr);
+ for (i = 0; i < lbr_nr; i++) {
lbr = entries ? &entries[i] : NULL;
e = &lbr_stack->entries[i];
@@ -1016,7 +1016,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
static void intel_pmu_arch_lbr_read(struct cpu_hw_events *cpuc,
struct perf_branch_stack *lbr_stack)
{
- intel_pmu_store_lbr(NULL, lbr_stack);
+ intel_pmu_store_lbr(NULL, x86_pmu.lbr_nr, lbr_stack);
}
static void
@@ -1026,14 +1026,15 @@ intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc,
struct x86_perf_task_context_arch_lbr_xsave *xsave = cpuc->lbr_xsave;
if (!xsave) {
- intel_pmu_store_lbr(NULL, lbr_stack);
+ intel_pmu_store_lbr(NULL, x86_pmu.lbr_nr, lbr_stack);
return;
}
xsave->header.xfeatures = 0;
xsaves(&xsave->xsave, XFEATURE_MASK_LBR);
if (xsave->header.xfeatures & XFEATURE_MASK_LBR) {
- intel_pmu_store_lbr(xsave->lbr.entries, lbr_stack);
+ intel_pmu_store_lbr(xsave->lbr.entries,
+ x86_pmu.lbr_nr, lbr_stack);
} else {
/* Clear the LBR stack to avoid leaking stale entries. */
lbr_stack->nr = 0;
@@ -1313,7 +1314,7 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc,
lbr_stack->nr = lbr_nr;
}
-void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
+void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr, unsigned int lbr_nr)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
@@ -1324,7 +1325,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
else
cpu_lbr_stack(cpuc)->hw_idx = intel_pmu_lbr_tos();
- intel_pmu_store_lbr(lbr, cpu_lbr_stack(cpuc));
+ intel_pmu_store_lbr(lbr, min(lbr_nr, x86_pmu.lbr_nr),
+ cpu_lbr_stack(cpuc));
intel_pmu_lbr_filter(cpuc, cpu_lbr_stack(cpuc), cpuc->lbr_counters);
}
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index a4e53c99eaf5..1b3fa997d89d 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -1980,7 +1980,7 @@ void intel_pmu_pebs_late_setup(struct cpu_hw_events *cpuc);
void intel_pmu_drain_pebs_buffer(void);
-void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr);
+void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr, unsigned int lbr_nr);
void intel_pebs_init(void);
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* [Patch v6 20/20] perf/core: Don't overwrite shared LBR stack depth
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (18 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 19/20] perf/x86/intel: Pass LBR stack size into store_lbr() Dapeng Mi
@ 2026-11-06 4:32 ` Dapeng Mi
2026-11-06 5:17 ` sashiko-bot
2026-11-06 7:23 ` [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Mi, Dapeng
20 siblings, 1 reply; 45+ messages in thread
From: Dapeng Mi @ 2026-11-06 4:32 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
perf_sample_save_brstack() currently clamps the shared LBR stack depth
to min(event->attr.sample_max_stack, brs->nr). This is unsafe because
the global LBR stack is shared across all LBR events, and a small
sample_max_stack from one event can shrink the depth for all others in
the same PMI.
As a result, later events may output a shorter LBR stack than requested
even when they do not require a small sample depth. Do not overwrite the
shared LBR stack depth in perf_sample_save_brstack(); instead, keep the
per-event limit in a local variable so each event is constrained
independently without mutating global state.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: c53e14f1ea4a ("perf: Extend per event callchain limit to branch stack")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
include/linux/perf_event.h | 13 ++++++++++---
kernel/events/core.c | 8 ++++----
2 files changed, 14 insertions(+), 7 deletions(-)
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index 19de3f1cbd6e..b9a4a821c7c6 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -1424,12 +1424,19 @@ static inline bool has_branch_stack(struct perf_event *event)
return event->attr.sample_type & PERF_SAMPLE_BRANCH_STACK;
}
+static inline u64 max_lbr_stack_depth(struct perf_event *event,
+ struct perf_branch_stack *brs)
+{
+ return min_t(u16, event->attr.sample_max_stack, brs->nr);
+}
+
static inline void perf_sample_save_brstack(struct perf_sample_data *data,
struct perf_event *event,
struct perf_branch_stack *brs,
u64 *brs_cntr)
{
int size = sizeof(u64); /* nr */
+ u64 lbr_nr;
if (!has_branch_stack(event))
return;
@@ -1439,9 +1446,9 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data,
if (branch_sample_hw_index(event))
size += sizeof(u64);
- brs->nr = min_t(u16, event->attr.sample_max_stack, brs->nr);
+ lbr_nr = max_lbr_stack_depth(event, brs);
- size += brs->nr * sizeof(struct perf_branch_entry);
+ size += lbr_nr * sizeof(struct perf_branch_entry);
/*
* The extension space for counters is appended after the
@@ -1449,7 +1456,7 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data,
* of events of each branch.
*/
if (brs_cntr)
- size += brs->nr * sizeof(u64);
+ size += lbr_nr * sizeof(u64);
data->br_stack = brs;
data->br_stack_cntr = brs_cntr;
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 457d75ea7142..db2613c2e1a2 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8462,12 +8462,12 @@ void perf_output_sample(struct perf_output_handle *handle,
if (sample_type & PERF_SAMPLE_BRANCH_STACK) {
if (data->br_stack) {
+ u64 lbr_nr = max_lbr_stack_depth(event, data->br_stack);
size_t size;
- size = data->br_stack->nr
- * sizeof(struct perf_branch_entry);
+ size = lbr_nr * sizeof(struct perf_branch_entry);
- perf_output_put(handle, data->br_stack->nr);
+ perf_output_put(handle, lbr_nr);
if (branch_sample_hw_index(event))
perf_output_put(handle, data->br_stack->hw_idx);
perf_output_copy(handle, data->br_stack->entries, size);
@@ -8476,7 +8476,7 @@ void perf_output_sample(struct perf_output_handle *handle,
* right after the struct perf_branch_stack.
*/
if (data->br_stack_cntr) {
- size = data->br_stack->nr * sizeof(u64);
+ size = lbr_nr * sizeof(u64);
perf_output_copy(handle, data->br_stack_cntr, size);
}
} else {
--
2.34.1
^ permalink raw reply related [flat|nested] 45+ messages in thread* Re: [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues
2026-11-06 4:32 [Patch v6 00/20] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (19 preceding siblings ...)
2026-11-06 4:32 ` [Patch v6 20/20] perf/core: Don't overwrite shared LBR stack depth Dapeng Mi
@ 2026-11-06 7:23 ` Mi, Dapeng
20 siblings, 0 replies; 45+ messages in thread
From: Mi, Dapeng @ 2026-11-06 7:23 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao
Sashiko reported an issue on the patch 15/20 "perf/x86/intel: Refresh LBR
stack for non-PEBS LBR events when needed",
https://lore.kernel.org/all/sashiko-outbox-200872@kernel.org/.
It looks a new version is needed to fix the issue. Thanks.
On 11/6/2026 12:32 PM, Dapeng Mi wrote:
> This patch series fixes several x86 LBR sampling issues discovered during
> review and debugging. Most of them are related to BPF LBR snapshotting
> support.
>
> The series originally targeted the BPF stack out-of-bounds access issue,
> but additional problems were found during debugging and code review,
> including:
> - PMU and LBR being re-enabled unconditionally at BPF snapshotting exit,
> even when they had already been disabled before BPF snapshotting on both
> AMD and Intel.
> - LBR not being fully frozen in the PMI handler on Intel, which allows
> branches in the PMI path to pollute the BPF LBR snapshot.
> - BPF snapshotting overwriting the cached LBR stack of PEBS LBR events on
> Intel.
> - PEBS LBR events overwriting the LBR stack of non-PEBS LBR events on
> Intel when both overflow in the same PMI.
> - LBR counter events reporting zero counters on Intel when multiple
> LBR counter events overflow in the same PMI.
>
>
> Patch layout:
> - Patch 01: Add a sanity check to ensure the LBR stack does not exceed
> MAX_LBR_ENTRIES.
> - Patch 02: Validate that hardware LBR data has actually been stored in
> the XSAVE area before parsing it.
> - Patch 03-04: Fix the potential LBR stack out-of-bounds access issue on
> AMD and Intel separately.
> - Patch 05-08: Fix the incorrect PMU and LBR re-enabling issue
> in perf_snapshot_branch_stack() for AMD and Intel separately.
> - Patch 09: Freeze LBR throughout the PMI processing period and ensure
> all LBR events observe a consistent LBR stack in the PMI handler.
> - Patch 10: Add counted bounds tracking for perf_branch_stack{}.
> - Patch 11-12: Add a dedicated LBR stack for BPF LBR snapshotting to avoid
> overwriting the default LBR stack of other LBR events.
> - Patch 13: Ensure perf_snapshot_branch_stack() snapshots the live
> hardware LBR data instead of the cached PEBS LBR stack.
> - Patch 14: Refresh LBR stack in PMI only when LBR users are active for
> AMD.
> - Patch 15: Prevent PEBS LBR data from overwriting non-PEBS LBR data when
> both overflow in the same PMI.
> - Patch 16: Remove dead lbr_pebs_users and relative code.
> - Patch 17-18: Ensure LBR counters are not dropped when multiple LBR
> counter events overflow in the same PMI.
> - Patch 19: Clamp the parsed PEBS LBR stack depth to x86_pmu.lbr_nr to
> avoid LBR stack out-of-bounds access.
> - Patch 20: Fix the shared LBR stack depth truncation issue.
>
>
> Changes in v6:
> - Patch 01: Move LBR stack depth validation into common
> init_hw_perf_events(). (Peter)
> - Patch 10: New patch, add __counted_by() tracking for flexible array
> entries[] of perf_branch_stack{}. (Peter)
> - Patch 14: Refresh LBR stack only when LBR users are active for AMD.
> (Peter)
> - Patch 15: Refactor code to ensure no duplicated LBR stack reading.
> (Peter)
> - Patch 16: New patch, remove dead lbr_pebs_users.
>
> Changes in v5:
> - Patch 02: Clear xsave->header.xfeatures before calling the xsaves
> instructuion to read LBR states.
> - Patch 05-18: New patches to fix newly discovered issues described above.
>
> Changes in v4:
> - Patch 01: Disable LBR if parsed x86_pmu.lbr_nr exceeds
> MAX_LBR_ENTRIES (Sashiko).
> - Patch 05: New patch, which always snapshots live hardware LBR stack
> for BPF path (Sashiko).
>
> Changes in v3:
> - Patch 01: A new patch to ensure x86_pmu.lbr_nr not to exclude
> MAX_LBR_ENTRIES (Internal Sashiko).
> - Patch 03: Clean cached LBR stack when reseting hardware LBR registers,
> which prevents stale LBR stack leakage.
> - Patch 04: Don't reset cached LBR stack if all LBR events are PEBS
> events. This ensures PEBS captured LBR stack isn't cleared (Sashiko).
>
> Changes in v2:
> - Include a fix for Intel LBR.
> - Add Tested-by from Dapeng to the Intel fix.
> https://lore.kernel.org/all/eb6c7bda-0dcf-4fac-ad69-9f5b24abc21b@linux.intel.com/
>
> History:
> v5: https://lore.kernel.org/all/20261029043914.1395147-1-dapeng1.mi@linux.intel.com/
> v4: https://lore.kernel.org/all/20261021050711.4043703-1-dapeng1.mi@linux.intel.com/
> v3: https://lore.kernel.org/all/20261019070836.2972403-1-dapeng1.mi@linux.intel.com/
> v2: https://lore.kernel.org/all/cover.1792064123.git.sandipan.das@amd.com/
> v1: https://lore.kernel.org/all/20261007181002.4005262-1-sandipan.das@amd.com/
>
> Dapeng Mi (19):
> perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
> perf/x86/intel: Validate LBR XSAVE state before updating LBR stack
> perf/x86/intel/lbr: Limit branch snapshots to valid entries
> perf/x86: Add lbr_enabled to track LBR state
> perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
> perf/x86/intel: Add helper to enable PMU without LBR
> perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
> perf/x86/intel: Keep LBR frozen throughout PMI handling
> perf: Add counted-bounds tracking for perf_branch_stack
> perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
> perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting
> perf/x86/intel: Always snapshot live hardware LBR state for BPF
> perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active
> perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
> perf/x86/intel: Remove dead lbr_pebs_users
> perf/x86/intel: Clear raw LBR counters in the LBR stack
> perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler
> perf/x86/intel: Pass LBR stack size into store_lbr()
> perf/core: Don't overwrite shared LBR stack depth
>
> Sandipan Das (1):
> perf/x86/amd/lbr: Limit branch snapshots to valid entries
>
> arch/powerpc/perf/core-book3s.c | 43 +++++---
> arch/x86/events/amd/brs.c | 7 +-
> arch/x86/events/amd/core.c | 21 ++--
> arch/x86/events/amd/lbr.c | 48 +++++----
> arch/x86/events/core.c | 8 +-
> arch/x86/events/intel/core.c | 130 ++++++++++++++++++++----
> arch/x86/events/intel/ds.c | 20 ++--
> arch/x86/events/intel/lbr.c | 170 +++++++++++++++++++-------------
> arch/x86/events/perf_event.h | 45 +++++++--
> drivers/perf/arm_brbe.c | 5 +
> include/linux/perf_event.h | 15 ++-
> kernel/events/core.c | 8 +-
> 12 files changed, 365 insertions(+), 155 deletions(-)
>
>
> base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
^ permalink raw reply [flat|nested] 45+ messages in thread