* [Patch v7 01/24] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:50 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 02/24] perf/x86/intel: Validate LBR XSAVE state before updating LBR stack Dapeng Mi
` (22 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
Disable LBR if x86_pmu.lbr_nr is larger than MAX_LBR_ENTRIES, preventing
out-of-bounds access in the cached LBR stack handling path.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/core.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index 4e344fb1ae3e..df3470c2f9cc 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -2645,6 +2645,12 @@ static int __init init_hw_perf_events(void)
if (!x86_pmu.config_mask)
x86_pmu.config_mask = X86_RAW_EVENT_MASK;
+ if (x86_pmu.lbr_nr > MAX_LBR_ENTRIES) {
+ pr_warn("LBR depth %d exceeds max %d, disabling LBR\n",
+ x86_pmu.lbr_nr, MAX_LBR_ENTRIES);
+ x86_pmu.lbr_nr = 0;
+ }
+
perf_events_lapic_init();
register_nmi_handler(NMI_LOCAL, perf_event_nmi_handler, 0, "PMI");
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [Patch v7 02/24] perf/x86/intel: Validate LBR XSAVE state before updating LBR stack
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
2026-11-18 3:14 ` [Patch v7 01/24] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:48 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 03/24] perf/x86/amd/lbr: Limit branch snapshots to valid entries Dapeng Mi
` (21 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
intel_pmu_arch_lbr_read_xsave() did not verify whether LBR data was
actually present in the XSAVE area. When the LBR state was in the INIT
state, the hardware might not have written valid data to the XSAVE
buffer, allowing stale or incomplete data to be cached.
Add an explicit presence check for the LBR XSAVE state before updating
the cached LBR stack. Also clear the cached LBR stack in
intel_pmu_lbr_reset() to prevent stale entries from being reused across
context switches and similar transitions.
Fixes: c085fb877467 ("perf/x86/intel/lbr: Support XSAVES for arch LBR read")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/lbr.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 0b446d8912e4..3d41385c4924 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -198,6 +198,7 @@ void intel_pmu_lbr_reset(void)
x86_pmu.lbr_reset();
+ cpuc->lbr_stack.nr = 0;
cpuc->last_task_ctx = NULL;
cpuc->last_log_id = 0;
if (!cpu_feature_enabled(X86_FEATURE_ARCH_LBR) && cpuc->lbr_select)
@@ -1010,9 +1011,15 @@ static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)
intel_pmu_store_lbr(cpuc, NULL);
return;
}
+ xsave->header.xfeatures = 0;
xsaves(&xsave->xsave, XFEATURE_MASK_LBR);
- intel_pmu_store_lbr(cpuc, xsave->lbr.entries);
+ if (xsave->header.xfeatures & XFEATURE_MASK_LBR) {
+ intel_pmu_store_lbr(cpuc, xsave->lbr.entries);
+ } else {
+ /* Clear the LBR stack to avoid leaking stale entries. */
+ cpuc->lbr_stack.nr = 0;
+ }
}
void intel_pmu_lbr_read(void)
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [Patch v7 03/24] perf/x86/amd/lbr: Limit branch snapshots to valid entries
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
2026-11-18 3:14 ` [Patch v7 01/24] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES Dapeng Mi
2026-11-18 3:14 ` [Patch v7 02/24] perf/x86/intel: Validate LBR XSAVE state before updating LBR stack Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:34 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 04/24] perf/x86/intel: " Dapeng Mi
` (20 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Sashiko, Dapeng Mi
From: Sandipan Das <sandipan.das@amd.com>
The branch snapshot callback used by bpf_get_branch_snapshot() copies
up to x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the
last LBR read kept fewer branches. The entries past lbr_stack.nr are
invalid, but they end up reaching BPF programs.
Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when
amd_pmu_lbr_read() skips reading the registers, so that a stale count
is never reused.
Fixes: a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-163130@kernel.org/
Signed-off-by: Sandipan Das <sandipan.das@amd.com>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/amd/core.c | 2 +-
arch/x86/events/amd/lbr.c | 4 +++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
index 49b6b8fce566..bf631f60757c 100644
--- a/arch/x86/events/amd/core.c
+++ b/arch/x86/events/amd/core.c
@@ -944,7 +944,7 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
cpuc = this_cpu_ptr(&cpu_hw_events);
amd_pmu_lbr_read();
- cnt = min(cnt, x86_pmu.lbr_nr);
+ cnt = min(cnt, cpuc->lbr_stack.nr);
memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
amd_pmu_v2_enable_all(0);
diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c
index a55646fcb846..6c93b8042c73 100644
--- a/arch/x86/events/amd/lbr.c
+++ b/arch/x86/events/amd/lbr.c
@@ -168,8 +168,10 @@ void amd_pmu_lbr_read(void)
struct branch_entry entry;
int out = 0, idx, i;
- if (!cpuc->lbr_users)
+ if (!cpuc->lbr_users) {
+ cpuc->lbr_stack.nr = 0;
return;
+ }
for (i = 0; i < x86_pmu.lbr_nr; i++) {
entry.from.full = amd_pmu_lbr_get_from(i);
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [Patch v7 04/24] perf/x86/intel: Limit branch snapshots to valid entries
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (2 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 03/24] perf/x86/amd/lbr: Limit branch snapshots to valid entries Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:37 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 05/24] perf/x86: Add lbr_enabled to track LBR state Dapeng Mi
` (19 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
The branch snapshot callback used by bpf_get_branch_snapshot() always
copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the
last LBR read kept fewer branches. The entries past lbr_stack.nr hold
invalid branches which end up reaching BPF programs.
Limit copying to lbr_stack.nr entries, and reset lbr_stack.nr when there
are no active LBR events or the guest owns the LBR hardware, so that
stale or guest LBR data can't leak.
For the case where all LBR events are PEBS events, hardware LBR reads are
currently skipped; that can leave stale LBR stack data from previous PEBS
sampling in place and may be inconsistent with the intended semantics of
bpf_get_branch_snapshot(). This will be addressed by making the BPF
snapshot always read the live hardware LBR state.
Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-163130@kernel.org/
Co-developed-by: Sandipan Das <sandipan.das@amd.com>
Signed-off-by: Sandipan Das <sandipan.das@amd.com>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 2 +-
arch/x86/events/intel/lbr.c | 9 +++++++--
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 0a34d674df59..5dfb9e9156e8 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2822,7 +2822,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
intel_pmu_lbr_read();
- cnt = min_t(unsigned int, cnt, x86_pmu.lbr_nr);
+ cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
intel_pmu_enable_all(0);
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 3d41385c4924..79303fb9b046 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -1026,14 +1026,19 @@ void intel_pmu_lbr_read(void)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+ if (!cpuc->lbr_users || vlbr_exclude_host()) {
+ /* Reset lbr_stack.nr to prevent stale data leakage. */
+ cpuc->lbr_stack.nr = 0;
+ return;
+ }
+
/*
* Don't read when all LBRs users are using adaptive PEBS.
*
* This could be smarter and actually check the event,
* but this simple approach seems to work for now.
*/
- if (!cpuc->lbr_users || vlbr_exclude_host() ||
- cpuc->lbr_users == cpuc->lbr_pebs_users)
+ if (cpuc->lbr_users == cpuc->lbr_pebs_users)
return;
x86_pmu.lbr_read(cpuc);
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* Re: [Patch v7 04/24] perf/x86/intel: Limit branch snapshots to valid entries
2026-11-18 3:14 ` [Patch v7 04/24] perf/x86/intel: " Dapeng Mi
@ 2026-11-18 3:37 ` sashiko-bot
0 siblings, 0 replies; 49+ messages in thread
From: sashiko-bot @ 2026-11-18 3:37 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users
> The branch snapshot callback used by bpf_get_branch_snapshot() always
> copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the
> last LBR read kept fewer branches. The entries past lbr_stack.nr hold
> invalid branches which end up reaching BPF programs.
>
> Limit copying to lbr_stack.nr entries, and reset lbr_stack.nr when there
> are no active LBR events or the guest owns the LBR hardware, so that
> stale or guest LBR data can't leak.
>
> For the case where all LBR events are PEBS events, hardware LBR reads are
> currently skipped; that can leave stale LBR stack data from previous PEBS
> sampling in place and may be inconsistent with the intended semantics of
> bpf_get_branch_snapshot(). This will be addressed by making the BPF
> snapshot always read the live hardware LBR state.
>
> Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Closes: https://lore.kernel.org/all/sashiko-outbox-163130@kernel.org/
> Co-developed-by: Sandipan Das <sandipan.das@amd.com>
> Signed-off-by: Sandipan Das <sandipan.das@amd.com>
> Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261118031453.1353625-1-dapeng1.mi@linux.intel.com?part=4
^ permalink raw reply [flat|nested] 49+ messages in thread
* [Patch v7 05/24] perf/x86: Add lbr_enabled to track LBR state
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (3 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 04/24] perf/x86/intel: " Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:58 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 06/24] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled Dapeng Mi
` (18 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
Add a lbr_enabled flag to track whether LBR is enabled. This allows the
branch snapshot callbacks to avoid blindly re-enabling LBR in
perf_snapshot_branch_stack() without checking whether LBR has already
been disabled.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/amd/lbr.c | 2 ++
arch/x86/events/intel/core.c | 5 +++++
arch/x86/events/intel/lbr.c | 2 ++
arch/x86/events/perf_event.h | 9 +++++++++
4 files changed, 18 insertions(+)
diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c
index 6c93b8042c73..400c94ecded0 100644
--- a/arch/x86/events/amd/lbr.c
+++ b/arch/x86/events/amd/lbr.c
@@ -399,6 +399,8 @@ void amd_pmu_lbr_enable_all(void)
if (!cpuc->lbr_users || !x86_pmu.lbr_nr)
return;
+ cpuc->lbr_enabled = 1;
+
/* Set hardware branch filter */
if (cpuc->lbr_select) {
lbr_select = cpuc->lbr_sel->config & LBR_SELECT_MASK;
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 5dfb9e9156e8..8ca7dc8c60db 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -4001,6 +4001,11 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
* It needs to be restored when leaving the handler.
*/
pmu_enabled = cpuc->enabled;
+
+ /* If PMU version < 4, LBR freeze disables LBR. */
+ if (x86_pmu.version < 4)
+ cpuc->lbr_enabled = 0;
+
/*
* In general, the early ACK is only applied for old platforms.
* For the big core starts from Haswell, the late ACK should be
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 79303fb9b046..fe8b303839d3 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -132,6 +132,8 @@ static void __intel_pmu_lbr_enable(bool pmi)
if (pmi && x86_pmu.version >= 4)
return;
+ cpuc->lbr_enabled = 1;
+
/*
* No need to reprogram LBR_SELECT in a PMI, as it
* did not change.
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index e274802ef062..a4aab8130e92 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -473,6 +473,7 @@ struct cpu_hw_events {
unsigned long active_mask[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
unsigned long dirty[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
int enabled;
+ int lbr_enabled;
int n_events; /* the # of events in the below arrays */
int n_added; /* the # last events in the below arrays;
@@ -1676,8 +1677,11 @@ int amd_pmu_lbr_hw_config(struct perf_event *event);
static __always_inline void __amd_pmu_lbr_disable(void)
{
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
u64 dbg_ctl, dbg_extn_cfg;
+ cpuc->lbr_enabled = 0;
+
rdmsrq(MSR_AMD_DBG_EXTN_CFG, dbg_extn_cfg);
wrmsrq(MSR_AMD_DBG_EXTN_CFG, dbg_extn_cfg & ~DBG_EXTN_CFG_LBRV2EN);
@@ -1829,13 +1833,18 @@ static __always_inline void __intel_pmu_pebs_disable_all(void)
static __always_inline void __intel_pmu_arch_lbr_disable(void)
{
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+
+ cpuc->lbr_enabled = 0;
wrmsrq(MSR_ARCH_LBR_CTL, 0);
}
static __always_inline void __intel_pmu_lbr_disable(void)
{
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
u64 debugctl;
+ cpuc->lbr_enabled = 0;
rdmsrq(MSR_IA32_DEBUGCTLMSR, debugctl);
debugctl &= ~(DEBUGCTLMSR_LBR | DEBUGCTLMSR_FREEZE_LBRS_ON_PMI);
wrmsrq(MSR_IA32_DEBUGCTLMSR, debugctl);
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [Patch v7 06/24] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (4 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 05/24] perf/x86: Add lbr_enabled to track LBR state Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:43 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 07/24] perf/x86/intel: Add helper to enable PMU without LBR Dapeng Mi
` (17 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
amd_pmu_v2_snapshot_branch_stack() currently re-enables the PMU and LBR
unconditionally even when they were already disabled in PMI context.
This can cause events such as cycles and instructions to be counted
while the PMI handler is running, which is not intended.
Only restore the PMU and LBR on the BPF path if they were originally
enabled.
Fixes: a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-179823@kernel.org/
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Cc: Sandipan Das <sandipan.das@amd.com>
---
arch/x86/events/amd/core.c | 15 ++++++++++++---
1 file changed, 12 insertions(+), 3 deletions(-)
diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
index bf631f60757c..dd0e24f26c6b 100644
--- a/arch/x86/events/amd/core.c
+++ b/arch/x86/events/amd/core.c
@@ -932,22 +932,31 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
{
struct cpu_hw_events *cpuc;
unsigned long flags;
+ int pmu_enabled;
+ int lbr_enabled;
/*
* The sequence of steps to freeze LBR should be completely inlined
* and contain no branches to minimize contamination of LBR snapshot
*/
local_irq_save(flags);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
+ pmu_enabled = cpuc->enabled;
+ lbr_enabled = cpuc->lbr_enabled;
+ cpuc->enabled = 0;
amd_pmu_core_disable_all();
__amd_pmu_lbr_disable();
- cpuc = this_cpu_ptr(&cpu_hw_events);
-
amd_pmu_lbr_read();
cnt = min(cnt, cpuc->lbr_stack.nr);
memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
- amd_pmu_v2_enable_all(0);
+ if (pmu_enabled) {
+ cpuc->enabled = 1;
+ amd_pmu_core_enable_all();
+ }
+ if (lbr_enabled)
+ amd_pmu_lbr_enable_all();
local_irq_restore(flags);
return cnt;
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* Re: [Patch v7 06/24] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
2026-11-18 3:14 ` [Patch v7 06/24] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled Dapeng Mi
@ 2026-11-18 3:43 ` sashiko-bot
0 siblings, 0 replies; 49+ messages in thread
From: sashiko-bot @ 2026-11-18 3:43 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users, bpf
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] perf/x86/amd: PMU enabled before LBR in
amd_pmu_v2_snapshot_branch_stack()
--
commit 496e4315c6fe26a4c320a0d5de1185d312c47c38
Author: Dapeng Mi <dapeng1.mi@linux.intel.com>
perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
This commit saves the enabled states of the core PMU and LBR in
amd_pmu_v2_snapshot_branch_stack() and only restores them if they were
enabled before the snapshot was taken.
> diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
> index bf631f60757cc..dd0e24f26c6b0 100644
> --- a/arch/x86/events/amd/core.c
> +++ b/arch/x86/events/amd/core.c
> @@ -932,22 +932,31 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
> {
> struct cpu_hw_events *cpuc;
> unsigned long flags;
> + int pmu_enabled;
> + int lbr_enabled;
>
> /*
> * The sequence of steps to freeze LBR should be completely inlined
> * and contain no branches to minimize contamination of LBR snapshot
> */
> local_irq_save(flags);
> + cpuc = this_cpu_ptr(&cpu_hw_events);
> + pmu_enabled = cpuc->enabled;
> + lbr_enabled = cpuc->lbr_enabled;
> + cpuc->enabled = 0;
> amd_pmu_core_disable_all();
> __amd_pmu_lbr_disable();
>
> - cpuc = this_cpu_ptr(&cpu_hw_events);
> -
> amd_pmu_lbr_read();
> cnt = min(cnt, cpuc->lbr_stack.nr);
> memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
>
> - amd_pmu_v2_enable_all(0);
> + if (pmu_enabled) {
> + cpuc->enabled = 1;
> + amd_pmu_core_enable_all();
> + }
> + if (lbr_enabled)
> + amd_pmu_lbr_enable_all();
[Severity: Low]
Should amd_pmu_lbr_enable_all() run before amd_pmu_core_enable_all(),
matching amd_pmu_v2_enable_all()?
Calling amd_pmu_core_enable_all() first restarts the hardware PMCs while
amd_pmu_lbr_enable_all() is still reprogramming the LBR MSRs, so the PMCs
count the MSR accesses in amd_pmu_lbr_enable_all() and can overflow before
DEBUGCTLMSR_FREEZE_LBRS_ON_PMI and DBG_EXTN_CFG_LBRV2EN are restored.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261118031453.1353625-1-dapeng1.mi@linux.intel.com?part=6
^ permalink raw reply [flat|nested] 49+ messages in thread
* [Patch v7 07/24] perf/x86/intel: Add helper to enable PMU without LBR
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (5 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 06/24] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:28 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 08/24] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled Dapeng Mi
` (16 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
Currently, __intel_pmu_enable_all() also re-enables LBR in addition to
the PMU. Split out the PMU-only path into a new helper,
___intel_pmu_enable_all(), so the PMU can be enabled independently. This
allows the branch snapshot path to re-enable the PMU without touching LBR.
Also remove the unused added argument from __intel_pmu_enable_all().
No functional change intended.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 8ca7dc8c60db..1a33963c5ee2 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2783,13 +2783,11 @@ static __always_inline void intel_pmu_disable_all(void)
intel_pmu_lbr_disable_all();
}
-static void __intel_pmu_enable_all(int added, bool pmi)
+static void ___intel_pmu_enable_all(void)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
u64 intel_ctrl = hybrid(cpuc->pmu, intel_ctrl);
- intel_pmu_lbr_enable_all(pmi);
-
if (cpuc->fixed_ctrl_val != cpuc->active_fixed_ctrl_val) {
wrmsrq(MSR_ARCH_PERFMON_FIXED_CTR_CTRL, cpuc->fixed_ctrl_val);
cpuc->active_fixed_ctrl_val = cpuc->fixed_ctrl_val;
@@ -2809,10 +2807,16 @@ static void __intel_pmu_enable_all(int added, bool pmi)
}
}
+static void __intel_pmu_enable_all(bool pmi)
+{
+ intel_pmu_lbr_enable_all(pmi);
+ ___intel_pmu_enable_all();
+}
+
static void intel_pmu_enable_all(int added)
{
static_call_cond(x86_pmu_pebs_enable_all)();
- __intel_pmu_enable_all(added, false);
+ __intel_pmu_enable_all(false);
}
static noinline int
@@ -4056,7 +4060,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
/* Only restore PMU state when it's active. See x86_pmu_disable(). */
cpuc->enabled = pmu_enabled;
if (pmu_enabled)
- __intel_pmu_enable_all(0, true);
+ __intel_pmu_enable_all(true);
intel_bts_enable_local();
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [Patch v7 08/24] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (6 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 07/24] perf/x86/intel: Add helper to enable PMU without LBR Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:39 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 09/24] perf/x86/intel: Keep LBR frozen throughout PMI handling Dapeng Mi
` (15 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
__intel_pmu_snapshot_branch_stack() currently re-enables the PMU and LBR
unconditionally even when they were already disabled in PMI context.
This can cause events such as cycles and instructions to be counted
while the PMI handler is running, which is not intended.
Only restore the PMU and LBR on the BPF path if they were originally
enabled.
Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-179823@kernel.org/
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 39 ++++++++++++++++++++++++++++++++----
1 file changed, 35 insertions(+), 4 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 1a33963c5ee2..b8b4e5d8a0e2 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2821,7 +2821,8 @@ static void intel_pmu_enable_all(int added)
static noinline int
__intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
- unsigned int cnt, unsigned long flags)
+ unsigned int cnt, unsigned long flags,
+ int pmu_enabled, int lbr_enabled)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
@@ -2829,7 +2830,21 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
- intel_pmu_enable_all(0);
+
+ /*
+ * This path may be entered from the PMI handler through
+ * bpf_overflow_handler(), so only re-enable what was enabled
+ * on entry. On PMU version < 4 the LBR freeze disables LBR,
+ * so it must stay disabled. On version 4 and later, LBR is
+ * only frozen and can be re-enabled.
+ */
+ if (pmu_enabled) {
+ cpuc->enabled = 1;
+ ___intel_pmu_enable_all();
+ }
+ if (lbr_enabled)
+ intel_pmu_lbr_enable_all(false);
+
local_irq_restore(flags);
return cnt;
}
@@ -2837,27 +2852,43 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
static int
intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
{
+ struct cpu_hw_events *cpuc;
unsigned long flags;
+ int pmu_enabled;
+ int lbr_enabled;
/* must not have branches... */
local_irq_save(flags);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
+ pmu_enabled = cpuc->enabled;
+ lbr_enabled = cpuc->lbr_enabled;
+ cpuc->enabled = 0;
__intel_pmu_disable_all(false); /* we don't care about BTS */
__intel_pmu_lbr_disable();
/* ... until here */
- return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
+ return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
+ pmu_enabled, lbr_enabled);
}
static int
intel_pmu_snapshot_arch_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
{
+ struct cpu_hw_events *cpuc;
unsigned long flags;
+ int pmu_enabled;
+ int lbr_enabled;
/* must not have branches... */
local_irq_save(flags);
+ cpuc = this_cpu_ptr(&cpu_hw_events);
+ pmu_enabled = cpuc->enabled;
+ lbr_enabled = cpuc->lbr_enabled;
+ cpuc->enabled = 0;
__intel_pmu_disable_all(false); /* we don't care about BTS */
__intel_pmu_arch_lbr_disable();
/* ... until here */
- return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
+ return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
+ pmu_enabled, lbr_enabled);
}
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [Patch v7 09/24] perf/x86/intel: Keep LBR frozen throughout PMI handling
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (7 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 08/24] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:42 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 10/24] perf: Add counted-bounds tracking for perf_branch_stack Dapeng Mi
` (14 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
On PMU version 4 and later, the streamlined freeze sets
PERF_GLOBAL_STATUS.LBR_FRZ to freeze LBR when a counter overflows.
However, the current PMI handler clears the LBR freeze bit too early,
allowing branch operations that occur during PMI handling to be recorded
into the LBR stack.
This can pollute or overwrite the intended branch snapshot, especially
when bpf_get_branch_snapshot() is invoked from bpf_overflow_handler().
In that case, branches recorded while processing the PMI are mixed into
the original LBR data.
Defer unfreezing until after PMI handling completes so the LBR remains
frozen for the entire PMI path and no branch activity from the handler
is recorded.
Fixes: 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-179823@kernel.org/
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index b8b4e5d8a0e2..2e9e2127e34e 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -4063,7 +4063,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
loops = 0;
again:
intel_pmu_lbr_read();
- intel_pmu_ack_status(status);
+
+ /*
+ * Keep the LBR state frozen throughout PMI handling so that the PMI
+ * processing itself does not pollute the LBR state.
+ */
+ intel_pmu_ack_status(status & ~GLOBAL_STATUS_LBRS_FROZEN);
+
if (++loops > 100) {
static bool warned;
@@ -4082,10 +4088,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
* Repeat if there is more work to be done:
*/
status = intel_pmu_get_status();
- if (status)
+ if (status & ~GLOBAL_STATUS_LBRS_FROZEN)
goto again;
done:
+ if (status & GLOBAL_STATUS_LBRS_FROZEN)
+ intel_pmu_ack_status(status & GLOBAL_STATUS_LBRS_FROZEN);
+
if (mid_ack)
apic_write(APIC_LVTPC, APIC_DM_NMI);
/* Only restore PMU state when it's active. See x86_pmu_disable(). */
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [Patch v7 10/24] perf: Add counted-bounds tracking for perf_branch_stack
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (8 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 09/24] perf/x86/intel: Keep LBR frozen throughout PMI handling Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:35 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 11/24] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks Dapeng Mi
` (13 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Madhavan Srinivasan,
Michael Ellerman, Nicholas Piggin, Christophe Leroy (CS GROUP),
Will Deacon, Mark Rutland, linux-arm-kernel
struct perf_branch_stack uses a trailing flexible array member,
entries[], whose valid length is tracked by nr. When this structure
is embedded as a value member in cpu_hw_events, the compiler can treat
the trailing flexible array as having a fixed zero-length layout under
-fstrict-flex-arrays=3 and UBSan, which can lead to strict array-bounds
diagnostics when entries[] is indexed.
Annotate entries[] with __counted_by(nr) so the compiler and sanitizer
track the array length from the nr field. Because a structure containing
a counted flexible array cannot be embedded as a value member, switch
the per-CPU LBR/BHRB stacks to raw backing arrays and add accessors that
return the struct perf_branch_stack view.
Also initialize the branch-stack count before populating entries[] to
avoid false-positive bounds checks while the stack is being filled.
No functional change intended.
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Cc: Sandipan Das <sandipan.das@amd.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: "Christophe Leroy (CS GROUP)" <chleroy@kernel.org>
Cc: Will Deacon <will@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
---
arch/powerpc/perf/core-book3s.c | 47 +++++++++++------
arch/x86/events/amd/brs.c | 7 +--
arch/x86/events/amd/core.c | 10 ++--
arch/x86/events/amd/lbr.c | 44 +++++++++-------
arch/x86/events/core.c | 2 +-
arch/x86/events/intel/core.c | 5 +-
arch/x86/events/intel/ds.c | 12 +++--
arch/x86/events/intel/lbr.c | 89 +++++++++++++++++----------------
arch/x86/events/perf_event.h | 19 ++++++-
drivers/perf/arm_brbe.c | 5 ++
include/linux/perf_event.h | 2 +-
11 files changed, 149 insertions(+), 93 deletions(-)
diff --git a/arch/powerpc/perf/core-book3s.c b/arch/powerpc/perf/core-book3s.c
index 10ded865583c..07e54f634c6d 100644
--- a/arch/powerpc/perf/core-book3s.c
+++ b/arch/powerpc/perf/core-book3s.c
@@ -30,6 +30,8 @@
#define BHRB_PREDICTION 0x0000000000000001
#define BHRB_EA 0xFFFFFFFFFFFFFFFCUL
+#define BHRB_STACK_SIZE struct_size_t(struct perf_branch_stack, entries, BHRB_MAX_ENTRIES)
+
struct cpu_hw_events {
int n_events;
int n_percpu;
@@ -54,8 +56,7 @@ struct cpu_hw_events {
u64 bhrb_filter; /* BHRB HW branch filter */
unsigned int bhrb_users;
void *bhrb_context;
- struct perf_branch_stack bhrb_stack;
- struct perf_branch_entry bhrb_entries[BHRB_MAX_ENTRIES];
+ u64 __bhrb_stack[BHRB_STACK_SIZE / sizeof(u64)];
u64 ic_init;
/* Store the PMC values */
@@ -177,6 +178,20 @@ static bool regs_use_siar(struct pt_regs *regs)
return ((TRAP(regs) == INTERRUPT_PERFMON) && regs->result);
}
+static __always_inline struct perf_branch_stack *cpu_bhrb_stack(struct cpu_hw_events *cpuc)
+{
+ return (struct perf_branch_stack *)cpuc->__bhrb_stack;
+}
+
+static __always_inline void cpu_init_bhrb_stack(struct cpu_hw_events *cpuc, int nr)
+{
+ /*
+ * Initialize the branch-stack count before populating entries[]
+ * to avoid UBSAN complaints from the __counted_by(nr) flexible array.
+ */
+ cpu_bhrb_stack(cpuc)->nr = nr;
+}
+
/*
* Things that are specific to 64-bit implementations.
*/
@@ -487,12 +502,15 @@ static __u64 power_pmu_bhrb_to(u64 addr)
/* Processing BHRB entries */
static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events *cpuhw)
{
+ struct perf_branch_stack *bhrb = cpu_bhrb_stack(cpuhw);
+ int r_index, u_index, pred;
u64 val;
u64 addr;
- int r_index, u_index, pred;
r_index = 0;
u_index = 0;
+
+ cpu_init_bhrb_stack(cpuhw, ppmu->bhrb_nr);
while (r_index < ppmu->bhrb_nr) {
/* Assembly read function */
val = read_bhrb(r_index++);
@@ -541,9 +559,9 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events *
/* Target branches use two entries
* (ie. computed gotos/XL form)
*/
- cpuhw->bhrb_entries[u_index].to = addr;
- cpuhw->bhrb_entries[u_index].mispred = pred;
- cpuhw->bhrb_entries[u_index].predicted = ~pred;
+ bhrb->entries[u_index].to = addr;
+ bhrb->entries[u_index].mispred = pred;
+ bhrb->entries[u_index].predicted = ~pred;
/* Get from address in next entry */
val = read_bhrb(r_index++);
@@ -554,22 +572,21 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events *
r_index--;
addr = 0;
}
- cpuhw->bhrb_entries[u_index].from = addr;
+ bhrb->entries[u_index].from = addr;
} else {
/* Branches to immediate field
(ie I or B form) */
- cpuhw->bhrb_entries[u_index].from = addr;
- cpuhw->bhrb_entries[u_index].to =
- power_pmu_bhrb_to(addr);
- cpuhw->bhrb_entries[u_index].mispred = pred;
- cpuhw->bhrb_entries[u_index].predicted = ~pred;
+ bhrb->entries[u_index].from = addr;
+ bhrb->entries[u_index].to = power_pmu_bhrb_to(addr);
+ bhrb->entries[u_index].mispred = pred;
+ bhrb->entries[u_index].predicted = ~pred;
}
u_index++;
}
}
- cpuhw->bhrb_stack.nr = u_index;
- cpuhw->bhrb_stack.hw_idx = -1ULL;
+ bhrb->nr = u_index;
+ bhrb->hw_idx = -1ULL;
return;
}
@@ -2332,7 +2349,7 @@ static void record_and_restart(struct perf_event *event, unsigned long val,
struct cpu_hw_events *cpuhw;
cpuhw = this_cpu_ptr(&cpu_hw_events);
power_pmu_bhrb_read(event, cpuhw);
- perf_sample_save_brstack(&data, event, &cpuhw->bhrb_stack, NULL);
+ perf_sample_save_brstack(&data, event, cpu_bhrb_stack(cpuhw), NULL);
}
if (event->attr.sample_type & PERF_SAMPLE_DATA_SRC &&
diff --git a/arch/x86/events/amd/brs.c b/arch/x86/events/amd/brs.c
index 54b13faba116..1906d1257646 100644
--- a/arch/x86/events/amd/brs.c
+++ b/arch/x86/events/amd/brs.c
@@ -282,7 +282,6 @@ void amd_brs_drain(void)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct perf_event *event = cpuc->events[0];
- struct perf_branch_entry *br = cpuc->lbr_entries;
union amd_debug_extn_cfg cfg;
u32 i, nr = 0, num, tos, start;
u32 shift = 64 - boot_cpu_data.x86_virt_bits;
@@ -317,6 +316,8 @@ void amd_brs_drain(void)
num = tos - start + 1;
+ cpu_init_lbr_stack(cpuc, num);
+
/*
* BRS is only one pass (saturation) from MSROFF to depth-1
* MSROFF wraps to zero when buffer is full
@@ -343,7 +344,7 @@ void amd_brs_drain(void)
if (!amd_brs_match_plm(event, from, to))
continue;
- br[nr] = (struct perf_branch_entry){
+ cpu_lbr_stack(cpuc)->entries[nr] = (struct perf_branch_entry){
.from = from,
.to = to,
};
@@ -351,7 +352,7 @@ void amd_brs_drain(void)
}
empty:
/* Record number of sampled branches */
- cpuc->lbr_stack.nr = nr;
+ cpu_lbr_stack(cpuc)->nr = nr;
}
/*
diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
index dd0e24f26c6b..7c6a9135eb22 100644
--- a/arch/x86/events/amd/core.c
+++ b/arch/x86/events/amd/core.c
@@ -928,7 +928,8 @@ static int amd_pmu_handle_irq(struct pt_regs *regs)
* details. It's up to caller to provide enough space in *entries* to fit all
* LBR records, otherwise returned result will be truncated to *cnt* entries.
*/
-static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
+static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries,
+ unsigned int cnt)
{
struct cpu_hw_events *cpuc;
unsigned long flags;
@@ -948,8 +949,9 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
__amd_pmu_lbr_disable();
amd_pmu_lbr_read();
- cnt = min(cnt, cpuc->lbr_stack.nr);
- memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
+ cnt = min(cnt, cpu_lbr_stack(cpuc)->nr);
+ memcpy(entries, cpu_lbr_stack(cpuc)->entries,
+ sizeof(struct perf_branch_entry) * cnt);
if (pmu_enabled) {
cpuc->enabled = 1;
@@ -1023,7 +1025,7 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)
if (!x86_perf_event_set_period(event))
continue;
- perf_sample_save_brstack(&data, event, &cpuc->lbr_stack, NULL);
+ perf_sample_save_brstack(&data, event, cpu_lbr_stack(cpuc), NULL);
perf_event_overflow(event, &data, regs);
}
diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c
index 400c94ecded0..3e4e5e984649 100644
--- a/arch/x86/events/amd/lbr.c
+++ b/arch/x86/events/amd/lbr.c
@@ -104,15 +104,18 @@ static void amd_pmu_lbr_filter(void)
bool compress = false;
bool fused_only = false;
u64 from, to;
+ int lbr_nr;
/* If sampling all branches, there is nothing to filter */
if (((br_sel & X86_BR_ALL) == X86_BR_ALL) &&
((br_sel & X86_BR_TYPE_SAVE) != X86_BR_TYPE_SAVE))
fused_only = true;
- for (i = 0; i < cpuc->lbr_stack.nr; i++) {
- from = cpuc->lbr_entries[i].from;
- to = cpuc->lbr_entries[i].to;
+ for (i = 0; i < cpu_lbr_stack(cpuc)->nr; i++) {
+ struct perf_branch_entry *be = &cpu_lbr_stack(cpuc)->entries[i];
+
+ from = be->from;
+ to = be->to;
type = branch_type_fused(from, to, 0, &offset);
/*
@@ -121,37 +124,41 @@ static void amd_pmu_lbr_filter(void)
* actual branch
*/
if (offset) {
- cpuc->lbr_entries[i].from += offset;
+ be->from += offset;
if (fused_only)
continue;
}
/* If type does not correspond, then discard */
if (type == X86_BR_NONE || (br_sel & type) != type ||
- (!(br_sel & X86_BR_KERNEL) && kernel_ip(cpuc->lbr_entries[i].from))) {
- cpuc->lbr_entries[i].from = 0; /* mark invalid */
+ (!(br_sel & X86_BR_KERNEL) && kernel_ip(be->from))) {
+ be->from = 0; /* mark invalid */
compress = true;
}
if ((br_sel & X86_BR_TYPE_SAVE) == X86_BR_TYPE_SAVE)
- cpuc->lbr_entries[i].type = common_branch_type(type);
+ be->type = common_branch_type(type);
}
if (!compress)
return;
/* Remove all invalid entries */
- for (i = 0; i < cpuc->lbr_stack.nr; ) {
- if (!cpuc->lbr_entries[i].from) {
+ lbr_nr = cpu_lbr_stack(cpuc)->nr;
+ for (i = 0; i < lbr_nr; ) {
+ if (!cpu_lbr_stack(cpuc)->entries[i].from) {
j = i;
- while (++j < cpuc->lbr_stack.nr)
- cpuc->lbr_entries[j - 1] = cpuc->lbr_entries[j];
- cpuc->lbr_stack.nr--;
- if (!cpuc->lbr_entries[i].from)
+ while (++j < lbr_nr) {
+ cpu_lbr_stack(cpuc)->entries[j - 1] =
+ cpu_lbr_stack(cpuc)->entries[j];
+ }
+ lbr_nr--;
+ if (!cpu_lbr_stack(cpuc)->entries[i].from)
continue;
}
i++;
}
+ cpu_lbr_stack(cpuc)->nr = lbr_nr;
}
static const int lbr_spec_map[PERF_BR_SPEC_MAX] = {
@@ -164,15 +171,16 @@ static const int lbr_spec_map[PERF_BR_SPEC_MAX] = {
void amd_pmu_lbr_read(void)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
- struct perf_branch_entry *br = cpuc->lbr_entries;
struct branch_entry entry;
int out = 0, idx, i;
if (!cpuc->lbr_users) {
- cpuc->lbr_stack.nr = 0;
+ cpu_lbr_stack(cpuc)->nr = 0;
return;
}
+ cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
+
for (i = 0; i < x86_pmu.lbr_nr; i++) {
entry.from.full = amd_pmu_lbr_get_from(i);
entry.to.full = amd_pmu_lbr_get_to(i);
@@ -204,7 +212,7 @@ void amd_pmu_lbr_read(void)
*/
idx = (entry.to.split.valid << 1) | entry.to.split.spec;
- br[out] = (struct perf_branch_entry){
+ cpu_lbr_stack(cpuc)->entries[out] = (struct perf_branch_entry){
.from = sign_ext_branch_ip(entry.from.split.ip),
.to = sign_ext_branch_ip(entry.to.split.ip),
.mispred = entry.from.split.mispredict,
@@ -214,13 +222,13 @@ void amd_pmu_lbr_read(void)
out++;
}
- cpuc->lbr_stack.nr = out;
+ cpu_lbr_stack(cpuc)->nr = out;
/*
* Internal register renaming always ensures that LBR From[0] and
* LBR To[0] always represent the TOS
*/
- cpuc->lbr_stack.hw_idx = 0;
+ cpu_lbr_stack(cpuc)->hw_idx = 0;
/* Perform further software filtering */
amd_pmu_lbr_filter();
diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index df3470c2f9cc..0c94b5fd8a16 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -2197,7 +2197,7 @@ int x86_pmu_handle_irq(struct pt_regs *regs)
perf_sample_data_init(&data, 0, last_period);
- perf_sample_save_brstack(&data, event, &cpuc->lbr_stack, NULL);
+ perf_sample_save_brstack(&data, event, cpu_lbr_stack(cpuc), NULL);
perf_event_overflow(event, &data, regs);
}
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 2e9e2127e34e..6547555192cf 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2827,9 +2827,10 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
intel_pmu_lbr_read();
- cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
+ cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr);
- memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
+ memcpy(entries, cpu_lbr_stack(cpuc)->entries,
+ sizeof(struct perf_branch_entry) * cnt);
/*
* This path may be entered from the PMI handler through
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 8444670cee4a..332754e1da4f 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -2010,8 +2010,7 @@ void intel_pmu_pebs_disable_all(void)
static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
- unsigned long from = cpuc->lbr_entries[0].from;
- unsigned long old_to, to = cpuc->lbr_entries[0].to;
+ unsigned long from, old_to, to;
unsigned long ip = regs->ip;
int is_64bit = 0;
void *kaddr;
@@ -2026,7 +2025,12 @@ static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)
/*
* No LBR entry, no basic block, no rewinding
*/
- if (!cpuc->lbr_stack.nr || !from || !to)
+ if (!cpu_lbr_stack(cpuc)->nr)
+ return 0;
+
+ from = cpu_lbr_stack(cpuc)->entries[0].from;
+ to = cpu_lbr_stack(cpuc)->entries[0].to;
+ if (!from || !to)
return 0;
/*
@@ -2318,7 +2322,7 @@ static void setup_pebs_fixed_sample_data(struct perf_event *event,
if (x86_pmu.intel_cap.pebs_format >= 3)
setup_pebs_time(event, data, pebs->tsc);
- perf_sample_save_brstack(data, event, &cpuc->lbr_stack, NULL);
+ perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
}
static void adaptive_pebs_save_regs(struct pt_regs *regs,
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index fe8b303839d3..15d779f92553 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -200,7 +200,7 @@ void intel_pmu_lbr_reset(void)
x86_pmu.lbr_reset();
- cpuc->lbr_stack.nr = 0;
+ cpu_lbr_stack(cpuc)->nr = 0;
cpuc->last_task_ctx = NULL;
cpuc->last_log_id = 0;
if (!cpu_feature_enabled(X86_FEATURE_ARCH_LBR) && cpuc->lbr_select)
@@ -743,10 +743,10 @@ void intel_pmu_lbr_disable_all(void)
void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
{
unsigned long mask = x86_pmu.lbr_nr - 1;
- struct perf_branch_entry *br = cpuc->lbr_entries;
u64 tos = intel_pmu_lbr_tos();
int i;
+ cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
for (i = 0; i < x86_pmu.lbr_nr; i++) {
unsigned long lbr_idx = (tos - i) & mask;
union {
@@ -759,14 +759,13 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
rdmsrq(x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr);
- *br = (struct perf_branch_entry){
+ cpu_lbr_stack(cpuc)->entries[i] = (struct perf_branch_entry){
.from = msr_lastbranch.from,
.to = msr_lastbranch.to,
};
- br++;
}
- cpuc->lbr_stack.nr = i;
- cpuc->lbr_stack.hw_idx = tos;
+ cpu_lbr_stack(cpuc)->nr = i;
+ cpu_lbr_stack(cpuc)->hw_idx = tos;
}
/*
@@ -778,7 +777,6 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
{
bool need_info = false, call_stack = false;
unsigned long mask = x86_pmu.lbr_nr - 1;
- struct perf_branch_entry *br = cpuc->lbr_entries;
u64 tos = intel_pmu_lbr_tos();
int i;
int out = 0;
@@ -790,6 +788,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
call_stack = true;
}
+ cpu_init_lbr_stack(cpuc, num);
for (i = 0; i < num; i++) {
unsigned long lbr_idx = (tos - i) & mask;
u64 from, to, mis = 0, pred = 0, in_tx = 0, abort = 0;
@@ -850,7 +849,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
if (abort && x86_pmu.lbr_double_abort && out > 0)
out--;
- br[out] = (struct perf_branch_entry){
+ cpu_lbr_stack(cpuc)->entries[out] = (struct perf_branch_entry){
.from = from,
.to = to,
.mispred = mis,
@@ -861,8 +860,8 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
};
out++;
}
- cpuc->lbr_stack.nr = out;
- cpuc->lbr_stack.hw_idx = tos;
+ cpu_lbr_stack(cpuc)->nr = out;
+ cpu_lbr_stack(cpuc)->hw_idx = tos;
}
static DEFINE_STATIC_KEY_FALSE(x86_lbr_mispred);
@@ -906,15 +905,14 @@ static_assert((64 - PERF_BRANCH_ENTRY_INFO_BITS_MAX) > LBR_INFO_BR_CNTR_NUM * LB
static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
struct lbr_entry *entries)
{
- struct perf_branch_entry *e;
struct lbr_entry *lbr;
u64 from, to, info;
bool mispred;
int i;
+ cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
for (i = 0; i < x86_pmu.lbr_nr; i++) {
lbr = entries ? &entries[i] : NULL;
- e = &cpuc->lbr_entries[i];
from = rdlbr_from(i, lbr);
/*
@@ -928,7 +926,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
mispred = get_lbr_mispred(info);
- *e = (struct perf_branch_entry){
+ cpu_lbr_stack(cpuc)->entries[i] = (struct perf_branch_entry){
.from = from,
.to = to,
.mispred = mispred,
@@ -938,18 +936,18 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
.cycles = get_lbr_cycles(info),
.type = get_lbr_br_type(info),
/*
- * Leverage the reserved field of
- * cpuc->lbr_entries[i] to temporarily store the
- * branch counters information. The later code will
- * decide what content can be disclosed to the perf
- * tool. Pleae see intel_pmu_lbr_counters_reorder().
+ * Leverage the reserved field of LBR entries to
+ * temporarily store the branch counters information.
+ * The later code will decide what content can be
+ * disclosed to the perf tool.
+ * Please see intel_pmu_lbr_counters_reorder().
*/
.reserved = (info >> LBR_INFO_BR_CNTR_OFFSET) &
LBR_INFO_BR_CNTR_FULL_MASK,
};
}
- cpuc->lbr_stack.nr = i;
+ cpu_lbr_stack(cpuc)->nr = i;
}
/*
@@ -975,15 +973,15 @@ static void intel_pmu_lbr_counters_reorder(struct cpu_hw_events *cpuc,
WARN_ON_ONCE(!pos);
- for (i = 0; i < cpuc->lbr_stack.nr; i++) {
- src = cpuc->lbr_entries[i].reserved;
+ for (i = 0; i < cpu_lbr_stack(cpuc)->nr; i++) {
+ src = cpu_lbr_stack(cpuc)->entries[i].reserved;
dst = 0;
for (j = 0; j < pos; j++) {
cnt = (src >> (order[j] * LBR_INFO_BR_CNTR_BITS)) & LBR_INFO_BR_CNTR_MASK;
dst |= cnt << j * LBR_INFO_BR_CNTR_BITS;
}
cpuc->lbr_counters[i] = dst;
- cpuc->lbr_entries[i].reserved = 0;
+ cpu_lbr_stack(cpuc)->entries[i].reserved = 0;
}
}
@@ -993,11 +991,11 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
{
if (is_branch_counters_group(event)) {
intel_pmu_lbr_counters_reorder(cpuc, event);
- perf_sample_save_brstack(data, event, &cpuc->lbr_stack, cpuc->lbr_counters);
+ perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), cpuc->lbr_counters);
return;
}
- perf_sample_save_brstack(data, event, &cpuc->lbr_stack, NULL);
+ perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
}
static void intel_pmu_arch_lbr_read(struct cpu_hw_events *cpuc)
@@ -1020,7 +1018,7 @@ static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)
intel_pmu_store_lbr(cpuc, xsave->lbr.entries);
} else {
/* Clear the LBR stack to avoid leaking stale entries. */
- cpuc->lbr_stack.nr = 0;
+ cpu_lbr_stack(cpuc)->nr = 0;
}
}
@@ -1030,7 +1028,7 @@ void intel_pmu_lbr_read(void)
if (!cpuc->lbr_users || vlbr_exclude_host()) {
/* Reset lbr_stack.nr to prevent stale data leakage. */
- cpuc->lbr_stack.nr = 0;
+ cpu_lbr_stack(cpuc)->nr = 0;
return;
}
@@ -1230,21 +1228,23 @@ static const int arch_lbr_br_type_map[ARCH_LBR_BR_TYPE_MAP_MAX] = {
static void
intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
{
- u64 from, to;
int br_sel = cpuc->br_sel;
int i, j, type, from_plm, to_plm;
bool compress = false;
+ u64 from, to;
+ int lbr_nr;
/* if sampling all branches, then nothing to filter */
if (((br_sel & X86_BR_ALL) == X86_BR_ALL) &&
((br_sel & X86_BR_TYPE_SAVE) != X86_BR_TYPE_SAVE))
return;
- for (i = 0; i < cpuc->lbr_stack.nr; i++) {
+ for (i = 0; i < cpu_lbr_stack(cpuc)->nr; i++) {
+ struct perf_branch_entry *be = &cpu_lbr_stack(cpuc)->entries[i];
- from = cpuc->lbr_entries[i].from;
- to = cpuc->lbr_entries[i].to;
- type = cpuc->lbr_entries[i].type;
+ from = be->from;
+ to = be->to;
+ type = be->type;
/*
* Parse the branch type recorded in LBR_x_INFO MSR.
@@ -1256,9 +1256,9 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
to_plm = kernel_ip(to) ? X86_BR_KERNEL : X86_BR_USER;
type = arch_lbr_br_type_map[type] | to_plm;
} else
- type = branch_type(from, to, cpuc->lbr_entries[i].abort);
+ type = branch_type(from, to, be->abort);
if (type != X86_BR_NONE && (br_sel & X86_BR_ANYTX)) {
- if (cpuc->lbr_entries[i].in_tx)
+ if (be->in_tx)
type |= X86_BR_IN_TX;
else
type |= X86_BR_NO_TX;
@@ -1272,31 +1272,34 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
*/
if (type == X86_BR_NONE || (br_sel & type) != type ||
(!(br_sel & X86_BR_KERNEL) && (from_plm & X86_BR_KERNEL))) {
- cpuc->lbr_entries[i].from = 0;
+ be->from = 0;
compress = true;
}
if ((br_sel & X86_BR_TYPE_SAVE) == X86_BR_TYPE_SAVE)
- cpuc->lbr_entries[i].type = common_branch_type(type);
+ be->type = common_branch_type(type);
}
if (!compress)
return;
/* remove all entries with from=0 */
- for (i = 0; i < cpuc->lbr_stack.nr; ) {
- if (!cpuc->lbr_entries[i].from) {
+ lbr_nr = cpu_lbr_stack(cpuc)->nr;
+ for (i = 0; i < lbr_nr; ) {
+ if (!cpu_lbr_stack(cpuc)->entries[i].from) {
j = i;
- while (++j < cpuc->lbr_stack.nr) {
- cpuc->lbr_entries[j-1] = cpuc->lbr_entries[j];
+ while (++j < lbr_nr) {
+ cpu_lbr_stack(cpuc)->entries[j-1] =
+ cpu_lbr_stack(cpuc)->entries[j];
cpuc->lbr_counters[j-1] = cpuc->lbr_counters[j];
}
- cpuc->lbr_stack.nr--;
- if (!cpuc->lbr_entries[i].from)
+ lbr_nr--;
+ if (!cpu_lbr_stack(cpuc)->entries[i].from)
continue;
}
i++;
}
+ cpu_lbr_stack(cpuc)->nr = lbr_nr;
}
void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
@@ -1306,9 +1309,9 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
/* Cannot get TOS for large PEBS and Arch LBR */
if (cpu_feature_enabled(X86_FEATURE_ARCH_LBR) ||
(cpuc->n_pebs == cpuc->n_large_pebs))
- cpuc->lbr_stack.hw_idx = -1ULL;
+ cpu_lbr_stack(cpuc)->hw_idx = -1ULL;
else
- cpuc->lbr_stack.hw_idx = intel_pmu_lbr_tos();
+ cpu_lbr_stack(cpuc)->hw_idx = intel_pmu_lbr_tos();
intel_pmu_store_lbr(cpuc, lbr);
intel_pmu_lbr_filter(cpuc);
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index a4aab8130e92..e3d625a6d7eb 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -447,6 +447,8 @@ struct intel_excl_cntrs {
struct x86_perf_task_context;
#define MAX_LBR_ENTRIES 32
+#define LBR_STACK_SIZE struct_size_t(struct perf_branch_stack, entries, MAX_LBR_ENTRIES)
+
enum {
LBR_FORMAT_32 = 0x00,
LBR_FORMAT_LIP = 0x01,
@@ -525,8 +527,7 @@ struct cpu_hw_events {
*/
int lbr_users;
int lbr_pebs_users;
- struct perf_branch_stack lbr_stack;
- struct perf_branch_entry lbr_entries[MAX_LBR_ENTRIES];
+ u64 __lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
u64 lbr_counters[MAX_LBR_ENTRIES]; /* branch stack extra */
union {
struct er_account *lbr_sel;
@@ -588,6 +589,20 @@ struct cpu_hw_events {
struct pmu *pmu;
};
+static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_events *cpuc)
+{
+ return (struct perf_branch_stack *)cpuc->__lbr_stack;
+}
+
+static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr)
+{
+ /*
+ * Initialize the branch-stack count before populating entries[]
+ * to avoid UBSAN complaints from the __counted_by(nr) flexible array.
+ */
+ cpu_lbr_stack(cpuc)->nr = nr;
+}
+
#define __EVENT_CONSTRAINT_RANGE(c, e, n, m, w, o, f) { \
{ .idxmsk64 = (n) }, \
.code = (c), \
diff --git a/drivers/perf/arm_brbe.c b/drivers/perf/arm_brbe.c
index 073d14584741..e2b32b239de1 100644
--- a/drivers/perf/arm_brbe.c
+++ b/drivers/perf/arm_brbe.c
@@ -763,6 +763,11 @@ void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack,
prepare_event_branch_type_mask(branch_sample_type, event_type_mask);
+ /*
+ * Initialize the branch_stack->nr before populating entries[]
+ * to avoid UBSAN complaints from the __counted_by(nr) flexible array.
+ */
+ branch_stack->nr = nr_hw;
for (int bank = 0; bank < nr_banks; bank++) {
int nr_remaining = nr_hw - (bank * BRBE_BANK_MAX_ENTRIES);
int nr_this_bank = min(nr_remaining, BRBE_BANK_MAX_ENTRIES);
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index eb3f7bab498c..19de3f1cbd6e 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -115,7 +115,7 @@ static __always_inline bool perf_raw_frag_last(const struct perf_raw_frag *frag)
struct perf_branch_stack {
u64 nr;
u64 hw_idx;
- struct perf_branch_entry entries[];
+ struct perf_branch_entry entries[] __counted_by(nr);
};
struct task_struct;
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* Re: [Patch v7 10/24] perf: Add counted-bounds tracking for perf_branch_stack
2026-11-18 3:14 ` [Patch v7 10/24] perf: Add counted-bounds tracking for perf_branch_stack Dapeng Mi
@ 2026-11-18 3:35 ` sashiko-bot
0 siblings, 0 replies; 49+ messages in thread
From: sashiko-bot @ 2026-11-18 3:35 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users
> struct perf_branch_stack uses a trailing flexible array member,
> entries[], whose valid length is tracked by nr. When this structure
> is embedded as a value member in cpu_hw_events, the compiler can treat
> the trailing flexible array as having a fixed zero-length layout under
> -fstrict-flex-arrays=3 and UBSan, which can lead to strict array-bounds
> diagnostics when entries[] is indexed.
>
> Annotate entries[] with __counted_by(nr) so the compiler and sanitizer
> track the array length from the nr field. Because a structure containing
> a counted flexible array cannot be embedded as a value member, switch
> the per-CPU LBR/BHRB stacks to raw backing arrays and add accessors that
> return the struct perf_branch_stack view.
>
> Also initialize the branch-stack count before populating entries[] to
> avoid false-positive bounds checks while the stack is being filled.
>
> No functional change intended.
>
> Suggested-by: Peter Zijlstra <peterz@infradead.org>
> Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261118031453.1353625-1-dapeng1.mi@linux.intel.com?part=10
^ permalink raw reply [flat|nested] 49+ messages in thread
* [Patch v7 11/24] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (9 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 10/24] perf: Add counted-bounds tracking for perf_branch_stack Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:32 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 12/24] perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting Dapeng Mi
` (12 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
The LBR read helpers currently assume the default stack,
cpuc->__lbr_stack, and therefore cannot be reused for other LBR
stacks that may be in use. Extend the *_lbr_read() helpers to take
an explicit LBR stack argument, so they can operate on any LBR stack
instead of only the default one.
Do the same for intel_pmu_store_lbr() and intel_pmu_lbr_filter(), and
pass the LBR counters array explicitly as well, since it is filtered
together with the stack.
No functional change intended.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/amd/brs.c | 2 +-
arch/x86/events/amd/lbr.c | 2 +-
arch/x86/events/intel/core.c | 4 +-
arch/x86/events/intel/lbr.c | 89 ++++++++++++++++++++----------------
arch/x86/events/perf_event.h | 16 ++++---
5 files changed, 64 insertions(+), 49 deletions(-)
diff --git a/arch/x86/events/amd/brs.c b/arch/x86/events/amd/brs.c
index 1906d1257646..475e3699d3ca 100644
--- a/arch/x86/events/amd/brs.c
+++ b/arch/x86/events/amd/brs.c
@@ -316,7 +316,7 @@ void amd_brs_drain(void)
num = tos - start + 1;
- cpu_init_lbr_stack(cpuc, num);
+ cpu_init_lbr_stack(cpu_lbr_stack(cpuc), num);
/*
* BRS is only one pass (saturation) from MSROFF to depth-1
diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c
index 3e4e5e984649..95898a1d9901 100644
--- a/arch/x86/events/amd/lbr.c
+++ b/arch/x86/events/amd/lbr.c
@@ -179,7 +179,7 @@ void amd_pmu_lbr_read(void)
return;
}
- cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
+ cpu_init_lbr_stack(cpu_lbr_stack(cpuc), x86_pmu.lbr_nr);
for (i = 0; i < x86_pmu.lbr_nr; i++) {
entry.from.full = amd_pmu_lbr_get_from(i);
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 6547555192cf..357c8eb7ee08 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2826,7 +2826,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
- intel_pmu_lbr_read();
+ intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr);
memcpy(entries, cpu_lbr_stack(cpuc)->entries,
@@ -4063,7 +4063,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
loops = 0;
again:
- intel_pmu_lbr_read();
+ intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
/*
* Keep the LBR state frozen throughout PMI handling so that the PMI
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 15d779f92553..b60719adcc1c 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -105,7 +105,9 @@
#define ARCH_LBR_CTL_MASK 0x7f000e
-static void intel_pmu_lbr_filter(struct cpu_hw_events *cpuc);
+static void intel_pmu_lbr_filter(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack,
+ u64 *lbr_counters);
static __always_inline bool is_lbr_call_stack_bit_set(u64 config)
{
@@ -740,13 +742,14 @@ void intel_pmu_lbr_disable_all(void)
}
}
-void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
+void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack)
{
unsigned long mask = x86_pmu.lbr_nr - 1;
u64 tos = intel_pmu_lbr_tos();
int i;
- cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
+ cpu_init_lbr_stack(lbr_stack, x86_pmu.lbr_nr);
for (i = 0; i < x86_pmu.lbr_nr; i++) {
unsigned long lbr_idx = (tos - i) & mask;
union {
@@ -759,13 +762,13 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
rdmsrq(x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr);
- cpu_lbr_stack(cpuc)->entries[i] = (struct perf_branch_entry){
+ lbr_stack->entries[i] = (struct perf_branch_entry){
.from = msr_lastbranch.from,
.to = msr_lastbranch.to,
};
}
- cpu_lbr_stack(cpuc)->nr = i;
- cpu_lbr_stack(cpuc)->hw_idx = tos;
+ lbr_stack->nr = i;
+ lbr_stack->hw_idx = tos;
}
/*
@@ -773,7 +776,8 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
* is the same as the linear address, allowing us to merge the LIP and EIP
* LBR formats.
*/
-void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
+void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack)
{
bool need_info = false, call_stack = false;
unsigned long mask = x86_pmu.lbr_nr - 1;
@@ -788,7 +792,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
call_stack = true;
}
- cpu_init_lbr_stack(cpuc, num);
+ cpu_init_lbr_stack(lbr_stack, num);
for (i = 0; i < num; i++) {
unsigned long lbr_idx = (tos - i) & mask;
u64 from, to, mis = 0, pred = 0, in_tx = 0, abort = 0;
@@ -849,7 +853,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
if (abort && x86_pmu.lbr_double_abort && out > 0)
out--;
- cpu_lbr_stack(cpuc)->entries[out] = (struct perf_branch_entry){
+ lbr_stack->entries[out] = (struct perf_branch_entry){
.from = from,
.to = to,
.mispred = mis,
@@ -860,8 +864,8 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
};
out++;
}
- cpu_lbr_stack(cpuc)->nr = out;
- cpu_lbr_stack(cpuc)->hw_idx = tos;
+ lbr_stack->nr = out;
+ lbr_stack->hw_idx = tos;
}
static DEFINE_STATIC_KEY_FALSE(x86_lbr_mispred);
@@ -902,17 +906,19 @@ static __always_inline u16 get_lbr_cycles(u64 info)
static_assert((64 - PERF_BRANCH_ENTRY_INFO_BITS_MAX) > LBR_INFO_BR_CNTR_NUM * LBR_INFO_BR_CNTR_BITS);
-static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
- struct lbr_entry *entries)
+static void intel_pmu_store_lbr(struct lbr_entry *entries,
+ struct perf_branch_stack *lbr_stack)
{
+ struct perf_branch_entry *e;
struct lbr_entry *lbr;
u64 from, to, info;
bool mispred;
int i;
- cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
+ cpu_init_lbr_stack(lbr_stack, x86_pmu.lbr_nr);
for (i = 0; i < x86_pmu.lbr_nr; i++) {
lbr = entries ? &entries[i] : NULL;
+ e = &lbr_stack->entries[i];
from = rdlbr_from(i, lbr);
/*
@@ -926,7 +932,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
mispred = get_lbr_mispred(info);
- cpu_lbr_stack(cpuc)->entries[i] = (struct perf_branch_entry){
+ *e = (struct perf_branch_entry){
.from = from,
.to = to,
.mispred = mispred,
@@ -947,7 +953,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
};
}
- cpu_lbr_stack(cpuc)->nr = i;
+ lbr_stack->nr = i;
}
/*
@@ -998,37 +1004,41 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
}
-static void intel_pmu_arch_lbr_read(struct cpu_hw_events *cpuc)
+static void intel_pmu_arch_lbr_read(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack)
{
- intel_pmu_store_lbr(cpuc, NULL);
+ intel_pmu_store_lbr(NULL, lbr_stack);
}
-static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)
+static void
+intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack)
{
struct x86_perf_task_context_arch_lbr_xsave *xsave = cpuc->lbr_xsave;
if (!xsave) {
- intel_pmu_store_lbr(cpuc, NULL);
+ intel_pmu_store_lbr(NULL, lbr_stack);
return;
}
xsave->header.xfeatures = 0;
xsaves(&xsave->xsave, XFEATURE_MASK_LBR);
if (xsave->header.xfeatures & XFEATURE_MASK_LBR) {
- intel_pmu_store_lbr(cpuc, xsave->lbr.entries);
+ intel_pmu_store_lbr(xsave->lbr.entries, lbr_stack);
} else {
/* Clear the LBR stack to avoid leaking stale entries. */
- cpu_lbr_stack(cpuc)->nr = 0;
+ lbr_stack->nr = 0;
}
}
-void intel_pmu_lbr_read(void)
+void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,
+ u64 *lbr_counters)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
if (!cpuc->lbr_users || vlbr_exclude_host()) {
- /* Reset lbr_stack.nr to prevent stale data leakage. */
- cpu_lbr_stack(cpuc)->nr = 0;
+ /* Reset lbr_stack->nr to prevent stale data leakage. */
+ lbr_stack->nr = 0;
return;
}
@@ -1041,9 +1051,9 @@ void intel_pmu_lbr_read(void)
if (cpuc->lbr_users == cpuc->lbr_pebs_users)
return;
- x86_pmu.lbr_read(cpuc);
+ x86_pmu.lbr_read(cpuc, lbr_stack);
- intel_pmu_lbr_filter(cpuc);
+ intel_pmu_lbr_filter(cpuc, lbr_stack, lbr_counters);
}
/*
@@ -1226,7 +1236,9 @@ static const int arch_lbr_br_type_map[ARCH_LBR_BR_TYPE_MAP_MAX] = {
* in PERF_SAMPLE_BRANCH_STACK sample may vary.
*/
static void
-intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
+intel_pmu_lbr_filter(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack,
+ u64 *lbr_counters)
{
int br_sel = cpuc->br_sel;
int i, j, type, from_plm, to_plm;
@@ -1239,8 +1251,8 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
((br_sel & X86_BR_TYPE_SAVE) != X86_BR_TYPE_SAVE))
return;
- for (i = 0; i < cpu_lbr_stack(cpuc)->nr; i++) {
- struct perf_branch_entry *be = &cpu_lbr_stack(cpuc)->entries[i];
+ for (i = 0; i < lbr_stack->nr; i++) {
+ struct perf_branch_entry *be = &lbr_stack->entries[i];
from = be->from;
to = be->to;
@@ -1284,22 +1296,21 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)
return;
/* remove all entries with from=0 */
- lbr_nr = cpu_lbr_stack(cpuc)->nr;
+ lbr_nr = lbr_stack->nr;
for (i = 0; i < lbr_nr; ) {
- if (!cpu_lbr_stack(cpuc)->entries[i].from) {
+ if (!lbr_stack->entries[i].from) {
j = i;
while (++j < lbr_nr) {
- cpu_lbr_stack(cpuc)->entries[j-1] =
- cpu_lbr_stack(cpuc)->entries[j];
- cpuc->lbr_counters[j-1] = cpuc->lbr_counters[j];
+ lbr_stack->entries[j-1] = lbr_stack->entries[j];
+ lbr_counters[j-1] = lbr_counters[j];
}
lbr_nr--;
- if (!cpu_lbr_stack(cpuc)->entries[i].from)
+ if (!lbr_stack->entries[i].from)
continue;
}
i++;
}
- cpu_lbr_stack(cpuc)->nr = lbr_nr;
+ lbr_stack->nr = lbr_nr;
}
void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
@@ -1313,8 +1324,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
else
cpu_lbr_stack(cpuc)->hw_idx = intel_pmu_lbr_tos();
- intel_pmu_store_lbr(cpuc, lbr);
- intel_pmu_lbr_filter(cpuc);
+ intel_pmu_store_lbr(lbr, cpu_lbr_stack(cpuc));
+ intel_pmu_lbr_filter(cpuc, cpu_lbr_stack(cpuc), cpuc->lbr_counters);
}
/*
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index e3d625a6d7eb..fa0af528cd1e 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -594,13 +594,13 @@ static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_eve
return (struct perf_branch_stack *)cpuc->__lbr_stack;
}
-static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr)
+static __always_inline void cpu_init_lbr_stack(struct perf_branch_stack *lbr_stack, int nr)
{
/*
* Initialize the branch-stack count before populating entries[]
* to avoid UBSAN complaints from the __counted_by(nr) flexible array.
*/
- cpu_lbr_stack(cpuc)->nr = nr;
+ lbr_stack->nr = nr;
}
#define __EVENT_CONSTRAINT_RANGE(c, e, n, m, w, o, f) { \
@@ -1207,7 +1207,8 @@ struct x86_pmu {
unsigned int lbr_counters:4;
void (*lbr_reset)(void);
- void (*lbr_read)(struct cpu_hw_events *cpuc);
+ void (*lbr_read)(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack);
void (*lbr_save)(void *ctx);
void (*lbr_restore)(void *ctx);
@@ -2007,11 +2008,14 @@ void intel_pmu_lbr_enable_all(bool pmi);
void intel_pmu_lbr_disable_all(void);
-void intel_pmu_lbr_read(void);
+void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,
+ u64 *lbr_counters);
-void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc);
+void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack);
-void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc);
+void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc,
+ struct perf_branch_stack *lbr_stack);
void intel_pmu_lbr_save(void *ctx);
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [Patch v7 12/24] perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (10 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 11/24] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:43 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 13/24] perf/x86/intel: Always snapshot live hardware LBR state for BPF Dapeng Mi
` (11 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
BPF LBR snapshotting currently reuses the default LBR stack
(cpuc->__lbr_stack) when storing sampled data. This is unsafe when the
BPF helper runs in PMI context via bpf_overflow_handler(): it can
overwrite the cached LBR state for active PEBS LBR events, causing
incorrect LBR data to be reported to userspace. When the PEBS and BPF
stacks have different depths, this can also lead to out-of-bounds reads
when the LBR stack is later exported.
Use a dedicated LBR stack for BPF snapshotting so the BPF path no
longer clobbers the shared LBR state used by other perf events.
Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 15 ++++++++++++---
1 file changed, 12 insertions(+), 3 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 357c8eb7ee08..b560f2e96848 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2819,17 +2819,26 @@ static void intel_pmu_enable_all(int added)
__intel_pmu_enable_all(false);
}
+struct lbr_snapshot {
+ u64 __lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
+ u64 counters[MAX_LBR_ENTRIES];
+};
+static DEFINE_PER_CPU(struct lbr_snapshot, lbr_snapshot);
+
static noinline int
__intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
unsigned int cnt, unsigned long flags,
int pmu_enabled, int lbr_enabled)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+ struct lbr_snapshot *bpf = this_cpu_ptr(&lbr_snapshot);
+ struct perf_branch_stack *lbr_stack;
- intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
- cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr);
+ lbr_stack = (struct perf_branch_stack *)bpf->__lbr_stack;
+ intel_pmu_lbr_read(lbr_stack, bpf->counters);
+ cnt = min_t(unsigned int, cnt, lbr_stack->nr);
- memcpy(entries, cpu_lbr_stack(cpuc)->entries,
+ memcpy(entries, lbr_stack->entries,
sizeof(struct perf_branch_entry) * cnt);
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [Patch v7 13/24] perf/x86/intel: Always snapshot live hardware LBR state for BPF
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (11 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 12/24] perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:46 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 14/24] perf/x86/intel: Account lbr_pebs_users for arch-PEBS events Dapeng Mi
` (10 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
__intel_pmu_snapshot_branch_stack() is intended to capture the current
live hardware LBR state for the BPF path. However, that snapshot is
currently skipped when all LBR users are PEBS events.
BPF LBR snapshotting now uses its own dedicated LBR stack, so there is
no longer a risk of clobbering the shared default LBR stack. Move the
lbr_pebs_users check into the PMI handler, which is the only caller that
relies on it, so the BPF path always snapshots the live hardware LBR
state.
Fixes: c22ac2a3d4bd ("perf: Enable branch record for software events")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 7 ++++++-
arch/x86/events/intel/lbr.c | 9 ---------
2 files changed, 6 insertions(+), 10 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index b560f2e96848..84b134bb91ee 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -4072,7 +4072,12 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
loops = 0;
again:
- intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
+ /*
+ * No need to read the LBR stack when all LBR users are PEBS events,
+ * since the PEBS handler updates it.
+ */
+ if (cpuc->lbr_users != cpuc->lbr_pebs_users)
+ intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
/*
* Keep the LBR state frozen throughout PMI handling so that the PMI
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index b60719adcc1c..6c7123c1f9ae 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -1042,15 +1042,6 @@ void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,
return;
}
- /*
- * Don't read when all LBRs users are using adaptive PEBS.
- *
- * This could be smarter and actually check the event,
- * but this simple approach seems to work for now.
- */
- if (cpuc->lbr_users == cpuc->lbr_pebs_users)
- return;
-
x86_pmu.lbr_read(cpuc, lbr_stack);
intel_pmu_lbr_filter(cpuc, lbr_stack, lbr_counters);
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [Patch v7 14/24] perf/x86/intel: Account lbr_pebs_users for arch-PEBS events
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (12 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 13/24] perf/x86/intel: Always snapshot live hardware LBR state for BPF Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:34 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 15/24] perf/x86/intel: Set PERF_EV_CAP_SIBLING for pure branch counter events Dapeng Mi
` (9 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
lbr_pebs_users tracks how many PEBS events are currently using LBR.
It is updated for legacy PEBS events, but arch-PEBS events were missing
from that accounting.
Add the missing lbr_pebs_users accounting for arch-PEBS as well.
Fixes: d243d0bb64af ("perf/x86/intel: Initialize architectural PEBS")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-179823@kernel.org/
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/lbr.c | 4 ++--
arch/x86/events/perf_event.h | 7 +++++++
2 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 6c7123c1f9ae..3a282352b0d2 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -616,7 +616,7 @@ void intel_pmu_lbr_add(struct perf_event *event)
* be 'new'. Conversely, a new event can get installed through the
* context switch path for the first time.
*/
- if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip > 0)
+ if (intel_pmu_has_pebs_lbr(event->pmu) && event->attr.precise_ip > 0)
cpuc->lbr_pebs_users++;
perf_sched_cb_inc(event->pmu);
if (!cpuc->lbr_users++ && !event->total_time_running)
@@ -687,7 +687,7 @@ void intel_pmu_lbr_del(struct perf_event *event)
if (event->hw.flags & PERF_X86_EVENT_LBR_SELECT)
cpuc->lbr_select = 0;
- if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip > 0)
+ if (intel_pmu_has_pebs_lbr(event->pmu) && event->attr.precise_ip > 0)
cpuc->lbr_pebs_users--;
cpuc->lbr_users--;
WARN_ON_ONCE(cpuc->lbr_users < 0);
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index fa0af528cd1e..8d64bfcdc82d 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -2096,6 +2096,13 @@ static inline bool intel_pmu_has_pebs(void)
return x86_pmu.ds_pebs || x86_pmu.arch_pebs;
}
+static inline bool intel_pmu_has_pebs_lbr(struct pmu *pmu)
+{
+ if (x86_pmu.arch_pebs)
+ return hybrid(pmu, arch_pebs_cap).caps & ARCH_PEBS_LBR;
+ return x86_pmu.intel_cap.pebs_baseline;
+}
+
#else /* CONFIG_CPU_SUP_INTEL */
static inline void reserve_ds_buffers(void)
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [Patch v7 15/24] perf/x86/intel: Set PERF_EV_CAP_SIBLING for pure branch counter events
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (13 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 14/24] perf/x86/intel: Account lbr_pebs_users for arch-PEBS events Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 4:05 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 16/24] perf/x86/intel: Account LBR user counts " Dapeng Mi
` (8 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
A pure branch counter event (i.e. an event which only applies
PERF_SAMPLE_BRANCH_COUNTERS without requiring the branch stack itself)
doesn't set up the LBR on its own. It relies on being grouped with a
normal branch event, which configures the LBR for it. Such an event
cannot work standalone.
Set the PERF_EV_CAP_SIBLING flag for pure branch counter events, so
that the generic perf core schedules them out once the group leader is
removed from the group.
Opportunistically refine the code style.
Fixes: 33744916196b ("perf/x86/intel: Support branch counters logging")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 84b134bb91ee..fd4de141ef28 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -5130,10 +5130,17 @@ static int intel_pmu_hw_config(struct perf_event *event)
* require any branch stack setup.
* Clear the bit to avoid unnecessary branch stack setup.
*/
- if (0 == (event->attr.branch_sample_type &
- ~(PERF_SAMPLE_BRANCH_PLM_ALL |
- PERF_SAMPLE_BRANCH_COUNTERS)))
- event->hw.flags &= ~PERF_X86_EVENT_NEEDS_BRANCH_STACK;
+ if (!(event->attr.branch_sample_type &
+ ~(PERF_SAMPLE_BRANCH_PLM_ALL | PERF_SAMPLE_BRANCH_COUNTERS))) {
+ event->hw.flags &= ~PERF_X86_EVENT_NEEDS_BRANCH_STACK;
+ /*
+ * A pure branch counter event doesn't set up the LBR
+ * itself. It must stay grouped with a normal branch
+ * event which configures the LBR for it. Otherwise,
+ * the LBR cannot be configured correctly.
+ */
+ event->event_caps |= PERF_EV_CAP_SIBLING;
+ }
/*
* Force the leader to be a LBR event. So LBRs can be reset
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* Re: [Patch v7 15/24] perf/x86/intel: Set PERF_EV_CAP_SIBLING for pure branch counter events
2026-11-18 3:14 ` [Patch v7 15/24] perf/x86/intel: Set PERF_EV_CAP_SIBLING for pure branch counter events Dapeng Mi
@ 2026-11-18 4:05 ` sashiko-bot
0 siblings, 0 replies; 49+ messages in thread
From: sashiko-bot @ 2026-11-18 4:05 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users, bpf
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] perf/x86/intel: incomplete group detach handling in
intel_pmu_hw_config()
--
commit 2f6a2a62d0ff0bf2794f702072c776c06e629ff3
Author: Dapeng Mi <dapeng1.mi@linux.intel.com>
perf/x86/intel: Set PERF_EV_CAP_SIBLING for pure branch counter events
This commit sets PERF_EV_CAP_SIBLING on pure branch counter events in
intel_pmu_hw_config() so that the perf core schedules them out if the
group leader is detached.
> diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
> index 84b134bb91eea..fd4de141ef281 100644
> --- a/arch/x86/events/intel/core.c
> +++ b/arch/x86/events/intel/core.c
> @@ -5130,10 +5130,17 @@ static int intel_pmu_hw_config(struct perf_event *event)
> * require any branch stack setup.
> * Clear the bit to avoid unnecessary branch stack setup.
> */
> - if (0 == (event->attr.branch_sample_type &
> - ~(PERF_SAMPLE_BRANCH_PLM_ALL |
> - PERF_SAMPLE_BRANCH_COUNTERS)))
> - event->hw.flags &= ~PERF_X86_EVENT_NEEDS_BRANCH_STACK;
> + if (!(event->attr.branch_sample_type &
> + ~(PERF_SAMPLE_BRANCH_PLM_ALL | PERF_SAMPLE_BRANCH_COUNTERS))) {
> + event->hw.flags &= ~PERF_X86_EVENT_NEEDS_BRANCH_STACK;
> + /*
> + * A pure branch counter event doesn't set up the LBR
> + * itself. It must stay grouped with a normal branch
> + * event which configures the LBR for it. Otherwise,
> + * the LBR cannot be configured correctly.
> + */
> + event->event_caps |= PERF_EV_CAP_SIBLING;
> + }
>
> /*
> * Force the leader to be a LBR event. So LBRs can be reset
[Severity: Medium]
What happens on group detach when a branch counter sibling has other
branch_sample_type filter bits set alongside PERF_SAMPLE_BRANCH_COUNTERS?
First, a counting sibling in PERF_SAMPLE_READ mode with a branch filter
(such as PERF_SAMPLE_BRANCH_ANY | PERF_SAMPLE_BRANCH_COUNTERS, which
"perf record -e '{e1,e2}:S' -j any,counter" configures) skips setting
PERF_X86_EVENT_NEEDS_BRANCH_STACK earlier in intel_pmu_hw_config():
arch/x86/events/intel/core.c:intel_pmu_hw_config() {
if (needs_branch_stack(event)) {
/* Avoid branch stack setup for counting events in SAMPLE READ */
if (is_sampling_event(event) ||
!(event->attr.sample_type & PERF_SAMPLE_READ))
event->hw.flags |= PERF_X86_EVENT_NEEDS_BRANCH_STACK;
}
}
Because branch_sample_type contains PERF_SAMPLE_BRANCH_ANY, the condition
here is false and PERF_EV_CAP_SIBLING is not set either. If the group
leader is closed, perf_group_detach() promotes this sibling to a
standalone leader that stays scheduled with ARCH_PERFMON_EVENTSEL_BR_CNTR
enabled in hardware without LBR configured.
Second, for a sampling sibling that retains
PERF_X86_EVENT_NEEDS_BRANCH_STACK and has PERF_SAMPLE_BRANCH_COUNTERS set,
intel_pmu_hw_config() only sets PERF_X86_EVENT_BRANCH_COUNTERS on
leader->hw.flags and not on event->hw.flags. When the original leader is
closed and perf_promote_sibling_to_leader() sets sibling->group_leader =
sibling, is_branch_counters_group(sibling) becomes false, so
intel_pmu_lbr_save_brstack() skips intel_pmu_lbr_counters_reorder() and
intel_pmu_lbr_del() skips intel_pmu_lbr_reset().
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261118031453.1353625-1-dapeng1.mi@linux.intel.com?part=15
^ permalink raw reply [flat|nested] 49+ messages in thread
* [Patch v7 16/24] perf/x86/intel: Account LBR user counts for pure branch counter events
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (14 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 15/24] perf/x86/intel: Set PERF_EV_CAP_SIBLING for pure branch counter events Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:49 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 17/24] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed Dapeng Mi
` (7 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
intel_pmu_hw_config() clears PERF_X86_EVENT_NEEDS_BRANCH_STACK for pure
branch counter events. As a result, intel_pmu_add_event() and
intel_pmu_del_event() do not account lbr_users and lbr_pebs_users
correctly for these events.
Add intel_pmu_account_pure_branch_counter() to adjust the LBR user counts
when pure branch counter events are added or removed.
Fixes: 33744916196b ("perf/x86/intel: Support branch counters logging")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 39 ++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index fd4de141ef28..5491cd4f4c6f 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -3168,10 +3168,47 @@ static __always_inline bool intel_pmu_needs_branch_stack(struct perf_event *even
return event->hw.flags & PERF_X86_EVENT_NEEDS_BRANCH_STACK;
}
+static inline bool is_pure_branch_counter(struct perf_event *event)
+{
+ return is_sampling_event(event) &&
+ has_branch_stack(event) &&
+ branch_sample_counters(event) &&
+ !intel_pmu_needs_branch_stack(event);
+}
+
+/*
+ * A pure branch counter event clears PERF_X86_EVENT_NEEDS_BRANCH_STACK,
+ * so intel_pmu_lbr_add()/intel_pmu_lbr_del() do not update lbr_users
+ * and lbr_pebs_users. Handle the accounting explicitly here instead.
+ * See intel_pmu_hw_config() for details.
+ */
+static void
+intel_pmu_account_pure_branch_counter(struct perf_event *event, bool add)
+{
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+
+ if (!x86_pmu.lbr_nr)
+ return;
+
+ if (add) {
+ cpuc->lbr_users++;
+ if (intel_pmu_has_pebs_lbr(event->pmu) && event->attr.precise_ip)
+ cpuc->lbr_pebs_users++;
+ } else {
+ cpuc->lbr_users--;
+ if (intel_pmu_has_pebs_lbr(event->pmu) && event->attr.precise_ip)
+ cpuc->lbr_pebs_users--;
+ WARN_ON_ONCE(cpuc->lbr_users < 0);
+ WARN_ON_ONCE(cpuc->lbr_pebs_users < 0);
+ }
+}
+
static void intel_pmu_del_event(struct perf_event *event)
{
if (intel_pmu_needs_branch_stack(event))
intel_pmu_lbr_del(event);
+ if (is_pure_branch_counter(event))
+ intel_pmu_account_pure_branch_counter(event, false);
if (event->attr.precise_ip)
intel_pmu_pebs_del(event);
if (is_pebs_counter_event_group(event) ||
@@ -3734,6 +3771,8 @@ static void intel_pmu_add_event(struct perf_event *event)
intel_pmu_pebs_add(event);
if (intel_pmu_needs_branch_stack(event))
intel_pmu_lbr_add(event);
+ if (is_pure_branch_counter(event))
+ intel_pmu_account_pure_branch_counter(event, true);
if (is_pebs_counter_event_group(event) ||
is_acr_event_group(event))
this_cpu_ptr(&cpu_hw_events)->n_late_setup++;
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [Patch v7 17/24] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (15 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 16/24] perf/x86/intel: Account LBR user counts " Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:14 ` [Patch v7 18/24] perf/x86/intel: Fix draining of un-overflowed PEBS buffer in PMI handler Dapeng Mi
` (6 subsequent siblings)
23 siblings, 0 replies; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
PEBS and non-PEBS events share the same LBR stack in cpuc->__lbr_stack.
In the PMI handler, the non-PEBS path refreshes the LBR state before the
PEBS buffer is drained.
When PEBS and non-PEBS LBR events overflow in the same PMI, draining the
PEBS buffer can overwrite the shared LBR stack with the PEBS snapshot.
This leaves stale LBR data from the PEBS record in place, so userspace
can see the wrong branch stack instead of the live LBR state at PMI
entry.
Track whether the LBR stack still contains the live state from PMI entry
with a cpuc->lbr_stack_live flag. The flag starts out clear on each pass
of the PMI handler and is set once the live LBR state has been read.
Draining PEBS records with LBR data clears it again, so a non-PEBS LBR
event re-reads the stack only when it is needed. Since the LBR state is
frozen for the duration of PMI handling, it only needs to be read once
for multiple non-PEBS LBR events in the same PMI.
Fixes: c22497f5838c ("perf/x86/intel: Support adaptive PEBS v4")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 18 ++++++++++++++----
arch/x86/events/intel/lbr.c | 1 +
arch/x86/events/perf_event.h | 1 +
3 files changed, 16 insertions(+), 4 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 5491cd4f4c6f..aa7034c5ea41 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -4054,8 +4054,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
perf_sample_data_init(&data, 0, last_period);
- if (has_branch_stack(event))
+ if (has_branch_stack(event)) {
+ if (!cpuc->lbr_stack_live) {
+ intel_pmu_lbr_read(cpu_lbr_stack(cpuc),
+ cpuc->lbr_counters);
+ cpuc->lbr_stack_live = true;
+ }
intel_pmu_lbr_save_brstack(&data, cpuc, event);
+ }
x86_pmu_clear_perf_regs(regs);
x86_pmu_update_perf_regs(event, &data, regs, false);
@@ -4111,12 +4117,16 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
loops = 0;
again:
+ cpuc->lbr_stack_live = false;
/*
- * No need to read the LBR stack when all LBR users are PEBS events,
- * since the PEBS handler updates it.
+ * If PEBS can't record LBRs, draining PEBS can't overwrite the
+ * LBR stack, so read it here at PMI entry. Otherwise, it is read
+ * after PEBS draining when needed.
*/
- if (cpuc->lbr_users != cpuc->lbr_pebs_users)
+ if (cpuc->lbr_users > 0 && !intel_pmu_has_pebs_lbr(cpuc->pmu)) {
intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
+ cpuc->lbr_stack_live = true;
+ }
/*
* Keep the LBR state frozen throughout PMI handling so that the PMI
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 3a282352b0d2..740d5a5ba4c3 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -1317,6 +1317,7 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
intel_pmu_store_lbr(lbr, cpu_lbr_stack(cpuc));
intel_pmu_lbr_filter(cpuc, cpu_lbr_stack(cpuc), cpuc->lbr_counters);
+ cpuc->lbr_stack_live = false;
}
/*
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index 8d64bfcdc82d..4e6cf5c1e865 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -525,6 +525,7 @@ struct cpu_hw_events {
/*
* Intel LBR bits
*/
+ bool lbr_stack_live; /* If LBR stack holds the live LBR state */
int lbr_users;
int lbr_pebs_users;
u64 __lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [Patch v7 18/24] perf/x86/intel: Fix draining of un-overflowed PEBS buffer in PMI handler
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (16 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 17/24] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:46 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 19/24] perf/x86/intel: Drain PEBS buffer for branch events with PEBS LBR users Dapeng Mi
` (5 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
When a regular counter which is read by a PEBS counter-event group
overflows, handle_pmi_common() drains the pending PEBS records to
ensure the stale counter snapshots are processed before handling the
counter's overflow. Currently this is done by invoking the
x86_pmu_drain_pebs() static call with a non-NULL regs argument.
This is incorrect. At this point the PEBS buffer has not reached its
interrupt threshold yet (a real PEBS buffer overflow would have
already been handled earlier), and the PMI being serviced belongs to
the regular counter, not to the PEBS events. With a non-NULL regs,
the drained records are processed with perf_event_overflow(), which
charges a PMI to the PEBS events via __perf_event_account_interrupt().
These spurious interrupt counts may falsely trigger the PMI throttling
and unexpectedly disable the PEBS events.
Call intel_pmu_drain_pebs_buffer() instead. It passes a NULL regs to
x86_pmu_drain_pebs(), so the records are emitted with
perf_event_output(), which skips the overflow handling and throttle
accounting.
Fixes: e02e9b0374c3 ("perf/x86/intel: Support PEBS counters snapshotting")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 2 +-
arch/x86/events/intel/ds.c | 8 ++++++++
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index aa7034c5ea41..67275a873dba 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -4045,7 +4045,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
* The PEBS buffer has to be drained before handling the A-PMI
*/
if (is_pebs_counter_event_group(event))
- static_call(x86_pmu_drain_pebs)(regs, &data);
+ intel_pmu_drain_pebs_buffer();
last_period = event->hw.last_period;
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 332754e1da4f..9fe84e4880d9 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -1262,6 +1262,14 @@ int intel_pmu_drain_bts_buffer(void)
return 1;
}
+/*
+ * Drain the pending PEBS buffer for the current CPU.
+ * This helper should only be used to drain a PEBS buffer which
+ * hasn't overflowed yet, e.g. on a context switch with large PEBS,
+ * because the NULL regs argument passed to x86_pmu_drain_pebs()
+ * makes the records be processed with perf_event_output() rather
+ * than perf_event_overflow().
+ */
void intel_pmu_drain_pebs_buffer(void)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* Re: [Patch v7 18/24] perf/x86/intel: Fix draining of un-overflowed PEBS buffer in PMI handler
2026-11-18 3:14 ` [Patch v7 18/24] perf/x86/intel: Fix draining of un-overflowed PEBS buffer in PMI handler Dapeng Mi
@ 2026-11-18 3:46 ` sashiko-bot
0 siblings, 0 replies; 49+ messages in thread
From: sashiko-bot @ 2026-11-18 3:46 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users
> When a regular counter which is read by a PEBS counter-event group
> overflows, handle_pmi_common() drains the pending PEBS records to
> ensure the stale counter snapshots are processed before handling the
> counter's overflow. Currently this is done by invoking the
> x86_pmu_drain_pebs() static call with a non-NULL regs argument.
>
> This is incorrect. At this point the PEBS buffer has not reached its
> interrupt threshold yet (a real PEBS buffer overflow would have
> already been handled earlier), and the PMI being serviced belongs to
> the regular counter, not to the PEBS events. With a non-NULL regs,
> the drained records are processed with perf_event_overflow(), which
> charges a PMI to the PEBS events via __perf_event_account_interrupt().
> These spurious interrupt counts may falsely trigger the PMI throttling
> and unexpectedly disable the PEBS events.
>
> Call intel_pmu_drain_pebs_buffer() instead. It passes a NULL regs to
> x86_pmu_drain_pebs(), so the records are emitted with
> perf_event_output(), which skips the overflow handling and throttle
> accounting.
>
> Fixes: e02e9b0374c3 ("perf/x86/intel: Support PEBS counters snapshotting")
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261118031453.1353625-1-dapeng1.mi@linux.intel.com?part=18
^ permalink raw reply [flat|nested] 49+ messages in thread
* [Patch v7 19/24] perf/x86/intel: Drain PEBS buffer for branch events with PEBS LBR users
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (17 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 18/24] perf/x86/intel: Fix draining of un-overflowed PEBS buffer in PMI handler Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:52 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 20/24] perf/x86/intel: Clear raw LBR counters in the LBR stack Dapeng Mi
` (4 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
If an event needs a branch stack while PEBS LBR users are active, the
PEBS buffer may get drained before the event's sample is output. The
drain overwrites the live LBR snapshot read by the PMI handler, so the
stale PEBS LBR stack is reported to userspace instead.
This can happen, for example, when:
- The event belongs to a SAMPLE_READ group that includes a PEBS event
snapshotting the LBR stack, and a later perf_event_overflow() calls
perf_output_read_group(), which drains the PEBS buffer via
intel_pmu_read_event();
- The overflow triggers group throttling, disabling a PEBS LBR event and
causing intel_pmu_disable_event() to drain the PEBS buffer before the
live LBR snapshot is output.
Thus, drain the PEBS buffer whenever the overflowed event needs a branch
stack and PEBS LBR users are active.
Fixes: c22497f5838c ("perf/x86/intel: Support adaptive PEBS v4")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 54 ++++++++++++++++++++++++++----------
1 file changed, 39 insertions(+), 15 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 67275a873dba..ccd2c57931b3 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -4027,24 +4027,48 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
continue;
/*
- * There may be unprocessed PEBS records in the PEBS buffer,
- * which still stores the previous values.
- * Process those records first before handling the latest value.
- * For example,
- * A is a regular counter
- * B is a PEBS event which reads A
- * C is a PEBS event
+ * Drain any pending PEBS records before handling the overflow
+ * for the event in two cases:
*
- * The following can happen:
- * B-assist A=1
- * C A=2
- * B-assist A=3
- * A-overflow-PMI A=4
- * C-assist-PMI (PEBS buffer) A=5
+ * 1) The event belongs to a PEBS counter event group. Other
+ * PEBS events may record the previous count for this event.
+ * If those buffered PEBS records are processed only after
+ * intel_pmu_save_and_restart() updates the count, the counter
+ * value will be corrupted.
*
- * The PEBS buffer has to be drained before handling the A-PMI
+ * Example:
+ * A = regular counter
+ * B = PEBS event that reads A
+ * C = PEBS event
+ *
+ * Sequence:
+ * B-assist A=1
+ * C A=2
+ * B-assist A=3
+ * A-overflow-PMI A=4
+ * C-assist-PMI (PEBS) A=5
+ *
+ * Therefore, the PEBS buffer must be drained before handling
+ * the A overflow.
+ *
+ * 2) The event needs a branch stack, but active PEBS LBR users
+ * may drain the PEBS buffer and overwrite the live LBR
+ * snapshot updated by the PMI handler. This can happen,
+ * for example, when:
+ *
+ * - the event belongs to a SAMPLE_READ group that includes
+ * a PEBS event which snapshots the LBR stack; a later
+ * perf_event_overflow() may call perf_output_read_group(),
+ * which drains the PEBS buffer via intel_pmu_read_event();
+ * - the overflow triggers group throttling, disabling a PEBS
+ * LBR event and causing intel_pmu_disable_event() to drain
+ * the PEBS buffer before the live LBR snapshot is output.
+ *
+ * Therefore, drain the PEBS buffer whenever the event needs
+ * a branch stack and PEBS LBR users are active.
*/
- if (is_pebs_counter_event_group(event))
+ if (is_pebs_counter_event_group(event) ||
+ (has_branch_stack(event) && cpuc->lbr_pebs_users > 0))
intel_pmu_drain_pebs_buffer();
last_period = event->hw.last_period;
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* Re: [Patch v7 19/24] perf/x86/intel: Drain PEBS buffer for branch events with PEBS LBR users
2026-11-18 3:14 ` [Patch v7 19/24] perf/x86/intel: Drain PEBS buffer for branch events with PEBS LBR users Dapeng Mi
@ 2026-11-18 3:52 ` sashiko-bot
0 siblings, 0 replies; 49+ messages in thread
From: sashiko-bot @ 2026-11-18 3:52 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users
> If an event needs a branch stack while PEBS LBR users are active, the
> PEBS buffer may get drained before the event's sample is output. The
> drain overwrites the live LBR snapshot read by the PMI handler, so the
> stale PEBS LBR stack is reported to userspace instead.
>
> This can happen, for example, when:
> - The event belongs to a SAMPLE_READ group that includes a PEBS event
> snapshotting the LBR stack, and a later perf_event_overflow() calls
> perf_output_read_group(), which drains the PEBS buffer via
> intel_pmu_read_event();
> - The overflow triggers group throttling, disabling a PEBS LBR event and
> causing intel_pmu_disable_event() to drain the PEBS buffer before the
> live LBR snapshot is output.
>
> Thus, drain the PEBS buffer whenever the overflowed event needs a branch
> stack and PEBS LBR users are active.
>
> Fixes: c22497f5838c ("perf/x86/intel: Support adaptive PEBS v4")
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261118031453.1353625-1-dapeng1.mi@linux.intel.com?part=19
^ permalink raw reply [flat|nested] 49+ messages in thread
* [Patch v7 20/24] perf/x86/intel: Clear raw LBR counters in the LBR stack
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (18 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 19/24] perf/x86/intel: Drain PEBS buffer for branch events with PEBS LBR users Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:50 ` sashiko-bot
2026-11-18 3:54 ` bot+bpf-ci
2026-11-18 3:14 ` [Patch v7 21/24] perf/x86/intel: Refresh LBR stack for LBR counter events in PMI handler Dapeng Mi
` (3 subsequent siblings)
23 siblings, 2 replies; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
The raw branch counters are stored in the reserved field of each LBR
entry. The branch counters path clears that field after reordering
the counters, but the BPF snapshot path and the regular branch stack
path leave the raw values in place, so they leak to userspace.
Clear the raw counters on those paths too.
Fixes: 33744916196b ("perf/x86/intel: Support branch counters logging")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 1 +
arch/x86/events/intel/lbr.c | 13 +++++++++++++
arch/x86/events/perf_event.h | 2 ++
3 files changed, 16 insertions(+)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index ccd2c57931b3..fc672bc69ae3 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2836,6 +2836,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
lbr_stack = (struct perf_branch_stack *)bpf->__lbr_stack;
intel_pmu_lbr_read(lbr_stack, bpf->counters);
+ intel_pmu_lbr_clear_raw_counters(lbr_stack);
cnt = min_t(unsigned int, cnt, lbr_stack->nr);
memcpy(entries, lbr_stack->entries,
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 740d5a5ba4c3..4736debffb0b 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -991,6 +991,18 @@ static void intel_pmu_lbr_counters_reorder(struct cpu_hw_events *cpuc,
}
}
+void intel_pmu_lbr_clear_raw_counters(struct perf_branch_stack *lbr_stack)
+{
+ int i;
+
+ if (!(x86_pmu.flags & PMU_FL_BR_CNTR))
+ return;
+
+ /* The reserved field holds the raw branch counters. */
+ for (i = 0; i < lbr_stack->nr; i++)
+ lbr_stack->entries[i].reserved = 0;
+}
+
void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
struct cpu_hw_events *cpuc,
struct perf_event *event)
@@ -1001,6 +1013,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
return;
}
+ intel_pmu_lbr_clear_raw_counters(cpu_lbr_stack(cpuc));
perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
}
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index 4e6cf5c1e865..733123fc4ffe 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -1986,6 +1986,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr);
void intel_pebs_init(void);
+void intel_pmu_lbr_clear_raw_counters(struct perf_branch_stack *lbr_stack);
+
void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
struct cpu_hw_events *cpuc,
struct perf_event *event);
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* Re: [Patch v7 20/24] perf/x86/intel: Clear raw LBR counters in the LBR stack
2026-11-18 3:14 ` [Patch v7 20/24] perf/x86/intel: Clear raw LBR counters in the LBR stack Dapeng Mi
@ 2026-11-18 3:50 ` sashiko-bot
2026-11-18 3:54 ` bot+bpf-ci
1 sibling, 0 replies; 49+ messages in thread
From: sashiko-bot @ 2026-11-18 3:50 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users
> The raw branch counters are stored in the reserved field of each LBR
> entry. The branch counters path clears that field after reordering
> the counters, but the BPF snapshot path and the regular branch stack
> path leave the raw values in place, so they leak to userspace.
>
> Clear the raw counters on those paths too.
>
> Fixes: 33744916196b ("perf/x86/intel: Support branch counters logging")
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261118031453.1353625-1-dapeng1.mi@linux.intel.com?part=20
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: [Patch v7 20/24] perf/x86/intel: Clear raw LBR counters in the LBR stack
2026-11-18 3:14 ` [Patch v7 20/24] perf/x86/intel: Clear raw LBR counters in the LBR stack Dapeng Mi
2026-11-18 3:50 ` sashiko-bot
@ 2026-11-18 3:54 ` bot+bpf-ci
1 sibling, 0 replies; 49+ messages in thread
From: bot+bpf-ci @ 2026-11-18 3:54 UTC (permalink / raw)
To: dapeng1.mi, peterz, mingo, acme, namhyung, irogers, adrian.hunter,
alexander.shishkin, ak, eranian, sandipan.das, andrii, song,
ravi.bangoria, ananth.narayan
Cc: linux-kernel, linux-perf-users, bpf, dapeng1.mi, zide.chen,
thomas.falcon, xudong.hao, dapeng1.mi, sashiko-bot, ast, andrii,
daniel, martin.lau, eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 2324 bytes --]
> perf/x86/intel: Clear raw LBR counters in the LBR stack
TL;DR: Clearing raw LBR counters in the shared stack may zero them before a
later branch-counter event in the same PMI reads them, giving all-zero counter
samples until a later commit fixes it.
> diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
> --- a/arch/x86/events/intel/lbr.c
> +++ b/arch/x86/events/intel/lbr.c
[ ... ]
> @@ -1001,6 +1013,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
> return;
> }
>
> + intel_pmu_lbr_clear_raw_counters(cpu_lbr_stack(cpuc));
> perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
> }
Can this clear the counters before a later event in the same PMI has
consumed them?
handle_pmi_common() reads the LBR stack only once per PMI and reuses it for
every overflowing branch event:
if (has_branch_stack(event)) {
if (!cpuc->lbr_stack_live) {
intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
cpuc->lbr_stack_live = true;
}
intel_pmu_lbr_save_brstack(&data, cpuc, event);
}
Suppose event A is an ordinary LBR event on counter 0 and event B is the
sampling event of a branch counters group on counter 1, and both overflow in
the same PMI. A is handled first, and intel_pmu_lbr_save_brstack() calls
intel_pmu_lbr_clear_raw_counters(), which zeroes entries[i].reserved in the
shared cpu_lbr_stack().
When B is handled, lbr_stack_live is still true so the stack is not re-read,
and intel_pmu_lbr_counters_reorder() reads reserved == 0 for every entry.
B's sample then reports all-zero branch counters.
Before this patch the non-counters path left reserved alone, so B got the
correct counters in this ordering.
The later commit 57fc5222dbd3 ("perf/x86/intel: Refresh LBR stack for LBR
counter events in PMI handler") addresses this case by re-reading the LBR
stack before a branch counters group event once an earlier LBR event has
cleared it. Until that commit, a bisect landing on this patch would see
wrong branch counter samples in this scenario.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/44231525404
^ permalink raw reply [flat|nested] 49+ messages in thread
* [Patch v7 21/24] perf/x86/intel: Refresh LBR stack for LBR counter events in PMI handler
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (19 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 20/24] perf/x86/intel: Clear raw LBR counters in the LBR stack Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:45 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 22/24] perf/x86/intel: Pass LBR stack size into intel_pmu_store_lbr() Dapeng Mi
` (2 subsequent siblings)
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
When handling an LBR counter event, the raw LBR counters in the reserved
field of each LBR entry are cleared after the stack is processed and
reported to userspace. This prevents raw counter data from leaking.
However, if multiple events overflow in the same PMI, an LBR counter
event may not be the first one processed. In that case, the raw LBR
counters may already have been cleared by the previous LBR event, so the
later LBR counter event cannot read the correct values.
Refresh the LBR stack before processing an LBR counter event when
multiple events are handled in a single PMI. This restores the correct
raw LBR counter data for the event in this corner case.
This situation is rare, so the extra refresh should have negligible
performance impact.
Fixes: 33744916196b ("perf/x86/intel: Support branch counters logging")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index fc672bc69ae3..b4632b791e26 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -3906,8 +3906,9 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
{
struct perf_sample_data data;
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
- int bit;
+ bool lbr_counter_cleared = false;
int handled = 0;
+ int bit;
inc_perf_irq_stat();
@@ -4080,12 +4081,19 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
perf_sample_data_init(&data, 0, last_period);
if (has_branch_stack(event)) {
- if (!cpuc->lbr_stack_live) {
+ /*
+ * Re-read the LBR stack if it no longer holds the live state,
+ * or if an earlier LBR event already cleared the raw counters
+ * that this event needs.
+ */
+ if (!cpuc->lbr_stack_live ||
+ (lbr_counter_cleared && is_branch_counters_group(event))) {
intel_pmu_lbr_read(cpu_lbr_stack(cpuc),
cpuc->lbr_counters);
cpuc->lbr_stack_live = true;
}
intel_pmu_lbr_save_brstack(&data, cpuc, event);
+ lbr_counter_cleared = true;
}
x86_pmu_clear_perf_regs(regs);
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* Re: [Patch v7 21/24] perf/x86/intel: Refresh LBR stack for LBR counter events in PMI handler
2026-11-18 3:14 ` [Patch v7 21/24] perf/x86/intel: Refresh LBR stack for LBR counter events in PMI handler Dapeng Mi
@ 2026-11-18 3:45 ` sashiko-bot
0 siblings, 0 replies; 49+ messages in thread
From: sashiko-bot @ 2026-11-18 3:45 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users
> When handling an LBR counter event, the raw LBR counters in the reserved
> field of each LBR entry are cleared after the stack is processed and
> reported to userspace. This prevents raw counter data from leaking.
>
> However, if multiple events overflow in the same PMI, an LBR counter
> event may not be the first one processed. In that case, the raw LBR
> counters may already have been cleared by the previous LBR event, so the
> later LBR counter event cannot read the correct values.
>
> Refresh the LBR stack before processing an LBR counter event when
> multiple events are handled in a single PMI. This restores the correct
> raw LBR counter data for the event in this corner case.
>
> This situation is rare, so the extra refresh should have negligible
> performance impact.
>
> Fixes: 33744916196b ("perf/x86/intel: Support branch counters logging")
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261118031453.1353625-1-dapeng1.mi@linux.intel.com?part=21
^ permalink raw reply [flat|nested] 49+ messages in thread
* [Patch v7 22/24] perf/x86/intel: Pass LBR stack size into intel_pmu_store_lbr()
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (20 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 21/24] perf/x86/intel: Refresh LBR stack for LBR counter events in PMI handler Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:46 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 23/24] perf/core: Don't mutate shared branch stack depth Dapeng Mi
2026-11-18 3:14 ` [Patch v7 24/24] perf/x86/intel: Fix incorrect PERF_METRICS overflow bit clearing Dapeng Mi
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko
intel_pmu_store_lbr() assumes the LBR stack always contains
x86_pmu.lbr_nr entries, but arch-PEBS can use a smaller configured depth.
This can result in out-of-bounds reads when the stack is shallower than
the hardware maximum.
Add a lbr_nr parameter to the helper so the code only processes the
available entries and avoids overrunning the LBR buffer.
Fixes: d21954c8a0ff ("perf/x86/intel: Process arch-PEBS records or record fragments")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/ds.c | 11 ++++++-----
arch/x86/events/intel/lbr.c | 18 ++++++++++--------
arch/x86/events/perf_event.h | 2 +-
3 files changed, 17 insertions(+), 14 deletions(-)
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 9fe84e4880d9..1bf9e9b4a8c9 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -2603,12 +2603,13 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event,
if (format_group & PEBS_DATACFG_LBRS) {
struct lbr_entry *lbr = next_record;
- int num_lbr = ((format_group >> PEBS_DATACFG_LBR_SHIFT)
- & 0xff) + 1;
+ unsigned int num_lbr;
+
+ num_lbr = ((format_group >> PEBS_DATACFG_LBR_SHIFT) & 0xff) + 1;
next_record = next_record + num_lbr * sizeof(struct lbr_entry);
if (has_branch_stack(event)) {
- intel_pmu_store_pebs_lbrs(lbr);
+ intel_pmu_store_pebs_lbrs(lbr, num_lbr);
intel_pmu_lbr_save_brstack(data, cpuc, event);
}
}
@@ -2789,7 +2790,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event,
if (header->lbr) {
struct arch_pebs_lbr_header *lbr_header = next_record;
struct lbr_entry *lbr;
- int num_lbr;
+ unsigned int num_lbr;
next_record = lbr_header + 1;
lbr = next_record;
@@ -2800,7 +2801,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event,
next_record += num_lbr * sizeof(struct lbr_entry);
if (has_branch_stack(event)) {
- intel_pmu_store_pebs_lbrs(lbr);
+ intel_pmu_store_pebs_lbrs(lbr, num_lbr);
intel_pmu_lbr_save_brstack(data, cpuc, event);
}
}
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 4736debffb0b..73b3fbb3b823 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -906,7 +906,7 @@ static __always_inline u16 get_lbr_cycles(u64 info)
static_assert((64 - PERF_BRANCH_ENTRY_INFO_BITS_MAX) > LBR_INFO_BR_CNTR_NUM * LBR_INFO_BR_CNTR_BITS);
-static void intel_pmu_store_lbr(struct lbr_entry *entries,
+static void intel_pmu_store_lbr(struct lbr_entry *entries, unsigned int lbr_nr,
struct perf_branch_stack *lbr_stack)
{
struct perf_branch_entry *e;
@@ -915,8 +915,8 @@ static void intel_pmu_store_lbr(struct lbr_entry *entries,
bool mispred;
int i;
- cpu_init_lbr_stack(lbr_stack, x86_pmu.lbr_nr);
- for (i = 0; i < x86_pmu.lbr_nr; i++) {
+ cpu_init_lbr_stack(lbr_stack, lbr_nr);
+ for (i = 0; i < lbr_nr; i++) {
lbr = entries ? &entries[i] : NULL;
e = &lbr_stack->entries[i];
@@ -1020,7 +1020,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
static void intel_pmu_arch_lbr_read(struct cpu_hw_events *cpuc,
struct perf_branch_stack *lbr_stack)
{
- intel_pmu_store_lbr(NULL, lbr_stack);
+ intel_pmu_store_lbr(NULL, x86_pmu.lbr_nr, lbr_stack);
}
static void
@@ -1030,14 +1030,15 @@ intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc,
struct x86_perf_task_context_arch_lbr_xsave *xsave = cpuc->lbr_xsave;
if (!xsave) {
- intel_pmu_store_lbr(NULL, lbr_stack);
+ intel_pmu_store_lbr(NULL, x86_pmu.lbr_nr, lbr_stack);
return;
}
xsave->header.xfeatures = 0;
xsaves(&xsave->xsave, XFEATURE_MASK_LBR);
if (xsave->header.xfeatures & XFEATURE_MASK_LBR) {
- intel_pmu_store_lbr(xsave->lbr.entries, lbr_stack);
+ intel_pmu_store_lbr(xsave->lbr.entries,
+ x86_pmu.lbr_nr, lbr_stack);
} else {
/* Clear the LBR stack to avoid leaking stale entries. */
lbr_stack->nr = 0;
@@ -1317,7 +1318,7 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc,
lbr_stack->nr = lbr_nr;
}
-void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
+void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr, unsigned int lbr_nr)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
@@ -1328,7 +1329,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
else
cpu_lbr_stack(cpuc)->hw_idx = intel_pmu_lbr_tos();
- intel_pmu_store_lbr(lbr, cpu_lbr_stack(cpuc));
+ intel_pmu_store_lbr(lbr, min(lbr_nr, x86_pmu.lbr_nr),
+ cpu_lbr_stack(cpuc));
intel_pmu_lbr_filter(cpuc, cpu_lbr_stack(cpuc), cpuc->lbr_counters);
cpuc->lbr_stack_live = false;
}
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index 733123fc4ffe..4269a47ee47e 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -1982,7 +1982,7 @@ void intel_pmu_pebs_late_setup(struct cpu_hw_events *cpuc);
void intel_pmu_drain_pebs_buffer(void);
-void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr);
+void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr, unsigned int lbr_nr);
void intel_pebs_init(void);
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [Patch v7 23/24] perf/core: Don't mutate shared branch stack depth
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (21 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 22/24] perf/x86/intel: Pass LBR stack size into intel_pmu_store_lbr() Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:51 ` sashiko-bot
2026-11-18 3:14 ` [Patch v7 24/24] perf/x86/intel: Fix incorrect PERF_METRICS overflow bit clearing Dapeng Mi
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi, Sashiko, KP Singh,
Matt Bobrowski, Jiri Olsa, Ihor Solodrai, Daniel Xu
perf_sample_save_brstack() clamps the branch stack depth by directly
overwriting brs->nr with min(event->attr.sample_max_stack, brs->nr).
This is unsafe because the branch stack buffer is shared by all branch
events in the same PMI: once one event with a small sample_max_stack
truncates brs->nr, every subsequent event processed in that PMI sees
the shrunken depth and silently loses branch records it asked for.
Add perf_event_brstack_depth(), which computes the per-event depth
without modifying the shared branch stack. Use it in the relevant callers
so each event is limited only by its own sample_max_stack, without
clobbering the shared LBR state.
bpf_read_branch_records() also relied on the clamped brs->nr, so switch
it to perf_event_brstack_depth() to preserve the same per-event limit for
records returned to BPF programs.
Fixes: c53e14f1ea4a ("perf: Extend per event callchain limit to branch stack")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Cc: KP Singh <kpsingh@kernel.org>
Cc: Matt Bobrowski <matt@bobrowski.net>
Cc: Song Liu <song@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Ihor Solodrai <ihor.solodrai@linux.dev>
Cc: Daniel Xu <dxu@dxuuu.xyz>
---
include/linux/perf_event.h | 13 ++++++++++---
kernel/events/core.c | 8 ++++----
kernel/trace/bpf_trace.c | 6 ++++--
3 files changed, 18 insertions(+), 9 deletions(-)
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index 19de3f1cbd6e..35d3cc4cc07e 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -1424,12 +1424,19 @@ static inline bool has_branch_stack(struct perf_event *event)
return event->attr.sample_type & PERF_SAMPLE_BRANCH_STACK;
}
+static inline u64 perf_event_brstack_depth(struct perf_event *event,
+ struct perf_branch_stack *brs)
+{
+ return min_t(u16, event->attr.sample_max_stack, brs->nr);
+}
+
static inline void perf_sample_save_brstack(struct perf_sample_data *data,
struct perf_event *event,
struct perf_branch_stack *brs,
u64 *brs_cntr)
{
int size = sizeof(u64); /* nr */
+ u64 nr;
if (!has_branch_stack(event))
return;
@@ -1439,9 +1446,9 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data,
if (branch_sample_hw_index(event))
size += sizeof(u64);
- brs->nr = min_t(u16, event->attr.sample_max_stack, brs->nr);
+ nr = perf_event_brstack_depth(event, brs);
- size += brs->nr * sizeof(struct perf_branch_entry);
+ size += nr * sizeof(struct perf_branch_entry);
/*
* The extension space for counters is appended after the
@@ -1449,7 +1456,7 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data,
* of events of each branch.
*/
if (brs_cntr)
- size += brs->nr * sizeof(u64);
+ size += nr * sizeof(u64);
data->br_stack = brs;
data->br_stack_cntr = brs_cntr;
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 457d75ea7142..8b80ca788523 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8462,12 +8462,12 @@ void perf_output_sample(struct perf_output_handle *handle,
if (sample_type & PERF_SAMPLE_BRANCH_STACK) {
if (data->br_stack) {
+ u64 nr = perf_event_brstack_depth(event, data->br_stack);
size_t size;
- size = data->br_stack->nr
- * sizeof(struct perf_branch_entry);
+ size = nr * sizeof(struct perf_branch_entry);
- perf_output_put(handle, data->br_stack->nr);
+ perf_output_put(handle, nr);
if (branch_sample_hw_index(event))
perf_output_put(handle, data->br_stack->hw_idx);
perf_output_copy(handle, data->br_stack->entries, size);
@@ -8476,7 +8476,7 @@ void perf_output_sample(struct perf_output_handle *handle,
* right after the struct perf_branch_stack.
*/
if (data->br_stack_cntr) {
- size = data->br_stack->nr * sizeof(u64);
+ size = nr * sizeof(u64);
perf_output_copy(handle, data->br_stack_cntr, size);
}
} else {
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 2aec56d7f3d3..7c42bbf05c2f 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -1549,6 +1549,7 @@ BPF_CALL_4(bpf_read_branch_records, struct bpf_perf_event_data_kern *, ctx,
static const u32 br_entry_size = sizeof(struct perf_branch_entry);
struct perf_branch_stack *br_stack = ctx->data->br_stack;
u32 to_copy;
+ u64 nr;
if (unlikely(flags & ~BPF_F_GET_BRANCH_RECORDS_SIZE))
return -EINVAL;
@@ -1559,13 +1560,14 @@ BPF_CALL_4(bpf_read_branch_records, struct bpf_perf_event_data_kern *, ctx,
if (unlikely(!br_stack))
return -ENOENT;
+ nr = perf_event_brstack_depth(ctx->event, br_stack);
if (flags & BPF_F_GET_BRANCH_RECORDS_SIZE)
- return br_stack->nr * br_entry_size;
+ return nr * br_entry_size;
if (!buf || (size % br_entry_size != 0))
return -EINVAL;
- to_copy = min_t(u32, br_stack->nr * br_entry_size, size);
+ to_copy = min_t(u32, nr * br_entry_size, size);
memcpy(buf, br_stack->entries, to_copy);
return to_copy;
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* Re: [Patch v7 23/24] perf/core: Don't mutate shared branch stack depth
2026-11-18 3:14 ` [Patch v7 23/24] perf/core: Don't mutate shared branch stack depth Dapeng Mi
@ 2026-11-18 3:51 ` sashiko-bot
0 siblings, 0 replies; 49+ messages in thread
From: sashiko-bot @ 2026-11-18 3:51 UTC (permalink / raw)
To: Dapeng Mi; +Cc: linux-perf-users
> perf_sample_save_brstack() clamps the branch stack depth by directly
> overwriting brs->nr with min(event->attr.sample_max_stack, brs->nr).
> This is unsafe because the branch stack buffer is shared by all branch
> events in the same PMI: once one event with a small sample_max_stack
> truncates brs->nr, every subsequent event processed in that PMI sees
> the shrunken depth and silently loses branch records it asked for.
>
> Add perf_event_brstack_depth(), which computes the per-event depth
> without modifying the shared branch stack. Use it in the relevant callers
> so each event is limited only by its own sample_max_stack, without
> clobbering the shared LBR state.
>
> bpf_read_branch_records() also relied on the clamped brs->nr, so switch
> it to perf_event_brstack_depth() to preserve the same per-event limit for
> records returned to BPF programs.
>
> Fixes: c53e14f1ea4a ("perf: Extend per event callchain limit to branch stack")
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261118031453.1353625-1-dapeng1.mi@linux.intel.com?part=23
^ permalink raw reply [flat|nested] 49+ messages in thread
* [Patch v7 24/24] perf/x86/intel: Fix incorrect PERF_METRICS overflow bit clearing
2026-11-18 3:14 [Patch v7 00/24] perf/x86: Fix multiple LBR sampling issues Dapeng Mi
` (22 preceding siblings ...)
2026-11-18 3:14 ` [Patch v7 23/24] perf/core: Don't mutate shared branch stack depth Dapeng Mi
@ 2026-11-18 3:14 ` Dapeng Mi
2026-11-18 3:46 ` sashiko-bot
23 siblings, 1 reply; 49+ messages in thread
From: Dapeng Mi @ 2026-11-18 3:14 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane, Sandipan Das, Andrii Nakryiko,
Song Liu, Ravi Bangoria, Ananth Narayan
Cc: linux-kernel, linux-perf-users, bpf, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
The overflow status bit for PERF_METRICS is cleared incorrectly in
handle_pmi_common(). The code clears the raw bit index value instead of
the corresponding BIT_ULL(...) mask, which leaves the overflow flag in
an incorrect state.
Define GLOBAL_STATUS_PERF_METRICS_OVF as the proper bit mask and use it
when clearing the PERF_METRICS overflow bit in handle_pmi_common().
Fixes: a5f5e1238f4f ("perf/x86/intel: Don't clear perf metrics overflow bit unconditionally")
Fixes: d21954c8a0ff ("perf/x86/intel: Process arch-PEBS records or record fragments")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/core.c | 4 ++--
arch/x86/include/asm/perf_event.h | 1 +
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index b4632b791e26..d7284eb73227 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -3972,7 +3972,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
*/
if (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS] &&
is_pebs_counter_event_group(cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS]))
- status &= ~GLOBAL_STATUS_PERF_METRICS_OVF_BIT;
+ status &= ~GLOBAL_STATUS_PERF_METRICS_OVF;
}
/*
@@ -3987,7 +3987,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
if (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS] &&
is_pebs_counter_event_group(cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS]))
- status &= ~GLOBAL_STATUS_PERF_METRICS_OVF_BIT;
+ status &= ~GLOBAL_STATUS_PERF_METRICS_OVF;
}
/*
diff --git a/arch/x86/include/asm/perf_event.h b/arch/x86/include/asm/perf_event.h
index 5c92d43bbef8..63e1d1bcd01a 100644
--- a/arch/x86/include/asm/perf_event.h
+++ b/arch/x86/include/asm/perf_event.h
@@ -452,6 +452,7 @@ static inline bool is_topdown_idx(int idx)
#define GLOBAL_STATUS_ARCH_PEBS_THRESHOLD_BIT 54
#define GLOBAL_STATUS_ARCH_PEBS_THRESHOLD BIT_ULL(GLOBAL_STATUS_ARCH_PEBS_THRESHOLD_BIT)
#define GLOBAL_STATUS_PERF_METRICS_OVF_BIT 48
+#define GLOBAL_STATUS_PERF_METRICS_OVF BIT_ULL(GLOBAL_STATUS_PERF_METRICS_OVF_BIT)
#define GLOBAL_CTRL_EN_PERF_METRICS BIT_ULL(48)
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread