* [PATCH 1/2] perf/x86/intel: Correct pt_regs->flags update for PEBS path
@ 2026-09-08 7:51 Dapeng Mi
2026-09-08 7:51 ` [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER Dapeng Mi
2026-09-08 8:05 ` [PATCH 1/2] perf/x86/intel: Correct pt_regs->flags update for PEBS path sashiko-bot
0 siblings, 2 replies; 20+ messages in thread
From: Dapeng Mi @ 2026-09-08 7:51 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin,
Andi Kleen, Eranian Stephane
Cc: linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen,
Falcon Thomas, Xudong Hao, Dapeng Mi
pt_regs->flags holds the saved CPU FLAGS register. In the PEBS path,
it was incorrectly set to PERF_EFLAGS_EXACT instead of being populated
from the PEBS flags snapshot.
Update pt_regs->flags from PEBS GPR flags if GPRs group is present.
Fixes: c22497f5838c ("perf/x86/intel: Support adaptive PEBS v4")
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
arch/x86/events/intel/ds.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 8940f0292229..d0bb767a0fef 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -2432,7 +2432,7 @@ static inline void __setup_pebs_basic_group(struct perf_event *event,
{
/* The ip in basic is EventingIP */
set_linear_ip(regs, ip);
- regs->flags = PERF_EFLAGS_EXACT;
+ regs->flags |= PERF_EFLAGS_EXACT;
setup_pebs_time(event, data, tsc);
if (sample_type & PERF_SAMPLE_WEIGHT_STRUCT)
@@ -2444,9 +2444,17 @@ static inline void __setup_pebs_gpr_group(struct perf_event *event,
struct pebs_gprs *gprs,
u64 sample_type)
{
+ /*
+ * Update flags with PEBS data. PERF_EFLAGS_EXACT must be set
+ * in previous basic group handling.
+ */
+ regs->flags = gprs->flags | PERF_EFLAGS_EXACT;
+
if (event->attr.precise_ip < 2) {
set_linear_ip(regs, gprs->ip);
regs->flags &= ~PERF_EFLAGS_EXACT;
+ } else if (regs->flags & X86_VM_MASK) {
+ regs->flags ^= (PERF_EFLAGS_VM | X86_VM_MASK);
}
if (sample_type & (PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_REGS_USER))
base-commit: 609d28fba41e977425eff9848e553740bbf155ad
--
2.34.1
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-08 7:51 [PATCH 1/2] perf/x86/intel: Correct pt_regs->flags update for PEBS path Dapeng Mi @ 2026-09-08 7:51 ` Dapeng Mi 2026-09-08 8:15 ` sashiko-bot ` (2 more replies) 2026-09-08 8:05 ` [PATCH 1/2] perf/x86/intel: Correct pt_regs->flags update for PEBS path sashiko-bot 1 sibling, 3 replies; 20+ messages in thread From: Dapeng Mi @ 2026-09-08 7:51 UTC (permalink / raw) To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin, Andi Kleen, Eranian Stephane Cc: linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Dapeng Mi, Gennady Kupava, Ravi Bangoria PERF_SAMPLE_STACK_USER needs to return the user stack and user registers to user space when the PMI exits. Since the skid from the PEBS/IBS sample and PMI delivery, the PEBS/IBS register snapshot (especially IP/SP/BP) can diverge from the user stack at PMI return. That mismatch breaks DWARF unwinding. Precise sampling provides no benefit in this case, so disable PEBS/IBS precise sampling and allow only PMI-based sampling when PERF_SAMPLE_STACK_USER is requested. Reported-by: Gennady Kupava <gennady.kupava@gmail.com> Closes: https://lore.kernel.org/all/CAPu-DQqF0aF6=GS8Z6KKWeeX_V5LiXeKU_rJQZC+uGg8zuTPNw@mail.gmail.com/ Cc: Ravi Bangoria <ravi.bangoria@amd.com> Fixes: c5ebcedb566e ("perf: Add ability to attach user stack dump to sample") Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com> --- arch/x86/events/amd/ibs.c | 3 +++ arch/x86/events/core.c | 3 +++ arch/x86/events/perf_event.h | 19 +++++++++++++++++++ 3 files changed, 25 insertions(+) diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c index 3531f9c23b8c..c67b659d64e2 100644 --- a/arch/x86/events/amd/ibs.c +++ b/arch/x86/events/amd/ibs.c @@ -327,6 +327,9 @@ static int perf_ibs_init(struct perf_event *event) if (has_branch_stack(event)) return -EOPNOTSUPP; + if (!x86_pmu_allow_sample_user_stack(event, true)) + return -EINVAL; + /* handle exclude_{user,kernel} in the IRQ handler */ if (event->attr.exclude_host || event->attr.exclude_guest || event->attr.exclude_idle) diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c index 8b3ea0adb965..ceb5038d7565 100644 --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -594,6 +594,9 @@ int x86_pmu_hw_config(struct perf_event *event) /* There's no sense in having PEBS for non sampling events: */ if (!is_sampling_event(event)) return -EINVAL; + + if (!x86_pmu_allow_sample_user_stack(event, false)) + return -EINVAL; } /* * check that PEBS LBR correction does not conflict with diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h index 71ed5b2acea2..9fbcf2bcef51 100644 --- a/arch/x86/events/perf_event.h +++ b/arch/x86/events/perf_event.h @@ -1274,6 +1274,25 @@ static inline bool is_counter_pair(struct hw_perf_event *hwc) return hwc->flags & PERF_X86_EVENT_PAIR; } +static inline bool +x86_pmu_allow_sample_user_stack(struct perf_event *event, bool ibs) +{ + if (!(event->attr.sample_type & PERF_SAMPLE_STACK_USER)) + return true; + + /* + * PERF_SAMPLE_STACK_USER needs to return the user stack and + * user registers to user space when the PMI exits. Since the skid + * from the PEBS/IBS sample and PMI delivery, the PEBS/IBS register + * snapshot (especially IP/SP/BP) can diverge from the user stack + * at PMI return. That mismatch breaks DWARF unwinding. + * + * Therefore, disable precise sampling for PERF_SAMPLE_STACK_USER + * and permit only PMI-based sampling for this case. + */ + return !(event->attr.precise_ip || ibs); +} + static inline void __x86_pmu_enable_event(struct hw_perf_event *hwc, u64 enable_mask) { -- 2.34.1 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-08 7:51 ` [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER Dapeng Mi @ 2026-09-08 8:15 ` sashiko-bot 2026-09-08 8:47 ` Mi, Dapeng 2026-09-08 8:49 ` Peter Zijlstra 2026-09-08 8:49 ` Mi, Dapeng 2 siblings, 1 reply; 20+ messages in thread From: sashiko-bot @ 2026-09-08 8:15 UTC (permalink / raw) To: Dapeng Mi; +Cc: linux-perf-users Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] ABI Breakage / Overly Broad Restriction on Precise Sampling with User Stacks -- commit 4f30379fa3ce88165bf1e61c1c825c26d5810b23 Author: Dapeng Mi <dapeng1.mi@linux.intel.com> perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER This commit disables precise sampling (PEBS/IBS) when PERF_SAMPLE_STACK_USER is requested. It attempts to prevent DWARF unwinding failures caused by skid between the PEBS/IBS register snapshot and the user stack at PMI return. Closes: https://lore.kernel.org/all/CAPu-DQqF0aF6=GS8Z6KKWeeX_V5LiXeKU_rJQZC+uGg8zuTPNw@mail.gmail.com/ Fixes: c5ebcedb566e ("perf: Add ability to attach user stack dump to sample") > diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c > index 3531f9c23b8c9..c67b659d64e2a 100644 > --- a/arch/x86/events/amd/ibs.c > +++ b/arch/x86/events/amd/ibs.c > @@ -327,6 +327,9 @@ static int perf_ibs_init(struct perf_event *event) > if (has_branch_stack(event)) > return -EOPNOTSUPP; > > + if (!x86_pmu_allow_sample_user_stack(event, true)) > + return -EINVAL; [Severity: High] Does this change unconditionally reject AMD IBS events when user stacks are requested, since the second parameter is hardcoded to true? Returning -EINVAL to sys_perf_event_open for configurations that were previously valid breaks backward compatibility. Userspace applications relying on AMD IBS with PERF_SAMPLE_STACK_USER might now fail unexpectedly. > + > /* handle exclude_{user,kernel} in the IRQ handler */ > if (event->attr.exclude_host || event->attr.exclude_guest || > diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c > index d1af33d96d0a3..a26ea8baea731 100644 > --- a/arch/x86/events/core.c > +++ b/arch/x86/events/core.c > @@ -596,6 +596,9 @@ int x86_pmu_hw_config(struct perf_event *event) > /* There's no sense in having PEBS for non sampling events: */ > if (!is_sampling_event(event)) > return -EINVAL; > + > + if (!x86_pmu_allow_sample_user_stack(event, false)) > + return -EINVAL; [Severity: High] Can this break existing perf setups for Intel PEBS events? Tools issuing commands like perf record -e cycles:pp --call-graph dwarf rely on precise_ip and PERF_SAMPLE_STACK_USER. Rejecting this setup with -EINVAL prevents previously accepted workflows from functioning. > } > /* > diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h > index eae24bb35dc14..ca52597cc39a1 100644 > --- a/arch/x86/events/perf_event.h > +++ b/arch/x86/events/perf_event.h [ ... ] > @@ -1273,6 +1273,25 @@ static inline bool is_counter_pair(struct hw_perf_event *hwc) > return hwc->flags & PERF_X86_EVENT_PAIR; > } > > +static inline bool > +x86_pmu_allow_sample_user_stack(struct perf_event *event, bool ibs) > +{ > + if (!(event->attr.sample_type & PERF_SAMPLE_STACK_USER)) > + return true; > + > + /* > + * PERF_SAMPLE_STACK_USER needs to return the user stack and > + * user registers to user space when the PMI exits. Since the skid > + * from the PEBS/IBS sample and PMI delivery, the PEBS/IBS register > + * snapshot (especially IP/SP/BP) can diverge from the user stack > + * at PMI return. That mismatch breaks DWARF unwinding. > + * > + * Therefore, disable precise sampling for PERF_SAMPLE_STACK_USER > + * and permit only PMI-based sampling for this case. > + */ > + return !(event->attr.precise_ip || ibs); [Severity: High] Is the assumption about skid accurate for kernel-space profiling? When profiling a syscall or with exclude_user=1, the user-space stack and registers are frozen upon kernel entry, meaning there is zero skid between them. By unconditionally banning precise sampling with user stacks, does this unintentionally prevent the valid use-case of precisely profiling kernel events while still capturing accurate user callgraphs? > +} > + -- Sashiko AI review · https://sashiko.dev/#/patchset/20260908075102.540715-1-dapeng1.mi@linux.intel.com?part=2 ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-08 8:15 ` sashiko-bot @ 2026-09-08 8:47 ` Mi, Dapeng 0 siblings, 0 replies; 20+ messages in thread From: Mi, Dapeng @ 2026-09-08 8:47 UTC (permalink / raw) To: sashiko-reviews; +Cc: linux-perf-users On 9/8/2026 4:15 PM, sashiko-bot@kernel.org wrote: > Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: > - [High] ABI Breakage / Overly Broad Restriction on Precise Sampling with User Stacks > -- > > commit 4f30379fa3ce88165bf1e61c1c825c26d5810b23 > Author: Dapeng Mi <dapeng1.mi@linux.intel.com> > > perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER > > This commit disables precise sampling (PEBS/IBS) when PERF_SAMPLE_STACK_USER > is requested. It attempts to prevent DWARF unwinding failures caused by skid > between the PEBS/IBS register snapshot and the user stack at PMI return. > > Closes: https://lore.kernel.org/all/CAPu-DQqF0aF6=GS8Z6KKWeeX_V5LiXeKU_rJQZC+uGg8zuTPNw@mail.gmail.com/ > Fixes: c5ebcedb566e ("perf: Add ability to attach user stack dump to sample") > >> diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c >> index 3531f9c23b8c9..c67b659d64e2a 100644 >> --- a/arch/x86/events/amd/ibs.c >> +++ b/arch/x86/events/amd/ibs.c >> @@ -327,6 +327,9 @@ static int perf_ibs_init(struct perf_event *event) >> if (has_branch_stack(event)) >> return -EOPNOTSUPP; >> >> + if (!x86_pmu_allow_sample_user_stack(event, true)) >> + return -EINVAL; > [Severity: High] > Does this change unconditionally reject AMD IBS events when user stacks are > requested, since the second parameter is hardcoded to true? > > Returning -EINVAL to sys_perf_event_open for configurations that were > previously valid breaks backward compatibility. Userspace applications > relying on AMD IBS with PERF_SAMPLE_STACK_USER might now fail unexpectedly. I'm not an IBS expert, but it looks there are two ways to trigger IBS sampling, one is to set the precise_ip to 1, just like PEBS, the other is directly to create IBS events. Both of these 2 cases should be disabled to sample user stack, otherwise returned user stack and IP would be inconsistent and break DWARF unwinding. So only PMI-based user stack sampling is allowed. > >> + >> /* handle exclude_{user,kernel} in the IRQ handler */ >> if (event->attr.exclude_host || event->attr.exclude_guest || >> diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c >> index d1af33d96d0a3..a26ea8baea731 100644 >> --- a/arch/x86/events/core.c >> +++ b/arch/x86/events/core.c >> @@ -596,6 +596,9 @@ int x86_pmu_hw_config(struct perf_event *event) >> /* There's no sense in having PEBS for non sampling events: */ >> if (!is_sampling_event(event)) >> return -EINVAL; >> + >> + if (!x86_pmu_allow_sample_user_stack(event, false)) >> + return -EINVAL; > [Severity: High] > Can this break existing perf setups for Intel PEBS events? > > Tools issuing commands like perf record -e cycles:pp --call-graph dwarf > rely on precise_ip and PERF_SAMPLE_STACK_USER. Rejecting this setup with > -EINVAL prevents previously accepted workflows from functioning. Yes, but it seems there is no better way to fix this issue. An alternative way to fix this issue is still to return the PMI context register state rather than the PEBS precise registers for user stack sampling, but this actually falls back the imprecise PMI-based sampling. This even make the thing worse, user requires to get precise samplings, but perf silently returns imprecise records, this would mislead user. > >> } >> /* >> diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h >> index eae24bb35dc14..ca52597cc39a1 100644 >> --- a/arch/x86/events/perf_event.h >> +++ b/arch/x86/events/perf_event.h > [ ... ] > >> @@ -1273,6 +1273,25 @@ static inline bool is_counter_pair(struct hw_perf_event *hwc) >> return hwc->flags & PERF_X86_EVENT_PAIR; >> } >> >> +static inline bool >> +x86_pmu_allow_sample_user_stack(struct perf_event *event, bool ibs) >> +{ >> + if (!(event->attr.sample_type & PERF_SAMPLE_STACK_USER)) >> + return true; >> + >> + /* >> + * PERF_SAMPLE_STACK_USER needs to return the user stack and >> + * user registers to user space when the PMI exits. Since the skid >> + * from the PEBS/IBS sample and PMI delivery, the PEBS/IBS register >> + * snapshot (especially IP/SP/BP) can diverge from the user stack >> + * at PMI return. That mismatch breaks DWARF unwinding. >> + * >> + * Therefore, disable precise sampling for PERF_SAMPLE_STACK_USER >> + * and permit only PMI-based sampling for this case. >> + */ >> + return !(event->attr.precise_ip || ibs); > [Severity: High] > Is the assumption about skid accurate for kernel-space profiling? No, the skid happens on from the PEBS/IBS record generation between PMI delivery. The skid is inevitable and would lead to inconsistency between PEBS/IBS register snapshot and user stack. Thanks. > > When profiling a syscall or with exclude_user=1, the user-space stack and > registers are frozen upon kernel entry, meaning there is zero skid between > them. > > By unconditionally banning precise sampling with user stacks, does this > unintentionally prevent the valid use-case of precisely profiling kernel > events while still capturing accurate user callgraphs? > >> +} >> + ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-08 7:51 ` [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER Dapeng Mi 2026-09-08 8:15 ` sashiko-bot @ 2026-09-08 8:49 ` Peter Zijlstra 2026-09-08 8:56 ` Mi, Dapeng 2026-09-08 8:49 ` Mi, Dapeng 2 siblings, 1 reply; 20+ messages in thread From: Peter Zijlstra @ 2026-09-08 8:49 UTC (permalink / raw) To: Dapeng Mi Cc: Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin, Andi Kleen, Eranian Stephane, linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava, Ravi Bangoria On Tue, Sep 08, 2026 at 03:51:02PM +0800, Dapeng Mi wrote: > PERF_SAMPLE_STACK_USER needs to return the user stack and user registers > to user space when the PMI exits. Since the skid from the PEBS/IBS sample > and PMI delivery, the PEBS/IBS register snapshot (especially IP/SP/BP) > can diverge from the user stack at PMI return. That mismatch breaks DWARF > unwinding. > > Precise sampling provides no benefit in this case, so disable PEBS/IBS > precise sampling and allow only PMI-based sampling when > PERF_SAMPLE_STACK_USER is requested. > > Reported-by: Gennady Kupava <gennady.kupava@gmail.com> > Closes: https://lore.kernel.org/all/CAPu-DQqF0aF6=GS8Z6KKWeeX_V5LiXeKU_rJQZC+uGg8zuTPNw@mail.gmail.com/ > Cc: Ravi Bangoria <ravi.bangoria@amd.com> > Fixes: c5ebcedb566e ("perf: Add ability to attach user stack dump to sample") > Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com> This breaks long standing existing behaviour. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-08 8:49 ` Peter Zijlstra @ 2026-09-08 8:56 ` Mi, Dapeng 2026-09-08 10:19 ` Peter Zijlstra 0 siblings, 1 reply; 20+ messages in thread From: Mi, Dapeng @ 2026-09-08 8:56 UTC (permalink / raw) To: Peter Zijlstra Cc: Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin, Andi Kleen, Eranian Stephane, linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava, Ravi Bangoria On 9/8/2026 4:49 PM, Peter Zijlstra wrote: > On Tue, Sep 08, 2026 at 03:51:02PM +0800, Dapeng Mi wrote: >> PERF_SAMPLE_STACK_USER needs to return the user stack and user registers >> to user space when the PMI exits. Since the skid from the PEBS/IBS sample >> and PMI delivery, the PEBS/IBS register snapshot (especially IP/SP/BP) >> can diverge from the user stack at PMI return. That mismatch breaks DWARF >> unwinding. >> >> Precise sampling provides no benefit in this case, so disable PEBS/IBS >> precise sampling and allow only PMI-based sampling when >> PERF_SAMPLE_STACK_USER is requested. >> >> Reported-by: Gennady Kupava <gennady.kupava@gmail.com> >> Closes: https://lore.kernel.org/all/CAPu-DQqF0aF6=GS8Z6KKWeeX_V5LiXeKU_rJQZC+uGg8zuTPNw@mail.gmail.com/ >> Cc: Ravi Bangoria <ravi.bangoria@amd.com> >> Fixes: c5ebcedb566e ("perf: Add ability to attach user stack dump to sample") >> Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com> > This breaks long standing existing behaviour. Yeah, but it seems there is no better way to fix this issue. An alternative way to fix this issue is still to return the PMI context register state rather than the PEBS precise registers for user stack sampling, but this actually falls back the imprecise PMI-based sampling. In my opinion, it could even make the thing worse. User requires to get precise samplings, but perf silently returns imprecise records, this would mislead user. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-08 8:56 ` Mi, Dapeng @ 2026-09-08 10:19 ` Peter Zijlstra 2026-09-08 15:10 ` Andi Kleen 2026-09-09 1:20 ` Mi, Dapeng 0 siblings, 2 replies; 20+ messages in thread From: Peter Zijlstra @ 2026-09-08 10:19 UTC (permalink / raw) To: Mi, Dapeng Cc: Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin, Andi Kleen, Eranian Stephane, linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava, Ravi Bangoria On Tue, Sep 08, 2026 at 04:56:22PM +0800, Mi, Dapeng wrote: > > On 9/8/2026 4:49 PM, Peter Zijlstra wrote: > > On Tue, Sep 08, 2026 at 03:51:02PM +0800, Dapeng Mi wrote: > >> PERF_SAMPLE_STACK_USER needs to return the user stack and user registers > >> to user space when the PMI exits. Since the skid from the PEBS/IBS sample > >> and PMI delivery, the PEBS/IBS register snapshot (especially IP/SP/BP) > >> can diverge from the user stack at PMI return. That mismatch breaks DWARF > >> unwinding. > >> > >> Precise sampling provides no benefit in this case, so disable PEBS/IBS > >> precise sampling and allow only PMI-based sampling when > >> PERF_SAMPLE_STACK_USER is requested. > >> > >> Reported-by: Gennady Kupava <gennady.kupava@gmail.com> > >> Closes: https://lore.kernel.org/all/CAPu-DQqF0aF6=GS8Z6KKWeeX_V5LiXeKU_rJQZC+uGg8zuTPNw@mail.gmail.com/ > >> Cc: Ravi Bangoria <ravi.bangoria@amd.com> > >> Fixes: c5ebcedb566e ("perf: Add ability to attach user stack dump to sample") > >> Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com> > > This breaks long standing existing behaviour. > > Yeah, but it seems there is no better way to fix this issue. Breaking things that worked before isn't fixing.. people get upset. > An alternative way to fix this issue is still to return the PMI > context register state rather than the PEBS precise registers for user > stack sampling, but this actually falls back the imprecise PMI-based > sampling. That's what we already do, no? I have distinct memories of making the stack unwind use the NMI regs rather then the PEBS regs. > In my opinion, it could even make the thing worse. User > requires to get precise samplings, but perf silently returns imprecise > records, this would mislead user. Mostly just the unwind might be off a little, the rest is accurate. This has been the case 'forever'. Performance analysis isn't for silly people, if they can't deal with a little fuzz then perhaps they're in the wrong business. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-08 10:19 ` Peter Zijlstra @ 2026-09-08 15:10 ` Andi Kleen 2026-09-08 20:56 ` Ian Rogers 2026-09-09 1:20 ` Mi, Dapeng 1 sibling, 1 reply; 20+ messages in thread From: Andi Kleen @ 2026-09-08 15:10 UTC (permalink / raw) To: Peter Zijlstra Cc: Mi, Dapeng, Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin, Eranian Stephane, linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava, Ravi Bangoria > That's what we already do, no? I have distinct memories of making the > stack unwind use the NMI regs rather then the PEBS regs. > > > In my opinion, it could even make the thing worse. User > > requires to get precise samplings, but perf silently returns imprecise > > records, this would mislead user. > > Mostly just the unwind might be off a little, the rest is accurate. This > has been the case 'forever'. Performance analysis isn't for silly > people, if they can't deal with a little fuzz then perhaps they're in > the wrong business. Is the main problem that the stack doesn't agree? Perhaps there could be a check for regs->rsp == pebs->user rsp (if in user space) to detect problematic samples. The question is how to report it and who should do the checking. It may need new fields in the ABI either to communicate the extra PEBS RSP or a bit to indicate that there might be a mismatch. I guess checking in the kernel and reporting an error might be simpler and maybe cleaner, but it would likely limit more advanced recovery possibilities. Are there other mismatches that break the unwinding? Perhaps the same for RBP? -Andi ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-08 15:10 ` Andi Kleen @ 2026-09-08 20:56 ` Ian Rogers 2026-09-09 0:59 ` Mi, Dapeng ` (2 more replies) 0 siblings, 3 replies; 20+ messages in thread From: Ian Rogers @ 2026-09-08 20:56 UTC (permalink / raw) To: Andi Kleen Cc: Peter Zijlstra, Mi, Dapeng, Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Adrian Hunter, Alexander Shishkin, Eranian Stephane, linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava, Ravi Bangoria On Tue, Sep 8, 2026 at 8:10 AM Andi Kleen <ak@linux.intel.com> wrote: > > > That's what we already do, no? I have distinct memories of making the > > stack unwind use the NMI regs rather then the PEBS regs. > > > > > In my opinion, it could even make the thing worse. User > > > requires to get precise samplings, but perf silently returns imprecise > > > records, this would mislead user. > > > > Mostly just the unwind might be off a little, the rest is accurate. This > > has been the case 'forever'. Performance analysis isn't for silly > > people, if they can't deal with a little fuzz then perhaps they're in > > the wrong business. > > Is the main problem that the stack doesn't agree? Perhaps there > could be a check for regs->rsp == pebs->user rsp (if in user space) > to detect problematic samples. > > The question is how to report it and who should do the checking. > > It may need new fields in the ABI either to communicate the extra PEBS RSP > or a bit to indicate that there might be a mismatch. > > I guess checking in the kernel and reporting an error might be simpler > and maybe cleaner, but it would likely limit more advanced recovery > possibilities. > > Are there other mismatches that break the unwinding? Perhaps the same > for RBP? For DWARF unwinding any register may be the source of a frame pointer (e.g. the OpenSSL library would use R11 rather than RBP). There is redundancy on x86 you can sample the PERF_REG_X86_IP register in the user register and there is PERF_SAMPLE_IP in the sample event itself. My understanding is that IBS can only sample IP and so for precise samples we can use PERF_SAMPLE_IP as the precise location and the user register PERF_REG_X86_IP as the interrupt IP - this would match the other register values in the interrupt. In DWARF unwinding, we initialize the register state using the sampled user registers: https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/unwind-libdw.c?h=perf-tools-next#n270 and on x86 we sample all registers for DWARF unwinding: https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/arch/x86/include/perf_regs.h?h=perf-tools-next#n20 https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/perf-regs-arch/perf_regs_x86.c?h=perf-tools-next#n238 Having PERF_SAMPLE_IP be precise and the user registers from the interrupt I believe works for AMD IBS and ARM SPE, but for Intel PEBS there is the ability to use the PEBS register samples for other non-redundant registers. In the x86 driver could we disable PEBS sampling for these registers when doing user stack sampling, so that the sampled user registers match the stack sample? We can keep the PERF_SAMPLE_IP precise, and make all the registers precise when there is no stack sampling. Thanks, Ian ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-08 20:56 ` Ian Rogers @ 2026-09-09 0:59 ` Mi, Dapeng 2026-09-09 1:28 ` Ravi Bangoria 2026-09-09 8:11 ` Peter Zijlstra 2026-09-09 14:14 ` Namhyung Kim 2 siblings, 1 reply; 20+ messages in thread From: Mi, Dapeng @ 2026-09-09 0:59 UTC (permalink / raw) To: Ian Rogers, Andi Kleen Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Adrian Hunter, Alexander Shishkin, Eranian Stephane, linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava, Ravi Bangoria On 9/9/2026 4:56 AM, Ian Rogers wrote: > On Tue, Sep 8, 2026 at 8:10 AM Andi Kleen <ak@linux.intel.com> wrote: >>> That's what we already do, no? I have distinct memories of making the >>> stack unwind use the NMI regs rather then the PEBS regs. >>> >>>> In my opinion, it could even make the thing worse. User >>>> requires to get precise samplings, but perf silently returns imprecise >>>> records, this would mislead user. >>> Mostly just the unwind might be off a little, the rest is accurate. This >>> has been the case 'forever'. Performance analysis isn't for silly >>> people, if they can't deal with a little fuzz then perhaps they're in >>> the wrong business. >> Is the main problem that the stack doesn't agree? Perhaps there >> could be a check for regs->rsp == pebs->user rsp (if in user space) >> to detect problematic samples. >> >> The question is how to report it and who should do the checking. >> >> It may need new fields in the ABI either to communicate the extra PEBS RSP >> or a bit to indicate that there might be a mismatch. >> >> I guess checking in the kernel and reporting an error might be simpler >> and maybe cleaner, but it would likely limit more advanced recovery >> possibilities. >> >> Are there other mismatches that break the unwinding? Perhaps the same >> for RBP? > For DWARF unwinding any register may be the source of a frame pointer > (e.g. the OpenSSL library would use R11 rather than RBP). > > There is redundancy on x86 you can sample the PERF_REG_X86_IP register > in the user register and there is PERF_SAMPLE_IP in the sample event > itself. > > My understanding is that IBS can only sample IP and so for precise > samples we can use PERF_SAMPLE_IP as the precise location and the user > register PERF_REG_X86_IP as the interrupt IP - this would match the > other register values in the interrupt. > > In DWARF unwinding, we initialize the register state using the sampled > user registers: > https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/unwind-libdw.c?h=perf-tools-next#n270 > and on x86 we sample all registers for DWARF unwinding: > https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/arch/x86/include/perf_regs.h?h=perf-tools-next#n20 > https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/perf-regs-arch/perf_regs_x86.c?h=perf-tools-next#n238 > > Having PERF_SAMPLE_IP be precise and the user registers from the > interrupt I believe works for AMD IBS and ARM SPE, but for Intel PEBS > there is the ability to use the PEBS register samples for other > non-redundant registers. In the x86 driver could we disable PEBS > sampling for these registers when doing user stack sampling, so that > the sampled user registers match the stack sample? We can keep the > PERF_SAMPLE_IP precise, and make all the registers precise when there > is no stack sampling. That sounds the best way to fix this issue by decoupling PERF_SAMPLE_IP with PERF_REG_X86_IP. Then we can keep the precise SAMPLE_IP and the PMI context user register snapshot simultaneously. I would post v2 patch with this fix. Thanks. > > Thanks, > Ian ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-09 0:59 ` Mi, Dapeng @ 2026-09-09 1:28 ` Ravi Bangoria 2026-09-09 1:59 ` Mi, Dapeng 0 siblings, 1 reply; 20+ messages in thread From: Ravi Bangoria @ 2026-09-09 1:28 UTC (permalink / raw) To: Mi, Dapeng, Ian Rogers Cc: Peter Zijlstra, Andi Kleen, Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Adrian Hunter, Alexander Shishkin, Eranian Stephane, linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava, Ravi Bangoria >> Having PERF_SAMPLE_IP be precise and the user registers from the >> interrupt I believe works for AMD IBS and ARM SPE, but for Intel PEBS >> there is the ability to use the PEBS register samples for other >> non-redundant registers. In the x86 driver could we disable PEBS >> sampling for these registers when doing user stack sampling, so that >> the sampled user registers match the stack sample? We can keep the >> PERF_SAMPLE_IP precise, and make all the registers precise when there >> is no stack sampling. > > That sounds the best way to fix this issue by decoupling PERF_SAMPLE_IP > with PERF_REG_X86_IP. Then we can keep the precise SAMPLE_IP and the PMI > context user register snapshot simultaneously. With this, PERF_SAMPLE_IP and PERF_REG_X86_IP might be from different privileges. So, any perf code that depends on user_mode(regs) (e.g. perf_exclude_event(), _REGS_USER, _REGS_INTR, header->misc, etc.) also needs to be inspected/modified accordingly. Thanks, Ravi ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-09 1:28 ` Ravi Bangoria @ 2026-09-09 1:59 ` Mi, Dapeng 0 siblings, 0 replies; 20+ messages in thread From: Mi, Dapeng @ 2026-09-09 1:59 UTC (permalink / raw) To: Ravi Bangoria, Ian Rogers Cc: Peter Zijlstra, Andi Kleen, Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Adrian Hunter, Alexander Shishkin, Eranian Stephane, linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava On 9/9/2026 9:28 AM, Ravi Bangoria wrote: >>> Having PERF_SAMPLE_IP be precise and the user registers from the >>> interrupt I believe works for AMD IBS and ARM SPE, but for Intel PEBS >>> there is the ability to use the PEBS register samples for other >>> non-redundant registers. In the x86 driver could we disable PEBS >>> sampling for these registers when doing user stack sampling, so that >>> the sampled user registers match the stack sample? We can keep the >>> PERF_SAMPLE_IP precise, and make all the registers precise when there >>> is no stack sampling. >> That sounds the best way to fix this issue by decoupling PERF_SAMPLE_IP >> with PERF_REG_X86_IP. Then we can keep the precise SAMPLE_IP and the PMI >> context user register snapshot simultaneously. > With this, PERF_SAMPLE_IP and PERF_REG_X86_IP might be from different > privileges. So, any perf code that depends on user_mode(regs) (e.g. > perf_exclude_event(), _REGS_USER, _REGS_INTR, header->misc, etc.) also > needs to be inspected/modified accordingly. Yeah, this introduces the inconsistency between PERF_SAMPLE_IP and the pt_regs, but I suppose it should be fine. I didn't look at the details yet, but all places which call user_mode(regs) should not be impacted in theory. As long as we ensure the PERF_SAMPLE_IP doesn't leak security information like kernel address, I suppose it should be fine. Thanks. > > Thanks, > Ravi ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-08 20:56 ` Ian Rogers 2026-09-09 0:59 ` Mi, Dapeng @ 2026-09-09 8:11 ` Peter Zijlstra 2026-09-09 9:36 ` Mi, Dapeng 2026-09-09 14:14 ` Namhyung Kim 2 siblings, 1 reply; 20+ messages in thread From: Peter Zijlstra @ 2026-09-09 8:11 UTC (permalink / raw) To: Ian Rogers Cc: Andi Kleen, Mi, Dapeng, Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Adrian Hunter, Alexander Shishkin, Eranian Stephane, linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava, Ravi Bangoria On Tue, Sep 08, 2026 at 01:56:56PM -0700, Ian Rogers wrote: > On Tue, Sep 8, 2026 at 8:10 AM Andi Kleen <ak@linux.intel.com> wrote: > > > > > That's what we already do, no? I have distinct memories of making the > > > stack unwind use the NMI regs rather then the PEBS regs. > > > > > > > In my opinion, it could even make the thing worse. User > > > > requires to get precise samplings, but perf silently returns imprecise > > > > records, this would mislead user. > > > > > > Mostly just the unwind might be off a little, the rest is accurate. This > > > has been the case 'forever'. Performance analysis isn't for silly > > > people, if they can't deal with a little fuzz then perhaps they're in > > > the wrong business. > > > > Is the main problem that the stack doesn't agree? Perhaps there > > could be a check for regs->rsp == pebs->user rsp (if in user space) > > to detect problematic samples. > > > > The question is how to report it and who should do the checking. > > > > It may need new fields in the ABI either to communicate the extra PEBS RSP > > or a bit to indicate that there might be a mismatch. > > > > I guess checking in the kernel and reporting an error might be simpler > > and maybe cleaner, but it would likely limit more advanced recovery > > possibilities. > > > > Are there other mismatches that break the unwinding? Perhaps the same > > for RBP? > > For DWARF unwinding any register may be the source of a frame pointer > (e.g. the OpenSSL library would use R11 rather than RBP). > > There is redundancy on x86 you can sample the PERF_REG_X86_IP register > in the user register and there is PERF_SAMPLE_IP in the sample event > itself. > > My understanding is that IBS can only sample IP and so for precise > samples we can use PERF_SAMPLE_IP as the precise location and the user > register PERF_REG_X86_IP as the interrupt IP - this would match the > other register values in the interrupt. > > In DWARF unwinding, we initialize the register state using the sampled > user registers: Oh, I had trouble reading yesterday :/ This is about USER_STACK, not CALLCHAIN. I think we should try very hard to not use USER_STACK, it is an abomination. Instead we really should improve CALLCHAIN to be more useful. There are a pile of patches for kernel based unwinders, including for .eh_frame (if only I had time to actually go look at them). And we should probably look at doing a shadow stack based unwinder as well. This USER_STACK is really the absolute worst possible option. And perhaps refusing PEBS+USER_STACK makes sense. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-09 8:11 ` Peter Zijlstra @ 2026-09-09 9:36 ` Mi, Dapeng 2026-09-09 19:30 ` Namhyung Kim 0 siblings, 1 reply; 20+ messages in thread From: Mi, Dapeng @ 2026-09-09 9:36 UTC (permalink / raw) To: Peter Zijlstra, Ian Rogers Cc: Andi Kleen, Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Adrian Hunter, Alexander Shishkin, Eranian Stephane, linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava, Ravi Bangoria On 9/9/2026 4:11 PM, Peter Zijlstra wrote: > On Tue, Sep 08, 2026 at 01:56:56PM -0700, Ian Rogers wrote: >> On Tue, Sep 8, 2026 at 8:10 AM Andi Kleen <ak@linux.intel.com> wrote: >>>> That's what we already do, no? I have distinct memories of making the >>>> stack unwind use the NMI regs rather then the PEBS regs. >>>> >>>>> In my opinion, it could even make the thing worse. User >>>>> requires to get precise samplings, but perf silently returns imprecise >>>>> records, this would mislead user. >>>> Mostly just the unwind might be off a little, the rest is accurate. This >>>> has been the case 'forever'. Performance analysis isn't for silly >>>> people, if they can't deal with a little fuzz then perhaps they're in >>>> the wrong business. >>> Is the main problem that the stack doesn't agree? Perhaps there >>> could be a check for regs->rsp == pebs->user rsp (if in user space) >>> to detect problematic samples. >>> >>> The question is how to report it and who should do the checking. >>> >>> It may need new fields in the ABI either to communicate the extra PEBS RSP >>> or a bit to indicate that there might be a mismatch. >>> >>> I guess checking in the kernel and reporting an error might be simpler >>> and maybe cleaner, but it would likely limit more advanced recovery >>> possibilities. >>> >>> Are there other mismatches that break the unwinding? Perhaps the same >>> for RBP? >> For DWARF unwinding any register may be the source of a frame pointer >> (e.g. the OpenSSL library would use R11 rather than RBP). >> >> There is redundancy on x86 you can sample the PERF_REG_X86_IP register >> in the user register and there is PERF_SAMPLE_IP in the sample event >> itself. >> >> My understanding is that IBS can only sample IP and so for precise >> samples we can use PERF_SAMPLE_IP as the precise location and the user >> register PERF_REG_X86_IP as the interrupt IP - this would match the >> other register values in the interrupt. >> >> In DWARF unwinding, we initialize the register state using the sampled >> user registers: > Oh, I had trouble reading yesterday :/ This is about USER_STACK, not > CALLCHAIN. Yes, this is about USR_STACK. The CALLCHAIN doesn't suffer this issue since perf already returns an IP chain and user space can directly map them to the symbols without depending on any register or stack snapshots. :) > > I think we should try very hard to not use USER_STACK, it is an > abomination. Instead we really should improve CALLCHAIN to be more > useful. There are a pile of patches for kernel based unwinders, > including for .eh_frame (if only I had time to actually go look at > them). > > And we should probably look at doing a shadow stack based unwinder as > well. > > This USER_STACK is really the absolute worst possible option. And > perhaps refusing PEBS+USER_STACK makes sense. If no others insist to implement the precise USER_STACK sampling (what Ian suggested), I would give up for sending the drafted precise USER_STACK sampling patches. Per my understanding, refusing precise USER_STACK sampling what current version does is a cleaner and simpler way. Supporting precise USER_STACK sampling inevitably complexes the PEBS/IBS handling. BTW, currently perf tools already support the events creation fallback. As long as user doesn't explicitly require precise USER_STACK sampling, the USER_STACK sampling (--call-graph dwarf) would automatically fallback to the PMI-based USER_STACK sampling after the initial precise USER_STACK sampling try fails. So it won't really lead to the USER_STACK sampling fails. Thanks. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-09 9:36 ` Mi, Dapeng @ 2026-09-09 19:30 ` Namhyung Kim 2026-09-10 0:12 ` Mi, Dapeng 0 siblings, 1 reply; 20+ messages in thread From: Namhyung Kim @ 2026-09-09 19:30 UTC (permalink / raw) To: Mi, Dapeng Cc: Peter Zijlstra, Ian Rogers, Andi Kleen, Ingo Molnar, Arnaldo Carvalho de Melo, Adrian Hunter, Alexander Shishkin, Eranian Stephane, linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava, Ravi Bangoria Hello, I missed this thread before sending my previous reply. On Wed, Sep 09, 2026 at 05:36:45PM +0800, Mi, Dapeng wrote: > > On 9/9/2026 4:11 PM, Peter Zijlstra wrote: > > On Tue, Sep 08, 2026 at 01:56:56PM -0700, Ian Rogers wrote: > >> On Tue, Sep 8, 2026 at 8:10 AM Andi Kleen <ak@linux.intel.com> wrote: > >>>> That's what we already do, no? I have distinct memories of making the > >>>> stack unwind use the NMI regs rather then the PEBS regs. > >>>> > >>>>> In my opinion, it could even make the thing worse. User > >>>>> requires to get precise samplings, but perf silently returns imprecise > >>>>> records, this would mislead user. > >>>> Mostly just the unwind might be off a little, the rest is accurate. This > >>>> has been the case 'forever'. Performance analysis isn't for silly > >>>> people, if they can't deal with a little fuzz then perhaps they're in > >>>> the wrong business. > >>> Is the main problem that the stack doesn't agree? Perhaps there > >>> could be a check for regs->rsp == pebs->user rsp (if in user space) > >>> to detect problematic samples. > >>> > >>> The question is how to report it and who should do the checking. > >>> > >>> It may need new fields in the ABI either to communicate the extra PEBS RSP > >>> or a bit to indicate that there might be a mismatch. > >>> > >>> I guess checking in the kernel and reporting an error might be simpler > >>> and maybe cleaner, but it would likely limit more advanced recovery > >>> possibilities. > >>> > >>> Are there other mismatches that break the unwinding? Perhaps the same > >>> for RBP? > >> For DWARF unwinding any register may be the source of a frame pointer > >> (e.g. the OpenSSL library would use R11 rather than RBP). > >> > >> There is redundancy on x86 you can sample the PERF_REG_X86_IP register > >> in the user register and there is PERF_SAMPLE_IP in the sample event > >> itself. > >> > >> My understanding is that IBS can only sample IP and so for precise > >> samples we can use PERF_SAMPLE_IP as the precise location and the user > >> register PERF_REG_X86_IP as the interrupt IP - this would match the > >> other register values in the interrupt. > >> > >> In DWARF unwinding, we initialize the register state using the sampled > >> user registers: > > Oh, I had trouble reading yesterday :/ This is about USER_STACK, not > > CALLCHAIN. > > Yes, this is about USR_STACK. The CALLCHAIN doesn't suffer this issue since > perf already returns an IP chain and user space can directly map them to > the symbols without depending on any register or stack snapshots. :) > > > > > > I think we should try very hard to not use USER_STACK, it is an > > abomination. Instead we really should improve CALLCHAIN to be more > > useful. There are a pile of patches for kernel based unwinders, > > including for .eh_frame (if only I had time to actually go look at > > them). That would be great! > > > > And we should probably look at doing a shadow stack based unwinder as > > well. Cool. It'd be nice to see that happen. > > > > This USER_STACK is really the absolute worst possible option. And > > perhaps refusing PEBS+USER_STACK makes sense. > > If no others insist to implement the precise USER_STACK sampling (what Ian > suggested), I would give up for sending the drafted precise USER_STACK > sampling patches. Per my understanding, refusing precise USER_STACK > sampling what current version does is a cleaner and simpler way. > Supporting precise USER_STACK sampling inevitably complexes the PEBS/IBS > handling. I think precise is for IP and it's not clear if it applies to other REGS too. But agree that it'd be simpler to reject. > > BTW, currently perf tools already support the events creation fallback. As > long as user doesn't explicitly require precise USER_STACK sampling, the > USER_STACK sampling (--call-graph dwarf) would automatically fallback to > the PMI-based USER_STACK sampling after the initial precise USER_STACK > sampling try fails. So it won't really lead to the USER_STACK sampling fails. That's true. perf tools can fallback to auto-reduce the precise level unless it's requested manually. If we go on this direction, I think we should mention this in the man page though. Probably the condition is precise_ip + USER_REGS + USER_STACK. Thanks, Namhyung ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-09 19:30 ` Namhyung Kim @ 2026-09-10 0:12 ` Mi, Dapeng 0 siblings, 0 replies; 20+ messages in thread From: Mi, Dapeng @ 2026-09-10 0:12 UTC (permalink / raw) To: Namhyung Kim Cc: Peter Zijlstra, Ian Rogers, Andi Kleen, Ingo Molnar, Arnaldo Carvalho de Melo, Adrian Hunter, Alexander Shishkin, Eranian Stephane, linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava, Ravi Bangoria On 9/10/2026 3:30 AM, Namhyung Kim wrote: > Hello, > > I missed this thread before sending my previous reply. > > On Wed, Sep 09, 2026 at 05:36:45PM +0800, Mi, Dapeng wrote: >> On 9/9/2026 4:11 PM, Peter Zijlstra wrote: >>> On Tue, Sep 08, 2026 at 01:56:56PM -0700, Ian Rogers wrote: >>>> On Tue, Sep 8, 2026 at 8:10 AM Andi Kleen <ak@linux.intel.com> wrote: >>>>>> That's what we already do, no? I have distinct memories of making the >>>>>> stack unwind use the NMI regs rather then the PEBS regs. >>>>>> >>>>>>> In my opinion, it could even make the thing worse. User >>>>>>> requires to get precise samplings, but perf silently returns imprecise >>>>>>> records, this would mislead user. >>>>>> Mostly just the unwind might be off a little, the rest is accurate. This >>>>>> has been the case 'forever'. Performance analysis isn't for silly >>>>>> people, if they can't deal with a little fuzz then perhaps they're in >>>>>> the wrong business. >>>>> Is the main problem that the stack doesn't agree? Perhaps there >>>>> could be a check for regs->rsp == pebs->user rsp (if in user space) >>>>> to detect problematic samples. >>>>> >>>>> The question is how to report it and who should do the checking. >>>>> >>>>> It may need new fields in the ABI either to communicate the extra PEBS RSP >>>>> or a bit to indicate that there might be a mismatch. >>>>> >>>>> I guess checking in the kernel and reporting an error might be simpler >>>>> and maybe cleaner, but it would likely limit more advanced recovery >>>>> possibilities. >>>>> >>>>> Are there other mismatches that break the unwinding? Perhaps the same >>>>> for RBP? >>>> For DWARF unwinding any register may be the source of a frame pointer >>>> (e.g. the OpenSSL library would use R11 rather than RBP). >>>> >>>> There is redundancy on x86 you can sample the PERF_REG_X86_IP register >>>> in the user register and there is PERF_SAMPLE_IP in the sample event >>>> itself. >>>> >>>> My understanding is that IBS can only sample IP and so for precise >>>> samples we can use PERF_SAMPLE_IP as the precise location and the user >>>> register PERF_REG_X86_IP as the interrupt IP - this would match the >>>> other register values in the interrupt. >>>> >>>> In DWARF unwinding, we initialize the register state using the sampled >>>> user registers: >>> Oh, I had trouble reading yesterday :/ This is about USER_STACK, not >>> CALLCHAIN. >> Yes, this is about USR_STACK. The CALLCHAIN doesn't suffer this issue since >> perf already returns an IP chain and user space can directly map them to >> the symbols without depending on any register or stack snapshots. :) >> >> >>> I think we should try very hard to not use USER_STACK, it is an >>> abomination. Instead we really should improve CALLCHAIN to be more >>> useful. There are a pile of patches for kernel based unwinders, >>> including for .eh_frame (if only I had time to actually go look at >>> them). > That would be great! > >>> And we should probably look at doing a shadow stack based unwinder as >>> well. > Cool. It'd be nice to see that happen. > >>> This USER_STACK is really the absolute worst possible option. And >>> perhaps refusing PEBS+USER_STACK makes sense. >> If no others insist to implement the precise USER_STACK sampling (what Ian >> suggested), I would give up for sending the drafted precise USER_STACK >> sampling patches. Per my understanding, refusing precise USER_STACK >> sampling what current version does is a cleaner and simpler way. >> Supporting precise USER_STACK sampling inevitably complexes the PEBS/IBS >> handling. > I think precise is for IP and it's not clear if it applies to other REGS > too. But agree that it'd be simpler to reject. > >> BTW, currently perf tools already support the events creation fallback. As >> long as user doesn't explicitly require precise USER_STACK sampling, the >> USER_STACK sampling (--call-graph dwarf) would automatically fallback to >> the PMI-based USER_STACK sampling after the initial precise USER_STACK >> sampling try fails. So it won't really lead to the USER_STACK sampling fails. > That's true. perf tools can fallback to auto-reduce the precise level > unless it's requested manually. If we go on this direction, I think we > should mention this in the man page though. Probably the condition is > precise_ip + USER_REGS + USER_STACK. Good idea. I would add an independent perf-tools patch to mention this once we decide to go on this direction. Thanks. > > Thanks, > Namhyung ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-08 20:56 ` Ian Rogers 2026-09-09 0:59 ` Mi, Dapeng 2026-09-09 8:11 ` Peter Zijlstra @ 2026-09-09 14:14 ` Namhyung Kim 2 siblings, 0 replies; 20+ messages in thread From: Namhyung Kim @ 2026-09-09 14:14 UTC (permalink / raw) To: Ian Rogers Cc: Andi Kleen, Peter Zijlstra, Mi, Dapeng, Ingo Molnar, Arnaldo Carvalho de Melo, Adrian Hunter, Alexander Shishkin, Eranian Stephane, linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava, Ravi Bangoria Hello, On Tue, Sep 08, 2026 at 01:56:56PM -0700, Ian Rogers wrote: > On Tue, Sep 8, 2026 at 8:10 AM Andi Kleen <ak@linux.intel.com> wrote: > > > > > That's what we already do, no? I have distinct memories of making the > > > stack unwind use the NMI regs rather then the PEBS regs. > > > > > > > In my opinion, it could even make the thing worse. User > > > > requires to get precise samplings, but perf silently returns imprecise > > > > records, this would mislead user. > > > > > > Mostly just the unwind might be off a little, the rest is accurate. This > > > has been the case 'forever'. Performance analysis isn't for silly > > > people, if they can't deal with a little fuzz then perhaps they're in > > > the wrong business. > > > > Is the main problem that the stack doesn't agree? Perhaps there > > could be a check for regs->rsp == pebs->user rsp (if in user space) > > to detect problematic samples. > > > > The question is how to report it and who should do the checking. > > > > It may need new fields in the ABI either to communicate the extra PEBS RSP > > or a bit to indicate that there might be a mismatch. > > > > I guess checking in the kernel and reporting an error might be simpler > > and maybe cleaner, but it would likely limit more advanced recovery > > possibilities. > > > > Are there other mismatches that break the unwinding? Perhaps the same > > for RBP? > > For DWARF unwinding any register may be the source of a frame pointer > (e.g. the OpenSSL library would use R11 rather than RBP). > > There is redundancy on x86 you can sample the PERF_REG_X86_IP register > in the user register and there is PERF_SAMPLE_IP in the sample event > itself. > > My understanding is that IBS can only sample IP and so for precise > samples we can use PERF_SAMPLE_IP as the precise location and the user > register PERF_REG_X86_IP as the interrupt IP - this would match the > other register values in the interrupt. > > In DWARF unwinding, we initialize the register state using the sampled > user registers: > https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/unwind-libdw.c?h=perf-tools-next#n270 > and on x86 we sample all registers for DWARF unwinding: > https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/arch/x86/include/perf_regs.h?h=perf-tools-next#n20 > https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/perf-regs-arch/perf_regs_x86.c?h=perf-tools-next#n238 > > Having PERF_SAMPLE_IP be precise and the user registers from the > interrupt I believe works for AMD IBS and ARM SPE, but for Intel PEBS > there is the ability to use the PEBS register samples for other > non-redundant registers. In the x86 driver could we disable PEBS > sampling for these registers when doing user stack sampling, so that > the sampled user registers match the stack sample? We can keep the > PERF_SAMPLE_IP precise, and make all the registers precise when there > is no stack sampling. +1. I think it's better to keep PERF_SAMPLE_IP precise as requested and pick appropriate registers based on PERF_SAMPLE_USER_STACK. Thanks, Namhyung ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-08 10:19 ` Peter Zijlstra 2026-09-08 15:10 ` Andi Kleen @ 2026-09-09 1:20 ` Mi, Dapeng 1 sibling, 0 replies; 20+ messages in thread From: Mi, Dapeng @ 2026-09-09 1:20 UTC (permalink / raw) To: Peter Zijlstra Cc: Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin, Andi Kleen, Eranian Stephane, linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava, Ravi Bangoria On 9/8/2026 6:19 PM, Peter Zijlstra wrote: > On Tue, Sep 08, 2026 at 04:56:22PM +0800, Mi, Dapeng wrote: >> On 9/8/2026 4:49 PM, Peter Zijlstra wrote: >>> On Tue, Sep 08, 2026 at 03:51:02PM +0800, Dapeng Mi wrote: >>>> PERF_SAMPLE_STACK_USER needs to return the user stack and user registers >>>> to user space when the PMI exits. Since the skid from the PEBS/IBS sample >>>> and PMI delivery, the PEBS/IBS register snapshot (especially IP/SP/BP) >>>> can diverge from the user stack at PMI return. That mismatch breaks DWARF >>>> unwinding. >>>> >>>> Precise sampling provides no benefit in this case, so disable PEBS/IBS >>>> precise sampling and allow only PMI-based sampling when >>>> PERF_SAMPLE_STACK_USER is requested. >>>> >>>> Reported-by: Gennady Kupava <gennady.kupava@gmail.com> >>>> Closes: https://lore.kernel.org/all/CAPu-DQqF0aF6=GS8Z6KKWeeX_V5LiXeKU_rJQZC+uGg8zuTPNw@mail.gmail.com/ >>>> Cc: Ravi Bangoria <ravi.bangoria@amd.com> >>>> Fixes: c5ebcedb566e ("perf: Add ability to attach user stack dump to sample") >>>> Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com> >>> This breaks long standing existing behaviour. >> Yeah, but it seems there is no better way to fix this issue. > Breaking things that worked before isn't fixing.. people get upset. > >> An alternative way to fix this issue is still to return the PMI >> context register state rather than the PEBS precise registers for user >> stack sampling, but this actually falls back the imprecise PMI-based >> sampling. > That's what we already do, no? I have distinct memories of making the > stack unwind use the NMI regs rather then the PEBS regs. Unfortunately it's not. :( Currently pt_regs->ip would be unconditionally overwritten by PEBS/IBS snapshotted IP register value, and then the pt_regs->ip is used to generated the SAMPLE_IP. if (filtered_sample_type & PERF_SAMPLE_IP) { data->ip = perf_instruction_pointer(event, regs); data->sample_flags |= PERF_SAMPLE_IP; } As Ian suggested, the better way to fix this issue could be to decouple PERF_SAMPLE_IP and PERF_REG_X86_IP. PERF_SAMPLE_IP still stores the precise IP from PEBS/IBS, but the whole user register snapshot keeps the PMI context registers. DWARF depends on the user register snapshot to unwind the call chain instead of PERF_SAMPLE_IP. I would follow this way and send V2 patches. Thanks. > >> In my opinion, it could even make the thing worse. User >> requires to get precise samplings, but perf silently returns imprecise >> records, this would mislead user. > Mostly just the unwind might be off a little, the rest is accurate. This > has been the case 'forever'. Performance analysis isn't for silly > people, if they can't deal with a little fuzz then perhaps they're in > the wrong business. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER 2026-09-08 7:51 ` [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER Dapeng Mi 2026-09-08 8:15 ` sashiko-bot 2026-09-08 8:49 ` Peter Zijlstra @ 2026-09-08 8:49 ` Mi, Dapeng 2 siblings, 0 replies; 20+ messages in thread From: Mi, Dapeng @ 2026-09-08 8:49 UTC (permalink / raw) To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Ian Rogers, Adrian Hunter, Alexander Shishkin, Andi Kleen, Eranian Stephane Cc: linux-kernel, linux-perf-users, Dapeng Mi, Zide Chen, Falcon Thomas, Xudong Hao, Gennady Kupava, Ravi Bangoria The patch is only validated on Intel platforms. @Ravi may help to validate on AMD platforms. Thanks. On 9/8/2026 3:51 PM, Dapeng Mi wrote: > PERF_SAMPLE_STACK_USER needs to return the user stack and user registers > to user space when the PMI exits. Since the skid from the PEBS/IBS sample > and PMI delivery, the PEBS/IBS register snapshot (especially IP/SP/BP) > can diverge from the user stack at PMI return. That mismatch breaks DWARF > unwinding. > > Precise sampling provides no benefit in this case, so disable PEBS/IBS > precise sampling and allow only PMI-based sampling when > PERF_SAMPLE_STACK_USER is requested. > > Reported-by: Gennady Kupava <gennady.kupava@gmail.com> > Closes: https://lore.kernel.org/all/CAPu-DQqF0aF6=GS8Z6KKWeeX_V5LiXeKU_rJQZC+uGg8zuTPNw@mail.gmail.com/ > Cc: Ravi Bangoria <ravi.bangoria@amd.com> > Fixes: c5ebcedb566e ("perf: Add ability to attach user stack dump to sample") > Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com> > --- > arch/x86/events/amd/ibs.c | 3 +++ > arch/x86/events/core.c | 3 +++ > arch/x86/events/perf_event.h | 19 +++++++++++++++++++ > 3 files changed, 25 insertions(+) > > diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c > index 3531f9c23b8c..c67b659d64e2 100644 > --- a/arch/x86/events/amd/ibs.c > +++ b/arch/x86/events/amd/ibs.c > @@ -327,6 +327,9 @@ static int perf_ibs_init(struct perf_event *event) > if (has_branch_stack(event)) > return -EOPNOTSUPP; > > + if (!x86_pmu_allow_sample_user_stack(event, true)) > + return -EINVAL; > + > /* handle exclude_{user,kernel} in the IRQ handler */ > if (event->attr.exclude_host || event->attr.exclude_guest || > event->attr.exclude_idle) > diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c > index 8b3ea0adb965..ceb5038d7565 100644 > --- a/arch/x86/events/core.c > +++ b/arch/x86/events/core.c > @@ -594,6 +594,9 @@ int x86_pmu_hw_config(struct perf_event *event) > /* There's no sense in having PEBS for non sampling events: */ > if (!is_sampling_event(event)) > return -EINVAL; > + > + if (!x86_pmu_allow_sample_user_stack(event, false)) > + return -EINVAL; > } > /* > * check that PEBS LBR correction does not conflict with > diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h > index 71ed5b2acea2..9fbcf2bcef51 100644 > --- a/arch/x86/events/perf_event.h > +++ b/arch/x86/events/perf_event.h > @@ -1274,6 +1274,25 @@ static inline bool is_counter_pair(struct hw_perf_event *hwc) > return hwc->flags & PERF_X86_EVENT_PAIR; > } > > +static inline bool > +x86_pmu_allow_sample_user_stack(struct perf_event *event, bool ibs) > +{ > + if (!(event->attr.sample_type & PERF_SAMPLE_STACK_USER)) > + return true; > + > + /* > + * PERF_SAMPLE_STACK_USER needs to return the user stack and > + * user registers to user space when the PMI exits. Since the skid > + * from the PEBS/IBS sample and PMI delivery, the PEBS/IBS register > + * snapshot (especially IP/SP/BP) can diverge from the user stack > + * at PMI return. That mismatch breaks DWARF unwinding. > + * > + * Therefore, disable precise sampling for PERF_SAMPLE_STACK_USER > + * and permit only PMI-based sampling for this case. > + */ > + return !(event->attr.precise_ip || ibs); > +} > + > static inline void __x86_pmu_enable_event(struct hw_perf_event *hwc, > u64 enable_mask) > { ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/2] perf/x86/intel: Correct pt_regs->flags update for PEBS path 2026-09-08 7:51 [PATCH 1/2] perf/x86/intel: Correct pt_regs->flags update for PEBS path Dapeng Mi 2026-09-08 7:51 ` [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER Dapeng Mi @ 2026-09-08 8:05 ` sashiko-bot 1 sibling, 0 replies; 20+ messages in thread From: sashiko-bot @ 2026-09-08 8:05 UTC (permalink / raw) To: Dapeng Mi; +Cc: linux-perf-users > pt_regs->flags holds the saved CPU FLAGS register. In the PEBS path, > it was incorrectly set to PERF_EFLAGS_EXACT instead of being populated > from the PEBS flags snapshot. > > Update pt_regs->flags from PEBS GPR flags if GPRs group is present. > > Fixes: c22497f5838c ("perf/x86/intel: Support adaptive PEBS v4") > Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com> Sashiko has reviewed this patch and found no issues. It looks great! -- Sashiko AI review · https://sashiko.dev/#/patchset/20260908075102.540715-1-dapeng1.mi@linux.intel.com?part=1 ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2026-09-10 0:12 UTC | newest] Thread overview: 20+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-09-08 7:51 [PATCH 1/2] perf/x86/intel: Correct pt_regs->flags update for PEBS path Dapeng Mi 2026-09-08 7:51 ` [PATCH 2/2] perf/x86: Disable precise sampling for PERF_SAMPLE_STACK_USER Dapeng Mi 2026-09-08 8:15 ` sashiko-bot 2026-09-08 8:47 ` Mi, Dapeng 2026-09-08 8:49 ` Peter Zijlstra 2026-09-08 8:56 ` Mi, Dapeng 2026-09-08 10:19 ` Peter Zijlstra 2026-09-08 15:10 ` Andi Kleen 2026-09-08 20:56 ` Ian Rogers 2026-09-09 0:59 ` Mi, Dapeng 2026-09-09 1:28 ` Ravi Bangoria 2026-09-09 1:59 ` Mi, Dapeng 2026-09-09 8:11 ` Peter Zijlstra 2026-09-09 9:36 ` Mi, Dapeng 2026-09-09 19:30 ` Namhyung Kim 2026-09-10 0:12 ` Mi, Dapeng 2026-09-09 14:14 ` Namhyung Kim 2026-09-09 1:20 ` Mi, Dapeng 2026-09-08 8:49 ` Mi, Dapeng 2026-09-08 8:05 ` [PATCH 1/2] perf/x86/intel: Correct pt_regs->flags update for PEBS path sashiko-bot
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.