[Patch v3 0/8] perf/x86: Miscellaneous PMU bug fixes and optimizations

[Patch v3 0/8] perf/x86: Miscellaneous PMU bug fixes and optimizations

[Dapeng Mi]

Changes since v2:
- Patch 6/8: Fix the typo in comments (Thomas).
- Patch 8/8: New patch which prevents drain_pebs() reentry.

Changes since v1:
- Patch 3/7: Ensure cpuc->pmu is not the static pmu before calling
  hybrid_pmu(cpuc->pmu) in intel_pmu_cpu_dead() (Sashiko).

This series fixes recently found x86 core PMU bugs. Most of bugs are
found by Sashiko in reviewing the "Support SIMD/eGPRs/SSP registers 
sampling for perf" patch-set, e.g.,
https://lore.kernel.org/all/20260706022123.480411F000E9@smtp.kernel.org/
https://lore.kernel.org/all/20260706021852.DE2ED1F000E9@smtp.kernel.org/

The patch 7/8 optimizes ACR handling in match_prev_assignment() and
mitigate the performance overhead.

The patch 8/8 enhances intel_pmu_drain_pebs_buffer() to prevent
drain_pebs() reentry.

Tests:
Below test cases are run on Diamonds and Novalake. No issues are found.
- Perf stat test
  $ perf test 119
- Perf record tests 
  $ perf test 155
- Perf record LBR tests
  $ perf test 156

History:
v2: https://lore.kernel.org/all/20260713082734.3162099-1-dapeng1.mi@linux.intel.com/ 
v1: https://lore.kernel.org/all/20260710065128.1799838-1-dapeng1.mi@linux.intel.com/

Dapeng Mi (8):
  perf/x86: Unregister PMI handler on PMU init failure
  perf/x86: Free hybrid state on PMU init failure
  perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts
  perf/x86/intel: Unwind cpuc state if PEBS buffer setup fails
  perf/x86: Remove stale fixed counter helper and fix hybrid PMU access
  perf/x86/intel: Fix intel_cap handling on hybrid PMUs
  perf/x86: Optimize ACR handling in match_prev_assignment()
  perf/x86/intel: Prevent drain_pebs() reentry

 arch/x86/events/core.c       |  45 ++++++++++++----
 arch/x86/events/intel/core.c | 101 ++++++++++++++++++++++++++---------
 arch/x86/events/intel/ds.c   |   7 ---
 arch/x86/events/perf_event.h |  12 ++---
 4 files changed, 113 insertions(+), 52 deletions(-)


base-commit: edda9051e267b7390c7ce24b1b71434414ad156e
-- 
2.34.1

[Patch v3 1/8] perf/x86: Unregister PMI handler on PMU init failure

[Dapeng Mi]

Fix an NMI handler leak in init_hw_perf_events(). When PMU
initialization fails after register_nmi_handler(), the error path
exits without calling unregister_nmi_handler(), leaving a stale
NMI_LOCAL "PMI" handler registered. Add the missing call before
clearing x86_pmu state.

Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
---
 arch/x86/events/core.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index af0b67ffb43d..872d07a5fa80 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -2219,7 +2219,7 @@ static int __init init_hw_perf_events(void)
 	err = cpuhp_setup_state(CPUHP_PERF_X86_PREPARE, "perf/x86:prepare",
 				x86_pmu_prepare_cpu, x86_pmu_dead_cpu);
 	if (err)
-		return err;
+		goto pmi_unregister;
 
 	err = cpuhp_setup_state(CPUHP_AP_PERF_X86_STARTING,
 				"perf/x86:starting", x86_pmu_starting_cpu,
@@ -2273,6 +2273,8 @@ static int __init init_hw_perf_events(void)
 	cpuhp_remove_state(CPUHP_AP_PERF_X86_STARTING);
 out:
 	cpuhp_remove_state(CPUHP_PERF_X86_PREPARE);
+pmi_unregister:
+	unregister_nmi_handler(NMI_LOCAL, "PMI");
 out_bad_pmu:
 	memset(&x86_pmu, 0, sizeof(x86_pmu));
 	return err;
-- 
2.34.1

Re: [Patch v3 1/8] perf/x86: Unregister PMI handler on PMU init failure

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Pre-existing Use-After-Free window for `cpuc->pmu` if `perf_pmu_register()` fails on hybrid systems.
--

--- Patch [1]: [Patch v3 1/8] perf/x86: Unregister PMI handler on PMU init failure ---
commit 841a0ed2adb8a95774faf3b5508e90fdd97d9135
Author: Dapeng Mi <dapeng1.mi@linux.intel.com>

perf/x86: Unregister PMI handler on PMU init failure

This commit fixes an NMI handler leak in init_hw_perf_events() by unregistering
the NMI handler when PMU initialization fails before clearing the x86_pmu state.

> diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
> index af0b67ffb43d8..872d07a5fa80d 100644
> --- a/arch/x86/events/core.c
> +++ b/arch/x86/events/core.c
[ ... ]
> @@ -2273,6 +2273,8 @@ static int __init init_hw_perf_events(void)
>  	cpuhp_remove_state(CPUHP_AP_PERF_X86_STARTING);
>  out:
>  	cpuhp_remove_state(CPUHP_PERF_X86_PREPARE);
> +pmi_unregister:
> +	unregister_nmi_handler(NMI_LOCAL, "PMI");
>  out_bad_pmu:

[Severity: High]
This is a pre-existing issue, but does this error path leave a dangling
pointer that could lead to a Use-After-Free?

On hybrid systems, if perf_pmu_register() fails, the error path frees the
x86_pmu.hybrid_pmu structures. However, CPUHP_AP_PERF_X86_STARTING was already
set up, so cpuc->pmu on all online CPUs was initialized to point to these
now-freed structures.

The CPU hotplug state rollback doesn't set cpuc->pmu back to NULL. Since
x86_pmu_static_call_update() has already routed x86_pmu_guest_get_msrs to
intel_guest_get_msrs(), and the perf_is_hybrid static key remains enabled,
could a subsequent KVM guest launch trigger a UAF?

When KVM calls perf_guest_get_msrs(), it jumps to intel_guest_get_msrs(),
which unconditionally dereferences the dangling cpuc->pmu pointer:

arch/x86/events/intel/core.c:intel_guest_get_msrs() {
    ...
    u64 intel_ctrl = hybrid(cpuc->pmu, intel_ctrl);
    ...
}

>  	memset(&x86_pmu, 0, sizeof(x86_pmu));
>  	return err;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260717080342.1879573-1-dapeng1.mi@linux.intel.com?part=1

[Patch v3 2/8] perf/x86: Free hybrid state on PMU init failure

[Dapeng Mi]

If PMU initialization fails, for example in check_hw_exists(), hybrid
state can be left partially initialized: x86_pmu.hybrid_pmu is not freed
and perf_is_hybrid remains set. This can leak memory and leave stale
hybrid state reachable after a failed init path.

Add x86_pmu_free_hybrid() and use it on PMU init failure paths so all
hybrid-related state is consistently reset.

Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
---
 arch/x86/events/core.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index 872d07a5fa80..6c63b27e11e6 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -2130,6 +2130,17 @@ void x86_pmu_show_pmu_cap(struct pmu *pmu)
 	pr_info("... global_ctrl mask:          %016llx\n", hybrid(pmu, intel_ctrl));
 }
 
+static void x86_pmu_free_hybrid(void)
+{
+	if (!x86_pmu.hybrid_pmu)
+		return;
+
+	static_branch_disable(&perf_is_hybrid);
+	kfree(x86_pmu.hybrid_pmu);
+	x86_pmu.hybrid_pmu = NULL;
+	x86_pmu.num_hybrid_pmus = 0;
+}
+
 static int __init init_hw_perf_events(void)
 {
 	struct x86_pmu_quirk *quirk;
@@ -2258,9 +2269,6 @@ static int __init init_hw_perf_events(void)
 			for (j = 0; j < i; j++)
 				perf_pmu_unregister(&x86_pmu.hybrid_pmu[j].pmu);
 			pr_warn("Failed to register hybrid PMUs\n");
-			kfree(x86_pmu.hybrid_pmu);
-			x86_pmu.hybrid_pmu = NULL;
-			x86_pmu.num_hybrid_pmus = 0;
 			goto out2;
 		}
 	}
@@ -2276,6 +2284,7 @@ static int __init init_hw_perf_events(void)
 pmi_unregister:
 	unregister_nmi_handler(NMI_LOCAL, "PMI");
 out_bad_pmu:
+	x86_pmu_free_hybrid();
 	memset(&x86_pmu, 0, sizeof(x86_pmu));
 	return err;
 }
-- 
2.34.1

[Patch v3 3/8] perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts

[Dapeng Mi]

In failure paths, cpuc->pmu can still point to the global static pmu
instead of an embedded x86_hybrid_pmu::pmu. Calling hybrid_pmu() on
that pointer causes an invalid container conversion and may lead to
out-of-bounds access.

This can happen in at least two cases:
- init_hybrid_pmu() fails check_hw_exists() and leaves cpuc->pmu as-is.
- CPU hotplug fails between CPUHP_PERF_X86_PREPARE and
CPUHP_AP_PERF_X86_STARTING, and rollback invokes intel_pmu_cpu_dead().

Fix both paths by:
- Clear cpuc->pmu to NULL when check_hw_exists() fails.
- Validat that cpuc->pmu is not the global static pmu before calling
  hybrid_pmu() in intel_pmu_cpu_dead().

A new helper x86_get_static_pmu() is added to get the global static pmu.

Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
---
 arch/x86/events/core.c       | 5 +++++
 arch/x86/events/intel/core.c | 7 +++++--
 arch/x86/events/perf_event.h | 1 +
 3 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index 6c63b27e11e6..a02f303a9151 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -790,6 +790,11 @@ int is_x86_event(struct perf_event *event)
 	return false;
 }
 
+inline struct pmu *x86_get_static_pmu(void)
+{
+	return &pmu;
+}
+
 struct pmu *x86_get_pmu(unsigned int cpu)
 {
 	struct cpu_hw_events *cpuc = &per_cpu(cpu_hw_events, cpu);
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index b39c6ce0efb5..a991fc4f1575 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -6329,8 +6329,10 @@ static bool init_hybrid_pmu(int cpu)
 
 	intel_pmu_check_hybrid_pmus(pmu);
 
-	if (!check_hw_exists(&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask))
+	if (!check_hw_exists(&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) {
+		cpuc->pmu = NULL;
 		return false;
+	}
 
 	pr_info("%s PMU driver: ", pmu->name);
 
@@ -6475,11 +6477,12 @@ void intel_cpuc_finish(struct cpu_hw_events *cpuc)
 static void intel_pmu_cpu_dead(int cpu)
 {
 	struct cpu_hw_events *cpuc = &per_cpu(cpu_hw_events, cpu);
+	struct pmu *pmu = x86_get_static_pmu();
 
 	release_arch_pebs_buf_on_cpu(cpu);
 	intel_cpuc_finish(cpuc);
 
-	if (is_hybrid() && cpuc->pmu)
+	if (is_hybrid() && cpuc->pmu && cpuc->pmu != pmu)
 		cpumask_clear_cpu(cpu, &hybrid_pmu(cpuc->pmu)->supported_cpus);
 }
 
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index a8afea8d38f0..01ae287cde16 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -1161,6 +1161,7 @@ static struct perf_pmu_format_hybrid_attr format_attr_hybrid_##_name = {\
 	.pmu_type	= _pmu,						\
 }
 
+struct pmu *x86_get_static_pmu(void);
 struct pmu *x86_get_pmu(unsigned int cpu);
 extern struct x86_pmu x86_pmu __read_mostly;
 
-- 
2.34.1

Re: [Patch v3 3/8] perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] Setting `cpuc->pmu = NULL` on initialization failure triggers a user-triggerable `WARN_ON_ONCE` in `x86_get_pmu()`.

Pre-existing issues:
- [High] Unchecked MSR writes in NMI handler cause a #GP fault on asymmetric PMU configurations.
--

--- Patch [3]: [Patch v3 3/8] perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts ---
commit 2ef12c59a0f0eb0a83a45dd11cf187ca55beecdf
Author: Dapeng Mi <dapeng1.mi@linux.intel.com>

perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts
    
Fix invalid container conversions in failure paths by clearing cpuc->pmu to
NULL when check_hw_exists() fails, and validating cpuc->pmu in
intel_pmu_cpu_dead(). A new helper x86_get_static_pmu() is added.

> diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
> index b39c6ce0efb5c..a991fc4f1575f 100644
> --- a/arch/x86/events/intel/core.c
> +++ b/arch/x86/events/intel/core.c
> @@ -6329,8 +6329,10 @@ static bool init_hybrid_pmu(int cpu)
>  
>  	intel_pmu_check_hybrid_pmus(pmu);
>  
> -	if (!check_hw_exists(&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask))
> +	if (!check_hw_exists(&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) {
> +		cpuc->pmu = NULL;

[Severity: High]
Does setting cpuc->pmu to NULL here trigger a warning in x86_get_pmu()?

If check_hw_exists() fails (e.g., due to a hypervisor restricting PMU MSRs),
we now set cpuc->pmu to NULL but leave the CPU in the possible_cpus mask.

If a user later opens a perf event utilizing Architectural LBRs (like perf
record -b), x86_reserve_hardware() calls reserve_lbr_buffers() which
iterates over all possible CPUs:

arch/x86/events/intel/lbr.c:reserve_lbr_buffers() {
	...
	for_each_possible_cpu(cpu) {
		kmem_cache = x86_get_pmu(cpu)->task_ctx_cache;
	...
}

Because the failed CPU's cpuc->pmu is NULL, x86_get_pmu() hits its strict
WARN_ON_ONCE(!cpuc->pmu) assertion. If panic_on_warn=1 is set, could this
upgrade a graceful PMU initialization failure into a local panic?

Since intel_pmu_cpu_dead() now safely guards against the invalid cast with
cpuc->pmu != pmu, is this assignment necessary?

[Severity: High]
This is a pre-existing issue, but does failing check_hw_exists() leave the
CPU vulnerable to a #GP fault during a spurious NMI?

If a CPU is brought online but its PMU MSRs are disabled by a hypervisor,
initialization for that CPU aborts here. However, perf_event_nmi_handler()
is registered globally if at least one CPU succeeds.

If an active perf event runs on a healthy CPU (so active_events > 0), and a
spurious or broadcast NMI strikes the broken CPU, it will execute:

intel_pmu_handle_irq()
    __intel_pmu_disable_all(true)
        wrmsrq(MSR_CORE_PERF_GLOBAL_CTRL, 0);

Could accessing this non-existent MSR on the failed CPU trigger an unchecked
#GP fault and kernel panic?

>  		return false;
> +	}
>  
>  	pr_info("%s PMU driver: ", pmu->name);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260717080342.1879573-1-dapeng1.mi@linux.intel.com?part=3

[Patch v3 4/8] perf/x86/intel: Unwind cpuc state if PEBS buffer setup fails

[Dapeng Mi]

intel_pmu_cpu_prepare() allocates per-CPU perf state first and then sets
up the arch PEBS buffer. If alloc_arch_pebs_buf_on_cpu() fails,
the previously allocated cpuc resources are left behind.

Make the failure path call intel_cpuc_finish(cpuc) to release the per-CPU
state allocated by intel_cpuc_prepare().

Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
---
 arch/x86/events/intel/core.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index a991fc4f1575..b47d2f00ac13 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -5924,13 +5924,20 @@ int intel_cpuc_prepare(struct cpu_hw_events *cpuc, int cpu)
 
 static int intel_pmu_cpu_prepare(int cpu)
 {
+	struct cpu_hw_events *cpuc = &per_cpu(cpu_hw_events, cpu);
 	int ret;
 
-	ret = intel_cpuc_prepare(&per_cpu(cpu_hw_events, cpu), cpu);
+	ret = intel_cpuc_prepare(cpuc, cpu);
 	if (ret)
 		return ret;
 
-	return alloc_arch_pebs_buf_on_cpu(cpu);
+	ret = alloc_arch_pebs_buf_on_cpu(cpu);
+	if (ret) {
+		intel_cpuc_finish(cpuc);
+		return ret;
+	}
+
+	return 0;
 }
 
 static void flip_smm_bit(void *data)
-- 
2.34.1

[Patch v3 5/8] perf/x86: Remove stale fixed counter helper and fix hybrid PMU access

[Dapeng Mi]

On hybrid systems, init_hw_perf_events() can call check_hw_exists() with
the global PMU pointer after perf_is_hybrid is set. In that case,
fixed_counter_disabled() uses hybrid() on a non-hybrid PMU object, so the
intel_ctrl access is taken from the wrong layout and can read out of
bounds.

fixed_counter_disabled() was added in commit 32451614da2a
("perf/x86/intel: Support CPUID 10.ECX to disable fixed counters"), when
fixed counters were tracked via num_fixed_counters. Today fixed counters
are represented by fixed_cntr_mask, so this helper is obsolete.

Remove fixed_counter_disabled() and its callers, and rely directly on the
fixed-counter bitmask. With the helper gone, check_hw_exists() no longer
needs a PMU argument, so drop that parameter as well. This removes the
invalid hybrid access and closes the out-of-bounds read risk.

Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
---
 arch/x86/events/core.c       | 8 ++------
 arch/x86/events/intel/core.c | 4 +---
 arch/x86/events/perf_event.h | 9 +--------
 3 files changed, 4 insertions(+), 17 deletions(-)

diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index a02f303a9151..143a6e735d9e 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -265,7 +265,7 @@ static void release_pmc_hardware(void) {}
 
 #endif
 
-bool check_hw_exists(struct pmu *pmu, unsigned long *cntr_mask,
+bool check_hw_exists(unsigned long *cntr_mask,
 		     unsigned long *fixed_cntr_mask)
 {
 	u64 val, val_fail = -1, val_new= ~0;
@@ -297,8 +297,6 @@ bool check_hw_exists(struct pmu *pmu, unsigned long *cntr_mask,
 		if (ret)
 			goto msr_fail;
 		for_each_set_bit(i, fixed_cntr_mask, X86_PMC_IDX_MAX) {
-			if (fixed_counter_disabled(i, pmu))
-				continue;
 			if (val & (0x03ULL << i*4)) {
 				bios_fail = 1;
 				val_fail = val;
@@ -1618,8 +1616,6 @@ void perf_event_print_debug(void)
 			cpu, idx, prev_left);
 	}
 	for_each_set_bit(idx, fixed_cntr_mask, X86_PMC_IDX_MAX) {
-		if (fixed_counter_disabled(idx, cpuc->pmu))
-			continue;
 		rdmsrq(x86_pmu_fixed_ctr_addr(idx), pmc_count);
 
 		pr_info("CPU#%d: fixed-PMC%d count: %016llx\n",
@@ -2180,7 +2176,7 @@ static int __init init_hw_perf_events(void)
 	pmu_check_apic();
 
 	/* sanity check that the hardware exists or is emulated */
-	if (!check_hw_exists(&pmu, x86_pmu.cntr_mask, x86_pmu.fixed_cntr_mask))
+	if (!check_hw_exists(x86_pmu.cntr_mask, x86_pmu.fixed_cntr_mask))
 		goto out_bad_pmu;
 
 	pr_cont("%s PMU driver.\n", x86_pmu.name);
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index b47d2f00ac13..c418176065f6 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -3713,8 +3713,6 @@ static void intel_pmu_reset(void)
 		wrmsrq_safe(x86_pmu_event_addr(idx),  0ull);
 	}
 	for_each_set_bit(idx, fixed_cntr_mask, INTEL_PMC_MAX_FIXED) {
-		if (fixed_counter_disabled(idx, cpuc->pmu))
-			continue;
 		wrmsrq_safe(x86_pmu_fixed_ctr_addr(idx), 0ull);
 	}
 
@@ -6336,7 +6334,7 @@ static bool init_hybrid_pmu(int cpu)
 
 	intel_pmu_check_hybrid_pmus(pmu);
 
-	if (!check_hw_exists(&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) {
+	if (!check_hw_exists(pmu->cntr_mask, pmu->fixed_cntr_mask)) {
 		cpuc->pmu = NULL;
 		return false;
 	}
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index 01ae287cde16..cc9cfaae4f01 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -1243,7 +1243,7 @@ static inline int x86_pmu_rdpmc_index(int index)
 	return x86_pmu.rdpmc_index ? x86_pmu.rdpmc_index(index) : index;
 }
 
-bool check_hw_exists(struct pmu *pmu, unsigned long *cntr_mask,
+bool check_hw_exists(unsigned long *cntr_mask,
 		     unsigned long *fixed_cntr_mask);
 
 int x86_add_exclusive(unsigned int what);
@@ -1456,13 +1456,6 @@ ssize_t events_hybrid_sysfs_show(struct device *dev,
 				 struct device_attribute *attr,
 				 char *page);
 
-static inline bool fixed_counter_disabled(int i, struct pmu *pmu)
-{
-	u64 intel_ctrl = hybrid(pmu, intel_ctrl);
-
-	return !(intel_ctrl >> (i + INTEL_PMC_IDX_FIXED));
-}
-
 #ifdef CONFIG_CPU_SUP_AMD
 
 int amd_pmu_init(void);
-- 
2.34.1

[Patch v3 6/8] perf/x86/intel: Fix intel_cap handling on hybrid PMUs

[Dapeng Mi]

intel_cap (IA32_PERF_CAPABILITIES) updates are currently tied to
X86_FEATURE_ARCH_PERFMON_EXT, but these are independent feature paths.
As a result, hybrid PMU capability state can be updated under the wrong
condition.

Also, intel_pmu_broken_perf_cap() is too narrow. Per RPL018, the missing
PERF_METRICS_AVAILABLE bit affects both Raptor Lake and Meteor Lake
parts, not only the currently covered subset.

Move intel_cap updates out of the ARCH_PERFMON_EXT-gated path, extend
intel_pmu_broken_perf_cap() coverage to both RPL and MTL families, and
introduce intel_update_pmu_caps() to centralize PMU capability updates.

Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
---
 arch/x86/events/intel/core.c | 41 ++++++++++++++++++++++++++----------
 1 file changed, 30 insertions(+), 11 deletions(-)

diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index c418176065f6..361f8e0ab36c 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -6147,8 +6147,15 @@ static void intel_pmu_check_extra_regs(struct extra_reg *extra_regs);
 
 static inline bool intel_pmu_broken_perf_cap(void)
 {
-	/* The Perf Metric (Bit 15) is always cleared */
-	if (boot_cpu_data.x86_vfm == INTEL_METEORLAKE ||
+	/*
+	 * The Perf Metric (Bit 15) is always cleared on P-core of
+	 * RPL and MTL. Details can be found in RPL018 erratum.
+	 * https://edc.intel.com/content/www/us/en/design/products/platforms/details/raptor-lake-s/13th-generation-core-processor-specification-update/errata-details/
+	 */
+	if (boot_cpu_data.x86_vfm == INTEL_RAPTORLAKE ||
+	    boot_cpu_data.x86_vfm == INTEL_RAPTORLAKE_P ||
+	    boot_cpu_data.x86_vfm == INTEL_RAPTORLAKE_S ||
+	    boot_cpu_data.x86_vfm == INTEL_METEORLAKE ||
 	    boot_cpu_data.x86_vfm == INTEL_METEORLAKE_L)
 		return true;
 
@@ -6183,7 +6190,7 @@ static inline void __intel_update_large_pebs_flags(struct pmu *pmu)
 
 #define counter_mask(_gp, _fixed) ((_gp) | ((u64)(_fixed) << INTEL_PMC_IDX_FIXED))
 
-static void update_pmu_cap(struct pmu *pmu)
+static void update_pmu_cap_from_perfmonext(struct pmu *pmu)
 {
 	unsigned int eax, ebx, ecx, edx;
 	union cpuid35_eax eax_0;
@@ -6241,10 +6248,24 @@ static void update_pmu_cap(struct pmu *pmu)
 		WARN_ON(x86_pmu.arch_pebs == 1);
 		x86_pmu.arch_pebs = 0;
 	}
+}
+
+static void intel_update_pmu_caps(struct pmu *pmu)
+{
+	if (this_cpu_has(X86_FEATURE_ARCH_PERFMON_EXT))
+		update_pmu_cap_from_perfmonext(pmu);
 
-	if (!intel_pmu_broken_perf_cap()) {
-		/* Perf Metric (Bit 15) and PEBS via PT (Bit 16) are hybrid enumeration */
-		rdmsrq(MSR_IA32_PERF_CAPABILITIES, hybrid(pmu, intel_cap).capabilities);
+	if (is_hybrid() && this_cpu_has(X86_FEATURE_PDCM)) {
+		rdmsrq(MSR_IA32_PERF_CAPABILITIES,
+		       hybrid(pmu, intel_cap).capabilities);
+
+		/*
+		 * Restore perf_metrics on platforms with broken
+		 * perf_capablities.
+		 */
+		if (intel_pmu_broken_perf_cap() &&
+		    hybrid_pmu(pmu)->pmu_type == hybrid_big)
+			hybrid(pmu, intel_cap).perf_metrics = 1;
 	}
 }
 
@@ -6329,9 +6350,7 @@ static bool init_hybrid_pmu(int cpu)
 	if (!cpumask_empty(&pmu->supported_cpus))
 		goto end;
 
-	if (this_cpu_has(X86_FEATURE_ARCH_PERFMON_EXT))
-		update_pmu_cap(&pmu->pmu);
-
+	intel_update_pmu_caps(&pmu->pmu);
 	intel_pmu_check_hybrid_pmus(pmu);
 
 	if (!check_hw_exists(pmu->cntr_mask, pmu->fixed_cntr_mask)) {
@@ -8828,8 +8847,8 @@ __init int intel_pmu_init(void)
 	 * from the leaf 0xa. The core specific update will be done later
 	 * when a new type is online.
 	 */
-	if (!is_hybrid() && boot_cpu_has(X86_FEATURE_ARCH_PERFMON_EXT))
-		update_pmu_cap(NULL);
+	if (!is_hybrid())
+		intel_update_pmu_caps(NULL);
 
 	if (x86_pmu.arch_pebs) {
 		static_call_update(intel_pmu_disable_event_ext,
-- 
2.34.1

[Patch v3 7/8] perf/x86: Optimize ACR handling in match_prev_assignment()

[Dapeng Mi]

match_prev_assignment() currently forces a mismatch for ACR events, so
ACR counter indices are reprogrammed on every scheduling pass. That
causes avoidable overhead because disable and enable paths must touch
multiple MSRs.

The previous ACR assignment is already cached in acr_cfg_b[]. Use that
state to compare the newly computed ACR counter indices in hwc->config1
against the cached value in acr_cfg_b[hwc->idx]. If they match, skip
unnecessary disable and enable work.

Also tighten is_acr_self_reload_event() so it first verifies the event
is an ACR event before testing for the self-reload case.

Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Reviewed-by: Thomas Falcon <thomas.falcon@intel.com>
---
 arch/x86/events/core.c       | 13 ++++++++++++-
 arch/x86/events/perf_event.h |  2 +-
 2 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index 143a6e735d9e..8b3ea0adb965 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -1297,6 +1297,17 @@ int x86_perf_rdpmc_index(struct perf_event *event)
 	return event->hw.event_base_rdpmc;
 }
 
+static inline bool acr_match_prev_indices(struct perf_event *event,
+					  struct cpu_hw_events *cpuc)
+{
+	struct hw_perf_event *hwc = &event->hw;
+
+	if (!is_acr_event_group(event))
+		return true;
+	/* ACR counter indices don't change. */
+	return hwc->config1 == cpuc->acr_cfg_b[hwc->idx];
+}
+
 static inline int match_prev_assignment(struct perf_event *event,
 					struct cpu_hw_events *cpuc,
 					int i)
@@ -1306,7 +1317,7 @@ static inline int match_prev_assignment(struct perf_event *event,
 	return hwc->idx == cpuc->assign[i] &&
 	       hwc->last_cpu == smp_processor_id() &&
 	       hwc->last_tag == cpuc->tags[i] &&
-	       !is_acr_event_group(event);
+	       acr_match_prev_indices(event, cpuc);
 }
 
 static void x86_pmu_start(struct perf_event *event, int flags);
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index cc9cfaae4f01..fa381110f7a7 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -141,7 +141,7 @@ static inline bool is_acr_self_reload_event(struct perf_event *event)
 {
 	struct hw_perf_event *hwc = &event->hw;
 
-	if (hwc->idx < 0)
+	if (hwc->idx < 0 || !is_acr_event_group(event))
 		return false;
 
 	return test_bit(hwc->idx, (unsigned long *)&hwc->config1);
-- 
2.34.1

[Patch v3 8/8] perf/x86/intel: Prevent drain_pebs() reentry

[Dapeng Mi]

The PEBS buffer is shared by all events on a CPU, so drain_pebs() must
not run concurrently. If it is reentered, one instance may observe stale
buffer state and potentially access out-of-bound memory.

Most invocations happen in NMI context, which naturally prevents reentry.
However, drain_pebs() is also reachable from process context via
intel_pmu_drain_pebs_buffer().

In those paths, the PMU is often already disabled, but not guaranteed.
For example, __intel_pmu_pebs_disable() only disables the target counter,
so other active counters can still raise a PMI and interrupt an in-flight
drain_pebs().

Introduce __intel_pmu_quiesce() and __intel_pmu_resume() helpers and
use them in intel_pmu_drain_pebs_buffer() to disable the full PMU
around the drain_pebs() call, preventing reentry.

Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
---
 arch/x86/events/intel/core.c | 40 ++++++++++++++++++++++++++++--------
 arch/x86/events/intel/ds.c   |  7 -------
 2 files changed, 32 insertions(+), 15 deletions(-)

diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 361f8e0ab36c..726b39b9bba9 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -3125,6 +3125,36 @@ static void intel_pmu_del_event(struct perf_event *event)
 		this_cpu_ptr(&cpu_hw_events)->n_late_setup--;
 }
 
+static inline void __intel_pmu_quiesce(bool pmu_enabled)
+{
+	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+
+	cpuc->enabled = 0;
+	if (pmu_enabled)
+		intel_pmu_disable_all();
+}
+
+static inline void __intel_pmu_resume(bool pmu_enabled)
+{
+	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+
+	cpuc->enabled = pmu_enabled;
+	if (pmu_enabled)
+		intel_pmu_enable_all(0);
+}
+
+void intel_pmu_drain_pebs_buffer(void)
+{
+	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+	bool pmu_enabled = cpuc->enabled;
+	struct perf_sample_data data;
+
+	/* Disable PMU so no new PMI can interrupt and re-enter drain_pebs(). */
+	__intel_pmu_quiesce(pmu_enabled);
+	static_call(x86_pmu_drain_pebs)(NULL, &data);
+	__intel_pmu_resume(pmu_enabled);
+}
+
 static int icl_set_topdown_event_period(struct perf_event *event)
 {
 	struct hw_perf_event *hwc = &event->hw;
@@ -3322,10 +3352,7 @@ static void intel_pmu_read_event(struct perf_event *event)
 		if (is_metric_event(event) && (cpuc->txn_flags & PERF_PMU_TXN_READ))
 			return;
 
-		cpuc->enabled = 0;
-		if (pmu_enabled)
-			intel_pmu_disable_all();
-
+		__intel_pmu_quiesce(pmu_enabled);
 		/*
 		 * If the PEBS counters snapshotting is enabled,
 		 * the topdown event is available in PEBS records.
@@ -3334,10 +3361,7 @@ static void intel_pmu_read_event(struct perf_event *event)
 			static_call(intel_pmu_update_topdown_event)(event, NULL);
 		else
 			intel_pmu_drain_pebs_buffer();
-
-		cpuc->enabled = pmu_enabled;
-		if (pmu_enabled)
-			intel_pmu_enable_all(0);
+		__intel_pmu_resume(pmu_enabled);
 
 		return;
 	}
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index e86e4ba91e1b..7f8b98d1837d 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -1240,13 +1240,6 @@ int intel_pmu_drain_bts_buffer(void)
 	return 1;
 }
 
-void intel_pmu_drain_pebs_buffer(void)
-{
-	struct perf_sample_data data;
-
-	static_call(x86_pmu_drain_pebs)(NULL, &data);
-}
-
 /*
  * PEBS
  */
-- 
2.34.1

Re: [Patch v3 8/8] perf/x86/intel: Prevent drain_pebs() reentry

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] TOCTOU race in intel_pmu_drain_bts_buffer leading to duplicate BTS records.
--

--- Patch [8]: [Patch v3 8/8] perf/x86/intel: Prevent drain_pebs() reentry ---
commit 77ebceadaa828dfb1bed75e560551cf29b1ae9f5
Author: Dapeng Mi <dapeng1.mi@linux.intel.com>

perf/x86/intel: Prevent drain_pebs() reentry

The PEBS buffer is shared by all events on a CPU, so drain_pebs() must
not run concurrently. This commit introduces __intel_pmu_quiesce() and
__intel_pmu_resume() helpers to disable the full PMU around the
drain_pebs() call, preventing reentry.

> diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
> index e86e4ba91e1b3..7f8b98d1837dc 100644
> --- a/arch/x86/events/intel/ds.c
> +++ b/arch/x86/events/intel/ds.c
> @@ -1240,13 +1240,6 @@ int intel_pmu_drain_bts_buffer(void)
>  	return 1;
>  }

[Severity: High]
This is a pre-existing issue, but since you are fixing the drain reentry 
race for PEBS, does intel_pmu_drain_bts_buffer() need similar protection?

If a user configures a BTS event alongside another active counter and 
disables the BTS event, the stop IPI can call this sequence without 
globally disabling the PMU:

__perf_event_stop()
  x86_pmu_stop()
    intel_pmu_disable_event()
      intel_pmu_drain_bts_buffer()

If a PMI interrupts execution precisely between reading the top pointer and 
resetting the index:

arch/x86/events/intel/ds.c:intel_pmu_drain_bts_buffer() {
    ...
    base = (struct bts_record *)(unsigned long)ds->bts_buffer_base;
    top  = (struct bts_record *)(unsigned long)ds->bts_index;

    if (top <= base)
        return 0;

    memset(&regs, 0, sizeof(regs));

    <-- PMI FIRES HERE -->

    ds->bts_index = ds->bts_buffer_base;
    ...
}

Could the NMI handler unconditionally call intel_pmu_drain_bts_buffer(), 
draining the records and resetting the index, only for the interrupted 
context to resume and push the exact same records again using its stale 
top pointer?

Would it make sense to wrap the BTS draining logic with the newly 
introduced __intel_pmu_quiesce() and __intel_pmu_resume() helpers as well?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260717080342.1879573-1-dapeng1.mi@linux.intel.com?part=8