From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A35AA37A854; Mon, 6 Jul 2026 02:02:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.19 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783303325; cv=none; b=aJtCNXKxv09CgPCzGQVQSy00urO/as2DF81C38n1GY/yGGkCQOHXzO8DsPd+RiPKaHwQ5iJyhxEId5PoRYgNcHrZVtdWE28qIBaf5xdkzkSoiMi9ivHk84NDgt8bEdHW7qjJrTnqmYyIEiQLy3BCDCJ2tNRIPNdlsnM6f0sWMVs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783303325; c=relaxed/simple; bh=V6XFfUNNcIrBxrs5T0sdWbPSWt4p6m5wyzJh1QTK3rY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fdNFdGBQH+AFG5dU0NGu85XU4DzmjcXeG41moJkYlHWZ7UuH6CkAFzaixkREj6UVMuPLInrJ8vJ371QtlZSmVicVm60VevTBt8apetjolRC7vWAla8xrGSOUH7R8b2072gQJBCfbLS9gwMMGacg2HiHyxL2MP+Y+PAGuZe2Ik1E= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ktpgA70F; arc=none smtp.client-ip=198.175.65.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ktpgA70F" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1783303324; x=1814839324; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=V6XFfUNNcIrBxrs5T0sdWbPSWt4p6m5wyzJh1QTK3rY=; b=ktpgA70FdDRMHi2HxWh7ff41vaOvOxR2zU62gTgs/RUgxHj+AGOMgNuB 0VHI5wGIiT2nCwQRuRPdhoYwD6HULo9mYRQCRMcbEGLSKhe+kSLJO/T9Y TqJPT6kVRdXPALL+RBTKJdkceOf6+ys7EHvN1CBJGg5ToEE/U2V1wUwuI 3Fkvd7qo1TzUsaVubAgC/GkHw2GQ8o2yBBrmk6Nvx58rjHSVJkvg1UEaP Zh6Xi2guxygLaXkQFyvSsfw8uyU1RpShnzV+wuFBK/OuDs1UcK9i6XFN5 5gPDoiAdftcPyjdXvHUHMhd7lOjf4dbEt/V9G8Pp9veV/ABcCO7m8x537 g==; X-CSE-ConnectionGUID: sk0Qq9u+S9+/LdKMfj6Mqw== X-CSE-MsgGUID: vniq6pgAR4eIvLO3bFXcPw== X-IronPort-AV: E=McAfee;i="6800,10657,11838"; a="83911655" X-IronPort-AV: E=Sophos;i="6.25,149,1779174000"; d="scan'208";a="83911655" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Jul 2026 19:02:04 -0700 X-CSE-ConnectionGUID: 6QuJVwHcRYuTYDeGHKUH1A== X-CSE-MsgGUID: YKxqtzH+RcmHxotnRajBBA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,149,1779174000"; d="scan'208";a="247191330" Received: from spr.sh.intel.com ([10.112.229.196]) by fmviesa009.fm.intel.com with ESMTP; 05 Jul 2026 19:01:57 -0700 From: Dapeng Mi To: Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , Thomas Gleixner , Dave Hansen , Ian Rogers , Adrian Hunter , Jiri Olsa , Alexander Shishkin , Andi Kleen , Eranian Stephane Cc: Mark Rutland , broonie@kernel.org, Ravi Bangoria , linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, Zide Chen , Falcon Thomas , Dapeng Mi , Xudong Hao , Dapeng Mi , Kan Liang Subject: [Patch v9 11/24] perf/x86: Enable XMM register sampling for non-PEBS events Date: Mon, 6 Jul 2026 09:54:26 +0800 Message-Id: <20260706015439.3040804-12-dapeng1.mi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260706015439.3040804-1-dapeng1.mi@linux.intel.com> References: <20260706015439.3040804-1-dapeng1.mi@linux.intel.com> Precedence: bulk X-Mailing-List: linux-perf-users@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Previously, XMM register sampling was only available for PEBS events starting from Icelake. Extend the support to non-PEBS events using the xsaves instruction, thereby completing the feature set. To implement this, a 64-byte aligned buffer is required. A per-CPU ext_regs_buf is introduced to store SIMD and other registers, with an approximate size of 2K. The buffer is allocated using kzalloc_node(), ensuring natural and 64-byte alignment for all kmalloc() allocations with powers of 2. XMM sampling for non-PEBS events is supported in the REGS_INTR case. Support for REGS_USER will be added in a subsequent patch. For PEBS events, XMM register sampling data is directly retrieved from PEBS records. Future support for additional vector registers (YMM/ZMM/OPMASK) is planned. An ext_regs_mask is added to track the supported vector register groups. Co-developed-by: Kan Liang Signed-off-by: Kan Liang Signed-off-by: Dapeng Mi --- arch/x86/events/core.c | 183 ++++++++++++++++++++++++++++-- arch/x86/events/intel/core.c | 32 ++++++ arch/x86/events/intel/ds.c | 12 +- arch/x86/events/perf_event.h | 13 +++ arch/x86/include/asm/fpu/xstate.h | 2 + arch/x86/include/asm/perf_event.h | 5 +- arch/x86/kernel/fpu/xstate.c | 2 +- 7 files changed, 234 insertions(+), 15 deletions(-) diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c index d83ea02e2457..3e52610510cd 100644 --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -410,6 +410,56 @@ set_ext_hw_attr(struct hw_perf_event *hwc, struct perf_event *event) return x86_pmu_extra_regs(val, event); } +static DEFINE_PER_CPU(struct xregs_state *, ext_regs_buf); + +static void release_ext_regs_buffers(void) +{ + int cpu; + + if (!x86_pmu.ext_regs_mask) + return; + + for_each_possible_cpu(cpu) { + kfree(per_cpu(ext_regs_buf, cpu)); + per_cpu(ext_regs_buf, cpu) = NULL; + } +} + +static void reserve_ext_regs_buffers(void) +{ + bool compacted = cpu_feature_enabled(X86_FEATURE_XCOMPACTED); + unsigned int size; + int cpu; + + if (!x86_pmu.ext_regs_mask) + return; + + /* +64 bytes for the 64 bytes alignment request of xsave area. */ + size = xstate_calculate_size(x86_pmu.ext_regs_mask, compacted) + 64; + + for_each_possible_cpu(cpu) { + per_cpu(ext_regs_buf, cpu) = kzalloc_node(size, GFP_KERNEL, + cpu_to_node(cpu)); + if (WARN_ON_ONCE(!per_cpu(ext_regs_buf, cpu))) + goto err; + } + + return; + +err: + release_ext_regs_buffers(); +} + +static inline struct xregs_state *get_ext_regs_buf(int cpu) +{ + void *buf = per_cpu(ext_regs_buf, cpu); + struct xregs_state *xsave; + + xsave = buf ? PTR_ALIGN(buf, 64) : NULL; + + return xsave; +} + int x86_reserve_hardware(void) { int err = 0; @@ -422,6 +472,7 @@ int x86_reserve_hardware(void) } else { reserve_ds_buffers(); reserve_lbr_buffers(); + reserve_ext_regs_buffers(); } } if (!err) @@ -438,6 +489,7 @@ void x86_release_hardware(void) release_pmc_hardware(); release_ds_buffers(); release_lbr_buffers(); + release_ext_regs_buffers(); mutex_unlock(&pmc_reserve_mutex); } } @@ -655,18 +707,20 @@ int x86_pmu_hw_config(struct perf_event *event) return -EINVAL; } - /* sample_regs_user never support XMM registers */ - if (unlikely(event->attr.sample_regs_user & PERF_REG_EXTENDED_MASK)) - return -EINVAL; - /* - * Besides the general purpose registers, XMM registers may - * be collected in PEBS on some platforms, e.g. Icelake - */ - if (unlikely(event->attr.sample_regs_intr & PERF_REG_EXTENDED_MASK)) { - if (!(event->pmu->capabilities & PERF_PMU_CAP_EXTENDED_REGS)) - return -EINVAL; + if (event->attr.sample_type & PERF_SAMPLE_REGS_INTR) { + /* + * Besides the general purpose registers, XMM registers may + * be collected as well. + */ + if (event->attr.sample_regs_intr & PERF_REG_EXTENDED_MASK) { + if (!(event->pmu->capabilities & PERF_PMU_CAP_EXTENDED_REGS)) + return -EINVAL; + } + } - if (!event->attr.precise_ip) + if (event->attr.sample_type & PERF_SAMPLE_REGS_USER) { + /* XMM registers sampling for REGS_USER is not supported yet. */ + if (event->attr.sample_regs_user & PERF_REG_EXTENDED_MASK) return -EINVAL; } @@ -1705,6 +1759,113 @@ static void x86_pmu_del(struct perf_event *event, int flags) static_call_cond(x86_pmu_del)(event); } +void x86_pmu_clear_perf_regs(struct pt_regs *regs) +{ + struct x86_perf_regs *perf_regs = container_of(regs, struct x86_perf_regs, regs); + + perf_regs->xmm_regs = NULL; +} + +static void update_perf_regs(struct x86_perf_regs *perf_regs, + struct xregs_state *xsave, u64 bitmap) +{ + u64 mask; + + if (!xsave) + return; + + /* Restrict to features actually saved by XSAVES */ + mask = bitmap & xsave->header.xfeatures; + + if (mask & XFEATURE_MASK_SSE) + perf_regs->xmm_space = xsave->i387.xmm_space; +} + +/* + * The x86 specific variant of perf_sample_regs_intr(). + * It would be extended to add more SIMD registers sampling support + * in later patches. + */ +static void x86_pmu_update_regs_intr(struct perf_event *event, + struct perf_sample_data *data, + struct pt_regs *regs, + bool exclude_kernel) +{ + if (exclude_kernel && !user_mode(regs)) { + data->regs_intr.regs = NULL; + data->regs_intr.abi = PERF_SAMPLE_REGS_ABI_NONE; + } else { + data->regs_intr.regs = regs; + data->regs_intr.abi = perf_reg_abi(current); + } + + data->dyn_size += sizeof(u64); + if (data->regs_intr.regs) { + data->dyn_size += hweight64(event->attr.sample_regs_intr) * + sizeof(u64); + } + + /* + * Set PERF_SAMPLE_REGS_INTR to bypass perf_sample_regs_intr() call + * in perf_prepare_sample() function. + */ + data->sample_flags |= PERF_SAMPLE_REGS_INTR; +} + +static void x86_pmu_sample_xregs(struct perf_event *event, + struct perf_sample_data *data, + u64 ignore_mask) +{ + struct xregs_state *xsave = get_ext_regs_buf(smp_processor_id()); + u64 sample_type = event->attr.sample_type; + struct x86_perf_regs *perf_regs; + u64 intr_mask = 0; + u64 mask = 0; + + if (WARN_ON_ONCE(!xsave) || !in_nmi()) + return; + + if ((sample_type & PERF_SAMPLE_REGS_INTR) && + (event->attr.sample_regs_intr & PERF_REG_EXTENDED_MASK)) + mask |= XFEATURE_MASK_SSE; + + mask &= x86_pmu.ext_regs_mask; + + if ((sample_type & PERF_SAMPLE_REGS_INTR) && data->regs_intr.regs) + intr_mask = mask & ~ignore_mask; + + if (intr_mask) { + perf_regs = container_of(data->regs_intr.regs, + struct x86_perf_regs, regs); + xsave->header.xfeatures = 0; + xsaves_nmi(xsave, mask); + update_perf_regs(perf_regs, xsave, intr_mask); + } +} + +void x86_pmu_update_perf_regs(struct perf_event *event, + struct perf_sample_data *data, + struct pt_regs *regs, + u64 ignore_mask) +{ + u64 sample_type = event->attr.sample_type; + + if (!((sample_type & PERF_SAMPLE_REGS_INTR) && + (event->attr.sample_regs_intr & PERF_REG_EXTENDED_MASK))) + return; + + if (sample_type & PERF_SAMPLE_REGS_INTR) { + x86_pmu_update_regs_intr(event, data, regs, + event->attr.exclude_kernel); + } + + /* + * ignore_mask indicates the PEBS sampled extended regs + * which are unnecessary to sample again. + */ + x86_pmu_sample_xregs(event, data, ignore_mask); +} + int x86_pmu_handle_irq(struct pt_regs *regs) { struct perf_sample_data data; diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c index 11a0c4dd2026..93ac6591cb3e 100644 --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -3927,6 +3927,9 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status) if (has_branch_stack(event)) intel_pmu_lbr_save_brstack(&data, cpuc, event); + x86_pmu_clear_perf_regs(regs); + x86_pmu_update_perf_regs(event, &data, regs, 0); + perf_event_overflow(event, &data, regs); } @@ -6162,9 +6165,38 @@ static inline void __intel_update_pmu_xregs_caps(struct pmu *pmu) struct pmu *dest_pmu = pmu ? pmu : x86_get_pmu(smp_processor_id()); u64 caps = hybrid(pmu, arch_pebs_cap).caps; + /* + * Extend the vector registers support to non-PEBS. + * The feature is limited to newer Intel machines with + * PEBS V4+ or archPerfmonExt (0x23) enabled for now. + * In theory, the vector registers can be retrieved as + * long as the CPU supports. The support for the old + * generations may be added later if there is a + * requirement. + * Only support the extension when XSAVES is available. + */ + if (!boot_cpu_has(X86_FEATURE_XSAVES)) + return; + + if (!boot_cpu_has(X86_FEATURE_XMM) || + !cpu_has_xfeatures(XFEATURE_MASK_SSE, NULL)) + return; + + /* + * On current hybrid platforms, P-cores and E-cores expose the same + * XSAVE feature set. Therefore, using the global x86_pmu.ext_regs_mask + * is sufficient to represent the hardware-supported XSAVE features. + */ + x86_pmu.ext_regs_mask |= XFEATURE_MASK_SSE; + + /* PEBS supported case */ if ((x86_pmu.arch_pebs && (caps & ARCH_PEBS_VECR_XMM)) || (x86_pmu.intel_cap.pebs_format >= 4 && x86_pmu.intel_cap.pebs_baseline)) dest_pmu->capabilities |= PERF_PMU_CAP_EXTENDED_REGS; + + /* PEBS unsupported case (e.g., guest) */ + if (!x86_pmu.intel_cap.pebs_format) + dest_pmu->capabilities |= PERF_PMU_CAP_EXTENDED_REGS; } static inline void __intel_update_large_pebs_flags(struct pmu *pmu) diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c index 78b9c0dcb14d..e2bb53f138ee 100644 --- a/arch/x86/events/intel/ds.c +++ b/arch/x86/events/intel/ds.c @@ -2508,6 +2508,7 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event, struct pebs_meminfo *meminfo = NULL; struct pebs_gprs *gprs = NULL; struct x86_perf_regs *perf_regs; + u64 ignore_mask = 0; u64 format_group; u16 retire; @@ -2515,7 +2516,7 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event, return; perf_regs = container_of(regs, struct x86_perf_regs, regs); - perf_regs->xmm_regs = NULL; + x86_pmu_clear_perf_regs(regs); format_group = basic->format_group; @@ -2562,6 +2563,7 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event, if (format_group & PEBS_DATACFG_XMMS) { struct pebs_xmm *xmm = next_record; + ignore_mask |= XFEATURE_MASK_SSE; next_record = xmm + 1; perf_regs->xmm_regs = xmm->xmm; } @@ -2600,6 +2602,8 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event, next_record += nr * sizeof(u64); } + x86_pmu_update_perf_regs(event, data, regs, ignore_mask); + WARN_ONCE(next_record != __pebs + basic->format_size, "PEBS record size %u, expected %llu, config %llx\n", basic->format_size, @@ -2625,6 +2629,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event, struct arch_pebs_aux *meminfo = NULL; struct arch_pebs_gprs *gprs = NULL; struct x86_perf_regs *perf_regs; + u64 ignore_mask = 0; void *next_record; void *at = __pebs; @@ -2632,7 +2637,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event, return; perf_regs = container_of(regs, struct x86_perf_regs, regs); - perf_regs->xmm_regs = NULL; + x86_pmu_clear_perf_regs(regs); __setup_perf_sample_data(event, iregs, data); @@ -2687,6 +2692,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event, next_record += sizeof(struct arch_pebs_xer_header); + ignore_mask |= XFEATURE_MASK_SSE; xmm = next_record; perf_regs->xmm_regs = xmm->xmm; next_record = xmm + 1; @@ -2734,6 +2740,8 @@ static void setup_arch_pebs_sample_data(struct perf_event *event, at = at + header->size; goto again; } + + x86_pmu_update_perf_regs(event, data, regs, ignore_mask); } static inline void * diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h index a8afea8d38f0..70fa7ec21673 100644 --- a/arch/x86/events/perf_event.h +++ b/arch/x86/events/perf_event.h @@ -1028,6 +1028,12 @@ struct x86_pmu { struct extra_reg *extra_regs; unsigned int flags; + /* + * Extended regs, e.g., vector registers + * Utilize the same format as the XFEATURE_MASK_* + */ + u64 ext_regs_mask; + /* * Intel host/guest support (KVM) */ @@ -1314,6 +1320,13 @@ void x86_pmu_enable_event(struct perf_event *event); int x86_pmu_handle_irq(struct pt_regs *regs); +void x86_pmu_clear_perf_regs(struct pt_regs *regs); + +void x86_pmu_update_perf_regs(struct perf_event *event, + struct perf_sample_data *data, + struct pt_regs *regs, + u64 ignore_mask); + void x86_pmu_show_pmu_cap(struct pmu *pmu); static inline int x86_pmu_num_counters(struct pmu *pmu) diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index 38fa8ff26559..19dec5f0b1c7 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -112,6 +112,8 @@ void xsaves(struct xregs_state *xsave, u64 mask); void xrstors(struct xregs_state *xsave, u64 mask); void xsaves_nmi(struct xregs_state *xsave, u64 mask); +unsigned int xstate_calculate_size(u64 xfeatures, bool compacted); + int xfd_enable_feature(u64 xfd_err); #ifdef CONFIG_X86_64 diff --git a/arch/x86/include/asm/perf_event.h b/arch/x86/include/asm/perf_event.h index 1eb13673e889..619e0ae915e1 100644 --- a/arch/x86/include/asm/perf_event.h +++ b/arch/x86/include/asm/perf_event.h @@ -728,7 +728,10 @@ extern void perf_events_lapic_init(void); struct pt_regs; struct x86_perf_regs { struct pt_regs regs; - u64 *xmm_regs; + union { + u64 *xmm_regs; + u32 *xmm_space; /* for xsaves */ + }; }; extern unsigned long perf_arch_instruction_pointer(struct pt_regs *regs); diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index d39c01546dee..3e7f5fb5bfaf 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -587,7 +587,7 @@ static bool __init check_xstate_against_struct(int nr) return true; } -static unsigned int xstate_calculate_size(u64 xfeatures, bool compacted) +unsigned int xstate_calculate_size(u64 xfeatures, bool compacted) { unsigned int topmost = fls64(xfeatures) - 1; unsigned int offset, i; -- 2.34.1