[PATCH v6 0/3] arm64: Add BRBE support for bpf_get_branch_snapshot()

[PATCH v6 0/3] arm64: Add BRBE support for bpf_get_branch_snapshot()

[Puranjay Mohan]

Changelog:
v5: https://lore.kernel.org/all/20260616155716.2631508-1-puranjay@kernel.org/
Changes in v6:
- Split the two perf/core patches out into their own series [3]. They
  fix bugs of their own and do not depend on this one.
- Add raw_local_daif_save() and raw_local_daif_restore(), so the
  snapshot can mask exceptions without trace_hardirqs_off() evicting
  the records it is about to read (new patch 1).
- Rework the snapshot. Clear PMCR_EL0.E across the BRBFCR_EL1
  read-modify-write so that a freeze cannot set PAUSED behind the saved
  value, and unpause before restarting the counters, since a paused
  BRBE cannot freeze for its own overflow handler (James Clark).
- Skip the pause entirely when BRBE is disabled or already frozen, and
  invalidate only when the snapshot was the one to pause it.
- Run valid_brbe_version() before masking exceptions; the static call
  is only reached from BPF programs, which run with migration disabled.
- Fold cpu_has_brbe() into valid_brbe_version() and drop the
  arm_brbe_ prefix from the snapshot callback (James Clark).
- Derive the selftest skip threshold from the waste and hit limits it
  depends on, so the two cannot drift apart.

v4: https://lore.kernel.org/all/20260527121207.2312181-1-puranjay@kernel.org/
Changes in v5:
- Rework patch 2: drop the UAPI union. Instead rename the helper to perf_clear_branch_entry() and
  clear the entry with a single
  	*br = (struct perf_branch_entry){ }.
  This stays kernel-internal, with no changes to the uapi/ or tools/ headers (James Clark).
- Add an isb() to the BRBCR_EL1 == 0 early-exit in the snapshot, for consistency with the other
  two exit paths (bpf-ci).
- Add Rob Herring's Reviewed-by to patch 3

v3: https://lore.kernel.org/all/20260413185740.3286146-1-puranjay@kernel.org/
Changes in v4:
- Fix leaking branch records when scheduled task has an unrelated perf event (Sashiko)
- Update tools/include/uapi/linux/perf_event.h as well for patch 2
- Introduce cpu_has_brbe() and use it in
  arm_brbe_snapshot_branch_stack(0 to make sure we don't run on a CPU
  without BRBE.
- Add explicit isb() after after writing to SYS_BRBFCR_EL1.
- Rebase on latest arm64 tree.

v2: https://lore.kernel.org/all/20260318171706.2840512-1-puranjay@kernel.org/
Changes in v3:
- Move NULL pmu_ctx fix from arm_pmuv3.c to perf core (Leo Yan)
- Use union to clear branch entry bitfields instead of per-field
  zeroing (Leo Yan)
- Remove per-CPU brbe_active flag; check BRBCR_EL1 == 0 instead (Rob
  Herring)
- Remove redundant valid_brbidr() check in snapshot path (Rob Herring)
- Introduce for_each_brbe_entry() iterator to deduplicate bank
  iteration (Rob Herring)
- Include perf core maintainers (Leo Yan, Rob Herring)

v1: https://lore.kernel.org/all/20260313180352.3800358-1-puranjay@kernel.org/
Changes in v2:
- Rebased on arm64/for-next/core
- Add per-CPU brbe_active flag to guard against UNDEFINED sysreg access
  on non-BRBE CPUs in heterogeneous big.LITTLE systems.
- Fix pre-existing bug in perf_clear_branch_entry_bitfields() that missed
  zeroing new_type and priv bitfields, added as a separate patch with
  Fixes tags (new patch 2).
- Use architecture-specific selftest threshold (#if defined(__aarch64__))
  instead of raising the global threshold, to preserve x86 regression
  detection.

RFC: https://lore.kernel.org/all/20260102214043.1410242-1-puranjay@kernel.org/
Changes from RFC:
 - Fix pre-existing NULL pointer dereference in armv8pmu_sched_task()
   found by Leo Yan during testing (patch 1)
 - Pause BRBE before local_daif_save() to avoid branch pollution from
   trace_hardirqs_off()
 - Use local_daif_save() to prevent pNMI race from counter overflow
   (Mark Rutland)
 - Reuse perf_entry_from_brbe_regset() instead of duplicating register
   read logic, by making it accept NULL event (Mark Rutland)
 - Invalidate BRBE after reading to maintain record contiguity for
   other consumers (Mark Rutland)
 - Adjust selftest wasted_entries threshold for ARM64 (patch 3)
 - Tested on ARM FVP with BRBE enabled

This series enables the bpf_get_branch_snapshot() BPF helper on ARM64
by implementing the perf_snapshot_branch_stack static call for ARM's
Branch Record Buffer Extension (BRBE).

bpf_get_branch_snapshot() [1] allows BPF programs to capture hardware
branch records on-demand from any BPF tracing context. This was
previously only available on x86 (Intel LBR) since v5.16. With BRBE
available on ARMv9, this series closes the gap for ARM64.

Usage model
-----------

The helper works in conjunction with perf events. The userspace
component of the BPF application opens a perf event with
PERF_SAMPLE_BRANCH_STACK on each CPU, which configures the hardware
to continuously record branches into BRBE (on ARM64) or LBR (on x86).
A BPF program attached to a tracepoint, kprobe, or fentry hook can
then call bpf_get_branch_snapshot() to snapshot the branch buffer at
any point. Without an active perf event, BRBE is not recording and
the buffer is empty.

On-demand branch snapshots from BPF are useful for diagnosing which
specific code path was taken inside a function. Stack traces only show
function boundaries, but branch records reveal the exact sequence of
jumps, calls, and returns within a function -- making it possible to
identify which specific error check triggered a failure, or which
callback implementation was invoked through a function pointer.

For example, retsnoop [2] is a BPF-based tool for non-intrusive
mass-tracing of kernel internals. Its LBR mode (--lbr) creates per-CPU
perf events with PERF_SAMPLE_BRANCH_STACK and then uses
bpf_get_branch_snapshot() in its fentry/fexit BPF programs to capture
branch records whenever a traced function returns an error.

Consider debugging a bpf() syscall that returns -EINVAL when creating
a BPF map with invalid parameters. Running retsnoop on an ARM64 FVP
with BRBE to trace the bpf() syscall and array_map_alloc_check():

  $ retsnoop -e '*sys_bpf' -a 'array_map_alloc_check' --lbr=any \
             -F -k vmlinux --debug full-lbr
  $ simfail bpf-bad-map-max-entries-array  # in another terminal

Output of retsnoop:

  --- fentry BPF program (entries #63-#17) ---

  [#63-#59] __htab_map_lookup_elem: hash table walk with memcmp        (hashtab.c)
  [#58] __htab_map_lookup_elem+0x98  -> dump_bpf_prog+0xc850           (hashtab.c:750)
  [#57-#55] ... dump_bpf_prog internal branches ...
  [#54] dump_bpf_prog+0xcab8        -> bpf_get_current_pid_tgid+0x0    (helpers.c:225)
  [#53] bpf_get_current_pid_tgid+0x1c -> dump_bpf_prog+0xcabc          (helpers.c:225)
  [#52-#51] ... dump_bpf_prog -> __htab_map_lookup_elem ...
  [#50-#47] __htab_map_lookup_elem: htab_map_hash (jhash2), select_bucket
  [#46-#42] lookup_nulls_elem_raw: hash chain walk with memcmp         (hashtab.c:717)
  [#41] __htab_map_lookup_elem+0x98  -> dump_bpf_prog+0xcaf8           (hashtab.c:750)
  [#40-#37] ... dump_bpf_prog -> bpf_ktime_get_ns ...
  [#36] bpf_ktime_get_ns+0x10       -> ktime_get_mono_fast_ns+0x0      (helpers.c:178)
  [#35-#32] ktime_get_mono_fast_ns: tk_clock_read -> arch_counter_get_cntpct
  [#31] ktime_get_mono_fast_ns+0x9c -> bpf_ktime_get_ns+0x14           (timekeeping.c:493)
  [#30] bpf_ktime_get_ns+0x18       -> dump_bpf_prog+0xcd50            (helpers.c:178)
  [#29-#25] ... dump_bpf_prog internal branches ...
  [#24] dump_bpf_prog+0x11b28       -> __bpf_prog_exit_recur+0x0       (trampoline.c:1190)
  [#23-#17] __bpf_prog_exit_recur: rcu_read_unlock, migrate_enable     (trampoline.c:1195)

  --- array_map_alloc_check (entries #16-#12) ---

  [#16] dump_bpf_prog+0x11b38       -> array_map_alloc_check+0x8       (arraymap.c:55)
  [#15] array_map_alloc_check+0x18  -> array_map_alloc_check+0xb8      (arraymap.c:56)
        . bpf_map_attr_numa_node       . bpf_map_attr_numa_node
  [#14] array_map_alloc_check+0xbc  -> array_map_alloc_check+0x20      (arraymap.c:59)
        . bpf_map_attr_numa_node
  [#13] array_map_alloc_check+0x24  -> array_map_alloc_check+0x94      (arraymap.c:64)
  [#12] array_map_alloc_check+0x98  -> dump_bpf_prog+0x11b3c           (arraymap.c:82)

  --- fexit trampoline overhead (entries #11-#00) ---

  [#11] dump_bpf_prog+0x11b5c       -> __bpf_prog_enter_recur+0x0      (trampoline.c:1145)
  [#10-#03] __bpf_prog_enter_recur: rcu_read_lock, migrate_disable     (trampoline.c:1146)
  [#02] __bpf_prog_enter_recur+0x114 -> dump_bpf_prog+0x11b60          (trampoline.c:1157)
  [#01] dump_bpf_prog+0x11b6c       -> dump_bpf_prog+0xd230
  [#00] dump_bpf_prog+0xd340        -> brbe_snapshot_branch_stack+0x0     (arm_brbe.c:814)

                   el0t_64_sync+0x168
                   el0t_64_sync_handler+0x98
                   el0_svc+0x28
                   do_el0_svc+0x4c
                   invoke_syscall.constprop.0+0x54
    373us [-EINVAL] __arm64_sys_bpf+0x8
                    __sys_bpf+0x87c
                    map_create+0x120
     95us [-EINVAL] array_map_alloc_check+0x8

The FVP's BRBE buffer has 64 entries (BRBE supports 8, 16, 32, or
64). Of these, entries #63-#17 (47) are consumed by the fentry BPF
trampoline that ran before the function, and entries #11-#00 (12)
are consumed by the fexit trampoline that runs after. Entry #00
shows the very last branch recorded before BRBE is paused: the call
into brbe_snapshot_branch_stack().

The 5 useful entries (#16-#12) show the exact path taken inside
array_map_alloc_check(). Record #14 shows a jump from line 56
(bpf_map_attr_numa_node) to line 59 (the if-condition), and #13
shows an immediate jump from line 59 (attr->max_entries == 0) to
line 64 (return -EINVAL), skipping lines 60-63. This pinpoints
max_entries==0 as the cause -- a diagnosis impossible with stack
traces alone.

[1] 856c02dbce4f ("bpf: Introduce helper bpf_get_branch_snapshot")
[2] https://github.com/anakryiko/retsnoop
[3] https://lore.kernel.org/all/20260810133540.1947118-1-puranjay@kernel.org/

Puranjay Mohan (3):
  arm64: Add raw variants of local_daif_save() and local_daif_restore()
  perf/arm64: Add BRBE support for bpf_get_branch_snapshot()
  selftests/bpf: Adjust entry thresholds for ARM64 BRBE

 arch/arm64/include/asm/daifflags.h            |  41 +++++-
 arch/arm64/kernel/suspend.c                   |   7 +-
 drivers/perf/arm_brbe.c                       | 135 ++++++++++++++----
 drivers/perf/arm_brbe.h                       |   9 ++
 drivers/perf/arm_pmuv3.c                      |   5 +-
 .../bpf/prog_tests/get_branch_snapshot.c      |  28 ++--
 6 files changed, 175 insertions(+), 50 deletions(-)


base-commit: 03a2e393994ca27567fe060e066806589c547fbc
-- 
2.53.0-Meta

[PATCH v6 1/3] arm64: Add raw variants of local_daif_save() and local_daif_restore()

[Puranjay Mohan]

local_daif_save() unconditionally calls trace_hardirqs_off(), and
local_daif_restore() the matching trace_hardirqs_on(). A caller already
in a hardirqs-off context, or one that must not run tracing code between
masking exceptions and its next step, has no way to opt out.

cpu_suspend() is one such caller and already carries a comment asking for
this. Sampling the BRBE branch record buffer is another: the tracing
calls generate branches that evict the records about to be read.

Split the tracing out into raw_local_daif_save() and
raw_local_daif_restore(), and convert cpu_suspend() to the raw save with
a lockdep_assert_irqs_disabled() for the precondition it now relies on.
Its restore stays traced, to re-arm the irqsoff tracer for the resume
path. The PMR unmasking cpu_suspend() needs is in raw_local_daif_mask(),
so it is unaffected.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 arch/arm64/include/asm/daifflags.h | 41 +++++++++++++++++++++++++-----
 arch/arm64/kernel/suspend.c        |  7 ++---
 2 files changed, 36 insertions(+), 12 deletions(-)

diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
index 795b351284673..c3adba0985bda 100644
--- a/arch/arm64/include/asm/daifflags.h
+++ b/arch/arm64/include/asm/daifflags.h
@@ -18,8 +18,7 @@
 #define DAIF_MASK		(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
 
 
-/* mask/save/unmask/restore all exceptions, including interrupts. */
-static __always_inline void local_daif_mask(void)
+static __always_inline void raw_local_daif_mask(void)
 {
 	WARN_ON(system_has_prio_mask_debugging() &&
 		(read_sysreg_s(SYS_ICC_PMR_EL1) == (GIC_PRIO_IRQOFF |
@@ -34,6 +33,12 @@ static __always_inline void local_daif_mask(void)
 	/* Don't really care for a dsb here, we don't intend to enable IRQs */
 	if (system_uses_irq_prio_masking())
 		gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
+}
+
+/* mask/save/unmask/restore all exceptions, including interrupts. */
+static __always_inline void local_daif_mask(void)
+{
+	raw_local_daif_mask();
 
 	trace_hardirqs_off();
 }
@@ -53,18 +58,29 @@ static __always_inline unsigned long local_daif_save_flags(void)
 	return flags;
 }
 
-static __always_inline unsigned long local_daif_save(void)
+static __always_inline unsigned long raw_local_daif_save(void)
 {
 	unsigned long flags;
 
 	flags = local_daif_save_flags();
 
-	local_daif_mask();
+	raw_local_daif_mask();
 
 	return flags;
 }
 
-static __always_inline void local_daif_restore(unsigned long flags)
+static __always_inline unsigned long local_daif_save(void)
+{
+	unsigned long flags;
+
+	flags = raw_local_daif_save();
+
+	trace_hardirqs_off();
+
+	return flags;
+}
+
+static __always_inline void __local_daif_restore(unsigned long flags, bool trace)
 {
 	bool irq_disabled = flags & PSR_I_BIT;
 
@@ -72,7 +88,8 @@ static __always_inline void local_daif_restore(unsigned long flags)
 		(read_sysreg(daif) & (PSR_I_BIT | PSR_F_BIT)) != (PSR_I_BIT | PSR_F_BIT));
 
 	if (!irq_disabled) {
-		trace_hardirqs_on();
+		if (trace)
+			trace_hardirqs_on();
 
 		if (system_uses_irq_prio_masking()) {
 			gic_write_pmr(GIC_PRIO_IRQON);
@@ -116,10 +133,20 @@ static __always_inline void local_daif_restore(unsigned long flags)
 
 	write_sysreg(flags, daif);
 
-	if (irq_disabled)
+	if (irq_disabled && trace)
 		trace_hardirqs_off();
 }
 
+static __always_inline void local_daif_restore(unsigned long flags)
+{
+	__local_daif_restore(flags, true);
+}
+
+static __always_inline void raw_local_daif_restore(unsigned long flags)
+{
+	__local_daif_restore(flags, false);
+}
+
 /*
  * Called by synchronous exception handlers to restore the DAIF bits that were
  * modified by taking an exception.
diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c
index c41724a40b756..461948ad15ec8 100644
--- a/arch/arm64/kernel/suspend.c
+++ b/arch/arm64/kernel/suspend.c
@@ -117,14 +117,11 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 	 * updates to mdscr register (saved and restored along with
 	 * general purpose registers) from kernel debuggers.
 	 *
-	 * Strictly speaking the trace_hardirqs_off() here is superfluous,
-	 * hardirqs should be firmly off by now. This really ought to use
-	 * something like raw_local_daif_save().
-	 *
 	 * This also unmasks interrupts in PMR in order to reliably
 	 * resume if we're using pseudo-NMIs.
 	 */
-	flags = local_daif_save();
+	lockdep_assert_irqs_disabled();
+	flags = raw_local_daif_save();
 
 	/*
 	 * Function graph tracer state gets inconsistent when the kernel
-- 
2.53.0-Meta

[PATCH v6 2/3] perf/arm64: Add BRBE support for bpf_get_branch_snapshot()

[Puranjay Mohan]

Implement the perf_snapshot_branch_stack static call for BRBE.

brbe_enable() and brbe_disable() run from the overflow interrupt and the
add/remove IPIs, so the snapshot masks exceptions before sampling BRBE
state. It pauses BRBE next, with trace_hardirqs_off() and the read after
that so neither evicts a record. valid_brbe_version() runs before the
mask; the static call is only reached from BPF programs, which run with
migration disabled.

BRBFCR_EL1 is read before the pause and written back after the read, and
a freeze in between would set PAUSED behind the saved value. PMCR_EL0.E
is cleared across that window: the overflow driven conditions of Arm ARM
(DDI 0487 M.a) D19.3 need a bit in PMOVSCLR_EL0, which a stopped counter
cannot set, and RLBQZR needs BRBCR_EL1.FZPSS, which brbe_enable() leaves
clear. The counters restart only after the unpause, since a paused BRBE
cannot freeze for its own overflow handler.

Branches are missed while paused, so the buffer is invalidated before
recording resumes rather than left with a hole, and a concurrent branch
stack event loses what it had collected. BRBE that arrived frozen is left
alone; it stopped before the snapshot ran, so its records are intact and
belong to whoever froze it.

Records are read with a NULL event, which skips the per-event software
filter. The hardware filters still apply, so the caller sees the union of
the privilege and branch type filters of all active events. That matches
x86 and is acceptable because bpf_get_branch_snapshot() is only reachable
from program types already requiring CAP_BPF and CAP_PERFMON.

bpf_get_branch_snapshot() returns -ENOENT when called from a BPF program
attached to the CPU PMU's own overflow handler, since armv8pmu_stop()
zeroes BRBCR_EL1 before the handler runs.

Introduce for_each_brbe_entry() to share bank iteration between
brbe_read_filtered_entries() and the snapshot.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 drivers/perf/arm_brbe.c  | 135 +++++++++++++++++++++++++++++++--------
 drivers/perf/arm_brbe.h  |   9 +++
 drivers/perf/arm_pmuv3.c |   5 +-
 3 files changed, 121 insertions(+), 28 deletions(-)

diff --git a/drivers/perf/arm_brbe.c b/drivers/perf/arm_brbe.c
index ba554e0c846c4..9e7e69c43b17d 100644
--- a/drivers/perf/arm_brbe.c
+++ b/drivers/perf/arm_brbe.c
@@ -9,6 +9,8 @@
 #include <linux/types.h>
 #include <linux/bitmap.h>
 #include <linux/perf/arm_pmu.h>
+#include <linux/perf/arm_pmuv3.h>
+#include <asm/daifflags.h>
 #include "arm_brbe.h"
 
 #define BRBFCR_EL1_BRANCH_FILTERS (BRBFCR_EL1_DIRECT   | \
@@ -250,10 +252,13 @@ static bool valid_brbidr(u64 brbidr)
 	return valid_brbe_format(brbe_format) && valid_brbe_cc(brbe_cc) && valid_brbe_nr(brbe_nr);
 }
 
-static bool valid_brbe_version(int brbe_version)
+static __always_inline bool valid_brbe_version(void)
 {
-	return brbe_version == ID_AA64DFR0_EL1_BRBE_IMP ||
-	       brbe_version == ID_AA64DFR0_EL1_BRBE_BRBE_V1P1;
+	u64 aa64dfr0 = read_sysreg_s(SYS_ID_AA64DFR0_EL1);
+	int brbe = cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_EL1_BRBE_SHIFT);
+
+	return brbe == ID_AA64DFR0_EL1_BRBE_IMP ||
+	       brbe == ID_AA64DFR0_EL1_BRBE_BRBE_V1P1;
 }
 
 static void select_brbe_bank(int bank)
@@ -271,6 +276,20 @@ static void select_brbe_bank(int bank)
 	isb();
 }
 
+static inline void __brbe_advance(int *bank, int *idx, int nr_hw)
+{
+	if (++(*idx) >= BRBE_BANK_MAX_ENTRIES &&
+	    *bank * BRBE_BANK_MAX_ENTRIES + *idx < nr_hw) {
+		*idx = 0;
+		select_brbe_bank(++(*bank));
+	}
+}
+
+#define for_each_brbe_entry(idx, nr_hw)					\
+	for (int __bank = (select_brbe_bank(0), 0), idx = 0;		\
+	     __bank * BRBE_BANK_MAX_ENTRIES + idx < (nr_hw);		\
+	     __brbe_advance(&__bank, &idx, (nr_hw)))
+
 static bool __read_brbe_regset(struct brbe_regset *entry, int idx)
 {
 	entry->brbinf = get_brbinf_reg(idx);
@@ -474,11 +493,9 @@ unsigned int brbe_num_branch_records(const struct arm_pmu *armpmu)
 
 void brbe_probe(struct arm_pmu *armpmu)
 {
-	u64 brbidr, aa64dfr0 = read_sysreg_s(SYS_ID_AA64DFR0_EL1);
-	u32 brbe;
+	u64 brbidr;
 
-	brbe = cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_EL1_BRBE_SHIFT);
-	if (!valid_brbe_version(brbe))
+	if (!valid_brbe_version())
 		return;
 
 	brbidr = read_sysreg_s(SYS_BRBIDR0_EL1);
@@ -575,7 +592,7 @@ static void brbe_set_perf_entry_type(struct perf_branch_entry *entry, u64 brbinf
 	}
 }
 
-static int brbinf_get_perf_priv(u64 brbinf)
+static int brbinf_get_perf_priv(u64 brbinf, const struct perf_event *event)
 {
 	int brbe_el = brbinf_get_el(brbinf);
 
@@ -589,11 +606,13 @@ static int brbinf_get_perf_priv(u64 brbinf)
 			return PERF_BR_PRIV_KERNEL;
 		return PERF_BR_PRIV_HV;
 	default:
-		pr_warn_once("%d - unknown branch privilege captured\n", brbe_el);
+		if (event)
+			pr_warn_once("%d - unknown branch privilege captured\n", brbe_el);
 		return PERF_BR_PRIV_UNKNOWN;
 	}
 }
 
+/* @event is NULL for the snapshot: no software filter, and no printing. */
 static bool perf_entry_from_brbe_regset(int index, struct perf_branch_entry *entry,
 					const struct perf_event *event)
 {
@@ -618,10 +637,10 @@ static bool perf_entry_from_brbe_regset(int index, struct perf_branch_entry *ent
 
 	brbe_set_perf_entry_type(entry, brbinf);
 
-	if (!branch_sample_no_cycles(event))
+	if (!event || !branch_sample_no_cycles(event))
 		entry->cycles = brbinf_get_cycles(brbinf);
 
-	if (!branch_sample_no_flags(event)) {
+	if (!event || !branch_sample_no_flags(event)) {
 		/* Mispredict info is available for source only and complete branch records. */
 		if (!brbe_record_is_target_only(brbinf)) {
 			entry->mispred = brbinf_get_mispredict(brbinf);
@@ -633,7 +652,7 @@ static bool perf_entry_from_brbe_regset(int index, struct perf_branch_entry *ent
 		 * nor it is being supported in the kernel. Just warn here once
 		 * if TME related information shows up rather unexpectedly.
 		 */
-		if (brbinf_get_lastfailed(brbinf) || brbinf_get_in_tx(brbinf))
+		if (event && (brbinf_get_lastfailed(brbinf) || brbinf_get_in_tx(brbinf)))
 			pr_warn_once("Unknown transaction states\n");
 	}
 
@@ -642,7 +661,7 @@ static bool perf_entry_from_brbe_regset(int index, struct perf_branch_entry *ent
 	 * branch records.
 	 */
 	if (!brbe_record_is_source_only(brbinf))
-		entry->priv = brbinf_get_perf_priv(brbinf);
+		entry->priv = brbinf_get_perf_priv(brbinf, event);
 
 	return true;
 }
@@ -774,32 +793,94 @@ void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack,
 {
 	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
 	int nr_hw = brbe_num_branch_records(cpu_pmu);
-	int nr_banks = DIV_ROUND_UP(nr_hw, BRBE_BANK_MAX_ENTRIES);
 	int nr_filtered = 0;
 	u64 branch_sample_type = event->attr.branch_sample_type;
 	DECLARE_BITMAP(event_type_mask, PERF_BR_ARM64_MAX);
 
 	prepare_event_branch_type_mask(branch_sample_type, event_type_mask);
 
-	for (int bank = 0; bank < nr_banks; bank++) {
-		int nr_remaining = nr_hw - (bank * BRBE_BANK_MAX_ENTRIES);
-		int nr_this_bank = min(nr_remaining, BRBE_BANK_MAX_ENTRIES);
+	for_each_brbe_entry(i, nr_hw) {
+		struct perf_branch_entry *pbe = &branch_stack->entries[nr_filtered];
 
-		select_brbe_bank(bank);
+		if (!perf_entry_from_brbe_regset(i, pbe, event))
+			break;
 
-		for (int i = 0; i < nr_this_bank; i++) {
-			struct perf_branch_entry *pbe = &branch_stack->entries[nr_filtered];
+		if (!filter_branch_record(pbe, branch_sample_type, event_type_mask))
+			continue;
 
-			if (!perf_entry_from_brbe_regset(i, pbe, event))
-				goto done;
+		nr_filtered++;
+	}
 
-			if (!filter_branch_record(pbe, branch_sample_type, event_type_mask))
-				continue;
+	branch_stack->nr = nr_filtered;
+}
 
-			nr_filtered++;
+int brbe_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
+{
+	u64 brbidr, brbfcr, brbcr, pmcr;
+	int nr_hw, nr_copied = 0;
+	bool paused_here = false;
+	unsigned long flags;
+
+	/* Called with migration disabled, so this is the CPU read below. */
+	if (!valid_brbe_version())
+		return 0;
+
+	flags = raw_local_daif_save();
+
+	brbcr = read_sysreg_s(SYS_BRBCR_EL1);
+	brbfcr = read_sysreg_s(SYS_BRBFCR_EL1);
+
+	/*
+	 * PAUSED is only ever set behind our back, so finding it set is final.
+	 * likely() keeps this inline; a taken branch costs a record.
+	 */
+	if (likely(brbcr && !(brbfcr & BRBFCR_EL1_PAUSED))) {
+		/*
+		 * Stop the counters so a freeze cannot set PAUSED behind the
+		 * value read below; the D19.3 conditions need a PMOVSCLR_EL0
+		 * bit, which a stopped counter cannot set.
+		 */
+		pmcr = read_pmcr();
+		write_pmcr(pmcr & ~ARMV8_PMU_PMCR_E);
+		isb();
+
+		brbfcr = read_sysreg_s(SYS_BRBFCR_EL1);
+		paused_here = !(brbfcr & BRBFCR_EL1_PAUSED);
+
+		write_sysreg_s(brbfcr | BRBFCR_EL1_PAUSED, SYS_BRBFCR_EL1);
+		isb();
+	}
+
+	trace_hardirqs_off();
+
+	/* Records outlive brbe_disable(). */
+	if (brbcr) {
+		brbidr = read_sysreg_s(SYS_BRBIDR0_EL1);
+		nr_hw = min_t(int, FIELD_GET(BRBIDR0_EL1_NUMREC_MASK, brbidr),
+			      BRBIDR0_EL1_NUMREC_64);
+
+		for_each_brbe_entry(i, nr_hw) {
+			if (nr_copied >= cnt)
+				break;
+
+			if (!perf_entry_from_brbe_regset(i, &entries[nr_copied], NULL))
+				break;
+
+			nr_copied++;
 		}
 	}
 
-done:
-	branch_stack->nr = nr_filtered;
+	if (paused_here) {
+		/* Branches were missed, so discard rather than leave a hole. */
+		brbe_invalidate();
+
+		/* Unpause first; a paused BRBE cannot freeze on overflow. */
+		write_sysreg_s(brbfcr, SYS_BRBFCR_EL1);
+		isb();
+		write_pmcr(pmcr);
+	}
+
+	local_daif_restore(flags);
+
+	return nr_copied;
 }
diff --git a/drivers/perf/arm_brbe.h b/drivers/perf/arm_brbe.h
index b7c7d8796c86f..0cc25737e5d49 100644
--- a/drivers/perf/arm_brbe.h
+++ b/drivers/perf/arm_brbe.h
@@ -10,6 +10,7 @@
 struct arm_pmu;
 struct perf_branch_stack;
 struct perf_event;
+struct perf_branch_entry;
 
 #ifdef CONFIG_ARM64_BRBE
 void brbe_probe(struct arm_pmu *arm_pmu);
@@ -22,6 +23,8 @@ void brbe_disable(void);
 bool brbe_branch_attr_valid(struct perf_event *event);
 void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack,
 				const struct perf_event *event);
+int brbe_snapshot_branch_stack(struct perf_branch_entry *entries,
+			       unsigned int cnt);
 #else
 static inline void brbe_probe(struct arm_pmu *arm_pmu) { }
 static inline unsigned int brbe_num_branch_records(const struct arm_pmu *armpmu)
@@ -44,4 +47,10 @@ static void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack,
 				       const struct perf_event *event)
 {
 }
+
+static inline int brbe_snapshot_branch_stack(struct perf_branch_entry *entries,
+					     unsigned int cnt)
+{
+	return 0;
+}
 #endif
diff --git a/drivers/perf/arm_pmuv3.c b/drivers/perf/arm_pmuv3.c
index 03359e078301f..c1afc1c5b4bc7 100644
--- a/drivers/perf/arm_pmuv3.c
+++ b/drivers/perf/arm_pmuv3.c
@@ -1495,8 +1495,11 @@ static int armv8_pmu_init(struct arm_pmu *cpu_pmu, char *name,
 	cpu_pmu->set_event_filter	= armv8pmu_set_event_filter;
 
 	cpu_pmu->pmu.event_idx		= armv8pmu_user_event_idx;
-	if (brbe_num_branch_records(cpu_pmu))
+	if (brbe_num_branch_records(cpu_pmu)) {
 		cpu_pmu->pmu.sched_task		= armv8pmu_sched_task;
+		static_call_update(perf_snapshot_branch_stack,
+				   brbe_snapshot_branch_stack);
+	}
 
 	cpu_pmu->name			= name;
 	cpu_pmu->map_event		= map_event;
-- 
2.53.0-Meta

[PATCH v6 3/3] selftests/bpf: Adjust entry thresholds for ARM64 BRBE

[Puranjay Mohan]

The test bounds how many branch entries are spent reaching the snapshot
helper. The limit of 10 was calibrated on x86, which spends about 7.

arm64 spends up to 14: eight in the BPF trampoline, which makes
out-of-line calls and branches more than x86 does, and the rest in
masking exceptions before the branch buffer can be paused.

Raise the limit to 18 on arm64, leaving room for compiler and
configuration variation.

The skip threshold has to move with it. BRBE implements 8, 16, 32 or 64
records, so on a 16 record buffer the existing 'total_entries < 16'
check does not skip, but 14 wasted entries leave only 2 for the hit test
and it fails instead. Derive the minimum from the two limits it depends
on, so the arms cannot drift apart.

This leaves 8 and 16 record implementations skipped, since the required
hit count is absolute and does not scale with the buffer.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 .../bpf/prog_tests/get_branch_snapshot.c      | 28 ++++++++++++-------
 1 file changed, 18 insertions(+), 10 deletions(-)

diff --git a/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c b/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c
index 0394a1156d99d..e06ae6798dbec 100644
--- a/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c
+++ b/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c
@@ -108,21 +108,29 @@ void serial_test_get_branch_snapshot(void)
 
 	trigger_module_test_read(100);
 
-	if (skel->bss->total_entries < 16) {
+	/* Reaching the snapshot helper costs a few entries: about 7 on x86_64
+	 * and about 14 on arm64, whose BPF trampoline branches more. Bound it
+	 * so that a regression shows up here.
+	 *
+	 * Derive the buffer minimum from the two limits it depends on, so the
+	 * arms cannot drift apart.
+	 */
+#if defined(__aarch64__)
+#define WASTED_ENTRIES_MAX	18
+#else
+#define WASTED_ENTRIES_MAX	10
+#endif
+#define TEST1_HITS_MIN		7
+#define TOTAL_ENTRIES_MIN	(WASTED_ENTRIES_MAX - 1 + TEST1_HITS_MIN)
+
+	if (skel->bss->total_entries < TOTAL_ENTRIES_MIN) {
 		/* too few entries for the hit/waste test */
 		test__skip();
 		goto cleanup;
 	}
 
-	ASSERT_GT(skel->bss->test1_hits, 6, "find_looptest_in_lbr");
-
-	/* Given we stop LBR in software, we will waste a few entries.
-	 * But we should try to waste as few as possible entries. We are at
-	 * about 7 on x86_64 systems.
-	 * Add a check for < 10 so that we get heads-up when something
-	 * changes and wastes too many entries.
-	 */
-	ASSERT_LT(skel->bss->wasted_entries, 10, "check_wasted_entries");
+	ASSERT_GT(skel->bss->test1_hits, TEST1_HITS_MIN - 1, "find_looptest_in_lbr");
+	ASSERT_LT(skel->bss->wasted_entries, WASTED_ENTRIES_MAX, "check_wasted_entries");
 
 cleanup:
 	get_branch_snapshot__destroy(skel);
-- 
2.53.0-Meta

Re: [PATCH v6 2/3] perf/arm64: Add BRBE support for bpf_get_branch_snapshot()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Missing `isb()` context synchronization barrier after writing to the `PMCR_EL0` system register in `brbe_snapshot_branch_stack`.
--

--- Patch [2]: [PATCH v6 2/3] perf/arm64: Add BRBE support for bpf_get_branch_snapshot() ---
commit a7708fa08fe97675c17bb6e7a2098acd91d49bfb
Author: Puranjay Mohan <puranjay@kernel.org>

perf/arm64: Add BRBE support for bpf_get_branch_snapshot()

This commit implements the perf_snapshot_branch_stack static call for BRBE,
allowing bpf_get_branch_snapshot() to capture hardware branch records. It
manages BRBE state by pausing it during the snapshot, freezing the counters,
and handling buffer invalidation to avoid data corruption.

> diff --git a/drivers/perf/arm_brbe.c b/drivers/perf/arm_brbe.c
> index ba554e0c846c4..9e7e69c43b17d 100644
> --- a/drivers/perf/arm_brbe.c
> +++ b/drivers/perf/arm_brbe.c

[ ... ]

> +int brbe_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
> +{

[ ... ]

> +	if (paused_here) {
> +		/* Branches were missed, so discard rather than leave a hole. */
> +		brbe_invalidate();
> +
> +		/* Unpause first; a paused BRBE cannot freeze on overflow. */
> +		write_sysreg_s(brbfcr, SYS_BRBFCR_EL1);
> +		isb();
> +		write_pmcr(pmcr);

[Severity: Medium]
Is a context synchronization barrier missing after this write to PMCR_EL0?

Architecture invariants for ARM64 usually require writes to control-plane
system registers to be immediately followed by an isb() to ensure the CPU
pipeline observes the new configuration.

Could the lack of an isb() here allow the CPU to continue operating under a
stale PMU configuration for several cycles, potentially missing performance
counter and BRBE branch events immediately following the unpause?

> +	}
> +
> +	local_daif_restore(flags);
> +
> +	return nr_copied;
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260810164622.1829843-1-puranjay@kernel.org?part=2