* [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support
@ 2026-10-17 5:45 Ian Rogers
2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
` (5 more replies)
0 siblings, 6 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Namhyung Kim
Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, rcu
This series adds a new BPF sample filter predicate, `lock_held(<lock>)`
(along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`,
`lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W`
mode suffixes), to `perf record --filter`, allowing perf event samples to
be recorded only when a specified kernel lock is held (or not held with
`!`) by the running kernel thread or CPU. It also extends both
`lock_held()` and `perf lock contention --use-bpf` to support RCU and
SRCU.
For example:
# perf record -a -g -e cycles --filter 'lock_held(&cgroup_mutex)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' -- ...
# perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ...
# perf lock contention -a -b -Y rcu,srcu -- sleep 1
Supported `<lock>` specifications match `perf lock contention`:
- Global/static kernel lock symbols (with or without leading `&`, e.g.
`lock_held(&cgroup_mutex)` or `lock_held(cgroup_mutex)`)
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`,
`lock_held_read(&dentry)`)
- Well-known per-process, semi-global, and RCU locks: `mmap_lock`,
`siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`
- Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`,
`rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`,
`percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`,
`rcu:W`, `semaphore`
Limitations and weaknesses in determining whether a lock is held:
- Inlined lock/unlock operations:
- When the kernel is configured to inline spinlock or rwlock operations
(`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`,
`CONFIG_INLINE_WRITE_LOCK`, etc., which are enabled on many
`!CONFIG_PREEMPT_RT` / `!CONFIG_PROVE_LOCKING` builds unless
`CONFIG_UNINLINE_SPIN_UNLOCK` or lockdep/preemption forces them out of
line), `_raw_spin_lock*` / `_raw_spin_unlock*` and
`_raw_read_lock*` / `_raw_write_lock*` are inlined into callers and
have no BTF function entry points for `fentry`/`fexit` to attach to.
- Fast-path `percpu_down_read()` / `percpu_up_read()` and
`srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined
per-CPU counter increments without function calls, so only
`percpu_down_write()` / `percpu_up_write()` and out-of-line
`__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked.
- On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to
`preempt_disable()` and does not maintain
`current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)`
read-side inspection requires `CONFIG_PREEMPT_RCU`.
- Lack of an inspectable owner field on non-exclusive / spinning locks:
- While `struct mutex`, `struct rt_mutex`, and write-locked
`struct rw_semaphore` store the owning `task_struct *` in `lock->owner`
(allowing `perf_sample_filter` to inspect kernel memory directly on
sample overflow when filtering by lock address), spinlocks, rwlocks,
semaphores, SRCU, and reader-locked `rw_semaphore`s (where
`__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`)
do not maintain an authoritative per-task owner pointer.
- Tracking these locks (as well as slab-cache and lock-type filters)
relies on `fexit` (acquire) and `fentry` (release) BPF hooks
maintaining the `held_locks` BPF hash map:
- Any such lock already held before `perf record` attaches its BPF
programs will not be present in `held_locks` until it is released
and re-acquired.
- The attached `fentry`/`fexit` hooks execute on every kernel lock
acquire and release for the hooked lock family across the system,
adding tracing overhead outside of perf sample handling.
Patch overview:
1. "perf lock contention: Share common BPF and lock type definitions in
lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and
`LCB_F_*` flags, CO-RE structs, and common BPF lock helpers
(`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`,
`get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`,
`do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`.
2. "perf bpf-filter: Add lock_held() sample filter": Implements
`lock_held()`, `lock_held_read()`/`lock_held_shared()`,
`lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode
suffixes using direct lock owner inspection where possible (`mutex`,
`rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF map
updated by selectively attached `fentry`/`fexit` hooks.
3. "perf test: Add tests for lock_held() BPF sample filter": Adds unit
tests in `tests/parse-events.c` and shell tests in
`tests/shell/record_bpf_filter.sh`.
4. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU": Supports `rcu` (`rcu:R` via direct
`current->rcu_read_lock_nesting` inspection with zero tracing hooks;
`rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks),
`rcu-trace` (via `current->trc_reader_nesting`), and `srcu`
(`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W`
via `synchronize_srcu*()` / `srcu_barrier()`).
5. "perf lock contention: Extend BPF lock contention to RCU and SRCU":
Extends `perf lock contention --use-bpf` to track RCU and SRCU
grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`).
Assisted-by: Antigravity:gemini-3.1-pro
Ian Rogers (5):
perf lock contention: Share common BPF and lock type definitions in
lock_data.h
perf bpf-filter: Add lock_held() sample filter
perf test: Add tests for lock_held() BPF sample filter
perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU
perf lock contention: Extend BPF lock contention to RCU and SRCU
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/Documentation/perf-record.txt | 51 +-
tools/perf/builtin-lock.c | 27 -
tools/perf/tests/parse-events.c | 121 ++
tools/perf/tests/shell/lock_contention.sh | 14 +
tools/perf/tests/shell/record_bpf_filter.sh | 119 ++
tools/perf/util/bpf-filter.c | 1070 ++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 46 +-
tools/perf/util/bpf_lock_contention.c | 167 ++-
.../perf/util/bpf_skel/lock_contention.bpf.c | 410 +++----
tools/perf/util/bpf_skel/lock_data.h | 360 +++++-
tools/perf/util/bpf_skel/sample-filter.h | 71 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 1051 ++++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 +
tools/perf/util/lock-contention.c | 22 +
tools/perf/util/lock-contention.h | 15 +
tools/perf/util/parse-events.c | 3 +-
19 files changed, 3273 insertions(+), 332 deletions(-)
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply [flat|nested] 106+ messages in thread
* [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
@ 2026-10-17 5:45 ` Ian Rogers
2026-10-17 5:52 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers
` (4 subsequent siblings)
5 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Namhyung Kim
Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, rcu
Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers
(get_lock_owner, get_current_mmap_lock, get_current_siglock,
get_lock_addr, check_lock_type, do_collect_zone_lock,
do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into
tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF
sample filtering.
Also add rw_semaphore.count and struct rt_mutex_base to the minimal
vmlinux.h and extend get_lock_owner() to support rt_mutex and check
writer-ownership and handoff flags.
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/builtin-lock.c | 27 --
.../perf/util/bpf_skel/lock_contention.bpf.c | 265 +--------------
tools/perf/util/bpf_skel/lock_data.h | 316 ++++++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 +
tools/perf/util/lock-contention.c | 18 +
tools/perf/util/lock-contention.h | 13 +
6 files changed, 370 insertions(+), 274 deletions(-)
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index d5c0d55cb82d..feaf1cfd08cf 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -1477,33 +1477,6 @@ static void sort_result(void)
}
}
-static const struct {
- unsigned int flags;
- /*
- * Name of the lock flags (access), with delimeter ':'.
- * For example, rwsem:R of rwsem:W.
- */
- const char *flags_name;
- /* Name of the lock (type), for example, rwlock or rwsem. */
- const char *lock_name;
-} lock_type_table[] = {
- { 0, "semaphore", "semaphore" },
- { LCB_F_SPIN, "spinlock", "spinlock" },
- { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
- { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
- { LCB_F_READ, "rwsem:R", "rwsem" },
- { LCB_F_WRITE, "rwsem:W", "rwsem" },
- { LCB_F_RT, "rt-mutex", "rt-mutex" },
- { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
- { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
- { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
- { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
- { LCB_F_MUTEX, "mutex", "mutex" },
- { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
- /* alias for optimistic spinning only */
- { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
-};
-
static const char *get_type_flags_name(unsigned int flags)
{
flags &= LCB_F_TYPE_MASK;
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index 0d9c6f55050e..0f2118006005 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -8,23 +8,9 @@
#include "lock_data.h"
-/* for collect_lock_syms(). 4096 was rejected by the verifier */
-#define MAX_CPUS 1024
-
-/* for collect_zone_lock(). It should be more than the actual zones. */
-#define MAX_ZONES 10
-
/* for do_lock_delay(). Arbitrarily set to 1 million. */
#define MAX_LOOP (1U << 20)
-/* lock contention flags from include/trace/events/lock.h */
-#define LCB_F_SPIN (1U << 0)
-#define LCB_F_READ (1U << 1)
-#define LCB_F_WRITE (1U << 2)
-#define LCB_F_RT (1U << 3)
-#define LCB_F_PERCPU (1U << 4)
-#define LCB_F_MUTEX (1U << 5)
-
/* callstack storage */
struct {
__uint(type, BPF_MAP_TYPE_STACK_TRACE);
@@ -96,13 +82,6 @@ struct {
__uint(max_entries, MAX_ENTRIES);
} task_data SEC(".maps");
-struct {
- __uint(type, BPF_MAP_TYPE_HASH);
- __uint(key_size, sizeof(__u64));
- __uint(value_size, sizeof(__u32));
- __uint(max_entries, MAX_ENTRIES);
-} lock_syms SEC(".maps");
-
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(key_size, sizeof(__u32));
@@ -138,6 +117,13 @@ struct {
__uint(max_entries, 1);
} cgroup_filter SEC(".maps");
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u32));
+ __uint(max_entries, MAX_ENTRIES);
+} lock_syms SEC(".maps");
+
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(key_size, sizeof(long));
@@ -159,22 +145,6 @@ struct {
__uint(max_entries, 1);
} lock_delays SEC(".maps");
-struct rw_semaphore___old {
- struct task_struct *owner;
-} __attribute__((preserve_access_index));
-
-struct rw_semaphore___new {
- atomic_long_t owner;
-} __attribute__((preserve_access_index));
-
-struct mm_struct___old {
- struct rw_semaphore mmap_sem;
-} __attribute__((preserve_access_index));
-
-struct mm_struct___new {
- struct rw_semaphore mmap_lock;
-} __attribute__((preserve_access_index));
-
struct cas_ctx {
struct contention_data *data;
u64 duration;
@@ -182,8 +152,6 @@ struct cas_ctx {
int min_done;
};
-extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak;
-
/* control flags */
const volatile int has_cpu;
const volatile int has_task;
@@ -202,6 +170,11 @@ const volatile int lock_delay;
/* determine the key of lock stat */
const volatile int aggr_mode;
+const volatile __u64 contig_page_data_addr;
+const volatile __u64 node_data_addr;
+const volatile int nr_nodes;
+const volatile int sizeof_zone;
+
int enabled;
int perf_subsys_id = -1;
@@ -222,8 +195,6 @@ int data_map_full;
struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak;
void bpf_task_release(struct task_struct *p) __ksym __weak;
-static inline __u32 check_lock_type(__u64 lock, __u32 flags);
-
static inline __u64 get_current_cgroup_id(void)
{
struct task_struct *task;
@@ -347,85 +318,6 @@ static inline int update_task_data(struct task_struct *task)
return 0;
}
-#ifndef __has_builtin
-# define __has_builtin(x) 0
-#endif
-
-static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
-{
- struct task_struct *task;
- __u64 owner = 0;
-
- if (flags & LCB_F_MUTEX) {
- struct mutex *mutex = (void *)lock;
- owner = BPF_CORE_READ(mutex, owner.counter);
- } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
- /*
- * Support for the BPF_TYPE_MATCHES argument to the
- * __builtin_preserve_type_info builtin was added at some point during
- * development of clang 15 and it's what is needed for
- * bpf_core_type_matches.
- */
-#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
- if (bpf_core_type_matches(struct rw_semaphore___old)) {
- struct rw_semaphore___old *rwsem = (void *)lock;
- owner = (unsigned long)BPF_CORE_READ(rwsem, owner);
- } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
- struct rw_semaphore___new *rwsem = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
- }
-#else
- /* assume new struct */
- struct rw_semaphore *rwsem = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
-#endif
- }
-
- if (!owner)
- return NULL;
-
- task = (void *)(owner & ~7UL);
- return task;
-}
-
-static inline __u32 check_lock_type(__u64 lock, __u32 flags)
-{
- struct task_struct *curr;
- struct mm_struct___old *mm_old;
- struct mm_struct___new *mm_new;
- struct sighand_struct *sighand;
-
- switch (flags) {
- case LCB_F_READ: /* rwsem */
- case LCB_F_WRITE:
- curr = bpf_get_current_task_btf();
- if (curr->mm == NULL)
- break;
- mm_new = (void *)curr->mm;
- if (bpf_core_field_exists(mm_new->mmap_lock)) {
- if (&mm_new->mmap_lock == (void *)lock)
- return LCD_F_MMAP_LOCK;
- break;
- }
- mm_old = (void *)curr->mm;
- if (bpf_core_field_exists(mm_old->mmap_sem)) {
- if (&mm_old->mmap_sem == (void *)lock)
- return LCD_F_MMAP_LOCK;
- }
- break;
- case LCB_F_SPIN: /* spinlock */
- curr = bpf_get_current_task_btf();
- sighand = curr->sighand;
-
- if (sighand && &sighand->siglock == (void *)lock)
- return LCD_F_SIGHAND_LOCK;
- break;
- default:
- break;
- }
- return 0;
-}
-
static inline long delay_callback(__u64 idx, void *arg)
{
__u64 target = *(__u64 *)arg;
@@ -908,145 +800,24 @@ int contention_end(u64 *ctx)
return 0;
}
-extern struct rq runqueues __ksym;
-
-const volatile __u64 contig_page_data_addr;
-const volatile __u64 node_data_addr;
-const volatile int nr_nodes;
-const volatile int sizeof_zone;
-
-struct rq___old {
- raw_spinlock_t lock;
-} __attribute__((preserve_access_index));
-
-struct rq___new {
- raw_spinlock_t __lock;
-} __attribute__((preserve_access_index));
-
-static void collect_zone_lock(void)
-{
- __u64 nr_zones, zone_off;
- __u64 lock_addr, lock_off;
- __u32 lock_flag = LOCK_CLASS_ZONE_LOCK;
-
- zone_off = offsetof(struct pglist_data, node_zones);
- lock_off = offsetof(struct zone, lock);
-
- if (contig_page_data_addr) {
- struct pglist_data *contig_page_data;
-
- contig_page_data = (void *)(long)contig_page_data_addr;
- nr_zones = BPF_CORE_READ(contig_page_data, nr_zones);
-
- for (int i = 0; i < MAX_ZONES; i++) {
- __u64 zone_addr;
-
- if (i >= nr_zones)
- break;
-
- zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off;
- lock_addr = zone_addr + lock_off;
-
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
- } else if (nr_nodes > 0) {
- struct pglist_data **node_data = (void *)(long)node_data_addr;
-
- for (int i = 0; i < nr_nodes; i++) {
- struct pglist_data *pgdat = NULL;
- int err;
-
- err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
- if (err < 0 || pgdat == NULL)
- break;
-
- nr_zones = BPF_CORE_READ(pgdat, nr_zones);
- for (int k = 0; k < MAX_ZONES; k++) {
- __u64 zone_addr;
-
- if (k >= nr_zones)
- break;
-
- zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off;
- lock_addr = zone_addr + lock_off;
-
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
- }
- }
-}
-
SEC("raw_tp/bpf_test_finish")
-int BPF_PROG(collect_lock_syms)
+int BPF_PROG(end_timestamp)
{
- __u64 lock_addr, lock_off;
- __u32 lock_flag;
-
- if (bpf_core_field_exists(struct rq___new, __lock))
- lock_off = offsetof(struct rq___new, __lock);
- else
- lock_off = offsetof(struct rq___old, lock);
-
- for (int i = 0; i < MAX_CPUS; i++) {
- struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
-
- if (rq == NULL)
- break;
-
- lock_addr = (__u64)(void *)rq + lock_off;
- lock_flag = LOCK_CLASS_RQLOCK;
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
-
- collect_zone_lock();
-
+ end_ts = bpf_ktime_get_ns();
return 0;
}
SEC("raw_tp/bpf_test_finish")
-int BPF_PROG(end_timestamp)
+int BPF_PROG(collect_lock_syms)
{
- end_ts = bpf_ktime_get_ns();
- return 0;
+ return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
}
-/*
- * bpf_iter__kmem_cache added recently so old kernels don't have it in the
- * vmlinux.h. But we cannot add it here since it will cause a compiler error
- * due to redefinition of the struct on later kernels.
- *
- * So it uses a CO-RE trick to access the member only if it has the type.
- * This will support both old and new kernels without compiler errors.
- */
-struct bpf_iter__kmem_cache___new {
- struct kmem_cache *s;
-} __attribute__((preserve_access_index));
-
SEC("iter/kmem_cache")
int slab_cache_iter(void *ctx)
{
- struct kmem_cache *s = NULL;
- struct slab_cache_data d;
- const char *nameptr;
-
- if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) {
- struct bpf_iter__kmem_cache___new *iter = ctx;
-
- s = iter->s;
- }
-
- if (s == NULL)
- return 0;
-
- nameptr = s->name;
- bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr);
-
- d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT;
- if (d.id >= LCB_F_SLAB_ID_END)
- return 0;
-
- bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST);
- return 0;
+ return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
}
char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index 652e114e6b87..fed445c9d8b4 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -45,6 +45,14 @@ struct contention_task_data {
#define LCB_F_SLAB_ID_END (1U << 26)
#define LCB_F_SLAB_ID_MASK 0x03FF0000U
+/* lock contention flags from include/trace/events/lock.h */
+#define LCB_F_SPIN (1U << 0)
+#define LCB_F_READ (1U << 1)
+#define LCB_F_WRITE (1U << 2)
+#define LCB_F_RT (1U << 3)
+#define LCB_F_PERCPU (1U << 4)
+#define LCB_F_MUTEX (1U << 5)
+
#define LCB_F_TYPE_MAX (1U << 7)
#define LCB_F_TYPE_MASK 0x0000007FU
@@ -76,4 +84,312 @@ struct slab_cache_data {
char name[SLAB_NAME_MAX];
};
+#ifdef __BPF__
+
+/* for collect_lock_syms(). 4096 was rejected by the verifier */
+#define MAX_CPUS 1024
+
+/* for collect_zone_lock(). It should be more than the actual zones. */
+#define MAX_ZONES 10
+
+extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak;
+extern struct rq runqueues __ksym __weak;
+
+#ifndef __has_builtin
+# define __has_builtin(x) 0
+#endif
+
+struct rw_semaphore___old {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
+struct rw_semaphore___new {
+ atomic_long_t count;
+ atomic_long_t owner;
+} __attribute__((preserve_access_index));
+
+struct mm_struct___old {
+ struct rw_semaphore mmap_sem;
+} __attribute__((preserve_access_index));
+
+struct mm_struct___new {
+ struct rw_semaphore mmap_lock;
+} __attribute__((preserve_access_index));
+
+struct rq___old {
+ raw_spinlock_t lock;
+} __attribute__((preserve_access_index));
+
+struct rq___new {
+ raw_spinlock_t __lock;
+} __attribute__((preserve_access_index));
+
+/*
+ * bpf_iter__kmem_cache added recently so old kernels don't have it in the
+ * vmlinux.h. But we cannot add it here since it will cause a compiler error
+ * due to redefinition of the struct on later kernels.
+ *
+ * So it uses a CO-RE trick to access the member only if it has the type.
+ * This will support both old and new kernels without compiler errors.
+ */
+struct bpf_iter__kmem_cache___new {
+ struct kmem_cache *s;
+} __attribute__((preserve_access_index));
+
+static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
+{
+ struct task_struct *task;
+ __u64 owner = 0;
+
+ if (flags & LCB_F_MUTEX) {
+ struct mutex *mutex = (void *)lock;
+
+ owner = BPF_CORE_READ(mutex, owner.counter);
+ if (owner & 0x4UL)
+ return NULL;
+ } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
+ struct rw_semaphore___new *rwsem = (void *)lock;
+
+ if (flags == LCB_F_WRITE && bpf_core_field_exists(rwsem->count.counter)) {
+ __u64 count = BPF_CORE_READ(rwsem, count.counter);
+
+ /* Must have RWSEM_WRITER_LOCKED (bit 0) set */
+ if (!(count & 1UL))
+ return NULL;
+ }
+ /*
+ * Support for the BPF_TYPE_MATCHES argument to the
+ * __builtin_preserve_type_info builtin was added at some point during
+ * development of clang 15 and it's what is needed for
+ * bpf_core_type_matches.
+ */
+#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
+ if (bpf_core_type_matches(struct rw_semaphore___old)) {
+ struct rw_semaphore___old *old = (void *)lock;
+
+ owner = (unsigned long)BPF_CORE_READ(old, owner);
+ } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
+ owner = BPF_CORE_READ(rwsem, owner.counter);
+ }
+#else
+ /* assume new struct */
+ owner = BPF_CORE_READ(rwsem, owner.counter);
+#endif
+ /* RWSEM_READER_OWNED is bit 0 of owner */
+ if (flags == LCB_F_WRITE && (owner & 1UL))
+ return NULL;
+ } else if (flags == LCB_F_RT) {
+ struct rt_mutex_base *rtm = (void *)lock;
+
+ if (bpf_core_type_exists(struct rt_mutex_base)) {
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ }
+ }
+
+ if (!owner)
+ return NULL;
+
+ task = (void *)(owner & ~7UL);
+ return task;
+}
+
+static __always_inline __u64 get_current_mmap_lock(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+ struct mm_struct___new *mm_new = NULL;
+ struct mm_struct___old *mm_old = NULL;
+ __u64 mm;
+
+ if (!curr)
+ return 0;
+
+ mm = (__u64)BPF_CORE_READ(task, mm);
+ if (!mm)
+ return 0;
+
+ if (bpf_core_field_exists(mm_new->mmap_lock))
+ return mm + bpf_core_field_offset(mm_new->mmap_lock);
+
+ if (bpf_core_field_exists(mm_old->mmap_sem))
+ return mm + bpf_core_field_offset(mm_old->mmap_sem);
+
+ return 0;
+}
+
+static __always_inline __u64 get_current_siglock(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+ struct sighand_struct *sighand = NULL;
+ __u64 sh;
+
+ if (!curr)
+ return 0;
+
+ sh = (__u64)BPF_CORE_READ(task, sighand);
+ if (!sh)
+ return 0;
+
+ return sh + bpf_core_field_offset(sighand->siglock);
+}
+
+/*
+ * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar.
+ *
+ * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the
+ * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed
+ * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such
+ * as `bpf_get_kmem_cache(u64 addr)`.
+ */
+static __always_inline __u64 get_lock_addr(void *ctx)
+{
+ __u64 lock_addr = 0;
+
+ bpf_get_func_arg(ctx, 0, &lock_addr);
+ return lock_addr;
+}
+
+static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags)
+{
+ __u64 curr;
+
+ switch (flags) {
+ case LCB_F_READ: /* rwsem */
+ case LCB_F_WRITE:
+ curr = bpf_get_current_task();
+ if (get_current_mmap_lock(curr) == lock)
+ return LCD_F_MMAP_LOCK;
+ break;
+ case LCB_F_SPIN: /* spinlock */
+ curr = bpf_get_current_task();
+ if (get_current_siglock(curr) == lock)
+ return LCD_F_SIGHAND_LOCK;
+ break;
+ default:
+ break;
+ }
+ return 0;
+}
+
+static __always_inline void do_collect_zone_lock(void *lock_syms_map,
+ __u64 contig_page_data_addr,
+ __u64 node_data_addr,
+ int nr_nodes,
+ int sizeof_zone)
+{
+ __u64 nr_zones, zone_off;
+ __u64 lock_addr, lock_off;
+ __u32 lock_flag = LOCK_CLASS_ZONE_LOCK;
+
+ zone_off = offsetof(struct pglist_data, node_zones);
+ lock_off = offsetof(struct zone, lock);
+
+ if (contig_page_data_addr) {
+ struct pglist_data *contig_page_data;
+
+ contig_page_data = (void *)(long)contig_page_data_addr;
+ nr_zones = BPF_CORE_READ(contig_page_data, nr_zones);
+
+ for (int i = 0; i < MAX_ZONES; i++) {
+ __u64 zone_addr;
+
+ if (i >= nr_zones)
+ break;
+
+ zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off;
+ lock_addr = zone_addr + lock_off;
+
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ } else if (nr_nodes > 0) {
+ struct pglist_data **node_data = (void *)(long)node_data_addr;
+
+ for (int i = 0; i < nr_nodes; i++) {
+ struct pglist_data *pgdat = NULL;
+ int err;
+
+ err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
+ if (err < 0)
+ break;
+ if (pgdat == NULL)
+ continue;
+
+ nr_zones = BPF_CORE_READ(pgdat, nr_zones);
+ for (int k = 0; k < MAX_ZONES; k++) {
+ __u64 zone_addr;
+
+ if (k >= nr_zones)
+ break;
+
+ zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off;
+ lock_addr = zone_addr + lock_off;
+
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ }
+ }
+}
+
+static __always_inline int do_collect_lock_syms(void *lock_syms_map,
+ __u64 contig_page_data_addr,
+ __u64 node_data_addr,
+ int nr_nodes,
+ int sizeof_zone)
+{
+ __u64 lock_addr, lock_off;
+ __u32 lock_flag;
+
+ if (bpf_core_field_exists(struct rq___new, __lock))
+ lock_off = offsetof(struct rq___new, __lock);
+ else
+ lock_off = offsetof(struct rq___old, lock);
+
+ if (&runqueues) {
+ for (int i = 0; i < MAX_CPUS; i++) {
+ struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
+
+ if (rq == NULL)
+ break;
+
+ lock_addr = (__u64)(void *)rq + lock_off;
+ lock_flag = LOCK_CLASS_RQLOCK;
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ }
+
+ do_collect_zone_lock(lock_syms_map, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
+
+ return 0;
+}
+
+static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map,
+ __u32 *slab_cache_id)
+{
+ struct kmem_cache *s = NULL;
+ struct slab_cache_data d = {};
+ const char *nameptr;
+
+ if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) {
+ struct bpf_iter__kmem_cache___new *iter = ctx;
+
+ s = iter->s;
+ }
+
+ if (s == NULL)
+ return 0;
+
+ nameptr = s->name;
+ bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr);
+
+ d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT;
+ if (d.id >= LCB_F_SLAB_ID_END)
+ return 0;
+
+ bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST);
+ return 0;
+}
+
+#endif /* __BPF__ */
+
#endif /* UTIL_BPF_SKEL_LOCK_DATA_H */
diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
index 94e42b022ab1..b75ddc6846bb 100644
--- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
+++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
@@ -76,6 +76,7 @@ struct signal_struct {
} __attribute__((preserve_access_index));
struct rw_semaphore {
+ atomic_long_t count;
atomic_long_t owner;
} __attribute__((preserve_access_index));
@@ -83,6 +84,10 @@ struct mutex {
atomic_long_t owner;
} __attribute__((preserve_access_index));
+struct rt_mutex_base {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
struct kernfs_node {
u64 id;
} __attribute__((preserve_access_index));
diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c
index 119a7206f3cd..a1bb78d6e29c 100644
--- a/tools/perf/util/lock-contention.c
+++ b/tools/perf/util/lock-contention.c
@@ -22,6 +22,24 @@ struct callstack_filter {
static LIST_HEAD(callstack_filters);
struct hlist_head *lockhash_table;
+const struct perf_lock_type lock_type_table[] = {
+ { 0, "semaphore", "semaphore" },
+ { LCB_F_SPIN, "spinlock", "spinlock" },
+ { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
+ { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
+ { LCB_F_READ, "rwsem:R", "rwsem" },
+ { LCB_F_WRITE, "rwsem:W", "rwsem" },
+ { LCB_F_RT, "rt-mutex", "rt-mutex" },
+ { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
+ { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
+ { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
+ { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
+ { LCB_F_MUTEX, "mutex", "mutex" },
+ { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
+ /* alias for optimistic spinning only */
+ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
+};
+
int parse_call_stack(const struct option *opt __maybe_unused, const char *str,
int unset __maybe_unused)
{
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index 59c94190b092..68e9bd77e8a9 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -129,6 +129,19 @@ struct thread_stat {
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+struct perf_lock_type {
+ unsigned int flags;
+ /*
+ * Name of the lock flags (access), with delimiter ':'.
+ * For example, rwsem:R or rwsem:W.
+ */
+ const char *flags_name;
+ /* Name of the lock (type), for example, rwlock or rwsem. */
+ const char *lock_name;
+};
+
+extern const struct perf_lock_type lock_type_table[14];
+
struct evlist;
struct machine;
struct target;
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter
2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
@ 2026-10-17 5:45 ` Ian Rogers
2026-10-17 6:09 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers
` (3 subsequent siblings)
5 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Namhyung Kim
Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, rcu
Add BPF sample filter predicates lock_held(<lock>),
lock_held_read(<lock>) / lock_held_shared(<lock>), and
lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R'
and ':W' suffixes on <lock>) to allow recording perf event samples only
when a specified kernel lock is held (or not held with '!').
Supported <lock> specifications match perf lock contention:
- Global/static kernel lock symbols (with or without '&', e.g.
lock_held(&cgroup_mutex) or lock_held(cgroup_mutex))
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with '&' (e.g. lock_held(&dentry),
lock_held_read(&dentry))
- Per-process and semi-global locks: mmap_lock, siglock, rq_lock,
zone_lock
- Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W,
rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W,
semaphore
Supported lock hold modes by lock family:
- Both read/shared and write/exclusive: rwsem (including mmap_lock)
and rwlock.
- Write/exclusive only: mutex, spinlock (including siglock, rq_lock,
zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write()
/ percpu_up_write(); fast-path percpu_down_read() / percpu_up_read()
are inlined per-CPU counter operations without function entry points).
When evaluating the filter on sample overflow, perf_sample_filter first
inspects the lock's owner field directly in kernel memory where
possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls
back to a held_locks BPF hash map (tracking separate read_count and
write_count) populated by fentry/fexit hooks on kernel lock/unlock
functions. To minimize tracing overhead, only the BPF hooks for the
requested lock family and mode are autoloaded and attached.
Example usage:
# perf record -a -g -e cycles --filter 'lock_held(&mmap_lock)' \
-- perf bench mem mmap -t 2 -l 100
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ]
# perf report --stdio --no-children
# Overhead Command Shared Object Symbol
# ........ ........ ................. ........................
#
18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms
|
---clear_page_erms
get_page_from_freelist
__alloc_pages_noprof
folio_alloc_mpol_noprof
vma_alloc_folio_noprof
do_anonymous_page
__handle_mm_fault
handle_mm_fault
do_user_addr_fault
exc_page_fault
asm_exc_page_fault
worker_thread
Read/shared vs write/exclusive holding can also be filtered separately:
# perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' ...
# perf record -a -g -e cycles --filter 'lock_held_write(&mmap_lock)' ...
# perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' ...
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 41 +-
tools/perf/util/bpf-filter.c | 934 ++++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 46 +-
tools/perf/util/bpf_skel/sample-filter.h | 59 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 920 ++++++++++++++++++
tools/perf/util/parse-events.c | 3 +-
8 files changed, 2021 insertions(+), 26 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 2351f7b4f5fc..246ae085744d 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -204,7 +204,46 @@ OPTIONS
ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr,
code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat,
p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock,
- mem_dtlb, mem_blk, mem_hops, uid, gid
+ mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>),
+ lock_held_read(<lock>), lock_held_shared(<lock>),
+ lock_held_write(<lock>), lock_held_exclusive(<lock>)
+
+ The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters
+ samples based on whether the specified kernel lock is currently held
+ by the kernel thread or CPU. To differentiate between shared/read
+ holding and exclusive/write holding, use 'lock_held_read(<lock>)'
+ (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)'
+ (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to
+ <lock> (e.g. 'lock_held(&mmap_lock:R)' or 'lock_held(&mmap_lock:W)').
+ <lock> can be a kernel lock symbol name (with or without a leading
+ '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'),
+ a hexadecimal lock address, a slab cache name prefixed with '&'
+ (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known
+ lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock
+ type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
+ 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
+ 'semaphore'). Both read/shared and write/exclusive tracking are
+ supported for 'rwsem' (including 'mmap_lock') and 'rwlock';
+ 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'),
+ 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()')
+ support write/exclusive tracking only.
+
+ Note that 'lock_held(<lock>)' introduces overhead outside of sample
+ overflow handling: in addition to checking the lock's owner field on
+ each sample where possible (such as for mutexes, rt-mutexes, and
+ write-locked rwsems), it attaches BPF fentry/fexit programs to kernel
+ lock acquire and release functions to track held locks (such as
+ spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a
+ BPF map. These BPF hooks run on every kernel lock acquisition and
+ release for the attached lock family, even when no perf sample is
+ being taken. To minimize this overhead, only the hooks for the
+ relevant lock family and mode are attached when known (for example,
+ 'lock_held_read(&mmap_lock)' only hooks rwsem read functions,
+ 'lock_held_write(&mmap_lock)' only hooks rwsem write functions,
+ '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks
+ spinlock functions), whereas slab filters ('&<slab>'), raw hex
+ addresses, or symbols without a recognizable lock-type suffix attach
+ hooks across all lock families (including high-frequency spinlocks).
The <operator> can be one of:
==, !=, >, >=, <, <=, &
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 72e83ac1d6a4..35d6f7c63868 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -40,21 +40,35 @@
* The 'dropped' map is to keep how many samples it dropped by the filter and
* it will be reported as lost samples.
*/
+#include <errno.h>
+#include <stdio.h>
#include <stdlib.h>
+#include <stdbool.h>
+#include <stddef.h>
+#include <string.h>
#include <fcntl.h>
+#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/stat.h>
#include <bpf/bpf.h>
+#include <bpf/btf.h>
#include <linux/err.h>
+#include <linux/kernel.h>
#include <linux/list.h>
+#include <linux/string.h>
+#include <linux/zalloc.h>
#include <api/fs/fs.h>
#include <internal/xyarray.h>
#include <perf/threadmap.h>
+#include <symbol/kallsyms.h>
#include "util/cap.h"
#include "util/debug.h"
#include "util/evsel.h"
+#include "util/lock-contention.h"
+#include "util/string2.h"
+#include "util/symbol_conf.h"
#include "util/target.h"
#include "util/thread_map.h"
#include "util/bpf-utils.h"
@@ -63,6 +77,7 @@
#include <util/bpf-filter-flex.h>
#include <util/bpf-filter-bison.h>
+#include "bpf_skel/lock_data.h"
#include "bpf_skel/sample-filter.h"
#include "bpf_skel/sample_filter.skel.h"
@@ -127,7 +142,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
(evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
return 0;
- if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
+ if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID ||
+ expr->term == PBF_TERM_LOCK_HELD) {
/* Not dependent on the sample_type as computed from a BPF helper. */
return 0;
}
@@ -155,9 +171,713 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
return -1;
}
-static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry)
+/*
+ * Program groups for `lock_held(...)` BPF tracing hooks.
+ *
+ * When a `lock_held(...)` filter is specified, only the BPF tracing program
+ * groups required for the requested lock(s) are autoloaded and attached.
+ * If no `lock_held(...)` filter is present, none of these programs are loaded.
+ */
+enum lock_prog_group {
+ LOCK_PROG_MUTEX = 1U << 0,
+ LOCK_PROG_RWSEM_READ = 1U << 1,
+ LOCK_PROG_RWSEM_WRITE = 1U << 2,
+ LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE,
+ LOCK_PROG_SPIN = 1U << 3,
+ LOCK_PROG_RWLOCK_READ = 1U << 4,
+ LOCK_PROG_RWLOCK_WRITE = 1U << 5,
+ LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE,
+ LOCK_PROG_RTMUTEX = 1U << 6,
+ LOCK_PROG_SEM = 1U << 7,
+ LOCK_PROG_PERCPU_RWSEM = 1U << 8,
+ LOCK_PROG_ALL = 0x1ffU,
+};
+
+static unsigned int lock_flags_to_prog_group(unsigned int flags)
+{
+ if (flags & LCB_F_MUTEX)
+ return LOCK_PROG_MUTEX;
+ if (flags & LCB_F_RT)
+ return LOCK_PROG_RTMUTEX;
+ if (flags & LCB_F_PERCPU)
+ return LOCK_PROG_PERCPU_RWSEM;
+ if (flags == LCB_F_SPIN)
+ return LOCK_PROG_SPIN;
+ if (flags & LCB_F_SPIN)
+ return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE;
+ if (flags & LCB_F_READ)
+ return LOCK_PROG_RWSEM_READ;
+ if (flags & LCB_F_WRITE)
+ return LOCK_PROG_RWSEM_WRITE;
+ return LOCK_PROG_SEM;
+}
+
+struct lock_prog_desc {
+ size_t prog_offset;
+ const char *btf_func;
+ unsigned int group;
+};
+
+#define LOCK_FPROG(field, func, grp) \
+ { offsetof(struct sample_filter_bpf, progs.field), func, grp }
+
+static const struct lock_prog_desc lock_progs[] = {
+ /* Mutex */
+ LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
+ LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
+ /* RW semaphore */
+ LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested",
+ LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested",
+ LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM),
+ LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE),
+ /* Spinlock */
+ LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN),
+ /* RWlock */
+ LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE),
+ /* RT-Mutex */
+ LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible",
+ LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX),
+ /* Semaphore */
+ LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM),
+ LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM),
+ /* Percpu-RWsem */
+ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
+ LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+};
+
+#undef LOCK_FPROG
+
+static int split_lock_name_mode(const char *name, int expr_part,
+ char *base_buf, size_t buf_sz,
+ bool *has_amp, unsigned int *mode_out)
+{
+ const char *bare;
+ size_t len;
+ unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK;
+ unsigned int suffix_mode = 0;
+
+ if (!name || !*name)
+ return -EINVAL;
+
+ *has_amp = (name[0] == '&');
+ bare = *has_amp ? name + 1 : name;
+ len = strlen(bare);
+ if (len == 0)
+ return -EINVAL;
+
+ if (len > 2 && !strcmp(bare + len - 2, ":R")) {
+ suffix_mode = PBF_LOCK_MODE_READ;
+ len -= 2;
+ } else if (len > 2 && !strcmp(bare + len - 2, ":W")) {
+ suffix_mode = PBF_LOCK_MODE_WRITE;
+ len -= 2;
+ }
+
+ if (len == 0 || len >= buf_sz)
+ return -EINVAL;
+
+ if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY &&
+ suffix_mode && expr_mode != suffix_mode)
+ return -EINVAL;
+
+ memcpy(base_buf, bare, len);
+ base_buf[len] = '\0';
+
+ if (suffix_mode)
+ *mode_out = suffix_mode;
+ else if (expr_mode)
+ *mode_out = expr_mode;
+ else
+ *mode_out = PBF_LOCK_MODE_ANY;
+
+ return 0;
+}
+
+/*
+ * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R",
+ * "spinlock", "percpu-rwsem").
+ *
+ * A lock type name may correspond to up to two flag combinations (e.g.
+ * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed
+ * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is
+ * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`.
+ */
+static bool parse_lock_type_name(const char *name, unsigned int mode,
+ struct sample_filter_bpf *skel,
+ u64 *encoded_pair, unsigned int *prog_groups)
+{
+ bool found = false;
+ u32 flag1 = 0xff, flag2 = 0xff;
+ unsigned int groups = 0;
+ const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name;
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) {
+ u32 f = lock_type_table[i].flags;
+ bool is_read = (f & LCB_F_READ) != 0;
+
+ if (strcmp(lock_type_table[i].lock_name, lookup) &&
+ strcmp(lock_type_table[i].flags_name, name))
+ continue;
+
+ /*
+ * percpu_down_read() / percpu_up_read() fast paths are inlined
+ * per-CPU counter increments without function entry points, so
+ * only writer holding (percpu_down_write / percpu_up_write) can
+ * be tracked for percpu-rwsem. rwlock-rt is also not hooked.
+ */
+ if ((f & LCB_F_PERCPU) && is_read)
+ continue;
+ if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE)))
+ continue;
+
+ if (is_read && !(mode & PBF_LOCK_MODE_READ))
+ continue;
+ if (!is_read && !(mode & PBF_LOCK_MODE_WRITE))
+ continue;
+
+ if (!found)
+ flag1 = f;
+ else if (f != flag1)
+ flag2 = f;
+ if (skel && f < LCB_F_TYPE_MAX) {
+ skel->bss->has_lock_type = 1;
+ skel->bss->lock_type_filter[f] = 1;
+ }
+ groups |= lock_flags_to_prog_group(f);
+ found = true;
+ }
+
+ if (!found)
+ return false;
+
+ if (encoded_pair)
+ *encoded_pair = flag1 | ((u64)flag2 << 8);
+ if (prog_groups)
+ *prog_groups |= groups;
+ return true;
+}
+
+struct lock_sym_cb_args {
+ const char *name;
+ u64 addr;
+ bool found;
+};
+
+static int find_lock_sym_cb(void *arg, const char *name,
+ char type __maybe_unused, u64 start)
+{
+ struct lock_sym_cb_args *args = arg;
+
+ if (!strcmp(name, args->name)) {
+ args->found = true;
+ args->addr = start;
+ if (start != 0)
+ return 1;
+ }
+ return 0;
+}
+
+static int find_kernel_lock_sym(const char *name, u64 *addr)
{
- int i = 0;
+ struct lock_sym_cb_args args = {
+ .name = name,
+ };
+ const char *kallsyms_filename = "/proc/kallsyms";
+
+ if (symbol_conf.kallsyms_name != NULL)
+ kallsyms_filename = symbol_conf.kallsyms_name;
+
+ if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0)
+ return -ENOENT;
+
+ if (!args.found)
+ return -ENOENT;
+
+ if (args.addr == 0) {
+ pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n",
+ name);
+ return -EACCES;
+ }
+
+ *addr = args.addr;
+ return 0;
+}
+
+static bool parse_lock_hex_addr(const char *str, u64 *addr)
+{
+ char *end;
+ unsigned long long val;
+
+ if (!str || !*str)
+ return false;
+
+ errno = 0;
+ val = strtoull(str, &end, 16);
+ if (*end != '\0' || errno != 0 || val == 0)
+ return false;
+
+ *addr = val;
+ return true;
+}
+
+static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf)
+{
+ u64 addr = 0;
+ char *buf = NULL, *p;
+ size_t len;
+ long last = -1;
+ int ret;
+
+ if (!btf)
+ return;
+
+ ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT);
+ if (ret < 0)
+ return;
+
+ ret = btf__resolve_size(btf, ret);
+ if (ret < 0)
+ return;
+
+ skel->rodata->sizeof_zone = ret;
+
+ if (find_kernel_lock_sym("contig_page_data", &addr) == 0) {
+ skel->rodata->contig_page_data_addr = addr;
+ return;
+ }
+
+ if (find_kernel_lock_sym("node_data", &addr) == 0)
+ skel->rodata->node_data_addr = addr;
+
+ if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) {
+ p = buf;
+ while (p && *p) {
+ char *end;
+
+ last = strtol(p, &end, 0);
+ if (end == p)
+ break;
+ p = end;
+ if (*p == ',' || *p == '-' || *p == '\n')
+ p++;
+ }
+ skel->rodata->nr_nodes = last + 1;
+ free(buf);
+ }
+}
+
+/*
+ * Enable or disable autoload for each lock-tracking BPF program in the
+ * skeleton before `sample_filter_bpf__load()` is called.
+ *
+ * Each program is enabled only if its group is in `needed_groups` and its
+ * target kernel function or tracepoint exists in the running kernel's BTF.
+ */
+static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
+ unsigned int needed_groups, bool needs_lock_syms,
+ bool needs_slab_iter)
+{
+ unsigned int enabled_progs = 0;
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) {
+ struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset;
+ struct bpf_program *prog = *progp;
+ bool enable = false;
+
+ if ((needed_groups & lock_progs[i].group) && btf &&
+ lock_progs[i].btf_func) {
+ if (btf__find_by_name_kind(btf, lock_progs[i].btf_func,
+ BTF_KIND_FUNC) > 0)
+ enable = true;
+ }
+
+ if (enable)
+ enabled_progs++;
+ bpf_program__set_autoload(prog, enable);
+ }
+
+ bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms);
+ if (needs_slab_iter && btf &&
+ btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) {
+ bpf_program__set_autoload(skel->progs.slab_cache_iter, true);
+ } else {
+ bpf_program__set_autoload(skel->progs.slab_cache_iter, false);
+ }
+
+ if (needed_groups && !enabled_progs) {
+ pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
+ return -ENOTSUP;
+ }
+ return 0;
+}
+
+/*
+ * Pre-scan a single filter expression prior to loading the BPF skeleton in
+ * order to determine which BPF maps need to be sized and which lock tracing
+ * program groups need to be autoloaded.
+ */
+static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
+ bool *has_lock_filter,
+ unsigned int *prog_groups,
+ bool *needs_lock_syms,
+ bool *needs_slab_iter)
+{
+ char base[128];
+ bool has_amp;
+ unsigned int mode;
+ u64 dummy;
+
+ if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name)
+ return;
+
+ *has_lock_filter = true;
+ if (split_lock_name_mode(expr->lock_name, expr->part, base,
+ sizeof(base), &has_amp, &mode) < 0)
+ return;
+
+ if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
+ if (mode & PBF_LOCK_MODE_READ)
+ *prog_groups |= LOCK_PROG_RWSEM_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ *prog_groups |= LOCK_PROG_RWSEM_WRITE;
+ return;
+ }
+ if (!strcmp(base, "siglock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ return;
+ }
+ if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *needs_lock_syms = true;
+ return;
+ }
+ if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, prog_groups))
+ return;
+
+ if (!has_amp && parse_lock_hex_addr(base, &dummy)) {
+ *prog_groups |= LOCK_PROG_ALL;
+ return;
+ }
+
+ if (has_amp)
+ *needs_slab_iter = true;
+
+ if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) {
+ *prog_groups |= LOCK_PROG_RTMUTEX;
+ } else if (strtailcmp(base, "mutex") == 0) {
+ *prog_groups |= LOCK_PROG_MUTEX;
+ } else if (strtailcmp(base, "percpu_rwsem") == 0 ||
+ strtailcmp(base, "pcpu_rwsem") == 0) {
+ *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
+ } else if (strtailcmp(base, "rwsem") == 0) {
+ if (mode & PBF_LOCK_MODE_READ)
+ *prog_groups |= LOCK_PROG_RWSEM_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ *prog_groups |= LOCK_PROG_RWSEM_WRITE;
+ } else if (strtailcmp(base, "spinlock") == 0) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ } else if (strtailcmp(base, "rwlock") == 0) {
+ if (mode & PBF_LOCK_MODE_READ)
+ *prog_groups |= LOCK_PROG_RWLOCK_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ *prog_groups |= LOCK_PROG_RWLOCK_WRITE;
+ } else {
+ *prog_groups |= LOCK_PROG_ALL;
+ }
+
+ if (!has_amp && find_kernel_lock_sym(base, &dummy) != 0)
+ *needs_slab_iter = true;
+}
+
+static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter,
+ unsigned int *prog_groups,
+ bool *needs_lock_syms,
+ bool *needs_slab_iter)
+{
+ struct perf_bpf_filter_expr *expr, *group;
+
+ *has_lock_filter = false;
+ *prog_groups = 0;
+ *needs_lock_syms = false;
+ *needs_slab_iter = false;
+
+ list_for_each_entry(expr, &evsel->bpf_filters, list) {
+ prescan_one_lock_expr(expr, has_lock_filter, prog_groups,
+ needs_lock_syms, needs_slab_iter);
+ if (expr->op == PBF_OP_GROUP_BEGIN) {
+ list_for_each_entry(group, &expr->groups, list) {
+ prescan_one_lock_expr(group, has_lock_filter,
+ prog_groups, needs_lock_syms,
+ needs_slab_iter);
+ }
+ }
+ }
+}
+
+static void run_slab_cache_iter(struct sample_filter_bpf *skel)
+{
+ struct bpf_link *link;
+ int fd;
+ char buf[256];
+
+ if (!bpf_program__autoload(skel->progs.slab_cache_iter))
+ return;
+
+ link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL);
+ if (!link || IS_ERR(link)) {
+ pr_debug("cannot attach slab cache iter\n");
+ return;
+ }
+
+ fd = bpf_iter_create(bpf_link__fd(link));
+ if (fd >= 0) {
+ while (read(fd, buf, sizeof(buf)) > 0)
+ continue;
+ close(fd);
+ }
+
+ bpf_link__destroy(link);
+}
+
+static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
+ u64 *kmem_cache_addr)
+{
+ int cache_fd;
+ u64 key, *prev_key = NULL;
+
+ if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter))
+ return false;
+
+ cache_fd = bpf_map__fd(skel->maps.slab_caches);
+ while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) {
+ struct slab_cache_data data;
+
+ if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0)
+ break;
+
+ if (!strcmp(data.name, name)) {
+ *kmem_cache_addr = key;
+ return true;
+ }
+ prev_key = &key;
+ }
+ return false;
+}
+
+/*
+ * Resolve a `lock_held(<name>)` filter expression into `entry->part`
+ * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate
+ * the corresponding BPF filter maps and BSS control flags.
+ *
+ * Resolution order:
+ * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
+ * `rq_lock`, `zone_lock` (with or without leading '&').
+ * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`,
+ * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`,
+ * `semaphore`, etc.
+ * 3. Hexadecimal lock address (when no leading '&').
+ * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`).
+ * 5. Global/static kernel lock symbol in kallsyms (with or without leading '&',
+ * e.g. `&cgroup_mutex` or `cgroup_mutex`).
+ * 6. Slab cache name without leading '&' fallback.
+ */
+static int resolve_lock_expr(struct sample_filter_bpf *skel,
+ struct perf_bpf_filter_expr *expr,
+ struct perf_bpf_filter_entry *entry)
+{
+ const char *name = expr->lock_name;
+ char base[128];
+ bool has_amp;
+ unsigned int mode;
+ u64 addr = 0, encoded_pair = 0;
+ u8 one = 1;
+ int ret;
+
+ if (split_lock_name_mode(name, expr->part, base, sizeof(base),
+ &has_amp, &mode) < 0) {
+ pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: "");
+ return -EINVAL;
+ }
+
+ if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
+ entry->part = PBF_LOCK_MMAP | mode;
+ entry->target = 0;
+ if (skel)
+ skel->bss->has_mmap_lock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "siglock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_SIGHAND | mode;
+ entry->target = 0;
+ if (skel)
+ skel->bss->has_siglock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "rq_lock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_CLASS | mode;
+ entry->target = LOCK_CLASS_RQLOCK;
+ if (skel)
+ skel->bss->has_lock_class = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "zone_lock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_CLASS | mode;
+ entry->target = LOCK_CLASS_ZONE_LOCK;
+ if (skel)
+ skel->bss->has_lock_class = 1;
+ return 0;
+ }
+
+ if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
+ entry->part = PBF_LOCK_TYPE | mode;
+ entry->target = encoded_pair;
+ return 0;
+ }
+
+ if (!has_amp && parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) {
+ pr_err("Error: Lock type '%s' does not support the requested read/write mode\n",
+ name);
+ return -EINVAL;
+ }
+
+ if (!has_amp && parse_lock_hex_addr(base, &addr)) {
+ entry->part = PBF_LOCK_ADDR | mode;
+ entry->target = addr;
+ if (skel) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
+ }
+ return 0;
+ }
+
+ if (has_amp && find_slab_cache(skel, base, &addr)) {
+ int fd = bpf_map__fd(skel->maps.slab_filter);
+
+ entry->part = PBF_LOCK_SLAB | mode;
+ entry->target = addr;
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_slab = 1;
+ return 0;
+ }
+
+ ret = find_kernel_lock_sym(base, &addr);
+ if (ret == 0) {
+ entry->part = PBF_LOCK_ADDR | mode;
+ entry->target = addr;
+ if (skel) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
+ }
+ return 0;
+ }
+ if (ret == -EACCES)
+ return ret;
+
+ if (!has_amp && find_slab_cache(skel, base, &addr)) {
+ int fd = bpf_map__fd(skel->maps.slab_filter);
+
+ entry->part = PBF_LOCK_SLAB | mode;
+ entry->target = addr;
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_slab = 1;
+ return 0;
+ }
+
+ pr_err("Error: Cannot resolve kernel lock '%s'\n", name);
+ return -EINVAL;
+}
+
+static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct perf_bpf_filter_entry *entry)
+{
+ int i = 0, ret;
struct perf_bpf_filter_expr *expr;
list_for_each_entry(expr, &evsel->bpf_filters, list) {
@@ -171,6 +891,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].part = expr->part;
entry[i].term = expr->term;
entry[i].value = expr->val;
+ entry[i].target = 0;
+ if (expr->term == PBF_TERM_LOCK_HELD) {
+ ret = resolve_lock_expr(skel, expr, &entry[i]);
+ if (ret < 0)
+ return ret;
+ }
i++;
if (expr->op == PBF_OP_GROUP_BEGIN) {
@@ -184,6 +910,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].part = group->part;
entry[i].term = group->term;
entry[i].value = group->val;
+ entry[i].target = 0;
+ if (group->term == PBF_TERM_LOCK_HELD) {
+ ret = resolve_lock_expr(skel, group, &entry[i]);
+ if (ret < 0)
+ return ret;
+ }
i++;
}
@@ -423,10 +1155,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
{
int i, x, y, fd, ret;
struct sample_filter_bpf *skel = NULL;
+ struct btf *btf = NULL;
struct bpf_program *prog;
struct bpf_link *link;
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
+ bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false;
+ unsigned int prog_groups = 0;
#if LIBBPF_CURRENT_VERSION_GEQ(1, 7)
DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts,
.dont_enable = true);
@@ -438,15 +1173,24 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
if (entry == NULL)
return -1;
- ret = get_filter_entries(evsel, entry);
- if (ret < 0) {
- pr_err("Failed to process filter entries\n");
- goto err;
- }
+ prescan_lock_filters(evsel, &has_lock_filter, &prog_groups,
+ &needs_lock_syms, &needs_slab_iter);
- if (needs_idx_hash && geteuid() != 0) {
+ if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
int zero = 0;
+ if (has_lock_filter) {
+ pr_err("Error: lock_held BPF filter requires CAP_BPF capability\n");
+ ret = -EPERM;
+ goto err;
+ }
+
+ ret = get_filter_entries(evsel, NULL, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
+
/* The filters map is shared among other processes */
ret = create_idx_hash(evsel, entry);
if (ret < 0)
@@ -484,13 +1228,69 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return 0;
}
- skel = sample_filter_bpf__open_and_load();
+ skel = sample_filter_bpf__open();
if (!skel) {
+ ret = -errno;
+ pr_err("Failed to open perf sample-filter BPF skeleton\n");
+ goto err;
+ }
+
+ if (has_lock_filter) {
+ btf = btf__load_vmlinux_btf();
+ bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES);
+ bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS);
+ if (needs_slab_iter) {
+ bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS);
+ bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES);
+ }
+ if (needs_lock_syms) {
+ bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
+ init_numa_data(skel, btf);
+ }
+ ret = configure_lock_progs(skel, btf, prog_groups, needs_lock_syms,
+ needs_slab_iter);
+ if (ret < 0)
+ goto err;
+ } else {
+ configure_lock_progs(skel, NULL, 0, false, false);
+ }
+
+ if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
pr_err("Failed to load perf sample-filter BPF skeleton\n");
goto err;
}
+ if (has_lock_filter && needs_slab_iter)
+ run_slab_cache_iter(skel);
+
+ ret = get_filter_entries(evsel, skel, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
+
+ if (has_lock_filter) {
+ if (needs_lock_syms && skel->bss->has_lock_class) {
+ DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts,
+ .flags = BPF_F_TEST_RUN_ON_CPU,
+ );
+ int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms);
+
+ bpf_prog_test_run_opts(prog_fd, &opts);
+ }
+ bpf_program__set_autoattach(skel->progs.perf_sample_filter, false);
+ bpf_program__set_autoattach(skel->progs.collect_lock_syms, false);
+ bpf_program__set_autoattach(skel->progs.slab_cache_iter, false);
+
+ ret = sample_filter_bpf__attach(skel);
+ if (ret < 0) {
+ pr_err("Failed to attach lock sample-filter BPF programs\n");
+ goto err;
+ }
+ skel->bss->lock_enabled = 1;
+ }
+
i = 0;
fd = bpf_map__fd(skel->maps.filters);
@@ -513,11 +1313,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
}
}
}
+ btf__free(btf);
free(entry);
evsel->bpf_skel = skel;
return 0;
err:
+ btf__free(btf);
free(entry);
if (!list_empty(&pinned_filters)) {
struct pinned_filter_idx *pfi, *tmp;
@@ -532,6 +1334,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return ret;
}
+void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr)
+{
+ struct perf_bpf_filter_expr *group, *tmp;
+
+ if (!expr)
+ return;
+
+ list_for_each_entry_safe(group, tmp, &expr->groups, list) {
+ list_del(&group->list);
+ perf_bpf_filter_expr__delete(group);
+ }
+ zfree(&expr->lock_name);
+ free(expr);
+}
+
int perf_bpf_filter__destroy(struct evsel *evsel)
{
struct perf_bpf_filter_expr *expr, *tmp;
@@ -539,7 +1356,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel)
list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) {
list_del(&expr->list);
- free(expr);
+ perf_bpf_filter_expr__delete(expr);
}
sample_filter_bpf__destroy(evsel->bpf_skel);
@@ -591,7 +1408,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
{
struct perf_bpf_filter_expr *expr;
- expr = malloc(sizeof(*expr));
+ expr = zalloc(sizeof(*expr));
if (expr != NULL) {
expr->term = term;
expr->part = part;
@@ -602,6 +1419,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
return expr;
}
+struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
+ int mode,
+ enum perf_bpf_filter_op op,
+ unsigned long val)
+{
+ struct perf_bpf_filter_expr *expr;
+
+ if (!lock_name || !*lock_name)
+ return NULL;
+
+ expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val);
+ if (!expr)
+ return NULL;
+
+ expr->lock_name = strdup(lock_name);
+ if (!expr->lock_name) {
+ free(expr);
+ return NULL;
+ }
+ return expr;
+}
+
static bool check_bpf_filter_capable(void)
{
int fd;
@@ -622,14 +1461,11 @@ static bool check_bpf_filter_capable(void)
return false;
}
-int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
+int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
{
YY_BUFFER_STATE buffer;
int ret;
- if (!check_bpf_filter_capable())
- return -EPERM;
-
buffer = perf_bpf_filter__scan_string(str);
ret = perf_bpf_filter_parse(expr_head);
@@ -641,6 +1477,53 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
return ret;
}
+int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
+{
+ if (!check_bpf_filter_capable())
+ return -EPERM;
+
+ return __perf_bpf_filter__parse(expr_head, str);
+}
+
+static int pin_filter_object(struct sample_filter_bpf *skel, const char *path)
+{
+ char *prog_path = NULL;
+ int ret;
+
+ ret = bpf_object__pin_maps(skel->obj, path);
+ if (ret < 0)
+ return ret;
+
+ if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
+ bpf_object__unpin_maps(skel->obj, path);
+ return -ENOMEM;
+ }
+
+ ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path);
+ if (ret < 0)
+ bpf_object__unpin_maps(skel->obj, path);
+
+ free(prog_path);
+ return ret;
+}
+
+static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path)
+{
+ char *prog_path = NULL;
+ int ret, map_ret;
+
+ if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
+ bpf_object__unpin_maps(skel->obj, path);
+ return -ENOMEM;
+ }
+
+ ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path);
+ free(prog_path);
+
+ map_ret = bpf_object__unpin_maps(skel->obj, path);
+ return ret < 0 ? ret : map_ret;
+}
+
int perf_bpf_filter__pin(void)
{
struct sample_filter_bpf *skel;
@@ -654,6 +1537,8 @@ int perf_bpf_filter__pin(void)
goto err;
}
+ configure_lock_progs(skel, NULL, 0, false, false);
+
/* pinned program will use pid-hash */
bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH);
@@ -674,7 +1559,7 @@ int perf_bpf_filter__pin(void)
goto err;
}
- ret = bpf_object__pin(skel->obj, path);
+ ret = pin_filter_object(skel, path);
if (ret < 0) {
pr_err("Failed to pin BPF filter objects\n");
goto err;
@@ -683,7 +1568,7 @@ int perf_bpf_filter__pin(void)
/* setup access permissions for the pinned objects */
dir_fd = open(path, O_PATH);
if (dir_fd < 0) {
- bpf_object__unpin(skel->obj, path);
+ unpin_filter_object(skel, path);
ret = dir_fd;
goto err;
}
@@ -727,6 +1612,8 @@ int perf_bpf_filter__pin(void)
err_close:
close(dir_fd);
+ if (ret < 0)
+ unpin_filter_object(skel, path);
err:
free(path);
@@ -740,13 +1627,20 @@ int perf_bpf_filter__unpin(void)
char *path = NULL;
int ret = -1;
- skel = sample_filter_bpf__open_and_load();
+ skel = sample_filter_bpf__open();
if (!skel) {
ret = -errno;
pr_err("Failed to open perf sample-filter BPF skeleton\n");
goto err;
}
+ configure_lock_progs(skel, NULL, 0, false, false);
+ if (sample_filter_bpf__load(skel) < 0) {
+ ret = -errno;
+ pr_err("Failed to load perf sample-filter BPF skeleton\n");
+ goto err;
+ }
+
if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(),
PERF_BPF_FILTER_PIN_PATH) < 0) {
ret = -errno;
@@ -754,7 +1648,7 @@ int perf_bpf_filter__unpin(void)
goto err;
}
- ret = bpf_object__unpin(skel->obj, path);
+ ret = unpin_filter_object(skel, path);
err:
free(path);
diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h
index 818c554b91b2..fda95ff226b3 100644
--- a/tools/perf/util/bpf-filter.h
+++ b/tools/perf/util/bpf-filter.h
@@ -14,6 +14,13 @@ struct perf_bpf_filter_expr {
int part;
enum perf_bpf_filter_term term;
unsigned long val;
+ /*
+ * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions
+ * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R").
+ * Resolved into `entry.part` and `entry.target` during
+ * perf_bpf_filter__prepare().
+ */
+ char *lock_name;
};
struct evsel;
@@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
int part,
enum perf_bpf_filter_op op,
unsigned long val);
+struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
+ int mode,
+ enum perf_bpf_filter_op op,
+ unsigned long val);
+void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr);
+int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
int perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target);
int perf_bpf_filter__destroy(struct evsel *evsel);
diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l
index 1be9df6550fc..b4022110f1e5 100644
--- a/tools/perf/util/bpf-filter.l
+++ b/tools/perf/util/bpf-filter.l
@@ -1,9 +1,12 @@
%option prefix="perf_bpf_filter_"
%option noyywrap
+%x LOCK_ARG
+
%{
#include <stdio.h>
#include <stdlib.h>
+#include <string.h>
#include <linux/perf_event.h>
#include "bpf-filter.h"
@@ -76,7 +79,7 @@ static int path_or_error(void)
num_dec [0-9]+
num_hex 0[Xx][0-9a-fA-F]+
space [ \t]+
-path [^ \t\n,]+
+path [^ \t\n,()!=>&|<]+
ident [_a-zA-Z][_a-zA-Z0-9]+
%%
@@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); }
uid { return sample(PBF_TERM_UID); }
gid { return sample(PBF_TERM_GID); }
cgroup { return sample_path(PBF_TERM_CGROUP); }
-
+lock_held[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY;
+ return BFT_LOCK_HELD;
+ }
+(lock_held_read|lock_held_shared)[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ;
+ return BFT_LOCK_HELD;
+ }
+(lock_held_write|lock_held_exclusive)[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE;
+ return BFT_LOCK_HELD;
+ }
+
+<LOCK_ARG>{space} { }
+<LOCK_ARG>[^ \t\n()]+ {
+ perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text);
+ return BFT_LOCK_NAME;
+ }
+<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; }
+<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; }
+
+"!" { return '!'; }
"==" { return operator(PBF_OP_EQ); }
"!=" { return operator(PBF_OP_NEQ); }
">" { return operator(PBF_OP_GT); }
diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y
index 5a79a8e7a45b..6ee6edef13f7 100644
--- a/tools/perf/util/bpf-filter.y
+++ b/tools/perf/util/bpf-filter.y
@@ -11,6 +11,7 @@
#include <string.h>
#include <linux/compiler.h>
#include <linux/list.h>
+#include <linux/zalloc.h>
#include "bpf-filter.h"
#include "cgroup.h"
@@ -31,6 +32,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
{
unsigned long num;
char *path;
+ char *lock_name;
+ struct {
+ char *name;
+ int mode;
+ } lock;
struct {
enum perf_bpf_filter_term term;
int part;
@@ -40,12 +46,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
}
%token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH
+%token BFT_LOCK_HELD BFT_LOCK_NAME
%type <expr> filter_term filter_expr
-%destructor { free ($$); } <expr>
+%destructor { perf_bpf_filter_expr__delete($$); } <expr>
+%destructor { free($$); } <lock_name>
+%destructor { free($$.name); } <lock>
%type <sample> BFT_SAMPLE BFT_SAMPLE_PATH
%type <op> BFT_OP
-%type <num> BFT_NUM
+%type <num> BFT_NUM BFT_LOCK_HELD
%type <path> BFT_PATH
+%type <lock_name> BFT_LOCK_NAME
+%type <lock> lock_term
%%
@@ -105,5 +116,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH
$$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id);
cgroup__put(cgrp);
}
+|
+lock_term
+{
+ $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1);
+ zfree(&$1.name);
+ if (!$$)
+ YYERROR;
+}
+|
+'!' lock_term
+{
+ $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0);
+ zfree(&$2.name);
+ if (!$$)
+ YYERROR;
+}
+|
+lock_term BFT_OP BFT_NUM
+{
+ $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3);
+ zfree(&$1.name);
+ if (!$$)
+ YYERROR;
+}
+
+lock_term:
+BFT_LOCK_HELD BFT_LOCK_NAME ')'
+{
+ $$.name = $2;
+ $$.mode = $1;
+}
%%
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index 683fec85e71e..cd711d1d04b6 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -53,14 +53,73 @@ enum perf_bpf_filter_term {
/* Terms computed from BPF helpers. */
PBF_TERM_UID,
PBF_TERM_GID,
+ /*
+ * Kernel lock held predicate (`lock_held(...)`). Evaluates to 1 if the
+ * specified kernel lock is currently held by the running task/CPU, or
+ * 0 otherwise. The lock specification kind is stored in `entry.part`
+ * (`enum perf_bpf_lock_target_kind`) and the resolved target address,
+ * class ID, or type flag encoding is stored in `entry.target`.
+ */
+ PBF_TERM_LOCK_HELD,
};
+/*
+ * Lock target categories for PBF_TERM_LOCK_HELD filters.
+ * Stored in `perf_bpf_filter_entry.part`.
+ */
+enum perf_bpf_lock_target_kind {
+ PBF_LOCK_NONE = 0,
+ /*
+ * Specific global or static lock address (resolved from kallsyms or
+ * given as a hex address). `entry.target` holds the kernel address.
+ */
+ PBF_LOCK_ADDR,
+ /*
+ * Dynamic lock allocated inside a slab cache (`&slab_name`).
+ * `entry.target` holds the kernel `struct kmem_cache *` address.
+ */
+ PBF_LOCK_SLAB,
+ /*
+ * Per-process `current->mm->mmap_lock` (or `mmap_sem`).
+ */
+ PBF_LOCK_MMAP,
+ /*
+ * Per-process `current->sighand->siglock`.
+ */
+ PBF_LOCK_SIGHAND,
+ /*
+ * Semi-global lock class (`rq_lock` or `zone_lock`).
+ * `entry.target` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or
+ * LOCK_CLASS_ZONE_LOCK).
+ */
+ PBF_LOCK_CLASS,
+ /*
+ * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.).
+ * `entry.target` packs up to two `LCB_F_*` flag combinations in the
+ * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
+ */
+ PBF_LOCK_TYPE,
+};
+
+/*
+ * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD:
+ * Bits 0..3: `enum perf_bpf_lock_target_kind`
+ * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding)
+ * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding)
+ */
+#define PBF_LOCK_KIND_MASK 0x0fU
+#define PBF_LOCK_MODE_READ (1U << 4)
+#define PBF_LOCK_MODE_WRITE (1U << 5)
+#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE)
+#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY
+
/* BPF map entry for filtering */
struct perf_bpf_filter_entry {
enum perf_bpf_filter_op op;
__u32 part; /* sub-sample type info when it has multiple values */
enum perf_bpf_filter_term term;
__u64 value;
+ __u64 target; /* lock address, kmem_cache *, class, or type flags */
};
struct idx_hash_key {
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index e5666d4c1722..fbd2e51815d7 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -5,8 +5,47 @@
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
+#include "lock_data.h"
#include "sample-filter.h"
+/*
+ * Key for the `held_locks` BPF hash map tracking currently held kernel locks.
+ *
+ * - `addr_or_id`:
+ * - PBF_LOCK_ADDR: kernel virtual address of the lock instance
+ * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK)
+ * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address
+ * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags
+ * - `owner_id`:
+ * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`),
+ * since the holder may sleep and migrate across CPUs while holding the lock.
+ * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks
+ * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption
+ * is disabled on the holding CPU and spinlocks such as `rq_lock` may be
+ * acquired by one task and released by the next task across `context_switch()`.
+ * - `kind`: `enum perf_bpf_lock_target_kind`.
+ */
+struct held_lock_key {
+ __u64 addr_or_id;
+ __u32 owner_id;
+ __u32 kind;
+};
+
+/*
+ * Value for `held_locks` entries.
+ * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`)
+ * and exclusive/write acquisitions (including nested class/slab/type
+ * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of
+ * the lock so release tracepoints that do not carry flags (such as
+ * `lock:contended_release`) can decrement matching class/slab/type entries.
+ */
+struct held_lock_val {
+ __u32 read_count;
+ __u32 write_count;
+ __u32 flags;
+ __u32 pad;
+};
+
/* BPF map that will be filled by user space */
struct filters {
__uint(type, BPF_MAP_TYPE_HASH);
@@ -43,10 +82,198 @@ struct lost_count {
__uint(max_entries, 1);
} dropped SEC(".maps");
+/* Currently held kernel locks tracked by fentry/fexit and lock tracepoints. */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, struct held_lock_key);
+ __type(value, struct held_lock_val);
+ __uint(max_entries, 1);
+} held_locks SEC(".maps");
+
+/* Target lock addresses for PBF_LOCK_ADDR filters. */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, __u64);
+ __type(value, __u8);
+ __uint(max_entries, 1);
+} lock_addr_filter SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u32));
+ __uint(max_entries, 1);
+} lock_syms SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u8));
+ __uint(max_entries, 1);
+} slab_filter SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(struct slab_cache_data));
+ __uint(max_entries, 1);
+} slab_caches SEC(".maps");
+
volatile const int use_idx_hash;
+const volatile __u64 contig_page_data_addr;
+const volatile __u64 node_data_addr;
+const volatile int nr_nodes;
+const volatile int sizeof_zone;
+
+__u32 slab_cache_id;
+
+/* Control flags set by userspace when lock_held(...) filters are active. */
+int lock_enabled;
+int has_lock_addr;
+int has_lock_slab;
+int has_lock_class;
+int has_lock_type;
+int has_mmap_lock;
+int has_siglock;
+__u8 lock_type_filter[LCB_F_TYPE_MAX];
+
void *bpf_cast_to_kern_ctx(void *) __ksym;
+/*
+ * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`.
+ *
+ * For locks that maintain a task_struct pointer in their `owner` field
+ * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`),
+ * `perf_sample_filter` can inspect the lock structure directly in kernel memory
+ * and compare the owner against `current` (`bpf_get_current_task()`).
+ *
+ * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear
+ * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and
+ * spinlocks are checked via the `held_locks` BPF map instead.
+ */
+static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr)
+{
+ if (!curr)
+ return false;
+
+ return (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr ||
+ (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr ||
+ (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr;
+}
+
+static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode)
+{
+ if (!val)
+ return false;
+ if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0)
+ return true;
+ if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0)
+ return true;
+ return false;
+}
+
+static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind,
+ __u32 mode, __u32 pid, __u32 cpu_id)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = pid,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+
+ if (pid) {
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+ }
+
+ key.owner_id = cpu_id;
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+
+ return 0;
+}
+
+static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode,
+ __u32 pid, __u32 cpu_id)
+{
+ __u32 flag1 = target & 0xff;
+ __u32 flag2 = (target >> 8) & 0xff;
+
+ if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ return 0;
+}
+
+static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry)
+{
+ __u64 curr;
+ __u32 pid;
+ __u32 cpu_id;
+ __u32 kind;
+ __u32 mode;
+ __u64 lock_addr;
+
+ /*
+ * Pinned filters (use_idx_hash != 0) share a world-writable filters
+ * map and do not support lock_held(). Reject evaluation when pinned
+ * or when lock_enabled is not set so unprivileged users cannot probe
+ * arbitrary kernel addresses via entry->target.
+ */
+ if (use_idx_hash || !lock_enabled)
+ return 0;
+
+ curr = bpf_get_current_task();
+ pid = (__u32)bpf_get_current_pid_tgid();
+ cpu_id = 0x80000000U | bpf_get_smp_processor_id();
+ kind = entry->part & PBF_LOCK_KIND_MASK;
+ mode = entry->part & PBF_LOCK_MODE_MASK;
+
+ if (!mode)
+ mode = PBF_LOCK_MODE_ANY;
+
+ switch (kind) {
+ case PBF_LOCK_ADDR:
+ lock_addr = entry->target;
+ if ((mode & PBF_LOCK_MODE_WRITE) &&
+ is_direct_lock_owner(lock_addr, curr))
+ return 1;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_MMAP:
+ lock_addr = get_current_mmap_lock(curr);
+ if (!lock_addr)
+ return 0;
+ if ((mode & PBF_LOCK_MODE_WRITE) &&
+ (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr)
+ return 1;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_SIGHAND:
+ lock_addr = get_current_siglock(curr);
+ if (!lock_addr)
+ return 0;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_CLASS:
+ return is_lock_in_map(entry->target, PBF_LOCK_CLASS, mode, pid, cpu_id);
+
+ case PBF_LOCK_SLAB:
+ return is_lock_in_map(entry->target, PBF_LOCK_SLAB, mode, pid, cpu_id);
+
+ case PBF_LOCK_TYPE:
+ return is_lock_type_in_map(entry->target, mode, pid, cpu_id);
+
+ default:
+ return 0;
+ }
+}
+
/* new kernel perf_sample_data definition */
struct perf_sample_data___new {
__u64 sample_flags;
@@ -175,6 +402,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
return bpf_get_current_uid_gid() & 0xFFFFFFFF;
case PBF_TERM_GID:
return bpf_get_current_uid_gid() >> 32;
+ case PBF_TERM_LOCK_HELD:
+ return check_lock_held(entry);
case PBF_TERM_NONE:
case __PBF_UNUSED_TERM4:
case __PBF_UNUSED_TERM5:
@@ -295,4 +524,695 @@ int perf_sample_filter(void *ctx)
return 0;
}
+static __always_inline __u32 get_lock_owner_id(__u32 flags)
+{
+ __u32 owner_id;
+
+ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN)
+ return 0x80000000U | bpf_get_smp_processor_id();
+
+ owner_id = (__u32)bpf_get_current_pid_tgid();
+ if (!owner_id)
+ owner_id = 0x80000000U | bpf_get_smp_processor_id();
+ return owner_id;
+}
+
+static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id,
+ __u32 kind, __u32 flags, bool acquired)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = owner_id,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (acquired) {
+ if (val) {
+ if (is_read)
+ __sync_fetch_and_add(&val->read_count, 1);
+ else
+ __sync_fetch_and_add(&val->write_count, 1);
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags,
+ };
+
+ bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY);
+ }
+ } else if (val) {
+ if (is_read) {
+ if (val->read_count <= 1 && val->write_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->read_count > 0)
+ __sync_fetch_and_add(&val->read_count, -1);
+ } else {
+ if (val->write_count <= 1 && val->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->write_count > 0)
+ __sync_fetch_and_add(&val->write_count, -1);
+ }
+ }
+}
+
+static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id)
+{
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ bool matched = false;
+ bool is_read = (flags & LCB_F_READ) != 0;
+ __u32 class_id = 0;
+ __u64 kmem_cache_addr = 0;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool match_type = false;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
+ matched = true;
+
+ if (has_mmap_lock || has_siglock) {
+ __u32 ltype = check_lock_type(lock_addr, flags);
+
+ if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
+ (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
+ matched = true;
+ }
+
+ if (has_lock_class && flags == LCB_F_SPIN) {
+ __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
+
+ if (cid) {
+ class_id = *cid;
+ matched = true;
+ }
+ }
+
+ if (has_lock_slab && bpf_get_kmem_cache) {
+ kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
+ if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr))
+ matched = true;
+ else
+ kmem_cache_addr = 0;
+ }
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) {
+ match_type = true;
+ matched = true;
+ }
+
+ if (!matched)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (existing) {
+ /* Non-reader locks are not recursive; avoid double-counting */
+ if (!is_read && existing->write_count > 0)
+ return 0;
+ if (is_read)
+ __sync_fetch_and_add(&existing->read_count, 1);
+ else
+ __sync_fetch_and_add(&existing->write_count, 1);
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags,
+ };
+
+ if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0)
+ return 0;
+ }
+
+ if (class_id)
+ update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, flags, true);
+ if (kmem_cache_addr) {
+ update_held_lock_count((__u64)kmem_cache_addr, owner_id,
+ PBF_LOCK_SLAB, flags, true);
+ }
+ if (match_type)
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, true);
+
+ return 0;
+}
+
+static __always_inline int record_lock_acquired(void *ctx, __u32 flags)
+{
+ return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags));
+}
+
+static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags)
+{
+ __u32 owner_id = get_lock_owner_id(flags);
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (!existing)
+ return 0;
+
+ if (is_read) {
+ if (existing->read_count <= 1 && existing->write_count == 0)
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ else if (existing->read_count > 0)
+ __sync_fetch_and_add(&existing->read_count, -1);
+ } else {
+ if (existing->write_count == 0 && existing->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ else if (existing->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ else
+ existing->write_count = 0;
+ }
+
+ if (has_lock_class && flags == LCB_F_SPIN) {
+ __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
+
+ if (cid)
+ update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false);
+ }
+
+ if (has_lock_slab && bpf_get_kmem_cache) {
+ __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
+
+ if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) {
+ update_held_lock_count(kmem_cache_addr, owner_id,
+ PBF_LOCK_SLAB, flags, false);
+ }
+ }
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag])
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false);
+
+ return 0;
+}
+
+static __always_inline int record_lock_released(void *ctx, __u32 flags)
+{
+ return record_lock_released_addr(get_lock_addr(ctx), flags);
+}
+
+/* Mutex acquire/release hooks */
+SEC("fexit/mutex_lock")
+int BPF_PROG(fexit_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_nested")
+int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io")
+int BPF_PROG(fexit_mutex_lock_io, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io_nested")
+int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_interruptible")
+int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_interruptible_nested")
+int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_killable")
+int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_trylock")
+int BPF_PROG(fexit_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fentry/mutex_unlock")
+int BPF_PROG(fentry_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_MUTEX);
+}
+
+/* RW semaphore acquire/release hooks */
+SEC("fexit/down_read")
+int BPF_PROG(fexit_down_read, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_nested")
+int BPF_PROG(fexit_down_read_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_interruptible")
+int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable")
+int BPF_PROG(fexit_down_read_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable_nested")
+int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_trylock")
+int BPF_PROG(fexit_down_read_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_write")
+int BPF_PROG(fexit_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_nested")
+int BPF_PROG(fexit_down_write_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_killable")
+int BPF_PROG(fexit_down_write_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_killable_nested")
+int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_trylock")
+int BPF_PROG(fexit_down_write_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/downgrade_write")
+int BPF_PROG(fexit_downgrade_write, void *sem)
+{
+ record_lock_acquired(ctx, LCB_F_READ);
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+SEC("fentry/up_read")
+int BPF_PROG(fentry_up_read, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_READ);
+}
+
+SEC("fentry/up_write")
+int BPF_PROG(fentry_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+/* Spinlock acquire/release hooks */
+SEC("fexit/_raw_spin_lock")
+int BPF_PROG(fexit_raw_spin_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_nested")
+int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq")
+int BPF_PROG(fexit_raw_spin_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq_disable")
+int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave")
+int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave_nested")
+int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_bh")
+int BPF_PROG(fexit_raw_spin_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_trylock")
+int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fexit/_raw_spin_trylock_bh")
+int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fentry/_raw_spin_unlock")
+int BPF_PROG(fentry_raw_spin_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq")
+int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq_enable")
+int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irqrestore")
+int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_bh")
+int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+/* RWlock acquire/release hooks */
+SEC("fexit/_raw_read_lock")
+int BPF_PROG(fexit_raw_read_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_lock_irq")
+int BPF_PROG(fexit_raw_read_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_lock_irqsave")
+int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_lock_bh")
+int BPF_PROG(fexit_raw_read_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_trylock")
+int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+ return 0;
+}
+
+SEC("fentry/_raw_read_unlock")
+int BPF_PROG(fentry_raw_read_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irq")
+int BPF_PROG(fentry_raw_read_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irqrestore")
+int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_bh")
+int BPF_PROG(fentry_raw_read_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_write_lock")
+int BPF_PROG(fexit_raw_write_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_nested")
+int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irq")
+int BPF_PROG(fexit_raw_write_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irqsave")
+int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_bh")
+int BPF_PROG(fexit_raw_write_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_trylock")
+int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fentry/_raw_write_unlock")
+int BPF_PROG(fentry_raw_write_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irq")
+int BPF_PROG(fentry_raw_write_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irqrestore")
+int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_bh")
+int BPF_PROG(fentry_raw_write_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+/* RT-Mutex acquire/release hooks */
+SEC("fexit/rt_mutex_lock")
+int BPF_PROG(fexit_rt_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_nested")
+int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_interruptible")
+int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_lock_killable")
+int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_trylock")
+int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fentry/rt_mutex_unlock")
+int BPF_PROG(fentry_rt_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_RT);
+}
+
+/* Semaphore acquire/release hooks */
+SEC("fexit/down")
+int BPF_PROG(fexit_down, void *sem)
+{
+ return record_lock_acquired(ctx, 0);
+}
+
+SEC("fexit/down_interruptible")
+int BPF_PROG(fexit_down_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_killable")
+int BPF_PROG(fexit_down_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_timeout")
+int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_trylock")
+int BPF_PROG(fexit_down_trylock, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fentry/up")
+int BPF_PROG(fentry_up, void *sem)
+{
+ return record_lock_released(ctx, 0);
+}
+
+/* Percpu-rwsem write acquire/release hooks */
+SEC("fexit/percpu_down_write")
+int BPF_PROG(fexit_percpu_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
+SEC("fentry/percpu_up_write")
+int BPF_PROG(fentry_percpu_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
+SEC("raw_tp/bpf_test_finish")
+int BPF_PROG(collect_lock_syms)
+{
+ return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
+}
+
+SEC("iter/kmem_cache")
+int slab_cache_iter(void *ctx)
+{
+ return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
+}
+
char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index 2f3317ce17f0..1fcb424b75c0 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist,
/* Will a tracepoint filter work for str or should a BPF filter be used? */
static bool is_possible_tp_filter(const char *str)
{
- return strstr(str, "uid") == NULL;
+ return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL &&
+ strstr(str, "lock_held") == NULL;
}
static int set_filter(struct evsel *evsel, const void *arg)
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v1 3/5] perf test: Add tests for lock_held() BPF sample filter
2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers
@ 2026-10-17 5:45 ` Ian Rogers
2026-10-17 5:49 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
` (2 subsequent siblings)
5 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Namhyung Kim
Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, rcu
Add unit tests in parse-events.c for parsing lock_held(),
lock_held_read(), and lock_held_exclusive() BPF filter expressions
(standalone predicates, negated predicates, and compound group
expressions) and extend record_bpf_filter.sh with lock_held(),
lock_held_read()/lock_held_write(), and ':R'/':W' recording tests.
Example output:
# perf test -v 'Parsing of BPF sample filters'
5: Parse event definition strings :
5.7: Parsing of BPF sample filters : Ok
# perf test -v 'perf record sample filtering (by BPF) tests'
112: perf record sample filtering (by BPF) tests :
Checking BPF-filter privilege
Basic bpf-filter test
Basic bpf-filter test [Success]
Failing bpf-filter test
Failing bpf-filter test [Success]
Group bpf-filter test
Group bpf-filter test [Success]
Multiple bpf-filter test
Multiple bpf-filter test [Success]
Cgroup bpf-filter test
Cgroup bpf-filter test [Success]
Lock-held bpf-filter test
Lock-held bpf-filter test [Success]
112: perf record sample filtering (by BPF) tests : Ok
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/tests/parse-events.c | 108 ++++++++++++++++++++
tools/perf/tests/shell/record_bpf_filter.sh | 99 ++++++++++++++++++
2 files changed, 207 insertions(+)
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 5f6f95c8a5b4..4cf9192762f2 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -8,9 +8,11 @@
#include "debug.h"
#include "pmu.h"
#include "pmus.h"
+#include "bpf-filter.h"
#include "strbuf.h"
#include <dirent.h>
#include <errno.h>
+#include <string.h>
#include "fncache.h"
#include <sys/types.h>
#include <sys/stat.h>
@@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused,
return ret;
}
+#ifdef HAVE_BPF_SKEL
+static void clear_bpf_filter_exprs(struct list_head *expr_head)
+{
+ struct perf_bpf_filter_expr *expr, *tmp;
+
+ list_for_each_entry_safe(expr, tmp, expr_head, list) {
+ list_del(&expr->list);
+ perf_bpf_filter_expr__delete(expr);
+ }
+}
+
+static int do_test_bpf_filters(struct list_head *expr_head)
+{
+ struct perf_bpf_filter_expr *expr, *group;
+ const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100";
+
+ /* Simple lock_held(&cgroup_mutex) */
+ TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)",
+ __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is &cgroup_mutex",
+ expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /* Negated !lock_held(cgroup_mutex) */
+ TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)",
+ __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 0", expr->val == 0);
+ TEST_ASSERT_VAL("lock_name is cgroup_mutex",
+ expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /* Read/shared and write/exclusive predicates */
+ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)",
+ __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is &mmap_lock",
+ expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock"));
+ clear_bpf_filter_exprs(expr_head);
+
+ TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)",
+ __perf_bpf_filter__parse(expr_head,
+ "!lock_held_exclusive(mmap_lock)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 0", expr->val == 0);
+ TEST_ASSERT_VAL("lock_name is mmap_lock",
+ expr->lock_name && !strcmp(expr->lock_name, "mmap_lock"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /*
+ * Comparison and compound group:
+ * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100
+ */
+ TEST_ASSERT_VAL("parse compound lock_held filter",
+ __perf_bpf_filter__parse(expr_head, compound) == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN);
+ TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups));
+ group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("first group member lock_name",
+ group->lock_name && !strcmp(group->lock_name, "rwsem:R"));
+ TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1);
+ group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("second group member lock_name",
+ group->lock_name && !strcmp(group->lock_name, "&dentry"));
+ TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0);
+ expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100);
+
+ return TEST_OK;
+}
+#endif
+
+static int test__bpf_filters(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
+{
+#ifdef HAVE_BPF_SKEL
+ struct list_head expr_head = LIST_HEAD_INIT(expr_head);
+ int ret = do_test_bpf_filters(&expr_head);
+
+ clear_bpf_filter_exprs(&expr_head);
+ return ret;
+#else
+ return TEST_SKIP;
+#endif
+}
+
static struct test_case tests__parse_events[] = {
TEST_CASE_REASON("Test event parsing",
events2,
@@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = {
"no aliases in sysfs"),
TEST_CASE("Parsing of aliased events", pmu_events_alias2),
TEST_CASE("Parsing of terms (event modifiers)", terms2),
+ TEST_CASE("Parsing of BPF sample filters", bpf_filters),
{ .name = NULL, }
};
diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh
index 383574cb3bd3..7ec75d969bd6 100755
--- a/tools/perf/tests/shell/record_bpf_filter.sh
+++ b/tools/perf/tests/shell/record_bpf_filter.sh
@@ -176,6 +176,101 @@ test_bpf_filter_cgroup() {
echo "Cgroup bpf-filter test [Success]"
}
+test_bpf_filter_lock_held() {
+ echo "Lock-held bpf-filter test"
+ lock_sym="cgroup_mutex"
+
+ if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms
+ then
+ lock_sym="slab_mutex"
+ fi
+
+ if ! perf record -e task-clock --filter "lock_held(&${lock_sym})" \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if [ "$(id -u)" != 0 ]
+ then
+ echo "Lock-held bpf-filter test [Skipped permission]"
+ return
+ fi
+ if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms
+ then
+ echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]"
+ return
+ fi
+ echo "Lock-held bpf-filter test [Failed &${lock_sym}]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \
+ -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed mmap_lock]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(rq_lock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed rq_lock]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(spinlock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock type]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock \
+ --filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read/write(&mmap_lock)]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(&mmap_lock:R) || lock_held(&mmap_lock:W)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed &mmap_lock:R/:W]"
+ err=1
+ return
+ fi
+
+ if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \
+ -o /dev/null true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]"
+ err=1
+ return
+ fi
+
+ if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \
+ -o /dev/null true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]"
+ err=1
+ return
+ fi
+
+ echo "Lock-held bpf-filter test [Success]"
+}
+
test_bpf_filter_priv
if [ $err = 0 ]; then
@@ -198,5 +293,9 @@ if [ $err = 0 ]; then
test_bpf_filter_cgroup
fi
+if [ $err = 0 ]; then
+ test_bpf_filter_lock_held
+fi
+
cleanup
exit $err
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (2 preceding siblings ...)
2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers
@ 2026-10-17 5:45 ` Ian Rogers
2026-10-17 5:56 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
5 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Namhyung Kim
Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, rcu
Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks
Trace RCU, and SRCU locks:
- 'rcu' (supports both read/shared and write/exclusive modes):
- Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)',
'lock_held(rcu_read_lock)'): directly inspects
current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on
CONFIG_PREEMPT_RCU kernels), accounting for the baseline
rcu_read_lock() nesting acquired by the perf overflow handler itself
(bpf_overflow_handler() and do_perf_sw_event()) with zero extra
fentry/fexit tracing hooks.
- Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'):
attaches fentry/fexit hooks to synchronize_rcu(),
synchronize_rcu_expedited(), synchronize_rcu_tasks(), and
rcu_barrier() to track grace-period and barrier waits.
- 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both
read/shared and write/exclusive modes):
- Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches
fexit/__srcu_read_lock and fentry/__srcu_read_unlock.
- Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'):
attaches fentry/fexit on synchronize_srcu(),
synchronize_srcu_expedited(), and srcu_barrier().
- 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports
read/shared mode only): directly inspects
current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels
with zero extra tracing hooks.
Example usage:
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ]
# perf report --stdio --no-children
# Overhead Command Shared Object Symbol
# ........ ....... ................. ........................
#
14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu
|
---__d_lookup_rcu
lookup_fast
walk_component
link_path_walk
path_lookupat
filename_lookup
vfs_statx
__do_sys_newstat
do_syscall_64
entry_SYSCALL_64_after_hwframe
# perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1
# perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 42 +++--
tools/perf/tests/parse-events.c | 13 ++
tools/perf/tests/shell/record_bpf_filter.sh | 20 +++
tools/perf/util/bpf-filter.c | 158 +++++++++++++++++--
tools/perf/util/bpf_skel/lock_data.h | 44 +++++-
tools/perf/util/bpf_skel/sample-filter.h | 12 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 135 +++++++++++++++-
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 +
tools/perf/util/lock-contention.c | 4 +
tools/perf/util/lock-contention.h | 4 +-
10 files changed, 402 insertions(+), 32 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 246ae085744d..c8ccfb0e0ce6 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -219,28 +219,38 @@ OPTIONS
'&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'),
a hexadecimal lock address, a slab cache name prefixed with '&'
(e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known
- lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock
- type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
- 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
- 'semaphore'). Both read/shared and write/exclusive tracking are
- supported for 'rwsem' (including 'mmap_lock') and 'rwlock';
+ lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock', 'rcu',
+ 'rcu-trace'), or a lock type ('mutex', 'spinlock', 'rwsem', 'rwsem:R',
+ 'rwsem:W', 'rwlock', 'rwlock:R', 'rwlock:W', 'rt-mutex',
+ 'percpu-rwsem', 'pcpu-sem:W', 'srcu', 'srcu:R', 'srcu:W', 'rcu:R',
+ 'rcu:W', 'semaphore'). Both read/shared and write/exclusive tracking
+ are supported for 'rwsem' (including 'mmap_lock'), 'rwlock', 'rcu',
+ and 'srcu' (where read/shared mode matches 'rcu_read_lock()' /
+ 'srcu_read_lock()' critical sections and write/exclusive mode matches
+ grace-period waits 'synchronize_rcu*()', 'synchronize_srcu*()',
+ 'rcu_barrier()', and 'srcu_barrier()'); 'rcu-trace'
+ ('rcu_read_lock_trace()') supports read/shared tracking only; and
'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'),
'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()')
support write/exclusive tracking only.
Note that 'lock_held(<lock>)' introduces overhead outside of sample
overflow handling: in addition to checking the lock's owner field on
- each sample where possible (such as for mutexes, rt-mutexes, and
- write-locked rwsems), it attaches BPF fentry/fexit programs to kernel
- lock acquire and release functions to track held locks (such as
- spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a
- BPF map. These BPF hooks run on every kernel lock acquisition and
- release for the attached lock family, even when no perf sample is
- being taken. To minimize this overhead, only the hooks for the
- relevant lock family and mode are attached when known (for example,
- 'lock_held_read(&mmap_lock)' only hooks rwsem read functions,
- 'lock_held_write(&mmap_lock)' only hooks rwsem write functions,
- '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks
+ each sample where possible (such as for mutexes, rt-mutexes,
+ write-locked rwsems, and 'rcu:R' / 'rcu-trace' which inspect
+ 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting'
+ directly with no extra tracing hooks), it attaches BPF fentry/fexit
+ programs to kernel lock acquire and release functions to track held
+ locks (such as spinlocks, rwlocks, reader-locked rwsems, SRCU locks,
+ RCU synchronize waits, and slab/type filters) in a BPF map. These
+ BPF hooks run on every kernel lock acquisition and release for the
+ attached lock family, even when no perf sample is being taken. To
+ minimize this overhead, only the hooks for the relevant lock family
+ and mode are attached when known (for example, 'lock_held_read(rcu)'
+ and 'rcu-trace' attach no extra hooks, 'lock_held_read(&mmap_lock)'
+ only hooks rwsem read functions, 'lock_held_write(&mmap_lock)' only
+ hooks rwsem write functions, '&cgroup_mutex' only hooks mutex
+ functions, 'srcu' only hooks SRCU functions, and 'rq_lock' only hooks
spinlock functions), whereas slab filters ('&<slab>'), raw hex
addresses, or symbols without a recognizable lock-type suffix attach
hooks across all lock families (including high-frequency spinlocks).
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 4cf9192762f2..730466b7ebf6 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head)
expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex"));
clear_bpf_filter_exprs(expr_head);
+ /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */
+ TEST_ASSERT_VAL("parse lock_held(rcu)",
+ __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is rcu",
+ expr->lock_name && !strcmp(expr->lock_name, "rcu"));
+ clear_bpf_filter_exprs(expr_head);
+
/* Read/shared and write/exclusive predicates */
TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)",
__perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0);
diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh
index 7ec75d969bd6..39ba628022db 100755
--- a/tools/perf/tests/shell/record_bpf_filter.sh
+++ b/tools/perf/tests/shell/record_bpf_filter.sh
@@ -235,6 +235,26 @@ test_bpf_filter_lock_held() {
return
fi
+ if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed rcu:W/srcu]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \
+ -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]"
+ err=1
+ return
+ fi
+ fi
+
if ! perf record -e task-clock \
--filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \
-o "${perfdata}" true 2> /dev/null
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 35d6f7c63868..d4d7f7a50f18 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -190,11 +190,19 @@ enum lock_prog_group {
LOCK_PROG_RTMUTEX = 1U << 6,
LOCK_PROG_SEM = 1U << 7,
LOCK_PROG_PERCPU_RWSEM = 1U << 8,
- LOCK_PROG_ALL = 0x1ffU,
+ LOCK_PROG_SRCU_READ = 1U << 9,
+ LOCK_PROG_SRCU_WRITE = 1U << 10,
+ LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE,
+ LOCK_PROG_RCU_WRITE = 1U << 11,
+ LOCK_PROG_ALL = 0xfffU,
};
static unsigned int lock_flags_to_prog_group(unsigned int flags)
{
+ if (flags & LCB_F_RCU)
+ return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0;
+ if (flags & LCB_F_SRCU)
+ return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE;
if (flags & LCB_F_MUTEX)
return LOCK_PROG_MUTEX;
if (flags & LCB_F_RT)
@@ -308,6 +316,32 @@ static const struct lock_prog_desc lock_progs[] = {
/* Percpu-RWsem */
LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+ /* SRCU */
+ LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe",
+ LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe",
+ LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited",
+ LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited",
+ LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE),
+ /* RCU grace-period synchronize / barrier */
+ LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited",
+ LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited",
+ LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE),
};
#undef LOCK_FPROG
@@ -616,6 +650,17 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
*needs_lock_syms = true;
return;
}
+ if (!strcmp(base, "rcu")) {
+ if (mode & PBF_LOCK_MODE_WRITE)
+ *prog_groups |= LOCK_PROG_RCU_WRITE;
+ return;
+ }
+ if (!strcmp(base, "rcu_read_lock") ||
+ !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") ||
+ !strcmp(base, "rcu_read_lock_trace")) {
+ /* Inspected directly on task_struct; no tracing hooks needed. */
+ return;
+ }
if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, prog_groups))
return;
@@ -646,6 +691,11 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
*prog_groups |= LOCK_PROG_RWLOCK_READ;
if (mode & PBF_LOCK_MODE_WRITE)
*prog_groups |= LOCK_PROG_RWLOCK_WRITE;
+ } else if (strtailcmp(base, "srcu") == 0) {
+ if (mode & PBF_LOCK_MODE_READ)
+ *prog_groups |= LOCK_PROG_SRCU_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ *prog_groups |= LOCK_PROG_SRCU_WRITE;
} else {
*prog_groups |= LOCK_PROG_ALL;
}
@@ -729,16 +779,54 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
return false;
}
+static u64 rcu_min_depth(struct evsel *evsel)
+{
+ if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE &&
+ evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK &&
+ evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK)
+ return 2;
+ return 1;
+}
+
+static bool btf_has_task_field(struct btf *btf, const char *field_name)
+{
+ const struct btf_type *t;
+ const struct btf_member *m;
+ s32 id;
+ u16 i, vlen;
+
+ if (!btf)
+ return false;
+
+ id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT);
+ if (id < 0)
+ return false;
+
+ t = btf__type_by_id(btf, id);
+ if (!t)
+ return false;
+
+ m = btf_members(t);
+ vlen = btf_vlen(t);
+ for (i = 0; i < vlen; i++, m++) {
+ const char *name = btf__name_by_offset(btf, m->name_off);
+
+ if (name && !strcmp(name, field_name))
+ return true;
+ }
+ return false;
+}
+
/*
* Resolve a `lock_held(<name>)` filter expression into `entry->part`
* (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate
* the corresponding BPF filter maps and BSS control flags.
*
* Resolution order:
- * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
- * `rq_lock`, `zone_lock` (with or without leading '&').
+ * 1. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`,
+ * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` (with or without leading '&').
* 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`,
- * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`,
+ * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, `srcu`,
* `semaphore`, etc.
* 3. Hexadecimal lock address (when no leading '&').
* 4. Slab cache name (when prefixed with '&', e.g. `&dentry`).
@@ -746,8 +834,8 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
* e.g. `&cgroup_mutex` or `cgroup_mutex`).
* 6. Slab cache name without leading '&' fallback.
*/
-static int resolve_lock_expr(struct sample_filter_bpf *skel,
- struct perf_bpf_filter_expr *expr,
+static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct btf *btf, struct perf_bpf_filter_expr *expr,
struct perf_bpf_filter_entry *entry)
{
const char *name = expr->lock_name;
@@ -811,6 +899,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel,
return 0;
}
+ if (!strcmp(base, "rcu_read_lock")) {
+ if (mode == PBF_LOCK_MODE_WRITE) {
+ pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
+ return -EINVAL;
+ }
+ if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
+ pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ;
+ entry->target = rcu_min_depth(evsel);
+ return 0;
+ }
+
+ if (!strcmp(base, "rcu")) {
+ if ((mode & PBF_LOCK_MODE_READ) && btf &&
+ !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
+ pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU | mode;
+ entry->target = rcu_min_depth(evsel);
+ if ((mode & PBF_LOCK_MODE_WRITE) && skel)
+ skel->bss->has_rcu_lock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") ||
+ !strcmp(base, "rcu_read_lock_trace")) {
+ if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) {
+ pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
+ return -EINVAL;
+ }
+ if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) {
+ pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ;
+ entry->target = 0;
+ return 0;
+ }
+
if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
entry->part = PBF_LOCK_TYPE | mode;
entry->target = encoded_pair;
@@ -875,7 +1005,7 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel,
}
static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
- struct perf_bpf_filter_entry *entry)
+ struct btf *btf, struct perf_bpf_filter_entry *entry)
{
int i = 0, ret;
struct perf_bpf_filter_expr *expr;
@@ -893,7 +1023,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske
entry[i].value = expr->val;
entry[i].target = 0;
if (expr->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(skel, expr, &entry[i]);
+ ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]);
if (ret < 0)
return ret;
}
@@ -912,7 +1042,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske
entry[i].value = group->val;
entry[i].target = 0;
if (group->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(skel, group, &entry[i]);
+ ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]);
if (ret < 0)
return ret;
}
@@ -1185,7 +1315,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- ret = get_filter_entries(evsel, NULL, entry);
+ ret = get_filter_entries(evsel, NULL, NULL, entry);
if (ret < 0) {
pr_err("Failed to process filter entries\n");
goto err;
@@ -1247,6 +1377,12 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
init_numa_data(skel, btf);
}
+ if (prog_groups & LOCK_PROG_RCU_WRITE) {
+ u64 rcu_addr = 1ULL;
+
+ find_kernel_lock_sym("rcu_state", &rcu_addr);
+ skel->rodata->rcu_state_addr = rcu_addr;
+ }
ret = configure_lock_progs(skel, btf, prog_groups, needs_lock_syms,
needs_slab_iter);
if (ret < 0)
@@ -1264,7 +1400,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
if (has_lock_filter && needs_slab_iter)
run_slab_cache_iter(skel);
- ret = get_filter_entries(evsel, skel, entry);
+ ret = get_filter_entries(evsel, skel, btf, entry);
if (ret < 0) {
pr_err("Failed to process filter entries\n");
goto err;
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index fed445c9d8b4..e32ba230d5f5 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -39,6 +39,7 @@ struct contention_task_data {
*/
#define LCD_F_MMAP_LOCK (1U << 31)
#define LCD_F_SIGHAND_LOCK (1U << 30)
+#define LCD_F_RCU_LOCK (1U << 29)
#define LCB_F_SLAB_ID_SHIFT 16
#define LCB_F_SLAB_ID_START (1U << 16)
@@ -52,9 +53,11 @@ struct contention_task_data {
#define LCB_F_RT (1U << 3)
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+#define LCB_F_SRCU (1U << 6)
+#define LCB_F_RCU (1U << 7)
-#define LCB_F_TYPE_MAX (1U << 7)
-#define LCB_F_TYPE_MASK 0x0000007FU
+#define LCB_F_TYPE_MAX (1U << 8)
+#define LCB_F_TYPE_MASK 0x000000FFU
#define SLAB_NAME_MAX 28
@@ -233,6 +236,40 @@ static __always_inline __u64 get_current_siglock(__u64 curr)
return sh + bpf_core_field_offset(sighand->siglock);
}
+/*
+ * Check if `curr` is inside a preemptible RCU read-side critical section
+ * (`rcu_read_lock()`).
+ *
+ * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires
+ * `rcu_read_lock()` before invoking `perf_sample_filter` (and
+ * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for
+ * non-clock software events), `min_depth` is the baseline nesting count
+ * contributed by the perf event overflow handler itself (1 or 2).
+ */
+static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth)
+{
+ struct task_struct *task = (void *)curr;
+
+ if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting))
+ return false;
+
+ return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth;
+}
+
+/*
+ * Check if `curr` is inside a Tasks Trace RCU read-side critical section
+ * (`rcu_read_lock_trace()`).
+ */
+static __always_inline bool is_rcu_trace_lock_held(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+
+ if (!curr || !bpf_core_field_exists(task->trc_reader_nesting))
+ return false;
+
+ return BPF_CORE_READ(task, trc_reader_nesting) > 0;
+}
+
/*
* Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar.
*
@@ -265,6 +302,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags)
if (get_current_siglock(curr) == lock)
return LCD_F_SIGHAND_LOCK;
break;
+ case LCB_F_RCU:
+ case LCB_F_RCU | LCB_F_WRITE:
+ return LCD_F_RCU_LOCK;
default:
break;
}
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index cd711d1d04b6..37c108608bbc 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -99,6 +99,18 @@ enum perf_bpf_lock_target_kind {
* low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
*/
PBF_LOCK_TYPE,
+ /*
+ * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`).
+ * Inspected directly via `current->rcu_read_lock_nesting > entry.target`,
+ * where `entry.target` is the baseline RCU nesting depth added by the
+ * kernel perf overflow path.
+ */
+ PBF_LOCK_RCU,
+ /*
+ * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`).
+ * Inspected directly via `current->trc_reader_nesting > 0`.
+ */
+ PBF_LOCK_RCU_TRACE,
};
/*
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index fbd2e51815d7..22d9b01e5851 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -123,6 +123,7 @@ volatile const int use_idx_hash;
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
+const volatile __u64 rcu_state_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
@@ -136,6 +137,7 @@ int has_lock_class;
int has_lock_type;
int has_mmap_lock;
int has_siglock;
+int has_rcu_lock;
__u8 lock_type_filter[LCB_F_TYPE_MAX];
void *bpf_cast_to_kern_ctx(void *) __ksym;
@@ -269,6 +271,20 @@ static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry
case PBF_LOCK_TYPE:
return is_lock_type_in_map(entry->target, mode, pid, cpu_id);
+ case PBF_LOCK_RCU:
+ if ((mode & PBF_LOCK_MODE_READ) &&
+ is_rcu_lock_held(curr, (int)entry->target))
+ return 1;
+ if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr &&
+ is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id))
+ return 1;
+ return 0;
+
+ case PBF_LOCK_RCU_TRACE:
+ if ((mode & PBF_LOCK_MODE_READ) && is_rcu_trace_lock_held(curr))
+ return 1;
+ return 0;
+
default:
return 0;
}
@@ -600,11 +616,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla
if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
matched = true;
- if (has_mmap_lock || has_siglock) {
+ if (has_mmap_lock || has_siglock || has_rcu_lock) {
__u32 ltype = check_lock_type(lock_addr, flags);
if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
- (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
+ (has_siglock && ltype == LCD_F_SIGHAND_LOCK) ||
+ (has_rcu_lock && ltype == LCD_F_RCU_LOCK))
matched = true;
}
@@ -1202,6 +1219,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem)
return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
}
+/* SRCU acquire/release and synchronize hooks */
+SEC("fexit/__srcu_read_lock")
+int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/__srcu_read_unlock")
+int BPF_PROG(fentry_srcu_read_unlock, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fexit/__srcu_read_lock_nmisafe")
+int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/__srcu_read_unlock_nmisafe")
+int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/synchronize_srcu")
+int BPF_PROG(fentry_synchronize_srcu, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu")
+int BPF_PROG(fexit_synchronize_srcu, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_srcu_expedited")
+int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu_expedited")
+int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fentry/srcu_barrier")
+int BPF_PROG(fentry_srcu_barrier, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/srcu_barrier")
+int BPF_PROG(fexit_srcu_barrier, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+/* RCU grace-period synchronize / barrier hooks */
+SEC("fentry/synchronize_rcu")
+int BPF_PROG(fentry_synchronize_rcu)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu")
+int BPF_PROG(fexit_synchronize_rcu)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_rcu_expedited")
+int BPF_PROG(fentry_synchronize_rcu_expedited)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu_expedited")
+int BPF_PROG(fexit_synchronize_rcu_expedited)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_rcu_tasks")
+int BPF_PROG(fentry_synchronize_rcu_tasks)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu_tasks")
+int BPF_PROG(fexit_synchronize_rcu_tasks)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/rcu_barrier")
+int BPF_PROG(fentry_rcu_barrier)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/rcu_barrier")
+int BPF_PROG(fexit_rcu_barrier)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
SEC("raw_tp/bpf_test_finish")
int BPF_PROG(collect_lock_syms)
{
diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
index b75ddc6846bb..1fb91a829d54 100644
--- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
+++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
@@ -119,6 +119,8 @@ struct task_struct {
struct signal_struct *signal;
struct sighand_struct *sighand;
struct css_set *cgroups;
+ int rcu_read_lock_nesting;
+ int trc_reader_nesting;
} __attribute__((preserve_access_index));
struct trace_entry {
diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c
index a1bb78d6e29c..b45b075f3346 100644
--- a/tools/perf/util/lock-contention.c
+++ b/tools/perf/util/lock-contention.c
@@ -38,6 +38,10 @@ const struct perf_lock_type lock_type_table[] = {
{ LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
/* alias for optimistic spinning only */
{ LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
+ { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" },
+ { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" },
+ { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" },
+ { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" },
};
int parse_call_stack(const struct option *opt __maybe_unused, const char *str,
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index 68e9bd77e8a9..88b2c11244ad 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -128,6 +128,8 @@ struct thread_stat {
#define LCB_F_RT (1U << 3)
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+#define LCB_F_SRCU (1U << 6)
+#define LCB_F_RCU (1U << 7)
struct perf_lock_type {
unsigned int flags;
@@ -140,7 +142,7 @@ struct perf_lock_type {
const char *lock_name;
};
-extern const struct perf_lock_type lock_type_table[14];
+extern const struct perf_lock_type lock_type_table[18];
struct evlist;
struct machine;
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU and SRCU
2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (3 preceding siblings ...)
2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
@ 2026-10-17 5:45 ` Ian Rogers
2026-10-17 6:08 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
5 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Namhyung Kim
Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, rcu
Extend 'perf lock contention --use-bpf' to track RCU and SRCU
grace-period and barrier waits:
- Attach fentry/fexit programs to synchronize_rcu(),
synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(),
synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier()
when available in BTF.
- Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu',
'-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct
symbol names).
Example usage:
# perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1
total wait avg wait type caller
35.12 ms 8.78 ms rcu:W cleanup_net+0x284
7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114
# perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1
total wait avg wait address symbol
35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu)
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/tests/shell/lock_contention.sh | 14 ++
tools/perf/util/bpf_lock_contention.c | 167 ++++++++++++++--
.../perf/util/bpf_skel/lock_contention.bpf.c | 183 ++++++++++++++----
4 files changed, 322 insertions(+), 49 deletions(-)
diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt
index c17b3e318169..bfeb98c976e3 100644
--- a/tools/perf/Documentation/perf-lock.txt
+++ b/tools/perf/Documentation/perf-lock.txt
@@ -188,11 +188,14 @@ CONTENTION OPTIONS
Show lock contention only for given lock types (comma separated list).
Available values are:
semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W,
- rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem,
- pcpu-sem:R, pcpu-sem:W, mutex
+ rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem,
+ pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W
Note that RW-variant of locks have :R and :W suffix. Names without the
suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W.
+ For RCU and SRCU with --use-bpf, contention tracks grace-period and
+ barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(),
+ srcu_barrier()).
-L::
--lock-filter=<value>::
diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh
index 5df9a0cd48a3..c6d37c910d4b 100755
--- a/tools/perf/tests/shell/lock_contention.sh
+++ b/tools/perf/tests/shell/lock_contention.sh
@@ -181,6 +181,13 @@ test_type_filter()
err=1
exit
fi
+
+ perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result}
+ if [ "$(grep -c -v rcu "${result}")" != "0" ]; then
+ echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")"
+ err=1
+ exit
+ fi
}
test_lock_filter()
@@ -224,6 +231,13 @@ test_lock_filter()
err=1
exit
fi
+
+ perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result}
+ if [ "$(grep -c -v rcu "${result}")" != "0" ]; then
+ echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")"
+ err=1
+ exit
+ fi
}
test_stack_filter()
diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c
index c20bd075664e..fa7df9187b08 100644
--- a/tools/perf/util/bpf_lock_contention.c
+++ b/tools/perf/util/bpf_lock_contention.c
@@ -16,6 +16,7 @@
#include <bpf/bpf.h>
#include <bpf/btf.h>
#include <inttypes.h>
+#include <unistd.h>
#include "bpf_skel/lock_contention.skel.h"
#include "bpf_skel/lock_data.h"
@@ -180,6 +181,129 @@ static void init_numa_data(struct lock_contention *con)
free(buf);
}
+static bool has_bpf_fentry_support(struct btf *btf)
+{
+ /*
+ * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and
+ * bpf_get_func_arg() helper availability (5.17+) used by
+ * get_lock_addr() in the SRCU fentry/fexit hooks.
+ */
+ struct bpf_insn insns[] = {
+ { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 },
+ { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3,
+ .src_reg = BPF_REG_10 },
+ { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 },
+ { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg },
+ { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 },
+ { .code = BPF_JMP | BPF_EXIT },
+ };
+ LIBBPF_OPTS(bpf_prog_load_opts, opts,
+ .expected_attach_type = BPF_TRACE_FENTRY,
+ );
+ s32 btf_id;
+ int prog_fd, link_fd;
+
+ if (!btf)
+ return false;
+
+ btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC);
+ if (btf_id <= 0)
+ return false;
+
+ opts.attach_btf_id = btf_id;
+ prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL",
+ insns, ARRAY_SIZE(insns), &opts);
+ if (prog_fd < 0)
+ return false;
+
+ link_fd = bpf_raw_tracepoint_open(NULL, prog_fd);
+ if (link_fd >= 0)
+ close(link_fd);
+ close(prog_fd);
+ return link_fd >= 0;
+}
+
+static void configure_rcu_contention_progs(struct lock_contention *con,
+ bool has_rcu_lock)
+{
+ static const struct {
+ size_t prog_offset;
+ const char *func;
+ bool is_srcu;
+ } rcu_progs[] = {
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu),
+ "synchronize_rcu", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu),
+ "synchronize_rcu", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited),
+ "synchronize_rcu_expedited", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited),
+ "synchronize_rcu_expedited", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks),
+ "synchronize_rcu_tasks", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks),
+ "synchronize_rcu_tasks", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier),
+ "rcu_barrier", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier),
+ "rcu_barrier", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu),
+ "synchronize_srcu", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu),
+ "synchronize_srcu", true },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited),
+ "synchronize_srcu_expedited", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited),
+ "synchronize_srcu_expedited", true },
+ { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier),
+ "srcu_barrier", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier),
+ "srcu_barrier", true },
+ };
+ struct symbol *sym;
+ struct map *kmap;
+ bool need_rcu = true, need_srcu = true;
+ bool fentry_ok;
+
+ sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap);
+ if (sym) {
+ skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start);
+ map__put(kmap);
+ } else {
+ skel->rodata->rcu_state_addr = 1ULL;
+ }
+
+ if (con->filters->nr_types) {
+ need_rcu = false;
+ need_srcu = false;
+ for (int i = 0; i < con->filters->nr_types; i++) {
+ if (con->filters->types[i] & LCB_F_RCU)
+ need_rcu = true;
+ if (con->filters->types[i] & LCB_F_SRCU)
+ need_srcu = true;
+ }
+ }
+
+ if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) {
+ if (!has_rcu_lock && !con->filters->nr_addrs)
+ need_rcu = false;
+ if (!con->filters->nr_addrs && !con->filters->nr_slabs)
+ need_srcu = false;
+ }
+
+ fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf);
+
+ for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) {
+ struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset;
+ bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu;
+ bool enable = fentry_ok && want &&
+ btf__find_by_name_kind(con->btf, rcu_progs[i].func,
+ BTF_KIND_FUNC) > 0;
+
+ bpf_program__set_autoload(*progp, enable);
+ }
+}
+
int lock_contention_prepare(struct lock_contention *con)
{
int i, fd;
@@ -187,6 +311,7 @@ int lock_contention_prepare(struct lock_contention *con)
struct evlist *evlist = con->evlist;
struct target *target = con->target;
bool has_mmap_lock = false;
+ bool has_rcu_lock = false;
/* make sure it loads the kernel map before lookup */
map__load(machine__kernel_map(con->machine));
@@ -200,6 +325,7 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64));
bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries);
bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries);
+ bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries);
if (con->aggr_mode == LOCK_AGGR_TASK)
bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries);
@@ -249,6 +375,10 @@ int lock_contention_prepare(struct lock_contention *con)
has_mmap_lock = true;
continue;
}
+ if (!strcmp(con->filters->syms[i], "rcu")) {
+ has_rcu_lock = true;
+ continue;
+ }
sym = machine__find_kernel_symbol_by_name(con->machine,
con->filters->syms[i],
@@ -269,7 +399,7 @@ int lock_contention_prepare(struct lock_contention *con)
addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start);
con->filters->addrs = addrs;
}
- naddrs = con->filters->nr_addrs ?: has_mmap_lock;
+ naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock);
skel->rodata->has_addr = 1;
}
@@ -305,6 +435,7 @@ int lock_contention_prepare(struct lock_contention *con)
skel->rodata->needs_callstack = con->save_callstack;
skel->rodata->lock_owner = con->owner;
skel->rodata->has_mmap_lock = has_mmap_lock;
+ skel->rodata->has_rcu_lock = has_rcu_lock;
if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) {
if (cgroup_is_v2("perf_event"))
@@ -321,6 +452,7 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map__set_max_entries(skel->maps.slab_filter, nslabs);
init_numa_data(con);
+ configure_rcu_contention_progs(con, has_rcu_lock);
if (lock_contention_bpf__load(skel) < 0) {
pr_err("Failed to load lock-contention BPF skeleton\n");
@@ -503,23 +635,30 @@ static void account_end_timestamp(struct lock_contention *con)
enum lock_aggr_mode aggr_mode = con->aggr_mode;
struct tstamp_data ts_data, *cpu_data;
- /* Iterate per-task tstamp map (key = TID) */
- ts_fd = bpf_map__fd(skel->maps.tstamp);
+ /* Iterate per-task tstamp maps (key = TID) */
+ int ts_fds[] = {
+ bpf_map__fd(skel->maps.tstamp),
+ bpf_map__fd(skel->maps.tstamp_rcu),
+ };
+
stat_fd = bpf_map__fd(skel->maps.lock_stat);
- prev_key = NULL;
- while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) {
- if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) {
- int pid = key;
+ for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) {
+ ts_fd = ts_fds[i];
+ prev_key = NULL;
+ while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) {
+ if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) {
+ int pid = key;
- if (aggr_mode == LOCK_AGGR_TASK && con->owner)
- pid = ts_data.flags;
+ if (aggr_mode == LOCK_AGGR_TASK && con->owner)
+ pid = ts_data.flags;
- update_lock_stat(stat_fd, pid, end_ts, aggr_mode,
- &ts_data);
- }
+ update_lock_stat(stat_fd, pid, end_ts, aggr_mode,
+ &ts_data);
+ }
- prev_key = &key;
+ prev_key = &key;
+ }
}
/* Now it'll check per-cpu tstamp map which doesn't have TID. */
@@ -604,6 +743,8 @@ static const char *lock_contention_get_name(struct lock_contention *con,
return "mmap_lock";
if (flags & LCD_F_SIGHAND_LOCK)
return "siglock";
+ if (flags & LCD_F_RCU_LOCK)
+ return "rcu";
/* global locks with symbols */
sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap);
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index 0f2118006005..257e3402e815 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -59,6 +59,18 @@ struct {
__uint(max_entries, MAX_ENTRIES);
} tstamp SEC(".maps");
+/*
+ * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits
+ * so they do not collide with inner sleeping lock contentions (such as
+ * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, int);
+ __type(value, struct tstamp_data);
+ __uint(max_entries, MAX_ENTRIES);
+} tstamp_rcu SEC(".maps");
+
/* maintain per-CPU timestamp at the beginning of contention */
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
@@ -160,6 +172,7 @@ const volatile int has_addr;
const volatile int has_cgroup;
const volatile int has_slab;
const volatile int has_mmap_lock;
+const volatile int has_rcu_lock;
const volatile int needs_callstack;
const volatile int stack_skip;
const volatile int lock_owner;
@@ -172,6 +185,7 @@ const volatile int aggr_mode;
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
+const volatile __u64 rcu_state_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
@@ -218,7 +232,7 @@ static inline __u64 get_current_cgroup_id(void)
return BPF_CORE_READ(cgrp, kn, id);
}
-static inline int can_record(u64 *ctx)
+static __always_inline int can_record(__u64 lock, __u32 flags)
{
bool is_addr_ok = false;
@@ -242,7 +256,6 @@ static inline int can_record(u64 *ctx)
if (has_type) {
__u8 *ok;
- __u32 flags = (__u32)ctx[1];
ok = bpf_map_lookup_elem(&type_filter, &flags);
if (!ok)
@@ -251,10 +264,9 @@ static inline int can_record(u64 *ctx)
if (has_addr) {
__u8 *ok;
- __u64 addr = ctx[0];
- ok = bpf_map_lookup_elem(&addr_filter, &addr);
- if (!ok && !has_slab && !has_mmap_lock)
+ ok = bpf_map_lookup_elem(&addr_filter, &lock);
+ if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock)
return 0;
is_addr_ok = !!ok;
@@ -272,26 +284,27 @@ static inline int can_record(u64 *ctx)
if (is_addr_ok)
return 1;
- /* slab and mmap_lock are part of the addr_filter */
+ /* slab, mmap_lock, and rcu_lock are part of the addr_filter */
if (has_slab && bpf_get_kmem_cache) {
__u8 *ok;
- __u64 addr = ctx[0];
long kmem_cache_addr;
- kmem_cache_addr = (long)bpf_get_kmem_cache(addr);
+ kmem_cache_addr = (long)bpf_get_kmem_cache(lock);
ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr);
if (ok)
return 1;
- else if (!has_mmap_lock)
+ else if (!has_mmap_lock && !has_rcu_lock)
return 0;
}
- if (has_mmap_lock) {
- __u64 lock = ctx[0];
- __u32 flag = ctx[1];
+ if (has_mmap_lock || has_rcu_lock) {
+ __u32 ltype = check_lock_type(lock, flags);
- if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK)
- return 0;
+ if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK)
+ return 1;
+ if (has_rcu_lock && ltype == LCD_F_RCU_LOCK)
+ return 1;
+ return 0;
}
return 1;
@@ -351,6 +364,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
{
__u32 pid;
struct tstamp_data *pelem;
+ void *map;
/* Use per-cpu array map for spinlock and rwlock */
if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) {
@@ -363,8 +377,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
return pelem;
}
+ map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp;
pid = bpf_get_current_pid_tgid();
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ pelem = bpf_map_lookup_elem(map, &pid);
/* Do not update the element for nested locks */
if (pelem && pelem->lock)
return NULL;
@@ -372,12 +387,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
if (pelem == NULL) {
struct tstamp_data zero = {};
- if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) {
+ if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) {
__sync_fetch_and_add(&task_fail, 1);
return NULL;
}
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ pelem = bpf_map_lookup_elem(map, &pid);
if (pelem == NULL) {
__sync_fetch_and_add(&task_fail, 1);
return NULL;
@@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags)
}
}
-SEC("tp_btf/contention_begin")
-int contention_begin(u64 *ctx)
+static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags)
{
struct tstamp_data *pelem;
- if (!enabled || !can_record(ctx))
+ if (!enabled || !can_record(lock, flags))
return 0;
- pelem = get_tstamp_elem(ctx[1]);
+ pelem = get_tstamp_elem(flags);
if (pelem == NULL)
return 0;
pelem->timestamp = bpf_ktime_get_ns();
- pelem->lock = (__u64)ctx[0];
- pelem->flags = (__u32)ctx[1];
+ pelem->lock = lock;
+ pelem->flags = flags;
if (aggr_mode == LOCK_AGGR_CGROUP)
pelem->cgroup_id = get_current_cgroup_id();
@@ -588,8 +602,7 @@ int contention_begin(u64 *ctx)
return 0;
}
-SEC("tp_btf/contention_end")
-int contention_end(u64 *ctx)
+static __always_inline int do_contention_end(__u64 lock, __s32 ret)
{
__u32 pid = 0, idx = 0;
struct tstamp_data *pelem;
@@ -597,7 +610,7 @@ int contention_end(u64 *ctx)
struct contention_data *data;
__u64 timestamp;
__u64 duration;
- bool need_delete = false;
+ void *ts_map = NULL;
if (!enabled)
return 0;
@@ -613,14 +626,19 @@ int contention_end(u64 *ctx)
*/
pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx);
if (pelem && pelem->lock) {
- if (pelem->lock != ctx[0])
+ if (pelem->lock != lock)
return 0;
} else {
pid = bpf_get_current_pid_tgid();
pelem = bpf_map_lookup_elem(&tstamp, &pid);
- if (!pelem || pelem->lock != ctx[0])
- return 0;
- need_delete = true;
+ if (pelem && pelem->lock == lock) {
+ ts_map = &tstamp;
+ } else {
+ pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid);
+ if (!pelem || pelem->lock != lock)
+ return 0;
+ ts_map = &tstamp_rcu;
+ }
}
timestamp = bpf_ktime_get_ns();
@@ -648,7 +666,6 @@ int contention_end(u64 *ctx)
*/
else {
u32 i = 0;
- s32 ret = (s32)ctx[1];
u64 *buf;
otdata->timestamp = timestamp;
@@ -713,7 +730,7 @@ int contention_end(u64 *ctx)
if (lock_owner)
key.pid = pelem->flags;
else {
- if (!need_delete)
+ if (!ts_map)
pid = bpf_get_current_pid_tgid();
key.pid = pid;
}
@@ -795,11 +812,109 @@ int contention_end(u64 *ctx)
check_lock_delay(pelem->lock);
pelem->lock = 0;
- if (need_delete)
- bpf_map_delete_elem(&tstamp, &pid);
+ if (ts_map)
+ bpf_map_delete_elem(ts_map, &pid);
return 0;
}
+SEC("tp_btf/contention_begin")
+int contention_begin(u64 *ctx)
+{
+ return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]);
+}
+
+SEC("tp_btf/contention_end")
+int contention_end(u64 *ctx)
+{
+ return do_contention_end((__u64)ctx[0], (__s32)ctx[1]);
+}
+
+/* RCU grace-period synchronize / barrier contention hooks */
+SEC("fentry/synchronize_rcu")
+int BPF_PROG(fentry_synchronize_rcu)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu")
+int BPF_PROG(fexit_synchronize_rcu)
+{
+ return do_contention_end(rcu_state_addr, 0);
+}
+
+SEC("fentry/synchronize_rcu_expedited")
+int BPF_PROG(fentry_synchronize_rcu_expedited)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu_expedited")
+int BPF_PROG(fexit_synchronize_rcu_expedited)
+{
+ return do_contention_end(rcu_state_addr, 0);
+}
+
+SEC("fentry/synchronize_rcu_tasks")
+int BPF_PROG(fentry_synchronize_rcu_tasks)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu_tasks")
+int BPF_PROG(fexit_synchronize_rcu_tasks)
+{
+ return do_contention_end(rcu_state_addr, 0);
+}
+
+SEC("fentry/rcu_barrier")
+int BPF_PROG(fentry_rcu_barrier)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/rcu_barrier")
+int BPF_PROG(fexit_rcu_barrier)
+{
+ return do_contention_end(rcu_state_addr, 0);
+}
+
+/* SRCU grace-period synchronize / barrier contention hooks */
+SEC("fentry/synchronize_srcu")
+int BPF_PROG(fentry_synchronize_srcu, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu")
+int BPF_PROG(fexit_synchronize_srcu, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0);
+}
+
+SEC("fentry/synchronize_srcu_expedited")
+int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu_expedited")
+int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0);
+}
+
+SEC("fentry/srcu_barrier")
+int BPF_PROG(fentry_srcu_barrier, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/srcu_barrier")
+int BPF_PROG(fexit_srcu_barrier, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0);
+}
+
SEC("raw_tp/bpf_test_finish")
int BPF_PROG(end_timestamp)
{
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* Re: [PATCH v1 3/5] perf test: Add tests for lock_held() BPF sample filter
2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers
@ 2026-10-17 5:49 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-17 5:49 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Add unit tests in parse-events.c for parsing lock_held(),
> lock_held_read(), and lock_held_exclusive() BPF filter expressions
> (standalone predicates, negated predicates, and compound group
> expressions) and extend record_bpf_filter.sh with lock_held(),
> lock_held_read()/lock_held_write(), and ':R'/':W' recording tests.
>
> Example output:
>
> # perf test -v 'Parsing of BPF sample filters'
> 5: Parse event definition strings :
> 5.7: Parsing of BPF sample filters : Ok
>
> # perf test -v 'perf record sample filtering (by BPF) tests'
> 112: perf record sample filtering (by BPF) tests :
> Checking BPF-filter privilege
> Basic bpf-filter test
> Basic bpf-filter test [Success]
> Failing bpf-filter test
> Failing bpf-filter test [Success]
> Group bpf-filter test
> Group bpf-filter test [Success]
> Multiple bpf-filter test
> Multiple bpf-filter test [Success]
> Cgroup bpf-filter test
> Cgroup bpf-filter test [Success]
> Lock-held bpf-filter test
> Lock-held bpf-filter test [Success]
> 112: perf record sample filtering (by BPF) tests : Ok
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=3
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
@ 2026-10-17 5:52 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-17 5:52 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers
> (get_lock_owner, get_current_mmap_lock, get_current_siglock,
> get_lock_addr, check_lock_type, do_collect_zone_lock,
> do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into
> tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF
> sample filtering.
>
> Also add rw_semaphore.count and struct rt_mutex_base to the minimal
> vmlinux.h and extend get_lock_owner() to support rt_mutex and check
> writer-ownership and handoff flags.
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=1
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
@ 2026-10-17 5:56 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-17 5:56 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks
> Trace RCU, and SRCU locks:
>
> - 'rcu' (supports both read/shared and write/exclusive modes):
> - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)',
> 'lock_held(rcu_read_lock)'): directly inspects
> current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on
> CONFIG_PREEMPT_RCU kernels), accounting for the baseline
> rcu_read_lock() nesting acquired by the perf overflow handler itself
> (bpf_overflow_handler() and do_perf_sw_event()) with zero extra
> fentry/fexit tracing hooks.
> - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'):
> attaches fentry/fexit hooks to synchronize_rcu(),
> synchronize_rcu_expedited(), synchronize_rcu_tasks(), and
> rcu_barrier() to track grace-period and barrier waits.
> [ ... ]
>
> # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1
> # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=4
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU and SRCU
2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
@ 2026-10-17 6:08 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-17 6:08 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf lock contention: wrong caller stack skip in do_contention_begin() for fentry
--
commit 9ca74d6f51b3408e9a151b639dd644cd10562df1
Author: Ian Rogers <irogers@google.com>
perf lock contention: Extend BPF lock contention to RCU and SRCU
This commit extends 'perf lock contention --use-bpf' to track RCU and SRCU
grace-period and barrier waits using fentry/fexit BPF programs and adds
lock type and name filtering support for rcu and srcu.
> diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
> index 0f2118006005a..257e3402e8151 100644
> --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
> +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
[ ... ]
> @@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags)
> }
> }
>
> -SEC("tp_btf/contention_begin")
> -int contention_begin(u64 *ctx)
> +static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags)
> {
> struct tstamp_data *pelem;
>
> - if (!enabled || !can_record(ctx))
> + if (!enabled || !can_record(lock, flags))
> return 0;
>
> - pelem = get_tstamp_elem(ctx[1]);
> + pelem = get_tstamp_elem(flags);
> if (pelem == NULL)
> return 0;
>
> pelem->timestamp = bpf_ktime_get_ns();
> - pelem->lock = (__u64)ctx[0];
> - pelem->flags = (__u32)ctx[1];
> + pelem->lock = lock;
> + pelem->flags = flags;
> if (aggr_mode == LOCK_AGGR_CGROUP)
> pelem->cgroup_id = get_current_cgroup_id();
>
[ ... ]
> @@ -795,11 +812,109 @@ int contention_end(u64 *ctx)
> check_lock_delay(pelem->lock);
>
> pelem->lock = 0;
> - if (need_delete)
> - bpf_map_delete_elem(&tstamp, &pid);
> + if (ts_map)
> + bpf_map_delete_elem(ts_map, &pid);
> return 0;
> }
>
> +SEC("tp_btf/contention_begin")
> +int contention_begin(u64 *ctx)
> +{
> + return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]);
> +}
> +
> +SEC("tp_btf/contention_end")
> +int contention_end(u64 *ctx)
> +{
> + return do_contention_end((__u64)ctx[0], (__s32)ctx[1]);
> +}
> +
> +/* RCU grace-period synchronize / barrier contention hooks */
> +SEC("fentry/synchronize_rcu")
> +int BPF_PROG(fentry_synchronize_rcu)
> +{
> + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
> +}
[Severity: Medium]
In do_contention_begin() in tools/perf/util/bpf_skel/lock_contention.bpf.c,
bpf_get_stackid() uses stack_skip (which defaults to CONTENTION_STACK_SKIP =
4) unconditionally for both tp_btf/contention_begin and the new fentry RCU
and SRCU programs:
skip_owner:
pelem->stack_id = bpf_get_stackid(ctx, &stacks,
BPF_F_FAST_STACK_CMP | stack_skip);
For tp_btf/contention_begin, the tracepoint fires inside a lock slowpath via
tracepoint dispatch:
caller -> mutex_lock -> __mutex_lock_slowpath -> __mutex_lock ->
trace_contention_begin -> __bpf_trace_contention_begin -> bpf_trace_run2 ->
bpf_prog
Skipping 4 frames lands inside the locking functions (.sched.text,
.spinlock.text, or trace_contention_begin), which lock_contention_get_name()
in tools/perf/util/bpf_lock_contention.c then skips via
machine__is_lock_function() to reach the direct caller:
/* LOCK_AGGR_CALLER: skip lock internal functions */
while (machine__is_lock_function(machine, stack_trace[idx]) &&
idx < con->max_stack - 1)
idx++;
addr = stack_trace[idx];
In contrast, fentry_synchronize_rcu() (and the other six RCU/SRCU fentry
programs) attach directly to the top-level API functions. In an fentry
program, the callchain captured by bpf_get_stackid_raw_tp() ->
perf_callchain_kernel() only has two frames before the traced function
itself: ip[0] is the BPF prog, ip[1] is the BPF trampoline, ip[2] is the
traced function (e.g., synchronize_rcu), ip[3] is the direct caller, and
ip[4] is the caller's caller.
Won't skipping stack_skip = 4 frames in bpf_get_stackid() skip past ip[3],
dropping the direct caller from the recorded stack trace, attributing RCU
and SRCU contention in LOCK_AGGR_CALLER mode to the caller's caller (ip[4]),
and causing callstack filtering (-S) in match_callstack_filter() to miss the
direct caller?
Also, if the fentry stack skip is adjusted so that the traced RCU/SRCU
function itself can appear at stack_trace[0], machine__is_lock_function() in
tools/perf/util/machine.c only checks .sched.text, .spinlock.text,
__traceiter_contention_begin, and trace_contention_begin. Since
synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(), and srcu_barrier()
are not in .sched.text or .spinlock.text, should machine__is_lock_function()
or the fentry skip depth account for those functions as well?
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=5
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter
2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers
@ 2026-10-17 6:09 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-17 6:09 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [Medium] perf bpf-filter: held_locks leak and count corruption in fexit_downgrade_write()
- [Medium] perf bpf-filter: prescan_one_lock_expr() misses percpu_rw_semaphore symbols
- [Medium] perf bpf-filter: configure_lock_progs() ignores missing inlined spin/rwlock unlocks
- [Medium] perf bpf-filter: missing CONFIG_DEBUG_LOCK_ALLOC mutex hooks in lock_progs
- [Medium] perf bpf-filter: unguarded mutex->owner read in is_direct_lock_owner() on RT
--
commit 362a7c2a8d1d1d0355b0ac1b022fdffe73aae6c0
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Add lock_held() sample filter
This commit adds BPF sample filter predicates lock_held(), lock_held_read(),
lock_held_shared(), lock_held_write(), and lock_held_exclusive() to record
perf event samples only when a specified kernel lock is held. It uses direct
kernel lock owner inspection where possible and falls back to a held_locks
BPF hash map populated by fentry/fexit hooks on kernel lock and unlock
functions.
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 72e83ac1d6a4b..35d6f7c638684 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
[ ... ]
> +static const struct lock_prog_desc lock_progs[] = {
> + /* Mutex */
> + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
> + LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
[Severity: Medium]
On kernels built with CONFIG_DEBUG_LOCK_ALLOC=y, will mutex acquisitions via
mutex_lock_killable(), mutex_trylock(), and mutex_lock_nest_lock() be missed
by the held_locks map?
In include/linux/mutex.h, when CONFIG_DEBUG_LOCK_ALLOC=y, those functions are
macros that expand to different underlying entry points:
#define mutex_lock(lock) mutex_lock_nested(lock, 0)
#define mutex_lock_interruptible(lock) mutex_lock_interruptible_nested(lock, 0)
#define mutex_lock_killable(lock) _mutex_lock_killable(lock, 0, NULL)
#define mutex_lock_io(lock) mutex_lock_io_nested(lock, 0)
#define mutex_lock_nest_lock(lock, nest_lock) _mutex_lock_nest_lock(lock, &(nest_lock)->dep_map)
#define mutex_trylock(lock) _mutex_trylock_nest_lock(lock, NULL)
While lock_progs[] and sample_filter.bpf.c hook mutex_lock_nested,
mutex_lock_interruptible_nested, and mutex_lock_io_nested, they only hook
mutex_lock_killable and mutex_trylock rather than _mutex_lock_killable,
_mutex_trylock_nest_lock, and _mutex_lock_nest_lock.
For filters that rely on held_locks rather than direct owner inspection (such
as PBF_LOCK_SLAB and PBF_LOCK_TYPE), wouldn't mutexes acquired via
mutex_lock_killable() or mutex_trylock() go untracked on lockdep-enabled
kernels?
[ ... ]
> +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
> + unsigned int needed_groups, bool needs_lock_syms,
> + bool needs_slab_iter)
> +{
> + unsigned int enabled_progs = 0;
> +
> + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) {
> + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset;
> + struct bpf_program *prog = *progp;
> + bool enable = false;
> +
> + if ((needed_groups & lock_progs[i].group) && btf &&
> + lock_progs[i].btf_func) {
> + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func,
> + BTF_KIND_FUNC) > 0)
> + enable = true;
> + }
> +
> + if (enable)
> + enabled_progs++;
> + bpf_program__set_autoload(prog, enable);
> + }
> +
> + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms);
> + if (needs_slab_iter && btf &&
> + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) {
> + bpf_program__set_autoload(skel->progs.slab_cache_iter, true);
> + } else {
> + bpf_program__set_autoload(skel->progs.slab_cache_iter, false);
> + }
> +
> + if (needed_groups && !enabled_progs) {
> + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
> + return -ENOTSUP;
> + }
> + return 0;
> +}
[Severity: Medium]
What happens on kernels where spinlock and rwlock unlock functions are
inlined?
In kernel/Kconfig.locks, when !CONFIG_PREEMPTION && !CONFIG_DEBUG_SPINLOCK
(for example CONFIG_PREEMPT_NONE or CONFIG_PREEMPT_VOLUNTARY without
CONFIG_PREEMPT_DYNAMIC), CONFIG_UNINLINE_SPIN_UNLOCK is not set and
CONFIG_INLINE_SPIN_UNLOCK_IRQ, CONFIG_INLINE_READ_UNLOCK,
CONFIG_INLINE_READ_UNLOCK_IRQ, CONFIG_INLINE_WRITE_UNLOCK, and
CONFIG_INLINE_WRITE_UNLOCK_IRQ are enabled.
In that configuration, _raw_spin_unlock, _raw_spin_unlock_irq,
_raw_read_unlock, _raw_read_unlock_irq, _raw_write_unlock, and
_raw_write_unlock_irq are inlined and absent from vmlinux BTF, while the
acquire functions and *_unlock_irqrestore / *_unlock_bh remain out-of-line
in BTF.
Because configure_lock_progs() enables each program individually if its BTF
function exists and only fails when enabled_progs == 0, it will attach the
spinlock and rwlock acquire hooks without the corresponding unlock hooks.
Wouldn't any spinlock or rwlock released via an inlined unlock (including
rq_lock, zone_lock, and siglock) remain permanently recorded as held in
held_locks?
> +/*
> + * Pre-scan a single filter expression prior to loading the BPF skeleton in
> + * order to determine which BPF maps need to be sized and which lock tracing
> + * program groups need to be autoloaded.
> + */
> +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
> + bool *has_lock_filter,
> + unsigned int *prog_groups,
> + bool *needs_lock_syms,
> + bool *needs_slab_iter)
> +{
[ ... ]
> + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) {
> + *prog_groups |= LOCK_PROG_RTMUTEX;
> + } else if (strtailcmp(base, "mutex") == 0) {
> + *prog_groups |= LOCK_PROG_MUTEX;
> + } else if (strtailcmp(base, "percpu_rwsem") == 0 ||
> + strtailcmp(base, "pcpu_rwsem") == 0) {
> + *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
> + } else if (strtailcmp(base, "rwsem") == 0) {
> + if (mode & PBF_LOCK_MODE_READ)
> + *prog_groups |= LOCK_PROG_RWSEM_READ;
> + if (mode & PBF_LOCK_MODE_WRITE)
> + *prog_groups |= LOCK_PROG_RWSEM_WRITE;
[Severity: Medium]
Can prescan_one_lock_expr() misclassify kernel struct percpu_rw_semaphore
symbols as regular rwsems?
Kernel percpu_rw_semaphore locks defined with DEFINE_PERCPU_RWSEM or
DEFINE_STATIC_PERCPU_RWSEM typically end in _rwsem rather than _percpu_rwsem
(for example cgroup_threadgroup_rwsem, file_rwsem, or scx_fork_rwsem).
In prescan_one_lock_expr(), those symbols match strtailcmp(base, "rwsem") == 0
and only enable LOCK_PROG_RWSEM_READ and/or LOCK_PROG_RWSEM_WRITE, leaving
LOCK_PROG_PERCPU_RWSEM (fexit_percpu_down_write and fentry_percpu_up_write)
disabled. Since is_direct_lock_owner() only inspects struct mutex, struct
rw_semaphore, and struct rt_mutex_base, won't filtering on a
percpu_rw_semaphore symbol like cgroup_threadgroup_rwsem always evaluate to 0?
[ ... ]
> diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
> index e5666d4c17228..fbd2e51815d75 100644
> --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
> +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
[ ... ]
> +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr)
> +{
> + if (!curr)
> + return false;
> +
> + return (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr ||
> + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr ||
> + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr;
> +}
[Severity: Medium]
Will calling get_lock_owner(lock_addr, LCB_F_MUTEX) from
is_direct_lock_owner() break loading sample_filter_bpf on CONFIG_PREEMPT_RT=y
kernels?
perf_sample_filter() inlines perf_get_sample() -> check_lock_held() ->
is_direct_lock_owner(), which calls get_lock_owner(lock_addr, LCB_F_MUTEX) in
tools/perf/util/bpf_skel/lock_data.h:
if (flags & LCB_F_MUTEX) {
struct mutex *mutex = (void *)lock;
owner = BPF_CORE_READ(mutex, owner.counter);
if (owner & 0x4UL)
return NULL;
}
On CONFIG_PREEMPT_RT=y kernels, struct mutex wraps struct rt_mutex_base
rtmutex and does not have an owner field. Because get_lock_owner() does not
guard BPF_CORE_READ(mutex, owner.counter) with bpf_core_field_exists(),
wouldn't libbpf poison the instruction during CO-RE relocation and cause the
BPF verifier to reject perf_sample_filter on PREEMPT_RT kernels even when
lock_held() is not used?
[ ... ]
> +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags)
> +{
> + __u32 owner_id = get_lock_owner_id(flags);
> + struct held_lock_key addr_key = {
> + .addr_or_id = lock_addr,
> + .owner_id = owner_id,
> + .kind = PBF_LOCK_ADDR,
> + };
> + struct held_lock_val *existing;
> + __u32 type_flag = flags & LCB_F_TYPE_MASK;
> + bool is_read = (flags & LCB_F_READ) != 0;
> +
> + if (!lock_enabled || !lock_addr)
> + return 0;
> +
> + existing = bpf_map_lookup_elem(&held_locks, &addr_key);
> + if (!existing)
> + return 0;
> +
> + if (is_read) {
> + if (existing->read_count <= 1 && existing->write_count == 0)
> + bpf_map_delete_elem(&held_locks, &addr_key);
> + else if (existing->read_count > 0)
> + __sync_fetch_and_add(&existing->read_count, -1);
> + } else {
> + if (existing->write_count == 0 && existing->read_count == 0)
> + bpf_map_delete_elem(&held_locks, &addr_key);
> + else if (existing->read_count == 0)
> + bpf_map_delete_elem(&held_locks, &addr_key);
> + else
> + existing->write_count = 0;
> + }
[ ... ]
> +SEC("fexit/downgrade_write")
> +int BPF_PROG(fexit_downgrade_write, void *sem)
> +{
> + record_lock_acquired(ctx, LCB_F_READ);
> + return record_lock_released(ctx, LCB_F_WRITE);
> +}
[Severity: Medium]
Can fexit_downgrade_write() leak held_locks map entries under write-only rwsem
filters and corrupt PBF_LOCK_SLAB / PBF_LOCK_TYPE write counts?
First, in lock_progs[], fexit_downgrade_write has group LOCK_PROG_RWSEM
(LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE), while fentry_up_read has group
LOCK_PROG_RWSEM_READ:
LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM),
LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ),
LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE),
When a write-only filter such as lock_held_write(&mmap_lock) is used,
prescan_one_lock_expr() sets only LOCK_PROG_RWSEM_WRITE. Because
(needed_groups & lock_progs[i].group) is non-zero for LOCK_PROG_RWSEM,
configure_lock_progs() enables fexit_downgrade_write while leaving
fentry_up_read disabled.
When the kernel calls downgrade_write() (for example mmap_write_downgrade()),
fexit_downgrade_write() calls record_lock_acquired(ctx, LCB_F_READ), setting
existing->read_count = 1, and then calls record_lock_released(ctx,
LCB_F_WRITE), which sees existing->read_count != 0 and sets
existing->write_count = 0 instead of deleting addr_key. Since fentry_up_read
is not attached, up_read() never decrements read_count, leaking the held_locks
entry (and any PBF_LOCK_SLAB read_count).
Second, because fexit_downgrade_write() calls record_lock_acquired(ctx,
LCB_F_READ) before record_lock_released(ctx, LCB_F_WRITE), if downgrade_write()
is called on an rwsem whose initial down_write() was not recorded in
held_locks (such as under a read-only filter where LOCK_PROG_RWSEM_WRITE is
disabled, or when acquired before lock_enabled = 1), record_lock_acquired(ctx,
LCB_F_READ) creates addr_key with read_count = 1 and write_count = 0. The
subsequent record_lock_released(ctx, LCB_F_WRITE) then finds existing != NULL
even though existing->write_count == 0, and proceeds to decrement write_count
for PBF_LOCK_SLAB and PBF_LOCK_TYPE.
Should fexit_downgrade_write release LCB_F_WRITE before acquiring LCB_F_READ
(and only record LCB_F_READ when LOCK_PROG_RWSEM_READ is active), and should
record_lock_released_addr() verify that the corresponding read_count or
write_count was non-zero before decrementing the aggregate slab/type counts?
[ ... ]
完美！ Let's double check all rules:
- No markdown headers, no backticks, no ALL CAPS shouting (except inside quoted code or config/macro names that are literally in code). Wait: `CONFIG_DEBUG_LOCK_ALLOC=y`, `PBF_LOCK_SLAB`, `PBF_LOCK_TYPE`, `CONFIG_PREEMPT_RT=y`, `LOCK_PROG_RWSEM` are exact code identifiers. Let's make sure there are no non-code ALL CAPS words.
- No line numbers mentioned anywhere.
- Ends with a blank line.
- Severity tags `[Severity: Medium]` on their own line right above each inline comment block.
- All 5 findings included.
Everything is clean and strictly follows instructions. I'm ready to output.完美! Let's emit the final response.commit 362a7c2a8d1d1d0355b0ac1b022fdffe73aae6c0
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Add lock_held() sample filter
This commit adds BPF sample filter predicates lock_held(), lock_held_read(),
lock_held_shared(), lock_held_write(), and lock_held_exclusive() to record
perf event samples only when a specified kernel lock is held. It uses direct
kernel lock owner inspection where possible and falls back to a held_locks
BPF hash map populated by fentry/fexit hooks on kernel lock and unlock
functions.
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 72e83ac1d6a4b..35d6f7c638684 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
[ ... ]
> +static const struct lock_prog_desc lock_progs[] = {
> + /* Mutex */
> + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
> + LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
[Severity: Medium]
On kernels built with CONFIG_DEBUG_LOCK_ALLOC=y, will mutex acquisitions via
mutex_lock_killable(), mutex_trylock(), and mutex_lock_nest_lock() be missed
by the held_locks map?
In include/linux/mutex.h, when CONFIG_DEBUG_LOCK_ALLOC=y, those functions are
macros that expand to different underlying entry points:
#define mutex_lock(lock) mutex_lock_nested(lock, 0)
#define mutex_lock_interruptible(lock) mutex_lock_interruptible_nested(lock, 0)
#define mutex_lock_killable(lock) _mutex_lock_killable(lock, 0, NULL)
#define mutex_lock_io(lock) mutex_lock_io_nested(lock, 0)
#define mutex_lock_nest_lock(lock, nest_lock) _mutex_lock_nest_lock(lock, &(nest_lock)->dep_map)
#define mutex_trylock(lock) _mutex_trylock_nest_lock(lock, NULL)
While lock_progs[] and sample_filter.bpf.c hook mutex_lock_nested,
mutex_lock_interruptible_nested, and mutex_lock_io_nested, they only hook
mutex_lock_killable and mutex_trylock rather than _mutex_lock_killable,
_mutex_trylock_nest_lock, and _mutex_lock_nest_lock.
For filters that rely on held_locks rather than direct owner inspection (such
as PBF_LOCK_SLAB and PBF_LOCK_TYPE), wouldn't mutexes acquired via
mutex_lock_killable() or mutex_trylock() go untracked on lockdep-enabled
kernels?
[ ... ]
> +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
> + unsigned int needed_groups, bool needs_lock_syms,
> + bool needs_slab_iter)
> +{
> + unsigned int enabled_progs = 0;
> +
> + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) {
> + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset;
> + struct bpf_program *prog = *progp;
> + bool enable = false;
> +
> + if ((needed_groups & lock_progs[i].group) && btf &&
> + lock_progs[i].btf_func) {
> + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func,
> + BTF_KIND_FUNC) > 0)
> + enable = true;
> + }
> +
> + if (enable)
> + enabled_progs++;
> + bpf_program__set_autoload(prog, enable);
> + }
> +
> + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms);
> + if (needs_slab_iter && btf &&
> + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) {
> + bpf_program__set_autoload(skel->progs.slab_cache_iter, true);
> + } else {
> + bpf_program__set_autoload(skel->progs.slab_cache_iter, false);
> + }
> +
> + if (needed_groups && !enabled_progs) {
> + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
> + return -ENOTSUP;
> + }
> + return 0;
> +}
[Severity: Medium]
What happens on kernels where spinlock and rwlock unlock functions are
inlined?
In kernel/Kconfig.locks, when !CONFIG_PREEMPTION && !CONFIG_DEBUG_SPINLOCK
(for example CONFIG_PREEMPT_NONE or CONFIG_PREEMPT_VOLUNTARY without
CONFIG_PREEMPT_DYNAMIC), CONFIG_UNINLINE_SPIN_UNLOCK is not set and
CONFIG_INLINE_SPIN_UNLOCK_IRQ, CONFIG_INLINE_READ_UNLOCK,
CONFIG_INLINE_READ_UNLOCK_IRQ, CONFIG_INLINE_WRITE_UNLOCK, and
CONFIG_INLINE_WRITE_UNLOCK_IRQ are enabled.
In that configuration, _raw_spin_unlock, _raw_spin_unlock_irq,
_raw_read_unlock, _raw_read_unlock_irq, _raw_write_unlock, and
_raw_write_unlock_irq are inlined and absent from vmlinux BTF, while the
acquire functions and *_unlock_irqrestore / *_unlock_bh remain out-of-line
in BTF.
Because configure_lock_progs() enables each program individually if its BTF
function exists and only fails when enabled_progs == 0, it will attach the
spinlock and rwlock acquire hooks without the corresponding unlock hooks.
Wouldn't any spinlock or rwlock released via an inlined unlock (including
rq_lock, zone_lock, and siglock) remain permanently recorded as held in
held_locks?
> +/*
> + * Pre-scan a single filter expression prior to loading the BPF skeleton in
> + * order to determine which BPF maps need to be sized and which lock tracing
> + * program groups need to be autoloaded.
> + */
> +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
> + bool *has_lock_filter,
> + unsigned int *prog_groups,
> + bool *needs_lock_syms,
> + bool *needs_slab_iter)
> +{
[ ... ]
> + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) {
> + *prog_groups |= LOCK_PROG_RTMUTEX;
> + } else if (strtailcmp(base, "mutex") == 0) {
> + *prog_groups |= LOCK_PROG_MUTEX;
> + } else if (strtailcmp(base, "percpu_rwsem") == 0 ||
> + strtailcmp(base, "pcpu_rwsem") == 0) {
> + *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
> + } else if (strtailcmp(base, "rwsem") == 0) {
> + if (mode & PBF_LOCK_MODE_READ)
> + *prog_groups |= LOCK_PROG_RWSEM_READ;
> + if (mode & PBF_LOCK_MODE_WRITE)
> + *prog_groups |= LOCK_PROG_RWSEM_WRITE;
[Severity: Medium]
Can prescan_one_lock_expr() misclassify kernel struct percpu_rw_semaphore
symbols as regular rwsems?
Kernel percpu_rw_semaphore locks defined with DEFINE_PERCPU_RWSEM or
DEFINE_STATIC_PERCPU_RWSEM typically end in _rwsem rather than _percpu_rwsem
(for example cgroup_threadgroup_rwsem, file_rwsem, or scx_fork_rwsem).
In prescan_one_lock_expr(), those symbols match strtailcmp(base, "rwsem") == 0
and only enable LOCK_PROG_RWSEM_READ and/or LOCK_PROG_RWSEM_WRITE, leaving
LOCK_PROG_PERCPU_RWSEM (fexit_percpu_down_write and fentry_percpu_up_write)
disabled. Since is_direct_lock_owner() only inspects struct mutex, struct
rw_semaphore, and struct rt_mutex_base, won't filtering on a
percpu_rw_semaphore symbol like cgroup_threadgroup_rwsem always evaluate to 0?
[ ... ]
> diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
> index e5666d4c17228..fbd2e51815d75 100644
> --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
> +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
[ ... ]
> +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr)
> +{
> + if (!curr)
> + return false;
> +
> + return (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr ||
> + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr ||
> + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr;
> +}
[Severity: Medium]
Will calling get_lock_owner(lock_addr, LCB_F_MUTEX) from
is_direct_lock_owner() break loading sample_filter_bpf on CONFIG_PREEMPT_RT=y
kernels?
perf_sample_filter() inlines perf_get_sample() -> check_lock_held() ->
is_direct_lock_owner(), which calls get_lock_owner(lock_addr, LCB_F_MUTEX) in
tools/perf/util/bpf_skel/lock_data.h:
if (flags & LCB_F_MUTEX) {
struct mutex *mutex = (void *)lock;
owner = BPF_CORE_READ(mutex, owner.counter);
if (owner & 0x4UL)
return NULL;
}
On CONFIG_PREEMPT_RT=y kernels, struct mutex wraps struct rt_mutex_base
rtmutex and does not have an owner field. Because get_lock_owner() does not
guard BPF_CORE_READ(mutex, owner.counter) with bpf_core_field_exists(),
wouldn't libbpf poison the instruction during CO-RE relocation and cause the
BPF verifier to reject perf_sample_filter on PREEMPT_RT kernels even when
lock_held() is not used?
[ ... ]
> +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags)
> +{
> + __u32 owner_id = get_lock_owner_id(flags);
> + struct held_lock_key addr_key = {
> + .addr_or_id = lock_addr,
> + .owner_id = owner_id,
> + .kind = PBF_LOCK_ADDR,
> + };
> + struct held_lock_val *existing;
> + __u32 type_flag = flags & LCB_F_TYPE_MASK;
> + bool is_read = (flags & LCB_F_READ) != 0;
> +
> + if (!lock_enabled || !lock_addr)
> + return 0;
> +
> + existing = bpf_map_lookup_elem(&held_locks, &addr_key);
> + if (!existing)
> + return 0;
> +
> + if (is_read) {
> + if (existing->read_count <= 1 && existing->write_count == 0)
> + bpf_map_delete_elem(&held_locks, &addr_key);
> + else if (existing->read_count > 0)
> + __sync_fetch_and_add(&existing->read_count, -1);
> + } else {
> + if (existing->write_count == 0 && existing->read_count == 0)
> + bpf_map_delete_elem(&held_locks, &addr_key);
> + else if (existing->read_count == 0)
> + bpf_map_delete_elem(&held_locks, &addr_key);
> + else
> + existing->write_count = 0;
> + }
[ ... ]
> +SEC("fexit/downgrade_write")
> +int BPF_PROG(fexit_downgrade_write, void *sem)
> +{
> + record_lock_acquired(ctx, LCB_F_READ);
> + return record_lock_released(ctx, LCB_F_WRITE);
> +}
[Severity: Medium]
Can fexit_downgrade_write() leak held_locks map entries under write-only rwsem
filters and corrupt PBF_LOCK_SLAB / PBF_LOCK_TYPE write counts?
First, in lock_progs[], fexit_downgrade_write has group LOCK_PROG_RWSEM
(LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE), while fentry_up_read has group
LOCK_PROG_RWSEM_READ:
LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM),
LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ),
LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE),
When a write-only filter such as lock_held_write(&mmap_lock) is used,
prescan_one_lock_expr() sets only LOCK_PROG_RWSEM_WRITE. Because
(needed_groups & lock_progs[i].group) is non-zero for LOCK_PROG_RWSEM,
configure_lock_progs() enables fexit_downgrade_write while leaving
fentry_up_read disabled.
When the kernel calls downgrade_write() (for example mmap_write_downgrade()),
fexit_downgrade_write() calls record_lock_acquired(ctx, LCB_F_READ), setting
existing->read_count = 1, and then calls record_lock_released(ctx,
LCB_F_WRITE), which sees existing->read_count != 0 and sets
existing->write_count = 0 instead of deleting addr_key. Since fentry_up_read
is not attached, up_read() never decrements read_count, leaking the held_locks
entry (and any PBF_LOCK_SLAB read_count).
Second, because fexit_downgrade_write() calls record_lock_acquired(ctx,
LCB_F_READ) before record_lock_released(ctx, LCB_F_WRITE), if downgrade_write()
is called on an rwsem whose initial down_write() was not recorded in
held_locks (such as under a read-only filter where LOCK_PROG_RWSEM_WRITE is
disabled, or when acquired before lock_enabled = 1), record_lock_acquired(ctx,
LCB_F_READ) creates addr_key with read_count = 1 and write_count = 0. The
subsequent record_lock_released(ctx, LCB_F_WRITE) then finds existing != NULL
even though existing->write_count == 0, and proceeds to decrement write_count
for PBF_LOCK_SLAB and PBF_LOCK_TYPE.
Should fexit_downgrade_write release LCB_F_WRITE before acquiring LCB_F_READ
(and only record LCB_F_READ when LOCK_PROG_RWSEM_READ is active), and should
record_lock_released_addr() verify that the corresponding read_count or
write_count was non-zero before decrementing the aggregate slab/type counts?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=2
^ permalink raw reply [flat|nested] 106+ messages in thread
* [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support
2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (4 preceding siblings ...)
2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
@ 2026-10-17 17:14 ` Ian Rogers
2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
` (6 more replies)
5 siblings, 7 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, rcu
This series adds a new BPF sample filter predicate, `lock_held(<lock>)`
(along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`,
`lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W`
mode suffixes), to `perf record --filter`, allowing perf event samples to
be recorded only when a specified kernel lock is held (or not held with
`!`) by the running kernel thread or CPU. It also extends both
`lock_held()` and `perf lock contention --use-bpf` to support RCU and
SRCU, and fixes a pre-existing `lock_stat` filtering bug in `perf lock
contention`.
For example:
# perf record -a -g -e cycles --filter 'lock_held(&cgroup_mutex)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' -- ...
# perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ...
# perf lock contention -a -b -Y rcu,srcu -- sleep 1
Supported `<lock>` specifications match `perf lock contention`:
- Global/static kernel lock symbols (with or without leading `&`, e.g.
`lock_held(&cgroup_mutex)` or `lock_held(cgroup_mutex)`)
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`,
`lock_held_read(&dentry)`)
- Well-known per-process, semi-global, and RCU locks: `mmap_lock`,
`siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`
- Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`,
`rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`,
`percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`,
`rcu:W`, `semaphore`
Limitations and weaknesses in determining whether a lock is held:
- Inlined lock/unlock operations:
- When the kernel is configured to inline spinlock or rwlock operations
(`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`,
`CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`,
`CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` /
`_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are
inlined into callers and have no BTF function entry points for
`fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that
both acquire and release entry points exist in BTF before enabling a
lock family.
- Fast-path `percpu_down_read()` / `percpu_up_read()` and
`srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined
per-CPU counter increments without function calls, so only
`percpu_down_write()` / `percpu_up_write()` and out-of-line
`__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked.
- On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to
`preempt_disable()` and does not maintain
`current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)`
read-side inspection requires `CONFIG_PREEMPT_RCU`.
- Lack of an inspectable owner field on non-exclusive / spinning locks:
- While `struct mutex`, `struct rt_mutex`, and write-locked
`struct rw_semaphore` store the owning `task_struct *` in `lock->owner`
(allowing `perf_sample_filter` to inspect kernel memory directly on
sample overflow when filtering by lock address), spinlocks, rwlocks,
semaphores, SRCU, and reader-locked `rw_semaphore`s (where
`__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`)
do not maintain an authoritative per-task owner pointer.
- Tracking these locks (as well as slab-cache and lock-type filters)
relies on `fexit` (acquire) and `fentry` (release) BPF hooks
maintaining the `held_locks` BPF hash map:
- Any such lock already held before `perf record` attaches its BPF
programs will not be present in `held_locks` until it is released
and re-acquired.
- The attached `fentry`/`fexit` hooks execute on every kernel lock
acquire and release for the hooked lock family across the system,
adding tracing overhead outside of perf sample handling.
Patch overview:
1. "perf lock contention: Share common BPF and lock type definitions in
lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and
`LCB_F_*` flags, CO-RE structs, and common BPF lock helpers
(`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`,
`get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`,
`do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`.
2. "perf bpf-filter: Add lock_held() sample filter": Implements
`lock_held()`, `lock_held_read()`/`lock_held_shared()`,
`lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode
suffixes using direct lock owner inspection where possible (`mutex`,
`rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF map
updated by selectively attached `fentry`/`fexit` hooks.
3. "perf test: Add tests for lock_held() BPF sample filter": Adds unit
tests in `tests/parse-events.c` and shell tests in
`tests/shell/record_bpf_filter.sh`.
4. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU": Supports `rcu` (`rcu:R` via direct
`current->rcu_read_lock_nesting` inspection with zero tracing hooks;
`rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks),
`rcu-trace` (via `current->trc_reader_nesting`), and `srcu`
(`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W`
via `synchronize_srcu*()` / `srcu_barrier()`).
5. "perf lock contention: Extend BPF lock contention to RCU and SRCU":
Extends `perf lock contention --use-bpf` to track RCU and SRCU
grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`).
6. "perf lock contention: Filter events before creating lock_stat":
Fixes `report_lock_contention_begin_event()` in `builtin-lock.c` to
apply `-Y`, `-L`, and `-S` filters before calling
`lock_stat_findnew()`.
Changes in v2:
- Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists`
and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE
relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels.
- Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points
(`_mutex_lock_nest_lock`, `_mutex_lock_killable`,
`_mutex_trylock_nest_lock`, `_down_write_nest_lock`,
`_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to
`sample_filter.bpf.c` and `lock_progs[]`.
- Verify in `supported_lock_prog_groups()` that both acquire and all
unlock BTF entry points exist for a lock family before enabling its
tracing hooks, preventing acquire-only attachment when spinlock/rwlock
unlock functions are inlined.
- Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when
a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`).
- Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write`
on `track_rwsem_read` and verify non-zero read/write counts in
`record_lock_released_addr()` before decrementing aggregate slab/type
counts.
- Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in
`lock_contention.bpf.c` and recognize `synchronize_rcu*`,
`synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in
`machine__is_lock_function()`.
- Add patch 6 fixing `report_lock_contention_begin_event()` to check
type, address, and callstack filters before creating `lock_stat`
entries.
Assisted-by: Antigravity:gemini-3.1-pro
Ian Rogers (6):
perf lock contention: Share common BPF and lock type definitions in
lock_data.h
perf bpf-filter: Add lock_held() sample filter
perf test: Add tests for lock_held() BPF sample filter
perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU
perf lock contention: Extend BPF lock contention to RCU and SRCU
perf lock contention: Filter events before creating lock_stat
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/Documentation/perf-record.txt | 51 +-
tools/perf/builtin-lock.c | 88 +-
tools/perf/tests/parse-events.c | 121 ++
tools/perf/tests/shell/lock_contention.sh | 14 +
tools/perf/tests/shell/record_bpf_filter.sh | 123 ++
tools/perf/util/bpf-filter.c | 1188 ++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 46 +-
tools/perf/util/bpf_lock_contention.c | 167 ++-
.../perf/util/bpf_skel/lock_contention.bpf.c | 427 +++---
tools/perf/util/bpf_skel/lock_data.h | 368 ++++-
tools/perf/util/bpf_skel/sample-filter.h | 71 +
tools/perf/util/bpf_skel/sample_filter.bpf.c | 1096 +++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 +
tools/perf/util/lock-contention.c | 22 +
tools/perf/util/lock-contention.h | 15 +
tools/perf/util/machine.c | 11 +
tools/perf/util/parse-events.c | 3 +-
20 files changed, 3507 insertions(+), 362 deletions(-)
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply [flat|nested] 106+ messages in thread
* [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
@ 2026-10-17 17:14 ` Ian Rogers
2026-10-17 17:31 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers
` (5 subsequent siblings)
6 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, rcu
Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers
(get_lock_owner, get_current_mmap_lock, get_current_siglock,
get_lock_addr, check_lock_type, do_collect_zone_lock,
do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into
tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF
sample filtering.
Also add rw_semaphore.count and struct rt_mutex_base to the minimal
vmlinux.h and extend get_lock_owner() to support rt_mutex and check
writer-ownership and handoff flags.
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/builtin-lock.c | 27 --
.../perf/util/bpf_skel/lock_contention.bpf.c | 265 +-------------
tools/perf/util/bpf_skel/lock_data.h | 324 ++++++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 +
tools/perf/util/lock-contention.c | 18 +
tools/perf/util/lock-contention.h | 13 +
6 files changed, 378 insertions(+), 274 deletions(-)
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index d5c0d55cb82d..feaf1cfd08cf 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -1477,33 +1477,6 @@ static void sort_result(void)
}
}
-static const struct {
- unsigned int flags;
- /*
- * Name of the lock flags (access), with delimeter ':'.
- * For example, rwsem:R of rwsem:W.
- */
- const char *flags_name;
- /* Name of the lock (type), for example, rwlock or rwsem. */
- const char *lock_name;
-} lock_type_table[] = {
- { 0, "semaphore", "semaphore" },
- { LCB_F_SPIN, "spinlock", "spinlock" },
- { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
- { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
- { LCB_F_READ, "rwsem:R", "rwsem" },
- { LCB_F_WRITE, "rwsem:W", "rwsem" },
- { LCB_F_RT, "rt-mutex", "rt-mutex" },
- { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
- { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
- { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
- { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
- { LCB_F_MUTEX, "mutex", "mutex" },
- { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
- /* alias for optimistic spinning only */
- { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
-};
-
static const char *get_type_flags_name(unsigned int flags)
{
flags &= LCB_F_TYPE_MASK;
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index 0d9c6f55050e..0f2118006005 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -8,23 +8,9 @@
#include "lock_data.h"
-/* for collect_lock_syms(). 4096 was rejected by the verifier */
-#define MAX_CPUS 1024
-
-/* for collect_zone_lock(). It should be more than the actual zones. */
-#define MAX_ZONES 10
-
/* for do_lock_delay(). Arbitrarily set to 1 million. */
#define MAX_LOOP (1U << 20)
-/* lock contention flags from include/trace/events/lock.h */
-#define LCB_F_SPIN (1U << 0)
-#define LCB_F_READ (1U << 1)
-#define LCB_F_WRITE (1U << 2)
-#define LCB_F_RT (1U << 3)
-#define LCB_F_PERCPU (1U << 4)
-#define LCB_F_MUTEX (1U << 5)
-
/* callstack storage */
struct {
__uint(type, BPF_MAP_TYPE_STACK_TRACE);
@@ -96,13 +82,6 @@ struct {
__uint(max_entries, MAX_ENTRIES);
} task_data SEC(".maps");
-struct {
- __uint(type, BPF_MAP_TYPE_HASH);
- __uint(key_size, sizeof(__u64));
- __uint(value_size, sizeof(__u32));
- __uint(max_entries, MAX_ENTRIES);
-} lock_syms SEC(".maps");
-
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(key_size, sizeof(__u32));
@@ -138,6 +117,13 @@ struct {
__uint(max_entries, 1);
} cgroup_filter SEC(".maps");
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u32));
+ __uint(max_entries, MAX_ENTRIES);
+} lock_syms SEC(".maps");
+
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(key_size, sizeof(long));
@@ -159,22 +145,6 @@ struct {
__uint(max_entries, 1);
} lock_delays SEC(".maps");
-struct rw_semaphore___old {
- struct task_struct *owner;
-} __attribute__((preserve_access_index));
-
-struct rw_semaphore___new {
- atomic_long_t owner;
-} __attribute__((preserve_access_index));
-
-struct mm_struct___old {
- struct rw_semaphore mmap_sem;
-} __attribute__((preserve_access_index));
-
-struct mm_struct___new {
- struct rw_semaphore mmap_lock;
-} __attribute__((preserve_access_index));
-
struct cas_ctx {
struct contention_data *data;
u64 duration;
@@ -182,8 +152,6 @@ struct cas_ctx {
int min_done;
};
-extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak;
-
/* control flags */
const volatile int has_cpu;
const volatile int has_task;
@@ -202,6 +170,11 @@ const volatile int lock_delay;
/* determine the key of lock stat */
const volatile int aggr_mode;
+const volatile __u64 contig_page_data_addr;
+const volatile __u64 node_data_addr;
+const volatile int nr_nodes;
+const volatile int sizeof_zone;
+
int enabled;
int perf_subsys_id = -1;
@@ -222,8 +195,6 @@ int data_map_full;
struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak;
void bpf_task_release(struct task_struct *p) __ksym __weak;
-static inline __u32 check_lock_type(__u64 lock, __u32 flags);
-
static inline __u64 get_current_cgroup_id(void)
{
struct task_struct *task;
@@ -347,85 +318,6 @@ static inline int update_task_data(struct task_struct *task)
return 0;
}
-#ifndef __has_builtin
-# define __has_builtin(x) 0
-#endif
-
-static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
-{
- struct task_struct *task;
- __u64 owner = 0;
-
- if (flags & LCB_F_MUTEX) {
- struct mutex *mutex = (void *)lock;
- owner = BPF_CORE_READ(mutex, owner.counter);
- } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
- /*
- * Support for the BPF_TYPE_MATCHES argument to the
- * __builtin_preserve_type_info builtin was added at some point during
- * development of clang 15 and it's what is needed for
- * bpf_core_type_matches.
- */
-#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
- if (bpf_core_type_matches(struct rw_semaphore___old)) {
- struct rw_semaphore___old *rwsem = (void *)lock;
- owner = (unsigned long)BPF_CORE_READ(rwsem, owner);
- } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
- struct rw_semaphore___new *rwsem = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
- }
-#else
- /* assume new struct */
- struct rw_semaphore *rwsem = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
-#endif
- }
-
- if (!owner)
- return NULL;
-
- task = (void *)(owner & ~7UL);
- return task;
-}
-
-static inline __u32 check_lock_type(__u64 lock, __u32 flags)
-{
- struct task_struct *curr;
- struct mm_struct___old *mm_old;
- struct mm_struct___new *mm_new;
- struct sighand_struct *sighand;
-
- switch (flags) {
- case LCB_F_READ: /* rwsem */
- case LCB_F_WRITE:
- curr = bpf_get_current_task_btf();
- if (curr->mm == NULL)
- break;
- mm_new = (void *)curr->mm;
- if (bpf_core_field_exists(mm_new->mmap_lock)) {
- if (&mm_new->mmap_lock == (void *)lock)
- return LCD_F_MMAP_LOCK;
- break;
- }
- mm_old = (void *)curr->mm;
- if (bpf_core_field_exists(mm_old->mmap_sem)) {
- if (&mm_old->mmap_sem == (void *)lock)
- return LCD_F_MMAP_LOCK;
- }
- break;
- case LCB_F_SPIN: /* spinlock */
- curr = bpf_get_current_task_btf();
- sighand = curr->sighand;
-
- if (sighand && &sighand->siglock == (void *)lock)
- return LCD_F_SIGHAND_LOCK;
- break;
- default:
- break;
- }
- return 0;
-}
-
static inline long delay_callback(__u64 idx, void *arg)
{
__u64 target = *(__u64 *)arg;
@@ -908,145 +800,24 @@ int contention_end(u64 *ctx)
return 0;
}
-extern struct rq runqueues __ksym;
-
-const volatile __u64 contig_page_data_addr;
-const volatile __u64 node_data_addr;
-const volatile int nr_nodes;
-const volatile int sizeof_zone;
-
-struct rq___old {
- raw_spinlock_t lock;
-} __attribute__((preserve_access_index));
-
-struct rq___new {
- raw_spinlock_t __lock;
-} __attribute__((preserve_access_index));
-
-static void collect_zone_lock(void)
-{
- __u64 nr_zones, zone_off;
- __u64 lock_addr, lock_off;
- __u32 lock_flag = LOCK_CLASS_ZONE_LOCK;
-
- zone_off = offsetof(struct pglist_data, node_zones);
- lock_off = offsetof(struct zone, lock);
-
- if (contig_page_data_addr) {
- struct pglist_data *contig_page_data;
-
- contig_page_data = (void *)(long)contig_page_data_addr;
- nr_zones = BPF_CORE_READ(contig_page_data, nr_zones);
-
- for (int i = 0; i < MAX_ZONES; i++) {
- __u64 zone_addr;
-
- if (i >= nr_zones)
- break;
-
- zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off;
- lock_addr = zone_addr + lock_off;
-
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
- } else if (nr_nodes > 0) {
- struct pglist_data **node_data = (void *)(long)node_data_addr;
-
- for (int i = 0; i < nr_nodes; i++) {
- struct pglist_data *pgdat = NULL;
- int err;
-
- err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
- if (err < 0 || pgdat == NULL)
- break;
-
- nr_zones = BPF_CORE_READ(pgdat, nr_zones);
- for (int k = 0; k < MAX_ZONES; k++) {
- __u64 zone_addr;
-
- if (k >= nr_zones)
- break;
-
- zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off;
- lock_addr = zone_addr + lock_off;
-
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
- }
- }
-}
-
SEC("raw_tp/bpf_test_finish")
-int BPF_PROG(collect_lock_syms)
+int BPF_PROG(end_timestamp)
{
- __u64 lock_addr, lock_off;
- __u32 lock_flag;
-
- if (bpf_core_field_exists(struct rq___new, __lock))
- lock_off = offsetof(struct rq___new, __lock);
- else
- lock_off = offsetof(struct rq___old, lock);
-
- for (int i = 0; i < MAX_CPUS; i++) {
- struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
-
- if (rq == NULL)
- break;
-
- lock_addr = (__u64)(void *)rq + lock_off;
- lock_flag = LOCK_CLASS_RQLOCK;
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
-
- collect_zone_lock();
-
+ end_ts = bpf_ktime_get_ns();
return 0;
}
SEC("raw_tp/bpf_test_finish")
-int BPF_PROG(end_timestamp)
+int BPF_PROG(collect_lock_syms)
{
- end_ts = bpf_ktime_get_ns();
- return 0;
+ return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
}
-/*
- * bpf_iter__kmem_cache added recently so old kernels don't have it in the
- * vmlinux.h. But we cannot add it here since it will cause a compiler error
- * due to redefinition of the struct on later kernels.
- *
- * So it uses a CO-RE trick to access the member only if it has the type.
- * This will support both old and new kernels without compiler errors.
- */
-struct bpf_iter__kmem_cache___new {
- struct kmem_cache *s;
-} __attribute__((preserve_access_index));
-
SEC("iter/kmem_cache")
int slab_cache_iter(void *ctx)
{
- struct kmem_cache *s = NULL;
- struct slab_cache_data d;
- const char *nameptr;
-
- if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) {
- struct bpf_iter__kmem_cache___new *iter = ctx;
-
- s = iter->s;
- }
-
- if (s == NULL)
- return 0;
-
- nameptr = s->name;
- bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr);
-
- d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT;
- if (d.id >= LCB_F_SLAB_ID_END)
- return 0;
-
- bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST);
- return 0;
+ return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
}
char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index 652e114e6b87..bf39128e0be1 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -45,6 +45,14 @@ struct contention_task_data {
#define LCB_F_SLAB_ID_END (1U << 26)
#define LCB_F_SLAB_ID_MASK 0x03FF0000U
+/* lock contention flags from include/trace/events/lock.h */
+#define LCB_F_SPIN (1U << 0)
+#define LCB_F_READ (1U << 1)
+#define LCB_F_WRITE (1U << 2)
+#define LCB_F_RT (1U << 3)
+#define LCB_F_PERCPU (1U << 4)
+#define LCB_F_MUTEX (1U << 5)
+
#define LCB_F_TYPE_MAX (1U << 7)
#define LCB_F_TYPE_MASK 0x0000007FU
@@ -76,4 +84,320 @@ struct slab_cache_data {
char name[SLAB_NAME_MAX];
};
+#ifdef __BPF__
+
+/* for collect_lock_syms(). 4096 was rejected by the verifier */
+#define MAX_CPUS 1024
+
+/* for collect_zone_lock(). It should be more than the actual zones. */
+#define MAX_ZONES 10
+
+extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak;
+extern struct rq runqueues __ksym __weak;
+
+#ifndef __has_builtin
+# define __has_builtin(x) 0
+#endif
+
+struct rw_semaphore___old {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
+struct rw_semaphore___new {
+ atomic_long_t count;
+ atomic_long_t owner;
+} __attribute__((preserve_access_index));
+
+struct mm_struct___old {
+ struct rw_semaphore mmap_sem;
+} __attribute__((preserve_access_index));
+
+struct mm_struct___new {
+ struct rw_semaphore mmap_lock;
+} __attribute__((preserve_access_index));
+
+struct rq___old {
+ raw_spinlock_t lock;
+} __attribute__((preserve_access_index));
+
+struct rq___new {
+ raw_spinlock_t __lock;
+} __attribute__((preserve_access_index));
+
+/*
+ * bpf_iter__kmem_cache added recently so old kernels don't have it in the
+ * vmlinux.h. But we cannot add it here since it will cause a compiler error
+ * due to redefinition of the struct on later kernels.
+ *
+ * So it uses a CO-RE trick to access the member only if it has the type.
+ * This will support both old and new kernels without compiler errors.
+ */
+struct bpf_iter__kmem_cache___new {
+ struct kmem_cache *s;
+} __attribute__((preserve_access_index));
+
+static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
+{
+ struct task_struct *task;
+ __u64 owner = 0;
+
+ if (flags & LCB_F_MUTEX) {
+ struct mutex *mutex = (void *)lock;
+
+ if (bpf_core_field_exists(mutex->owner.counter)) {
+ owner = BPF_CORE_READ(mutex, owner.counter);
+ if (owner & 0x4UL)
+ return NULL;
+ } else if (bpf_core_type_exists(struct rt_mutex_base)) {
+ struct rt_mutex_base *rtm = (void *)lock;
+
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ }
+ } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
+ struct rw_semaphore___new *rwsem = (void *)lock;
+
+ if (flags == LCB_F_WRITE && bpf_core_field_exists(rwsem->count.counter)) {
+ __u64 count = BPF_CORE_READ(rwsem, count.counter);
+
+ /* Must have RWSEM_WRITER_LOCKED (bit 0) set */
+ if (!(count & 1UL))
+ return NULL;
+ }
+ /*
+ * Support for the BPF_TYPE_MATCHES argument to the
+ * __builtin_preserve_type_info builtin was added at some point during
+ * development of clang 15 and it's what is needed for
+ * bpf_core_type_matches.
+ */
+#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
+ if (bpf_core_type_matches(struct rw_semaphore___old)) {
+ struct rw_semaphore___old *old = (void *)lock;
+
+ owner = (unsigned long)BPF_CORE_READ(old, owner);
+ } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
+ owner = BPF_CORE_READ(rwsem, owner.counter);
+ }
+#else
+ /* assume new struct */
+ if (bpf_core_field_exists(rwsem->owner.counter))
+ owner = BPF_CORE_READ(rwsem, owner.counter);
+#endif
+ /* RWSEM_READER_OWNED is bit 0 of owner */
+ if (flags == LCB_F_WRITE && (owner & 1UL))
+ return NULL;
+ } else if (flags == LCB_F_RT) {
+ struct rt_mutex_base *rtm = (void *)lock;
+
+ if (bpf_core_type_exists(struct rt_mutex_base)) {
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ }
+ }
+
+ if (!owner)
+ return NULL;
+
+ task = (void *)(owner & ~7UL);
+ return task;
+}
+
+static __always_inline __u64 get_current_mmap_lock(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+ struct mm_struct___new *mm_new = NULL;
+ struct mm_struct___old *mm_old = NULL;
+ __u64 mm;
+
+ if (!curr)
+ return 0;
+
+ mm = (__u64)BPF_CORE_READ(task, mm);
+ if (!mm)
+ return 0;
+
+ if (bpf_core_field_exists(mm_new->mmap_lock))
+ return mm + bpf_core_field_offset(mm_new->mmap_lock);
+
+ if (bpf_core_field_exists(mm_old->mmap_sem))
+ return mm + bpf_core_field_offset(mm_old->mmap_sem);
+
+ return 0;
+}
+
+static __always_inline __u64 get_current_siglock(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+ struct sighand_struct *sighand = NULL;
+ __u64 sh;
+
+ if (!curr)
+ return 0;
+
+ sh = (__u64)BPF_CORE_READ(task, sighand);
+ if (!sh)
+ return 0;
+
+ return sh + bpf_core_field_offset(sighand->siglock);
+}
+
+/*
+ * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar.
+ *
+ * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the
+ * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed
+ * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such
+ * as `bpf_get_kmem_cache(u64 addr)`.
+ */
+static __always_inline __u64 get_lock_addr(void *ctx)
+{
+ __u64 lock_addr = 0;
+
+ bpf_get_func_arg(ctx, 0, &lock_addr);
+ return lock_addr;
+}
+
+static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags)
+{
+ __u64 curr;
+
+ switch (flags) {
+ case LCB_F_READ: /* rwsem */
+ case LCB_F_WRITE:
+ curr = bpf_get_current_task();
+ if (get_current_mmap_lock(curr) == lock)
+ return LCD_F_MMAP_LOCK;
+ break;
+ case LCB_F_SPIN: /* spinlock */
+ curr = bpf_get_current_task();
+ if (get_current_siglock(curr) == lock)
+ return LCD_F_SIGHAND_LOCK;
+ break;
+ default:
+ break;
+ }
+ return 0;
+}
+
+static __always_inline void do_collect_zone_lock(void *lock_syms_map,
+ __u64 contig_page_data_addr,
+ __u64 node_data_addr,
+ int nr_nodes,
+ int sizeof_zone)
+{
+ __u64 nr_zones, zone_off;
+ __u64 lock_addr, lock_off;
+ __u32 lock_flag = LOCK_CLASS_ZONE_LOCK;
+
+ zone_off = offsetof(struct pglist_data, node_zones);
+ lock_off = offsetof(struct zone, lock);
+
+ if (contig_page_data_addr) {
+ struct pglist_data *contig_page_data;
+
+ contig_page_data = (void *)(long)contig_page_data_addr;
+ nr_zones = BPF_CORE_READ(contig_page_data, nr_zones);
+
+ for (int i = 0; i < MAX_ZONES; i++) {
+ __u64 zone_addr;
+
+ if (i >= nr_zones)
+ break;
+
+ zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off;
+ lock_addr = zone_addr + lock_off;
+
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ } else if (nr_nodes > 0) {
+ struct pglist_data **node_data = (void *)(long)node_data_addr;
+
+ for (int i = 0; i < nr_nodes; i++) {
+ struct pglist_data *pgdat = NULL;
+ int err;
+
+ err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
+ if (err < 0)
+ break;
+ if (pgdat == NULL)
+ continue;
+
+ nr_zones = BPF_CORE_READ(pgdat, nr_zones);
+ for (int k = 0; k < MAX_ZONES; k++) {
+ __u64 zone_addr;
+
+ if (k >= nr_zones)
+ break;
+
+ zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off;
+ lock_addr = zone_addr + lock_off;
+
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ }
+ }
+}
+
+static __always_inline int do_collect_lock_syms(void *lock_syms_map,
+ __u64 contig_page_data_addr,
+ __u64 node_data_addr,
+ int nr_nodes,
+ int sizeof_zone)
+{
+ __u64 lock_addr, lock_off;
+ __u32 lock_flag;
+
+ if (bpf_core_field_exists(struct rq___new, __lock))
+ lock_off = offsetof(struct rq___new, __lock);
+ else
+ lock_off = offsetof(struct rq___old, lock);
+
+ if (&runqueues) {
+ for (int i = 0; i < MAX_CPUS; i++) {
+ struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
+
+ if (rq == NULL)
+ break;
+
+ lock_addr = (__u64)(void *)rq + lock_off;
+ lock_flag = LOCK_CLASS_RQLOCK;
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ }
+
+ do_collect_zone_lock(lock_syms_map, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
+
+ return 0;
+}
+
+static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map,
+ __u32 *slab_cache_id)
+{
+ struct kmem_cache *s = NULL;
+ struct slab_cache_data d = {};
+ const char *nameptr;
+
+ if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) {
+ struct bpf_iter__kmem_cache___new *iter = ctx;
+
+ s = iter->s;
+ }
+
+ if (s == NULL)
+ return 0;
+
+ nameptr = s->name;
+ bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr);
+
+ d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT;
+ if (d.id >= LCB_F_SLAB_ID_END)
+ return 0;
+
+ bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST);
+ return 0;
+}
+
+#endif /* __BPF__ */
+
#endif /* UTIL_BPF_SKEL_LOCK_DATA_H */
diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
index 94e42b022ab1..b75ddc6846bb 100644
--- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
+++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
@@ -76,6 +76,7 @@ struct signal_struct {
} __attribute__((preserve_access_index));
struct rw_semaphore {
+ atomic_long_t count;
atomic_long_t owner;
} __attribute__((preserve_access_index));
@@ -83,6 +84,10 @@ struct mutex {
atomic_long_t owner;
} __attribute__((preserve_access_index));
+struct rt_mutex_base {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
struct kernfs_node {
u64 id;
} __attribute__((preserve_access_index));
diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c
index 119a7206f3cd..a1bb78d6e29c 100644
--- a/tools/perf/util/lock-contention.c
+++ b/tools/perf/util/lock-contention.c
@@ -22,6 +22,24 @@ struct callstack_filter {
static LIST_HEAD(callstack_filters);
struct hlist_head *lockhash_table;
+const struct perf_lock_type lock_type_table[] = {
+ { 0, "semaphore", "semaphore" },
+ { LCB_F_SPIN, "spinlock", "spinlock" },
+ { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
+ { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
+ { LCB_F_READ, "rwsem:R", "rwsem" },
+ { LCB_F_WRITE, "rwsem:W", "rwsem" },
+ { LCB_F_RT, "rt-mutex", "rt-mutex" },
+ { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
+ { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
+ { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
+ { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
+ { LCB_F_MUTEX, "mutex", "mutex" },
+ { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
+ /* alias for optimistic spinning only */
+ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
+};
+
int parse_call_stack(const struct option *opt __maybe_unused, const char *str,
int unset __maybe_unused)
{
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index 59c94190b092..68e9bd77e8a9 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -129,6 +129,19 @@ struct thread_stat {
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+struct perf_lock_type {
+ unsigned int flags;
+ /*
+ * Name of the lock flags (access), with delimiter ':'.
+ * For example, rwsem:R or rwsem:W.
+ */
+ const char *flags_name;
+ /* Name of the lock (type), for example, rwlock or rwsem. */
+ const char *lock_name;
+};
+
+extern const struct perf_lock_type lock_type_table[14];
+
struct evlist;
struct machine;
struct target;
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter
2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
@ 2026-10-17 17:14 ` Ian Rogers
2026-10-17 17:35 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers
` (4 subsequent siblings)
6 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, rcu
Add BPF sample filter predicates lock_held(<lock>),
lock_held_read(<lock>) / lock_held_shared(<lock>), and
lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R'
and ':W' suffixes on <lock>) to allow recording perf event samples only
when a specified kernel lock is held (or not held with '!').
Supported <lock> specifications match perf lock contention:
- Global/static kernel lock symbols (with or without '&', e.g.
lock_held(&cgroup_mutex) or lock_held(cgroup_mutex))
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with '&' (e.g. lock_held(&dentry),
lock_held_read(&dentry))
- Per-process and semi-global locks: mmap_lock, siglock, rq_lock,
zone_lock
- Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W,
rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W,
semaphore
Supported lock hold modes by lock family:
- Both read/shared and write/exclusive: rwsem (including mmap_lock)
and rwlock.
- Write/exclusive only: mutex, spinlock (including siglock, rq_lock,
zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write()
/ percpu_up_write(); fast-path percpu_down_read() / percpu_up_read()
are inlined per-CPU counter operations without function entry points).
When evaluating the filter on sample overflow, perf_sample_filter first
inspects the lock's owner field directly in kernel memory where
possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls
back to a held_locks BPF hash map (tracking separate read_count and
write_count) populated by fentry/fexit hooks on kernel lock/unlock
functions. To minimize tracing overhead, only the BPF hooks for the
requested lock family and mode are autoloaded and attached.
Example usage:
# perf record -a -g -e cycles --filter 'lock_held(&mmap_lock)' \
-- perf bench mem mmap -t 2 -l 100
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ]
# perf report --stdio --no-children
# Overhead Command Shared Object Symbol
# ........ ........ ................. ........................
#
18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms
|
---clear_page_erms
get_page_from_freelist
__alloc_pages_noprof
folio_alloc_mpol_noprof
vma_alloc_folio_noprof
do_anonymous_page
__handle_mm_fault
handle_mm_fault
do_user_addr_fault
exc_page_fault
asm_exc_page_fault
worker_thread
Read/shared vs write/exclusive holding can also be filtered separately:
# perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' ...
# perf record -a -g -e cycles --filter 'lock_held_write(&mmap_lock)' ...
# perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' ...
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 41 +-
tools/perf/util/bpf-filter.c | 1036 +++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 46 +-
tools/perf/util/bpf_skel/sample-filter.h | 59 +
tools/perf/util/bpf_skel/sample_filter.bpf.c | 965 ++++++++++++++++
tools/perf/util/parse-events.c | 3 +-
8 files changed, 2168 insertions(+), 26 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 2351f7b4f5fc..246ae085744d 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -204,7 +204,46 @@ OPTIONS
ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr,
code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat,
p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock,
- mem_dtlb, mem_blk, mem_hops, uid, gid
+ mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>),
+ lock_held_read(<lock>), lock_held_shared(<lock>),
+ lock_held_write(<lock>), lock_held_exclusive(<lock>)
+
+ The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters
+ samples based on whether the specified kernel lock is currently held
+ by the kernel thread or CPU. To differentiate between shared/read
+ holding and exclusive/write holding, use 'lock_held_read(<lock>)'
+ (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)'
+ (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to
+ <lock> (e.g. 'lock_held(&mmap_lock:R)' or 'lock_held(&mmap_lock:W)').
+ <lock> can be a kernel lock symbol name (with or without a leading
+ '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'),
+ a hexadecimal lock address, a slab cache name prefixed with '&'
+ (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known
+ lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock
+ type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
+ 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
+ 'semaphore'). Both read/shared and write/exclusive tracking are
+ supported for 'rwsem' (including 'mmap_lock') and 'rwlock';
+ 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'),
+ 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()')
+ support write/exclusive tracking only.
+
+ Note that 'lock_held(<lock>)' introduces overhead outside of sample
+ overflow handling: in addition to checking the lock's owner field on
+ each sample where possible (such as for mutexes, rt-mutexes, and
+ write-locked rwsems), it attaches BPF fentry/fexit programs to kernel
+ lock acquire and release functions to track held locks (such as
+ spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a
+ BPF map. These BPF hooks run on every kernel lock acquisition and
+ release for the attached lock family, even when no perf sample is
+ being taken. To minimize this overhead, only the hooks for the
+ relevant lock family and mode are attached when known (for example,
+ 'lock_held_read(&mmap_lock)' only hooks rwsem read functions,
+ 'lock_held_write(&mmap_lock)' only hooks rwsem write functions,
+ '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks
+ spinlock functions), whereas slab filters ('&<slab>'), raw hex
+ addresses, or symbols without a recognizable lock-type suffix attach
+ hooks across all lock families (including high-frequency spinlocks).
The <operator> can be one of:
==, !=, >, >=, <, <=, &
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 72e83ac1d6a4..2f4c6bd85f89 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -40,21 +40,35 @@
* The 'dropped' map is to keep how many samples it dropped by the filter and
* it will be reported as lost samples.
*/
+#include <errno.h>
+#include <stdio.h>
#include <stdlib.h>
+#include <stdbool.h>
+#include <stddef.h>
+#include <string.h>
#include <fcntl.h>
+#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/stat.h>
#include <bpf/bpf.h>
+#include <bpf/btf.h>
#include <linux/err.h>
+#include <linux/kernel.h>
#include <linux/list.h>
+#include <linux/string.h>
+#include <linux/zalloc.h>
#include <api/fs/fs.h>
#include <internal/xyarray.h>
#include <perf/threadmap.h>
+#include <symbol/kallsyms.h>
#include "util/cap.h"
#include "util/debug.h"
#include "util/evsel.h"
+#include "util/lock-contention.h"
+#include "util/string2.h"
+#include "util/symbol_conf.h"
#include "util/target.h"
#include "util/thread_map.h"
#include "util/bpf-utils.h"
@@ -63,6 +77,7 @@
#include <util/bpf-filter-flex.h>
#include <util/bpf-filter-bison.h>
+#include "bpf_skel/lock_data.h"
#include "bpf_skel/sample-filter.h"
#include "bpf_skel/sample_filter.skel.h"
@@ -127,7 +142,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
(evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
return 0;
- if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
+ if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID ||
+ expr->term == PBF_TERM_LOCK_HELD) {
/* Not dependent on the sample_type as computed from a BPF helper. */
return 0;
}
@@ -155,9 +171,814 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
return -1;
}
-static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry)
+/*
+ * Program groups for `lock_held(...)` BPF tracing hooks.
+ *
+ * When a `lock_held(...)` filter is specified, only the BPF tracing program
+ * groups required for the requested lock(s) are autoloaded and attached.
+ * If no `lock_held(...)` filter is present, none of these programs are loaded.
+ */
+enum lock_prog_group {
+ LOCK_PROG_MUTEX = 1U << 0,
+ LOCK_PROG_RWSEM_READ = 1U << 1,
+ LOCK_PROG_RWSEM_WRITE = 1U << 2,
+ LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE,
+ LOCK_PROG_SPIN = 1U << 3,
+ LOCK_PROG_RWLOCK_READ = 1U << 4,
+ LOCK_PROG_RWLOCK_WRITE = 1U << 5,
+ LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE,
+ LOCK_PROG_RTMUTEX = 1U << 6,
+ LOCK_PROG_SEM = 1U << 7,
+ LOCK_PROG_PERCPU_RWSEM = 1U << 8,
+ LOCK_PROG_ALL = 0x1ffU,
+};
+
+static unsigned int lock_flags_to_prog_group(unsigned int flags)
+{
+ if (flags & LCB_F_MUTEX)
+ return LOCK_PROG_MUTEX;
+ if (flags & LCB_F_RT)
+ return LOCK_PROG_RTMUTEX;
+ if (flags & LCB_F_PERCPU)
+ return LOCK_PROG_PERCPU_RWSEM;
+ if (flags == LCB_F_SPIN)
+ return LOCK_PROG_SPIN;
+ if (flags & LCB_F_SPIN)
+ return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE;
+ if (flags & LCB_F_READ)
+ return LOCK_PROG_RWSEM_READ;
+ if (flags & LCB_F_WRITE)
+ return LOCK_PROG_RWSEM_WRITE;
+ return LOCK_PROG_SEM;
+}
+
+struct lock_prog_desc {
+ size_t prog_offset;
+ const char *btf_func;
+ unsigned int group;
+};
+
+#define LOCK_FPROG(field, func, grp) \
+ { offsetof(struct sample_filter_bpf, progs.field), func, grp }
+
+static const struct lock_prog_desc lock_progs[] = {
+ /* Mutex */
+ LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
+ LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
+ /* RW semaphore */
+ LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested",
+ LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested",
+ LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM),
+ LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE),
+ /* Spinlock */
+ LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN),
+ /* RWlock */
+ LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE),
+ /* RT-Mutex */
+ LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible",
+ LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX),
+ /* Semaphore */
+ LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM),
+ LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM),
+ /* Percpu-RWsem */
+ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
+ LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+};
+
+#undef LOCK_FPROG
+
+static int split_lock_name_mode(const char *name, int expr_part,
+ char *base_buf, size_t buf_sz,
+ bool *has_amp, unsigned int *mode_out)
+{
+ const char *bare;
+ size_t len;
+ unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK;
+ unsigned int suffix_mode = 0;
+
+ if (!name || !*name)
+ return -EINVAL;
+
+ *has_amp = (name[0] == '&');
+ bare = *has_amp ? name + 1 : name;
+ len = strlen(bare);
+ if (len == 0)
+ return -EINVAL;
+
+ if (len > 2 && !strcmp(bare + len - 2, ":R")) {
+ suffix_mode = PBF_LOCK_MODE_READ;
+ len -= 2;
+ } else if (len > 2 && !strcmp(bare + len - 2, ":W")) {
+ suffix_mode = PBF_LOCK_MODE_WRITE;
+ len -= 2;
+ }
+
+ if (len == 0 || len >= buf_sz)
+ return -EINVAL;
+
+ if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY &&
+ suffix_mode && expr_mode != suffix_mode)
+ return -EINVAL;
+
+ memcpy(base_buf, bare, len);
+ base_buf[len] = '\0';
+
+ if (suffix_mode)
+ *mode_out = suffix_mode;
+ else if (expr_mode)
+ *mode_out = expr_mode;
+ else
+ *mode_out = PBF_LOCK_MODE_ANY;
+
+ return 0;
+}
+
+/*
+ * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R",
+ * "spinlock", "percpu-rwsem").
+ *
+ * A lock type name may correspond to up to two flag combinations (e.g.
+ * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed
+ * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is
+ * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`.
+ */
+static bool parse_lock_type_name(const char *name, unsigned int mode,
+ struct sample_filter_bpf *skel,
+ u64 *encoded_pair, unsigned int *prog_groups)
+{
+ bool found = false;
+ u32 flag1 = 0xff, flag2 = 0xff;
+ unsigned int groups = 0;
+ const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name;
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) {
+ u32 f = lock_type_table[i].flags;
+ bool is_read = (f & LCB_F_READ) != 0;
+
+ if (strcmp(lock_type_table[i].lock_name, lookup) &&
+ strcmp(lock_type_table[i].flags_name, name))
+ continue;
+
+ /*
+ * percpu_down_read() / percpu_up_read() fast paths are inlined
+ * per-CPU counter increments without function entry points, so
+ * only writer holding (percpu_down_write / percpu_up_write) can
+ * be tracked for percpu-rwsem. rwlock-rt is also not hooked.
+ */
+ if ((f & LCB_F_PERCPU) && is_read)
+ continue;
+ if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE)))
+ continue;
+
+ if (is_read && !(mode & PBF_LOCK_MODE_READ))
+ continue;
+ if (!is_read && !(mode & PBF_LOCK_MODE_WRITE))
+ continue;
+
+ if (!found)
+ flag1 = f;
+ else if (f != flag1)
+ flag2 = f;
+ if (skel && f < LCB_F_TYPE_MAX) {
+ skel->bss->has_lock_type = 1;
+ skel->bss->lock_type_filter[f] = 1;
+ }
+ groups |= lock_flags_to_prog_group(f);
+ found = true;
+ }
+
+ if (!found)
+ return false;
+
+ if (encoded_pair)
+ *encoded_pair = flag1 | ((u64)flag2 << 8);
+ if (prog_groups)
+ *prog_groups |= groups;
+ return true;
+}
+
+struct lock_sym_cb_args {
+ const char *name;
+ u64 addr;
+ bool found;
+};
+
+static int find_lock_sym_cb(void *arg, const char *name,
+ char type __maybe_unused, u64 start)
+{
+ struct lock_sym_cb_args *args = arg;
+
+ if (!strcmp(name, args->name)) {
+ args->found = true;
+ args->addr = start;
+ if (start != 0)
+ return 1;
+ }
+ return 0;
+}
+
+static int find_kernel_lock_sym(const char *name, u64 *addr)
+{
+ struct lock_sym_cb_args args = {
+ .name = name,
+ };
+ const char *kallsyms_filename = "/proc/kallsyms";
+
+ if (symbol_conf.kallsyms_name != NULL)
+ kallsyms_filename = symbol_conf.kallsyms_name;
+
+ if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0)
+ return -ENOENT;
+
+ if (!args.found)
+ return -ENOENT;
+
+ if (args.addr == 0) {
+ pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n",
+ name);
+ return -EACCES;
+ }
+
+ *addr = args.addr;
+ return 0;
+}
+
+static bool parse_lock_hex_addr(const char *str, u64 *addr)
+{
+ char *end;
+ unsigned long long val;
+
+ if (!str || !*str)
+ return false;
+
+ errno = 0;
+ val = strtoull(str, &end, 16);
+ if (*end != '\0' || errno != 0 || val == 0)
+ return false;
+
+ *addr = val;
+ return true;
+}
+
+static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf)
+{
+ u64 addr = 0;
+ char *buf = NULL, *p;
+ size_t len;
+ long last = -1;
+ int ret;
+
+ if (!btf)
+ return;
+
+ ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT);
+ if (ret < 0)
+ return;
+
+ ret = btf__resolve_size(btf, ret);
+ if (ret < 0)
+ return;
+
+ skel->rodata->sizeof_zone = ret;
+
+ if (find_kernel_lock_sym("contig_page_data", &addr) == 0) {
+ skel->rodata->contig_page_data_addr = addr;
+ return;
+ }
+
+ if (find_kernel_lock_sym("node_data", &addr) == 0)
+ skel->rodata->node_data_addr = addr;
+
+ if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) {
+ p = buf;
+ while (p && *p) {
+ char *end;
+
+ last = strtol(p, &end, 0);
+ if (end == p)
+ break;
+ p = end;
+ if (*p == ',' || *p == '-' || *p == '\n')
+ p++;
+ }
+ skel->rodata->nr_nodes = last + 1;
+ free(buf);
+ }
+}
+
+static bool btf_has_func(struct btf *btf, const char *name)
+{
+ return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0;
+}
+
+static unsigned int supported_lock_prog_groups(struct btf *btf)
+{
+ unsigned int supported = 0;
+
+ if (!btf)
+ return 0;
+
+ if ((btf_has_func(btf, "mutex_lock") || btf_has_func(btf, "mutex_lock_nested")) &&
+ btf_has_func(btf, "mutex_unlock"))
+ supported |= LOCK_PROG_MUTEX;
+
+ if ((btf_has_func(btf, "down_read") || btf_has_func(btf, "down_read_nested")) &&
+ btf_has_func(btf, "up_read"))
+ supported |= LOCK_PROG_RWSEM_READ;
+
+ if ((btf_has_func(btf, "down_write") || btf_has_func(btf, "down_write_nested")) &&
+ btf_has_func(btf, "up_write"))
+ supported |= LOCK_PROG_RWSEM_WRITE;
+
+ /*
+ * Require both acquire and all unlock entry points in BTF so we never
+ * attach spinlock/rwlock acquire hooks when unlock functions are
+ * inlined (e.g. !CONFIG_UNINLINE_SPIN_UNLOCK or CONFIG_INLINE_*_UNLOCK).
+ */
+ if ((btf_has_func(btf, "_raw_spin_lock") || btf_has_func(btf, "_raw_spin_lock_nested")) &&
+ btf_has_func(btf, "_raw_spin_unlock") &&
+ btf_has_func(btf, "_raw_spin_unlock_irq") &&
+ btf_has_func(btf, "_raw_spin_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_spin_unlock_bh"))
+ supported |= LOCK_PROG_SPIN;
+
+ if (btf_has_func(btf, "_raw_read_lock") &&
+ btf_has_func(btf, "_raw_read_unlock") &&
+ btf_has_func(btf, "_raw_read_unlock_irq") &&
+ btf_has_func(btf, "_raw_read_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_read_unlock_bh"))
+ supported |= LOCK_PROG_RWLOCK_READ;
+
+ if ((btf_has_func(btf, "_raw_write_lock") || btf_has_func(btf, "_raw_write_lock_nested")) &&
+ btf_has_func(btf, "_raw_write_unlock") &&
+ btf_has_func(btf, "_raw_write_unlock_irq") &&
+ btf_has_func(btf, "_raw_write_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_write_unlock_bh"))
+ supported |= LOCK_PROG_RWLOCK_WRITE;
+
+ if ((btf_has_func(btf, "rt_mutex_lock") || btf_has_func(btf, "rt_mutex_lock_nested")) &&
+ btf_has_func(btf, "rt_mutex_unlock"))
+ supported |= LOCK_PROG_RTMUTEX;
+
+ if (btf_has_func(btf, "down") && btf_has_func(btf, "up"))
+ supported |= LOCK_PROG_SEM;
+
+ if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write"))
+ supported |= LOCK_PROG_PERCPU_RWSEM;
+
+ return supported;
+}
+
+/*
+ * Enable or disable autoload for each lock-tracking BPF program in the
+ * skeleton before `sample_filter_bpf__load()` is called.
+ *
+ * Each program is enabled only if its group is in `needed_groups` and its
+ * target kernel function or tracepoint exists in the running kernel's BTF.
+ */
+static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
+ unsigned int needed_groups,
+ unsigned int required_groups,
+ bool needs_lock_syms, bool needs_slab_iter)
+{
+ unsigned int supported_groups = supported_lock_prog_groups(btf);
+ unsigned int enabled_progs = 0;
+
+ if ((required_groups & supported_groups) != required_groups) {
+ pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n");
+ return -ENOTSUP;
+ }
+
+ needed_groups &= supported_groups;
+ if (skel)
+ skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ);
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) {
+ struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset;
+ struct bpf_program *prog = *progp;
+ bool enable = false;
+
+ if ((needed_groups & lock_progs[i].group) && btf &&
+ lock_progs[i].btf_func) {
+ if (btf__find_by_name_kind(btf, lock_progs[i].btf_func,
+ BTF_KIND_FUNC) > 0)
+ enable = true;
+ }
+
+ if (enable)
+ enabled_progs++;
+ bpf_program__set_autoload(prog, enable);
+ }
+
+ bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms);
+ if (needs_slab_iter && btf &&
+ btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) {
+ bpf_program__set_autoload(skel->progs.slab_cache_iter, true);
+ } else {
+ bpf_program__set_autoload(skel->progs.slab_cache_iter, false);
+ }
+
+ if (needed_groups && !enabled_progs) {
+ pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
+ return -ENOTSUP;
+ }
+ return 0;
+}
+
+/*
+ * Pre-scan a single filter expression prior to loading the BPF skeleton in
+ * order to determine which BPF maps need to be sized and which lock tracing
+ * program groups need to be autoloaded.
+ */
+static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
+ bool *has_lock_filter,
+ unsigned int *prog_groups,
+ unsigned int *required_groups,
+ bool *needs_lock_syms,
+ bool *needs_slab_iter)
+{
+ char base[128];
+ bool has_amp;
+ unsigned int mode, groups = 0;
+ u64 dummy;
+
+ if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name)
+ return;
+
+ *has_lock_filter = true;
+ if (split_lock_name_mode(expr->lock_name, expr->part, base,
+ sizeof(base), &has_amp, &mode) < 0)
+ return;
+
+ if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
+ if (mode & PBF_LOCK_MODE_READ)
+ groups |= LOCK_PROG_RWSEM_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ groups |= LOCK_PROG_RWSEM_WRITE;
+ *prog_groups |= groups;
+ *required_groups |= groups;
+ return;
+ }
+ if (!strcmp(base, "siglock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ return;
+ }
+ if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ *needs_lock_syms = true;
+ return;
+ }
+ if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, &groups)) {
+ *prog_groups |= groups;
+ *required_groups |= groups;
+ return;
+ }
+
+ if (!has_amp && parse_lock_hex_addr(base, &dummy)) {
+ *prog_groups |= LOCK_PROG_ALL;
+ return;
+ }
+
+ if (has_amp)
+ *needs_slab_iter = true;
+
+ if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) {
+ *prog_groups |= LOCK_PROG_RTMUTEX;
+ *required_groups |= LOCK_PROG_RTMUTEX;
+ } else if (strtailcmp(base, "mutex") == 0) {
+ *prog_groups |= LOCK_PROG_MUTEX;
+ *required_groups |= LOCK_PROG_MUTEX;
+ } else if (strtailcmp(base, "percpu_rwsem") == 0 ||
+ strtailcmp(base, "pcpu_rwsem") == 0) {
+ *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
+ *required_groups |= LOCK_PROG_PERCPU_RWSEM;
+ } else if (strtailcmp(base, "rwsem") == 0) {
+ if (mode & PBF_LOCK_MODE_READ) {
+ *prog_groups |= LOCK_PROG_RWSEM_READ;
+ *required_groups |= LOCK_PROG_RWSEM_READ;
+ }
+ if (mode & PBF_LOCK_MODE_WRITE) {
+ *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM;
+ *required_groups |= LOCK_PROG_RWSEM_WRITE;
+ }
+ } else if (strtailcmp(base, "spinlock") == 0) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ } else if (strtailcmp(base, "rwlock") == 0) {
+ if (mode & PBF_LOCK_MODE_READ)
+ groups |= LOCK_PROG_RWLOCK_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ groups |= LOCK_PROG_RWLOCK_WRITE;
+ *prog_groups |= groups;
+ *required_groups |= groups;
+ } else {
+ *prog_groups |= LOCK_PROG_ALL;
+ }
+
+ if (!has_amp && find_kernel_lock_sym(base, &dummy) != 0)
+ *needs_slab_iter = true;
+}
+
+static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter,
+ unsigned int *prog_groups,
+ unsigned int *required_groups,
+ bool *needs_lock_syms,
+ bool *needs_slab_iter)
+{
+ struct perf_bpf_filter_expr *expr, *group;
+
+ *has_lock_filter = false;
+ *prog_groups = 0;
+ *required_groups = 0;
+ *needs_lock_syms = false;
+ *needs_slab_iter = false;
+
+ list_for_each_entry(expr, &evsel->bpf_filters, list) {
+ prescan_one_lock_expr(expr, has_lock_filter, prog_groups,
+ required_groups, needs_lock_syms,
+ needs_slab_iter);
+ if (expr->op == PBF_OP_GROUP_BEGIN) {
+ list_for_each_entry(group, &expr->groups, list) {
+ prescan_one_lock_expr(group, has_lock_filter,
+ prog_groups, required_groups,
+ needs_lock_syms, needs_slab_iter);
+ }
+ }
+ }
+}
+
+static void run_slab_cache_iter(struct sample_filter_bpf *skel)
+{
+ struct bpf_link *link;
+ int fd;
+ char buf[256];
+
+ if (!bpf_program__autoload(skel->progs.slab_cache_iter))
+ return;
+
+ link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL);
+ if (!link || IS_ERR(link)) {
+ pr_debug("cannot attach slab cache iter\n");
+ return;
+ }
+
+ fd = bpf_iter_create(bpf_link__fd(link));
+ if (fd >= 0) {
+ while (read(fd, buf, sizeof(buf)) > 0)
+ continue;
+ close(fd);
+ }
+
+ bpf_link__destroy(link);
+}
+
+static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
+ u64 *kmem_cache_addr)
+{
+ int cache_fd;
+ u64 key, *prev_key = NULL;
+
+ if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter))
+ return false;
+
+ cache_fd = bpf_map__fd(skel->maps.slab_caches);
+ while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) {
+ struct slab_cache_data data;
+
+ if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0)
+ break;
+
+ if (!strcmp(data.name, name)) {
+ *kmem_cache_addr = key;
+ return true;
+ }
+ prev_key = &key;
+ }
+ return false;
+}
+
+/*
+ * Resolve a `lock_held(<name>)` filter expression into `entry->part`
+ * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate
+ * the corresponding BPF filter maps and BSS control flags.
+ *
+ * Resolution order:
+ * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
+ * `rq_lock`, `zone_lock` (with or without leading '&').
+ * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`,
+ * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`,
+ * `semaphore`, etc.
+ * 3. Hexadecimal lock address (when no leading '&').
+ * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`).
+ * 5. Global/static kernel lock symbol in kallsyms (with or without leading '&',
+ * e.g. `&cgroup_mutex` or `cgroup_mutex`).
+ * 6. Slab cache name without leading '&' fallback.
+ */
+static int resolve_lock_expr(struct sample_filter_bpf *skel,
+ struct perf_bpf_filter_expr *expr,
+ struct perf_bpf_filter_entry *entry)
+{
+ const char *name = expr->lock_name;
+ char base[128];
+ bool has_amp;
+ unsigned int mode;
+ u64 addr = 0, encoded_pair = 0;
+ u8 one = 1;
+ int ret;
+
+ if (split_lock_name_mode(name, expr->part, base, sizeof(base),
+ &has_amp, &mode) < 0) {
+ pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: "");
+ return -EINVAL;
+ }
+
+ if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
+ entry->part = PBF_LOCK_MMAP | mode;
+ entry->target = 0;
+ if (skel)
+ skel->bss->has_mmap_lock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "siglock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_SIGHAND | mode;
+ entry->target = 0;
+ if (skel)
+ skel->bss->has_siglock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "rq_lock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_CLASS | mode;
+ entry->target = LOCK_CLASS_RQLOCK;
+ if (skel)
+ skel->bss->has_lock_class = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "zone_lock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_CLASS | mode;
+ entry->target = LOCK_CLASS_ZONE_LOCK;
+ if (skel)
+ skel->bss->has_lock_class = 1;
+ return 0;
+ }
+
+ if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
+ entry->part = PBF_LOCK_TYPE | mode;
+ entry->target = encoded_pair;
+ return 0;
+ }
+
+ if (!has_amp && parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) {
+ pr_err("Error: Lock type '%s' does not support the requested read/write mode\n",
+ name);
+ return -EINVAL;
+ }
+
+ if (!has_amp && parse_lock_hex_addr(base, &addr)) {
+ entry->part = PBF_LOCK_ADDR | mode;
+ entry->target = addr;
+ if (skel) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
+ }
+ return 0;
+ }
+
+ if (has_amp && find_slab_cache(skel, base, &addr)) {
+ int fd = bpf_map__fd(skel->maps.slab_filter);
+
+ entry->part = PBF_LOCK_SLAB | mode;
+ entry->target = addr;
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_slab = 1;
+ return 0;
+ }
+
+ ret = find_kernel_lock_sym(base, &addr);
+ if (ret == 0) {
+ entry->part = PBF_LOCK_ADDR | mode;
+ entry->target = addr;
+ if (skel) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
+ }
+ return 0;
+ }
+ if (ret == -EACCES)
+ return ret;
+
+ if (!has_amp && find_slab_cache(skel, base, &addr)) {
+ int fd = bpf_map__fd(skel->maps.slab_filter);
+
+ entry->part = PBF_LOCK_SLAB | mode;
+ entry->target = addr;
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_slab = 1;
+ return 0;
+ }
+
+ pr_err("Error: Cannot resolve kernel lock '%s'\n", name);
+ return -EINVAL;
+}
+
+static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct perf_bpf_filter_entry *entry)
{
- int i = 0;
+ int i = 0, ret;
struct perf_bpf_filter_expr *expr;
list_for_each_entry(expr, &evsel->bpf_filters, list) {
@@ -171,6 +992,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].part = expr->part;
entry[i].term = expr->term;
entry[i].value = expr->val;
+ entry[i].target = 0;
+ if (expr->term == PBF_TERM_LOCK_HELD) {
+ ret = resolve_lock_expr(skel, expr, &entry[i]);
+ if (ret < 0)
+ return ret;
+ }
i++;
if (expr->op == PBF_OP_GROUP_BEGIN) {
@@ -184,6 +1011,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].part = group->part;
entry[i].term = group->term;
entry[i].value = group->val;
+ entry[i].target = 0;
+ if (group->term == PBF_TERM_LOCK_HELD) {
+ ret = resolve_lock_expr(skel, group, &entry[i]);
+ if (ret < 0)
+ return ret;
+ }
i++;
}
@@ -423,10 +1256,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
{
int i, x, y, fd, ret;
struct sample_filter_bpf *skel = NULL;
+ struct btf *btf = NULL;
struct bpf_program *prog;
struct bpf_link *link;
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
+ bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false;
+ unsigned int prog_groups = 0, required_groups = 0;
#if LIBBPF_CURRENT_VERSION_GEQ(1, 7)
DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts,
.dont_enable = true);
@@ -438,15 +1274,25 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
if (entry == NULL)
return -1;
- ret = get_filter_entries(evsel, entry);
- if (ret < 0) {
- pr_err("Failed to process filter entries\n");
- goto err;
- }
+ prescan_lock_filters(evsel, &has_lock_filter, &prog_groups,
+ &required_groups, &needs_lock_syms,
+ &needs_slab_iter);
- if (needs_idx_hash && geteuid() != 0) {
+ if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
int zero = 0;
+ if (has_lock_filter) {
+ pr_err("Error: lock_held BPF filter requires CAP_BPF capability\n");
+ ret = -EPERM;
+ goto err;
+ }
+
+ ret = get_filter_entries(evsel, NULL, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
+
/* The filters map is shared among other processes */
ret = create_idx_hash(evsel, entry);
if (ret < 0)
@@ -484,13 +1330,69 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return 0;
}
- skel = sample_filter_bpf__open_and_load();
+ skel = sample_filter_bpf__open();
if (!skel) {
+ ret = -errno;
+ pr_err("Failed to open perf sample-filter BPF skeleton\n");
+ goto err;
+ }
+
+ if (has_lock_filter) {
+ btf = btf__load_vmlinux_btf();
+ bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES);
+ bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS);
+ if (needs_slab_iter) {
+ bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS);
+ bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES);
+ }
+ if (needs_lock_syms) {
+ bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
+ init_numa_data(skel, btf);
+ }
+ ret = configure_lock_progs(skel, btf, prog_groups, required_groups,
+ needs_lock_syms, needs_slab_iter);
+ if (ret < 0)
+ goto err;
+ } else {
+ configure_lock_progs(skel, NULL, 0, 0, false, false);
+ }
+
+ if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
pr_err("Failed to load perf sample-filter BPF skeleton\n");
goto err;
}
+ if (has_lock_filter && needs_slab_iter)
+ run_slab_cache_iter(skel);
+
+ ret = get_filter_entries(evsel, skel, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
+
+ if (has_lock_filter) {
+ if (needs_lock_syms && skel->bss->has_lock_class) {
+ DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts,
+ .flags = BPF_F_TEST_RUN_ON_CPU,
+ );
+ int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms);
+
+ bpf_prog_test_run_opts(prog_fd, &opts);
+ }
+ bpf_program__set_autoattach(skel->progs.perf_sample_filter, false);
+ bpf_program__set_autoattach(skel->progs.collect_lock_syms, false);
+ bpf_program__set_autoattach(skel->progs.slab_cache_iter, false);
+
+ ret = sample_filter_bpf__attach(skel);
+ if (ret < 0) {
+ pr_err("Failed to attach lock sample-filter BPF programs\n");
+ goto err;
+ }
+ skel->bss->lock_enabled = 1;
+ }
+
i = 0;
fd = bpf_map__fd(skel->maps.filters);
@@ -513,11 +1415,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
}
}
}
+ btf__free(btf);
free(entry);
evsel->bpf_skel = skel;
return 0;
err:
+ btf__free(btf);
free(entry);
if (!list_empty(&pinned_filters)) {
struct pinned_filter_idx *pfi, *tmp;
@@ -532,6 +1436,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return ret;
}
+void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr)
+{
+ struct perf_bpf_filter_expr *group, *tmp;
+
+ if (!expr)
+ return;
+
+ list_for_each_entry_safe(group, tmp, &expr->groups, list) {
+ list_del(&group->list);
+ perf_bpf_filter_expr__delete(group);
+ }
+ zfree(&expr->lock_name);
+ free(expr);
+}
+
int perf_bpf_filter__destroy(struct evsel *evsel)
{
struct perf_bpf_filter_expr *expr, *tmp;
@@ -539,7 +1458,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel)
list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) {
list_del(&expr->list);
- free(expr);
+ perf_bpf_filter_expr__delete(expr);
}
sample_filter_bpf__destroy(evsel->bpf_skel);
@@ -591,7 +1510,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
{
struct perf_bpf_filter_expr *expr;
- expr = malloc(sizeof(*expr));
+ expr = zalloc(sizeof(*expr));
if (expr != NULL) {
expr->term = term;
expr->part = part;
@@ -602,6 +1521,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
return expr;
}
+struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
+ int mode,
+ enum perf_bpf_filter_op op,
+ unsigned long val)
+{
+ struct perf_bpf_filter_expr *expr;
+
+ if (!lock_name || !*lock_name)
+ return NULL;
+
+ expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val);
+ if (!expr)
+ return NULL;
+
+ expr->lock_name = strdup(lock_name);
+ if (!expr->lock_name) {
+ free(expr);
+ return NULL;
+ }
+ return expr;
+}
+
static bool check_bpf_filter_capable(void)
{
int fd;
@@ -622,14 +1563,11 @@ static bool check_bpf_filter_capable(void)
return false;
}
-int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
+int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
{
YY_BUFFER_STATE buffer;
int ret;
- if (!check_bpf_filter_capable())
- return -EPERM;
-
buffer = perf_bpf_filter__scan_string(str);
ret = perf_bpf_filter_parse(expr_head);
@@ -641,6 +1579,53 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
return ret;
}
+int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
+{
+ if (!check_bpf_filter_capable())
+ return -EPERM;
+
+ return __perf_bpf_filter__parse(expr_head, str);
+}
+
+static int pin_filter_object(struct sample_filter_bpf *skel, const char *path)
+{
+ char *prog_path = NULL;
+ int ret;
+
+ ret = bpf_object__pin_maps(skel->obj, path);
+ if (ret < 0)
+ return ret;
+
+ if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
+ bpf_object__unpin_maps(skel->obj, path);
+ return -ENOMEM;
+ }
+
+ ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path);
+ if (ret < 0)
+ bpf_object__unpin_maps(skel->obj, path);
+
+ free(prog_path);
+ return ret;
+}
+
+static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path)
+{
+ char *prog_path = NULL;
+ int ret, map_ret;
+
+ if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
+ bpf_object__unpin_maps(skel->obj, path);
+ return -ENOMEM;
+ }
+
+ ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path);
+ free(prog_path);
+
+ map_ret = bpf_object__unpin_maps(skel->obj, path);
+ return ret < 0 ? ret : map_ret;
+}
+
int perf_bpf_filter__pin(void)
{
struct sample_filter_bpf *skel;
@@ -654,6 +1639,8 @@ int perf_bpf_filter__pin(void)
goto err;
}
+ configure_lock_progs(skel, NULL, 0, 0, false, false);
+
/* pinned program will use pid-hash */
bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH);
@@ -674,7 +1661,7 @@ int perf_bpf_filter__pin(void)
goto err;
}
- ret = bpf_object__pin(skel->obj, path);
+ ret = pin_filter_object(skel, path);
if (ret < 0) {
pr_err("Failed to pin BPF filter objects\n");
goto err;
@@ -683,7 +1670,7 @@ int perf_bpf_filter__pin(void)
/* setup access permissions for the pinned objects */
dir_fd = open(path, O_PATH);
if (dir_fd < 0) {
- bpf_object__unpin(skel->obj, path);
+ unpin_filter_object(skel, path);
ret = dir_fd;
goto err;
}
@@ -727,6 +1714,8 @@ int perf_bpf_filter__pin(void)
err_close:
close(dir_fd);
+ if (ret < 0)
+ unpin_filter_object(skel, path);
err:
free(path);
@@ -740,13 +1729,20 @@ int perf_bpf_filter__unpin(void)
char *path = NULL;
int ret = -1;
- skel = sample_filter_bpf__open_and_load();
+ skel = sample_filter_bpf__open();
if (!skel) {
ret = -errno;
pr_err("Failed to open perf sample-filter BPF skeleton\n");
goto err;
}
+ configure_lock_progs(skel, NULL, 0, 0, false, false);
+ if (sample_filter_bpf__load(skel) < 0) {
+ ret = -errno;
+ pr_err("Failed to load perf sample-filter BPF skeleton\n");
+ goto err;
+ }
+
if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(),
PERF_BPF_FILTER_PIN_PATH) < 0) {
ret = -errno;
@@ -754,7 +1750,7 @@ int perf_bpf_filter__unpin(void)
goto err;
}
- ret = bpf_object__unpin(skel->obj, path);
+ ret = unpin_filter_object(skel, path);
err:
free(path);
diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h
index 818c554b91b2..fda95ff226b3 100644
--- a/tools/perf/util/bpf-filter.h
+++ b/tools/perf/util/bpf-filter.h
@@ -14,6 +14,13 @@ struct perf_bpf_filter_expr {
int part;
enum perf_bpf_filter_term term;
unsigned long val;
+ /*
+ * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions
+ * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R").
+ * Resolved into `entry.part` and `entry.target` during
+ * perf_bpf_filter__prepare().
+ */
+ char *lock_name;
};
struct evsel;
@@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
int part,
enum perf_bpf_filter_op op,
unsigned long val);
+struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
+ int mode,
+ enum perf_bpf_filter_op op,
+ unsigned long val);
+void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr);
+int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
int perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target);
int perf_bpf_filter__destroy(struct evsel *evsel);
diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l
index 1be9df6550fc..b4022110f1e5 100644
--- a/tools/perf/util/bpf-filter.l
+++ b/tools/perf/util/bpf-filter.l
@@ -1,9 +1,12 @@
%option prefix="perf_bpf_filter_"
%option noyywrap
+%x LOCK_ARG
+
%{
#include <stdio.h>
#include <stdlib.h>
+#include <string.h>
#include <linux/perf_event.h>
#include "bpf-filter.h"
@@ -76,7 +79,7 @@ static int path_or_error(void)
num_dec [0-9]+
num_hex 0[Xx][0-9a-fA-F]+
space [ \t]+
-path [^ \t\n,]+
+path [^ \t\n,()!=>&|<]+
ident [_a-zA-Z][_a-zA-Z0-9]+
%%
@@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); }
uid { return sample(PBF_TERM_UID); }
gid { return sample(PBF_TERM_GID); }
cgroup { return sample_path(PBF_TERM_CGROUP); }
-
+lock_held[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY;
+ return BFT_LOCK_HELD;
+ }
+(lock_held_read|lock_held_shared)[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ;
+ return BFT_LOCK_HELD;
+ }
+(lock_held_write|lock_held_exclusive)[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE;
+ return BFT_LOCK_HELD;
+ }
+
+<LOCK_ARG>{space} { }
+<LOCK_ARG>[^ \t\n()]+ {
+ perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text);
+ return BFT_LOCK_NAME;
+ }
+<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; }
+<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; }
+
+"!" { return '!'; }
"==" { return operator(PBF_OP_EQ); }
"!=" { return operator(PBF_OP_NEQ); }
">" { return operator(PBF_OP_GT); }
diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y
index 5a79a8e7a45b..6ee6edef13f7 100644
--- a/tools/perf/util/bpf-filter.y
+++ b/tools/perf/util/bpf-filter.y
@@ -11,6 +11,7 @@
#include <string.h>
#include <linux/compiler.h>
#include <linux/list.h>
+#include <linux/zalloc.h>
#include "bpf-filter.h"
#include "cgroup.h"
@@ -31,6 +32,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
{
unsigned long num;
char *path;
+ char *lock_name;
+ struct {
+ char *name;
+ int mode;
+ } lock;
struct {
enum perf_bpf_filter_term term;
int part;
@@ -40,12 +46,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
}
%token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH
+%token BFT_LOCK_HELD BFT_LOCK_NAME
%type <expr> filter_term filter_expr
-%destructor { free ($$); } <expr>
+%destructor { perf_bpf_filter_expr__delete($$); } <expr>
+%destructor { free($$); } <lock_name>
+%destructor { free($$.name); } <lock>
%type <sample> BFT_SAMPLE BFT_SAMPLE_PATH
%type <op> BFT_OP
-%type <num> BFT_NUM
+%type <num> BFT_NUM BFT_LOCK_HELD
%type <path> BFT_PATH
+%type <lock_name> BFT_LOCK_NAME
+%type <lock> lock_term
%%
@@ -105,5 +116,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH
$$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id);
cgroup__put(cgrp);
}
+|
+lock_term
+{
+ $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1);
+ zfree(&$1.name);
+ if (!$$)
+ YYERROR;
+}
+|
+'!' lock_term
+{
+ $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0);
+ zfree(&$2.name);
+ if (!$$)
+ YYERROR;
+}
+|
+lock_term BFT_OP BFT_NUM
+{
+ $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3);
+ zfree(&$1.name);
+ if (!$$)
+ YYERROR;
+}
+
+lock_term:
+BFT_LOCK_HELD BFT_LOCK_NAME ')'
+{
+ $$.name = $2;
+ $$.mode = $1;
+}
%%
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index 683fec85e71e..cd711d1d04b6 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -53,14 +53,73 @@ enum perf_bpf_filter_term {
/* Terms computed from BPF helpers. */
PBF_TERM_UID,
PBF_TERM_GID,
+ /*
+ * Kernel lock held predicate (`lock_held(...)`). Evaluates to 1 if the
+ * specified kernel lock is currently held by the running task/CPU, or
+ * 0 otherwise. The lock specification kind is stored in `entry.part`
+ * (`enum perf_bpf_lock_target_kind`) and the resolved target address,
+ * class ID, or type flag encoding is stored in `entry.target`.
+ */
+ PBF_TERM_LOCK_HELD,
};
+/*
+ * Lock target categories for PBF_TERM_LOCK_HELD filters.
+ * Stored in `perf_bpf_filter_entry.part`.
+ */
+enum perf_bpf_lock_target_kind {
+ PBF_LOCK_NONE = 0,
+ /*
+ * Specific global or static lock address (resolved from kallsyms or
+ * given as a hex address). `entry.target` holds the kernel address.
+ */
+ PBF_LOCK_ADDR,
+ /*
+ * Dynamic lock allocated inside a slab cache (`&slab_name`).
+ * `entry.target` holds the kernel `struct kmem_cache *` address.
+ */
+ PBF_LOCK_SLAB,
+ /*
+ * Per-process `current->mm->mmap_lock` (or `mmap_sem`).
+ */
+ PBF_LOCK_MMAP,
+ /*
+ * Per-process `current->sighand->siglock`.
+ */
+ PBF_LOCK_SIGHAND,
+ /*
+ * Semi-global lock class (`rq_lock` or `zone_lock`).
+ * `entry.target` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or
+ * LOCK_CLASS_ZONE_LOCK).
+ */
+ PBF_LOCK_CLASS,
+ /*
+ * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.).
+ * `entry.target` packs up to two `LCB_F_*` flag combinations in the
+ * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
+ */
+ PBF_LOCK_TYPE,
+};
+
+/*
+ * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD:
+ * Bits 0..3: `enum perf_bpf_lock_target_kind`
+ * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding)
+ * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding)
+ */
+#define PBF_LOCK_KIND_MASK 0x0fU
+#define PBF_LOCK_MODE_READ (1U << 4)
+#define PBF_LOCK_MODE_WRITE (1U << 5)
+#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE)
+#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY
+
/* BPF map entry for filtering */
struct perf_bpf_filter_entry {
enum perf_bpf_filter_op op;
__u32 part; /* sub-sample type info when it has multiple values */
enum perf_bpf_filter_term term;
__u64 value;
+ __u64 target; /* lock address, kmem_cache *, class, or type flags */
};
struct idx_hash_key {
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index e5666d4c1722..480ba4b49793 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -5,8 +5,47 @@
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
+#include "lock_data.h"
#include "sample-filter.h"
+/*
+ * Key for the `held_locks` BPF hash map tracking currently held kernel locks.
+ *
+ * - `addr_or_id`:
+ * - PBF_LOCK_ADDR: kernel virtual address of the lock instance
+ * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK)
+ * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address
+ * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags
+ * - `owner_id`:
+ * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`),
+ * since the holder may sleep and migrate across CPUs while holding the lock.
+ * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks
+ * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption
+ * is disabled on the holding CPU and spinlocks such as `rq_lock` may be
+ * acquired by one task and released by the next task across `context_switch()`.
+ * - `kind`: `enum perf_bpf_lock_target_kind`.
+ */
+struct held_lock_key {
+ __u64 addr_or_id;
+ __u32 owner_id;
+ __u32 kind;
+};
+
+/*
+ * Value for `held_locks` entries.
+ * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`)
+ * and exclusive/write acquisitions (including nested class/slab/type
+ * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of
+ * the lock so release tracepoints that do not carry flags (such as
+ * `lock:contended_release`) can decrement matching class/slab/type entries.
+ */
+struct held_lock_val {
+ __u32 read_count;
+ __u32 write_count;
+ __u32 flags;
+ __u32 pad;
+};
+
/* BPF map that will be filled by user space */
struct filters {
__uint(type, BPF_MAP_TYPE_HASH);
@@ -43,10 +82,199 @@ struct lost_count {
__uint(max_entries, 1);
} dropped SEC(".maps");
+/* Currently held kernel locks tracked by fentry/fexit and lock tracepoints. */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, struct held_lock_key);
+ __type(value, struct held_lock_val);
+ __uint(max_entries, 1);
+} held_locks SEC(".maps");
+
+/* Target lock addresses for PBF_LOCK_ADDR filters. */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, __u64);
+ __type(value, __u8);
+ __uint(max_entries, 1);
+} lock_addr_filter SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u32));
+ __uint(max_entries, 1);
+} lock_syms SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u8));
+ __uint(max_entries, 1);
+} slab_filter SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(struct slab_cache_data));
+ __uint(max_entries, 1);
+} slab_caches SEC(".maps");
+
volatile const int use_idx_hash;
+const volatile __u64 contig_page_data_addr;
+const volatile __u64 node_data_addr;
+const volatile int nr_nodes;
+const volatile int sizeof_zone;
+
+__u32 slab_cache_id;
+
+/* Control flags set by userspace when lock_held(...) filters are active. */
+int lock_enabled;
+int has_lock_addr;
+int has_lock_slab;
+int has_lock_class;
+int has_lock_type;
+int has_mmap_lock;
+int has_siglock;
+int track_rwsem_read;
+__u8 lock_type_filter[LCB_F_TYPE_MAX];
+
void *bpf_cast_to_kern_ctx(void *) __ksym;
+/*
+ * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`.
+ *
+ * For locks that maintain a task_struct pointer in their `owner` field
+ * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`),
+ * `perf_sample_filter` can inspect the lock structure directly in kernel memory
+ * and compare the owner against `current` (`bpf_get_current_task()`).
+ *
+ * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear
+ * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and
+ * spinlocks are checked via the `held_locks` BPF map instead.
+ */
+static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr)
+{
+ if (!curr)
+ return false;
+
+ return (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr ||
+ (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr ||
+ (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr;
+}
+
+static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode)
+{
+ if (!val)
+ return false;
+ if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0)
+ return true;
+ if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0)
+ return true;
+ return false;
+}
+
+static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind,
+ __u32 mode, __u32 pid, __u32 cpu_id)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = pid,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+
+ if (pid) {
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+ }
+
+ key.owner_id = cpu_id;
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+
+ return 0;
+}
+
+static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode,
+ __u32 pid, __u32 cpu_id)
+{
+ __u32 flag1 = target & 0xff;
+ __u32 flag2 = (target >> 8) & 0xff;
+
+ if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ return 0;
+}
+
+static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry)
+{
+ __u64 curr;
+ __u32 pid;
+ __u32 cpu_id;
+ __u32 kind;
+ __u32 mode;
+ __u64 lock_addr;
+
+ /*
+ * Pinned filters (use_idx_hash != 0) share a world-writable filters
+ * map and do not support lock_held(). Reject evaluation when pinned
+ * or when lock_enabled is not set so unprivileged users cannot probe
+ * arbitrary kernel addresses via entry->target.
+ */
+ if (use_idx_hash || !lock_enabled)
+ return 0;
+
+ curr = bpf_get_current_task();
+ pid = (__u32)bpf_get_current_pid_tgid();
+ cpu_id = 0x80000000U | bpf_get_smp_processor_id();
+ kind = entry->part & PBF_LOCK_KIND_MASK;
+ mode = entry->part & PBF_LOCK_MODE_MASK;
+
+ if (!mode)
+ mode = PBF_LOCK_MODE_ANY;
+
+ switch (kind) {
+ case PBF_LOCK_ADDR:
+ lock_addr = entry->target;
+ if ((mode & PBF_LOCK_MODE_WRITE) &&
+ is_direct_lock_owner(lock_addr, curr))
+ return 1;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_MMAP:
+ lock_addr = get_current_mmap_lock(curr);
+ if (!lock_addr)
+ return 0;
+ if ((mode & PBF_LOCK_MODE_WRITE) &&
+ (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr)
+ return 1;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_SIGHAND:
+ lock_addr = get_current_siglock(curr);
+ if (!lock_addr)
+ return 0;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_CLASS:
+ return is_lock_in_map(entry->target, PBF_LOCK_CLASS, mode, pid, cpu_id);
+
+ case PBF_LOCK_SLAB:
+ return is_lock_in_map(entry->target, PBF_LOCK_SLAB, mode, pid, cpu_id);
+
+ case PBF_LOCK_TYPE:
+ return is_lock_type_in_map(entry->target, mode, pid, cpu_id);
+
+ default:
+ return 0;
+ }
+}
+
/* new kernel perf_sample_data definition */
struct perf_sample_data___new {
__u64 sample_flags;
@@ -175,6 +403,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
return bpf_get_current_uid_gid() & 0xFFFFFFFF;
case PBF_TERM_GID:
return bpf_get_current_uid_gid() >> 32;
+ case PBF_TERM_LOCK_HELD:
+ return check_lock_held(entry);
case PBF_TERM_NONE:
case __PBF_UNUSED_TERM4:
case __PBF_UNUSED_TERM5:
@@ -295,4 +525,739 @@ int perf_sample_filter(void *ctx)
return 0;
}
+static __always_inline __u32 get_lock_owner_id(__u32 flags)
+{
+ __u32 owner_id;
+
+ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN)
+ return 0x80000000U | bpf_get_smp_processor_id();
+
+ owner_id = (__u32)bpf_get_current_pid_tgid();
+ if (!owner_id)
+ owner_id = 0x80000000U | bpf_get_smp_processor_id();
+ return owner_id;
+}
+
+static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id,
+ __u32 kind, __u32 flags, bool acquired)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = owner_id,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (acquired) {
+ if (val) {
+ if (is_read)
+ __sync_fetch_and_add(&val->read_count, 1);
+ else
+ __sync_fetch_and_add(&val->write_count, 1);
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags,
+ };
+
+ bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY);
+ }
+ } else if (val) {
+ if (is_read) {
+ if (val->read_count <= 1 && val->write_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->read_count > 0)
+ __sync_fetch_and_add(&val->read_count, -1);
+ } else {
+ if (val->write_count <= 1 && val->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->write_count > 0)
+ __sync_fetch_and_add(&val->write_count, -1);
+ }
+ }
+}
+
+static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id)
+{
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ bool matched = false;
+ bool is_read = (flags & LCB_F_READ) != 0;
+ __u32 class_id = 0;
+ __u64 kmem_cache_addr = 0;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool match_type = false;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
+ matched = true;
+
+ if (has_mmap_lock || has_siglock) {
+ __u32 ltype = check_lock_type(lock_addr, flags);
+
+ if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
+ (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
+ matched = true;
+ }
+
+ if (has_lock_class && flags == LCB_F_SPIN) {
+ __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
+
+ if (cid) {
+ class_id = *cid;
+ matched = true;
+ }
+ }
+
+ if (has_lock_slab && bpf_get_kmem_cache) {
+ kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
+ if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr))
+ matched = true;
+ else
+ kmem_cache_addr = 0;
+ }
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) {
+ match_type = true;
+ matched = true;
+ }
+
+ if (!matched)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (existing) {
+ /* Non-reader locks are not recursive; avoid double-counting */
+ if (!is_read && existing->write_count > 0)
+ return 0;
+ if (is_read)
+ __sync_fetch_and_add(&existing->read_count, 1);
+ else
+ __sync_fetch_and_add(&existing->write_count, 1);
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags,
+ };
+
+ if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0)
+ return 0;
+ }
+
+ if (class_id)
+ update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, flags, true);
+ if (kmem_cache_addr) {
+ update_held_lock_count((__u64)kmem_cache_addr, owner_id,
+ PBF_LOCK_SLAB, flags, true);
+ }
+ if (match_type)
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, true);
+
+ return 0;
+}
+
+static __always_inline int record_lock_acquired(void *ctx, __u32 flags)
+{
+ return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags));
+}
+
+static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags)
+{
+ __u32 owner_id = get_lock_owner_id(flags);
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (!existing)
+ return 0;
+
+ if (is_read) {
+ if (existing->read_count == 0)
+ return 0;
+ if (existing->read_count == 1 && existing->write_count == 0)
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ else
+ __sync_fetch_and_add(&existing->read_count, -1);
+ } else {
+ if (existing->write_count == 0)
+ return 0;
+ if (existing->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ else
+ existing->write_count = 0;
+ }
+
+ if (has_lock_class && flags == LCB_F_SPIN) {
+ __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
+
+ if (cid)
+ update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false);
+ }
+
+ if (has_lock_slab && bpf_get_kmem_cache) {
+ __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
+
+ if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) {
+ update_held_lock_count(kmem_cache_addr, owner_id,
+ PBF_LOCK_SLAB, flags, false);
+ }
+ }
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag])
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false);
+
+ return 0;
+}
+
+static __always_inline int record_lock_released(void *ctx, __u32 flags)
+{
+ return record_lock_released_addr(get_lock_addr(ctx), flags);
+}
+
+/* Mutex acquire/release hooks */
+SEC("fexit/mutex_lock")
+int BPF_PROG(fexit_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_nested")
+int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/_mutex_lock_nest_lock")
+int BPF_PROG(fexit__mutex_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io")
+int BPF_PROG(fexit_mutex_lock_io, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io_nested")
+int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_interruptible")
+int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_interruptible_nested")
+int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_killable")
+int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/_mutex_lock_killable")
+int BPF_PROG(fexit__mutex_lock_killable, void *lock, unsigned int subclass,
+ void *nest_lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_trylock")
+int BPF_PROG(fexit_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/_mutex_trylock_nest_lock")
+int BPF_PROG(fexit__mutex_trylock_nest_lock, void *lock, void *nest_lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fentry/mutex_unlock")
+int BPF_PROG(fentry_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_MUTEX);
+}
+
+/* RW semaphore acquire/release hooks */
+SEC("fexit/down_read")
+int BPF_PROG(fexit_down_read, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_nested")
+int BPF_PROG(fexit_down_read_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_interruptible")
+int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable")
+int BPF_PROG(fexit_down_read_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable_nested")
+int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_trylock")
+int BPF_PROG(fexit_down_read_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_write")
+int BPF_PROG(fexit_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_nested")
+int BPF_PROG(fexit_down_write_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/_down_write_nest_lock")
+int BPF_PROG(fexit__down_write_nest_lock, void *sem, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_killable")
+int BPF_PROG(fexit_down_write_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_killable_nested")
+int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_trylock")
+int BPF_PROG(fexit_down_write_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/downgrade_write")
+int BPF_PROG(fexit_downgrade_write, void *sem)
+{
+ if (track_rwsem_read)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+SEC("fentry/up_read")
+int BPF_PROG(fentry_up_read, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_READ);
+}
+
+SEC("fentry/up_write")
+int BPF_PROG(fentry_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+/* Spinlock acquire/release hooks */
+SEC("fexit/_raw_spin_lock")
+int BPF_PROG(fexit_raw_spin_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_nested")
+int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_nest_lock")
+int BPF_PROG(fexit_raw_spin_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq")
+int BPF_PROG(fexit_raw_spin_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq_disable")
+int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave")
+int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave_nested")
+int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_bh")
+int BPF_PROG(fexit_raw_spin_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_trylock")
+int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fexit/_raw_spin_trylock_bh")
+int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fentry/_raw_spin_unlock")
+int BPF_PROG(fentry_raw_spin_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq")
+int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq_enable")
+int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irqrestore")
+int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_bh")
+int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+/* RWlock acquire/release hooks */
+SEC("fexit/_raw_read_lock")
+int BPF_PROG(fexit_raw_read_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_lock_irq")
+int BPF_PROG(fexit_raw_read_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_lock_irqsave")
+int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_lock_bh")
+int BPF_PROG(fexit_raw_read_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_trylock")
+int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+ return 0;
+}
+
+SEC("fentry/_raw_read_unlock")
+int BPF_PROG(fentry_raw_read_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irq")
+int BPF_PROG(fentry_raw_read_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irqrestore")
+int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_bh")
+int BPF_PROG(fentry_raw_read_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_write_lock")
+int BPF_PROG(fexit_raw_write_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_nested")
+int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irq")
+int BPF_PROG(fexit_raw_write_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irqsave")
+int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_bh")
+int BPF_PROG(fexit_raw_write_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_trylock")
+int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fentry/_raw_write_unlock")
+int BPF_PROG(fentry_raw_write_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irq")
+int BPF_PROG(fentry_raw_write_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irqrestore")
+int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_bh")
+int BPF_PROG(fentry_raw_write_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+/* RT-Mutex acquire/release hooks */
+SEC("fexit/rt_mutex_lock")
+int BPF_PROG(fexit_rt_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_nested")
+int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/_rt_mutex_lock_nest_lock")
+int BPF_PROG(fexit__rt_mutex_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_interruptible")
+int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_lock_killable")
+int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_trylock")
+int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fentry/rt_mutex_unlock")
+int BPF_PROG(fentry_rt_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_RT);
+}
+
+/* Semaphore acquire/release hooks */
+SEC("fexit/down")
+int BPF_PROG(fexit_down, void *sem)
+{
+ return record_lock_acquired(ctx, 0);
+}
+
+SEC("fexit/down_interruptible")
+int BPF_PROG(fexit_down_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_killable")
+int BPF_PROG(fexit_down_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_timeout")
+int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_trylock")
+int BPF_PROG(fexit_down_trylock, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fentry/up")
+int BPF_PROG(fentry_up, void *sem)
+{
+ return record_lock_released(ctx, 0);
+}
+
+/* Percpu-rwsem write acquire/release hooks */
+SEC("fexit/percpu_down_write")
+int BPF_PROG(fexit_percpu_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
+SEC("fentry/percpu_up_write")
+int BPF_PROG(fentry_percpu_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
+SEC("raw_tp/bpf_test_finish")
+int BPF_PROG(collect_lock_syms)
+{
+ return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
+}
+
+SEC("iter/kmem_cache")
+int slab_cache_iter(void *ctx)
+{
+ return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
+}
+
char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index 2f3317ce17f0..1fcb424b75c0 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist,
/* Will a tracepoint filter work for str or should a BPF filter be used? */
static bool is_possible_tp_filter(const char *str)
{
- return strstr(str, "uid") == NULL;
+ return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL &&
+ strstr(str, "lock_held") == NULL;
}
static int set_filter(struct evsel *evsel, const void *arg)
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v2 3/6] perf test: Add tests for lock_held() BPF sample filter
2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-17 17:14 ` [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers
@ 2026-10-17 17:14 ` Ian Rogers
2026-10-17 17:17 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
` (3 subsequent siblings)
6 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, rcu
Add unit tests in parse-events.c for parsing lock_held(),
lock_held_read(), and lock_held_exclusive() BPF filter expressions
(standalone predicates, negated predicates, and compound group
expressions) and extend record_bpf_filter.sh with lock_held(),
lock_held_read()/lock_held_write(), and ':R'/':W' recording tests.
Example output:
# perf test -v 'Parsing of BPF sample filters'
5: Parse event definition strings :
5.7: Parsing of BPF sample filters : Ok
# perf test -v 'perf record sample filtering (by BPF) tests'
112: perf record sample filtering (by BPF) tests :
Checking BPF-filter privilege
Basic bpf-filter test
Basic bpf-filter test [Success]
Failing bpf-filter test
Failing bpf-filter test [Success]
Group bpf-filter test
Group bpf-filter test [Success]
Multiple bpf-filter test
Multiple bpf-filter test [Success]
Cgroup bpf-filter test
Cgroup bpf-filter test [Success]
Lock-held bpf-filter test
Lock-held bpf-filter test [Success]
112: perf record sample filtering (by BPF) tests : Ok
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/tests/parse-events.c | 108 ++++++++++++++++++++
tools/perf/tests/shell/record_bpf_filter.sh | 103 +++++++++++++++++++
2 files changed, 211 insertions(+)
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 5f6f95c8a5b4..4cf9192762f2 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -8,9 +8,11 @@
#include "debug.h"
#include "pmu.h"
#include "pmus.h"
+#include "bpf-filter.h"
#include "strbuf.h"
#include <dirent.h>
#include <errno.h>
+#include <string.h>
#include "fncache.h"
#include <sys/types.h>
#include <sys/stat.h>
@@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused,
return ret;
}
+#ifdef HAVE_BPF_SKEL
+static void clear_bpf_filter_exprs(struct list_head *expr_head)
+{
+ struct perf_bpf_filter_expr *expr, *tmp;
+
+ list_for_each_entry_safe(expr, tmp, expr_head, list) {
+ list_del(&expr->list);
+ perf_bpf_filter_expr__delete(expr);
+ }
+}
+
+static int do_test_bpf_filters(struct list_head *expr_head)
+{
+ struct perf_bpf_filter_expr *expr, *group;
+ const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100";
+
+ /* Simple lock_held(&cgroup_mutex) */
+ TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)",
+ __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is &cgroup_mutex",
+ expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /* Negated !lock_held(cgroup_mutex) */
+ TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)",
+ __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 0", expr->val == 0);
+ TEST_ASSERT_VAL("lock_name is cgroup_mutex",
+ expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /* Read/shared and write/exclusive predicates */
+ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)",
+ __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is &mmap_lock",
+ expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock"));
+ clear_bpf_filter_exprs(expr_head);
+
+ TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)",
+ __perf_bpf_filter__parse(expr_head,
+ "!lock_held_exclusive(mmap_lock)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 0", expr->val == 0);
+ TEST_ASSERT_VAL("lock_name is mmap_lock",
+ expr->lock_name && !strcmp(expr->lock_name, "mmap_lock"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /*
+ * Comparison and compound group:
+ * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100
+ */
+ TEST_ASSERT_VAL("parse compound lock_held filter",
+ __perf_bpf_filter__parse(expr_head, compound) == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN);
+ TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups));
+ group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("first group member lock_name",
+ group->lock_name && !strcmp(group->lock_name, "rwsem:R"));
+ TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1);
+ group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("second group member lock_name",
+ group->lock_name && !strcmp(group->lock_name, "&dentry"));
+ TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0);
+ expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100);
+
+ return TEST_OK;
+}
+#endif
+
+static int test__bpf_filters(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
+{
+#ifdef HAVE_BPF_SKEL
+ struct list_head expr_head = LIST_HEAD_INIT(expr_head);
+ int ret = do_test_bpf_filters(&expr_head);
+
+ clear_bpf_filter_exprs(&expr_head);
+ return ret;
+#else
+ return TEST_SKIP;
+#endif
+}
+
static struct test_case tests__parse_events[] = {
TEST_CASE_REASON("Test event parsing",
events2,
@@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = {
"no aliases in sysfs"),
TEST_CASE("Parsing of aliased events", pmu_events_alias2),
TEST_CASE("Parsing of terms (event modifiers)", terms2),
+ TEST_CASE("Parsing of BPF sample filters", bpf_filters),
{ .name = NULL, }
};
diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh
index 383574cb3bd3..adaa5363d0b7 100755
--- a/tools/perf/tests/shell/record_bpf_filter.sh
+++ b/tools/perf/tests/shell/record_bpf_filter.sh
@@ -176,6 +176,105 @@ test_bpf_filter_cgroup() {
echo "Cgroup bpf-filter test [Success]"
}
+test_bpf_filter_lock_held() {
+ echo "Lock-held bpf-filter test"
+ lock_sym="cgroup_mutex"
+
+ if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms
+ then
+ lock_sym="slab_mutex"
+ fi
+
+ if ! perf record -e task-clock --filter "lock_held(&${lock_sym})" \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if [ "$(id -u)" != 0 ]
+ then
+ echo "Lock-held bpf-filter test [Skipped permission]"
+ return
+ fi
+ if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms
+ then
+ echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]"
+ return
+ fi
+ echo "Lock-held bpf-filter test [Failed &${lock_sym}]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \
+ -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed mmap_lock]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \
+ -o /dev/null true 2>&1 | grep -q "Kernel BTF lacks required"
+ then
+ echo "Lock-held bpf-filter test [Failed rq_lock/spinlock]"
+ err=1
+ return
+ fi
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(rwsem)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock type]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock \
+ --filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read/write(&mmap_lock)]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(&mmap_lock:R) || lock_held(&mmap_lock:W)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed &mmap_lock:R/:W]"
+ err=1
+ return
+ fi
+
+ if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \
+ -o /dev/null true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]"
+ err=1
+ return
+ fi
+
+ if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \
+ -o /dev/null true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]"
+ err=1
+ return
+ fi
+
+ echo "Lock-held bpf-filter test [Success]"
+}
+
test_bpf_filter_priv
if [ $err = 0 ]; then
@@ -198,5 +297,9 @@ if [ $err = 0 ]; then
test_bpf_filter_cgroup
fi
+if [ $err = 0 ]; then
+ test_bpf_filter_lock_held
+fi
+
cleanup
exit $err
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (2 preceding siblings ...)
2026-10-17 17:14 ` [PATCH v2 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers
@ 2026-10-17 17:14 ` Ian Rogers
2026-10-17 17:25 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
` (2 subsequent siblings)
6 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, rcu
Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks
Trace RCU, and SRCU locks:
- 'rcu' (supports both read/shared and write/exclusive modes):
- Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)',
'lock_held(rcu_read_lock)'): directly inspects
current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on
CONFIG_PREEMPT_RCU kernels), accounting for the baseline
rcu_read_lock() nesting acquired by the perf overflow handler itself
(bpf_overflow_handler() and do_perf_sw_event()) with zero extra
fentry/fexit tracing hooks.
- Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'):
attaches fentry/fexit hooks to synchronize_rcu(),
synchronize_rcu_expedited(), synchronize_rcu_tasks(), and
rcu_barrier() to track grace-period and barrier waits.
- 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both
read/shared and write/exclusive modes):
- Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches
fexit/__srcu_read_lock and fentry/__srcu_read_unlock.
- Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'):
attaches fentry/fexit on synchronize_srcu(),
synchronize_srcu_expedited(), and srcu_barrier().
- 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports
read/shared mode only): directly inspects
current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels
with zero extra tracing hooks.
Example usage:
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ]
# perf report --stdio --no-children
# Overhead Command Shared Object Symbol
# ........ ....... ................. ........................
#
14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu
|
---__d_lookup_rcu
lookup_fast
walk_component
link_path_walk
path_lookupat
filename_lookup
vfs_statx
__do_sys_newstat
do_syscall_64
entry_SYSCALL_64_after_hwframe
# perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1
# perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1
Signed-off-by: Ian Rogers <irogers@google.com>
# Conflicts:
# tools/perf/util/bpf-filter.c
# tools/perf/util/bpf_skel/sample_filter.bpf.c
---
tools/perf/Documentation/perf-record.txt | 42 +++--
tools/perf/tests/parse-events.c | 13 ++
tools/perf/tests/shell/record_bpf_filter.sh | 20 +++
tools/perf/util/bpf-filter.c | 174 +++++++++++++++++--
tools/perf/util/bpf_skel/lock_data.h | 44 ++++-
tools/perf/util/bpf_skel/sample-filter.h | 12 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 135 +++++++++++++-
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 +
tools/perf/util/lock-contention.c | 4 +
tools/perf/util/lock-contention.h | 4 +-
10 files changed, 418 insertions(+), 32 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 246ae085744d..c8ccfb0e0ce6 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -219,28 +219,38 @@ OPTIONS
'&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'),
a hexadecimal lock address, a slab cache name prefixed with '&'
(e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known
- lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock
- type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
- 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
- 'semaphore'). Both read/shared and write/exclusive tracking are
- supported for 'rwsem' (including 'mmap_lock') and 'rwlock';
+ lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock', 'rcu',
+ 'rcu-trace'), or a lock type ('mutex', 'spinlock', 'rwsem', 'rwsem:R',
+ 'rwsem:W', 'rwlock', 'rwlock:R', 'rwlock:W', 'rt-mutex',
+ 'percpu-rwsem', 'pcpu-sem:W', 'srcu', 'srcu:R', 'srcu:W', 'rcu:R',
+ 'rcu:W', 'semaphore'). Both read/shared and write/exclusive tracking
+ are supported for 'rwsem' (including 'mmap_lock'), 'rwlock', 'rcu',
+ and 'srcu' (where read/shared mode matches 'rcu_read_lock()' /
+ 'srcu_read_lock()' critical sections and write/exclusive mode matches
+ grace-period waits 'synchronize_rcu*()', 'synchronize_srcu*()',
+ 'rcu_barrier()', and 'srcu_barrier()'); 'rcu-trace'
+ ('rcu_read_lock_trace()') supports read/shared tracking only; and
'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'),
'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()')
support write/exclusive tracking only.
Note that 'lock_held(<lock>)' introduces overhead outside of sample
overflow handling: in addition to checking the lock's owner field on
- each sample where possible (such as for mutexes, rt-mutexes, and
- write-locked rwsems), it attaches BPF fentry/fexit programs to kernel
- lock acquire and release functions to track held locks (such as
- spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a
- BPF map. These BPF hooks run on every kernel lock acquisition and
- release for the attached lock family, even when no perf sample is
- being taken. To minimize this overhead, only the hooks for the
- relevant lock family and mode are attached when known (for example,
- 'lock_held_read(&mmap_lock)' only hooks rwsem read functions,
- 'lock_held_write(&mmap_lock)' only hooks rwsem write functions,
- '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks
+ each sample where possible (such as for mutexes, rt-mutexes,
+ write-locked rwsems, and 'rcu:R' / 'rcu-trace' which inspect
+ 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting'
+ directly with no extra tracing hooks), it attaches BPF fentry/fexit
+ programs to kernel lock acquire and release functions to track held
+ locks (such as spinlocks, rwlocks, reader-locked rwsems, SRCU locks,
+ RCU synchronize waits, and slab/type filters) in a BPF map. These
+ BPF hooks run on every kernel lock acquisition and release for the
+ attached lock family, even when no perf sample is being taken. To
+ minimize this overhead, only the hooks for the relevant lock family
+ and mode are attached when known (for example, 'lock_held_read(rcu)'
+ and 'rcu-trace' attach no extra hooks, 'lock_held_read(&mmap_lock)'
+ only hooks rwsem read functions, 'lock_held_write(&mmap_lock)' only
+ hooks rwsem write functions, '&cgroup_mutex' only hooks mutex
+ functions, 'srcu' only hooks SRCU functions, and 'rq_lock' only hooks
spinlock functions), whereas slab filters ('&<slab>'), raw hex
addresses, or symbols without a recognizable lock-type suffix attach
hooks across all lock families (including high-frequency spinlocks).
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 4cf9192762f2..730466b7ebf6 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head)
expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex"));
clear_bpf_filter_exprs(expr_head);
+ /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */
+ TEST_ASSERT_VAL("parse lock_held(rcu)",
+ __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is rcu",
+ expr->lock_name && !strcmp(expr->lock_name, "rcu"));
+ clear_bpf_filter_exprs(expr_head);
+
/* Read/shared and write/exclusive predicates */
TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)",
__perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0);
diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh
index adaa5363d0b7..dc1c1a02eaad 100755
--- a/tools/perf/tests/shell/record_bpf_filter.sh
+++ b/tools/perf/tests/shell/record_bpf_filter.sh
@@ -239,6 +239,26 @@ test_bpf_filter_lock_held() {
return
fi
+ if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed rcu:W/srcu]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \
+ -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]"
+ err=1
+ return
+ fi
+ fi
+
if ! perf record -e task-clock \
--filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \
-o "${perfdata}" true 2> /dev/null
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 2f4c6bd85f89..6261a79b1455 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -190,11 +190,19 @@ enum lock_prog_group {
LOCK_PROG_RTMUTEX = 1U << 6,
LOCK_PROG_SEM = 1U << 7,
LOCK_PROG_PERCPU_RWSEM = 1U << 8,
- LOCK_PROG_ALL = 0x1ffU,
+ LOCK_PROG_SRCU_READ = 1U << 9,
+ LOCK_PROG_SRCU_WRITE = 1U << 10,
+ LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE,
+ LOCK_PROG_RCU_WRITE = 1U << 11,
+ LOCK_PROG_ALL = 0xfffU,
};
static unsigned int lock_flags_to_prog_group(unsigned int flags)
{
+ if (flags & LCB_F_RCU)
+ return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0;
+ if (flags & LCB_F_SRCU)
+ return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE;
if (flags & LCB_F_MUTEX)
return LOCK_PROG_MUTEX;
if (flags & LCB_F_RT)
@@ -314,6 +322,32 @@ static const struct lock_prog_desc lock_progs[] = {
/* Percpu-RWsem */
LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+ /* SRCU */
+ LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe",
+ LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe",
+ LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited",
+ LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited",
+ LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE),
+ /* RCU grace-period synchronize / barrier */
+ LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited",
+ LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited",
+ LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE),
};
#undef LOCK_FPROG
@@ -597,6 +631,18 @@ static unsigned int supported_lock_prog_groups(struct btf *btf)
if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write"))
supported |= LOCK_PROG_PERCPU_RWSEM;
+ if ((btf_has_func(btf, "__srcu_read_lock") ||
+ btf_has_func(btf, "__srcu_read_lock_nmisafe")) &&
+ (btf_has_func(btf, "__srcu_read_unlock") ||
+ btf_has_func(btf, "__srcu_read_unlock_nmisafe")))
+ supported |= LOCK_PROG_SRCU_READ;
+
+ if (btf_has_func(btf, "synchronize_srcu"))
+ supported |= LOCK_PROG_SRCU_WRITE;
+
+ if (btf_has_func(btf, "synchronize_rcu"))
+ supported |= LOCK_PROG_RCU_WRITE;
+
return supported;
}
@@ -701,6 +747,19 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
*needs_lock_syms = true;
return;
}
+ if (!strcmp(base, "rcu")) {
+ if (mode & PBF_LOCK_MODE_WRITE) {
+ *prog_groups |= LOCK_PROG_RCU_WRITE;
+ *required_groups |= LOCK_PROG_RCU_WRITE;
+ }
+ return;
+ }
+ if (!strcmp(base, "rcu_read_lock") ||
+ !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") ||
+ !strcmp(base, "rcu_read_lock_trace")) {
+ /* Inspected directly on task_struct; no tracing hooks needed. */
+ return;
+ }
if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, &groups)) {
*prog_groups |= groups;
*required_groups |= groups;
@@ -744,6 +803,13 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
groups |= LOCK_PROG_RWLOCK_WRITE;
*prog_groups |= groups;
*required_groups |= groups;
+ } else if (strtailcmp(base, "srcu") == 0) {
+ if (mode & PBF_LOCK_MODE_READ)
+ groups |= LOCK_PROG_SRCU_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ groups |= LOCK_PROG_SRCU_WRITE;
+ *prog_groups |= groups;
+ *required_groups |= groups;
} else {
*prog_groups |= LOCK_PROG_ALL;
}
@@ -830,16 +896,54 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
return false;
}
+static u64 rcu_min_depth(struct evsel *evsel)
+{
+ if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE &&
+ evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK &&
+ evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK)
+ return 2;
+ return 1;
+}
+
+static bool btf_has_task_field(struct btf *btf, const char *field_name)
+{
+ const struct btf_type *t;
+ const struct btf_member *m;
+ s32 id;
+ u16 i, vlen;
+
+ if (!btf)
+ return false;
+
+ id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT);
+ if (id < 0)
+ return false;
+
+ t = btf__type_by_id(btf, id);
+ if (!t)
+ return false;
+
+ m = btf_members(t);
+ vlen = btf_vlen(t);
+ for (i = 0; i < vlen; i++, m++) {
+ const char *name = btf__name_by_offset(btf, m->name_off);
+
+ if (name && !strcmp(name, field_name))
+ return true;
+ }
+ return false;
+}
+
/*
* Resolve a `lock_held(<name>)` filter expression into `entry->part`
* (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate
* the corresponding BPF filter maps and BSS control flags.
*
* Resolution order:
- * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
- * `rq_lock`, `zone_lock` (with or without leading '&').
+ * 1. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`,
+ * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` (with or without leading '&').
* 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`,
- * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`,
+ * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, `srcu`,
* `semaphore`, etc.
* 3. Hexadecimal lock address (when no leading '&').
* 4. Slab cache name (when prefixed with '&', e.g. `&dentry`).
@@ -847,8 +951,8 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
* e.g. `&cgroup_mutex` or `cgroup_mutex`).
* 6. Slab cache name without leading '&' fallback.
*/
-static int resolve_lock_expr(struct sample_filter_bpf *skel,
- struct perf_bpf_filter_expr *expr,
+static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct btf *btf, struct perf_bpf_filter_expr *expr,
struct perf_bpf_filter_entry *entry)
{
const char *name = expr->lock_name;
@@ -912,6 +1016,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel,
return 0;
}
+ if (!strcmp(base, "rcu_read_lock")) {
+ if (mode == PBF_LOCK_MODE_WRITE) {
+ pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
+ return -EINVAL;
+ }
+ if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
+ pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ;
+ entry->target = rcu_min_depth(evsel);
+ return 0;
+ }
+
+ if (!strcmp(base, "rcu")) {
+ if ((mode & PBF_LOCK_MODE_READ) && btf &&
+ !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
+ pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU | mode;
+ entry->target = rcu_min_depth(evsel);
+ if ((mode & PBF_LOCK_MODE_WRITE) && skel)
+ skel->bss->has_rcu_lock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") ||
+ !strcmp(base, "rcu_read_lock_trace")) {
+ if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) {
+ pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
+ return -EINVAL;
+ }
+ if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) {
+ pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ;
+ entry->target = 0;
+ return 0;
+ }
+
if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
entry->part = PBF_LOCK_TYPE | mode;
entry->target = encoded_pair;
@@ -976,7 +1122,7 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel,
}
static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
- struct perf_bpf_filter_entry *entry)
+ struct btf *btf, struct perf_bpf_filter_entry *entry)
{
int i = 0, ret;
struct perf_bpf_filter_expr *expr;
@@ -994,7 +1140,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske
entry[i].value = expr->val;
entry[i].target = 0;
if (expr->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(skel, expr, &entry[i]);
+ ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]);
if (ret < 0)
return ret;
}
@@ -1013,7 +1159,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske
entry[i].value = group->val;
entry[i].target = 0;
if (group->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(skel, group, &entry[i]);
+ ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]);
if (ret < 0)
return ret;
}
@@ -1287,7 +1433,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- ret = get_filter_entries(evsel, NULL, entry);
+ ret = get_filter_entries(evsel, NULL, NULL, entry);
if (ret < 0) {
pr_err("Failed to process filter entries\n");
goto err;
@@ -1349,6 +1495,12 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
init_numa_data(skel, btf);
}
+ if (prog_groups & LOCK_PROG_RCU_WRITE) {
+ u64 rcu_addr = 1ULL;
+
+ find_kernel_lock_sym("rcu_state", &rcu_addr);
+ skel->rodata->rcu_state_addr = rcu_addr;
+ }
ret = configure_lock_progs(skel, btf, prog_groups, required_groups,
needs_lock_syms, needs_slab_iter);
if (ret < 0)
@@ -1366,7 +1518,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
if (has_lock_filter && needs_slab_iter)
run_slab_cache_iter(skel);
- ret = get_filter_entries(evsel, skel, entry);
+ ret = get_filter_entries(evsel, skel, btf, entry);
if (ret < 0) {
pr_err("Failed to process filter entries\n");
goto err;
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index bf39128e0be1..85570798baae 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -39,6 +39,7 @@ struct contention_task_data {
*/
#define LCD_F_MMAP_LOCK (1U << 31)
#define LCD_F_SIGHAND_LOCK (1U << 30)
+#define LCD_F_RCU_LOCK (1U << 29)
#define LCB_F_SLAB_ID_SHIFT 16
#define LCB_F_SLAB_ID_START (1U << 16)
@@ -52,9 +53,11 @@ struct contention_task_data {
#define LCB_F_RT (1U << 3)
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+#define LCB_F_SRCU (1U << 6)
+#define LCB_F_RCU (1U << 7)
-#define LCB_F_TYPE_MAX (1U << 7)
-#define LCB_F_TYPE_MASK 0x0000007FU
+#define LCB_F_TYPE_MAX (1U << 8)
+#define LCB_F_TYPE_MASK 0x000000FFU
#define SLAB_NAME_MAX 28
@@ -241,6 +244,40 @@ static __always_inline __u64 get_current_siglock(__u64 curr)
return sh + bpf_core_field_offset(sighand->siglock);
}
+/*
+ * Check if `curr` is inside a preemptible RCU read-side critical section
+ * (`rcu_read_lock()`).
+ *
+ * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires
+ * `rcu_read_lock()` before invoking `perf_sample_filter` (and
+ * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for
+ * non-clock software events), `min_depth` is the baseline nesting count
+ * contributed by the perf event overflow handler itself (1 or 2).
+ */
+static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth)
+{
+ struct task_struct *task = (void *)curr;
+
+ if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting))
+ return false;
+
+ return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth;
+}
+
+/*
+ * Check if `curr` is inside a Tasks Trace RCU read-side critical section
+ * (`rcu_read_lock_trace()`).
+ */
+static __always_inline bool is_rcu_trace_lock_held(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+
+ if (!curr || !bpf_core_field_exists(task->trc_reader_nesting))
+ return false;
+
+ return BPF_CORE_READ(task, trc_reader_nesting) > 0;
+}
+
/*
* Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar.
*
@@ -273,6 +310,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags)
if (get_current_siglock(curr) == lock)
return LCD_F_SIGHAND_LOCK;
break;
+ case LCB_F_RCU:
+ case LCB_F_RCU | LCB_F_WRITE:
+ return LCD_F_RCU_LOCK;
default:
break;
}
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index cd711d1d04b6..37c108608bbc 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -99,6 +99,18 @@ enum perf_bpf_lock_target_kind {
* low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
*/
PBF_LOCK_TYPE,
+ /*
+ * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`).
+ * Inspected directly via `current->rcu_read_lock_nesting > entry.target`,
+ * where `entry.target` is the baseline RCU nesting depth added by the
+ * kernel perf overflow path.
+ */
+ PBF_LOCK_RCU,
+ /*
+ * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`).
+ * Inspected directly via `current->trc_reader_nesting > 0`.
+ */
+ PBF_LOCK_RCU_TRACE,
};
/*
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index 480ba4b49793..6f3adbbe0289 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -123,6 +123,7 @@ volatile const int use_idx_hash;
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
+const volatile __u64 rcu_state_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
@@ -136,6 +137,7 @@ int has_lock_class;
int has_lock_type;
int has_mmap_lock;
int has_siglock;
+int has_rcu_lock;
int track_rwsem_read;
__u8 lock_type_filter[LCB_F_TYPE_MAX];
@@ -270,6 +272,20 @@ static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry
case PBF_LOCK_TYPE:
return is_lock_type_in_map(entry->target, mode, pid, cpu_id);
+ case PBF_LOCK_RCU:
+ if ((mode & PBF_LOCK_MODE_READ) &&
+ is_rcu_lock_held(curr, (int)entry->target))
+ return 1;
+ if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr &&
+ is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id))
+ return 1;
+ return 0;
+
+ case PBF_LOCK_RCU_TRACE:
+ if ((mode & PBF_LOCK_MODE_READ) && is_rcu_trace_lock_held(curr))
+ return 1;
+ return 0;
+
default:
return 0;
}
@@ -601,11 +617,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla
if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
matched = true;
- if (has_mmap_lock || has_siglock) {
+ if (has_mmap_lock || has_siglock || has_rcu_lock) {
__u32 ltype = check_lock_type(lock_addr, flags);
if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
- (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
+ (has_siglock && ltype == LCD_F_SIGHAND_LOCK) ||
+ (has_rcu_lock && ltype == LCD_F_RCU_LOCK))
matched = true;
}
@@ -1247,6 +1264,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem)
return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
}
+/* SRCU acquire/release and synchronize hooks */
+SEC("fexit/__srcu_read_lock")
+int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/__srcu_read_unlock")
+int BPF_PROG(fentry_srcu_read_unlock, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fexit/__srcu_read_lock_nmisafe")
+int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/__srcu_read_unlock_nmisafe")
+int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/synchronize_srcu")
+int BPF_PROG(fentry_synchronize_srcu, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu")
+int BPF_PROG(fexit_synchronize_srcu, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_srcu_expedited")
+int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu_expedited")
+int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fentry/srcu_barrier")
+int BPF_PROG(fentry_srcu_barrier, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/srcu_barrier")
+int BPF_PROG(fexit_srcu_barrier, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+/* RCU grace-period synchronize / barrier hooks */
+SEC("fentry/synchronize_rcu")
+int BPF_PROG(fentry_synchronize_rcu)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu")
+int BPF_PROG(fexit_synchronize_rcu)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_rcu_expedited")
+int BPF_PROG(fentry_synchronize_rcu_expedited)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu_expedited")
+int BPF_PROG(fexit_synchronize_rcu_expedited)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_rcu_tasks")
+int BPF_PROG(fentry_synchronize_rcu_tasks)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu_tasks")
+int BPF_PROG(fexit_synchronize_rcu_tasks)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/rcu_barrier")
+int BPF_PROG(fentry_rcu_barrier)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/rcu_barrier")
+int BPF_PROG(fexit_rcu_barrier)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
SEC("raw_tp/bpf_test_finish")
int BPF_PROG(collect_lock_syms)
{
diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
index b75ddc6846bb..1fb91a829d54 100644
--- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
+++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
@@ -119,6 +119,8 @@ struct task_struct {
struct signal_struct *signal;
struct sighand_struct *sighand;
struct css_set *cgroups;
+ int rcu_read_lock_nesting;
+ int trc_reader_nesting;
} __attribute__((preserve_access_index));
struct trace_entry {
diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c
index a1bb78d6e29c..b45b075f3346 100644
--- a/tools/perf/util/lock-contention.c
+++ b/tools/perf/util/lock-contention.c
@@ -38,6 +38,10 @@ const struct perf_lock_type lock_type_table[] = {
{ LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
/* alias for optimistic spinning only */
{ LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
+ { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" },
+ { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" },
+ { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" },
+ { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" },
};
int parse_call_stack(const struct option *opt __maybe_unused, const char *str,
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index 68e9bd77e8a9..88b2c11244ad 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -128,6 +128,8 @@ struct thread_stat {
#define LCB_F_RT (1U << 3)
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+#define LCB_F_SRCU (1U << 6)
+#define LCB_F_RCU (1U << 7)
struct perf_lock_type {
unsigned int flags;
@@ -140,7 +142,7 @@ struct perf_lock_type {
const char *lock_name;
};
-extern const struct perf_lock_type lock_type_table[14];
+extern const struct perf_lock_type lock_type_table[18];
struct evlist;
struct machine;
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU and SRCU
2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (3 preceding siblings ...)
2026-10-17 17:14 ` [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
@ 2026-10-17 17:14 ` Ian Rogers
2026-10-17 17:30 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers
2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
6 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, rcu
Extend 'perf lock contention --use-bpf' to track RCU and SRCU
grace-period and barrier waits:
- Attach fentry/fexit programs to synchronize_rcu(),
synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(),
synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier()
when available in BTF.
- Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu',
'-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct
symbol names).
Example usage:
# perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1
total wait avg wait type caller
35.12 ms 8.78 ms rcu:W cleanup_net+0x284
7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114
# perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1
total wait avg wait address symbol
35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu)
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/tests/shell/lock_contention.sh | 14 ++
tools/perf/util/bpf_lock_contention.c | 167 +++++++++++++--
.../perf/util/bpf_skel/lock_contention.bpf.c | 200 ++++++++++++++----
tools/perf/util/machine.c | 11 +
5 files changed, 348 insertions(+), 51 deletions(-)
diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt
index c17b3e318169..bfeb98c976e3 100644
--- a/tools/perf/Documentation/perf-lock.txt
+++ b/tools/perf/Documentation/perf-lock.txt
@@ -188,11 +188,14 @@ CONTENTION OPTIONS
Show lock contention only for given lock types (comma separated list).
Available values are:
semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W,
- rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem,
- pcpu-sem:R, pcpu-sem:W, mutex
+ rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem,
+ pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W
Note that RW-variant of locks have :R and :W suffix. Names without the
suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W.
+ For RCU and SRCU with --use-bpf, contention tracks grace-period and
+ barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(),
+ srcu_barrier()).
-L::
--lock-filter=<value>::
diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh
index 5df9a0cd48a3..c6d37c910d4b 100755
--- a/tools/perf/tests/shell/lock_contention.sh
+++ b/tools/perf/tests/shell/lock_contention.sh
@@ -181,6 +181,13 @@ test_type_filter()
err=1
exit
fi
+
+ perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result}
+ if [ "$(grep -c -v rcu "${result}")" != "0" ]; then
+ echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")"
+ err=1
+ exit
+ fi
}
test_lock_filter()
@@ -224,6 +231,13 @@ test_lock_filter()
err=1
exit
fi
+
+ perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result}
+ if [ "$(grep -c -v rcu "${result}")" != "0" ]; then
+ echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")"
+ err=1
+ exit
+ fi
}
test_stack_filter()
diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c
index c20bd075664e..fa7df9187b08 100644
--- a/tools/perf/util/bpf_lock_contention.c
+++ b/tools/perf/util/bpf_lock_contention.c
@@ -16,6 +16,7 @@
#include <bpf/bpf.h>
#include <bpf/btf.h>
#include <inttypes.h>
+#include <unistd.h>
#include "bpf_skel/lock_contention.skel.h"
#include "bpf_skel/lock_data.h"
@@ -180,6 +181,129 @@ static void init_numa_data(struct lock_contention *con)
free(buf);
}
+static bool has_bpf_fentry_support(struct btf *btf)
+{
+ /*
+ * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and
+ * bpf_get_func_arg() helper availability (5.17+) used by
+ * get_lock_addr() in the SRCU fentry/fexit hooks.
+ */
+ struct bpf_insn insns[] = {
+ { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 },
+ { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3,
+ .src_reg = BPF_REG_10 },
+ { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 },
+ { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg },
+ { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 },
+ { .code = BPF_JMP | BPF_EXIT },
+ };
+ LIBBPF_OPTS(bpf_prog_load_opts, opts,
+ .expected_attach_type = BPF_TRACE_FENTRY,
+ );
+ s32 btf_id;
+ int prog_fd, link_fd;
+
+ if (!btf)
+ return false;
+
+ btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC);
+ if (btf_id <= 0)
+ return false;
+
+ opts.attach_btf_id = btf_id;
+ prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL",
+ insns, ARRAY_SIZE(insns), &opts);
+ if (prog_fd < 0)
+ return false;
+
+ link_fd = bpf_raw_tracepoint_open(NULL, prog_fd);
+ if (link_fd >= 0)
+ close(link_fd);
+ close(prog_fd);
+ return link_fd >= 0;
+}
+
+static void configure_rcu_contention_progs(struct lock_contention *con,
+ bool has_rcu_lock)
+{
+ static const struct {
+ size_t prog_offset;
+ const char *func;
+ bool is_srcu;
+ } rcu_progs[] = {
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu),
+ "synchronize_rcu", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu),
+ "synchronize_rcu", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited),
+ "synchronize_rcu_expedited", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited),
+ "synchronize_rcu_expedited", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks),
+ "synchronize_rcu_tasks", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks),
+ "synchronize_rcu_tasks", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier),
+ "rcu_barrier", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier),
+ "rcu_barrier", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu),
+ "synchronize_srcu", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu),
+ "synchronize_srcu", true },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited),
+ "synchronize_srcu_expedited", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited),
+ "synchronize_srcu_expedited", true },
+ { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier),
+ "srcu_barrier", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier),
+ "srcu_barrier", true },
+ };
+ struct symbol *sym;
+ struct map *kmap;
+ bool need_rcu = true, need_srcu = true;
+ bool fentry_ok;
+
+ sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap);
+ if (sym) {
+ skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start);
+ map__put(kmap);
+ } else {
+ skel->rodata->rcu_state_addr = 1ULL;
+ }
+
+ if (con->filters->nr_types) {
+ need_rcu = false;
+ need_srcu = false;
+ for (int i = 0; i < con->filters->nr_types; i++) {
+ if (con->filters->types[i] & LCB_F_RCU)
+ need_rcu = true;
+ if (con->filters->types[i] & LCB_F_SRCU)
+ need_srcu = true;
+ }
+ }
+
+ if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) {
+ if (!has_rcu_lock && !con->filters->nr_addrs)
+ need_rcu = false;
+ if (!con->filters->nr_addrs && !con->filters->nr_slabs)
+ need_srcu = false;
+ }
+
+ fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf);
+
+ for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) {
+ struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset;
+ bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu;
+ bool enable = fentry_ok && want &&
+ btf__find_by_name_kind(con->btf, rcu_progs[i].func,
+ BTF_KIND_FUNC) > 0;
+
+ bpf_program__set_autoload(*progp, enable);
+ }
+}
+
int lock_contention_prepare(struct lock_contention *con)
{
int i, fd;
@@ -187,6 +311,7 @@ int lock_contention_prepare(struct lock_contention *con)
struct evlist *evlist = con->evlist;
struct target *target = con->target;
bool has_mmap_lock = false;
+ bool has_rcu_lock = false;
/* make sure it loads the kernel map before lookup */
map__load(machine__kernel_map(con->machine));
@@ -200,6 +325,7 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64));
bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries);
bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries);
+ bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries);
if (con->aggr_mode == LOCK_AGGR_TASK)
bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries);
@@ -249,6 +375,10 @@ int lock_contention_prepare(struct lock_contention *con)
has_mmap_lock = true;
continue;
}
+ if (!strcmp(con->filters->syms[i], "rcu")) {
+ has_rcu_lock = true;
+ continue;
+ }
sym = machine__find_kernel_symbol_by_name(con->machine,
con->filters->syms[i],
@@ -269,7 +399,7 @@ int lock_contention_prepare(struct lock_contention *con)
addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start);
con->filters->addrs = addrs;
}
- naddrs = con->filters->nr_addrs ?: has_mmap_lock;
+ naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock);
skel->rodata->has_addr = 1;
}
@@ -305,6 +435,7 @@ int lock_contention_prepare(struct lock_contention *con)
skel->rodata->needs_callstack = con->save_callstack;
skel->rodata->lock_owner = con->owner;
skel->rodata->has_mmap_lock = has_mmap_lock;
+ skel->rodata->has_rcu_lock = has_rcu_lock;
if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) {
if (cgroup_is_v2("perf_event"))
@@ -321,6 +452,7 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map__set_max_entries(skel->maps.slab_filter, nslabs);
init_numa_data(con);
+ configure_rcu_contention_progs(con, has_rcu_lock);
if (lock_contention_bpf__load(skel) < 0) {
pr_err("Failed to load lock-contention BPF skeleton\n");
@@ -503,23 +635,30 @@ static void account_end_timestamp(struct lock_contention *con)
enum lock_aggr_mode aggr_mode = con->aggr_mode;
struct tstamp_data ts_data, *cpu_data;
- /* Iterate per-task tstamp map (key = TID) */
- ts_fd = bpf_map__fd(skel->maps.tstamp);
+ /* Iterate per-task tstamp maps (key = TID) */
+ int ts_fds[] = {
+ bpf_map__fd(skel->maps.tstamp),
+ bpf_map__fd(skel->maps.tstamp_rcu),
+ };
+
stat_fd = bpf_map__fd(skel->maps.lock_stat);
- prev_key = NULL;
- while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) {
- if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) {
- int pid = key;
+ for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) {
+ ts_fd = ts_fds[i];
+ prev_key = NULL;
+ while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) {
+ if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) {
+ int pid = key;
- if (aggr_mode == LOCK_AGGR_TASK && con->owner)
- pid = ts_data.flags;
+ if (aggr_mode == LOCK_AGGR_TASK && con->owner)
+ pid = ts_data.flags;
- update_lock_stat(stat_fd, pid, end_ts, aggr_mode,
- &ts_data);
- }
+ update_lock_stat(stat_fd, pid, end_ts, aggr_mode,
+ &ts_data);
+ }
- prev_key = &key;
+ prev_key = &key;
+ }
}
/* Now it'll check per-cpu tstamp map which doesn't have TID. */
@@ -604,6 +743,8 @@ static const char *lock_contention_get_name(struct lock_contention *con,
return "mmap_lock";
if (flags & LCD_F_SIGHAND_LOCK)
return "siglock";
+ if (flags & LCD_F_RCU_LOCK)
+ return "rcu";
/* global locks with symbols */
sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap);
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index 0f2118006005..58d45c43150b 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -59,6 +59,18 @@ struct {
__uint(max_entries, MAX_ENTRIES);
} tstamp SEC(".maps");
+/*
+ * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits
+ * so they do not collide with inner sleeping lock contentions (such as
+ * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, int);
+ __type(value, struct tstamp_data);
+ __uint(max_entries, MAX_ENTRIES);
+} tstamp_rcu SEC(".maps");
+
/* maintain per-CPU timestamp at the beginning of contention */
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
@@ -160,6 +172,7 @@ const volatile int has_addr;
const volatile int has_cgroup;
const volatile int has_slab;
const volatile int has_mmap_lock;
+const volatile int has_rcu_lock;
const volatile int needs_callstack;
const volatile int stack_skip;
const volatile int lock_owner;
@@ -172,6 +185,7 @@ const volatile int aggr_mode;
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
+const volatile __u64 rcu_state_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
@@ -218,7 +232,7 @@ static inline __u64 get_current_cgroup_id(void)
return BPF_CORE_READ(cgrp, kn, id);
}
-static inline int can_record(u64 *ctx)
+static __always_inline int can_record(__u64 lock, __u32 flags)
{
bool is_addr_ok = false;
@@ -242,7 +256,6 @@ static inline int can_record(u64 *ctx)
if (has_type) {
__u8 *ok;
- __u32 flags = (__u32)ctx[1];
ok = bpf_map_lookup_elem(&type_filter, &flags);
if (!ok)
@@ -251,10 +264,9 @@ static inline int can_record(u64 *ctx)
if (has_addr) {
__u8 *ok;
- __u64 addr = ctx[0];
- ok = bpf_map_lookup_elem(&addr_filter, &addr);
- if (!ok && !has_slab && !has_mmap_lock)
+ ok = bpf_map_lookup_elem(&addr_filter, &lock);
+ if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock)
return 0;
is_addr_ok = !!ok;
@@ -272,26 +284,27 @@ static inline int can_record(u64 *ctx)
if (is_addr_ok)
return 1;
- /* slab and mmap_lock are part of the addr_filter */
+ /* slab, mmap_lock, and rcu_lock are part of the addr_filter */
if (has_slab && bpf_get_kmem_cache) {
__u8 *ok;
- __u64 addr = ctx[0];
long kmem_cache_addr;
- kmem_cache_addr = (long)bpf_get_kmem_cache(addr);
+ kmem_cache_addr = (long)bpf_get_kmem_cache(lock);
ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr);
if (ok)
return 1;
- else if (!has_mmap_lock)
+ else if (!has_mmap_lock && !has_rcu_lock)
return 0;
}
- if (has_mmap_lock) {
- __u64 lock = ctx[0];
- __u32 flag = ctx[1];
+ if (has_mmap_lock || has_rcu_lock) {
+ __u32 ltype = check_lock_type(lock, flags);
- if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK)
- return 0;
+ if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK)
+ return 1;
+ if (has_rcu_lock && ltype == LCD_F_RCU_LOCK)
+ return 1;
+ return 0;
}
return 1;
@@ -351,6 +364,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
{
__u32 pid;
struct tstamp_data *pelem;
+ void *map;
/* Use per-cpu array map for spinlock and rwlock */
if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) {
@@ -363,8 +377,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
return pelem;
}
+ map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp;
pid = bpf_get_current_pid_tgid();
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ pelem = bpf_map_lookup_elem(map, &pid);
/* Do not update the element for nested locks */
if (pelem && pelem->lock)
return NULL;
@@ -372,12 +387,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
if (pelem == NULL) {
struct tstamp_data zero = {};
- if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) {
+ if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) {
__sync_fetch_and_add(&task_fail, 1);
return NULL;
}
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ pelem = bpf_map_lookup_elem(map, &pid);
if (pelem == NULL) {
__sync_fetch_and_add(&task_fail, 1);
return NULL;
@@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags)
}
}
-SEC("tp_btf/contention_begin")
-int contention_begin(u64 *ctx)
+static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags)
{
struct tstamp_data *pelem;
- if (!enabled || !can_record(ctx))
+ if (!enabled || !can_record(lock, flags))
return 0;
- pelem = get_tstamp_elem(ctx[1]);
+ pelem = get_tstamp_elem(flags);
if (pelem == NULL)
return 0;
pelem->timestamp = bpf_ktime_get_ns();
- pelem->lock = (__u64)ctx[0];
- pelem->flags = (__u32)ctx[1];
+ pelem->lock = lock;
+ pelem->flags = flags;
if (aggr_mode == LOCK_AGGR_CGROUP)
pelem->cgroup_id = get_current_cgroup_id();
@@ -559,8 +573,21 @@ int contention_begin(u64 *ctx)
}
}
skip_owner:
- pelem->stack_id = bpf_get_stackid(ctx, &stacks,
- BPF_F_FAST_STACK_CMP | stack_skip);
+ {
+ int skip = stack_skip;
+
+ /*
+ * Unlike tp_btf/contention_begin (which fires inside a
+ * lock slowpath via trace_contention_begin), fentry
+ * RCU/SRCU programs attach directly to the top-level
+ * wait API, which is 2 frames shallower.
+ */
+ if (flags & (LCB_F_RCU | LCB_F_SRCU))
+ skip = stack_skip > 2 ? stack_skip - 2 : 0;
+
+ pelem->stack_id = bpf_get_stackid(ctx, &stacks,
+ BPF_F_FAST_STACK_CMP | skip);
+ }
if (pelem->stack_id < 0)
__sync_fetch_and_add(&stack_fail, 1);
} else if (aggr_mode == LOCK_AGGR_TASK) {
@@ -588,8 +615,7 @@ int contention_begin(u64 *ctx)
return 0;
}
-SEC("tp_btf/contention_end")
-int contention_end(u64 *ctx)
+static __always_inline int do_contention_end(__u64 lock, __s32 ret)
{
__u32 pid = 0, idx = 0;
struct tstamp_data *pelem;
@@ -597,7 +623,7 @@ int contention_end(u64 *ctx)
struct contention_data *data;
__u64 timestamp;
__u64 duration;
- bool need_delete = false;
+ void *ts_map = NULL;
if (!enabled)
return 0;
@@ -613,14 +639,19 @@ int contention_end(u64 *ctx)
*/
pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx);
if (pelem && pelem->lock) {
- if (pelem->lock != ctx[0])
+ if (pelem->lock != lock)
return 0;
} else {
pid = bpf_get_current_pid_tgid();
pelem = bpf_map_lookup_elem(&tstamp, &pid);
- if (!pelem || pelem->lock != ctx[0])
- return 0;
- need_delete = true;
+ if (pelem && pelem->lock == lock) {
+ ts_map = &tstamp;
+ } else {
+ pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid);
+ if (!pelem || pelem->lock != lock)
+ return 0;
+ ts_map = &tstamp_rcu;
+ }
}
timestamp = bpf_ktime_get_ns();
@@ -648,7 +679,6 @@ int contention_end(u64 *ctx)
*/
else {
u32 i = 0;
- s32 ret = (s32)ctx[1];
u64 *buf;
otdata->timestamp = timestamp;
@@ -713,7 +743,7 @@ int contention_end(u64 *ctx)
if (lock_owner)
key.pid = pelem->flags;
else {
- if (!need_delete)
+ if (!ts_map)
pid = bpf_get_current_pid_tgid();
key.pid = pid;
}
@@ -795,11 +825,109 @@ int contention_end(u64 *ctx)
check_lock_delay(pelem->lock);
pelem->lock = 0;
- if (need_delete)
- bpf_map_delete_elem(&tstamp, &pid);
+ if (ts_map)
+ bpf_map_delete_elem(ts_map, &pid);
return 0;
}
+SEC("tp_btf/contention_begin")
+int contention_begin(u64 *ctx)
+{
+ return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]);
+}
+
+SEC("tp_btf/contention_end")
+int contention_end(u64 *ctx)
+{
+ return do_contention_end((__u64)ctx[0], (__s32)ctx[1]);
+}
+
+/* RCU grace-period synchronize / barrier contention hooks */
+SEC("fentry/synchronize_rcu")
+int BPF_PROG(fentry_synchronize_rcu)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu")
+int BPF_PROG(fexit_synchronize_rcu)
+{
+ return do_contention_end(rcu_state_addr, 0);
+}
+
+SEC("fentry/synchronize_rcu_expedited")
+int BPF_PROG(fentry_synchronize_rcu_expedited)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu_expedited")
+int BPF_PROG(fexit_synchronize_rcu_expedited)
+{
+ return do_contention_end(rcu_state_addr, 0);
+}
+
+SEC("fentry/synchronize_rcu_tasks")
+int BPF_PROG(fentry_synchronize_rcu_tasks)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu_tasks")
+int BPF_PROG(fexit_synchronize_rcu_tasks)
+{
+ return do_contention_end(rcu_state_addr, 0);
+}
+
+SEC("fentry/rcu_barrier")
+int BPF_PROG(fentry_rcu_barrier)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/rcu_barrier")
+int BPF_PROG(fexit_rcu_barrier)
+{
+ return do_contention_end(rcu_state_addr, 0);
+}
+
+/* SRCU grace-period synchronize / barrier contention hooks */
+SEC("fentry/synchronize_srcu")
+int BPF_PROG(fentry_synchronize_srcu, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu")
+int BPF_PROG(fexit_synchronize_srcu, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0);
+}
+
+SEC("fentry/synchronize_srcu_expedited")
+int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu_expedited")
+int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0);
+}
+
+SEC("fentry/srcu_barrier")
+int BPF_PROG(fentry_srcu_barrier, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/srcu_barrier")
+int BPF_PROG(fexit_srcu_barrier, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0);
+}
+
SEC("raw_tp/bpf_test_finish")
int BPF_PROG(end_timestamp)
{
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 1d9d3bf57720..0a0f223b3335 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -3379,6 +3379,17 @@ bool machine__is_lock_function(struct machine *machine, u64 addr)
return true;
}
+ {
+ struct symbol *sym = machine__find_kernel_symbol(machine, addr, NULL);
+
+ if (sym && (!strncmp(sym->name, "synchronize_rcu", 15) ||
+ !strncmp(sym->name, "synchronize_srcu", 16) ||
+ !strcmp(sym->name, "__synchronize_srcu") ||
+ !strcmp(sym->name, "rcu_barrier") ||
+ !strcmp(sym->name, "srcu_barrier")))
+ return true;
+ }
+
return false;
}
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat
2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (4 preceding siblings ...)
2026-10-17 17:14 ` [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
@ 2026-10-17 17:14 ` Ian Rogers
2026-10-17 17:17 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
6 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, rcu
In report_lock_contention_begin_event(), lock_stat_findnew() was
called before checking filters.nr_types (-Y), filters.nr_addrs (-L),
and match_callstack_filter() (-S).
When a filtered-out lock event (such as a mutex contention when
running with -Y spinlock) shared the same aggregation key as a
subsequent matching event (for example, contention on the mutex's
internal wait_lock spinlock from the same caller), the first event
created the lock_stat entry with the filtered-out lock's flags and the
second event reused that entry, causing non-spinlock entries to appear
in -Y spinlock output.
Check the type, address, and callstack filters before looking up or
creating the lock_stat entry.
Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/builtin-lock.c | 61 +++++++++++++++++++++------------------
1 file changed, 33 insertions(+), 28 deletions(-)
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index feaf1cfd08cf..a031709a2cc2 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -975,6 +975,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
struct machine *machine = &session->machines.host;
struct map *kmap;
struct symbol *sym;
+ u64 *callstack = NULL;
ret = get_key_by_aggr_mode(&key, addr, sample);
if (ret < 0)
@@ -1010,33 +1011,6 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
}
}
- ls = lock_stat_find(key);
- if (!ls) {
- char buf[128];
- const char *name = "";
-
- switch (aggr_mode) {
- case LOCK_AGGR_ADDR:
- sym = machine__find_kernel_symbol(machine, key, &kmap);
- if (sym)
- name = sym->name;
- break;
- case LOCK_AGGR_CALLER:
- name = buf;
- if (lock_contention_caller(sample, buf, sizeof(buf)) < 0)
- name = "Unknown";
- break;
- case LOCK_AGGR_CGROUP:
- case LOCK_AGGR_TASK:
- default:
- break;
- }
-
- ls = lock_stat_findnew(key, name, flags);
- if (!ls)
- return -ENOMEM;
- }
-
if (filters.nr_types) {
bool found = false;
@@ -1066,7 +1040,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
}
if (needs_callstack()) {
- u64 *callstack = get_callstack(sample, max_stack_depth);
+ callstack = get_callstack(sample, max_stack_depth);
if (callstack == NULL)
return 0;
@@ -1074,7 +1048,38 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
free(callstack);
return 0;
}
+ }
+
+ ls = lock_stat_find(key);
+ if (!ls) {
+ char buf[128];
+ const char *name = "";
+
+ switch (aggr_mode) {
+ case LOCK_AGGR_ADDR:
+ sym = machine__find_kernel_symbol(machine, key, &kmap);
+ if (sym)
+ name = sym->name;
+ break;
+ case LOCK_AGGR_CALLER:
+ name = buf;
+ if (lock_contention_caller(sample, buf, sizeof(buf)) < 0)
+ name = "Unknown";
+ break;
+ case LOCK_AGGR_CGROUP:
+ case LOCK_AGGR_TASK:
+ default:
+ break;
+ }
+
+ ls = lock_stat_findnew(key, name, flags);
+ if (!ls) {
+ free(callstack);
+ return -ENOMEM;
+ }
+ }
+ if (callstack) {
if (ls->callstack == NULL)
ls->callstack = callstack;
else
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* Re: [PATCH v2 3/6] perf test: Add tests for lock_held() BPF sample filter
2026-10-17 17:14 ` [PATCH v2 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers
@ 2026-10-17 17:17 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-17 17:17 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Add unit tests in parse-events.c for parsing lock_held(),
> lock_held_read(), and lock_held_exclusive() BPF filter expressions
> (standalone predicates, negated predicates, and compound group
> expressions) and extend record_bpf_filter.sh with lock_held(),
> lock_held_read()/lock_held_write(), and ':R'/':W' recording tests.
>
> Example output:
>
> # perf test -v 'Parsing of BPF sample filters'
> 5: Parse event definition strings :
> 5.7: Parsing of BPF sample filters : Ok
>
> # perf test -v 'perf record sample filtering (by BPF) tests'
> 112: perf record sample filtering (by BPF) tests :
> Checking BPF-filter privilege
> Basic bpf-filter test
> Basic bpf-filter test [Success]
> Failing bpf-filter test
> Failing bpf-filter test [Success]
> Group bpf-filter test
> Group bpf-filter test [Success]
> Multiple bpf-filter test
> Multiple bpf-filter test [Success]
> Cgroup bpf-filter test
> Cgroup bpf-filter test [Success]
> Lock-held bpf-filter test
> Lock-held bpf-filter test [Success]
> 112: perf record sample filtering (by BPF) tests : Ok
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017171437.2066343-1-irogers@google.com?part=3
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat
2026-10-17 17:14 ` [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers
@ 2026-10-17 17:17 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-17 17:17 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> In report_lock_contention_begin_event(), lock_stat_findnew() was
> called before checking filters.nr_types (-Y), filters.nr_addrs (-L),
> and match_callstack_filter() (-S).
>
> When a filtered-out lock event (such as a mutex contention when
> running with -Y spinlock) shared the same aggregation key as a
> subsequent matching event (for example, contention on the mutex's
> internal wait_lock spinlock from the same caller), the first event
> created the lock_stat entry with the filtered-out lock's flags and the
> second event reused that entry, causing non-spinlock entries to appear
> in -Y spinlock output.
>
> Check the type, address, and callstack filters before looking up or
> creating the lock_stat entry.
>
> Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option")
> Assisted-by: Antigravity:gemini-3.1-pro
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017171437.2066343-1-irogers@google.com?part=6
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
2026-10-17 17:14 ` [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
@ 2026-10-17 17:25 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-17 17:25 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks
> Trace RCU, and SRCU locks:
>
> - 'rcu' (supports both read/shared and write/exclusive modes):
> - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)',
> 'lock_held(rcu_read_lock)'): directly inspects
> current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on
> CONFIG_PREEMPT_RCU kernels), accounting for the baseline
> rcu_read_lock() nesting acquired by the perf overflow handler itself
> (bpf_overflow_handler() and do_perf_sw_event()) with zero extra
> fentry/fexit tracing hooks.
> - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'):
> attaches fentry/fexit hooks to synchronize_rcu(),
> synchronize_rcu_expedited(), synchronize_rcu_tasks(), and
> rcu_barrier() to track grace-period and barrier waits.
> [ ... ]
>
> # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1
> # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017171437.2066343-1-irogers@google.com?part=4
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU and SRCU
2026-10-17 17:14 ` [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
@ 2026-10-17 17:30 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-17 17:30 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf lock: rcu_state_addr collides with rcu_state.node[0].lock in BPF
--
commit 66aefea7c5ca7c7c5ee421770f454d75f5c24b84
Author: Ian Rogers <irogers@google.com>
perf lock contention: Extend BPF lock contention to RCU and SRCU
This commit extends 'perf lock contention --use-bpf' to track RCU and SRCU
grace-period and barrier waits using fentry/fexit BPF programs and adds
support for filtering by RCU and SRCU lock types and names.
> diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c
> index c20bd075664e1..fa7df9187b08b 100644
> --- a/tools/perf/util/bpf_lock_contention.c
> +++ b/tools/perf/util/bpf_lock_contention.c
[ ... ]
> @@ -180,6 +181,129 @@ static void init_numa_data(struct lock_contention *con)
[ ... ]
> +static void configure_rcu_contention_progs(struct lock_contention *con,
> + bool has_rcu_lock)
> +{
[ ... ]
> + sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap);
> + if (sym) {
> + skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start);
> + map__put(kmap);
> + } else {
> + skel->rodata->rcu_state_addr = 1ULL;
> + }
[ ... ]
> diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
> index 0f2118006005a..58d45c43150b5 100644
> --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
> +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
[ ... ]
> @@ -613,14 +639,19 @@ int contention_end(u64 *ctx)
> */
> pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx);
> if (pelem && pelem->lock) {
> - if (pelem->lock != ctx[0])
> + if (pelem->lock != lock)
> return 0;
> } else {
> pid = bpf_get_current_pid_tgid();
> pelem = bpf_map_lookup_elem(&tstamp, &pid);
> - if (!pelem || pelem->lock != ctx[0])
> - return 0;
> - need_delete = true;
> + if (pelem && pelem->lock == lock) {
> + ts_map = &tstamp;
> + } else {
> + pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid);
> + if (!pelem || pelem->lock != lock)
> + return 0;
> + ts_map = &tstamp_rcu;
> + }
> }
[Severity: Medium]
Can setting rcu_state_addr to the start address of rcu_state and sharing
do_contention_end() between tp_btf/contention_end and the RCU/SRCU fexit
hooks cause lock address collisions and corrupted wait tracking?
In kernel/rcu/tree.h, struct rcu_state begins at offset 0 with
node[NUM_RCU_NODES], and struct rcu_node begins at offset 0 with its
raw_spinlock_t lock:
kernel/rcu/tree.h:
struct rcu_node {
raw_spinlock_t __private lock;
...
};
struct rcu_state {
struct rcu_node node[NUM_RCU_NODES];
...
};
Because of this layout, &rcu_state.node[0].lock has the exact same address
as rcu_state (rcu_state_addr). Together with do_contention_end() not
distinguishing whether its caller is tp_btf/contention_end or an RCU/SRCU
fexit program, three issues can occur:
1. When filtering for RCU (for example with '-Y rcu', '-Y rcu:W', or
'-L rcu'), a task executing synchronize_rcu(), synchronize_rcu_expedited(),
or rcu_barrier() records an active entry in tstamp_rcu[pid] with
pelem->lock == rcu_state_addr. If that task contends on
&rcu_state.node[0].lock during grace-period initialization or waiting,
tp_btf/contention_begin(lock = rcu_state_addr, flags = LCB_F_SPIN) is
rejected by can_record(), leaving tstamp_cpu empty. When the spinlock is
acquired and tp_btf/contention_end(lock = rcu_state_addr) fires,
do_contention_end() falls through tstamp_cpu and tstamp into
tstamp_rcu[pid], matches pelem->lock == rcu_state_addr, records the short
spinlock wait duration as the RCU wait, and deletes tstamp_rcu[pid] before
the fexit hook runs.
2. If an RCU or SRCU fexit program runs on a CPU where tstamp_cpu happens
to have an active spinlock entry (pelem->lock != 0 and pelem->lock != lock),
do_contention_end() returns 0 at the tstamp_cpu check without looking up or
deleting tstamp_rcu[pid]. That leaves a stale entry in tstamp_rcu[pid] that
blocks subsequent RCU/SRCU waits for that task in get_tstamp_elem() and
charges end_ts - pelem->timestamp in account_end_timestamp().
3. In LOCK_AGGR_ADDR mode ('perf lock con -a -b -l'), both
do_contention_end() and lock_contention_read() key lock_stat entries by
lock address:
tools/perf/util/bpf_skel/lock_contention.bpf.c:do_contention_end()
case LOCK_AGGR_ADDR:
key.lock_addr_or_cgroup = pelem->lock;
Because &rcu_state.node[0].lock and rcu_state_addr share the same address,
root rcu_node spinlock contentions (LCB_F_SPIN) and RCU grace-period waits
(LCB_F_RCU | LCB_F_WRITE) merge into a single lock_stat entry under
whichever lock type was recorded first.
Would it make sense to pass a distinct synthetic address or offset for RCU
waits (or disambiguate the lock key), and pass an explicit flag to
do_contention_end() so that RCU/SRCU fexit hooks only inspect tstamp_rcu
while tp_btf/contention_end only inspects tstamp_cpu and tstamp?
[ ... ]
> @@ -795,11 +825,109 @@ int contention_end(u64 *ctx)
> check_lock_delay(pelem->lock);
>
> pelem->lock = 0;
> - if (need_delete)
> - bpf_map_delete_elem(&tstamp, &pid);
> + if (ts_map)
> + bpf_map_delete_elem(ts_map, &pid);
> return 0;
> }
>
> +SEC("tp_btf/contention_begin")
> +int contention_begin(u64 *ctx)
> +{
> + return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]);
> +}
> +
> +SEC("tp_btf/contention_end")
> +int contention_end(u64 *ctx)
> +{
> + return do_contention_end((__u64)ctx[0], (__s32)ctx[1]);
> +}
> +
> +/* RCU grace-period synchronize / barrier contention hooks */
> +SEC("fentry/synchronize_rcu")
> +int BPF_PROG(fentry_synchronize_rcu)
> +{
> + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
> +}
> +
> +SEC("fexit/synchronize_rcu")
> +int BPF_PROG(fexit_synchronize_rcu)
> +{
> + return do_contention_end(rcu_state_addr, 0);
> +}
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017171437.2066343-1-irogers@google.com?part=5
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
@ 2026-10-17 17:31 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-17 17:31 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf lock: get_lock_owner() conflates waiter and owner rwsem flags
--
commit bc2b5a2f6b16ec842dcc1aeeebcdc72f657fd11b
Author: Ian Rogers <irogers@google.com>
perf lock contention: Share common BPF and lock type definitions in lock_data.h
This commit moves lock_type_table, LCB_F_* flags, CO-RE structs, and common
BPF lock helpers into tools/perf/util/bpf_skel/lock_data.h so they can be
shared with BPF sample filtering. It also extends get_lock_owner() to support
rt_mutex and check writer-ownership and handoff flags.
> diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
> index 652e114e6b877..bf39128e0be15 100644
> --- a/tools/perf/util/bpf_skel/lock_data.h
> +++ b/tools/perf/util/bpf_skel/lock_data.h
> @@ -76,4 +84,320 @@ struct slab_cache_data {
[ ... ]
> +static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
> +{
> + struct task_struct *task;
> + __u64 owner = 0;
> +
> + if (flags & LCB_F_MUTEX) {
> + struct mutex *mutex = (void *)lock;
> +
> + if (bpf_core_field_exists(mutex->owner.counter)) {
> + owner = BPF_CORE_READ(mutex, owner.counter);
> + if (owner & 0x4UL)
> + return NULL;
> + } else if (bpf_core_type_exists(struct rt_mutex_base)) {
> + struct rt_mutex_base *rtm = (void *)lock;
> +
> + owner = (__u64)BPF_CORE_READ(rtm, owner);
> + return (void *)(owner & ~1UL);
> + }
> + } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
> + struct rw_semaphore___new *rwsem = (void *)lock;
> +
> + if (flags == LCB_F_WRITE && bpf_core_field_exists(rwsem->count.counter)) {
> + __u64 count = BPF_CORE_READ(rwsem, count.counter);
> +
> + /* Must have RWSEM_WRITER_LOCKED (bit 0) set */
> + if (!(count & 1UL))
> + return NULL;
> + }
> + /*
> + * Support for the BPF_TYPE_MATCHES argument to the
> + * __builtin_preserve_type_info builtin was added at some point during
> + * development of clang 15 and it's what is needed for
> + * bpf_core_type_matches.
> + */
> +#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
> + if (bpf_core_type_matches(struct rw_semaphore___old)) {
> + struct rw_semaphore___old *old = (void *)lock;
> +
> + owner = (unsigned long)BPF_CORE_READ(old, owner);
> + } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
> + owner = BPF_CORE_READ(rwsem, owner.counter);
> + }
> +#else
> + /* assume new struct */
> + if (bpf_core_field_exists(rwsem->owner.counter))
> + owner = BPF_CORE_READ(rwsem, owner.counter);
> +#endif
> + /* RWSEM_READER_OWNED is bit 0 of owner */
> + if (flags == LCB_F_WRITE && (owner & 1UL))
> + return NULL;
[Severity: Medium]
In tools/perf/util/bpf_skel/lock_contention.bpf.c, contention_begin() calls
get_lock_owner(pelem->lock, pelem->flags) for both owner stack tracing and
LOCK_AGGR_TASK owner PID attribution:
tools/perf/util/bpf_skel/lock_contention.bpf.c:contention_begin() {
...
task = get_lock_owner(pelem->lock, pelem->flags);
if (!task)
goto skip_owner;
...
if (lock_owner) {
task = get_lock_owner(pelem->lock, pelem->flags);
...
}
Here pelem->flags comes from trace_contention_begin() and represents the
contending waiter's mode (LCB_F_READ in rwsem_down_read_slowpath() or
LCB_F_WRITE in rwsem_down_write_slowpath()), rather than the lock owner's
mode.
When a writer contends on a reader-owned rw_semaphore, pelem->flags is
LCB_F_WRITE while rwsem->count does not have RWSEM_WRITER_LOCKED (bit 0) set
and rwsem->owner has RWSEM_READER_OWNED (bit 0) set. Conversely, when a
reader contends (flags == LCB_F_READ), both checks are bypassed.
Will checking flags == LCB_F_WRITE in get_lock_owner() cause it to always
return NULL whenever a writer contends on a reader-owned rw_semaphore,
dropping owner attribution and owner callstacks in perf lock contention -b -o?
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017171437.2066343-1-irogers@google.com?part=1
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter
2026-10-17 17:14 ` [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers
@ 2026-10-17 17:35 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-17 17:35 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] perf bpf-filter: rwlock read_count leak in record_lock_acquired_owner()
- [Medium] perf bpf-filter: missing error return on node_data lookup in init_numa_data()
--
commit 782c7a3aaec8d457ca5bf045258b33f6bbfbcae8
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Add lock_held() sample filter
This commit adds BPF sample filter predicates lock_held(<lock>),
lock_held_read(<lock>) / lock_held_shared(<lock>), and
lock_held_write(<lock>) / lock_held_exclusive(<lock>) to record perf event
samples only when a specified kernel lock is held. It evaluates lock
ownership directly in kernel memory where possible and falls back to a
held_locks BPF hash map populated by fentry/fexit hooks on kernel
lock/unlock functions.
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 72e83ac1d6a4b..2f4c6bd85f89f 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
[ ... ]
> @@ -155,9 +171,814 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
[ ... ]
> +static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf)
> +{
> + u64 addr = 0;
> + char *buf = NULL, *p;
> + size_t len;
> + long last = -1;
> + int ret;
> +
> + if (!btf)
> + return;
> +
> + ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT);
> + if (ret < 0)
> + return;
> +
> + ret = btf__resolve_size(btf, ret);
> + if (ret < 0)
> + return;
> +
> + skel->rodata->sizeof_zone = ret;
> +
> + if (find_kernel_lock_sym("contig_page_data", &addr) == 0) {
> + skel->rodata->contig_page_data_addr = addr;
> + return;
> + }
> +
> + if (find_kernel_lock_sym("node_data", &addr) == 0)
> + skel->rodata->node_data_addr = addr;
> +
> + if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) {
> + p = buf;
> + while (p && *p) {
> + char *end;
> +
> + last = strtol(p, &end, 0);
> + if (end == p)
> + break;
> + p = end;
> + if (*p == ',' || *p == '-' || *p == '\n')
> + p++;
> + }
> + skel->rodata->nr_nodes = last + 1;
> + free(buf);
> + }
> +}
[Severity: Medium]
What happens in init_numa_data() when find_kernel_lock_sym("node_data",
&addr) fails (for example, when /proc/kallsyms zeroes symbol addresses
under kptr_restrict for a non-root user with CAP_BPF)?
Unlike init_numa_data() in tools/perf/util/bpf_lock_contention.c, this does
not return early on failure, so skel->rodata->node_data_addr remains 0
while skel->rodata->nr_nodes is populated from devices/system/node/online.
When collect_lock_syms later calls do_collect_lock_syms() ->
do_collect_zone_lock() in tools/perf/util/bpf_skel/lock_data.h:
} else if (nr_nodes > 0) {
struct pglist_data **node_data = (void *)(long)node_data_addr;
for (int i = 0; i < nr_nodes; i++) {
struct pglist_data *pgdat = NULL;
int err;
err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
will this attempt to read from kernel address 0x0 and leave lock_syms
unpopulated without reporting an error for lock_held(zone_lock)?
[ ... ]
> diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
> index e5666d4c17228..480ba4b49793d 100644
> --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
> +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
[ ... ]
> @@ -295,4 +525,739 @@ int perf_sample_filter(void *ctx)
[ ... ]
> +static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id)
> +{
[ ... ]
> + existing = bpf_map_lookup_elem(&held_locks, &addr_key);
> + if (existing) {
> + /* Non-reader locks are not recursive; avoid double-counting */
> + if (!is_read && existing->write_count > 0)
> + return 0;
> + if (is_read)
> + __sync_fetch_and_add(&existing->read_count, 1);
> + else
> + __sync_fetch_and_add(&existing->write_count, 1);
> + } else {
> + struct held_lock_val first = {
> + .read_count = is_read ? 1 : 0,
> + .write_count = is_read ? 0 : 1,
> + .flags = flags,
> + };
> +
> + if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0)
> + return 0;
> + }
> +
> + if (class_id)
> + update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, flags, true);
> + if (kmem_cache_addr) {
> + update_held_lock_count((__u64)kmem_cache_addr, owner_id,
> + PBF_LOCK_SLAB, flags, true);
> + }
> + if (match_type)
> + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, true);
> +
> + return 0;
> +}
[Severity: Medium]
Can record_lock_acquired_owner() leak read_count in the held_locks map when
rwlocks are acquired via read_lock_irq() or read_lock_bh() on kernels with
CONFIG_GENERIC_LOCKBREAK=y and CONFIG_DEBUG_LOCK_ALLOC=n?
In kernel/locking/spinlock.c, BUILD_LOCK_OPS(read, rwlock, __acquires_shared)
implements __raw_read_lock_irq() and __raw_read_lock_bh() by calling the
noinline function _raw_read_lock_irqsave():
static void __lockfunc __raw_##op##_lock_irq(locktype##_t *lock) \
lock_ctx_op(lock) \
{ \
_raw_##op##_lock_irqsave(lock); \
} \
\
static void __lockfunc __raw_##op##_lock_bh(locktype##_t *lock) \
lock_ctx_op(lock) \
{ \
unsigned long flags; \
flags = _raw_##op##_lock_irqsave(lock); \
local_bh_disable(); \
local_irq_restore(flags); \
}
Since LOCK_PROG_RWLOCK_READ attaches fexit hooks to _raw_read_lock_irq,
_raw_read_lock_bh, and _raw_read_lock_irqsave, a single read_lock_irq() or
read_lock_bh() call fires both the inner fexit_raw_read_lock_irqsave hook
and the outer fexit_raw_read_lock_irq / fexit_raw_read_lock_bh hook.
Because is_read is true for LCB_F_SPIN | LCB_F_READ, the nested-call check
(!is_read && existing->write_count > 0) is bypassed and read_count is
incremented twice (for PBF_LOCK_ADDR as well as any active PBF_LOCK_SLAB or
PBF_LOCK_TYPE entries). On release, _raw_read_unlock_irq() and
_raw_read_unlock_bh() do not call _raw_read_unlock_irqrestore(), so
record_lock_released() only runs once, leaving read_count permanently
elevated in held_locks.
[ ... ]
> +/* RWlock acquire/release hooks */
> +SEC("fexit/_raw_read_lock")
> +int BPF_PROG(fexit_raw_read_lock, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
> +}
> +
> +SEC("fexit/_raw_read_lock_irq")
> +int BPF_PROG(fexit_raw_read_lock_irq, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
> +}
> +
> +SEC("fexit/_raw_read_lock_irqsave")
> +int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
> +}
> +
> +SEC("fexit/_raw_read_lock_bh")
> +int BPF_PROG(fexit_raw_read_lock_bh, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
> +}
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017171437.2066343-1-irogers@google.com?part=2
^ permalink raw reply [flat|nested] 106+ messages in thread
* [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support
2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (5 preceding siblings ...)
2026-10-17 17:14 ` [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers
@ 2026-10-18 5:37 ` Ian Rogers
2026-10-18 5:37 ` [PATCH v3 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
` (7 more replies)
6 siblings, 8 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-18 5:37 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, rcu
This series adds a new BPF sample filter predicate, `lock_held(<lock>)`
(along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`,
`lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W`
mode suffixes), to `perf record --filter`, allowing perf event samples to
be recorded only when a specified kernel lock is held (or not held with
`!`) by the running kernel thread or CPU. It also extends both
`lock_held()` and `perf lock contention --use-bpf` to support RCU and
SRCU, and fixes a pre-existing `lock_stat` filtering bug in `perf lock
contention`.
For example:
# perf record -a -g -e cycles --filter 'lock_held(&cgroup_mutex)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' -- ...
# perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ...
# perf lock contention -a -b -Y rcu,srcu -- sleep 1
An important use-case is detecting memory allocations or direct memory
reclaim occurring while kernel locks are held, which can stall lock
waiters for extended periods under memory pressure:
# Direct memory reclaim triggered while holding any mutex or rwsem:
# perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \
--filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
# Page allocations permitted to enter direct reclaim (__GFP_DIRECT_RECLAIM)
# while holding a mutex or rwsem:
# perf record -a -g -e kmem:mm_page_alloc \
--filter 'gfp_flags & 0x400' \
--filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
# Major page faults occurring while holding a mutex or rwsem:
# perf record -a -g -e major-faults \
--filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
Supported `<lock>` specifications match `perf lock contention`:
- Global/static kernel lock symbols (with or without leading `&`, e.g.
`lock_held(&cgroup_mutex)` or `lock_held(cgroup_mutex)`)
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`,
`lock_held_read(&dentry)`)
- Well-known per-process, semi-global, and RCU locks: `mmap_lock`,
`siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`
- Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`,
`rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`,
`percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`,
`rcu:W`, `semaphore`
Limitations and weaknesses in determining whether a lock is held:
- Inlined lock/unlock operations:
- When the kernel is configured to inline spinlock or rwlock operations
(`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`,
`CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`,
`CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` /
`_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are
inlined into callers and have no BTF function entry points for
`fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that
both acquire and release entry points exist in BTF before enabling a
lock family.
- Fast-path `percpu_down_read()` / `percpu_up_read()` and
`srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined
per-CPU counter increments without function calls, so only
`percpu_down_write()` / `percpu_up_write()` and out-of-line
`__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked.
- On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to
`preempt_disable()` and does not maintain
`current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)`
read-side inspection requires `CONFIG_PREEMPT_RCU`.
- Lack of an inspectable owner field on non-exclusive / spinning locks:
- While `struct mutex`, `struct rt_mutex`, and write-locked
`struct rw_semaphore` store the owning `task_struct *` in `lock->owner`
(allowing `perf_sample_filter` to inspect kernel memory directly on
sample overflow when filtering by lock address), spinlocks, rwlocks,
semaphores, SRCU, and reader-locked `rw_semaphore`s (where
`__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`)
do not maintain an authoritative per-task owner pointer.
- Tracking these locks (as well as slab-cache and lock-type filters)
relies on `fexit` (acquire) and `fentry` (release) BPF hooks
maintaining the `held_locks` BPF hash map:
- Any such lock already held before `perf record` attaches its BPF
programs will not be present in `held_locks` until it is released
and re-acquired.
- The attached `fentry`/`fexit` hooks execute on every kernel lock
acquire and release for the hooked lock family across the system,
adding tracing overhead outside of perf sample handling.
Patch overview:
1. "perf lock contention: Share common BPF and lock type definitions in
lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and
`LCB_F_*` flags, CO-RE structs, and common BPF lock helpers
(`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`,
`get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`,
`do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`.
2. "perf bpf-filter: Add lock_held() sample filter": Implements
`lock_held()`, `lock_held_read()`/`lock_held_shared()`,
`lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode
suffixes using direct lock owner inspection where possible (`mutex`,
`rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF map
updated by selectively attached `fentry`/`fexit` hooks, supporting
both PMU/software events (`SEC("perf_event")`) and tracepoint events
(`SEC("tracepoint")`).
3. "perf test: Add tests for lock_held() BPF sample filter": Adds unit
tests in `tests/parse-events.c` and shell tests in
`tests/shell/record_bpf_filter.sh`.
4. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU": Supports `rcu` (`rcu:R` via direct
`current->rcu_read_lock_nesting` inspection with zero tracing hooks;
`rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks),
`rcu-trace` (via `current->trc_reader_nesting`), and `srcu`
(`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W`
via `synchronize_srcu*()` / `srcu_barrier()`).
5. "perf lock contention: Extend BPF lock contention to RCU and SRCU":
Extends `perf lock contention --use-bpf` to track RCU and SRCU
grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`).
6. "perf lock contention: Filter events before creating lock_stat":
Fixes `report_lock_contention_begin_event()` in `builtin-lock.c` to
apply `-Y`, `-L`, and `-S` filters before calling
`lock_stat_findnew()`.
Changes in v3:
- Move the writer-only `rw_semaphore` check out of `get_lock_owner()` in
`lock_data.h` into `is_rwsem_writer_owner()` in `sample_filter.bpf.c`
so `perf lock contention -b -o` still reports reader owners when a
writer contends on a reader-owned `rw_semaphore`, and scope
`is_direct_lock_owner()` by lock type (`PBF_LOCK_DIRECT_MUTEX`,
`PBF_LOCK_DIRECT_RWSEM`, `PBF_LOCK_DIRECT_RTMUTEX`).
- Add `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF
sample filters (`lock_held()`, `uid`, `gid`, etc.) can attach to
`PERF_TYPE_TRACEPOINT` events (such as `kmem:mm_page_alloc` and
`vmscan:mm_vmscan_direct_reclaim_begin`), check `libbpf_get_error()`
when attaching, and document memory allocation / direct reclaim
filtering with `lock_held()`.
- Return early in `init_numa_data()` in `bpf-filter.c` when `node_data`
cannot be resolved and check `node_data_addr != 0` in
`do_collect_zone_lock()`.
- Avoid double-counting `_raw_read_lock_irq` / `_raw_read_lock_bh` in
`sample_filter.bpf.c` on `CONFIG_GENERIC_LOCKBREAK=y &&
!CONFIG_DEBUG_LOCK_ALLOC` kernels where they call
`_raw_read_lock_irqsave()`.
- Offset `rcu_state_addr` by 1 byte so it is distinct from
`&rcu_state.node[0].lock` at offset 0 of `rcu_state`, and pass `bool
is_rcu` to `do_contention_end()` in `lock_contention.bpf.c` so
`tp_btf/contention_end` and RCU/SRCU `fexit` hooks never cross-match
timestamp maps.
Changes in v2:
- Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists`
and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE
relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels.
- Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points
(`_mutex_lock_nest_lock`, `_mutex_lock_killable`,
`_mutex_trylock_nest_lock`, `_down_write_nest_lock`,
`_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to
`sample_filter.bpf.c` and `lock_progs[]`.
- Verify in `supported_lock_prog_groups()` that both acquire and all
unlock BTF entry points exist for a lock family before enabling its
tracing hooks, preventing acquire-only attachment when spinlock/rwlock
unlock functions are inlined.
- Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when
a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`).
- Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write`
on `track_rwsem_read` and verify non-zero read/write counts in
`record_lock_released_addr()` before decrementing aggregate slab/type
counts.
- Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in
`lock_contention.bpf.c` and recognize `synchronize_rcu*`,
`synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in
`machine__is_lock_function()`.
- Add patch 6 fixing `report_lock_contention_begin_event()` to check
type, address, and callstack filters before creating `lock_stat`
entries.
Assisted-by: Antigravity:gemini-3.1-pro
Ian Rogers (6):
perf lock contention: Share common BPF and lock type definitions in
lock_data.h
perf bpf-filter: Add lock_held() sample filter
perf test: Add tests for lock_held() BPF sample filter
perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU
perf lock contention: Extend BPF lock contention to RCU and SRCU
perf lock contention: Filter events before creating lock_stat
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/Documentation/perf-record.txt | 64 +-
tools/perf/builtin-lock.c | 88 +-
tools/perf/tests/parse-events.c | 121 ++
tools/perf/tests/shell/lock_contention.sh | 14 +
tools/perf/tests/shell/record_bpf_filter.sh | 134 ++
tools/perf/util/bpf-filter.c | 1271 ++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 46 +-
tools/perf/util/bpf_lock_contention.c | 171 ++-
.../perf/util/bpf_skel/lock_contention.bpf.c | 448 +++---
tools/perf/util/bpf_skel/lock_data.h | 360 ++++-
tools/perf/util/bpf_skel/sample-filter.h | 77 +
tools/perf/util/bpf_skel/sample_filter.bpf.c | 1225 ++++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 +
tools/perf/util/lock-contention.c | 22 +
tools/perf/util/lock-contention.h | 15 +
tools/perf/util/machine.c | 11 +
tools/perf/util/parse-events.c | 3 +-
20 files changed, 3741 insertions(+), 387 deletions(-)
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply [flat|nested] 106+ messages in thread
* [PATCH v3 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
@ 2026-10-18 5:37 ` Ian Rogers
2026-10-18 5:50 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers
` (6 subsequent siblings)
7 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-18 5:37 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, rcu
Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers
(get_lock_owner, get_current_mmap_lock, get_current_siglock,
get_lock_addr, check_lock_type, do_collect_zone_lock,
do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into
tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF
sample filtering.
Also add rw_semaphore.count and struct rt_mutex_base to the minimal
vmlinux.h and extend get_lock_owner() to support rt_mutex and check
the mutex handoff flag.
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/builtin-lock.c | 27 --
.../perf/util/bpf_skel/lock_contention.bpf.c | 265 +--------------
tools/perf/util/bpf_skel/lock_data.h | 316 ++++++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 +
tools/perf/util/lock-contention.c | 18 +
tools/perf/util/lock-contention.h | 13 +
6 files changed, 370 insertions(+), 274 deletions(-)
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index d5c0d55cb82d..feaf1cfd08cf 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -1477,33 +1477,6 @@ static void sort_result(void)
}
}
-static const struct {
- unsigned int flags;
- /*
- * Name of the lock flags (access), with delimeter ':'.
- * For example, rwsem:R of rwsem:W.
- */
- const char *flags_name;
- /* Name of the lock (type), for example, rwlock or rwsem. */
- const char *lock_name;
-} lock_type_table[] = {
- { 0, "semaphore", "semaphore" },
- { LCB_F_SPIN, "spinlock", "spinlock" },
- { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
- { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
- { LCB_F_READ, "rwsem:R", "rwsem" },
- { LCB_F_WRITE, "rwsem:W", "rwsem" },
- { LCB_F_RT, "rt-mutex", "rt-mutex" },
- { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
- { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
- { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
- { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
- { LCB_F_MUTEX, "mutex", "mutex" },
- { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
- /* alias for optimistic spinning only */
- { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
-};
-
static const char *get_type_flags_name(unsigned int flags)
{
flags &= LCB_F_TYPE_MASK;
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index 0d9c6f55050e..0f2118006005 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -8,23 +8,9 @@
#include "lock_data.h"
-/* for collect_lock_syms(). 4096 was rejected by the verifier */
-#define MAX_CPUS 1024
-
-/* for collect_zone_lock(). It should be more than the actual zones. */
-#define MAX_ZONES 10
-
/* for do_lock_delay(). Arbitrarily set to 1 million. */
#define MAX_LOOP (1U << 20)
-/* lock contention flags from include/trace/events/lock.h */
-#define LCB_F_SPIN (1U << 0)
-#define LCB_F_READ (1U << 1)
-#define LCB_F_WRITE (1U << 2)
-#define LCB_F_RT (1U << 3)
-#define LCB_F_PERCPU (1U << 4)
-#define LCB_F_MUTEX (1U << 5)
-
/* callstack storage */
struct {
__uint(type, BPF_MAP_TYPE_STACK_TRACE);
@@ -96,13 +82,6 @@ struct {
__uint(max_entries, MAX_ENTRIES);
} task_data SEC(".maps");
-struct {
- __uint(type, BPF_MAP_TYPE_HASH);
- __uint(key_size, sizeof(__u64));
- __uint(value_size, sizeof(__u32));
- __uint(max_entries, MAX_ENTRIES);
-} lock_syms SEC(".maps");
-
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(key_size, sizeof(__u32));
@@ -138,6 +117,13 @@ struct {
__uint(max_entries, 1);
} cgroup_filter SEC(".maps");
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u32));
+ __uint(max_entries, MAX_ENTRIES);
+} lock_syms SEC(".maps");
+
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(key_size, sizeof(long));
@@ -159,22 +145,6 @@ struct {
__uint(max_entries, 1);
} lock_delays SEC(".maps");
-struct rw_semaphore___old {
- struct task_struct *owner;
-} __attribute__((preserve_access_index));
-
-struct rw_semaphore___new {
- atomic_long_t owner;
-} __attribute__((preserve_access_index));
-
-struct mm_struct___old {
- struct rw_semaphore mmap_sem;
-} __attribute__((preserve_access_index));
-
-struct mm_struct___new {
- struct rw_semaphore mmap_lock;
-} __attribute__((preserve_access_index));
-
struct cas_ctx {
struct contention_data *data;
u64 duration;
@@ -182,8 +152,6 @@ struct cas_ctx {
int min_done;
};
-extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak;
-
/* control flags */
const volatile int has_cpu;
const volatile int has_task;
@@ -202,6 +170,11 @@ const volatile int lock_delay;
/* determine the key of lock stat */
const volatile int aggr_mode;
+const volatile __u64 contig_page_data_addr;
+const volatile __u64 node_data_addr;
+const volatile int nr_nodes;
+const volatile int sizeof_zone;
+
int enabled;
int perf_subsys_id = -1;
@@ -222,8 +195,6 @@ int data_map_full;
struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak;
void bpf_task_release(struct task_struct *p) __ksym __weak;
-static inline __u32 check_lock_type(__u64 lock, __u32 flags);
-
static inline __u64 get_current_cgroup_id(void)
{
struct task_struct *task;
@@ -347,85 +318,6 @@ static inline int update_task_data(struct task_struct *task)
return 0;
}
-#ifndef __has_builtin
-# define __has_builtin(x) 0
-#endif
-
-static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
-{
- struct task_struct *task;
- __u64 owner = 0;
-
- if (flags & LCB_F_MUTEX) {
- struct mutex *mutex = (void *)lock;
- owner = BPF_CORE_READ(mutex, owner.counter);
- } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
- /*
- * Support for the BPF_TYPE_MATCHES argument to the
- * __builtin_preserve_type_info builtin was added at some point during
- * development of clang 15 and it's what is needed for
- * bpf_core_type_matches.
- */
-#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
- if (bpf_core_type_matches(struct rw_semaphore___old)) {
- struct rw_semaphore___old *rwsem = (void *)lock;
- owner = (unsigned long)BPF_CORE_READ(rwsem, owner);
- } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
- struct rw_semaphore___new *rwsem = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
- }
-#else
- /* assume new struct */
- struct rw_semaphore *rwsem = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
-#endif
- }
-
- if (!owner)
- return NULL;
-
- task = (void *)(owner & ~7UL);
- return task;
-}
-
-static inline __u32 check_lock_type(__u64 lock, __u32 flags)
-{
- struct task_struct *curr;
- struct mm_struct___old *mm_old;
- struct mm_struct___new *mm_new;
- struct sighand_struct *sighand;
-
- switch (flags) {
- case LCB_F_READ: /* rwsem */
- case LCB_F_WRITE:
- curr = bpf_get_current_task_btf();
- if (curr->mm == NULL)
- break;
- mm_new = (void *)curr->mm;
- if (bpf_core_field_exists(mm_new->mmap_lock)) {
- if (&mm_new->mmap_lock == (void *)lock)
- return LCD_F_MMAP_LOCK;
- break;
- }
- mm_old = (void *)curr->mm;
- if (bpf_core_field_exists(mm_old->mmap_sem)) {
- if (&mm_old->mmap_sem == (void *)lock)
- return LCD_F_MMAP_LOCK;
- }
- break;
- case LCB_F_SPIN: /* spinlock */
- curr = bpf_get_current_task_btf();
- sighand = curr->sighand;
-
- if (sighand && &sighand->siglock == (void *)lock)
- return LCD_F_SIGHAND_LOCK;
- break;
- default:
- break;
- }
- return 0;
-}
-
static inline long delay_callback(__u64 idx, void *arg)
{
__u64 target = *(__u64 *)arg;
@@ -908,145 +800,24 @@ int contention_end(u64 *ctx)
return 0;
}
-extern struct rq runqueues __ksym;
-
-const volatile __u64 contig_page_data_addr;
-const volatile __u64 node_data_addr;
-const volatile int nr_nodes;
-const volatile int sizeof_zone;
-
-struct rq___old {
- raw_spinlock_t lock;
-} __attribute__((preserve_access_index));
-
-struct rq___new {
- raw_spinlock_t __lock;
-} __attribute__((preserve_access_index));
-
-static void collect_zone_lock(void)
-{
- __u64 nr_zones, zone_off;
- __u64 lock_addr, lock_off;
- __u32 lock_flag = LOCK_CLASS_ZONE_LOCK;
-
- zone_off = offsetof(struct pglist_data, node_zones);
- lock_off = offsetof(struct zone, lock);
-
- if (contig_page_data_addr) {
- struct pglist_data *contig_page_data;
-
- contig_page_data = (void *)(long)contig_page_data_addr;
- nr_zones = BPF_CORE_READ(contig_page_data, nr_zones);
-
- for (int i = 0; i < MAX_ZONES; i++) {
- __u64 zone_addr;
-
- if (i >= nr_zones)
- break;
-
- zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off;
- lock_addr = zone_addr + lock_off;
-
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
- } else if (nr_nodes > 0) {
- struct pglist_data **node_data = (void *)(long)node_data_addr;
-
- for (int i = 0; i < nr_nodes; i++) {
- struct pglist_data *pgdat = NULL;
- int err;
-
- err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
- if (err < 0 || pgdat == NULL)
- break;
-
- nr_zones = BPF_CORE_READ(pgdat, nr_zones);
- for (int k = 0; k < MAX_ZONES; k++) {
- __u64 zone_addr;
-
- if (k >= nr_zones)
- break;
-
- zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off;
- lock_addr = zone_addr + lock_off;
-
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
- }
- }
-}
-
SEC("raw_tp/bpf_test_finish")
-int BPF_PROG(collect_lock_syms)
+int BPF_PROG(end_timestamp)
{
- __u64 lock_addr, lock_off;
- __u32 lock_flag;
-
- if (bpf_core_field_exists(struct rq___new, __lock))
- lock_off = offsetof(struct rq___new, __lock);
- else
- lock_off = offsetof(struct rq___old, lock);
-
- for (int i = 0; i < MAX_CPUS; i++) {
- struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
-
- if (rq == NULL)
- break;
-
- lock_addr = (__u64)(void *)rq + lock_off;
- lock_flag = LOCK_CLASS_RQLOCK;
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
-
- collect_zone_lock();
-
+ end_ts = bpf_ktime_get_ns();
return 0;
}
SEC("raw_tp/bpf_test_finish")
-int BPF_PROG(end_timestamp)
+int BPF_PROG(collect_lock_syms)
{
- end_ts = bpf_ktime_get_ns();
- return 0;
+ return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
}
-/*
- * bpf_iter__kmem_cache added recently so old kernels don't have it in the
- * vmlinux.h. But we cannot add it here since it will cause a compiler error
- * due to redefinition of the struct on later kernels.
- *
- * So it uses a CO-RE trick to access the member only if it has the type.
- * This will support both old and new kernels without compiler errors.
- */
-struct bpf_iter__kmem_cache___new {
- struct kmem_cache *s;
-} __attribute__((preserve_access_index));
-
SEC("iter/kmem_cache")
int slab_cache_iter(void *ctx)
{
- struct kmem_cache *s = NULL;
- struct slab_cache_data d;
- const char *nameptr;
-
- if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) {
- struct bpf_iter__kmem_cache___new *iter = ctx;
-
- s = iter->s;
- }
-
- if (s == NULL)
- return 0;
-
- nameptr = s->name;
- bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr);
-
- d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT;
- if (d.id >= LCB_F_SLAB_ID_END)
- return 0;
-
- bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST);
- return 0;
+ return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
}
char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index 652e114e6b87..8cf24f07204c 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -45,6 +45,14 @@ struct contention_task_data {
#define LCB_F_SLAB_ID_END (1U << 26)
#define LCB_F_SLAB_ID_MASK 0x03FF0000U
+/* lock contention flags from include/trace/events/lock.h */
+#define LCB_F_SPIN (1U << 0)
+#define LCB_F_READ (1U << 1)
+#define LCB_F_WRITE (1U << 2)
+#define LCB_F_RT (1U << 3)
+#define LCB_F_PERCPU (1U << 4)
+#define LCB_F_MUTEX (1U << 5)
+
#define LCB_F_TYPE_MAX (1U << 7)
#define LCB_F_TYPE_MASK 0x0000007FU
@@ -76,4 +84,312 @@ struct slab_cache_data {
char name[SLAB_NAME_MAX];
};
+#ifdef __BPF__
+
+/* for collect_lock_syms(). 4096 was rejected by the verifier */
+#define MAX_CPUS 1024
+
+/* for collect_zone_lock(). It should be more than the actual zones. */
+#define MAX_ZONES 10
+
+extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak;
+extern struct rq runqueues __ksym __weak;
+
+#ifndef __has_builtin
+# define __has_builtin(x) 0
+#endif
+
+struct rw_semaphore___old {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
+struct rw_semaphore___new {
+ atomic_long_t count;
+ atomic_long_t owner;
+} __attribute__((preserve_access_index));
+
+struct mm_struct___old {
+ struct rw_semaphore mmap_sem;
+} __attribute__((preserve_access_index));
+
+struct mm_struct___new {
+ struct rw_semaphore mmap_lock;
+} __attribute__((preserve_access_index));
+
+struct rq___old {
+ raw_spinlock_t lock;
+} __attribute__((preserve_access_index));
+
+struct rq___new {
+ raw_spinlock_t __lock;
+} __attribute__((preserve_access_index));
+
+/*
+ * bpf_iter__kmem_cache added recently so old kernels don't have it in the
+ * vmlinux.h. But we cannot add it here since it will cause a compiler error
+ * due to redefinition of the struct on later kernels.
+ *
+ * So it uses a CO-RE trick to access the member only if it has the type.
+ * This will support both old and new kernels without compiler errors.
+ */
+struct bpf_iter__kmem_cache___new {
+ struct kmem_cache *s;
+} __attribute__((preserve_access_index));
+
+static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
+{
+ struct task_struct *task;
+ __u64 owner = 0;
+
+ if (flags & LCB_F_MUTEX) {
+ struct mutex *mutex = (void *)lock;
+
+ if (bpf_core_field_exists(mutex->owner.counter)) {
+ owner = BPF_CORE_READ(mutex, owner.counter);
+ if (owner & 0x4UL)
+ return NULL;
+ } else if (bpf_core_type_exists(struct rt_mutex_base)) {
+ struct rt_mutex_base *rtm = (void *)lock;
+
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ }
+ } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
+ /*
+ * Support for the BPF_TYPE_MATCHES argument to the
+ * __builtin_preserve_type_info builtin was added at some point during
+ * development of clang 15 and it's what is needed for
+ * bpf_core_type_matches.
+ */
+#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
+ if (bpf_core_type_matches(struct rw_semaphore___old)) {
+ struct rw_semaphore___old *old = (void *)lock;
+
+ owner = (unsigned long)BPF_CORE_READ(old, owner);
+ } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
+ struct rw_semaphore___new *new = (void *)lock;
+
+ owner = BPF_CORE_READ(new, owner.counter);
+ }
+#else
+ /* assume new struct */
+ struct rw_semaphore___new *new = (void *)lock;
+
+ if (bpf_core_field_exists(new->owner.counter))
+ owner = BPF_CORE_READ(new, owner.counter);
+#endif
+ } else if (flags == LCB_F_RT) {
+ struct rt_mutex_base *rtm = (void *)lock;
+
+ if (bpf_core_type_exists(struct rt_mutex_base)) {
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ }
+ }
+
+ if (!owner)
+ return NULL;
+
+ task = (void *)(owner & ~7UL);
+ return task;
+}
+
+static __always_inline __u64 get_current_mmap_lock(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+ struct mm_struct___new *mm_new = NULL;
+ struct mm_struct___old *mm_old = NULL;
+ __u64 mm;
+
+ if (!curr)
+ return 0;
+
+ mm = (__u64)BPF_CORE_READ(task, mm);
+ if (!mm)
+ return 0;
+
+ if (bpf_core_field_exists(mm_new->mmap_lock))
+ return mm + bpf_core_field_offset(mm_new->mmap_lock);
+
+ if (bpf_core_field_exists(mm_old->mmap_sem))
+ return mm + bpf_core_field_offset(mm_old->mmap_sem);
+
+ return 0;
+}
+
+static __always_inline __u64 get_current_siglock(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+ struct sighand_struct *sighand = NULL;
+ __u64 sh;
+
+ if (!curr)
+ return 0;
+
+ sh = (__u64)BPF_CORE_READ(task, sighand);
+ if (!sh)
+ return 0;
+
+ return sh + bpf_core_field_offset(sighand->siglock);
+}
+
+/*
+ * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar.
+ *
+ * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the
+ * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed
+ * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such
+ * as `bpf_get_kmem_cache(u64 addr)`.
+ */
+static __always_inline __u64 get_lock_addr(void *ctx)
+{
+ __u64 lock_addr = 0;
+
+ bpf_get_func_arg(ctx, 0, &lock_addr);
+ return lock_addr;
+}
+
+static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags)
+{
+ __u64 curr;
+
+ switch (flags) {
+ case LCB_F_READ: /* rwsem */
+ case LCB_F_WRITE:
+ curr = bpf_get_current_task();
+ if (get_current_mmap_lock(curr) == lock)
+ return LCD_F_MMAP_LOCK;
+ break;
+ case LCB_F_SPIN: /* spinlock */
+ curr = bpf_get_current_task();
+ if (get_current_siglock(curr) == lock)
+ return LCD_F_SIGHAND_LOCK;
+ break;
+ default:
+ break;
+ }
+ return 0;
+}
+
+static __always_inline void do_collect_zone_lock(void *lock_syms_map,
+ __u64 contig_page_data_addr,
+ __u64 node_data_addr,
+ int nr_nodes,
+ int sizeof_zone)
+{
+ __u64 nr_zones, zone_off;
+ __u64 lock_addr, lock_off;
+ __u32 lock_flag = LOCK_CLASS_ZONE_LOCK;
+
+ zone_off = offsetof(struct pglist_data, node_zones);
+ lock_off = offsetof(struct zone, lock);
+
+ if (contig_page_data_addr) {
+ struct pglist_data *contig_page_data;
+
+ contig_page_data = (void *)(long)contig_page_data_addr;
+ nr_zones = BPF_CORE_READ(contig_page_data, nr_zones);
+
+ for (int i = 0; i < MAX_ZONES; i++) {
+ __u64 zone_addr;
+
+ if (i >= nr_zones)
+ break;
+
+ zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off;
+ lock_addr = zone_addr + lock_off;
+
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ } else if (node_data_addr && nr_nodes > 0) {
+ struct pglist_data **node_data = (void *)(long)node_data_addr;
+
+ for (int i = 0; i < nr_nodes; i++) {
+ struct pglist_data *pgdat = NULL;
+ int err;
+
+ err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
+ if (err < 0)
+ break;
+ if (pgdat == NULL)
+ continue;
+
+ nr_zones = BPF_CORE_READ(pgdat, nr_zones);
+ for (int k = 0; k < MAX_ZONES; k++) {
+ __u64 zone_addr;
+
+ if (k >= nr_zones)
+ break;
+
+ zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off;
+ lock_addr = zone_addr + lock_off;
+
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ }
+ }
+}
+
+static __always_inline int do_collect_lock_syms(void *lock_syms_map,
+ __u64 contig_page_data_addr,
+ __u64 node_data_addr,
+ int nr_nodes,
+ int sizeof_zone)
+{
+ __u64 lock_addr, lock_off;
+ __u32 lock_flag;
+
+ if (bpf_core_field_exists(struct rq___new, __lock))
+ lock_off = offsetof(struct rq___new, __lock);
+ else
+ lock_off = offsetof(struct rq___old, lock);
+
+ if (&runqueues) {
+ for (int i = 0; i < MAX_CPUS; i++) {
+ struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
+
+ if (rq == NULL)
+ break;
+
+ lock_addr = (__u64)(void *)rq + lock_off;
+ lock_flag = LOCK_CLASS_RQLOCK;
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ }
+
+ do_collect_zone_lock(lock_syms_map, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
+
+ return 0;
+}
+
+static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map,
+ __u32 *slab_cache_id)
+{
+ struct kmem_cache *s = NULL;
+ struct slab_cache_data d = {};
+ const char *nameptr;
+
+ if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) {
+ struct bpf_iter__kmem_cache___new *iter = ctx;
+
+ s = iter->s;
+ }
+
+ if (s == NULL)
+ return 0;
+
+ nameptr = s->name;
+ bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr);
+
+ d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT;
+ if (d.id >= LCB_F_SLAB_ID_END)
+ return 0;
+
+ bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST);
+ return 0;
+}
+
+#endif /* __BPF__ */
+
#endif /* UTIL_BPF_SKEL_LOCK_DATA_H */
diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
index 94e42b022ab1..b75ddc6846bb 100644
--- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
+++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
@@ -76,6 +76,7 @@ struct signal_struct {
} __attribute__((preserve_access_index));
struct rw_semaphore {
+ atomic_long_t count;
atomic_long_t owner;
} __attribute__((preserve_access_index));
@@ -83,6 +84,10 @@ struct mutex {
atomic_long_t owner;
} __attribute__((preserve_access_index));
+struct rt_mutex_base {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
struct kernfs_node {
u64 id;
} __attribute__((preserve_access_index));
diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c
index 119a7206f3cd..a1bb78d6e29c 100644
--- a/tools/perf/util/lock-contention.c
+++ b/tools/perf/util/lock-contention.c
@@ -22,6 +22,24 @@ struct callstack_filter {
static LIST_HEAD(callstack_filters);
struct hlist_head *lockhash_table;
+const struct perf_lock_type lock_type_table[] = {
+ { 0, "semaphore", "semaphore" },
+ { LCB_F_SPIN, "spinlock", "spinlock" },
+ { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
+ { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
+ { LCB_F_READ, "rwsem:R", "rwsem" },
+ { LCB_F_WRITE, "rwsem:W", "rwsem" },
+ { LCB_F_RT, "rt-mutex", "rt-mutex" },
+ { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
+ { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
+ { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
+ { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
+ { LCB_F_MUTEX, "mutex", "mutex" },
+ { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
+ /* alias for optimistic spinning only */
+ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
+};
+
int parse_call_stack(const struct option *opt __maybe_unused, const char *str,
int unset __maybe_unused)
{
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index 59c94190b092..68e9bd77e8a9 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -129,6 +129,19 @@ struct thread_stat {
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+struct perf_lock_type {
+ unsigned int flags;
+ /*
+ * Name of the lock flags (access), with delimiter ':'.
+ * For example, rwsem:R or rwsem:W.
+ */
+ const char *flags_name;
+ /* Name of the lock (type), for example, rwlock or rwsem. */
+ const char *lock_name;
+};
+
+extern const struct perf_lock_type lock_type_table[14];
+
struct evlist;
struct machine;
struct target;
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v3 2/6] perf bpf-filter: Add lock_held() sample filter
2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-18 5:37 ` [PATCH v3 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
@ 2026-10-18 5:37 ` Ian Rogers
2026-10-18 6:12 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers
` (5 subsequent siblings)
7 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-18 5:37 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, rcu
Add BPF sample filter predicates lock_held(<lock>),
lock_held_read(<lock>) / lock_held_shared(<lock>), and
lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R'
and ':W' suffixes on <lock>) to allow recording perf event samples only
when a specified kernel lock is held (or not held with '!').
Supported <lock> specifications match perf lock contention:
- Global/static kernel lock symbols (with or without '&', e.g.
lock_held(&cgroup_mutex) or lock_held(cgroup_mutex))
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with '&' (e.g. lock_held(&dentry),
lock_held_read(&dentry))
- Per-process and semi-global locks: mmap_lock, siglock, rq_lock,
zone_lock
- Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W,
rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W,
semaphore
Supported lock hold modes by lock family:
- Both read/shared and write/exclusive: rwsem (including mmap_lock)
and rwlock.
- Write/exclusive only: mutex, spinlock (including siglock, rq_lock,
zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write()
/ percpu_up_write(); fast-path percpu_down_read() / percpu_up_read()
are inlined per-CPU counter operations without function entry points).
When evaluating the filter on sample overflow, perf_sample_filter first
inspects the lock's owner field directly in kernel memory where
possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls
back to a held_locks BPF hash map (tracking separate read_count and
write_count) populated by fentry/fexit hooks on kernel lock/unlock
functions. To minimize tracing overhead, only the BPF hooks for the
requested lock family and mode are autoloaded and attached.
Example usage:
# perf record -a -g -e cycles --filter 'lock_held(&mmap_lock)' \
-- perf bench mem mmap -t 2 -l 100
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ]
# perf report --stdio --no-children
# Overhead Command Shared Object Symbol
# ........ ........ ................. ........................
#
18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms
|
---clear_page_erms
get_page_from_freelist
__alloc_pages_noprof
folio_alloc_mpol_noprof
vma_alloc_folio_noprof
do_anonymous_page
__handle_mm_fault
handle_mm_fault
do_user_addr_fault
exc_page_fault
asm_exc_page_fault
worker_thread
Read/shared vs write/exclusive holding can also be filtered separately:
# perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' ...
# perf record -a -g -e cycles --filter 'lock_held_write(&mmap_lock)' ...
# perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' ...
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 54 +-
tools/perf/util/bpf-filter.c | 1114 +++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 46 +-
tools/perf/util/bpf_skel/sample-filter.h | 65 +
tools/perf/util/bpf_skel/sample_filter.bpf.c | 1094 +++++++++++++++++
tools/perf/util/parse-events.c | 3 +-
8 files changed, 2380 insertions(+), 40 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 2351f7b4f5fc..39dfe63fe3a8 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -204,7 +204,59 @@ OPTIONS
ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr,
code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat,
p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock,
- mem_dtlb, mem_blk, mem_hops, uid, gid
+ mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>),
+ lock_held_read(<lock>), lock_held_shared(<lock>),
+ lock_held_write(<lock>), lock_held_exclusive(<lock>)
+
+ The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters
+ samples based on whether the specified kernel lock is currently held
+ by the kernel thread or CPU. To differentiate between shared/read
+ holding and exclusive/write holding, use 'lock_held_read(<lock>)'
+ (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)'
+ (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to
+ <lock> (e.g. 'lock_held(&mmap_lock:R)' or 'lock_held(&mmap_lock:W)').
+ <lock> can be a kernel lock symbol name (with or without a leading
+ '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'),
+ a hexadecimal lock address, a slab cache name prefixed with '&'
+ (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known
+ lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock
+ type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
+ 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
+ 'semaphore'). Both read/shared and write/exclusive tracking are
+ supported for 'rwsem' (including 'mmap_lock') and 'rwlock';
+ 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'),
+ 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()')
+ support write/exclusive tracking only.
+
+ For example, 'lock_held()' can be combined with memory allocation or
+ reclaim events to detect allocations or direct memory reclaim occurring
+ while a kernel lock is held (which can stall lock waiters for extended
+ periods):
+
+ # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+ # perf record -a -g -e kmem:mm_page_alloc \
+ --filter 'gfp_flags & 0x400' \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+ # perf record -a -g -e major-faults \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+
+ Note that 'lock_held(<lock>)' introduces overhead outside of sample
+ overflow handling: in addition to checking the lock's owner field on
+ each sample where possible (such as for mutexes, rt-mutexes, and
+ write-locked rwsems), it attaches BPF fentry/fexit programs to kernel
+ lock acquire and release functions to track held locks (such as
+ spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a
+ BPF map. These BPF hooks run on every kernel lock acquisition and
+ release for the attached lock family, even when no perf sample is
+ being taken. To minimize this overhead, only the hooks for the
+ relevant lock family and mode are attached when known (for example,
+ 'lock_held_read(&mmap_lock)' only hooks rwsem read functions,
+ 'lock_held_write(&mmap_lock)' only hooks rwsem write functions,
+ '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks
+ spinlock functions), whereas slab filters ('&<slab>'), raw hex
+ addresses, or symbols without a recognizable lock-type suffix attach
+ hooks across all lock families (including high-frequency spinlocks).
The <operator> can be one of:
==, !=, >, >=, <, <=, &
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 72e83ac1d6a4..0c133b31b5d5 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -40,21 +40,35 @@
* The 'dropped' map is to keep how many samples it dropped by the filter and
* it will be reported as lost samples.
*/
+#include <errno.h>
+#include <stdio.h>
#include <stdlib.h>
+#include <stdbool.h>
+#include <stddef.h>
+#include <string.h>
#include <fcntl.h>
+#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/stat.h>
#include <bpf/bpf.h>
+#include <bpf/btf.h>
#include <linux/err.h>
+#include <linux/kernel.h>
#include <linux/list.h>
+#include <linux/string.h>
+#include <linux/zalloc.h>
#include <api/fs/fs.h>
#include <internal/xyarray.h>
#include <perf/threadmap.h>
+#include <symbol/kallsyms.h>
#include "util/cap.h"
#include "util/debug.h"
#include "util/evsel.h"
+#include "util/lock-contention.h"
+#include "util/string2.h"
+#include "util/symbol_conf.h"
#include "util/target.h"
#include "util/thread_map.h"
#include "util/bpf-utils.h"
@@ -63,6 +77,7 @@
#include <util/bpf-filter-flex.h>
#include <util/bpf-filter-bison.h>
+#include "bpf_skel/lock_data.h"
#include "bpf_skel/sample-filter.h"
#include "bpf_skel/sample_filter.skel.h"
@@ -123,15 +138,27 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
{
const struct perf_sample_info *info;
- if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END &&
- (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
- return 0;
-
- if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
+ if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID ||
+ expr->term == PBF_TERM_LOCK_HELD) {
/* Not dependent on the sample_type as computed from a BPF helper. */
return 0;
}
+ if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT) {
+ if (expr->term == PBF_TERM_TID || expr->term == PBF_TERM_CPU ||
+ expr->term == PBF_TERM_TIME || expr->term == PBF_TERM_CGROUP)
+ return 0;
+ if (expr->op != PBF_OP_GROUP_BEGIN) {
+ pr_err("Error: %s tracepoint event does not support BPF sample term %d\n",
+ evsel__name(evsel), expr->term);
+ return -1;
+ }
+ }
+
+ if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END &&
+ (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
+ return 0;
+
if (expr->op == PBF_OP_GROUP_BEGIN) {
struct perf_bpf_filter_expr *group;
@@ -155,9 +182,831 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
return -1;
}
-static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry)
+/*
+ * Program groups for `lock_held(...)` BPF tracing hooks.
+ *
+ * When a `lock_held(...)` filter is specified, only the BPF tracing program
+ * groups required for the requested lock(s) are autoloaded and attached.
+ * If no `lock_held(...)` filter is present, none of these programs are loaded.
+ */
+enum lock_prog_group {
+ LOCK_PROG_MUTEX = 1U << 0,
+ LOCK_PROG_RWSEM_READ = 1U << 1,
+ LOCK_PROG_RWSEM_WRITE = 1U << 2,
+ LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE,
+ LOCK_PROG_SPIN = 1U << 3,
+ LOCK_PROG_RWLOCK_READ = 1U << 4,
+ LOCK_PROG_RWLOCK_WRITE = 1U << 5,
+ LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE,
+ LOCK_PROG_RTMUTEX = 1U << 6,
+ LOCK_PROG_SEM = 1U << 7,
+ LOCK_PROG_PERCPU_RWSEM = 1U << 8,
+ LOCK_PROG_ALL = 0x1ffU,
+};
+
+static unsigned int lock_flags_to_prog_group(unsigned int flags)
+{
+ if (flags & LCB_F_MUTEX)
+ return LOCK_PROG_MUTEX;
+ if (flags & LCB_F_RT)
+ return LOCK_PROG_RTMUTEX;
+ if (flags & LCB_F_PERCPU)
+ return LOCK_PROG_PERCPU_RWSEM;
+ if (flags == LCB_F_SPIN)
+ return LOCK_PROG_SPIN;
+ if (flags & LCB_F_SPIN)
+ return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE;
+ if (flags & LCB_F_READ)
+ return LOCK_PROG_RWSEM_READ;
+ if (flags & LCB_F_WRITE)
+ return LOCK_PROG_RWSEM_WRITE;
+ return LOCK_PROG_SEM;
+}
+
+struct lock_prog_desc {
+ size_t prog_offset;
+ const char *btf_func;
+ unsigned int group;
+};
+
+#define LOCK_FPROG(field, func, grp) \
+ { offsetof(struct sample_filter_bpf, progs.field), func, grp }
+
+static const struct lock_prog_desc lock_progs[] = {
+ /* Mutex */
+ LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
+ LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
+ /* RW semaphore */
+ LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested",
+ LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested",
+ LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM),
+ LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE),
+ /* Spinlock */
+ LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN),
+ /* RWlock */
+ LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE),
+ /* RT-Mutex */
+ LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible",
+ LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX),
+ /* Semaphore */
+ LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM),
+ LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM),
+ /* Percpu-RWsem */
+ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
+ LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+};
+
+#undef LOCK_FPROG
+
+static int split_lock_name_mode(const char *name, int expr_part,
+ char *base_buf, size_t buf_sz,
+ bool *has_amp, unsigned int *mode_out)
+{
+ const char *bare;
+ size_t len;
+ unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK;
+ unsigned int suffix_mode = 0;
+
+ if (!name || !*name)
+ return -EINVAL;
+
+ *has_amp = (name[0] == '&');
+ bare = *has_amp ? name + 1 : name;
+ len = strlen(bare);
+ if (len == 0)
+ return -EINVAL;
+
+ if (len > 2 && !strcmp(bare + len - 2, ":R")) {
+ suffix_mode = PBF_LOCK_MODE_READ;
+ len -= 2;
+ } else if (len > 2 && !strcmp(bare + len - 2, ":W")) {
+ suffix_mode = PBF_LOCK_MODE_WRITE;
+ len -= 2;
+ }
+
+ if (len == 0 || len >= buf_sz)
+ return -EINVAL;
+
+ if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY &&
+ suffix_mode && expr_mode != suffix_mode)
+ return -EINVAL;
+
+ memcpy(base_buf, bare, len);
+ base_buf[len] = '\0';
+
+ if (suffix_mode)
+ *mode_out = suffix_mode;
+ else if (expr_mode)
+ *mode_out = expr_mode;
+ else
+ *mode_out = PBF_LOCK_MODE_ANY;
+
+ return 0;
+}
+
+/*
+ * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R",
+ * "spinlock", "percpu-rwsem").
+ *
+ * A lock type name may correspond to up to two flag combinations (e.g.
+ * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed
+ * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is
+ * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`.
+ */
+static bool parse_lock_type_name(const char *name, unsigned int mode,
+ struct sample_filter_bpf *skel,
+ u64 *encoded_pair, unsigned int *prog_groups)
+{
+ bool found = false;
+ u32 flag1 = 0xff, flag2 = 0xff;
+ unsigned int groups = 0;
+ const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name;
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) {
+ u32 f = lock_type_table[i].flags;
+ bool is_read = (f & LCB_F_READ) != 0;
+
+ if (strcmp(lock_type_table[i].lock_name, lookup) &&
+ strcmp(lock_type_table[i].flags_name, name))
+ continue;
+
+ /*
+ * percpu_down_read() / percpu_up_read() fast paths are inlined
+ * per-CPU counter increments without function entry points, so
+ * only writer holding (percpu_down_write / percpu_up_write) can
+ * be tracked for percpu-rwsem. rwlock-rt is also not hooked.
+ */
+ if ((f & LCB_F_PERCPU) && is_read)
+ continue;
+ if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE)))
+ continue;
+
+ if (is_read && !(mode & PBF_LOCK_MODE_READ))
+ continue;
+ if (!is_read && !(mode & PBF_LOCK_MODE_WRITE))
+ continue;
+
+ if (!found)
+ flag1 = f;
+ else if (f != flag1)
+ flag2 = f;
+ if (skel && f < LCB_F_TYPE_MAX) {
+ skel->bss->has_lock_type = 1;
+ skel->bss->lock_type_filter[f] = 1;
+ }
+ groups |= lock_flags_to_prog_group(f);
+ found = true;
+ }
+
+ if (!found)
+ return false;
+
+ if (encoded_pair)
+ *encoded_pair = flag1 | ((u64)flag2 << 8);
+ if (prog_groups)
+ *prog_groups |= groups;
+ return true;
+}
+
+struct lock_sym_cb_args {
+ const char *name;
+ u64 addr;
+ bool found;
+};
+
+static int find_lock_sym_cb(void *arg, const char *name,
+ char type __maybe_unused, u64 start)
+{
+ struct lock_sym_cb_args *args = arg;
+
+ if (!strcmp(name, args->name)) {
+ args->found = true;
+ args->addr = start;
+ if (start != 0)
+ return 1;
+ }
+ return 0;
+}
+
+static int find_kernel_lock_sym(const char *name, u64 *addr)
+{
+ struct lock_sym_cb_args args = {
+ .name = name,
+ };
+ const char *kallsyms_filename = "/proc/kallsyms";
+
+ if (symbol_conf.kallsyms_name != NULL)
+ kallsyms_filename = symbol_conf.kallsyms_name;
+
+ if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0)
+ return -ENOENT;
+
+ if (!args.found)
+ return -ENOENT;
+
+ if (args.addr == 0) {
+ pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n",
+ name);
+ return -EACCES;
+ }
+
+ *addr = args.addr;
+ return 0;
+}
+
+static bool parse_lock_hex_addr(const char *str, u64 *addr)
+{
+ char *end;
+ unsigned long long val;
+
+ if (!str || !*str)
+ return false;
+
+ errno = 0;
+ val = strtoull(str, &end, 16);
+ if (*end != '\0' || errno != 0 || val == 0)
+ return false;
+
+ *addr = val;
+ return true;
+}
+
+static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf)
+{
+ u64 addr = 0;
+ char *buf = NULL, *p;
+ size_t len;
+ long last = -1;
+ int ret;
+
+ if (!btf)
+ return;
+
+ ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT);
+ if (ret < 0)
+ return;
+
+ ret = btf__resolve_size(btf, ret);
+ if (ret < 0)
+ return;
+
+ skel->rodata->sizeof_zone = ret;
+
+ if (find_kernel_lock_sym("contig_page_data", &addr) == 0) {
+ skel->rodata->contig_page_data_addr = addr;
+ return;
+ }
+
+ if (find_kernel_lock_sym("node_data", &addr) != 0)
+ return;
+
+ skel->rodata->node_data_addr = addr;
+
+ if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) {
+ p = buf;
+ while (p && *p) {
+ char *end;
+
+ last = strtol(p, &end, 0);
+ if (end == p)
+ break;
+ p = end;
+ if (*p == ',' || *p == '-' || *p == '\n')
+ p++;
+ }
+ skel->rodata->nr_nodes = last + 1;
+ free(buf);
+ }
+}
+
+static bool btf_has_func(struct btf *btf, const char *name)
+{
+ return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0;
+}
+
+static unsigned int supported_lock_prog_groups(struct btf *btf)
+{
+ unsigned int supported = 0;
+
+ if (!btf)
+ return 0;
+
+ if ((btf_has_func(btf, "mutex_lock") || btf_has_func(btf, "mutex_lock_nested")) &&
+ btf_has_func(btf, "mutex_unlock"))
+ supported |= LOCK_PROG_MUTEX;
+
+ if ((btf_has_func(btf, "down_read") || btf_has_func(btf, "down_read_nested")) &&
+ btf_has_func(btf, "up_read"))
+ supported |= LOCK_PROG_RWSEM_READ;
+
+ if ((btf_has_func(btf, "down_write") || btf_has_func(btf, "down_write_nested")) &&
+ btf_has_func(btf, "up_write"))
+ supported |= LOCK_PROG_RWSEM_WRITE;
+
+ /*
+ * Require both acquire and all unlock entry points in BTF so we never
+ * attach spinlock/rwlock acquire hooks when unlock functions are
+ * inlined (e.g. !CONFIG_UNINLINE_SPIN_UNLOCK or CONFIG_INLINE_*_UNLOCK).
+ */
+ if ((btf_has_func(btf, "_raw_spin_lock") || btf_has_func(btf, "_raw_spin_lock_nested")) &&
+ btf_has_func(btf, "_raw_spin_unlock") &&
+ btf_has_func(btf, "_raw_spin_unlock_irq") &&
+ btf_has_func(btf, "_raw_spin_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_spin_unlock_bh"))
+ supported |= LOCK_PROG_SPIN;
+
+ if (btf_has_func(btf, "_raw_read_lock") &&
+ btf_has_func(btf, "_raw_read_unlock") &&
+ btf_has_func(btf, "_raw_read_unlock_irq") &&
+ btf_has_func(btf, "_raw_read_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_read_unlock_bh"))
+ supported |= LOCK_PROG_RWLOCK_READ;
+
+ if ((btf_has_func(btf, "_raw_write_lock") || btf_has_func(btf, "_raw_write_lock_nested")) &&
+ btf_has_func(btf, "_raw_write_unlock") &&
+ btf_has_func(btf, "_raw_write_unlock_irq") &&
+ btf_has_func(btf, "_raw_write_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_write_unlock_bh"))
+ supported |= LOCK_PROG_RWLOCK_WRITE;
+
+ if ((btf_has_func(btf, "rt_mutex_lock") || btf_has_func(btf, "rt_mutex_lock_nested")) &&
+ btf_has_func(btf, "rt_mutex_unlock"))
+ supported |= LOCK_PROG_RTMUTEX;
+
+ if (btf_has_func(btf, "down") && btf_has_func(btf, "up"))
+ supported |= LOCK_PROG_SEM;
+
+ if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write"))
+ supported |= LOCK_PROG_PERCPU_RWSEM;
+
+ return supported;
+}
+
+/*
+ * Enable or disable autoload for each lock-tracking BPF program in the
+ * skeleton before `sample_filter_bpf__load()` is called.
+ *
+ * Each program is enabled only if its group is in `needed_groups` and its
+ * target kernel function or tracepoint exists in the running kernel's BTF.
+ */
+static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
+ unsigned int needed_groups,
+ unsigned int required_groups,
+ bool needs_lock_syms, bool needs_slab_iter,
+ bool is_tracepoint)
+{
+ unsigned int supported_groups = supported_lock_prog_groups(btf);
+ unsigned int enabled_progs = 0;
+
+ bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
+ bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
+
+ if ((required_groups & supported_groups) != required_groups) {
+ pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n");
+ return -ENOTSUP;
+ }
+
+ needed_groups &= supported_groups;
+ if (skel)
+ skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ);
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) {
+ struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset;
+ struct bpf_program *prog = *progp;
+ bool enable = false;
+
+ if ((needed_groups & lock_progs[i].group) && btf &&
+ lock_progs[i].btf_func) {
+ if (btf__find_by_name_kind(btf, lock_progs[i].btf_func,
+ BTF_KIND_FUNC) > 0)
+ enable = true;
+ }
+
+ if (enable)
+ enabled_progs++;
+ bpf_program__set_autoload(prog, enable);
+ }
+
+ bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms);
+ if (needs_slab_iter && btf &&
+ btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) {
+ bpf_program__set_autoload(skel->progs.slab_cache_iter, true);
+ } else {
+ bpf_program__set_autoload(skel->progs.slab_cache_iter, false);
+ }
+
+ if (needed_groups && !enabled_progs) {
+ pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
+ return -ENOTSUP;
+ }
+ return 0;
+}
+
+/*
+ * Pre-scan a single filter expression prior to loading the BPF skeleton in
+ * order to determine which BPF maps need to be sized and which lock tracing
+ * program groups need to be autoloaded.
+ */
+static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
+ bool *has_lock_filter,
+ unsigned int *prog_groups,
+ unsigned int *required_groups,
+ bool *needs_lock_syms,
+ bool *needs_slab_iter)
+{
+ char base[128];
+ bool has_amp;
+ unsigned int mode, groups = 0;
+ u64 dummy;
+
+ if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name)
+ return;
+
+ *has_lock_filter = true;
+ if (split_lock_name_mode(expr->lock_name, expr->part, base,
+ sizeof(base), &has_amp, &mode) < 0)
+ return;
+
+ if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
+ if (mode & PBF_LOCK_MODE_READ)
+ groups |= LOCK_PROG_RWSEM_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ groups |= LOCK_PROG_RWSEM_WRITE;
+ *prog_groups |= groups;
+ *required_groups |= groups;
+ return;
+ }
+ if (!strcmp(base, "siglock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ return;
+ }
+ if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ *needs_lock_syms = true;
+ return;
+ }
+ if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, &groups)) {
+ *prog_groups |= groups;
+ *required_groups |= groups;
+ return;
+ }
+
+ if (!has_amp && parse_lock_hex_addr(base, &dummy)) {
+ *prog_groups |= LOCK_PROG_ALL;
+ return;
+ }
+
+ if (has_amp)
+ *needs_slab_iter = true;
+
+ if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) {
+ *prog_groups |= LOCK_PROG_RTMUTEX;
+ *required_groups |= LOCK_PROG_RTMUTEX;
+ } else if (strtailcmp(base, "mutex") == 0) {
+ *prog_groups |= LOCK_PROG_MUTEX;
+ *required_groups |= LOCK_PROG_MUTEX;
+ } else if (strtailcmp(base, "percpu_rwsem") == 0 ||
+ strtailcmp(base, "pcpu_rwsem") == 0) {
+ *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
+ *required_groups |= LOCK_PROG_PERCPU_RWSEM;
+ } else if (strtailcmp(base, "rwsem") == 0) {
+ if (mode & PBF_LOCK_MODE_READ) {
+ *prog_groups |= LOCK_PROG_RWSEM_READ;
+ *required_groups |= LOCK_PROG_RWSEM_READ;
+ }
+ if (mode & PBF_LOCK_MODE_WRITE) {
+ *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM;
+ *required_groups |= LOCK_PROG_RWSEM_WRITE;
+ }
+ } else if (strtailcmp(base, "spinlock") == 0) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ } else if (strtailcmp(base, "rwlock") == 0) {
+ if (mode & PBF_LOCK_MODE_READ)
+ groups |= LOCK_PROG_RWLOCK_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ groups |= LOCK_PROG_RWLOCK_WRITE;
+ *prog_groups |= groups;
+ *required_groups |= groups;
+ } else {
+ *prog_groups |= LOCK_PROG_ALL;
+ }
+
+ if (!has_amp && find_kernel_lock_sym(base, &dummy) != 0)
+ *needs_slab_iter = true;
+}
+
+static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter,
+ unsigned int *prog_groups,
+ unsigned int *required_groups,
+ bool *needs_lock_syms,
+ bool *needs_slab_iter)
+{
+ struct perf_bpf_filter_expr *expr, *group;
+
+ *has_lock_filter = false;
+ *prog_groups = 0;
+ *required_groups = 0;
+ *needs_lock_syms = false;
+ *needs_slab_iter = false;
+
+ list_for_each_entry(expr, &evsel->bpf_filters, list) {
+ prescan_one_lock_expr(expr, has_lock_filter, prog_groups,
+ required_groups, needs_lock_syms,
+ needs_slab_iter);
+ if (expr->op == PBF_OP_GROUP_BEGIN) {
+ list_for_each_entry(group, &expr->groups, list) {
+ prescan_one_lock_expr(group, has_lock_filter,
+ prog_groups, required_groups,
+ needs_lock_syms, needs_slab_iter);
+ }
+ }
+ }
+}
+
+static void run_slab_cache_iter(struct sample_filter_bpf *skel)
+{
+ struct bpf_link *link;
+ int fd;
+ char buf[256];
+
+ if (!bpf_program__autoload(skel->progs.slab_cache_iter))
+ return;
+
+ link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL);
+ if (!link || IS_ERR(link)) {
+ pr_debug("cannot attach slab cache iter\n");
+ return;
+ }
+
+ fd = bpf_iter_create(bpf_link__fd(link));
+ if (fd >= 0) {
+ while (read(fd, buf, sizeof(buf)) > 0)
+ continue;
+ close(fd);
+ }
+
+ bpf_link__destroy(link);
+}
+
+static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
+ u64 *kmem_cache_addr)
+{
+ int cache_fd;
+ u64 key, *prev_key = NULL;
+
+ if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter))
+ return false;
+
+ cache_fd = bpf_map__fd(skel->maps.slab_caches);
+ while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) {
+ struct slab_cache_data data;
+
+ if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0)
+ break;
+
+ if (!strcmp(data.name, name)) {
+ *kmem_cache_addr = key;
+ return true;
+ }
+ prev_key = &key;
+ }
+ return false;
+}
+
+/*
+ * Resolve a `lock_held(<name>)` filter expression into `entry->part`
+ * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate
+ * the corresponding BPF filter maps and BSS control flags.
+ *
+ * Resolution order:
+ * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
+ * `rq_lock`, `zone_lock` (with or without leading '&').
+ * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`,
+ * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`,
+ * `semaphore`, etc.
+ * 3. Hexadecimal lock address (when no leading '&').
+ * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`).
+ * 5. Global/static kernel lock symbol in kallsyms (with or without leading '&',
+ * e.g. `&cgroup_mutex` or `cgroup_mutex`).
+ * 6. Slab cache name without leading '&' fallback.
+ */
+static int resolve_lock_expr(struct sample_filter_bpf *skel,
+ struct perf_bpf_filter_expr *expr,
+ struct perf_bpf_filter_entry *entry)
{
- int i = 0;
+ const char *name = expr->lock_name;
+ char base[128];
+ bool has_amp;
+ unsigned int mode;
+ u64 addr = 0, encoded_pair = 0;
+ u8 one = 1;
+ int ret;
+
+ if (split_lock_name_mode(name, expr->part, base, sizeof(base),
+ &has_amp, &mode) < 0) {
+ pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: "");
+ return -EINVAL;
+ }
+
+ if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
+ entry->part = PBF_LOCK_MMAP | mode;
+ entry->target = 0;
+ if (skel)
+ skel->bss->has_mmap_lock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "siglock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_SIGHAND | mode;
+ entry->target = 0;
+ if (skel)
+ skel->bss->has_siglock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "rq_lock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_CLASS | mode;
+ entry->target = LOCK_CLASS_RQLOCK;
+ if (skel)
+ skel->bss->has_lock_class = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "zone_lock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_CLASS | mode;
+ entry->target = LOCK_CLASS_ZONE_LOCK;
+ if (skel)
+ skel->bss->has_lock_class = 1;
+ return 0;
+ }
+
+ if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
+ entry->part = PBF_LOCK_TYPE | mode;
+ entry->target = encoded_pair;
+ return 0;
+ }
+
+ if (!has_amp && parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) {
+ pr_err("Error: Lock type '%s' does not support the requested read/write mode\n",
+ name);
+ return -EINVAL;
+ }
+
+ if (!has_amp && parse_lock_hex_addr(base, &addr)) {
+ entry->part = PBF_LOCK_ADDR | mode;
+ entry->target = addr;
+ if (skel) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
+ }
+ return 0;
+ }
+
+ if (has_amp && find_slab_cache(skel, base, &addr)) {
+ int fd = bpf_map__fd(skel->maps.slab_filter);
+
+ entry->part = PBF_LOCK_SLAB | mode;
+ entry->target = addr;
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_slab = 1;
+ return 0;
+ }
+
+ ret = find_kernel_lock_sym(base, &addr);
+ if (ret == 0) {
+ unsigned int direct_flag = 0;
+
+ if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0)
+ direct_flag = PBF_LOCK_DIRECT_RTMUTEX;
+ else if (strtailcmp(base, "mutex") == 0)
+ direct_flag = PBF_LOCK_DIRECT_MUTEX;
+ else if (strtailcmp(base, "rwsem") == 0 &&
+ strtailcmp(base, "percpu_rwsem") != 0 &&
+ strtailcmp(base, "pcpu_rwsem") != 0)
+ direct_flag = PBF_LOCK_DIRECT_RWSEM;
+
+ entry->part = PBF_LOCK_ADDR | mode | direct_flag;
+ entry->target = addr;
+ if (skel) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
+ }
+ return 0;
+ }
+ if (ret == -EACCES)
+ return ret;
+
+ if (!has_amp && find_slab_cache(skel, base, &addr)) {
+ int fd = bpf_map__fd(skel->maps.slab_filter);
+
+ entry->part = PBF_LOCK_SLAB | mode;
+ entry->target = addr;
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_slab = 1;
+ return 0;
+ }
+
+ pr_err("Error: Cannot resolve kernel lock '%s'\n", name);
+ return -EINVAL;
+}
+
+static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct perf_bpf_filter_entry *entry)
+{
+ int i = 0, ret;
struct perf_bpf_filter_expr *expr;
list_for_each_entry(expr, &evsel->bpf_filters, list) {
@@ -171,6 +1020,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].part = expr->part;
entry[i].term = expr->term;
entry[i].value = expr->val;
+ entry[i].target = 0;
+ if (expr->term == PBF_TERM_LOCK_HELD) {
+ ret = resolve_lock_expr(skel, expr, &entry[i]);
+ if (ret < 0)
+ return ret;
+ }
i++;
if (expr->op == PBF_OP_GROUP_BEGIN) {
@@ -184,6 +1039,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].part = group->part;
entry[i].term = group->term;
entry[i].value = group->val;
+ entry[i].target = 0;
+ if (group->term == PBF_TERM_LOCK_HELD) {
+ ret = resolve_lock_expr(skel, group, &entry[i]);
+ if (ret < 0)
+ return ret;
+ }
i++;
}
@@ -423,10 +1284,14 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
{
int i, x, y, fd, ret;
struct sample_filter_bpf *skel = NULL;
+ struct btf *btf = NULL;
struct bpf_program *prog;
struct bpf_link *link;
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
+ bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
+ bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false;
+ unsigned int prog_groups = 0, required_groups = 0;
#if LIBBPF_CURRENT_VERSION_GEQ(1, 7)
DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts,
.dont_enable = true);
@@ -438,15 +1303,25 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
if (entry == NULL)
return -1;
- ret = get_filter_entries(evsel, entry);
- if (ret < 0) {
- pr_err("Failed to process filter entries\n");
- goto err;
- }
+ prescan_lock_filters(evsel, &has_lock_filter, &prog_groups,
+ &required_groups, &needs_lock_syms,
+ &needs_slab_iter);
- if (needs_idx_hash && geteuid() != 0) {
+ if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
int zero = 0;
+ if (has_lock_filter || is_tracepoint) {
+ pr_err("Error: BPF filter requires CAP_BPF capability\n");
+ ret = -EPERM;
+ goto err;
+ }
+
+ ret = get_filter_entries(evsel, NULL, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
+
/* The filters map is shared among other processes */
ret = create_idx_hash(evsel, entry);
if (ret < 0)
@@ -484,13 +1359,71 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return 0;
}
- skel = sample_filter_bpf__open_and_load();
+ skel = sample_filter_bpf__open();
if (!skel) {
+ ret = -errno;
+ pr_err("Failed to open perf sample-filter BPF skeleton\n");
+ goto err;
+ }
+
+ if (has_lock_filter) {
+ btf = btf__load_vmlinux_btf();
+ bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES);
+ bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS);
+ if (needs_slab_iter) {
+ bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS);
+ bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES);
+ }
+ if (needs_lock_syms) {
+ bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
+ init_numa_data(skel, btf);
+ }
+ ret = configure_lock_progs(skel, btf, prog_groups, required_groups,
+ needs_lock_syms, needs_slab_iter,
+ is_tracepoint);
+ if (ret < 0)
+ goto err;
+ } else {
+ configure_lock_progs(skel, NULL, 0, 0, false, false, is_tracepoint);
+ }
+
+ if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
pr_err("Failed to load perf sample-filter BPF skeleton\n");
goto err;
}
+ if (has_lock_filter && needs_slab_iter)
+ run_slab_cache_iter(skel);
+
+ ret = get_filter_entries(evsel, skel, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
+
+ if (has_lock_filter) {
+ if (needs_lock_syms && skel->bss->has_lock_class) {
+ DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts,
+ .flags = BPF_F_TEST_RUN_ON_CPU,
+ );
+ int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms);
+
+ bpf_prog_test_run_opts(prog_fd, &opts);
+ }
+ bpf_program__set_autoattach(skel->progs.perf_sample_filter, false);
+ bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false);
+ bpf_program__set_autoattach(skel->progs.collect_lock_syms, false);
+ bpf_program__set_autoattach(skel->progs.slab_cache_iter, false);
+
+ ret = sample_filter_bpf__attach(skel);
+ if (ret < 0) {
+ pr_err("Failed to attach lock sample-filter BPF programs\n");
+ goto err;
+ }
+ skel->bss->lock_enabled = 1;
+ }
+
i = 0;
fd = bpf_map__fd(skel->maps.filters);
@@ -501,23 +1434,44 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- prog = skel->progs.perf_sample_filter;
- for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
- for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
- link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y),
- &pe_opts);
- if (IS_ERR(link)) {
- pr_err("Failed to attach perf sample-filter program\n");
- ret = PTR_ERR(link);
- goto err;
+ if (is_tracepoint) {
+ /*
+ * For PERF_TYPE_TRACEPOINT, PERF_EVENT_IOC_SET_BPF attaches the
+ * BPF program to event->tp_event->prog_array on the shared
+ * trace_event_call, which perf_trace_run_bpf_submit() executes
+ * for all CPUs and tasks before iterating the per-CPU perf
+ * events. Attach once on the first FD so tp_event->prog_array
+ * does not get duplicate copies for each per-CPU FD.
+ */
+ prog = skel->progs.perf_tp_sample_filter;
+ link = bpf_program__attach_perf_event_opts(prog, FD(evsel, 0, 0),
+ &pe_opts);
+ ret = libbpf_get_error(link);
+ if (ret) {
+ pr_err("Failed to attach perf tracepoint sample-filter program\n");
+ goto err;
+ }
+ } else {
+ prog = skel->progs.perf_sample_filter;
+ for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
+ for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
+ link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y),
+ &pe_opts);
+ ret = libbpf_get_error(link);
+ if (ret) {
+ pr_err("Failed to attach perf sample-filter program\n");
+ goto err;
+ }
}
}
}
+ btf__free(btf);
free(entry);
evsel->bpf_skel = skel;
return 0;
err:
+ btf__free(btf);
free(entry);
if (!list_empty(&pinned_filters)) {
struct pinned_filter_idx *pfi, *tmp;
@@ -532,6 +1486,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return ret;
}
+void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr)
+{
+ struct perf_bpf_filter_expr *group, *tmp;
+
+ if (!expr)
+ return;
+
+ list_for_each_entry_safe(group, tmp, &expr->groups, list) {
+ list_del(&group->list);
+ perf_bpf_filter_expr__delete(group);
+ }
+ zfree(&expr->lock_name);
+ free(expr);
+}
+
int perf_bpf_filter__destroy(struct evsel *evsel)
{
struct perf_bpf_filter_expr *expr, *tmp;
@@ -539,7 +1508,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel)
list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) {
list_del(&expr->list);
- free(expr);
+ perf_bpf_filter_expr__delete(expr);
}
sample_filter_bpf__destroy(evsel->bpf_skel);
@@ -591,7 +1560,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
{
struct perf_bpf_filter_expr *expr;
- expr = malloc(sizeof(*expr));
+ expr = zalloc(sizeof(*expr));
if (expr != NULL) {
expr->term = term;
expr->part = part;
@@ -602,6 +1571,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
return expr;
}
+struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
+ int mode,
+ enum perf_bpf_filter_op op,
+ unsigned long val)
+{
+ struct perf_bpf_filter_expr *expr;
+
+ if (!lock_name || !*lock_name)
+ return NULL;
+
+ expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val);
+ if (!expr)
+ return NULL;
+
+ expr->lock_name = strdup(lock_name);
+ if (!expr->lock_name) {
+ free(expr);
+ return NULL;
+ }
+ return expr;
+}
+
static bool check_bpf_filter_capable(void)
{
int fd;
@@ -622,14 +1613,11 @@ static bool check_bpf_filter_capable(void)
return false;
}
-int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
+int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
{
YY_BUFFER_STATE buffer;
int ret;
- if (!check_bpf_filter_capable())
- return -EPERM;
-
buffer = perf_bpf_filter__scan_string(str);
ret = perf_bpf_filter_parse(expr_head);
@@ -641,6 +1629,53 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
return ret;
}
+int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
+{
+ if (!check_bpf_filter_capable())
+ return -EPERM;
+
+ return __perf_bpf_filter__parse(expr_head, str);
+}
+
+static int pin_filter_object(struct sample_filter_bpf *skel, const char *path)
+{
+ char *prog_path = NULL;
+ int ret;
+
+ ret = bpf_object__pin_maps(skel->obj, path);
+ if (ret < 0)
+ return ret;
+
+ if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
+ bpf_object__unpin_maps(skel->obj, path);
+ return -ENOMEM;
+ }
+
+ ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path);
+ if (ret < 0)
+ bpf_object__unpin_maps(skel->obj, path);
+
+ free(prog_path);
+ return ret;
+}
+
+static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path)
+{
+ char *prog_path = NULL;
+ int ret, map_ret;
+
+ if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
+ bpf_object__unpin_maps(skel->obj, path);
+ return -ENOMEM;
+ }
+
+ ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path);
+ free(prog_path);
+
+ map_ret = bpf_object__unpin_maps(skel->obj, path);
+ return ret < 0 ? ret : map_ret;
+}
+
int perf_bpf_filter__pin(void)
{
struct sample_filter_bpf *skel;
@@ -654,6 +1689,8 @@ int perf_bpf_filter__pin(void)
goto err;
}
+ configure_lock_progs(skel, NULL, 0, 0, false, false, false);
+
/* pinned program will use pid-hash */
bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH);
@@ -674,7 +1711,7 @@ int perf_bpf_filter__pin(void)
goto err;
}
- ret = bpf_object__pin(skel->obj, path);
+ ret = pin_filter_object(skel, path);
if (ret < 0) {
pr_err("Failed to pin BPF filter objects\n");
goto err;
@@ -683,8 +1720,8 @@ int perf_bpf_filter__pin(void)
/* setup access permissions for the pinned objects */
dir_fd = open(path, O_PATH);
if (dir_fd < 0) {
- bpf_object__unpin(skel->obj, path);
- ret = dir_fd;
+ ret = -errno;
+ unpin_filter_object(skel, path);
goto err;
}
@@ -727,6 +1764,8 @@ int perf_bpf_filter__pin(void)
err_close:
close(dir_fd);
+ if (ret < 0)
+ unpin_filter_object(skel, path);
err:
free(path);
@@ -740,13 +1779,20 @@ int perf_bpf_filter__unpin(void)
char *path = NULL;
int ret = -1;
- skel = sample_filter_bpf__open_and_load();
+ skel = sample_filter_bpf__open();
if (!skel) {
ret = -errno;
pr_err("Failed to open perf sample-filter BPF skeleton\n");
goto err;
}
+ configure_lock_progs(skel, NULL, 0, 0, false, false, false);
+ if (sample_filter_bpf__load(skel) < 0) {
+ ret = -errno;
+ pr_err("Failed to load perf sample-filter BPF skeleton\n");
+ goto err;
+ }
+
if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(),
PERF_BPF_FILTER_PIN_PATH) < 0) {
ret = -errno;
@@ -754,7 +1800,7 @@ int perf_bpf_filter__unpin(void)
goto err;
}
- ret = bpf_object__unpin(skel->obj, path);
+ ret = unpin_filter_object(skel, path);
err:
free(path);
diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h
index 818c554b91b2..fda95ff226b3 100644
--- a/tools/perf/util/bpf-filter.h
+++ b/tools/perf/util/bpf-filter.h
@@ -14,6 +14,13 @@ struct perf_bpf_filter_expr {
int part;
enum perf_bpf_filter_term term;
unsigned long val;
+ /*
+ * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions
+ * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R").
+ * Resolved into `entry.part` and `entry.target` during
+ * perf_bpf_filter__prepare().
+ */
+ char *lock_name;
};
struct evsel;
@@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
int part,
enum perf_bpf_filter_op op,
unsigned long val);
+struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
+ int mode,
+ enum perf_bpf_filter_op op,
+ unsigned long val);
+void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr);
+int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
int perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target);
int perf_bpf_filter__destroy(struct evsel *evsel);
diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l
index 1be9df6550fc..b4022110f1e5 100644
--- a/tools/perf/util/bpf-filter.l
+++ b/tools/perf/util/bpf-filter.l
@@ -1,9 +1,12 @@
%option prefix="perf_bpf_filter_"
%option noyywrap
+%x LOCK_ARG
+
%{
#include <stdio.h>
#include <stdlib.h>
+#include <string.h>
#include <linux/perf_event.h>
#include "bpf-filter.h"
@@ -76,7 +79,7 @@ static int path_or_error(void)
num_dec [0-9]+
num_hex 0[Xx][0-9a-fA-F]+
space [ \t]+
-path [^ \t\n,]+
+path [^ \t\n,()!=>&|<]+
ident [_a-zA-Z][_a-zA-Z0-9]+
%%
@@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); }
uid { return sample(PBF_TERM_UID); }
gid { return sample(PBF_TERM_GID); }
cgroup { return sample_path(PBF_TERM_CGROUP); }
-
+lock_held[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY;
+ return BFT_LOCK_HELD;
+ }
+(lock_held_read|lock_held_shared)[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ;
+ return BFT_LOCK_HELD;
+ }
+(lock_held_write|lock_held_exclusive)[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE;
+ return BFT_LOCK_HELD;
+ }
+
+<LOCK_ARG>{space} { }
+<LOCK_ARG>[^ \t\n()]+ {
+ perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text);
+ return BFT_LOCK_NAME;
+ }
+<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; }
+<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; }
+
+"!" { return '!'; }
"==" { return operator(PBF_OP_EQ); }
"!=" { return operator(PBF_OP_NEQ); }
">" { return operator(PBF_OP_GT); }
diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y
index 5a79a8e7a45b..6ee6edef13f7 100644
--- a/tools/perf/util/bpf-filter.y
+++ b/tools/perf/util/bpf-filter.y
@@ -11,6 +11,7 @@
#include <string.h>
#include <linux/compiler.h>
#include <linux/list.h>
+#include <linux/zalloc.h>
#include "bpf-filter.h"
#include "cgroup.h"
@@ -31,6 +32,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
{
unsigned long num;
char *path;
+ char *lock_name;
+ struct {
+ char *name;
+ int mode;
+ } lock;
struct {
enum perf_bpf_filter_term term;
int part;
@@ -40,12 +46,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
}
%token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH
+%token BFT_LOCK_HELD BFT_LOCK_NAME
%type <expr> filter_term filter_expr
-%destructor { free ($$); } <expr>
+%destructor { perf_bpf_filter_expr__delete($$); } <expr>
+%destructor { free($$); } <lock_name>
+%destructor { free($$.name); } <lock>
%type <sample> BFT_SAMPLE BFT_SAMPLE_PATH
%type <op> BFT_OP
-%type <num> BFT_NUM
+%type <num> BFT_NUM BFT_LOCK_HELD
%type <path> BFT_PATH
+%type <lock_name> BFT_LOCK_NAME
+%type <lock> lock_term
%%
@@ -105,5 +116,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH
$$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id);
cgroup__put(cgrp);
}
+|
+lock_term
+{
+ $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1);
+ zfree(&$1.name);
+ if (!$$)
+ YYERROR;
+}
+|
+'!' lock_term
+{
+ $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0);
+ zfree(&$2.name);
+ if (!$$)
+ YYERROR;
+}
+|
+lock_term BFT_OP BFT_NUM
+{
+ $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3);
+ zfree(&$1.name);
+ if (!$$)
+ YYERROR;
+}
+
+lock_term:
+BFT_LOCK_HELD BFT_LOCK_NAME ')'
+{
+ $$.name = $2;
+ $$.mode = $1;
+}
%%
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index 683fec85e71e..215998897243 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -53,14 +53,79 @@ enum perf_bpf_filter_term {
/* Terms computed from BPF helpers. */
PBF_TERM_UID,
PBF_TERM_GID,
+ /*
+ * Kernel lock held predicate (`lock_held(...)`). Evaluates to 1 if the
+ * specified kernel lock is currently held by the running task/CPU, or
+ * 0 otherwise. The lock specification kind is stored in `entry.part`
+ * (`enum perf_bpf_lock_target_kind`) and the resolved target address,
+ * class ID, or type flag encoding is stored in `entry.target`.
+ */
+ PBF_TERM_LOCK_HELD,
};
+/*
+ * Lock target categories for PBF_TERM_LOCK_HELD filters.
+ * Stored in `perf_bpf_filter_entry.part`.
+ */
+enum perf_bpf_lock_target_kind {
+ PBF_LOCK_NONE = 0,
+ /*
+ * Specific global or static lock address (resolved from kallsyms or
+ * given as a hex address). `entry.target` holds the kernel address.
+ */
+ PBF_LOCK_ADDR,
+ /*
+ * Dynamic lock allocated inside a slab cache (`&slab_name`).
+ * `entry.target` holds the kernel `struct kmem_cache *` address.
+ */
+ PBF_LOCK_SLAB,
+ /*
+ * Per-process `current->mm->mmap_lock` (or `mmap_sem`).
+ */
+ PBF_LOCK_MMAP,
+ /*
+ * Per-process `current->sighand->siglock`.
+ */
+ PBF_LOCK_SIGHAND,
+ /*
+ * Semi-global lock class (`rq_lock` or `zone_lock`).
+ * `entry.target` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or
+ * LOCK_CLASS_ZONE_LOCK).
+ */
+ PBF_LOCK_CLASS,
+ /*
+ * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.).
+ * `entry.target` packs up to two `LCB_F_*` flag combinations in the
+ * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
+ */
+ PBF_LOCK_TYPE,
+};
+
+/*
+ * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD:
+ * Bits 0..3: `enum perf_bpf_lock_target_kind`
+ * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding)
+ * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding)
+ * Bit 6: PBF_LOCK_DIRECT_MUTEX (inspect struct mutex owner)
+ * Bit 7: PBF_LOCK_DIRECT_RWSEM (inspect struct rw_semaphore writer owner)
+ * Bit 8: PBF_LOCK_DIRECT_RTMUTEX (inspect struct rt_mutex_base owner)
+ */
+#define PBF_LOCK_KIND_MASK 0x0fU
+#define PBF_LOCK_MODE_READ (1U << 4)
+#define PBF_LOCK_MODE_WRITE (1U << 5)
+#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE)
+#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY
+#define PBF_LOCK_DIRECT_MUTEX (1U << 6)
+#define PBF_LOCK_DIRECT_RWSEM (1U << 7)
+#define PBF_LOCK_DIRECT_RTMUTEX (1U << 8)
+
/* BPF map entry for filtering */
struct perf_bpf_filter_entry {
enum perf_bpf_filter_op op;
__u32 part; /* sub-sample type info when it has multiple values */
enum perf_bpf_filter_term term;
__u64 value;
+ __u64 target; /* lock address, kmem_cache *, class, or type flags */
};
struct idx_hash_key {
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index e5666d4c1722..583a5b98b929 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -5,8 +5,47 @@
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
+#include "lock_data.h"
#include "sample-filter.h"
+/*
+ * Key for the `held_locks` BPF hash map tracking currently held kernel locks.
+ *
+ * - `addr_or_id`:
+ * - PBF_LOCK_ADDR: kernel virtual address of the lock instance
+ * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK)
+ * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address
+ * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags
+ * - `owner_id`:
+ * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`),
+ * since the holder may sleep and migrate across CPUs while holding the lock.
+ * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks
+ * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption
+ * is disabled on the holding CPU and spinlocks such as `rq_lock` may be
+ * acquired by one task and released by the next task across `context_switch()`.
+ * - `kind`: `enum perf_bpf_lock_target_kind`.
+ */
+struct held_lock_key {
+ __u64 addr_or_id;
+ __u32 owner_id;
+ __u32 kind;
+};
+
+/*
+ * Value for `held_locks` entries.
+ * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`)
+ * and exclusive/write acquisitions (including nested class/slab/type
+ * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of
+ * the lock so release tracepoints that do not carry flags (such as
+ * `lock:contended_release`) can decrement matching class/slab/type entries.
+ */
+struct held_lock_val {
+ __u32 read_count;
+ __u32 write_count;
+ __u32 flags;
+ __u32 pad;
+};
+
/* BPF map that will be filled by user space */
struct filters {
__uint(type, BPF_MAP_TYPE_HASH);
@@ -43,10 +82,220 @@ struct lost_count {
__uint(max_entries, 1);
} dropped SEC(".maps");
+/* Currently held kernel locks tracked by fentry/fexit and lock tracepoints. */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, struct held_lock_key);
+ __type(value, struct held_lock_val);
+ __uint(max_entries, 1);
+} held_locks SEC(".maps");
+
+/* Target lock addresses for PBF_LOCK_ADDR filters. */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, __u64);
+ __type(value, __u8);
+ __uint(max_entries, 1);
+} lock_addr_filter SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u32));
+ __uint(max_entries, 1);
+} lock_syms SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u8));
+ __uint(max_entries, 1);
+} slab_filter SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(struct slab_cache_data));
+ __uint(max_entries, 1);
+} slab_caches SEC(".maps");
+
volatile const int use_idx_hash;
+const volatile __u64 contig_page_data_addr;
+const volatile __u64 node_data_addr;
+const volatile int nr_nodes;
+const volatile int sizeof_zone;
+
+__u32 slab_cache_id;
+
+/* Control flags set by userspace when lock_held(...) filters are active. */
+int lock_enabled;
+int has_lock_addr;
+int has_lock_slab;
+int has_lock_class;
+int has_lock_type;
+int has_mmap_lock;
+int has_siglock;
+int track_rwsem_read;
+__u8 lock_type_filter[LCB_F_TYPE_MAX];
+
void *bpf_cast_to_kern_ctx(void *) __ksym;
+/*
+ * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`.
+ *
+ * For locks that maintain a task_struct pointer in their `owner` field
+ * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`),
+ * `perf_sample_filter` can inspect the lock structure directly in kernel memory
+ * and compare the owner against `current` (`bpf_get_current_task()`).
+ *
+ * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear
+ * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and
+ * spinlocks are checked via the `held_locks` BPF map instead.
+ */
+static __always_inline bool is_rwsem_writer_owner(__u64 lock_addr, __u64 curr)
+{
+ struct rw_semaphore___new *rwsem = (void *)lock_addr;
+
+ if (bpf_core_field_exists(rwsem->count.counter) &&
+ !(BPF_CORE_READ(rwsem, count.counter) & 1UL))
+ return false;
+ if (bpf_core_field_exists(rwsem->owner.counter) &&
+ (BPF_CORE_READ(rwsem, owner.counter) & 1UL))
+ return false;
+
+ return (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr;
+}
+
+static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __u32 part)
+{
+ if (!curr)
+ return false;
+
+ if ((part & PBF_LOCK_DIRECT_MUTEX) &&
+ (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr)
+ return true;
+ if ((part & PBF_LOCK_DIRECT_RWSEM) &&
+ is_rwsem_writer_owner(lock_addr, curr))
+ return true;
+ if ((part & PBF_LOCK_DIRECT_RTMUTEX) &&
+ (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr)
+ return true;
+ return false;
+}
+
+static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode)
+{
+ if (!val)
+ return false;
+ if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0)
+ return true;
+ if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0)
+ return true;
+ return false;
+}
+
+static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind,
+ __u32 mode, __u32 pid, __u32 cpu_id)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = pid,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+
+ if (pid) {
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+ }
+
+ key.owner_id = cpu_id;
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+
+ return 0;
+}
+
+static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode,
+ __u32 pid, __u32 cpu_id)
+{
+ __u32 flag1 = target & 0xff;
+ __u32 flag2 = (target >> 8) & 0xff;
+
+ if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ return 0;
+}
+
+static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry)
+{
+ __u64 curr;
+ __u32 pid;
+ __u32 cpu_id;
+ __u32 kind;
+ __u32 mode;
+ __u64 lock_addr;
+
+ /*
+ * Pinned filters (use_idx_hash != 0) share a world-writable filters
+ * map and do not support lock_held(). Reject evaluation when pinned
+ * or when lock_enabled is not set so unprivileged users cannot probe
+ * arbitrary kernel addresses via entry->target.
+ */
+ if (use_idx_hash || !lock_enabled)
+ return 0;
+
+ curr = bpf_get_current_task();
+ pid = (__u32)bpf_get_current_pid_tgid();
+ cpu_id = 0x80000000U | bpf_get_smp_processor_id();
+ kind = entry->part & PBF_LOCK_KIND_MASK;
+ mode = entry->part & PBF_LOCK_MODE_MASK;
+
+ if (!mode)
+ mode = PBF_LOCK_MODE_ANY;
+
+ switch (kind) {
+ case PBF_LOCK_ADDR:
+ lock_addr = entry->target;
+ if ((mode & PBF_LOCK_MODE_WRITE) &&
+ is_direct_lock_owner(lock_addr, curr, entry->part))
+ return 1;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_MMAP:
+ lock_addr = get_current_mmap_lock(curr);
+ if (!lock_addr)
+ return 0;
+ if ((mode & PBF_LOCK_MODE_WRITE) &&
+ is_rwsem_writer_owner(lock_addr, curr))
+ return 1;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_SIGHAND:
+ lock_addr = get_current_siglock(curr);
+ if (!lock_addr)
+ return 0;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_CLASS:
+ return is_lock_in_map(entry->target, PBF_LOCK_CLASS, mode, pid, cpu_id);
+
+ case PBF_LOCK_SLAB:
+ return is_lock_in_map(entry->target, PBF_LOCK_SLAB, mode, pid, cpu_id);
+
+ case PBF_LOCK_TYPE:
+ return is_lock_type_in_map(entry->target, mode, pid, cpu_id);
+
+ default:
+ return 0;
+ }
+}
+
/* new kernel perf_sample_data definition */
struct perf_sample_data___new {
__u64 sample_flags;
@@ -175,6 +424,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
return bpf_get_current_uid_gid() & 0xFFFFFFFF;
case PBF_TERM_GID:
return bpf_get_current_uid_gid() >> 32;
+ case PBF_TERM_LOCK_HELD:
+ return check_lock_held(entry);
case PBF_TERM_NONE:
case __PBF_UNUSED_TERM4:
case __PBF_UNUSED_TERM5:
@@ -192,6 +443,30 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
return 0;
}
+static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry *entry)
+{
+ switch (entry->term) {
+ case PBF_TERM_TID:
+ if (entry->part)
+ return bpf_get_current_pid_tgid() >> 32;
+ return (__u32)bpf_get_current_pid_tgid();
+ case PBF_TERM_CPU:
+ return bpf_get_smp_processor_id();
+ case PBF_TERM_TIME:
+ return bpf_ktime_get_ns();
+ case PBF_TERM_CGROUP:
+ return bpf_get_current_cgroup_id();
+ case PBF_TERM_UID:
+ return bpf_get_current_uid_gid() & 0xFFFFFFFF;
+ case PBF_TERM_GID:
+ return bpf_get_current_uid_gid() >> 32;
+ case PBF_TERM_LOCK_HELD:
+ return check_lock_held(entry);
+ default:
+ return 0;
+ }
+}
+
#define CHECK_RESULT(data, op, val) \
if (!(data op val)) { \
if (!in_group) \
@@ -295,4 +570,823 @@ int perf_sample_filter(void *ctx)
return 0;
}
+/* BPF program to be called for tracepoint perf events */
+SEC("tracepoint")
+int perf_tp_sample_filter(void *ctx)
+{
+ struct perf_bpf_filter_entry *entry;
+ __u64 sample_data;
+ int in_group = 0;
+ int group_result = 0;
+ int i, k = 0;
+ int *losts;
+
+ entry = bpf_map_lookup_elem(&filters, &k);
+ if (entry == NULL)
+ goto drop;
+
+ for (i = 0; i < MAX_FILTERS; i++) {
+ sample_data = perf_get_common_sample(&entry[i]);
+
+ switch (entry[i].op) {
+ case PBF_OP_EQ:
+ CHECK_RESULT(sample_data, ==, entry[i].value)
+ break;
+ case PBF_OP_NEQ:
+ CHECK_RESULT(sample_data, !=, entry[i].value)
+ break;
+ case PBF_OP_GT:
+ CHECK_RESULT(sample_data, >, entry[i].value)
+ break;
+ case PBF_OP_GE:
+ CHECK_RESULT(sample_data, >=, entry[i].value)
+ break;
+ case PBF_OP_LT:
+ CHECK_RESULT(sample_data, <, entry[i].value)
+ break;
+ case PBF_OP_LE:
+ CHECK_RESULT(sample_data, <=, entry[i].value)
+ break;
+ case PBF_OP_AND:
+ CHECK_RESULT(sample_data, &, entry[i].value)
+ break;
+ case PBF_OP_GROUP_BEGIN:
+ in_group = 1;
+ group_result = 0;
+ break;
+ case PBF_OP_GROUP_END:
+ if (group_result == 0)
+ goto drop;
+ in_group = 0;
+ break;
+ case PBF_OP_DONE:
+ return 1;
+ }
+ }
+ return 1;
+
+drop:
+ losts = bpf_map_lookup_elem(&dropped, &k);
+ if (losts != NULL)
+ __sync_fetch_and_add(losts, 1);
+
+ return 0;
+}
+
+static __always_inline __u32 get_lock_owner_id(__u32 flags)
+{
+ __u32 owner_id;
+
+ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN)
+ return 0x80000000U | bpf_get_smp_processor_id();
+
+ owner_id = (__u32)bpf_get_current_pid_tgid();
+ if (!owner_id)
+ owner_id = 0x80000000U | bpf_get_smp_processor_id();
+ return owner_id;
+}
+
+static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id,
+ __u32 kind, __u32 flags, bool acquired)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = owner_id,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (acquired) {
+ if (val) {
+ if (is_read)
+ __sync_fetch_and_add(&val->read_count, 1);
+ else
+ __sync_fetch_and_add(&val->write_count, 1);
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags,
+ };
+
+ bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY);
+ }
+ } else if (val) {
+ if (is_read) {
+ if (val->read_count <= 1 && val->write_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->read_count > 0)
+ __sync_fetch_and_add(&val->read_count, -1);
+ } else {
+ if (val->write_count <= 1 && val->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->write_count > 0)
+ __sync_fetch_and_add(&val->write_count, -1);
+ }
+ }
+}
+
+#define LCB_F_READ_IRQSAVE (1U << 30)
+#define LCB_F_READ_IRQ_BH (1U << 31)
+
+static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id)
+{
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ bool matched = false;
+ bool is_read = (flags & LCB_F_READ) != 0;
+ __u32 class_id = 0;
+ __u64 kmem_cache_addr = 0;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool match_type = false;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
+ matched = true;
+
+ if (has_mmap_lock || has_siglock) {
+ __u32 ltype = check_lock_type(lock_addr, type_flag);
+
+ if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
+ (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
+ matched = true;
+ }
+
+ if (has_lock_class && type_flag == LCB_F_SPIN) {
+ __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
+
+ if (cid) {
+ class_id = *cid;
+ matched = true;
+ }
+ }
+
+ if (has_lock_slab && bpf_get_kmem_cache) {
+ kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
+ if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr))
+ matched = true;
+ else
+ kmem_cache_addr = 0;
+ }
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) {
+ match_type = true;
+ matched = true;
+ }
+
+ if (!matched)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (existing) {
+ /*
+ * On CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC
+ * kernels, _raw_read_lock_irq() and _raw_read_lock_bh() call
+ * the out-of-line _raw_read_lock_irqsave(). Clear the marker
+ * and skip the duplicate outer fexit increment.
+ */
+ if ((flags & LCB_F_READ_IRQ_BH) &&
+ (existing->flags & LCB_F_READ_IRQSAVE)) {
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ return 0;
+ }
+ /* Non-reader locks are not recursive; avoid double-counting */
+ if (!is_read && existing->write_count > 0)
+ return 0;
+ if (is_read) {
+ __sync_fetch_and_add(&existing->read_count, 1);
+ if (flags & LCB_F_READ_IRQSAVE)
+ existing->flags |= LCB_F_READ_IRQSAVE;
+ else
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ } else {
+ __sync_fetch_and_add(&existing->write_count, 1);
+ }
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags & (LCB_F_TYPE_MASK | LCB_F_READ_IRQSAVE),
+ };
+
+ if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0)
+ return 0;
+ }
+
+ if (class_id)
+ update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, type_flag, true);
+ if (kmem_cache_addr) {
+ update_held_lock_count((__u64)kmem_cache_addr, owner_id,
+ PBF_LOCK_SLAB, type_flag, true);
+ }
+ if (match_type)
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true);
+
+ return 0;
+}
+
+static __always_inline int record_lock_acquired(void *ctx, __u32 flags)
+{
+ return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags));
+}
+
+static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags)
+{
+ __u32 owner_id = get_lock_owner_id(flags);
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (!existing)
+ return 0;
+
+ if (is_read) {
+ if (existing->read_count == 0)
+ return 0;
+ if (existing->read_count == 1 && existing->write_count == 0) {
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ } else {
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ __sync_fetch_and_add(&existing->read_count, -1);
+ }
+ } else {
+ if (existing->write_count == 0)
+ return 0;
+ if (existing->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ else
+ existing->write_count = 0;
+ }
+
+ if (has_lock_class && flags == LCB_F_SPIN) {
+ __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
+
+ if (cid)
+ update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false);
+ }
+
+ if (has_lock_slab && bpf_get_kmem_cache) {
+ __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
+
+ if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) {
+ update_held_lock_count(kmem_cache_addr, owner_id,
+ PBF_LOCK_SLAB, flags, false);
+ }
+ }
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag])
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false);
+
+ return 0;
+}
+
+static __always_inline int record_lock_released(void *ctx, __u32 flags)
+{
+ return record_lock_released_addr(get_lock_addr(ctx), flags);
+}
+
+/* Mutex acquire/release hooks */
+SEC("fexit/mutex_lock")
+int BPF_PROG(fexit_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_nested")
+int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/_mutex_lock_nest_lock")
+int BPF_PROG(fexit__mutex_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io")
+int BPF_PROG(fexit_mutex_lock_io, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io_nested")
+int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_interruptible")
+int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_interruptible_nested")
+int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_killable")
+int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/_mutex_lock_killable")
+int BPF_PROG(fexit__mutex_lock_killable, void *lock, unsigned int subclass,
+ void *nest_lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_trylock")
+int BPF_PROG(fexit_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/_mutex_trylock_nest_lock")
+int BPF_PROG(fexit__mutex_trylock_nest_lock, void *lock, void *nest_lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fentry/mutex_unlock")
+int BPF_PROG(fentry_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_MUTEX);
+}
+
+/* RW semaphore acquire/release hooks */
+SEC("fexit/down_read")
+int BPF_PROG(fexit_down_read, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_nested")
+int BPF_PROG(fexit_down_read_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_interruptible")
+int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable")
+int BPF_PROG(fexit_down_read_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable_nested")
+int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_trylock")
+int BPF_PROG(fexit_down_read_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_write")
+int BPF_PROG(fexit_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_nested")
+int BPF_PROG(fexit_down_write_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/_down_write_nest_lock")
+int BPF_PROG(fexit__down_write_nest_lock, void *sem, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_killable")
+int BPF_PROG(fexit_down_write_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_killable_nested")
+int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_trylock")
+int BPF_PROG(fexit_down_write_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/downgrade_write")
+int BPF_PROG(fexit_downgrade_write, void *sem)
+{
+ if (track_rwsem_read)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+SEC("fentry/up_read")
+int BPF_PROG(fentry_up_read, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_READ);
+}
+
+SEC("fentry/up_write")
+int BPF_PROG(fentry_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+/* Spinlock acquire/release hooks */
+SEC("fexit/_raw_spin_lock")
+int BPF_PROG(fexit_raw_spin_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_nested")
+int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_nest_lock")
+int BPF_PROG(fexit_raw_spin_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq")
+int BPF_PROG(fexit_raw_spin_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq_disable")
+int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave")
+int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave_nested")
+int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_bh")
+int BPF_PROG(fexit_raw_spin_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_trylock")
+int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fexit/_raw_spin_trylock_bh")
+int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fentry/_raw_spin_unlock")
+int BPF_PROG(fentry_raw_spin_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq")
+int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq_enable")
+int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irqrestore")
+int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_bh")
+int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+/* RWlock acquire/release hooks */
+SEC("fexit/_raw_read_lock")
+int BPF_PROG(fexit_raw_read_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_lock_irq")
+int BPF_PROG(fexit_raw_read_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH);
+}
+
+SEC("fexit/_raw_read_lock_irqsave")
+int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQSAVE);
+}
+
+SEC("fexit/_raw_read_lock_bh")
+int BPF_PROG(fexit_raw_read_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH);
+}
+
+SEC("fexit/_raw_read_trylock")
+int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+ return 0;
+}
+
+SEC("fentry/_raw_read_unlock")
+int BPF_PROG(fentry_raw_read_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irq")
+int BPF_PROG(fentry_raw_read_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irqrestore")
+int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_bh")
+int BPF_PROG(fentry_raw_read_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_write_lock")
+int BPF_PROG(fexit_raw_write_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_nested")
+int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irq")
+int BPF_PROG(fexit_raw_write_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irqsave")
+int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_bh")
+int BPF_PROG(fexit_raw_write_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_trylock")
+int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fentry/_raw_write_unlock")
+int BPF_PROG(fentry_raw_write_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irq")
+int BPF_PROG(fentry_raw_write_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irqrestore")
+int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_bh")
+int BPF_PROG(fentry_raw_write_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+/* RT-Mutex acquire/release hooks */
+SEC("fexit/rt_mutex_lock")
+int BPF_PROG(fexit_rt_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_nested")
+int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/_rt_mutex_lock_nest_lock")
+int BPF_PROG(fexit__rt_mutex_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_interruptible")
+int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_lock_killable")
+int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_trylock")
+int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fentry/rt_mutex_unlock")
+int BPF_PROG(fentry_rt_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_RT);
+}
+
+/* Semaphore acquire/release hooks */
+SEC("fexit/down")
+int BPF_PROG(fexit_down, void *sem)
+{
+ return record_lock_acquired(ctx, 0);
+}
+
+SEC("fexit/down_interruptible")
+int BPF_PROG(fexit_down_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_killable")
+int BPF_PROG(fexit_down_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_timeout")
+int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_trylock")
+int BPF_PROG(fexit_down_trylock, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fentry/up")
+int BPF_PROG(fentry_up, void *sem)
+{
+ return record_lock_released(ctx, 0);
+}
+
+/* Percpu-rwsem write acquire/release hooks */
+SEC("fexit/percpu_down_write")
+int BPF_PROG(fexit_percpu_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
+SEC("fentry/percpu_up_write")
+int BPF_PROG(fentry_percpu_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
+SEC("raw_tp/bpf_test_finish")
+int BPF_PROG(collect_lock_syms)
+{
+ return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
+}
+
+SEC("iter/kmem_cache")
+int slab_cache_iter(void *ctx)
+{
+ return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
+}
+
char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index 2f3317ce17f0..1fcb424b75c0 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist,
/* Will a tracepoint filter work for str or should a BPF filter be used? */
static bool is_possible_tp_filter(const char *str)
{
- return strstr(str, "uid") == NULL;
+ return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL &&
+ strstr(str, "lock_held") == NULL;
}
static int set_filter(struct evsel *evsel, const void *arg)
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v3 3/6] perf test: Add tests for lock_held() BPF sample filter
2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-18 5:37 ` [PATCH v3 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-18 5:37 ` [PATCH v3 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers
@ 2026-10-18 5:37 ` Ian Rogers
2026-10-18 5:41 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
` (4 subsequent siblings)
7 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-18 5:37 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, rcu
Add unit tests in parse-events.c for parsing lock_held(),
lock_held_read(), and lock_held_exclusive() BPF filter expressions
(standalone predicates, negated predicates, and compound group
expressions) and extend record_bpf_filter.sh with lock_held(),
lock_held_read()/lock_held_write(), and ':R'/':W' recording tests.
Example output:
# perf test -v 'Parsing of BPF sample filters'
5: Parse event definition strings :
5.7: Parsing of BPF sample filters : Ok
# perf test -v 'perf record sample filtering (by BPF) tests'
112: perf record sample filtering (by BPF) tests :
Checking BPF-filter privilege
Basic bpf-filter test
Basic bpf-filter test [Success]
Failing bpf-filter test
Failing bpf-filter test [Success]
Group bpf-filter test
Group bpf-filter test [Success]
Multiple bpf-filter test
Multiple bpf-filter test [Success]
Cgroup bpf-filter test
Cgroup bpf-filter test [Success]
Lock-held bpf-filter test
Lock-held bpf-filter test [Success]
112: perf record sample filtering (by BPF) tests : Ok
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/tests/parse-events.c | 108 +++++++++++++++++++
tools/perf/tests/shell/record_bpf_filter.sh | 114 ++++++++++++++++++++
2 files changed, 222 insertions(+)
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 5f6f95c8a5b4..4cf9192762f2 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -8,9 +8,11 @@
#include "debug.h"
#include "pmu.h"
#include "pmus.h"
+#include "bpf-filter.h"
#include "strbuf.h"
#include <dirent.h>
#include <errno.h>
+#include <string.h>
#include "fncache.h"
#include <sys/types.h>
#include <sys/stat.h>
@@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused,
return ret;
}
+#ifdef HAVE_BPF_SKEL
+static void clear_bpf_filter_exprs(struct list_head *expr_head)
+{
+ struct perf_bpf_filter_expr *expr, *tmp;
+
+ list_for_each_entry_safe(expr, tmp, expr_head, list) {
+ list_del(&expr->list);
+ perf_bpf_filter_expr__delete(expr);
+ }
+}
+
+static int do_test_bpf_filters(struct list_head *expr_head)
+{
+ struct perf_bpf_filter_expr *expr, *group;
+ const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100";
+
+ /* Simple lock_held(&cgroup_mutex) */
+ TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)",
+ __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is &cgroup_mutex",
+ expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /* Negated !lock_held(cgroup_mutex) */
+ TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)",
+ __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 0", expr->val == 0);
+ TEST_ASSERT_VAL("lock_name is cgroup_mutex",
+ expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /* Read/shared and write/exclusive predicates */
+ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)",
+ __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is &mmap_lock",
+ expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock"));
+ clear_bpf_filter_exprs(expr_head);
+
+ TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)",
+ __perf_bpf_filter__parse(expr_head,
+ "!lock_held_exclusive(mmap_lock)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 0", expr->val == 0);
+ TEST_ASSERT_VAL("lock_name is mmap_lock",
+ expr->lock_name && !strcmp(expr->lock_name, "mmap_lock"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /*
+ * Comparison and compound group:
+ * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100
+ */
+ TEST_ASSERT_VAL("parse compound lock_held filter",
+ __perf_bpf_filter__parse(expr_head, compound) == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN);
+ TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups));
+ group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("first group member lock_name",
+ group->lock_name && !strcmp(group->lock_name, "rwsem:R"));
+ TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1);
+ group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("second group member lock_name",
+ group->lock_name && !strcmp(group->lock_name, "&dentry"));
+ TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0);
+ expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100);
+
+ return TEST_OK;
+}
+#endif
+
+static int test__bpf_filters(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
+{
+#ifdef HAVE_BPF_SKEL
+ struct list_head expr_head = LIST_HEAD_INIT(expr_head);
+ int ret = do_test_bpf_filters(&expr_head);
+
+ clear_bpf_filter_exprs(&expr_head);
+ return ret;
+#else
+ return TEST_SKIP;
+#endif
+}
+
static struct test_case tests__parse_events[] = {
TEST_CASE_REASON("Test event parsing",
events2,
@@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = {
"no aliases in sysfs"),
TEST_CASE("Parsing of aliased events", pmu_events_alias2),
TEST_CASE("Parsing of terms (event modifiers)", terms2),
+ TEST_CASE("Parsing of BPF sample filters", bpf_filters),
{ .name = NULL, }
};
diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh
index 383574cb3bd3..9334b8b9b1c2 100755
--- a/tools/perf/tests/shell/record_bpf_filter.sh
+++ b/tools/perf/tests/shell/record_bpf_filter.sh
@@ -176,6 +176,116 @@ test_bpf_filter_cgroup() {
echo "Cgroup bpf-filter test [Success]"
}
+test_bpf_filter_lock_held() {
+ echo "Lock-held bpf-filter test"
+ lock_sym="cgroup_mutex"
+
+ if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms
+ then
+ lock_sym="slab_mutex"
+ fi
+
+ if ! perf record -e task-clock --filter "lock_held(&${lock_sym})" \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if [ "$(id -u)" != 0 ]
+ then
+ echo "Lock-held bpf-filter test [Skipped permission]"
+ return
+ fi
+ if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms
+ then
+ echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]"
+ return
+ fi
+ echo "Lock-held bpf-filter test [Failed &${lock_sym}]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \
+ -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed mmap_lock]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \
+ -o /dev/null true 2>&1 | grep -q "Kernel BTF lacks required"
+ then
+ echo "Lock-held bpf-filter test [Failed rq_lock/spinlock]"
+ err=1
+ return
+ fi
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(rwsem)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock type]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock \
+ --filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read/write(&mmap_lock)]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(&mmap_lock:R) || lock_held(&mmap_lock:W)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed &mmap_lock:R/:W]"
+ err=1
+ return
+ fi
+
+ if perf list tracepoint 2> /dev/null | grep -q kmem:mm_page_alloc
+ then
+ if ! perf record -e kmem:mm_page_alloc --filter 'lock_held(mutex) || lock_held(rwsem)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed tracepoint kmem:mm_page_alloc]"
+ err=1
+ return
+ fi
+ fi
+
+ if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \
+ -o /dev/null true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]"
+ err=1
+ return
+ fi
+
+ if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \
+ -o /dev/null true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]"
+ err=1
+ return
+ fi
+
+ echo "Lock-held bpf-filter test [Success]"
+}
+
test_bpf_filter_priv
if [ $err = 0 ]; then
@@ -198,5 +308,9 @@ if [ $err = 0 ]; then
test_bpf_filter_cgroup
fi
+if [ $err = 0 ]; then
+ test_bpf_filter_lock_held
+fi
+
cleanup
exit $err
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v3 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (2 preceding siblings ...)
2026-10-18 5:37 ` [PATCH v3 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers
@ 2026-10-18 5:37 ` Ian Rogers
2026-10-18 5:51 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
` (3 subsequent siblings)
7 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-18 5:37 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, rcu
Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks
Trace RCU, and SRCU locks:
- 'rcu' (supports both read/shared and write/exclusive modes):
- Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)',
'lock_held(rcu_read_lock)'): directly inspects
current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on
CONFIG_PREEMPT_RCU kernels), accounting for the baseline
rcu_read_lock() nesting acquired by the perf overflow handler itself
(bpf_overflow_handler() and do_perf_sw_event()) with zero extra
fentry/fexit tracing hooks.
- Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'):
attaches fentry/fexit hooks to synchronize_rcu(),
synchronize_rcu_expedited(), synchronize_rcu_tasks(), and
rcu_barrier() to track grace-period and barrier waits.
- 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both
read/shared and write/exclusive modes):
- Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches
fexit/__srcu_read_lock and fentry/__srcu_read_unlock.
- Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'):
attaches fentry/fexit on synchronize_srcu(),
synchronize_srcu_expedited(), and srcu_barrier().
- 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports
read/shared mode only): directly inspects
current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels
with zero extra tracing hooks.
Example usage:
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ]
# perf report --stdio --no-children
# Overhead Command Shared Object Symbol
# ........ ....... ................. ........................
#
14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu
|
---__d_lookup_rcu
lookup_fast
walk_component
link_path_walk
path_lookupat
filename_lookup
vfs_statx
__do_sys_newstat
do_syscall_64
entry_SYSCALL_64_after_hwframe
# perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1
# perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1
Signed-off-by: Ian Rogers <irogers@google.com>
# Conflicts:
# tools/perf/util/bpf-filter.c
# tools/perf/util/bpf_skel/sample_filter.bpf.c
# Conflicts:
# tools/perf/util/bpf_skel/sample_filter.bpf.c
---
tools/perf/Documentation/perf-record.txt | 42 +++--
tools/perf/tests/parse-events.c | 13 ++
tools/perf/tests/shell/record_bpf_filter.sh | 20 +++
tools/perf/util/bpf-filter.c | 179 +++++++++++++++++--
tools/perf/util/bpf_skel/lock_data.h | 44 ++++-
tools/perf/util/bpf_skel/sample-filter.h | 12 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 135 +++++++++++++-
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 +
tools/perf/util/lock-contention.c | 4 +
tools/perf/util/lock-contention.h | 4 +-
10 files changed, 423 insertions(+), 32 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 39dfe63fe3a8..f45d2eec901c 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -219,11 +219,17 @@ OPTIONS
'&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'),
a hexadecimal lock address, a slab cache name prefixed with '&'
(e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known
- lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock
- type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
- 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
- 'semaphore'). Both read/shared and write/exclusive tracking are
- supported for 'rwsem' (including 'mmap_lock') and 'rwlock';
+ lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock', 'rcu',
+ 'rcu-trace'), or a lock type ('mutex', 'spinlock', 'rwsem', 'rwsem:R',
+ 'rwsem:W', 'rwlock', 'rwlock:R', 'rwlock:W', 'rt-mutex',
+ 'percpu-rwsem', 'pcpu-sem:W', 'srcu', 'srcu:R', 'srcu:W', 'rcu:R',
+ 'rcu:W', 'semaphore'). Both read/shared and write/exclusive tracking
+ are supported for 'rwsem' (including 'mmap_lock'), 'rwlock', 'rcu',
+ and 'srcu' (where read/shared mode matches 'rcu_read_lock()' /
+ 'srcu_read_lock()' critical sections and write/exclusive mode matches
+ grace-period waits 'synchronize_rcu*()', 'synchronize_srcu*()',
+ 'rcu_barrier()', and 'srcu_barrier()'); 'rcu-trace'
+ ('rcu_read_lock_trace()') supports read/shared tracking only; and
'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'),
'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()')
support write/exclusive tracking only.
@@ -243,17 +249,21 @@ OPTIONS
Note that 'lock_held(<lock>)' introduces overhead outside of sample
overflow handling: in addition to checking the lock's owner field on
- each sample where possible (such as for mutexes, rt-mutexes, and
- write-locked rwsems), it attaches BPF fentry/fexit programs to kernel
- lock acquire and release functions to track held locks (such as
- spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a
- BPF map. These BPF hooks run on every kernel lock acquisition and
- release for the attached lock family, even when no perf sample is
- being taken. To minimize this overhead, only the hooks for the
- relevant lock family and mode are attached when known (for example,
- 'lock_held_read(&mmap_lock)' only hooks rwsem read functions,
- 'lock_held_write(&mmap_lock)' only hooks rwsem write functions,
- '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks
+ each sample where possible (such as for mutexes, rt-mutexes,
+ write-locked rwsems, and 'rcu:R' / 'rcu-trace' which inspect
+ 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting'
+ directly with no extra tracing hooks), it attaches BPF fentry/fexit
+ programs to kernel lock acquire and release functions to track held
+ locks (such as spinlocks, rwlocks, reader-locked rwsems, SRCU locks,
+ RCU synchronize waits, and slab/type filters) in a BPF map. These
+ BPF hooks run on every kernel lock acquisition and release for the
+ attached lock family, even when no perf sample is being taken. To
+ minimize this overhead, only the hooks for the relevant lock family
+ and mode are attached when known (for example, 'lock_held_read(rcu)'
+ and 'rcu-trace' attach no extra hooks, 'lock_held_read(&mmap_lock)'
+ only hooks rwsem read functions, 'lock_held_write(&mmap_lock)' only
+ hooks rwsem write functions, '&cgroup_mutex' only hooks mutex
+ functions, 'srcu' only hooks SRCU functions, and 'rq_lock' only hooks
spinlock functions), whereas slab filters ('&<slab>'), raw hex
addresses, or symbols without a recognizable lock-type suffix attach
hooks across all lock families (including high-frequency spinlocks).
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 4cf9192762f2..730466b7ebf6 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head)
expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex"));
clear_bpf_filter_exprs(expr_head);
+ /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */
+ TEST_ASSERT_VAL("parse lock_held(rcu)",
+ __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is rcu",
+ expr->lock_name && !strcmp(expr->lock_name, "rcu"));
+ clear_bpf_filter_exprs(expr_head);
+
/* Read/shared and write/exclusive predicates */
TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)",
__perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0);
diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh
index 9334b8b9b1c2..6b58f1556c15 100755
--- a/tools/perf/tests/shell/record_bpf_filter.sh
+++ b/tools/perf/tests/shell/record_bpf_filter.sh
@@ -239,6 +239,26 @@ test_bpf_filter_lock_held() {
return
fi
+ if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed rcu:W/srcu]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \
+ -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]"
+ err=1
+ return
+ fi
+ fi
+
if ! perf record -e task-clock \
--filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \
-o "${perfdata}" true 2> /dev/null
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 0c133b31b5d5..4a5ef78e2690 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -201,11 +201,19 @@ enum lock_prog_group {
LOCK_PROG_RTMUTEX = 1U << 6,
LOCK_PROG_SEM = 1U << 7,
LOCK_PROG_PERCPU_RWSEM = 1U << 8,
- LOCK_PROG_ALL = 0x1ffU,
+ LOCK_PROG_SRCU_READ = 1U << 9,
+ LOCK_PROG_SRCU_WRITE = 1U << 10,
+ LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE,
+ LOCK_PROG_RCU_WRITE = 1U << 11,
+ LOCK_PROG_ALL = 0xfffU,
};
static unsigned int lock_flags_to_prog_group(unsigned int flags)
{
+ if (flags & LCB_F_RCU)
+ return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0;
+ if (flags & LCB_F_SRCU)
+ return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE;
if (flags & LCB_F_MUTEX)
return LOCK_PROG_MUTEX;
if (flags & LCB_F_RT)
@@ -325,6 +333,32 @@ static const struct lock_prog_desc lock_progs[] = {
/* Percpu-RWsem */
LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+ /* SRCU */
+ LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe",
+ LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe",
+ LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited",
+ LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited",
+ LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE),
+ /* RCU grace-period synchronize / barrier */
+ LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited",
+ LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited",
+ LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE),
};
#undef LOCK_FPROG
@@ -610,6 +644,18 @@ static unsigned int supported_lock_prog_groups(struct btf *btf)
if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write"))
supported |= LOCK_PROG_PERCPU_RWSEM;
+ if ((btf_has_func(btf, "__srcu_read_lock") ||
+ btf_has_func(btf, "__srcu_read_lock_nmisafe")) &&
+ (btf_has_func(btf, "__srcu_read_unlock") ||
+ btf_has_func(btf, "__srcu_read_unlock_nmisafe")))
+ supported |= LOCK_PROG_SRCU_READ;
+
+ if (btf_has_func(btf, "synchronize_srcu"))
+ supported |= LOCK_PROG_SRCU_WRITE;
+
+ if (btf_has_func(btf, "synchronize_rcu"))
+ supported |= LOCK_PROG_RCU_WRITE;
+
return supported;
}
@@ -718,6 +764,19 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
*needs_lock_syms = true;
return;
}
+ if (!strcmp(base, "rcu")) {
+ if (mode & PBF_LOCK_MODE_WRITE) {
+ *prog_groups |= LOCK_PROG_RCU_WRITE;
+ *required_groups |= LOCK_PROG_RCU_WRITE;
+ }
+ return;
+ }
+ if (!strcmp(base, "rcu_read_lock") ||
+ !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") ||
+ !strcmp(base, "rcu_read_lock_trace")) {
+ /* Inspected directly on task_struct; no tracing hooks needed. */
+ return;
+ }
if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, &groups)) {
*prog_groups |= groups;
*required_groups |= groups;
@@ -761,6 +820,13 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
groups |= LOCK_PROG_RWLOCK_WRITE;
*prog_groups |= groups;
*required_groups |= groups;
+ } else if (strtailcmp(base, "srcu") == 0) {
+ if (mode & PBF_LOCK_MODE_READ)
+ groups |= LOCK_PROG_SRCU_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ groups |= LOCK_PROG_SRCU_WRITE;
+ *prog_groups |= groups;
+ *required_groups |= groups;
} else {
*prog_groups |= LOCK_PROG_ALL;
}
@@ -847,16 +913,54 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
return false;
}
+static u64 rcu_min_depth(struct evsel *evsel)
+{
+ if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE &&
+ evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK &&
+ evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK)
+ return 2;
+ return 1;
+}
+
+static bool btf_has_task_field(struct btf *btf, const char *field_name)
+{
+ const struct btf_type *t;
+ const struct btf_member *m;
+ s32 id;
+ u16 i, vlen;
+
+ if (!btf)
+ return false;
+
+ id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT);
+ if (id < 0)
+ return false;
+
+ t = btf__type_by_id(btf, id);
+ if (!t)
+ return false;
+
+ m = btf_members(t);
+ vlen = btf_vlen(t);
+ for (i = 0; i < vlen; i++, m++) {
+ const char *name = btf__name_by_offset(btf, m->name_off);
+
+ if (name && !strcmp(name, field_name))
+ return true;
+ }
+ return false;
+}
+
/*
* Resolve a `lock_held(<name>)` filter expression into `entry->part`
* (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate
* the corresponding BPF filter maps and BSS control flags.
*
* Resolution order:
- * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
- * `rq_lock`, `zone_lock` (with or without leading '&').
+ * 1. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`,
+ * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` (with or without leading '&').
* 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`,
- * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`,
+ * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, `srcu`,
* `semaphore`, etc.
* 3. Hexadecimal lock address (when no leading '&').
* 4. Slab cache name (when prefixed with '&', e.g. `&dentry`).
@@ -864,8 +968,8 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
* e.g. `&cgroup_mutex` or `cgroup_mutex`).
* 6. Slab cache name without leading '&' fallback.
*/
-static int resolve_lock_expr(struct sample_filter_bpf *skel,
- struct perf_bpf_filter_expr *expr,
+static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct btf *btf, struct perf_bpf_filter_expr *expr,
struct perf_bpf_filter_entry *entry)
{
const char *name = expr->lock_name;
@@ -929,6 +1033,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel,
return 0;
}
+ if (!strcmp(base, "rcu_read_lock")) {
+ if (mode == PBF_LOCK_MODE_WRITE) {
+ pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
+ return -EINVAL;
+ }
+ if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
+ pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ;
+ entry->target = rcu_min_depth(evsel);
+ return 0;
+ }
+
+ if (!strcmp(base, "rcu")) {
+ if ((mode & PBF_LOCK_MODE_READ) && btf &&
+ !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
+ pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU | mode;
+ entry->target = rcu_min_depth(evsel);
+ if ((mode & PBF_LOCK_MODE_WRITE) && skel)
+ skel->bss->has_rcu_lock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") ||
+ !strcmp(base, "rcu_read_lock_trace")) {
+ if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) {
+ pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
+ return -EINVAL;
+ }
+ if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) {
+ pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ;
+ entry->target = 0;
+ return 0;
+ }
+
if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
entry->part = PBF_LOCK_TYPE | mode;
entry->target = encoded_pair;
@@ -1004,7 +1150,7 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel,
}
static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
- struct perf_bpf_filter_entry *entry)
+ struct btf *btf, struct perf_bpf_filter_entry *entry)
{
int i = 0, ret;
struct perf_bpf_filter_expr *expr;
@@ -1022,7 +1168,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske
entry[i].value = expr->val;
entry[i].target = 0;
if (expr->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(skel, expr, &entry[i]);
+ ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]);
if (ret < 0)
return ret;
}
@@ -1041,7 +1187,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske
entry[i].value = group->val;
entry[i].target = 0;
if (group->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(skel, group, &entry[i]);
+ ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]);
if (ret < 0)
return ret;
}
@@ -1316,7 +1462,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- ret = get_filter_entries(evsel, NULL, entry);
+ ret = get_filter_entries(evsel, NULL, NULL, entry);
if (ret < 0) {
pr_err("Failed to process filter entries\n");
goto err;
@@ -1378,6 +1524,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
init_numa_data(skel, btf);
}
+ if (prog_groups & LOCK_PROG_RCU_WRITE) {
+ u64 rcu_addr = 1ULL;
+
+ /*
+ * Offset by 1 byte so rcu_state_addr is distinct from
+ * &rcu_state.node[0].lock at offset 0 of rcu_state.
+ */
+ if (find_kernel_lock_sym("rcu_state", &rcu_addr) == 0)
+ rcu_addr += 1;
+ skel->rodata->rcu_state_addr = rcu_addr;
+ }
ret = configure_lock_progs(skel, btf, prog_groups, required_groups,
needs_lock_syms, needs_slab_iter,
is_tracepoint);
@@ -1396,7 +1553,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
if (has_lock_filter && needs_slab_iter)
run_slab_cache_iter(skel);
- ret = get_filter_entries(evsel, skel, entry);
+ ret = get_filter_entries(evsel, skel, btf, entry);
if (ret < 0) {
pr_err("Failed to process filter entries\n");
goto err;
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index 8cf24f07204c..d11ad1746d09 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -39,6 +39,7 @@ struct contention_task_data {
*/
#define LCD_F_MMAP_LOCK (1U << 31)
#define LCD_F_SIGHAND_LOCK (1U << 30)
+#define LCD_F_RCU_LOCK (1U << 29)
#define LCB_F_SLAB_ID_SHIFT 16
#define LCB_F_SLAB_ID_START (1U << 16)
@@ -52,9 +53,11 @@ struct contention_task_data {
#define LCB_F_RT (1U << 3)
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+#define LCB_F_SRCU (1U << 6)
+#define LCB_F_RCU (1U << 7)
-#define LCB_F_TYPE_MAX (1U << 7)
-#define LCB_F_TYPE_MASK 0x0000007FU
+#define LCB_F_TYPE_MAX (1U << 8)
+#define LCB_F_TYPE_MASK 0x000000FFU
#define SLAB_NAME_MAX 28
@@ -233,6 +236,40 @@ static __always_inline __u64 get_current_siglock(__u64 curr)
return sh + bpf_core_field_offset(sighand->siglock);
}
+/*
+ * Check if `curr` is inside a preemptible RCU read-side critical section
+ * (`rcu_read_lock()`).
+ *
+ * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires
+ * `rcu_read_lock()` before invoking `perf_sample_filter` (and
+ * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for
+ * non-clock software events), `min_depth` is the baseline nesting count
+ * contributed by the perf event overflow handler itself (1 or 2).
+ */
+static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth)
+{
+ struct task_struct *task = (void *)curr;
+
+ if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting))
+ return false;
+
+ return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth;
+}
+
+/*
+ * Check if `curr` is inside a Tasks Trace RCU read-side critical section
+ * (`rcu_read_lock_trace()`).
+ */
+static __always_inline bool is_rcu_trace_lock_held(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+
+ if (!curr || !bpf_core_field_exists(task->trc_reader_nesting))
+ return false;
+
+ return BPF_CORE_READ(task, trc_reader_nesting) > 0;
+}
+
/*
* Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar.
*
@@ -265,6 +302,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags)
if (get_current_siglock(curr) == lock)
return LCD_F_SIGHAND_LOCK;
break;
+ case LCB_F_RCU:
+ case LCB_F_RCU | LCB_F_WRITE:
+ return LCD_F_RCU_LOCK;
default:
break;
}
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index 215998897243..f6a030abfb0d 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -99,6 +99,18 @@ enum perf_bpf_lock_target_kind {
* low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
*/
PBF_LOCK_TYPE,
+ /*
+ * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`).
+ * Inspected directly via `current->rcu_read_lock_nesting > entry.target`,
+ * where `entry.target` is the baseline RCU nesting depth added by the
+ * kernel perf overflow path.
+ */
+ PBF_LOCK_RCU,
+ /*
+ * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`).
+ * Inspected directly via `current->trc_reader_nesting > 0`.
+ */
+ PBF_LOCK_RCU_TRACE,
};
/*
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index 583a5b98b929..776425c27a4a 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -123,6 +123,7 @@ volatile const int use_idx_hash;
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
+const volatile __u64 rcu_state_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
@@ -136,6 +137,7 @@ int has_lock_class;
int has_lock_type;
int has_mmap_lock;
int has_siglock;
+int has_rcu_lock;
int track_rwsem_read;
__u8 lock_type_filter[LCB_F_TYPE_MAX];
@@ -291,6 +293,20 @@ static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry
case PBF_LOCK_TYPE:
return is_lock_type_in_map(entry->target, mode, pid, cpu_id);
+ case PBF_LOCK_RCU:
+ if ((mode & PBF_LOCK_MODE_READ) &&
+ is_rcu_lock_held(curr, (int)entry->target))
+ return 1;
+ if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr &&
+ is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id))
+ return 1;
+ return 0;
+
+ case PBF_LOCK_RCU_TRACE:
+ if ((mode & PBF_LOCK_MODE_READ) && is_rcu_trace_lock_held(curr))
+ return 1;
+ return 0;
+
default:
return 0;
}
@@ -712,11 +728,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla
if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
matched = true;
- if (has_mmap_lock || has_siglock) {
+ if (has_mmap_lock || has_siglock || has_rcu_lock) {
__u32 ltype = check_lock_type(lock_addr, type_flag);
if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
- (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
+ (has_siglock && ltype == LCD_F_SIGHAND_LOCK) ||
+ (has_rcu_lock && ltype == LCD_F_RCU_LOCK))
matched = true;
}
@@ -1376,6 +1393,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem)
return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
}
+/* SRCU acquire/release and synchronize hooks */
+SEC("fexit/__srcu_read_lock")
+int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/__srcu_read_unlock")
+int BPF_PROG(fentry_srcu_read_unlock, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fexit/__srcu_read_lock_nmisafe")
+int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/__srcu_read_unlock_nmisafe")
+int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/synchronize_srcu")
+int BPF_PROG(fentry_synchronize_srcu, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu")
+int BPF_PROG(fexit_synchronize_srcu, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_srcu_expedited")
+int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu_expedited")
+int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fentry/srcu_barrier")
+int BPF_PROG(fentry_srcu_barrier, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/srcu_barrier")
+int BPF_PROG(fexit_srcu_barrier, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+/* RCU grace-period synchronize / barrier hooks */
+SEC("fentry/synchronize_rcu")
+int BPF_PROG(fentry_synchronize_rcu)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu")
+int BPF_PROG(fexit_synchronize_rcu)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_rcu_expedited")
+int BPF_PROG(fentry_synchronize_rcu_expedited)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu_expedited")
+int BPF_PROG(fexit_synchronize_rcu_expedited)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_rcu_tasks")
+int BPF_PROG(fentry_synchronize_rcu_tasks)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu_tasks")
+int BPF_PROG(fexit_synchronize_rcu_tasks)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/rcu_barrier")
+int BPF_PROG(fentry_rcu_barrier)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/rcu_barrier")
+int BPF_PROG(fexit_rcu_barrier)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
SEC("raw_tp/bpf_test_finish")
int BPF_PROG(collect_lock_syms)
{
diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
index b75ddc6846bb..1fb91a829d54 100644
--- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
+++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
@@ -119,6 +119,8 @@ struct task_struct {
struct signal_struct *signal;
struct sighand_struct *sighand;
struct css_set *cgroups;
+ int rcu_read_lock_nesting;
+ int trc_reader_nesting;
} __attribute__((preserve_access_index));
struct trace_entry {
diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c
index a1bb78d6e29c..b45b075f3346 100644
--- a/tools/perf/util/lock-contention.c
+++ b/tools/perf/util/lock-contention.c
@@ -38,6 +38,10 @@ const struct perf_lock_type lock_type_table[] = {
{ LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
/* alias for optimistic spinning only */
{ LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
+ { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" },
+ { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" },
+ { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" },
+ { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" },
};
int parse_call_stack(const struct option *opt __maybe_unused, const char *str,
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index 68e9bd77e8a9..88b2c11244ad 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -128,6 +128,8 @@ struct thread_stat {
#define LCB_F_RT (1U << 3)
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+#define LCB_F_SRCU (1U << 6)
+#define LCB_F_RCU (1U << 7)
struct perf_lock_type {
unsigned int flags;
@@ -140,7 +142,7 @@ struct perf_lock_type {
const char *lock_name;
};
-extern const struct perf_lock_type lock_type_table[14];
+extern const struct perf_lock_type lock_type_table[18];
struct evlist;
struct machine;
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v3 5/6] perf lock contention: Extend BPF lock contention to RCU and SRCU
2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (3 preceding siblings ...)
2026-10-18 5:37 ` [PATCH v3 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
@ 2026-10-18 5:37 ` Ian Rogers
2026-10-18 5:48 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers
` (2 subsequent siblings)
7 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-18 5:37 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, rcu
Extend 'perf lock contention --use-bpf' to track RCU and SRCU
grace-period and barrier waits:
- Attach fentry/fexit programs to synchronize_rcu(),
synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(),
synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier()
when available in BTF.
- Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu',
'-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct
symbol names).
Example usage:
# perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1
total wait avg wait type caller
35.12 ms 8.78 ms rcu:W cleanup_net+0x284
7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114
# perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1
total wait avg wait address symbol
35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu)
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/tests/shell/lock_contention.sh | 14 ++
tools/perf/util/bpf_lock_contention.c | 171 ++++++++++++--
.../perf/util/bpf_skel/lock_contention.bpf.c | 221 ++++++++++++++----
tools/perf/util/machine.c | 11 +
5 files changed, 362 insertions(+), 62 deletions(-)
diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt
index c17b3e318169..bfeb98c976e3 100644
--- a/tools/perf/Documentation/perf-lock.txt
+++ b/tools/perf/Documentation/perf-lock.txt
@@ -188,11 +188,14 @@ CONTENTION OPTIONS
Show lock contention only for given lock types (comma separated list).
Available values are:
semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W,
- rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem,
- pcpu-sem:R, pcpu-sem:W, mutex
+ rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem,
+ pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W
Note that RW-variant of locks have :R and :W suffix. Names without the
suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W.
+ For RCU and SRCU with --use-bpf, contention tracks grace-period and
+ barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(),
+ srcu_barrier()).
-L::
--lock-filter=<value>::
diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh
index 5df9a0cd48a3..c6d37c910d4b 100755
--- a/tools/perf/tests/shell/lock_contention.sh
+++ b/tools/perf/tests/shell/lock_contention.sh
@@ -181,6 +181,13 @@ test_type_filter()
err=1
exit
fi
+
+ perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result}
+ if [ "$(grep -c -v rcu "${result}")" != "0" ]; then
+ echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")"
+ err=1
+ exit
+ fi
}
test_lock_filter()
@@ -224,6 +231,13 @@ test_lock_filter()
err=1
exit
fi
+
+ perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result}
+ if [ "$(grep -c -v rcu "${result}")" != "0" ]; then
+ echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")"
+ err=1
+ exit
+ fi
}
test_stack_filter()
diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c
index c20bd075664e..dfb384f1c5e6 100644
--- a/tools/perf/util/bpf_lock_contention.c
+++ b/tools/perf/util/bpf_lock_contention.c
@@ -16,6 +16,7 @@
#include <bpf/bpf.h>
#include <bpf/btf.h>
#include <inttypes.h>
+#include <unistd.h>
#include "bpf_skel/lock_contention.skel.h"
#include "bpf_skel/lock_data.h"
@@ -180,6 +181,133 @@ static void init_numa_data(struct lock_contention *con)
free(buf);
}
+static bool has_bpf_fentry_support(struct btf *btf)
+{
+ /*
+ * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and
+ * bpf_get_func_arg() helper availability (5.17+) used by
+ * get_lock_addr() in the SRCU fentry/fexit hooks.
+ */
+ struct bpf_insn insns[] = {
+ { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 },
+ { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3,
+ .src_reg = BPF_REG_10 },
+ { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 },
+ { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg },
+ { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 },
+ { .code = BPF_JMP | BPF_EXIT },
+ };
+ LIBBPF_OPTS(bpf_prog_load_opts, opts,
+ .expected_attach_type = BPF_TRACE_FENTRY,
+ );
+ s32 btf_id;
+ int prog_fd, link_fd;
+
+ if (!btf)
+ return false;
+
+ btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC);
+ if (btf_id <= 0)
+ return false;
+
+ opts.attach_btf_id = btf_id;
+ prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL",
+ insns, ARRAY_SIZE(insns), &opts);
+ if (prog_fd < 0)
+ return false;
+
+ link_fd = bpf_raw_tracepoint_open(NULL, prog_fd);
+ if (link_fd >= 0)
+ close(link_fd);
+ close(prog_fd);
+ return link_fd >= 0;
+}
+
+static void configure_rcu_contention_progs(struct lock_contention *con,
+ bool has_rcu_lock)
+{
+ static const struct {
+ size_t prog_offset;
+ const char *func;
+ bool is_srcu;
+ } rcu_progs[] = {
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu),
+ "synchronize_rcu", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu),
+ "synchronize_rcu", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited),
+ "synchronize_rcu_expedited", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited),
+ "synchronize_rcu_expedited", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks),
+ "synchronize_rcu_tasks", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks),
+ "synchronize_rcu_tasks", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier),
+ "rcu_barrier", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier),
+ "rcu_barrier", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu),
+ "synchronize_srcu", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu),
+ "synchronize_srcu", true },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited),
+ "synchronize_srcu_expedited", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited),
+ "synchronize_srcu_expedited", true },
+ { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier),
+ "srcu_barrier", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier),
+ "srcu_barrier", true },
+ };
+ struct symbol *sym;
+ struct map *kmap;
+ bool need_rcu = true, need_srcu = true;
+ bool fentry_ok;
+
+ sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap);
+ if (sym) {
+ /*
+ * Offset by 1 byte so rcu_state_addr is distinct from
+ * &rcu_state.node[0].lock at offset 0 of rcu_state.
+ */
+ skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start) + 1;
+ map__put(kmap);
+ } else {
+ skel->rodata->rcu_state_addr = 1ULL;
+ }
+
+ if (con->filters->nr_types) {
+ need_rcu = false;
+ need_srcu = false;
+ for (int i = 0; i < con->filters->nr_types; i++) {
+ if (con->filters->types[i] & LCB_F_RCU)
+ need_rcu = true;
+ if (con->filters->types[i] & LCB_F_SRCU)
+ need_srcu = true;
+ }
+ }
+
+ if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) {
+ if (!has_rcu_lock && !con->filters->nr_addrs)
+ need_rcu = false;
+ if (!con->filters->nr_addrs && !con->filters->nr_slabs)
+ need_srcu = false;
+ }
+
+ fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf);
+
+ for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) {
+ struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset;
+ bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu;
+ bool enable = fentry_ok && want &&
+ btf__find_by_name_kind(con->btf, rcu_progs[i].func,
+ BTF_KIND_FUNC) > 0;
+
+ bpf_program__set_autoload(*progp, enable);
+ }
+}
+
int lock_contention_prepare(struct lock_contention *con)
{
int i, fd;
@@ -187,6 +315,7 @@ int lock_contention_prepare(struct lock_contention *con)
struct evlist *evlist = con->evlist;
struct target *target = con->target;
bool has_mmap_lock = false;
+ bool has_rcu_lock = false;
/* make sure it loads the kernel map before lookup */
map__load(machine__kernel_map(con->machine));
@@ -200,6 +329,7 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64));
bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries);
bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries);
+ bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries);
if (con->aggr_mode == LOCK_AGGR_TASK)
bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries);
@@ -249,6 +379,10 @@ int lock_contention_prepare(struct lock_contention *con)
has_mmap_lock = true;
continue;
}
+ if (!strcmp(con->filters->syms[i], "rcu")) {
+ has_rcu_lock = true;
+ continue;
+ }
sym = machine__find_kernel_symbol_by_name(con->machine,
con->filters->syms[i],
@@ -269,7 +403,7 @@ int lock_contention_prepare(struct lock_contention *con)
addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start);
con->filters->addrs = addrs;
}
- naddrs = con->filters->nr_addrs ?: has_mmap_lock;
+ naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock);
skel->rodata->has_addr = 1;
}
@@ -305,6 +439,7 @@ int lock_contention_prepare(struct lock_contention *con)
skel->rodata->needs_callstack = con->save_callstack;
skel->rodata->lock_owner = con->owner;
skel->rodata->has_mmap_lock = has_mmap_lock;
+ skel->rodata->has_rcu_lock = has_rcu_lock;
if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) {
if (cgroup_is_v2("perf_event"))
@@ -321,6 +456,7 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map__set_max_entries(skel->maps.slab_filter, nslabs);
init_numa_data(con);
+ configure_rcu_contention_progs(con, has_rcu_lock);
if (lock_contention_bpf__load(skel) < 0) {
pr_err("Failed to load lock-contention BPF skeleton\n");
@@ -503,23 +639,30 @@ static void account_end_timestamp(struct lock_contention *con)
enum lock_aggr_mode aggr_mode = con->aggr_mode;
struct tstamp_data ts_data, *cpu_data;
- /* Iterate per-task tstamp map (key = TID) */
- ts_fd = bpf_map__fd(skel->maps.tstamp);
+ /* Iterate per-task tstamp maps (key = TID) */
+ int ts_fds[] = {
+ bpf_map__fd(skel->maps.tstamp),
+ bpf_map__fd(skel->maps.tstamp_rcu),
+ };
+
stat_fd = bpf_map__fd(skel->maps.lock_stat);
- prev_key = NULL;
- while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) {
- if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) {
- int pid = key;
+ for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) {
+ ts_fd = ts_fds[i];
+ prev_key = NULL;
+ while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) {
+ if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) {
+ int pid = key;
- if (aggr_mode == LOCK_AGGR_TASK && con->owner)
- pid = ts_data.flags;
+ if (aggr_mode == LOCK_AGGR_TASK && con->owner)
+ pid = ts_data.flags;
- update_lock_stat(stat_fd, pid, end_ts, aggr_mode,
- &ts_data);
- }
+ update_lock_stat(stat_fd, pid, end_ts, aggr_mode,
+ &ts_data);
+ }
- prev_key = &key;
+ prev_key = &key;
+ }
}
/* Now it'll check per-cpu tstamp map which doesn't have TID. */
@@ -604,6 +747,8 @@ static const char *lock_contention_get_name(struct lock_contention *con,
return "mmap_lock";
if (flags & LCD_F_SIGHAND_LOCK)
return "siglock";
+ if (flags & LCD_F_RCU_LOCK)
+ return "rcu";
/* global locks with symbols */
sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap);
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index 0f2118006005..26936f329dba 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -59,6 +59,18 @@ struct {
__uint(max_entries, MAX_ENTRIES);
} tstamp SEC(".maps");
+/*
+ * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits
+ * so they do not collide with inner sleeping lock contentions (such as
+ * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, int);
+ __type(value, struct tstamp_data);
+ __uint(max_entries, MAX_ENTRIES);
+} tstamp_rcu SEC(".maps");
+
/* maintain per-CPU timestamp at the beginning of contention */
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
@@ -160,6 +172,7 @@ const volatile int has_addr;
const volatile int has_cgroup;
const volatile int has_slab;
const volatile int has_mmap_lock;
+const volatile int has_rcu_lock;
const volatile int needs_callstack;
const volatile int stack_skip;
const volatile int lock_owner;
@@ -172,6 +185,7 @@ const volatile int aggr_mode;
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
+const volatile __u64 rcu_state_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
@@ -218,7 +232,7 @@ static inline __u64 get_current_cgroup_id(void)
return BPF_CORE_READ(cgrp, kn, id);
}
-static inline int can_record(u64 *ctx)
+static __always_inline int can_record(__u64 lock, __u32 flags)
{
bool is_addr_ok = false;
@@ -242,7 +256,6 @@ static inline int can_record(u64 *ctx)
if (has_type) {
__u8 *ok;
- __u32 flags = (__u32)ctx[1];
ok = bpf_map_lookup_elem(&type_filter, &flags);
if (!ok)
@@ -251,10 +264,9 @@ static inline int can_record(u64 *ctx)
if (has_addr) {
__u8 *ok;
- __u64 addr = ctx[0];
- ok = bpf_map_lookup_elem(&addr_filter, &addr);
- if (!ok && !has_slab && !has_mmap_lock)
+ ok = bpf_map_lookup_elem(&addr_filter, &lock);
+ if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock)
return 0;
is_addr_ok = !!ok;
@@ -272,26 +284,27 @@ static inline int can_record(u64 *ctx)
if (is_addr_ok)
return 1;
- /* slab and mmap_lock are part of the addr_filter */
+ /* slab, mmap_lock, and rcu_lock are part of the addr_filter */
if (has_slab && bpf_get_kmem_cache) {
__u8 *ok;
- __u64 addr = ctx[0];
long kmem_cache_addr;
- kmem_cache_addr = (long)bpf_get_kmem_cache(addr);
+ kmem_cache_addr = (long)bpf_get_kmem_cache(lock);
ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr);
if (ok)
return 1;
- else if (!has_mmap_lock)
+ else if (!has_mmap_lock && !has_rcu_lock)
return 0;
}
- if (has_mmap_lock) {
- __u64 lock = ctx[0];
- __u32 flag = ctx[1];
+ if (has_mmap_lock || has_rcu_lock) {
+ __u32 ltype = check_lock_type(lock, flags);
- if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK)
- return 0;
+ if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK)
+ return 1;
+ if (has_rcu_lock && ltype == LCD_F_RCU_LOCK)
+ return 1;
+ return 0;
}
return 1;
@@ -351,6 +364,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
{
__u32 pid;
struct tstamp_data *pelem;
+ void *map;
/* Use per-cpu array map for spinlock and rwlock */
if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) {
@@ -363,8 +377,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
return pelem;
}
+ map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp;
pid = bpf_get_current_pid_tgid();
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ pelem = bpf_map_lookup_elem(map, &pid);
/* Do not update the element for nested locks */
if (pelem && pelem->lock)
return NULL;
@@ -372,12 +387,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
if (pelem == NULL) {
struct tstamp_data zero = {};
- if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) {
+ if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) {
__sync_fetch_and_add(&task_fail, 1);
return NULL;
}
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ pelem = bpf_map_lookup_elem(map, &pid);
if (pelem == NULL) {
__sync_fetch_and_add(&task_fail, 1);
return NULL;
@@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags)
}
}
-SEC("tp_btf/contention_begin")
-int contention_begin(u64 *ctx)
+static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags)
{
struct tstamp_data *pelem;
- if (!enabled || !can_record(ctx))
+ if (!enabled || !can_record(lock, flags))
return 0;
- pelem = get_tstamp_elem(ctx[1]);
+ pelem = get_tstamp_elem(flags);
if (pelem == NULL)
return 0;
pelem->timestamp = bpf_ktime_get_ns();
- pelem->lock = (__u64)ctx[0];
- pelem->flags = (__u32)ctx[1];
+ pelem->lock = lock;
+ pelem->flags = flags;
if (aggr_mode == LOCK_AGGR_CGROUP)
pelem->cgroup_id = get_current_cgroup_id();
@@ -559,8 +573,21 @@ int contention_begin(u64 *ctx)
}
}
skip_owner:
- pelem->stack_id = bpf_get_stackid(ctx, &stacks,
- BPF_F_FAST_STACK_CMP | stack_skip);
+ {
+ int skip = stack_skip;
+
+ /*
+ * Unlike tp_btf/contention_begin (which fires inside a
+ * lock slowpath via trace_contention_begin), fentry
+ * RCU/SRCU programs attach directly to the top-level
+ * wait API, which is 2 frames shallower.
+ */
+ if (flags & (LCB_F_RCU | LCB_F_SRCU))
+ skip = stack_skip > 2 ? stack_skip - 2 : 0;
+
+ pelem->stack_id = bpf_get_stackid(ctx, &stacks,
+ BPF_F_FAST_STACK_CMP | skip);
+ }
if (pelem->stack_id < 0)
__sync_fetch_and_add(&stack_fail, 1);
} else if (aggr_mode == LOCK_AGGR_TASK) {
@@ -588,8 +615,7 @@ int contention_begin(u64 *ctx)
return 0;
}
-SEC("tp_btf/contention_end")
-int contention_end(u64 *ctx)
+static __always_inline int do_contention_end(__u64 lock, __s32 ret, bool is_rcu)
{
__u32 pid = 0, idx = 0;
struct tstamp_data *pelem;
@@ -597,30 +623,34 @@ int contention_end(u64 *ctx)
struct contention_data *data;
__u64 timestamp;
__u64 duration;
- bool need_delete = false;
+ void *ts_map = NULL;
if (!enabled)
return 0;
/*
- * For spinlock and rwlock, it needs to get the timestamp for the
- * per-cpu map. However, contention_end does not have the flags
- * so it cannot know whether it reads percpu or hash map.
- *
- * Try per-cpu map first and check if there's active contention.
- * If it is, do not read hash map because it cannot go to sleeping
- * locks before releasing the spinning locks.
+ * RCU/SRCU fexit hooks always use the dedicated per-task tstamp_rcu
+ * map. For tp_btf/contention_end, try the per-cpu map first (used by
+ * spinlocks and rwlocks) and fall back to the per-task tstamp map.
*/
- pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx);
- if (pelem && pelem->lock) {
- if (pelem->lock != ctx[0])
- return 0;
- } else {
+ if (is_rcu) {
pid = bpf_get_current_pid_tgid();
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
- if (!pelem || pelem->lock != ctx[0])
+ pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid);
+ if (!pelem || pelem->lock != lock)
return 0;
- need_delete = true;
+ ts_map = &tstamp_rcu;
+ } else {
+ pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx);
+ if (pelem && pelem->lock) {
+ if (pelem->lock != lock)
+ return 0;
+ } else {
+ pid = bpf_get_current_pid_tgid();
+ pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ if (!pelem || pelem->lock != lock)
+ return 0;
+ ts_map = &tstamp;
+ }
}
timestamp = bpf_ktime_get_ns();
@@ -648,7 +678,6 @@ int contention_end(u64 *ctx)
*/
else {
u32 i = 0;
- s32 ret = (s32)ctx[1];
u64 *buf;
otdata->timestamp = timestamp;
@@ -713,7 +742,7 @@ int contention_end(u64 *ctx)
if (lock_owner)
key.pid = pelem->flags;
else {
- if (!need_delete)
+ if (!ts_map)
pid = bpf_get_current_pid_tgid();
key.pid = pid;
}
@@ -795,11 +824,109 @@ int contention_end(u64 *ctx)
check_lock_delay(pelem->lock);
pelem->lock = 0;
- if (need_delete)
- bpf_map_delete_elem(&tstamp, &pid);
+ if (ts_map)
+ bpf_map_delete_elem(ts_map, &pid);
return 0;
}
+SEC("tp_btf/contention_begin")
+int contention_begin(u64 *ctx)
+{
+ return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]);
+}
+
+SEC("tp_btf/contention_end")
+int contention_end(u64 *ctx)
+{
+ return do_contention_end((__u64)ctx[0], (__s32)ctx[1], false);
+}
+
+/* RCU grace-period synchronize / barrier contention hooks */
+SEC("fentry/synchronize_rcu")
+int BPF_PROG(fentry_synchronize_rcu)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu")
+int BPF_PROG(fexit_synchronize_rcu)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+SEC("fentry/synchronize_rcu_expedited")
+int BPF_PROG(fentry_synchronize_rcu_expedited)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu_expedited")
+int BPF_PROG(fexit_synchronize_rcu_expedited)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+SEC("fentry/synchronize_rcu_tasks")
+int BPF_PROG(fentry_synchronize_rcu_tasks)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu_tasks")
+int BPF_PROG(fexit_synchronize_rcu_tasks)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+SEC("fentry/rcu_barrier")
+int BPF_PROG(fentry_rcu_barrier)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/rcu_barrier")
+int BPF_PROG(fexit_rcu_barrier)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+/* SRCU grace-period synchronize / barrier contention hooks */
+SEC("fentry/synchronize_srcu")
+int BPF_PROG(fentry_synchronize_srcu, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu")
+int BPF_PROG(fexit_synchronize_srcu, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0, true);
+}
+
+SEC("fentry/synchronize_srcu_expedited")
+int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu_expedited")
+int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0, true);
+}
+
+SEC("fentry/srcu_barrier")
+int BPF_PROG(fentry_srcu_barrier, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/srcu_barrier")
+int BPF_PROG(fexit_srcu_barrier, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0, true);
+}
+
SEC("raw_tp/bpf_test_finish")
int BPF_PROG(end_timestamp)
{
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 1d9d3bf57720..0a0f223b3335 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -3379,6 +3379,17 @@ bool machine__is_lock_function(struct machine *machine, u64 addr)
return true;
}
+ {
+ struct symbol *sym = machine__find_kernel_symbol(machine, addr, NULL);
+
+ if (sym && (!strncmp(sym->name, "synchronize_rcu", 15) ||
+ !strncmp(sym->name, "synchronize_srcu", 16) ||
+ !strcmp(sym->name, "__synchronize_srcu") ||
+ !strcmp(sym->name, "rcu_barrier") ||
+ !strcmp(sym->name, "srcu_barrier")))
+ return true;
+ }
+
return false;
}
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v3 6/6] perf lock contention: Filter events before creating lock_stat
2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (4 preceding siblings ...)
2026-10-18 5:37 ` [PATCH v3 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
@ 2026-10-18 5:37 ` Ian Rogers
2026-10-18 5:40 ` sashiko-bot
2026-10-19 7:00 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Namhyung Kim
2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers
7 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-18 5:37 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, rcu
In report_lock_contention_begin_event(), lock_stat_findnew() was
called before checking filters.nr_types (-Y), filters.nr_addrs (-L),
and match_callstack_filter() (-S).
When a filtered-out lock event (such as a mutex contention when
running with -Y spinlock) shared the same aggregation key as a
subsequent matching event (for example, contention on the mutex's
internal wait_lock spinlock from the same caller), the first event
created the lock_stat entry with the filtered-out lock's flags and the
second event reused that entry, causing non-spinlock entries to appear
in -Y spinlock output.
Check the type, address, and callstack filters before looking up or
creating the lock_stat entry.
Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option")
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/builtin-lock.c | 61 +++++++++++++++++++++------------------
1 file changed, 33 insertions(+), 28 deletions(-)
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index feaf1cfd08cf..a031709a2cc2 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -975,6 +975,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
struct machine *machine = &session->machines.host;
struct map *kmap;
struct symbol *sym;
+ u64 *callstack = NULL;
ret = get_key_by_aggr_mode(&key, addr, sample);
if (ret < 0)
@@ -1010,33 +1011,6 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
}
}
- ls = lock_stat_find(key);
- if (!ls) {
- char buf[128];
- const char *name = "";
-
- switch (aggr_mode) {
- case LOCK_AGGR_ADDR:
- sym = machine__find_kernel_symbol(machine, key, &kmap);
- if (sym)
- name = sym->name;
- break;
- case LOCK_AGGR_CALLER:
- name = buf;
- if (lock_contention_caller(sample, buf, sizeof(buf)) < 0)
- name = "Unknown";
- break;
- case LOCK_AGGR_CGROUP:
- case LOCK_AGGR_TASK:
- default:
- break;
- }
-
- ls = lock_stat_findnew(key, name, flags);
- if (!ls)
- return -ENOMEM;
- }
-
if (filters.nr_types) {
bool found = false;
@@ -1066,7 +1040,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
}
if (needs_callstack()) {
- u64 *callstack = get_callstack(sample, max_stack_depth);
+ callstack = get_callstack(sample, max_stack_depth);
if (callstack == NULL)
return 0;
@@ -1074,7 +1048,38 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
free(callstack);
return 0;
}
+ }
+
+ ls = lock_stat_find(key);
+ if (!ls) {
+ char buf[128];
+ const char *name = "";
+
+ switch (aggr_mode) {
+ case LOCK_AGGR_ADDR:
+ sym = machine__find_kernel_symbol(machine, key, &kmap);
+ if (sym)
+ name = sym->name;
+ break;
+ case LOCK_AGGR_CALLER:
+ name = buf;
+ if (lock_contention_caller(sample, buf, sizeof(buf)) < 0)
+ name = "Unknown";
+ break;
+ case LOCK_AGGR_CGROUP:
+ case LOCK_AGGR_TASK:
+ default:
+ break;
+ }
+
+ ls = lock_stat_findnew(key, name, flags);
+ if (!ls) {
+ free(callstack);
+ return -ENOMEM;
+ }
+ }
+ if (callstack) {
if (ls->callstack == NULL)
ls->callstack = callstack;
else
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* Re: [PATCH v3 6/6] perf lock contention: Filter events before creating lock_stat
2026-10-18 5:37 ` [PATCH v3 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers
@ 2026-10-18 5:40 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-18 5:40 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> In report_lock_contention_begin_event(), lock_stat_findnew() was
> called before checking filters.nr_types (-Y), filters.nr_addrs (-L),
> and match_callstack_filter() (-S).
>
> When a filtered-out lock event (such as a mutex contention when
> running with -Y spinlock) shared the same aggregation key as a
> subsequent matching event (for example, contention on the mutex's
> internal wait_lock spinlock from the same caller), the first event
> created the lock_stat entry with the filtered-out lock's flags and the
> second event reused that entry, causing non-spinlock entries to appear
> in -Y spinlock output.
>
> Check the type, address, and callstack filters before looking up or
> creating the lock_stat entry.
>
> Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option")
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261018053710.2572636-1-irogers@google.com?part=6
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v3 3/6] perf test: Add tests for lock_held() BPF sample filter
2026-10-18 5:37 ` [PATCH v3 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers
@ 2026-10-18 5:41 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-18 5:41 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Add unit tests in parse-events.c for parsing lock_held(),
> lock_held_read(), and lock_held_exclusive() BPF filter expressions
> (standalone predicates, negated predicates, and compound group
> expressions) and extend record_bpf_filter.sh with lock_held(),
> lock_held_read()/lock_held_write(), and ':R'/':W' recording tests.
>
> Example output:
>
> # perf test -v 'Parsing of BPF sample filters'
> 5: Parse event definition strings :
> 5.7: Parsing of BPF sample filters : Ok
>
> # perf test -v 'perf record sample filtering (by BPF) tests'
> 112: perf record sample filtering (by BPF) tests :
> Checking BPF-filter privilege
> Basic bpf-filter test
> Basic bpf-filter test [Success]
> Failing bpf-filter test
> Failing bpf-filter test [Success]
> Group bpf-filter test
> Group bpf-filter test [Success]
> Multiple bpf-filter test
> Multiple bpf-filter test [Success]
> Cgroup bpf-filter test
> Cgroup bpf-filter test [Success]
> Lock-held bpf-filter test
> Lock-held bpf-filter test [Success]
> 112: perf record sample filtering (by BPF) tests : Ok
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261018053710.2572636-1-irogers@google.com?part=3
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v3 5/6] perf lock contention: Extend BPF lock contention to RCU and SRCU
2026-10-18 5:37 ` [PATCH v3 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
@ 2026-10-18 5:48 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-18 5:48 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Extend 'perf lock contention --use-bpf' to track RCU and SRCU
> grace-period and barrier waits:
>
> - Attach fentry/fexit programs to synchronize_rcu(),
> synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(),
> synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier()
> when available in BTF.
> - Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu',
> '-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct
> symbol names).
>
> Example usage:
>
> # perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1
> total wait avg wait type caller
>
> 35.12 ms 8.78 ms rcu:W cleanup_net+0x284
> 7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114
>
> # perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1
> total wait avg wait address symbol
>
> 35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu)
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261018053710.2572636-1-irogers@google.com?part=5
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v3 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-18 5:37 ` [PATCH v3 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
@ 2026-10-18 5:50 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-18 5:50 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] perf lock: missing CO-RE flavor struct for mutex in get_lock_owner()
--
commit 93be88d7858bcce932211b385210b6872f154e7a
Author: Ian Rogers <irogers@google.com>
perf lock contention: Share common BPF and lock type definitions in lock_data.h
This commit moves lock_type_table, LCB_F_* flags, common BPF lock helpers,
and shared CO-RE structs into tools/perf/util/bpf_skel/lock_data.h so they
can be reused by BPF sample filtering. It also extends get_lock_owner() to
support rt_mutex and check the mutex handoff flag.
> diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
> index 652e114e6b877..8cf24f07204c5 100644
> --- a/tools/perf/util/bpf_skel/lock_data.h
> +++ b/tools/perf/util/bpf_skel/lock_data.h
[ ... ]
> +static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
> +{
> + struct task_struct *task;
> + __u64 owner = 0;
> +
> + if (flags & LCB_F_MUTEX) {
> + struct mutex *mutex = (void *)lock;
> +
> + if (bpf_core_field_exists(mutex->owner.counter)) {
> + owner = BPF_CORE_READ(mutex, owner.counter);
> + if (owner & 0x4UL)
> + return NULL;
> + } else if (bpf_core_type_exists(struct rt_mutex_base)) {
> + struct rt_mutex_base *rtm = (void *)lock;
> +
> + owner = (__u64)BPF_CORE_READ(rtm, owner);
> + return (void *)(owner & ~1UL);
> + }
[Severity: Low]
When building perf with GEN_VMLINUX_H=1 against a CONFIG_PREEMPT_RT=y
kernel, vmlinux.h is generated from the kernel BTF where struct mutex is
defined as:
include/linux/mutex_types.h:
context_lock_struct(mutex) {
struct rt_mutex_base rtmutex;
#ifdef CONFIG_DEBUG_LOCK_ALLOC
struct lockdep_map dep_map;
#endif
};
Because struct mutex has no owner member in that configuration, and
bpf_core_field_exists(mutex->owner.counter) and BPF_CORE_READ(mutex,
owner.counter) expand to C member accesses on struct mutex, will this fail
to compile?
Should this use a CO-RE flavor struct (similar to struct rw_semaphore___new
and struct mm_struct___new above) to access owner.counter?
[ ... ]
>
>
>
>
[]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261018053710.2572636-1-irogers@google.com?part=1
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v3 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
2026-10-18 5:37 ` [PATCH v3 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
@ 2026-10-18 5:51 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-18 5:51 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf bpf-filter: btf_has_task_field() misses anon struct under RANDSTRUCT
--
commit 5ff446ff868169d426bb172f1b13dc459b646352
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
This commit extends the lock_held(<lock>) BPF sample filter to support RCU,
Tasks Trace RCU, and SRCU locks. For RCU and Tasks Trace RCU read-side
critical sections, it inspects task_struct nesting fields via CO-RE, while
attaching fentry/fexit hooks for SRCU and RCU synchronize/barrier waits.
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 0c133b31b5d54..4a5ef78e26900 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
[ ... ]
> @@ -847,16 +913,54 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
> return false;
> }
>
> +static u64 rcu_min_depth(struct evsel *evsel)
> +{
> + if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE &&
> + evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK &&
> + evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK)
> + return 2;
> + return 1;
> +}
> +
> +static bool btf_has_task_field(struct btf *btf, const char *field_name)
> +{
> + const struct btf_type *t;
> + const struct btf_member *m;
> + s32 id;
> + u16 i, vlen;
> +
> + if (!btf)
> + return false;
> +
> + id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT);
> + if (id < 0)
> + return false;
> +
> + t = btf__type_by_id(btf, id);
> + if (!t)
> + return false;
> +
> + m = btf_members(t);
> + vlen = btf_vlen(t);
> + for (i = 0; i < vlen; i++, m++) {
> + const char *name = btf__name_by_offset(btf, m->name_off);
> +
> + if (name && !strcmp(name, field_name))
> + return true;
> + }
> + return false;
> +}
[Severity: Medium]
In include/linux/sched.h, both rcu_read_lock_nesting and trc_reader_nesting
are declared inside the randomized_struct_fields_start /
randomized_struct_fields_end block of struct task_struct:
include/linux/sched.h:
struct task_struct {
...
randomized_struct_fields_start
...
#ifdef CONFIG_PREEMPT_RCU
int rcu_read_lock_nesting;
...
#endif
#ifdef CONFIG_TASKS_TRACE_RCU
int trc_reader_nesting;
...
#endif
...
randomized_struct_fields_end
};
When CONFIG_RANDSTRUCT is enabled, randomized_struct_fields_start wraps those
fields inside an anonymous struct member of task_struct, which appears in
vmlinux BTF as a member with m->name_off == 0 pointing to an inner
BTF_KIND_STRUCT.
While libbpf CO-RE in sample_filter.bpf.c recursively descends into anonymous
structs and unions, btf_has_task_field() only iterates over the direct
top-level members of task_struct.
Will btf_has_task_field() return false on CONFIG_RANDSTRUCT kernels, causing
resolve_lock_expr() to reject lock_held(rcu) and lock_held(rcu-trace) with
-EOPNOTSUPP?
[ ... ]
> @@ -929,6 +1033,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel,
> return 0;
> }
>
> + if (!strcmp(base, "rcu_read_lock")) {
> + if (mode == PBF_LOCK_MODE_WRITE) {
> + pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
> + return -EINVAL;
> + }
> + if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
> + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
> + return -EOPNOTSUPP;
> + }
> + entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ;
> + entry->target = rcu_min_depth(evsel);
> + return 0;
> + }
[ ... ]
—
🤖 Generated by Linux Kernel Patch Reviewer (Model: gemini-3.1-pro-preview)
Prompts used:
- bpf.md (https://raw.githubusercontent.com/masoncl/review-prompts/main/bpf.md)
- btf.md (https://raw.githubusercontent.com/masoncl/review-prompts/main/btf.md)
- inline-template.md (https://raw.githubusercontent.com/masoncl/review-prompts/main/inline-template.md)
- libbpf.md (https://raw.githubusercontent.com/masoncl/review-prompts/main/libbpf.md)
- perf.md (https://raw.githubusercontent.com/masoncl/review-prompts/main/perf.md)
- rcu.md (https://raw.githubusercontent.com/masoncl/review-prompts/main/rcu.md)
- tracing.md (https://raw.githubusercontent.com/masoncl/review-prompts/main/tracing.md)
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261018053710.2572636-1-irogers@google.com?part=4
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v3 2/6] perf bpf-filter: Add lock_held() sample filter
2026-10-18 5:37 ` [PATCH v3 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers
@ 2026-10-18 6:12 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-18 6:12 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Medium] perf bpf-filter: unreachable BTF check in configure_lock_progs()
- [Medium] perf bpf-filter: read mode accepted for exclusive locks in resolve_lock_expr()
- [Medium] perf bpf-filter: unhandled mutex:spin entry in parse_lock_type_name()
- [Medium] perf bpf-filter: leak of bpf_link in perf_bpf_filter__prepare()
--
commit 62241acfb0cf66852754002cba52ca231185bd12
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Add lock_held() sample filter
This commit adds BPF sample filter predicates lock_held(), lock_held_read(),
lock_held_shared(), lock_held_write(), and lock_held_exclusive() to record
perf event samples only when a specified kernel lock is held. It evaluates
lock ownership by inspecting kernel lock owner fields directly where possible
and falling back to a held_locks BPF hash map populated by fentry/fexit hooks
on kernel lock and unlock functions.
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 72e83ac1d6a4b..0c133b31b5d54 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
[ ... ]
> +static bool parse_lock_type_name(const char *name, unsigned int mode,
> + struct sample_filter_bpf *skel,
> + u64 *encoded_pair, unsigned int *prog_groups)
> +{
> + bool found = false;
> + u32 flag1 = 0xff, flag2 = 0xff;
> + unsigned int groups = 0;
> + const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name;
> +
> + for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) {
> + u32 f = lock_type_table[i].flags;
> + bool is_read = (f & LCB_F_READ) != 0;
> +
> + if (strcmp(lock_type_table[i].lock_name, lookup) &&
> + strcmp(lock_type_table[i].flags_name, name))
> + continue;
> +
> + /*
> + * percpu_down_read() / percpu_up_read() fast paths are inlined
> + * per-CPU counter increments without function entry points, so
> + * only writer holding (percpu_down_write / percpu_up_write) can
> + * be tracked for percpu-rwsem. rwlock-rt is also not hooked.
> + */
> + if ((f & LCB_F_PERCPU) && is_read)
> + continue;
> + if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE)))
> + continue;
> +
> + if (is_read && !(mode & PBF_LOCK_MODE_READ))
> + continue;
> + if (!is_read && !(mode & PBF_LOCK_MODE_WRITE))
> + continue;
> +
> + if (!found)
> + flag1 = f;
> + else if (f != flag1)
> + flag2 = f;
> + if (skel && f < LCB_F_TYPE_MAX) {
> + skel->bss->has_lock_type = 1;
> + skel->bss->lock_type_filter[f] = 1;
> + }
> + groups |= lock_flags_to_prog_group(f);
> + found = true;
> + }
[Severity: Medium]
Should parse_lock_type_name() also skip the LCB_F_MUTEX | LCB_F_SPIN entries
in lock_type_table?
lock_type_table contains both { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }
and { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, whereas all
mutex hooks in sample_filter.bpf.c record acquisitions with LCB_F_MUTEX:
tools/perf/util/bpf_skel/sample_filter.bpf.c:
SEC("fexit/mutex_lock")
int BPF_PROG(fexit_mutex_lock, void *lock)
{
return record_lock_acquired(ctx, LCB_F_MUTEX);
}
Because LCB_F_MUTEX | LCB_F_SPIN is never recorded in held_locks, filters
like lock_held(mutex:spin) or lock_held(mutex-spin) are accepted and attach
all mutex fentry/fexit hooks, but only set
lock_type_filter[LCB_F_MUTEX | LCB_F_SPIN] = 1 and always evaluate to 0.
Furthermore, lock_held(mutex) encodes flag2 = LCB_F_MUTEX | LCB_F_SPIN,
causing is_lock_type_in_map() to perform two extra BPF map lookups on every
sample whenever no mutex is held.
[ ... ]
> +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
> + unsigned int needed_groups,
> + unsigned int required_groups,
> + bool needs_lock_syms, bool needs_slab_iter,
> + bool is_tracepoint)
> +{
> + unsigned int supported_groups = supported_lock_prog_groups(btf);
> + unsigned int enabled_progs = 0;
> +
> + bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
> + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
> +
> + if ((required_groups & supported_groups) != required_groups) {
> + pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n");
> + return -ENOTSUP;
> + }
> +
> + needed_groups &= supported_groups;
> + if (skel)
> + skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ);
> +
> + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) {
> + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset;
> + struct bpf_program *prog = *progp;
> + bool enable = false;
> +
> + if ((needed_groups & lock_progs[i].group) && btf &&
> + lock_progs[i].btf_func) {
> + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func,
> + BTF_KIND_FUNC) > 0)
> + enable = true;
> + }
> +
> + if (enable)
> + enabled_progs++;
> + bpf_program__set_autoload(prog, enable);
> + }
> +
> + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms);
> + if (needs_slab_iter && btf &&
> + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) {
> + bpf_program__set_autoload(skel->progs.slab_cache_iter, true);
> + } else {
> + bpf_program__set_autoload(skel->progs.slab_cache_iter, false);
> + }
> +
> + if (needed_groups && !enabled_progs) {
> + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
> + return -ENOTSUP;
> + }
> + return 0;
> +}
[Severity: Medium]
Can this final check in configure_lock_progs() ever trigger?
Earlier in configure_lock_progs(), needed_groups is masked in-place via
needed_groups &= supported_groups. Because every group bit set by
supported_lock_prog_groups(btf) guarantees that its corresponding BTF
functions exist in lock_progs[], enabled_progs will only be 0 when
(needed_groups & supported_groups) is 0. However, in that case
needed_groups &= supported_groups has already cleared needed_groups to 0, so
if (needed_groups && !enabled_progs) always evaluates to false.
When a filter specifies a target with required_groups == 0 and
needed_groups == LOCK_PROG_ALL (such as a hex lock address, a slab cache like
&dentry, or a lock symbol without a recognized type suffix like
&tasklist_lock) on a kernel without BTF or lacking the lock entry points,
configure_lock_progs() will silently succeed with zero lock-tracking programs
enabled instead of returning -ENOTSUP.
[ ... ]
> +static int resolve_lock_expr(struct sample_filter_bpf *skel,
> + struct perf_bpf_filter_expr *expr,
> + struct perf_bpf_filter_entry *entry)
> +{
[ ... ]
> + ret = find_kernel_lock_sym(base, &addr);
> + if (ret == 0) {
> + unsigned int direct_flag = 0;
> +
> + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0)
> + direct_flag = PBF_LOCK_DIRECT_RTMUTEX;
> + else if (strtailcmp(base, "mutex") == 0)
> + direct_flag = PBF_LOCK_DIRECT_MUTEX;
> + else if (strtailcmp(base, "rwsem") == 0 &&
> + strtailcmp(base, "percpu_rwsem") != 0 &&
> + strtailcmp(base, "pcpu_rwsem") != 0)
> + direct_flag = PBF_LOCK_DIRECT_RWSEM;
> +
> + entry->part = PBF_LOCK_ADDR | mode | direct_flag;
> + entry->target = addr;
> + if (skel) {
> + int fd = bpf_map__fd(skel->maps.lock_addr_filter);
> +
> + bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
> + skel->bss->has_lock_addr = 1;
> + }
> + return 0;
> + }
[Severity: Medium]
Should resolve_lock_expr() reject read-only mode (!(mode & PBF_LOCK_MODE_WRITE))
when a global/static kernel lock symbol has an exclusive-only suffix?
In prescan_one_lock_expr(), symbols ending in rt_mutex, rtmutex, mutex,
percpu_rwsem, pcpu_rwsem, or spinlock only enable their respective exclusive
BPF program groups, which only increment write_count.
If a user specifies read/shared mode on such a symbol (for example,
lock_held_read(&cgroup_mutex) or lock_held(&cgroup_mutex:R)),
resolve_lock_expr() accepts it without returning -EINVAL, attaches the
exclusive lock hooks, and sets mode to PBF_LOCK_MODE_READ. At runtime in
check_lock_held(), is_direct_lock_owner() is skipped when
mode == PBF_LOCK_MODE_READ and match_held_val() checks val->read_count > 0,
which can never be non-zero.
[ ... ]
> @@ -501,23 +1434,44 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> goto err;
> }
>
> - prog = skel->progs.perf_sample_filter;
> - for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
> - for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
> - link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y),
> - &pe_opts);
> - if (IS_ERR(link)) {
> - pr_err("Failed to attach perf sample-filter program\n");
> - ret = PTR_ERR(link);
> - goto err;
> + if (is_tracepoint) {
> + /*
> + * For PERF_TYPE_TRACEPOINT, PERF_EVENT_IOC_SET_BPF attaches the
> + * BPF program to event->tp_event->prog_array on the shared
> + * trace_event_call, which perf_trace_run_bpf_submit() executes
> + * for all CPUs and tasks before iterating the per-CPU perf
> + * events. Attach once on the first FD so tp_event->prog_array
> + * does not get duplicate copies for each per-CPU FD.
> + */
> + prog = skel->progs.perf_tp_sample_filter;
> + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, 0, 0),
> + &pe_opts);
> + ret = libbpf_get_error(link);
> + if (ret) {
> + pr_err("Failed to attach perf tracepoint sample-filter program\n");
> + goto err;
> + }
> + } else {
> + prog = skel->progs.perf_sample_filter;
> + for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
> + for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
> + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y),
> + &pe_opts);
> + ret = libbpf_get_error(link);
> + if (ret) {
> + pr_err("Failed to attach perf sample-filter program\n");
> + goto err;
> + }
> }
> }
> }
[Severity: Medium]
Does perf_bpf_filter__prepare() leak the struct bpf_link returned by
bpf_program__attach_perf_event_opts() in the new is_tracepoint branch (as
well as in the existing per-FD loop)?
bpf_program__attach_perf_event_opts() heap-allocates a struct bpf_link_perf,
creates a kernel BPF_PERF_EVENT link fd holding references to the perf_event
file and BPF program, and returns ownership of the struct bpf_link * to the
caller without registering it in skel->links. Storing it in the local link
variable without saving or destroying it in perf_bpf_filter__destroy() (or on
the err path) leaks both the struct bpf_link_perf allocation and the kernel
link fd.
Because the leaked link fd holds a reference to the underlying perf_event
file, closing the evsel fds does not release the perf_event or detach
perf_tp_sample_filter from tp_event->prog_array while the process remains
alive (for example, across repeated runs in perf stat -r N).
[ ... ]
> @@ -532,6 +1486,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> return ret;
> }
>
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261018053710.2572636-1-irogers@google.com?part=2
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support
2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (5 preceding siblings ...)
2026-10-18 5:37 ` [PATCH v3 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers
@ 2026-10-19 7:00 ` Namhyung Kim
2026-10-19 19:08 ` Ian Rogers
2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers
7 siblings, 1 reply; 106+ messages in thread
From: Namhyung Kim @ 2026-10-19 7:00 UTC (permalink / raw)
To: Ian Rogers
Cc: acme, adrian.hunter, james.clark, jolsa, linux-kernel,
linux-perf-users, mingo, peterz, rcu
Hi Ian,
On Sat, Oct 17, 2026 at 10:37:04PM -0700, Ian Rogers wrote:
> This series adds a new BPF sample filter predicate, `lock_held(<lock>)`
> (along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`,
> `lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W`
> mode suffixes), to `perf record --filter`, allowing perf event samples to
> be recorded only when a specified kernel lock is held (or not held with
> `!`) by the running kernel thread or CPU. It also extends both
> `lock_held()` and `perf lock contention --use-bpf` to support RCU and
> SRCU, and fixes a pre-existing `lock_stat` filtering bug in `perf lock
> contention`.
Awesome! Looks really useful!
>
> For example:
>
> # perf record -a -g -e cycles --filter 'lock_held(&cgroup_mutex)' -- ...
> # perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' -- ...
> # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' -- ...
> # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ...
> # perf lock contention -a -b -Y rcu,srcu -- sleep 1
>
> An important use-case is detecting memory allocations or direct memory
> reclaim occurring while kernel locks are held, which can stall lock
> waiters for extended periods under memory pressure:
>
> # Direct memory reclaim triggered while holding any mutex or rwsem:
> # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \
> --filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
>
> # Page allocations permitted to enter direct reclaim (__GFP_DIRECT_RECLAIM)
> # while holding a mutex or rwsem:
> # perf record -a -g -e kmem:mm_page_alloc \
> --filter 'gfp_flags & 0x400' \
> --filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
It seems the first filter is processed by the kernel tracepoint
filtering logic and the second is passed to the BPF filter, right?
Does the ordering matter?
>
> # Major page faults occurring while holding a mutex or rwsem:
> # perf record -a -g -e major-faults \
> --filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
>
> Supported `<lock>` specifications match `perf lock contention`:
> - Global/static kernel lock symbols (with or without leading `&`, e.g.
> `lock_held(&cgroup_mutex)` or `lock_held(cgroup_mutex)`)
It used to have no '&' for global locks. While having it is more
natural, I think it's better to leave it for slab locks so that we can
distinguish them easily.
> - Hexadecimal kernel lock addresses
> - Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`,
> `lock_held_read(&dentry)`)
> - Well-known per-process, semi-global, and RCU locks: `mmap_lock`,
> `siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`
> - Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`,
> `rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`,
> `percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`,
> `rcu:W`, `semaphore`
>
> Limitations and weaknesses in determining whether a lock is held:
> - Inlined lock/unlock operations:
> - When the kernel is configured to inline spinlock or rwlock operations
> (`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`,
> `CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`,
> `CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` /
> `_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are
> inlined into callers and have no BTF function entry points for
> `fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that
> both acquire and release entry points exist in BTF before enabling a
> lock family.
> - Fast-path `percpu_down_read()` / `percpu_up_read()` and
> `srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined
> per-CPU counter increments without function calls, so only
> `percpu_down_write()` / `percpu_up_write()` and out-of-line
> `__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked.
> - On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to
> `preempt_disable()` and does not maintain
> `current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)`
> read-side inspection requires `CONFIG_PREEMPT_RCU`.
Looks ok. What happens if user specified unsupported locks - e.g. rcu:R
on !CONFIG_PREEMPT_RCU.
> - Lack of an inspectable owner field on non-exclusive / spinning locks:
> - While `struct mutex`, `struct rt_mutex`, and write-locked
> `struct rw_semaphore` store the owning `task_struct *` in `lock->owner`
> (allowing `perf_sample_filter` to inspect kernel memory directly on
> sample overflow when filtering by lock address), spinlocks, rwlocks,
> semaphores, SRCU, and reader-locked `rw_semaphore`s (where
> `__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`)
> do not maintain an authoritative per-task owner pointer.
So do you use the owner field for the supported locks? Then maybe it
can skip tracing mutex_lock() and mutex_unlock(). Does it already do?
> - Tracking these locks (as well as slab-cache and lock-type filters)
> relies on `fexit` (acquire) and `fentry` (release) BPF hooks
> maintaining the `held_locks` BPF hash map:
> - Any such lock already held before `perf record` attaches its BPF
> programs will not be present in `held_locks` until it is released
> and re-acquired.
Right, but this can be mitigated when it runs for long.
> - The attached `fentry`/`fexit` hooks execute on every kernel lock
> acquire and release for the hooked lock family across the system,
> adding tracing overhead outside of perf sample handling.
This is a real problem. As it touches the fastpath it can cause a lot
of overhead if locking is frequent. But having this capability would be
very useful anyway!
Thanks,
Namhyung
>
> Patch overview:
> 1. "perf lock contention: Share common BPF and lock type definitions in
> lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and
> `LCB_F_*` flags, CO-RE structs, and common BPF lock helpers
> (`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`,
> `get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`,
> `do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`.
> 2. "perf bpf-filter: Add lock_held() sample filter": Implements
> `lock_held()`, `lock_held_read()`/`lock_held_shared()`,
> `lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode
> suffixes using direct lock owner inspection where possible (`mutex`,
> `rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF map
> updated by selectively attached `fentry`/`fexit` hooks, supporting
> both PMU/software events (`SEC("perf_event")`) and tracepoint events
> (`SEC("tracepoint")`).
> 3. "perf test: Add tests for lock_held() BPF sample filter": Adds unit
> tests in `tests/parse-events.c` and shell tests in
> `tests/shell/record_bpf_filter.sh`.
> 4. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
> and SRCU": Supports `rcu` (`rcu:R` via direct
> `current->rcu_read_lock_nesting` inspection with zero tracing hooks;
> `rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks),
> `rcu-trace` (via `current->trc_reader_nesting`), and `srcu`
> (`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W`
> via `synchronize_srcu*()` / `srcu_barrier()`).
> 5. "perf lock contention: Extend BPF lock contention to RCU and SRCU":
> Extends `perf lock contention --use-bpf` to track RCU and SRCU
> grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`).
> 6. "perf lock contention: Filter events before creating lock_stat":
> Fixes `report_lock_contention_begin_event()` in `builtin-lock.c` to
> apply `-Y`, `-L`, and `-S` filters before calling
> `lock_stat_findnew()`.
>
> Changes in v3:
> - Move the writer-only `rw_semaphore` check out of `get_lock_owner()` in
> `lock_data.h` into `is_rwsem_writer_owner()` in `sample_filter.bpf.c`
> so `perf lock contention -b -o` still reports reader owners when a
> writer contends on a reader-owned `rw_semaphore`, and scope
> `is_direct_lock_owner()` by lock type (`PBF_LOCK_DIRECT_MUTEX`,
> `PBF_LOCK_DIRECT_RWSEM`, `PBF_LOCK_DIRECT_RTMUTEX`).
> - Add `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF
> sample filters (`lock_held()`, `uid`, `gid`, etc.) can attach to
> `PERF_TYPE_TRACEPOINT` events (such as `kmem:mm_page_alloc` and
> `vmscan:mm_vmscan_direct_reclaim_begin`), check `libbpf_get_error()`
> when attaching, and document memory allocation / direct reclaim
> filtering with `lock_held()`.
> - Return early in `init_numa_data()` in `bpf-filter.c` when `node_data`
> cannot be resolved and check `node_data_addr != 0` in
> `do_collect_zone_lock()`.
> - Avoid double-counting `_raw_read_lock_irq` / `_raw_read_lock_bh` in
> `sample_filter.bpf.c` on `CONFIG_GENERIC_LOCKBREAK=y &&
> !CONFIG_DEBUG_LOCK_ALLOC` kernels where they call
> `_raw_read_lock_irqsave()`.
> - Offset `rcu_state_addr` by 1 byte so it is distinct from
> `&rcu_state.node[0].lock` at offset 0 of `rcu_state`, and pass `bool
> is_rcu` to `do_contention_end()` in `lock_contention.bpf.c` so
> `tp_btf/contention_end` and RCU/SRCU `fexit` hooks never cross-match
> timestamp maps.
>
> Changes in v2:
> - Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists`
> and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE
> relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels.
> - Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points
> (`_mutex_lock_nest_lock`, `_mutex_lock_killable`,
> `_mutex_trylock_nest_lock`, `_down_write_nest_lock`,
> `_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to
> `sample_filter.bpf.c` and `lock_progs[]`.
> - Verify in `supported_lock_prog_groups()` that both acquire and all
> unlock BTF entry points exist for a lock family before enabling its
> tracing hooks, preventing acquire-only attachment when spinlock/rwlock
> unlock functions are inlined.
> - Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when
> a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`).
> - Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write`
> on `track_rwsem_read` and verify non-zero read/write counts in
> `record_lock_released_addr()` before decrementing aggregate slab/type
> counts.
> - Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in
> `lock_contention.bpf.c` and recognize `synchronize_rcu*`,
> `synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in
> `machine__is_lock_function()`.
> - Add patch 6 fixing `report_lock_contention_begin_event()` to check
> type, address, and callstack filters before creating `lock_stat`
> entries.
>
> Assisted-by: Antigravity:gemini-3.1-pro
>
> Ian Rogers (6):
> perf lock contention: Share common BPF and lock type definitions in
> lock_data.h
> perf bpf-filter: Add lock_held() sample filter
> perf test: Add tests for lock_held() BPF sample filter
> perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
> and SRCU
> perf lock contention: Extend BPF lock contention to RCU and SRCU
> perf lock contention: Filter events before creating lock_stat
>
> tools/perf/Documentation/perf-lock.txt | 7 +-
> tools/perf/Documentation/perf-record.txt | 64 +-
> tools/perf/builtin-lock.c | 88 +-
> tools/perf/tests/parse-events.c | 121 ++
> tools/perf/tests/shell/lock_contention.sh | 14 +
> tools/perf/tests/shell/record_bpf_filter.sh | 134 ++
> tools/perf/util/bpf-filter.c | 1271 ++++++++++++++++-
> tools/perf/util/bpf-filter.h | 13 +
> tools/perf/util/bpf-filter.l | 31 +-
> tools/perf/util/bpf-filter.y | 46 +-
> tools/perf/util/bpf_lock_contention.c | 171 ++-
> .../perf/util/bpf_skel/lock_contention.bpf.c | 448 +++---
> tools/perf/util/bpf_skel/lock_data.h | 360 ++++-
> tools/perf/util/bpf_skel/sample-filter.h | 77 +
> tools/perf/util/bpf_skel/sample_filter.bpf.c | 1225 ++++++++++++++++
> tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 +
> tools/perf/util/lock-contention.c | 22 +
> tools/perf/util/lock-contention.h | 15 +
> tools/perf/util/machine.c | 11 +
> tools/perf/util/parse-events.c | 3 +-
> 20 files changed, 3741 insertions(+), 387 deletions(-)
>
> --
> 2.56.0.212.g129ede20d9-goog
>
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support
2026-10-19 7:00 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Namhyung Kim
@ 2026-10-19 19:08 ` Ian Rogers
0 siblings, 0 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-19 19:08 UTC (permalink / raw)
To: Namhyung Kim
Cc: acme, adrian.hunter, james.clark, jolsa, linux-kernel,
linux-perf-users, mingo, peterz, rcu
On Mon, Oct 19, 2026 at 12:00 AM Namhyung Kim <namhyung@kernel.org> wrote:
>
> Hi Ian,
>
> On Sat, Oct 17, 2026 at 10:37:04PM -0700, Ian Rogers wrote:
> > This series adds a new BPF sample filter predicate, `lock_held(<lock>)`
> > (along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`,
> > `lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W`
> > mode suffixes), to `perf record --filter`, allowing perf event samples to
> > be recorded only when a specified kernel lock is held (or not held with
> > `!`) by the running kernel thread or CPU. It also extends both
> > `lock_held()` and `perf lock contention --use-bpf` to support RCU and
> > SRCU, and fixes a pre-existing `lock_stat` filtering bug in `perf lock
> > contention`.
>
> Awesome! Looks really useful!
>
> >
> > For example:
> >
> > # perf record -a -g -e cycles --filter 'lock_held(&cgroup_mutex)' -- ...
> > # perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' -- ...
> > # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' -- ...
> > # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ...
> > # perf lock contention -a -b -Y rcu,srcu -- sleep 1
> >
> > An important use-case is detecting memory allocations or direct memory
> > reclaim occurring while kernel locks are held, which can stall lock
> > waiters for extended periods under memory pressure:
> >
> > # Direct memory reclaim triggered while holding any mutex or rwsem:
> > # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \
> > --filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
> >
> > # Page allocations permitted to enter direct reclaim (__GFP_DIRECT_RECLAIM)
> > # while holding a mutex or rwsem:
> > # perf record -a -g -e kmem:mm_page_alloc \
> > --filter 'gfp_flags & 0x400' \
> > --filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
>
> It seems the first filter is processed by the kernel tracepoint
> filtering logic and the second is passed to the BPF filter, right?
> Does the ordering matter?
Right — set_filter() in util/parse-events.c inspects each --filter
string independently with is_possible_tp_filter(): expressions without
BPF-only terms (uid, gid, lock_held) are appended to evsel->filter
(the kernel tracepoint filter), while expressions with lock_held() are
added to evsel->bpf_filters.
The command-line order of the two --filter options does not matter: in
evlist__apply_filters() both evsel->filter (PERF_EVENT_IOC_SET_FILTER)
and evsel->bpf_filters (PERF_EVENT_IOC_SET_BPF) are applied to the
tracepoint event, and in the kernel (perf_trace_run_bpf_submit() ->
perf_tp_event()) both must match (logical AND) for the sample to be
recorded.
> >
> > # Major page faults occurring while holding a mutex or rwsem:
> > # perf record -a -g -e major-faults \
> > --filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
> >
> > Supported `<lock>` specifications match `perf lock contention`:
> > - Global/static kernel lock symbols (with or without leading `&`, e.g.
> > `lock_held(&cgroup_mutex)` or `lock_held(cgroup_mutex)`)
>
> It used to have no '&' for global locks. While having it is more
> natural, I think it's better to leave it for slab locks so that we can
> distinguish them easily.
Good point — in v4 I updated this so '&' is reserved strictly for slab
cache locks (e.g. lock_held(&dentry)), and global/static lock symbols
and well-known locks do not use '&' (e.g. lock_held(cgroup_mutex),
lock_held_read(mmap_lock)). This also avoids running the kmem_cache
BPF iterator unless a '&<slab>' filter is specified.
> > - Hexadecimal kernel lock addresses
> > - Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`,
> > `lock_held_read(&dentry)`)
> > - Well-known per-process, semi-global, and RCU locks: `mmap_lock`,
> > `siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`
> > - Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`,
> > `rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`,
> > `percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`,
> > `rcu:W`, `semaphore`
> >
> > Limitations and weaknesses in determining whether a lock is held:
> > - Inlined lock/unlock operations:
> > - When the kernel is configured to inline spinlock or rwlock operations
> > (`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`,
> > `CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`,
> > `CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` /
> > `_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are
> > inlined into callers and have no BTF function entry points for
> > `fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that
> > both acquire and release entry points exist in BTF before enabling a
> > lock family.
> > - Fast-path `percpu_down_read()` / `percpu_up_read()` and
> > `srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined
> > per-CPU counter increments without function calls, so only
> > `percpu_down_write()` / `percpu_up_write()` and out-of-line
> > `__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked.
> > - On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to
> > `preempt_disable()` and does not maintain
> > `current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)`
> > read-side inspection requires `CONFIG_PREEMPT_RCU`.
>
> Looks ok. What happens if user specified unsupported locks - e.g. rcu:R
> on !CONFIG_PREEMPT_RCU.
resolve_lock_expr() checks vmlinux BTF for task_struct's
rcu_read_lock_nesting field (and trc_reader_nesting for rcu-trace) and
configure_lock_progs() checks that both the acquire and unlock BTF
entry points exist for any required lock family. If the kernel lacks
support, perf record prints an error message (e.g. "Error:
lock_held(rcu) requires CONFIG_PREEMPT_RCU" or "Error: Kernel BTF
lacks required lock/unlock tracing entry points") and fails
immediately with -EOPNOTSUPP / -ENOTSUP.
> > - Lack of an inspectable owner field on non-exclusive / spinning locks:
> > - While `struct mutex`, `struct rt_mutex`, and write-locked
> > `struct rw_semaphore` store the owning `task_struct *` in `lock->owner`
> > (allowing `perf_sample_filter` to inspect kernel memory directly on
> > sample overflow when filtering by lock address), spinlocks, rwlocks,
> > semaphores, SRCU, and reader-locked `rw_semaphore`s (where
> > `__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`)
> > do not maintain an authoritative per-task owner pointer.
>
> So do you use the owner field for the supported locks? Then maybe it
> can skip tracing mutex_lock() and mutex_unlock(). Does it already do?
Great catch — in v3 it was checking lock->owner in perf_sample_filter,
but prescan_one_lock_expr() was still enabling the fentry/fexit hooks
for *_mutex, *_rt_mutex, and mmap_lock:W. In v4 I will update
prescan_one_lock_expr() to skip attaching fentry/fexit hooks
altogether for specific *_mutex symbols, *_rt_mutex symbols, and
write-locked mmap_lock (mmap_lock:W / lock_held_write(mmap_lock)), so
those filters only inspect lock->owner on sample overflow with zero
lock fastpath tracing overhead.
> > - Tracking these locks (as well as slab-cache and lock-type filters)
> > relies on `fexit` (acquire) and `fentry` (release) BPF hooks
> > maintaining the `held_locks` BPF hash map:
> > - Any such lock already held before `perf record` attaches its BPF
> > programs will not be present in `held_locks` until it is released
> > and re-acquired.
>
> Right, but this can be mitigated when it runs for long.
>
>
> > - The attached `fentry`/`fexit` hooks execute on every kernel lock
> > acquire and release for the hooked lock family across the system,
> > adding tracing overhead outside of perf sample handling.
>
> This is a real problem. As it touches the fastpath it can cause a lot
> of overhead if locking is frequent. But having this capability would be
> very useful anyway!
Agreed. We may also be able to optimize or compromise between lock
performance and observability once this gets adopted.
Thanks,
Ian
> Thanks,
> Namhyung
>
> >
> > Patch overview:
> > 1. "perf lock contention: Share common BPF and lock type definitions in
> > lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and
> > `LCB_F_*` flags, CO-RE structs, and common BPF lock helpers
> > (`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`,
> > `get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`,
> > `do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`.
> > 2. "perf bpf-filter: Add lock_held() sample filter": Implements
> > `lock_held()`, `lock_held_read()`/`lock_held_shared()`,
> > `lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode
> > suffixes using direct lock owner inspection where possible (`mutex`,
> > `rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF map
> > updated by selectively attached `fentry`/`fexit` hooks, supporting
> > both PMU/software events (`SEC("perf_event")`) and tracepoint events
> > (`SEC("tracepoint")`).
> > 3. "perf test: Add tests for lock_held() BPF sample filter": Adds unit
> > tests in `tests/parse-events.c` and shell tests in
> > `tests/shell/record_bpf_filter.sh`.
> > 4. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
> > and SRCU": Supports `rcu` (`rcu:R` via direct
> > `current->rcu_read_lock_nesting` inspection with zero tracing hooks;
> > `rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks),
> > `rcu-trace` (via `current->trc_reader_nesting`), and `srcu`
> > (`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W`
> > via `synchronize_srcu*()` / `srcu_barrier()`).
> > 5. "perf lock contention: Extend BPF lock contention to RCU and SRCU":
> > Extends `perf lock contention --use-bpf` to track RCU and SRCU
> > grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`).
> > 6. "perf lock contention: Filter events before creating lock_stat":
> > Fixes `report_lock_contention_begin_event()` in `builtin-lock.c` to
> > apply `-Y`, `-L`, and `-S` filters before calling
> > `lock_stat_findnew()`.
> >
> > Changes in v3:
> > - Move the writer-only `rw_semaphore` check out of `get_lock_owner()` in
> > `lock_data.h` into `is_rwsem_writer_owner()` in `sample_filter.bpf.c`
> > so `perf lock contention -b -o` still reports reader owners when a
> > writer contends on a reader-owned `rw_semaphore`, and scope
> > `is_direct_lock_owner()` by lock type (`PBF_LOCK_DIRECT_MUTEX`,
> > `PBF_LOCK_DIRECT_RWSEM`, `PBF_LOCK_DIRECT_RTMUTEX`).
> > - Add `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF
> > sample filters (`lock_held()`, `uid`, `gid`, etc.) can attach to
> > `PERF_TYPE_TRACEPOINT` events (such as `kmem:mm_page_alloc` and
> > `vmscan:mm_vmscan_direct_reclaim_begin`), check `libbpf_get_error()`
> > when attaching, and document memory allocation / direct reclaim
> > filtering with `lock_held()`.
> > - Return early in `init_numa_data()` in `bpf-filter.c` when `node_data`
> > cannot be resolved and check `node_data_addr != 0` in
> > `do_collect_zone_lock()`.
> > - Avoid double-counting `_raw_read_lock_irq` / `_raw_read_lock_bh` in
> > `sample_filter.bpf.c` on `CONFIG_GENERIC_LOCKBREAK=y &&
> > !CONFIG_DEBUG_LOCK_ALLOC` kernels where they call
> > `_raw_read_lock_irqsave()`.
> > - Offset `rcu_state_addr` by 1 byte so it is distinct from
> > `&rcu_state.node[0].lock` at offset 0 of `rcu_state`, and pass `bool
> > is_rcu` to `do_contention_end()` in `lock_contention.bpf.c` so
> > `tp_btf/contention_end` and RCU/SRCU `fexit` hooks never cross-match
> > timestamp maps.
> >
> > Changes in v2:
> > - Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists`
> > and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE
> > relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels.
> > - Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points
> > (`_mutex_lock_nest_lock`, `_mutex_lock_killable`,
> > `_mutex_trylock_nest_lock`, `_down_write_nest_lock`,
> > `_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to
> > `sample_filter.bpf.c` and `lock_progs[]`.
> > - Verify in `supported_lock_prog_groups()` that both acquire and all
> > unlock BTF entry points exist for a lock family before enabling its
> > tracing hooks, preventing acquire-only attachment when spinlock/rwlock
> > unlock functions are inlined.
> > - Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when
> > a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`).
> > - Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write`
> > on `track_rwsem_read` and verify non-zero read/write counts in
> > `record_lock_released_addr()` before decrementing aggregate slab/type
> > counts.
> > - Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in
> > `lock_contention.bpf.c` and recognize `synchronize_rcu*`,
> > `synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in
> > `machine__is_lock_function()`.
> > - Add patch 6 fixing `report_lock_contention_begin_event()` to check
> > type, address, and callstack filters before creating `lock_stat`
> > entries.
> >
> > Assisted-by: Antigravity:gemini-3.1-pro
> >
> > Ian Rogers (6):
> > perf lock contention: Share common BPF and lock type definitions in
> > lock_data.h
> > perf bpf-filter: Add lock_held() sample filter
> > perf test: Add tests for lock_held() BPF sample filter
> > perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
> > and SRCU
> > perf lock contention: Extend BPF lock contention to RCU and SRCU
> > perf lock contention: Filter events before creating lock_stat
> >
> > tools/perf/Documentation/perf-lock.txt | 7 +-
> > tools/perf/Documentation/perf-record.txt | 64 +-
> > tools/perf/builtin-lock.c | 88 +-
> > tools/perf/tests/parse-events.c | 121 ++
> > tools/perf/tests/shell/lock_contention.sh | 14 +
> > tools/perf/tests/shell/record_bpf_filter.sh | 134 ++
> > tools/perf/util/bpf-filter.c | 1271 ++++++++++++++++-
> > tools/perf/util/bpf-filter.h | 13 +
> > tools/perf/util/bpf-filter.l | 31 +-
> > tools/perf/util/bpf-filter.y | 46 +-
> > tools/perf/util/bpf_lock_contention.c | 171 ++-
> > .../perf/util/bpf_skel/lock_contention.bpf.c | 448 +++---
> > tools/perf/util/bpf_skel/lock_data.h | 360 ++++-
> > tools/perf/util/bpf_skel/sample-filter.h | 77 +
> > tools/perf/util/bpf_skel/sample_filter.bpf.c | 1225 ++++++++++++++++
> > tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 +
> > tools/perf/util/lock-contention.c | 22 +
> > tools/perf/util/lock-contention.h | 15 +
> > tools/perf/util/machine.c | 11 +
> > tools/perf/util/parse-events.c | 3 +-
> > 20 files changed, 3741 insertions(+), 387 deletions(-)
> >
> > --
> > 2.56.0.212.g129ede20d9-goog
> >
^ permalink raw reply [flat|nested] 106+ messages in thread
* [PATCH v4 0/7] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support
2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (6 preceding siblings ...)
2026-10-19 7:00 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Namhyung Kim
@ 2026-10-19 22:09 ` Ian Rogers
2026-10-19 22:09 ` [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat Ian Rogers
` (7 more replies)
7 siblings, 8 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, eranian, ddhariwal, rcu
This series adds a new BPF sample filter predicate, `lock_held(<lock>)`
(along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`,
`lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W`
mode suffixes), to `perf record --filter`, allowing perf event samples to
be recorded only when a specified kernel lock is held (or not held with
`!`) by the running kernel thread or CPU. It also fixes a pre-existing
`lock_stat` filtering bug in `perf lock contention`, enables BPF sample
filters on tracepoint events (`PERF_TYPE_TRACEPOINT`), and extends both
`lock_held()` and `perf lock contention --use-bpf` to support RCU and
SRCU.
For example:
# perf record -a -g -e cycles --filter 'lock_held(cgroup_mutex)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(mmap_lock)' -- ...
# perf record -a -g -e cycles --filter 'lock_held(mmap_lock:W)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ...
# perf lock contention -a -b -Y rcu,srcu -- sleep 1
An important use-case is detecting memory allocations or direct memory
reclaim occurring while kernel locks are held, which can stall lock
waiters for extended periods under memory pressure:
# Direct memory reclaim triggered while holding any mutex or rwsem:
# perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \
--filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
# Page allocations permitted to enter direct reclaim (__GFP_DIRECT_RECLAIM)
# while holding a mutex or rwsem:
# perf record -a -g -e kmem:mm_page_alloc \
--filter 'gfp_flags & 0x400' \
--filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
# Major page faults occurring while holding a mutex or rwsem:
# perf record -a -g -e major-faults \
--filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
Supported `<lock>` specifications match `perf lock contention`:
- Global/static kernel lock symbols (e.g. `lock_held(cgroup_mutex)`)
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`,
`lock_held_read(&dentry)`)
- Well-known per-process, semi-global, and RCU locks: `mmap_lock`,
`siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`
- Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`,
`rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`,
`percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`,
`rcu:W`, `semaphore`
Limitations and weaknesses in determining whether a lock is held:
- Inlined lock/unlock operations:
- When the kernel is configured to inline spinlock or rwlock operations
(`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`,
`CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`,
`CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` /
`_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are
inlined into callers and have no BTF function entry points for
`fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that
both acquire and release entry points exist in BTF before enabling a
lock family.
- Fast-path `percpu_down_read()` / `percpu_up_read()` and
`srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined
per-CPU counter increments without function calls, so only
`percpu_down_write()` / `percpu_up_write()` and out-of-line
`__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked.
- On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to
`preempt_disable()` and does not maintain
`current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)`
read-side inspection requires `CONFIG_PREEMPT_RCU`.
- Lack of an inspectable owner field on non-exclusive / spinning locks:
- While `struct mutex`, `struct rt_mutex`, and write-locked
`struct rw_semaphore` store the owning `task_struct *` in `lock->owner`
(allowing `perf_sample_filter` to inspect kernel memory directly on
sample overflow when filtering by lock address without attaching any
`fentry`/`fexit` hooks for `*_mutex`, `*_rt_mutex`, and `mmap_lock:W`),
spinlocks, rwlocks, semaphores, SRCU, and reader-locked
`rw_semaphore`s (where `__up_read()` does not clear `sem->owner` when
`!CONFIG_DEBUG_RWSEMS`) do not maintain an authoritative per-task
owner pointer.
- Tracking these locks (as well as slab-cache and lock-type filters)
relies on `fexit` (acquire) and `fentry` (release) BPF hooks
maintaining the `held_locks` BPF LRU hash map:
- Any such lock already held before `perf record` attaches its BPF
programs will not be present in `held_locks` until it is released
and re-acquired.
- The attached `fentry`/`fexit` hooks execute on every kernel lock
acquire and release for the hooked lock family across the system,
adding tracing overhead outside of perf sample handling.
- Cross-task semaphore signaling (`down*()` / `up()`):
- Counting `struct semaphore`s do not enforce same-task ownership in the
kernel: when a semaphore is used as a completion/signaling mechanism
(`down()` in one task woken by `up()` in another task or interrupt
handler, which proposed kernel dependency trackers like `CONFIG_DEPT`
model as wait/event dependencies bounded by execution context rather
than lock ownership), `fexit/down` and `fentry/up` do not pair on the
same task ID. Semaphore hooks are therefore only enabled when
`semaphore` or a `*_sem` symbol (such as `console_sem`) is explicitly
specified, and `held_locks` uses `BPF_MAP_TYPE_LRU_HASH` so any
unpaired cross-task entries are automatically evicted.
Patch overview (patches 1 and 2 are standalone fixes independent of the
rest of the series):
1. "perf lock contention: Filter events before creating lock_stat":
Fixes `report_lock_contention_begin_event()` in `builtin-lock.c` to
apply `-Y`, `-L`, and `-S` filters before calling
`lock_stat_findnew()`.
2. "perf bpf-filter: Support BPF sample filters on tracepoint events":
Adds `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF
sample filters can attach to `PERF_TYPE_TRACEPOINT` events (where
`__perf_event_set_bpf_prog()` in `kernel/events/core.c` requires
`BPF_PROG_TYPE_TRACEPOINT` rather than `BPF_PROG_TYPE_PERF_EVENT`).
3. "perf lock contention: Share common BPF and lock type definitions in
lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and
`LCB_F_*` flags, CO-RE structs, and common BPF lock helpers
(`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`,
`get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`,
`do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`.
4. "perf bpf-filter: Add lock_held() sample filter": Implements
`lock_held()`, `lock_held_read()`/`lock_held_shared()`,
`lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode
suffixes using direct lock owner inspection where possible (`mutex`,
`rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF LRU
hash map updated by selectively attached `fentry`/`fexit` hooks.
5. "perf test: Add tests for lock_held() BPF sample filter": Adds unit
tests in `tests/parse-events.c` and shell tests in
`tests/shell/record_bpf_filter.sh`.
6. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU": Supports `rcu` (`rcu:R` via direct
`current->rcu_read_lock_nesting` inspection with zero tracing hooks;
`rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks),
`rcu-trace` (via `current->trc_reader_nesting`), and `srcu`
(`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W`
via `synchronize_srcu*()` / `srcu_barrier()`).
7. "perf lock contention: Extend BPF lock contention to RCU and SRCU":
Extends `perf lock contention --use-bpf` to track RCU and SRCU
grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`).
Changes in v4:
- Move the two bugfix commits (`Filter events before creating lock_stat`
and `Support BPF sample filters on tracepoint events`) to patches 1
and 2 at the start of the series for easier cherry-picking.
- Reserve leading `&` strictly for slab cache locks (e.g.
`lock_held(&dentry)`) so global/static lock symbols and well-known
locks do not use `&` (e.g. `lock_held(cgroup_mutex)`,
`lock_held_read(mmap_lock)`), matching `perf lock contention`
(Namhyung).
- Skip attaching `fentry`/`fexit` hooks when filtering on a specific
`mutex` symbol (`*_mutex`), `rt_mutex` symbol (`*_rt_mutex`), or
write-locked `mmap_lock` (`mmap_lock:W` / `lock_held_write(mmap_lock)`),
evaluating them purely via direct `owner` inspection (Namhyung).
- Use a CO-RE flavor struct `struct mutex___non_rt` in `get_lock_owner()`
in `lock_data.h` so `GEN_VMLINUX_H=1` builds succeed on
`CONFIG_PREEMPT_RT=y` kernels.
- Switch `held_locks` to `BPF_MAP_TYPE_LRU_HASH`, exclude `LOCK_PROG_SEM`
from `LOCK_PROG_ALL` unless `semaphore` or a `*_sem` symbol is
explicitly requested, and document cross-task semaphore signaling
limitations (and comparison with `CONFIG_DEPT`).
- Skip `LCB_F_MUTEX | LCB_F_SPIN` in `parse_lock_type_name()`, reject
read-only mode on exclusive-only lock symbol suffixes in
`resolve_lock_expr()`, check `needed_groups & supported_groups` before
masking `needed_groups` in `configure_lock_progs()`, and store
`skel->links.perf_tp_sample_filter` so `sample_filter_bpf__destroy()`
cleans up the tracepoint BPF link.
- Recursively traverse anonymous structs/unions in `btf_has_task_field()`
so `lock_held(rcu)` and `lock_held(rcu-trace)` work on
`CONFIG_RANDSTRUCT` kernels.
Changes in v3:
- Move the writer-only `rw_semaphore` check out of `get_lock_owner()` in
`lock_data.h` into `is_rwsem_writer_owner()` in `sample_filter.bpf.c`
so `perf lock contention -b -o` still reports reader owners when a
writer contends on a reader-owned `rw_semaphore`, and scope
`is_direct_lock_owner()` by lock type (`PBF_LOCK_DIRECT_MUTEX`,
`PBF_LOCK_DIRECT_RWSEM`, `PBF_LOCK_DIRECT_RTMUTEX`).
- Add `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF
sample filters (`lock_held()`, `uid`, `gid`, etc.) can attach to
`PERF_TYPE_TRACEPOINT` events (such as `kmem:mm_page_alloc` and
`vmscan:mm_vmscan_direct_reclaim_begin`), check `libbpf_get_error()`
when attaching, and document memory allocation / direct reclaim
filtering with `lock_held()`.
- Return early in `init_numa_data()` in `bpf-filter.c` when `node_data`
cannot be resolved and check `node_data_addr != 0` in
`do_collect_zone_lock()`.
- Avoid double-counting `_raw_read_lock_irq` / `_raw_read_lock_bh` in
`sample_filter.bpf.c` on `CONFIG_GENERIC_LOCKBREAK=y &&
!CONFIG_DEBUG_LOCK_ALLOC` kernels where they call
`_raw_read_lock_irqsave()`.
- Offset `rcu_state_addr` by 1 byte so it is distinct from
`&rcu_state.node[0].lock` at offset 0 of `rcu_state`, and pass `bool
is_rcu` to `do_contention_end()` in `lock_contention.bpf.c` so
`tp_btf/contention_end` and RCU/SRCU `fexit` hooks never cross-match
timestamp maps.
Changes in v2:
- Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists`
and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE
relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels.
- Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points
(`_mutex_lock_nest_lock`, `_mutex_lock_killable`,
`_mutex_trylock_nest_lock`, `_down_write_nest_lock`,
`_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to
`sample_filter.bpf.c` and `lock_progs[]`.
- Verify in `supported_lock_prog_groups()` that both acquire and all
unlock BTF entry points exist for a lock family before enabling its
tracing hooks, preventing acquire-only attachment when spinlock/rwlock
unlock functions are inlined.
- Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when
a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`).
- Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write`
on `track_rwsem_read` and verify non-zero read/write counts in
`record_lock_released_addr()` before decrementing aggregate slab/type
counts.
- Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in
`lock_contention.bpf.c` and recognize `synchronize_rcu*`,
`synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in
`machine__is_lock_function()`.
- Add patch fixing `report_lock_contention_begin_event()` to check
type, address, and callstack filters before creating `lock_stat`
entries.
Assisted-by: Antigravity:gemini-3.1-pro
Ian Rogers (7):
perf lock contention: Filter events before creating lock_stat
perf bpf-filter: Support BPF sample filters on tracepoint events
perf lock contention: Share common BPF and lock type definitions in
lock_data.h
perf bpf-filter: Add lock_held() sample filter
perf test: Add tests for lock_held() BPF sample filter
perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU
perf lock contention: Extend BPF lock contention to RCU and SRCU
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/Documentation/perf-record.txt | 66 +-
tools/perf/builtin-lock.c | 88 +-
tools/perf/tests/parse-events.c | 121 ++
tools/perf/tests/shell/lock_contention.sh | 14 +
tools/perf/tests/shell/record_bpf_filter.sh | 142 ++
tools/perf/util/bpf-filter.c | 1300 ++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 46 +-
tools/perf/util/bpf_lock_contention.c | 171 ++-
.../perf/util/bpf_skel/lock_contention.bpf.c | 448 +++---
tools/perf/util/bpf_skel/lock_data.h | 364 ++++-
tools/perf/util/bpf_skel/sample-filter.h | 77 +
tools/perf/util/bpf_skel/sample_filter.bpf.c | 1231 ++++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 +
tools/perf/util/lock-contention.c | 22 +
tools/perf/util/lock-contention.h | 15 +
tools/perf/util/machine.c | 11 +
tools/perf/util/parse-events.c | 3 +-
20 files changed, 3790 insertions(+), 387 deletions(-)
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply [flat|nested] 106+ messages in thread
* [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat
2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers
@ 2026-10-19 22:09 ` Ian Rogers
2026-10-19 22:14 ` sashiko-bot
2026-10-20 4:34 ` Namhyung Kim
2026-10-19 22:09 ` [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
` (6 subsequent siblings)
7 siblings, 2 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, eranian, ddhariwal, rcu
In report_lock_contention_begin_event(), lock_stat_findnew() was
called before checking filters.nr_types (-Y), filters.nr_addrs (-L),
and match_callstack_filter() (-S).
When a filtered-out lock event (such as a mutex contention when
running with -Y spinlock) shared the same aggregation key as a
subsequent matching event (for example, contention on the mutex's
internal wait_lock spinlock from the same caller), the first event
created the lock_stat entry with the filtered-out lock's flags and the
second event reused that entry, causing non-spinlock entries to appear
in -Y spinlock output.
Check the type, address, and callstack filters before looking up or
creating the lock_stat entry.
Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option")
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/builtin-lock.c | 61 +++++++++++++++++++++------------------
1 file changed, 33 insertions(+), 28 deletions(-)
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index d5c0d55cb82d..778923c92f42 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -975,6 +975,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
struct machine *machine = &session->machines.host;
struct map *kmap;
struct symbol *sym;
+ u64 *callstack = NULL;
ret = get_key_by_aggr_mode(&key, addr, sample);
if (ret < 0)
@@ -1010,33 +1011,6 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
}
}
- ls = lock_stat_find(key);
- if (!ls) {
- char buf[128];
- const char *name = "";
-
- switch (aggr_mode) {
- case LOCK_AGGR_ADDR:
- sym = machine__find_kernel_symbol(machine, key, &kmap);
- if (sym)
- name = sym->name;
- break;
- case LOCK_AGGR_CALLER:
- name = buf;
- if (lock_contention_caller(sample, buf, sizeof(buf)) < 0)
- name = "Unknown";
- break;
- case LOCK_AGGR_CGROUP:
- case LOCK_AGGR_TASK:
- default:
- break;
- }
-
- ls = lock_stat_findnew(key, name, flags);
- if (!ls)
- return -ENOMEM;
- }
-
if (filters.nr_types) {
bool found = false;
@@ -1066,7 +1040,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
}
if (needs_callstack()) {
- u64 *callstack = get_callstack(sample, max_stack_depth);
+ callstack = get_callstack(sample, max_stack_depth);
if (callstack == NULL)
return 0;
@@ -1074,7 +1048,38 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
free(callstack);
return 0;
}
+ }
+
+ ls = lock_stat_find(key);
+ if (!ls) {
+ char buf[128];
+ const char *name = "";
+
+ switch (aggr_mode) {
+ case LOCK_AGGR_ADDR:
+ sym = machine__find_kernel_symbol(machine, key, &kmap);
+ if (sym)
+ name = sym->name;
+ break;
+ case LOCK_AGGR_CALLER:
+ name = buf;
+ if (lock_contention_caller(sample, buf, sizeof(buf)) < 0)
+ name = "Unknown";
+ break;
+ case LOCK_AGGR_CGROUP:
+ case LOCK_AGGR_TASK:
+ default:
+ break;
+ }
+
+ ls = lock_stat_findnew(key, name, flags);
+ if (!ls) {
+ free(callstack);
+ return -ENOMEM;
+ }
+ }
+ if (callstack) {
if (ls->callstack == NULL)
ls->callstack = callstack;
else
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events
2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers
2026-10-19 22:09 ` [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat Ian Rogers
@ 2026-10-19 22:09 ` Ian Rogers
2026-10-19 22:26 ` sashiko-bot
2026-10-20 5:41 ` Namhyung Kim
2026-10-19 22:09 ` [PATCH v4 3/7] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
` (5 subsequent siblings)
7 siblings, 2 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, eranian, ddhariwal, rcu
Commit 466db4275edd ("perf parse-events: Add parse_uid_filter helper")
updated set_filter() to route non-tracepoint filter terms (such as uid
and gid) on PERF_TYPE_TRACEPOINT events to evsel->bpf_filters instead
of the ftrace tracepoint predicate filter.
However, in the kernel's __perf_event_set_bpf_prog()
(kernel/events/core.c), attaching a BPF program to a perf_event fd
branches on perf_event_is_tracing(event):
- Non-tracing events (hardware PMU, software events, breakpoints) call
perf_event_set_bpf_handler(), which requires BPF_PROG_TYPE_PERF_EVENT
(SEC("perf_event")).
- Tracepoint events (event->pmu == &perf_tracepoint) reject
BPF_PROG_TYPE_PERF_EVENT with -EINVAL and instead require
BPF_PROG_TYPE_TRACEPOINT (SEC("tracepoint")), which is attached to
event->tp_event->prog_array and executed in
perf_trace_run_bpf_submit() before perf_tp_event(), dropping the
sample if the BPF program returns 0. Furthermore, the
bpf_cast_to_kern_ctx() kfunc is only permitted in
BPF_PROG_TYPE_PERF_EVENT programs.
Add SEC("tracepoint") int perf_tp_sample_filter(void *ctx) in
sample_filter.bpf.c to evaluate context-independent filter terms (uid,
gid, tid, cpu, time, cgroup) using BPF helpers without calling
bpf_cast_to_kern_ctx(). When preparing a BPF filter on a
PERF_TYPE_TRACEPOINT evsel, autoload perf_tp_sample_filter instead of
perf_sample_filter and attach it once to the shared trace_event_call.
Before:
# perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true
libbpf: prog 'perf_sample_filter': failed to attach to perf_event FD 3: Invalid argument
Failed to attach perf sample-filter program
failed to set filter "BPF" on event kmem:mm_page_alloc with 22 (Invalid argument)
After:
# perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.024 MB perf.data (42 samples) ]
Fixes: 466db4275edd ("perf parse-events: Add parse_uid_filter helper")
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/bpf-filter.c | 136 ++++++++++++++++---
tools/perf/util/bpf_skel/sample_filter.bpf.c | 85 ++++++++++++
2 files changed, 201 insertions(+), 20 deletions(-)
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 72e83ac1d6a4..4e99af4764f1 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -123,15 +123,26 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
{
const struct perf_sample_info *info;
- if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END &&
- (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
- return 0;
-
if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
/* Not dependent on the sample_type as computed from a BPF helper. */
return 0;
}
+ if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT) {
+ if (expr->term == PBF_TERM_TID || expr->term == PBF_TERM_CPU ||
+ expr->term == PBF_TERM_TIME || expr->term == PBF_TERM_CGROUP)
+ return 0;
+ if (expr->op != PBF_OP_GROUP_BEGIN) {
+ pr_err("Error: %s tracepoint event does not support BPF sample term %d\n",
+ evsel__name(evsel), expr->term);
+ return -1;
+ }
+ }
+
+ if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END &&
+ (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
+ return 0;
+
if (expr->op == PBF_OP_GROUP_BEGIN) {
struct perf_bpf_filter_expr *group;
@@ -427,6 +438,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
struct bpf_link *link;
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
+ bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
#if LIBBPF_CURRENT_VERSION_GEQ(1, 7)
DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts,
.dont_enable = true);
@@ -444,9 +456,15 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- if (needs_idx_hash && geteuid() != 0) {
+ if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
int zero = 0;
+ if (is_tracepoint) {
+ pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n");
+ ret = -EPERM;
+ goto err;
+ }
+
/* The filters map is shared among other processes */
ret = create_idx_hash(evsel, entry);
if (ret < 0)
@@ -484,8 +502,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return 0;
}
- skel = sample_filter_bpf__open_and_load();
+ skel = sample_filter_bpf__open();
if (!skel) {
+ ret = -errno;
+ pr_err("Failed to open perf sample-filter BPF skeleton\n");
+ goto err;
+ }
+
+ bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
+ bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
+
+ if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
pr_err("Failed to load perf sample-filter BPF skeleton\n");
goto err;
@@ -501,15 +528,35 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- prog = skel->progs.perf_sample_filter;
- for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
- for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
- link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y),
- &pe_opts);
- if (IS_ERR(link)) {
- pr_err("Failed to attach perf sample-filter program\n");
- ret = PTR_ERR(link);
- goto err;
+ if (is_tracepoint) {
+ /*
+ * For PERF_TYPE_TRACEPOINT, PERF_EVENT_IOC_SET_BPF attaches the
+ * BPF program to event->tp_event->prog_array on the shared
+ * trace_event_call, which perf_trace_run_bpf_submit() executes
+ * for all CPUs and tasks before iterating the per-CPU perf
+ * events. Attach once on the first FD so tp_event->prog_array
+ * does not get duplicate copies for each per-CPU FD.
+ */
+ prog = skel->progs.perf_tp_sample_filter;
+ link = bpf_program__attach_perf_event_opts(prog, FD(evsel, 0, 0),
+ &pe_opts);
+ ret = libbpf_get_error(link);
+ if (ret) {
+ pr_err("Failed to attach perf tracepoint sample-filter program\n");
+ goto err;
+ }
+ skel->links.perf_tp_sample_filter = link;
+ } else {
+ prog = skel->progs.perf_sample_filter;
+ for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
+ for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
+ link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y),
+ &pe_opts);
+ ret = libbpf_get_error(link);
+ if (ret) {
+ pr_err("Failed to attach perf sample-filter program\n");
+ goto err;
+ }
}
}
}
@@ -641,6 +688,45 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
return ret;
}
+static int pin_filter_object(struct sample_filter_bpf *skel, const char *path)
+{
+ char *prog_path = NULL;
+ int ret;
+
+ ret = bpf_object__pin_maps(skel->obj, path);
+ if (ret < 0)
+ return ret;
+
+ if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
+ bpf_object__unpin_maps(skel->obj, path);
+ return -ENOMEM;
+ }
+
+ ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path);
+ if (ret < 0)
+ bpf_object__unpin_maps(skel->obj, path);
+
+ free(prog_path);
+ return ret;
+}
+
+static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path)
+{
+ char *prog_path = NULL;
+ int ret, map_ret;
+
+ if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
+ bpf_object__unpin_maps(skel->obj, path);
+ return -ENOMEM;
+ }
+
+ ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path);
+ free(prog_path);
+
+ map_ret = bpf_object__unpin_maps(skel->obj, path);
+ return ret < 0 ? ret : map_ret;
+}
+
int perf_bpf_filter__pin(void)
{
struct sample_filter_bpf *skel;
@@ -655,6 +741,7 @@ int perf_bpf_filter__pin(void)
}
/* pinned program will use pid-hash */
+ bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH);
bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH);
@@ -674,7 +761,7 @@ int perf_bpf_filter__pin(void)
goto err;
}
- ret = bpf_object__pin(skel->obj, path);
+ ret = pin_filter_object(skel, path);
if (ret < 0) {
pr_err("Failed to pin BPF filter objects\n");
goto err;
@@ -683,8 +770,8 @@ int perf_bpf_filter__pin(void)
/* setup access permissions for the pinned objects */
dir_fd = open(path, O_PATH);
if (dir_fd < 0) {
- bpf_object__unpin(skel->obj, path);
- ret = dir_fd;
+ ret = -errno;
+ unpin_filter_object(skel, path);
goto err;
}
@@ -727,6 +814,8 @@ int perf_bpf_filter__pin(void)
err_close:
close(dir_fd);
+ if (ret < 0)
+ unpin_filter_object(skel, path);
err:
free(path);
@@ -740,13 +829,20 @@ int perf_bpf_filter__unpin(void)
char *path = NULL;
int ret = -1;
- skel = sample_filter_bpf__open_and_load();
+ skel = sample_filter_bpf__open();
if (!skel) {
ret = -errno;
pr_err("Failed to open perf sample-filter BPF skeleton\n");
goto err;
}
+ bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
+ if (sample_filter_bpf__load(skel) < 0) {
+ ret = -errno;
+ pr_err("Failed to load perf sample-filter BPF skeleton\n");
+ goto err;
+ }
+
if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(),
PERF_BPF_FILTER_PIN_PATH) < 0) {
ret = -errno;
@@ -754,7 +850,7 @@ int perf_bpf_filter__unpin(void)
goto err;
}
- ret = bpf_object__unpin(skel->obj, path);
+ ret = unpin_filter_object(skel, path);
err:
free(path);
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index e5666d4c1722..576fbed270e5 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -192,6 +192,28 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
return 0;
}
+static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry *entry)
+{
+ switch (entry->term) {
+ case PBF_TERM_TID:
+ if (entry->part)
+ return bpf_get_current_pid_tgid() >> 32;
+ return (__u32)bpf_get_current_pid_tgid();
+ case PBF_TERM_CPU:
+ return bpf_get_smp_processor_id();
+ case PBF_TERM_TIME:
+ return bpf_ktime_get_ns();
+ case PBF_TERM_CGROUP:
+ return bpf_get_current_cgroup_id();
+ case PBF_TERM_UID:
+ return bpf_get_current_uid_gid() & 0xFFFFFFFF;
+ case PBF_TERM_GID:
+ return bpf_get_current_uid_gid() >> 32;
+ default:
+ return 0;
+ }
+}
+
#define CHECK_RESULT(data, op, val) \
if (!(data op val)) { \
if (!in_group) \
@@ -295,4 +317,67 @@ int perf_sample_filter(void *ctx)
return 0;
}
+/* BPF program to be called for tracepoint perf events */
+SEC("tracepoint")
+int perf_tp_sample_filter(void *ctx)
+{
+ struct perf_bpf_filter_entry *entry;
+ __u64 sample_data;
+ int in_group = 0;
+ int group_result = 0;
+ int i, k = 0;
+ int *losts;
+
+ entry = bpf_map_lookup_elem(&filters, &k);
+ if (entry == NULL)
+ goto drop;
+
+ for (i = 0; i < MAX_FILTERS; i++) {
+ sample_data = perf_get_common_sample(&entry[i]);
+
+ switch (entry[i].op) {
+ case PBF_OP_EQ:
+ CHECK_RESULT(sample_data, ==, entry[i].value)
+ break;
+ case PBF_OP_NEQ:
+ CHECK_RESULT(sample_data, !=, entry[i].value)
+ break;
+ case PBF_OP_GT:
+ CHECK_RESULT(sample_data, >, entry[i].value)
+ break;
+ case PBF_OP_GE:
+ CHECK_RESULT(sample_data, >=, entry[i].value)
+ break;
+ case PBF_OP_LT:
+ CHECK_RESULT(sample_data, <, entry[i].value)
+ break;
+ case PBF_OP_LE:
+ CHECK_RESULT(sample_data, <=, entry[i].value)
+ break;
+ case PBF_OP_AND:
+ CHECK_RESULT(sample_data, &, entry[i].value)
+ break;
+ case PBF_OP_GROUP_BEGIN:
+ in_group = 1;
+ group_result = 0;
+ break;
+ case PBF_OP_GROUP_END:
+ if (group_result == 0)
+ goto drop;
+ in_group = 0;
+ break;
+ case PBF_OP_DONE:
+ return 1;
+ }
+ }
+ return 1;
+
+drop:
+ losts = bpf_map_lookup_elem(&dropped, &k);
+ if (losts != NULL)
+ __sync_fetch_and_add(losts, 1);
+
+ return 0;
+}
+
char LICENSE[] SEC("license") = "Dual BSD/GPL";
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v4 3/7] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers
2026-10-19 22:09 ` [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat Ian Rogers
2026-10-19 22:09 ` [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
@ 2026-10-19 22:09 ` Ian Rogers
2026-10-19 22:17 ` sashiko-bot
2026-10-19 22:09 ` [PATCH v4 4/7] perf bpf-filter: Add lock_held() sample filter Ian Rogers
` (4 subsequent siblings)
7 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, eranian, ddhariwal, rcu
Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers
(get_lock_owner, get_current_mmap_lock, get_current_siglock,
get_lock_addr, check_lock_type, do_collect_zone_lock,
do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into
tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF
sample filtering.
Also add rw_semaphore.count and struct rt_mutex_base to the minimal
vmlinux.h and extend get_lock_owner() to support rt_mutex and check
the mutex handoff flag.
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/builtin-lock.c | 27 --
.../perf/util/bpf_skel/lock_contention.bpf.c | 265 +--------------
tools/perf/util/bpf_skel/lock_data.h | 320 ++++++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 +
tools/perf/util/lock-contention.c | 18 +
tools/perf/util/lock-contention.h | 13 +
6 files changed, 374 insertions(+), 274 deletions(-)
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index 778923c92f42..a031709a2cc2 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -1482,33 +1482,6 @@ static void sort_result(void)
}
}
-static const struct {
- unsigned int flags;
- /*
- * Name of the lock flags (access), with delimeter ':'.
- * For example, rwsem:R of rwsem:W.
- */
- const char *flags_name;
- /* Name of the lock (type), for example, rwlock or rwsem. */
- const char *lock_name;
-} lock_type_table[] = {
- { 0, "semaphore", "semaphore" },
- { LCB_F_SPIN, "spinlock", "spinlock" },
- { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
- { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
- { LCB_F_READ, "rwsem:R", "rwsem" },
- { LCB_F_WRITE, "rwsem:W", "rwsem" },
- { LCB_F_RT, "rt-mutex", "rt-mutex" },
- { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
- { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
- { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
- { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
- { LCB_F_MUTEX, "mutex", "mutex" },
- { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
- /* alias for optimistic spinning only */
- { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
-};
-
static const char *get_type_flags_name(unsigned int flags)
{
flags &= LCB_F_TYPE_MASK;
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index 0d9c6f55050e..0f2118006005 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -8,23 +8,9 @@
#include "lock_data.h"
-/* for collect_lock_syms(). 4096 was rejected by the verifier */
-#define MAX_CPUS 1024
-
-/* for collect_zone_lock(). It should be more than the actual zones. */
-#define MAX_ZONES 10
-
/* for do_lock_delay(). Arbitrarily set to 1 million. */
#define MAX_LOOP (1U << 20)
-/* lock contention flags from include/trace/events/lock.h */
-#define LCB_F_SPIN (1U << 0)
-#define LCB_F_READ (1U << 1)
-#define LCB_F_WRITE (1U << 2)
-#define LCB_F_RT (1U << 3)
-#define LCB_F_PERCPU (1U << 4)
-#define LCB_F_MUTEX (1U << 5)
-
/* callstack storage */
struct {
__uint(type, BPF_MAP_TYPE_STACK_TRACE);
@@ -96,13 +82,6 @@ struct {
__uint(max_entries, MAX_ENTRIES);
} task_data SEC(".maps");
-struct {
- __uint(type, BPF_MAP_TYPE_HASH);
- __uint(key_size, sizeof(__u64));
- __uint(value_size, sizeof(__u32));
- __uint(max_entries, MAX_ENTRIES);
-} lock_syms SEC(".maps");
-
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(key_size, sizeof(__u32));
@@ -138,6 +117,13 @@ struct {
__uint(max_entries, 1);
} cgroup_filter SEC(".maps");
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u32));
+ __uint(max_entries, MAX_ENTRIES);
+} lock_syms SEC(".maps");
+
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(key_size, sizeof(long));
@@ -159,22 +145,6 @@ struct {
__uint(max_entries, 1);
} lock_delays SEC(".maps");
-struct rw_semaphore___old {
- struct task_struct *owner;
-} __attribute__((preserve_access_index));
-
-struct rw_semaphore___new {
- atomic_long_t owner;
-} __attribute__((preserve_access_index));
-
-struct mm_struct___old {
- struct rw_semaphore mmap_sem;
-} __attribute__((preserve_access_index));
-
-struct mm_struct___new {
- struct rw_semaphore mmap_lock;
-} __attribute__((preserve_access_index));
-
struct cas_ctx {
struct contention_data *data;
u64 duration;
@@ -182,8 +152,6 @@ struct cas_ctx {
int min_done;
};
-extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak;
-
/* control flags */
const volatile int has_cpu;
const volatile int has_task;
@@ -202,6 +170,11 @@ const volatile int lock_delay;
/* determine the key of lock stat */
const volatile int aggr_mode;
+const volatile __u64 contig_page_data_addr;
+const volatile __u64 node_data_addr;
+const volatile int nr_nodes;
+const volatile int sizeof_zone;
+
int enabled;
int perf_subsys_id = -1;
@@ -222,8 +195,6 @@ int data_map_full;
struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak;
void bpf_task_release(struct task_struct *p) __ksym __weak;
-static inline __u32 check_lock_type(__u64 lock, __u32 flags);
-
static inline __u64 get_current_cgroup_id(void)
{
struct task_struct *task;
@@ -347,85 +318,6 @@ static inline int update_task_data(struct task_struct *task)
return 0;
}
-#ifndef __has_builtin
-# define __has_builtin(x) 0
-#endif
-
-static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
-{
- struct task_struct *task;
- __u64 owner = 0;
-
- if (flags & LCB_F_MUTEX) {
- struct mutex *mutex = (void *)lock;
- owner = BPF_CORE_READ(mutex, owner.counter);
- } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
- /*
- * Support for the BPF_TYPE_MATCHES argument to the
- * __builtin_preserve_type_info builtin was added at some point during
- * development of clang 15 and it's what is needed for
- * bpf_core_type_matches.
- */
-#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
- if (bpf_core_type_matches(struct rw_semaphore___old)) {
- struct rw_semaphore___old *rwsem = (void *)lock;
- owner = (unsigned long)BPF_CORE_READ(rwsem, owner);
- } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
- struct rw_semaphore___new *rwsem = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
- }
-#else
- /* assume new struct */
- struct rw_semaphore *rwsem = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
-#endif
- }
-
- if (!owner)
- return NULL;
-
- task = (void *)(owner & ~7UL);
- return task;
-}
-
-static inline __u32 check_lock_type(__u64 lock, __u32 flags)
-{
- struct task_struct *curr;
- struct mm_struct___old *mm_old;
- struct mm_struct___new *mm_new;
- struct sighand_struct *sighand;
-
- switch (flags) {
- case LCB_F_READ: /* rwsem */
- case LCB_F_WRITE:
- curr = bpf_get_current_task_btf();
- if (curr->mm == NULL)
- break;
- mm_new = (void *)curr->mm;
- if (bpf_core_field_exists(mm_new->mmap_lock)) {
- if (&mm_new->mmap_lock == (void *)lock)
- return LCD_F_MMAP_LOCK;
- break;
- }
- mm_old = (void *)curr->mm;
- if (bpf_core_field_exists(mm_old->mmap_sem)) {
- if (&mm_old->mmap_sem == (void *)lock)
- return LCD_F_MMAP_LOCK;
- }
- break;
- case LCB_F_SPIN: /* spinlock */
- curr = bpf_get_current_task_btf();
- sighand = curr->sighand;
-
- if (sighand && &sighand->siglock == (void *)lock)
- return LCD_F_SIGHAND_LOCK;
- break;
- default:
- break;
- }
- return 0;
-}
-
static inline long delay_callback(__u64 idx, void *arg)
{
__u64 target = *(__u64 *)arg;
@@ -908,145 +800,24 @@ int contention_end(u64 *ctx)
return 0;
}
-extern struct rq runqueues __ksym;
-
-const volatile __u64 contig_page_data_addr;
-const volatile __u64 node_data_addr;
-const volatile int nr_nodes;
-const volatile int sizeof_zone;
-
-struct rq___old {
- raw_spinlock_t lock;
-} __attribute__((preserve_access_index));
-
-struct rq___new {
- raw_spinlock_t __lock;
-} __attribute__((preserve_access_index));
-
-static void collect_zone_lock(void)
-{
- __u64 nr_zones, zone_off;
- __u64 lock_addr, lock_off;
- __u32 lock_flag = LOCK_CLASS_ZONE_LOCK;
-
- zone_off = offsetof(struct pglist_data, node_zones);
- lock_off = offsetof(struct zone, lock);
-
- if (contig_page_data_addr) {
- struct pglist_data *contig_page_data;
-
- contig_page_data = (void *)(long)contig_page_data_addr;
- nr_zones = BPF_CORE_READ(contig_page_data, nr_zones);
-
- for (int i = 0; i < MAX_ZONES; i++) {
- __u64 zone_addr;
-
- if (i >= nr_zones)
- break;
-
- zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off;
- lock_addr = zone_addr + lock_off;
-
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
- } else if (nr_nodes > 0) {
- struct pglist_data **node_data = (void *)(long)node_data_addr;
-
- for (int i = 0; i < nr_nodes; i++) {
- struct pglist_data *pgdat = NULL;
- int err;
-
- err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
- if (err < 0 || pgdat == NULL)
- break;
-
- nr_zones = BPF_CORE_READ(pgdat, nr_zones);
- for (int k = 0; k < MAX_ZONES; k++) {
- __u64 zone_addr;
-
- if (k >= nr_zones)
- break;
-
- zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off;
- lock_addr = zone_addr + lock_off;
-
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
- }
- }
-}
-
SEC("raw_tp/bpf_test_finish")
-int BPF_PROG(collect_lock_syms)
+int BPF_PROG(end_timestamp)
{
- __u64 lock_addr, lock_off;
- __u32 lock_flag;
-
- if (bpf_core_field_exists(struct rq___new, __lock))
- lock_off = offsetof(struct rq___new, __lock);
- else
- lock_off = offsetof(struct rq___old, lock);
-
- for (int i = 0; i < MAX_CPUS; i++) {
- struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
-
- if (rq == NULL)
- break;
-
- lock_addr = (__u64)(void *)rq + lock_off;
- lock_flag = LOCK_CLASS_RQLOCK;
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
-
- collect_zone_lock();
-
+ end_ts = bpf_ktime_get_ns();
return 0;
}
SEC("raw_tp/bpf_test_finish")
-int BPF_PROG(end_timestamp)
+int BPF_PROG(collect_lock_syms)
{
- end_ts = bpf_ktime_get_ns();
- return 0;
+ return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
}
-/*
- * bpf_iter__kmem_cache added recently so old kernels don't have it in the
- * vmlinux.h. But we cannot add it here since it will cause a compiler error
- * due to redefinition of the struct on later kernels.
- *
- * So it uses a CO-RE trick to access the member only if it has the type.
- * This will support both old and new kernels without compiler errors.
- */
-struct bpf_iter__kmem_cache___new {
- struct kmem_cache *s;
-} __attribute__((preserve_access_index));
-
SEC("iter/kmem_cache")
int slab_cache_iter(void *ctx)
{
- struct kmem_cache *s = NULL;
- struct slab_cache_data d;
- const char *nameptr;
-
- if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) {
- struct bpf_iter__kmem_cache___new *iter = ctx;
-
- s = iter->s;
- }
-
- if (s == NULL)
- return 0;
-
- nameptr = s->name;
- bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr);
-
- d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT;
- if (d.id >= LCB_F_SLAB_ID_END)
- return 0;
-
- bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST);
- return 0;
+ return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
}
char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index 652e114e6b87..43043d62ea9f 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -45,6 +45,14 @@ struct contention_task_data {
#define LCB_F_SLAB_ID_END (1U << 26)
#define LCB_F_SLAB_ID_MASK 0x03FF0000U
+/* lock contention flags from include/trace/events/lock.h */
+#define LCB_F_SPIN (1U << 0)
+#define LCB_F_READ (1U << 1)
+#define LCB_F_WRITE (1U << 2)
+#define LCB_F_RT (1U << 3)
+#define LCB_F_PERCPU (1U << 4)
+#define LCB_F_MUTEX (1U << 5)
+
#define LCB_F_TYPE_MAX (1U << 7)
#define LCB_F_TYPE_MASK 0x0000007FU
@@ -76,4 +84,316 @@ struct slab_cache_data {
char name[SLAB_NAME_MAX];
};
+#ifdef __BPF__
+
+/* for collect_lock_syms(). 4096 was rejected by the verifier */
+#define MAX_CPUS 1024
+
+/* for collect_zone_lock(). It should be more than the actual zones. */
+#define MAX_ZONES 10
+
+extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak;
+extern struct rq runqueues __ksym __weak;
+
+#ifndef __has_builtin
+# define __has_builtin(x) 0
+#endif
+
+struct mutex___non_rt {
+ atomic_long_t owner;
+} __attribute__((preserve_access_index));
+
+struct rw_semaphore___old {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
+struct rw_semaphore___new {
+ atomic_long_t count;
+ atomic_long_t owner;
+} __attribute__((preserve_access_index));
+
+struct mm_struct___old {
+ struct rw_semaphore mmap_sem;
+} __attribute__((preserve_access_index));
+
+struct mm_struct___new {
+ struct rw_semaphore mmap_lock;
+} __attribute__((preserve_access_index));
+
+struct rq___old {
+ raw_spinlock_t lock;
+} __attribute__((preserve_access_index));
+
+struct rq___new {
+ raw_spinlock_t __lock;
+} __attribute__((preserve_access_index));
+
+/*
+ * bpf_iter__kmem_cache added recently so old kernels don't have it in the
+ * vmlinux.h. But we cannot add it here since it will cause a compiler error
+ * due to redefinition of the struct on later kernels.
+ *
+ * So it uses a CO-RE trick to access the member only if it has the type.
+ * This will support both old and new kernels without compiler errors.
+ */
+struct bpf_iter__kmem_cache___new {
+ struct kmem_cache *s;
+} __attribute__((preserve_access_index));
+
+static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
+{
+ struct task_struct *task;
+ __u64 owner = 0;
+
+ if (flags & LCB_F_MUTEX) {
+ struct mutex___non_rt *mutex = (void *)lock;
+
+ if (bpf_core_field_exists(mutex->owner.counter)) {
+ owner = BPF_CORE_READ(mutex, owner.counter);
+ if (owner & 0x4UL)
+ return NULL;
+ } else if (bpf_core_type_exists(struct rt_mutex_base)) {
+ struct rt_mutex_base *rtm = (void *)lock;
+
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ }
+ } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
+ /*
+ * Support for the BPF_TYPE_MATCHES argument to the
+ * __builtin_preserve_type_info builtin was added at some point during
+ * development of clang 15 and it's what is needed for
+ * bpf_core_type_matches.
+ */
+#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
+ if (bpf_core_type_matches(struct rw_semaphore___old)) {
+ struct rw_semaphore___old *old = (void *)lock;
+
+ owner = (unsigned long)BPF_CORE_READ(old, owner);
+ } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
+ struct rw_semaphore___new *new = (void *)lock;
+
+ owner = BPF_CORE_READ(new, owner.counter);
+ }
+#else
+ /* assume new struct */
+ struct rw_semaphore___new *new = (void *)lock;
+
+ if (bpf_core_field_exists(new->owner.counter))
+ owner = BPF_CORE_READ(new, owner.counter);
+#endif
+ } else if (flags == LCB_F_RT) {
+ struct rt_mutex_base *rtm = (void *)lock;
+
+ if (bpf_core_type_exists(struct rt_mutex_base)) {
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ }
+ }
+
+ if (!owner)
+ return NULL;
+
+ task = (void *)(owner & ~7UL);
+ return task;
+}
+
+static __always_inline __u64 get_current_mmap_lock(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+ struct mm_struct___new *mm_new = NULL;
+ struct mm_struct___old *mm_old = NULL;
+ __u64 mm;
+
+ if (!curr)
+ return 0;
+
+ mm = (__u64)BPF_CORE_READ(task, mm);
+ if (!mm)
+ return 0;
+
+ if (bpf_core_field_exists(mm_new->mmap_lock))
+ return mm + bpf_core_field_offset(mm_new->mmap_lock);
+
+ if (bpf_core_field_exists(mm_old->mmap_sem))
+ return mm + bpf_core_field_offset(mm_old->mmap_sem);
+
+ return 0;
+}
+
+static __always_inline __u64 get_current_siglock(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+ struct sighand_struct *sighand = NULL;
+ __u64 sh;
+
+ if (!curr)
+ return 0;
+
+ sh = (__u64)BPF_CORE_READ(task, sighand);
+ if (!sh)
+ return 0;
+
+ return sh + bpf_core_field_offset(sighand->siglock);
+}
+
+/*
+ * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar.
+ *
+ * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the
+ * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed
+ * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such
+ * as `bpf_get_kmem_cache(u64 addr)`.
+ */
+static __always_inline __u64 get_lock_addr(void *ctx)
+{
+ __u64 lock_addr = 0;
+
+ bpf_get_func_arg(ctx, 0, &lock_addr);
+ return lock_addr;
+}
+
+static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags)
+{
+ __u64 curr;
+
+ switch (flags) {
+ case LCB_F_READ: /* rwsem */
+ case LCB_F_WRITE:
+ curr = bpf_get_current_task();
+ if (get_current_mmap_lock(curr) == lock)
+ return LCD_F_MMAP_LOCK;
+ break;
+ case LCB_F_SPIN: /* spinlock */
+ curr = bpf_get_current_task();
+ if (get_current_siglock(curr) == lock)
+ return LCD_F_SIGHAND_LOCK;
+ break;
+ default:
+ break;
+ }
+ return 0;
+}
+
+static __always_inline void do_collect_zone_lock(void *lock_syms_map,
+ __u64 contig_page_data_addr,
+ __u64 node_data_addr,
+ int nr_nodes,
+ int sizeof_zone)
+{
+ __u64 nr_zones, zone_off;
+ __u64 lock_addr, lock_off;
+ __u32 lock_flag = LOCK_CLASS_ZONE_LOCK;
+
+ zone_off = offsetof(struct pglist_data, node_zones);
+ lock_off = offsetof(struct zone, lock);
+
+ if (contig_page_data_addr) {
+ struct pglist_data *contig_page_data;
+
+ contig_page_data = (void *)(long)contig_page_data_addr;
+ nr_zones = BPF_CORE_READ(contig_page_data, nr_zones);
+
+ for (int i = 0; i < MAX_ZONES; i++) {
+ __u64 zone_addr;
+
+ if (i >= nr_zones)
+ break;
+
+ zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off;
+ lock_addr = zone_addr + lock_off;
+
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ } else if (node_data_addr && nr_nodes > 0) {
+ struct pglist_data **node_data = (void *)(long)node_data_addr;
+
+ for (int i = 0; i < nr_nodes; i++) {
+ struct pglist_data *pgdat = NULL;
+ int err;
+
+ err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
+ if (err < 0)
+ break;
+ if (pgdat == NULL)
+ continue;
+
+ nr_zones = BPF_CORE_READ(pgdat, nr_zones);
+ for (int k = 0; k < MAX_ZONES; k++) {
+ __u64 zone_addr;
+
+ if (k >= nr_zones)
+ break;
+
+ zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off;
+ lock_addr = zone_addr + lock_off;
+
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ }
+ }
+}
+
+static __always_inline int do_collect_lock_syms(void *lock_syms_map,
+ __u64 contig_page_data_addr,
+ __u64 node_data_addr,
+ int nr_nodes,
+ int sizeof_zone)
+{
+ __u64 lock_addr, lock_off;
+ __u32 lock_flag;
+
+ if (bpf_core_field_exists(struct rq___new, __lock))
+ lock_off = offsetof(struct rq___new, __lock);
+ else
+ lock_off = offsetof(struct rq___old, lock);
+
+ if (&runqueues) {
+ for (int i = 0; i < MAX_CPUS; i++) {
+ struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
+
+ if (rq == NULL)
+ break;
+
+ lock_addr = (__u64)(void *)rq + lock_off;
+ lock_flag = LOCK_CLASS_RQLOCK;
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ }
+
+ do_collect_zone_lock(lock_syms_map, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
+
+ return 0;
+}
+
+static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map,
+ __u32 *slab_cache_id)
+{
+ struct kmem_cache *s = NULL;
+ struct slab_cache_data d = {};
+ const char *nameptr;
+
+ if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) {
+ struct bpf_iter__kmem_cache___new *iter = ctx;
+
+ s = iter->s;
+ }
+
+ if (s == NULL)
+ return 0;
+
+ nameptr = s->name;
+ bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr);
+
+ d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT;
+ if (d.id >= LCB_F_SLAB_ID_END)
+ return 0;
+
+ bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST);
+ return 0;
+}
+
+#endif /* __BPF__ */
+
#endif /* UTIL_BPF_SKEL_LOCK_DATA_H */
diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
index 94e42b022ab1..b75ddc6846bb 100644
--- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
+++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
@@ -76,6 +76,7 @@ struct signal_struct {
} __attribute__((preserve_access_index));
struct rw_semaphore {
+ atomic_long_t count;
atomic_long_t owner;
} __attribute__((preserve_access_index));
@@ -83,6 +84,10 @@ struct mutex {
atomic_long_t owner;
} __attribute__((preserve_access_index));
+struct rt_mutex_base {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
struct kernfs_node {
u64 id;
} __attribute__((preserve_access_index));
diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c
index 119a7206f3cd..a1bb78d6e29c 100644
--- a/tools/perf/util/lock-contention.c
+++ b/tools/perf/util/lock-contention.c
@@ -22,6 +22,24 @@ struct callstack_filter {
static LIST_HEAD(callstack_filters);
struct hlist_head *lockhash_table;
+const struct perf_lock_type lock_type_table[] = {
+ { 0, "semaphore", "semaphore" },
+ { LCB_F_SPIN, "spinlock", "spinlock" },
+ { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
+ { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
+ { LCB_F_READ, "rwsem:R", "rwsem" },
+ { LCB_F_WRITE, "rwsem:W", "rwsem" },
+ { LCB_F_RT, "rt-mutex", "rt-mutex" },
+ { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
+ { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
+ { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
+ { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
+ { LCB_F_MUTEX, "mutex", "mutex" },
+ { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
+ /* alias for optimistic spinning only */
+ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
+};
+
int parse_call_stack(const struct option *opt __maybe_unused, const char *str,
int unset __maybe_unused)
{
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index 59c94190b092..68e9bd77e8a9 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -129,6 +129,19 @@ struct thread_stat {
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+struct perf_lock_type {
+ unsigned int flags;
+ /*
+ * Name of the lock flags (access), with delimiter ':'.
+ * For example, rwsem:R or rwsem:W.
+ */
+ const char *flags_name;
+ /* Name of the lock (type), for example, rwlock or rwsem. */
+ const char *lock_name;
+};
+
+extern const struct perf_lock_type lock_type_table[14];
+
struct evlist;
struct machine;
struct target;
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v4 4/7] perf bpf-filter: Add lock_held() sample filter
2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers
` (2 preceding siblings ...)
2026-10-19 22:09 ` [PATCH v4 3/7] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
@ 2026-10-19 22:09 ` Ian Rogers
2026-10-19 22:28 ` sashiko-bot
2026-10-19 22:09 ` [PATCH v4 5/7] perf test: Add tests for lock_held() BPF " Ian Rogers
` (3 subsequent siblings)
7 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, eranian, ddhariwal, rcu
Add BPF sample filter predicates lock_held(<lock>),
lock_held_read(<lock>) / lock_held_shared(<lock>), and
lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R'
and ':W' suffixes on <lock>) to allow recording perf event samples only
when a specified kernel lock is held (or not held with '!').
Supported <lock> specifications match perf lock contention:
- Global/static kernel lock symbols (e.g. lock_held(cgroup_mutex))
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with '&' (e.g. lock_held(&dentry),
lock_held_read(&dentry))
- Per-process and semi-global locks: mmap_lock, siglock, rq_lock,
zone_lock
- Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W,
rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W,
semaphore
Supported lock hold modes by lock family:
- Both read/shared and write/exclusive: rwsem (including mmap_lock)
and rwlock.
- Write/exclusive only: mutex, spinlock (including siglock, rq_lock,
zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write()
/ percpu_up_write(); fast-path percpu_down_read() / percpu_up_read()
are inlined per-CPU counter operations without function entry points).
When evaluating the filter on sample overflow, perf_sample_filter first
inspects the lock's owner field directly in kernel memory where
possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls
back to a held_locks BPF LRU hash map (tracking separate read_count and
write_count) populated by fentry/fexit hooks on kernel lock/unlock
functions. When filtering on a specific mutex symbol (*_mutex),
rt-mutex symbol (*_rt_mutex), or write-locked mmap_lock (mmap_lock:W /
lock_held_write(mmap_lock)), no fentry/fexit hooks are attached at all.
Note that counting semaphores (down*() / up()) do not enforce same-task
ownership in the kernel: when a semaphore is used as a cross-task
completion/signaling mechanism (down() in one task woken by up() in
another task or interrupt handler, which proposed kernel trackers like
CONFIG_DEPT model as wait/event dependencies bounded by execution
context rather than lock ownership), fexit/down and fentry/up will not
pair on the same task ID. Semaphore hooks are therefore only enabled
when 'semaphore' or a '*_sem' symbol (such as console_sem) is
explicitly specified, and held_locks uses BPF_MAP_TYPE_LRU_HASH so any
unpaired cross-task entries are automatically evicted.
Example usage:
# perf record -a -g -e cycles --filter 'lock_held(mmap_lock)' \
-- perf bench mem mmap -t 2 -l 100
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ]
# perf report --stdio --no-children
# Overhead Command Shared Object Symbol
# ........ ........ ................. ........................
#
18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms
|
---clear_page_erms
get_page_from_freelist
__alloc_pages_noprof
folio_alloc_mpol_noprof
vma_alloc_folio_noprof
do_anonymous_page
__handle_mm_fault
handle_mm_fault
do_user_addr_fault
exc_page_fault
asm_exc_page_fault
worker_thread
Read/shared vs write/exclusive holding can also be filtered separately:
# perf record -a -g -e cycles --filter 'lock_held_read(mmap_lock)' ...
# perf record -a -g -e cycles --filter 'lock_held_write(mmap_lock)' ...
# perf record -a -g -e cycles --filter 'lock_held(mmap_lock:W)' ...
Signed-off-by: Ian Rogers <irogers@google.com>
# Conflicts:
# tools/perf/util/bpf-filter.c
# tools/perf/util/bpf_skel/sample_filter.bpf.c
---
tools/perf/Documentation/perf-record.txt | 58 +-
tools/perf/util/bpf-filter.c | 1004 ++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 46 +-
tools/perf/util/bpf_skel/sample-filter.h | 65 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 1015 ++++++++++++++++++
tools/perf/util/parse-events.c | 3 +-
8 files changed, 2209 insertions(+), 26 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 2351f7b4f5fc..8496b20987a0 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -204,7 +204,63 @@ OPTIONS
ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr,
code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat,
p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock,
- mem_dtlb, mem_blk, mem_hops, uid, gid
+ mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>),
+ lock_held_read(<lock>), lock_held_shared(<lock>),
+ lock_held_write(<lock>), lock_held_exclusive(<lock>)
+
+ The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters
+ samples based on whether the specified kernel lock is currently held
+ by the kernel thread or CPU. To differentiate between shared/read
+ holding and exclusive/write holding, use 'lock_held_read(<lock>)'
+ (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)'
+ (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to
+ <lock> (e.g. 'lock_held(mmap_lock:R)' or 'lock_held(mmap_lock:W)').
+ <lock> can be a kernel lock symbol name (e.g.
+ 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache
+ name prefixed with '&' (e.g. 'lock_held(&dentry)',
+ 'lock_held_read(&dentry)'), a well-known lock name ('mmap_lock',
+ 'siglock', 'rq_lock', 'zone_lock'), or a lock type ('mutex',
+ 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R',
+ 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore').
+ Both read/shared and write/exclusive tracking are supported for
+ 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', 'spinlock'
+ (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex',
+ 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support
+ write/exclusive tracking only. Note that 'semaphore' tracking
+ ('down*()' / 'up()') assumes binary mutex-like usage where the
+ acquiring task also releases the semaphore (such as 'console_sem'),
+ rather than cross-task completion signaling.
+
+ For example, 'lock_held()' can be combined with memory allocation or
+ reclaim events to detect allocations or direct memory reclaim occurring
+ while a kernel lock is held (which can stall lock waiters for extended
+ periods):
+
+ # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+ # perf record -a -g -e kmem:mm_page_alloc \
+ --filter 'gfp_flags & 0x400' \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+ # perf record -a -g -e major-faults \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+
+ When filtering on a specific mutex symbol ('*_mutex'), rt-mutex symbol
+ ('*_rt_mutex'), or write-locked 'mmap_lock' ('lock_held_write(mmap_lock)'
+ or 'lock_held(mmap_lock:W)'), 'perf_sample_filter' inspects the lock's
+ 'owner' field directly in kernel memory on each sample without attaching
+ any BPF fentry/fexit hooks. For locks that do not maintain an
+ authoritative per-task owner pointer (such as spinlocks, rwlocks,
+ semaphores, reader-locked rwsems, and slab/type filters), it attaches
+ BPF fentry/fexit programs to kernel lock acquire and release functions
+ to track held locks in a BPF map. These BPF hooks run on every kernel
+ lock acquisition and release for the attached lock family, even when no
+ perf sample is being taken. To minimize this overhead, only the hooks
+ for the relevant lock family and mode are attached when known (for
+ example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions
+ and 'rq_lock' only hooks spinlock functions), whereas slab filters
+ ('&<slab>'), raw hex addresses, or symbols without a recognizable
+ lock-type suffix attach hooks across all lock families (including
+ high-frequency spinlocks).
The <operator> can be one of:
==, !=, >, >=, <, <=, &
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 4e99af4764f1..ab9b58f6cb71 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -40,21 +40,35 @@
* The 'dropped' map is to keep how many samples it dropped by the filter and
* it will be reported as lost samples.
*/
+#include <errno.h>
+#include <stdio.h>
#include <stdlib.h>
+#include <stdbool.h>
+#include <stddef.h>
+#include <string.h>
#include <fcntl.h>
+#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/stat.h>
#include <bpf/bpf.h>
+#include <bpf/btf.h>
#include <linux/err.h>
+#include <linux/kernel.h>
#include <linux/list.h>
+#include <linux/string.h>
+#include <linux/zalloc.h>
#include <api/fs/fs.h>
#include <internal/xyarray.h>
#include <perf/threadmap.h>
+#include <symbol/kallsyms.h>
#include "util/cap.h"
#include "util/debug.h"
#include "util/evsel.h"
+#include "util/lock-contention.h"
+#include "util/string2.h"
+#include "util/symbol_conf.h"
#include "util/target.h"
#include "util/thread_map.h"
#include "util/bpf-utils.h"
@@ -63,6 +77,7 @@
#include <util/bpf-filter-flex.h>
#include <util/bpf-filter-bison.h>
+#include "bpf_skel/lock_data.h"
#include "bpf_skel/sample-filter.h"
#include "bpf_skel/sample_filter.skel.h"
@@ -123,7 +138,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
{
const struct perf_sample_info *info;
- if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
+ if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID ||
+ expr->term == PBF_TERM_LOCK_HELD) {
/* Not dependent on the sample_type as computed from a BPF helper. */
return 0;
}
@@ -166,9 +182,844 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
return -1;
}
-static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry)
+/*
+ * Program groups for `lock_held(...)` BPF tracing hooks.
+ *
+ * When a `lock_held(...)` filter is specified, only the BPF tracing program
+ * groups required for the requested lock(s) are autoloaded and attached.
+ * If no `lock_held(...)` filter is present, none of these programs are loaded.
+ */
+enum lock_prog_group {
+ LOCK_PROG_MUTEX = 1U << 0,
+ LOCK_PROG_RWSEM_READ = 1U << 1,
+ LOCK_PROG_RWSEM_WRITE = 1U << 2,
+ LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE,
+ LOCK_PROG_SPIN = 1U << 3,
+ LOCK_PROG_RWLOCK_READ = 1U << 4,
+ LOCK_PROG_RWLOCK_WRITE = 1U << 5,
+ LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE,
+ LOCK_PROG_RTMUTEX = 1U << 6,
+ LOCK_PROG_SEM = 1U << 7,
+ LOCK_PROG_PERCPU_RWSEM = 1U << 8,
+ LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM,
+};
+
+static unsigned int lock_flags_to_prog_group(unsigned int flags)
+{
+ if (flags & LCB_F_MUTEX)
+ return LOCK_PROG_MUTEX;
+ if (flags & LCB_F_RT)
+ return LOCK_PROG_RTMUTEX;
+ if (flags & LCB_F_PERCPU)
+ return LOCK_PROG_PERCPU_RWSEM;
+ if (flags == LCB_F_SPIN)
+ return LOCK_PROG_SPIN;
+ if (flags & LCB_F_SPIN)
+ return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE;
+ if (flags & LCB_F_READ)
+ return LOCK_PROG_RWSEM_READ;
+ if (flags & LCB_F_WRITE)
+ return LOCK_PROG_RWSEM_WRITE;
+ return LOCK_PROG_SEM;
+}
+
+struct lock_prog_desc {
+ size_t prog_offset;
+ const char *btf_func;
+ unsigned int group;
+};
+
+#define LOCK_FPROG(field, func, grp) \
+ { offsetof(struct sample_filter_bpf, progs.field), func, grp }
+
+static const struct lock_prog_desc lock_progs[] = {
+ /* Mutex */
+ LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
+ LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
+ /* RW semaphore */
+ LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested",
+ LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested",
+ LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM),
+ LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE),
+ /* Spinlock */
+ LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN),
+ /* RWlock */
+ LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE),
+ /* RT-Mutex */
+ LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible",
+ LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX),
+ /* Semaphore */
+ LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM),
+ LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM),
+ /* Percpu-RWsem */
+ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
+ LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+};
+
+#undef LOCK_FPROG
+
+static int split_lock_name_mode(const char *name, int expr_part,
+ char *base_buf, size_t buf_sz,
+ bool *has_amp, unsigned int *mode_out)
+{
+ const char *bare;
+ size_t len;
+ unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK;
+ unsigned int suffix_mode = 0;
+
+ if (!name || !*name)
+ return -EINVAL;
+
+ *has_amp = (name[0] == '&');
+ bare = *has_amp ? name + 1 : name;
+ len = strlen(bare);
+ if (len == 0)
+ return -EINVAL;
+
+ if (len > 2 && !strcmp(bare + len - 2, ":R")) {
+ suffix_mode = PBF_LOCK_MODE_READ;
+ len -= 2;
+ } else if (len > 2 && !strcmp(bare + len - 2, ":W")) {
+ suffix_mode = PBF_LOCK_MODE_WRITE;
+ len -= 2;
+ }
+
+ if (len == 0 || len >= buf_sz)
+ return -EINVAL;
+
+ if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY &&
+ suffix_mode && expr_mode != suffix_mode)
+ return -EINVAL;
+
+ memcpy(base_buf, bare, len);
+ base_buf[len] = '\0';
+
+ if (suffix_mode)
+ *mode_out = suffix_mode;
+ else if (expr_mode)
+ *mode_out = expr_mode;
+ else
+ *mode_out = PBF_LOCK_MODE_ANY;
+
+ return 0;
+}
+
+/*
+ * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R",
+ * "spinlock", "percpu-rwsem").
+ *
+ * A lock type name may correspond to up to two flag combinations (e.g.
+ * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed
+ * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is
+ * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`.
+ */
+static bool parse_lock_type_name(const char *name, unsigned int mode,
+ struct sample_filter_bpf *skel,
+ u64 *encoded_pair, unsigned int *prog_groups)
+{
+ bool found = false;
+ u32 flag1 = 0xff, flag2 = 0xff;
+ unsigned int groups = 0;
+ const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name;
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) {
+ u32 f = lock_type_table[i].flags;
+ bool is_read = (f & LCB_F_READ) != 0;
+
+ if (strcmp(lock_type_table[i].lock_name, lookup) &&
+ strcmp(lock_type_table[i].flags_name, name))
+ continue;
+
+ /*
+ * percpu_down_read() / percpu_up_read() fast paths are inlined
+ * per-CPU counter increments without function entry points, so
+ * only writer holding (percpu_down_write / percpu_up_write) can
+ * be tracked for percpu-rwsem. rwlock-rt is also not hooked.
+ */
+ if ((f & LCB_F_PERCPU) && is_read)
+ continue;
+ if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE)))
+ continue;
+ if ((f & LCB_F_MUTEX) && (f & LCB_F_SPIN))
+ continue;
+
+ if (is_read && !(mode & PBF_LOCK_MODE_READ))
+ continue;
+ if (!is_read && !(mode & PBF_LOCK_MODE_WRITE))
+ continue;
+
+ if (!found)
+ flag1 = f;
+ else if (f != flag1)
+ flag2 = f;
+ if (skel && f < LCB_F_TYPE_MAX) {
+ skel->bss->has_lock_type = 1;
+ skel->bss->lock_type_filter[f] = 1;
+ }
+ groups |= lock_flags_to_prog_group(f);
+ found = true;
+ }
+
+ if (!found)
+ return false;
+
+ if (encoded_pair)
+ *encoded_pair = flag1 | ((u64)flag2 << 8);
+ if (prog_groups)
+ *prog_groups |= groups;
+ return true;
+}
+
+struct lock_sym_cb_args {
+ const char *name;
+ u64 addr;
+ bool found;
+};
+
+static int find_lock_sym_cb(void *arg, const char *name,
+ char type __maybe_unused, u64 start)
+{
+ struct lock_sym_cb_args *args = arg;
+
+ if (!strcmp(name, args->name)) {
+ args->found = true;
+ args->addr = start;
+ if (start != 0)
+ return 1;
+ }
+ return 0;
+}
+
+static int find_kernel_lock_sym(const char *name, u64 *addr)
+{
+ struct lock_sym_cb_args args = {
+ .name = name,
+ };
+ const char *kallsyms_filename = "/proc/kallsyms";
+
+ if (symbol_conf.kallsyms_name != NULL)
+ kallsyms_filename = symbol_conf.kallsyms_name;
+
+ if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0)
+ return -ENOENT;
+
+ if (!args.found)
+ return -ENOENT;
+
+ if (args.addr == 0) {
+ pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n",
+ name);
+ return -EACCES;
+ }
+
+ *addr = args.addr;
+ return 0;
+}
+
+static bool parse_lock_hex_addr(const char *str, u64 *addr)
+{
+ char *end;
+ unsigned long long val;
+
+ if (!str || !*str)
+ return false;
+
+ errno = 0;
+ val = strtoull(str, &end, 16);
+ if (*end != '\0' || errno != 0 || val == 0)
+ return false;
+
+ *addr = val;
+ return true;
+}
+
+static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf)
+{
+ u64 addr = 0;
+ char *buf = NULL, *p;
+ size_t len;
+ long last = -1;
+ int ret;
+
+ if (!btf)
+ return;
+
+ ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT);
+ if (ret < 0)
+ return;
+
+ ret = btf__resolve_size(btf, ret);
+ if (ret < 0)
+ return;
+
+ skel->rodata->sizeof_zone = ret;
+
+ if (find_kernel_lock_sym("contig_page_data", &addr) == 0) {
+ skel->rodata->contig_page_data_addr = addr;
+ return;
+ }
+
+ if (find_kernel_lock_sym("node_data", &addr) != 0)
+ return;
+
+ skel->rodata->node_data_addr = addr;
+
+ if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) {
+ p = buf;
+ while (p && *p) {
+ char *end;
+
+ last = strtol(p, &end, 0);
+ if (end == p)
+ break;
+ p = end;
+ if (*p == ',' || *p == '-' || *p == '\n')
+ p++;
+ }
+ skel->rodata->nr_nodes = last + 1;
+ free(buf);
+ }
+}
+
+static bool btf_has_func(struct btf *btf, const char *name)
+{
+ return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0;
+}
+
+static unsigned int supported_lock_prog_groups(struct btf *btf)
+{
+ unsigned int supported = 0;
+
+ if (!btf)
+ return 0;
+
+ if ((btf_has_func(btf, "mutex_lock") || btf_has_func(btf, "mutex_lock_nested")) &&
+ btf_has_func(btf, "mutex_unlock"))
+ supported |= LOCK_PROG_MUTEX;
+
+ if ((btf_has_func(btf, "down_read") || btf_has_func(btf, "down_read_nested")) &&
+ btf_has_func(btf, "up_read"))
+ supported |= LOCK_PROG_RWSEM_READ;
+
+ if ((btf_has_func(btf, "down_write") || btf_has_func(btf, "down_write_nested")) &&
+ btf_has_func(btf, "up_write"))
+ supported |= LOCK_PROG_RWSEM_WRITE;
+
+ /*
+ * Require both acquire and all unlock entry points in BTF so we never
+ * attach spinlock/rwlock acquire hooks when unlock functions are
+ * inlined (e.g. !CONFIG_UNINLINE_SPIN_UNLOCK or CONFIG_INLINE_*_UNLOCK).
+ */
+ if ((btf_has_func(btf, "_raw_spin_lock") || btf_has_func(btf, "_raw_spin_lock_nested")) &&
+ btf_has_func(btf, "_raw_spin_unlock") &&
+ btf_has_func(btf, "_raw_spin_unlock_irq") &&
+ btf_has_func(btf, "_raw_spin_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_spin_unlock_bh"))
+ supported |= LOCK_PROG_SPIN;
+
+ if (btf_has_func(btf, "_raw_read_lock") &&
+ btf_has_func(btf, "_raw_read_unlock") &&
+ btf_has_func(btf, "_raw_read_unlock_irq") &&
+ btf_has_func(btf, "_raw_read_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_read_unlock_bh"))
+ supported |= LOCK_PROG_RWLOCK_READ;
+
+ if ((btf_has_func(btf, "_raw_write_lock") || btf_has_func(btf, "_raw_write_lock_nested")) &&
+ btf_has_func(btf, "_raw_write_unlock") &&
+ btf_has_func(btf, "_raw_write_unlock_irq") &&
+ btf_has_func(btf, "_raw_write_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_write_unlock_bh"))
+ supported |= LOCK_PROG_RWLOCK_WRITE;
+
+ if ((btf_has_func(btf, "rt_mutex_lock") || btf_has_func(btf, "rt_mutex_lock_nested")) &&
+ btf_has_func(btf, "rt_mutex_unlock"))
+ supported |= LOCK_PROG_RTMUTEX;
+
+ if (btf_has_func(btf, "down") && btf_has_func(btf, "up"))
+ supported |= LOCK_PROG_SEM;
+
+ if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write"))
+ supported |= LOCK_PROG_PERCPU_RWSEM;
+
+ return supported;
+}
+
+/*
+ * Enable or disable autoload for each lock-tracking BPF program in the
+ * skeleton before `sample_filter_bpf__load()` is called.
+ *
+ * Each program is enabled only if its group is in `needed_groups` and its
+ * target kernel function or tracepoint exists in the running kernel's BTF.
+ */
+static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
+ unsigned int needed_groups,
+ unsigned int required_groups,
+ bool needs_lock_syms, bool needs_slab_iter,
+ bool is_tracepoint)
+{
+ unsigned int supported_groups = supported_lock_prog_groups(btf);
+ unsigned int enabled_progs = 0;
+
+ bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
+ bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
+
+ if ((required_groups & supported_groups) != required_groups ||
+ (needed_groups && !(needed_groups & supported_groups))) {
+ pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n");
+ return -ENOTSUP;
+ }
+
+ needed_groups &= supported_groups;
+ if (skel)
+ skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ);
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) {
+ struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset;
+ struct bpf_program *prog = *progp;
+ bool enable = false;
+
+ if ((needed_groups & lock_progs[i].group) && btf &&
+ lock_progs[i].btf_func) {
+ if (btf__find_by_name_kind(btf, lock_progs[i].btf_func,
+ BTF_KIND_FUNC) > 0)
+ enable = true;
+ }
+
+ if (enable)
+ enabled_progs++;
+ bpf_program__set_autoload(prog, enable);
+ }
+
+ bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms);
+ if (needs_slab_iter && btf &&
+ btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) {
+ bpf_program__set_autoload(skel->progs.slab_cache_iter, true);
+ } else {
+ bpf_program__set_autoload(skel->progs.slab_cache_iter, false);
+ }
+
+ if (needed_groups && !enabled_progs) {
+ pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
+ return -ENOTSUP;
+ }
+ return 0;
+}
+
+/*
+ * Pre-scan a single filter expression prior to loading the BPF skeleton in
+ * order to determine which BPF maps need to be sized and which lock tracing
+ * program groups need to be autoloaded.
+ */
+static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
+ bool *has_lock_filter,
+ unsigned int *prog_groups,
+ unsigned int *required_groups,
+ bool *needs_lock_syms,
+ bool *needs_slab_iter)
+{
+ char base[128];
+ bool has_amp;
+ unsigned int mode, groups = 0;
+ u64 dummy;
+
+ if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name)
+ return;
+
+ *has_lock_filter = true;
+ if (split_lock_name_mode(expr->lock_name, expr->part, base,
+ sizeof(base), &has_amp, &mode) < 0)
+ return;
+
+ /* Leading '&' is reserved for slab cache locks (e.g. '&dentry'). */
+ if (has_amp) {
+ *needs_slab_iter = true;
+ *prog_groups |= LOCK_PROG_ALL;
+ return;
+ }
+
+ if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
+ /*
+ * Write ownership of mm->mmap_lock is inspected directly via
+ * is_rwsem_writer_owner() without tracing hooks; only reader
+ * tracking requires LOCK_PROG_RWSEM_READ.
+ */
+ if (mode & PBF_LOCK_MODE_READ) {
+ *prog_groups |= LOCK_PROG_RWSEM_READ;
+ *required_groups |= LOCK_PROG_RWSEM_READ;
+ }
+ return;
+ }
+ if (!strcmp(base, "siglock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ return;
+ }
+ if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ *needs_lock_syms = true;
+ return;
+ }
+ if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) {
+ *prog_groups |= groups;
+ *required_groups |= groups;
+ return;
+ }
+
+ if (parse_lock_hex_addr(base, &dummy)) {
+ *prog_groups |= LOCK_PROG_ALL;
+ return;
+ }
+
+ if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0 ||
+ strtailcmp(base, "mutex") == 0) {
+ /* Inspected directly via lock->owner; no tracing hooks needed. */
+ return;
+ } else if (strtailcmp(base, "percpu_rwsem") == 0 ||
+ strtailcmp(base, "pcpu_rwsem") == 0) {
+ *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
+ *required_groups |= LOCK_PROG_PERCPU_RWSEM;
+ } else if (strtailcmp(base, "rwsem") == 0) {
+ if (mode & PBF_LOCK_MODE_READ) {
+ *prog_groups |= LOCK_PROG_RWSEM_READ;
+ *required_groups |= LOCK_PROG_RWSEM_READ;
+ }
+ if (mode & PBF_LOCK_MODE_WRITE) {
+ *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM;
+ *required_groups |= LOCK_PROG_RWSEM_WRITE;
+ }
+ } else if (strtailcmp(base, "spinlock") == 0) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ } else if (strtailcmp(base, "sem") == 0) {
+ *prog_groups |= LOCK_PROG_SEM;
+ *required_groups |= LOCK_PROG_SEM;
+ } else if (strtailcmp(base, "rwlock") == 0) {
+ if (mode & PBF_LOCK_MODE_READ)
+ groups |= LOCK_PROG_RWLOCK_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ groups |= LOCK_PROG_RWLOCK_WRITE;
+ *prog_groups |= groups;
+ *required_groups |= groups;
+ } else {
+ *prog_groups |= LOCK_PROG_ALL;
+ }
+}
+
+static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter,
+ unsigned int *prog_groups,
+ unsigned int *required_groups,
+ bool *needs_lock_syms,
+ bool *needs_slab_iter)
+{
+ struct perf_bpf_filter_expr *expr, *group;
+
+ *has_lock_filter = false;
+ *prog_groups = 0;
+ *required_groups = 0;
+ *needs_lock_syms = false;
+ *needs_slab_iter = false;
+
+ list_for_each_entry(expr, &evsel->bpf_filters, list) {
+ prescan_one_lock_expr(expr, has_lock_filter, prog_groups,
+ required_groups, needs_lock_syms,
+ needs_slab_iter);
+ if (expr->op == PBF_OP_GROUP_BEGIN) {
+ list_for_each_entry(group, &expr->groups, list) {
+ prescan_one_lock_expr(group, has_lock_filter,
+ prog_groups, required_groups,
+ needs_lock_syms, needs_slab_iter);
+ }
+ }
+ }
+}
+
+static void run_slab_cache_iter(struct sample_filter_bpf *skel)
+{
+ struct bpf_link *link;
+ int fd;
+ char buf[256];
+
+ if (!bpf_program__autoload(skel->progs.slab_cache_iter))
+ return;
+
+ link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL);
+ if (!link || IS_ERR(link)) {
+ pr_debug("cannot attach slab cache iter\n");
+ return;
+ }
+
+ fd = bpf_iter_create(bpf_link__fd(link));
+ if (fd >= 0) {
+ while (read(fd, buf, sizeof(buf)) > 0)
+ continue;
+ close(fd);
+ }
+
+ bpf_link__destroy(link);
+}
+
+static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
+ u64 *kmem_cache_addr)
+{
+ int cache_fd;
+ u64 key, *prev_key = NULL;
+
+ if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter))
+ return false;
+
+ cache_fd = bpf_map__fd(skel->maps.slab_caches);
+ while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) {
+ struct slab_cache_data data;
+
+ if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0)
+ break;
+
+ if (!strcmp(data.name, name)) {
+ *kmem_cache_addr = key;
+ return true;
+ }
+ prev_key = &key;
+ }
+ return false;
+}
+
+/*
+ * Resolve a `lock_held(<name>)` filter expression into `entry->part`
+ * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate
+ * the corresponding BPF filter maps and BSS control flags.
+ *
+ * Resolution order:
+ * 1. Slab cache name (when prefixed with '&', e.g. `&dentry`).
+ * 2. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
+ * `rq_lock`, `zone_lock`.
+ * 3. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`,
+ * `rwlock`, `percpu-rwsem`, `semaphore`, etc.
+ * 4. Hexadecimal lock address.
+ * 5. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`).
+ */
+static int resolve_lock_expr(struct sample_filter_bpf *skel,
+ struct perf_bpf_filter_expr *expr,
+ struct perf_bpf_filter_entry *entry)
{
- int i = 0;
+ const char *name = expr->lock_name;
+ char base[128];
+ bool has_amp;
+ unsigned int mode;
+ u64 addr = 0, encoded_pair = 0;
+ u8 one = 1;
+ int ret;
+
+ if (split_lock_name_mode(name, expr->part, base, sizeof(base),
+ &has_amp, &mode) < 0) {
+ pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: "");
+ return -EINVAL;
+ }
+
+ if (has_amp) {
+ if (find_slab_cache(skel, base, &addr)) {
+ int fd = bpf_map__fd(skel->maps.slab_filter);
+
+ entry->part = PBF_LOCK_SLAB | mode;
+ entry->target = addr;
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_slab = 1;
+ return 0;
+ }
+ pr_err("Error: Cannot resolve slab cache '%s'\n", name);
+ return -EINVAL;
+ }
+
+ if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
+ entry->part = PBF_LOCK_MMAP | mode;
+ entry->target = 0;
+ if (skel && (mode & PBF_LOCK_MODE_READ))
+ skel->bss->has_mmap_lock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "siglock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_SIGHAND | mode;
+ entry->target = 0;
+ if (skel)
+ skel->bss->has_siglock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "rq_lock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_CLASS | mode;
+ entry->target = LOCK_CLASS_RQLOCK;
+ if (skel)
+ skel->bss->has_lock_class = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "zone_lock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_CLASS | mode;
+ entry->target = LOCK_CLASS_ZONE_LOCK;
+ if (skel)
+ skel->bss->has_lock_class = 1;
+ return 0;
+ }
+
+ if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
+ entry->part = PBF_LOCK_TYPE | mode;
+ entry->target = encoded_pair;
+ return 0;
+ }
+
+ if (parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) {
+ pr_err("Error: Lock type '%s' does not support the requested read/write mode\n",
+ name);
+ return -EINVAL;
+ }
+
+ if (parse_lock_hex_addr(base, &addr)) {
+ entry->part = PBF_LOCK_ADDR | mode;
+ entry->target = addr;
+ if (skel) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
+ }
+ return 0;
+ }
+
+ ret = find_kernel_lock_sym(base, &addr);
+ if (ret == 0) {
+ unsigned int direct_flag = 0;
+
+ if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0)
+ direct_flag = PBF_LOCK_DIRECT_RTMUTEX;
+ else if (strtailcmp(base, "mutex") == 0)
+ direct_flag = PBF_LOCK_DIRECT_MUTEX;
+ else if (strtailcmp(base, "rwsem") == 0 &&
+ strtailcmp(base, "percpu_rwsem") != 0 &&
+ strtailcmp(base, "pcpu_rwsem") != 0)
+ direct_flag = PBF_LOCK_DIRECT_RWSEM;
+
+ if ((direct_flag == PBF_LOCK_DIRECT_MUTEX ||
+ direct_flag == PBF_LOCK_DIRECT_RTMUTEX ||
+ strtailcmp(base, "percpu_rwsem") == 0 ||
+ strtailcmp(base, "pcpu_rwsem") == 0 ||
+ strtailcmp(base, "spinlock") == 0 ||
+ (strtailcmp(base, "sem") == 0 &&
+ strtailcmp(base, "rwsem") != 0)) &&
+ !(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+
+ entry->part = PBF_LOCK_ADDR | mode | direct_flag;
+ entry->target = addr;
+ if (skel && direct_flag != PBF_LOCK_DIRECT_MUTEX &&
+ direct_flag != PBF_LOCK_DIRECT_RTMUTEX) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
+ }
+ return 0;
+ }
+ if (ret == -EACCES)
+ return ret;
+
+ pr_err("Error: Cannot resolve kernel lock '%s'\n", name);
+ return -EINVAL;
+}
+
+static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct perf_bpf_filter_entry *entry)
+{
+ int i = 0, ret;
struct perf_bpf_filter_expr *expr;
list_for_each_entry(expr, &evsel->bpf_filters, list) {
@@ -182,6 +1033,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].part = expr->part;
entry[i].term = expr->term;
entry[i].value = expr->val;
+ entry[i].target = 0;
+ if (expr->term == PBF_TERM_LOCK_HELD) {
+ ret = resolve_lock_expr(skel, expr, &entry[i]);
+ if (ret < 0)
+ return ret;
+ }
i++;
if (expr->op == PBF_OP_GROUP_BEGIN) {
@@ -195,6 +1052,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].part = group->part;
entry[i].term = group->term;
entry[i].value = group->val;
+ entry[i].target = 0;
+ if (group->term == PBF_TERM_LOCK_HELD) {
+ ret = resolve_lock_expr(skel, group, &entry[i]);
+ if (ret < 0)
+ return ret;
+ }
i++;
}
@@ -434,11 +1297,14 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
{
int i, x, y, fd, ret;
struct sample_filter_bpf *skel = NULL;
+ struct btf *btf = NULL;
struct bpf_program *prog;
struct bpf_link *link;
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
+ bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false;
+ unsigned int prog_groups = 0, required_groups = 0;
#if LIBBPF_CURRENT_VERSION_GEQ(1, 7)
DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts,
.dont_enable = true);
@@ -450,21 +1316,25 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
if (entry == NULL)
return -1;
- ret = get_filter_entries(evsel, entry);
- if (ret < 0) {
- pr_err("Failed to process filter entries\n");
- goto err;
- }
+ prescan_lock_filters(evsel, &has_lock_filter, &prog_groups,
+ &required_groups, &needs_lock_syms,
+ &needs_slab_iter);
if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
int zero = 0;
- if (is_tracepoint) {
- pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n");
+ if (has_lock_filter || is_tracepoint) {
+ pr_err("Error: BPF filter requires CAP_BPF capability\n");
ret = -EPERM;
goto err;
}
+ ret = get_filter_entries(evsel, NULL, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
+
/* The filters map is shared among other processes */
ret = create_idx_hash(evsel, entry);
if (ret < 0)
@@ -509,8 +1379,26 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
- bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
+ if (has_lock_filter) {
+ btf = btf__load_vmlinux_btf();
+ bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES);
+ bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS);
+ if (needs_slab_iter) {
+ bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS);
+ bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES);
+ }
+ if (needs_lock_syms) {
+ bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
+ init_numa_data(skel, btf);
+ }
+ ret = configure_lock_progs(skel, btf, prog_groups, required_groups,
+ needs_lock_syms, needs_slab_iter,
+ is_tracepoint);
+ if (ret < 0)
+ goto err;
+ } else {
+ configure_lock_progs(skel, NULL, 0, 0, false, false, is_tracepoint);
+ }
if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
@@ -518,6 +1406,37 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
+ if (has_lock_filter && needs_slab_iter)
+ run_slab_cache_iter(skel);
+
+ ret = get_filter_entries(evsel, skel, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
+
+ if (has_lock_filter) {
+ if (needs_lock_syms && skel->bss->has_lock_class) {
+ DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts,
+ .flags = BPF_F_TEST_RUN_ON_CPU,
+ );
+ int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms);
+
+ bpf_prog_test_run_opts(prog_fd, &opts);
+ }
+ bpf_program__set_autoattach(skel->progs.perf_sample_filter, false);
+ bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false);
+ bpf_program__set_autoattach(skel->progs.collect_lock_syms, false);
+ bpf_program__set_autoattach(skel->progs.slab_cache_iter, false);
+
+ ret = sample_filter_bpf__attach(skel);
+ if (ret < 0) {
+ pr_err("Failed to attach lock sample-filter BPF programs\n");
+ goto err;
+ }
+ skel->bss->lock_enabled = 1;
+ }
+
i = 0;
fd = bpf_map__fd(skel->maps.filters);
@@ -560,11 +1479,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
}
}
}
+ btf__free(btf);
free(entry);
evsel->bpf_skel = skel;
return 0;
err:
+ btf__free(btf);
free(entry);
if (!list_empty(&pinned_filters)) {
struct pinned_filter_idx *pfi, *tmp;
@@ -579,6 +1500,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return ret;
}
+void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr)
+{
+ struct perf_bpf_filter_expr *group, *tmp;
+
+ if (!expr)
+ return;
+
+ list_for_each_entry_safe(group, tmp, &expr->groups, list) {
+ list_del(&group->list);
+ perf_bpf_filter_expr__delete(group);
+ }
+ zfree(&expr->lock_name);
+ free(expr);
+}
+
int perf_bpf_filter__destroy(struct evsel *evsel)
{
struct perf_bpf_filter_expr *expr, *tmp;
@@ -586,7 +1522,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel)
list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) {
list_del(&expr->list);
- free(expr);
+ perf_bpf_filter_expr__delete(expr);
}
sample_filter_bpf__destroy(evsel->bpf_skel);
@@ -638,7 +1574,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
{
struct perf_bpf_filter_expr *expr;
- expr = malloc(sizeof(*expr));
+ expr = zalloc(sizeof(*expr));
if (expr != NULL) {
expr->term = term;
expr->part = part;
@@ -649,6 +1585,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
return expr;
}
+struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
+ int mode,
+ enum perf_bpf_filter_op op,
+ unsigned long val)
+{
+ struct perf_bpf_filter_expr *expr;
+
+ if (!lock_name || !*lock_name)
+ return NULL;
+
+ expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val);
+ if (!expr)
+ return NULL;
+
+ expr->lock_name = strdup(lock_name);
+ if (!expr->lock_name) {
+ free(expr);
+ return NULL;
+ }
+ return expr;
+}
+
static bool check_bpf_filter_capable(void)
{
int fd;
@@ -669,14 +1627,11 @@ static bool check_bpf_filter_capable(void)
return false;
}
-int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
+int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
{
YY_BUFFER_STATE buffer;
int ret;
- if (!check_bpf_filter_capable())
- return -EPERM;
-
buffer = perf_bpf_filter__scan_string(str);
ret = perf_bpf_filter_parse(expr_head);
@@ -688,6 +1643,14 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
return ret;
}
+int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
+{
+ if (!check_bpf_filter_capable())
+ return -EPERM;
+
+ return __perf_bpf_filter__parse(expr_head, str);
+}
+
static int pin_filter_object(struct sample_filter_bpf *skel, const char *path)
{
char *prog_path = NULL;
@@ -740,8 +1703,9 @@ int perf_bpf_filter__pin(void)
goto err;
}
+ configure_lock_progs(skel, NULL, 0, 0, false, false, false);
+
/* pinned program will use pid-hash */
- bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH);
bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH);
@@ -836,7 +1800,7 @@ int perf_bpf_filter__unpin(void)
goto err;
}
- bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
+ configure_lock_progs(skel, NULL, 0, 0, false, false, false);
if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
pr_err("Failed to load perf sample-filter BPF skeleton\n");
diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h
index 818c554b91b2..fda95ff226b3 100644
--- a/tools/perf/util/bpf-filter.h
+++ b/tools/perf/util/bpf-filter.h
@@ -14,6 +14,13 @@ struct perf_bpf_filter_expr {
int part;
enum perf_bpf_filter_term term;
unsigned long val;
+ /*
+ * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions
+ * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R").
+ * Resolved into `entry.part` and `entry.target` during
+ * perf_bpf_filter__prepare().
+ */
+ char *lock_name;
};
struct evsel;
@@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
int part,
enum perf_bpf_filter_op op,
unsigned long val);
+struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
+ int mode,
+ enum perf_bpf_filter_op op,
+ unsigned long val);
+void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr);
+int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
int perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target);
int perf_bpf_filter__destroy(struct evsel *evsel);
diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l
index 1be9df6550fc..b4022110f1e5 100644
--- a/tools/perf/util/bpf-filter.l
+++ b/tools/perf/util/bpf-filter.l
@@ -1,9 +1,12 @@
%option prefix="perf_bpf_filter_"
%option noyywrap
+%x LOCK_ARG
+
%{
#include <stdio.h>
#include <stdlib.h>
+#include <string.h>
#include <linux/perf_event.h>
#include "bpf-filter.h"
@@ -76,7 +79,7 @@ static int path_or_error(void)
num_dec [0-9]+
num_hex 0[Xx][0-9a-fA-F]+
space [ \t]+
-path [^ \t\n,]+
+path [^ \t\n,()!=>&|<]+
ident [_a-zA-Z][_a-zA-Z0-9]+
%%
@@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); }
uid { return sample(PBF_TERM_UID); }
gid { return sample(PBF_TERM_GID); }
cgroup { return sample_path(PBF_TERM_CGROUP); }
-
+lock_held[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY;
+ return BFT_LOCK_HELD;
+ }
+(lock_held_read|lock_held_shared)[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ;
+ return BFT_LOCK_HELD;
+ }
+(lock_held_write|lock_held_exclusive)[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE;
+ return BFT_LOCK_HELD;
+ }
+
+<LOCK_ARG>{space} { }
+<LOCK_ARG>[^ \t\n()]+ {
+ perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text);
+ return BFT_LOCK_NAME;
+ }
+<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; }
+<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; }
+
+"!" { return '!'; }
"==" { return operator(PBF_OP_EQ); }
"!=" { return operator(PBF_OP_NEQ); }
">" { return operator(PBF_OP_GT); }
diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y
index 5a79a8e7a45b..6ee6edef13f7 100644
--- a/tools/perf/util/bpf-filter.y
+++ b/tools/perf/util/bpf-filter.y
@@ -11,6 +11,7 @@
#include <string.h>
#include <linux/compiler.h>
#include <linux/list.h>
+#include <linux/zalloc.h>
#include "bpf-filter.h"
#include "cgroup.h"
@@ -31,6 +32,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
{
unsigned long num;
char *path;
+ char *lock_name;
+ struct {
+ char *name;
+ int mode;
+ } lock;
struct {
enum perf_bpf_filter_term term;
int part;
@@ -40,12 +46,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
}
%token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH
+%token BFT_LOCK_HELD BFT_LOCK_NAME
%type <expr> filter_term filter_expr
-%destructor { free ($$); } <expr>
+%destructor { perf_bpf_filter_expr__delete($$); } <expr>
+%destructor { free($$); } <lock_name>
+%destructor { free($$.name); } <lock>
%type <sample> BFT_SAMPLE BFT_SAMPLE_PATH
%type <op> BFT_OP
-%type <num> BFT_NUM
+%type <num> BFT_NUM BFT_LOCK_HELD
%type <path> BFT_PATH
+%type <lock_name> BFT_LOCK_NAME
+%type <lock> lock_term
%%
@@ -105,5 +116,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH
$$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id);
cgroup__put(cgrp);
}
+|
+lock_term
+{
+ $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1);
+ zfree(&$1.name);
+ if (!$$)
+ YYERROR;
+}
+|
+'!' lock_term
+{
+ $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0);
+ zfree(&$2.name);
+ if (!$$)
+ YYERROR;
+}
+|
+lock_term BFT_OP BFT_NUM
+{
+ $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3);
+ zfree(&$1.name);
+ if (!$$)
+ YYERROR;
+}
+
+lock_term:
+BFT_LOCK_HELD BFT_LOCK_NAME ')'
+{
+ $$.name = $2;
+ $$.mode = $1;
+}
%%
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index 683fec85e71e..215998897243 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -53,14 +53,79 @@ enum perf_bpf_filter_term {
/* Terms computed from BPF helpers. */
PBF_TERM_UID,
PBF_TERM_GID,
+ /*
+ * Kernel lock held predicate (`lock_held(...)`). Evaluates to 1 if the
+ * specified kernel lock is currently held by the running task/CPU, or
+ * 0 otherwise. The lock specification kind is stored in `entry.part`
+ * (`enum perf_bpf_lock_target_kind`) and the resolved target address,
+ * class ID, or type flag encoding is stored in `entry.target`.
+ */
+ PBF_TERM_LOCK_HELD,
};
+/*
+ * Lock target categories for PBF_TERM_LOCK_HELD filters.
+ * Stored in `perf_bpf_filter_entry.part`.
+ */
+enum perf_bpf_lock_target_kind {
+ PBF_LOCK_NONE = 0,
+ /*
+ * Specific global or static lock address (resolved from kallsyms or
+ * given as a hex address). `entry.target` holds the kernel address.
+ */
+ PBF_LOCK_ADDR,
+ /*
+ * Dynamic lock allocated inside a slab cache (`&slab_name`).
+ * `entry.target` holds the kernel `struct kmem_cache *` address.
+ */
+ PBF_LOCK_SLAB,
+ /*
+ * Per-process `current->mm->mmap_lock` (or `mmap_sem`).
+ */
+ PBF_LOCK_MMAP,
+ /*
+ * Per-process `current->sighand->siglock`.
+ */
+ PBF_LOCK_SIGHAND,
+ /*
+ * Semi-global lock class (`rq_lock` or `zone_lock`).
+ * `entry.target` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or
+ * LOCK_CLASS_ZONE_LOCK).
+ */
+ PBF_LOCK_CLASS,
+ /*
+ * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.).
+ * `entry.target` packs up to two `LCB_F_*` flag combinations in the
+ * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
+ */
+ PBF_LOCK_TYPE,
+};
+
+/*
+ * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD:
+ * Bits 0..3: `enum perf_bpf_lock_target_kind`
+ * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding)
+ * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding)
+ * Bit 6: PBF_LOCK_DIRECT_MUTEX (inspect struct mutex owner)
+ * Bit 7: PBF_LOCK_DIRECT_RWSEM (inspect struct rw_semaphore writer owner)
+ * Bit 8: PBF_LOCK_DIRECT_RTMUTEX (inspect struct rt_mutex_base owner)
+ */
+#define PBF_LOCK_KIND_MASK 0x0fU
+#define PBF_LOCK_MODE_READ (1U << 4)
+#define PBF_LOCK_MODE_WRITE (1U << 5)
+#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE)
+#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY
+#define PBF_LOCK_DIRECT_MUTEX (1U << 6)
+#define PBF_LOCK_DIRECT_RWSEM (1U << 7)
+#define PBF_LOCK_DIRECT_RTMUTEX (1U << 8)
+
/* BPF map entry for filtering */
struct perf_bpf_filter_entry {
enum perf_bpf_filter_op op;
__u32 part; /* sub-sample type info when it has multiple values */
enum perf_bpf_filter_term term;
__u64 value;
+ __u64 target; /* lock address, kmem_cache *, class, or type flags */
};
struct idx_hash_key {
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index 576fbed270e5..31bce5e55b62 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -5,8 +5,47 @@
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
+#include "lock_data.h"
#include "sample-filter.h"
+/*
+ * Key for the `held_locks` BPF hash map tracking currently held kernel locks.
+ *
+ * - `addr_or_id`:
+ * - PBF_LOCK_ADDR: kernel virtual address of the lock instance
+ * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK)
+ * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address
+ * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags
+ * - `owner_id`:
+ * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`),
+ * since the holder may sleep and migrate across CPUs while holding the lock.
+ * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks
+ * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption
+ * is disabled on the holding CPU and spinlocks such as `rq_lock` may be
+ * acquired by one task and released by the next task across `context_switch()`.
+ * - `kind`: `enum perf_bpf_lock_target_kind`.
+ */
+struct held_lock_key {
+ __u64 addr_or_id;
+ __u32 owner_id;
+ __u32 kind;
+};
+
+/*
+ * Value for `held_locks` entries.
+ * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`)
+ * and exclusive/write acquisitions (including nested class/slab/type
+ * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of
+ * the lock so release tracepoints that do not carry flags (such as
+ * `lock:contended_release`) can decrement matching class/slab/type entries.
+ */
+struct held_lock_val {
+ __u32 read_count;
+ __u32 write_count;
+ __u32 flags;
+ __u32 pad;
+};
+
/* BPF map that will be filled by user space */
struct filters {
__uint(type, BPF_MAP_TYPE_HASH);
@@ -43,10 +82,226 @@ struct lost_count {
__uint(max_entries, 1);
} dropped SEC(".maps");
+/*
+ * Currently held kernel locks tracked by fentry/fexit hooks.
+ * Uses BPF_MAP_TYPE_LRU_HASH so that if a counting semaphore (down/up) or
+ * non-owner rwsem is acquired in one task and released in another task or
+ * interrupt handler (cross-context signaling), any orphaned entry is
+ * automatically evicted rather than exhausting the map.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_LRU_HASH);
+ __type(key, struct held_lock_key);
+ __type(value, struct held_lock_val);
+ __uint(max_entries, 1);
+} held_locks SEC(".maps");
+
+/* Target lock addresses for PBF_LOCK_ADDR filters. */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, __u64);
+ __type(value, __u8);
+ __uint(max_entries, 1);
+} lock_addr_filter SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u32));
+ __uint(max_entries, 1);
+} lock_syms SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u8));
+ __uint(max_entries, 1);
+} slab_filter SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(struct slab_cache_data));
+ __uint(max_entries, 1);
+} slab_caches SEC(".maps");
+
volatile const int use_idx_hash;
+const volatile __u64 contig_page_data_addr;
+const volatile __u64 node_data_addr;
+const volatile int nr_nodes;
+const volatile int sizeof_zone;
+
+__u32 slab_cache_id;
+
+/* Control flags set by userspace when lock_held(...) filters are active. */
+int lock_enabled;
+int has_lock_addr;
+int has_lock_slab;
+int has_lock_class;
+int has_lock_type;
+int has_mmap_lock;
+int has_siglock;
+int track_rwsem_read;
+__u8 lock_type_filter[LCB_F_TYPE_MAX];
+
void *bpf_cast_to_kern_ctx(void *) __ksym;
+/*
+ * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`.
+ *
+ * For locks that maintain a task_struct pointer in their `owner` field
+ * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`),
+ * `perf_sample_filter` can inspect the lock structure directly in kernel memory
+ * and compare the owner against `current` (`bpf_get_current_task()`).
+ *
+ * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear
+ * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and
+ * spinlocks are checked via the `held_locks` BPF map instead.
+ */
+static __always_inline bool is_rwsem_writer_owner(__u64 lock_addr, __u64 curr)
+{
+ struct rw_semaphore___new *rwsem = (void *)lock_addr;
+
+ if (bpf_core_field_exists(rwsem->count.counter) &&
+ !(BPF_CORE_READ(rwsem, count.counter) & 1UL))
+ return false;
+ if (bpf_core_field_exists(rwsem->owner.counter) &&
+ (BPF_CORE_READ(rwsem, owner.counter) & 1UL))
+ return false;
+
+ return (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr;
+}
+
+static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __u32 part)
+{
+ if (!curr)
+ return false;
+
+ if ((part & PBF_LOCK_DIRECT_MUTEX) &&
+ (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr)
+ return true;
+ if ((part & PBF_LOCK_DIRECT_RWSEM) &&
+ is_rwsem_writer_owner(lock_addr, curr))
+ return true;
+ if ((part & PBF_LOCK_DIRECT_RTMUTEX) &&
+ (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr)
+ return true;
+ return false;
+}
+
+static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode)
+{
+ if (!val)
+ return false;
+ if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0)
+ return true;
+ if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0)
+ return true;
+ return false;
+}
+
+static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind,
+ __u32 mode, __u32 pid, __u32 cpu_id)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = pid,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+
+ if (pid) {
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+ }
+
+ key.owner_id = cpu_id;
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+
+ return 0;
+}
+
+static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode,
+ __u32 pid, __u32 cpu_id)
+{
+ __u32 flag1 = target & 0xff;
+ __u32 flag2 = (target >> 8) & 0xff;
+
+ if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ return 0;
+}
+
+static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry)
+{
+ __u64 curr;
+ __u32 pid;
+ __u32 cpu_id;
+ __u32 kind;
+ __u32 mode;
+ __u64 lock_addr;
+
+ /*
+ * Pinned filters (use_idx_hash != 0) share a world-writable filters
+ * map and do not support lock_held(). Reject evaluation when pinned
+ * or when lock_enabled is not set so unprivileged users cannot probe
+ * arbitrary kernel addresses via entry->target.
+ */
+ if (use_idx_hash || !lock_enabled)
+ return 0;
+
+ curr = bpf_get_current_task();
+ pid = (__u32)bpf_get_current_pid_tgid();
+ cpu_id = 0x80000000U | bpf_get_smp_processor_id();
+ kind = entry->part & PBF_LOCK_KIND_MASK;
+ mode = entry->part & PBF_LOCK_MODE_MASK;
+
+ if (!mode)
+ mode = PBF_LOCK_MODE_ANY;
+
+ switch (kind) {
+ case PBF_LOCK_ADDR:
+ lock_addr = entry->target;
+ if ((mode & PBF_LOCK_MODE_WRITE) &&
+ is_direct_lock_owner(lock_addr, curr, entry->part))
+ return 1;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_MMAP:
+ lock_addr = get_current_mmap_lock(curr);
+ if (!lock_addr)
+ return 0;
+ if ((mode & PBF_LOCK_MODE_WRITE) &&
+ is_rwsem_writer_owner(lock_addr, curr))
+ return 1;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_SIGHAND:
+ lock_addr = get_current_siglock(curr);
+ if (!lock_addr)
+ return 0;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_CLASS:
+ return is_lock_in_map(entry->target, PBF_LOCK_CLASS, mode, pid, cpu_id);
+
+ case PBF_LOCK_SLAB:
+ return is_lock_in_map(entry->target, PBF_LOCK_SLAB, mode, pid, cpu_id);
+
+ case PBF_LOCK_TYPE:
+ return is_lock_type_in_map(entry->target, mode, pid, cpu_id);
+
+ default:
+ return 0;
+ }
+}
+
/* new kernel perf_sample_data definition */
struct perf_sample_data___new {
__u64 sample_flags;
@@ -175,6 +430,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
return bpf_get_current_uid_gid() & 0xFFFFFFFF;
case PBF_TERM_GID:
return bpf_get_current_uid_gid() >> 32;
+ case PBF_TERM_LOCK_HELD:
+ return check_lock_held(entry);
case PBF_TERM_NONE:
case __PBF_UNUSED_TERM4:
case __PBF_UNUSED_TERM5:
@@ -209,6 +466,8 @@ static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry
return bpf_get_current_uid_gid() & 0xFFFFFFFF;
case PBF_TERM_GID:
return bpf_get_current_uid_gid() >> 32;
+ case PBF_TERM_LOCK_HELD:
+ return check_lock_held(entry);
default:
return 0;
}
@@ -380,4 +639,760 @@ int perf_tp_sample_filter(void *ctx)
return 0;
}
+static __always_inline __u32 get_lock_owner_id(__u32 flags)
+{
+ __u32 owner_id;
+
+ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN)
+ return 0x80000000U | bpf_get_smp_processor_id();
+
+ owner_id = (__u32)bpf_get_current_pid_tgid();
+ if (!owner_id)
+ owner_id = 0x80000000U | bpf_get_smp_processor_id();
+ return owner_id;
+}
+
+static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id,
+ __u32 kind, __u32 flags, bool acquired)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = owner_id,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (acquired) {
+ if (val) {
+ if (is_read)
+ __sync_fetch_and_add(&val->read_count, 1);
+ else
+ __sync_fetch_and_add(&val->write_count, 1);
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags,
+ };
+
+ bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY);
+ }
+ } else if (val) {
+ if (is_read) {
+ if (val->read_count <= 1 && val->write_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->read_count > 0)
+ __sync_fetch_and_add(&val->read_count, -1);
+ } else {
+ if (val->write_count <= 1 && val->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->write_count > 0)
+ __sync_fetch_and_add(&val->write_count, -1);
+ }
+ }
+}
+
+#define LCB_F_READ_IRQSAVE (1U << 30)
+#define LCB_F_READ_IRQ_BH (1U << 31)
+
+static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id)
+{
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ bool matched = false;
+ bool is_read = (flags & LCB_F_READ) != 0;
+ __u32 class_id = 0;
+ __u64 kmem_cache_addr = 0;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool match_type = false;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
+ matched = true;
+
+ if (has_mmap_lock || has_siglock) {
+ __u32 ltype = check_lock_type(lock_addr, type_flag);
+
+ if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
+ (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
+ matched = true;
+ }
+
+ if (has_lock_class && type_flag == LCB_F_SPIN) {
+ __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
+
+ if (cid) {
+ class_id = *cid;
+ matched = true;
+ }
+ }
+
+ if (has_lock_slab && bpf_get_kmem_cache) {
+ kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
+ if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr))
+ matched = true;
+ else
+ kmem_cache_addr = 0;
+ }
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) {
+ match_type = true;
+ matched = true;
+ }
+
+ if (!matched)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (existing) {
+ /*
+ * On CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC
+ * kernels, _raw_read_lock_irq() and _raw_read_lock_bh() call
+ * the out-of-line _raw_read_lock_irqsave(). Clear the marker
+ * and skip the duplicate outer fexit increment.
+ */
+ if ((flags & LCB_F_READ_IRQ_BH) &&
+ (existing->flags & LCB_F_READ_IRQSAVE)) {
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ return 0;
+ }
+ /* Non-reader locks are not recursive; avoid double-counting */
+ if (!is_read && existing->write_count > 0)
+ return 0;
+ if (is_read) {
+ __sync_fetch_and_add(&existing->read_count, 1);
+ if (flags & LCB_F_READ_IRQSAVE)
+ existing->flags |= LCB_F_READ_IRQSAVE;
+ else
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ } else {
+ __sync_fetch_and_add(&existing->write_count, 1);
+ }
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags & (LCB_F_TYPE_MASK | LCB_F_READ_IRQSAVE),
+ };
+
+ if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0)
+ return 0;
+ }
+
+ if (class_id)
+ update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, type_flag, true);
+ if (kmem_cache_addr) {
+ update_held_lock_count((__u64)kmem_cache_addr, owner_id,
+ PBF_LOCK_SLAB, type_flag, true);
+ }
+ if (match_type)
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true);
+
+ return 0;
+}
+
+static __always_inline int record_lock_acquired(void *ctx, __u32 flags)
+{
+ return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags));
+}
+
+static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags)
+{
+ __u32 owner_id = get_lock_owner_id(flags);
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (!existing)
+ return 0;
+
+ if (is_read) {
+ if (existing->read_count == 0)
+ return 0;
+ if (existing->read_count == 1 && existing->write_count == 0) {
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ } else {
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ __sync_fetch_and_add(&existing->read_count, -1);
+ }
+ } else {
+ if (existing->write_count == 0)
+ return 0;
+ if (existing->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ else
+ existing->write_count = 0;
+ }
+
+ if (has_lock_class && flags == LCB_F_SPIN) {
+ __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
+
+ if (cid)
+ update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false);
+ }
+
+ if (has_lock_slab && bpf_get_kmem_cache) {
+ __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
+
+ if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) {
+ update_held_lock_count(kmem_cache_addr, owner_id,
+ PBF_LOCK_SLAB, flags, false);
+ }
+ }
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag])
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false);
+
+ return 0;
+}
+
+static __always_inline int record_lock_released(void *ctx, __u32 flags)
+{
+ return record_lock_released_addr(get_lock_addr(ctx), flags);
+}
+
+/* Mutex acquire/release hooks */
+SEC("fexit/mutex_lock")
+int BPF_PROG(fexit_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_nested")
+int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/_mutex_lock_nest_lock")
+int BPF_PROG(fexit__mutex_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io")
+int BPF_PROG(fexit_mutex_lock_io, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io_nested")
+int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_interruptible")
+int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_interruptible_nested")
+int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_killable")
+int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/_mutex_lock_killable")
+int BPF_PROG(fexit__mutex_lock_killable, void *lock, unsigned int subclass,
+ void *nest_lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_trylock")
+int BPF_PROG(fexit_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/_mutex_trylock_nest_lock")
+int BPF_PROG(fexit__mutex_trylock_nest_lock, void *lock, void *nest_lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fentry/mutex_unlock")
+int BPF_PROG(fentry_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_MUTEX);
+}
+
+/* RW semaphore acquire/release hooks */
+SEC("fexit/down_read")
+int BPF_PROG(fexit_down_read, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_nested")
+int BPF_PROG(fexit_down_read_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_interruptible")
+int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable")
+int BPF_PROG(fexit_down_read_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable_nested")
+int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_trylock")
+int BPF_PROG(fexit_down_read_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_write")
+int BPF_PROG(fexit_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_nested")
+int BPF_PROG(fexit_down_write_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/_down_write_nest_lock")
+int BPF_PROG(fexit__down_write_nest_lock, void *sem, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_killable")
+int BPF_PROG(fexit_down_write_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_killable_nested")
+int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_trylock")
+int BPF_PROG(fexit_down_write_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/downgrade_write")
+int BPF_PROG(fexit_downgrade_write, void *sem)
+{
+ if (track_rwsem_read)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+SEC("fentry/up_read")
+int BPF_PROG(fentry_up_read, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_READ);
+}
+
+SEC("fentry/up_write")
+int BPF_PROG(fentry_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+/* Spinlock acquire/release hooks */
+SEC("fexit/_raw_spin_lock")
+int BPF_PROG(fexit_raw_spin_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_nested")
+int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_nest_lock")
+int BPF_PROG(fexit_raw_spin_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq")
+int BPF_PROG(fexit_raw_spin_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq_disable")
+int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave")
+int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave_nested")
+int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_bh")
+int BPF_PROG(fexit_raw_spin_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_trylock")
+int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fexit/_raw_spin_trylock_bh")
+int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fentry/_raw_spin_unlock")
+int BPF_PROG(fentry_raw_spin_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq")
+int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq_enable")
+int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irqrestore")
+int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_bh")
+int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+/* RWlock acquire/release hooks */
+SEC("fexit/_raw_read_lock")
+int BPF_PROG(fexit_raw_read_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_lock_irq")
+int BPF_PROG(fexit_raw_read_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH);
+}
+
+SEC("fexit/_raw_read_lock_irqsave")
+int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQSAVE);
+}
+
+SEC("fexit/_raw_read_lock_bh")
+int BPF_PROG(fexit_raw_read_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH);
+}
+
+SEC("fexit/_raw_read_trylock")
+int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+ return 0;
+}
+
+SEC("fentry/_raw_read_unlock")
+int BPF_PROG(fentry_raw_read_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irq")
+int BPF_PROG(fentry_raw_read_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irqrestore")
+int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_bh")
+int BPF_PROG(fentry_raw_read_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_write_lock")
+int BPF_PROG(fexit_raw_write_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_nested")
+int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irq")
+int BPF_PROG(fexit_raw_write_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irqsave")
+int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_bh")
+int BPF_PROG(fexit_raw_write_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_trylock")
+int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fentry/_raw_write_unlock")
+int BPF_PROG(fentry_raw_write_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irq")
+int BPF_PROG(fentry_raw_write_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irqrestore")
+int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_bh")
+int BPF_PROG(fentry_raw_write_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+/* RT-Mutex acquire/release hooks */
+SEC("fexit/rt_mutex_lock")
+int BPF_PROG(fexit_rt_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_nested")
+int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/_rt_mutex_lock_nest_lock")
+int BPF_PROG(fexit__rt_mutex_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_interruptible")
+int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_lock_killable")
+int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_trylock")
+int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fentry/rt_mutex_unlock")
+int BPF_PROG(fentry_rt_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_RT);
+}
+
+/* Semaphore acquire/release hooks */
+SEC("fexit/down")
+int BPF_PROG(fexit_down, void *sem)
+{
+ return record_lock_acquired(ctx, 0);
+}
+
+SEC("fexit/down_interruptible")
+int BPF_PROG(fexit_down_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_killable")
+int BPF_PROG(fexit_down_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_timeout")
+int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_trylock")
+int BPF_PROG(fexit_down_trylock, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fentry/up")
+int BPF_PROG(fentry_up, void *sem)
+{
+ return record_lock_released(ctx, 0);
+}
+
+/* Percpu-rwsem write acquire/release hooks */
+SEC("fexit/percpu_down_write")
+int BPF_PROG(fexit_percpu_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
+SEC("fentry/percpu_up_write")
+int BPF_PROG(fentry_percpu_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
+SEC("raw_tp/bpf_test_finish")
+int BPF_PROG(collect_lock_syms)
+{
+ return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
+}
+
+SEC("iter/kmem_cache")
+int slab_cache_iter(void *ctx)
+{
+ return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
+}
+
char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index 2f3317ce17f0..1fcb424b75c0 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist,
/* Will a tracepoint filter work for str or should a BPF filter be used? */
static bool is_possible_tp_filter(const char *str)
{
- return strstr(str, "uid") == NULL;
+ return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL &&
+ strstr(str, "lock_held") == NULL;
}
static int set_filter(struct evsel *evsel, const void *arg)
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v4 5/7] perf test: Add tests for lock_held() BPF sample filter
2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers
` (3 preceding siblings ...)
2026-10-19 22:09 ` [PATCH v4 4/7] perf bpf-filter: Add lock_held() sample filter Ian Rogers
@ 2026-10-19 22:09 ` Ian Rogers
2026-10-19 22:12 ` sashiko-bot
2026-10-19 22:09 ` [PATCH v4 6/7] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
` (2 subsequent siblings)
7 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, eranian, ddhariwal, rcu
Add unit tests in parse-events.c for parsing lock_held(),
lock_held_read(), and lock_held_exclusive() BPF filter expressions
(standalone predicates, negated predicates, and compound group
expressions) and extend record_bpf_filter.sh with lock_held(),
lock_held_read()/lock_held_write(), and ':R'/':W' recording tests.
Example output:
# perf test -v 'Parsing of BPF sample filters'
5: Parse event definition strings :
5.7: Parsing of BPF sample filters : Ok
# perf test -v 'perf record sample filtering (by BPF) tests'
112: perf record sample filtering (by BPF) tests :
Checking BPF-filter privilege
Basic bpf-filter test
Basic bpf-filter test [Success]
Failing bpf-filter test
Failing bpf-filter test [Success]
Group bpf-filter test
Group bpf-filter test [Success]
Multiple bpf-filter test
Multiple bpf-filter test [Success]
Cgroup bpf-filter test
Cgroup bpf-filter test [Success]
Lock-held bpf-filter test
Lock-held bpf-filter test [Success]
112: perf record sample filtering (by BPF) tests : Ok
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/tests/parse-events.c | 108 +++++++++++++++++
tools/perf/tests/shell/record_bpf_filter.sh | 122 ++++++++++++++++++++
2 files changed, 230 insertions(+)
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 5f6f95c8a5b4..4cf9192762f2 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -8,9 +8,11 @@
#include "debug.h"
#include "pmu.h"
#include "pmus.h"
+#include "bpf-filter.h"
#include "strbuf.h"
#include <dirent.h>
#include <errno.h>
+#include <string.h>
#include "fncache.h"
#include <sys/types.h>
#include <sys/stat.h>
@@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused,
return ret;
}
+#ifdef HAVE_BPF_SKEL
+static void clear_bpf_filter_exprs(struct list_head *expr_head)
+{
+ struct perf_bpf_filter_expr *expr, *tmp;
+
+ list_for_each_entry_safe(expr, tmp, expr_head, list) {
+ list_del(&expr->list);
+ perf_bpf_filter_expr__delete(expr);
+ }
+}
+
+static int do_test_bpf_filters(struct list_head *expr_head)
+{
+ struct perf_bpf_filter_expr *expr, *group;
+ const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100";
+
+ /* Simple lock_held(&cgroup_mutex) */
+ TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)",
+ __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is &cgroup_mutex",
+ expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /* Negated !lock_held(cgroup_mutex) */
+ TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)",
+ __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 0", expr->val == 0);
+ TEST_ASSERT_VAL("lock_name is cgroup_mutex",
+ expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /* Read/shared and write/exclusive predicates */
+ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)",
+ __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is &mmap_lock",
+ expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock"));
+ clear_bpf_filter_exprs(expr_head);
+
+ TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)",
+ __perf_bpf_filter__parse(expr_head,
+ "!lock_held_exclusive(mmap_lock)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 0", expr->val == 0);
+ TEST_ASSERT_VAL("lock_name is mmap_lock",
+ expr->lock_name && !strcmp(expr->lock_name, "mmap_lock"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /*
+ * Comparison and compound group:
+ * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100
+ */
+ TEST_ASSERT_VAL("parse compound lock_held filter",
+ __perf_bpf_filter__parse(expr_head, compound) == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN);
+ TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups));
+ group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("first group member lock_name",
+ group->lock_name && !strcmp(group->lock_name, "rwsem:R"));
+ TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1);
+ group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("second group member lock_name",
+ group->lock_name && !strcmp(group->lock_name, "&dentry"));
+ TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0);
+ expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100);
+
+ return TEST_OK;
+}
+#endif
+
+static int test__bpf_filters(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
+{
+#ifdef HAVE_BPF_SKEL
+ struct list_head expr_head = LIST_HEAD_INIT(expr_head);
+ int ret = do_test_bpf_filters(&expr_head);
+
+ clear_bpf_filter_exprs(&expr_head);
+ return ret;
+#else
+ return TEST_SKIP;
+#endif
+}
+
static struct test_case tests__parse_events[] = {
TEST_CASE_REASON("Test event parsing",
events2,
@@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = {
"no aliases in sysfs"),
TEST_CASE("Parsing of aliased events", pmu_events_alias2),
TEST_CASE("Parsing of terms (event modifiers)", terms2),
+ TEST_CASE("Parsing of BPF sample filters", bpf_filters),
{ .name = NULL, }
};
diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh
index 383574cb3bd3..ebaf8ee41622 100755
--- a/tools/perf/tests/shell/record_bpf_filter.sh
+++ b/tools/perf/tests/shell/record_bpf_filter.sh
@@ -176,6 +176,124 @@ test_bpf_filter_cgroup() {
echo "Cgroup bpf-filter test [Success]"
}
+test_bpf_filter_lock_held() {
+ echo "Lock-held bpf-filter test"
+ lock_sym="cgroup_mutex"
+
+ if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms
+ then
+ lock_sym="slab_mutex"
+ fi
+
+ if ! perf record -e task-clock --filter "lock_held(${lock_sym})" \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if [ "$(id -u)" != 0 ]
+ then
+ echo "Lock-held bpf-filter test [Skipped permission]"
+ return
+ fi
+ if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms
+ then
+ echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]"
+ return
+ fi
+ echo "Lock-held bpf-filter test [Failed ${lock_sym}]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(&dentry)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed &dentry]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \
+ -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed mmap_lock]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \
+ -o /dev/null true 2>&1 | grep -q "Kernel BTF lacks required"
+ then
+ echo "Lock-held bpf-filter test [Failed rq_lock/spinlock]"
+ err=1
+ return
+ fi
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(rwsem)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock type]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock \
+ --filter 'lock_held_read(mmap_lock) || lock_held_write(mmap_lock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read/write(mmap_lock)]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mmap_lock:R) || lock_held(mmap_lock:W)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed mmap_lock:R/:W]"
+ err=1
+ return
+ fi
+
+ if perf list tracepoint 2> /dev/null | grep -q kmem:mm_page_alloc
+ then
+ if ! perf record -e kmem:mm_page_alloc --filter 'lock_held(mutex) || lock_held(rwsem)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed tracepoint kmem:mm_page_alloc]"
+ err=1
+ return
+ fi
+ fi
+
+ if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \
+ -o /dev/null true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]"
+ err=1
+ return
+ fi
+
+ if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \
+ -o /dev/null true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]"
+ err=1
+ return
+ fi
+
+ echo "Lock-held bpf-filter test [Success]"
+}
+
test_bpf_filter_priv
if [ $err = 0 ]; then
@@ -198,5 +316,9 @@ if [ $err = 0 ]; then
test_bpf_filter_cgroup
fi
+if [ $err = 0 ]; then
+ test_bpf_filter_lock_held
+fi
+
cleanup
exit $err
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v4 6/7] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers
` (4 preceding siblings ...)
2026-10-19 22:09 ` [PATCH v4 5/7] perf test: Add tests for lock_held() BPF " Ian Rogers
@ 2026-10-19 22:09 ` Ian Rogers
2026-10-19 22:19 ` sashiko-bot
2026-10-19 22:09 ` [PATCH v4 7/7] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
7 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, eranian, ddhariwal, rcu
Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks
Trace RCU, and SRCU locks:
- 'rcu' (supports both read/shared and write/exclusive modes):
- Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)',
'lock_held(rcu_read_lock)'): directly inspects
current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on
CONFIG_PREEMPT_RCU kernels), accounting for the baseline
rcu_read_lock() nesting acquired by the perf overflow handler itself
(bpf_overflow_handler() and do_perf_sw_event()) with zero extra
fentry/fexit tracing hooks.
- Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'):
attaches fentry/fexit hooks to synchronize_rcu(),
synchronize_rcu_expedited(), synchronize_rcu_tasks(), and
rcu_barrier() to track grace-period and barrier waits.
- 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both
read/shared and write/exclusive modes):
- Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches
fexit/__srcu_read_lock and fentry/__srcu_read_unlock.
- Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'):
attaches fentry/fexit on synchronize_srcu(),
synchronize_srcu_expedited(), and srcu_barrier().
- 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports
read/shared mode only): directly inspects
current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels
with zero extra tracing hooks.
Example usage:
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ]
# perf report --stdio --no-children
# Overhead Command Shared Object Symbol
# ........ ....... ................. ........................
#
14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu
|
---__d_lookup_rcu
lookup_fast
walk_component
link_path_walk
path_lookupat
filename_lookup
vfs_statx
__do_sys_newstat
do_syscall_64
entry_SYSCALL_64_after_hwframe
# perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1
# perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1
Signed-off-by: Ian Rogers <irogers@google.com>
# Conflicts:
# tools/perf/util/bpf-filter.c
# tools/perf/util/bpf_skel/sample_filter.bpf.c
# Conflicts:
# tools/perf/util/bpf_skel/sample_filter.bpf.c
# Conflicts:
# tools/perf/util/bpf-filter.c
# Conflicts:
# tools/perf/Documentation/perf-record.txt
# tools/perf/util/bpf-filter.c
---
tools/perf/Documentation/perf-record.txt | 44 +++--
tools/perf/tests/parse-events.c | 13 ++
tools/perf/tests/shell/record_bpf_filter.sh | 20 ++
tools/perf/util/bpf-filter.c | 194 +++++++++++++++++--
tools/perf/util/bpf_skel/lock_data.h | 44 ++++-
tools/perf/util/bpf_skel/sample-filter.h | 12 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 135 ++++++++++++-
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 +
tools/perf/util/lock-contention.c | 4 +
tools/perf/util/lock-contention.h | 4 +-
10 files changed, 438 insertions(+), 34 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 8496b20987a0..3eaebdc9efc2 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -219,11 +219,17 @@ OPTIONS
'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache
name prefixed with '&' (e.g. 'lock_held(&dentry)',
'lock_held_read(&dentry)'), a well-known lock name ('mmap_lock',
- 'siglock', 'rq_lock', 'zone_lock'), or a lock type ('mutex',
- 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R',
- 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore').
- Both read/shared and write/exclusive tracking are supported for
- 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', 'spinlock'
+ 'siglock', 'rq_lock', 'zone_lock', 'rcu', 'rcu-trace'), or a lock
+ type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
+ 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
+ 'srcu', 'srcu:R', 'srcu:W', 'rcu:R', 'rcu:W', 'semaphore'). Both
+ read/shared and write/exclusive tracking are supported for 'rwsem'
+ (including 'mmap_lock'), 'rwlock', 'rcu', and 'srcu' (where
+ read/shared mode matches 'rcu_read_lock()' / 'srcu_read_lock()'
+ critical sections and write/exclusive mode matches grace-period
+ waits 'synchronize_rcu*()', 'synchronize_srcu*()', 'rcu_barrier()',
+ and 'srcu_barrier()'); 'rcu-trace' ('rcu_read_lock_trace()')
+ supports read/shared tracking only; and 'mutex', 'spinlock'
(including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex',
'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support
write/exclusive tracking only. Note that 'semaphore' tracking
@@ -245,19 +251,21 @@ OPTIONS
--filter 'lock_held(mutex) || lock_held(rwsem)'
When filtering on a specific mutex symbol ('*_mutex'), rt-mutex symbol
- ('*_rt_mutex'), or write-locked 'mmap_lock' ('lock_held_write(mmap_lock)'
- or 'lock_held(mmap_lock:W)'), 'perf_sample_filter' inspects the lock's
- 'owner' field directly in kernel memory on each sample without attaching
- any BPF fentry/fexit hooks. For locks that do not maintain an
- authoritative per-task owner pointer (such as spinlocks, rwlocks,
- semaphores, reader-locked rwsems, and slab/type filters), it attaches
- BPF fentry/fexit programs to kernel lock acquire and release functions
- to track held locks in a BPF map. These BPF hooks run on every kernel
- lock acquisition and release for the attached lock family, even when no
- perf sample is being taken. To minimize this overhead, only the hooks
- for the relevant lock family and mode are attached when known (for
- example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions
- and 'rq_lock' only hooks spinlock functions), whereas slab filters
+ ('*_rt_mutex'), write-locked 'mmap_lock' ('lock_held_write(mmap_lock)'
+ or 'lock_held(mmap_lock:W)'), or 'rcu:R' / 'rcu-trace' (which inspect
+ 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting'),
+ 'perf_sample_filter' inspects the kernel fields directly on each sample
+ without attaching any BPF fentry/fexit hooks. For locks that do not
+ maintain an authoritative per-task owner pointer (such as spinlocks,
+ rwlocks, semaphores, reader-locked rwsems, SRCU locks, RCU synchronize
+ waits, and slab/type filters), it attaches BPF fentry/fexit programs to
+ kernel lock acquire and release functions to track held locks in a BPF
+ map. These BPF hooks run on every kernel lock acquisition and release
+ for the attached lock family, even when no perf sample is being taken.
+ To minimize this overhead, only the hooks for the relevant lock family
+ and mode are attached when known (for example, 'lock_held_read(mmap_lock)'
+ only hooks rwsem read functions, 'srcu' only hooks SRCU functions, and
+ 'rq_lock' only hooks spinlock functions), whereas slab filters
('&<slab>'), raw hex addresses, or symbols without a recognizable
lock-type suffix attach hooks across all lock families (including
high-frequency spinlocks).
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 4cf9192762f2..730466b7ebf6 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head)
expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex"));
clear_bpf_filter_exprs(expr_head);
+ /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */
+ TEST_ASSERT_VAL("parse lock_held(rcu)",
+ __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is rcu",
+ expr->lock_name && !strcmp(expr->lock_name, "rcu"));
+ clear_bpf_filter_exprs(expr_head);
+
/* Read/shared and write/exclusive predicates */
TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)",
__perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0);
diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh
index ebaf8ee41622..47f697f53a85 100755
--- a/tools/perf/tests/shell/record_bpf_filter.sh
+++ b/tools/perf/tests/shell/record_bpf_filter.sh
@@ -247,6 +247,26 @@ test_bpf_filter_lock_held() {
return
fi
+ if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed rcu:W/srcu]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \
+ -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]"
+ err=1
+ return
+ fi
+ fi
+
if ! perf record -e task-clock \
--filter 'lock_held_read(mmap_lock) || lock_held_write(mmap_lock)' \
-o "${perfdata}" true 2> /dev/null
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index ab9b58f6cb71..887b5c406429 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -201,11 +201,19 @@ enum lock_prog_group {
LOCK_PROG_RTMUTEX = 1U << 6,
LOCK_PROG_SEM = 1U << 7,
LOCK_PROG_PERCPU_RWSEM = 1U << 8,
- LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM,
+ LOCK_PROG_SRCU_READ = 1U << 9,
+ LOCK_PROG_SRCU_WRITE = 1U << 10,
+ LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE,
+ LOCK_PROG_RCU_WRITE = 1U << 11,
+ LOCK_PROG_ALL = 0xfffU & ~LOCK_PROG_SEM,
};
static unsigned int lock_flags_to_prog_group(unsigned int flags)
{
+ if (flags & LCB_F_RCU)
+ return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0;
+ if (flags & LCB_F_SRCU)
+ return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE;
if (flags & LCB_F_MUTEX)
return LOCK_PROG_MUTEX;
if (flags & LCB_F_RT)
@@ -325,6 +333,32 @@ static const struct lock_prog_desc lock_progs[] = {
/* Percpu-RWsem */
LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+ /* SRCU */
+ LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe",
+ LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe",
+ LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited",
+ LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited",
+ LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE),
+ /* RCU grace-period synchronize / barrier */
+ LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited",
+ LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited",
+ LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE),
};
#undef LOCK_FPROG
@@ -612,6 +646,18 @@ static unsigned int supported_lock_prog_groups(struct btf *btf)
if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write"))
supported |= LOCK_PROG_PERCPU_RWSEM;
+ if ((btf_has_func(btf, "__srcu_read_lock") ||
+ btf_has_func(btf, "__srcu_read_lock_nmisafe")) &&
+ (btf_has_func(btf, "__srcu_read_unlock") ||
+ btf_has_func(btf, "__srcu_read_unlock_nmisafe")))
+ supported |= LOCK_PROG_SRCU_READ;
+
+ if (btf_has_func(btf, "synchronize_srcu"))
+ supported |= LOCK_PROG_SRCU_WRITE;
+
+ if (btf_has_func(btf, "synchronize_rcu"))
+ supported |= LOCK_PROG_RCU_WRITE;
+
return supported;
}
@@ -731,6 +777,19 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
*needs_lock_syms = true;
return;
}
+ if (!strcmp(base, "rcu")) {
+ if (mode & PBF_LOCK_MODE_WRITE) {
+ *prog_groups |= LOCK_PROG_RCU_WRITE;
+ *required_groups |= LOCK_PROG_RCU_WRITE;
+ }
+ return;
+ }
+ if (!strcmp(base, "rcu_read_lock") ||
+ !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") ||
+ !strcmp(base, "rcu_read_lock_trace")) {
+ /* Inspected directly on task_struct; no tracing hooks needed. */
+ return;
+ }
if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) {
*prog_groups |= groups;
*required_groups |= groups;
@@ -772,6 +831,13 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
groups |= LOCK_PROG_RWLOCK_WRITE;
*prog_groups |= groups;
*required_groups |= groups;
+ } else if (strtailcmp(base, "srcu") == 0) {
+ if (mode & PBF_LOCK_MODE_READ)
+ groups |= LOCK_PROG_SRCU_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ groups |= LOCK_PROG_SRCU_WRITE;
+ *prog_groups |= groups;
+ *required_groups |= groups;
} else {
*prog_groups |= LOCK_PROG_ALL;
}
@@ -855,6 +921,59 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
return false;
}
+static u64 rcu_min_depth(struct evsel *evsel)
+{
+ if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE &&
+ evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK &&
+ evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK)
+ return 2;
+ return 1;
+}
+
+static bool btf_struct_has_field(const struct btf *btf, s32 type_id,
+ const char *field_name)
+{
+ const struct btf_type *t;
+ const struct btf_member *m;
+ u16 i, vlen;
+
+ type_id = btf__resolve_type(btf, type_id);
+ if (type_id < 0)
+ return false;
+
+ t = btf__type_by_id(btf, type_id);
+ if (!t || (!btf_is_struct(t) && !btf_is_union(t)))
+ return false;
+
+ m = btf_members(t);
+ vlen = btf_vlen(t);
+ for (i = 0; i < vlen; i++, m++) {
+ const char *name = btf__name_by_offset(btf, m->name_off);
+
+ if (name && *name) {
+ if (!strcmp(name, field_name))
+ return true;
+ } else if (btf_struct_has_field(btf, m->type, field_name)) {
+ return true;
+ }
+ }
+ return false;
+}
+
+static bool btf_has_task_field(struct btf *btf, const char *field_name)
+{
+ s32 id;
+
+ if (!btf)
+ return false;
+
+ id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT);
+ if (id < 0)
+ return false;
+
+ return btf_struct_has_field(btf, id, field_name);
+}
+
/*
* Resolve a `lock_held(<name>)` filter expression into `entry->part`
* (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate
@@ -862,15 +981,15 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
*
* Resolution order:
* 1. Slab cache name (when prefixed with '&', e.g. `&dentry`).
- * 2. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
- * `rq_lock`, `zone_lock`.
+ * 2. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`,
+ * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`.
* 3. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`,
- * `rwlock`, `percpu-rwsem`, `semaphore`, etc.
+ * `rwlock`, `percpu-rwsem`, `srcu`, `semaphore`, etc.
* 4. Hexadecimal lock address.
* 5. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`).
*/
-static int resolve_lock_expr(struct sample_filter_bpf *skel,
- struct perf_bpf_filter_expr *expr,
+static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct btf *btf, struct perf_bpf_filter_expr *expr,
struct perf_bpf_filter_entry *entry)
{
const char *name = expr->lock_name;
@@ -948,6 +1067,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel,
return 0;
}
+ if (!strcmp(base, "rcu_read_lock")) {
+ if (mode == PBF_LOCK_MODE_WRITE) {
+ pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
+ return -EINVAL;
+ }
+ if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
+ pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ;
+ entry->target = rcu_min_depth(evsel);
+ return 0;
+ }
+
+ if (!strcmp(base, "rcu")) {
+ if ((mode & PBF_LOCK_MODE_READ) && btf &&
+ !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
+ pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU | mode;
+ entry->target = rcu_min_depth(evsel);
+ if ((mode & PBF_LOCK_MODE_WRITE) && skel)
+ skel->bss->has_rcu_lock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") ||
+ !strcmp(base, "rcu_read_lock_trace")) {
+ if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) {
+ pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
+ return -EINVAL;
+ }
+ if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) {
+ pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ;
+ entry->target = 0;
+ return 0;
+ }
+
if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
entry->part = PBF_LOCK_TYPE | mode;
entry->target = encoded_pair;
@@ -1017,7 +1178,7 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel,
}
static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
- struct perf_bpf_filter_entry *entry)
+ struct btf *btf, struct perf_bpf_filter_entry *entry)
{
int i = 0, ret;
struct perf_bpf_filter_expr *expr;
@@ -1035,7 +1196,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske
entry[i].value = expr->val;
entry[i].target = 0;
if (expr->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(skel, expr, &entry[i]);
+ ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]);
if (ret < 0)
return ret;
}
@@ -1054,7 +1215,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske
entry[i].value = group->val;
entry[i].target = 0;
if (group->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(skel, group, &entry[i]);
+ ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]);
if (ret < 0)
return ret;
}
@@ -1329,7 +1490,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- ret = get_filter_entries(evsel, NULL, entry);
+ ret = get_filter_entries(evsel, NULL, NULL, entry);
if (ret < 0) {
pr_err("Failed to process filter entries\n");
goto err;
@@ -1391,6 +1552,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
init_numa_data(skel, btf);
}
+ if (prog_groups & LOCK_PROG_RCU_WRITE) {
+ u64 rcu_addr = 1ULL;
+
+ /*
+ * Offset by 1 byte so rcu_state_addr is distinct from
+ * &rcu_state.node[0].lock at offset 0 of rcu_state.
+ */
+ if (find_kernel_lock_sym("rcu_state", &rcu_addr) == 0)
+ rcu_addr += 1;
+ skel->rodata->rcu_state_addr = rcu_addr;
+ }
ret = configure_lock_progs(skel, btf, prog_groups, required_groups,
needs_lock_syms, needs_slab_iter,
is_tracepoint);
@@ -1409,7 +1581,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
if (has_lock_filter && needs_slab_iter)
run_slab_cache_iter(skel);
- ret = get_filter_entries(evsel, skel, entry);
+ ret = get_filter_entries(evsel, skel, btf, entry);
if (ret < 0) {
pr_err("Failed to process filter entries\n");
goto err;
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index 43043d62ea9f..136ffbffbfff 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -39,6 +39,7 @@ struct contention_task_data {
*/
#define LCD_F_MMAP_LOCK (1U << 31)
#define LCD_F_SIGHAND_LOCK (1U << 30)
+#define LCD_F_RCU_LOCK (1U << 29)
#define LCB_F_SLAB_ID_SHIFT 16
#define LCB_F_SLAB_ID_START (1U << 16)
@@ -52,9 +53,11 @@ struct contention_task_data {
#define LCB_F_RT (1U << 3)
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+#define LCB_F_SRCU (1U << 6)
+#define LCB_F_RCU (1U << 7)
-#define LCB_F_TYPE_MAX (1U << 7)
-#define LCB_F_TYPE_MASK 0x0000007FU
+#define LCB_F_TYPE_MAX (1U << 8)
+#define LCB_F_TYPE_MASK 0x000000FFU
#define SLAB_NAME_MAX 28
@@ -237,6 +240,40 @@ static __always_inline __u64 get_current_siglock(__u64 curr)
return sh + bpf_core_field_offset(sighand->siglock);
}
+/*
+ * Check if `curr` is inside a preemptible RCU read-side critical section
+ * (`rcu_read_lock()`).
+ *
+ * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires
+ * `rcu_read_lock()` before invoking `perf_sample_filter` (and
+ * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for
+ * non-clock software events), `min_depth` is the baseline nesting count
+ * contributed by the perf event overflow handler itself (1 or 2).
+ */
+static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth)
+{
+ struct task_struct *task = (void *)curr;
+
+ if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting))
+ return false;
+
+ return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth;
+}
+
+/*
+ * Check if `curr` is inside a Tasks Trace RCU read-side critical section
+ * (`rcu_read_lock_trace()`).
+ */
+static __always_inline bool is_rcu_trace_lock_held(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+
+ if (!curr || !bpf_core_field_exists(task->trc_reader_nesting))
+ return false;
+
+ return BPF_CORE_READ(task, trc_reader_nesting) > 0;
+}
+
/*
* Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar.
*
@@ -269,6 +306,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags)
if (get_current_siglock(curr) == lock)
return LCD_F_SIGHAND_LOCK;
break;
+ case LCB_F_RCU:
+ case LCB_F_RCU | LCB_F_WRITE:
+ return LCD_F_RCU_LOCK;
default:
break;
}
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index 215998897243..f6a030abfb0d 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -99,6 +99,18 @@ enum perf_bpf_lock_target_kind {
* low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
*/
PBF_LOCK_TYPE,
+ /*
+ * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`).
+ * Inspected directly via `current->rcu_read_lock_nesting > entry.target`,
+ * where `entry.target` is the baseline RCU nesting depth added by the
+ * kernel perf overflow path.
+ */
+ PBF_LOCK_RCU,
+ /*
+ * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`).
+ * Inspected directly via `current->trc_reader_nesting > 0`.
+ */
+ PBF_LOCK_RCU_TRACE,
};
/*
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index 31bce5e55b62..a94b493ee6b2 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -129,6 +129,7 @@ volatile const int use_idx_hash;
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
+const volatile __u64 rcu_state_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
@@ -142,6 +143,7 @@ int has_lock_class;
int has_lock_type;
int has_mmap_lock;
int has_siglock;
+int has_rcu_lock;
int track_rwsem_read;
__u8 lock_type_filter[LCB_F_TYPE_MAX];
@@ -297,6 +299,20 @@ static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry
case PBF_LOCK_TYPE:
return is_lock_type_in_map(entry->target, mode, pid, cpu_id);
+ case PBF_LOCK_RCU:
+ if ((mode & PBF_LOCK_MODE_READ) &&
+ is_rcu_lock_held(curr, (int)entry->target))
+ return 1;
+ if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr &&
+ is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id))
+ return 1;
+ return 0;
+
+ case PBF_LOCK_RCU_TRACE:
+ if ((mode & PBF_LOCK_MODE_READ) && is_rcu_trace_lock_held(curr))
+ return 1;
+ return 0;
+
default:
return 0;
}
@@ -718,11 +734,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla
if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
matched = true;
- if (has_mmap_lock || has_siglock) {
+ if (has_mmap_lock || has_siglock || has_rcu_lock) {
__u32 ltype = check_lock_type(lock_addr, type_flag);
if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
- (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
+ (has_siglock && ltype == LCD_F_SIGHAND_LOCK) ||
+ (has_rcu_lock && ltype == LCD_F_RCU_LOCK))
matched = true;
}
@@ -1382,6 +1399,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem)
return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
}
+/* SRCU acquire/release and synchronize hooks */
+SEC("fexit/__srcu_read_lock")
+int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/__srcu_read_unlock")
+int BPF_PROG(fentry_srcu_read_unlock, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fexit/__srcu_read_lock_nmisafe")
+int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/__srcu_read_unlock_nmisafe")
+int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/synchronize_srcu")
+int BPF_PROG(fentry_synchronize_srcu, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu")
+int BPF_PROG(fexit_synchronize_srcu, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_srcu_expedited")
+int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu_expedited")
+int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fentry/srcu_barrier")
+int BPF_PROG(fentry_srcu_barrier, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/srcu_barrier")
+int BPF_PROG(fexit_srcu_barrier, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+/* RCU grace-period synchronize / barrier hooks */
+SEC("fentry/synchronize_rcu")
+int BPF_PROG(fentry_synchronize_rcu)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu")
+int BPF_PROG(fexit_synchronize_rcu)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_rcu_expedited")
+int BPF_PROG(fentry_synchronize_rcu_expedited)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu_expedited")
+int BPF_PROG(fexit_synchronize_rcu_expedited)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_rcu_tasks")
+int BPF_PROG(fentry_synchronize_rcu_tasks)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu_tasks")
+int BPF_PROG(fexit_synchronize_rcu_tasks)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/rcu_barrier")
+int BPF_PROG(fentry_rcu_barrier)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/rcu_barrier")
+int BPF_PROG(fexit_rcu_barrier)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
SEC("raw_tp/bpf_test_finish")
int BPF_PROG(collect_lock_syms)
{
diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
index b75ddc6846bb..1fb91a829d54 100644
--- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
+++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
@@ -119,6 +119,8 @@ struct task_struct {
struct signal_struct *signal;
struct sighand_struct *sighand;
struct css_set *cgroups;
+ int rcu_read_lock_nesting;
+ int trc_reader_nesting;
} __attribute__((preserve_access_index));
struct trace_entry {
diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c
index a1bb78d6e29c..b45b075f3346 100644
--- a/tools/perf/util/lock-contention.c
+++ b/tools/perf/util/lock-contention.c
@@ -38,6 +38,10 @@ const struct perf_lock_type lock_type_table[] = {
{ LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
/* alias for optimistic spinning only */
{ LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
+ { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" },
+ { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" },
+ { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" },
+ { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" },
};
int parse_call_stack(const struct option *opt __maybe_unused, const char *str,
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index 68e9bd77e8a9..88b2c11244ad 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -128,6 +128,8 @@ struct thread_stat {
#define LCB_F_RT (1U << 3)
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+#define LCB_F_SRCU (1U << 6)
+#define LCB_F_RCU (1U << 7)
struct perf_lock_type {
unsigned int flags;
@@ -140,7 +142,7 @@ struct perf_lock_type {
const char *lock_name;
};
-extern const struct perf_lock_type lock_type_table[14];
+extern const struct perf_lock_type lock_type_table[18];
struct evlist;
struct machine;
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v4 7/7] perf lock contention: Extend BPF lock contention to RCU and SRCU
2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers
` (5 preceding siblings ...)
2026-10-19 22:09 ` [PATCH v4 6/7] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
@ 2026-10-19 22:09 ` Ian Rogers
2026-10-19 22:19 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
7 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-19 22:09 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users,
mingo, peterz, eranian, ddhariwal, rcu
Extend 'perf lock contention --use-bpf' to track RCU and SRCU
grace-period and barrier waits:
- Attach fentry/fexit programs to synchronize_rcu(),
synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(),
synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier()
when available in BTF.
- Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu',
'-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct
symbol names).
Example usage:
# perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1
total wait avg wait type caller
35.12 ms 8.78 ms rcu:W cleanup_net+0x284
7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114
# perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1
total wait avg wait address symbol
35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu)
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/tests/shell/lock_contention.sh | 14 ++
tools/perf/util/bpf_lock_contention.c | 171 ++++++++++++--
.../perf/util/bpf_skel/lock_contention.bpf.c | 221 ++++++++++++++----
tools/perf/util/machine.c | 11 +
5 files changed, 362 insertions(+), 62 deletions(-)
diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt
index c17b3e318169..bfeb98c976e3 100644
--- a/tools/perf/Documentation/perf-lock.txt
+++ b/tools/perf/Documentation/perf-lock.txt
@@ -188,11 +188,14 @@ CONTENTION OPTIONS
Show lock contention only for given lock types (comma separated list).
Available values are:
semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W,
- rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem,
- pcpu-sem:R, pcpu-sem:W, mutex
+ rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem,
+ pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W
Note that RW-variant of locks have :R and :W suffix. Names without the
suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W.
+ For RCU and SRCU with --use-bpf, contention tracks grace-period and
+ barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(),
+ srcu_barrier()).
-L::
--lock-filter=<value>::
diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh
index 5df9a0cd48a3..c6d37c910d4b 100755
--- a/tools/perf/tests/shell/lock_contention.sh
+++ b/tools/perf/tests/shell/lock_contention.sh
@@ -181,6 +181,13 @@ test_type_filter()
err=1
exit
fi
+
+ perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result}
+ if [ "$(grep -c -v rcu "${result}")" != "0" ]; then
+ echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")"
+ err=1
+ exit
+ fi
}
test_lock_filter()
@@ -224,6 +231,13 @@ test_lock_filter()
err=1
exit
fi
+
+ perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result}
+ if [ "$(grep -c -v rcu "${result}")" != "0" ]; then
+ echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")"
+ err=1
+ exit
+ fi
}
test_stack_filter()
diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c
index c20bd075664e..dfb384f1c5e6 100644
--- a/tools/perf/util/bpf_lock_contention.c
+++ b/tools/perf/util/bpf_lock_contention.c
@@ -16,6 +16,7 @@
#include <bpf/bpf.h>
#include <bpf/btf.h>
#include <inttypes.h>
+#include <unistd.h>
#include "bpf_skel/lock_contention.skel.h"
#include "bpf_skel/lock_data.h"
@@ -180,6 +181,133 @@ static void init_numa_data(struct lock_contention *con)
free(buf);
}
+static bool has_bpf_fentry_support(struct btf *btf)
+{
+ /*
+ * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and
+ * bpf_get_func_arg() helper availability (5.17+) used by
+ * get_lock_addr() in the SRCU fentry/fexit hooks.
+ */
+ struct bpf_insn insns[] = {
+ { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 },
+ { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3,
+ .src_reg = BPF_REG_10 },
+ { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 },
+ { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg },
+ { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 },
+ { .code = BPF_JMP | BPF_EXIT },
+ };
+ LIBBPF_OPTS(bpf_prog_load_opts, opts,
+ .expected_attach_type = BPF_TRACE_FENTRY,
+ );
+ s32 btf_id;
+ int prog_fd, link_fd;
+
+ if (!btf)
+ return false;
+
+ btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC);
+ if (btf_id <= 0)
+ return false;
+
+ opts.attach_btf_id = btf_id;
+ prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL",
+ insns, ARRAY_SIZE(insns), &opts);
+ if (prog_fd < 0)
+ return false;
+
+ link_fd = bpf_raw_tracepoint_open(NULL, prog_fd);
+ if (link_fd >= 0)
+ close(link_fd);
+ close(prog_fd);
+ return link_fd >= 0;
+}
+
+static void configure_rcu_contention_progs(struct lock_contention *con,
+ bool has_rcu_lock)
+{
+ static const struct {
+ size_t prog_offset;
+ const char *func;
+ bool is_srcu;
+ } rcu_progs[] = {
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu),
+ "synchronize_rcu", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu),
+ "synchronize_rcu", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited),
+ "synchronize_rcu_expedited", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited),
+ "synchronize_rcu_expedited", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks),
+ "synchronize_rcu_tasks", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks),
+ "synchronize_rcu_tasks", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier),
+ "rcu_barrier", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier),
+ "rcu_barrier", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu),
+ "synchronize_srcu", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu),
+ "synchronize_srcu", true },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited),
+ "synchronize_srcu_expedited", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited),
+ "synchronize_srcu_expedited", true },
+ { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier),
+ "srcu_barrier", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier),
+ "srcu_barrier", true },
+ };
+ struct symbol *sym;
+ struct map *kmap;
+ bool need_rcu = true, need_srcu = true;
+ bool fentry_ok;
+
+ sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap);
+ if (sym) {
+ /*
+ * Offset by 1 byte so rcu_state_addr is distinct from
+ * &rcu_state.node[0].lock at offset 0 of rcu_state.
+ */
+ skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start) + 1;
+ map__put(kmap);
+ } else {
+ skel->rodata->rcu_state_addr = 1ULL;
+ }
+
+ if (con->filters->nr_types) {
+ need_rcu = false;
+ need_srcu = false;
+ for (int i = 0; i < con->filters->nr_types; i++) {
+ if (con->filters->types[i] & LCB_F_RCU)
+ need_rcu = true;
+ if (con->filters->types[i] & LCB_F_SRCU)
+ need_srcu = true;
+ }
+ }
+
+ if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) {
+ if (!has_rcu_lock && !con->filters->nr_addrs)
+ need_rcu = false;
+ if (!con->filters->nr_addrs && !con->filters->nr_slabs)
+ need_srcu = false;
+ }
+
+ fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf);
+
+ for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) {
+ struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset;
+ bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu;
+ bool enable = fentry_ok && want &&
+ btf__find_by_name_kind(con->btf, rcu_progs[i].func,
+ BTF_KIND_FUNC) > 0;
+
+ bpf_program__set_autoload(*progp, enable);
+ }
+}
+
int lock_contention_prepare(struct lock_contention *con)
{
int i, fd;
@@ -187,6 +315,7 @@ int lock_contention_prepare(struct lock_contention *con)
struct evlist *evlist = con->evlist;
struct target *target = con->target;
bool has_mmap_lock = false;
+ bool has_rcu_lock = false;
/* make sure it loads the kernel map before lookup */
map__load(machine__kernel_map(con->machine));
@@ -200,6 +329,7 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64));
bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries);
bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries);
+ bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries);
if (con->aggr_mode == LOCK_AGGR_TASK)
bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries);
@@ -249,6 +379,10 @@ int lock_contention_prepare(struct lock_contention *con)
has_mmap_lock = true;
continue;
}
+ if (!strcmp(con->filters->syms[i], "rcu")) {
+ has_rcu_lock = true;
+ continue;
+ }
sym = machine__find_kernel_symbol_by_name(con->machine,
con->filters->syms[i],
@@ -269,7 +403,7 @@ int lock_contention_prepare(struct lock_contention *con)
addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start);
con->filters->addrs = addrs;
}
- naddrs = con->filters->nr_addrs ?: has_mmap_lock;
+ naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock);
skel->rodata->has_addr = 1;
}
@@ -305,6 +439,7 @@ int lock_contention_prepare(struct lock_contention *con)
skel->rodata->needs_callstack = con->save_callstack;
skel->rodata->lock_owner = con->owner;
skel->rodata->has_mmap_lock = has_mmap_lock;
+ skel->rodata->has_rcu_lock = has_rcu_lock;
if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) {
if (cgroup_is_v2("perf_event"))
@@ -321,6 +456,7 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map__set_max_entries(skel->maps.slab_filter, nslabs);
init_numa_data(con);
+ configure_rcu_contention_progs(con, has_rcu_lock);
if (lock_contention_bpf__load(skel) < 0) {
pr_err("Failed to load lock-contention BPF skeleton\n");
@@ -503,23 +639,30 @@ static void account_end_timestamp(struct lock_contention *con)
enum lock_aggr_mode aggr_mode = con->aggr_mode;
struct tstamp_data ts_data, *cpu_data;
- /* Iterate per-task tstamp map (key = TID) */
- ts_fd = bpf_map__fd(skel->maps.tstamp);
+ /* Iterate per-task tstamp maps (key = TID) */
+ int ts_fds[] = {
+ bpf_map__fd(skel->maps.tstamp),
+ bpf_map__fd(skel->maps.tstamp_rcu),
+ };
+
stat_fd = bpf_map__fd(skel->maps.lock_stat);
- prev_key = NULL;
- while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) {
- if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) {
- int pid = key;
+ for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) {
+ ts_fd = ts_fds[i];
+ prev_key = NULL;
+ while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) {
+ if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) {
+ int pid = key;
- if (aggr_mode == LOCK_AGGR_TASK && con->owner)
- pid = ts_data.flags;
+ if (aggr_mode == LOCK_AGGR_TASK && con->owner)
+ pid = ts_data.flags;
- update_lock_stat(stat_fd, pid, end_ts, aggr_mode,
- &ts_data);
- }
+ update_lock_stat(stat_fd, pid, end_ts, aggr_mode,
+ &ts_data);
+ }
- prev_key = &key;
+ prev_key = &key;
+ }
}
/* Now it'll check per-cpu tstamp map which doesn't have TID. */
@@ -604,6 +747,8 @@ static const char *lock_contention_get_name(struct lock_contention *con,
return "mmap_lock";
if (flags & LCD_F_SIGHAND_LOCK)
return "siglock";
+ if (flags & LCD_F_RCU_LOCK)
+ return "rcu";
/* global locks with symbols */
sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap);
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index 0f2118006005..26936f329dba 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -59,6 +59,18 @@ struct {
__uint(max_entries, MAX_ENTRIES);
} tstamp SEC(".maps");
+/*
+ * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits
+ * so they do not collide with inner sleeping lock contentions (such as
+ * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, int);
+ __type(value, struct tstamp_data);
+ __uint(max_entries, MAX_ENTRIES);
+} tstamp_rcu SEC(".maps");
+
/* maintain per-CPU timestamp at the beginning of contention */
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
@@ -160,6 +172,7 @@ const volatile int has_addr;
const volatile int has_cgroup;
const volatile int has_slab;
const volatile int has_mmap_lock;
+const volatile int has_rcu_lock;
const volatile int needs_callstack;
const volatile int stack_skip;
const volatile int lock_owner;
@@ -172,6 +185,7 @@ const volatile int aggr_mode;
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
+const volatile __u64 rcu_state_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
@@ -218,7 +232,7 @@ static inline __u64 get_current_cgroup_id(void)
return BPF_CORE_READ(cgrp, kn, id);
}
-static inline int can_record(u64 *ctx)
+static __always_inline int can_record(__u64 lock, __u32 flags)
{
bool is_addr_ok = false;
@@ -242,7 +256,6 @@ static inline int can_record(u64 *ctx)
if (has_type) {
__u8 *ok;
- __u32 flags = (__u32)ctx[1];
ok = bpf_map_lookup_elem(&type_filter, &flags);
if (!ok)
@@ -251,10 +264,9 @@ static inline int can_record(u64 *ctx)
if (has_addr) {
__u8 *ok;
- __u64 addr = ctx[0];
- ok = bpf_map_lookup_elem(&addr_filter, &addr);
- if (!ok && !has_slab && !has_mmap_lock)
+ ok = bpf_map_lookup_elem(&addr_filter, &lock);
+ if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock)
return 0;
is_addr_ok = !!ok;
@@ -272,26 +284,27 @@ static inline int can_record(u64 *ctx)
if (is_addr_ok)
return 1;
- /* slab and mmap_lock are part of the addr_filter */
+ /* slab, mmap_lock, and rcu_lock are part of the addr_filter */
if (has_slab && bpf_get_kmem_cache) {
__u8 *ok;
- __u64 addr = ctx[0];
long kmem_cache_addr;
- kmem_cache_addr = (long)bpf_get_kmem_cache(addr);
+ kmem_cache_addr = (long)bpf_get_kmem_cache(lock);
ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr);
if (ok)
return 1;
- else if (!has_mmap_lock)
+ else if (!has_mmap_lock && !has_rcu_lock)
return 0;
}
- if (has_mmap_lock) {
- __u64 lock = ctx[0];
- __u32 flag = ctx[1];
+ if (has_mmap_lock || has_rcu_lock) {
+ __u32 ltype = check_lock_type(lock, flags);
- if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK)
- return 0;
+ if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK)
+ return 1;
+ if (has_rcu_lock && ltype == LCD_F_RCU_LOCK)
+ return 1;
+ return 0;
}
return 1;
@@ -351,6 +364,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
{
__u32 pid;
struct tstamp_data *pelem;
+ void *map;
/* Use per-cpu array map for spinlock and rwlock */
if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) {
@@ -363,8 +377,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
return pelem;
}
+ map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp;
pid = bpf_get_current_pid_tgid();
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ pelem = bpf_map_lookup_elem(map, &pid);
/* Do not update the element for nested locks */
if (pelem && pelem->lock)
return NULL;
@@ -372,12 +387,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
if (pelem == NULL) {
struct tstamp_data zero = {};
- if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) {
+ if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) {
__sync_fetch_and_add(&task_fail, 1);
return NULL;
}
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ pelem = bpf_map_lookup_elem(map, &pid);
if (pelem == NULL) {
__sync_fetch_and_add(&task_fail, 1);
return NULL;
@@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags)
}
}
-SEC("tp_btf/contention_begin")
-int contention_begin(u64 *ctx)
+static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags)
{
struct tstamp_data *pelem;
- if (!enabled || !can_record(ctx))
+ if (!enabled || !can_record(lock, flags))
return 0;
- pelem = get_tstamp_elem(ctx[1]);
+ pelem = get_tstamp_elem(flags);
if (pelem == NULL)
return 0;
pelem->timestamp = bpf_ktime_get_ns();
- pelem->lock = (__u64)ctx[0];
- pelem->flags = (__u32)ctx[1];
+ pelem->lock = lock;
+ pelem->flags = flags;
if (aggr_mode == LOCK_AGGR_CGROUP)
pelem->cgroup_id = get_current_cgroup_id();
@@ -559,8 +573,21 @@ int contention_begin(u64 *ctx)
}
}
skip_owner:
- pelem->stack_id = bpf_get_stackid(ctx, &stacks,
- BPF_F_FAST_STACK_CMP | stack_skip);
+ {
+ int skip = stack_skip;
+
+ /*
+ * Unlike tp_btf/contention_begin (which fires inside a
+ * lock slowpath via trace_contention_begin), fentry
+ * RCU/SRCU programs attach directly to the top-level
+ * wait API, which is 2 frames shallower.
+ */
+ if (flags & (LCB_F_RCU | LCB_F_SRCU))
+ skip = stack_skip > 2 ? stack_skip - 2 : 0;
+
+ pelem->stack_id = bpf_get_stackid(ctx, &stacks,
+ BPF_F_FAST_STACK_CMP | skip);
+ }
if (pelem->stack_id < 0)
__sync_fetch_and_add(&stack_fail, 1);
} else if (aggr_mode == LOCK_AGGR_TASK) {
@@ -588,8 +615,7 @@ int contention_begin(u64 *ctx)
return 0;
}
-SEC("tp_btf/contention_end")
-int contention_end(u64 *ctx)
+static __always_inline int do_contention_end(__u64 lock, __s32 ret, bool is_rcu)
{
__u32 pid = 0, idx = 0;
struct tstamp_data *pelem;
@@ -597,30 +623,34 @@ int contention_end(u64 *ctx)
struct contention_data *data;
__u64 timestamp;
__u64 duration;
- bool need_delete = false;
+ void *ts_map = NULL;
if (!enabled)
return 0;
/*
- * For spinlock and rwlock, it needs to get the timestamp for the
- * per-cpu map. However, contention_end does not have the flags
- * so it cannot know whether it reads percpu or hash map.
- *
- * Try per-cpu map first and check if there's active contention.
- * If it is, do not read hash map because it cannot go to sleeping
- * locks before releasing the spinning locks.
+ * RCU/SRCU fexit hooks always use the dedicated per-task tstamp_rcu
+ * map. For tp_btf/contention_end, try the per-cpu map first (used by
+ * spinlocks and rwlocks) and fall back to the per-task tstamp map.
*/
- pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx);
- if (pelem && pelem->lock) {
- if (pelem->lock != ctx[0])
- return 0;
- } else {
+ if (is_rcu) {
pid = bpf_get_current_pid_tgid();
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
- if (!pelem || pelem->lock != ctx[0])
+ pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid);
+ if (!pelem || pelem->lock != lock)
return 0;
- need_delete = true;
+ ts_map = &tstamp_rcu;
+ } else {
+ pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx);
+ if (pelem && pelem->lock) {
+ if (pelem->lock != lock)
+ return 0;
+ } else {
+ pid = bpf_get_current_pid_tgid();
+ pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ if (!pelem || pelem->lock != lock)
+ return 0;
+ ts_map = &tstamp;
+ }
}
timestamp = bpf_ktime_get_ns();
@@ -648,7 +678,6 @@ int contention_end(u64 *ctx)
*/
else {
u32 i = 0;
- s32 ret = (s32)ctx[1];
u64 *buf;
otdata->timestamp = timestamp;
@@ -713,7 +742,7 @@ int contention_end(u64 *ctx)
if (lock_owner)
key.pid = pelem->flags;
else {
- if (!need_delete)
+ if (!ts_map)
pid = bpf_get_current_pid_tgid();
key.pid = pid;
}
@@ -795,11 +824,109 @@ int contention_end(u64 *ctx)
check_lock_delay(pelem->lock);
pelem->lock = 0;
- if (need_delete)
- bpf_map_delete_elem(&tstamp, &pid);
+ if (ts_map)
+ bpf_map_delete_elem(ts_map, &pid);
return 0;
}
+SEC("tp_btf/contention_begin")
+int contention_begin(u64 *ctx)
+{
+ return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]);
+}
+
+SEC("tp_btf/contention_end")
+int contention_end(u64 *ctx)
+{
+ return do_contention_end((__u64)ctx[0], (__s32)ctx[1], false);
+}
+
+/* RCU grace-period synchronize / barrier contention hooks */
+SEC("fentry/synchronize_rcu")
+int BPF_PROG(fentry_synchronize_rcu)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu")
+int BPF_PROG(fexit_synchronize_rcu)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+SEC("fentry/synchronize_rcu_expedited")
+int BPF_PROG(fentry_synchronize_rcu_expedited)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu_expedited")
+int BPF_PROG(fexit_synchronize_rcu_expedited)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+SEC("fentry/synchronize_rcu_tasks")
+int BPF_PROG(fentry_synchronize_rcu_tasks)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu_tasks")
+int BPF_PROG(fexit_synchronize_rcu_tasks)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+SEC("fentry/rcu_barrier")
+int BPF_PROG(fentry_rcu_barrier)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/rcu_barrier")
+int BPF_PROG(fexit_rcu_barrier)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+/* SRCU grace-period synchronize / barrier contention hooks */
+SEC("fentry/synchronize_srcu")
+int BPF_PROG(fentry_synchronize_srcu, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu")
+int BPF_PROG(fexit_synchronize_srcu, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0, true);
+}
+
+SEC("fentry/synchronize_srcu_expedited")
+int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu_expedited")
+int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0, true);
+}
+
+SEC("fentry/srcu_barrier")
+int BPF_PROG(fentry_srcu_barrier, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/srcu_barrier")
+int BPF_PROG(fexit_srcu_barrier, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0, true);
+}
+
SEC("raw_tp/bpf_test_finish")
int BPF_PROG(end_timestamp)
{
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 1d9d3bf57720..0a0f223b3335 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -3379,6 +3379,17 @@ bool machine__is_lock_function(struct machine *machine, u64 addr)
return true;
}
+ {
+ struct symbol *sym = machine__find_kernel_symbol(machine, addr, NULL);
+
+ if (sym && (!strncmp(sym->name, "synchronize_rcu", 15) ||
+ !strncmp(sym->name, "synchronize_srcu", 16) ||
+ !strcmp(sym->name, "__synchronize_srcu") ||
+ !strcmp(sym->name, "rcu_barrier") ||
+ !strcmp(sym->name, "srcu_barrier")))
+ return true;
+ }
+
return false;
}
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* Re: [PATCH v4 5/7] perf test: Add tests for lock_held() BPF sample filter
2026-10-19 22:09 ` [PATCH v4 5/7] perf test: Add tests for lock_held() BPF " Ian Rogers
@ 2026-10-19 22:12 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-19 22:12 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Add unit tests in parse-events.c for parsing lock_held(),
> lock_held_read(), and lock_held_exclusive() BPF filter expressions
> (standalone predicates, negated predicates, and compound group
> expressions) and extend record_bpf_filter.sh with lock_held(),
> lock_held_read()/lock_held_write(), and ':R'/':W' recording tests.
>
> Example output:
>
> # perf test -v 'Parsing of BPF sample filters'
> 5: Parse event definition strings :
> 5.7: Parsing of BPF sample filters : Ok
>
> # perf test -v 'perf record sample filtering (by BPF) tests'
> 112: perf record sample filtering (by BPF) tests :
> Checking BPF-filter privilege
> Basic bpf-filter test
> Basic bpf-filter test [Success]
> Failing bpf-filter test
> Failing bpf-filter test [Success]
> Group bpf-filter test
> Group bpf-filter test [Success]
> Multiple bpf-filter test
> Multiple bpf-filter test [Success]
> Cgroup bpf-filter test
> Cgroup bpf-filter test [Success]
> Lock-held bpf-filter test
> Lock-held bpf-filter test [Success]
> 112: perf record sample filtering (by BPF) tests : Ok
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019220915.3943592-1-irogers@google.com?part=5
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat
2026-10-19 22:09 ` [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat Ian Rogers
@ 2026-10-19 22:14 ` sashiko-bot
2026-10-20 4:34 ` Namhyung Kim
1 sibling, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-19 22:14 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> In report_lock_contention_begin_event(), lock_stat_findnew() was
> called before checking filters.nr_types (-Y), filters.nr_addrs (-L),
> and match_callstack_filter() (-S).
>
> When a filtered-out lock event (such as a mutex contention when
> running with -Y spinlock) shared the same aggregation key as a
> subsequent matching event (for example, contention on the mutex's
> internal wait_lock spinlock from the same caller), the first event
> created the lock_stat entry with the filtered-out lock's flags and the
> second event reused that entry, causing non-spinlock entries to appear
> in -Y spinlock output.
>
> Check the type, address, and callstack filters before looking up or
> creating the lock_stat entry.
>
> Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option")
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019220915.3943592-1-irogers@google.com?part=1
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v4 3/7] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-19 22:09 ` [PATCH v4 3/7] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
@ 2026-10-19 22:17 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-19 22:17 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers
> (get_lock_owner, get_current_mmap_lock, get_current_siglock,
> get_lock_addr, check_lock_type, do_collect_zone_lock,
> do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into
> tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF
> sample filtering.
>
> Also add rw_semaphore.count and struct rt_mutex_base to the minimal
> vmlinux.h and extend get_lock_owner() to support rt_mutex and check
> the mutex handoff flag.
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019220915.3943592-1-irogers@google.com?part=3
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v4 6/7] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
2026-10-19 22:09 ` [PATCH v4 6/7] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
@ 2026-10-19 22:19 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-19 22:19 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks
> Trace RCU, and SRCU locks:
>
> - 'rcu' (supports both read/shared and write/exclusive modes):
> - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)',
> 'lock_held(rcu_read_lock)'): directly inspects
> current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on
> CONFIG_PREEMPT_RCU kernels), accounting for the baseline
> rcu_read_lock() nesting acquired by the perf overflow handler itself
> (bpf_overflow_handler() and do_perf_sw_event()) with zero extra
> fentry/fexit tracing hooks.
> - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'):
> attaches fentry/fexit hooks to synchronize_rcu(),
> synchronize_rcu_expedited(), synchronize_rcu_tasks(), and
> rcu_barrier() to track grace-period and barrier waits.
> [ ... ]
>
> # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1
> # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019220915.3943592-1-irogers@google.com?part=6
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v4 7/7] perf lock contention: Extend BPF lock contention to RCU and SRCU
2026-10-19 22:09 ` [PATCH v4 7/7] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
@ 2026-10-19 22:19 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-19 22:19 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Extend 'perf lock contention --use-bpf' to track RCU and SRCU
> grace-period and barrier waits:
>
> - Attach fentry/fexit programs to synchronize_rcu(),
> synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(),
> synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier()
> when available in BTF.
> - Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu',
> '-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct
> symbol names).
>
> Example usage:
>
> # perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1
> total wait avg wait type caller
>
> 35.12 ms 8.78 ms rcu:W cleanup_net+0x284
> 7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114
>
> # perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1
> total wait avg wait address symbol
>
> 35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu)
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019220915.3943592-1-irogers@google.com?part=7
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events
2026-10-19 22:09 ` [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
@ 2026-10-19 22:26 ` sashiko-bot
2026-10-20 5:41 ` Namhyung Kim
1 sibling, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-19 22:26 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] perf bpf-filter: double close of tracepoint fd via perf_bpf_filter__prepare()
- [Medium] perf bpf-filter: missing tracepoint BPF terms in is_possible_tp_filter()
--
commit d87f98375f9a6f9592030472ebfbf88f3edd6fdd
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Support BPF sample filters on tracepoint events
This commit adds a tracepoint BPF program (perf_tp_sample_filter) to
evaluate context-independent filter terms (uid, gid, tid, cpu, time,
cgroup) on PERF_TYPE_TRACEPOINT events, and attaches it once to the
shared trace_event_call in perf_bpf_filter__prepare().
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 72e83ac1d6a4b..4e99af4764f1d 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
> @@ -123,15 +123,26 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
> {
> const struct perf_sample_info *info;
>
> - if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END &&
> - (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
> - return 0;
> -
> if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
> /* Not dependent on the sample_type as computed from a BPF helper. */
> return 0;
> }
>
> + if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT) {
> + if (expr->term == PBF_TERM_TID || expr->term == PBF_TERM_CPU ||
> + expr->term == PBF_TERM_TIME || expr->term == PBF_TERM_CGROUP)
> + return 0;
[Severity: Medium]
Can standalone tracepoint BPF filters using gid, tid, pid, cpu, time, or
cgroup actually reach perf_bpf_filter__parse() from --filter?
In tools/perf/util/parse-events.c, set_filter() checks
is_possible_tp_filter(str) before deciding whether to route a
PERF_TYPE_TRACEPOINT filter to evsel__append_tp_filter() or
perf_bpf_filter__parse():
tools/perf/util/parse-events.c:is_possible_tp_filter() {
return strstr(str, "uid") == NULL;
}
tools/perf/util/parse-events.c:set_filter() {
...
if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT && is_possible_tp_filter(str)) {
if (evsel__append_tp_filter(evsel, str) < 0) {
...
}
Because is_possible_tp_filter() only checks for "uid" in this commit,
passing --filter 'gid == 0' or --filter 'cpu == 0' on a tracepoint event
routes the expression to the ftrace tracepoint predicate filter via
evsel__append_tp_filter(), which fails with -EINVAL in
PERF_EVENT_IOC_SET_FILTER when the tracepoint lacks matching format fields.
Although a later commit in the series ("perf bpf-filter: Add lock_held()
sample filter") adds "gid" and "lock_held" to is_possible_tp_filter(), the
tid, pid, cpu, time, and cgroup terms supported here in check_sample_flags()
and perf_get_common_sample() still remain unrouted to perf_bpf_filter__parse()
unless combined with uid, gid, or lock_held in the same filter string.
Should is_possible_tp_filter() also check for gid, tid, pid, cpu, time, and
cgroup?
> + if (expr->op != PBF_OP_GROUP_BEGIN) {
> + pr_err("Error: %s tracepoint event does not support BPF sample term %d\n",
> + evsel__name(evsel), expr->term);
> + return -1;
> + }
> + }
[ ... ]
> @@ -501,15 +528,35 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> goto err;
> }
>
> - prog = skel->progs.perf_sample_filter;
> - for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
> - for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
> - link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y),
> - &pe_opts);
> - if (IS_ERR(link)) {
> - pr_err("Failed to attach perf sample-filter program\n");
> - ret = PTR_ERR(link);
> - goto err;
> + if (is_tracepoint) {
> + /*
> + * For PERF_TYPE_TRACEPOINT, PERF_EVENT_IOC_SET_BPF attaches the
> + * BPF program to event->tp_event->prog_array on the shared
> + * trace_event_call, which perf_trace_run_bpf_submit() executes
> + * for all CPUs and tasks before iterating the per-CPU perf
> + * events. Attach once on the first FD so tp_event->prog_array
> + * does not get duplicate copies for each per-CPU FD.
> + */
> + prog = skel->progs.perf_tp_sample_filter;
> + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, 0, 0),
> + &pe_opts);
> + ret = libbpf_get_error(link);
> + if (ret) {
> + pr_err("Failed to attach perf tracepoint sample-filter program\n");
> + goto err;
> + }
> + skel->links.perf_tp_sample_filter = link;
[Severity: Medium]
Does storing link in skel->links.perf_tp_sample_filter cause FD(evsel, 0, 0)
to be closed twice during teardown?
In tools/lib/bpf/libbpf.c, bpf_program__attach_perf_event_opts() takes
ownership of the passed perf_event_fd without duplicating it, and its
detach callback closes that descriptor when the bpf_link is destroyed:
tools/lib/bpf/libbpf.c:bpf_link_perf_detach() {
struct bpf_link_perf *perf_link = container_of(link, struct bpf_link_perf, link);
int err = 0;
if (ioctl(perf_link->perf_event_fd, PERF_EVENT_IOC_DISABLE, 0) < 0)
err = -errno;
if (perf_link->perf_event_fd != link->fd)
close(perf_link->perf_event_fd);
close(link->fd);
...
}
During teardown (for example in evlist__put()), evlist__close() closes all
descriptors in evsel->core.fd, including FD(evsel, 0, 0), before
evlist__purge() destroys the evsels:
evlist__put()
-> evlist__close()
-> perf_evsel__close_cpu()
-> perf_evsel__close_fd_cpu() /* closes FD(evsel, 0, 0) */
-> evlist__purge()
-> evsel__put()
-> evsel__exit()
-> perf_bpf_filter__destroy()
-> sample_filter_bpf__destroy()
-> bpf_link__destroy(skel->links.perf_tp_sample_filter)
-> bpf_link_perf_detach() /* ioctls and closes FD(evsel, 0, 0) again */
In the non-tracepoint branch below, the returned link is not stored in
skel->links, or if skel->links.perf_tp_sample_filter is kept, should
FD(evsel, 0, 0) be duplicated before passing it to
bpf_program__attach_perf_event_opts()?
[ ... ]
>
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019220915.3943592-1-irogers@google.com?part=2
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v4 4/7] perf bpf-filter: Add lock_held() sample filter
2026-10-19 22:09 ` [PATCH v4 4/7] perf bpf-filter: Add lock_held() sample filter Ian Rogers
@ 2026-10-19 22:28 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-19 22:28 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf bpf-filter: wrong lock suffix match via strtailcmp() on short names
--
commit 30b0f8cede3e3d90dc8406410f23666b838bbc6b
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Add lock_held() sample filter
This commit adds BPF sample filter predicates lock_held(), lock_held_read(),
lock_held_shared(), lock_held_write(), and lock_held_exclusive() to record
perf event samples only when a specified kernel lock is held. It evaluates
lock ownership directly in kernel memory where possible and falls back to an
LRU hash map populated by fentry/fexit hooks on lock/unlock functions.
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 4e99af4764f1d..ab9b58f6cb713 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
[ ... ]
> @@ -166,9 +182,844 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
[ ... ]
> +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
> + bool *has_lock_filter,
> + unsigned int *prog_groups,
> + unsigned int *required_groups,
> + bool *needs_lock_syms,
> + bool *needs_slab_iter)
> +{
[ ... ]
> + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0 ||
> + strtailcmp(base, "mutex") == 0) {
> + /* Inspected directly via lock->owner; no tracing hooks needed. */
> + return;
> + } else if (strtailcmp(base, "percpu_rwsem") == 0 ||
> + strtailcmp(base, "pcpu_rwsem") == 0) {
> + *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
> + *required_groups |= LOCK_PROG_PERCPU_RWSEM;
> + } else if (strtailcmp(base, "rwsem") == 0) {
> + if (mode & PBF_LOCK_MODE_READ) {
> + *prog_groups |= LOCK_PROG_RWSEM_READ;
> + *required_groups |= LOCK_PROG_RWSEM_READ;
> + }
> + if (mode & PBF_LOCK_MODE_WRITE) {
> + *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM;
> + *required_groups |= LOCK_PROG_RWSEM_WRITE;
> + }
> + } else if (strtailcmp(base, "spinlock") == 0) {
> + *prog_groups |= LOCK_PROG_SPIN;
> + *required_groups |= LOCK_PROG_SPIN;
> + } else if (strtailcmp(base, "sem") == 0) {
> + *prog_groups |= LOCK_PROG_SEM;
> + *required_groups |= LOCK_PROG_SEM;
> + } else if (strtailcmp(base, "rwlock") == 0) {
> + if (mode & PBF_LOCK_MODE_READ)
> + groups |= LOCK_PROG_RWLOCK_READ;
> + if (mode & PBF_LOCK_MODE_WRITE)
> + groups |= LOCK_PROG_RWLOCK_WRITE;
> + *prog_groups |= groups;
> + *required_groups |= groups;
> + } else {
> + *prog_groups |= LOCK_PROG_ALL;
> + }
> +}
[ ... ]
> +static int resolve_lock_expr(struct sample_filter_bpf *skel,
> + struct perf_bpf_filter_expr *expr,
> + struct perf_bpf_filter_entry *entry)
> +{
[ ... ]
> + ret = find_kernel_lock_sym(base, &addr);
> + if (ret == 0) {
> + unsigned int direct_flag = 0;
> +
> + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0)
> + direct_flag = PBF_LOCK_DIRECT_RTMUTEX;
> + else if (strtailcmp(base, "mutex") == 0)
> + direct_flag = PBF_LOCK_DIRECT_MUTEX;
> + else if (strtailcmp(base, "rwsem") == 0 &&
> + strtailcmp(base, "percpu_rwsem") != 0 &&
> + strtailcmp(base, "pcpu_rwsem") != 0)
> + direct_flag = PBF_LOCK_DIRECT_RWSEM;
> +
> + if ((direct_flag == PBF_LOCK_DIRECT_MUTEX ||
> + direct_flag == PBF_LOCK_DIRECT_RTMUTEX ||
> + strtailcmp(base, "percpu_rwsem") == 0 ||
> + strtailcmp(base, "pcpu_rwsem") == 0 ||
> + strtailcmp(base, "spinlock") == 0 ||
> + (strtailcmp(base, "sem") == 0 &&
> + strtailcmp(base, "rwsem") != 0)) &&
> + !(mode & PBF_LOCK_MODE_WRITE)) {
> + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
> + name);
> + return -EINVAL;
> + }
[Severity: Medium]
Can strtailcmp() in prescan_one_lock_expr() and resolve_lock_expr()
misclassify short kernel lock symbol names?
In tools/perf/util/string.c, strtailcmp() compares backward from the end of
both strings and returns 0 as soon as either string reaches the start:
tools/perf/util/string.c:strtailcmp() {
int i1 = strlen(s1);
int i2 = strlen(s2);
while (--i1 >= 0 && --i2 >= 0) {
if (s1[i1] != s2[i2])
return s1[i1] - s2[i2];
}
return 0;
}
Because strtailcmp(base, suffix) does not require base to be at least as long
as suffix, a lock symbol in kallsyms that is shorter than suffix and matches
its tail will return 0:
- A lock symbol named "lock" (such as a static mutex or rwsem named lock)
matches strtailcmp("lock", "spinlock") == 0, causing prescan_one_lock_expr()
to only enable LOCK_PROG_SPIN instead of LOCK_PROG_ALL and
resolve_lock_expr() to reject lock_held_read(lock) with -EINVAL.
- A short rwsem symbol such as "cpu_rwsem" or "_rwsem" matches
strtailcmp(base, "percpu_rwsem") == 0, attaching LOCK_PROG_PERCPU_RWSEM
instead of LOCK_PROG_RWSEM_READ/WRITE, omitting PBF_LOCK_DIRECT_RWSEM, and
rejecting read-mode filtering with -EINVAL.
- A short mutex symbol such as "_mutex" matches
strtailcmp(base, "rt_mutex") == 0, setting PBF_LOCK_DIRECT_RTMUTEX instead
of PBF_LOCK_DIRECT_MUTEX.
Should this use a helper that checks strlen(base) >= strlen(suffix) before
comparing the suffix?
> +
> + entry->part = PBF_LOCK_ADDR | mode | direct_flag;
> + entry->target = addr;
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261019220915.3943592-1-irogers@google.com?part=4
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat
2026-10-19 22:09 ` [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat Ian Rogers
2026-10-19 22:14 ` sashiko-bot
@ 2026-10-20 4:34 ` Namhyung Kim
2026-10-21 21:39 ` Arnaldo Carvalho de Melo
1 sibling, 1 reply; 106+ messages in thread
From: Namhyung Kim @ 2026-10-20 4:34 UTC (permalink / raw)
To: Ian Rogers
Cc: acme, adrian.hunter, james.clark, jolsa, linux-kernel,
linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu
On Mon, Oct 19, 2026 at 03:09:09PM -0700, Ian Rogers wrote:
> In report_lock_contention_begin_event(), lock_stat_findnew() was
> called before checking filters.nr_types (-Y), filters.nr_addrs (-L),
> and match_callstack_filter() (-S).
>
> When a filtered-out lock event (such as a mutex contention when
> running with -Y spinlock) shared the same aggregation key as a
> subsequent matching event (for example, contention on the mutex's
> internal wait_lock spinlock from the same caller), the first event
> created the lock_stat entry with the filtered-out lock's flags and the
> second event reused that entry, causing non-spinlock entries to appear
> in -Y spinlock output.
>
> Check the type, address, and callstack filters before looking up or
> creating the lock_stat entry.
>
> Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option")
> Signed-off-by: Ian Rogers <irogers@google.com>
Reviewed-by: Namhyung Kim <namhyung@kernel.org>
Thanks,
Namhyung
> ---
> tools/perf/builtin-lock.c | 61 +++++++++++++++++++++------------------
> 1 file changed, 33 insertions(+), 28 deletions(-)
>
> diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
> index d5c0d55cb82d..778923c92f42 100644
> --- a/tools/perf/builtin-lock.c
> +++ b/tools/perf/builtin-lock.c
> @@ -975,6 +975,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
> struct machine *machine = &session->machines.host;
> struct map *kmap;
> struct symbol *sym;
> + u64 *callstack = NULL;
>
> ret = get_key_by_aggr_mode(&key, addr, sample);
> if (ret < 0)
> @@ -1010,33 +1011,6 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
> }
> }
>
> - ls = lock_stat_find(key);
> - if (!ls) {
> - char buf[128];
> - const char *name = "";
> -
> - switch (aggr_mode) {
> - case LOCK_AGGR_ADDR:
> - sym = machine__find_kernel_symbol(machine, key, &kmap);
> - if (sym)
> - name = sym->name;
> - break;
> - case LOCK_AGGR_CALLER:
> - name = buf;
> - if (lock_contention_caller(sample, buf, sizeof(buf)) < 0)
> - name = "Unknown";
> - break;
> - case LOCK_AGGR_CGROUP:
> - case LOCK_AGGR_TASK:
> - default:
> - break;
> - }
> -
> - ls = lock_stat_findnew(key, name, flags);
> - if (!ls)
> - return -ENOMEM;
> - }
> -
> if (filters.nr_types) {
> bool found = false;
>
> @@ -1066,7 +1040,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
> }
>
> if (needs_callstack()) {
> - u64 *callstack = get_callstack(sample, max_stack_depth);
> + callstack = get_callstack(sample, max_stack_depth);
> if (callstack == NULL)
> return 0;
>
> @@ -1074,7 +1048,38 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
> free(callstack);
> return 0;
> }
> + }
> +
> + ls = lock_stat_find(key);
> + if (!ls) {
> + char buf[128];
> + const char *name = "";
> +
> + switch (aggr_mode) {
> + case LOCK_AGGR_ADDR:
> + sym = machine__find_kernel_symbol(machine, key, &kmap);
> + if (sym)
> + name = sym->name;
> + break;
> + case LOCK_AGGR_CALLER:
> + name = buf;
> + if (lock_contention_caller(sample, buf, sizeof(buf)) < 0)
> + name = "Unknown";
> + break;
> + case LOCK_AGGR_CGROUP:
> + case LOCK_AGGR_TASK:
> + default:
> + break;
> + }
> +
> + ls = lock_stat_findnew(key, name, flags);
> + if (!ls) {
> + free(callstack);
> + return -ENOMEM;
> + }
> + }
>
> + if (callstack) {
> if (ls->callstack == NULL)
> ls->callstack = callstack;
> else
> --
> 2.56.0.212.g129ede20d9-goog
>
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events
2026-10-19 22:09 ` [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
2026-10-19 22:26 ` sashiko-bot
@ 2026-10-20 5:41 ` Namhyung Kim
2026-10-20 15:30 ` Ian Rogers
1 sibling, 1 reply; 106+ messages in thread
From: Namhyung Kim @ 2026-10-20 5:41 UTC (permalink / raw)
To: Ian Rogers
Cc: acme, adrian.hunter, james.clark, jolsa, linux-kernel,
linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu
On Mon, Oct 19, 2026 at 03:09:10PM -0700, Ian Rogers wrote:
> Commit 466db4275edd ("perf parse-events: Add parse_uid_filter helper")
> updated set_filter() to route non-tracepoint filter terms (such as uid
> and gid) on PERF_TYPE_TRACEPOINT events to evsel->bpf_filters instead
> of the ftrace tracepoint predicate filter.
>
> However, in the kernel's __perf_event_set_bpf_prog()
> (kernel/events/core.c), attaching a BPF program to a perf_event fd
> branches on perf_event_is_tracing(event):
> - Non-tracing events (hardware PMU, software events, breakpoints) call
> perf_event_set_bpf_handler(), which requires BPF_PROG_TYPE_PERF_EVENT
> (SEC("perf_event")).
> - Tracepoint events (event->pmu == &perf_tracepoint) reject
> BPF_PROG_TYPE_PERF_EVENT with -EINVAL and instead require
> BPF_PROG_TYPE_TRACEPOINT (SEC("tracepoint")), which is attached to
> event->tp_event->prog_array and executed in
> perf_trace_run_bpf_submit() before perf_tp_event(), dropping the
> sample if the BPF program returns 0. Furthermore, the
> bpf_cast_to_kern_ctx() kfunc is only permitted in
> BPF_PROG_TYPE_PERF_EVENT programs.
>
> Add SEC("tracepoint") int perf_tp_sample_filter(void *ctx) in
> sample_filter.bpf.c to evaluate context-independent filter terms (uid,
> gid, tid, cpu, time, cgroup) using BPF helpers without calling
> bpf_cast_to_kern_ctx(). When preparing a BPF filter on a
> PERF_TYPE_TRACEPOINT evsel, autoload perf_tp_sample_filter instead of
> perf_sample_filter and attach it once to the shared trace_event_call.
>
> Before:
> # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true
> libbpf: prog 'perf_sample_filter': failed to attach to perf_event FD 3: Invalid argument
> Failed to attach perf sample-filter program
> failed to set filter "BPF" on event kmem:mm_page_alloc with 22 (Invalid argument)
>
> After:
> # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.024 MB perf.data (42 samples) ]
>
> Fixes: 466db4275edd ("perf parse-events: Add parse_uid_filter helper")
> Signed-off-by: Ian Rogers <irogers@google.com>
> ---
> tools/perf/util/bpf-filter.c | 136 ++++++++++++++++---
> tools/perf/util/bpf_skel/sample_filter.bpf.c | 85 ++++++++++++
> 2 files changed, 201 insertions(+), 20 deletions(-)
>
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 72e83ac1d6a4..4e99af4764f1 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
> @@ -123,15 +123,26 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
> {
> const struct perf_sample_info *info;
>
> - if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END &&
> - (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
> - return 0;
> -
> if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
> /* Not dependent on the sample_type as computed from a BPF helper. */
> return 0;
> }
>
> + if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT) {
> + if (expr->term == PBF_TERM_TID || expr->term == PBF_TERM_CPU ||
> + expr->term == PBF_TERM_TIME || expr->term == PBF_TERM_CGROUP)
> + return 0;
> + if (expr->op != PBF_OP_GROUP_BEGIN) {
> + pr_err("Error: %s tracepoint event does not support BPF sample term %d\n",
> + evsel__name(evsel), expr->term);
> + return -1;
> + }
> + }
> +
> + if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END &&
> + (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
> + return 0;
> +
> if (expr->op == PBF_OP_GROUP_BEGIN) {
> struct perf_bpf_filter_expr *group;
>
> @@ -427,6 +438,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> struct bpf_link *link;
> struct perf_bpf_filter_entry *entry;
> bool needs_idx_hash = !target__has_cpu(target);
> + bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
> #if LIBBPF_CURRENT_VERSION_GEQ(1, 7)
> DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts,
> .dont_enable = true);
> @@ -444,9 +456,15 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> goto err;
> }
>
> - if (needs_idx_hash && geteuid() != 0) {
> + if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
> int zero = 0;
>
> + if (is_tracepoint) {
> + pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n");
> + ret = -EPERM;
> + goto err;
> + }
> +
> /* The filters map is shared among other processes */
> ret = create_idx_hash(evsel, entry);
> if (ret < 0)
> @@ -484,8 +502,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> return 0;
> }
>
> - skel = sample_filter_bpf__open_and_load();
> + skel = sample_filter_bpf__open();
> if (!skel) {
> + ret = -errno;
> + pr_err("Failed to open perf sample-filter BPF skeleton\n");
> + goto err;
> + }
> +
> + bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
> + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
> +
> + if (sample_filter_bpf__load(skel) < 0) {
> ret = -errno;
> pr_err("Failed to load perf sample-filter BPF skeleton\n");
> goto err;
> @@ -501,15 +528,35 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> goto err;
> }
>
> - prog = skel->progs.perf_sample_filter;
> - for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
> - for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
> - link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y),
> - &pe_opts);
> - if (IS_ERR(link)) {
> - pr_err("Failed to attach perf sample-filter program\n");
> - ret = PTR_ERR(link);
> - goto err;
> + if (is_tracepoint) {
> + /*
> + * For PERF_TYPE_TRACEPOINT, PERF_EVENT_IOC_SET_BPF attaches the
> + * BPF program to event->tp_event->prog_array on the shared
> + * trace_event_call, which perf_trace_run_bpf_submit() executes
> + * for all CPUs and tasks before iterating the per-CPU perf
> + * events. Attach once on the first FD so tp_event->prog_array
> + * does not get duplicate copies for each per-CPU FD.
> + */
> + prog = skel->progs.perf_tp_sample_filter;
> + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, 0, 0),
> + &pe_opts);
> + ret = libbpf_get_error(link);
> + if (ret) {
> + pr_err("Failed to attach perf tracepoint sample-filter program\n");
> + goto err;
> + }
> + skel->links.perf_tp_sample_filter = link;
> + } else {
> + prog = skel->progs.perf_sample_filter;
> + for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
> + for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
> + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y),
> + &pe_opts);
This link should be saves somewhere. I've sent a fix for it.
> + ret = libbpf_get_error(link);
> + if (ret) {
> + pr_err("Failed to attach perf sample-filter program\n");
> + goto err;
> + }
> }
> }
> }
> @@ -641,6 +688,45 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
> return ret;
> }
>
> +static int pin_filter_object(struct sample_filter_bpf *skel, const char *path)
> +{
> + char *prog_path = NULL;
> + int ret;
> +
> + ret = bpf_object__pin_maps(skel->obj, path);
> + if (ret < 0)
> + return ret;
> +
> + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
> + bpf_object__unpin_maps(skel->obj, path);
> + return -ENOMEM;
> + }
> +
> + ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path);
> + if (ret < 0)
> + bpf_object__unpin_maps(skel->obj, path);
> +
> + free(prog_path);
> + return ret;
> +}
> +
> +static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path)
> +{
> + char *prog_path = NULL;
> + int ret, map_ret;
> +
> + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
> + bpf_object__unpin_maps(skel->obj, path);
> + return -ENOMEM;
> + }
> +
> + ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path);
> + free(prog_path);
> +
> + map_ret = bpf_object__unpin_maps(skel->obj, path);
> + return ret < 0 ? ret : map_ret;
> +}
> +
> int perf_bpf_filter__pin(void)
> {
> struct sample_filter_bpf *skel;
> @@ -655,6 +741,7 @@ int perf_bpf_filter__pin(void)
> }
>
> /* pinned program will use pid-hash */
> + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
> bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
> bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH);
> bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH);
> @@ -674,7 +761,7 @@ int perf_bpf_filter__pin(void)
> goto err;
> }
>
> - ret = bpf_object__pin(skel->obj, path);
> + ret = pin_filter_object(skel, path);
Why is this change needed?
Thanks,
Namhyung
> if (ret < 0) {
> pr_err("Failed to pin BPF filter objects\n");
> goto err;
> @@ -683,8 +770,8 @@ int perf_bpf_filter__pin(void)
> /* setup access permissions for the pinned objects */
> dir_fd = open(path, O_PATH);
> if (dir_fd < 0) {
> - bpf_object__unpin(skel->obj, path);
> - ret = dir_fd;
> + ret = -errno;
> + unpin_filter_object(skel, path);
> goto err;
> }
>
> @@ -727,6 +814,8 @@ int perf_bpf_filter__pin(void)
>
> err_close:
> close(dir_fd);
> + if (ret < 0)
> + unpin_filter_object(skel, path);
>
> err:
> free(path);
> @@ -740,13 +829,20 @@ int perf_bpf_filter__unpin(void)
> char *path = NULL;
> int ret = -1;
>
> - skel = sample_filter_bpf__open_and_load();
> + skel = sample_filter_bpf__open();
> if (!skel) {
> ret = -errno;
> pr_err("Failed to open perf sample-filter BPF skeleton\n");
> goto err;
> }
>
> + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
> + if (sample_filter_bpf__load(skel) < 0) {
> + ret = -errno;
> + pr_err("Failed to load perf sample-filter BPF skeleton\n");
> + goto err;
> + }
> +
> if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(),
> PERF_BPF_FILTER_PIN_PATH) < 0) {
> ret = -errno;
> @@ -754,7 +850,7 @@ int perf_bpf_filter__unpin(void)
> goto err;
> }
>
> - ret = bpf_object__unpin(skel->obj, path);
> + ret = unpin_filter_object(skel, path);
>
> err:
> free(path);
> diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
> index e5666d4c1722..576fbed270e5 100644
> --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
> +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
> @@ -192,6 +192,28 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
> return 0;
> }
>
> +static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry *entry)
> +{
> + switch (entry->term) {
> + case PBF_TERM_TID:
> + if (entry->part)
> + return bpf_get_current_pid_tgid() >> 32;
> + return (__u32)bpf_get_current_pid_tgid();
> + case PBF_TERM_CPU:
> + return bpf_get_smp_processor_id();
> + case PBF_TERM_TIME:
> + return bpf_ktime_get_ns();
> + case PBF_TERM_CGROUP:
> + return bpf_get_current_cgroup_id();
> + case PBF_TERM_UID:
> + return bpf_get_current_uid_gid() & 0xFFFFFFFF;
> + case PBF_TERM_GID:
> + return bpf_get_current_uid_gid() >> 32;
> + default:
> + return 0;
> + }
> +}
> +
> #define CHECK_RESULT(data, op, val) \
> if (!(data op val)) { \
> if (!in_group) \
> @@ -295,4 +317,67 @@ int perf_sample_filter(void *ctx)
> return 0;
> }
>
> +/* BPF program to be called for tracepoint perf events */
> +SEC("tracepoint")
> +int perf_tp_sample_filter(void *ctx)
> +{
> + struct perf_bpf_filter_entry *entry;
> + __u64 sample_data;
> + int in_group = 0;
> + int group_result = 0;
> + int i, k = 0;
> + int *losts;
> +
> + entry = bpf_map_lookup_elem(&filters, &k);
> + if (entry == NULL)
> + goto drop;
> +
> + for (i = 0; i < MAX_FILTERS; i++) {
> + sample_data = perf_get_common_sample(&entry[i]);
> +
> + switch (entry[i].op) {
> + case PBF_OP_EQ:
> + CHECK_RESULT(sample_data, ==, entry[i].value)
> + break;
> + case PBF_OP_NEQ:
> + CHECK_RESULT(sample_data, !=, entry[i].value)
> + break;
> + case PBF_OP_GT:
> + CHECK_RESULT(sample_data, >, entry[i].value)
> + break;
> + case PBF_OP_GE:
> + CHECK_RESULT(sample_data, >=, entry[i].value)
> + break;
> + case PBF_OP_LT:
> + CHECK_RESULT(sample_data, <, entry[i].value)
> + break;
> + case PBF_OP_LE:
> + CHECK_RESULT(sample_data, <=, entry[i].value)
> + break;
> + case PBF_OP_AND:
> + CHECK_RESULT(sample_data, &, entry[i].value)
> + break;
> + case PBF_OP_GROUP_BEGIN:
> + in_group = 1;
> + group_result = 0;
> + break;
> + case PBF_OP_GROUP_END:
> + if (group_result == 0)
> + goto drop;
> + in_group = 0;
> + break;
> + case PBF_OP_DONE:
> + return 1;
> + }
> + }
> + return 1;
> +
> +drop:
> + losts = bpf_map_lookup_elem(&dropped, &k);
> + if (losts != NULL)
> + __sync_fetch_and_add(losts, 1);
> +
> + return 0;
> +}
> +
> char LICENSE[] SEC("license") = "Dual BSD/GPL";
> --
> 2.56.0.212.g129ede20d9-goog
>
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events
2026-10-20 5:41 ` Namhyung Kim
@ 2026-10-20 15:30 ` Ian Rogers
2026-10-20 22:29 ` Namhyung Kim
0 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-20 15:30 UTC (permalink / raw)
To: Namhyung Kim
Cc: acme, adrian.hunter, james.clark, jolsa, linux-kernel,
linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu
On Mon, Oct 19, 2026 at 10:41 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> On Mon, Oct 19, 2026 at 03:09:10PM -0700, Ian Rogers wrote:
> > Commit 466db4275edd ("perf parse-events: Add parse_uid_filter helper")
> > updated set_filter() to route non-tracepoint filter terms (such as uid
> > and gid) on PERF_TYPE_TRACEPOINT events to evsel->bpf_filters instead
> > of the ftrace tracepoint predicate filter.
> >
> > However, in the kernel's __perf_event_set_bpf_prog()
> > (kernel/events/core.c), attaching a BPF program to a perf_event fd
> > branches on perf_event_is_tracing(event):
> > - Non-tracing events (hardware PMU, software events, breakpoints) call
> > perf_event_set_bpf_handler(), which requires BPF_PROG_TYPE_PERF_EVENT
> > (SEC("perf_event")).
> > - Tracepoint events (event->pmu == &perf_tracepoint) reject
> > BPF_PROG_TYPE_PERF_EVENT with -EINVAL and instead require
> > BPF_PROG_TYPE_TRACEPOINT (SEC("tracepoint")), which is attached to
> > event->tp_event->prog_array and executed in
> > perf_trace_run_bpf_submit() before perf_tp_event(), dropping the
> > sample if the BPF program returns 0. Furthermore, the
> > bpf_cast_to_kern_ctx() kfunc is only permitted in
> > BPF_PROG_TYPE_PERF_EVENT programs.
> >
> > Add SEC("tracepoint") int perf_tp_sample_filter(void *ctx) in
> > sample_filter.bpf.c to evaluate context-independent filter terms (uid,
> > gid, tid, cpu, time, cgroup) using BPF helpers without calling
> > bpf_cast_to_kern_ctx(). When preparing a BPF filter on a
> > PERF_TYPE_TRACEPOINT evsel, autoload perf_tp_sample_filter instead of
> > perf_sample_filter and attach it once to the shared trace_event_call.
> >
> > Before:
> > # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true
> > libbpf: prog 'perf_sample_filter': failed to attach to perf_event FD 3: Invalid argument
> > Failed to attach perf sample-filter program
> > failed to set filter "BPF" on event kmem:mm_page_alloc with 22 (Invalid argument)
> >
> > After:
> > # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true
> > [ perf record: Woken up 1 times to write data ]
> > [ perf record: Captured and wrote 0.024 MB perf.data (42 samples) ]
> >
> > Fixes: 466db4275edd ("perf parse-events: Add parse_uid_filter helper")
> > Signed-off-by: Ian Rogers <irogers@google.com>
> > ---
> > tools/perf/util/bpf-filter.c | 136 ++++++++++++++++---
> > tools/perf/util/bpf_skel/sample_filter.bpf.c | 85 ++++++++++++
> > 2 files changed, 201 insertions(+), 20 deletions(-)
> >
> > diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> > index 72e83ac1d6a4..4e99af4764f1 100644
> > --- a/tools/perf/util/bpf-filter.c
> > +++ b/tools/perf/util/bpf-filter.c
> > @@ -123,15 +123,26 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
> > {
> > const struct perf_sample_info *info;
> >
> > - if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END &&
> > - (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
> > - return 0;
> > -
> > if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
> > /* Not dependent on the sample_type as computed from a BPF helper. */
> > return 0;
> > }
> >
> > + if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT) {
> > + if (expr->term == PBF_TERM_TID || expr->term == PBF_TERM_CPU ||
> > + expr->term == PBF_TERM_TIME || expr->term == PBF_TERM_CGROUP)
> > + return 0;
> > + if (expr->op != PBF_OP_GROUP_BEGIN) {
> > + pr_err("Error: %s tracepoint event does not support BPF sample term %d\n",
> > + evsel__name(evsel), expr->term);
> > + return -1;
> > + }
> > + }
> > +
> > + if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END &&
> > + (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
> > + return 0;
> > +
> > if (expr->op == PBF_OP_GROUP_BEGIN) {
> > struct perf_bpf_filter_expr *group;
> >
> > @@ -427,6 +438,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> > struct bpf_link *link;
> > struct perf_bpf_filter_entry *entry;
> > bool needs_idx_hash = !target__has_cpu(target);
> > + bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
> > #if LIBBPF_CURRENT_VERSION_GEQ(1, 7)
> > DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts,
> > .dont_enable = true);
> > @@ -444,9 +456,15 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> > goto err;
> > }
> >
> > - if (needs_idx_hash && geteuid() != 0) {
> > + if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
> > int zero = 0;
> >
> > + if (is_tracepoint) {
> > + pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n");
> > + ret = -EPERM;
> > + goto err;
> > + }
> > +
> > /* The filters map is shared among other processes */
> > ret = create_idx_hash(evsel, entry);
> > if (ret < 0)
> > @@ -484,8 +502,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> > return 0;
> > }
> >
> > - skel = sample_filter_bpf__open_and_load();
> > + skel = sample_filter_bpf__open();
> > if (!skel) {
> > + ret = -errno;
> > + pr_err("Failed to open perf sample-filter BPF skeleton\n");
> > + goto err;
> > + }
> > +
> > + bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
> > + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
> > +
> > + if (sample_filter_bpf__load(skel) < 0) {
> > ret = -errno;
> > pr_err("Failed to load perf sample-filter BPF skeleton\n");
> > goto err;
> > @@ -501,15 +528,35 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> > goto err;
> > }
> >
> > - prog = skel->progs.perf_sample_filter;
> > - for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
> > - for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
> > - link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y),
> > - &pe_opts);
> > - if (IS_ERR(link)) {
> > - pr_err("Failed to attach perf sample-filter program\n");
> > - ret = PTR_ERR(link);
> > - goto err;
> > + if (is_tracepoint) {
> > + /*
> > + * For PERF_TYPE_TRACEPOINT, PERF_EVENT_IOC_SET_BPF attaches the
> > + * BPF program to event->tp_event->prog_array on the shared
> > + * trace_event_call, which perf_trace_run_bpf_submit() executes
> > + * for all CPUs and tasks before iterating the per-CPU perf
> > + * events. Attach once on the first FD so tp_event->prog_array
> > + * does not get duplicate copies for each per-CPU FD.
> > + */
> > + prog = skel->progs.perf_tp_sample_filter;
> > + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, 0, 0),
> > + &pe_opts);
> > + ret = libbpf_get_error(link);
> > + if (ret) {
> > + pr_err("Failed to attach perf tracepoint sample-filter program\n");
> > + goto err;
> > + }
> > + skel->links.perf_tp_sample_filter = link;
> > + } else {
> > + prog = skel->progs.perf_sample_filter;
> > + for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
> > + for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
> > + link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y),
> > + &pe_opts);
>
> This link should be saves somewhere. I've sent a fix for it.
Good catch! I'll follow up on that change.
> > + ret = libbpf_get_error(link);
> > + if (ret) {
> > + pr_err("Failed to attach perf sample-filter program\n");
> > + goto err;
> > + }
> > }
> > }
> > }
> > @@ -641,6 +688,45 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
> > return ret;
> > }
> >
> > +static int pin_filter_object(struct sample_filter_bpf *skel, const char *path)
> > +{
> > + char *prog_path = NULL;
> > + int ret;
> > +
> > + ret = bpf_object__pin_maps(skel->obj, path);
> > + if (ret < 0)
> > + return ret;
> > +
> > + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
> > + bpf_object__unpin_maps(skel->obj, path);
> > + return -ENOMEM;
> > + }
> > +
> > + ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path);
> > + if (ret < 0)
> > + bpf_object__unpin_maps(skel->obj, path);
> > +
> > + free(prog_path);
> > + return ret;
> > +}
> > +
> > +static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path)
> > +{
> > + char *prog_path = NULL;
> > + int ret, map_ret;
> > +
> > + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
> > + bpf_object__unpin_maps(skel->obj, path);
> > + return -ENOMEM;
> > + }
> > +
> > + ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path);
> > + free(prog_path);
> > +
> > + map_ret = bpf_object__unpin_maps(skel->obj, path);
> > + return ret < 0 ? ret : map_ret;
> > +}
> > +
> > int perf_bpf_filter__pin(void)
> > {
> > struct sample_filter_bpf *skel;
> > @@ -655,6 +741,7 @@ int perf_bpf_filter__pin(void)
> > }
> >
> > /* pinned program will use pid-hash */
> > + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
> > bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
> > bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH);
> > bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH);
> > @@ -674,7 +761,7 @@ int perf_bpf_filter__pin(void)
> > goto err;
> > }
> >
> > - ret = bpf_object__pin(skel->obj, path);
> > + ret = pin_filter_object(skel, path);
>
> Why is this change needed?
Before this patch, sample_filter.bpf.c only contained a single BPF
program (perf_sample_filter), which was always loaded. This patch adds
a second program (perf_tp_sample_filter) to the same skeleton (and
patch 4 adds the lock-tracking programs to it as well), and uses
bpf_program__set_autoload() before loading the skeleton so we can
select only the program(s) needed for a given event.
In perf_bpf_filter__pin() and perf_bpf_filter__unpin() (for 'perf
record --setup-filter pin|unpin'), only perf_sample_filter is used,
because the pinned filter mechanism shares a single program across
privileged sessions by looking up kctx->event->id via
bpf_cast_to_kern_ctx(), which is only available to SEC("perf_event")
programs. So autoload is disabled for perf_tp_sample_filter (and in
patch 4, for the lock-tracking programs) when pinning and unpinning.
A consequence of disabling autoload on any program in the skeleton is
that libbpf's bpf_object__pin() and bpf_object__unpin() fail:
- bpf_object__pin() calls bpf_object__pin_maps() and
bpf_object__pin_programs().
- While bpf_object__pin_maps() skips maps that have autocreate
disabled, bpf_object__pin_programs() and bpf_object__unpin_programs()
iterate over every program in skel->obj without checking whether
autoload was enabled.
- When bpf_program__pin() or bpf_program__unpin() is called on a
program that wasn't loaded (prog->fd < 0), libbpf returns -EINVAL
("prog 'perf_tp_sample_filter': can't pin/unpin program that wasn't
loaded").
pin_filter_object() and unpin_filter_object() avoid this by calling
bpf_object__pin_maps() / bpf_object__unpin_maps() for the maps and
explicitly pinning/unpinning only the loaded perf_sample_filter
program.
Thanks,
Ian
> Thanks,
> Namhyung
>
>
> > if (ret < 0) {
> > pr_err("Failed to pin BPF filter objects\n");
> > goto err;
> > @@ -683,8 +770,8 @@ int perf_bpf_filter__pin(void)
> > /* setup access permissions for the pinned objects */
> > dir_fd = open(path, O_PATH);
> > if (dir_fd < 0) {
> > - bpf_object__unpin(skel->obj, path);
> > - ret = dir_fd;
> > + ret = -errno;
> > + unpin_filter_object(skel, path);
> > goto err;
> > }
> >
> > @@ -727,6 +814,8 @@ int perf_bpf_filter__pin(void)
> >
> > err_close:
> > close(dir_fd);
> > + if (ret < 0)
> > + unpin_filter_object(skel, path);
> >
> > err:
> > free(path);
> > @@ -740,13 +829,20 @@ int perf_bpf_filter__unpin(void)
> > char *path = NULL;
> > int ret = -1;
> >
> > - skel = sample_filter_bpf__open_and_load();
> > + skel = sample_filter_bpf__open();
> > if (!skel) {
> > ret = -errno;
> > pr_err("Failed to open perf sample-filter BPF skeleton\n");
> > goto err;
> > }
> >
> > + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
> > + if (sample_filter_bpf__load(skel) < 0) {
> > + ret = -errno;
> > + pr_err("Failed to load perf sample-filter BPF skeleton\n");
> > + goto err;
> > + }
> > +
> > if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(),
> > PERF_BPF_FILTER_PIN_PATH) < 0) {
> > ret = -errno;
> > @@ -754,7 +850,7 @@ int perf_bpf_filter__unpin(void)
> > goto err;
> > }
> >
> > - ret = bpf_object__unpin(skel->obj, path);
> > + ret = unpin_filter_object(skel, path);
> >
> > err:
> > free(path);
> > diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
> > index e5666d4c1722..576fbed270e5 100644
> > --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
> > +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
> > @@ -192,6 +192,28 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
> > return 0;
> > }
> >
> > +static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry *entry)
> > +{
> > + switch (entry->term) {
> > + case PBF_TERM_TID:
> > + if (entry->part)
> > + return bpf_get_current_pid_tgid() >> 32;
> > + return (__u32)bpf_get_current_pid_tgid();
> > + case PBF_TERM_CPU:
> > + return bpf_get_smp_processor_id();
> > + case PBF_TERM_TIME:
> > + return bpf_ktime_get_ns();
> > + case PBF_TERM_CGROUP:
> > + return bpf_get_current_cgroup_id();
> > + case PBF_TERM_UID:
> > + return bpf_get_current_uid_gid() & 0xFFFFFFFF;
> > + case PBF_TERM_GID:
> > + return bpf_get_current_uid_gid() >> 32;
> > + default:
> > + return 0;
> > + }
> > +}
> > +
> > #define CHECK_RESULT(data, op, val) \
> > if (!(data op val)) { \
> > if (!in_group) \
> > @@ -295,4 +317,67 @@ int perf_sample_filter(void *ctx)
> > return 0;
> > }
> >
> > +/* BPF program to be called for tracepoint perf events */
> > +SEC("tracepoint")
> > +int perf_tp_sample_filter(void *ctx)
> > +{
> > + struct perf_bpf_filter_entry *entry;
> > + __u64 sample_data;
> > + int in_group = 0;
> > + int group_result = 0;
> > + int i, k = 0;
> > + int *losts;
> > +
> > + entry = bpf_map_lookup_elem(&filters, &k);
> > + if (entry == NULL)
> > + goto drop;
> > +
> > + for (i = 0; i < MAX_FILTERS; i++) {
> > + sample_data = perf_get_common_sample(&entry[i]);
> > +
> > + switch (entry[i].op) {
> > + case PBF_OP_EQ:
> > + CHECK_RESULT(sample_data, ==, entry[i].value)
> > + break;
> > + case PBF_OP_NEQ:
> > + CHECK_RESULT(sample_data, !=, entry[i].value)
> > + break;
> > + case PBF_OP_GT:
> > + CHECK_RESULT(sample_data, >, entry[i].value)
> > + break;
> > + case PBF_OP_GE:
> > + CHECK_RESULT(sample_data, >=, entry[i].value)
> > + break;
> > + case PBF_OP_LT:
> > + CHECK_RESULT(sample_data, <, entry[i].value)
> > + break;
> > + case PBF_OP_LE:
> > + CHECK_RESULT(sample_data, <=, entry[i].value)
> > + break;
> > + case PBF_OP_AND:
> > + CHECK_RESULT(sample_data, &, entry[i].value)
> > + break;
> > + case PBF_OP_GROUP_BEGIN:
> > + in_group = 1;
> > + group_result = 0;
> > + break;
> > + case PBF_OP_GROUP_END:
> > + if (group_result == 0)
> > + goto drop;
> > + in_group = 0;
> > + break;
> > + case PBF_OP_DONE:
> > + return 1;
> > + }
> > + }
> > + return 1;
> > +
> > +drop:
> > + losts = bpf_map_lookup_elem(&dropped, &k);
> > + if (losts != NULL)
> > + __sync_fetch_and_add(losts, 1);
> > +
> > + return 0;
> > +}
> > +
> > char LICENSE[] SEC("license") = "Dual BSD/GPL";
> > --
> > 2.56.0.212.g129ede20d9-goog
> >
^ permalink raw reply [flat|nested] 106+ messages in thread
* [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support
2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers
` (6 preceding siblings ...)
2026-10-19 22:09 ` [PATCH v4 7/7] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
@ 2026-10-20 20:58 ` Ian Rogers
2026-10-20 20:58 ` [PATCH v5 1/8] perf lock contention: Filter events before creating lock_stat Ian Rogers
` (8 more replies)
7 siblings, 9 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
This series adds a new BPF sample filter predicate, `lock_held(<lock>)`
(along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`,
`lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W`
mode suffixes), to `perf record --filter`, allowing perf event samples to
be recorded only when a specified kernel lock is held (or not held with
`!`) by the running kernel thread or CPU. It also fixes a pre-existing
`lock_stat` filtering bug in `perf lock contention`, fixes a `bpf_link`
memory leak in `perf_bpf_filter__prepare()`, enables BPF sample filters on
tracepoint events (`PERF_TYPE_TRACEPOINT`), and extends both `lock_held()`
and `perf lock contention --use-bpf` to support RCU and SRCU.
For example:
# perf record -a -g -e cycles --filter 'lock_held(cgroup_mutex)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(mmap_lock)' -- ...
# perf record -a -g -e cycles --filter 'lock_held(mmap_lock:W)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ...
# perf lock contention -a -b -Y rcu,srcu -- sleep 1
An important use-case is detecting memory allocations or direct memory
reclaim occurring while kernel locks are held, which can stall lock
waiters for extended periods under memory pressure:
# Direct memory reclaim triggered while holding any mutex or rwsem:
# perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \
--filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
# Page allocations permitted to enter direct reclaim (__GFP_DIRECT_RECLAIM)
# while holding a mutex or rwsem:
# perf record -a -g -e kmem:mm_page_alloc \
--filter 'gfp_flags & 0x400' \
--filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
# Major page faults occurring while holding a mutex or rwsem:
# perf record -a -g -e major-faults \
--filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
Supported `<lock>` specifications match `perf lock contention`:
- Global/static kernel lock symbols (e.g. `lock_held(cgroup_mutex)`)
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`,
`lock_held_read(&dentry)`)
- Well-known per-process, semi-global, and RCU locks: `mmap_lock`,
`siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`
- Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`,
`rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`,
`percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`,
`rcu:W`, `semaphore`
Limitations and weaknesses in determining whether a lock is held:
- Inlined lock/unlock operations:
- When the kernel is configured to inline spinlock or rwlock operations
(`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`,
`CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`,
`CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` /
`_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are
inlined into callers and have no BTF function entry points for
`fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that
both acquire and release entry points exist in BTF before enabling a
lock family.
- Fast-path `percpu_down_read()` / `percpu_up_read()` and
`srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined
per-CPU counter increments without function calls, so only
`percpu_down_write()` / `percpu_up_write()` and out-of-line
`__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked.
- On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to
`preempt_disable()` and does not maintain
`current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)`
read-side inspection requires `CONFIG_PREEMPT_RCU`.
- Lack of an inspectable owner field on non-exclusive / spinning locks:
- While `struct mutex`, `struct rt_mutex`, and write-locked
`struct rw_semaphore` store the owning `task_struct *` in `lock->owner`
(allowing `perf_sample_filter` to inspect kernel memory directly on
sample overflow when filtering by lock address without attaching any
`fentry`/`fexit` hooks for `*_mutex`, `*_rt_mutex`, and `mmap_lock:W`),
spinlocks, rwlocks, semaphores, SRCU, and reader-locked
`rw_semaphore`s (where `__up_read()` does not clear `sem->owner` when
`!CONFIG_DEBUG_RWSEMS`) do not maintain an authoritative per-task
owner pointer.
- Tracking these locks (as well as slab-cache and lock-type filters)
relies on `fexit` (acquire) and `fentry` (release) BPF hooks
maintaining the `held_locks` BPF LRU hash map:
- Any such lock already held before `perf record` attaches its BPF
programs will not be present in `held_locks` until it is released
and re-acquired.
- The attached `fentry`/`fexit` hooks execute on every kernel lock
acquire and release for the hooked lock family across the system,
adding tracing overhead outside of perf sample handling.
- Cross-task semaphore signaling (`down*()` / `up()`):
- Counting `struct semaphore`s do not enforce same-task ownership in the
kernel: when a semaphore is used as a completion/signaling mechanism
(`down()` in one task woken by `up()` in another task or interrupt
handler, which proposed kernel dependency trackers like `CONFIG_DEPT`
model as wait/event dependencies bounded by execution context rather
than lock ownership), `fexit/down` and `fentry/up` do not pair on the
same task ID. Semaphore hooks are therefore only enabled when
`semaphore` or a `*_sem` symbol (such as `console_sem`) is explicitly
specified, and `held_locks` uses `BPF_MAP_TYPE_LRU_HASH` so any
unpaired cross-task entries are automatically evicted.
Patch overview (patches 1-3 are standalone fixes independent of the rest
of the series):
1. "perf lock contention: Filter events before creating lock_stat":
Fixes `report_lock_contention_begin_event()` in `builtin-lock.c` to
apply `-Y`, `-L`, and `-S` filters before calling
`lock_stat_findnew()`.
2. "perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF":
Attaches `perf_sample_filter` directly via `PERF_EVENT_IOC_SET_BPF` in
`perf_bpf_filter__prepare()`, matching the pinned filter path and
avoiding leaking `struct bpf_link_perf` or double-closing
`FD(evsel, x, y)`.
3. "perf bpf-filter: Support BPF sample filters on tracepoint events":
Adds `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF
sample filters can attach to `PERF_TYPE_TRACEPOINT` events (where
`__perf_event_set_bpf_prog()` in `kernel/events/core.c` requires
`BPF_PROG_TYPE_TRACEPOINT` rather than `BPF_PROG_TYPE_PERF_EVENT`).
4. "perf lock contention: Share common BPF and lock type definitions in
lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and
`LCB_F_*` flags, CO-RE structs, and common BPF lock helpers
(`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`,
`get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`,
`do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`.
5. "perf bpf-filter: Add lock_held() sample filter": Implements
`lock_held()`, `lock_held_read()`/`lock_held_shared()`,
`lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode
suffixes using direct lock owner inspection where possible (`mutex`,
`rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF LRU
hash map updated by selectively attached `fentry`/`fexit` hooks.
6. "perf test: Add tests for lock_held() BPF sample filter": Adds unit
tests in `tests/parse-events.c` and shell tests in
`tests/shell/record_bpf_filter.sh`.
7. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU": Supports `rcu` (`rcu:R` via direct
`current->rcu_read_lock_nesting` inspection with zero tracing hooks;
`rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks),
`rcu-trace` (via `current->trc_reader_nesting`), and `srcu`
(`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W`
via `synchronize_srcu*()` / `srcu_barrier()`).
8. "perf lock contention: Extend BPF lock contention to RCU and SRCU":
Extends `perf lock contention --use-bpf` to track RCU and SRCU
grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`).
Changes in v5:
- Add patch 2 ("perf bpf-filter: Fix bpf_link leak by using
PERF_EVENT_IOC_SET_BPF") attaching BPF sample filters via
`ioctl(..., PERF_EVENT_IOC_SET_BPF, fd)` instead of
`bpf_program__attach_perf_event_opts()`, fixing the `bpf_link` memory
leak without double-closing `FD(evsel, x, y)`, and use the same
attachment in patch 3 for tracepoint sample filters (Namhyung).
- Expand patch 3's commit message to detail why `pin_filter_object()`
and `unpin_filter_object()` replace `bpf_object__pin()` and
`bpf_object__unpin()` when `autoload` is disabled on
`perf_tp_sample_filter` (Namhyung).
- Store the resolved lock target in `perf_bpf_filter_entry.value` (with
`op` normalized to `PBF_OP_EQ` or `PBF_OP_NEQ`), keeping
`struct perf_bpf_filter_entry` unchanged at 24 bytes for backwards
compatibility with pinned `filters` maps.
- Add a CO-RE `struct rt_mutex___old` fallback in `get_lock_owner()` in
`lock_data.h` for kernels prior to v5.15, and pass baseline nesting
depth to `is_rcu_trace_lock_held()` for syscall tracepoints.
Changes in v4:
- Move the two bugfix commits (`Filter events before creating lock_stat`
and `Support BPF sample filters on tracepoint events`) to patches 1
and 2 at the start of the series for easier cherry-picking.
- Reserve leading `&` strictly for slab cache locks (e.g.
`lock_held(&dentry)`) so global/static lock symbols and well-known
locks do not use `&` (e.g. `lock_held(cgroup_mutex)`,
`lock_held_read(mmap_lock)`), matching `perf lock contention`
(Namhyung).
- Skip attaching `fentry`/`fexit` hooks when filtering on a specific
`mutex` symbol (`*_mutex`), `rt_mutex` symbol (`*_rt_mutex`), or
write-locked `mmap_lock` (`mmap_lock:W` / `lock_held_write(mmap_lock)`),
evaluating them purely via direct `owner` inspection (Namhyung).
- Use a CO-RE flavor struct `struct mutex___non_rt` in `get_lock_owner()`
in `lock_data.h` so `GEN_VMLINUX_H=1` builds succeed on
`CONFIG_PREEMPT_RT=y` kernels.
- Switch `held_locks` to `BPF_MAP_TYPE_LRU_HASH`, exclude `LOCK_PROG_SEM`
from `LOCK_PROG_ALL` unless `semaphore` or a `*_sem` symbol is
explicitly requested, and document cross-task semaphore signaling
limitations (and comparison with `CONFIG_DEPT`).
- Skip `LCB_F_MUTEX | LCB_F_SPIN` in `parse_lock_type_name()`, reject
read-only mode on exclusive-only lock symbol suffixes in
`resolve_lock_expr()`, and check `needed_groups & supported_groups`
before masking `needed_groups` in `configure_lock_progs()`.
- Recursively traverse anonymous structs/unions in `btf_has_task_field()`
so `lock_held(rcu)` and `lock_held(rcu-trace)` work on
`CONFIG_RANDSTRUCT` kernels.
Changes in v3:
- Move the writer-only `rw_semaphore` check out of `get_lock_owner()` in
`lock_data.h` into `is_rwsem_writer_owner()` in `sample_filter.bpf.c`
so `perf lock contention -b -o` still reports reader owners when a
writer contends on a reader-owned `rw_semaphore`, and scope
`is_direct_lock_owner()` by lock type (`PBF_LOCK_DIRECT_MUTEX`,
`PBF_LOCK_DIRECT_RWSEM`, `PBF_LOCK_DIRECT_RTMUTEX`).
- Add `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF
sample filters (`lock_held()`, `uid`, `gid`, etc.) can attach to
`PERF_TYPE_TRACEPOINT` events (such as `kmem:mm_page_alloc` and
`vmscan:mm_vmscan_direct_reclaim_begin`), check `libbpf_get_error()`
when attaching, and document memory allocation / direct reclaim
filtering with `lock_held()`.
- Return early in `init_numa_data()` in `bpf-filter.c` when `node_data`
cannot be resolved and check `node_data_addr != 0` in
`do_collect_zone_lock()`.
- Avoid double-counting `_raw_read_lock_irq` / `_raw_read_lock_bh` in
`sample_filter.bpf.c` on `CONFIG_GENERIC_LOCKBREAK=y &&
!CONFIG_DEBUG_LOCK_ALLOC` kernels where they call
`_raw_read_lock_irqsave()`.
- Offset `rcu_state_addr` by 1 byte so it is distinct from
`&rcu_state.node[0].lock` at offset 0 of `rcu_state`, and pass `bool
is_rcu` to `do_contention_end()` in `lock_contention.bpf.c` so
`tp_btf/contention_end` and RCU/SRCU `fexit` hooks never cross-match
timestamp maps.
Changes in v2:
- Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists`
and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE
relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels.
- Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points
(`_mutex_lock_nest_lock`, `_mutex_lock_killable`,
`_mutex_trylock_nest_lock`, `_down_write_nest_lock`,
`_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to
`sample_filter.bpf.c` and `lock_progs[]`.
- Verify in `supported_lock_prog_groups()` that both acquire and all
unlock BTF entry points exist for a lock family before enabling its
tracing hooks, preventing acquire-only attachment when spinlock/rwlock
unlock functions are inlined.
- Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when
a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`).
- Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write`
on `track_rwsem_read` and verify non-zero read/write counts in
`record_lock_released_addr()` before decrementing aggregate slab/type
counts.
- Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in
`lock_contention.bpf.c` and recognize `synchronize_rcu*`,
`synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in
`machine__is_lock_function()`.
- Add patch fixing `report_lock_contention_begin_event()` to check
type, address, and callstack filters before creating `lock_stat`
entries.
Assisted-by: Antigravity:gemini-3.1-pro
Ian Rogers (8):
perf lock contention: Filter events before creating lock_stat
perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF
perf bpf-filter: Support BPF sample filters on tracepoint events
perf lock contention: Share common BPF and lock type definitions in
lock_data.h
perf bpf-filter: Add lock_held() sample filter
perf test: Add tests for lock_held() BPF sample filter
perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU
perf lock contention: Extend BPF lock contention to RCU and SRCU
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/Documentation/perf-record.txt | 66 +-
tools/perf/builtin-lock.c | 88 +-
tools/perf/tests/parse-events.c | 121 ++
tools/perf/tests/shell/lock_contention.sh | 14 +
tools/perf/tests/shell/record_bpf_filter.sh | 142 ++
tools/perf/util/bpf-filter.c | 1417 ++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 47 +-
tools/perf/util/bpf_lock_contention.c | 171 +-
.../perf/util/bpf_skel/lock_contention.bpf.c | 448 +++---
tools/perf/util/bpf_skel/lock_data.h | 385 ++++-
tools/perf/util/bpf_skel/sample-filter.h | 80 +
tools/perf/util/bpf_skel/sample_filter.bpf.c | 1241 +++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 +
tools/perf/util/lock-contention.c | 23 +
tools/perf/util/lock-contention.h | 15 +
tools/perf/util/machine.c | 11 +
tools/perf/util/parse-events.c | 3 +-
20 files changed, 3937 insertions(+), 393 deletions(-)
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply [flat|nested] 106+ messages in thread
* [PATCH v5 1/8] perf lock contention: Filter events before creating lock_stat
2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
@ 2026-10-20 20:58 ` Ian Rogers
2026-10-20 22:38 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 2/8] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers
` (7 subsequent siblings)
8 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
In report_lock_contention_begin_event(), lock_stat_findnew() was
called before checking filters.nr_types (-Y), filters.nr_addrs (-L),
and match_callstack_filter() (-S).
When a filtered-out lock event (such as a mutex contention when
running with -Y spinlock) shared the same aggregation key as a
subsequent matching event (for example, contention on the mutex's
internal wait_lock spinlock from the same caller), the first event
created the lock_stat entry with the filtered-out lock's flags and the
second event reused that entry, causing non-spinlock entries to appear
in -Y spinlock output.
Check the type, address, and callstack filters before looking up or
creating the lock_stat entry.
Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option")
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/builtin-lock.c | 61 +++++++++++++++++++++------------------
1 file changed, 33 insertions(+), 28 deletions(-)
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index d5c0d55cb82d..778923c92f42 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -975,6 +975,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
struct machine *machine = &session->machines.host;
struct map *kmap;
struct symbol *sym;
+ u64 *callstack = NULL;
ret = get_key_by_aggr_mode(&key, addr, sample);
if (ret < 0)
@@ -1010,33 +1011,6 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
}
}
- ls = lock_stat_find(key);
- if (!ls) {
- char buf[128];
- const char *name = "";
-
- switch (aggr_mode) {
- case LOCK_AGGR_ADDR:
- sym = machine__find_kernel_symbol(machine, key, &kmap);
- if (sym)
- name = sym->name;
- break;
- case LOCK_AGGR_CALLER:
- name = buf;
- if (lock_contention_caller(sample, buf, sizeof(buf)) < 0)
- name = "Unknown";
- break;
- case LOCK_AGGR_CGROUP:
- case LOCK_AGGR_TASK:
- default:
- break;
- }
-
- ls = lock_stat_findnew(key, name, flags);
- if (!ls)
- return -ENOMEM;
- }
-
if (filters.nr_types) {
bool found = false;
@@ -1066,7 +1040,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
}
if (needs_callstack()) {
- u64 *callstack = get_callstack(sample, max_stack_depth);
+ callstack = get_callstack(sample, max_stack_depth);
if (callstack == NULL)
return 0;
@@ -1074,7 +1048,38 @@ static int report_lock_contention_begin_event(struct perf_sample *sample)
free(callstack);
return 0;
}
+ }
+
+ ls = lock_stat_find(key);
+ if (!ls) {
+ char buf[128];
+ const char *name = "";
+
+ switch (aggr_mode) {
+ case LOCK_AGGR_ADDR:
+ sym = machine__find_kernel_symbol(machine, key, &kmap);
+ if (sym)
+ name = sym->name;
+ break;
+ case LOCK_AGGR_CALLER:
+ name = buf;
+ if (lock_contention_caller(sample, buf, sizeof(buf)) < 0)
+ name = "Unknown";
+ break;
+ case LOCK_AGGR_CGROUP:
+ case LOCK_AGGR_TASK:
+ default:
+ break;
+ }
+
+ ls = lock_stat_findnew(key, name, flags);
+ if (!ls) {
+ free(callstack);
+ return -ENOMEM;
+ }
+ }
+ if (callstack) {
if (ls->callstack == NULL)
ls->callstack = callstack;
else
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v5 2/8] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF
2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-20 20:58 ` [PATCH v5 1/8] perf lock contention: Filter events before creating lock_stat Ian Rogers
@ 2026-10-20 20:58 ` Ian Rogers
2026-10-20 22:41 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 3/8] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
` (6 subsequent siblings)
8 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
In perf_bpf_filter__prepare(), the unpinned filter path attaches
skel->progs.perf_sample_filter to each FD(evsel, x, y) using
bpf_program__attach_perf_event_opts() and ignores the returned
struct bpf_link *, leaking the allocated struct bpf_link_perf and its
BPF_PERF_EVENT link fd.
Furthermore, bpf_program__attach_perf_event_opts() is designed for
callers that transfer ownership of the perf_event fd to the returned
bpf_link: when bpf_link__destroy() is called, bpf_link_perf_detach()
unconditionally disables and closes perf_link->perf_event_fd. Because
evsel->core.fd owns FD(evsel, x, y) and closes them in evlist__close(),
destroying the returned bpf_link would double-close the perf_event file
descriptors.
Fix this by attaching the program directly with
ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd), matching the pinned
filter path right above it in perf_bpf_filter__prepare(). In the
kernel, PERF_EVENT_IOC_SET_BPF takes a reference on the BPF program and
_free_event() -> perf_event_free_bpf_prog() automatically detaches and
releases it when the perf_event fd is closed. This also avoids enabling
the event prematurely, allowing pe_opts (.dont_enable = true) and the
LIBBPF_CURRENT_VERSION_GEQ(1, 7) guard to be dropped.
Before (built with EXTRA_CFLAGS="-fsanitize=address"):
# perf record -e cycles --filter 'period > 1000' -- true
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.024 MB perf.data ]
=================================================================
==51328==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 1568 byte(s) in 28 object(s) allocated from:
#0 0x7ffbe76ef24f in calloc (/lib64/libasan.so.8+0xef24f)
#1 0x625df7 in bpf_program__attach_perf_event_opts (tools/perf/perf+0x625df7)
#2 0xbeddc0 in perf_bpf_filter__prepare util/bpf-filter.c:507
#3 0x8f3def in evlist__apply_filters util/evlist.c:1421
#4 0x44bbcf in record__open tools/perf/builtin-record.c:1469
#5 0x453f9e in __cmd_record tools/perf/builtin-record.c:2594
#6 0x45dadb in cmd_record tools/perf/builtin-record.c:4427
#7 0x5df56b in run_builtin tools/perf/perf.c:383
#8 0x5dfa0e in handle_internal_command tools/perf/perf.c:436
#9 0x5dfddc in run_argv tools/perf/perf.c:480
#10 0x5e0333 in main tools/perf/perf.c:581
SUMMARY: AddressSanitizer: 1568 byte(s) leaked in 28 allocation(s).
After (built with EXTRA_CFLAGS="-fsanitize=address"):
# perf record -e cycles --filter 'period > 1000' -- true
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.024 MB perf.data ]
Reported-by: Namhyung Kim <namhyung@kernel.org>
Closes: https://lore.kernel.org/linux-perf-users/20261020053226.55936-1-namhyung@kernel.org/
Fixes: 56ec9457a4a2 ("perf bpf filter: Implement event sample filtering")
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/bpf-filter.c | 17 ++++-------------
1 file changed, 4 insertions(+), 13 deletions(-)
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 72e83ac1d6a4..1596b08fe4e6 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -423,16 +423,8 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
{
int i, x, y, fd, ret;
struct sample_filter_bpf *skel = NULL;
- struct bpf_program *prog;
- struct bpf_link *link;
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
-#if LIBBPF_CURRENT_VERSION_GEQ(1, 7)
- DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts,
- .dont_enable = true);
-#else
- DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts);
-#endif
entry = calloc(MAX_FILTERS, sizeof(*entry));
if (entry == NULL)
@@ -501,14 +493,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- prog = skel->progs.perf_sample_filter;
+ fd = bpf_program__fd(skel->progs.perf_sample_filter);
for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
- link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y),
- &pe_opts);
- if (IS_ERR(link)) {
+ ret = ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd);
+ if (ret < 0) {
+ ret = -errno;
pr_err("Failed to attach perf sample-filter program\n");
- ret = PTR_ERR(link);
goto err;
}
}
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v5 3/8] perf bpf-filter: Support BPF sample filters on tracepoint events
2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-20 20:58 ` [PATCH v5 1/8] perf lock contention: Filter events before creating lock_stat Ian Rogers
2026-10-20 20:58 ` [PATCH v5 2/8] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers
@ 2026-10-20 20:58 ` Ian Rogers
2026-10-20 22:57 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
` (5 subsequent siblings)
8 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Commit 466db4275edd ("perf parse-events: Add parse_uid_filter helper")
updated set_filter() to route non-tracepoint filter terms (such as uid
and gid) on PERF_TYPE_TRACEPOINT events to evsel->bpf_filters instead
of the ftrace tracepoint predicate filter.
However, in the kernel's __perf_event_set_bpf_prog()
(kernel/events/core.c), attaching a BPF program to a perf_event fd
branches on perf_event_is_tracing(event):
- Non-tracing events (hardware PMU, software events, breakpoints) call
perf_event_set_bpf_handler(), which requires BPF_PROG_TYPE_PERF_EVENT
(SEC("perf_event")).
- Tracepoint events (event->pmu == &perf_tracepoint) reject
BPF_PROG_TYPE_PERF_EVENT with -EINVAL and instead require
BPF_PROG_TYPE_TRACEPOINT (SEC("tracepoint")), which is attached to
event->tp_event->prog_array and executed in
perf_trace_run_bpf_submit() before perf_tp_event(), dropping the
sample if the BPF program returns 0. Furthermore, the
bpf_cast_to_kern_ctx() kfunc is only permitted in
BPF_PROG_TYPE_PERF_EVENT programs.
Add SEC("tracepoint") int perf_tp_sample_filter(void *ctx) in
sample_filter.bpf.c to evaluate context-independent filter terms (uid,
gid, tid, cpu, time, cgroup) using BPF helpers without calling
bpf_cast_to_kern_ctx(). When preparing a BPF filter on a
PERF_TYPE_TRACEPOINT evsel, autoload perf_tp_sample_filter instead of
perf_sample_filter and attach it once to the shared trace_event_call.
Because returning 0 from a BPF_PROG_TYPE_TRACEPOINT program filters at
the shared trace_event_call->prog_array prior to perf_tp_event(),
tracepoint BPF filters are never pinned for unprivileged use and
require CAP_BPF or root privileges.
Because perf_bpf_filter__pin() and perf_bpf_filter__unpin() only load
perf_sample_filter and disable autoload on perf_tp_sample_filter,
libbpf's bpf_object__pin() and bpf_object__unpin() can no longer be used
directly (as bpf_object__pin_programs() and bpf_object__unpin_programs()
iterate over all programs in the object without checking autoload and
fail with -EINVAL when a program has not been loaded). Introduce
pin_filter_object() and unpin_filter_object() to pin/unpin the maps and
only the loaded perf_sample_filter program.
Before:
# perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true
libbpf: prog 'perf_sample_filter': failed to attach to perf_event FD 3: Invalid argument
Failed to attach perf sample-filter program
failed to set filter "BPF" on event kmem:mm_page_alloc with 22 (Invalid argument)
After:
# perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.024 MB perf.data (42 samples) ]
Fixes: 466db4275edd ("perf parse-events: Add parse_uid_filter helper")
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/bpf-filter.c | 132 ++++++++++++++++---
tools/perf/util/bpf_skel/sample_filter.bpf.c | 95 +++++++++++++
2 files changed, 208 insertions(+), 19 deletions(-)
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 1596b08fe4e6..ac0e7cf40f3a 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -123,15 +123,26 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
{
const struct perf_sample_info *info;
- if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END &&
- (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
- return 0;
-
if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
/* Not dependent on the sample_type as computed from a BPF helper. */
return 0;
}
+ if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT) {
+ if (expr->term == PBF_TERM_TID || expr->term == PBF_TERM_CPU ||
+ expr->term == PBF_TERM_TIME || expr->term == PBF_TERM_CGROUP)
+ return 0;
+ if (expr->op != PBF_OP_GROUP_BEGIN) {
+ pr_err("Error: %s tracepoint event does not support BPF sample term %d\n",
+ evsel__name(evsel), expr->term);
+ return -1;
+ }
+ }
+
+ if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END &&
+ (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
+ return 0;
+
if (expr->op == PBF_OP_GROUP_BEGIN) {
struct perf_bpf_filter_expr *group;
@@ -425,6 +436,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
struct sample_filter_bpf *skel = NULL;
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
+ bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
entry = calloc(MAX_FILTERS, sizeof(*entry));
if (entry == NULL)
@@ -436,7 +448,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- if (needs_idx_hash && geteuid() != 0) {
+ if (is_tracepoint && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
+ pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n");
+ ret = -EPERM;
+ goto err;
+ }
+
+ if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
int zero = 0;
/* The filters map is shared among other processes */
@@ -476,8 +494,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return 0;
}
- skel = sample_filter_bpf__open_and_load();
+ skel = sample_filter_bpf__open();
if (!skel) {
+ ret = -errno;
+ pr_err("Failed to open perf sample-filter BPF skeleton\n");
+ goto err;
+ }
+
+ bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
+ bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
+
+ if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
pr_err("Failed to load perf sample-filter BPF skeleton\n");
goto err;
@@ -493,14 +520,32 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- fd = bpf_program__fd(skel->progs.perf_sample_filter);
- for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
- for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
- ret = ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd);
- if (ret < 0) {
- ret = -errno;
- pr_err("Failed to attach perf sample-filter program\n");
- goto err;
+ if (is_tracepoint) {
+ /*
+ * For PERF_TYPE_TRACEPOINT, PERF_EVENT_IOC_SET_BPF attaches the
+ * BPF program to event->tp_event->prog_array on the shared
+ * trace_event_call, which perf_trace_run_bpf_submit() executes
+ * for all CPUs and tasks before iterating the per-CPU perf
+ * events. Attach once on the first FD so tp_event->prog_array
+ * does not get duplicate copies for each per-CPU FD.
+ */
+ fd = bpf_program__fd(skel->progs.perf_tp_sample_filter);
+ ret = ioctl(FD(evsel, 0, 0), PERF_EVENT_IOC_SET_BPF, fd);
+ if (ret < 0) {
+ ret = -errno;
+ pr_err("Failed to attach perf tracepoint sample-filter program\n");
+ goto err;
+ }
+ } else {
+ fd = bpf_program__fd(skel->progs.perf_sample_filter);
+ for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
+ for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
+ ret = ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd);
+ if (ret < 0) {
+ ret = -errno;
+ pr_err("Failed to attach perf sample-filter program\n");
+ goto err;
+ }
}
}
}
@@ -632,6 +677,45 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
return ret;
}
+static int pin_filter_object(struct sample_filter_bpf *skel, const char *path)
+{
+ char *prog_path = NULL;
+ int ret;
+
+ ret = bpf_object__pin_maps(skel->obj, path);
+ if (ret < 0)
+ return ret;
+
+ if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
+ bpf_object__unpin_maps(skel->obj, path);
+ return -ENOMEM;
+ }
+
+ ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path);
+ if (ret < 0)
+ bpf_object__unpin_maps(skel->obj, path);
+
+ free(prog_path);
+ return ret;
+}
+
+static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path)
+{
+ char *prog_path = NULL;
+ int ret, map_ret;
+
+ if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
+ bpf_object__unpin_maps(skel->obj, path);
+ return -ENOMEM;
+ }
+
+ ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path);
+ free(prog_path);
+
+ map_ret = bpf_object__unpin_maps(skel->obj, path);
+ return ret < 0 ? ret : map_ret;
+}
+
int perf_bpf_filter__pin(void)
{
struct sample_filter_bpf *skel;
@@ -646,6 +730,7 @@ int perf_bpf_filter__pin(void)
}
/* pinned program will use pid-hash */
+ bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH);
bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH);
@@ -665,7 +750,7 @@ int perf_bpf_filter__pin(void)
goto err;
}
- ret = bpf_object__pin(skel->obj, path);
+ ret = pin_filter_object(skel, path);
if (ret < 0) {
pr_err("Failed to pin BPF filter objects\n");
goto err;
@@ -674,8 +759,8 @@ int perf_bpf_filter__pin(void)
/* setup access permissions for the pinned objects */
dir_fd = open(path, O_PATH);
if (dir_fd < 0) {
- bpf_object__unpin(skel->obj, path);
- ret = dir_fd;
+ ret = -errno;
+ unpin_filter_object(skel, path);
goto err;
}
@@ -718,6 +803,8 @@ int perf_bpf_filter__pin(void)
err_close:
close(dir_fd);
+ if (ret < 0)
+ unpin_filter_object(skel, path);
err:
free(path);
@@ -731,13 +818,20 @@ int perf_bpf_filter__unpin(void)
char *path = NULL;
int ret = -1;
- skel = sample_filter_bpf__open_and_load();
+ skel = sample_filter_bpf__open();
if (!skel) {
ret = -errno;
pr_err("Failed to open perf sample-filter BPF skeleton\n");
goto err;
}
+ bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
+ if (sample_filter_bpf__load(skel) < 0) {
+ ret = -errno;
+ pr_err("Failed to load perf sample-filter BPF skeleton\n");
+ goto err;
+ }
+
if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(),
PERF_BPF_FILTER_PIN_PATH) < 0) {
ret = -errno;
@@ -745,7 +839,7 @@ int perf_bpf_filter__unpin(void)
goto err;
}
- ret = bpf_object__unpin(skel->obj, path);
+ ret = unpin_filter_object(skel, path);
err:
free(path);
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index e5666d4c1722..a190374ed87a 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -192,6 +192,28 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
return 0;
}
+static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry *entry)
+{
+ switch (entry->term) {
+ case PBF_TERM_TID:
+ if (entry->part)
+ return bpf_get_current_pid_tgid() >> 32;
+ return (__u32)bpf_get_current_pid_tgid();
+ case PBF_TERM_CPU:
+ return bpf_get_smp_processor_id();
+ case PBF_TERM_TIME:
+ return bpf_ktime_get_ns();
+ case PBF_TERM_CGROUP:
+ return bpf_get_current_cgroup_id();
+ case PBF_TERM_UID:
+ return bpf_get_current_uid_gid() & 0xFFFFFFFF;
+ case PBF_TERM_GID:
+ return bpf_get_current_uid_gid() >> 32;
+ default:
+ return 0;
+ }
+}
+
#define CHECK_RESULT(data, op, val) \
if (!(data op val)) { \
if (!in_group) \
@@ -295,4 +317,77 @@ int perf_sample_filter(void *ctx)
return 0;
}
+/*
+ * BPF program to be called for tracepoint perf events.
+ *
+ * For PERF_TYPE_TRACEPOINT events, the kernel's __perf_event_set_bpf_prog()
+ * rejects BPF_PROG_TYPE_PERF_EVENT and only accepts BPF_PROG_TYPE_TRACEPOINT,
+ * attaching the program to the shared trace_event_call->prog_array executed
+ * by perf_trace_run_bpf_submit() prior to perf_tp_event(). Because returning 0
+ * filters the event at trace_event_call->prog_array (affecting the
+ * trace_event_call globally while attached), tracepoint BPF filters are never
+ * pinned for unprivileged use and require CAP_BPF / root privileges.
+ */
+SEC("tracepoint")
+int perf_tp_sample_filter(void *ctx)
+{
+ struct perf_bpf_filter_entry *entry;
+ __u64 sample_data;
+ int in_group = 0;
+ int group_result = 0;
+ int i, k = 0;
+ int *losts;
+
+ entry = bpf_map_lookup_elem(&filters, &k);
+ if (entry == NULL)
+ goto drop;
+
+ for (i = 0; i < MAX_FILTERS; i++) {
+ sample_data = perf_get_common_sample(&entry[i]);
+
+ switch (entry[i].op) {
+ case PBF_OP_EQ:
+ CHECK_RESULT(sample_data, ==, entry[i].value)
+ break;
+ case PBF_OP_NEQ:
+ CHECK_RESULT(sample_data, !=, entry[i].value)
+ break;
+ case PBF_OP_GT:
+ CHECK_RESULT(sample_data, >, entry[i].value)
+ break;
+ case PBF_OP_GE:
+ CHECK_RESULT(sample_data, >=, entry[i].value)
+ break;
+ case PBF_OP_LT:
+ CHECK_RESULT(sample_data, <, entry[i].value)
+ break;
+ case PBF_OP_LE:
+ CHECK_RESULT(sample_data, <=, entry[i].value)
+ break;
+ case PBF_OP_AND:
+ CHECK_RESULT(sample_data, &, entry[i].value)
+ break;
+ case PBF_OP_GROUP_BEGIN:
+ in_group = 1;
+ group_result = 0;
+ break;
+ case PBF_OP_GROUP_END:
+ if (group_result == 0)
+ goto drop;
+ in_group = 0;
+ break;
+ case PBF_OP_DONE:
+ return 1;
+ }
+ }
+ return 1;
+
+drop:
+ losts = bpf_map_lookup_elem(&dropped, &k);
+ if (losts != NULL)
+ __sync_fetch_and_add(losts, 1);
+
+ return 0;
+}
+
char LICENSE[] SEC("license") = "Dual BSD/GPL";
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (2 preceding siblings ...)
2026-10-20 20:58 ` [PATCH v5 3/8] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
@ 2026-10-20 20:58 ` Ian Rogers
2026-10-20 22:42 ` Namhyung Kim
2026-10-20 23:02 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter Ian Rogers
` (4 subsequent siblings)
8 siblings, 2 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers
(get_lock_owner, get_current_mmap_lock, get_current_siglock,
get_lock_addr, check_lock_type, do_collect_zone_lock,
do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into
tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF
sample filtering.
Also add rw_semaphore.count and struct rt_mutex_base to the minimal
vmlinux.h and extend get_lock_owner() to support rt_mutex and check
the mutex handoff flag.
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/builtin-lock.c | 27 --
.../perf/util/bpf_skel/lock_contention.bpf.c | 265 +-------------
tools/perf/util/bpf_skel/lock_data.h | 338 ++++++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 +
tools/perf/util/lock-contention.c | 19 +
tools/perf/util/lock-contention.h | 13 +
6 files changed, 393 insertions(+), 274 deletions(-)
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index 778923c92f42..a031709a2cc2 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -1482,33 +1482,6 @@ static void sort_result(void)
}
}
-static const struct {
- unsigned int flags;
- /*
- * Name of the lock flags (access), with delimeter ':'.
- * For example, rwsem:R of rwsem:W.
- */
- const char *flags_name;
- /* Name of the lock (type), for example, rwlock or rwsem. */
- const char *lock_name;
-} lock_type_table[] = {
- { 0, "semaphore", "semaphore" },
- { LCB_F_SPIN, "spinlock", "spinlock" },
- { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
- { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
- { LCB_F_READ, "rwsem:R", "rwsem" },
- { LCB_F_WRITE, "rwsem:W", "rwsem" },
- { LCB_F_RT, "rt-mutex", "rt-mutex" },
- { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
- { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
- { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
- { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
- { LCB_F_MUTEX, "mutex", "mutex" },
- { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
- /* alias for optimistic spinning only */
- { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
-};
-
static const char *get_type_flags_name(unsigned int flags)
{
flags &= LCB_F_TYPE_MASK;
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index 0d9c6f55050e..0f2118006005 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -8,23 +8,9 @@
#include "lock_data.h"
-/* for collect_lock_syms(). 4096 was rejected by the verifier */
-#define MAX_CPUS 1024
-
-/* for collect_zone_lock(). It should be more than the actual zones. */
-#define MAX_ZONES 10
-
/* for do_lock_delay(). Arbitrarily set to 1 million. */
#define MAX_LOOP (1U << 20)
-/* lock contention flags from include/trace/events/lock.h */
-#define LCB_F_SPIN (1U << 0)
-#define LCB_F_READ (1U << 1)
-#define LCB_F_WRITE (1U << 2)
-#define LCB_F_RT (1U << 3)
-#define LCB_F_PERCPU (1U << 4)
-#define LCB_F_MUTEX (1U << 5)
-
/* callstack storage */
struct {
__uint(type, BPF_MAP_TYPE_STACK_TRACE);
@@ -96,13 +82,6 @@ struct {
__uint(max_entries, MAX_ENTRIES);
} task_data SEC(".maps");
-struct {
- __uint(type, BPF_MAP_TYPE_HASH);
- __uint(key_size, sizeof(__u64));
- __uint(value_size, sizeof(__u32));
- __uint(max_entries, MAX_ENTRIES);
-} lock_syms SEC(".maps");
-
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(key_size, sizeof(__u32));
@@ -138,6 +117,13 @@ struct {
__uint(max_entries, 1);
} cgroup_filter SEC(".maps");
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u32));
+ __uint(max_entries, MAX_ENTRIES);
+} lock_syms SEC(".maps");
+
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(key_size, sizeof(long));
@@ -159,22 +145,6 @@ struct {
__uint(max_entries, 1);
} lock_delays SEC(".maps");
-struct rw_semaphore___old {
- struct task_struct *owner;
-} __attribute__((preserve_access_index));
-
-struct rw_semaphore___new {
- atomic_long_t owner;
-} __attribute__((preserve_access_index));
-
-struct mm_struct___old {
- struct rw_semaphore mmap_sem;
-} __attribute__((preserve_access_index));
-
-struct mm_struct___new {
- struct rw_semaphore mmap_lock;
-} __attribute__((preserve_access_index));
-
struct cas_ctx {
struct contention_data *data;
u64 duration;
@@ -182,8 +152,6 @@ struct cas_ctx {
int min_done;
};
-extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak;
-
/* control flags */
const volatile int has_cpu;
const volatile int has_task;
@@ -202,6 +170,11 @@ const volatile int lock_delay;
/* determine the key of lock stat */
const volatile int aggr_mode;
+const volatile __u64 contig_page_data_addr;
+const volatile __u64 node_data_addr;
+const volatile int nr_nodes;
+const volatile int sizeof_zone;
+
int enabled;
int perf_subsys_id = -1;
@@ -222,8 +195,6 @@ int data_map_full;
struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak;
void bpf_task_release(struct task_struct *p) __ksym __weak;
-static inline __u32 check_lock_type(__u64 lock, __u32 flags);
-
static inline __u64 get_current_cgroup_id(void)
{
struct task_struct *task;
@@ -347,85 +318,6 @@ static inline int update_task_data(struct task_struct *task)
return 0;
}
-#ifndef __has_builtin
-# define __has_builtin(x) 0
-#endif
-
-static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
-{
- struct task_struct *task;
- __u64 owner = 0;
-
- if (flags & LCB_F_MUTEX) {
- struct mutex *mutex = (void *)lock;
- owner = BPF_CORE_READ(mutex, owner.counter);
- } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
- /*
- * Support for the BPF_TYPE_MATCHES argument to the
- * __builtin_preserve_type_info builtin was added at some point during
- * development of clang 15 and it's what is needed for
- * bpf_core_type_matches.
- */
-#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
- if (bpf_core_type_matches(struct rw_semaphore___old)) {
- struct rw_semaphore___old *rwsem = (void *)lock;
- owner = (unsigned long)BPF_CORE_READ(rwsem, owner);
- } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
- struct rw_semaphore___new *rwsem = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
- }
-#else
- /* assume new struct */
- struct rw_semaphore *rwsem = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
-#endif
- }
-
- if (!owner)
- return NULL;
-
- task = (void *)(owner & ~7UL);
- return task;
-}
-
-static inline __u32 check_lock_type(__u64 lock, __u32 flags)
-{
- struct task_struct *curr;
- struct mm_struct___old *mm_old;
- struct mm_struct___new *mm_new;
- struct sighand_struct *sighand;
-
- switch (flags) {
- case LCB_F_READ: /* rwsem */
- case LCB_F_WRITE:
- curr = bpf_get_current_task_btf();
- if (curr->mm == NULL)
- break;
- mm_new = (void *)curr->mm;
- if (bpf_core_field_exists(mm_new->mmap_lock)) {
- if (&mm_new->mmap_lock == (void *)lock)
- return LCD_F_MMAP_LOCK;
- break;
- }
- mm_old = (void *)curr->mm;
- if (bpf_core_field_exists(mm_old->mmap_sem)) {
- if (&mm_old->mmap_sem == (void *)lock)
- return LCD_F_MMAP_LOCK;
- }
- break;
- case LCB_F_SPIN: /* spinlock */
- curr = bpf_get_current_task_btf();
- sighand = curr->sighand;
-
- if (sighand && &sighand->siglock == (void *)lock)
- return LCD_F_SIGHAND_LOCK;
- break;
- default:
- break;
- }
- return 0;
-}
-
static inline long delay_callback(__u64 idx, void *arg)
{
__u64 target = *(__u64 *)arg;
@@ -908,145 +800,24 @@ int contention_end(u64 *ctx)
return 0;
}
-extern struct rq runqueues __ksym;
-
-const volatile __u64 contig_page_data_addr;
-const volatile __u64 node_data_addr;
-const volatile int nr_nodes;
-const volatile int sizeof_zone;
-
-struct rq___old {
- raw_spinlock_t lock;
-} __attribute__((preserve_access_index));
-
-struct rq___new {
- raw_spinlock_t __lock;
-} __attribute__((preserve_access_index));
-
-static void collect_zone_lock(void)
-{
- __u64 nr_zones, zone_off;
- __u64 lock_addr, lock_off;
- __u32 lock_flag = LOCK_CLASS_ZONE_LOCK;
-
- zone_off = offsetof(struct pglist_data, node_zones);
- lock_off = offsetof(struct zone, lock);
-
- if (contig_page_data_addr) {
- struct pglist_data *contig_page_data;
-
- contig_page_data = (void *)(long)contig_page_data_addr;
- nr_zones = BPF_CORE_READ(contig_page_data, nr_zones);
-
- for (int i = 0; i < MAX_ZONES; i++) {
- __u64 zone_addr;
-
- if (i >= nr_zones)
- break;
-
- zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off;
- lock_addr = zone_addr + lock_off;
-
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
- } else if (nr_nodes > 0) {
- struct pglist_data **node_data = (void *)(long)node_data_addr;
-
- for (int i = 0; i < nr_nodes; i++) {
- struct pglist_data *pgdat = NULL;
- int err;
-
- err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
- if (err < 0 || pgdat == NULL)
- break;
-
- nr_zones = BPF_CORE_READ(pgdat, nr_zones);
- for (int k = 0; k < MAX_ZONES; k++) {
- __u64 zone_addr;
-
- if (k >= nr_zones)
- break;
-
- zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off;
- lock_addr = zone_addr + lock_off;
-
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
- }
- }
-}
-
SEC("raw_tp/bpf_test_finish")
-int BPF_PROG(collect_lock_syms)
+int BPF_PROG(end_timestamp)
{
- __u64 lock_addr, lock_off;
- __u32 lock_flag;
-
- if (bpf_core_field_exists(struct rq___new, __lock))
- lock_off = offsetof(struct rq___new, __lock);
- else
- lock_off = offsetof(struct rq___old, lock);
-
- for (int i = 0; i < MAX_CPUS; i++) {
- struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
-
- if (rq == NULL)
- break;
-
- lock_addr = (__u64)(void *)rq + lock_off;
- lock_flag = LOCK_CLASS_RQLOCK;
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
-
- collect_zone_lock();
-
+ end_ts = bpf_ktime_get_ns();
return 0;
}
SEC("raw_tp/bpf_test_finish")
-int BPF_PROG(end_timestamp)
+int BPF_PROG(collect_lock_syms)
{
- end_ts = bpf_ktime_get_ns();
- return 0;
+ return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
}
-/*
- * bpf_iter__kmem_cache added recently so old kernels don't have it in the
- * vmlinux.h. But we cannot add it here since it will cause a compiler error
- * due to redefinition of the struct on later kernels.
- *
- * So it uses a CO-RE trick to access the member only if it has the type.
- * This will support both old and new kernels without compiler errors.
- */
-struct bpf_iter__kmem_cache___new {
- struct kmem_cache *s;
-} __attribute__((preserve_access_index));
-
SEC("iter/kmem_cache")
int slab_cache_iter(void *ctx)
{
- struct kmem_cache *s = NULL;
- struct slab_cache_data d;
- const char *nameptr;
-
- if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) {
- struct bpf_iter__kmem_cache___new *iter = ctx;
-
- s = iter->s;
- }
-
- if (s == NULL)
- return 0;
-
- nameptr = s->name;
- bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr);
-
- d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT;
- if (d.id >= LCB_F_SLAB_ID_END)
- return 0;
-
- bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST);
- return 0;
+ return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
}
char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index 652e114e6b87..317d8ccae63e 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -45,6 +45,14 @@ struct contention_task_data {
#define LCB_F_SLAB_ID_END (1U << 26)
#define LCB_F_SLAB_ID_MASK 0x03FF0000U
+/* lock contention flags from include/trace/events/lock.h */
+#define LCB_F_SPIN (1U << 0)
+#define LCB_F_READ (1U << 1)
+#define LCB_F_WRITE (1U << 2)
+#define LCB_F_RT (1U << 3)
+#define LCB_F_PERCPU (1U << 4)
+#define LCB_F_MUTEX (1U << 5)
+
#define LCB_F_TYPE_MAX (1U << 7)
#define LCB_F_TYPE_MASK 0x0000007FU
@@ -76,4 +84,334 @@ struct slab_cache_data {
char name[SLAB_NAME_MAX];
};
+#ifdef __BPF__
+
+/* for collect_lock_syms(). 4096 was rejected by the verifier */
+#define MAX_CPUS 1024
+
+/* for collect_zone_lock(). It should be more than the actual zones. */
+#define MAX_ZONES 10
+
+extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak;
+extern struct rq runqueues __ksym __weak;
+
+#ifndef __has_builtin
+# define __has_builtin(x) 0
+#endif
+
+struct mutex___non_rt {
+ atomic_long_t owner;
+} __attribute__((preserve_access_index));
+
+struct rt_mutex___old {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
+struct rw_semaphore___old {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
+struct rw_semaphore___new {
+ atomic_long_t count;
+ atomic_long_t owner;
+} __attribute__((preserve_access_index));
+
+struct mm_struct___old {
+ struct rw_semaphore mmap_sem;
+} __attribute__((preserve_access_index));
+
+struct mm_struct___new {
+ struct rw_semaphore mmap_lock;
+} __attribute__((preserve_access_index));
+
+struct rq___old {
+ raw_spinlock_t lock;
+} __attribute__((preserve_access_index));
+
+struct rq___new {
+ raw_spinlock_t __lock;
+} __attribute__((preserve_access_index));
+
+/*
+ * bpf_iter__kmem_cache added recently so old kernels don't have it in the
+ * vmlinux.h. But we cannot add it here since it will cause a compiler error
+ * due to redefinition of the struct on later kernels.
+ *
+ * So it uses a CO-RE trick to access the member only if it has the type.
+ * This will support both old and new kernels without compiler errors.
+ */
+struct bpf_iter__kmem_cache___new {
+ struct kmem_cache *s;
+} __attribute__((preserve_access_index));
+
+static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
+{
+ struct task_struct *task;
+ __u64 owner = 0;
+
+ if (flags & LCB_F_MUTEX) {
+ struct mutex___non_rt *mutex = (void *)lock;
+
+ if (bpf_core_field_exists(mutex->owner.counter)) {
+ owner = BPF_CORE_READ(mutex, owner.counter);
+ if (owner & 0x4UL)
+ return NULL;
+ } else if (bpf_core_type_exists(struct rt_mutex_base)) {
+ struct rt_mutex_base *rtm = (void *)lock;
+
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ } else {
+ struct rt_mutex___old *rtm = (void *)lock;
+
+ if (bpf_core_field_exists(rtm->owner)) {
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ }
+ }
+ } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
+ /*
+ * Support for the BPF_TYPE_MATCHES argument to the
+ * __builtin_preserve_type_info builtin was added at some point during
+ * development of clang 15 and it's what is needed for
+ * bpf_core_type_matches.
+ */
+#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
+ if (bpf_core_type_matches(struct rw_semaphore___old)) {
+ struct rw_semaphore___old *old = (void *)lock;
+
+ owner = (unsigned long)BPF_CORE_READ(old, owner);
+ } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
+ struct rw_semaphore___new *new = (void *)lock;
+
+ owner = BPF_CORE_READ(new, owner.counter);
+ }
+#else
+ /* assume new struct */
+ struct rw_semaphore___new *new = (void *)lock;
+
+ if (bpf_core_field_exists(new->owner.counter))
+ owner = BPF_CORE_READ(new, owner.counter);
+#endif
+ } else if (flags == LCB_F_RT) {
+ if (bpf_core_type_exists(struct rt_mutex_base)) {
+ struct rt_mutex_base *rtm = (void *)lock;
+
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ } else {
+ struct rt_mutex___old *rtm = (void *)lock;
+
+ if (bpf_core_field_exists(rtm->owner)) {
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ }
+ }
+ }
+
+ if (!owner)
+ return NULL;
+
+ task = (void *)(owner & ~7UL);
+ return task;
+}
+
+static __always_inline __u64 get_current_mmap_lock(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+ struct mm_struct___new *mm_new = NULL;
+ struct mm_struct___old *mm_old = NULL;
+ __u64 mm;
+
+ if (!curr)
+ return 0;
+
+ mm = (__u64)BPF_CORE_READ(task, mm);
+ if (!mm)
+ return 0;
+
+ if (bpf_core_field_exists(mm_new->mmap_lock))
+ return mm + bpf_core_field_offset(mm_new->mmap_lock);
+
+ if (bpf_core_field_exists(mm_old->mmap_sem))
+ return mm + bpf_core_field_offset(mm_old->mmap_sem);
+
+ return 0;
+}
+
+static __always_inline __u64 get_current_siglock(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+ struct sighand_struct *sighand = NULL;
+ __u64 sh;
+
+ if (!curr)
+ return 0;
+
+ sh = (__u64)BPF_CORE_READ(task, sighand);
+ if (!sh)
+ return 0;
+
+ return sh + bpf_core_field_offset(sighand->siglock);
+}
+
+/*
+ * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar.
+ *
+ * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the
+ * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed
+ * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such
+ * as `bpf_get_kmem_cache(u64 addr)`.
+ */
+static __always_inline __u64 get_lock_addr(void *ctx)
+{
+ __u64 lock_addr = 0;
+
+ bpf_get_func_arg(ctx, 0, &lock_addr);
+ return lock_addr;
+}
+
+static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags)
+{
+ __u64 curr;
+
+ switch (flags) {
+ case LCB_F_READ: /* rwsem */
+ case LCB_F_WRITE:
+ curr = bpf_get_current_task();
+ if (get_current_mmap_lock(curr) == lock)
+ return LCD_F_MMAP_LOCK;
+ break;
+ case LCB_F_SPIN: /* spinlock */
+ curr = bpf_get_current_task();
+ if (get_current_siglock(curr) == lock)
+ return LCD_F_SIGHAND_LOCK;
+ break;
+ default:
+ break;
+ }
+ return 0;
+}
+
+static __always_inline void do_collect_zone_lock(void *lock_syms_map,
+ __u64 contig_page_data_addr,
+ __u64 node_data_addr,
+ int nr_nodes,
+ int sizeof_zone)
+{
+ __u64 nr_zones, zone_off;
+ __u64 lock_addr, lock_off;
+ __u32 lock_flag = LOCK_CLASS_ZONE_LOCK;
+
+ zone_off = offsetof(struct pglist_data, node_zones);
+ lock_off = offsetof(struct zone, lock);
+
+ if (contig_page_data_addr) {
+ struct pglist_data *contig_page_data;
+
+ contig_page_data = (void *)(long)contig_page_data_addr;
+ nr_zones = BPF_CORE_READ(contig_page_data, nr_zones);
+
+ for (int i = 0; i < MAX_ZONES; i++) {
+ __u64 zone_addr;
+
+ if (i >= nr_zones)
+ break;
+
+ zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off;
+ lock_addr = zone_addr + lock_off;
+
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ } else if (node_data_addr && nr_nodes > 0) {
+ struct pglist_data **node_data = (void *)(long)node_data_addr;
+
+ for (int i = 0; i < nr_nodes; i++) {
+ struct pglist_data *pgdat = NULL;
+ int err;
+
+ err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
+ if (err < 0)
+ break;
+ if (pgdat == NULL)
+ continue;
+
+ nr_zones = BPF_CORE_READ(pgdat, nr_zones);
+ for (int k = 0; k < MAX_ZONES; k++) {
+ __u64 zone_addr;
+
+ if (k >= nr_zones)
+ break;
+
+ zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off;
+ lock_addr = zone_addr + lock_off;
+
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ }
+ }
+}
+
+static __always_inline int do_collect_lock_syms(void *lock_syms_map,
+ __u64 contig_page_data_addr,
+ __u64 node_data_addr,
+ int nr_nodes,
+ int sizeof_zone)
+{
+ __u64 lock_addr, lock_off;
+ __u32 lock_flag;
+
+ if (bpf_core_field_exists(struct rq___new, __lock))
+ lock_off = offsetof(struct rq___new, __lock);
+ else
+ lock_off = offsetof(struct rq___old, lock);
+
+ if (&runqueues) {
+ for (int i = 0; i < MAX_CPUS; i++) {
+ struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
+
+ if (rq == NULL)
+ break;
+
+ lock_addr = (__u64)(void *)rq + lock_off;
+ lock_flag = LOCK_CLASS_RQLOCK;
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ }
+
+ do_collect_zone_lock(lock_syms_map, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
+
+ return 0;
+}
+
+static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map,
+ __u32 *slab_cache_id)
+{
+ struct kmem_cache *s = NULL;
+ struct slab_cache_data d = {};
+ const char *nameptr;
+
+ if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) {
+ struct bpf_iter__kmem_cache___new *iter = ctx;
+
+ s = iter->s;
+ }
+
+ if (s == NULL)
+ return 0;
+
+ nameptr = s->name;
+ bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr);
+
+ d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT;
+ if (d.id >= LCB_F_SLAB_ID_END)
+ return 0;
+
+ bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST);
+ return 0;
+}
+
+#endif /* __BPF__ */
+
#endif /* UTIL_BPF_SKEL_LOCK_DATA_H */
diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
index 94e42b022ab1..b75ddc6846bb 100644
--- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
+++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
@@ -76,6 +76,7 @@ struct signal_struct {
} __attribute__((preserve_access_index));
struct rw_semaphore {
+ atomic_long_t count;
atomic_long_t owner;
} __attribute__((preserve_access_index));
@@ -83,6 +84,10 @@ struct mutex {
atomic_long_t owner;
} __attribute__((preserve_access_index));
+struct rt_mutex_base {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
struct kernfs_node {
u64 id;
} __attribute__((preserve_access_index));
diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c
index 119a7206f3cd..f6e68d740860 100644
--- a/tools/perf/util/lock-contention.c
+++ b/tools/perf/util/lock-contention.c
@@ -6,6 +6,7 @@
#include "symbol.h"
#include <limits.h>
+#include <stdlib.h>
#include <string.h>
#include <linux/hash.h>
@@ -22,6 +23,24 @@ struct callstack_filter {
static LIST_HEAD(callstack_filters);
struct hlist_head *lockhash_table;
+const struct perf_lock_type lock_type_table[] = {
+ { 0, "semaphore", "semaphore" },
+ { LCB_F_SPIN, "spinlock", "spinlock" },
+ { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
+ { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
+ { LCB_F_READ, "rwsem:R", "rwsem" },
+ { LCB_F_WRITE, "rwsem:W", "rwsem" },
+ { LCB_F_RT, "rt-mutex", "rt-mutex" },
+ { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
+ { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
+ { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
+ { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
+ { LCB_F_MUTEX, "mutex", "mutex" },
+ { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
+ /* alias for optimistic spinning only */
+ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
+};
+
int parse_call_stack(const struct option *opt __maybe_unused, const char *str,
int unset __maybe_unused)
{
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index 59c94190b092..68e9bd77e8a9 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -129,6 +129,19 @@ struct thread_stat {
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+struct perf_lock_type {
+ unsigned int flags;
+ /*
+ * Name of the lock flags (access), with delimiter ':'.
+ * For example, rwsem:R or rwsem:W.
+ */
+ const char *flags_name;
+ /* Name of the lock (type), for example, rwlock or rwsem. */
+ const char *lock_name;
+};
+
+extern const struct perf_lock_type lock_type_table[14];
+
struct evlist;
struct machine;
struct target;
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter
2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (3 preceding siblings ...)
2026-10-20 20:58 ` [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
@ 2026-10-20 20:58 ` Ian Rogers
2026-10-20 22:49 ` Namhyung Kim
2026-10-20 23:17 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 6/8] perf test: Add tests for lock_held() BPF " Ian Rogers
` (3 subsequent siblings)
8 siblings, 2 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Add BPF sample filter predicates lock_held(<lock>),
lock_held_read(<lock>) / lock_held_shared(<lock>), and
lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R'
and ':W' suffixes on <lock>) to allow recording perf event samples only
when a specified kernel lock is held (or not held with '!').
Supported <lock> specifications match perf lock contention:
- Global/static kernel lock symbols (e.g. lock_held(cgroup_mutex))
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with '&' (e.g. lock_held(&dentry),
lock_held_read(&dentry))
- Per-process and semi-global locks: mmap_lock, siglock, rq_lock,
zone_lock
- Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W,
rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W,
semaphore
Supported lock hold modes by lock family:
- Both read/shared and write/exclusive: rwsem (including mmap_lock)
and rwlock.
- Write/exclusive only: mutex, spinlock (including siglock, rq_lock,
zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write()
/ percpu_up_write(); fast-path percpu_down_read() / percpu_up_read()
are inlined per-CPU counter operations without function entry points).
When evaluating the filter on sample overflow, perf_sample_filter first
inspects the lock's owner field directly in kernel memory where
possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls
back to a held_locks BPF LRU hash map (tracking separate read_count and
write_count) populated by fentry/fexit hooks on kernel lock/unlock
functions. When filtering on a specific mutex symbol (*_mutex),
rt-mutex symbol (*_rt_mutex), or write-locked mmap_lock (mmap_lock:W /
lock_held_write(mmap_lock)), no fentry/fexit hooks are attached at all.
Note that counting semaphores (down*() / up()) do not enforce same-task
ownership in the kernel: when a semaphore is used as a cross-task
completion/signaling mechanism (down() in one task woken by up() in
another task or interrupt handler, which proposed kernel trackers like
CONFIG_DEPT model as wait/event dependencies bounded by execution
context rather than lock ownership), fexit/down and fentry/up will not
pair on the same task ID. Semaphore hooks are therefore only enabled
when 'semaphore' or a '*_sem' symbol (such as console_sem) is
explicitly specified, and held_locks uses BPF_MAP_TYPE_LRU_HASH so any
unpaired cross-task entries are automatically evicted.
Example usage:
# perf record -a -g -e cycles --filter 'lock_held(mmap_lock)' \
-- perf bench mem mmap -t 2 -l 100
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ]
# perf report --stdio --no-children
# Overhead Command Shared Object Symbol
# ........ ........ ................. ........................
#
18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms
|
---clear_page_erms
get_page_from_freelist
__alloc_pages_noprof
folio_alloc_mpol_noprof
vma_alloc_folio_noprof
do_anonymous_page
__handle_mm_fault
handle_mm_fault
do_user_addr_fault
exc_page_fault
asm_exc_page_fault
worker_thread
Read/shared vs write/exclusive holding can also be filtered separately:
# perf record -a -g -e cycles --filter 'lock_held_read(mmap_lock)' ...
# perf record -a -g -e cycles --filter 'lock_held_write(mmap_lock)' ...
# perf record -a -g -e cycles --filter 'lock_held(mmap_lock:W)' ...
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 58 +-
tools/perf/util/bpf-filter.c | 1071 +++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 47 +-
tools/perf/util/bpf_skel/sample-filter.h | 66 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 1014 +++++++++++++++++
tools/perf/util/parse-events.c | 3 +-
8 files changed, 2277 insertions(+), 26 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 2351f7b4f5fc..8496b20987a0 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -204,7 +204,63 @@ OPTIONS
ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr,
code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat,
p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock,
- mem_dtlb, mem_blk, mem_hops, uid, gid
+ mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>),
+ lock_held_read(<lock>), lock_held_shared(<lock>),
+ lock_held_write(<lock>), lock_held_exclusive(<lock>)
+
+ The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters
+ samples based on whether the specified kernel lock is currently held
+ by the kernel thread or CPU. To differentiate between shared/read
+ holding and exclusive/write holding, use 'lock_held_read(<lock>)'
+ (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)'
+ (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to
+ <lock> (e.g. 'lock_held(mmap_lock:R)' or 'lock_held(mmap_lock:W)').
+ <lock> can be a kernel lock symbol name (e.g.
+ 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache
+ name prefixed with '&' (e.g. 'lock_held(&dentry)',
+ 'lock_held_read(&dentry)'), a well-known lock name ('mmap_lock',
+ 'siglock', 'rq_lock', 'zone_lock'), or a lock type ('mutex',
+ 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R',
+ 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore').
+ Both read/shared and write/exclusive tracking are supported for
+ 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', 'spinlock'
+ (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex',
+ 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support
+ write/exclusive tracking only. Note that 'semaphore' tracking
+ ('down*()' / 'up()') assumes binary mutex-like usage where the
+ acquiring task also releases the semaphore (such as 'console_sem'),
+ rather than cross-task completion signaling.
+
+ For example, 'lock_held()' can be combined with memory allocation or
+ reclaim events to detect allocations or direct memory reclaim occurring
+ while a kernel lock is held (which can stall lock waiters for extended
+ periods):
+
+ # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+ # perf record -a -g -e kmem:mm_page_alloc \
+ --filter 'gfp_flags & 0x400' \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+ # perf record -a -g -e major-faults \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+
+ When filtering on a specific mutex symbol ('*_mutex'), rt-mutex symbol
+ ('*_rt_mutex'), or write-locked 'mmap_lock' ('lock_held_write(mmap_lock)'
+ or 'lock_held(mmap_lock:W)'), 'perf_sample_filter' inspects the lock's
+ 'owner' field directly in kernel memory on each sample without attaching
+ any BPF fentry/fexit hooks. For locks that do not maintain an
+ authoritative per-task owner pointer (such as spinlocks, rwlocks,
+ semaphores, reader-locked rwsems, and slab/type filters), it attaches
+ BPF fentry/fexit programs to kernel lock acquire and release functions
+ to track held locks in a BPF map. These BPF hooks run on every kernel
+ lock acquisition and release for the attached lock family, even when no
+ perf sample is being taken. To minimize this overhead, only the hooks
+ for the relevant lock family and mode are attached when known (for
+ example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions
+ and 'rq_lock' only hooks spinlock functions), whereas slab filters
+ ('&<slab>'), raw hex addresses, or symbols without a recognizable
+ lock-type suffix attach hooks across all lock families (including
+ high-frequency spinlocks).
The <operator> can be one of:
==, !=, >, >=, <, <=, &
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index ac0e7cf40f3a..d5a85ee98d5f 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -40,21 +40,35 @@
* The 'dropped' map is to keep how many samples it dropped by the filter and
* it will be reported as lost samples.
*/
+#include <errno.h>
+#include <stdio.h>
#include <stdlib.h>
+#include <stdbool.h>
+#include <stddef.h>
+#include <string.h>
#include <fcntl.h>
+#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/stat.h>
#include <bpf/bpf.h>
+#include <bpf/btf.h>
#include <linux/err.h>
+#include <linux/kernel.h>
#include <linux/list.h>
+#include <linux/string.h>
+#include <linux/zalloc.h>
#include <api/fs/fs.h>
#include <internal/xyarray.h>
#include <perf/threadmap.h>
+#include <symbol/kallsyms.h>
#include "util/cap.h"
#include "util/debug.h"
#include "util/evsel.h"
+#include "util/lock-contention.h"
+#include "util/string2.h"
+#include "util/symbol_conf.h"
#include "util/target.h"
#include "util/thread_map.h"
#include "util/bpf-utils.h"
@@ -63,6 +77,7 @@
#include <util/bpf-filter-flex.h>
#include <util/bpf-filter-bison.h>
+#include "bpf_skel/lock_data.h"
#include "bpf_skel/sample-filter.h"
#include "bpf_skel/sample_filter.skel.h"
@@ -123,7 +138,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
{
const struct perf_sample_info *info;
- if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
+ if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID ||
+ expr->term == PBF_TERM_LOCK_HELD) {
/* Not dependent on the sample_type as computed from a BPF helper. */
return 0;
}
@@ -166,9 +182,913 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
return -1;
}
-static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry)
+/*
+ * Program groups for `lock_held(...)` BPF tracing hooks.
+ *
+ * When a `lock_held(...)` filter is specified, only the BPF tracing program
+ * groups required for the requested lock(s) are autoloaded and attached.
+ * If no `lock_held(...)` filter is present, none of these programs are loaded.
+ */
+enum lock_prog_group {
+ LOCK_PROG_MUTEX = 1U << 0,
+ LOCK_PROG_RWSEM_READ = 1U << 1,
+ LOCK_PROG_RWSEM_WRITE = 1U << 2,
+ LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE,
+ LOCK_PROG_SPIN = 1U << 3,
+ LOCK_PROG_RWLOCK_READ = 1U << 4,
+ LOCK_PROG_RWLOCK_WRITE = 1U << 5,
+ LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE,
+ LOCK_PROG_RTMUTEX = 1U << 6,
+ LOCK_PROG_SEM = 1U << 7,
+ LOCK_PROG_PERCPU_RWSEM = 1U << 8,
+ LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM,
+};
+
+static unsigned int lock_flags_to_prog_group(unsigned int flags)
+{
+ if (flags & LCB_F_MUTEX)
+ return LOCK_PROG_MUTEX;
+ if (flags & LCB_F_RT)
+ return LOCK_PROG_RTMUTEX;
+ if (flags & LCB_F_PERCPU)
+ return LOCK_PROG_PERCPU_RWSEM;
+ if (flags == LCB_F_SPIN)
+ return LOCK_PROG_SPIN;
+ if (flags & LCB_F_SPIN)
+ return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE;
+ if (flags & LCB_F_READ)
+ return LOCK_PROG_RWSEM_READ;
+ if (flags & LCB_F_WRITE)
+ return LOCK_PROG_RWSEM_WRITE;
+ return LOCK_PROG_SEM;
+}
+
+struct lock_prog_desc {
+ size_t prog_offset;
+ const char *btf_func;
+ unsigned int group;
+};
+
+#define LOCK_FPROG(field, func, grp) \
+ { offsetof(struct sample_filter_bpf, progs.field), func, grp }
+
+static const struct lock_prog_desc lock_progs[] = {
+ /* Mutex */
+ LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
+ LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
+ /* RW semaphore */
+ LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested",
+ LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested",
+ LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM),
+ LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE),
+ /* Spinlock */
+ LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN),
+ /* RWlock */
+ LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE),
+ /* RT-Mutex */
+ LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible",
+ LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX),
+ /* Semaphore */
+ LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM),
+ LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM),
+ /* Percpu-RWsem */
+ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
+ LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+};
+
+#undef LOCK_FPROG
+
+static int split_lock_name_mode(const char *name, int expr_part,
+ char *base_buf, size_t buf_sz,
+ bool *has_amp, unsigned int *mode_out)
+{
+ const char *bare;
+ size_t len;
+ unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK;
+ unsigned int suffix_mode = 0;
+
+ if (!name || !*name)
+ return -EINVAL;
+
+ *has_amp = (name[0] == '&');
+ bare = *has_amp ? name + 1 : name;
+ len = strlen(bare);
+ if (len == 0)
+ return -EINVAL;
+
+ if (len > 2 && !strcmp(bare + len - 2, ":R")) {
+ suffix_mode = PBF_LOCK_MODE_READ;
+ len -= 2;
+ } else if (len > 2 && !strcmp(bare + len - 2, ":W")) {
+ suffix_mode = PBF_LOCK_MODE_WRITE;
+ len -= 2;
+ }
+
+ if (len == 0 || len >= buf_sz)
+ return -EINVAL;
+
+ if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY &&
+ suffix_mode && expr_mode != suffix_mode)
+ return -EINVAL;
+
+ memcpy(base_buf, bare, len);
+ base_buf[len] = '\0';
+
+ if (suffix_mode)
+ *mode_out = suffix_mode;
+ else if (expr_mode)
+ *mode_out = expr_mode;
+ else
+ *mode_out = PBF_LOCK_MODE_ANY;
+
+ return 0;
+}
+
+/*
+ * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R",
+ * "spinlock", "percpu-rwsem").
+ *
+ * A lock type name may correspond to up to two flag combinations (e.g.
+ * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed
+ * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is
+ * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`.
+ */
+static bool parse_lock_type_name(const char *name, unsigned int mode,
+ struct sample_filter_bpf *skel,
+ u64 *encoded_pair, unsigned int *prog_groups)
+{
+ bool found = false;
+ u32 flag1 = 0xff, flag2 = 0xff;
+ unsigned int groups = 0;
+ const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name;
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) {
+ u32 f = lock_type_table[i].flags;
+ bool is_read = (f & LCB_F_READ) != 0;
+
+ if (strcmp(lock_type_table[i].lock_name, lookup) &&
+ strcmp(lock_type_table[i].flags_name, name))
+ continue;
+
+ /*
+ * percpu_down_read() / percpu_up_read() fast paths are inlined
+ * per-CPU counter increments without function entry points, so
+ * only writer holding (percpu_down_write / percpu_up_write) can
+ * be tracked for percpu-rwsem. rwlock-rt is also not hooked.
+ */
+ if ((f & LCB_F_PERCPU) && is_read)
+ continue;
+ if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE)))
+ continue;
+ if ((f & LCB_F_MUTEX) && (f & LCB_F_SPIN))
+ continue;
+
+ if (is_read && !(mode & PBF_LOCK_MODE_READ))
+ continue;
+ if (!is_read && !(mode & PBF_LOCK_MODE_WRITE))
+ continue;
+
+ if (!found)
+ flag1 = f;
+ else if (f != flag1)
+ flag2 = f;
+ if (skel && f < LCB_F_TYPE_MAX) {
+ skel->bss->has_lock_type = 1;
+ skel->bss->lock_type_filter[f] = 1;
+ }
+ groups |= lock_flags_to_prog_group(f);
+ found = true;
+ }
+
+ if (!found)
+ return false;
+
+ if (encoded_pair)
+ *encoded_pair = flag1 | ((u64)flag2 << 8);
+ if (prog_groups)
+ *prog_groups |= groups;
+ return true;
+}
+
+struct lock_sym_cb_args {
+ const char *name;
+ u64 addr;
+ bool found;
+};
+
+static int find_lock_sym_cb(void *arg, const char *name,
+ char type __maybe_unused, u64 start)
+{
+ struct lock_sym_cb_args *args = arg;
+
+ if (!strcmp(name, args->name)) {
+ args->found = true;
+ args->addr = start;
+ if (start != 0)
+ return 1;
+ }
+ return 0;
+}
+
+static int find_kernel_lock_sym(const char *name, u64 *addr)
+{
+ struct lock_sym_cb_args args = {
+ .name = name,
+ };
+ const char *kallsyms_filename = "/proc/kallsyms";
+
+ if (symbol_conf.kallsyms_name != NULL)
+ kallsyms_filename = symbol_conf.kallsyms_name;
+
+ if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0)
+ return -ENOENT;
+
+ if (!args.found)
+ return -ENOENT;
+
+ if (args.addr == 0) {
+ pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n",
+ name);
+ return -EACCES;
+ }
+
+ *addr = args.addr;
+ return 0;
+}
+
+static bool parse_lock_hex_addr(const char *str, u64 *addr)
+{
+ char *end;
+ unsigned long long val;
+
+ if (!str || !*str)
+ return false;
+
+ errno = 0;
+ val = strtoull(str, &end, 16);
+ if (*end != '\0' || errno != 0 || val == 0)
+ return false;
+
+ *addr = val;
+ return true;
+}
+
+static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf)
+{
+ u64 addr = 0;
+ char *buf = NULL, *p;
+ size_t len;
+ long last = -1;
+ int ret;
+
+ if (!btf)
+ return;
+
+ ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT);
+ if (ret < 0)
+ return;
+
+ ret = btf__resolve_size(btf, ret);
+ if (ret < 0)
+ return;
+
+ skel->rodata->sizeof_zone = ret;
+
+ if (find_kernel_lock_sym("contig_page_data", &addr) == 0) {
+ skel->rodata->contig_page_data_addr = addr;
+ return;
+ }
+
+ if (find_kernel_lock_sym("node_data", &addr) != 0)
+ return;
+
+ skel->rodata->node_data_addr = addr;
+
+ if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) {
+ p = buf;
+ while (p && *p) {
+ char *end;
+
+ last = strtol(p, &end, 0);
+ if (end == p)
+ break;
+ p = end;
+ if (*p == ',' || *p == '-' || *p == '\n')
+ p++;
+ }
+ skel->rodata->nr_nodes = last + 1;
+ free(buf);
+ }
+}
+
+static bool btf_has_func(struct btf *btf, const char *name)
+{
+ return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0;
+}
+
+static unsigned int supported_lock_prog_groups(struct btf *btf)
+{
+ unsigned int supported = 0;
+
+ if (!btf)
+ return 0;
+
+ if ((btf_has_func(btf, "mutex_lock") || btf_has_func(btf, "mutex_lock_nested")) &&
+ btf_has_func(btf, "mutex_unlock"))
+ supported |= LOCK_PROG_MUTEX;
+
+ if ((btf_has_func(btf, "down_read") || btf_has_func(btf, "down_read_nested")) &&
+ btf_has_func(btf, "up_read"))
+ supported |= LOCK_PROG_RWSEM_READ;
+
+ if ((btf_has_func(btf, "down_write") || btf_has_func(btf, "down_write_nested")) &&
+ btf_has_func(btf, "up_write"))
+ supported |= LOCK_PROG_RWSEM_WRITE;
+
+ /*
+ * Require both acquire and all unlock entry points in BTF so we never
+ * attach spinlock/rwlock acquire hooks when unlock functions are
+ * inlined (e.g. !CONFIG_UNINLINE_SPIN_UNLOCK or CONFIG_INLINE_*_UNLOCK).
+ */
+ if ((btf_has_func(btf, "_raw_spin_lock") || btf_has_func(btf, "_raw_spin_lock_nested")) &&
+ btf_has_func(btf, "_raw_spin_unlock") &&
+ btf_has_func(btf, "_raw_spin_unlock_irq") &&
+ btf_has_func(btf, "_raw_spin_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_spin_unlock_bh"))
+ supported |= LOCK_PROG_SPIN;
+
+ if (btf_has_func(btf, "_raw_read_lock") &&
+ btf_has_func(btf, "_raw_read_unlock") &&
+ btf_has_func(btf, "_raw_read_unlock_irq") &&
+ btf_has_func(btf, "_raw_read_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_read_unlock_bh"))
+ supported |= LOCK_PROG_RWLOCK_READ;
+
+ if ((btf_has_func(btf, "_raw_write_lock") || btf_has_func(btf, "_raw_write_lock_nested")) &&
+ btf_has_func(btf, "_raw_write_unlock") &&
+ btf_has_func(btf, "_raw_write_unlock_irq") &&
+ btf_has_func(btf, "_raw_write_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_write_unlock_bh"))
+ supported |= LOCK_PROG_RWLOCK_WRITE;
+
+ if ((btf_has_func(btf, "rt_mutex_lock") || btf_has_func(btf, "rt_mutex_lock_nested")) &&
+ btf_has_func(btf, "rt_mutex_unlock"))
+ supported |= LOCK_PROG_RTMUTEX;
+
+ if (btf_has_func(btf, "down") && btf_has_func(btf, "up"))
+ supported |= LOCK_PROG_SEM;
+
+ if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write"))
+ supported |= LOCK_PROG_PERCPU_RWSEM;
+
+ return supported;
+}
+
+/*
+ * Enable or disable autoload for each lock-tracking BPF program in the
+ * skeleton before `sample_filter_bpf__load()` is called.
+ *
+ * Each program is enabled only if its group is in `needed_groups` and its
+ * target kernel function or tracepoint exists in the running kernel's BTF.
+ */
+static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
+ unsigned int needed_groups,
+ unsigned int required_groups,
+ bool needs_lock_syms, bool needs_slab_iter,
+ bool is_tracepoint)
+{
+ unsigned int supported_groups = supported_lock_prog_groups(btf);
+ unsigned int enabled_progs = 0;
+
+ bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
+ bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
+
+ if ((required_groups & supported_groups) != required_groups ||
+ (needed_groups && !(needed_groups & supported_groups))) {
+ pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n");
+ return -ENOTSUP;
+ }
+
+ needed_groups &= supported_groups;
+ if (skel)
+ skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ);
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) {
+ struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset;
+ struct bpf_program *prog = *progp;
+ bool enable = false;
+
+ if ((needed_groups & lock_progs[i].group) && btf &&
+ lock_progs[i].btf_func) {
+ if (btf__find_by_name_kind(btf, lock_progs[i].btf_func,
+ BTF_KIND_FUNC) > 0)
+ enable = true;
+ }
+
+ if (enable)
+ enabled_progs++;
+ bpf_program__set_autoload(prog, enable);
+ }
+
+ bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms);
+ if (needs_slab_iter && btf &&
+ btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) {
+ bpf_program__set_autoload(skel->progs.slab_cache_iter, true);
+ } else {
+ bpf_program__set_autoload(skel->progs.slab_cache_iter, false);
+ }
+
+ if (needed_groups && !enabled_progs) {
+ pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
+ return -ENOTSUP;
+ }
+ return 0;
+}
+
+/*
+ * Pre-scan a single filter expression prior to loading the BPF skeleton in
+ * order to determine which BPF maps need to be sized and which lock tracing
+ * program groups need to be autoloaded.
+ */
+static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
+ bool *has_lock_filter,
+ unsigned int *prog_groups,
+ unsigned int *required_groups,
+ bool *needs_lock_syms,
+ bool *needs_slab_iter)
+{
+ char base[128];
+ bool has_amp;
+ unsigned int mode, groups = 0;
+ u64 dummy;
+
+ if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name)
+ return;
+
+ *has_lock_filter = true;
+ if (split_lock_name_mode(expr->lock_name, expr->part, base,
+ sizeof(base), &has_amp, &mode) < 0)
+ return;
+
+ /* Leading '&' is reserved for slab cache locks (e.g. '&dentry'). */
+ if (has_amp) {
+ *needs_slab_iter = true;
+ *prog_groups |= LOCK_PROG_ALL;
+ return;
+ }
+
+ if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
+ /*
+ * Write ownership of mm->mmap_lock is inspected directly via
+ * is_rwsem_writer_owner() without tracing hooks; only reader
+ * tracking requires LOCK_PROG_RWSEM_READ.
+ */
+ if (mode & PBF_LOCK_MODE_READ) {
+ *prog_groups |= LOCK_PROG_RWSEM_READ;
+ *required_groups |= LOCK_PROG_RWSEM_READ;
+ }
+ return;
+ }
+ if (!strcmp(base, "siglock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ return;
+ }
+ if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ *needs_lock_syms = true;
+ return;
+ }
+ if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) {
+ *prog_groups |= groups;
+ *required_groups |= groups;
+ return;
+ }
+
+ if (parse_lock_hex_addr(base, &dummy)) {
+ *prog_groups |= LOCK_PROG_ALL;
+ return;
+ }
+
+ if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0 ||
+ strtailcmp(base, "mutex") == 0) {
+ /* Inspected directly via lock->owner; no tracing hooks needed. */
+ return;
+ } else if (strtailcmp(base, "percpu_rwsem") == 0 ||
+ strtailcmp(base, "pcpu_rwsem") == 0) {
+ *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
+ *required_groups |= LOCK_PROG_PERCPU_RWSEM;
+ } else if (strtailcmp(base, "rwsem") == 0) {
+ if (mode & PBF_LOCK_MODE_READ) {
+ *prog_groups |= LOCK_PROG_RWSEM_READ;
+ *required_groups |= LOCK_PROG_RWSEM_READ;
+ }
+ if (mode & PBF_LOCK_MODE_WRITE) {
+ *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM;
+ *required_groups |= LOCK_PROG_RWSEM_WRITE;
+ }
+ } else if (strtailcmp(base, "spinlock") == 0) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ } else if (strtailcmp(base, "sem") == 0) {
+ *prog_groups |= LOCK_PROG_SEM;
+ *required_groups |= LOCK_PROG_SEM;
+ } else if (strtailcmp(base, "rwlock") == 0) {
+ if (mode & PBF_LOCK_MODE_READ)
+ groups |= LOCK_PROG_RWLOCK_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ groups |= LOCK_PROG_RWLOCK_WRITE;
+ *prog_groups |= groups;
+ *required_groups |= groups;
+ } else {
+ *prog_groups |= LOCK_PROG_ALL;
+ }
+}
+
+static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter,
+ unsigned int *prog_groups,
+ unsigned int *required_groups,
+ bool *needs_lock_syms,
+ bool *needs_slab_iter)
+{
+ struct perf_bpf_filter_expr *expr, *group;
+
+ *has_lock_filter = false;
+ *prog_groups = 0;
+ *required_groups = 0;
+ *needs_lock_syms = false;
+ *needs_slab_iter = false;
+
+ list_for_each_entry(expr, &evsel->bpf_filters, list) {
+ prescan_one_lock_expr(expr, has_lock_filter, prog_groups,
+ required_groups, needs_lock_syms,
+ needs_slab_iter);
+ if (expr->op == PBF_OP_GROUP_BEGIN) {
+ list_for_each_entry(group, &expr->groups, list) {
+ prescan_one_lock_expr(group, has_lock_filter,
+ prog_groups, required_groups,
+ needs_lock_syms, needs_slab_iter);
+ }
+ }
+ }
+}
+
+static void run_slab_cache_iter(struct sample_filter_bpf *skel)
+{
+ struct bpf_link *link;
+ int fd;
+ char buf[256];
+
+ if (!bpf_program__autoload(skel->progs.slab_cache_iter))
+ return;
+
+ link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL);
+ if (!link || IS_ERR(link)) {
+ pr_debug("cannot attach slab cache iter\n");
+ return;
+ }
+
+ fd = bpf_iter_create(bpf_link__fd(link));
+ if (fd >= 0) {
+ while (read(fd, buf, sizeof(buf)) > 0)
+ continue;
+ close(fd);
+ }
+
+ bpf_link__destroy(link);
+}
+
+static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
+ u64 *kmem_cache_addr)
+{
+ int cache_fd;
+ u64 key, *prev_key = NULL;
+
+ if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter))
+ return false;
+
+ cache_fd = bpf_map__fd(skel->maps.slab_caches);
+ while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) {
+ struct slab_cache_data data;
+
+ if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0)
+ break;
+
+ if (!strcmp(data.name, name)) {
+ *kmem_cache_addr = key;
+ return true;
+ }
+ prev_key = &key;
+ }
+ return false;
+}
+
+static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr,
+ enum perf_bpf_filter_op *op_out)
+{
+ switch (expr->op) {
+ case PBF_OP_EQ:
+ if (expr->val == 1) {
+ *op_out = PBF_OP_EQ;
+ return 0;
+ }
+ if (expr->val == 0) {
+ *op_out = PBF_OP_NEQ;
+ return 0;
+ }
+ break;
+ case PBF_OP_NEQ:
+ if (expr->val == 0) {
+ *op_out = PBF_OP_EQ;
+ return 0;
+ }
+ if (expr->val == 1) {
+ *op_out = PBF_OP_NEQ;
+ return 0;
+ }
+ break;
+ case PBF_OP_GT:
+ if (expr->val == 0) {
+ *op_out = PBF_OP_EQ;
+ return 0;
+ }
+ break;
+ case PBF_OP_GE:
+ if (expr->val == 1) {
+ *op_out = PBF_OP_EQ;
+ return 0;
+ }
+ break;
+ case PBF_OP_LT:
+ if (expr->val == 1) {
+ *op_out = PBF_OP_NEQ;
+ return 0;
+ }
+ break;
+ case PBF_OP_LE:
+ if (expr->val == 0) {
+ *op_out = PBF_OP_NEQ;
+ return 0;
+ }
+ break;
+ case PBF_OP_AND:
+ if (expr->val & 1) {
+ *op_out = PBF_OP_EQ;
+ return 0;
+ }
+ break;
+ case PBF_OP_GROUP_BEGIN:
+ case PBF_OP_GROUP_END:
+ case PBF_OP_DONE:
+ default:
+ break;
+ }
+ pr_err("Error: Invalid comparison for boolean lock_held(%s) predicate\n",
+ expr->lock_name ?: "");
+ return -EINVAL;
+}
+
+/*
+ * Resolve a `lock_held(<name>)` filter expression into `entry->op`,
+ * `entry->part` (`enum perf_bpf_lock_target_kind`), and `entry->value`,
+ * and populate the corresponding BPF filter maps and BSS control flags.
+ *
+ * Resolution order:
+ * 1. Slab cache name (when prefixed with '&', e.g. `&dentry`).
+ * 2. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
+ * `rq_lock`, `zone_lock`.
+ * 3. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`,
+ * `rwlock`, `percpu-rwsem`, `semaphore`, etc.
+ * 4. Hexadecimal lock address.
+ * 5. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`).
+ */
+static int resolve_lock_expr(struct sample_filter_bpf *skel,
+ struct perf_bpf_filter_expr *expr,
+ struct perf_bpf_filter_entry *entry)
+{
+ const char *name = expr->lock_name;
+ char base[128];
+ bool has_amp;
+ unsigned int mode;
+ u64 addr = 0, encoded_pair = 0;
+ u8 one = 1;
+ int ret;
+
+ ret = normalize_lock_expr_op(expr, &entry->op);
+ if (ret < 0)
+ return ret;
+
+ if (split_lock_name_mode(name, expr->part, base, sizeof(base),
+ &has_amp, &mode) < 0) {
+ pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: "");
+ return -EINVAL;
+ }
+
+ if (has_amp) {
+ if (find_slab_cache(skel, base, &addr)) {
+ int fd = bpf_map__fd(skel->maps.slab_filter);
+
+ entry->part = PBF_LOCK_SLAB | mode;
+ entry->value = addr;
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_slab = 1;
+ return 0;
+ }
+ pr_err("Error: Cannot resolve slab cache '%s'\n", name);
+ return -EINVAL;
+ }
+
+ if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
+ entry->part = PBF_LOCK_MMAP | mode;
+ entry->value = 0;
+ if (skel && (mode & PBF_LOCK_MODE_READ))
+ skel->bss->has_mmap_lock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "siglock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_SIGHAND | mode;
+ entry->value = 0;
+ if (skel)
+ skel->bss->has_siglock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "rq_lock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_CLASS | mode;
+ entry->value = LOCK_CLASS_RQLOCK;
+ if (skel)
+ skel->bss->has_lock_class = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "zone_lock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_CLASS | mode;
+ entry->value = LOCK_CLASS_ZONE_LOCK;
+ if (skel)
+ skel->bss->has_lock_class = 1;
+ return 0;
+ }
+
+ if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
+ entry->part = PBF_LOCK_TYPE | mode;
+ entry->value = encoded_pair;
+ return 0;
+ }
+
+ if (parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) {
+ pr_err("Error: Lock type '%s' does not support the requested read/write mode\n",
+ name);
+ return -EINVAL;
+ }
+
+ if (parse_lock_hex_addr(base, &addr)) {
+ entry->part = PBF_LOCK_ADDR | mode;
+ entry->value = addr;
+ if (skel) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
+ }
+ return 0;
+ }
+
+ ret = find_kernel_lock_sym(base, &addr);
+ if (ret == 0) {
+ unsigned int direct_flag = 0;
+
+ if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0)
+ direct_flag = PBF_LOCK_DIRECT_RTMUTEX;
+ else if (strtailcmp(base, "mutex") == 0)
+ direct_flag = PBF_LOCK_DIRECT_MUTEX;
+ else if (strtailcmp(base, "rwsem") == 0 &&
+ strtailcmp(base, "percpu_rwsem") != 0 &&
+ strtailcmp(base, "pcpu_rwsem") != 0)
+ direct_flag = PBF_LOCK_DIRECT_RWSEM;
+
+ if ((direct_flag == PBF_LOCK_DIRECT_MUTEX ||
+ direct_flag == PBF_LOCK_DIRECT_RTMUTEX ||
+ strtailcmp(base, "percpu_rwsem") == 0 ||
+ strtailcmp(base, "pcpu_rwsem") == 0 ||
+ strtailcmp(base, "spinlock") == 0 ||
+ (strtailcmp(base, "sem") == 0 &&
+ strtailcmp(base, "rwsem") != 0)) &&
+ !(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+
+ entry->part = PBF_LOCK_ADDR | mode | direct_flag;
+ entry->value = addr;
+ if (skel && direct_flag != PBF_LOCK_DIRECT_MUTEX &&
+ direct_flag != PBF_LOCK_DIRECT_RTMUTEX) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
+ }
+ return 0;
+ }
+ if (ret == -EACCES)
+ return ret;
+
+ pr_err("Error: Cannot resolve kernel lock '%s'\n", name);
+ return -EINVAL;
+}
+
+static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct perf_bpf_filter_entry *entry)
{
- int i = 0;
+ int i = 0, ret;
struct perf_bpf_filter_expr *expr;
list_for_each_entry(expr, &evsel->bpf_filters, list) {
@@ -182,6 +1102,11 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].part = expr->part;
entry[i].term = expr->term;
entry[i].value = expr->val;
+ if (expr->term == PBF_TERM_LOCK_HELD) {
+ ret = resolve_lock_expr(skel, expr, &entry[i]);
+ if (ret < 0)
+ return ret;
+ }
i++;
if (expr->op == PBF_OP_GROUP_BEGIN) {
@@ -195,6 +1120,11 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].part = group->part;
entry[i].term = group->term;
entry[i].value = group->val;
+ if (group->term == PBF_TERM_LOCK_HELD) {
+ ret = resolve_lock_expr(skel, group, &entry[i]);
+ if (ret < 0)
+ return ret;
+ }
i++;
}
@@ -434,22 +1364,24 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
{
int i, x, y, fd, ret;
struct sample_filter_bpf *skel = NULL;
+ struct btf *btf = NULL;
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
+ bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false;
+ unsigned int prog_groups = 0, required_groups = 0;
entry = calloc(MAX_FILTERS, sizeof(*entry));
if (entry == NULL)
return -1;
- ret = get_filter_entries(evsel, entry);
- if (ret < 0) {
- pr_err("Failed to process filter entries\n");
- goto err;
- }
+ prescan_lock_filters(evsel, &has_lock_filter, &prog_groups,
+ &required_groups, &needs_lock_syms,
+ &needs_slab_iter);
- if (is_tracepoint && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
- pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n");
+ if ((has_lock_filter || is_tracepoint) && geteuid() != 0 &&
+ !perf_cap__capable(CAP_BPF)) {
+ pr_err("Error: BPF filter requires CAP_BPF capability\n");
ret = -EPERM;
goto err;
}
@@ -457,6 +1389,11 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
int zero = 0;
+ ret = get_filter_entries(evsel, NULL, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
/* The filters map is shared among other processes */
ret = create_idx_hash(evsel, entry);
if (ret < 0)
@@ -501,8 +1438,26 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
- bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
+ if (has_lock_filter) {
+ btf = btf__load_vmlinux_btf();
+ bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES);
+ bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS);
+ if (needs_slab_iter) {
+ bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS);
+ bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES);
+ }
+ if (needs_lock_syms) {
+ bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
+ init_numa_data(skel, btf);
+ }
+ ret = configure_lock_progs(skel, btf, prog_groups, required_groups,
+ needs_lock_syms, needs_slab_iter,
+ is_tracepoint);
+ if (ret < 0)
+ goto err;
+ } else {
+ configure_lock_progs(skel, NULL, 0, 0, false, false, is_tracepoint);
+ }
if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
@@ -510,6 +1465,37 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
+ if (has_lock_filter && needs_slab_iter)
+ run_slab_cache_iter(skel);
+
+ ret = get_filter_entries(evsel, skel, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
+
+ if (has_lock_filter) {
+ if (needs_lock_syms && skel->bss->has_lock_class) {
+ DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts,
+ .flags = BPF_F_TEST_RUN_ON_CPU,
+ );
+ int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms);
+
+ bpf_prog_test_run_opts(prog_fd, &opts);
+ }
+ bpf_program__set_autoattach(skel->progs.perf_sample_filter, false);
+ bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false);
+ bpf_program__set_autoattach(skel->progs.collect_lock_syms, false);
+ bpf_program__set_autoattach(skel->progs.slab_cache_iter, false);
+
+ ret = sample_filter_bpf__attach(skel);
+ if (ret < 0) {
+ pr_err("Failed to attach lock sample-filter BPF programs\n");
+ goto err;
+ }
+ skel->bss->lock_enabled = 1;
+ }
+
i = 0;
fd = bpf_map__fd(skel->maps.filters);
@@ -549,11 +1535,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
}
}
}
+ btf__free(btf);
free(entry);
evsel->bpf_skel = skel;
return 0;
err:
+ btf__free(btf);
free(entry);
if (!list_empty(&pinned_filters)) {
struct pinned_filter_idx *pfi, *tmp;
@@ -568,6 +1556,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return ret;
}
+void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr)
+{
+ struct perf_bpf_filter_expr *group, *tmp;
+
+ if (!expr)
+ return;
+
+ list_for_each_entry_safe(group, tmp, &expr->groups, list) {
+ list_del(&group->list);
+ perf_bpf_filter_expr__delete(group);
+ }
+ zfree(&expr->lock_name);
+ free(expr);
+}
+
int perf_bpf_filter__destroy(struct evsel *evsel)
{
struct perf_bpf_filter_expr *expr, *tmp;
@@ -575,7 +1578,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel)
list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) {
list_del(&expr->list);
- free(expr);
+ perf_bpf_filter_expr__delete(expr);
}
sample_filter_bpf__destroy(evsel->bpf_skel);
@@ -627,7 +1630,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
{
struct perf_bpf_filter_expr *expr;
- expr = malloc(sizeof(*expr));
+ expr = zalloc(sizeof(*expr));
if (expr != NULL) {
expr->term = term;
expr->part = part;
@@ -638,6 +1641,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
return expr;
}
+struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
+ int mode,
+ enum perf_bpf_filter_op op,
+ unsigned long val)
+{
+ struct perf_bpf_filter_expr *expr;
+
+ if (!lock_name || !*lock_name)
+ return NULL;
+
+ expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val);
+ if (!expr)
+ return NULL;
+
+ expr->lock_name = strdup(lock_name);
+ if (!expr->lock_name) {
+ free(expr);
+ return NULL;
+ }
+ return expr;
+}
+
static bool check_bpf_filter_capable(void)
{
int fd;
@@ -658,14 +1683,11 @@ static bool check_bpf_filter_capable(void)
return false;
}
-int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
+int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
{
YY_BUFFER_STATE buffer;
int ret;
- if (!check_bpf_filter_capable())
- return -EPERM;
-
buffer = perf_bpf_filter__scan_string(str);
ret = perf_bpf_filter_parse(expr_head);
@@ -677,6 +1699,14 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
return ret;
}
+int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
+{
+ if (!check_bpf_filter_capable())
+ return -EPERM;
+
+ return __perf_bpf_filter__parse(expr_head, str);
+}
+
static int pin_filter_object(struct sample_filter_bpf *skel, const char *path)
{
char *prog_path = NULL;
@@ -729,8 +1759,9 @@ int perf_bpf_filter__pin(void)
goto err;
}
+ configure_lock_progs(skel, NULL, 0, 0, false, false, false);
+
/* pinned program will use pid-hash */
- bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH);
bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH);
@@ -825,7 +1856,7 @@ int perf_bpf_filter__unpin(void)
goto err;
}
- bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
+ configure_lock_progs(skel, NULL, 0, 0, false, false, false);
if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
pr_err("Failed to load perf sample-filter BPF skeleton\n");
diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h
index 818c554b91b2..fda95ff226b3 100644
--- a/tools/perf/util/bpf-filter.h
+++ b/tools/perf/util/bpf-filter.h
@@ -14,6 +14,13 @@ struct perf_bpf_filter_expr {
int part;
enum perf_bpf_filter_term term;
unsigned long val;
+ /*
+ * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions
+ * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R").
+ * Resolved into `entry.part` and `entry.target` during
+ * perf_bpf_filter__prepare().
+ */
+ char *lock_name;
};
struct evsel;
@@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
int part,
enum perf_bpf_filter_op op,
unsigned long val);
+struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
+ int mode,
+ enum perf_bpf_filter_op op,
+ unsigned long val);
+void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr);
+int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
int perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target);
int perf_bpf_filter__destroy(struct evsel *evsel);
diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l
index 1be9df6550fc..b4022110f1e5 100644
--- a/tools/perf/util/bpf-filter.l
+++ b/tools/perf/util/bpf-filter.l
@@ -1,9 +1,12 @@
%option prefix="perf_bpf_filter_"
%option noyywrap
+%x LOCK_ARG
+
%{
#include <stdio.h>
#include <stdlib.h>
+#include <string.h>
#include <linux/perf_event.h>
#include "bpf-filter.h"
@@ -76,7 +79,7 @@ static int path_or_error(void)
num_dec [0-9]+
num_hex 0[Xx][0-9a-fA-F]+
space [ \t]+
-path [^ \t\n,]+
+path [^ \t\n,()!=>&|<]+
ident [_a-zA-Z][_a-zA-Z0-9]+
%%
@@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); }
uid { return sample(PBF_TERM_UID); }
gid { return sample(PBF_TERM_GID); }
cgroup { return sample_path(PBF_TERM_CGROUP); }
-
+lock_held[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY;
+ return BFT_LOCK_HELD;
+ }
+(lock_held_read|lock_held_shared)[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ;
+ return BFT_LOCK_HELD;
+ }
+(lock_held_write|lock_held_exclusive)[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE;
+ return BFT_LOCK_HELD;
+ }
+
+<LOCK_ARG>{space} { }
+<LOCK_ARG>[^ \t\n()]+ {
+ perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text);
+ return BFT_LOCK_NAME;
+ }
+<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; }
+<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; }
+
+"!" { return '!'; }
"==" { return operator(PBF_OP_EQ); }
"!=" { return operator(PBF_OP_NEQ); }
">" { return operator(PBF_OP_GT); }
diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y
index 5a79a8e7a45b..5ae05d4608c6 100644
--- a/tools/perf/util/bpf-filter.y
+++ b/tools/perf/util/bpf-filter.y
@@ -8,9 +8,11 @@
#endif
#include <stdio.h>
+#include <stdlib.h>
#include <string.h>
#include <linux/compiler.h>
#include <linux/list.h>
+#include <linux/zalloc.h>
#include "bpf-filter.h"
#include "cgroup.h"
@@ -31,6 +33,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
{
unsigned long num;
char *path;
+ char *lock_name;
+ struct {
+ char *name;
+ int mode;
+ } lock;
struct {
enum perf_bpf_filter_term term;
int part;
@@ -40,12 +47,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
}
%token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH
+%token BFT_LOCK_HELD BFT_LOCK_NAME
%type <expr> filter_term filter_expr
-%destructor { free ($$); } <expr>
+%destructor { perf_bpf_filter_expr__delete($$); } <expr>
+%destructor { free($$); } <lock_name>
+%destructor { free($$.name); } <lock>
%type <sample> BFT_SAMPLE BFT_SAMPLE_PATH
%type <op> BFT_OP
-%type <num> BFT_NUM
+%type <num> BFT_NUM BFT_LOCK_HELD
%type <path> BFT_PATH
+%type <lock_name> BFT_LOCK_NAME
+%type <lock> lock_term
%%
@@ -105,5 +117,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH
$$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id);
cgroup__put(cgrp);
}
+|
+lock_term
+{
+ $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1);
+ zfree(&$1.name);
+ if (!$$)
+ YYERROR;
+}
+|
+'!' lock_term
+{
+ $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0);
+ zfree(&$2.name);
+ if (!$$)
+ YYERROR;
+}
+|
+lock_term BFT_OP BFT_NUM
+{
+ $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3);
+ zfree(&$1.name);
+ if (!$$)
+ YYERROR;
+}
+
+lock_term:
+BFT_LOCK_HELD BFT_LOCK_NAME ')'
+{
+ $$.name = $2;
+ $$.mode = $1;
+}
%%
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index 683fec85e71e..a65ee780d64d 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -53,8 +53,74 @@ enum perf_bpf_filter_term {
/* Terms computed from BPF helpers. */
PBF_TERM_UID,
PBF_TERM_GID,
+ /*
+ * Kernel lock held predicate (`lock_held(...)`). Evaluates whether the
+ * specified kernel lock is currently held by the running task/CPU.
+ * The lock specification kind is stored in `entry.part`
+ * (`enum perf_bpf_lock_target_kind`), `entry.op` is normalized to
+ * `PBF_OP_EQ` (match when held) or `PBF_OP_NEQ` (match when not held),
+ * and the resolved target address, class ID, or type flag encoding is
+ * stored in `entry.value`.
+ */
+ PBF_TERM_LOCK_HELD,
};
+/*
+ * Lock target categories for PBF_TERM_LOCK_HELD filters.
+ * Stored in `perf_bpf_filter_entry.part`.
+ */
+enum perf_bpf_lock_target_kind {
+ PBF_LOCK_NONE = 0,
+ /*
+ * Specific global or static lock address (resolved from kallsyms or
+ * given as a hex address). `entry.value` holds the kernel address.
+ */
+ PBF_LOCK_ADDR,
+ /*
+ * Dynamic lock allocated inside a slab cache (`&slab_name`).
+ * `entry.value` holds the kernel `struct kmem_cache *` address.
+ */
+ PBF_LOCK_SLAB,
+ /*
+ * Per-process `current->mm->mmap_lock` (or `mmap_sem`).
+ */
+ PBF_LOCK_MMAP,
+ /*
+ * Per-process `current->sighand->siglock`.
+ */
+ PBF_LOCK_SIGHAND,
+ /*
+ * Semi-global lock class (`rq_lock` or `zone_lock`).
+ * `entry.value` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or
+ * LOCK_CLASS_ZONE_LOCK).
+ */
+ PBF_LOCK_CLASS,
+ /*
+ * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.).
+ * `entry.value` packs up to two `LCB_F_*` flag combinations in the
+ * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
+ */
+ PBF_LOCK_TYPE,
+};
+
+/*
+ * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD:
+ * Bits 0..3: `enum perf_bpf_lock_target_kind`
+ * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding)
+ * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding)
+ * Bit 6: PBF_LOCK_DIRECT_MUTEX (inspect struct mutex owner)
+ * Bit 7: PBF_LOCK_DIRECT_RWSEM (inspect struct rw_semaphore writer owner)
+ * Bit 8: PBF_LOCK_DIRECT_RTMUTEX (inspect struct rt_mutex_base / rt_mutex owner)
+ */
+#define PBF_LOCK_KIND_MASK 0x0fU
+#define PBF_LOCK_MODE_READ (1U << 4)
+#define PBF_LOCK_MODE_WRITE (1U << 5)
+#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE)
+#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY
+#define PBF_LOCK_DIRECT_MUTEX (1U << 6)
+#define PBF_LOCK_DIRECT_RWSEM (1U << 7)
+#define PBF_LOCK_DIRECT_RTMUTEX (1U << 8)
+
/* BPF map entry for filtering */
struct perf_bpf_filter_entry {
enum perf_bpf_filter_op op;
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index a190374ed87a..d1192a93179e 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -5,8 +5,47 @@
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
+#include "lock_data.h"
#include "sample-filter.h"
+/*
+ * Key for the `held_locks` BPF hash map tracking currently held kernel locks.
+ *
+ * - `addr_or_id`:
+ * - PBF_LOCK_ADDR: kernel virtual address of the lock instance
+ * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK)
+ * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address
+ * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags
+ * - `owner_id`:
+ * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`),
+ * since the holder may sleep and migrate across CPUs while holding the lock.
+ * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks
+ * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption
+ * is disabled on the holding CPU and spinlocks such as `rq_lock` may be
+ * acquired by one task and released by the next task across `context_switch()`.
+ * - `kind`: `enum perf_bpf_lock_target_kind`.
+ */
+struct held_lock_key {
+ __u64 addr_or_id;
+ __u32 owner_id;
+ __u32 kind;
+};
+
+/*
+ * Value for `held_locks` entries.
+ * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`)
+ * and exclusive/write acquisitions (including nested class/slab/type
+ * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of
+ * the lock so release tracepoints that do not carry flags (such as
+ * `lock:contended_release`) can decrement matching class/slab/type entries.
+ */
+struct held_lock_val {
+ __u32 read_count;
+ __u32 write_count;
+ __u32 flags;
+ __u32 pad;
+};
+
/* BPF map that will be filled by user space */
struct filters {
__uint(type, BPF_MAP_TYPE_HASH);
@@ -43,10 +82,225 @@ struct lost_count {
__uint(max_entries, 1);
} dropped SEC(".maps");
+/*
+ * Currently held kernel locks tracked by fentry/fexit hooks.
+ * Uses BPF_MAP_TYPE_LRU_HASH so that if a counting semaphore (down/up) or
+ * non-owner rwsem is acquired in one task and released in another task or
+ * interrupt handler (cross-context signaling), any orphaned entry is
+ * automatically evicted rather than exhausting the map.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_LRU_HASH);
+ __type(key, struct held_lock_key);
+ __type(value, struct held_lock_val);
+ __uint(max_entries, 1);
+} held_locks SEC(".maps");
+
+/* Target lock addresses for PBF_LOCK_ADDR filters. */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, __u64);
+ __type(value, __u8);
+ __uint(max_entries, 1);
+} lock_addr_filter SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u32));
+ __uint(max_entries, 1);
+} lock_syms SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u8));
+ __uint(max_entries, 1);
+} slab_filter SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(struct slab_cache_data));
+ __uint(max_entries, 1);
+} slab_caches SEC(".maps");
+
volatile const int use_idx_hash;
+const volatile __u64 contig_page_data_addr;
+const volatile __u64 node_data_addr;
+const volatile int nr_nodes;
+const volatile int sizeof_zone;
+
+__u32 slab_cache_id;
+
+/* Control flags set by userspace when lock_held(...) filters are active. */
+int lock_enabled;
+int has_lock_addr;
+int has_lock_slab;
+int has_lock_class;
+int has_lock_type;
+int has_mmap_lock;
+int has_siglock;
+int track_rwsem_read;
+__u8 lock_type_filter[LCB_F_TYPE_MAX];
+
void *bpf_cast_to_kern_ctx(void *) __ksym;
+/*
+ * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`.
+ *
+ * For locks that maintain a task_struct pointer in their `owner` field
+ * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`),
+ * `perf_sample_filter` can inspect the lock structure directly in kernel memory
+ * and compare the owner against `current` (`bpf_get_current_task()`).
+ *
+ * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear
+ * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and
+ * spinlocks are checked via the `held_locks` BPF map instead.
+ */
+static __always_inline bool is_rwsem_writer_owner(__u64 lock_addr, __u64 curr)
+{
+ struct rw_semaphore___new *rwsem = (void *)lock_addr;
+
+ if (bpf_core_field_exists(rwsem->count.counter) &&
+ !(BPF_CORE_READ(rwsem, count.counter) & 1UL))
+ return false;
+ if (bpf_core_field_exists(rwsem->owner.counter) &&
+ (BPF_CORE_READ(rwsem, owner.counter) & 1UL))
+ return false;
+
+ return (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr;
+}
+
+static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __u32 part)
+{
+ if (!curr)
+ return false;
+
+ if ((part & PBF_LOCK_DIRECT_MUTEX) &&
+ (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr)
+ return true;
+ if ((part & PBF_LOCK_DIRECT_RWSEM) &&
+ is_rwsem_writer_owner(lock_addr, curr))
+ return true;
+ if ((part & PBF_LOCK_DIRECT_RTMUTEX) &&
+ (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr)
+ return true;
+ return false;
+}
+
+static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode)
+{
+ if (!val)
+ return false;
+ if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0)
+ return true;
+ if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0)
+ return true;
+ return false;
+}
+
+static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind,
+ __u32 mode, __u32 pid, __u32 cpu_id)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = pid,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+
+ if (pid) {
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+ }
+
+ key.owner_id = cpu_id;
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+
+ return 0;
+}
+
+static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode,
+ __u32 pid, __u32 cpu_id)
+{
+ __u32 flag1 = target & 0xff;
+ __u32 flag2 = (target >> 8) & 0xff;
+
+ if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ return 0;
+}
+
+static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry)
+{
+ __u64 curr = bpf_get_current_task();
+ __u32 pid = (__u32)bpf_get_current_pid_tgid();
+ __u32 cpu_id = 0x80000000U | bpf_get_smp_processor_id();
+ __u32 kind = entry->part & PBF_LOCK_KIND_MASK;
+ __u32 mode = entry->part & PBF_LOCK_MODE_MASK;
+ __u64 lock_addr;
+
+ if (!mode)
+ mode = PBF_LOCK_MODE_ANY;
+
+ switch (kind) {
+ case PBF_LOCK_ADDR:
+ lock_addr = entry->value;
+ if ((mode & PBF_LOCK_MODE_WRITE) &&
+ is_direct_lock_owner(lock_addr, curr, entry->part))
+ return true;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0;
+
+ case PBF_LOCK_MMAP:
+ lock_addr = get_current_mmap_lock(curr);
+ if (!lock_addr)
+ return false;
+ if ((mode & PBF_LOCK_MODE_WRITE) &&
+ is_rwsem_writer_owner(lock_addr, curr))
+ return true;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0;
+
+ case PBF_LOCK_SIGHAND:
+ lock_addr = get_current_siglock(curr);
+ if (!lock_addr)
+ return false;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0;
+
+ case PBF_LOCK_CLASS:
+ return is_lock_in_map(entry->value, PBF_LOCK_CLASS, mode, pid, cpu_id) != 0;
+
+ case PBF_LOCK_SLAB:
+ return is_lock_in_map(entry->value, PBF_LOCK_SLAB, mode, pid, cpu_id) != 0;
+
+ case PBF_LOCK_TYPE:
+ return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0;
+
+ default:
+ return false;
+ }
+}
+
+static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry)
+{
+ /*
+ * Pinned filters (use_idx_hash != 0) share a world-writable filters
+ * map and do not support lock_held(). Reject evaluation when pinned
+ * or when lock_enabled is not set so unprivileged users cannot probe
+ * arbitrary kernel addresses via entry->value.
+ */
+ if (use_idx_hash || !lock_enabled)
+ return entry->op == PBF_OP_EQ ? ~entry->value : entry->value;
+
+ return is_lock_held(entry) ? entry->value : ~entry->value;
+}
+
/* new kernel perf_sample_data definition */
struct perf_sample_data___new {
__u64 sample_flags;
@@ -175,6 +429,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
return bpf_get_current_uid_gid() & 0xFFFFFFFF;
case PBF_TERM_GID:
return bpf_get_current_uid_gid() >> 32;
+ case PBF_TERM_LOCK_HELD:
+ return check_lock_held(entry);
case PBF_TERM_NONE:
case __PBF_UNUSED_TERM4:
case __PBF_UNUSED_TERM5:
@@ -209,6 +465,8 @@ static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry
return bpf_get_current_uid_gid() & 0xFFFFFFFF;
case PBF_TERM_GID:
return bpf_get_current_uid_gid() >> 32;
+ case PBF_TERM_LOCK_HELD:
+ return check_lock_held(entry);
default:
return 0;
}
@@ -390,4 +648,760 @@ int perf_tp_sample_filter(void *ctx)
return 0;
}
+static __always_inline __u32 get_lock_owner_id(__u32 flags)
+{
+ __u32 owner_id;
+
+ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN)
+ return 0x80000000U | bpf_get_smp_processor_id();
+
+ owner_id = (__u32)bpf_get_current_pid_tgid();
+ if (!owner_id)
+ owner_id = 0x80000000U | bpf_get_smp_processor_id();
+ return owner_id;
+}
+
+static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id,
+ __u32 kind, __u32 flags, bool acquired)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = owner_id,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (acquired) {
+ if (val) {
+ if (is_read)
+ __sync_fetch_and_add(&val->read_count, 1);
+ else
+ __sync_fetch_and_add(&val->write_count, 1);
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags,
+ };
+
+ bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY);
+ }
+ } else if (val) {
+ if (is_read) {
+ if (val->read_count <= 1 && val->write_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->read_count > 0)
+ __sync_fetch_and_add(&val->read_count, -1);
+ } else {
+ if (val->write_count <= 1 && val->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->write_count > 0)
+ __sync_fetch_and_add(&val->write_count, -1);
+ }
+ }
+}
+
+#define LCB_F_READ_IRQSAVE (1U << 30)
+#define LCB_F_READ_IRQ_BH (1U << 31)
+
+static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id)
+{
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ bool matched = false;
+ bool is_read = (flags & LCB_F_READ) != 0;
+ __u32 class_id = 0;
+ __u64 kmem_cache_addr = 0;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool match_type = false;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
+ matched = true;
+
+ if (has_mmap_lock || has_siglock) {
+ __u32 ltype = check_lock_type(lock_addr, type_flag);
+
+ if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
+ (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
+ matched = true;
+ }
+
+ if (has_lock_class && type_flag == LCB_F_SPIN) {
+ __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
+
+ if (cid) {
+ class_id = *cid;
+ matched = true;
+ }
+ }
+
+ if (has_lock_slab && bpf_get_kmem_cache) {
+ kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
+ if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr))
+ matched = true;
+ else
+ kmem_cache_addr = 0;
+ }
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) {
+ match_type = true;
+ matched = true;
+ }
+
+ if (!matched)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (existing) {
+ /*
+ * On CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC
+ * kernels, _raw_read_lock_irq() and _raw_read_lock_bh() call
+ * the out-of-line _raw_read_lock_irqsave(). Clear the marker
+ * and skip the duplicate outer fexit increment.
+ */
+ if ((flags & LCB_F_READ_IRQ_BH) &&
+ (existing->flags & LCB_F_READ_IRQSAVE)) {
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ return 0;
+ }
+ /* Non-reader locks are not recursive; avoid double-counting */
+ if (!is_read && existing->write_count > 0)
+ return 0;
+ if (is_read) {
+ __sync_fetch_and_add(&existing->read_count, 1);
+ if (flags & LCB_F_READ_IRQSAVE)
+ existing->flags |= LCB_F_READ_IRQSAVE;
+ else
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ } else {
+ __sync_fetch_and_add(&existing->write_count, 1);
+ }
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags & (LCB_F_TYPE_MASK | LCB_F_READ_IRQSAVE),
+ };
+
+ if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0)
+ return 0;
+ }
+
+ if (class_id)
+ update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, type_flag, true);
+ if (kmem_cache_addr) {
+ update_held_lock_count((__u64)kmem_cache_addr, owner_id,
+ PBF_LOCK_SLAB, type_flag, true);
+ }
+ if (match_type)
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true);
+
+ return 0;
+}
+
+static __always_inline int record_lock_acquired(void *ctx, __u32 flags)
+{
+ return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags));
+}
+
+static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags)
+{
+ __u32 owner_id = get_lock_owner_id(flags);
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (!existing)
+ return 0;
+
+ if (is_read) {
+ if (existing->read_count == 0)
+ return 0;
+ if (existing->read_count == 1 && existing->write_count == 0) {
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ } else {
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ __sync_fetch_and_add(&existing->read_count, -1);
+ }
+ } else {
+ if (existing->write_count == 0)
+ return 0;
+ if (existing->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ else
+ existing->write_count = 0;
+ }
+
+ if (has_lock_class && flags == LCB_F_SPIN) {
+ __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
+
+ if (cid)
+ update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false);
+ }
+
+ if (has_lock_slab && bpf_get_kmem_cache) {
+ __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
+
+ if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) {
+ update_held_lock_count(kmem_cache_addr, owner_id,
+ PBF_LOCK_SLAB, flags, false);
+ }
+ }
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag])
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false);
+
+ return 0;
+}
+
+static __always_inline int record_lock_released(void *ctx, __u32 flags)
+{
+ return record_lock_released_addr(get_lock_addr(ctx), flags);
+}
+
+/* Mutex acquire/release hooks */
+SEC("fexit/mutex_lock")
+int BPF_PROG(fexit_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_nested")
+int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/_mutex_lock_nest_lock")
+int BPF_PROG(fexit__mutex_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io")
+int BPF_PROG(fexit_mutex_lock_io, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io_nested")
+int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_interruptible")
+int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_interruptible_nested")
+int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_killable")
+int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/_mutex_lock_killable")
+int BPF_PROG(fexit__mutex_lock_killable, void *lock, unsigned int subclass,
+ void *nest_lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_trylock")
+int BPF_PROG(fexit_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/_mutex_trylock_nest_lock")
+int BPF_PROG(fexit__mutex_trylock_nest_lock, void *lock, void *nest_lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fentry/mutex_unlock")
+int BPF_PROG(fentry_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_MUTEX);
+}
+
+/* RW semaphore acquire/release hooks */
+SEC("fexit/down_read")
+int BPF_PROG(fexit_down_read, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_nested")
+int BPF_PROG(fexit_down_read_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_interruptible")
+int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable")
+int BPF_PROG(fexit_down_read_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable_nested")
+int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_trylock")
+int BPF_PROG(fexit_down_read_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_write")
+int BPF_PROG(fexit_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_nested")
+int BPF_PROG(fexit_down_write_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/_down_write_nest_lock")
+int BPF_PROG(fexit__down_write_nest_lock, void *sem, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_killable")
+int BPF_PROG(fexit_down_write_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_killable_nested")
+int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_trylock")
+int BPF_PROG(fexit_down_write_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/downgrade_write")
+int BPF_PROG(fexit_downgrade_write, void *sem)
+{
+ if (track_rwsem_read)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+SEC("fentry/up_read")
+int BPF_PROG(fentry_up_read, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_READ);
+}
+
+SEC("fentry/up_write")
+int BPF_PROG(fentry_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+/* Spinlock acquire/release hooks */
+SEC("fexit/_raw_spin_lock")
+int BPF_PROG(fexit_raw_spin_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_nested")
+int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_nest_lock")
+int BPF_PROG(fexit_raw_spin_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq")
+int BPF_PROG(fexit_raw_spin_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq_disable")
+int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave")
+int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave_nested")
+int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_bh")
+int BPF_PROG(fexit_raw_spin_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_trylock")
+int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fexit/_raw_spin_trylock_bh")
+int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fentry/_raw_spin_unlock")
+int BPF_PROG(fentry_raw_spin_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq")
+int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq_enable")
+int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irqrestore")
+int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_bh")
+int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+/* RWlock acquire/release hooks */
+SEC("fexit/_raw_read_lock")
+int BPF_PROG(fexit_raw_read_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_lock_irq")
+int BPF_PROG(fexit_raw_read_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH);
+}
+
+SEC("fexit/_raw_read_lock_irqsave")
+int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQSAVE);
+}
+
+SEC("fexit/_raw_read_lock_bh")
+int BPF_PROG(fexit_raw_read_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH);
+}
+
+SEC("fexit/_raw_read_trylock")
+int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+ return 0;
+}
+
+SEC("fentry/_raw_read_unlock")
+int BPF_PROG(fentry_raw_read_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irq")
+int BPF_PROG(fentry_raw_read_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irqrestore")
+int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_bh")
+int BPF_PROG(fentry_raw_read_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_write_lock")
+int BPF_PROG(fexit_raw_write_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_nested")
+int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irq")
+int BPF_PROG(fexit_raw_write_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irqsave")
+int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_bh")
+int BPF_PROG(fexit_raw_write_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_trylock")
+int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fentry/_raw_write_unlock")
+int BPF_PROG(fentry_raw_write_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irq")
+int BPF_PROG(fentry_raw_write_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irqrestore")
+int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_bh")
+int BPF_PROG(fentry_raw_write_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+/* RT-Mutex acquire/release hooks */
+SEC("fexit/rt_mutex_lock")
+int BPF_PROG(fexit_rt_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_nested")
+int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/_rt_mutex_lock_nest_lock")
+int BPF_PROG(fexit__rt_mutex_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_interruptible")
+int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_lock_killable")
+int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_trylock")
+int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fentry/rt_mutex_unlock")
+int BPF_PROG(fentry_rt_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_RT);
+}
+
+/* Semaphore acquire/release hooks */
+SEC("fexit/down")
+int BPF_PROG(fexit_down, void *sem)
+{
+ return record_lock_acquired(ctx, 0);
+}
+
+SEC("fexit/down_interruptible")
+int BPF_PROG(fexit_down_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_killable")
+int BPF_PROG(fexit_down_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_timeout")
+int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_trylock")
+int BPF_PROG(fexit_down_trylock, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fentry/up")
+int BPF_PROG(fentry_up, void *sem)
+{
+ return record_lock_released(ctx, 0);
+}
+
+/* Percpu-rwsem write acquire/release hooks */
+SEC("fexit/percpu_down_write")
+int BPF_PROG(fexit_percpu_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
+SEC("fentry/percpu_up_write")
+int BPF_PROG(fentry_percpu_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
+SEC("raw_tp/bpf_test_finish")
+int BPF_PROG(collect_lock_syms)
+{
+ return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
+}
+
+SEC("iter/kmem_cache")
+int slab_cache_iter(void *ctx)
+{
+ return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
+}
+
char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index 2f3317ce17f0..1fcb424b75c0 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist,
/* Will a tracepoint filter work for str or should a BPF filter be used? */
static bool is_possible_tp_filter(const char *str)
{
- return strstr(str, "uid") == NULL;
+ return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL &&
+ strstr(str, "lock_held") == NULL;
}
static int set_filter(struct evsel *evsel, const void *arg)
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v5 6/8] perf test: Add tests for lock_held() BPF sample filter
2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (4 preceding siblings ...)
2026-10-20 20:58 ` [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter Ian Rogers
@ 2026-10-20 20:58 ` Ian Rogers
2026-10-20 23:20 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 7/8] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
` (2 subsequent siblings)
8 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Add unit tests in parse-events.c for parsing lock_held(),
lock_held_read(), and lock_held_exclusive() BPF filter expressions
(standalone predicates, negated predicates, and compound group
expressions) and extend record_bpf_filter.sh with lock_held(),
lock_held_read()/lock_held_write(), and ':R'/':W' recording tests.
Example output:
# perf test -v 'Parsing of BPF sample filters'
5: Parse event definition strings :
5.7: Parsing of BPF sample filters : Ok
# perf test -v 'perf record sample filtering (by BPF) tests'
112: perf record sample filtering (by BPF) tests :
Checking BPF-filter privilege
Basic bpf-filter test
Basic bpf-filter test [Success]
Failing bpf-filter test
Failing bpf-filter test [Success]
Group bpf-filter test
Group bpf-filter test [Success]
Multiple bpf-filter test
Multiple bpf-filter test [Success]
Cgroup bpf-filter test
Cgroup bpf-filter test [Success]
Lock-held bpf-filter test
Lock-held bpf-filter test [Success]
112: perf record sample filtering (by BPF) tests : Ok
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/tests/parse-events.c | 108 +++++++++++++++++
tools/perf/tests/shell/record_bpf_filter.sh | 122 ++++++++++++++++++++
2 files changed, 230 insertions(+)
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 5f6f95c8a5b4..4cf9192762f2 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -8,9 +8,11 @@
#include "debug.h"
#include "pmu.h"
#include "pmus.h"
+#include "bpf-filter.h"
#include "strbuf.h"
#include <dirent.h>
#include <errno.h>
+#include <string.h>
#include "fncache.h"
#include <sys/types.h>
#include <sys/stat.h>
@@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused,
return ret;
}
+#ifdef HAVE_BPF_SKEL
+static void clear_bpf_filter_exprs(struct list_head *expr_head)
+{
+ struct perf_bpf_filter_expr *expr, *tmp;
+
+ list_for_each_entry_safe(expr, tmp, expr_head, list) {
+ list_del(&expr->list);
+ perf_bpf_filter_expr__delete(expr);
+ }
+}
+
+static int do_test_bpf_filters(struct list_head *expr_head)
+{
+ struct perf_bpf_filter_expr *expr, *group;
+ const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100";
+
+ /* Simple lock_held(&cgroup_mutex) */
+ TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)",
+ __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is &cgroup_mutex",
+ expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /* Negated !lock_held(cgroup_mutex) */
+ TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)",
+ __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 0", expr->val == 0);
+ TEST_ASSERT_VAL("lock_name is cgroup_mutex",
+ expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /* Read/shared and write/exclusive predicates */
+ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)",
+ __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is &mmap_lock",
+ expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock"));
+ clear_bpf_filter_exprs(expr_head);
+
+ TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)",
+ __perf_bpf_filter__parse(expr_head,
+ "!lock_held_exclusive(mmap_lock)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 0", expr->val == 0);
+ TEST_ASSERT_VAL("lock_name is mmap_lock",
+ expr->lock_name && !strcmp(expr->lock_name, "mmap_lock"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /*
+ * Comparison and compound group:
+ * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100
+ */
+ TEST_ASSERT_VAL("parse compound lock_held filter",
+ __perf_bpf_filter__parse(expr_head, compound) == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN);
+ TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups));
+ group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("first group member lock_name",
+ group->lock_name && !strcmp(group->lock_name, "rwsem:R"));
+ TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1);
+ group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("second group member lock_name",
+ group->lock_name && !strcmp(group->lock_name, "&dentry"));
+ TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0);
+ expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100);
+
+ return TEST_OK;
+}
+#endif
+
+static int test__bpf_filters(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
+{
+#ifdef HAVE_BPF_SKEL
+ struct list_head expr_head = LIST_HEAD_INIT(expr_head);
+ int ret = do_test_bpf_filters(&expr_head);
+
+ clear_bpf_filter_exprs(&expr_head);
+ return ret;
+#else
+ return TEST_SKIP;
+#endif
+}
+
static struct test_case tests__parse_events[] = {
TEST_CASE_REASON("Test event parsing",
events2,
@@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = {
"no aliases in sysfs"),
TEST_CASE("Parsing of aliased events", pmu_events_alias2),
TEST_CASE("Parsing of terms (event modifiers)", terms2),
+ TEST_CASE("Parsing of BPF sample filters", bpf_filters),
{ .name = NULL, }
};
diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh
index 383574cb3bd3..ebaf8ee41622 100755
--- a/tools/perf/tests/shell/record_bpf_filter.sh
+++ b/tools/perf/tests/shell/record_bpf_filter.sh
@@ -176,6 +176,124 @@ test_bpf_filter_cgroup() {
echo "Cgroup bpf-filter test [Success]"
}
+test_bpf_filter_lock_held() {
+ echo "Lock-held bpf-filter test"
+ lock_sym="cgroup_mutex"
+
+ if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms
+ then
+ lock_sym="slab_mutex"
+ fi
+
+ if ! perf record -e task-clock --filter "lock_held(${lock_sym})" \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if [ "$(id -u)" != 0 ]
+ then
+ echo "Lock-held bpf-filter test [Skipped permission]"
+ return
+ fi
+ if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms
+ then
+ echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]"
+ return
+ fi
+ echo "Lock-held bpf-filter test [Failed ${lock_sym}]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(&dentry)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed &dentry]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \
+ -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed mmap_lock]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \
+ -o /dev/null true 2>&1 | grep -q "Kernel BTF lacks required"
+ then
+ echo "Lock-held bpf-filter test [Failed rq_lock/spinlock]"
+ err=1
+ return
+ fi
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(rwsem)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock type]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock \
+ --filter 'lock_held_read(mmap_lock) || lock_held_write(mmap_lock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read/write(mmap_lock)]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mmap_lock:R) || lock_held(mmap_lock:W)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed mmap_lock:R/:W]"
+ err=1
+ return
+ fi
+
+ if perf list tracepoint 2> /dev/null | grep -q kmem:mm_page_alloc
+ then
+ if ! perf record -e kmem:mm_page_alloc --filter 'lock_held(mutex) || lock_held(rwsem)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed tracepoint kmem:mm_page_alloc]"
+ err=1
+ return
+ fi
+ fi
+
+ if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \
+ -o /dev/null true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]"
+ err=1
+ return
+ fi
+
+ if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \
+ -o /dev/null true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]"
+ err=1
+ return
+ fi
+
+ echo "Lock-held bpf-filter test [Success]"
+}
+
test_bpf_filter_priv
if [ $err = 0 ]; then
@@ -198,5 +316,9 @@ if [ $err = 0 ]; then
test_bpf_filter_cgroup
fi
+if [ $err = 0 ]; then
+ test_bpf_filter_lock_held
+fi
+
cleanup
exit $err
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v5 7/8] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (5 preceding siblings ...)
2026-10-20 20:58 ` [PATCH v5 6/8] perf test: Add tests for lock_held() BPF " Ian Rogers
@ 2026-10-20 20:58 ` Ian Rogers
2026-10-20 23:29 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 8/8] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
8 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks
Trace RCU, and SRCU locks:
- 'rcu' (supports both read/shared and write/exclusive modes):
- Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)',
'lock_held(rcu_read_lock)'): directly inspects
current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on
CONFIG_PREEMPT_RCU kernels), accounting for the baseline
rcu_read_lock() nesting acquired by the perf overflow handler itself
(bpf_overflow_handler() and do_perf_sw_event()) with zero extra
fentry/fexit tracing hooks.
- Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'):
attaches fentry/fexit hooks to synchronize_rcu(),
synchronize_rcu_expedited(), synchronize_rcu_tasks(), and
rcu_barrier() to track grace-period and barrier waits.
- 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both
read/shared and write/exclusive modes):
- Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches
fexit/__srcu_read_lock and fentry/__srcu_read_unlock.
- Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'):
attaches fentry/fexit on synchronize_srcu(),
synchronize_srcu_expedited(), and srcu_barrier().
- 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports
read/shared mode only): directly inspects
current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels
with zero extra tracing hooks.
Example usage:
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ]
# perf report --stdio --no-children
# Overhead Command Shared Object Symbol
# ........ ....... ................. ........................
#
14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu
|
---__d_lookup_rcu
lookup_fast
walk_component
link_path_walk
path_lookupat
filename_lookup
vfs_statx
__do_sys_newstat
do_syscall_64
entry_SYSCALL_64_after_hwframe
# perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1
# perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 44 ++--
tools/perf/tests/parse-events.c | 13 +
tools/perf/tests/shell/record_bpf_filter.sh | 20 ++
tools/perf/util/bpf-filter.c | 243 ++++++++++++++++++-
tools/perf/util/bpf_skel/lock_data.h | 47 +++-
tools/perf/util/bpf_skel/sample-filter.h | 14 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 136 ++++++++++-
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 +
tools/perf/util/lock-contention.c | 4 +
tools/perf/util/lock-contention.h | 4 +-
10 files changed, 493 insertions(+), 34 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 8496b20987a0..3eaebdc9efc2 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -219,11 +219,17 @@ OPTIONS
'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache
name prefixed with '&' (e.g. 'lock_held(&dentry)',
'lock_held_read(&dentry)'), a well-known lock name ('mmap_lock',
- 'siglock', 'rq_lock', 'zone_lock'), or a lock type ('mutex',
- 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R',
- 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore').
- Both read/shared and write/exclusive tracking are supported for
- 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', 'spinlock'
+ 'siglock', 'rq_lock', 'zone_lock', 'rcu', 'rcu-trace'), or a lock
+ type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
+ 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
+ 'srcu', 'srcu:R', 'srcu:W', 'rcu:R', 'rcu:W', 'semaphore'). Both
+ read/shared and write/exclusive tracking are supported for 'rwsem'
+ (including 'mmap_lock'), 'rwlock', 'rcu', and 'srcu' (where
+ read/shared mode matches 'rcu_read_lock()' / 'srcu_read_lock()'
+ critical sections and write/exclusive mode matches grace-period
+ waits 'synchronize_rcu*()', 'synchronize_srcu*()', 'rcu_barrier()',
+ and 'srcu_barrier()'); 'rcu-trace' ('rcu_read_lock_trace()')
+ supports read/shared tracking only; and 'mutex', 'spinlock'
(including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex',
'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support
write/exclusive tracking only. Note that 'semaphore' tracking
@@ -245,19 +251,21 @@ OPTIONS
--filter 'lock_held(mutex) || lock_held(rwsem)'
When filtering on a specific mutex symbol ('*_mutex'), rt-mutex symbol
- ('*_rt_mutex'), or write-locked 'mmap_lock' ('lock_held_write(mmap_lock)'
- or 'lock_held(mmap_lock:W)'), 'perf_sample_filter' inspects the lock's
- 'owner' field directly in kernel memory on each sample without attaching
- any BPF fentry/fexit hooks. For locks that do not maintain an
- authoritative per-task owner pointer (such as spinlocks, rwlocks,
- semaphores, reader-locked rwsems, and slab/type filters), it attaches
- BPF fentry/fexit programs to kernel lock acquire and release functions
- to track held locks in a BPF map. These BPF hooks run on every kernel
- lock acquisition and release for the attached lock family, even when no
- perf sample is being taken. To minimize this overhead, only the hooks
- for the relevant lock family and mode are attached when known (for
- example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions
- and 'rq_lock' only hooks spinlock functions), whereas slab filters
+ ('*_rt_mutex'), write-locked 'mmap_lock' ('lock_held_write(mmap_lock)'
+ or 'lock_held(mmap_lock:W)'), or 'rcu:R' / 'rcu-trace' (which inspect
+ 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting'),
+ 'perf_sample_filter' inspects the kernel fields directly on each sample
+ without attaching any BPF fentry/fexit hooks. For locks that do not
+ maintain an authoritative per-task owner pointer (such as spinlocks,
+ rwlocks, semaphores, reader-locked rwsems, SRCU locks, RCU synchronize
+ waits, and slab/type filters), it attaches BPF fentry/fexit programs to
+ kernel lock acquire and release functions to track held locks in a BPF
+ map. These BPF hooks run on every kernel lock acquisition and release
+ for the attached lock family, even when no perf sample is being taken.
+ To minimize this overhead, only the hooks for the relevant lock family
+ and mode are attached when known (for example, 'lock_held_read(mmap_lock)'
+ only hooks rwsem read functions, 'srcu' only hooks SRCU functions, and
+ 'rq_lock' only hooks spinlock functions), whereas slab filters
('&<slab>'), raw hex addresses, or symbols without a recognizable
lock-type suffix attach hooks across all lock families (including
high-frequency spinlocks).
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 4cf9192762f2..730466b7ebf6 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head)
expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex"));
clear_bpf_filter_exprs(expr_head);
+ /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */
+ TEST_ASSERT_VAL("parse lock_held(rcu)",
+ __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is rcu",
+ expr->lock_name && !strcmp(expr->lock_name, "rcu"));
+ clear_bpf_filter_exprs(expr_head);
+
/* Read/shared and write/exclusive predicates */
TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)",
__perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0);
diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh
index ebaf8ee41622..47f697f53a85 100755
--- a/tools/perf/tests/shell/record_bpf_filter.sh
+++ b/tools/perf/tests/shell/record_bpf_filter.sh
@@ -247,6 +247,26 @@ test_bpf_filter_lock_held() {
return
fi
+ if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed rcu:W/srcu]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \
+ -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]"
+ err=1
+ return
+ fi
+ fi
+
if ! perf record -e task-clock \
--filter 'lock_held_read(mmap_lock) || lock_held_write(mmap_lock)' \
-o "${perfdata}" true 2> /dev/null
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index d5a85ee98d5f..08a6e49ca16e 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -201,11 +201,19 @@ enum lock_prog_group {
LOCK_PROG_RTMUTEX = 1U << 6,
LOCK_PROG_SEM = 1U << 7,
LOCK_PROG_PERCPU_RWSEM = 1U << 8,
- LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM,
+ LOCK_PROG_SRCU_READ = 1U << 9,
+ LOCK_PROG_SRCU_WRITE = 1U << 10,
+ LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE,
+ LOCK_PROG_RCU_WRITE = 1U << 11,
+ LOCK_PROG_ALL = 0xfffU & ~LOCK_PROG_SEM,
};
static unsigned int lock_flags_to_prog_group(unsigned int flags)
{
+ if (flags & LCB_F_RCU)
+ return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0;
+ if (flags & LCB_F_SRCU)
+ return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE;
if (flags & LCB_F_MUTEX)
return LOCK_PROG_MUTEX;
if (flags & LCB_F_RT)
@@ -325,6 +333,32 @@ static const struct lock_prog_desc lock_progs[] = {
/* Percpu-RWsem */
LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+ /* SRCU */
+ LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe",
+ LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe",
+ LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited",
+ LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited",
+ LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE),
+ /* RCU grace-period synchronize / barrier */
+ LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited",
+ LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited",
+ LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE),
};
#undef LOCK_FPROG
@@ -612,6 +646,18 @@ static unsigned int supported_lock_prog_groups(struct btf *btf)
if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write"))
supported |= LOCK_PROG_PERCPU_RWSEM;
+ if ((btf_has_func(btf, "__srcu_read_lock") ||
+ btf_has_func(btf, "__srcu_read_lock_nmisafe")) &&
+ (btf_has_func(btf, "__srcu_read_unlock") ||
+ btf_has_func(btf, "__srcu_read_unlock_nmisafe")))
+ supported |= LOCK_PROG_SRCU_READ;
+
+ if (btf_has_func(btf, "synchronize_srcu"))
+ supported |= LOCK_PROG_SRCU_WRITE;
+
+ if (btf_has_func(btf, "synchronize_rcu"))
+ supported |= LOCK_PROG_RCU_WRITE;
+
return supported;
}
@@ -731,6 +777,19 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
*needs_lock_syms = true;
return;
}
+ if (!strcmp(base, "rcu")) {
+ if (mode & PBF_LOCK_MODE_WRITE) {
+ *prog_groups |= LOCK_PROG_RCU_WRITE;
+ *required_groups |= LOCK_PROG_RCU_WRITE;
+ }
+ return;
+ }
+ if (!strcmp(base, "rcu_read_lock") ||
+ !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") ||
+ !strcmp(base, "rcu_read_lock_trace")) {
+ /* Inspected directly on task_struct; no tracing hooks needed. */
+ return;
+ }
if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) {
*prog_groups |= groups;
*required_groups |= groups;
@@ -772,6 +831,13 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
groups |= LOCK_PROG_RWLOCK_WRITE;
*prog_groups |= groups;
*required_groups |= groups;
+ } else if (strtailcmp(base, "srcu") == 0) {
+ if (mode & PBF_LOCK_MODE_READ)
+ groups |= LOCK_PROG_SRCU_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ groups |= LOCK_PROG_SRCU_WRITE;
+ *prog_groups |= groups;
+ *required_groups |= groups;
} else {
*prog_groups |= LOCK_PROG_ALL;
}
@@ -920,6 +986,108 @@ static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr,
return -EINVAL;
}
+static u64 rcu_min_depth(struct evsel *evsel)
+{
+ /*
+ * Non-sleepable BPF sample filter programs always run under at least
+ * one kernel rcu_read_lock():
+ * - PERF_TYPE_PERF_EVENT: bpf_overflow_handler() takes rcu_read_lock()
+ * (and perf_swevent_event() takes an outer rcu_read_lock() for
+ * non-timer PERF_TYPE_SOFTWARE events, giving depth 2).
+ * - Standard tracepoints: trace_call_bpf() takes rcu_read_lock().
+ * - Faultable syscall tracepoints: trace_call_bpf_faultable() invokes
+ * bpf_prog_run_array_sleepable(), which takes guard(rcu)() for
+ * non-sleepable programs (!prog->sleepable) such as
+ * perf_sample_filter_tp.
+ */
+ if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE &&
+ evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK &&
+ evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK)
+ return 2;
+ return 1;
+}
+
+static u64 rcu_trace_min_depth(struct evsel *evsel, struct btf *btf)
+{
+ const char *ev_name;
+
+ /*
+ * Only PERF_TYPE_PERF_EVENT and PERF_TYPE_TRACEPOINT events can attach
+ * sample_filter.bpf.c (kprobes and uprobes require BPF_PROG_TYPE_KPROBE
+ * in __perf_event_set_bpf_prog()).
+ *
+ * For syscall tracepoints:
+ * - raw_syscalls:sys_enter and raw_syscalls:sys_exit are defined with
+ * TRACE_EVENT_SYSCALL(), whose __do_trace_sys_enter/exit wrapper
+ * takes guard(rcu_tasks_trace)() (depth 1) before perf_trace_buf
+ * submission calls trace_call_bpf().
+ * - syscalls:sys_enter_* and syscalls:sys_exit_* register
+ * perf_syscall_enter/exit on the raw sys_enter/sys_exit tracepoints
+ * (1st guard(rcu_tasks_trace)()) and, on kernels with
+ * trace_call_bpf_faultable(), acquire a 2nd guard(rcu_tasks_trace)()
+ * in trace_call_bpf_faultable() (depth 2).
+ */
+ if (evsel && evsel->core.attr.type == PERF_TYPE_TRACEPOINT) {
+ ev_name = evsel__name(evsel);
+ if (!ev_name)
+ return 0;
+ if (strstarts(ev_name, "raw_syscalls:"))
+ return 1;
+ if (strstarts(ev_name, "syscalls:")) {
+ if (!btf ||
+ btf__find_by_name_kind(btf, "trace_call_bpf_faultable",
+ BTF_KIND_FUNC) > 0)
+ return 2;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+static bool btf_struct_has_field(const struct btf *btf, s32 type_id,
+ const char *field_name)
+{
+ const struct btf_type *t;
+ const struct btf_member *m;
+ u16 i, vlen;
+
+ type_id = btf__resolve_type(btf, type_id);
+ if (type_id < 0)
+ return false;
+
+ t = btf__type_by_id(btf, type_id);
+ if (!t || (!btf_is_struct(t) && !btf_is_union(t)))
+ return false;
+
+ m = btf_members(t);
+ vlen = btf_vlen(t);
+ for (i = 0; i < vlen; i++, m++) {
+ const char *name = btf__name_by_offset(btf, m->name_off);
+
+ if (name && *name) {
+ if (!strcmp(name, field_name))
+ return true;
+ } else if (btf_struct_has_field(btf, m->type, field_name)) {
+ return true;
+ }
+ }
+ return false;
+}
+
+static bool btf_has_task_field(struct btf *btf, const char *field_name)
+{
+ s32 id;
+
+ if (!btf)
+ return false;
+
+ id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT);
+ if (id < 0)
+ return false;
+
+ return btf_struct_has_field(btf, id, field_name);
+}
+
/*
* Resolve a `lock_held(<name>)` filter expression into `entry->op`,
* `entry->part` (`enum perf_bpf_lock_target_kind`), and `entry->value`,
@@ -927,15 +1095,15 @@ static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr,
*
* Resolution order:
* 1. Slab cache name (when prefixed with '&', e.g. `&dentry`).
- * 2. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
- * `rq_lock`, `zone_lock`.
+ * 2. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`,
+ * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`.
* 3. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`,
- * `rwlock`, `percpu-rwsem`, `semaphore`, etc.
+ * `rwlock`, `percpu-rwsem`, `srcu`, `semaphore`, etc.
* 4. Hexadecimal lock address.
* 5. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`).
*/
-static int resolve_lock_expr(struct sample_filter_bpf *skel,
- struct perf_bpf_filter_expr *expr,
+static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct btf *btf, struct perf_bpf_filter_expr *expr,
struct perf_bpf_filter_entry *entry)
{
const char *name = expr->lock_name;
@@ -1017,6 +1185,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel,
return 0;
}
+ if (!strcmp(base, "rcu_read_lock")) {
+ if (mode == PBF_LOCK_MODE_WRITE) {
+ pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
+ return -EINVAL;
+ }
+ if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
+ pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ;
+ entry->value = rcu_min_depth(evsel);
+ return 0;
+ }
+
+ if (!strcmp(base, "rcu")) {
+ if ((mode & PBF_LOCK_MODE_READ) && btf &&
+ !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
+ pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU | mode;
+ entry->value = rcu_min_depth(evsel);
+ if ((mode & PBF_LOCK_MODE_WRITE) && skel)
+ skel->bss->has_rcu_lock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") ||
+ !strcmp(base, "rcu_read_lock_trace")) {
+ if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) {
+ pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
+ return -EINVAL;
+ }
+ if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) {
+ pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ;
+ entry->value = rcu_trace_min_depth(evsel, btf);
+ return 0;
+ }
+
if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
entry->part = PBF_LOCK_TYPE | mode;
entry->value = encoded_pair;
@@ -1086,7 +1296,7 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel,
}
static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
- struct perf_bpf_filter_entry *entry)
+ struct btf *btf, struct perf_bpf_filter_entry *entry)
{
int i = 0, ret;
struct perf_bpf_filter_expr *expr;
@@ -1103,7 +1313,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske
entry[i].term = expr->term;
entry[i].value = expr->val;
if (expr->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(skel, expr, &entry[i]);
+ ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]);
if (ret < 0)
return ret;
}
@@ -1121,7 +1331,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske
entry[i].term = group->term;
entry[i].value = group->val;
if (group->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(skel, group, &entry[i]);
+ ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]);
if (ret < 0)
return ret;
}
@@ -1389,7 +1599,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
int zero = 0;
- ret = get_filter_entries(evsel, NULL, entry);
+ ret = get_filter_entries(evsel, NULL, NULL, entry);
if (ret < 0) {
pr_err("Failed to process filter entries\n");
goto err;
@@ -1450,6 +1660,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
init_numa_data(skel, btf);
}
+ if (prog_groups & LOCK_PROG_RCU_WRITE) {
+ u64 rcu_addr = 1ULL;
+
+ /*
+ * Offset by 1 byte so rcu_state_addr is distinct from
+ * &rcu_state.node[0].lock at offset 0 of rcu_state.
+ */
+ if (find_kernel_lock_sym("rcu_state", &rcu_addr) == 0)
+ rcu_addr += 1;
+ skel->rodata->rcu_state_addr = rcu_addr;
+ }
ret = configure_lock_progs(skel, btf, prog_groups, required_groups,
needs_lock_syms, needs_slab_iter,
is_tracepoint);
@@ -1468,7 +1689,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
if (has_lock_filter && needs_slab_iter)
run_slab_cache_iter(skel);
- ret = get_filter_entries(evsel, skel, entry);
+ ret = get_filter_entries(evsel, skel, btf, entry);
if (ret < 0) {
pr_err("Failed to process filter entries\n");
goto err;
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index 317d8ccae63e..12bed223f9a5 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -39,6 +39,7 @@ struct contention_task_data {
*/
#define LCD_F_MMAP_LOCK (1U << 31)
#define LCD_F_SIGHAND_LOCK (1U << 30)
+#define LCD_F_RCU_LOCK (1U << 29)
#define LCB_F_SLAB_ID_SHIFT 16
#define LCB_F_SLAB_ID_START (1U << 16)
@@ -52,9 +53,11 @@ struct contention_task_data {
#define LCB_F_RT (1U << 3)
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+#define LCB_F_SRCU (1U << 6)
+#define LCB_F_RCU (1U << 7)
-#define LCB_F_TYPE_MAX (1U << 7)
-#define LCB_F_TYPE_MASK 0x0000007FU
+#define LCB_F_TYPE_MAX (1U << 8)
+#define LCB_F_TYPE_MASK 0x000000FFU
#define SLAB_NAME_MAX 28
@@ -255,6 +258,43 @@ static __always_inline __u64 get_current_siglock(__u64 curr)
return sh + bpf_core_field_offset(sighand->siglock);
}
+/*
+ * Check if `curr` is inside a preemptible RCU read-side critical section
+ * (`rcu_read_lock()`).
+ *
+ * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires
+ * `rcu_read_lock()` before invoking `perf_sample_filter` (and
+ * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for
+ * non-clock software events), `min_depth` is the baseline nesting count
+ * contributed by the perf event overflow handler itself (1 or 2).
+ */
+static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth)
+{
+ struct task_struct *task = (void *)curr;
+
+ if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting))
+ return false;
+
+ return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth;
+}
+
+/*
+ * Check if `curr` is inside a Tasks Trace RCU read-side critical section
+ * (`rcu_read_lock_trace()`). `min_depth` accounts for any baseline nesting
+ * contributed by the kernel tracing wrapper itself (2 on faultable
+ * `syscalls:*` tracepoints via `__DECLARE_TRACE_SYSCALL` and
+ * `trace_call_bpf_faultable()`, 1 on `raw_syscalls:*`, 0 otherwise).
+ */
+static __always_inline bool is_rcu_trace_lock_held(__u64 curr, int min_depth)
+{
+ struct task_struct *task = (void *)curr;
+
+ if (!curr || !bpf_core_field_exists(task->trc_reader_nesting))
+ return false;
+
+ return BPF_CORE_READ(task, trc_reader_nesting) > min_depth;
+}
+
/*
* Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar.
*
@@ -287,6 +327,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags)
if (get_current_siglock(curr) == lock)
return LCD_F_SIGHAND_LOCK;
break;
+ case LCB_F_RCU:
+ case LCB_F_RCU | LCB_F_WRITE:
+ return LCD_F_RCU_LOCK;
default:
break;
}
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index a65ee780d64d..7abc530ed50d 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -101,6 +101,20 @@ enum perf_bpf_lock_target_kind {
* low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
*/
PBF_LOCK_TYPE,
+ /*
+ * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`).
+ * Inspected directly via `current->rcu_read_lock_nesting > entry.value`,
+ * where `entry.value` is the baseline RCU nesting depth added by the
+ * kernel perf overflow path.
+ */
+ PBF_LOCK_RCU,
+ /*
+ * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`).
+ * Inspected directly via `current->trc_reader_nesting > entry.value`,
+ * where `entry.value` is the baseline nesting depth added by the
+ * kernel event tracing path.
+ */
+ PBF_LOCK_RCU_TRACE,
};
/*
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index d1192a93179e..60fe4c3a99a2 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -129,6 +129,7 @@ volatile const int use_idx_hash;
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
+const volatile __u64 rcu_state_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
@@ -142,6 +143,7 @@ int has_lock_class;
int has_lock_type;
int has_mmap_lock;
int has_siglock;
+int has_rcu_lock;
int track_rwsem_read;
__u8 lock_type_filter[LCB_F_TYPE_MAX];
@@ -282,6 +284,21 @@ static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry)
case PBF_LOCK_TYPE:
return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0;
+ case PBF_LOCK_RCU:
+ if ((mode & PBF_LOCK_MODE_READ) &&
+ is_rcu_lock_held(curr, (int)entry->value))
+ return true;
+ if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr &&
+ is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id))
+ return true;
+ return false;
+
+ case PBF_LOCK_RCU_TRACE:
+ if ((mode & PBF_LOCK_MODE_READ) &&
+ is_rcu_trace_lock_held(curr, (int)entry->value))
+ return true;
+ return false;
+
default:
return false;
}
@@ -727,11 +744,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla
if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
matched = true;
- if (has_mmap_lock || has_siglock) {
+ if (has_mmap_lock || has_siglock || has_rcu_lock) {
__u32 ltype = check_lock_type(lock_addr, type_flag);
if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
- (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
+ (has_siglock && ltype == LCD_F_SIGHAND_LOCK) ||
+ (has_rcu_lock && ltype == LCD_F_RCU_LOCK))
matched = true;
}
@@ -1391,6 +1409,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem)
return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
}
+/* SRCU acquire/release and synchronize hooks */
+SEC("fexit/__srcu_read_lock")
+int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/__srcu_read_unlock")
+int BPF_PROG(fentry_srcu_read_unlock, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fexit/__srcu_read_lock_nmisafe")
+int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/__srcu_read_unlock_nmisafe")
+int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/synchronize_srcu")
+int BPF_PROG(fentry_synchronize_srcu, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu")
+int BPF_PROG(fexit_synchronize_srcu, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_srcu_expedited")
+int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu_expedited")
+int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fentry/srcu_barrier")
+int BPF_PROG(fentry_srcu_barrier, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/srcu_barrier")
+int BPF_PROG(fexit_srcu_barrier, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+/* RCU grace-period synchronize / barrier hooks */
+SEC("fentry/synchronize_rcu")
+int BPF_PROG(fentry_synchronize_rcu)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu")
+int BPF_PROG(fexit_synchronize_rcu)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_rcu_expedited")
+int BPF_PROG(fentry_synchronize_rcu_expedited)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu_expedited")
+int BPF_PROG(fexit_synchronize_rcu_expedited)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_rcu_tasks")
+int BPF_PROG(fentry_synchronize_rcu_tasks)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu_tasks")
+int BPF_PROG(fexit_synchronize_rcu_tasks)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/rcu_barrier")
+int BPF_PROG(fentry_rcu_barrier)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/rcu_barrier")
+int BPF_PROG(fexit_rcu_barrier)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
SEC("raw_tp/bpf_test_finish")
int BPF_PROG(collect_lock_syms)
{
diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
index b75ddc6846bb..1fb91a829d54 100644
--- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
+++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
@@ -119,6 +119,8 @@ struct task_struct {
struct signal_struct *signal;
struct sighand_struct *sighand;
struct css_set *cgroups;
+ int rcu_read_lock_nesting;
+ int trc_reader_nesting;
} __attribute__((preserve_access_index));
struct trace_entry {
diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c
index f6e68d740860..37415f7b7f2c 100644
--- a/tools/perf/util/lock-contention.c
+++ b/tools/perf/util/lock-contention.c
@@ -39,6 +39,10 @@ const struct perf_lock_type lock_type_table[] = {
{ LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
/* alias for optimistic spinning only */
{ LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
+ { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" },
+ { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" },
+ { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" },
+ { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" },
};
int parse_call_stack(const struct option *opt __maybe_unused, const char *str,
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index 68e9bd77e8a9..88b2c11244ad 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -128,6 +128,8 @@ struct thread_stat {
#define LCB_F_RT (1U << 3)
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+#define LCB_F_SRCU (1U << 6)
+#define LCB_F_RCU (1U << 7)
struct perf_lock_type {
unsigned int flags;
@@ -140,7 +142,7 @@ struct perf_lock_type {
const char *lock_name;
};
-extern const struct perf_lock_type lock_type_table[14];
+extern const struct perf_lock_type lock_type_table[18];
struct evlist;
struct machine;
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v5 8/8] perf lock contention: Extend BPF lock contention to RCU and SRCU
2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (6 preceding siblings ...)
2026-10-20 20:58 ` [PATCH v5 7/8] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
@ 2026-10-20 20:58 ` Ian Rogers
2026-10-20 23:37 ` sashiko-bot
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
8 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-20 20:58 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Extend 'perf lock contention --use-bpf' to track RCU and SRCU
grace-period and barrier waits:
- Attach fentry/fexit programs to synchronize_rcu(),
synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(),
synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier()
when available in BTF.
- Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu',
'-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct
symbol names).
Example usage:
# perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1
total wait avg wait type caller
35.12 ms 8.78 ms rcu:W cleanup_net+0x284
7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114
# perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1
total wait avg wait address symbol
35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu)
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/tests/shell/lock_contention.sh | 14 ++
tools/perf/util/bpf_lock_contention.c | 171 ++++++++++++--
.../perf/util/bpf_skel/lock_contention.bpf.c | 221 ++++++++++++++----
tools/perf/util/machine.c | 11 +
5 files changed, 362 insertions(+), 62 deletions(-)
diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt
index c17b3e318169..bfeb98c976e3 100644
--- a/tools/perf/Documentation/perf-lock.txt
+++ b/tools/perf/Documentation/perf-lock.txt
@@ -188,11 +188,14 @@ CONTENTION OPTIONS
Show lock contention only for given lock types (comma separated list).
Available values are:
semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W,
- rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem,
- pcpu-sem:R, pcpu-sem:W, mutex
+ rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem,
+ pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W
Note that RW-variant of locks have :R and :W suffix. Names without the
suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W.
+ For RCU and SRCU with --use-bpf, contention tracks grace-period and
+ barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(),
+ srcu_barrier()).
-L::
--lock-filter=<value>::
diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh
index 5df9a0cd48a3..999dcfd1ef26 100755
--- a/tools/perf/tests/shell/lock_contention.sh
+++ b/tools/perf/tests/shell/lock_contention.sh
@@ -181,6 +181,13 @@ test_type_filter()
err=1
exit
fi
+
+ perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result}
+ if [ "$(grep -c -v -w rcu "${result}")" != "0" ]; then
+ echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")"
+ err=1
+ exit
+ fi
}
test_lock_filter()
@@ -224,6 +231,13 @@ test_lock_filter()
err=1
exit
fi
+
+ perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result}
+ if [ "$(grep -c -v -w rcu "${result}")" != "0" ]; then
+ echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")"
+ err=1
+ exit
+ fi
}
test_stack_filter()
diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c
index c20bd075664e..dfb384f1c5e6 100644
--- a/tools/perf/util/bpf_lock_contention.c
+++ b/tools/perf/util/bpf_lock_contention.c
@@ -16,6 +16,7 @@
#include <bpf/bpf.h>
#include <bpf/btf.h>
#include <inttypes.h>
+#include <unistd.h>
#include "bpf_skel/lock_contention.skel.h"
#include "bpf_skel/lock_data.h"
@@ -180,6 +181,133 @@ static void init_numa_data(struct lock_contention *con)
free(buf);
}
+static bool has_bpf_fentry_support(struct btf *btf)
+{
+ /*
+ * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and
+ * bpf_get_func_arg() helper availability (5.17+) used by
+ * get_lock_addr() in the SRCU fentry/fexit hooks.
+ */
+ struct bpf_insn insns[] = {
+ { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 },
+ { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3,
+ .src_reg = BPF_REG_10 },
+ { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 },
+ { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg },
+ { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 },
+ { .code = BPF_JMP | BPF_EXIT },
+ };
+ LIBBPF_OPTS(bpf_prog_load_opts, opts,
+ .expected_attach_type = BPF_TRACE_FENTRY,
+ );
+ s32 btf_id;
+ int prog_fd, link_fd;
+
+ if (!btf)
+ return false;
+
+ btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC);
+ if (btf_id <= 0)
+ return false;
+
+ opts.attach_btf_id = btf_id;
+ prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL",
+ insns, ARRAY_SIZE(insns), &opts);
+ if (prog_fd < 0)
+ return false;
+
+ link_fd = bpf_raw_tracepoint_open(NULL, prog_fd);
+ if (link_fd >= 0)
+ close(link_fd);
+ close(prog_fd);
+ return link_fd >= 0;
+}
+
+static void configure_rcu_contention_progs(struct lock_contention *con,
+ bool has_rcu_lock)
+{
+ static const struct {
+ size_t prog_offset;
+ const char *func;
+ bool is_srcu;
+ } rcu_progs[] = {
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu),
+ "synchronize_rcu", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu),
+ "synchronize_rcu", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited),
+ "synchronize_rcu_expedited", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited),
+ "synchronize_rcu_expedited", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks),
+ "synchronize_rcu_tasks", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks),
+ "synchronize_rcu_tasks", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier),
+ "rcu_barrier", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier),
+ "rcu_barrier", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu),
+ "synchronize_srcu", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu),
+ "synchronize_srcu", true },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited),
+ "synchronize_srcu_expedited", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited),
+ "synchronize_srcu_expedited", true },
+ { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier),
+ "srcu_barrier", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier),
+ "srcu_barrier", true },
+ };
+ struct symbol *sym;
+ struct map *kmap;
+ bool need_rcu = true, need_srcu = true;
+ bool fentry_ok;
+
+ sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap);
+ if (sym) {
+ /*
+ * Offset by 1 byte so rcu_state_addr is distinct from
+ * &rcu_state.node[0].lock at offset 0 of rcu_state.
+ */
+ skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start) + 1;
+ map__put(kmap);
+ } else {
+ skel->rodata->rcu_state_addr = 1ULL;
+ }
+
+ if (con->filters->nr_types) {
+ need_rcu = false;
+ need_srcu = false;
+ for (int i = 0; i < con->filters->nr_types; i++) {
+ if (con->filters->types[i] & LCB_F_RCU)
+ need_rcu = true;
+ if (con->filters->types[i] & LCB_F_SRCU)
+ need_srcu = true;
+ }
+ }
+
+ if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) {
+ if (!has_rcu_lock && !con->filters->nr_addrs)
+ need_rcu = false;
+ if (!con->filters->nr_addrs && !con->filters->nr_slabs)
+ need_srcu = false;
+ }
+
+ fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf);
+
+ for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) {
+ struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset;
+ bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu;
+ bool enable = fentry_ok && want &&
+ btf__find_by_name_kind(con->btf, rcu_progs[i].func,
+ BTF_KIND_FUNC) > 0;
+
+ bpf_program__set_autoload(*progp, enable);
+ }
+}
+
int lock_contention_prepare(struct lock_contention *con)
{
int i, fd;
@@ -187,6 +315,7 @@ int lock_contention_prepare(struct lock_contention *con)
struct evlist *evlist = con->evlist;
struct target *target = con->target;
bool has_mmap_lock = false;
+ bool has_rcu_lock = false;
/* make sure it loads the kernel map before lookup */
map__load(machine__kernel_map(con->machine));
@@ -200,6 +329,7 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64));
bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries);
bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries);
+ bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries);
if (con->aggr_mode == LOCK_AGGR_TASK)
bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries);
@@ -249,6 +379,10 @@ int lock_contention_prepare(struct lock_contention *con)
has_mmap_lock = true;
continue;
}
+ if (!strcmp(con->filters->syms[i], "rcu")) {
+ has_rcu_lock = true;
+ continue;
+ }
sym = machine__find_kernel_symbol_by_name(con->machine,
con->filters->syms[i],
@@ -269,7 +403,7 @@ int lock_contention_prepare(struct lock_contention *con)
addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start);
con->filters->addrs = addrs;
}
- naddrs = con->filters->nr_addrs ?: has_mmap_lock;
+ naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock);
skel->rodata->has_addr = 1;
}
@@ -305,6 +439,7 @@ int lock_contention_prepare(struct lock_contention *con)
skel->rodata->needs_callstack = con->save_callstack;
skel->rodata->lock_owner = con->owner;
skel->rodata->has_mmap_lock = has_mmap_lock;
+ skel->rodata->has_rcu_lock = has_rcu_lock;
if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) {
if (cgroup_is_v2("perf_event"))
@@ -321,6 +456,7 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map__set_max_entries(skel->maps.slab_filter, nslabs);
init_numa_data(con);
+ configure_rcu_contention_progs(con, has_rcu_lock);
if (lock_contention_bpf__load(skel) < 0) {
pr_err("Failed to load lock-contention BPF skeleton\n");
@@ -503,23 +639,30 @@ static void account_end_timestamp(struct lock_contention *con)
enum lock_aggr_mode aggr_mode = con->aggr_mode;
struct tstamp_data ts_data, *cpu_data;
- /* Iterate per-task tstamp map (key = TID) */
- ts_fd = bpf_map__fd(skel->maps.tstamp);
+ /* Iterate per-task tstamp maps (key = TID) */
+ int ts_fds[] = {
+ bpf_map__fd(skel->maps.tstamp),
+ bpf_map__fd(skel->maps.tstamp_rcu),
+ };
+
stat_fd = bpf_map__fd(skel->maps.lock_stat);
- prev_key = NULL;
- while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) {
- if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) {
- int pid = key;
+ for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) {
+ ts_fd = ts_fds[i];
+ prev_key = NULL;
+ while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) {
+ if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) {
+ int pid = key;
- if (aggr_mode == LOCK_AGGR_TASK && con->owner)
- pid = ts_data.flags;
+ if (aggr_mode == LOCK_AGGR_TASK && con->owner)
+ pid = ts_data.flags;
- update_lock_stat(stat_fd, pid, end_ts, aggr_mode,
- &ts_data);
- }
+ update_lock_stat(stat_fd, pid, end_ts, aggr_mode,
+ &ts_data);
+ }
- prev_key = &key;
+ prev_key = &key;
+ }
}
/* Now it'll check per-cpu tstamp map which doesn't have TID. */
@@ -604,6 +747,8 @@ static const char *lock_contention_get_name(struct lock_contention *con,
return "mmap_lock";
if (flags & LCD_F_SIGHAND_LOCK)
return "siglock";
+ if (flags & LCD_F_RCU_LOCK)
+ return "rcu";
/* global locks with symbols */
sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap);
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index 0f2118006005..26936f329dba 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -59,6 +59,18 @@ struct {
__uint(max_entries, MAX_ENTRIES);
} tstamp SEC(".maps");
+/*
+ * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits
+ * so they do not collide with inner sleeping lock contentions (such as
+ * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, int);
+ __type(value, struct tstamp_data);
+ __uint(max_entries, MAX_ENTRIES);
+} tstamp_rcu SEC(".maps");
+
/* maintain per-CPU timestamp at the beginning of contention */
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
@@ -160,6 +172,7 @@ const volatile int has_addr;
const volatile int has_cgroup;
const volatile int has_slab;
const volatile int has_mmap_lock;
+const volatile int has_rcu_lock;
const volatile int needs_callstack;
const volatile int stack_skip;
const volatile int lock_owner;
@@ -172,6 +185,7 @@ const volatile int aggr_mode;
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
+const volatile __u64 rcu_state_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
@@ -218,7 +232,7 @@ static inline __u64 get_current_cgroup_id(void)
return BPF_CORE_READ(cgrp, kn, id);
}
-static inline int can_record(u64 *ctx)
+static __always_inline int can_record(__u64 lock, __u32 flags)
{
bool is_addr_ok = false;
@@ -242,7 +256,6 @@ static inline int can_record(u64 *ctx)
if (has_type) {
__u8 *ok;
- __u32 flags = (__u32)ctx[1];
ok = bpf_map_lookup_elem(&type_filter, &flags);
if (!ok)
@@ -251,10 +264,9 @@ static inline int can_record(u64 *ctx)
if (has_addr) {
__u8 *ok;
- __u64 addr = ctx[0];
- ok = bpf_map_lookup_elem(&addr_filter, &addr);
- if (!ok && !has_slab && !has_mmap_lock)
+ ok = bpf_map_lookup_elem(&addr_filter, &lock);
+ if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock)
return 0;
is_addr_ok = !!ok;
@@ -272,26 +284,27 @@ static inline int can_record(u64 *ctx)
if (is_addr_ok)
return 1;
- /* slab and mmap_lock are part of the addr_filter */
+ /* slab, mmap_lock, and rcu_lock are part of the addr_filter */
if (has_slab && bpf_get_kmem_cache) {
__u8 *ok;
- __u64 addr = ctx[0];
long kmem_cache_addr;
- kmem_cache_addr = (long)bpf_get_kmem_cache(addr);
+ kmem_cache_addr = (long)bpf_get_kmem_cache(lock);
ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr);
if (ok)
return 1;
- else if (!has_mmap_lock)
+ else if (!has_mmap_lock && !has_rcu_lock)
return 0;
}
- if (has_mmap_lock) {
- __u64 lock = ctx[0];
- __u32 flag = ctx[1];
+ if (has_mmap_lock || has_rcu_lock) {
+ __u32 ltype = check_lock_type(lock, flags);
- if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK)
- return 0;
+ if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK)
+ return 1;
+ if (has_rcu_lock && ltype == LCD_F_RCU_LOCK)
+ return 1;
+ return 0;
}
return 1;
@@ -351,6 +364,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
{
__u32 pid;
struct tstamp_data *pelem;
+ void *map;
/* Use per-cpu array map for spinlock and rwlock */
if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) {
@@ -363,8 +377,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
return pelem;
}
+ map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp;
pid = bpf_get_current_pid_tgid();
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ pelem = bpf_map_lookup_elem(map, &pid);
/* Do not update the element for nested locks */
if (pelem && pelem->lock)
return NULL;
@@ -372,12 +387,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
if (pelem == NULL) {
struct tstamp_data zero = {};
- if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) {
+ if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) {
__sync_fetch_and_add(&task_fail, 1);
return NULL;
}
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ pelem = bpf_map_lookup_elem(map, &pid);
if (pelem == NULL) {
__sync_fetch_and_add(&task_fail, 1);
return NULL;
@@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags)
}
}
-SEC("tp_btf/contention_begin")
-int contention_begin(u64 *ctx)
+static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags)
{
struct tstamp_data *pelem;
- if (!enabled || !can_record(ctx))
+ if (!enabled || !can_record(lock, flags))
return 0;
- pelem = get_tstamp_elem(ctx[1]);
+ pelem = get_tstamp_elem(flags);
if (pelem == NULL)
return 0;
pelem->timestamp = bpf_ktime_get_ns();
- pelem->lock = (__u64)ctx[0];
- pelem->flags = (__u32)ctx[1];
+ pelem->lock = lock;
+ pelem->flags = flags;
if (aggr_mode == LOCK_AGGR_CGROUP)
pelem->cgroup_id = get_current_cgroup_id();
@@ -559,8 +573,21 @@ int contention_begin(u64 *ctx)
}
}
skip_owner:
- pelem->stack_id = bpf_get_stackid(ctx, &stacks,
- BPF_F_FAST_STACK_CMP | stack_skip);
+ {
+ int skip = stack_skip;
+
+ /*
+ * Unlike tp_btf/contention_begin (which fires inside a
+ * lock slowpath via trace_contention_begin), fentry
+ * RCU/SRCU programs attach directly to the top-level
+ * wait API, which is 2 frames shallower.
+ */
+ if (flags & (LCB_F_RCU | LCB_F_SRCU))
+ skip = stack_skip > 2 ? stack_skip - 2 : 0;
+
+ pelem->stack_id = bpf_get_stackid(ctx, &stacks,
+ BPF_F_FAST_STACK_CMP | skip);
+ }
if (pelem->stack_id < 0)
__sync_fetch_and_add(&stack_fail, 1);
} else if (aggr_mode == LOCK_AGGR_TASK) {
@@ -588,8 +615,7 @@ int contention_begin(u64 *ctx)
return 0;
}
-SEC("tp_btf/contention_end")
-int contention_end(u64 *ctx)
+static __always_inline int do_contention_end(__u64 lock, __s32 ret, bool is_rcu)
{
__u32 pid = 0, idx = 0;
struct tstamp_data *pelem;
@@ -597,30 +623,34 @@ int contention_end(u64 *ctx)
struct contention_data *data;
__u64 timestamp;
__u64 duration;
- bool need_delete = false;
+ void *ts_map = NULL;
if (!enabled)
return 0;
/*
- * For spinlock and rwlock, it needs to get the timestamp for the
- * per-cpu map. However, contention_end does not have the flags
- * so it cannot know whether it reads percpu or hash map.
- *
- * Try per-cpu map first and check if there's active contention.
- * If it is, do not read hash map because it cannot go to sleeping
- * locks before releasing the spinning locks.
+ * RCU/SRCU fexit hooks always use the dedicated per-task tstamp_rcu
+ * map. For tp_btf/contention_end, try the per-cpu map first (used by
+ * spinlocks and rwlocks) and fall back to the per-task tstamp map.
*/
- pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx);
- if (pelem && pelem->lock) {
- if (pelem->lock != ctx[0])
- return 0;
- } else {
+ if (is_rcu) {
pid = bpf_get_current_pid_tgid();
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
- if (!pelem || pelem->lock != ctx[0])
+ pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid);
+ if (!pelem || pelem->lock != lock)
return 0;
- need_delete = true;
+ ts_map = &tstamp_rcu;
+ } else {
+ pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx);
+ if (pelem && pelem->lock) {
+ if (pelem->lock != lock)
+ return 0;
+ } else {
+ pid = bpf_get_current_pid_tgid();
+ pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ if (!pelem || pelem->lock != lock)
+ return 0;
+ ts_map = &tstamp;
+ }
}
timestamp = bpf_ktime_get_ns();
@@ -648,7 +678,6 @@ int contention_end(u64 *ctx)
*/
else {
u32 i = 0;
- s32 ret = (s32)ctx[1];
u64 *buf;
otdata->timestamp = timestamp;
@@ -713,7 +742,7 @@ int contention_end(u64 *ctx)
if (lock_owner)
key.pid = pelem->flags;
else {
- if (!need_delete)
+ if (!ts_map)
pid = bpf_get_current_pid_tgid();
key.pid = pid;
}
@@ -795,11 +824,109 @@ int contention_end(u64 *ctx)
check_lock_delay(pelem->lock);
pelem->lock = 0;
- if (need_delete)
- bpf_map_delete_elem(&tstamp, &pid);
+ if (ts_map)
+ bpf_map_delete_elem(ts_map, &pid);
return 0;
}
+SEC("tp_btf/contention_begin")
+int contention_begin(u64 *ctx)
+{
+ return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]);
+}
+
+SEC("tp_btf/contention_end")
+int contention_end(u64 *ctx)
+{
+ return do_contention_end((__u64)ctx[0], (__s32)ctx[1], false);
+}
+
+/* RCU grace-period synchronize / barrier contention hooks */
+SEC("fentry/synchronize_rcu")
+int BPF_PROG(fentry_synchronize_rcu)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu")
+int BPF_PROG(fexit_synchronize_rcu)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+SEC("fentry/synchronize_rcu_expedited")
+int BPF_PROG(fentry_synchronize_rcu_expedited)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu_expedited")
+int BPF_PROG(fexit_synchronize_rcu_expedited)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+SEC("fentry/synchronize_rcu_tasks")
+int BPF_PROG(fentry_synchronize_rcu_tasks)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu_tasks")
+int BPF_PROG(fexit_synchronize_rcu_tasks)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+SEC("fentry/rcu_barrier")
+int BPF_PROG(fentry_rcu_barrier)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/rcu_barrier")
+int BPF_PROG(fexit_rcu_barrier)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+/* SRCU grace-period synchronize / barrier contention hooks */
+SEC("fentry/synchronize_srcu")
+int BPF_PROG(fentry_synchronize_srcu, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu")
+int BPF_PROG(fexit_synchronize_srcu, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0, true);
+}
+
+SEC("fentry/synchronize_srcu_expedited")
+int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu_expedited")
+int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0, true);
+}
+
+SEC("fentry/srcu_barrier")
+int BPF_PROG(fentry_srcu_barrier, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/srcu_barrier")
+int BPF_PROG(fexit_srcu_barrier, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0, true);
+}
+
SEC("raw_tp/bpf_test_finish")
int BPF_PROG(end_timestamp)
{
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 1d9d3bf57720..0a0f223b3335 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -3379,6 +3379,17 @@ bool machine__is_lock_function(struct machine *machine, u64 addr)
return true;
}
+ {
+ struct symbol *sym = machine__find_kernel_symbol(machine, addr, NULL);
+
+ if (sym && (!strncmp(sym->name, "synchronize_rcu", 15) ||
+ !strncmp(sym->name, "synchronize_srcu", 16) ||
+ !strcmp(sym->name, "__synchronize_srcu") ||
+ !strcmp(sym->name, "rcu_barrier") ||
+ !strcmp(sym->name, "srcu_barrier")))
+ return true;
+ }
+
return false;
}
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* Re: [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events
2026-10-20 15:30 ` Ian Rogers
@ 2026-10-20 22:29 ` Namhyung Kim
0 siblings, 0 replies; 106+ messages in thread
From: Namhyung Kim @ 2026-10-20 22:29 UTC (permalink / raw)
To: Ian Rogers
Cc: acme, adrian.hunter, james.clark, jolsa, linux-kernel,
linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu
On Tue, Oct 20, 2026 at 08:30:26AM -0700, Ian Rogers wrote:
> On Mon, Oct 19, 2026 at 10:41 PM Namhyung Kim <namhyung@kernel.org> wrote:
> >
> > On Mon, Oct 19, 2026 at 03:09:10PM -0700, Ian Rogers wrote:
> > > Commit 466db4275edd ("perf parse-events: Add parse_uid_filter helper")
> > > updated set_filter() to route non-tracepoint filter terms (such as uid
> > > and gid) on PERF_TYPE_TRACEPOINT events to evsel->bpf_filters instead
> > > of the ftrace tracepoint predicate filter.
> > >
> > > However, in the kernel's __perf_event_set_bpf_prog()
> > > (kernel/events/core.c), attaching a BPF program to a perf_event fd
> > > branches on perf_event_is_tracing(event):
> > > - Non-tracing events (hardware PMU, software events, breakpoints) call
> > > perf_event_set_bpf_handler(), which requires BPF_PROG_TYPE_PERF_EVENT
> > > (SEC("perf_event")).
> > > - Tracepoint events (event->pmu == &perf_tracepoint) reject
> > > BPF_PROG_TYPE_PERF_EVENT with -EINVAL and instead require
> > > BPF_PROG_TYPE_TRACEPOINT (SEC("tracepoint")), which is attached to
> > > event->tp_event->prog_array and executed in
> > > perf_trace_run_bpf_submit() before perf_tp_event(), dropping the
> > > sample if the BPF program returns 0. Furthermore, the
> > > bpf_cast_to_kern_ctx() kfunc is only permitted in
> > > BPF_PROG_TYPE_PERF_EVENT programs.
> > >
> > > Add SEC("tracepoint") int perf_tp_sample_filter(void *ctx) in
> > > sample_filter.bpf.c to evaluate context-independent filter terms (uid,
> > > gid, tid, cpu, time, cgroup) using BPF helpers without calling
> > > bpf_cast_to_kern_ctx(). When preparing a BPF filter on a
> > > PERF_TYPE_TRACEPOINT evsel, autoload perf_tp_sample_filter instead of
> > > perf_sample_filter and attach it once to the shared trace_event_call.
> > >
> > > Before:
> > > # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true
> > > libbpf: prog 'perf_sample_filter': failed to attach to perf_event FD 3: Invalid argument
> > > Failed to attach perf sample-filter program
> > > failed to set filter "BPF" on event kmem:mm_page_alloc with 22 (Invalid argument)
> > >
> > > After:
> > > # perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true
> > > [ perf record: Woken up 1 times to write data ]
> > > [ perf record: Captured and wrote 0.024 MB perf.data (42 samples) ]
> > >
> > > Fixes: 466db4275edd ("perf parse-events: Add parse_uid_filter helper")
> > > Signed-off-by: Ian Rogers <irogers@google.com>
> > > ---
[SNIP]
> > > @@ -674,7 +761,7 @@ int perf_bpf_filter__pin(void)
> > > goto err;
> > > }
> > >
> > > - ret = bpf_object__pin(skel->obj, path);
> > > + ret = pin_filter_object(skel, path);
> >
> > Why is this change needed?
>
> Before this patch, sample_filter.bpf.c only contained a single BPF
> program (perf_sample_filter), which was always loaded. This patch adds
> a second program (perf_tp_sample_filter) to the same skeleton (and
> patch 4 adds the lock-tracking programs to it as well), and uses
> bpf_program__set_autoload() before loading the skeleton so we can
> select only the program(s) needed for a given event.
>
> In perf_bpf_filter__pin() and perf_bpf_filter__unpin() (for 'perf
> record --setup-filter pin|unpin'), only perf_sample_filter is used,
> because the pinned filter mechanism shares a single program across
> privileged sessions by looking up kctx->event->id via
> bpf_cast_to_kern_ctx(), which is only available to SEC("perf_event")
> programs. So autoload is disabled for perf_tp_sample_filter (and in
> patch 4, for the lock-tracking programs) when pinning and unpinning.
>
> A consequence of disabling autoload on any program in the skeleton is
> that libbpf's bpf_object__pin() and bpf_object__unpin() fail:
> - bpf_object__pin() calls bpf_object__pin_maps() and
> bpf_object__pin_programs().
> - While bpf_object__pin_maps() skips maps that have autocreate
> disabled, bpf_object__pin_programs() and bpf_object__unpin_programs()
> iterate over every program in skel->obj without checking whether
> autoload was enabled.
Looks like it needs a fix in the libbpf.
> - When bpf_program__pin() or bpf_program__unpin() is called on a
> program that wasn't loaded (prog->fd < 0), libbpf returns -EINVAL
> ("prog 'perf_tp_sample_filter': can't pin/unpin program that wasn't
> loaded").
>
> pin_filter_object() and unpin_filter_object() avoid this by calling
> bpf_object__pin_maps() / bpf_object__unpin_maps() for the maps and
> explicitly pinning/unpinning only the loaded perf_sample_filter
> program.
Thanks for the explanation. It'd be nice if we can have it in a comment.
Thanks,
Namhyung
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v5 1/8] perf lock contention: Filter events before creating lock_stat
2026-10-20 20:58 ` [PATCH v5 1/8] perf lock contention: Filter events before creating lock_stat Ian Rogers
@ 2026-10-20 22:38 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-20 22:38 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> In report_lock_contention_begin_event(), lock_stat_findnew() was
> called before checking filters.nr_types (-Y), filters.nr_addrs (-L),
> and match_callstack_filter() (-S).
>
> When a filtered-out lock event (such as a mutex contention when
> running with -Y spinlock) shared the same aggregation key as a
> subsequent matching event (for example, contention on the mutex's
> internal wait_lock spinlock from the same caller), the first event
> created the lock_stat entry with the filtered-out lock's flags and the
> second event reused that entry, causing non-spinlock entries to appear
> in -Y spinlock output.
>
> Check the type, address, and callstack filters before looking up or
> creating the lock_stat entry.
>
> Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option")
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261020205859.1080743-1-irogers@google.com?part=1
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v5 2/8] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF
2026-10-20 20:58 ` [PATCH v5 2/8] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers
@ 2026-10-20 22:41 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-20 22:41 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> In perf_bpf_filter__prepare(), the unpinned filter path attaches
> skel->progs.perf_sample_filter to each FD(evsel, x, y) using
> bpf_program__attach_perf_event_opts() and ignores the returned
> struct bpf_link *, leaking the allocated struct bpf_link_perf and its
> BPF_PERF_EVENT link fd.
>
> Furthermore, bpf_program__attach_perf_event_opts() is designed for
> callers that transfer ownership of the perf_event fd to the returned
> bpf_link: when bpf_link__destroy() is called, bpf_link_perf_detach()
> unconditionally disables and closes perf_link->perf_event_fd. Because
> evsel->core.fd owns FD(evsel, x, y) and closes them in evlist__close(),
> destroying the returned bpf_link would double-close the perf_event file
> descriptors.
>
> Fix this by attaching the program directly with
> [ ... ]
>
> Reported-by: Namhyung Kim <namhyung@kernel.org>
> Closes: https://lore.kernel.org/linux-perf-users/20261020053226.55936-1-namhyung@kernel.org/
> Fixes: 56ec9457a4a2 ("perf bpf filter: Implement event sample filtering")
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261020205859.1080743-1-irogers@google.com?part=2
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-20 20:58 ` [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
@ 2026-10-20 22:42 ` Namhyung Kim
2026-10-20 23:02 ` sashiko-bot
1 sibling, 0 replies; 106+ messages in thread
From: Namhyung Kim @ 2026-10-20 22:42 UTC (permalink / raw)
To: Ian Rogers
Cc: acme, adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
On Tue, Oct 20, 2026 at 01:58:54PM -0700, Ian Rogers wrote:
> Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers
> (get_lock_owner, get_current_mmap_lock, get_current_siglock,
> get_lock_addr, check_lock_type, do_collect_zone_lock,
> do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into
> tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF
> sample filtering.
This is fine.
>
> Also add rw_semaphore.count and struct rt_mutex_base to the minimal
> vmlinux.h and extend get_lock_owner() to support rt_mutex and check
> the mutex handoff flag.
Please do this in a separate commit. I'd like to see this kind of
commit only moves the existing code.
>
> Signed-off-by: Ian Rogers <irogers@google.com>
> ---
> tools/perf/builtin-lock.c | 27 --
> .../perf/util/bpf_skel/lock_contention.bpf.c | 265 +-------------
> tools/perf/util/bpf_skel/lock_data.h | 338 ++++++++++++++++++
> tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 +
> tools/perf/util/lock-contention.c | 19 +
> tools/perf/util/lock-contention.h | 13 +
> 6 files changed, 393 insertions(+), 274 deletions(-)
>
> diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
> index 778923c92f42..a031709a2cc2 100644
> --- a/tools/perf/builtin-lock.c
> +++ b/tools/perf/builtin-lock.c
> @@ -1482,33 +1482,6 @@ static void sort_result(void)
> }
> }
>
> -static const struct {
> - unsigned int flags;
> - /*
> - * Name of the lock flags (access), with delimeter ':'.
> - * For example, rwsem:R of rwsem:W.
> - */
> - const char *flags_name;
> - /* Name of the lock (type), for example, rwlock or rwsem. */
> - const char *lock_name;
> -} lock_type_table[] = {
> - { 0, "semaphore", "semaphore" },
> - { LCB_F_SPIN, "spinlock", "spinlock" },
> - { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
> - { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
> - { LCB_F_READ, "rwsem:R", "rwsem" },
> - { LCB_F_WRITE, "rwsem:W", "rwsem" },
> - { LCB_F_RT, "rt-mutex", "rt-mutex" },
> - { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
> - { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
> - { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
> - { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
> - { LCB_F_MUTEX, "mutex", "mutex" },
> - { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
> - /* alias for optimistic spinning only */
> - { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
> -};
> -
> static const char *get_type_flags_name(unsigned int flags)
> {
> flags &= LCB_F_TYPE_MASK;
> diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
> index 0d9c6f55050e..0f2118006005 100644
> --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
> +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
> @@ -8,23 +8,9 @@
>
> #include "lock_data.h"
>
> -/* for collect_lock_syms(). 4096 was rejected by the verifier */
> -#define MAX_CPUS 1024
> -
> -/* for collect_zone_lock(). It should be more than the actual zones. */
> -#define MAX_ZONES 10
> -
> /* for do_lock_delay(). Arbitrarily set to 1 million. */
> #define MAX_LOOP (1U << 20)
>
> -/* lock contention flags from include/trace/events/lock.h */
> -#define LCB_F_SPIN (1U << 0)
> -#define LCB_F_READ (1U << 1)
> -#define LCB_F_WRITE (1U << 2)
> -#define LCB_F_RT (1U << 3)
> -#define LCB_F_PERCPU (1U << 4)
> -#define LCB_F_MUTEX (1U << 5)
> -
> /* callstack storage */
> struct {
> __uint(type, BPF_MAP_TYPE_STACK_TRACE);
> @@ -96,13 +82,6 @@ struct {
> __uint(max_entries, MAX_ENTRIES);
> } task_data SEC(".maps");
>
> -struct {
> - __uint(type, BPF_MAP_TYPE_HASH);
> - __uint(key_size, sizeof(__u64));
> - __uint(value_size, sizeof(__u32));
> - __uint(max_entries, MAX_ENTRIES);
> -} lock_syms SEC(".maps");
> -
> struct {
> __uint(type, BPF_MAP_TYPE_HASH);
> __uint(key_size, sizeof(__u32));
> @@ -138,6 +117,13 @@ struct {
> __uint(max_entries, 1);
> } cgroup_filter SEC(".maps");
>
> +struct {
> + __uint(type, BPF_MAP_TYPE_HASH);
> + __uint(key_size, sizeof(__u64));
> + __uint(value_size, sizeof(__u32));
> + __uint(max_entries, MAX_ENTRIES);
> +} lock_syms SEC(".maps");
> +
Looks like un uncessasry change.
> struct {
> __uint(type, BPF_MAP_TYPE_HASH);
> __uint(key_size, sizeof(long));
> @@ -159,22 +145,6 @@ struct {
> __uint(max_entries, 1);
> } lock_delays SEC(".maps");
>
> -struct rw_semaphore___old {
> - struct task_struct *owner;
> -} __attribute__((preserve_access_index));
> -
> -struct rw_semaphore___new {
> - atomic_long_t owner;
> -} __attribute__((preserve_access_index));
> -
> -struct mm_struct___old {
> - struct rw_semaphore mmap_sem;
> -} __attribute__((preserve_access_index));
> -
> -struct mm_struct___new {
> - struct rw_semaphore mmap_lock;
> -} __attribute__((preserve_access_index));
> -
> struct cas_ctx {
> struct contention_data *data;
> u64 duration;
> @@ -182,8 +152,6 @@ struct cas_ctx {
[SNIP]
> diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
> index 652e114e6b87..317d8ccae63e 100644
> --- a/tools/perf/util/bpf_skel/lock_data.h
> +++ b/tools/perf/util/bpf_skel/lock_data.h
> @@ -45,6 +45,14 @@ struct contention_task_data {
> #define LCB_F_SLAB_ID_END (1U << 26)
> #define LCB_F_SLAB_ID_MASK 0x03FF0000U
>
> +/* lock contention flags from include/trace/events/lock.h */
> +#define LCB_F_SPIN (1U << 0)
> +#define LCB_F_READ (1U << 1)
> +#define LCB_F_WRITE (1U << 2)
> +#define LCB_F_RT (1U << 3)
> +#define LCB_F_PERCPU (1U << 4)
> +#define LCB_F_MUTEX (1U << 5)
> +
> #define LCB_F_TYPE_MAX (1U << 7)
> #define LCB_F_TYPE_MASK 0x0000007FU
>
> @@ -76,4 +84,334 @@ struct slab_cache_data {
> char name[SLAB_NAME_MAX];
> };
>
> +#ifdef __BPF__
Why is this needed?
Thanks,
Namhyung
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter
2026-10-20 20:58 ` [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter Ian Rogers
@ 2026-10-20 22:49 ` Namhyung Kim
2026-10-20 23:17 ` sashiko-bot
1 sibling, 0 replies; 106+ messages in thread
From: Namhyung Kim @ 2026-10-20 22:49 UTC (permalink / raw)
To: Ian Rogers
Cc: acme, adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
On Tue, Oct 20, 2026 at 01:58:55PM -0700, Ian Rogers wrote:
> Add BPF sample filter predicates lock_held(<lock>),
> lock_held_read(<lock>) / lock_held_shared(<lock>), and
> lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R'
> and ':W' suffixes on <lock>) to allow recording perf event samples only
> when a specified kernel lock is held (or not held with '!').
>
> Supported <lock> specifications match perf lock contention:
> - Global/static kernel lock symbols (e.g. lock_held(cgroup_mutex))
> - Hexadecimal kernel lock addresses
> - Slab cache names prefixed with '&' (e.g. lock_held(&dentry),
> lock_held_read(&dentry))
> - Per-process and semi-global locks: mmap_lock, siglock, rq_lock,
> zone_lock
> - Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W,
> rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W,
> semaphore
Can we do this step by step? I think you can add the core logic with,
for example, global mutex first. Then add other lock types, semi-
global, slab cache support and so on in the later patches.
I believe that would help reviewers.
Thanks,
Namhyung
>
> Supported lock hold modes by lock family:
> - Both read/shared and write/exclusive: rwsem (including mmap_lock)
> and rwlock.
> - Write/exclusive only: mutex, spinlock (including siglock, rq_lock,
> zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write()
> / percpu_up_write(); fast-path percpu_down_read() / percpu_up_read()
> are inlined per-CPU counter operations without function entry points).
>
> When evaluating the filter on sample overflow, perf_sample_filter first
> inspects the lock's owner field directly in kernel memory where
> possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls
> back to a held_locks BPF LRU hash map (tracking separate read_count and
> write_count) populated by fentry/fexit hooks on kernel lock/unlock
> functions. When filtering on a specific mutex symbol (*_mutex),
> rt-mutex symbol (*_rt_mutex), or write-locked mmap_lock (mmap_lock:W /
> lock_held_write(mmap_lock)), no fentry/fexit hooks are attached at all.
>
> Note that counting semaphores (down*() / up()) do not enforce same-task
> ownership in the kernel: when a semaphore is used as a cross-task
> completion/signaling mechanism (down() in one task woken by up() in
> another task or interrupt handler, which proposed kernel trackers like
> CONFIG_DEPT model as wait/event dependencies bounded by execution
> context rather than lock ownership), fexit/down and fentry/up will not
> pair on the same task ID. Semaphore hooks are therefore only enabled
> when 'semaphore' or a '*_sem' symbol (such as console_sem) is
> explicitly specified, and held_locks uses BPF_MAP_TYPE_LRU_HASH so any
> unpaired cross-task entries are automatically evicted.
>
> Example usage:
>
> # perf record -a -g -e cycles --filter 'lock_held(mmap_lock)' \
> -- perf bench mem mmap -t 2 -l 100
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ]
>
> # perf report --stdio --no-children
> # Overhead Command Shared Object Symbol
> # ........ ........ ................. ........................
> #
> 18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms
> |
> ---clear_page_erms
> get_page_from_freelist
> __alloc_pages_noprof
> folio_alloc_mpol_noprof
> vma_alloc_folio_noprof
> do_anonymous_page
> __handle_mm_fault
> handle_mm_fault
> do_user_addr_fault
> exc_page_fault
> asm_exc_page_fault
> worker_thread
>
> Read/shared vs write/exclusive holding can also be filtered separately:
>
> # perf record -a -g -e cycles --filter 'lock_held_read(mmap_lock)' ...
> # perf record -a -g -e cycles --filter 'lock_held_write(mmap_lock)' ...
> # perf record -a -g -e cycles --filter 'lock_held(mmap_lock:W)' ...
>
> Signed-off-by: Ian Rogers <irogers@google.com>
> ---
> tools/perf/Documentation/perf-record.txt | 58 +-
> tools/perf/util/bpf-filter.c | 1071 +++++++++++++++++-
> tools/perf/util/bpf-filter.h | 13 +
> tools/perf/util/bpf-filter.l | 31 +-
> tools/perf/util/bpf-filter.y | 47 +-
> tools/perf/util/bpf_skel/sample-filter.h | 66 ++
> tools/perf/util/bpf_skel/sample_filter.bpf.c | 1014 +++++++++++++++++
> tools/perf/util/parse-events.c | 3 +-
> 8 files changed, 2277 insertions(+), 26 deletions(-)
>
> diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
> index 2351f7b4f5fc..8496b20987a0 100644
> --- a/tools/perf/Documentation/perf-record.txt
> +++ b/tools/perf/Documentation/perf-record.txt
> @@ -204,7 +204,63 @@ OPTIONS
> ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr,
> code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat,
> p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock,
> - mem_dtlb, mem_blk, mem_hops, uid, gid
> + mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>),
> + lock_held_read(<lock>), lock_held_shared(<lock>),
> + lock_held_write(<lock>), lock_held_exclusive(<lock>)
> +
> + The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters
> + samples based on whether the specified kernel lock is currently held
> + by the kernel thread or CPU. To differentiate between shared/read
> + holding and exclusive/write holding, use 'lock_held_read(<lock>)'
> + (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)'
> + (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to
> + <lock> (e.g. 'lock_held(mmap_lock:R)' or 'lock_held(mmap_lock:W)').
> + <lock> can be a kernel lock symbol name (e.g.
> + 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache
> + name prefixed with '&' (e.g. 'lock_held(&dentry)',
> + 'lock_held_read(&dentry)'), a well-known lock name ('mmap_lock',
> + 'siglock', 'rq_lock', 'zone_lock'), or a lock type ('mutex',
> + 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R',
> + 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore').
> + Both read/shared and write/exclusive tracking are supported for
> + 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', 'spinlock'
> + (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex',
> + 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support
> + write/exclusive tracking only. Note that 'semaphore' tracking
> + ('down*()' / 'up()') assumes binary mutex-like usage where the
> + acquiring task also releases the semaphore (such as 'console_sem'),
> + rather than cross-task completion signaling.
> +
> + For example, 'lock_held()' can be combined with memory allocation or
> + reclaim events to detect allocations or direct memory reclaim occurring
> + while a kernel lock is held (which can stall lock waiters for extended
> + periods):
> +
> + # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \
> + --filter 'lock_held(mutex) || lock_held(rwsem)'
> + # perf record -a -g -e kmem:mm_page_alloc \
> + --filter 'gfp_flags & 0x400' \
> + --filter 'lock_held(mutex) || lock_held(rwsem)'
> + # perf record -a -g -e major-faults \
> + --filter 'lock_held(mutex) || lock_held(rwsem)'
> +
> + When filtering on a specific mutex symbol ('*_mutex'), rt-mutex symbol
> + ('*_rt_mutex'), or write-locked 'mmap_lock' ('lock_held_write(mmap_lock)'
> + or 'lock_held(mmap_lock:W)'), 'perf_sample_filter' inspects the lock's
> + 'owner' field directly in kernel memory on each sample without attaching
> + any BPF fentry/fexit hooks. For locks that do not maintain an
> + authoritative per-task owner pointer (such as spinlocks, rwlocks,
> + semaphores, reader-locked rwsems, and slab/type filters), it attaches
> + BPF fentry/fexit programs to kernel lock acquire and release functions
> + to track held locks in a BPF map. These BPF hooks run on every kernel
> + lock acquisition and release for the attached lock family, even when no
> + perf sample is being taken. To minimize this overhead, only the hooks
> + for the relevant lock family and mode are attached when known (for
> + example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions
> + and 'rq_lock' only hooks spinlock functions), whereas slab filters
> + ('&<slab>'), raw hex addresses, or symbols without a recognizable
> + lock-type suffix attach hooks across all lock families (including
> + high-frequency spinlocks).
>
> The <operator> can be one of:
> ==, !=, >, >=, <, <=, &
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index ac0e7cf40f3a..d5a85ee98d5f 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
> @@ -40,21 +40,35 @@
> * The 'dropped' map is to keep how many samples it dropped by the filter and
> * it will be reported as lost samples.
> */
> +#include <errno.h>
> +#include <stdio.h>
> #include <stdlib.h>
> +#include <stdbool.h>
> +#include <stddef.h>
> +#include <string.h>
> #include <fcntl.h>
> +#include <unistd.h>
> #include <sys/ioctl.h>
> #include <sys/stat.h>
>
> #include <bpf/bpf.h>
> +#include <bpf/btf.h>
> #include <linux/err.h>
> +#include <linux/kernel.h>
> #include <linux/list.h>
> +#include <linux/string.h>
> +#include <linux/zalloc.h>
> #include <api/fs/fs.h>
> #include <internal/xyarray.h>
> #include <perf/threadmap.h>
> +#include <symbol/kallsyms.h>
>
> #include "util/cap.h"
> #include "util/debug.h"
> #include "util/evsel.h"
> +#include "util/lock-contention.h"
> +#include "util/string2.h"
> +#include "util/symbol_conf.h"
> #include "util/target.h"
> #include "util/thread_map.h"
> #include "util/bpf-utils.h"
> @@ -63,6 +77,7 @@
> #include <util/bpf-filter-flex.h>
> #include <util/bpf-filter-bison.h>
>
> +#include "bpf_skel/lock_data.h"
> #include "bpf_skel/sample-filter.h"
> #include "bpf_skel/sample_filter.skel.h"
>
> @@ -123,7 +138,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
> {
> const struct perf_sample_info *info;
>
> - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
> + if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID ||
> + expr->term == PBF_TERM_LOCK_HELD) {
> /* Not dependent on the sample_type as computed from a BPF helper. */
> return 0;
> }
> @@ -166,9 +182,913 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
> return -1;
> }
>
> -static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry)
> +/*
> + * Program groups for `lock_held(...)` BPF tracing hooks.
> + *
> + * When a `lock_held(...)` filter is specified, only the BPF tracing program
> + * groups required for the requested lock(s) are autoloaded and attached.
> + * If no `lock_held(...)` filter is present, none of these programs are loaded.
> + */
> +enum lock_prog_group {
> + LOCK_PROG_MUTEX = 1U << 0,
> + LOCK_PROG_RWSEM_READ = 1U << 1,
> + LOCK_PROG_RWSEM_WRITE = 1U << 2,
> + LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE,
> + LOCK_PROG_SPIN = 1U << 3,
> + LOCK_PROG_RWLOCK_READ = 1U << 4,
> + LOCK_PROG_RWLOCK_WRITE = 1U << 5,
> + LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE,
> + LOCK_PROG_RTMUTEX = 1U << 6,
> + LOCK_PROG_SEM = 1U << 7,
> + LOCK_PROG_PERCPU_RWSEM = 1U << 8,
> + LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM,
> +};
> +
> +static unsigned int lock_flags_to_prog_group(unsigned int flags)
> +{
> + if (flags & LCB_F_MUTEX)
> + return LOCK_PROG_MUTEX;
> + if (flags & LCB_F_RT)
> + return LOCK_PROG_RTMUTEX;
> + if (flags & LCB_F_PERCPU)
> + return LOCK_PROG_PERCPU_RWSEM;
> + if (flags == LCB_F_SPIN)
> + return LOCK_PROG_SPIN;
> + if (flags & LCB_F_SPIN)
> + return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE;
> + if (flags & LCB_F_READ)
> + return LOCK_PROG_RWSEM_READ;
> + if (flags & LCB_F_WRITE)
> + return LOCK_PROG_RWSEM_WRITE;
> + return LOCK_PROG_SEM;
> +}
> +
> +struct lock_prog_desc {
> + size_t prog_offset;
> + const char *btf_func;
> + unsigned int group;
> +};
> +
> +#define LOCK_FPROG(field, func, grp) \
> + { offsetof(struct sample_filter_bpf, progs.field), func, grp }
> +
> +static const struct lock_prog_desc lock_progs[] = {
> + /* Mutex */
> + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
> + LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
> + /* RW semaphore */
> + LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ),
> + LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ),
> + LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ),
> + LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ),
> + LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested",
> + LOCK_PROG_RWSEM_READ),
> + LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ),
> + LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE),
> + LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE),
> + LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE),
> + LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE),
> + LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested",
> + LOCK_PROG_RWSEM_WRITE),
> + LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE),
> + LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM),
> + LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ),
> + LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE),
> + /* Spinlock */
> + LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested",
> + LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN),
> + LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN),
> + LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN),
> + LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable",
> + LOCK_PROG_SPIN),
> + LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore",
> + LOCK_PROG_SPIN),
> + LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN),
> + /* RWlock */
> + LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore",
> + LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore",
> + LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE),
> + /* RT-Mutex */
> + LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX),
> + LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX),
> + LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX),
> + LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible",
> + LOCK_PROG_RTMUTEX),
> + LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX),
> + LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX),
> + LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX),
> + /* Semaphore */
> + LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM),
> + LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM),
> + LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM),
> + LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM),
> + LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM),
> + LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM),
> + /* Percpu-RWsem */
> + LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
> + LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
> +};
> +
> +#undef LOCK_FPROG
> +
> +static int split_lock_name_mode(const char *name, int expr_part,
> + char *base_buf, size_t buf_sz,
> + bool *has_amp, unsigned int *mode_out)
> +{
> + const char *bare;
> + size_t len;
> + unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK;
> + unsigned int suffix_mode = 0;
> +
> + if (!name || !*name)
> + return -EINVAL;
> +
> + *has_amp = (name[0] == '&');
> + bare = *has_amp ? name + 1 : name;
> + len = strlen(bare);
> + if (len == 0)
> + return -EINVAL;
> +
> + if (len > 2 && !strcmp(bare + len - 2, ":R")) {
> + suffix_mode = PBF_LOCK_MODE_READ;
> + len -= 2;
> + } else if (len > 2 && !strcmp(bare + len - 2, ":W")) {
> + suffix_mode = PBF_LOCK_MODE_WRITE;
> + len -= 2;
> + }
> +
> + if (len == 0 || len >= buf_sz)
> + return -EINVAL;
> +
> + if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY &&
> + suffix_mode && expr_mode != suffix_mode)
> + return -EINVAL;
> +
> + memcpy(base_buf, bare, len);
> + base_buf[len] = '\0';
> +
> + if (suffix_mode)
> + *mode_out = suffix_mode;
> + else if (expr_mode)
> + *mode_out = expr_mode;
> + else
> + *mode_out = PBF_LOCK_MODE_ANY;
> +
> + return 0;
> +}
> +
> +/*
> + * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R",
> + * "spinlock", "percpu-rwsem").
> + *
> + * A lock type name may correspond to up to two flag combinations (e.g.
> + * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed
> + * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is
> + * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`.
> + */
> +static bool parse_lock_type_name(const char *name, unsigned int mode,
> + struct sample_filter_bpf *skel,
> + u64 *encoded_pair, unsigned int *prog_groups)
> +{
> + bool found = false;
> + u32 flag1 = 0xff, flag2 = 0xff;
> + unsigned int groups = 0;
> + const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name;
> +
> + for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) {
> + u32 f = lock_type_table[i].flags;
> + bool is_read = (f & LCB_F_READ) != 0;
> +
> + if (strcmp(lock_type_table[i].lock_name, lookup) &&
> + strcmp(lock_type_table[i].flags_name, name))
> + continue;
> +
> + /*
> + * percpu_down_read() / percpu_up_read() fast paths are inlined
> + * per-CPU counter increments without function entry points, so
> + * only writer holding (percpu_down_write / percpu_up_write) can
> + * be tracked for percpu-rwsem. rwlock-rt is also not hooked.
> + */
> + if ((f & LCB_F_PERCPU) && is_read)
> + continue;
> + if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE)))
> + continue;
> + if ((f & LCB_F_MUTEX) && (f & LCB_F_SPIN))
> + continue;
> +
> + if (is_read && !(mode & PBF_LOCK_MODE_READ))
> + continue;
> + if (!is_read && !(mode & PBF_LOCK_MODE_WRITE))
> + continue;
> +
> + if (!found)
> + flag1 = f;
> + else if (f != flag1)
> + flag2 = f;
> + if (skel && f < LCB_F_TYPE_MAX) {
> + skel->bss->has_lock_type = 1;
> + skel->bss->lock_type_filter[f] = 1;
> + }
> + groups |= lock_flags_to_prog_group(f);
> + found = true;
> + }
> +
> + if (!found)
> + return false;
> +
> + if (encoded_pair)
> + *encoded_pair = flag1 | ((u64)flag2 << 8);
> + if (prog_groups)
> + *prog_groups |= groups;
> + return true;
> +}
> +
> +struct lock_sym_cb_args {
> + const char *name;
> + u64 addr;
> + bool found;
> +};
> +
> +static int find_lock_sym_cb(void *arg, const char *name,
> + char type __maybe_unused, u64 start)
> +{
> + struct lock_sym_cb_args *args = arg;
> +
> + if (!strcmp(name, args->name)) {
> + args->found = true;
> + args->addr = start;
> + if (start != 0)
> + return 1;
> + }
> + return 0;
> +}
> +
> +static int find_kernel_lock_sym(const char *name, u64 *addr)
> +{
> + struct lock_sym_cb_args args = {
> + .name = name,
> + };
> + const char *kallsyms_filename = "/proc/kallsyms";
> +
> + if (symbol_conf.kallsyms_name != NULL)
> + kallsyms_filename = symbol_conf.kallsyms_name;
> +
> + if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0)
> + return -ENOENT;
> +
> + if (!args.found)
> + return -ENOENT;
> +
> + if (args.addr == 0) {
> + pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n",
> + name);
> + return -EACCES;
> + }
> +
> + *addr = args.addr;
> + return 0;
> +}
> +
> +static bool parse_lock_hex_addr(const char *str, u64 *addr)
> +{
> + char *end;
> + unsigned long long val;
> +
> + if (!str || !*str)
> + return false;
> +
> + errno = 0;
> + val = strtoull(str, &end, 16);
> + if (*end != '\0' || errno != 0 || val == 0)
> + return false;
> +
> + *addr = val;
> + return true;
> +}
> +
> +static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf)
> +{
> + u64 addr = 0;
> + char *buf = NULL, *p;
> + size_t len;
> + long last = -1;
> + int ret;
> +
> + if (!btf)
> + return;
> +
> + ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT);
> + if (ret < 0)
> + return;
> +
> + ret = btf__resolve_size(btf, ret);
> + if (ret < 0)
> + return;
> +
> + skel->rodata->sizeof_zone = ret;
> +
> + if (find_kernel_lock_sym("contig_page_data", &addr) == 0) {
> + skel->rodata->contig_page_data_addr = addr;
> + return;
> + }
> +
> + if (find_kernel_lock_sym("node_data", &addr) != 0)
> + return;
> +
> + skel->rodata->node_data_addr = addr;
> +
> + if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) {
> + p = buf;
> + while (p && *p) {
> + char *end;
> +
> + last = strtol(p, &end, 0);
> + if (end == p)
> + break;
> + p = end;
> + if (*p == ',' || *p == '-' || *p == '\n')
> + p++;
> + }
> + skel->rodata->nr_nodes = last + 1;
> + free(buf);
> + }
> +}
> +
> +static bool btf_has_func(struct btf *btf, const char *name)
> +{
> + return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0;
> +}
> +
> +static unsigned int supported_lock_prog_groups(struct btf *btf)
> +{
> + unsigned int supported = 0;
> +
> + if (!btf)
> + return 0;
> +
> + if ((btf_has_func(btf, "mutex_lock") || btf_has_func(btf, "mutex_lock_nested")) &&
> + btf_has_func(btf, "mutex_unlock"))
> + supported |= LOCK_PROG_MUTEX;
> +
> + if ((btf_has_func(btf, "down_read") || btf_has_func(btf, "down_read_nested")) &&
> + btf_has_func(btf, "up_read"))
> + supported |= LOCK_PROG_RWSEM_READ;
> +
> + if ((btf_has_func(btf, "down_write") || btf_has_func(btf, "down_write_nested")) &&
> + btf_has_func(btf, "up_write"))
> + supported |= LOCK_PROG_RWSEM_WRITE;
> +
> + /*
> + * Require both acquire and all unlock entry points in BTF so we never
> + * attach spinlock/rwlock acquire hooks when unlock functions are
> + * inlined (e.g. !CONFIG_UNINLINE_SPIN_UNLOCK or CONFIG_INLINE_*_UNLOCK).
> + */
> + if ((btf_has_func(btf, "_raw_spin_lock") || btf_has_func(btf, "_raw_spin_lock_nested")) &&
> + btf_has_func(btf, "_raw_spin_unlock") &&
> + btf_has_func(btf, "_raw_spin_unlock_irq") &&
> + btf_has_func(btf, "_raw_spin_unlock_irqrestore") &&
> + btf_has_func(btf, "_raw_spin_unlock_bh"))
> + supported |= LOCK_PROG_SPIN;
> +
> + if (btf_has_func(btf, "_raw_read_lock") &&
> + btf_has_func(btf, "_raw_read_unlock") &&
> + btf_has_func(btf, "_raw_read_unlock_irq") &&
> + btf_has_func(btf, "_raw_read_unlock_irqrestore") &&
> + btf_has_func(btf, "_raw_read_unlock_bh"))
> + supported |= LOCK_PROG_RWLOCK_READ;
> +
> + if ((btf_has_func(btf, "_raw_write_lock") || btf_has_func(btf, "_raw_write_lock_nested")) &&
> + btf_has_func(btf, "_raw_write_unlock") &&
> + btf_has_func(btf, "_raw_write_unlock_irq") &&
> + btf_has_func(btf, "_raw_write_unlock_irqrestore") &&
> + btf_has_func(btf, "_raw_write_unlock_bh"))
> + supported |= LOCK_PROG_RWLOCK_WRITE;
> +
> + if ((btf_has_func(btf, "rt_mutex_lock") || btf_has_func(btf, "rt_mutex_lock_nested")) &&
> + btf_has_func(btf, "rt_mutex_unlock"))
> + supported |= LOCK_PROG_RTMUTEX;
> +
> + if (btf_has_func(btf, "down") && btf_has_func(btf, "up"))
> + supported |= LOCK_PROG_SEM;
> +
> + if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write"))
> + supported |= LOCK_PROG_PERCPU_RWSEM;
> +
> + return supported;
> +}
> +
> +/*
> + * Enable or disable autoload for each lock-tracking BPF program in the
> + * skeleton before `sample_filter_bpf__load()` is called.
> + *
> + * Each program is enabled only if its group is in `needed_groups` and its
> + * target kernel function or tracepoint exists in the running kernel's BTF.
> + */
> +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
> + unsigned int needed_groups,
> + unsigned int required_groups,
> + bool needs_lock_syms, bool needs_slab_iter,
> + bool is_tracepoint)
> +{
> + unsigned int supported_groups = supported_lock_prog_groups(btf);
> + unsigned int enabled_progs = 0;
> +
> + bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
> + bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
> +
> + if ((required_groups & supported_groups) != required_groups ||
> + (needed_groups && !(needed_groups & supported_groups))) {
> + pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n");
> + return -ENOTSUP;
> + }
> +
> + needed_groups &= supported_groups;
> + if (skel)
> + skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ);
> +
> + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) {
> + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset;
> + struct bpf_program *prog = *progp;
> + bool enable = false;
> +
> + if ((needed_groups & lock_progs[i].group) && btf &&
> + lock_progs[i].btf_func) {
> + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func,
> + BTF_KIND_FUNC) > 0)
> + enable = true;
> + }
> +
> + if (enable)
> + enabled_progs++;
> + bpf_program__set_autoload(prog, enable);
> + }
> +
> + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms);
> + if (needs_slab_iter && btf &&
> + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) {
> + bpf_program__set_autoload(skel->progs.slab_cache_iter, true);
> + } else {
> + bpf_program__set_autoload(skel->progs.slab_cache_iter, false);
> + }
> +
> + if (needed_groups && !enabled_progs) {
> + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
> + return -ENOTSUP;
> + }
> + return 0;
> +}
> +
> +/*
> + * Pre-scan a single filter expression prior to loading the BPF skeleton in
> + * order to determine which BPF maps need to be sized and which lock tracing
> + * program groups need to be autoloaded.
> + */
> +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
> + bool *has_lock_filter,
> + unsigned int *prog_groups,
> + unsigned int *required_groups,
> + bool *needs_lock_syms,
> + bool *needs_slab_iter)
> +{
> + char base[128];
> + bool has_amp;
> + unsigned int mode, groups = 0;
> + u64 dummy;
> +
> + if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name)
> + return;
> +
> + *has_lock_filter = true;
> + if (split_lock_name_mode(expr->lock_name, expr->part, base,
> + sizeof(base), &has_amp, &mode) < 0)
> + return;
> +
> + /* Leading '&' is reserved for slab cache locks (e.g. '&dentry'). */
> + if (has_amp) {
> + *needs_slab_iter = true;
> + *prog_groups |= LOCK_PROG_ALL;
> + return;
> + }
> +
> + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
> + /*
> + * Write ownership of mm->mmap_lock is inspected directly via
> + * is_rwsem_writer_owner() without tracing hooks; only reader
> + * tracking requires LOCK_PROG_RWSEM_READ.
> + */
> + if (mode & PBF_LOCK_MODE_READ) {
> + *prog_groups |= LOCK_PROG_RWSEM_READ;
> + *required_groups |= LOCK_PROG_RWSEM_READ;
> + }
> + return;
> + }
> + if (!strcmp(base, "siglock")) {
> + *prog_groups |= LOCK_PROG_SPIN;
> + *required_groups |= LOCK_PROG_SPIN;
> + return;
> + }
> + if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) {
> + *prog_groups |= LOCK_PROG_SPIN;
> + *required_groups |= LOCK_PROG_SPIN;
> + *needs_lock_syms = true;
> + return;
> + }
> + if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) {
> + *prog_groups |= groups;
> + *required_groups |= groups;
> + return;
> + }
> +
> + if (parse_lock_hex_addr(base, &dummy)) {
> + *prog_groups |= LOCK_PROG_ALL;
> + return;
> + }
> +
> + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0 ||
> + strtailcmp(base, "mutex") == 0) {
> + /* Inspected directly via lock->owner; no tracing hooks needed. */
> + return;
> + } else if (strtailcmp(base, "percpu_rwsem") == 0 ||
> + strtailcmp(base, "pcpu_rwsem") == 0) {
> + *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
> + *required_groups |= LOCK_PROG_PERCPU_RWSEM;
> + } else if (strtailcmp(base, "rwsem") == 0) {
> + if (mode & PBF_LOCK_MODE_READ) {
> + *prog_groups |= LOCK_PROG_RWSEM_READ;
> + *required_groups |= LOCK_PROG_RWSEM_READ;
> + }
> + if (mode & PBF_LOCK_MODE_WRITE) {
> + *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM;
> + *required_groups |= LOCK_PROG_RWSEM_WRITE;
> + }
> + } else if (strtailcmp(base, "spinlock") == 0) {
> + *prog_groups |= LOCK_PROG_SPIN;
> + *required_groups |= LOCK_PROG_SPIN;
> + } else if (strtailcmp(base, "sem") == 0) {
> + *prog_groups |= LOCK_PROG_SEM;
> + *required_groups |= LOCK_PROG_SEM;
> + } else if (strtailcmp(base, "rwlock") == 0) {
> + if (mode & PBF_LOCK_MODE_READ)
> + groups |= LOCK_PROG_RWLOCK_READ;
> + if (mode & PBF_LOCK_MODE_WRITE)
> + groups |= LOCK_PROG_RWLOCK_WRITE;
> + *prog_groups |= groups;
> + *required_groups |= groups;
> + } else {
> + *prog_groups |= LOCK_PROG_ALL;
> + }
> +}
> +
> +static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter,
> + unsigned int *prog_groups,
> + unsigned int *required_groups,
> + bool *needs_lock_syms,
> + bool *needs_slab_iter)
> +{
> + struct perf_bpf_filter_expr *expr, *group;
> +
> + *has_lock_filter = false;
> + *prog_groups = 0;
> + *required_groups = 0;
> + *needs_lock_syms = false;
> + *needs_slab_iter = false;
> +
> + list_for_each_entry(expr, &evsel->bpf_filters, list) {
> + prescan_one_lock_expr(expr, has_lock_filter, prog_groups,
> + required_groups, needs_lock_syms,
> + needs_slab_iter);
> + if (expr->op == PBF_OP_GROUP_BEGIN) {
> + list_for_each_entry(group, &expr->groups, list) {
> + prescan_one_lock_expr(group, has_lock_filter,
> + prog_groups, required_groups,
> + needs_lock_syms, needs_slab_iter);
> + }
> + }
> + }
> +}
> +
> +static void run_slab_cache_iter(struct sample_filter_bpf *skel)
> +{
> + struct bpf_link *link;
> + int fd;
> + char buf[256];
> +
> + if (!bpf_program__autoload(skel->progs.slab_cache_iter))
> + return;
> +
> + link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL);
> + if (!link || IS_ERR(link)) {
> + pr_debug("cannot attach slab cache iter\n");
> + return;
> + }
> +
> + fd = bpf_iter_create(bpf_link__fd(link));
> + if (fd >= 0) {
> + while (read(fd, buf, sizeof(buf)) > 0)
> + continue;
> + close(fd);
> + }
> +
> + bpf_link__destroy(link);
> +}
> +
> +static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
> + u64 *kmem_cache_addr)
> +{
> + int cache_fd;
> + u64 key, *prev_key = NULL;
> +
> + if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter))
> + return false;
> +
> + cache_fd = bpf_map__fd(skel->maps.slab_caches);
> + while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) {
> + struct slab_cache_data data;
> +
> + if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0)
> + break;
> +
> + if (!strcmp(data.name, name)) {
> + *kmem_cache_addr = key;
> + return true;
> + }
> + prev_key = &key;
> + }
> + return false;
> +}
> +
> +static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr,
> + enum perf_bpf_filter_op *op_out)
> +{
> + switch (expr->op) {
> + case PBF_OP_EQ:
> + if (expr->val == 1) {
> + *op_out = PBF_OP_EQ;
> + return 0;
> + }
> + if (expr->val == 0) {
> + *op_out = PBF_OP_NEQ;
> + return 0;
> + }
> + break;
> + case PBF_OP_NEQ:
> + if (expr->val == 0) {
> + *op_out = PBF_OP_EQ;
> + return 0;
> + }
> + if (expr->val == 1) {
> + *op_out = PBF_OP_NEQ;
> + return 0;
> + }
> + break;
> + case PBF_OP_GT:
> + if (expr->val == 0) {
> + *op_out = PBF_OP_EQ;
> + return 0;
> + }
> + break;
> + case PBF_OP_GE:
> + if (expr->val == 1) {
> + *op_out = PBF_OP_EQ;
> + return 0;
> + }
> + break;
> + case PBF_OP_LT:
> + if (expr->val == 1) {
> + *op_out = PBF_OP_NEQ;
> + return 0;
> + }
> + break;
> + case PBF_OP_LE:
> + if (expr->val == 0) {
> + *op_out = PBF_OP_NEQ;
> + return 0;
> + }
> + break;
> + case PBF_OP_AND:
> + if (expr->val & 1) {
> + *op_out = PBF_OP_EQ;
> + return 0;
> + }
> + break;
> + case PBF_OP_GROUP_BEGIN:
> + case PBF_OP_GROUP_END:
> + case PBF_OP_DONE:
> + default:
> + break;
> + }
> + pr_err("Error: Invalid comparison for boolean lock_held(%s) predicate\n",
> + expr->lock_name ?: "");
> + return -EINVAL;
> +}
> +
> +/*
> + * Resolve a `lock_held(<name>)` filter expression into `entry->op`,
> + * `entry->part` (`enum perf_bpf_lock_target_kind`), and `entry->value`,
> + * and populate the corresponding BPF filter maps and BSS control flags.
> + *
> + * Resolution order:
> + * 1. Slab cache name (when prefixed with '&', e.g. `&dentry`).
> + * 2. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
> + * `rq_lock`, `zone_lock`.
> + * 3. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`,
> + * `rwlock`, `percpu-rwsem`, `semaphore`, etc.
> + * 4. Hexadecimal lock address.
> + * 5. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`).
> + */
> +static int resolve_lock_expr(struct sample_filter_bpf *skel,
> + struct perf_bpf_filter_expr *expr,
> + struct perf_bpf_filter_entry *entry)
> +{
> + const char *name = expr->lock_name;
> + char base[128];
> + bool has_amp;
> + unsigned int mode;
> + u64 addr = 0, encoded_pair = 0;
> + u8 one = 1;
> + int ret;
> +
> + ret = normalize_lock_expr_op(expr, &entry->op);
> + if (ret < 0)
> + return ret;
> +
> + if (split_lock_name_mode(name, expr->part, base, sizeof(base),
> + &has_amp, &mode) < 0) {
> + pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: "");
> + return -EINVAL;
> + }
> +
> + if (has_amp) {
> + if (find_slab_cache(skel, base, &addr)) {
> + int fd = bpf_map__fd(skel->maps.slab_filter);
> +
> + entry->part = PBF_LOCK_SLAB | mode;
> + entry->value = addr;
> + bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
> + skel->bss->has_lock_slab = 1;
> + return 0;
> + }
> + pr_err("Error: Cannot resolve slab cache '%s'\n", name);
> + return -EINVAL;
> + }
> +
> + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
> + entry->part = PBF_LOCK_MMAP | mode;
> + entry->value = 0;
> + if (skel && (mode & PBF_LOCK_MODE_READ))
> + skel->bss->has_mmap_lock = 1;
> + return 0;
> + }
> +
> + if (!strcmp(base, "siglock")) {
> + if (!(mode & PBF_LOCK_MODE_WRITE)) {
> + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
> + name);
> + return -EINVAL;
> + }
> + entry->part = PBF_LOCK_SIGHAND | mode;
> + entry->value = 0;
> + if (skel)
> + skel->bss->has_siglock = 1;
> + return 0;
> + }
> +
> + if (!strcmp(base, "rq_lock")) {
> + if (!(mode & PBF_LOCK_MODE_WRITE)) {
> + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
> + name);
> + return -EINVAL;
> + }
> + entry->part = PBF_LOCK_CLASS | mode;
> + entry->value = LOCK_CLASS_RQLOCK;
> + if (skel)
> + skel->bss->has_lock_class = 1;
> + return 0;
> + }
> +
> + if (!strcmp(base, "zone_lock")) {
> + if (!(mode & PBF_LOCK_MODE_WRITE)) {
> + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
> + name);
> + return -EINVAL;
> + }
> + entry->part = PBF_LOCK_CLASS | mode;
> + entry->value = LOCK_CLASS_ZONE_LOCK;
> + if (skel)
> + skel->bss->has_lock_class = 1;
> + return 0;
> + }
> +
> + if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
> + entry->part = PBF_LOCK_TYPE | mode;
> + entry->value = encoded_pair;
> + return 0;
> + }
> +
> + if (parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) {
> + pr_err("Error: Lock type '%s' does not support the requested read/write mode\n",
> + name);
> + return -EINVAL;
> + }
> +
> + if (parse_lock_hex_addr(base, &addr)) {
> + entry->part = PBF_LOCK_ADDR | mode;
> + entry->value = addr;
> + if (skel) {
> + int fd = bpf_map__fd(skel->maps.lock_addr_filter);
> +
> + bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
> + skel->bss->has_lock_addr = 1;
> + }
> + return 0;
> + }
> +
> + ret = find_kernel_lock_sym(base, &addr);
> + if (ret == 0) {
> + unsigned int direct_flag = 0;
> +
> + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0)
> + direct_flag = PBF_LOCK_DIRECT_RTMUTEX;
> + else if (strtailcmp(base, "mutex") == 0)
> + direct_flag = PBF_LOCK_DIRECT_MUTEX;
> + else if (strtailcmp(base, "rwsem") == 0 &&
> + strtailcmp(base, "percpu_rwsem") != 0 &&
> + strtailcmp(base, "pcpu_rwsem") != 0)
> + direct_flag = PBF_LOCK_DIRECT_RWSEM;
> +
> + if ((direct_flag == PBF_LOCK_DIRECT_MUTEX ||
> + direct_flag == PBF_LOCK_DIRECT_RTMUTEX ||
> + strtailcmp(base, "percpu_rwsem") == 0 ||
> + strtailcmp(base, "pcpu_rwsem") == 0 ||
> + strtailcmp(base, "spinlock") == 0 ||
> + (strtailcmp(base, "sem") == 0 &&
> + strtailcmp(base, "rwsem") != 0)) &&
> + !(mode & PBF_LOCK_MODE_WRITE)) {
> + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
> + name);
> + return -EINVAL;
> + }
> +
> + entry->part = PBF_LOCK_ADDR | mode | direct_flag;
> + entry->value = addr;
> + if (skel && direct_flag != PBF_LOCK_DIRECT_MUTEX &&
> + direct_flag != PBF_LOCK_DIRECT_RTMUTEX) {
> + int fd = bpf_map__fd(skel->maps.lock_addr_filter);
> +
> + bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
> + skel->bss->has_lock_addr = 1;
> + }
> + return 0;
> + }
> + if (ret == -EACCES)
> + return ret;
> +
> + pr_err("Error: Cannot resolve kernel lock '%s'\n", name);
> + return -EINVAL;
> +}
> +
> +static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
> + struct perf_bpf_filter_entry *entry)
> {
> - int i = 0;
> + int i = 0, ret;
> struct perf_bpf_filter_expr *expr;
>
> list_for_each_entry(expr, &evsel->bpf_filters, list) {
> @@ -182,6 +1102,11 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
> entry[i].part = expr->part;
> entry[i].term = expr->term;
> entry[i].value = expr->val;
> + if (expr->term == PBF_TERM_LOCK_HELD) {
> + ret = resolve_lock_expr(skel, expr, &entry[i]);
> + if (ret < 0)
> + return ret;
> + }
> i++;
>
> if (expr->op == PBF_OP_GROUP_BEGIN) {
> @@ -195,6 +1120,11 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
> entry[i].part = group->part;
> entry[i].term = group->term;
> entry[i].value = group->val;
> + if (group->term == PBF_TERM_LOCK_HELD) {
> + ret = resolve_lock_expr(skel, group, &entry[i]);
> + if (ret < 0)
> + return ret;
> + }
> i++;
> }
>
> @@ -434,22 +1364,24 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> {
> int i, x, y, fd, ret;
> struct sample_filter_bpf *skel = NULL;
> + struct btf *btf = NULL;
> struct perf_bpf_filter_entry *entry;
> bool needs_idx_hash = !target__has_cpu(target);
> bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
> + bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false;
> + unsigned int prog_groups = 0, required_groups = 0;
>
> entry = calloc(MAX_FILTERS, sizeof(*entry));
> if (entry == NULL)
> return -1;
>
> - ret = get_filter_entries(evsel, entry);
> - if (ret < 0) {
> - pr_err("Failed to process filter entries\n");
> - goto err;
> - }
> + prescan_lock_filters(evsel, &has_lock_filter, &prog_groups,
> + &required_groups, &needs_lock_syms,
> + &needs_slab_iter);
>
> - if (is_tracepoint && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
> - pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n");
> + if ((has_lock_filter || is_tracepoint) && geteuid() != 0 &&
> + !perf_cap__capable(CAP_BPF)) {
> + pr_err("Error: BPF filter requires CAP_BPF capability\n");
> ret = -EPERM;
> goto err;
> }
> @@ -457,6 +1389,11 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
> int zero = 0;
>
> + ret = get_filter_entries(evsel, NULL, entry);
> + if (ret < 0) {
> + pr_err("Failed to process filter entries\n");
> + goto err;
> + }
> /* The filters map is shared among other processes */
> ret = create_idx_hash(evsel, entry);
> if (ret < 0)
> @@ -501,8 +1438,26 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> goto err;
> }
>
> - bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
> - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
> + if (has_lock_filter) {
> + btf = btf__load_vmlinux_btf();
> + bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES);
> + bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS);
> + if (needs_slab_iter) {
> + bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS);
> + bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES);
> + }
> + if (needs_lock_syms) {
> + bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
> + init_numa_data(skel, btf);
> + }
> + ret = configure_lock_progs(skel, btf, prog_groups, required_groups,
> + needs_lock_syms, needs_slab_iter,
> + is_tracepoint);
> + if (ret < 0)
> + goto err;
> + } else {
> + configure_lock_progs(skel, NULL, 0, 0, false, false, is_tracepoint);
> + }
>
> if (sample_filter_bpf__load(skel) < 0) {
> ret = -errno;
> @@ -510,6 +1465,37 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> goto err;
> }
>
> + if (has_lock_filter && needs_slab_iter)
> + run_slab_cache_iter(skel);
> +
> + ret = get_filter_entries(evsel, skel, entry);
> + if (ret < 0) {
> + pr_err("Failed to process filter entries\n");
> + goto err;
> + }
> +
> + if (has_lock_filter) {
> + if (needs_lock_syms && skel->bss->has_lock_class) {
> + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts,
> + .flags = BPF_F_TEST_RUN_ON_CPU,
> + );
> + int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms);
> +
> + bpf_prog_test_run_opts(prog_fd, &opts);
> + }
> + bpf_program__set_autoattach(skel->progs.perf_sample_filter, false);
> + bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false);
> + bpf_program__set_autoattach(skel->progs.collect_lock_syms, false);
> + bpf_program__set_autoattach(skel->progs.slab_cache_iter, false);
> +
> + ret = sample_filter_bpf__attach(skel);
> + if (ret < 0) {
> + pr_err("Failed to attach lock sample-filter BPF programs\n");
> + goto err;
> + }
> + skel->bss->lock_enabled = 1;
> + }
> +
> i = 0;
> fd = bpf_map__fd(skel->maps.filters);
>
> @@ -549,11 +1535,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> }
> }
> }
> + btf__free(btf);
> free(entry);
> evsel->bpf_skel = skel;
> return 0;
>
> err:
> + btf__free(btf);
> free(entry);
> if (!list_empty(&pinned_filters)) {
> struct pinned_filter_idx *pfi, *tmp;
> @@ -568,6 +1556,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> return ret;
> }
>
> +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr)
> +{
> + struct perf_bpf_filter_expr *group, *tmp;
> +
> + if (!expr)
> + return;
> +
> + list_for_each_entry_safe(group, tmp, &expr->groups, list) {
> + list_del(&group->list);
> + perf_bpf_filter_expr__delete(group);
> + }
> + zfree(&expr->lock_name);
> + free(expr);
> +}
> +
> int perf_bpf_filter__destroy(struct evsel *evsel)
> {
> struct perf_bpf_filter_expr *expr, *tmp;
> @@ -575,7 +1578,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel)
>
> list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) {
> list_del(&expr->list);
> - free(expr);
> + perf_bpf_filter_expr__delete(expr);
> }
> sample_filter_bpf__destroy(evsel->bpf_skel);
>
> @@ -627,7 +1630,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
> {
> struct perf_bpf_filter_expr *expr;
>
> - expr = malloc(sizeof(*expr));
> + expr = zalloc(sizeof(*expr));
> if (expr != NULL) {
> expr->term = term;
> expr->part = part;
> @@ -638,6 +1641,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
> return expr;
> }
>
> +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
> + int mode,
> + enum perf_bpf_filter_op op,
> + unsigned long val)
> +{
> + struct perf_bpf_filter_expr *expr;
> +
> + if (!lock_name || !*lock_name)
> + return NULL;
> +
> + expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val);
> + if (!expr)
> + return NULL;
> +
> + expr->lock_name = strdup(lock_name);
> + if (!expr->lock_name) {
> + free(expr);
> + return NULL;
> + }
> + return expr;
> +}
> +
> static bool check_bpf_filter_capable(void)
> {
> int fd;
> @@ -658,14 +1683,11 @@ static bool check_bpf_filter_capable(void)
> return false;
> }
>
> -int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
> +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
> {
> YY_BUFFER_STATE buffer;
> int ret;
>
> - if (!check_bpf_filter_capable())
> - return -EPERM;
> -
> buffer = perf_bpf_filter__scan_string(str);
>
> ret = perf_bpf_filter_parse(expr_head);
> @@ -677,6 +1699,14 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
> return ret;
> }
>
> +int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
> +{
> + if (!check_bpf_filter_capable())
> + return -EPERM;
> +
> + return __perf_bpf_filter__parse(expr_head, str);
> +}
> +
> static int pin_filter_object(struct sample_filter_bpf *skel, const char *path)
> {
> char *prog_path = NULL;
> @@ -729,8 +1759,9 @@ int perf_bpf_filter__pin(void)
> goto err;
> }
>
> + configure_lock_progs(skel, NULL, 0, 0, false, false, false);
> +
> /* pinned program will use pid-hash */
> - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
> bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
> bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH);
> bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH);
> @@ -825,7 +1856,7 @@ int perf_bpf_filter__unpin(void)
> goto err;
> }
>
> - bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
> + configure_lock_progs(skel, NULL, 0, 0, false, false, false);
> if (sample_filter_bpf__load(skel) < 0) {
> ret = -errno;
> pr_err("Failed to load perf sample-filter BPF skeleton\n");
> diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h
> index 818c554b91b2..fda95ff226b3 100644
> --- a/tools/perf/util/bpf-filter.h
> +++ b/tools/perf/util/bpf-filter.h
> @@ -14,6 +14,13 @@ struct perf_bpf_filter_expr {
> int part;
> enum perf_bpf_filter_term term;
> unsigned long val;
> + /*
> + * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions
> + * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R").
> + * Resolved into `entry.part` and `entry.target` during
> + * perf_bpf_filter__prepare().
> + */
> + char *lock_name;
> };
>
> struct evsel;
> @@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
> int part,
> enum perf_bpf_filter_op op,
> unsigned long val);
> +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
> + int mode,
> + enum perf_bpf_filter_op op,
> + unsigned long val);
> +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr);
> +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
> int perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
> int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target);
> int perf_bpf_filter__destroy(struct evsel *evsel);
> diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l
> index 1be9df6550fc..b4022110f1e5 100644
> --- a/tools/perf/util/bpf-filter.l
> +++ b/tools/perf/util/bpf-filter.l
> @@ -1,9 +1,12 @@
> %option prefix="perf_bpf_filter_"
> %option noyywrap
>
> +%x LOCK_ARG
> +
> %{
> #include <stdio.h>
> #include <stdlib.h>
> +#include <string.h>
> #include <linux/perf_event.h>
>
> #include "bpf-filter.h"
> @@ -76,7 +79,7 @@ static int path_or_error(void)
> num_dec [0-9]+
> num_hex 0[Xx][0-9a-fA-F]+
> space [ \t]+
> -path [^ \t\n,]+
> +path [^ \t\n,()!=>&|<]+
> ident [_a-zA-Z][_a-zA-Z0-9]+
>
> %%
> @@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); }
> uid { return sample(PBF_TERM_UID); }
> gid { return sample(PBF_TERM_GID); }
> cgroup { return sample_path(PBF_TERM_CGROUP); }
> -
> +lock_held[ \t]*"(" {
> + BEGIN(LOCK_ARG);
> + perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY;
> + return BFT_LOCK_HELD;
> + }
> +(lock_held_read|lock_held_shared)[ \t]*"(" {
> + BEGIN(LOCK_ARG);
> + perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ;
> + return BFT_LOCK_HELD;
> + }
> +(lock_held_write|lock_held_exclusive)[ \t]*"(" {
> + BEGIN(LOCK_ARG);
> + perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE;
> + return BFT_LOCK_HELD;
> + }
> +
> +<LOCK_ARG>{space} { }
> +<LOCK_ARG>[^ \t\n()]+ {
> + perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text);
> + return BFT_LOCK_NAME;
> + }
> +<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; }
> +<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; }
> +
> +"!" { return '!'; }
> "==" { return operator(PBF_OP_EQ); }
> "!=" { return operator(PBF_OP_NEQ); }
> ">" { return operator(PBF_OP_GT); }
> diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y
> index 5a79a8e7a45b..5ae05d4608c6 100644
> --- a/tools/perf/util/bpf-filter.y
> +++ b/tools/perf/util/bpf-filter.y
> @@ -8,9 +8,11 @@
> #endif
>
> #include <stdio.h>
> +#include <stdlib.h>
> #include <string.h>
> #include <linux/compiler.h>
> #include <linux/list.h>
> +#include <linux/zalloc.h>
> #include "bpf-filter.h"
> #include "cgroup.h"
>
> @@ -31,6 +33,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
> {
> unsigned long num;
> char *path;
> + char *lock_name;
> + struct {
> + char *name;
> + int mode;
> + } lock;
> struct {
> enum perf_bpf_filter_term term;
> int part;
> @@ -40,12 +47,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
> }
>
> %token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH
> +%token BFT_LOCK_HELD BFT_LOCK_NAME
> %type <expr> filter_term filter_expr
> -%destructor { free ($$); } <expr>
> +%destructor { perf_bpf_filter_expr__delete($$); } <expr>
> +%destructor { free($$); } <lock_name>
> +%destructor { free($$.name); } <lock>
> %type <sample> BFT_SAMPLE BFT_SAMPLE_PATH
> %type <op> BFT_OP
> -%type <num> BFT_NUM
> +%type <num> BFT_NUM BFT_LOCK_HELD
> %type <path> BFT_PATH
> +%type <lock_name> BFT_LOCK_NAME
> +%type <lock> lock_term
>
> %%
>
> @@ -105,5 +117,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH
> $$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id);
> cgroup__put(cgrp);
> }
> +|
> +lock_term
> +{
> + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1);
> + zfree(&$1.name);
> + if (!$$)
> + YYERROR;
> +}
> +|
> +'!' lock_term
> +{
> + $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0);
> + zfree(&$2.name);
> + if (!$$)
> + YYERROR;
> +}
> +|
> +lock_term BFT_OP BFT_NUM
> +{
> + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3);
> + zfree(&$1.name);
> + if (!$$)
> + YYERROR;
> +}
> +
> +lock_term:
> +BFT_LOCK_HELD BFT_LOCK_NAME ')'
> +{
> + $$.name = $2;
> + $$.mode = $1;
> +}
>
> %%
> diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
> index 683fec85e71e..a65ee780d64d 100644
> --- a/tools/perf/util/bpf_skel/sample-filter.h
> +++ b/tools/perf/util/bpf_skel/sample-filter.h
> @@ -53,8 +53,74 @@ enum perf_bpf_filter_term {
> /* Terms computed from BPF helpers. */
> PBF_TERM_UID,
> PBF_TERM_GID,
> + /*
> + * Kernel lock held predicate (`lock_held(...)`). Evaluates whether the
> + * specified kernel lock is currently held by the running task/CPU.
> + * The lock specification kind is stored in `entry.part`
> + * (`enum perf_bpf_lock_target_kind`), `entry.op` is normalized to
> + * `PBF_OP_EQ` (match when held) or `PBF_OP_NEQ` (match when not held),
> + * and the resolved target address, class ID, or type flag encoding is
> + * stored in `entry.value`.
> + */
> + PBF_TERM_LOCK_HELD,
> };
>
> +/*
> + * Lock target categories for PBF_TERM_LOCK_HELD filters.
> + * Stored in `perf_bpf_filter_entry.part`.
> + */
> +enum perf_bpf_lock_target_kind {
> + PBF_LOCK_NONE = 0,
> + /*
> + * Specific global or static lock address (resolved from kallsyms or
> + * given as a hex address). `entry.value` holds the kernel address.
> + */
> + PBF_LOCK_ADDR,
> + /*
> + * Dynamic lock allocated inside a slab cache (`&slab_name`).
> + * `entry.value` holds the kernel `struct kmem_cache *` address.
> + */
> + PBF_LOCK_SLAB,
> + /*
> + * Per-process `current->mm->mmap_lock` (or `mmap_sem`).
> + */
> + PBF_LOCK_MMAP,
> + /*
> + * Per-process `current->sighand->siglock`.
> + */
> + PBF_LOCK_SIGHAND,
> + /*
> + * Semi-global lock class (`rq_lock` or `zone_lock`).
> + * `entry.value` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or
> + * LOCK_CLASS_ZONE_LOCK).
> + */
> + PBF_LOCK_CLASS,
> + /*
> + * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.).
> + * `entry.value` packs up to two `LCB_F_*` flag combinations in the
> + * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
> + */
> + PBF_LOCK_TYPE,
> +};
> +
> +/*
> + * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD:
> + * Bits 0..3: `enum perf_bpf_lock_target_kind`
> + * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding)
> + * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding)
> + * Bit 6: PBF_LOCK_DIRECT_MUTEX (inspect struct mutex owner)
> + * Bit 7: PBF_LOCK_DIRECT_RWSEM (inspect struct rw_semaphore writer owner)
> + * Bit 8: PBF_LOCK_DIRECT_RTMUTEX (inspect struct rt_mutex_base / rt_mutex owner)
> + */
> +#define PBF_LOCK_KIND_MASK 0x0fU
> +#define PBF_LOCK_MODE_READ (1U << 4)
> +#define PBF_LOCK_MODE_WRITE (1U << 5)
> +#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE)
> +#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY
> +#define PBF_LOCK_DIRECT_MUTEX (1U << 6)
> +#define PBF_LOCK_DIRECT_RWSEM (1U << 7)
> +#define PBF_LOCK_DIRECT_RTMUTEX (1U << 8)
> +
> /* BPF map entry for filtering */
> struct perf_bpf_filter_entry {
> enum perf_bpf_filter_op op;
> diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
> index a190374ed87a..d1192a93179e 100644
> --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
> +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
> @@ -5,8 +5,47 @@
> #include <bpf/bpf_tracing.h>
> #include <bpf/bpf_core_read.h>
>
> +#include "lock_data.h"
> #include "sample-filter.h"
>
> +/*
> + * Key for the `held_locks` BPF hash map tracking currently held kernel locks.
> + *
> + * - `addr_or_id`:
> + * - PBF_LOCK_ADDR: kernel virtual address of the lock instance
> + * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK)
> + * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address
> + * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags
> + * - `owner_id`:
> + * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`),
> + * since the holder may sleep and migrate across CPUs while holding the lock.
> + * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks
> + * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption
> + * is disabled on the holding CPU and spinlocks such as `rq_lock` may be
> + * acquired by one task and released by the next task across `context_switch()`.
> + * - `kind`: `enum perf_bpf_lock_target_kind`.
> + */
> +struct held_lock_key {
> + __u64 addr_or_id;
> + __u32 owner_id;
> + __u32 kind;
> +};
> +
> +/*
> + * Value for `held_locks` entries.
> + * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`)
> + * and exclusive/write acquisitions (including nested class/slab/type
> + * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of
> + * the lock so release tracepoints that do not carry flags (such as
> + * `lock:contended_release`) can decrement matching class/slab/type entries.
> + */
> +struct held_lock_val {
> + __u32 read_count;
> + __u32 write_count;
> + __u32 flags;
> + __u32 pad;
> +};
> +
> /* BPF map that will be filled by user space */
> struct filters {
> __uint(type, BPF_MAP_TYPE_HASH);
> @@ -43,10 +82,225 @@ struct lost_count {
> __uint(max_entries, 1);
> } dropped SEC(".maps");
>
> +/*
> + * Currently held kernel locks tracked by fentry/fexit hooks.
> + * Uses BPF_MAP_TYPE_LRU_HASH so that if a counting semaphore (down/up) or
> + * non-owner rwsem is acquired in one task and released in another task or
> + * interrupt handler (cross-context signaling), any orphaned entry is
> + * automatically evicted rather than exhausting the map.
> + */
> +struct {
> + __uint(type, BPF_MAP_TYPE_LRU_HASH);
> + __type(key, struct held_lock_key);
> + __type(value, struct held_lock_val);
> + __uint(max_entries, 1);
> +} held_locks SEC(".maps");
> +
> +/* Target lock addresses for PBF_LOCK_ADDR filters. */
> +struct {
> + __uint(type, BPF_MAP_TYPE_HASH);
> + __type(key, __u64);
> + __type(value, __u8);
> + __uint(max_entries, 1);
> +} lock_addr_filter SEC(".maps");
> +
> +struct {
> + __uint(type, BPF_MAP_TYPE_HASH);
> + __uint(key_size, sizeof(__u64));
> + __uint(value_size, sizeof(__u32));
> + __uint(max_entries, 1);
> +} lock_syms SEC(".maps");
> +
> +struct {
> + __uint(type, BPF_MAP_TYPE_HASH);
> + __uint(key_size, sizeof(__u64));
> + __uint(value_size, sizeof(__u8));
> + __uint(max_entries, 1);
> +} slab_filter SEC(".maps");
> +
> +struct {
> + __uint(type, BPF_MAP_TYPE_HASH);
> + __uint(key_size, sizeof(__u64));
> + __uint(value_size, sizeof(struct slab_cache_data));
> + __uint(max_entries, 1);
> +} slab_caches SEC(".maps");
> +
> volatile const int use_idx_hash;
>
> +const volatile __u64 contig_page_data_addr;
> +const volatile __u64 node_data_addr;
> +const volatile int nr_nodes;
> +const volatile int sizeof_zone;
> +
> +__u32 slab_cache_id;
> +
> +/* Control flags set by userspace when lock_held(...) filters are active. */
> +int lock_enabled;
> +int has_lock_addr;
> +int has_lock_slab;
> +int has_lock_class;
> +int has_lock_type;
> +int has_mmap_lock;
> +int has_siglock;
> +int track_rwsem_read;
> +__u8 lock_type_filter[LCB_F_TYPE_MAX];
> +
> void *bpf_cast_to_kern_ctx(void *) __ksym;
>
> +/*
> + * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`.
> + *
> + * For locks that maintain a task_struct pointer in their `owner` field
> + * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`),
> + * `perf_sample_filter` can inspect the lock structure directly in kernel memory
> + * and compare the owner against `current` (`bpf_get_current_task()`).
> + *
> + * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear
> + * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and
> + * spinlocks are checked via the `held_locks` BPF map instead.
> + */
> +static __always_inline bool is_rwsem_writer_owner(__u64 lock_addr, __u64 curr)
> +{
> + struct rw_semaphore___new *rwsem = (void *)lock_addr;
> +
> + if (bpf_core_field_exists(rwsem->count.counter) &&
> + !(BPF_CORE_READ(rwsem, count.counter) & 1UL))
> + return false;
> + if (bpf_core_field_exists(rwsem->owner.counter) &&
> + (BPF_CORE_READ(rwsem, owner.counter) & 1UL))
> + return false;
> +
> + return (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr;
> +}
> +
> +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __u32 part)
> +{
> + if (!curr)
> + return false;
> +
> + if ((part & PBF_LOCK_DIRECT_MUTEX) &&
> + (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr)
> + return true;
> + if ((part & PBF_LOCK_DIRECT_RWSEM) &&
> + is_rwsem_writer_owner(lock_addr, curr))
> + return true;
> + if ((part & PBF_LOCK_DIRECT_RTMUTEX) &&
> + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr)
> + return true;
> + return false;
> +}
> +
> +static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode)
> +{
> + if (!val)
> + return false;
> + if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0)
> + return true;
> + if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0)
> + return true;
> + return false;
> +}
> +
> +static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind,
> + __u32 mode, __u32 pid, __u32 cpu_id)
> +{
> + struct held_lock_key key = {
> + .addr_or_id = addr_or_id,
> + .owner_id = pid,
> + .kind = kind,
> + };
> + struct held_lock_val *val;
> +
> + if (pid) {
> + val = bpf_map_lookup_elem(&held_locks, &key);
> + if (match_held_val(val, mode))
> + return 1;
> + }
> +
> + key.owner_id = cpu_id;
> + val = bpf_map_lookup_elem(&held_locks, &key);
> + if (match_held_val(val, mode))
> + return 1;
> +
> + return 0;
> +}
> +
> +static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode,
> + __u32 pid, __u32 cpu_id)
> +{
> + __u32 flag1 = target & 0xff;
> + __u32 flag2 = (target >> 8) & 0xff;
> +
> + if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id))
> + return 1;
> + if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id))
> + return 1;
> + return 0;
> +}
> +
> +static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry)
> +{
> + __u64 curr = bpf_get_current_task();
> + __u32 pid = (__u32)bpf_get_current_pid_tgid();
> + __u32 cpu_id = 0x80000000U | bpf_get_smp_processor_id();
> + __u32 kind = entry->part & PBF_LOCK_KIND_MASK;
> + __u32 mode = entry->part & PBF_LOCK_MODE_MASK;
> + __u64 lock_addr;
> +
> + if (!mode)
> + mode = PBF_LOCK_MODE_ANY;
> +
> + switch (kind) {
> + case PBF_LOCK_ADDR:
> + lock_addr = entry->value;
> + if ((mode & PBF_LOCK_MODE_WRITE) &&
> + is_direct_lock_owner(lock_addr, curr, entry->part))
> + return true;
> + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0;
> +
> + case PBF_LOCK_MMAP:
> + lock_addr = get_current_mmap_lock(curr);
> + if (!lock_addr)
> + return false;
> + if ((mode & PBF_LOCK_MODE_WRITE) &&
> + is_rwsem_writer_owner(lock_addr, curr))
> + return true;
> + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0;
> +
> + case PBF_LOCK_SIGHAND:
> + lock_addr = get_current_siglock(curr);
> + if (!lock_addr)
> + return false;
> + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0;
> +
> + case PBF_LOCK_CLASS:
> + return is_lock_in_map(entry->value, PBF_LOCK_CLASS, mode, pid, cpu_id) != 0;
> +
> + case PBF_LOCK_SLAB:
> + return is_lock_in_map(entry->value, PBF_LOCK_SLAB, mode, pid, cpu_id) != 0;
> +
> + case PBF_LOCK_TYPE:
> + return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0;
> +
> + default:
> + return false;
> + }
> +}
> +
> +static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry)
> +{
> + /*
> + * Pinned filters (use_idx_hash != 0) share a world-writable filters
> + * map and do not support lock_held(). Reject evaluation when pinned
> + * or when lock_enabled is not set so unprivileged users cannot probe
> + * arbitrary kernel addresses via entry->value.
> + */
> + if (use_idx_hash || !lock_enabled)
> + return entry->op == PBF_OP_EQ ? ~entry->value : entry->value;
> +
> + return is_lock_held(entry) ? entry->value : ~entry->value;
> +}
> +
> /* new kernel perf_sample_data definition */
> struct perf_sample_data___new {
> __u64 sample_flags;
> @@ -175,6 +429,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
> return bpf_get_current_uid_gid() & 0xFFFFFFFF;
> case PBF_TERM_GID:
> return bpf_get_current_uid_gid() >> 32;
> + case PBF_TERM_LOCK_HELD:
> + return check_lock_held(entry);
> case PBF_TERM_NONE:
> case __PBF_UNUSED_TERM4:
> case __PBF_UNUSED_TERM5:
> @@ -209,6 +465,8 @@ static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry
> return bpf_get_current_uid_gid() & 0xFFFFFFFF;
> case PBF_TERM_GID:
> return bpf_get_current_uid_gid() >> 32;
> + case PBF_TERM_LOCK_HELD:
> + return check_lock_held(entry);
> default:
> return 0;
> }
> @@ -390,4 +648,760 @@ int perf_tp_sample_filter(void *ctx)
> return 0;
> }
>
> +static __always_inline __u32 get_lock_owner_id(__u32 flags)
> +{
> + __u32 owner_id;
> +
> + if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN)
> + return 0x80000000U | bpf_get_smp_processor_id();
> +
> + owner_id = (__u32)bpf_get_current_pid_tgid();
> + if (!owner_id)
> + owner_id = 0x80000000U | bpf_get_smp_processor_id();
> + return owner_id;
> +}
> +
> +static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id,
> + __u32 kind, __u32 flags, bool acquired)
> +{
> + struct held_lock_key key = {
> + .addr_or_id = addr_or_id,
> + .owner_id = owner_id,
> + .kind = kind,
> + };
> + struct held_lock_val *val;
> + bool is_read = (flags & LCB_F_READ) != 0;
> +
> + val = bpf_map_lookup_elem(&held_locks, &key);
> + if (acquired) {
> + if (val) {
> + if (is_read)
> + __sync_fetch_and_add(&val->read_count, 1);
> + else
> + __sync_fetch_and_add(&val->write_count, 1);
> + } else {
> + struct held_lock_val first = {
> + .read_count = is_read ? 1 : 0,
> + .write_count = is_read ? 0 : 1,
> + .flags = flags,
> + };
> +
> + bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY);
> + }
> + } else if (val) {
> + if (is_read) {
> + if (val->read_count <= 1 && val->write_count == 0)
> + bpf_map_delete_elem(&held_locks, &key);
> + else if (val->read_count > 0)
> + __sync_fetch_and_add(&val->read_count, -1);
> + } else {
> + if (val->write_count <= 1 && val->read_count == 0)
> + bpf_map_delete_elem(&held_locks, &key);
> + else if (val->write_count > 0)
> + __sync_fetch_and_add(&val->write_count, -1);
> + }
> + }
> +}
> +
> +#define LCB_F_READ_IRQSAVE (1U << 30)
> +#define LCB_F_READ_IRQ_BH (1U << 31)
> +
> +static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id)
> +{
> + struct held_lock_key addr_key = {
> + .addr_or_id = lock_addr,
> + .owner_id = owner_id,
> + .kind = PBF_LOCK_ADDR,
> + };
> + struct held_lock_val *existing;
> + bool matched = false;
> + bool is_read = (flags & LCB_F_READ) != 0;
> + __u32 class_id = 0;
> + __u64 kmem_cache_addr = 0;
> + __u32 type_flag = flags & LCB_F_TYPE_MASK;
> + bool match_type = false;
> +
> + if (!lock_enabled || !lock_addr)
> + return 0;
> +
> + if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
> + matched = true;
> +
> + if (has_mmap_lock || has_siglock) {
> + __u32 ltype = check_lock_type(lock_addr, type_flag);
> +
> + if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
> + (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
> + matched = true;
> + }
> +
> + if (has_lock_class && type_flag == LCB_F_SPIN) {
> + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
> +
> + if (cid) {
> + class_id = *cid;
> + matched = true;
> + }
> + }
> +
> + if (has_lock_slab && bpf_get_kmem_cache) {
> + kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
> + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr))
> + matched = true;
> + else
> + kmem_cache_addr = 0;
> + }
> +
> + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) {
> + match_type = true;
> + matched = true;
> + }
> +
> + if (!matched)
> + return 0;
> +
> + existing = bpf_map_lookup_elem(&held_locks, &addr_key);
> + if (existing) {
> + /*
> + * On CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC
> + * kernels, _raw_read_lock_irq() and _raw_read_lock_bh() call
> + * the out-of-line _raw_read_lock_irqsave(). Clear the marker
> + * and skip the duplicate outer fexit increment.
> + */
> + if ((flags & LCB_F_READ_IRQ_BH) &&
> + (existing->flags & LCB_F_READ_IRQSAVE)) {
> + existing->flags &= ~LCB_F_READ_IRQSAVE;
> + return 0;
> + }
> + /* Non-reader locks are not recursive; avoid double-counting */
> + if (!is_read && existing->write_count > 0)
> + return 0;
> + if (is_read) {
> + __sync_fetch_and_add(&existing->read_count, 1);
> + if (flags & LCB_F_READ_IRQSAVE)
> + existing->flags |= LCB_F_READ_IRQSAVE;
> + else
> + existing->flags &= ~LCB_F_READ_IRQSAVE;
> + } else {
> + __sync_fetch_and_add(&existing->write_count, 1);
> + }
> + } else {
> + struct held_lock_val first = {
> + .read_count = is_read ? 1 : 0,
> + .write_count = is_read ? 0 : 1,
> + .flags = flags & (LCB_F_TYPE_MASK | LCB_F_READ_IRQSAVE),
> + };
> +
> + if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0)
> + return 0;
> + }
> +
> + if (class_id)
> + update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, type_flag, true);
> + if (kmem_cache_addr) {
> + update_held_lock_count((__u64)kmem_cache_addr, owner_id,
> + PBF_LOCK_SLAB, type_flag, true);
> + }
> + if (match_type)
> + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true);
> +
> + return 0;
> +}
> +
> +static __always_inline int record_lock_acquired(void *ctx, __u32 flags)
> +{
> + return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags));
> +}
> +
> +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags)
> +{
> + __u32 owner_id = get_lock_owner_id(flags);
> + struct held_lock_key addr_key = {
> + .addr_or_id = lock_addr,
> + .owner_id = owner_id,
> + .kind = PBF_LOCK_ADDR,
> + };
> + struct held_lock_val *existing;
> + __u32 type_flag = flags & LCB_F_TYPE_MASK;
> + bool is_read = (flags & LCB_F_READ) != 0;
> +
> + if (!lock_enabled || !lock_addr)
> + return 0;
> +
> + existing = bpf_map_lookup_elem(&held_locks, &addr_key);
> + if (!existing)
> + return 0;
> +
> + if (is_read) {
> + if (existing->read_count == 0)
> + return 0;
> + if (existing->read_count == 1 && existing->write_count == 0) {
> + bpf_map_delete_elem(&held_locks, &addr_key);
> + } else {
> + existing->flags &= ~LCB_F_READ_IRQSAVE;
> + __sync_fetch_and_add(&existing->read_count, -1);
> + }
> + } else {
> + if (existing->write_count == 0)
> + return 0;
> + if (existing->read_count == 0)
> + bpf_map_delete_elem(&held_locks, &addr_key);
> + else
> + existing->write_count = 0;
> + }
> +
> + if (has_lock_class && flags == LCB_F_SPIN) {
> + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
> +
> + if (cid)
> + update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false);
> + }
> +
> + if (has_lock_slab && bpf_get_kmem_cache) {
> + __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
> +
> + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) {
> + update_held_lock_count(kmem_cache_addr, owner_id,
> + PBF_LOCK_SLAB, flags, false);
> + }
> + }
> +
> + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag])
> + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false);
> +
> + return 0;
> +}
> +
> +static __always_inline int record_lock_released(void *ctx, __u32 flags)
> +{
> + return record_lock_released_addr(get_lock_addr(ctx), flags);
> +}
> +
> +/* Mutex acquire/release hooks */
> +SEC("fexit/mutex_lock")
> +int BPF_PROG(fexit_mutex_lock, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_MUTEX);
> +}
> +
> +SEC("fexit/mutex_lock_nested")
> +int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass)
> +{
> + return record_lock_acquired(ctx, LCB_F_MUTEX);
> +}
> +
> +SEC("fexit/_mutex_lock_nest_lock")
> +int BPF_PROG(fexit__mutex_lock_nest_lock, void *lock, void *nest_lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_MUTEX);
> +}
> +
> +SEC("fexit/mutex_lock_io")
> +int BPF_PROG(fexit_mutex_lock_io, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_MUTEX);
> +}
> +
> +SEC("fexit/mutex_lock_io_nested")
> +int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass)
> +{
> + return record_lock_acquired(ctx, LCB_F_MUTEX);
> +}
> +
> +SEC("fexit/mutex_lock_interruptible")
> +int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, LCB_F_MUTEX);
> + return 0;
> +}
> +
> +SEC("fexit/mutex_lock_interruptible_nested")
> +int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, LCB_F_MUTEX);
> + return 0;
> +}
> +
> +SEC("fexit/mutex_lock_killable")
> +int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, LCB_F_MUTEX);
> + return 0;
> +}
> +
> +SEC("fexit/_mutex_lock_killable")
> +int BPF_PROG(fexit__mutex_lock_killable, void *lock, unsigned int subclass,
> + void *nest_lock, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, LCB_F_MUTEX);
> + return 0;
> +}
> +
> +SEC("fexit/mutex_trylock")
> +int BPF_PROG(fexit_mutex_trylock, void *lock, int ret)
> +{
> + if (ret)
> + record_lock_acquired(ctx, LCB_F_MUTEX);
> + return 0;
> +}
> +
> +SEC("fexit/_mutex_trylock_nest_lock")
> +int BPF_PROG(fexit__mutex_trylock_nest_lock, void *lock, void *nest_lock, int ret)
> +{
> + if (ret)
> + record_lock_acquired(ctx, LCB_F_MUTEX);
> + return 0;
> +}
> +
> +SEC("fentry/mutex_unlock")
> +int BPF_PROG(fentry_mutex_unlock, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_MUTEX);
> +}
> +
> +/* RW semaphore acquire/release hooks */
> +SEC("fexit/down_read")
> +int BPF_PROG(fexit_down_read, void *sem)
> +{
> + return record_lock_acquired(ctx, LCB_F_READ);
> +}
> +
> +SEC("fexit/down_read_nested")
> +int BPF_PROG(fexit_down_read_nested, void *sem, int subclass)
> +{
> + return record_lock_acquired(ctx, LCB_F_READ);
> +}
> +
> +SEC("fexit/down_read_interruptible")
> +int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, LCB_F_READ);
> + return 0;
> +}
> +
> +SEC("fexit/down_read_killable")
> +int BPF_PROG(fexit_down_read_killable, void *sem, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, LCB_F_READ);
> + return 0;
> +}
> +
> +SEC("fexit/down_read_killable_nested")
> +int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, LCB_F_READ);
> + return 0;
> +}
> +
> +SEC("fexit/down_read_trylock")
> +int BPF_PROG(fexit_down_read_trylock, void *sem, int ret)
> +{
> + if (ret)
> + record_lock_acquired(ctx, LCB_F_READ);
> + return 0;
> +}
> +
> +SEC("fexit/down_write")
> +int BPF_PROG(fexit_down_write, void *sem)
> +{
> + return record_lock_acquired(ctx, LCB_F_WRITE);
> +}
> +
> +SEC("fexit/down_write_nested")
> +int BPF_PROG(fexit_down_write_nested, void *sem, int subclass)
> +{
> + return record_lock_acquired(ctx, LCB_F_WRITE);
> +}
> +
> +SEC("fexit/_down_write_nest_lock")
> +int BPF_PROG(fexit__down_write_nest_lock, void *sem, void *nest_lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_WRITE);
> +}
> +
> +SEC("fexit/down_write_killable")
> +int BPF_PROG(fexit_down_write_killable, void *sem, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, LCB_F_WRITE);
> + return 0;
> +}
> +
> +SEC("fexit/down_write_killable_nested")
> +int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, LCB_F_WRITE);
> + return 0;
> +}
> +
> +SEC("fexit/down_write_trylock")
> +int BPF_PROG(fexit_down_write_trylock, void *sem, int ret)
> +{
> + if (ret)
> + record_lock_acquired(ctx, LCB_F_WRITE);
> + return 0;
> +}
> +
> +SEC("fexit/downgrade_write")
> +int BPF_PROG(fexit_downgrade_write, void *sem)
> +{
> + if (track_rwsem_read)
> + record_lock_acquired(ctx, LCB_F_READ);
> + return record_lock_released(ctx, LCB_F_WRITE);
> +}
> +
> +SEC("fentry/up_read")
> +int BPF_PROG(fentry_up_read, void *sem)
> +{
> + return record_lock_released(ctx, LCB_F_READ);
> +}
> +
> +SEC("fentry/up_write")
> +int BPF_PROG(fentry_up_write, void *sem)
> +{
> + return record_lock_released(ctx, LCB_F_WRITE);
> +}
> +
> +/* Spinlock acquire/release hooks */
> +SEC("fexit/_raw_spin_lock")
> +int BPF_PROG(fexit_raw_spin_lock, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN);
> +}
> +
> +SEC("fexit/_raw_spin_lock_nested")
> +int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN);
> +}
> +
> +SEC("fexit/_raw_spin_lock_nest_lock")
> +int BPF_PROG(fexit_raw_spin_lock_nest_lock, void *lock, void *nest_lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN);
> +}
> +
> +SEC("fexit/_raw_spin_lock_irq")
> +int BPF_PROG(fexit_raw_spin_lock_irq, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN);
> +}
> +
> +SEC("fexit/_raw_spin_lock_irq_disable")
> +int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN);
> +}
> +
> +SEC("fexit/_raw_spin_lock_irqsave")
> +int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN);
> +}
> +
> +SEC("fexit/_raw_spin_lock_irqsave_nested")
> +int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN);
> +}
> +
> +SEC("fexit/_raw_spin_lock_bh")
> +int BPF_PROG(fexit_raw_spin_lock_bh, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN);
> +}
> +
> +SEC("fexit/_raw_spin_trylock")
> +int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret)
> +{
> + if (ret)
> + record_lock_acquired(ctx, LCB_F_SPIN);
> + return 0;
> +}
> +
> +SEC("fexit/_raw_spin_trylock_bh")
> +int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret)
> +{
> + if (ret)
> + record_lock_acquired(ctx, LCB_F_SPIN);
> + return 0;
> +}
> +
> +SEC("fentry/_raw_spin_unlock")
> +int BPF_PROG(fentry_raw_spin_unlock, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_SPIN);
> +}
> +
> +SEC("fentry/_raw_spin_unlock_irq")
> +int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_SPIN);
> +}
> +
> +SEC("fentry/_raw_spin_unlock_irq_enable")
> +int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_SPIN);
> +}
> +
> +SEC("fentry/_raw_spin_unlock_irqrestore")
> +int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_SPIN);
> +}
> +
> +SEC("fentry/_raw_spin_unlock_bh")
> +int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_SPIN);
> +}
> +
> +/* RWlock acquire/release hooks */
> +SEC("fexit/_raw_read_lock")
> +int BPF_PROG(fexit_raw_read_lock, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
> +}
> +
> +SEC("fexit/_raw_read_lock_irq")
> +int BPF_PROG(fexit_raw_read_lock_irq, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH);
> +}
> +
> +SEC("fexit/_raw_read_lock_irqsave")
> +int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQSAVE);
> +}
> +
> +SEC("fexit/_raw_read_lock_bh")
> +int BPF_PROG(fexit_raw_read_lock_bh, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH);
> +}
> +
> +SEC("fexit/_raw_read_trylock")
> +int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret)
> +{
> + if (ret)
> + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
> + return 0;
> +}
> +
> +SEC("fentry/_raw_read_unlock")
> +int BPF_PROG(fentry_raw_read_unlock, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
> +}
> +
> +SEC("fentry/_raw_read_unlock_irq")
> +int BPF_PROG(fentry_raw_read_unlock_irq, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
> +}
> +
> +SEC("fentry/_raw_read_unlock_irqrestore")
> +int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
> +}
> +
> +SEC("fentry/_raw_read_unlock_bh")
> +int BPF_PROG(fentry_raw_read_unlock_bh, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
> +}
> +
> +SEC("fexit/_raw_write_lock")
> +int BPF_PROG(fexit_raw_write_lock, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
> +}
> +
> +SEC("fexit/_raw_write_lock_nested")
> +int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
> +}
> +
> +SEC("fexit/_raw_write_lock_irq")
> +int BPF_PROG(fexit_raw_write_lock_irq, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
> +}
> +
> +SEC("fexit/_raw_write_lock_irqsave")
> +int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
> +}
> +
> +SEC("fexit/_raw_write_lock_bh")
> +int BPF_PROG(fexit_raw_write_lock_bh, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
> +}
> +
> +SEC("fexit/_raw_write_trylock")
> +int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret)
> +{
> + if (ret)
> + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
> + return 0;
> +}
> +
> +SEC("fentry/_raw_write_unlock")
> +int BPF_PROG(fentry_raw_write_unlock, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
> +}
> +
> +SEC("fentry/_raw_write_unlock_irq")
> +int BPF_PROG(fentry_raw_write_unlock_irq, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
> +}
> +
> +SEC("fentry/_raw_write_unlock_irqrestore")
> +int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
> +}
> +
> +SEC("fentry/_raw_write_unlock_bh")
> +int BPF_PROG(fentry_raw_write_unlock_bh, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
> +}
> +
> +/* RT-Mutex acquire/release hooks */
> +SEC("fexit/rt_mutex_lock")
> +int BPF_PROG(fexit_rt_mutex_lock, void *lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_RT);
> +}
> +
> +SEC("fexit/rt_mutex_lock_nested")
> +int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass)
> +{
> + return record_lock_acquired(ctx, LCB_F_RT);
> +}
> +
> +SEC("fexit/_rt_mutex_lock_nest_lock")
> +int BPF_PROG(fexit__rt_mutex_lock_nest_lock, void *lock, void *nest_lock)
> +{
> + return record_lock_acquired(ctx, LCB_F_RT);
> +}
> +
> +SEC("fexit/rt_mutex_lock_interruptible")
> +int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, LCB_F_RT);
> + return 0;
> +}
> +
> +SEC("fexit/rt_mutex_lock_killable")
> +int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, LCB_F_RT);
> + return 0;
> +}
> +
> +SEC("fexit/rt_mutex_trylock")
> +int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret)
> +{
> + if (ret)
> + record_lock_acquired(ctx, LCB_F_RT);
> + return 0;
> +}
> +
> +SEC("fentry/rt_mutex_unlock")
> +int BPF_PROG(fentry_rt_mutex_unlock, void *lock)
> +{
> + return record_lock_released(ctx, LCB_F_RT);
> +}
> +
> +/* Semaphore acquire/release hooks */
> +SEC("fexit/down")
> +int BPF_PROG(fexit_down, void *sem)
> +{
> + return record_lock_acquired(ctx, 0);
> +}
> +
> +SEC("fexit/down_interruptible")
> +int BPF_PROG(fexit_down_interruptible, void *sem, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, 0);
> + return 0;
> +}
> +
> +SEC("fexit/down_killable")
> +int BPF_PROG(fexit_down_killable, void *sem, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, 0);
> + return 0;
> +}
> +
> +SEC("fexit/down_timeout")
> +int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, 0);
> + return 0;
> +}
> +
> +SEC("fexit/down_trylock")
> +int BPF_PROG(fexit_down_trylock, void *sem, int ret)
> +{
> + if (ret == 0)
> + record_lock_acquired(ctx, 0);
> + return 0;
> +}
> +
> +SEC("fentry/up")
> +int BPF_PROG(fentry_up, void *sem)
> +{
> + return record_lock_released(ctx, 0);
> +}
> +
> +/* Percpu-rwsem write acquire/release hooks */
> +SEC("fexit/percpu_down_write")
> +int BPF_PROG(fexit_percpu_down_write, void *sem)
> +{
> + return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE);
> +}
> +
> +SEC("fentry/percpu_up_write")
> +int BPF_PROG(fentry_percpu_up_write, void *sem)
> +{
> + return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
> +}
> +
> +SEC("raw_tp/bpf_test_finish")
> +int BPF_PROG(collect_lock_syms)
> +{
> + return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
> + node_data_addr, nr_nodes, sizeof_zone);
> +}
> +
> +SEC("iter/kmem_cache")
> +int slab_cache_iter(void *ctx)
> +{
> + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
> +}
> +
> char LICENSE[] SEC("license") = "Dual BSD/GPL";
> diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
> index 2f3317ce17f0..1fcb424b75c0 100644
> --- a/tools/perf/util/parse-events.c
> +++ b/tools/perf/util/parse-events.c
> @@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist,
> /* Will a tracepoint filter work for str or should a BPF filter be used? */
> static bool is_possible_tp_filter(const char *str)
> {
> - return strstr(str, "uid") == NULL;
> + return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL &&
> + strstr(str, "lock_held") == NULL;
> }
>
> static int set_filter(struct evsel *evsel, const void *arg)
> --
> 2.56.0.212.g129ede20d9-goog
>
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v5 3/8] perf bpf-filter: Support BPF sample filters on tracepoint events
2026-10-20 20:58 ` [PATCH v5 3/8] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
@ 2026-10-20 22:57 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-20 22:57 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf bpf-filter: tracepoint BPF filter terms bypassed in is_possible_tp_filter()
--
commit af3470a1e411bc452a8bb18ee78fdec67327457c
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Support BPF sample filters on tracepoint events
This commit adds a tracepoint bpf program (perf_tp_sample_filter) to
evaluate context-independent sample filter terms (uid, gid, tid, cpu,
time, and cgroup) on tracepoint events using bpf helpers. It also updates
perf_bpf_filter__prepare(), perf_bpf_filter__pin(), and
perf_bpf_filter__unpin() to handle loading and pinning with the new
tracepoint program.
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 1596b08fe4e6a..ac0e7cf40f3af 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
> @@ -123,15 +123,26 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
> {
> const struct perf_sample_info *info;
>
> - if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END &&
> - (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
> - return 0;
> -
> if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
> /* Not dependent on the sample_type as computed from a BPF helper. */
> return 0;
> }
>
> + if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT) {
> + if (expr->term == PBF_TERM_TID || expr->term == PBF_TERM_CPU ||
> + expr->term == PBF_TERM_TIME || expr->term == PBF_TERM_CGROUP)
> + return 0;
> + if (expr->op != PBF_OP_GROUP_BEGIN) {
> + pr_err("Error: %s tracepoint event does not support BPF sample term %d\n",
> + evsel__name(evsel), expr->term);
> + return -1;
> + }
> + }
[Severity: Medium]
check_sample_flags() in tools/perf/util/bpf-filter.c and
perf_get_common_sample() in tools/perf/util/bpf_skel/sample_filter.bpf.c
enable tracepoint bpf sample filtering for uid, gid, tid (and pid via
entry->part), cpu, time, and cgroup.
However, when parse_filter() calls set_filter() in
tools/perf/util/parse-events.c, tracepoint events are checked with
is_possible_tp_filter():
tools/perf/util/parse-events.c:is_possible_tp_filter() {
return strstr(str, "uid") == NULL;
}
tools/perf/util/parse-events.c:set_filter() {
...
if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT && is_possible_tp_filter(str)) {
if (evsel__append_tp_filter(evsel, str) < 0) {
fprintf(stderr,
"not enough memory to hold filter string\n");
return -1;
}
return 0;
}
...
if (!nr_addr_filters)
return perf_bpf_filter__parse(&evsel->bpf_filters, str);
}
A later commit in this series ("perf bpf-filter: Add lock_held() sample
filter") updates is_possible_tp_filter() to also check for "gid" and
"lock_held", but filters on tracepoint events using terms such as cgroup,
time, tid, or pid (or gid at this commit) without "uid" are still routed to
evsel__append_tp_filter() instead of perf_bpf_filter__parse(), causing the
tracepoint filter ioctl to fail when those terms are not tracepoint format
fields.
Should is_possible_tp_filter() in tools/perf/util/parse-events.c also
account for the other tracepoint bpf filter terms enabled here so that
set_filter() routes them to evsel->bpf_filters?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261020205859.1080743-1-irogers@google.com?part=3
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-20 20:58 ` [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-20 22:42 ` Namhyung Kim
@ 2026-10-20 23:02 ` sashiko-bot
1 sibling, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-20 23:02 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers
> (get_lock_owner, get_current_mmap_lock, get_current_siglock,
> get_lock_addr, check_lock_type, do_collect_zone_lock,
> do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into
> tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF
> sample filtering.
>
> Also add rw_semaphore.count and struct rt_mutex_base to the minimal
> vmlinux.h and extend get_lock_owner() to support rt_mutex and check
> the mutex handoff flag.
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261020205859.1080743-1-irogers@google.com?part=4
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter
2026-10-20 20:58 ` [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter Ian Rogers
2026-10-20 22:49 ` Namhyung Kim
@ 2026-10-20 23:17 ` sashiko-bot
1 sibling, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-20 23:17 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf bpf-filter: strtailcmp() misclassifies short lock symbol names
--
commit 16bcda5fecfa952837c229e2749fdc8c298eb5ca
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Add lock_held() sample filter
This commit adds BPF sample filter predicates lock_held(), lock_held_read(),
and lock_held_write() (along with :R and :W suffixes) to filter perf samples
based on whether a specified kernel lock is held. It supports direct owner
inspection for mutexes, rt-mutexes, and write-locked rwsems, and uses BPF
fentry/fexit hooks with an LRU hash map for other lock families and modes.
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index ac0e7cf40f3af..d5a85ee98d5fd 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
[ ... ]
> @@ -166,9 +182,913 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
[ ... ]
> +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
> + bool *has_lock_filter,
> + unsigned int *prog_groups,
> + unsigned int *required_groups,
> + bool *needs_lock_syms,
> + bool *needs_slab_iter)
> +{
[ ... ]
> + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0 ||
> + strtailcmp(base, "mutex") == 0) {
> + /* Inspected directly via lock->owner; no tracing hooks needed. */
> + return;
> + } else if (strtailcmp(base, "percpu_rwsem") == 0 ||
> + strtailcmp(base, "pcpu_rwsem") == 0) {
> + *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
> + *required_groups |= LOCK_PROG_PERCPU_RWSEM;
> + } else if (strtailcmp(base, "rwsem") == 0) {
> + if (mode & PBF_LOCK_MODE_READ) {
> + *prog_groups |= LOCK_PROG_RWSEM_READ;
> + *required_groups |= LOCK_PROG_RWSEM_READ;
> + }
> + if (mode & PBF_LOCK_MODE_WRITE) {
> + *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM;
> + *required_groups |= LOCK_PROG_RWSEM_WRITE;
> + }
> + } else if (strtailcmp(base, "spinlock") == 0) {
> + *prog_groups |= LOCK_PROG_SPIN;
> + *required_groups |= LOCK_PROG_SPIN;
> + } else if (strtailcmp(base, "sem") == 0) {
> + *prog_groups |= LOCK_PROG_SEM;
> + *required_groups |= LOCK_PROG_SEM;
> + } else if (strtailcmp(base, "rwlock") == 0) {
> + if (mode & PBF_LOCK_MODE_READ)
> + groups |= LOCK_PROG_RWLOCK_READ;
> + if (mode & PBF_LOCK_MODE_WRITE)
> + groups |= LOCK_PROG_RWLOCK_WRITE;
> + *prog_groups |= groups;
> + *required_groups |= groups;
> + } else {
> + *prog_groups |= LOCK_PROG_ALL;
> + }
> +}
[ ... ]
> +static int resolve_lock_expr(struct sample_filter_bpf *skel,
> + struct perf_bpf_filter_expr *expr,
> + struct perf_bpf_filter_entry *entry)
> +{
[ ... ]
> + ret = find_kernel_lock_sym(base, &addr);
> + if (ret == 0) {
> + unsigned int direct_flag = 0;
> +
> + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0)
> + direct_flag = PBF_LOCK_DIRECT_RTMUTEX;
> + else if (strtailcmp(base, "mutex") == 0)
> + direct_flag = PBF_LOCK_DIRECT_MUTEX;
> + else if (strtailcmp(base, "rwsem") == 0 &&
> + strtailcmp(base, "percpu_rwsem") != 0 &&
> + strtailcmp(base, "pcpu_rwsem") != 0)
> + direct_flag = PBF_LOCK_DIRECT_RWSEM;
> +
> + if ((direct_flag == PBF_LOCK_DIRECT_MUTEX ||
> + direct_flag == PBF_LOCK_DIRECT_RTMUTEX ||
> + strtailcmp(base, "percpu_rwsem") == 0 ||
> + strtailcmp(base, "pcpu_rwsem") == 0 ||
> + strtailcmp(base, "spinlock") == 0 ||
> + (strtailcmp(base, "sem") == 0 &&
> + strtailcmp(base, "rwsem") != 0)) &&
> + !(mode & PBF_LOCK_MODE_WRITE)) {
> + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
> + name);
> + return -EINVAL;
> + }
[Severity: Medium]
Can strtailcmp() in prescan_one_lock_expr() and resolve_lock_expr() in
tools/perf/util/bpf-filter.c misclassify lock symbol names that are shorter
than the compared suffix?
In tools/perf/util/string.c, strtailcmp() compares characters backward from
the end of both strings and returns 0 as soon as either string reaches the
beginning:
int strtailcmp(const char *s1, const char *s2)
{
int i1 = strlen(s1);
int i2 = strlen(s2);
while (--i1 >= 0 && --i2 >= 0) {
if (s1[i1] != s2[i2])
return s1[i1] - s2[i2];
}
return 0;
}
Because strtailcmp() is symmetric, it returns 0 when base is shorter than the
tested suffix and matches the tail of that suffix:
1. Passing a lock symbol named "lock" (such as the static mutexes named lock
in kernel/ksysfs.c or mm/folio.c) to lock_held(lock) or
lock_held_read(lock) matches strtailcmp("lock", "spinlock") == 0. Instead
of falling through to LOCK_PROG_ALL, prescan_one_lock_expr() only enables
LOCK_PROG_SPIN, and resolve_lock_expr() treats it as an exclusive spinlock
and rejects read mode with -EINVAL.
2. Any rw_semaphore symbol shorter than 12 characters whose name is a suffix
of "percpu_rwsem" or "pcpu_rwsem" (such as "cpu_rwsem" or "u_rwsem")
matches strtailcmp(base, "percpu_rwsem") == 0, causing
prescan_one_lock_expr() to only enable LOCK_PROG_PERCPU_RWSEM and
resolve_lock_expr() to omit PBF_LOCK_DIRECT_RWSEM and reject read mode
with -EINVAL.
3. Any mutex symbol named "_mutex" or "t_mutex" matches
strtailcmp(base, "rt_mutex") == 0, causing resolve_lock_expr() to set
PBF_LOCK_DIRECT_RTMUTEX instead of PBF_LOCK_DIRECT_MUTEX.
Would it be better to use str_ends_with() from linux/string.h here so that
base is required to be at least as long as the suffix?
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261020205859.1080743-1-irogers@google.com?part=5
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v5 6/8] perf test: Add tests for lock_held() BPF sample filter
2026-10-20 20:58 ` [PATCH v5 6/8] perf test: Add tests for lock_held() BPF " Ian Rogers
@ 2026-10-20 23:20 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-20 23:20 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Add unit tests in parse-events.c for parsing lock_held(),
> lock_held_read(), and lock_held_exclusive() BPF filter expressions
> (standalone predicates, negated predicates, and compound group
> expressions) and extend record_bpf_filter.sh with lock_held(),
> lock_held_read()/lock_held_write(), and ':R'/':W' recording tests.
>
> Example output:
>
> # perf test -v 'Parsing of BPF sample filters'
> 5: Parse event definition strings :
> 5.7: Parsing of BPF sample filters : Ok
>
> # perf test -v 'perf record sample filtering (by BPF) tests'
> 112: perf record sample filtering (by BPF) tests :
> Checking BPF-filter privilege
> Basic bpf-filter test
> Basic bpf-filter test [Success]
> Failing bpf-filter test
> Failing bpf-filter test [Success]
> Group bpf-filter test
> Group bpf-filter test [Success]
> Multiple bpf-filter test
> Multiple bpf-filter test [Success]
> Cgroup bpf-filter test
> Cgroup bpf-filter test [Success]
> Lock-held bpf-filter test
> Lock-held bpf-filter test [Success]
> 112: perf record sample filtering (by BPF) tests : Ok
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261020205859.1080743-1-irogers@google.com?part=6
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v5 7/8] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
2026-10-20 20:58 ` [PATCH v5 7/8] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
@ 2026-10-20 23:29 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-20 23:29 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks
> Trace RCU, and SRCU locks:
>
> - 'rcu' (supports both read/shared and write/exclusive modes):
> - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)',
> 'lock_held(rcu_read_lock)'): directly inspects
> current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on
> CONFIG_PREEMPT_RCU kernels), accounting for the baseline
> rcu_read_lock() nesting acquired by the perf overflow handler itself
> (bpf_overflow_handler() and do_perf_sw_event()) with zero extra
> fentry/fexit tracing hooks.
> - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'):
> attaches fentry/fexit hooks to synchronize_rcu(),
> synchronize_rcu_expedited(), synchronize_rcu_tasks(), and
> rcu_barrier() to track grace-period and barrier waits.
> [ ... ]
>
> # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1
> # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261020205859.1080743-1-irogers@google.com?part=7
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v5 8/8] perf lock contention: Extend BPF lock contention to RCU and SRCU
2026-10-20 20:58 ` [PATCH v5 8/8] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
@ 2026-10-20 23:37 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-20 23:37 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Extend 'perf lock contention --use-bpf' to track RCU and SRCU
> grace-period and barrier waits:
>
> - Attach fentry/fexit programs to synchronize_rcu(),
> synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(),
> synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier()
> when available in BTF.
> - Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu',
> '-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct
> symbol names).
>
> Example usage:
>
> # perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1
> total wait avg wait type caller
>
> 35.12 ms 8.78 ms rcu:W cleanup_net+0x284
> 7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114
>
> # perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1
> total wait avg wait address symbol
>
> 35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu)
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261020205859.1080743-1-irogers@google.com?part=8
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat
2026-10-20 4:34 ` Namhyung Kim
@ 2026-10-21 21:39 ` Arnaldo Carvalho de Melo
0 siblings, 0 replies; 106+ messages in thread
From: Arnaldo Carvalho de Melo @ 2026-10-21 21:39 UTC (permalink / raw)
To: Namhyung Kim
Cc: Ian Rogers, adrian.hunter, james.clark, jolsa, linux-kernel,
linux-perf-users, mingo, peterz, eranian, ddhariwal, rcu
On Mon, Oct 19, 2026 at 09:34:04PM -0700, Namhyung Kim wrote:
> On Mon, Oct 19, 2026 at 03:09:09PM -0700, Ian Rogers wrote:
> > In report_lock_contention_begin_event(), lock_stat_findnew() was
> > called before checking filters.nr_types (-Y), filters.nr_addrs (-L),
> > and match_callstack_filter() (-S).
> >
> > When a filtered-out lock event (such as a mutex contention when
> > running with -Y spinlock) shared the same aggregation key as a
> > subsequent matching event (for example, contention on the mutex's
> > internal wait_lock spinlock from the same caller), the first event
> > created the lock_stat entry with the filtered-out lock's flags and the
> > second event reused that entry, causing non-spinlock entries to appear
> > in -Y spinlock output.
> >
> > Check the type, address, and callstack filters before looking up or
> > creating the lock_stat entry.
> >
> > Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option")
> > Signed-off-by: Ian Rogers <irogers@google.com>
>
> Reviewed-by: Namhyung Kim <namhyung@kernel.org>
Cherry picked this one as its the first in the series.
- Arnaldo
^ permalink raw reply [flat|nested] 106+ messages in thread
* [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support
2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (7 preceding siblings ...)
2026-10-20 20:58 ` [PATCH v5 8/8] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
@ 2026-10-22 20:29 ` Ian Rogers
2026-10-22 20:29 ` [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers
` (10 more replies)
8 siblings, 11 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
This series adds a new BPF sample filter predicate, `lock_held(<lock>)`
(along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`,
`lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W`
mode suffixes), to `perf record --filter`, allowing perf event samples to
be recorded only when a specified kernel lock is held (or not held with
`!`) by the running kernel thread or CPU. It also fixes a `bpf_link`
memory leak in `perf_bpf_filter__prepare()`, enables BPF sample filters on
tracepoint events (`PERF_TYPE_TRACEPOINT`), and extends both `lock_held()`
and `perf lock contention --use-bpf` to support RCU and SRCU.
For example:
# perf record -a -g -e cycles --filter 'lock_held(cgroup_mutex)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(mmap_lock)' -- ...
# perf record -a -g -e cycles --filter 'lock_held(mmap_lock:W)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ...
# perf lock contention -a -b -Y rcu,srcu -- sleep 1
An important use-case is detecting memory allocations or direct memory
reclaim occurring while kernel locks are held, which can stall lock
waiters for extended periods under memory pressure:
# Direct memory reclaim triggered while holding any mutex or rwsem:
# perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \
--filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
# Page allocations permitted to enter direct reclaim (__GFP_DIRECT_RECLAIM)
# while holding a mutex or rwsem:
# perf record -a -g -e kmem:mm_page_alloc \
--filter 'gfp_flags & 0x400' \
--filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
# Major page faults occurring while holding a mutex or rwsem:
# perf record -a -g -e major-faults \
--filter 'lock_held(mutex) || lock_held(rwsem)' -- ...
Supported `<lock>` specifications match `perf lock contention`:
- Global/static kernel lock symbols (e.g. `lock_held(cgroup_mutex)`)
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`,
`lock_held_read(&dentry)`)
- Well-known per-process, semi-global, and RCU locks: `mmap_lock`,
`siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`
- Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`,
`rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`,
`percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`,
`rcu:W`, `semaphore`
Limitations and weaknesses in determining whether a lock is held:
- Inlined lock/unlock operations:
- When the kernel is configured to inline spinlock or rwlock operations
(`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`,
`CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`,
`CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` /
`_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are
inlined into callers and have no BTF function entry points for
`fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that
both acquire and release entry points exist in BTF before enabling a
lock family.
- Fast-path `percpu_down_read()` / `percpu_up_read()` and
`srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined
per-CPU counter increments without function calls, so only
`percpu_down_write()` / `percpu_up_write()` and out-of-line
`__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked.
- On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to
`preempt_disable()` and does not maintain
`current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)`
read-side inspection requires `CONFIG_PREEMPT_RCU`.
- Lack of an inspectable owner field on non-exclusive / spinning locks:
- While `struct mutex`, `struct rt_mutex`, and write-locked
`struct rw_semaphore` store the owning `task_struct *` in `lock->owner`
(allowing `perf_sample_filter` to inspect kernel memory directly on
sample overflow when filtering by lock address without attaching any
`fentry`/`fexit` hooks for `*_mutex`, `*_rt_mutex`, and `mmap_lock:W`),
spinlocks, rwlocks, semaphores, SRCU, and reader-locked
`rw_semaphore`s (where `__up_read()` does not clear `sem->owner` when
`!CONFIG_DEBUG_RWSEMS`) do not maintain an authoritative per-task
owner pointer.
- Tracking these locks (as well as slab-cache and lock-type filters)
relies on `fexit` (acquire) and `fentry` (release) BPF hooks
maintaining the `held_locks` BPF LRU hash map:
- Any such lock already held before `perf record` attaches its BPF
programs will not be present in `held_locks` until it is released
and re-acquired.
- The attached `fentry`/`fexit` hooks execute on every kernel lock
acquire and release for the hooked lock family across the system,
adding tracing overhead outside of perf sample handling.
- Cross-task semaphore signaling (`down*()` / `up()`):
- Counting `struct semaphore`s do not enforce same-task ownership in the
kernel: when a semaphore is used as a completion/signaling mechanism
(`down()` in one task woken by `up()` in another task or interrupt
handler, which proposed kernel dependency trackers like `CONFIG_DEPT`
model as wait/event dependencies bounded by execution context rather
than lock ownership), `fexit/down` and `fentry/up` do not pair on the
same task ID. Semaphore hooks are therefore only enabled when
`semaphore` or a `*_sem` symbol (such as `console_sem`) is explicitly
specified, and `held_locks` uses `BPF_MAP_TYPE_LRU_HASH` so any
unpaired cross-task entries are automatically evicted.
- Shared `trace_event_call->prog_array` on `PERF_TYPE_TRACEPOINT`:
- In `kernel/events/core.c`, `__perf_event_set_bpf_prog()` requires
`BPF_PROG_TYPE_TRACEPOINT` for `PERF_TYPE_TRACEPOINT` events and
attaches the program to `event->tp_event->prog_array` on the shared
`trace_event_call`. Because `perf_trace_run_bpf_submit()` executes
`trace_call_bpf()` before `perf_tp_event()`, returning 0 from
`perf_tp_sample_filter` filters that tracepoint invocation before
`perf_tp_event()` iterates `call->perf_events` (affecting concurrent
`perf_event` listeners on the same tracepoint while the privileged
`perf record` session is active).
Patch overview (patches 1-2 are standalone fixes independent of the rest
of the series):
1. "perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF":
Attaches `perf_sample_filter` directly via `PERF_EVENT_IOC_SET_BPF` in
`perf_bpf_filter__prepare()`, matching the pinned filter path and
avoiding leaking `struct bpf_link_perf` or double-closing
`FD(evsel, x, y)`.
2. "perf bpf-filter: Support BPF sample filters on tracepoint events":
Adds `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF
sample filters can attach to `PERF_TYPE_TRACEPOINT` events (where
`__perf_event_set_bpf_prog()` in `kernel/events/core.c` requires
`BPF_PROG_TYPE_TRACEPOINT` rather than `BPF_PROG_TYPE_PERF_EVENT`).
3. "perf lock contention: Share common BPF and lock type definitions in
lock_data.h": Pure refactoring moving `lock_type_table` to
`util/lock-contention.c` and `LCB_F_*` flags, CO-RE structs, and common
BPF lock helpers into `tools/perf/util/bpf_skel/lock_data.h`.
4. "perf lock contention: Extend get_lock_owner() for rt_mutex and mutex
handoff": Extends `get_lock_owner()` in `lock_data.h` to support
`rt_mutex` (`LCB_F_RT`), `CONFIG_PREEMPT_RT` mutexes, and
`MUTEX_FLAG_HANDOFF`.
5. "perf bpf-filter: Add lock_held() sample filter for global mutexes":
Adds the `lock_held()`, `lock_held_read()`/`lock_held_shared()`,
`lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` filter
syntax and evaluates global/static kernel mutexes via direct
`lock->owner` inspection.
6. "perf bpf-filter: Extend lock_held() to other kernel lock types":
Extends `lock_held()` to spinlocks, rwsems, rwlocks, rt-mutexes,
percpu-rwsems, and semaphores (as well as hex addresses and lock type
filters) using a `held_locks` BPF LRU hash map updated by selectively
attached `fentry`/`fexit` hooks.
7. "perf bpf-filter: Support per-process and semi-global locks in
lock_held()": Adds support for `mmap_lock`, `siglock`, `rq_lock`, and
`zone_lock`.
8. "perf bpf-filter: Support slab cache locks (&<name>) in lock_held()":
Adds support for slab cache locks prefixed with `&` (e.g.
`lock_held(&dentry)`) using the `kmem_cache` BPF iterator and `slab_caches`
map.
9. "perf test: Add tests for lock_held() BPF sample filter": Adds unit
tests in `tests/parse-events.c` and shell tests in
`tests/shell/record_bpf_filter.sh`.
10. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU": Supports `rcu` (`rcu:R` via direct
`current->rcu_read_lock_nesting` inspection with zero tracing hooks;
`rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks),
`rcu-trace` (via `current->trc_reader_nesting`), and `srcu`
(`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W`
via `synchronize_srcu*()` / `srcu_barrier()`).
11. "perf lock contention: Extend BPF lock contention to RCU and SRCU":
Extends `perf lock contention --use-bpf` to track RCU and SRCU
grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`).
Changes in v6:
- Drop "perf lock contention: Filter events before creating lock_stat"
as Arnaldo cherry-picked it.
- Add an inline comment above `pin_filter_object()` in `util/bpf-filter.c`
in patch 2 explaining why `bpf_object__pin()` / `bpf_object__unpin()`
cannot be used when programs in `sample_filter.bpf.c` have `autoload`
disabled (Namhyung).
- Split the `lock_data.h` refactoring into a pure code-movement commit
(patch 3) and a separate commit extending `get_lock_owner()` for
`rt_mutex` and `MUTEX_FLAG_HANDOFF` (patch 4), keep `lock_syms` in its
original location in `lock_contention.bpf.c`, and add a comment above
`#ifdef __BPF__` in `lock_data.h` explaining that the header is shared
between userspace and BPF programs (Namhyung).
- Split the `lock_held()` implementation into 4 incremental commits
(patches 5-8): (5) core parser, expression evaluation, and global mutex
support; (6) other kernel lock types (`spinlock`, `rwsem`, `rwlock`,
`rt-mutex`, `percpu-rwsem`, `semaphore`), hex addresses, and lock type
filters; (7) per-process (`mmap_lock`, `siglock`) and semi-global
(`rq_lock`, `zone_lock`) locks; and (8) slab cache locks (`&<name>`)
(Namhyung).
- Update `is_possible_tp_filter()` in `util/parse-events.c` in patch 2
to check all tracepoint BPF sample filter terms enabled by that patch
(`uid`, `gid`, `tid`, `pid`, `cpu`, `time`, `cgroup`) using
word-boundary matching (`has_filter_word()`) and `!evsel__field()` so
they are routed to `evsel->bpf_filters` when not matching a tracepoint
format field, and extend `bpf_tp_terms[]` with `lock_held*` in patch 5.
- Replace `strtailcmp(base, ...)` with `str_ends_with(base, ...)` in
`prescan_one_lock_expr()` and `resolve_lock_expr()` in
`util/bpf-filter.c` (patches 5, 6, and 10) so lock symbol names shorter
than a tested suffix (such as `lock` vs `spinlock`, or `u_rwsem` vs
`percpu_rwsem`) are not misclassified.
- Use a CO-RE flavor struct `struct rt_mutex_base___new` in `lock_data.h`
(patch 4) and add explicit standard C header includes (`<errno.h>`,
`<stdio.h>`, `<unistd.h>`, `<string.h>`, `<stddef.h>`, `<stdlib.h>`) in
patches 1, 2, and 11 for musl libc compatibility.
- Reject kernel text functions, module text functions, and absolute or
undefined symbols when resolving lock addresses from `/proc/kallsyms`,
and check for `__percpu_rwsem_rc_<name>` in BTF so `percpu_rw_semaphore`
symbols ending in `_rwsem` (such as `cgroup_threadgroup_rwsem` or
`file_rwsem`) do not set `PBF_LOCK_DIRECT_RWSEM` or allow read-only mode.
- Link to v5: https://lore.kernel.org/linux-perf-users/20261020205859.1080743-1-irogers@google.com/
Changes in v5:
- Add patch 2 ("perf bpf-filter: Fix bpf_link leak by using
PERF_EVENT_IOC_SET_BPF") attaching BPF sample filters via
`ioctl(..., PERF_EVENT_IOC_SET_BPF, fd)` instead of
`bpf_program__attach_perf_event_opts()`, fixing the `bpf_link` memory
leak without double-closing `FD(evsel, x, y)`, and use the same
attachment in patch 3 for tracepoint sample filters (Namhyung).
- Expand patch 3's commit message to detail why `pin_filter_object()`
and `unpin_filter_object()` replace `bpf_object__pin()` and
`bpf_object__unpin()` when `autoload` is disabled on
`perf_tp_sample_filter` (Namhyung).
- Store the resolved lock target in `perf_bpf_filter_entry.value` (with
`op` normalized to `PBF_OP_EQ` or `PBF_OP_NEQ`), keeping
`struct perf_bpf_filter_entry` unchanged at 24 bytes for backwards
compatibility with pinned `filters` maps.
- Add a CO-RE `struct rt_mutex___old` fallback in `get_lock_owner()` in
`lock_data.h` for kernels prior to v5.15, and pass baseline nesting
depth to `is_rcu_trace_lock_held()` for syscall tracepoints.
Changes in v4:
- Move the two bugfix commits (`Filter events before creating lock_stat`
and `Support BPF sample filters on tracepoint events`) to patches 1
and 2 at the start of the series for easier cherry-picking.
- Reserve leading `&` strictly for slab cache locks (e.g.
`lock_held(&dentry)`) so global/static lock symbols and well-known
locks do not use `&` (e.g. `lock_held(cgroup_mutex)`,
`lock_held_read(mmap_lock)`), matching `perf lock contention`
(Namhyung).
- Skip attaching `fentry`/`fexit` hooks when filtering on a specific
`mutex` symbol (`*_mutex`), `rt_mutex` symbol (`*_rt_mutex`), or
write-locked `mmap_lock` (`mmap_lock:W` / `lock_held_write(mmap_lock)`),
evaluating them purely via direct `owner` inspection (Namhyung).
- Use a CO-RE flavor struct `struct mutex___non_rt` in `get_lock_owner()`
in `lock_data.h` so `GEN_VMLINUX_H=1` builds succeed on
`CONFIG_PREEMPT_RT=y` kernels.
- Switch `held_locks` to `BPF_MAP_TYPE_LRU_HASH`, exclude `LOCK_PROG_SEM`
from `LOCK_PROG_ALL` unless `semaphore` or a `*_sem` symbol is
explicitly requested, and document cross-task semaphore signaling
limitations (and comparison with `CONFIG_DEPT`).
- Skip `LCB_F_MUTEX | LCB_F_SPIN` in `parse_lock_type_name()`, reject
read-only mode on exclusive-only lock symbol suffixes in
`resolve_lock_expr()`, and check `needed_groups & supported_groups`
before masking `needed_groups` in `configure_lock_progs()`.
- Recursively traverse anonymous structs/unions in `btf_has_task_field()`
so `lock_held(rcu)` and `lock_held(rcu-trace)` work on
`CONFIG_RANDSTRUCT` kernels.
Changes in v3:
- Move the writer-only `rw_semaphore` check out of `get_lock_owner()` in
`lock_data.h` into `is_rwsem_writer_owner()` in `sample_filter.bpf.c`
so `perf lock contention -b -o` still reports reader owners when a
writer contends on a reader-owned `rw_semaphore`, and scope
`is_direct_lock_owner()` by lock type (`PBF_LOCK_DIRECT_MUTEX`,
`PBF_LOCK_DIRECT_RWSEM`, `PBF_LOCK_DIRECT_RTMUTEX`).
- Add `SEC("tracepoint") int perf_tp_sample_filter(void *ctx)` so BPF
sample filters (`lock_held()`, `uid`, `gid`, etc.) can attach to
`PERF_TYPE_TRACEPOINT` events (such as `kmem:mm_page_alloc` and
`vmscan:mm_vmscan_direct_reclaim_begin`), check `libbpf_get_error()`
when attaching, and document memory allocation / direct reclaim
filtering with `lock_held()`.
- Return early in `init_numa_data()` in `bpf-filter.c` when `node_data`
cannot be resolved and check `node_data_addr != 0` in
`do_collect_zone_lock()`.
- Avoid double-counting `_raw_read_lock_irq` / `_raw_read_lock_bh` in
`sample_filter.bpf.c` on `CONFIG_GENERIC_LOCKBREAK=y &&
!CONFIG_DEBUG_LOCK_ALLOC` kernels where they call
`_raw_read_lock_irqsave()`.
- Offset `rcu_state_addr` by 1 byte so it is distinct from
`&rcu_state.node[0].lock` at offset 0 of `rcu_state`, and pass `bool
is_rcu` to `do_contention_end()` in `lock_contention.bpf.c` so
`tp_btf/contention_end` and RCU/SRCU `fexit` hooks never cross-match
timestamp maps.
Changes in v2:
- Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists`
and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE
relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels.
- Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points
(`_mutex_lock_nest_lock`, `_mutex_lock_killable`,
`_mutex_trylock_nest_lock`, `_down_write_nest_lock`,
`_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to
`sample_filter.bpf.c` and `lock_progs[]`.
- Verify in `supported_lock_prog_groups()` that both acquire and all
unlock BTF entry points exist for a lock family before enabling its
tracing hooks, preventing acquire-only attachment when spinlock/rwlock
unlock functions are inlined.
- Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when
a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`).
- Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write`
on `track_rwsem_read` and verify non-zero read/write counts in
`record_lock_released_addr()` before decrementing aggregate slab/type
counts.
- Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in
`lock_contention.bpf.c` and recognize `synchronize_rcu*`,
`synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in
`machine__is_lock_function()`.
- Add patch fixing `report_lock_contention_begin_event()` to check
type, address, and callstack filters before creating `lock_stat`
entries.
Assisted-by: Antigravity:gemini-3.1-pro
Ian Rogers (11):
perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF
perf bpf-filter: Support BPF sample filters on tracepoint events
perf lock contention: Share common BPF and lock type definitions in
lock_data.h
perf lock contention: Extend get_lock_owner() for rt_mutex and mutex
handoff
perf bpf-filter: Add lock_held() sample filter for global mutexes
perf bpf-filter: Extend lock_held() to other kernel lock types
perf bpf-filter: Support per-process and semi-global locks in
lock_held()
perf bpf-filter: Support slab cache locks (&<name>) in lock_held()
perf test: Add tests for lock_held() BPF sample filter
perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU
perf lock contention: Extend BPF lock contention to RCU and SRCU
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/Documentation/perf-record.txt | 74 +-
tools/perf/builtin-lock.c | 27 -
tools/perf/tests/parse-events.c | 121 ++
tools/perf/tests/shell/lock_contention.sh | 14 +
tools/perf/tests/shell/record_bpf_filter.sh | 142 ++
tools/perf/util/bpf-filter.c | 1475 ++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 47 +-
tools/perf/util/bpf_lock_contention.c | 174 +-
.../perf/util/bpf_skel/lock_contention.bpf.c | 431 ++---
tools/perf/util/bpf_skel/lock_data.h | 394 ++++-
tools/perf/util/bpf_skel/sample-filter.h | 80 +
tools/perf/util/bpf_skel/sample_filter.bpf.c | 1241 ++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 +
tools/perf/util/lock-contention.c | 23 +
tools/perf/util/lock-contention.h | 15 +
tools/perf/util/machine.c | 11 +
tools/perf/util/parse-events.c | 40 +-
20 files changed, 4008 insertions(+), 359 deletions(-)
--
2.56.0.307.g5fe7bd680c-goog
^ permalink raw reply [flat|nested] 106+ messages in thread
* [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
@ 2026-10-22 20:29 ` Ian Rogers
2026-10-22 20:35 ` sashiko-bot
2026-10-22 22:06 ` Namhyung Kim
2026-10-22 20:29 ` [PATCH v6 02/11] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
` (9 subsequent siblings)
10 siblings, 2 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
In perf_bpf_filter__prepare(), the unpinned filter path attaches
skel->progs.perf_sample_filter to each FD(evsel, x, y) using
bpf_program__attach_perf_event_opts() and ignores the returned
struct bpf_link *, leaking the allocated struct bpf_link_perf and its
BPF_PERF_EVENT link fd.
Furthermore, bpf_program__attach_perf_event_opts() is designed for
callers that transfer ownership of the perf_event fd to the returned
bpf_link: when bpf_link__destroy() is called, bpf_link_perf_detach()
unconditionally disables and closes perf_link->perf_event_fd. Because
evsel->core.fd owns FD(evsel, x, y) and closes them in evlist__close(),
destroying the returned bpf_link would double-close the perf_event file
descriptors.
Fix this by attaching the program directly with
ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd), matching the pinned
filter path right above it in perf_bpf_filter__prepare(). In the
kernel, PERF_EVENT_IOC_SET_BPF takes a reference on the BPF program and
_free_event() -> perf_event_free_bpf_prog() automatically detaches and
releases it when the perf_event fd is closed. This also avoids enabling
the event prematurely, allowing pe_opts (.dont_enable = true) and the
LIBBPF_CURRENT_VERSION_GEQ(1, 7) guard to be dropped.
Before (built with EXTRA_CFLAGS="-fsanitize=address"):
# perf record -e cycles --filter 'period > 1000' -- true
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.024 MB perf.data ]
=================================================================
==51328==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 1568 byte(s) in 28 object(s) allocated from:
#0 0x7ffbe76ef24f in calloc (/lib64/libasan.so.8+0xef24f)
#1 0x625df7 in bpf_program__attach_perf_event_opts (tools/perf/perf+0x625df7)
#2 0xbeddc0 in perf_bpf_filter__prepare util/bpf-filter.c:507
#3 0x8f3def in evlist__apply_filters util/evlist.c:1421
#4 0x44bbcf in record__open tools/perf/builtin-record.c:1469
#5 0x453f9e in __cmd_record tools/perf/builtin-record.c:2594
#6 0x45dadb in cmd_record tools/perf/builtin-record.c:4427
#7 0x5df56b in run_builtin tools/perf/perf.c:383
#8 0x5dfa0e in handle_internal_command tools/perf/perf.c:436
#9 0x5dfddc in run_argv tools/perf/perf.c:480
#10 0x5e0333 in main tools/perf/perf.c:581
SUMMARY: AddressSanitizer: 1568 byte(s) leaked in 28 allocation(s).
After (built with EXTRA_CFLAGS="-fsanitize=address"):
# perf record -e cycles --filter 'period > 1000' -- true
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.024 MB perf.data ]
Reported-by: Namhyung Kim <namhyung@kernel.org>
Closes: https://lore.kernel.org/linux-perf-users/20261020053226.55936-1-namhyung@kernel.org/
Fixes: 56ec9457a4a2 ("perf bpf filter: Implement event sample filtering")
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/bpf-filter.c | 18 +++++-------------
1 file changed, 5 insertions(+), 13 deletions(-)
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 72e83ac1d6a4..506414e5c407 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -40,6 +40,7 @@
* The 'dropped' map is to keep how many samples it dropped by the filter and
* it will be reported as lost samples.
*/
+#include <errno.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/ioctl.h>
@@ -423,16 +424,8 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
{
int i, x, y, fd, ret;
struct sample_filter_bpf *skel = NULL;
- struct bpf_program *prog;
- struct bpf_link *link;
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
-#if LIBBPF_CURRENT_VERSION_GEQ(1, 7)
- DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts,
- .dont_enable = true);
-#else
- DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts);
-#endif
entry = calloc(MAX_FILTERS, sizeof(*entry));
if (entry == NULL)
@@ -501,14 +494,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- prog = skel->progs.perf_sample_filter;
+ fd = bpf_program__fd(skel->progs.perf_sample_filter);
for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
- link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y),
- &pe_opts);
- if (IS_ERR(link)) {
+ ret = ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd);
+ if (ret < 0) {
+ ret = -errno;
pr_err("Failed to attach perf sample-filter program\n");
- ret = PTR_ERR(link);
goto err;
}
}
--
2.56.0.307.g5fe7bd680c-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v6 02/11] perf bpf-filter: Support BPF sample filters on tracepoint events
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-22 20:29 ` [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers
@ 2026-10-22 20:29 ` Ian Rogers
2026-10-22 20:55 ` sashiko-bot
2026-10-22 20:29 ` [PATCH v6 03/11] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
` (8 subsequent siblings)
10 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Commit 466db4275edd ("perf parse-events: Add parse_uid_filter helper")
updated set_filter() to route non-tracepoint filter terms (such as uid
and gid) on PERF_TYPE_TRACEPOINT events to evsel->bpf_filters instead
of the ftrace tracepoint predicate filter.
However, in the kernel's __perf_event_set_bpf_prog()
(kernel/events/core.c), attaching a BPF program to a perf_event fd
branches on perf_event_is_tracing(event):
- Non-tracing events (hardware PMU, software events, breakpoints) call
perf_event_set_bpf_handler(), which requires BPF_PROG_TYPE_PERF_EVENT
(SEC("perf_event")).
- Tracepoint events (event->pmu == &perf_tracepoint) reject
BPF_PROG_TYPE_PERF_EVENT with -EINVAL and instead require
BPF_PROG_TYPE_TRACEPOINT (SEC("tracepoint")), which is attached to
event->tp_event->prog_array and executed in
perf_trace_run_bpf_submit() before perf_tp_event(), dropping the
sample if the BPF program returns 0. Furthermore, the
bpf_cast_to_kern_ctx() kfunc is only permitted in
BPF_PROG_TYPE_PERF_EVENT programs.
Add SEC("tracepoint") int perf_tp_sample_filter(void *ctx) in
sample_filter.bpf.c to evaluate context-independent filter terms (uid,
gid, tid, cpu, time, cgroup) using BPF helpers without calling
bpf_cast_to_kern_ctx(). When preparing a BPF filter on a
PERF_TYPE_TRACEPOINT evsel, autoload perf_tp_sample_filter instead of
perf_sample_filter and attach it once to the shared trace_event_call.
Because returning 0 from a BPF_PROG_TYPE_TRACEPOINT program filters at
the shared trace_event_call->prog_array prior to perf_tp_event(),
tracepoint BPF filters are never pinned for unprivileged use and
require CAP_BPF or root privileges.
Because perf_bpf_filter__pin() and perf_bpf_filter__unpin() only load
perf_sample_filter and disable autoload on perf_tp_sample_filter,
libbpf's bpf_object__pin() and bpf_object__unpin() can no longer be used
directly (as bpf_object__pin_programs() and bpf_object__unpin_programs()
iterate over all programs in the object without checking autoload and
fail with -EINVAL when a program has not been loaded). Introduce
pin_filter_object() and unpin_filter_object() to pin/unpin the maps and
only the loaded perf_sample_filter program.
Before:
# perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true
libbpf: prog 'perf_sample_filter': failed to attach to perf_event FD 3: Invalid argument
Failed to attach perf sample-filter program
failed to set filter "BPF" on event kmem:mm_page_alloc with 22 (Invalid argument)
After:
# perf record -e kmem:mm_page_alloc --filter 'uid == 0' -- true
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.024 MB perf.data (42 samples) ]
Fixes: 466db4275edd ("perf parse-events: Add parse_uid_filter helper")
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 8 +-
tools/perf/util/bpf-filter.c | 144 ++++++++++++++++---
tools/perf/util/bpf_skel/sample_filter.bpf.c | 95 ++++++++++++
tools/perf/util/parse-events.c | 38 ++++-
4 files changed, 262 insertions(+), 23 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 2351f7b4f5fc..fcf1ea62bf1c 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -173,7 +173,13 @@ OPTIONS
A BPF filter can access the sample data and make a decision based on the
data. Users need to set an appropriate sample type to use the BPF
- filter. BPF filters need root privilege.
+ filter. BPF filters need root privilege (or CAP_BPF, or a pinned BPF
+ filter for non-tracepoint events). On tracepoint events, BPF filters
+ support context-independent terms ('uid', 'gid', 'tid', 'pid', 'cpu',
+ 'time', 'cgroup') by attaching a 'BPF_PROG_TYPE_TRACEPOINT' program to
+ the tracepoint's shared 'trace_event_call->prog_array', which filters
+ the tracepoint before 'perf_tp_event()' runs across concurrent
+ perf_event listeners on that tracepoint.
The sample data field can be specified in lower case letter. Multiple
filters can be separated with comma. For example,
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 506414e5c407..0d3c700d3dcd 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -41,8 +41,10 @@
* it will be reported as lost samples.
*/
#include <errno.h>
+#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
+#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/stat.h>
@@ -124,15 +126,26 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
{
const struct perf_sample_info *info;
- if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END &&
- (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
- return 0;
-
if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
/* Not dependent on the sample_type as computed from a BPF helper. */
return 0;
}
+ if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT) {
+ if (expr->term == PBF_TERM_TID || expr->term == PBF_TERM_CPU ||
+ expr->term == PBF_TERM_TIME || expr->term == PBF_TERM_CGROUP)
+ return 0;
+ if (expr->op != PBF_OP_GROUP_BEGIN) {
+ pr_err("Error: %s tracepoint event does not support BPF sample term %d\n",
+ evsel__name(evsel), expr->term);
+ return -1;
+ }
+ }
+
+ if (expr->term >= PBF_TERM_SAMPLE_START && expr->term <= PBF_TERM_SAMPLE_END &&
+ (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
+ return 0;
+
if (expr->op == PBF_OP_GROUP_BEGIN) {
struct perf_bpf_filter_expr *group;
@@ -426,6 +439,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
struct sample_filter_bpf *skel = NULL;
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
+ bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
entry = calloc(MAX_FILTERS, sizeof(*entry));
if (entry == NULL)
@@ -437,7 +451,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- if (needs_idx_hash && geteuid() != 0) {
+ if (is_tracepoint && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
+ pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n");
+ ret = -EPERM;
+ goto err;
+ }
+
+ if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
int zero = 0;
/* The filters map is shared among other processes */
@@ -477,8 +497,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return 0;
}
- skel = sample_filter_bpf__open_and_load();
+ skel = sample_filter_bpf__open();
if (!skel) {
+ ret = -errno;
+ pr_err("Failed to open perf sample-filter BPF skeleton\n");
+ goto err;
+ }
+
+ bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
+ bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
+
+ if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
pr_err("Failed to load perf sample-filter BPF skeleton\n");
goto err;
@@ -494,14 +523,32 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- fd = bpf_program__fd(skel->progs.perf_sample_filter);
- for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
- for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
- ret = ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd);
- if (ret < 0) {
- ret = -errno;
- pr_err("Failed to attach perf sample-filter program\n");
- goto err;
+ if (is_tracepoint) {
+ /*
+ * For PERF_TYPE_TRACEPOINT, PERF_EVENT_IOC_SET_BPF attaches the
+ * BPF program to event->tp_event->prog_array on the shared
+ * trace_event_call, which perf_trace_run_bpf_submit() executes
+ * for all CPUs and tasks before iterating the per-CPU perf
+ * events. Attach once on the first FD so tp_event->prog_array
+ * does not get duplicate copies for each per-CPU FD.
+ */
+ fd = bpf_program__fd(skel->progs.perf_tp_sample_filter);
+ ret = ioctl(FD(evsel, 0, 0), PERF_EVENT_IOC_SET_BPF, fd);
+ if (ret < 0) {
+ ret = -errno;
+ pr_err("Failed to attach perf tracepoint sample-filter program\n");
+ goto err;
+ }
+ } else {
+ fd = bpf_program__fd(skel->progs.perf_sample_filter);
+ for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
+ for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
+ ret = ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd);
+ if (ret < 0) {
+ ret = -errno;
+ pr_err("Failed to attach perf sample-filter program\n");
+ goto err;
+ }
}
}
}
@@ -633,6 +680,55 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
return ret;
}
+/*
+ * perf_bpf_filter__pin() and perf_bpf_filter__unpin() only load
+ * perf_sample_filter and disable autoload on other programs in the skeleton
+ * (such as perf_tp_sample_filter). While libbpf's bpf_object__pin_maps() skips
+ * maps that were not created, bpf_object__pin_programs() and
+ * bpf_object__unpin_programs() iterate over every program in skel->obj without
+ * checking whether autoload was enabled and fail with -EINVAL when a program
+ * was not loaded. Pin and unpin the maps and the loaded perf_sample_filter
+ * program explicitly instead of calling bpf_object__pin() / bpf_object__unpin().
+ */
+static int pin_filter_object(struct sample_filter_bpf *skel, const char *path)
+{
+ char *prog_path = NULL;
+ int ret;
+
+ ret = bpf_object__pin_maps(skel->obj, path);
+ if (ret < 0)
+ return ret;
+
+ if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
+ bpf_object__unpin_maps(skel->obj, path);
+ return -ENOMEM;
+ }
+
+ ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path);
+ if (ret < 0)
+ bpf_object__unpin_maps(skel->obj, path);
+
+ free(prog_path);
+ return ret;
+}
+
+static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path)
+{
+ char *prog_path = NULL;
+ int ret, map_ret;
+
+ if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
+ bpf_object__unpin_maps(skel->obj, path);
+ return -ENOMEM;
+ }
+
+ ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path);
+ free(prog_path);
+
+ map_ret = bpf_object__unpin_maps(skel->obj, path);
+ return ret < 0 ? ret : map_ret;
+}
+
int perf_bpf_filter__pin(void)
{
struct sample_filter_bpf *skel;
@@ -647,6 +743,7 @@ int perf_bpf_filter__pin(void)
}
/* pinned program will use pid-hash */
+ bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH);
bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH);
@@ -666,7 +763,7 @@ int perf_bpf_filter__pin(void)
goto err;
}
- ret = bpf_object__pin(skel->obj, path);
+ ret = pin_filter_object(skel, path);
if (ret < 0) {
pr_err("Failed to pin BPF filter objects\n");
goto err;
@@ -675,8 +772,8 @@ int perf_bpf_filter__pin(void)
/* setup access permissions for the pinned objects */
dir_fd = open(path, O_PATH);
if (dir_fd < 0) {
- bpf_object__unpin(skel->obj, path);
- ret = dir_fd;
+ ret = -errno;
+ unpin_filter_object(skel, path);
goto err;
}
@@ -719,6 +816,8 @@ int perf_bpf_filter__pin(void)
err_close:
close(dir_fd);
+ if (ret < 0)
+ unpin_filter_object(skel, path);
err:
free(path);
@@ -732,13 +831,20 @@ int perf_bpf_filter__unpin(void)
char *path = NULL;
int ret = -1;
- skel = sample_filter_bpf__open_and_load();
+ skel = sample_filter_bpf__open();
if (!skel) {
ret = -errno;
pr_err("Failed to open perf sample-filter BPF skeleton\n");
goto err;
}
+ bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
+ if (sample_filter_bpf__load(skel) < 0) {
+ ret = -errno;
+ pr_err("Failed to load perf sample-filter BPF skeleton\n");
+ goto err;
+ }
+
if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(),
PERF_BPF_FILTER_PIN_PATH) < 0) {
ret = -errno;
@@ -746,7 +852,7 @@ int perf_bpf_filter__unpin(void)
goto err;
}
- ret = bpf_object__unpin(skel->obj, path);
+ ret = unpin_filter_object(skel, path);
err:
free(path);
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index e5666d4c1722..a190374ed87a 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -192,6 +192,28 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
return 0;
}
+static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry *entry)
+{
+ switch (entry->term) {
+ case PBF_TERM_TID:
+ if (entry->part)
+ return bpf_get_current_pid_tgid() >> 32;
+ return (__u32)bpf_get_current_pid_tgid();
+ case PBF_TERM_CPU:
+ return bpf_get_smp_processor_id();
+ case PBF_TERM_TIME:
+ return bpf_ktime_get_ns();
+ case PBF_TERM_CGROUP:
+ return bpf_get_current_cgroup_id();
+ case PBF_TERM_UID:
+ return bpf_get_current_uid_gid() & 0xFFFFFFFF;
+ case PBF_TERM_GID:
+ return bpf_get_current_uid_gid() >> 32;
+ default:
+ return 0;
+ }
+}
+
#define CHECK_RESULT(data, op, val) \
if (!(data op val)) { \
if (!in_group) \
@@ -295,4 +317,77 @@ int perf_sample_filter(void *ctx)
return 0;
}
+/*
+ * BPF program to be called for tracepoint perf events.
+ *
+ * For PERF_TYPE_TRACEPOINT events, the kernel's __perf_event_set_bpf_prog()
+ * rejects BPF_PROG_TYPE_PERF_EVENT and only accepts BPF_PROG_TYPE_TRACEPOINT,
+ * attaching the program to the shared trace_event_call->prog_array executed
+ * by perf_trace_run_bpf_submit() prior to perf_tp_event(). Because returning 0
+ * filters the event at trace_event_call->prog_array (affecting the
+ * trace_event_call globally while attached), tracepoint BPF filters are never
+ * pinned for unprivileged use and require CAP_BPF / root privileges.
+ */
+SEC("tracepoint")
+int perf_tp_sample_filter(void *ctx)
+{
+ struct perf_bpf_filter_entry *entry;
+ __u64 sample_data;
+ int in_group = 0;
+ int group_result = 0;
+ int i, k = 0;
+ int *losts;
+
+ entry = bpf_map_lookup_elem(&filters, &k);
+ if (entry == NULL)
+ goto drop;
+
+ for (i = 0; i < MAX_FILTERS; i++) {
+ sample_data = perf_get_common_sample(&entry[i]);
+
+ switch (entry[i].op) {
+ case PBF_OP_EQ:
+ CHECK_RESULT(sample_data, ==, entry[i].value)
+ break;
+ case PBF_OP_NEQ:
+ CHECK_RESULT(sample_data, !=, entry[i].value)
+ break;
+ case PBF_OP_GT:
+ CHECK_RESULT(sample_data, >, entry[i].value)
+ break;
+ case PBF_OP_GE:
+ CHECK_RESULT(sample_data, >=, entry[i].value)
+ break;
+ case PBF_OP_LT:
+ CHECK_RESULT(sample_data, <, entry[i].value)
+ break;
+ case PBF_OP_LE:
+ CHECK_RESULT(sample_data, <=, entry[i].value)
+ break;
+ case PBF_OP_AND:
+ CHECK_RESULT(sample_data, &, entry[i].value)
+ break;
+ case PBF_OP_GROUP_BEGIN:
+ in_group = 1;
+ group_result = 0;
+ break;
+ case PBF_OP_GROUP_END:
+ if (group_result == 0)
+ goto drop;
+ in_group = 0;
+ break;
+ case PBF_OP_DONE:
+ return 1;
+ }
+ }
+ return 1;
+
+drop:
+ losts = bpf_map_lookup_elem(&dropped, &k);
+ if (losts != NULL)
+ __sync_fetch_and_add(losts, 1);
+
+ return 0;
+}
+
char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index 2f3317ce17f0..d6bab89ade5d 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -3,8 +3,10 @@
#include <linux/err.h>
#include <linux/list_sort.h>
#include <linux/zalloc.h>
+#include <ctype.h>
#include <dirent.h>
#include <errno.h>
+#include <string.h>
#include <sys/ioctl.h>
#include <sys/param.h>
#include "cpumap.h"
@@ -2593,10 +2595,40 @@ foreach_evsel_in_last_glob(struct evlist *evlist,
return 0;
}
+static bool is_filter_word_char(char c)
+{
+ return isalnum((unsigned char)c) || c == '_' || c == '/' || c == '.';
+}
+
+static bool has_filter_word(const char *str, const char *word)
+{
+ size_t len = strlen(word);
+ const char *p = str;
+
+ while ((p = strstr(p, word)) != NULL) {
+ bool start_ok = (p == str) || !is_filter_word_char(p[-1]);
+ bool end_ok = !is_filter_word_char(p[len]);
+
+ if (start_ok && end_ok)
+ return true;
+ p += len;
+ }
+ return false;
+}
+
/* Will a tracepoint filter work for str or should a BPF filter be used? */
-static bool is_possible_tp_filter(const char *str)
+static bool is_possible_tp_filter(struct evsel *evsel, const char *str)
{
- return strstr(str, "uid") == NULL;
+ static const char *const bpf_tp_terms[] = {
+ "uid", "gid", "tid", "pid", "cpu", "time", "cgroup",
+ };
+
+ for (size_t i = 0; i < ARRAY_SIZE(bpf_tp_terms); i++) {
+ if (has_filter_word(str, bpf_tp_terms[i]) &&
+ !evsel__field(evsel, bpf_tp_terms[i]))
+ return false;
+ }
+ return true;
}
static int set_filter(struct evsel *evsel, const void *arg)
@@ -2611,7 +2643,7 @@ static int set_filter(struct evsel *evsel, const void *arg)
return -1;
}
- if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT && is_possible_tp_filter(str)) {
+ if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT && is_possible_tp_filter(evsel, str)) {
if (evsel__append_tp_filter(evsel, str) < 0) {
fprintf(stderr,
"not enough memory to hold filter string\n");
--
2.56.0.307.g5fe7bd680c-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v6 03/11] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-22 20:29 ` [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers
2026-10-22 20:29 ` [PATCH v6 02/11] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
@ 2026-10-22 20:29 ` Ian Rogers
2026-10-22 20:36 ` sashiko-bot
2026-10-22 20:29 ` [PATCH v6 04/11] perf lock contention: Extend get_lock_owner() for rt_mutex and mutex handoff Ian Rogers
` (7 subsequent siblings)
10 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers
(get_lock_owner, get_current_mmap_lock, get_current_siglock,
get_lock_addr, check_lock_type, do_collect_zone_lock,
do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into
tools/perf/util/bpf_skel/lock_data.h and tools/perf/util/lock-contention.[ch]
so they can be reused by BPF sample filtering.
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/builtin-lock.c | 27 --
.../perf/util/bpf_skel/lock_contention.bpf.c | 235 +-------------
tools/perf/util/bpf_skel/lock_data.h | 303 ++++++++++++++++++
tools/perf/util/lock-contention.c | 19 ++
tools/perf/util/lock-contention.h | 13 +
5 files changed, 338 insertions(+), 259 deletions(-)
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index d5c0d55cb82d..feaf1cfd08cf 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -1477,33 +1477,6 @@ static void sort_result(void)
}
}
-static const struct {
- unsigned int flags;
- /*
- * Name of the lock flags (access), with delimeter ':'.
- * For example, rwsem:R of rwsem:W.
- */
- const char *flags_name;
- /* Name of the lock (type), for example, rwlock or rwsem. */
- const char *lock_name;
-} lock_type_table[] = {
- { 0, "semaphore", "semaphore" },
- { LCB_F_SPIN, "spinlock", "spinlock" },
- { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
- { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
- { LCB_F_READ, "rwsem:R", "rwsem" },
- { LCB_F_WRITE, "rwsem:W", "rwsem" },
- { LCB_F_RT, "rt-mutex", "rt-mutex" },
- { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
- { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
- { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
- { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
- { LCB_F_MUTEX, "mutex", "mutex" },
- { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
- /* alias for optimistic spinning only */
- { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
-};
-
static const char *get_type_flags_name(unsigned int flags)
{
flags &= LCB_F_TYPE_MASK;
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index 0d9c6f55050e..c70c3955e559 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -8,23 +8,9 @@
#include "lock_data.h"
-/* for collect_lock_syms(). 4096 was rejected by the verifier */
-#define MAX_CPUS 1024
-
-/* for collect_zone_lock(). It should be more than the actual zones. */
-#define MAX_ZONES 10
-
/* for do_lock_delay(). Arbitrarily set to 1 million. */
#define MAX_LOOP (1U << 20)
-/* lock contention flags from include/trace/events/lock.h */
-#define LCB_F_SPIN (1U << 0)
-#define LCB_F_READ (1U << 1)
-#define LCB_F_WRITE (1U << 2)
-#define LCB_F_RT (1U << 3)
-#define LCB_F_PERCPU (1U << 4)
-#define LCB_F_MUTEX (1U << 5)
-
/* callstack storage */
struct {
__uint(type, BPF_MAP_TYPE_STACK_TRACE);
@@ -159,22 +145,6 @@ struct {
__uint(max_entries, 1);
} lock_delays SEC(".maps");
-struct rw_semaphore___old {
- struct task_struct *owner;
-} __attribute__((preserve_access_index));
-
-struct rw_semaphore___new {
- atomic_long_t owner;
-} __attribute__((preserve_access_index));
-
-struct mm_struct___old {
- struct rw_semaphore mmap_sem;
-} __attribute__((preserve_access_index));
-
-struct mm_struct___new {
- struct rw_semaphore mmap_lock;
-} __attribute__((preserve_access_index));
-
struct cas_ctx {
struct contention_data *data;
u64 duration;
@@ -182,8 +152,6 @@ struct cas_ctx {
int min_done;
};
-extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak;
-
/* control flags */
const volatile int has_cpu;
const volatile int has_task;
@@ -222,8 +190,6 @@ int data_map_full;
struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak;
void bpf_task_release(struct task_struct *p) __ksym __weak;
-static inline __u32 check_lock_type(__u64 lock, __u32 flags);
-
static inline __u64 get_current_cgroup_id(void)
{
struct task_struct *task;
@@ -347,85 +313,6 @@ static inline int update_task_data(struct task_struct *task)
return 0;
}
-#ifndef __has_builtin
-# define __has_builtin(x) 0
-#endif
-
-static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
-{
- struct task_struct *task;
- __u64 owner = 0;
-
- if (flags & LCB_F_MUTEX) {
- struct mutex *mutex = (void *)lock;
- owner = BPF_CORE_READ(mutex, owner.counter);
- } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
- /*
- * Support for the BPF_TYPE_MATCHES argument to the
- * __builtin_preserve_type_info builtin was added at some point during
- * development of clang 15 and it's what is needed for
- * bpf_core_type_matches.
- */
-#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
- if (bpf_core_type_matches(struct rw_semaphore___old)) {
- struct rw_semaphore___old *rwsem = (void *)lock;
- owner = (unsigned long)BPF_CORE_READ(rwsem, owner);
- } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
- struct rw_semaphore___new *rwsem = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
- }
-#else
- /* assume new struct */
- struct rw_semaphore *rwsem = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
-#endif
- }
-
- if (!owner)
- return NULL;
-
- task = (void *)(owner & ~7UL);
- return task;
-}
-
-static inline __u32 check_lock_type(__u64 lock, __u32 flags)
-{
- struct task_struct *curr;
- struct mm_struct___old *mm_old;
- struct mm_struct___new *mm_new;
- struct sighand_struct *sighand;
-
- switch (flags) {
- case LCB_F_READ: /* rwsem */
- case LCB_F_WRITE:
- curr = bpf_get_current_task_btf();
- if (curr->mm == NULL)
- break;
- mm_new = (void *)curr->mm;
- if (bpf_core_field_exists(mm_new->mmap_lock)) {
- if (&mm_new->mmap_lock == (void *)lock)
- return LCD_F_MMAP_LOCK;
- break;
- }
- mm_old = (void *)curr->mm;
- if (bpf_core_field_exists(mm_old->mmap_sem)) {
- if (&mm_old->mmap_sem == (void *)lock)
- return LCD_F_MMAP_LOCK;
- }
- break;
- case LCB_F_SPIN: /* spinlock */
- curr = bpf_get_current_task_btf();
- sighand = curr->sighand;
-
- if (sighand && &sighand->siglock == (void *)lock)
- return LCD_F_SIGHAND_LOCK;
- break;
- default:
- break;
- }
- return 0;
-}
-
static inline long delay_callback(__u64 idx, void *arg)
{
__u64 target = *(__u64 *)arg;
@@ -908,99 +795,16 @@ int contention_end(u64 *ctx)
return 0;
}
-extern struct rq runqueues __ksym;
-
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
-struct rq___old {
- raw_spinlock_t lock;
-} __attribute__((preserve_access_index));
-
-struct rq___new {
- raw_spinlock_t __lock;
-} __attribute__((preserve_access_index));
-
-static void collect_zone_lock(void)
-{
- __u64 nr_zones, zone_off;
- __u64 lock_addr, lock_off;
- __u32 lock_flag = LOCK_CLASS_ZONE_LOCK;
-
- zone_off = offsetof(struct pglist_data, node_zones);
- lock_off = offsetof(struct zone, lock);
-
- if (contig_page_data_addr) {
- struct pglist_data *contig_page_data;
-
- contig_page_data = (void *)(long)contig_page_data_addr;
- nr_zones = BPF_CORE_READ(contig_page_data, nr_zones);
-
- for (int i = 0; i < MAX_ZONES; i++) {
- __u64 zone_addr;
-
- if (i >= nr_zones)
- break;
-
- zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off;
- lock_addr = zone_addr + lock_off;
-
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
- } else if (nr_nodes > 0) {
- struct pglist_data **node_data = (void *)(long)node_data_addr;
-
- for (int i = 0; i < nr_nodes; i++) {
- struct pglist_data *pgdat = NULL;
- int err;
-
- err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
- if (err < 0 || pgdat == NULL)
- break;
-
- nr_zones = BPF_CORE_READ(pgdat, nr_zones);
- for (int k = 0; k < MAX_ZONES; k++) {
- __u64 zone_addr;
-
- if (k >= nr_zones)
- break;
-
- zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off;
- lock_addr = zone_addr + lock_off;
-
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
- }
- }
-}
-
SEC("raw_tp/bpf_test_finish")
int BPF_PROG(collect_lock_syms)
{
- __u64 lock_addr, lock_off;
- __u32 lock_flag;
-
- if (bpf_core_field_exists(struct rq___new, __lock))
- lock_off = offsetof(struct rq___new, __lock);
- else
- lock_off = offsetof(struct rq___old, lock);
-
- for (int i = 0; i < MAX_CPUS; i++) {
- struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
-
- if (rq == NULL)
- break;
-
- lock_addr = (__u64)(void *)rq + lock_off;
- lock_flag = LOCK_CLASS_RQLOCK;
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
-
- collect_zone_lock();
-
- return 0;
+ return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
}
SEC("raw_tp/bpf_test_finish")
@@ -1010,43 +814,10 @@ int BPF_PROG(end_timestamp)
return 0;
}
-/*
- * bpf_iter__kmem_cache added recently so old kernels don't have it in the
- * vmlinux.h. But we cannot add it here since it will cause a compiler error
- * due to redefinition of the struct on later kernels.
- *
- * So it uses a CO-RE trick to access the member only if it has the type.
- * This will support both old and new kernels without compiler errors.
- */
-struct bpf_iter__kmem_cache___new {
- struct kmem_cache *s;
-} __attribute__((preserve_access_index));
-
SEC("iter/kmem_cache")
int slab_cache_iter(void *ctx)
{
- struct kmem_cache *s = NULL;
- struct slab_cache_data d;
- const char *nameptr;
-
- if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) {
- struct bpf_iter__kmem_cache___new *iter = ctx;
-
- s = iter->s;
- }
-
- if (s == NULL)
- return 0;
-
- nameptr = s->name;
- bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr);
-
- d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT;
- if (d.id >= LCB_F_SLAB_ID_END)
- return 0;
-
- bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST);
- return 0;
+ return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
}
char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index 652e114e6b87..91d464ac18bd 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -45,6 +45,14 @@ struct contention_task_data {
#define LCB_F_SLAB_ID_END (1U << 26)
#define LCB_F_SLAB_ID_MASK 0x03FF0000U
+/* lock contention flags from include/trace/events/lock.h */
+#define LCB_F_SPIN (1U << 0)
+#define LCB_F_READ (1U << 1)
+#define LCB_F_WRITE (1U << 2)
+#define LCB_F_RT (1U << 3)
+#define LCB_F_PERCPU (1U << 4)
+#define LCB_F_MUTEX (1U << 5)
+
#define LCB_F_TYPE_MAX (1U << 7)
#define LCB_F_TYPE_MASK 0x0000007FU
@@ -76,4 +84,299 @@ struct slab_cache_data {
char name[SLAB_NAME_MAX];
};
+/*
+ * Common BPF CO-RE structures and helper functions shared between
+ * lock_contention.bpf.c and sample_filter.bpf.c. Guarded by __BPF__
+ * because lock_data.h is also included by userspace code.
+ */
+#ifdef __BPF__
+
+/* for collect_lock_syms(). 4096 was rejected by the verifier */
+#define MAX_CPUS 1024
+
+/* for collect_zone_lock(). It should be more than the actual zones. */
+#define MAX_ZONES 10
+
+extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak;
+extern struct rq runqueues __ksym __weak;
+
+#ifndef __has_builtin
+# define __has_builtin(x) 0
+#endif
+
+struct rw_semaphore___old {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
+struct rw_semaphore___new {
+ atomic_long_t owner;
+} __attribute__((preserve_access_index));
+
+struct mm_struct___old {
+ struct rw_semaphore mmap_sem;
+} __attribute__((preserve_access_index));
+
+struct mm_struct___new {
+ struct rw_semaphore mmap_lock;
+} __attribute__((preserve_access_index));
+
+struct rq___old {
+ raw_spinlock_t lock;
+} __attribute__((preserve_access_index));
+
+struct rq___new {
+ raw_spinlock_t __lock;
+} __attribute__((preserve_access_index));
+
+/*
+ * bpf_iter__kmem_cache added recently so old kernels don't have it in the
+ * vmlinux.h. But we cannot add it here since it will cause a compiler error
+ * due to redefinition of the struct on later kernels.
+ *
+ * So it uses a CO-RE trick to access the member only if it has the type.
+ * This will support both old and new kernels without compiler errors.
+ */
+struct bpf_iter__kmem_cache___new {
+ struct kmem_cache *s;
+} __attribute__((preserve_access_index));
+
+static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
+{
+ struct task_struct *task;
+ __u64 owner = 0;
+
+ if (flags & LCB_F_MUTEX) {
+ struct mutex *mutex = (void *)lock;
+
+ owner = BPF_CORE_READ(mutex, owner.counter);
+ } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
+ /*
+ * Support for the BPF_TYPE_MATCHES argument to the
+ * __builtin_preserve_type_info builtin was added at some point during
+ * development of clang 15 and it's what is needed for
+ * bpf_core_type_matches.
+ */
+#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
+ if (bpf_core_type_matches(struct rw_semaphore___old)) {
+ struct rw_semaphore___old *rwsem = (void *)lock;
+
+ owner = (unsigned long)BPF_CORE_READ(rwsem, owner);
+ } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
+ struct rw_semaphore___new *rwsem = (void *)lock;
+
+ owner = BPF_CORE_READ(rwsem, owner.counter);
+ }
+#else
+ /* assume new struct */
+ struct rw_semaphore *rwsem = (void *)lock;
+
+ owner = BPF_CORE_READ(rwsem, owner.counter);
+#endif
+ }
+
+ if (!owner)
+ return NULL;
+
+ task = (void *)(owner & ~7UL);
+ return task;
+}
+
+static __always_inline __u64 get_current_mmap_lock(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+ struct mm_struct___new *mm_new = NULL;
+ struct mm_struct___old *mm_old = NULL;
+ __u64 mm;
+
+ if (!curr)
+ return 0;
+
+ mm = (__u64)BPF_CORE_READ(task, mm);
+ if (!mm)
+ return 0;
+
+ if (bpf_core_field_exists(mm_new->mmap_lock))
+ return mm + bpf_core_field_offset(mm_new->mmap_lock);
+
+ if (bpf_core_field_exists(mm_old->mmap_sem))
+ return mm + bpf_core_field_offset(mm_old->mmap_sem);
+
+ return 0;
+}
+
+static __always_inline __u64 get_current_siglock(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+ struct sighand_struct *sighand = NULL;
+ __u64 sh;
+
+ if (!curr)
+ return 0;
+
+ sh = (__u64)BPF_CORE_READ(task, sighand);
+ if (!sh)
+ return 0;
+
+ return sh + bpf_core_field_offset(sighand->siglock);
+}
+
+/*
+ * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar.
+ *
+ * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the
+ * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed
+ * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such
+ * as `bpf_get_kmem_cache(u64 addr)`.
+ */
+static __always_inline __u64 get_lock_addr(void *ctx)
+{
+ __u64 lock_addr = 0;
+
+ bpf_get_func_arg(ctx, 0, &lock_addr);
+ return lock_addr;
+}
+
+static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags)
+{
+ __u64 curr;
+
+ switch (flags) {
+ case LCB_F_READ: /* rwsem */
+ case LCB_F_WRITE:
+ curr = bpf_get_current_task();
+ if (get_current_mmap_lock(curr) == lock)
+ return LCD_F_MMAP_LOCK;
+ break;
+ case LCB_F_SPIN: /* spinlock */
+ curr = bpf_get_current_task();
+ if (get_current_siglock(curr) == lock)
+ return LCD_F_SIGHAND_LOCK;
+ break;
+ default:
+ break;
+ }
+ return 0;
+}
+
+static __always_inline void do_collect_zone_lock(void *lock_syms_map,
+ __u64 contig_page_data_addr,
+ __u64 node_data_addr,
+ int nr_nodes,
+ int sizeof_zone)
+{
+ __u64 nr_zones, zone_off;
+ __u64 lock_addr, lock_off;
+ __u32 lock_flag = LOCK_CLASS_ZONE_LOCK;
+
+ zone_off = offsetof(struct pglist_data, node_zones);
+ lock_off = offsetof(struct zone, lock);
+
+ if (contig_page_data_addr) {
+ struct pglist_data *contig_page_data;
+
+ contig_page_data = (void *)(long)contig_page_data_addr;
+ nr_zones = BPF_CORE_READ(contig_page_data, nr_zones);
+
+ for (int i = 0; i < MAX_ZONES; i++) {
+ __u64 zone_addr;
+
+ if (i >= nr_zones)
+ break;
+
+ zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off;
+ lock_addr = zone_addr + lock_off;
+
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ } else if (node_data_addr && nr_nodes > 0) {
+ struct pglist_data **node_data = (void *)(long)node_data_addr;
+
+ for (int i = 0; i < nr_nodes; i++) {
+ struct pglist_data *pgdat = NULL;
+ int err;
+
+ err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
+ if (err < 0)
+ break;
+ if (pgdat == NULL)
+ continue;
+
+ nr_zones = BPF_CORE_READ(pgdat, nr_zones);
+ for (int k = 0; k < MAX_ZONES; k++) {
+ __u64 zone_addr;
+
+ if (k >= nr_zones)
+ break;
+
+ zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off;
+ lock_addr = zone_addr + lock_off;
+
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ }
+ }
+}
+
+static __always_inline int do_collect_lock_syms(void *lock_syms_map,
+ __u64 contig_page_data_addr,
+ __u64 node_data_addr,
+ int nr_nodes,
+ int sizeof_zone)
+{
+ __u64 lock_addr, lock_off;
+ __u32 lock_flag;
+
+ if (bpf_core_field_exists(struct rq___new, __lock))
+ lock_off = offsetof(struct rq___new, __lock);
+ else
+ lock_off = offsetof(struct rq___old, lock);
+
+ if (&runqueues) {
+ for (int i = 0; i < MAX_CPUS; i++) {
+ struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
+
+ if (rq == NULL)
+ break;
+
+ lock_addr = (__u64)(void *)rq + lock_off;
+ lock_flag = LOCK_CLASS_RQLOCK;
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ }
+
+ do_collect_zone_lock(lock_syms_map, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
+
+ return 0;
+}
+
+static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map,
+ __u32 *slab_cache_id)
+{
+ struct kmem_cache *s = NULL;
+ struct slab_cache_data d = {};
+ const char *nameptr;
+
+ if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) {
+ struct bpf_iter__kmem_cache___new *iter = ctx;
+
+ s = iter->s;
+ }
+
+ if (s == NULL)
+ return 0;
+
+ nameptr = s->name;
+ bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr);
+
+ d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT;
+ if (d.id >= LCB_F_SLAB_ID_END)
+ return 0;
+
+ bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST);
+ return 0;
+}
+
+#endif /* __BPF__ */
+
#endif /* UTIL_BPF_SKEL_LOCK_DATA_H */
diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c
index 119a7206f3cd..f6e68d740860 100644
--- a/tools/perf/util/lock-contention.c
+++ b/tools/perf/util/lock-contention.c
@@ -6,6 +6,7 @@
#include "symbol.h"
#include <limits.h>
+#include <stdlib.h>
#include <string.h>
#include <linux/hash.h>
@@ -22,6 +23,24 @@ struct callstack_filter {
static LIST_HEAD(callstack_filters);
struct hlist_head *lockhash_table;
+const struct perf_lock_type lock_type_table[] = {
+ { 0, "semaphore", "semaphore" },
+ { LCB_F_SPIN, "spinlock", "spinlock" },
+ { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
+ { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
+ { LCB_F_READ, "rwsem:R", "rwsem" },
+ { LCB_F_WRITE, "rwsem:W", "rwsem" },
+ { LCB_F_RT, "rt-mutex", "rt-mutex" },
+ { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
+ { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
+ { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
+ { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
+ { LCB_F_MUTEX, "mutex", "mutex" },
+ { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
+ /* alias for optimistic spinning only */
+ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
+};
+
int parse_call_stack(const struct option *opt __maybe_unused, const char *str,
int unset __maybe_unused)
{
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index 59c94190b092..68e9bd77e8a9 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -129,6 +129,19 @@ struct thread_stat {
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+struct perf_lock_type {
+ unsigned int flags;
+ /*
+ * Name of the lock flags (access), with delimiter ':'.
+ * For example, rwsem:R or rwsem:W.
+ */
+ const char *flags_name;
+ /* Name of the lock (type), for example, rwlock or rwsem. */
+ const char *lock_name;
+};
+
+extern const struct perf_lock_type lock_type_table[14];
+
struct evlist;
struct machine;
struct target;
--
2.56.0.307.g5fe7bd680c-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v6 04/11] perf lock contention: Extend get_lock_owner() for rt_mutex and mutex handoff
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (2 preceding siblings ...)
2026-10-22 20:29 ` [PATCH v6 03/11] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
@ 2026-10-22 20:29 ` Ian Rogers
2026-10-22 20:41 ` sashiko-bot
2026-10-22 20:29 ` [PATCH v6 05/11] perf bpf-filter: Add lock_held() sample filter for global mutexes Ian Rogers
` (6 subsequent siblings)
10 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Extend get_lock_owner() in tools/perf/util/bpf_skel/lock_data.h to:
- Support rt_mutex locks (LCB_F_RT) and CONFIG_PREEMPT_RT mutexes via
struct rt_mutex_base and struct rt_mutex___old, masking out the low
waiter bit (0x1UL).
- Return NULL when a mutex has MUTEX_FLAG_HANDOFF (0x4UL) set on
owner.counter and is in the middle of being handed off rather than
actively held.
Also add rw_semaphore.count and struct rt_mutex_base to the fallback
tools/perf/util/bpf_skel/vmlinux/vmlinux.h.
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/util/bpf_skel/lock_data.h | 62 ++++++++++++++++++----
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 ++
2 files changed, 58 insertions(+), 9 deletions(-)
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index 91d464ac18bd..43d8f5030bc1 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -104,11 +104,24 @@ extern struct rq runqueues __ksym __weak;
# define __has_builtin(x) 0
#endif
+struct mutex___non_rt {
+ atomic_long_t owner;
+} __attribute__((preserve_access_index));
+
+struct rt_mutex_base___new {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
+struct rt_mutex___old {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
struct rw_semaphore___old {
struct task_struct *owner;
} __attribute__((preserve_access_index));
struct rw_semaphore___new {
+ atomic_long_t count;
atomic_long_t owner;
} __attribute__((preserve_access_index));
@@ -146,9 +159,25 @@ static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flag
__u64 owner = 0;
if (flags & LCB_F_MUTEX) {
- struct mutex *mutex = (void *)lock;
-
- owner = BPF_CORE_READ(mutex, owner.counter);
+ struct mutex___non_rt *mutex = (void *)lock;
+
+ if (bpf_core_field_exists(mutex->owner.counter)) {
+ owner = BPF_CORE_READ(mutex, owner.counter);
+ if (owner & 0x4UL)
+ return NULL;
+ } else if (bpf_core_type_exists(struct rt_mutex_base___new)) {
+ struct rt_mutex_base___new *rtm = (void *)lock;
+
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ } else {
+ struct rt_mutex___old *rtm = (void *)lock;
+
+ if (bpf_core_field_exists(rtm->owner)) {
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ }
+ }
} else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
/*
* Support for the BPF_TYPE_MATCHES argument to the
@@ -158,20 +187,35 @@ static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flag
*/
#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
if (bpf_core_type_matches(struct rw_semaphore___old)) {
- struct rw_semaphore___old *rwsem = (void *)lock;
+ struct rw_semaphore___old *old = (void *)lock;
- owner = (unsigned long)BPF_CORE_READ(rwsem, owner);
+ owner = (unsigned long)BPF_CORE_READ(old, owner);
} else if (bpf_core_type_matches(struct rw_semaphore___new)) {
- struct rw_semaphore___new *rwsem = (void *)lock;
+ struct rw_semaphore___new *new = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
+ owner = BPF_CORE_READ(new, owner.counter);
}
#else
/* assume new struct */
- struct rw_semaphore *rwsem = (void *)lock;
+ struct rw_semaphore___new *new = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
+ if (bpf_core_field_exists(new->owner.counter))
+ owner = BPF_CORE_READ(new, owner.counter);
#endif
+ } else if (flags == LCB_F_RT) {
+ if (bpf_core_type_exists(struct rt_mutex_base___new)) {
+ struct rt_mutex_base___new *rtm = (void *)lock;
+
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ } else {
+ struct rt_mutex___old *rtm = (void *)lock;
+
+ if (bpf_core_field_exists(rtm->owner)) {
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ }
+ }
}
if (!owner)
diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
index 94e42b022ab1..b75ddc6846bb 100644
--- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
+++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
@@ -76,6 +76,7 @@ struct signal_struct {
} __attribute__((preserve_access_index));
struct rw_semaphore {
+ atomic_long_t count;
atomic_long_t owner;
} __attribute__((preserve_access_index));
@@ -83,6 +84,10 @@ struct mutex {
atomic_long_t owner;
} __attribute__((preserve_access_index));
+struct rt_mutex_base {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
struct kernfs_node {
u64 id;
} __attribute__((preserve_access_index));
--
2.56.0.307.g5fe7bd680c-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v6 05/11] perf bpf-filter: Add lock_held() sample filter for global mutexes
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (3 preceding siblings ...)
2026-10-22 20:29 ` [PATCH v6 04/11] perf lock contention: Extend get_lock_owner() for rt_mutex and mutex handoff Ian Rogers
@ 2026-10-22 20:29 ` Ian Rogers
2026-10-22 20:46 ` sashiko-bot
2026-10-22 22:55 ` Namhyung Kim
2026-10-22 20:29 ` [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types Ian Rogers
` (5 subsequent siblings)
10 siblings, 2 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Add BPF sample filter predicates lock_held(<lock>),
lock_held_read(<lock>) / lock_held_shared(<lock>), and
lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R'
and ':W' suffixes on <lock>) to allow recording perf event samples only
when a specified kernel lock is held (or not held with '!').
In this initial step, support global/static kernel mutex symbols (e.g.
lock_held(cgroup_mutex)) resolved via /proc/kallsyms. On sample
overflow, perf_sample_filter and perf_tp_sample_filter inspect the
mutex's owner field directly in kernel memory via get_lock_owner()
without attaching any BPF fentry/fexit hooks.
Subsequent commits extend lock_held() to other kernel lock types,
per-process and semi-global locks, slab cache locks, and RCU/SRCU.
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 16 +-
tools/perf/util/bpf-filter.c | 350 ++++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 47 ++-
tools/perf/util/bpf_skel/sample-filter.h | 36 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 54 +++
tools/perf/util/parse-events.c | 2 +
8 files changed, 530 insertions(+), 19 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index fcf1ea62bf1c..445cc98a6c33 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -210,7 +210,21 @@ OPTIONS
ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr,
code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat,
p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock,
- mem_dtlb, mem_blk, mem_hops, uid, gid
+ mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>),
+ lock_held_read(<lock>), lock_held_shared(<lock>),
+ lock_held_write(<lock>), lock_held_exclusive(<lock>)
+
+ The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters
+ samples based on whether the specified kernel lock is currently held
+ by the kernel thread. To differentiate between shared/read holding
+ and exclusive/write holding, use 'lock_held_read(<lock>)' (alias
+ 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' (alias
+ 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to <lock>.
+ <lock> can be a kernel mutex symbol name (e.g.
+ 'lock_held(cgroup_mutex)'). When filtering on a mutex symbol
+ ('*_mutex'), 'perf_sample_filter' inspects the lock's 'owner' field
+ directly in kernel memory on each sample without attaching any BPF
+ fentry/fexit hooks.
The <operator> can be one of:
==, !=, >, >=, <, <=, &
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 0d3c700d3dcd..639db6c03137 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -43,6 +43,9 @@
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
+#include <stdbool.h>
+#include <stddef.h>
+#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
@@ -50,14 +53,19 @@
#include <bpf/bpf.h>
#include <linux/err.h>
+#include <linux/kernel.h>
#include <linux/list.h>
+#include <linux/string.h>
+#include <linux/zalloc.h>
#include <api/fs/fs.h>
#include <internal/xyarray.h>
#include <perf/threadmap.h>
+#include <symbol/kallsyms.h>
#include "util/cap.h"
#include "util/debug.h"
#include "util/evsel.h"
+#include "util/symbol_conf.h"
#include "util/target.h"
#include "util/thread_map.h"
#include "util/bpf-utils.h"
@@ -126,7 +134,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
{
const struct perf_sample_info *info;
- if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
+ if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID ||
+ expr->term == PBF_TERM_LOCK_HELD) {
/* Not dependent on the sample_type as computed from a BPF helper. */
return 0;
}
@@ -169,9 +178,263 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
return -1;
}
+static int split_lock_name_mode(const char *name, int expr_part,
+ char *base_buf, size_t buf_sz,
+ bool *has_amp, unsigned int *mode_out)
+{
+ const char *bare;
+ size_t len;
+ unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK;
+ unsigned int suffix_mode = 0;
+
+ if (!name || !*name)
+ return -EINVAL;
+
+ *has_amp = (name[0] == '&');
+ bare = *has_amp ? name + 1 : name;
+ len = strlen(bare);
+ if (len == 0)
+ return -EINVAL;
+
+ if (len > 2 && !strcmp(bare + len - 2, ":R")) {
+ suffix_mode = PBF_LOCK_MODE_READ;
+ len -= 2;
+ } else if (len > 2 && !strcmp(bare + len - 2, ":W")) {
+ suffix_mode = PBF_LOCK_MODE_WRITE;
+ len -= 2;
+ }
+
+ if (len == 0 || len >= buf_sz)
+ return -EINVAL;
+
+ if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY &&
+ suffix_mode && expr_mode != suffix_mode)
+ return -EINVAL;
+
+ memcpy(base_buf, bare, len);
+ base_buf[len] = '\0';
+
+ if (suffix_mode)
+ *mode_out = suffix_mode;
+ else if (expr_mode)
+ *mode_out = expr_mode;
+ else
+ *mode_out = PBF_LOCK_MODE_ANY;
+
+ return 0;
+}
+
+struct lock_sym_cb_args {
+ const char *name;
+ u64 addr;
+ bool in_text;
+ bool in_data;
+ bool found;
+};
+
+static int find_lock_sym_cb(void *arg, const char *name, char type, u64 start)
+{
+ struct lock_sym_cb_args *args = arg;
+
+ if (!strcmp(name, "_text") || !strcmp(name, "_stext") ||
+ !strcmp(name, "_sinittext")) {
+ args->in_text = true;
+ return 0;
+ }
+ if (!strcmp(name, "_etext") || !strcmp(name, "_einittext")) {
+ args->in_text = false;
+ return 0;
+ }
+ if (!strcmp(name, "_sdata") || !strcmp(name, "__bss_start")) {
+ args->in_text = false;
+ args->in_data = true;
+ return 0;
+ }
+ if (!strcmp(name, "__bss_stop")) {
+ args->in_data = false;
+ return 0;
+ }
+
+ /*
+ * Ignore kernel text functions, module text functions, and
+ * absolute/undefined symbols. On CONFIG_LTO_CLANG kernels, vmlinux
+ * .data symbols between _sdata and _edata may have kallsyms type
+ * 'T'/'t', so allow 'T'/'t' within vmlinux data/bss (args->in_data).
+ */
+ if (args->in_text || type == 'A' || type == 'a' ||
+ type == 'U' || type == 'u' ||
+ (!args->in_data && kallsyms__is_function(type)))
+ return 0;
+
+ if (!strcmp(name, args->name)) {
+ args->found = true;
+ args->addr = start;
+ if (start != 0)
+ return 1;
+ }
+ return 0;
+}
+
+static int find_kernel_lock_sym(const char *name, u64 *addr)
+{
+ struct lock_sym_cb_args args = {
+ .name = name,
+ };
+ const char *kallsyms_filename = "/proc/kallsyms";
+
+ if (symbol_conf.kallsyms_name != NULL)
+ kallsyms_filename = symbol_conf.kallsyms_name;
+
+ if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0)
+ return -ENOENT;
+
+ if (!args.found)
+ return -ENOENT;
+
+ if (args.addr == 0) {
+ pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n",
+ name);
+ return -EACCES;
+ }
+
+ *addr = args.addr;
+ return 0;
+}
+
+static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
+ bool *has_lock_filter)
+{
+ if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name)
+ return;
+
+ *has_lock_filter = true;
+}
+
+static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter)
+{
+ struct perf_bpf_filter_expr *expr, *group;
+
+ *has_lock_filter = false;
+
+ list_for_each_entry(expr, &evsel->bpf_filters, list) {
+ prescan_one_lock_expr(expr, has_lock_filter);
+ if (expr->op == PBF_OP_GROUP_BEGIN) {
+ list_for_each_entry(group, &expr->groups, list)
+ prescan_one_lock_expr(group, has_lock_filter);
+ }
+ }
+}
+
+static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr,
+ enum perf_bpf_filter_op *op_out)
+{
+ switch (expr->op) {
+ case PBF_OP_EQ:
+ if (expr->val == 1) {
+ *op_out = PBF_OP_EQ;
+ return 0;
+ }
+ if (expr->val == 0) {
+ *op_out = PBF_OP_NEQ;
+ return 0;
+ }
+ break;
+ case PBF_OP_NEQ:
+ if (expr->val == 0) {
+ *op_out = PBF_OP_EQ;
+ return 0;
+ }
+ if (expr->val == 1) {
+ *op_out = PBF_OP_NEQ;
+ return 0;
+ }
+ break;
+ case PBF_OP_GT:
+ if (expr->val == 0) {
+ *op_out = PBF_OP_EQ;
+ return 0;
+ }
+ break;
+ case PBF_OP_GE:
+ if (expr->val == 1) {
+ *op_out = PBF_OP_EQ;
+ return 0;
+ }
+ break;
+ case PBF_OP_LT:
+ if (expr->val == 1) {
+ *op_out = PBF_OP_NEQ;
+ return 0;
+ }
+ break;
+ case PBF_OP_LE:
+ if (expr->val == 0) {
+ *op_out = PBF_OP_NEQ;
+ return 0;
+ }
+ break;
+ case PBF_OP_AND:
+ if (expr->val & 1) {
+ *op_out = PBF_OP_EQ;
+ return 0;
+ }
+ break;
+ case PBF_OP_GROUP_BEGIN:
+ case PBF_OP_GROUP_END:
+ case PBF_OP_DONE:
+ default:
+ break;
+ }
+ pr_err("Error: Invalid comparison for boolean lock_held(%s) predicate\n",
+ expr->lock_name ?: "");
+ return -EINVAL;
+}
+
+static int resolve_lock_expr(struct perf_bpf_filter_expr *expr,
+ struct perf_bpf_filter_entry *entry)
+{
+ const char *name = expr->lock_name;
+ char base[128];
+ bool has_amp;
+ unsigned int mode;
+ u64 addr = 0;
+ int ret;
+
+ ret = normalize_lock_expr_op(expr, &entry->op);
+ if (ret < 0)
+ return ret;
+
+ if (split_lock_name_mode(name, expr->part, base, sizeof(base),
+ &has_amp, &mode) < 0 || has_amp) {
+ pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: "");
+ return -EINVAL;
+ }
+
+ if (str_ends_with(base, "mutex") &&
+ !str_ends_with(base, "rt_mutex") &&
+ !str_ends_with(base, "rtmutex")) {
+ ret = find_kernel_lock_sym(base, &addr);
+ if (ret == 0) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_ADDR | mode | PBF_LOCK_DIRECT_MUTEX;
+ entry->value = addr;
+ return 0;
+ }
+ if (ret == -EACCES)
+ return ret;
+ }
+
+ pr_err("Error: Cannot resolve kernel lock '%s'\n", name);
+ return -EINVAL;
+}
+
static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry)
{
- int i = 0;
+ int i = 0, ret;
struct perf_bpf_filter_expr *expr;
list_for_each_entry(expr, &evsel->bpf_filters, list) {
@@ -185,6 +448,11 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].part = expr->part;
entry[i].term = expr->term;
entry[i].value = expr->val;
+ if (expr->term == PBF_TERM_LOCK_HELD) {
+ ret = resolve_lock_expr(expr, &entry[i]);
+ if (ret < 0)
+ return ret;
+ }
i++;
if (expr->op == PBF_OP_GROUP_BEGIN) {
@@ -198,6 +466,11 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].part = group->part;
entry[i].term = group->term;
entry[i].value = group->val;
+ if (group->term == PBF_TERM_LOCK_HELD) {
+ ret = resolve_lock_expr(group, &entry[i]);
+ if (ret < 0)
+ return ret;
+ }
i++;
}
@@ -440,20 +713,24 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
+ bool has_lock_filter = false;
entry = calloc(MAX_FILTERS, sizeof(*entry));
if (entry == NULL)
return -1;
- ret = get_filter_entries(evsel, entry);
- if (ret < 0) {
- pr_err("Failed to process filter entries\n");
+ prescan_lock_filters(evsel, &has_lock_filter);
+
+ if ((has_lock_filter || is_tracepoint) && geteuid() != 0 &&
+ !perf_cap__capable(CAP_BPF)) {
+ pr_err("Error: BPF filter requires CAP_BPF capability\n");
+ ret = -EPERM;
goto err;
}
- if (is_tracepoint && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
- pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n");
- ret = -EPERM;
+ ret = get_filter_entries(evsel, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
goto err;
}
@@ -513,6 +790,9 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
+ if (has_lock_filter)
+ skel->bss->lock_enabled = 1;
+
i = 0;
fd = bpf_map__fd(skel->maps.filters);
@@ -571,6 +851,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return ret;
}
+void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr)
+{
+ struct perf_bpf_filter_expr *group, *tmp;
+
+ if (!expr)
+ return;
+
+ list_for_each_entry_safe(group, tmp, &expr->groups, list) {
+ list_del(&group->list);
+ perf_bpf_filter_expr__delete(group);
+ }
+ zfree(&expr->lock_name);
+ free(expr);
+}
+
int perf_bpf_filter__destroy(struct evsel *evsel)
{
struct perf_bpf_filter_expr *expr, *tmp;
@@ -578,7 +873,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel)
list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) {
list_del(&expr->list);
- free(expr);
+ perf_bpf_filter_expr__delete(expr);
}
sample_filter_bpf__destroy(evsel->bpf_skel);
@@ -630,7 +925,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
{
struct perf_bpf_filter_expr *expr;
- expr = malloc(sizeof(*expr));
+ expr = zalloc(sizeof(*expr));
if (expr != NULL) {
expr->term = term;
expr->part = part;
@@ -641,6 +936,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
return expr;
}
+struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
+ int mode,
+ enum perf_bpf_filter_op op,
+ unsigned long val)
+{
+ struct perf_bpf_filter_expr *expr;
+
+ if (!lock_name || !*lock_name)
+ return NULL;
+
+ expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val);
+ if (!expr)
+ return NULL;
+
+ expr->lock_name = strdup(lock_name);
+ if (!expr->lock_name) {
+ free(expr);
+ return NULL;
+ }
+ return expr;
+}
+
static bool check_bpf_filter_capable(void)
{
int fd;
@@ -661,14 +978,11 @@ static bool check_bpf_filter_capable(void)
return false;
}
-int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
+int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
{
YY_BUFFER_STATE buffer;
int ret;
- if (!check_bpf_filter_capable())
- return -EPERM;
-
buffer = perf_bpf_filter__scan_string(str);
ret = perf_bpf_filter_parse(expr_head);
@@ -680,6 +994,14 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
return ret;
}
+int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
+{
+ if (!check_bpf_filter_capable())
+ return -EPERM;
+
+ return __perf_bpf_filter__parse(expr_head, str);
+}
+
/*
* perf_bpf_filter__pin() and perf_bpf_filter__unpin() only load
* perf_sample_filter and disable autoload on other programs in the skeleton
diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h
index 818c554b91b2..fda95ff226b3 100644
--- a/tools/perf/util/bpf-filter.h
+++ b/tools/perf/util/bpf-filter.h
@@ -14,6 +14,13 @@ struct perf_bpf_filter_expr {
int part;
enum perf_bpf_filter_term term;
unsigned long val;
+ /*
+ * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions
+ * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R").
+ * Resolved into `entry.part` and `entry.target` during
+ * perf_bpf_filter__prepare().
+ */
+ char *lock_name;
};
struct evsel;
@@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
int part,
enum perf_bpf_filter_op op,
unsigned long val);
+struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
+ int mode,
+ enum perf_bpf_filter_op op,
+ unsigned long val);
+void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr);
+int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
int perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target);
int perf_bpf_filter__destroy(struct evsel *evsel);
diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l
index 1be9df6550fc..b4022110f1e5 100644
--- a/tools/perf/util/bpf-filter.l
+++ b/tools/perf/util/bpf-filter.l
@@ -1,9 +1,12 @@
%option prefix="perf_bpf_filter_"
%option noyywrap
+%x LOCK_ARG
+
%{
#include <stdio.h>
#include <stdlib.h>
+#include <string.h>
#include <linux/perf_event.h>
#include "bpf-filter.h"
@@ -76,7 +79,7 @@ static int path_or_error(void)
num_dec [0-9]+
num_hex 0[Xx][0-9a-fA-F]+
space [ \t]+
-path [^ \t\n,]+
+path [^ \t\n,()!=>&|<]+
ident [_a-zA-Z][_a-zA-Z0-9]+
%%
@@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); }
uid { return sample(PBF_TERM_UID); }
gid { return sample(PBF_TERM_GID); }
cgroup { return sample_path(PBF_TERM_CGROUP); }
-
+lock_held[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY;
+ return BFT_LOCK_HELD;
+ }
+(lock_held_read|lock_held_shared)[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ;
+ return BFT_LOCK_HELD;
+ }
+(lock_held_write|lock_held_exclusive)[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE;
+ return BFT_LOCK_HELD;
+ }
+
+<LOCK_ARG>{space} { }
+<LOCK_ARG>[^ \t\n()]+ {
+ perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text);
+ return BFT_LOCK_NAME;
+ }
+<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; }
+<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; }
+
+"!" { return '!'; }
"==" { return operator(PBF_OP_EQ); }
"!=" { return operator(PBF_OP_NEQ); }
">" { return operator(PBF_OP_GT); }
diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y
index 5a79a8e7a45b..5ae05d4608c6 100644
--- a/tools/perf/util/bpf-filter.y
+++ b/tools/perf/util/bpf-filter.y
@@ -8,9 +8,11 @@
#endif
#include <stdio.h>
+#include <stdlib.h>
#include <string.h>
#include <linux/compiler.h>
#include <linux/list.h>
+#include <linux/zalloc.h>
#include "bpf-filter.h"
#include "cgroup.h"
@@ -31,6 +33,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
{
unsigned long num;
char *path;
+ char *lock_name;
+ struct {
+ char *name;
+ int mode;
+ } lock;
struct {
enum perf_bpf_filter_term term;
int part;
@@ -40,12 +47,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
}
%token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH
+%token BFT_LOCK_HELD BFT_LOCK_NAME
%type <expr> filter_term filter_expr
-%destructor { free ($$); } <expr>
+%destructor { perf_bpf_filter_expr__delete($$); } <expr>
+%destructor { free($$); } <lock_name>
+%destructor { free($$.name); } <lock>
%type <sample> BFT_SAMPLE BFT_SAMPLE_PATH
%type <op> BFT_OP
-%type <num> BFT_NUM
+%type <num> BFT_NUM BFT_LOCK_HELD
%type <path> BFT_PATH
+%type <lock_name> BFT_LOCK_NAME
+%type <lock> lock_term
%%
@@ -105,5 +117,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH
$$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id);
cgroup__put(cgrp);
}
+|
+lock_term
+{
+ $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1);
+ zfree(&$1.name);
+ if (!$$)
+ YYERROR;
+}
+|
+'!' lock_term
+{
+ $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0);
+ zfree(&$2.name);
+ if (!$$)
+ YYERROR;
+}
+|
+lock_term BFT_OP BFT_NUM
+{
+ $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3);
+ zfree(&$1.name);
+ if (!$$)
+ YYERROR;
+}
+
+lock_term:
+BFT_LOCK_HELD BFT_LOCK_NAME ')'
+{
+ $$.name = $2;
+ $$.mode = $1;
+}
%%
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index 683fec85e71e..ca1cf586ed6e 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -53,8 +53,44 @@ enum perf_bpf_filter_term {
/* Terms computed from BPF helpers. */
PBF_TERM_UID,
PBF_TERM_GID,
+ /*
+ * Kernel lock held predicate (`lock_held(...)`). Evaluates whether the
+ * specified kernel lock is currently held by the running task/CPU.
+ * The lock specification kind is stored in `entry.part`
+ * (`enum perf_bpf_lock_target_kind`), `entry.op` is normalized to
+ * `PBF_OP_EQ` (match when held) or `PBF_OP_NEQ` (match when not held),
+ * and the resolved target address is stored in `entry.value`.
+ */
+ PBF_TERM_LOCK_HELD,
};
+/*
+ * Lock target categories for PBF_TERM_LOCK_HELD filters.
+ * Stored in `perf_bpf_filter_entry.part`.
+ */
+enum perf_bpf_lock_target_kind {
+ PBF_LOCK_NONE = 0,
+ /*
+ * Specific global or static lock address resolved from kallsyms.
+ * `entry.value` holds the kernel address.
+ */
+ PBF_LOCK_ADDR,
+};
+
+/*
+ * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD:
+ * Bits 0..3: `enum perf_bpf_lock_target_kind`
+ * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding)
+ * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding)
+ * Bit 6: PBF_LOCK_DIRECT_MUTEX (inspect struct mutex owner)
+ */
+#define PBF_LOCK_KIND_MASK 0x0fU
+#define PBF_LOCK_MODE_READ (1U << 4)
+#define PBF_LOCK_MODE_WRITE (1U << 5)
+#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE)
+#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY
+#define PBF_LOCK_DIRECT_MUTEX (1U << 6)
+
/* BPF map entry for filtering */
struct perf_bpf_filter_entry {
enum perf_bpf_filter_op op;
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index a190374ed87a..42556ab0e2d5 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -5,6 +5,7 @@
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
+#include "lock_data.h"
#include "sample-filter.h"
/* BPF map that will be filled by user space */
@@ -45,8 +46,57 @@ struct lost_count {
volatile const int use_idx_hash;
+/* Control flag set by userspace when lock_held(...) filters are active. */
+int lock_enabled;
+
void *bpf_cast_to_kern_ctx(void *) __ksym;
+static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __u32 part)
+{
+ if (!curr)
+ return false;
+
+ if ((part & PBF_LOCK_DIRECT_MUTEX) &&
+ (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr)
+ return true;
+ return false;
+}
+
+static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry)
+{
+ __u64 curr = bpf_get_current_task();
+ __u32 kind = entry->part & PBF_LOCK_KIND_MASK;
+ __u32 mode = entry->part & PBF_LOCK_MODE_MASK;
+
+ if (!mode)
+ mode = PBF_LOCK_MODE_ANY;
+
+ switch (kind) {
+ case PBF_LOCK_ADDR:
+ if ((mode & PBF_LOCK_MODE_WRITE) &&
+ is_direct_lock_owner(entry->value, curr, entry->part))
+ return true;
+ return false;
+
+ default:
+ return false;
+ }
+}
+
+static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry)
+{
+ /*
+ * Pinned filters (use_idx_hash != 0) share a world-writable filters
+ * map and do not support lock_held(). Reject evaluation when pinned
+ * or when lock_enabled is not set so unprivileged users cannot probe
+ * arbitrary kernel addresses via entry->value.
+ */
+ if (use_idx_hash || !lock_enabled)
+ return entry->op == PBF_OP_EQ ? ~entry->value : entry->value;
+
+ return is_lock_held(entry) ? entry->value : ~entry->value;
+}
+
/* new kernel perf_sample_data definition */
struct perf_sample_data___new {
__u64 sample_flags;
@@ -175,6 +225,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
return bpf_get_current_uid_gid() & 0xFFFFFFFF;
case PBF_TERM_GID:
return bpf_get_current_uid_gid() >> 32;
+ case PBF_TERM_LOCK_HELD:
+ return check_lock_held(entry);
case PBF_TERM_NONE:
case __PBF_UNUSED_TERM4:
case __PBF_UNUSED_TERM5:
@@ -209,6 +261,8 @@ static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry
return bpf_get_current_uid_gid() & 0xFFFFFFFF;
case PBF_TERM_GID:
return bpf_get_current_uid_gid() >> 32;
+ case PBF_TERM_LOCK_HELD:
+ return check_lock_held(entry);
default:
return 0;
}
diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index d6bab89ade5d..da687e9d1a67 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -2621,6 +2621,8 @@ static bool is_possible_tp_filter(struct evsel *evsel, const char *str)
{
static const char *const bpf_tp_terms[] = {
"uid", "gid", "tid", "pid", "cpu", "time", "cgroup",
+ "lock_held", "lock_held_read", "lock_held_shared",
+ "lock_held_write", "lock_held_exclusive",
};
for (size_t i = 0; i < ARRAY_SIZE(bpf_tp_terms); i++) {
--
2.56.0.307.g5fe7bd680c-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (4 preceding siblings ...)
2026-10-22 20:29 ` [PATCH v6 05/11] perf bpf-filter: Add lock_held() sample filter for global mutexes Ian Rogers
@ 2026-10-22 20:29 ` Ian Rogers
2026-10-22 20:55 ` sashiko-bot
2026-10-22 22:56 ` Namhyung Kim
2026-10-22 20:29 ` [PATCH v6 07/11] perf bpf-filter: Support per-process and semi-global locks in lock_held() Ian Rogers
` (4 subsequent siblings)
10 siblings, 2 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Extend the lock_held() BPF sample filter beyond global mutexes to
support:
- Other global/static kernel lock symbols (rwsem, spinlock, rwlock,
rt-mutex, percpu-rwsem, semaphore) and hexadecimal kernel lock
addresses.
- Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W,
rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W,
semaphore.
For rt-mutexes and write-locked rwsems, perf_sample_filter inspects the
lock's owner field directly in kernel memory. For locks and modes without
an authoritative per-task owner pointer (spinlocks, rwlocks, semaphores,
reader-locked rwsems, percpu-rwsems, and lock type filters), attach BPF
fentry/fexit hooks on kernel lock acquire and release functions to track
held locks in the held_locks BPF LRU hash map.
Note that counting semaphores (down*() / up()) do not enforce same-task
ownership in the kernel: when a semaphore is used as a cross-task
completion/signaling mechanism, fexit/down and fentry/up will not pair
on the same task ID. Semaphore hooks are therefore only enabled when
'semaphore' or a '*_sem' symbol (such as console_sem) is explicitly
specified, and held_locks uses BPF_MAP_TYPE_LRU_HASH so any unpaired
cross-task entries are automatically evicted.
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 53 +-
tools/perf/util/bpf-filter.c | 596 ++++++++++++-
tools/perf/util/bpf_skel/sample-filter.h | 17 +-
tools/perf/util/bpf_skel/sample_filter.bpf.c | 851 ++++++++++++++++++-
4 files changed, 1465 insertions(+), 52 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 445cc98a6c33..9fb5d15de135 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -216,15 +216,50 @@ OPTIONS
The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters
samples based on whether the specified kernel lock is currently held
- by the kernel thread. To differentiate between shared/read holding
- and exclusive/write holding, use 'lock_held_read(<lock>)' (alias
- 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' (alias
- 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to <lock>.
- <lock> can be a kernel mutex symbol name (e.g.
- 'lock_held(cgroup_mutex)'). When filtering on a mutex symbol
- ('*_mutex'), 'perf_sample_filter' inspects the lock's 'owner' field
- directly in kernel memory on each sample without attaching any BPF
- fentry/fexit hooks.
+ by the kernel thread or CPU. To differentiate between shared/read
+ holding and exclusive/write holding, use 'lock_held_read(<lock>)'
+ (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)'
+ (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to
+ <lock> (e.g. 'lock_held(rwsem:R)' or 'lock_held(rwsem:W)').
+ <lock> can be a kernel lock symbol name (e.g.
+ 'lock_held(cgroup_mutex)'), a hexadecimal lock address, or a lock type
+ ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
+ 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
+ 'semaphore'). Both read/shared and write/exclusive tracking are
+ supported for 'rwsem' and 'rwlock'; 'mutex', 'spinlock', 'rt-mutex',
+ 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support
+ write/exclusive tracking only. Note that 'semaphore' tracking
+ ('down*()' / 'up()') assumes binary mutex-like usage where the
+ acquiring task also releases the semaphore (such as 'console_sem'),
+ rather than cross-task completion signaling.
+
+ For example, 'lock_held()' can be combined with memory allocation or
+ reclaim events to detect allocations or direct memory reclaim occurring
+ while a kernel lock is held (which can stall lock waiters for extended
+ periods):
+
+ # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+ # perf record -a -g -e kmem:mm_page_alloc \
+ --filter 'gfp_flags & 0x400' \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+ # perf record -a -g -e major-faults \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+
+ When filtering on a specific mutex symbol ('*_mutex') or rt-mutex
+ symbol ('*_rt_mutex'), 'perf_sample_filter' inspects the lock's
+ 'owner' field directly in kernel memory on each sample without attaching
+ any BPF fentry/fexit hooks. For locks that do not maintain an
+ authoritative per-task owner pointer (such as spinlocks, rwlocks,
+ semaphores, reader-locked rwsems, and lock type filters), it attaches
+ BPF fentry/fexit programs to kernel lock acquire and release functions
+ to track held locks in a BPF map. These BPF hooks run on every kernel
+ lock acquisition and release for the attached lock family, even when no
+ perf sample is being taken. To minimize this overhead, only the hooks
+ for the relevant lock family and mode are attached when known, whereas
+ raw hex addresses or symbols without a recognizable lock-type suffix
+ attach hooks across all lock families (including high-frequency
+ spinlocks).
The <operator> can be one of:
==, !=, >, >=, <, <=, &
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 639db6c03137..8797f67698f7 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -52,6 +52,7 @@
#include <sys/stat.h>
#include <bpf/bpf.h>
+#include <bpf/btf.h>
#include <linux/err.h>
#include <linux/kernel.h>
#include <linux/list.h>
@@ -65,6 +66,7 @@
#include "util/cap.h"
#include "util/debug.h"
#include "util/evsel.h"
+#include "util/lock-contention.h"
#include "util/symbol_conf.h"
#include "util/target.h"
#include "util/thread_map.h"
@@ -74,6 +76,7 @@
#include <util/bpf-filter-flex.h>
#include <util/bpf-filter-bison.h>
+#include "bpf_skel/lock_data.h"
#include "bpf_skel/sample-filter.h"
#include "bpf_skel/sample_filter.skel.h"
@@ -178,6 +181,153 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
return -1;
}
+/*
+ * Program groups for `lock_held(...)` BPF tracing hooks.
+ *
+ * When a `lock_held(...)` filter is specified, only the BPF tracing program
+ * groups required for the requested lock(s) are autoloaded and attached.
+ * If no `lock_held(...)` filter is present, none of these programs are loaded.
+ */
+enum lock_prog_group {
+ LOCK_PROG_MUTEX = 1U << 0,
+ LOCK_PROG_RWSEM_READ = 1U << 1,
+ LOCK_PROG_RWSEM_WRITE = 1U << 2,
+ LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE,
+ LOCK_PROG_SPIN = 1U << 3,
+ LOCK_PROG_RWLOCK_READ = 1U << 4,
+ LOCK_PROG_RWLOCK_WRITE = 1U << 5,
+ LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE,
+ LOCK_PROG_RTMUTEX = 1U << 6,
+ LOCK_PROG_SEM = 1U << 7,
+ LOCK_PROG_PERCPU_RWSEM = 1U << 8,
+ LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM,
+};
+
+static unsigned int lock_flags_to_prog_group(unsigned int flags)
+{
+ if (flags & LCB_F_MUTEX)
+ return LOCK_PROG_MUTEX;
+ if (flags & LCB_F_RT)
+ return LOCK_PROG_RTMUTEX;
+ if (flags & LCB_F_PERCPU)
+ return LOCK_PROG_PERCPU_RWSEM;
+ if (flags == LCB_F_SPIN)
+ return LOCK_PROG_SPIN;
+ if (flags & LCB_F_SPIN)
+ return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE;
+ if (flags & LCB_F_READ)
+ return LOCK_PROG_RWSEM_READ;
+ if (flags & LCB_F_WRITE)
+ return LOCK_PROG_RWSEM_WRITE;
+ return LOCK_PROG_SEM;
+}
+
+struct lock_prog_desc {
+ size_t prog_offset;
+ const char *btf_func;
+ unsigned int group;
+};
+
+#define LOCK_FPROG(field, func, grp) \
+ { offsetof(struct sample_filter_bpf, progs.field), func, grp }
+
+static const struct lock_prog_desc lock_progs[] = {
+ /* Mutex */
+ LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
+ LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
+ /* RW semaphore */
+ LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested",
+ LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested",
+ LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM),
+ LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE),
+ /* Spinlock */
+ LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN),
+ /* RWlock */
+ LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE),
+ /* RT-Mutex */
+ LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible",
+ LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX),
+ /* Semaphore */
+ LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM),
+ LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM),
+ /* Percpu-RWsem */
+ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
+ LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+};
+
+#undef LOCK_FPROG
+
static int split_lock_name_mode(const char *name, int expr_part,
char *base_buf, size_t buf_sz,
bool *has_amp, unsigned int *mode_out)
@@ -224,6 +374,72 @@ static int split_lock_name_mode(const char *name, int expr_part,
return 0;
}
+/*
+ * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R",
+ * "spinlock", "percpu-rwsem").
+ *
+ * A lock type name may correspond to up to two flag combinations (e.g.
+ * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed
+ * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is
+ * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`.
+ */
+static bool parse_lock_type_name(const char *name, unsigned int mode,
+ struct sample_filter_bpf *skel,
+ u64 *encoded_pair, unsigned int *prog_groups)
+{
+ bool found = false;
+ u32 flag1 = 0xff, flag2 = 0xff;
+ unsigned int groups = 0;
+ const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name;
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) {
+ u32 f = lock_type_table[i].flags;
+ bool is_read = (f & LCB_F_READ) != 0;
+
+ if (strcmp(lock_type_table[i].lock_name, lookup) &&
+ strcmp(lock_type_table[i].flags_name, name))
+ continue;
+
+ /*
+ * percpu_down_read() / percpu_up_read() fast paths are inlined
+ * per-CPU counter increments without function entry points, so
+ * only writer holding (percpu_down_write / percpu_up_write) can
+ * be tracked for percpu-rwsem. rwlock-rt is also not hooked.
+ */
+ if ((f & LCB_F_PERCPU) && is_read)
+ continue;
+ if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE)))
+ continue;
+ if ((f & LCB_F_MUTEX) && (f & LCB_F_SPIN))
+ continue;
+
+ if (is_read && !(mode & PBF_LOCK_MODE_READ))
+ continue;
+ if (!is_read && !(mode & PBF_LOCK_MODE_WRITE))
+ continue;
+
+ if (!found)
+ flag1 = f;
+ else if (f != flag1)
+ flag2 = f;
+ if (skel && f < LCB_F_TYPE_MAX) {
+ skel->bss->has_lock_type = 1;
+ skel->bss->lock_type_filter[f] = 1;
+ }
+ groups |= lock_flags_to_prog_group(f);
+ found = true;
+ }
+
+ if (!found)
+ return false;
+
+ if (encoded_pair)
+ *encoded_pair = flag1 | ((u64)flag2 << 8);
+ if (prog_groups)
+ *prog_groups |= groups;
+ return true;
+}
+
struct lock_sym_cb_args {
const char *name;
u64 addr;
@@ -301,26 +517,225 @@ static int find_kernel_lock_sym(const char *name, u64 *addr)
return 0;
}
+static bool parse_lock_hex_addr(const char *str, u64 *addr)
+{
+ char *end;
+ unsigned long long val;
+
+ if (!str || !*str)
+ return false;
+
+ errno = 0;
+ val = strtoull(str, &end, 16);
+ if (*end != '\0' || errno != 0 || val == 0)
+ return false;
+
+ *addr = val;
+ return true;
+}
+
+static bool btf_has_func(struct btf *btf, const char *name)
+{
+ return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0;
+}
+
+static unsigned int supported_lock_prog_groups(struct btf *btf)
+{
+ unsigned int supported = 0;
+
+ if (!btf)
+ return 0;
+
+ if ((btf_has_func(btf, "mutex_lock") || btf_has_func(btf, "mutex_lock_nested")) &&
+ btf_has_func(btf, "mutex_unlock"))
+ supported |= LOCK_PROG_MUTEX;
+
+ if ((btf_has_func(btf, "down_read") || btf_has_func(btf, "down_read_nested")) &&
+ btf_has_func(btf, "up_read"))
+ supported |= LOCK_PROG_RWSEM_READ;
+
+ if ((btf_has_func(btf, "down_write") || btf_has_func(btf, "down_write_nested")) &&
+ btf_has_func(btf, "up_write"))
+ supported |= LOCK_PROG_RWSEM_WRITE;
+
+ /*
+ * Require both acquire and all unlock entry points in BTF so we never
+ * attach spinlock/rwlock acquire hooks when unlock functions are
+ * inlined (e.g. !CONFIG_UNINLINE_SPIN_UNLOCK or CONFIG_INLINE_*_UNLOCK).
+ */
+ if ((btf_has_func(btf, "_raw_spin_lock") || btf_has_func(btf, "_raw_spin_lock_nested")) &&
+ btf_has_func(btf, "_raw_spin_unlock") &&
+ btf_has_func(btf, "_raw_spin_unlock_irq") &&
+ btf_has_func(btf, "_raw_spin_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_spin_unlock_bh"))
+ supported |= LOCK_PROG_SPIN;
+
+ if (btf_has_func(btf, "_raw_read_lock") &&
+ btf_has_func(btf, "_raw_read_unlock") &&
+ btf_has_func(btf, "_raw_read_unlock_irq") &&
+ btf_has_func(btf, "_raw_read_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_read_unlock_bh"))
+ supported |= LOCK_PROG_RWLOCK_READ;
+
+ if ((btf_has_func(btf, "_raw_write_lock") || btf_has_func(btf, "_raw_write_lock_nested")) &&
+ btf_has_func(btf, "_raw_write_unlock") &&
+ btf_has_func(btf, "_raw_write_unlock_irq") &&
+ btf_has_func(btf, "_raw_write_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_write_unlock_bh"))
+ supported |= LOCK_PROG_RWLOCK_WRITE;
+
+ if ((btf_has_func(btf, "rt_mutex_lock") || btf_has_func(btf, "rt_mutex_lock_nested")) &&
+ btf_has_func(btf, "rt_mutex_unlock"))
+ supported |= LOCK_PROG_RTMUTEX;
+
+ if (btf_has_func(btf, "down") && btf_has_func(btf, "up"))
+ supported |= LOCK_PROG_SEM;
+
+ if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write"))
+ supported |= LOCK_PROG_PERCPU_RWSEM;
+
+ return supported;
+}
+
+/*
+ * Enable or disable autoload for each lock-tracking BPF program in the
+ * skeleton before `sample_filter_bpf__load()` is called.
+ *
+ * Each program is enabled only if its group is in `needed_groups` and its
+ * target kernel function or tracepoint exists in the running kernel's BTF.
+ */
+static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
+ unsigned int needed_groups,
+ unsigned int required_groups,
+ bool is_tracepoint)
+{
+ unsigned int supported_groups = supported_lock_prog_groups(btf);
+ unsigned int enabled_progs = 0;
+
+ bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
+ bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
+
+ if ((required_groups & supported_groups) != required_groups ||
+ (needed_groups && !(needed_groups & supported_groups))) {
+ pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n");
+ return -ENOTSUP;
+ }
+
+ needed_groups &= supported_groups;
+ if (skel)
+ skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ);
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) {
+ struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset;
+ struct bpf_program *prog = *progp;
+ bool enable = false;
+
+ if ((needed_groups & lock_progs[i].group) && btf &&
+ lock_progs[i].btf_func) {
+ if (btf__find_by_name_kind(btf, lock_progs[i].btf_func,
+ BTF_KIND_FUNC) > 0)
+ enable = true;
+ }
+
+ if (enable)
+ enabled_progs++;
+ bpf_program__set_autoload(prog, enable);
+ }
+
+ if (needed_groups && !enabled_progs) {
+ pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
+ return -ENOTSUP;
+ }
+ return 0;
+}
+
+/*
+ * Pre-scan a single filter expression prior to loading the BPF skeleton in
+ * order to determine which BPF maps need to be sized and which lock tracing
+ * program groups need to be autoloaded.
+ */
static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
- bool *has_lock_filter)
+ bool *has_lock_filter,
+ unsigned int *prog_groups,
+ unsigned int *required_groups)
{
+ char base[128];
+ bool has_amp;
+ unsigned int mode, groups = 0;
+ u64 dummy;
+
if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name)
return;
*has_lock_filter = true;
+ if (split_lock_name_mode(expr->lock_name, expr->part, base,
+ sizeof(base), &has_amp, &mode) < 0 || has_amp)
+ return;
+
+ if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) {
+ *prog_groups |= groups;
+ *required_groups |= groups;
+ return;
+ }
+
+ if (parse_lock_hex_addr(base, &dummy)) {
+ *prog_groups |= LOCK_PROG_ALL;
+ return;
+ }
+
+ if (str_ends_with(base, "rt_mutex") || str_ends_with(base, "rtmutex") ||
+ str_ends_with(base, "mutex")) {
+ /* Inspected directly via lock->owner; no tracing hooks needed. */
+ return;
+ } else if (str_ends_with(base, "percpu_rwsem") ||
+ str_ends_with(base, "pcpu_rwsem")) {
+ *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
+ *required_groups |= LOCK_PROG_PERCPU_RWSEM;
+ } else if (str_ends_with(base, "rwsem")) {
+ if (mode & PBF_LOCK_MODE_READ) {
+ *prog_groups |= LOCK_PROG_RWSEM_READ;
+ *required_groups |= LOCK_PROG_RWSEM_READ;
+ }
+ if (mode & PBF_LOCK_MODE_WRITE) {
+ *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM;
+ *required_groups |= LOCK_PROG_RWSEM_WRITE;
+ }
+ } else if (str_ends_with(base, "spinlock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ } else if (str_ends_with(base, "sem")) {
+ *prog_groups |= LOCK_PROG_SEM;
+ *required_groups |= LOCK_PROG_SEM;
+ } else if (str_ends_with(base, "rwlock")) {
+ if (mode & PBF_LOCK_MODE_READ)
+ groups |= LOCK_PROG_RWLOCK_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ groups |= LOCK_PROG_RWLOCK_WRITE;
+ *prog_groups |= groups;
+ *required_groups |= groups;
+ } else {
+ *prog_groups |= LOCK_PROG_ALL;
+ }
}
-static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter)
+static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter,
+ unsigned int *prog_groups,
+ unsigned int *required_groups)
{
struct perf_bpf_filter_expr *expr, *group;
*has_lock_filter = false;
+ *prog_groups = 0;
+ *required_groups = 0;
list_for_each_entry(expr, &evsel->bpf_filters, list) {
- prescan_one_lock_expr(expr, has_lock_filter);
+ prescan_one_lock_expr(expr, has_lock_filter, prog_groups,
+ required_groups);
if (expr->op == PBF_OP_GROUP_BEGIN) {
- list_for_each_entry(group, &expr->groups, list)
- prescan_one_lock_expr(group, has_lock_filter);
+ list_for_each_entry(group, &expr->groups, list) {
+ prescan_one_lock_expr(group, has_lock_filter,
+ prog_groups, required_groups);
+ }
}
}
}
@@ -390,14 +805,42 @@ static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr,
return -EINVAL;
}
-static int resolve_lock_expr(struct perf_bpf_filter_expr *expr,
+static bool is_percpu_rwsem_sym(struct btf *btf, const char *name)
+{
+ char rc_name[160];
+
+ if (str_ends_with(name, "percpu_rwsem") ||
+ str_ends_with(name, "pcpu_rwsem"))
+ return true;
+
+ if (!btf)
+ return false;
+
+ snprintf(rc_name, sizeof(rc_name), "__percpu_rwsem_rc_%s", name);
+ return btf__find_by_name_kind(btf, rc_name, BTF_KIND_VAR) > 0;
+}
+
+/*
+ * Resolve a `lock_held(<name>)` filter expression into `entry->op`,
+ * `entry->part` (`enum perf_bpf_lock_target_kind`), and `entry->value`,
+ * and populate the corresponding BPF filter maps and BSS control flags.
+ *
+ * Resolution order:
+ * 1. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`,
+ * `rwlock`, `percpu-rwsem`, `semaphore`, etc.
+ * 2. Hexadecimal lock address.
+ * 3. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`).
+ */
+static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf,
+ struct perf_bpf_filter_expr *expr,
struct perf_bpf_filter_entry *entry)
{
const char *name = expr->lock_name;
char base[128];
bool has_amp;
unsigned int mode;
- u64 addr = 0;
+ u64 addr = 0, encoded_pair = 0;
+ u8 one = 1;
int ret;
ret = normalize_lock_expr_op(expr, &entry->op);
@@ -410,29 +853,79 @@ static int resolve_lock_expr(struct perf_bpf_filter_expr *expr,
return -EINVAL;
}
- if (str_ends_with(base, "mutex") &&
- !str_ends_with(base, "rt_mutex") &&
- !str_ends_with(base, "rtmutex")) {
- ret = find_kernel_lock_sym(base, &addr);
- if (ret == 0) {
- if (!(mode & PBF_LOCK_MODE_WRITE)) {
- pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
- name);
- return -EINVAL;
- }
- entry->part = PBF_LOCK_ADDR | mode | PBF_LOCK_DIRECT_MUTEX;
- entry->value = addr;
- return 0;
+ if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
+ entry->part = PBF_LOCK_TYPE | mode;
+ entry->value = encoded_pair;
+ return 0;
+ }
+
+ if (parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) {
+ pr_err("Error: Lock type '%s' does not support the requested read/write mode\n",
+ name);
+ return -EINVAL;
+ }
+
+ if (parse_lock_hex_addr(base, &addr)) {
+ entry->part = PBF_LOCK_ADDR | mode;
+ entry->value = addr;
+ if (skel) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
}
- if (ret == -EACCES)
- return ret;
+ return 0;
}
+ if (btf_has_func(btf, base)) {
+ pr_err("Error: '%s' is a kernel function, not a lock\n", name);
+ return -EINVAL;
+ }
+
+ ret = find_kernel_lock_sym(base, &addr);
+ if (ret == 0) {
+ unsigned int direct_flag = 0;
+ bool is_pcpu_rwsem = is_percpu_rwsem_sym(btf, base);
+
+ if (str_ends_with(base, "rt_mutex") || str_ends_with(base, "rtmutex"))
+ direct_flag = PBF_LOCK_DIRECT_RTMUTEX;
+ else if (str_ends_with(base, "mutex"))
+ direct_flag = PBF_LOCK_DIRECT_MUTEX;
+ else if (str_ends_with(base, "rwsem") && !is_pcpu_rwsem)
+ direct_flag = PBF_LOCK_DIRECT_RWSEM;
+
+ if ((direct_flag == PBF_LOCK_DIRECT_MUTEX ||
+ direct_flag == PBF_LOCK_DIRECT_RTMUTEX ||
+ is_pcpu_rwsem ||
+ str_ends_with(base, "spinlock") ||
+ (str_ends_with(base, "sem") &&
+ !str_ends_with(base, "rwsem"))) &&
+ !(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+
+ entry->part = PBF_LOCK_ADDR | mode | direct_flag;
+ entry->value = addr;
+ if (skel && direct_flag != PBF_LOCK_DIRECT_MUTEX &&
+ direct_flag != PBF_LOCK_DIRECT_RTMUTEX) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
+ }
+ return 0;
+ }
+ if (ret == -EACCES)
+ return ret;
+
pr_err("Error: Cannot resolve kernel lock '%s'\n", name);
return -EINVAL;
}
-static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry)
+static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct btf *btf, struct perf_bpf_filter_entry *entry)
{
int i = 0, ret;
struct perf_bpf_filter_expr *expr;
@@ -449,7 +942,7 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].term = expr->term;
entry[i].value = expr->val;
if (expr->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(expr, &entry[i]);
+ ret = resolve_lock_expr(skel, btf, expr, &entry[i]);
if (ret < 0)
return ret;
}
@@ -467,7 +960,7 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].term = group->term;
entry[i].value = group->val;
if (group->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(group, &entry[i]);
+ ret = resolve_lock_expr(skel, btf, group, &entry[i]);
if (ret < 0)
return ret;
}
@@ -710,16 +1203,19 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
{
int i, x, y, fd, ret;
struct sample_filter_bpf *skel = NULL;
+ struct btf *btf = NULL;
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
bool has_lock_filter = false;
+ unsigned int prog_groups = 0, required_groups = 0;
entry = calloc(MAX_FILTERS, sizeof(*entry));
if (entry == NULL)
return -1;
- prescan_lock_filters(evsel, &has_lock_filter);
+ prescan_lock_filters(evsel, &has_lock_filter, &prog_groups,
+ &required_groups);
if ((has_lock_filter || is_tracepoint) && geteuid() != 0 &&
!perf_cap__capable(CAP_BPF)) {
@@ -728,15 +1224,14 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- ret = get_filter_entries(evsel, entry);
- if (ret < 0) {
- pr_err("Failed to process filter entries\n");
- goto err;
- }
-
if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
int zero = 0;
+ ret = get_filter_entries(evsel, NULL, NULL, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
/* The filters map is shared among other processes */
ret = create_idx_hash(evsel, entry);
if (ret < 0)
@@ -781,8 +1276,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
- bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
+ if (has_lock_filter) {
+ btf = btf__load_vmlinux_btf();
+ bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES);
+ bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS);
+ ret = configure_lock_progs(skel, btf, prog_groups, required_groups,
+ is_tracepoint);
+ if (ret < 0)
+ goto err;
+ } else {
+ configure_lock_progs(skel, NULL, 0, 0, is_tracepoint);
+ }
if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
@@ -790,8 +1294,23 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- if (has_lock_filter)
+ ret = get_filter_entries(evsel, skel, btf, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
+
+ if (has_lock_filter) {
+ bpf_program__set_autoattach(skel->progs.perf_sample_filter, false);
+ bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false);
+
+ ret = sample_filter_bpf__attach(skel);
+ if (ret < 0) {
+ pr_err("Failed to attach lock sample-filter BPF programs\n");
+ goto err;
+ }
skel->bss->lock_enabled = 1;
+ }
i = 0;
fd = bpf_map__fd(skel->maps.filters);
@@ -832,11 +1351,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
}
}
}
+ btf__free(btf);
free(entry);
evsel->bpf_skel = skel;
return 0;
err:
+ btf__free(btf);
free(entry);
if (!list_empty(&pinned_filters)) {
struct pinned_filter_idx *pfi, *tmp;
@@ -1064,8 +1585,9 @@ int perf_bpf_filter__pin(void)
goto err;
}
+ configure_lock_progs(skel, NULL, 0, 0, false);
+
/* pinned program will use pid-hash */
- bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH);
bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH);
@@ -1160,7 +1682,7 @@ int perf_bpf_filter__unpin(void)
goto err;
}
- bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
+ configure_lock_progs(skel, NULL, 0, 0, false);
if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
pr_err("Failed to load perf sample-filter BPF skeleton\n");
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index ca1cf586ed6e..c240eee782f8 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -59,7 +59,8 @@ enum perf_bpf_filter_term {
* The lock specification kind is stored in `entry.part`
* (`enum perf_bpf_lock_target_kind`), `entry.op` is normalized to
* `PBF_OP_EQ` (match when held) or `PBF_OP_NEQ` (match when not held),
- * and the resolved target address is stored in `entry.value`.
+ * and the resolved target address or type flag encoding is stored in
+ * `entry.value`.
*/
PBF_TERM_LOCK_HELD,
};
@@ -71,10 +72,16 @@ enum perf_bpf_filter_term {
enum perf_bpf_lock_target_kind {
PBF_LOCK_NONE = 0,
/*
- * Specific global or static lock address resolved from kallsyms.
- * `entry.value` holds the kernel address.
+ * Specific global or static lock address (resolved from kallsyms or
+ * given as a hex address). `entry.value` holds the kernel address.
*/
PBF_LOCK_ADDR,
+ /*
+ * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.).
+ * `entry.value` packs up to two `LCB_F_*` flag combinations in the
+ * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
+ */
+ PBF_LOCK_TYPE,
};
/*
@@ -83,6 +90,8 @@ enum perf_bpf_lock_target_kind {
* Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding)
* Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding)
* Bit 6: PBF_LOCK_DIRECT_MUTEX (inspect struct mutex owner)
+ * Bit 7: PBF_LOCK_DIRECT_RWSEM (inspect struct rw_semaphore writer owner)
+ * Bit 8: PBF_LOCK_DIRECT_RTMUTEX (inspect struct rt_mutex_base / rt_mutex owner)
*/
#define PBF_LOCK_KIND_MASK 0x0fU
#define PBF_LOCK_MODE_READ (1U << 4)
@@ -90,6 +99,8 @@ enum perf_bpf_lock_target_kind {
#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE)
#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY
#define PBF_LOCK_DIRECT_MUTEX (1U << 6)
+#define PBF_LOCK_DIRECT_RWSEM (1U << 7)
+#define PBF_LOCK_DIRECT_RTMUTEX (1U << 8)
/* BPF map entry for filtering */
struct perf_bpf_filter_entry {
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index 42556ab0e2d5..e527fbaf8a46 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -8,6 +8,44 @@
#include "lock_data.h"
#include "sample-filter.h"
+/*
+ * Key for the `held_locks` BPF hash map tracking currently held kernel locks.
+ *
+ * - `addr_or_id`:
+ * - PBF_LOCK_ADDR: kernel virtual address of the lock instance
+ * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK)
+ * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address
+ * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags
+ * - `owner_id`:
+ * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`),
+ * since the holder may sleep and migrate across CPUs while holding the lock.
+ * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks
+ * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption
+ * is disabled on the holding CPU and spinlocks such as `rq_lock` may be
+ * acquired by one task and released by the next task across `context_switch()`.
+ * - `kind`: `enum perf_bpf_lock_target_kind`.
+ */
+struct held_lock_key {
+ __u64 addr_or_id;
+ __u32 owner_id;
+ __u32 kind;
+};
+
+/*
+ * Value for `held_locks` entries.
+ * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`)
+ * and exclusive/write acquisitions (including nested class/slab/type
+ * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of
+ * the lock so release tracepoints that do not carry flags (such as
+ * `lock:contended_release`) can decrement matching class/slab/type entries.
+ */
+struct held_lock_val {
+ __u32 read_count;
+ __u32 write_count;
+ __u32 flags;
+ __u32 pad;
+};
+
/* BPF map that will be filled by user space */
struct filters {
__uint(type, BPF_MAP_TYPE_HASH);
@@ -44,13 +82,65 @@ struct lost_count {
__uint(max_entries, 1);
} dropped SEC(".maps");
+/*
+ * Currently held kernel locks tracked by fentry/fexit hooks.
+ * Uses BPF_MAP_TYPE_LRU_HASH so that if a counting semaphore (down/up) or
+ * non-owner rwsem is acquired in one task and released in another task or
+ * interrupt handler (cross-context signaling), any orphaned entry is
+ * automatically evicted rather than exhausting the map.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_LRU_HASH);
+ __type(key, struct held_lock_key);
+ __type(value, struct held_lock_val);
+ __uint(max_entries, 1);
+} held_locks SEC(".maps");
+
+/* Target lock addresses for PBF_LOCK_ADDR filters. */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, __u64);
+ __type(value, __u8);
+ __uint(max_entries, 1);
+} lock_addr_filter SEC(".maps");
+
volatile const int use_idx_hash;
-/* Control flag set by userspace when lock_held(...) filters are active. */
+/* Control flags set by userspace when lock_held(...) filters are active. */
int lock_enabled;
+int has_lock_addr;
+int has_lock_type;
+int track_rwsem_read;
+__u8 lock_type_filter[LCB_F_TYPE_MAX];
void *bpf_cast_to_kern_ctx(void *) __ksym;
+/*
+ * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`.
+ *
+ * For locks that maintain a task_struct pointer in their `owner` field
+ * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`),
+ * `perf_sample_filter` can inspect the lock structure directly in kernel memory
+ * and compare the owner against `current` (`bpf_get_current_task()`).
+ *
+ * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear
+ * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and
+ * spinlocks are checked via the `held_locks` BPF map instead.
+ */
+static __always_inline bool is_rwsem_writer_owner(__u64 lock_addr, __u64 curr)
+{
+ struct rw_semaphore___new *rwsem = (void *)lock_addr;
+
+ if (bpf_core_field_exists(rwsem->count.counter) &&
+ !(BPF_CORE_READ(rwsem, count.counter) & 1UL))
+ return false;
+ if (bpf_core_field_exists(rwsem->owner.counter) &&
+ (BPF_CORE_READ(rwsem, owner.counter) & 1UL))
+ return false;
+
+ return (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr;
+}
+
static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __u32 part)
{
if (!curr)
@@ -59,24 +149,85 @@ static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __
if ((part & PBF_LOCK_DIRECT_MUTEX) &&
(__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr)
return true;
+ if ((part & PBF_LOCK_DIRECT_RWSEM) &&
+ is_rwsem_writer_owner(lock_addr, curr))
+ return true;
+ if ((part & PBF_LOCK_DIRECT_RTMUTEX) &&
+ (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr)
+ return true;
return false;
}
+static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode)
+{
+ if (!val)
+ return false;
+ if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0)
+ return true;
+ if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0)
+ return true;
+ return false;
+}
+
+static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind,
+ __u32 mode, __u32 pid, __u32 cpu_id)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = pid,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+
+ if (pid) {
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+ }
+
+ key.owner_id = cpu_id;
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+
+ return 0;
+}
+
+static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode,
+ __u32 pid, __u32 cpu_id)
+{
+ __u32 flag1 = target & 0xff;
+ __u32 flag2 = (target >> 8) & 0xff;
+
+ if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ return 0;
+}
+
static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry)
{
__u64 curr = bpf_get_current_task();
+ __u32 pid = (__u32)bpf_get_current_pid_tgid();
+ __u32 cpu_id = 0x80000000U | bpf_get_smp_processor_id();
__u32 kind = entry->part & PBF_LOCK_KIND_MASK;
__u32 mode = entry->part & PBF_LOCK_MODE_MASK;
+ __u64 lock_addr;
if (!mode)
mode = PBF_LOCK_MODE_ANY;
switch (kind) {
case PBF_LOCK_ADDR:
+ lock_addr = entry->value;
if ((mode & PBF_LOCK_MODE_WRITE) &&
- is_direct_lock_owner(entry->value, curr, entry->part))
+ is_direct_lock_owner(lock_addr, curr, entry->part))
return true;
- return false;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0;
+
+ case PBF_LOCK_TYPE:
+ return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0;
default:
return false;
@@ -444,4 +595,698 @@ int perf_tp_sample_filter(void *ctx)
return 0;
}
+static __always_inline __u32 get_lock_owner_id(__u32 flags)
+{
+ __u32 owner_id;
+
+ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN)
+ return 0x80000000U | bpf_get_smp_processor_id();
+
+ owner_id = (__u32)bpf_get_current_pid_tgid();
+ if (!owner_id)
+ owner_id = 0x80000000U | bpf_get_smp_processor_id();
+ return owner_id;
+}
+
+static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id,
+ __u32 kind, __u32 flags, bool acquired)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = owner_id,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (acquired) {
+ if (val) {
+ if (is_read)
+ __sync_fetch_and_add(&val->read_count, 1);
+ else
+ __sync_fetch_and_add(&val->write_count, 1);
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags,
+ };
+
+ bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY);
+ }
+ } else if (val) {
+ if (is_read) {
+ if (val->read_count <= 1 && val->write_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->read_count > 0)
+ __sync_fetch_and_add(&val->read_count, -1);
+ } else {
+ if (val->write_count <= 1 && val->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->write_count > 0)
+ __sync_fetch_and_add(&val->write_count, -1);
+ }
+ }
+}
+
+#define LCB_F_READ_IRQSAVE (1U << 30)
+#define LCB_F_READ_IRQ_BH (1U << 31)
+
+static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id)
+{
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ bool matched = false;
+ bool is_read = (flags & LCB_F_READ) != 0;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool match_type = false;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
+ matched = true;
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) {
+ match_type = true;
+ matched = true;
+ }
+
+ if (!matched)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (existing) {
+ /*
+ * On CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC
+ * kernels, _raw_read_lock_irq() and _raw_read_lock_bh() call
+ * the out-of-line _raw_read_lock_irqsave(). Clear the marker
+ * and skip the duplicate outer fexit increment.
+ */
+ if ((flags & LCB_F_READ_IRQ_BH) &&
+ (existing->flags & LCB_F_READ_IRQSAVE)) {
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ return 0;
+ }
+ /* Non-reader locks are not recursive; avoid double-counting */
+ if (!is_read && existing->write_count > 0)
+ return 0;
+ if (is_read) {
+ __sync_fetch_and_add(&existing->read_count, 1);
+ if (flags & LCB_F_READ_IRQSAVE)
+ existing->flags |= LCB_F_READ_IRQSAVE;
+ else
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ } else {
+ __sync_fetch_and_add(&existing->write_count, 1);
+ }
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags & (LCB_F_TYPE_MASK | LCB_F_READ_IRQSAVE),
+ };
+
+ if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0)
+ return 0;
+ }
+
+ if (match_type)
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true);
+
+ return 0;
+}
+
+static __always_inline int record_lock_acquired(void *ctx, __u32 flags)
+{
+ return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags));
+}
+
+static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags)
+{
+ __u32 owner_id = get_lock_owner_id(flags);
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (!existing)
+ return 0;
+
+ if (is_read) {
+ if (existing->read_count == 0)
+ return 0;
+ if (existing->read_count == 1 && existing->write_count == 0) {
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ } else {
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ __sync_fetch_and_add(&existing->read_count, -1);
+ }
+ } else {
+ if (existing->write_count == 0)
+ return 0;
+ if (existing->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ else
+ existing->write_count = 0;
+ }
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag])
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false);
+
+ return 0;
+}
+
+static __always_inline int record_lock_released(void *ctx, __u32 flags)
+{
+ return record_lock_released_addr(get_lock_addr(ctx), flags);
+}
+
+/* Mutex acquire/release hooks */
+SEC("fexit/mutex_lock")
+int BPF_PROG(fexit_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_nested")
+int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/_mutex_lock_nest_lock")
+int BPF_PROG(fexit__mutex_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io")
+int BPF_PROG(fexit_mutex_lock_io, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io_nested")
+int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_interruptible")
+int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_interruptible_nested")
+int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_killable")
+int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/_mutex_lock_killable")
+int BPF_PROG(fexit__mutex_lock_killable, void *lock, unsigned int subclass,
+ void *nest_lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_trylock")
+int BPF_PROG(fexit_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/_mutex_trylock_nest_lock")
+int BPF_PROG(fexit__mutex_trylock_nest_lock, void *lock, void *nest_lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fentry/mutex_unlock")
+int BPF_PROG(fentry_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_MUTEX);
+}
+
+/* RW semaphore acquire/release hooks */
+SEC("fexit/down_read")
+int BPF_PROG(fexit_down_read, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_nested")
+int BPF_PROG(fexit_down_read_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_interruptible")
+int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable")
+int BPF_PROG(fexit_down_read_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable_nested")
+int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_trylock")
+int BPF_PROG(fexit_down_read_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_write")
+int BPF_PROG(fexit_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_nested")
+int BPF_PROG(fexit_down_write_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/_down_write_nest_lock")
+int BPF_PROG(fexit__down_write_nest_lock, void *sem, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_killable")
+int BPF_PROG(fexit_down_write_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_killable_nested")
+int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_trylock")
+int BPF_PROG(fexit_down_write_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/downgrade_write")
+int BPF_PROG(fexit_downgrade_write, void *sem)
+{
+ if (track_rwsem_read)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+SEC("fentry/up_read")
+int BPF_PROG(fentry_up_read, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_READ);
+}
+
+SEC("fentry/up_write")
+int BPF_PROG(fentry_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+/* Spinlock acquire/release hooks */
+SEC("fexit/_raw_spin_lock")
+int BPF_PROG(fexit_raw_spin_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_nested")
+int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_nest_lock")
+int BPF_PROG(fexit_raw_spin_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq")
+int BPF_PROG(fexit_raw_spin_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq_disable")
+int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave")
+int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave_nested")
+int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_bh")
+int BPF_PROG(fexit_raw_spin_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_trylock")
+int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fexit/_raw_spin_trylock_bh")
+int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fentry/_raw_spin_unlock")
+int BPF_PROG(fentry_raw_spin_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq")
+int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq_enable")
+int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irqrestore")
+int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_bh")
+int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+/* RWlock acquire/release hooks */
+SEC("fexit/_raw_read_lock")
+int BPF_PROG(fexit_raw_read_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_lock_irq")
+int BPF_PROG(fexit_raw_read_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH);
+}
+
+SEC("fexit/_raw_read_lock_irqsave")
+int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQSAVE);
+}
+
+SEC("fexit/_raw_read_lock_bh")
+int BPF_PROG(fexit_raw_read_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH);
+}
+
+SEC("fexit/_raw_read_trylock")
+int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+ return 0;
+}
+
+SEC("fentry/_raw_read_unlock")
+int BPF_PROG(fentry_raw_read_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irq")
+int BPF_PROG(fentry_raw_read_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irqrestore")
+int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_bh")
+int BPF_PROG(fentry_raw_read_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_write_lock")
+int BPF_PROG(fexit_raw_write_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_nested")
+int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irq")
+int BPF_PROG(fexit_raw_write_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irqsave")
+int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_bh")
+int BPF_PROG(fexit_raw_write_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_trylock")
+int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fentry/_raw_write_unlock")
+int BPF_PROG(fentry_raw_write_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irq")
+int BPF_PROG(fentry_raw_write_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irqrestore")
+int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_bh")
+int BPF_PROG(fentry_raw_write_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+/* RT-Mutex acquire/release hooks */
+SEC("fexit/rt_mutex_lock")
+int BPF_PROG(fexit_rt_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_nested")
+int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/_rt_mutex_lock_nest_lock")
+int BPF_PROG(fexit__rt_mutex_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_interruptible")
+int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_lock_killable")
+int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_trylock")
+int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fentry/rt_mutex_unlock")
+int BPF_PROG(fentry_rt_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_RT);
+}
+
+/* Semaphore acquire/release hooks */
+SEC("fexit/down")
+int BPF_PROG(fexit_down, void *sem)
+{
+ return record_lock_acquired(ctx, 0);
+}
+
+SEC("fexit/down_interruptible")
+int BPF_PROG(fexit_down_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_killable")
+int BPF_PROG(fexit_down_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_timeout")
+int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_trylock")
+int BPF_PROG(fexit_down_trylock, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fentry/up")
+int BPF_PROG(fentry_up, void *sem)
+{
+ return record_lock_released(ctx, 0);
+}
+
+/* Percpu-rwsem write acquire/release hooks */
+SEC("fexit/percpu_down_write")
+int BPF_PROG(fexit_percpu_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
+SEC("fentry/percpu_up_write")
+int BPF_PROG(fentry_percpu_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
char LICENSE[] SEC("license") = "Dual BSD/GPL";
--
2.56.0.307.g5fe7bd680c-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v6 07/11] perf bpf-filter: Support per-process and semi-global locks in lock_held()
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (5 preceding siblings ...)
2026-10-22 20:29 ` [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types Ian Rogers
@ 2026-10-22 20:29 ` Ian Rogers
2026-10-22 20:49 ` sashiko-bot
2026-10-22 20:30 ` [PATCH v6 08/11] perf bpf-filter: Support slab cache locks (&<name>) " Ian Rogers
` (3 subsequent siblings)
10 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-22 20:29 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Extend the lock_held() BPF sample filter to support well-known
per-process and semi-global kernel locks:
- mmap_lock (or mmap_sem): current->mm->mmap_lock, supporting both
read/shared (lock_held_read(mmap_lock) / lock_held(mmap_lock:R)) and
write/exclusive (lock_held_write(mmap_lock) / lock_held(mmap_lock:W))
modes. Write ownership is inspected directly via is_rwsem_writer_owner()
without attaching fentry/fexit hooks.
- siglock: current->sighand->siglock.
- rq_lock: per-CPU scheduler runqueue locks.
- zone_lock: memory management NUMA zone locks.
For rq_lock and zone_lock, reuse do_collect_lock_syms() from lock_data.h
via a raw_tp/bpf_test_finish program run at filter setup time to
populate the lock_syms map with each CPU's rq_lock and each NUMA node's
zone_lock addresses.
Example usage:
# perf record -a -g -e cycles --filter 'lock_held(mmap_lock)' \
-- perf bench mem mmap -t 2 -l 100
# perf record -a -g -e cycles --filter 'lock_held_read(mmap_lock)' ...
# perf record -a -g -e cycles --filter 'lock_held_write(mmap_lock)' ...
# perf record -a -g -e cycles --filter 'lock_held(mmap_lock:W)' ...
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 24 +--
tools/perf/util/bpf-filter.c | 169 +++++++++++++++++--
tools/perf/util/bpf_skel/sample-filter.h | 18 +-
tools/perf/util/bpf_skel/sample_filter.bpf.c | 67 ++++++++
4 files changed, 252 insertions(+), 26 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 9fb5d15de135..2b2aaaead76b 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -220,13 +220,15 @@ OPTIONS
holding and exclusive/write holding, use 'lock_held_read(<lock>)'
(alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)'
(alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to
- <lock> (e.g. 'lock_held(rwsem:R)' or 'lock_held(rwsem:W)').
+ <lock> (e.g. 'lock_held(mmap_lock:R)' or 'lock_held(mmap_lock:W)').
<lock> can be a kernel lock symbol name (e.g.
- 'lock_held(cgroup_mutex)'), a hexadecimal lock address, or a lock type
- ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
+ 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a well-known
+ lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock
+ type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
'semaphore'). Both read/shared and write/exclusive tracking are
- supported for 'rwsem' and 'rwlock'; 'mutex', 'spinlock', 'rt-mutex',
+ supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex',
+ 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex',
'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support
write/exclusive tracking only. Note that 'semaphore' tracking
('down*()' / 'up()') assumes binary mutex-like usage where the
@@ -246,8 +248,9 @@ OPTIONS
# perf record -a -g -e major-faults \
--filter 'lock_held(mutex) || lock_held(rwsem)'
- When filtering on a specific mutex symbol ('*_mutex') or rt-mutex
- symbol ('*_rt_mutex'), 'perf_sample_filter' inspects the lock's
+ When filtering on a specific mutex symbol ('*_mutex'), rt-mutex symbol
+ ('*_rt_mutex'), or write-locked 'mmap_lock' ('lock_held_write(mmap_lock)'
+ or 'lock_held(mmap_lock:W)'), 'perf_sample_filter' inspects the lock's
'owner' field directly in kernel memory on each sample without attaching
any BPF fentry/fexit hooks. For locks that do not maintain an
authoritative per-task owner pointer (such as spinlocks, rwlocks,
@@ -256,10 +259,11 @@ OPTIONS
to track held locks in a BPF map. These BPF hooks run on every kernel
lock acquisition and release for the attached lock family, even when no
perf sample is being taken. To minimize this overhead, only the hooks
- for the relevant lock family and mode are attached when known, whereas
- raw hex addresses or symbols without a recognizable lock-type suffix
- attach hooks across all lock families (including high-frequency
- spinlocks).
+ for the relevant lock family and mode are attached when known (for
+ example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions
+ and 'rq_lock' only hooks spinlock functions), whereas raw hex addresses
+ or symbols without a recognizable lock-type suffix attach hooks across
+ all lock families (including high-frequency spinlocks).
The <operator> can be one of:
==, !=, >, >=, <, <=, &
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 8797f67698f7..f542920f914f 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -534,6 +534,54 @@ static bool parse_lock_hex_addr(const char *str, u64 *addr)
return true;
}
+static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf)
+{
+ u64 addr = 0;
+ char *buf = NULL, *p;
+ size_t len;
+ long last = -1;
+ int ret;
+
+ if (!btf)
+ return;
+
+ ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT);
+ if (ret < 0)
+ return;
+
+ ret = btf__resolve_size(btf, ret);
+ if (ret < 0)
+ return;
+
+ skel->rodata->sizeof_zone = ret;
+
+ if (find_kernel_lock_sym("contig_page_data", &addr) == 0) {
+ skel->rodata->contig_page_data_addr = addr;
+ return;
+ }
+
+ if (find_kernel_lock_sym("node_data", &addr) != 0)
+ return;
+
+ skel->rodata->node_data_addr = addr;
+
+ if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) {
+ p = buf;
+ while (p && *p) {
+ char *end;
+
+ last = strtol(p, &end, 0);
+ if (end == p)
+ break;
+ p = end;
+ if (*p == ',' || *p == '-' || *p == '\n')
+ p++;
+ }
+ skel->rodata->nr_nodes = last + 1;
+ free(buf);
+ }
+}
+
static bool btf_has_func(struct btf *btf, const char *name)
{
return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0;
@@ -607,7 +655,7 @@ static unsigned int supported_lock_prog_groups(struct btf *btf)
static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
unsigned int needed_groups,
unsigned int required_groups,
- bool is_tracepoint)
+ bool needs_lock_syms, bool is_tracepoint)
{
unsigned int supported_groups = supported_lock_prog_groups(btf);
unsigned int enabled_progs = 0;
@@ -642,6 +690,8 @@ static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
bpf_program__set_autoload(prog, enable);
}
+ bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms);
+
if (needed_groups && !enabled_progs) {
pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
return -ENOTSUP;
@@ -657,7 +707,8 @@ static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
bool *has_lock_filter,
unsigned int *prog_groups,
- unsigned int *required_groups)
+ unsigned int *required_groups,
+ bool *needs_lock_syms)
{
char base[128];
bool has_amp;
@@ -672,6 +723,29 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
sizeof(base), &has_amp, &mode) < 0 || has_amp)
return;
+ if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
+ /*
+ * Write ownership of mm->mmap_lock is inspected directly via
+ * is_rwsem_writer_owner() without tracing hooks; only reader
+ * tracking requires LOCK_PROG_RWSEM_READ.
+ */
+ if (mode & PBF_LOCK_MODE_READ) {
+ *prog_groups |= LOCK_PROG_RWSEM_READ;
+ *required_groups |= LOCK_PROG_RWSEM_READ;
+ }
+ return;
+ }
+ if (!strcmp(base, "siglock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ return;
+ }
+ if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ *needs_lock_syms = true;
+ return;
+ }
if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) {
*prog_groups |= groups;
*required_groups |= groups;
@@ -720,21 +794,24 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter,
unsigned int *prog_groups,
- unsigned int *required_groups)
+ unsigned int *required_groups,
+ bool *needs_lock_syms)
{
struct perf_bpf_filter_expr *expr, *group;
*has_lock_filter = false;
*prog_groups = 0;
*required_groups = 0;
+ *needs_lock_syms = false;
list_for_each_entry(expr, &evsel->bpf_filters, list) {
prescan_one_lock_expr(expr, has_lock_filter, prog_groups,
- required_groups);
+ required_groups, needs_lock_syms);
if (expr->op == PBF_OP_GROUP_BEGIN) {
list_for_each_entry(group, &expr->groups, list) {
prescan_one_lock_expr(group, has_lock_filter,
- prog_groups, required_groups);
+ prog_groups, required_groups,
+ needs_lock_syms);
}
}
}
@@ -826,10 +903,12 @@ static bool is_percpu_rwsem_sym(struct btf *btf, const char *name)
* and populate the corresponding BPF filter maps and BSS control flags.
*
* Resolution order:
- * 1. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`,
+ * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
+ * `rq_lock`, `zone_lock`.
+ * 2. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`,
* `rwlock`, `percpu-rwsem`, `semaphore`, etc.
- * 2. Hexadecimal lock address.
- * 3. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`).
+ * 3. Hexadecimal lock address.
+ * 4. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`).
*/
static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf,
struct perf_bpf_filter_expr *expr,
@@ -853,6 +932,53 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf,
return -EINVAL;
}
+ if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
+ entry->part = PBF_LOCK_MMAP | mode;
+ entry->value = 0;
+ if (skel && (mode & PBF_LOCK_MODE_READ))
+ skel->bss->has_mmap_lock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "siglock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_SIGHAND | mode;
+ entry->value = 0;
+ if (skel)
+ skel->bss->has_siglock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "rq_lock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_CLASS | mode;
+ entry->value = LOCK_CLASS_RQLOCK;
+ if (skel)
+ skel->bss->has_lock_class = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "zone_lock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_CLASS | mode;
+ entry->value = LOCK_CLASS_ZONE_LOCK;
+ if (skel)
+ skel->bss->has_lock_class = 1;
+ return 0;
+ }
+
if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
entry->part = PBF_LOCK_TYPE | mode;
entry->value = encoded_pair;
@@ -1207,7 +1333,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
- bool has_lock_filter = false;
+ bool has_lock_filter = false, needs_lock_syms = false;
unsigned int prog_groups = 0, required_groups = 0;
entry = calloc(MAX_FILTERS, sizeof(*entry));
@@ -1215,7 +1341,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return -1;
prescan_lock_filters(evsel, &has_lock_filter, &prog_groups,
- &required_groups);
+ &required_groups, &needs_lock_syms);
if ((has_lock_filter || is_tracepoint) && geteuid() != 0 &&
!perf_cap__capable(CAP_BPF)) {
@@ -1280,12 +1406,16 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
btf = btf__load_vmlinux_btf();
bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES);
bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS);
+ if (needs_lock_syms) {
+ bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
+ init_numa_data(skel, btf);
+ }
ret = configure_lock_progs(skel, btf, prog_groups, required_groups,
- is_tracepoint);
+ needs_lock_syms, is_tracepoint);
if (ret < 0)
goto err;
} else {
- configure_lock_progs(skel, NULL, 0, 0, is_tracepoint);
+ configure_lock_progs(skel, NULL, 0, 0, false, is_tracepoint);
}
if (sample_filter_bpf__load(skel) < 0) {
@@ -1301,8 +1431,19 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
}
if (has_lock_filter) {
+ if (needs_lock_syms && skel->bss->has_lock_class) {
+ DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts);
+ int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms);
+
+ ret = bpf_prog_test_run_opts(prog_fd, &opts);
+ if (ret < 0) {
+ pr_err("Failed to collect kernel lock symbols\n");
+ goto err;
+ }
+ }
bpf_program__set_autoattach(skel->progs.perf_sample_filter, false);
bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false);
+ bpf_program__set_autoattach(skel->progs.collect_lock_syms, false);
ret = sample_filter_bpf__attach(skel);
if (ret < 0) {
@@ -1585,7 +1726,7 @@ int perf_bpf_filter__pin(void)
goto err;
}
- configure_lock_progs(skel, NULL, 0, 0, false);
+ configure_lock_progs(skel, NULL, 0, 0, false, false);
/* pinned program will use pid-hash */
bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
@@ -1682,7 +1823,7 @@ int perf_bpf_filter__unpin(void)
goto err;
}
- configure_lock_progs(skel, NULL, 0, 0, false);
+ configure_lock_progs(skel, NULL, 0, 0, false, false);
if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
pr_err("Failed to load perf sample-filter BPF skeleton\n");
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index c240eee782f8..836b398dc62b 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -59,8 +59,8 @@ enum perf_bpf_filter_term {
* The lock specification kind is stored in `entry.part`
* (`enum perf_bpf_lock_target_kind`), `entry.op` is normalized to
* `PBF_OP_EQ` (match when held) or `PBF_OP_NEQ` (match when not held),
- * and the resolved target address or type flag encoding is stored in
- * `entry.value`.
+ * and the resolved target address, class ID, or type flag encoding is
+ * stored in `entry.value`.
*/
PBF_TERM_LOCK_HELD,
};
@@ -76,6 +76,20 @@ enum perf_bpf_lock_target_kind {
* given as a hex address). `entry.value` holds the kernel address.
*/
PBF_LOCK_ADDR,
+ /*
+ * Per-process `current->mm->mmap_lock` (or `mmap_sem`).
+ */
+ PBF_LOCK_MMAP,
+ /*
+ * Per-process `current->sighand->siglock`.
+ */
+ PBF_LOCK_SIGHAND,
+ /*
+ * Semi-global lock class (`rq_lock` or `zone_lock`).
+ * `entry.value` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or
+ * LOCK_CLASS_ZONE_LOCK).
+ */
+ PBF_LOCK_CLASS,
/*
* Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.).
* `entry.value` packs up to two `LCB_F_*` flag combinations in the
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index e527fbaf8a46..2087fb05e4c7 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -104,12 +104,27 @@ struct {
__uint(max_entries, 1);
} lock_addr_filter SEC(".maps");
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u32));
+ __uint(max_entries, 1);
+} lock_syms SEC(".maps");
+
volatile const int use_idx_hash;
+const volatile __u64 contig_page_data_addr;
+const volatile __u64 node_data_addr;
+const volatile int nr_nodes;
+const volatile int sizeof_zone;
+
/* Control flags set by userspace when lock_held(...) filters are active. */
int lock_enabled;
int has_lock_addr;
+int has_lock_class;
int has_lock_type;
+int has_mmap_lock;
+int has_siglock;
int track_rwsem_read;
__u8 lock_type_filter[LCB_F_TYPE_MAX];
@@ -226,6 +241,24 @@ static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry)
return true;
return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0;
+ case PBF_LOCK_MMAP:
+ lock_addr = get_current_mmap_lock(curr);
+ if (!lock_addr)
+ return false;
+ if ((mode & PBF_LOCK_MODE_WRITE) &&
+ is_rwsem_writer_owner(lock_addr, curr))
+ return true;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0;
+
+ case PBF_LOCK_SIGHAND:
+ lock_addr = get_current_siglock(curr);
+ if (!lock_addr)
+ return false;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0;
+
+ case PBF_LOCK_CLASS:
+ return is_lock_in_map(entry->value, PBF_LOCK_CLASS, mode, pid, cpu_id) != 0;
+
case PBF_LOCK_TYPE:
return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0;
@@ -663,6 +696,7 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla
struct held_lock_val *existing;
bool matched = false;
bool is_read = (flags & LCB_F_READ) != 0;
+ __u32 class_id = 0;
__u32 type_flag = flags & LCB_F_TYPE_MASK;
bool match_type = false;
@@ -672,6 +706,23 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla
if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
matched = true;
+ if (has_mmap_lock || has_siglock) {
+ __u32 ltype = check_lock_type(lock_addr, type_flag);
+
+ if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
+ (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
+ matched = true;
+ }
+
+ if (has_lock_class && type_flag == LCB_F_SPIN) {
+ __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
+
+ if (cid) {
+ class_id = *cid;
+ matched = true;
+ }
+ }
+
if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) {
match_type = true;
matched = true;
@@ -716,6 +767,8 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla
return 0;
}
+ if (class_id)
+ update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, type_flag, true);
if (match_type)
update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true);
@@ -764,6 +817,13 @@ static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flag
existing->write_count = 0;
}
+ if (has_lock_class && flags == LCB_F_SPIN) {
+ __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
+
+ if (cid)
+ update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false);
+ }
+
if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag])
update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false);
@@ -1289,4 +1349,11 @@ int BPF_PROG(fentry_percpu_up_write, void *sem)
return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
}
+SEC("raw_tp/bpf_test_finish")
+int BPF_PROG(collect_lock_syms)
+{
+ return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
+}
+
char LICENSE[] SEC("license") = "Dual BSD/GPL";
--
2.56.0.307.g5fe7bd680c-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v6 08/11] perf bpf-filter: Support slab cache locks (&<name>) in lock_held()
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (6 preceding siblings ...)
2026-10-22 20:29 ` [PATCH v6 07/11] perf bpf-filter: Support per-process and semi-global locks in lock_held() Ian Rogers
@ 2026-10-22 20:30 ` Ian Rogers
2026-10-22 20:37 ` sashiko-bot
2026-10-22 20:30 ` [PATCH v6 09/11] perf test: Add tests for lock_held() BPF sample filter Ian Rogers
` (2 subsequent siblings)
10 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-22 20:30 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Extend the lock_held() BPF sample filter to support dynamically
allocated kernel locks inside slab caches prefixed with '&' (e.g.
lock_held(&dentry), lock_held_read(&dentry), lock_held_write(&dentry)).
Reuse do_slab_cache_iter() from lock_data.h via a SEC("iter/kmem_cache")
program to resolve the requested slab cache name to its kernel
struct kmem_cache * address, and use the bpf_get_kmem_cache() kfunc in
record_lock_acquired_owner() and record_lock_released_addr() to match
held locks allocated from that slab cache.
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 25 ++--
tools/perf/util/bpf-filter.c | 125 ++++++++++++++++---
tools/perf/util/bpf_skel/sample-filter.h | 5 +
tools/perf/util/bpf_skel/sample_filter.bpf.c | 48 +++++++
4 files changed, 175 insertions(+), 28 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 2b2aaaead76b..8fc3196ed93d 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -222,13 +222,15 @@ OPTIONS
(alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to
<lock> (e.g. 'lock_held(mmap_lock:R)' or 'lock_held(mmap_lock:W)').
<lock> can be a kernel lock symbol name (e.g.
- 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a well-known
- lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock
- type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
- 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
- 'semaphore'). Both read/shared and write/exclusive tracking are
- supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex',
- 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex',
+ 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache
+ name prefixed with '&' (e.g. 'lock_held(&dentry)',
+ 'lock_held_read(&dentry)'), a well-known lock name ('mmap_lock',
+ 'siglock', 'rq_lock', 'zone_lock'), or a lock type ('mutex',
+ 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R',
+ 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore').
+ Both read/shared and write/exclusive tracking are supported for
+ 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', 'spinlock'
+ (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex',
'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support
write/exclusive tracking only. Note that 'semaphore' tracking
('down*()' / 'up()') assumes binary mutex-like usage where the
@@ -254,16 +256,17 @@ OPTIONS
'owner' field directly in kernel memory on each sample without attaching
any BPF fentry/fexit hooks. For locks that do not maintain an
authoritative per-task owner pointer (such as spinlocks, rwlocks,
- semaphores, reader-locked rwsems, and lock type filters), it attaches
+ semaphores, reader-locked rwsems, and slab/type filters), it attaches
BPF fentry/fexit programs to kernel lock acquire and release functions
to track held locks in a BPF map. These BPF hooks run on every kernel
lock acquisition and release for the attached lock family, even when no
perf sample is being taken. To minimize this overhead, only the hooks
for the relevant lock family and mode are attached when known (for
example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions
- and 'rq_lock' only hooks spinlock functions), whereas raw hex addresses
- or symbols without a recognizable lock-type suffix attach hooks across
- all lock families (including high-frequency spinlocks).
+ and 'rq_lock' only hooks spinlock functions), whereas slab filters
+ ('&<slab>'), raw hex addresses, or symbols without a recognizable
+ lock-type suffix attach hooks across all lock families (including
+ high-frequency spinlocks).
The <operator> can be one of:
==, !=, >, >=, <, <=, &
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index f542920f914f..1a75d779e891 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -655,10 +655,14 @@ static unsigned int supported_lock_prog_groups(struct btf *btf)
static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
unsigned int needed_groups,
unsigned int required_groups,
- bool needs_lock_syms, bool is_tracepoint)
+ bool needs_lock_syms, bool needs_slab_iter,
+ bool is_tracepoint)
{
unsigned int supported_groups = supported_lock_prog_groups(btf);
unsigned int enabled_progs = 0;
+ bool has_slab_iter = needs_slab_iter && btf &&
+ btf__find_by_name_kind(btf, "bpf_iter__kmem_cache",
+ BTF_KIND_STRUCT) > 0;
bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
@@ -691,6 +695,7 @@ static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
}
bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms);
+ bpf_program__set_autoload(skel->progs.slab_cache_iter, has_slab_iter);
if (needed_groups && !enabled_progs) {
pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
@@ -708,7 +713,8 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
bool *has_lock_filter,
unsigned int *prog_groups,
unsigned int *required_groups,
- bool *needs_lock_syms)
+ bool *needs_lock_syms,
+ bool *needs_slab_iter)
{
char base[128];
bool has_amp;
@@ -720,9 +726,16 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
*has_lock_filter = true;
if (split_lock_name_mode(expr->lock_name, expr->part, base,
- sizeof(base), &has_amp, &mode) < 0 || has_amp)
+ sizeof(base), &has_amp, &mode) < 0)
return;
+ /* Leading '&' is reserved for slab cache locks (e.g. '&dentry'). */
+ if (has_amp) {
+ *needs_slab_iter = true;
+ *prog_groups |= LOCK_PROG_ALL;
+ return;
+ }
+
if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
/*
* Write ownership of mm->mmap_lock is inspected directly via
@@ -795,7 +808,8 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter,
unsigned int *prog_groups,
unsigned int *required_groups,
- bool *needs_lock_syms)
+ bool *needs_lock_syms,
+ bool *needs_slab_iter)
{
struct perf_bpf_filter_expr *expr, *group;
@@ -803,20 +817,72 @@ static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter,
*prog_groups = 0;
*required_groups = 0;
*needs_lock_syms = false;
+ *needs_slab_iter = false;
list_for_each_entry(expr, &evsel->bpf_filters, list) {
prescan_one_lock_expr(expr, has_lock_filter, prog_groups,
- required_groups, needs_lock_syms);
+ required_groups, needs_lock_syms,
+ needs_slab_iter);
if (expr->op == PBF_OP_GROUP_BEGIN) {
list_for_each_entry(group, &expr->groups, list) {
prescan_one_lock_expr(group, has_lock_filter,
prog_groups, required_groups,
- needs_lock_syms);
+ needs_lock_syms, needs_slab_iter);
}
}
}
}
+static void run_slab_cache_iter(struct sample_filter_bpf *skel)
+{
+ struct bpf_link *link;
+ int fd;
+ char buf[256];
+
+ if (!bpf_program__autoload(skel->progs.slab_cache_iter))
+ return;
+
+ link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL);
+ if (!link || IS_ERR(link)) {
+ pr_debug("cannot attach slab cache iter\n");
+ return;
+ }
+
+ fd = bpf_iter_create(bpf_link__fd(link));
+ if (fd >= 0) {
+ while (read(fd, buf, sizeof(buf)) > 0)
+ continue;
+ close(fd);
+ }
+
+ bpf_link__destroy(link);
+}
+
+static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
+ u64 *kmem_cache_addr)
+{
+ int cache_fd;
+ u64 key, *prev_key = NULL;
+
+ if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter))
+ return false;
+
+ cache_fd = bpf_map__fd(skel->maps.slab_caches);
+ while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) {
+ struct slab_cache_data data;
+
+ if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0)
+ break;
+
+ if (!strcmp(data.name, name)) {
+ *kmem_cache_addr = key;
+ return true;
+ }
+ prev_key = &key;
+ }
+ return false;
+}
+
static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr,
enum perf_bpf_filter_op *op_out)
{
@@ -903,12 +969,13 @@ static bool is_percpu_rwsem_sym(struct btf *btf, const char *name)
* and populate the corresponding BPF filter maps and BSS control flags.
*
* Resolution order:
- * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
+ * 1. Slab cache name (when prefixed with '&', e.g. `&dentry`).
+ * 2. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
* `rq_lock`, `zone_lock`.
- * 2. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`,
+ * 3. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`,
* `rwlock`, `percpu-rwsem`, `semaphore`, etc.
- * 3. Hexadecimal lock address.
- * 4. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`).
+ * 4. Hexadecimal lock address.
+ * 5. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`).
*/
static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf,
struct perf_bpf_filter_expr *expr,
@@ -927,11 +994,25 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf,
return ret;
if (split_lock_name_mode(name, expr->part, base, sizeof(base),
- &has_amp, &mode) < 0 || has_amp) {
+ &has_amp, &mode) < 0) {
pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: "");
return -EINVAL;
}
+ if (has_amp) {
+ if (find_slab_cache(skel, base, &addr)) {
+ int fd = bpf_map__fd(skel->maps.slab_filter);
+
+ entry->part = PBF_LOCK_SLAB | mode;
+ entry->value = addr;
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_slab = 1;
+ return 0;
+ }
+ pr_err("Error: Cannot resolve slab cache '%s'\n", name);
+ return -EINVAL;
+ }
+
if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
entry->part = PBF_LOCK_MMAP | mode;
entry->value = 0;
@@ -1333,7 +1414,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
- bool has_lock_filter = false, needs_lock_syms = false;
+ bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false;
unsigned int prog_groups = 0, required_groups = 0;
entry = calloc(MAX_FILTERS, sizeof(*entry));
@@ -1341,7 +1422,8 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return -1;
prescan_lock_filters(evsel, &has_lock_filter, &prog_groups,
- &required_groups, &needs_lock_syms);
+ &required_groups, &needs_lock_syms,
+ &needs_slab_iter);
if ((has_lock_filter || is_tracepoint) && geteuid() != 0 &&
!perf_cap__capable(CAP_BPF)) {
@@ -1406,16 +1488,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
btf = btf__load_vmlinux_btf();
bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES);
bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS);
+ if (needs_slab_iter) {
+ bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS);
+ bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES);
+ }
if (needs_lock_syms) {
bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
init_numa_data(skel, btf);
}
ret = configure_lock_progs(skel, btf, prog_groups, required_groups,
- needs_lock_syms, is_tracepoint);
+ needs_lock_syms, needs_slab_iter,
+ is_tracepoint);
if (ret < 0)
goto err;
} else {
- configure_lock_progs(skel, NULL, 0, 0, false, is_tracepoint);
+ configure_lock_progs(skel, NULL, 0, 0, false, false, is_tracepoint);
}
if (sample_filter_bpf__load(skel) < 0) {
@@ -1424,6 +1511,9 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
+ if (has_lock_filter && needs_slab_iter)
+ run_slab_cache_iter(skel);
+
ret = get_filter_entries(evsel, skel, btf, entry);
if (ret < 0) {
pr_err("Failed to process filter entries\n");
@@ -1444,6 +1534,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
bpf_program__set_autoattach(skel->progs.perf_sample_filter, false);
bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false);
bpf_program__set_autoattach(skel->progs.collect_lock_syms, false);
+ bpf_program__set_autoattach(skel->progs.slab_cache_iter, false);
ret = sample_filter_bpf__attach(skel);
if (ret < 0) {
@@ -1726,7 +1817,7 @@ int perf_bpf_filter__pin(void)
goto err;
}
- configure_lock_progs(skel, NULL, 0, 0, false, false);
+ configure_lock_progs(skel, NULL, 0, 0, false, false, false);
/* pinned program will use pid-hash */
bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
@@ -1823,7 +1914,7 @@ int perf_bpf_filter__unpin(void)
goto err;
}
- configure_lock_progs(skel, NULL, 0, 0, false, false);
+ configure_lock_progs(skel, NULL, 0, 0, false, false, false);
if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
pr_err("Failed to load perf sample-filter BPF skeleton\n");
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index 836b398dc62b..a65ee780d64d 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -76,6 +76,11 @@ enum perf_bpf_lock_target_kind {
* given as a hex address). `entry.value` holds the kernel address.
*/
PBF_LOCK_ADDR,
+ /*
+ * Dynamic lock allocated inside a slab cache (`&slab_name`).
+ * `entry.value` holds the kernel `struct kmem_cache *` address.
+ */
+ PBF_LOCK_SLAB,
/*
* Per-process `current->mm->mmap_lock` (or `mmap_sem`).
*/
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index 2087fb05e4c7..d1192a93179e 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -111,6 +111,20 @@ struct {
__uint(max_entries, 1);
} lock_syms SEC(".maps");
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u8));
+ __uint(max_entries, 1);
+} slab_filter SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(struct slab_cache_data));
+ __uint(max_entries, 1);
+} slab_caches SEC(".maps");
+
volatile const int use_idx_hash;
const volatile __u64 contig_page_data_addr;
@@ -118,9 +132,12 @@ const volatile __u64 node_data_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
+__u32 slab_cache_id;
+
/* Control flags set by userspace when lock_held(...) filters are active. */
int lock_enabled;
int has_lock_addr;
+int has_lock_slab;
int has_lock_class;
int has_lock_type;
int has_mmap_lock;
@@ -259,6 +276,9 @@ static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry)
case PBF_LOCK_CLASS:
return is_lock_in_map(entry->value, PBF_LOCK_CLASS, mode, pid, cpu_id) != 0;
+ case PBF_LOCK_SLAB:
+ return is_lock_in_map(entry->value, PBF_LOCK_SLAB, mode, pid, cpu_id) != 0;
+
case PBF_LOCK_TYPE:
return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0;
@@ -697,6 +717,7 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla
bool matched = false;
bool is_read = (flags & LCB_F_READ) != 0;
__u32 class_id = 0;
+ __u64 kmem_cache_addr = 0;
__u32 type_flag = flags & LCB_F_TYPE_MASK;
bool match_type = false;
@@ -723,6 +744,14 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla
}
}
+ if (has_lock_slab && bpf_get_kmem_cache) {
+ kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
+ if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr))
+ matched = true;
+ else
+ kmem_cache_addr = 0;
+ }
+
if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) {
match_type = true;
matched = true;
@@ -769,6 +798,10 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla
if (class_id)
update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, type_flag, true);
+ if (kmem_cache_addr) {
+ update_held_lock_count((__u64)kmem_cache_addr, owner_id,
+ PBF_LOCK_SLAB, type_flag, true);
+ }
if (match_type)
update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true);
@@ -824,6 +857,15 @@ static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flag
update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false);
}
+ if (has_lock_slab && bpf_get_kmem_cache) {
+ __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
+
+ if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) {
+ update_held_lock_count(kmem_cache_addr, owner_id,
+ PBF_LOCK_SLAB, flags, false);
+ }
+ }
+
if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag])
update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false);
@@ -1356,4 +1398,10 @@ int BPF_PROG(collect_lock_syms)
node_data_addr, nr_nodes, sizeof_zone);
}
+SEC("iter/kmem_cache")
+int slab_cache_iter(void *ctx)
+{
+ return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
+}
+
char LICENSE[] SEC("license") = "Dual BSD/GPL";
--
2.56.0.307.g5fe7bd680c-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v6 09/11] perf test: Add tests for lock_held() BPF sample filter
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (7 preceding siblings ...)
2026-10-22 20:30 ` [PATCH v6 08/11] perf bpf-filter: Support slab cache locks (&<name>) " Ian Rogers
@ 2026-10-22 20:30 ` Ian Rogers
2026-10-22 20:37 ` sashiko-bot
2026-10-22 20:30 ` [PATCH v6 10/11] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-22 20:30 ` [PATCH v6 11/11] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
10 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-22 20:30 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Add unit tests in parse-events.c for parsing lock_held(),
lock_held_read(), and lock_held_exclusive() BPF filter expressions
(standalone predicates, negated predicates, and compound group
expressions) and extend record_bpf_filter.sh with lock_held(),
lock_held_read()/lock_held_write(), and ':R'/':W' recording tests.
Example output:
# perf test -v 'Parsing of BPF sample filters'
5: Parse event definition strings :
5.7: Parsing of BPF sample filters : Ok
# perf test -v 'perf record sample filtering (by BPF) tests'
112: perf record sample filtering (by BPF) tests :
Checking BPF-filter privilege
Basic bpf-filter test
Basic bpf-filter test [Success]
Failing bpf-filter test
Failing bpf-filter test [Success]
Group bpf-filter test
Group bpf-filter test [Success]
Multiple bpf-filter test
Multiple bpf-filter test [Success]
Cgroup bpf-filter test
Cgroup bpf-filter test [Success]
Lock-held bpf-filter test
Lock-held bpf-filter test [Success]
112: perf record sample filtering (by BPF) tests : Ok
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/tests/parse-events.c | 108 +++++++++++++++++
tools/perf/tests/shell/record_bpf_filter.sh | 122 ++++++++++++++++++++
2 files changed, 230 insertions(+)
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 5f6f95c8a5b4..4cf9192762f2 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -8,9 +8,11 @@
#include "debug.h"
#include "pmu.h"
#include "pmus.h"
+#include "bpf-filter.h"
#include "strbuf.h"
#include <dirent.h>
#include <errno.h>
+#include <string.h>
#include "fncache.h"
#include <sys/types.h>
#include <sys/stat.h>
@@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused,
return ret;
}
+#ifdef HAVE_BPF_SKEL
+static void clear_bpf_filter_exprs(struct list_head *expr_head)
+{
+ struct perf_bpf_filter_expr *expr, *tmp;
+
+ list_for_each_entry_safe(expr, tmp, expr_head, list) {
+ list_del(&expr->list);
+ perf_bpf_filter_expr__delete(expr);
+ }
+}
+
+static int do_test_bpf_filters(struct list_head *expr_head)
+{
+ struct perf_bpf_filter_expr *expr, *group;
+ const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100";
+
+ /* Simple lock_held(&cgroup_mutex) */
+ TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)",
+ __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is &cgroup_mutex",
+ expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /* Negated !lock_held(cgroup_mutex) */
+ TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)",
+ __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 0", expr->val == 0);
+ TEST_ASSERT_VAL("lock_name is cgroup_mutex",
+ expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /* Read/shared and write/exclusive predicates */
+ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)",
+ __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is &mmap_lock",
+ expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock"));
+ clear_bpf_filter_exprs(expr_head);
+
+ TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)",
+ __perf_bpf_filter__parse(expr_head,
+ "!lock_held_exclusive(mmap_lock)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 0", expr->val == 0);
+ TEST_ASSERT_VAL("lock_name is mmap_lock",
+ expr->lock_name && !strcmp(expr->lock_name, "mmap_lock"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /*
+ * Comparison and compound group:
+ * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100
+ */
+ TEST_ASSERT_VAL("parse compound lock_held filter",
+ __perf_bpf_filter__parse(expr_head, compound) == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN);
+ TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups));
+ group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("first group member lock_name",
+ group->lock_name && !strcmp(group->lock_name, "rwsem:R"));
+ TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1);
+ group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("second group member lock_name",
+ group->lock_name && !strcmp(group->lock_name, "&dentry"));
+ TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0);
+ expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100);
+
+ return TEST_OK;
+}
+#endif
+
+static int test__bpf_filters(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
+{
+#ifdef HAVE_BPF_SKEL
+ struct list_head expr_head = LIST_HEAD_INIT(expr_head);
+ int ret = do_test_bpf_filters(&expr_head);
+
+ clear_bpf_filter_exprs(&expr_head);
+ return ret;
+#else
+ return TEST_SKIP;
+#endif
+}
+
static struct test_case tests__parse_events[] = {
TEST_CASE_REASON("Test event parsing",
events2,
@@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = {
"no aliases in sysfs"),
TEST_CASE("Parsing of aliased events", pmu_events_alias2),
TEST_CASE("Parsing of terms (event modifiers)", terms2),
+ TEST_CASE("Parsing of BPF sample filters", bpf_filters),
{ .name = NULL, }
};
diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh
index 383574cb3bd3..ebaf8ee41622 100755
--- a/tools/perf/tests/shell/record_bpf_filter.sh
+++ b/tools/perf/tests/shell/record_bpf_filter.sh
@@ -176,6 +176,124 @@ test_bpf_filter_cgroup() {
echo "Cgroup bpf-filter test [Success]"
}
+test_bpf_filter_lock_held() {
+ echo "Lock-held bpf-filter test"
+ lock_sym="cgroup_mutex"
+
+ if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms
+ then
+ lock_sym="slab_mutex"
+ fi
+
+ if ! perf record -e task-clock --filter "lock_held(${lock_sym})" \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if [ "$(id -u)" != 0 ]
+ then
+ echo "Lock-held bpf-filter test [Skipped permission]"
+ return
+ fi
+ if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms
+ then
+ echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]"
+ return
+ fi
+ echo "Lock-held bpf-filter test [Failed ${lock_sym}]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(&dentry)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed &dentry]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \
+ -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed mmap_lock]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \
+ -o /dev/null true 2>&1 | grep -q "Kernel BTF lacks required"
+ then
+ echo "Lock-held bpf-filter test [Failed rq_lock/spinlock]"
+ err=1
+ return
+ fi
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(rwsem)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock type]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock \
+ --filter 'lock_held_read(mmap_lock) || lock_held_write(mmap_lock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read/write(mmap_lock)]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mmap_lock:R) || lock_held(mmap_lock:W)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed mmap_lock:R/:W]"
+ err=1
+ return
+ fi
+
+ if perf list tracepoint 2> /dev/null | grep -q kmem:mm_page_alloc
+ then
+ if ! perf record -e kmem:mm_page_alloc --filter 'lock_held(mutex) || lock_held(rwsem)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed tracepoint kmem:mm_page_alloc]"
+ err=1
+ return
+ fi
+ fi
+
+ if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \
+ -o /dev/null true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]"
+ err=1
+ return
+ fi
+
+ if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \
+ -o /dev/null true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]"
+ err=1
+ return
+ fi
+
+ echo "Lock-held bpf-filter test [Success]"
+}
+
test_bpf_filter_priv
if [ $err = 0 ]; then
@@ -198,5 +316,9 @@ if [ $err = 0 ]; then
test_bpf_filter_cgroup
fi
+if [ $err = 0 ]; then
+ test_bpf_filter_lock_held
+fi
+
cleanup
exit $err
--
2.56.0.307.g5fe7bd680c-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v6 10/11] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (8 preceding siblings ...)
2026-10-22 20:30 ` [PATCH v6 09/11] perf test: Add tests for lock_held() BPF sample filter Ian Rogers
@ 2026-10-22 20:30 ` Ian Rogers
2026-10-22 20:44 ` sashiko-bot
2026-10-22 20:30 ` [PATCH v6 11/11] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
10 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-22 20:30 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks
Trace RCU, and SRCU locks:
- 'rcu' (supports both read/shared and write/exclusive modes):
- Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)',
'lock_held(rcu_read_lock)'): directly inspects
current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on
CONFIG_PREEMPT_RCU kernels), accounting for the baseline
rcu_read_lock() nesting acquired by the perf overflow handler itself
(bpf_overflow_handler() and do_perf_sw_event()) with zero extra
fentry/fexit tracing hooks.
- Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'):
attaches fentry/fexit hooks to synchronize_rcu(),
synchronize_rcu_expedited(), synchronize_rcu_tasks(), and
rcu_barrier() to track grace-period and barrier waits.
- 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both
read/shared and write/exclusive modes):
- Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches
fexit/__srcu_read_lock and fentry/__srcu_read_unlock.
- Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'):
attaches fentry/fexit on synchronize_srcu(),
synchronize_srcu_expedited(), and srcu_barrier().
- 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports
read/shared mode only): directly inspects
current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels
with zero extra tracing hooks.
Example usage:
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ]
# perf report --stdio --no-children
# Overhead Command Shared Object Symbol
# ........ ....... ................. ........................
#
14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu
|
---__d_lookup_rcu
lookup_fast
walk_component
link_path_walk
path_lookupat
filename_lookup
vfs_statx
__do_sys_newstat
do_syscall_64
entry_SYSCALL_64_after_hwframe
# perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1
# perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 44 ++--
tools/perf/tests/parse-events.c | 13 +
tools/perf/tests/shell/record_bpf_filter.sh | 20 ++
tools/perf/util/bpf-filter.c | 237 ++++++++++++++++++-
tools/perf/util/bpf_skel/lock_data.h | 47 +++-
tools/perf/util/bpf_skel/sample-filter.h | 14 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 136 ++++++++++-
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 +
tools/perf/util/lock-contention.c | 4 +
tools/perf/util/lock-contention.h | 4 +-
10 files changed, 490 insertions(+), 31 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 8fc3196ed93d..6869a465d8e3 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -225,11 +225,17 @@ OPTIONS
'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache
name prefixed with '&' (e.g. 'lock_held(&dentry)',
'lock_held_read(&dentry)'), a well-known lock name ('mmap_lock',
- 'siglock', 'rq_lock', 'zone_lock'), or a lock type ('mutex',
- 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', 'rwlock:R',
- 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', 'semaphore').
- Both read/shared and write/exclusive tracking are supported for
- 'rwsem' (including 'mmap_lock') and 'rwlock'; 'mutex', 'spinlock'
+ 'siglock', 'rq_lock', 'zone_lock', 'rcu', 'rcu-trace'), or a lock
+ type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
+ 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
+ 'srcu', 'srcu:R', 'srcu:W', 'rcu:R', 'rcu:W', 'semaphore'). Both
+ read/shared and write/exclusive tracking are supported for 'rwsem'
+ (including 'mmap_lock'), 'rwlock', 'rcu', and 'srcu' (where
+ read/shared mode matches 'rcu_read_lock()' / 'srcu_read_lock()'
+ critical sections and write/exclusive mode matches grace-period
+ waits 'synchronize_rcu*()', 'synchronize_srcu*()', 'rcu_barrier()',
+ and 'srcu_barrier()'); 'rcu-trace' ('rcu_read_lock_trace()')
+ supports read/shared tracking only; and 'mutex', 'spinlock'
(including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex',
'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support
write/exclusive tracking only. Note that 'semaphore' tracking
@@ -251,19 +257,21 @@ OPTIONS
--filter 'lock_held(mutex) || lock_held(rwsem)'
When filtering on a specific mutex symbol ('*_mutex'), rt-mutex symbol
- ('*_rt_mutex'), or write-locked 'mmap_lock' ('lock_held_write(mmap_lock)'
- or 'lock_held(mmap_lock:W)'), 'perf_sample_filter' inspects the lock's
- 'owner' field directly in kernel memory on each sample without attaching
- any BPF fentry/fexit hooks. For locks that do not maintain an
- authoritative per-task owner pointer (such as spinlocks, rwlocks,
- semaphores, reader-locked rwsems, and slab/type filters), it attaches
- BPF fentry/fexit programs to kernel lock acquire and release functions
- to track held locks in a BPF map. These BPF hooks run on every kernel
- lock acquisition and release for the attached lock family, even when no
- perf sample is being taken. To minimize this overhead, only the hooks
- for the relevant lock family and mode are attached when known (for
- example, 'lock_held_read(mmap_lock)' only hooks rwsem read functions
- and 'rq_lock' only hooks spinlock functions), whereas slab filters
+ ('*_rt_mutex'), write-locked 'mmap_lock' ('lock_held_write(mmap_lock)'
+ or 'lock_held(mmap_lock:W)'), or 'rcu:R' / 'rcu-trace' (which inspect
+ 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting'),
+ 'perf_sample_filter' inspects the kernel fields directly on each sample
+ without attaching any BPF fentry/fexit hooks. For locks that do not
+ maintain an authoritative per-task owner pointer (such as spinlocks,
+ rwlocks, semaphores, reader-locked rwsems, SRCU locks, RCU synchronize
+ waits, and slab/type filters), it attaches BPF fentry/fexit programs to
+ kernel lock acquire and release functions to track held locks in a BPF
+ map. These BPF hooks run on every kernel lock acquisition and release
+ for the attached lock family, even when no perf sample is being taken.
+ To minimize this overhead, only the hooks for the relevant lock family
+ and mode are attached when known (for example, 'lock_held_read(mmap_lock)'
+ only hooks rwsem read functions, 'srcu' only hooks SRCU functions, and
+ 'rq_lock' only hooks spinlock functions), whereas slab filters
('&<slab>'), raw hex addresses, or symbols without a recognizable
lock-type suffix attach hooks across all lock families (including
high-frequency spinlocks).
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 4cf9192762f2..730466b7ebf6 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head)
expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex"));
clear_bpf_filter_exprs(expr_head);
+ /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */
+ TEST_ASSERT_VAL("parse lock_held(rcu)",
+ __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is rcu",
+ expr->lock_name && !strcmp(expr->lock_name, "rcu"));
+ clear_bpf_filter_exprs(expr_head);
+
/* Read/shared and write/exclusive predicates */
TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)",
__perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0);
diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh
index ebaf8ee41622..47f697f53a85 100755
--- a/tools/perf/tests/shell/record_bpf_filter.sh
+++ b/tools/perf/tests/shell/record_bpf_filter.sh
@@ -247,6 +247,26 @@ test_bpf_filter_lock_held() {
return
fi
+ if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed rcu:W/srcu]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \
+ -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]"
+ err=1
+ return
+ fi
+ fi
+
if ! perf record -e task-clock \
--filter 'lock_held_read(mmap_lock) || lock_held_write(mmap_lock)' \
-o "${perfdata}" true 2> /dev/null
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 1a75d779e891..0161ba1f4769 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -200,11 +200,19 @@ enum lock_prog_group {
LOCK_PROG_RTMUTEX = 1U << 6,
LOCK_PROG_SEM = 1U << 7,
LOCK_PROG_PERCPU_RWSEM = 1U << 8,
- LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM,
+ LOCK_PROG_SRCU_READ = 1U << 9,
+ LOCK_PROG_SRCU_WRITE = 1U << 10,
+ LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE,
+ LOCK_PROG_RCU_WRITE = 1U << 11,
+ LOCK_PROG_ALL = 0xfffU & ~LOCK_PROG_SEM,
};
static unsigned int lock_flags_to_prog_group(unsigned int flags)
{
+ if (flags & LCB_F_RCU)
+ return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0;
+ if (flags & LCB_F_SRCU)
+ return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE;
if (flags & LCB_F_MUTEX)
return LOCK_PROG_MUTEX;
if (flags & LCB_F_RT)
@@ -324,6 +332,32 @@ static const struct lock_prog_desc lock_progs[] = {
/* Percpu-RWsem */
LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+ /* SRCU */
+ LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe",
+ LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe",
+ LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited",
+ LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited",
+ LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE),
+ /* RCU grace-period synchronize / barrier */
+ LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited",
+ LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited",
+ LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE),
};
#undef LOCK_FPROG
@@ -642,6 +676,18 @@ static unsigned int supported_lock_prog_groups(struct btf *btf)
if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write"))
supported |= LOCK_PROG_PERCPU_RWSEM;
+ if ((btf_has_func(btf, "__srcu_read_lock") ||
+ btf_has_func(btf, "__srcu_read_lock_nmisafe")) &&
+ (btf_has_func(btf, "__srcu_read_unlock") ||
+ btf_has_func(btf, "__srcu_read_unlock_nmisafe")))
+ supported |= LOCK_PROG_SRCU_READ;
+
+ if (btf_has_func(btf, "synchronize_srcu"))
+ supported |= LOCK_PROG_SRCU_WRITE;
+
+ if (btf_has_func(btf, "synchronize_rcu"))
+ supported |= LOCK_PROG_RCU_WRITE;
+
return supported;
}
@@ -759,6 +805,19 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
*needs_lock_syms = true;
return;
}
+ if (!strcmp(base, "rcu")) {
+ if (mode & PBF_LOCK_MODE_WRITE) {
+ *prog_groups |= LOCK_PROG_RCU_WRITE;
+ *required_groups |= LOCK_PROG_RCU_WRITE;
+ }
+ return;
+ }
+ if (!strcmp(base, "rcu_read_lock") ||
+ !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") ||
+ !strcmp(base, "rcu_read_lock_trace")) {
+ /* Inspected directly on task_struct; no tracing hooks needed. */
+ return;
+ }
if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) {
*prog_groups |= groups;
*required_groups |= groups;
@@ -800,6 +859,13 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
groups |= LOCK_PROG_RWLOCK_WRITE;
*prog_groups |= groups;
*required_groups |= groups;
+ } else if (str_ends_with(base, "srcu")) {
+ if (mode & PBF_LOCK_MODE_READ)
+ groups |= LOCK_PROG_SRCU_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ groups |= LOCK_PROG_SRCU_WRITE;
+ *prog_groups |= groups;
+ *required_groups |= groups;
} else {
*prog_groups |= LOCK_PROG_ALL;
}
@@ -963,6 +1029,108 @@ static bool is_percpu_rwsem_sym(struct btf *btf, const char *name)
return btf__find_by_name_kind(btf, rc_name, BTF_KIND_VAR) > 0;
}
+static u64 rcu_min_depth(struct evsel *evsel)
+{
+ /*
+ * Non-sleepable BPF sample filter programs always run under at least
+ * one kernel rcu_read_lock():
+ * - PERF_TYPE_PERF_EVENT: bpf_overflow_handler() takes rcu_read_lock()
+ * (and perf_swevent_event() takes an outer rcu_read_lock() for
+ * non-timer PERF_TYPE_SOFTWARE events, giving depth 2).
+ * - Standard tracepoints: trace_call_bpf() takes rcu_read_lock().
+ * - Faultable syscall tracepoints: trace_call_bpf_faultable() invokes
+ * bpf_prog_run_array_sleepable(), which takes guard(rcu)() for
+ * non-sleepable programs (!prog->sleepable) such as
+ * perf_sample_filter_tp.
+ */
+ if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE &&
+ evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK &&
+ evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK)
+ return 2;
+ return 1;
+}
+
+static u64 rcu_trace_min_depth(struct evsel *evsel, struct btf *btf)
+{
+ const char *ev_name;
+
+ /*
+ * Only PERF_TYPE_PERF_EVENT and PERF_TYPE_TRACEPOINT events can attach
+ * sample_filter.bpf.c (kprobes and uprobes require BPF_PROG_TYPE_KPROBE
+ * in __perf_event_set_bpf_prog()).
+ *
+ * For syscall tracepoints:
+ * - raw_syscalls:sys_enter and raw_syscalls:sys_exit are defined with
+ * TRACE_EVENT_SYSCALL(), whose __do_trace_sys_enter/exit wrapper
+ * takes guard(rcu_tasks_trace)() (depth 1) before perf_trace_buf
+ * submission calls trace_call_bpf().
+ * - syscalls:sys_enter_* and syscalls:sys_exit_* register
+ * perf_syscall_enter/exit on the raw sys_enter/sys_exit tracepoints
+ * (1st guard(rcu_tasks_trace)()) and, on kernels with
+ * trace_call_bpf_faultable(), acquire a 2nd guard(rcu_tasks_trace)()
+ * in trace_call_bpf_faultable() (depth 2).
+ */
+ if (evsel && evsel->core.attr.type == PERF_TYPE_TRACEPOINT) {
+ ev_name = evsel__name(evsel);
+ if (!ev_name)
+ return 0;
+ if (strstarts(ev_name, "raw_syscalls:"))
+ return 1;
+ if (strstarts(ev_name, "syscalls:")) {
+ if (!btf ||
+ btf__find_by_name_kind(btf, "trace_call_bpf_faultable",
+ BTF_KIND_FUNC) > 0)
+ return 2;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+static bool btf_struct_has_field(const struct btf *btf, s32 type_id,
+ const char *field_name)
+{
+ const struct btf_type *t;
+ const struct btf_member *m;
+ u16 i, vlen;
+
+ type_id = btf__resolve_type(btf, type_id);
+ if (type_id < 0)
+ return false;
+
+ t = btf__type_by_id(btf, type_id);
+ if (!t || (!btf_is_struct(t) && !btf_is_union(t)))
+ return false;
+
+ m = btf_members(t);
+ vlen = btf_vlen(t);
+ for (i = 0; i < vlen; i++, m++) {
+ const char *name = btf__name_by_offset(btf, m->name_off);
+
+ if (name && *name) {
+ if (!strcmp(name, field_name))
+ return true;
+ } else if (btf_struct_has_field(btf, m->type, field_name)) {
+ return true;
+ }
+ }
+ return false;
+}
+
+static bool btf_has_task_field(struct btf *btf, const char *field_name)
+{
+ s32 id;
+
+ if (!btf)
+ return false;
+
+ id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT);
+ if (id < 0)
+ return false;
+
+ return btf_struct_has_field(btf, id, field_name);
+}
+
/*
* Resolve a `lock_held(<name>)` filter expression into `entry->op`,
* `entry->part` (`enum perf_bpf_lock_target_kind`), and `entry->value`,
@@ -970,15 +1138,15 @@ static bool is_percpu_rwsem_sym(struct btf *btf, const char *name)
*
* Resolution order:
* 1. Slab cache name (when prefixed with '&', e.g. `&dentry`).
- * 2. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
- * `rq_lock`, `zone_lock`.
+ * 2. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`,
+ * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`.
* 3. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`,
- * `rwlock`, `percpu-rwsem`, `semaphore`, etc.
+ * `rwlock`, `percpu-rwsem`, `srcu`, `semaphore`, etc.
* 4. Hexadecimal lock address.
* 5. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`).
*/
-static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf,
- struct perf_bpf_filter_expr *expr,
+static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct btf *btf, struct perf_bpf_filter_expr *expr,
struct perf_bpf_filter_entry *entry)
{
const char *name = expr->lock_name;
@@ -1060,6 +1228,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf,
return 0;
}
+ if (!strcmp(base, "rcu_read_lock")) {
+ if (mode == PBF_LOCK_MODE_WRITE) {
+ pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
+ return -EINVAL;
+ }
+ if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
+ pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ;
+ entry->value = rcu_min_depth(evsel);
+ return 0;
+ }
+
+ if (!strcmp(base, "rcu")) {
+ if ((mode & PBF_LOCK_MODE_READ) && btf &&
+ !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
+ pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU | mode;
+ entry->value = rcu_min_depth(evsel);
+ if ((mode & PBF_LOCK_MODE_WRITE) && skel)
+ skel->bss->has_rcu_lock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") ||
+ !strcmp(base, "rcu_read_lock_trace")) {
+ if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) {
+ pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
+ return -EINVAL;
+ }
+ if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) {
+ pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ;
+ entry->value = rcu_trace_min_depth(evsel, btf);
+ return 0;
+ }
+
if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
entry->part = PBF_LOCK_TYPE | mode;
entry->value = encoded_pair;
@@ -1149,7 +1359,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske
entry[i].term = expr->term;
entry[i].value = expr->val;
if (expr->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(skel, btf, expr, &entry[i]);
+ ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]);
if (ret < 0)
return ret;
}
@@ -1167,7 +1377,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske
entry[i].term = group->term;
entry[i].value = group->val;
if (group->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(skel, btf, group, &entry[i]);
+ ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]);
if (ret < 0)
return ret;
}
@@ -1496,6 +1706,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
init_numa_data(skel, btf);
}
+ if (prog_groups & LOCK_PROG_RCU_WRITE) {
+ u64 rcu_addr = 1ULL;
+
+ /*
+ * Offset by 1 byte so rcu_state_addr is distinct from
+ * &rcu_state.node[0].lock at offset 0 of rcu_state.
+ */
+ if (find_kernel_lock_sym("rcu_state", &rcu_addr) == 0)
+ rcu_addr += 1;
+ skel->rodata->rcu_state_addr = rcu_addr;
+ }
ret = configure_lock_progs(skel, btf, prog_groups, required_groups,
needs_lock_syms, needs_slab_iter,
is_tracepoint);
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index 43d8f5030bc1..f3f8076fdd0b 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -39,6 +39,7 @@ struct contention_task_data {
*/
#define LCD_F_MMAP_LOCK (1U << 31)
#define LCD_F_SIGHAND_LOCK (1U << 30)
+#define LCD_F_RCU_LOCK (1U << 29)
#define LCB_F_SLAB_ID_SHIFT 16
#define LCB_F_SLAB_ID_START (1U << 16)
@@ -52,9 +53,11 @@ struct contention_task_data {
#define LCB_F_RT (1U << 3)
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+#define LCB_F_SRCU (1U << 6)
+#define LCB_F_RCU (1U << 7)
-#define LCB_F_TYPE_MAX (1U << 7)
-#define LCB_F_TYPE_MASK 0x0000007FU
+#define LCB_F_TYPE_MAX (1U << 8)
+#define LCB_F_TYPE_MASK 0x000000FFU
#define SLAB_NAME_MAX 28
@@ -264,6 +267,43 @@ static __always_inline __u64 get_current_siglock(__u64 curr)
return sh + bpf_core_field_offset(sighand->siglock);
}
+/*
+ * Check if `curr` is inside a preemptible RCU read-side critical section
+ * (`rcu_read_lock()`).
+ *
+ * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires
+ * `rcu_read_lock()` before invoking `perf_sample_filter` (and
+ * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for
+ * non-clock software events), `min_depth` is the baseline nesting count
+ * contributed by the perf event overflow handler itself (1 or 2).
+ */
+static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth)
+{
+ struct task_struct *task = (void *)curr;
+
+ if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting))
+ return false;
+
+ return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth;
+}
+
+/*
+ * Check if `curr` is inside a Tasks Trace RCU read-side critical section
+ * (`rcu_read_lock_trace()`). `min_depth` accounts for any baseline nesting
+ * contributed by the kernel tracing wrapper itself (2 on faultable
+ * `syscalls:*` tracepoints via `__DECLARE_TRACE_SYSCALL` and
+ * `trace_call_bpf_faultable()`, 1 on `raw_syscalls:*`, 0 otherwise).
+ */
+static __always_inline bool is_rcu_trace_lock_held(__u64 curr, int min_depth)
+{
+ struct task_struct *task = (void *)curr;
+
+ if (!curr || !bpf_core_field_exists(task->trc_reader_nesting))
+ return false;
+
+ return BPF_CORE_READ(task, trc_reader_nesting) > min_depth;
+}
+
/*
* Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar.
*
@@ -296,6 +336,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags)
if (get_current_siglock(curr) == lock)
return LCD_F_SIGHAND_LOCK;
break;
+ case LCB_F_RCU:
+ case LCB_F_RCU | LCB_F_WRITE:
+ return LCD_F_RCU_LOCK;
default:
break;
}
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index a65ee780d64d..7abc530ed50d 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -101,6 +101,20 @@ enum perf_bpf_lock_target_kind {
* low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
*/
PBF_LOCK_TYPE,
+ /*
+ * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`).
+ * Inspected directly via `current->rcu_read_lock_nesting > entry.value`,
+ * where `entry.value` is the baseline RCU nesting depth added by the
+ * kernel perf overflow path.
+ */
+ PBF_LOCK_RCU,
+ /*
+ * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`).
+ * Inspected directly via `current->trc_reader_nesting > entry.value`,
+ * where `entry.value` is the baseline nesting depth added by the
+ * kernel event tracing path.
+ */
+ PBF_LOCK_RCU_TRACE,
};
/*
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index d1192a93179e..60fe4c3a99a2 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -129,6 +129,7 @@ volatile const int use_idx_hash;
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
+const volatile __u64 rcu_state_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
@@ -142,6 +143,7 @@ int has_lock_class;
int has_lock_type;
int has_mmap_lock;
int has_siglock;
+int has_rcu_lock;
int track_rwsem_read;
__u8 lock_type_filter[LCB_F_TYPE_MAX];
@@ -282,6 +284,21 @@ static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry)
case PBF_LOCK_TYPE:
return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0;
+ case PBF_LOCK_RCU:
+ if ((mode & PBF_LOCK_MODE_READ) &&
+ is_rcu_lock_held(curr, (int)entry->value))
+ return true;
+ if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr &&
+ is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id))
+ return true;
+ return false;
+
+ case PBF_LOCK_RCU_TRACE:
+ if ((mode & PBF_LOCK_MODE_READ) &&
+ is_rcu_trace_lock_held(curr, (int)entry->value))
+ return true;
+ return false;
+
default:
return false;
}
@@ -727,11 +744,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla
if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
matched = true;
- if (has_mmap_lock || has_siglock) {
+ if (has_mmap_lock || has_siglock || has_rcu_lock) {
__u32 ltype = check_lock_type(lock_addr, type_flag);
if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
- (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
+ (has_siglock && ltype == LCD_F_SIGHAND_LOCK) ||
+ (has_rcu_lock && ltype == LCD_F_RCU_LOCK))
matched = true;
}
@@ -1391,6 +1409,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem)
return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
}
+/* SRCU acquire/release and synchronize hooks */
+SEC("fexit/__srcu_read_lock")
+int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/__srcu_read_unlock")
+int BPF_PROG(fentry_srcu_read_unlock, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fexit/__srcu_read_lock_nmisafe")
+int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/__srcu_read_unlock_nmisafe")
+int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/synchronize_srcu")
+int BPF_PROG(fentry_synchronize_srcu, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu")
+int BPF_PROG(fexit_synchronize_srcu, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_srcu_expedited")
+int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu_expedited")
+int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fentry/srcu_barrier")
+int BPF_PROG(fentry_srcu_barrier, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/srcu_barrier")
+int BPF_PROG(fexit_srcu_barrier, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+/* RCU grace-period synchronize / barrier hooks */
+SEC("fentry/synchronize_rcu")
+int BPF_PROG(fentry_synchronize_rcu)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu")
+int BPF_PROG(fexit_synchronize_rcu)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_rcu_expedited")
+int BPF_PROG(fentry_synchronize_rcu_expedited)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu_expedited")
+int BPF_PROG(fexit_synchronize_rcu_expedited)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_rcu_tasks")
+int BPF_PROG(fentry_synchronize_rcu_tasks)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu_tasks")
+int BPF_PROG(fexit_synchronize_rcu_tasks)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/rcu_barrier")
+int BPF_PROG(fentry_rcu_barrier)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/rcu_barrier")
+int BPF_PROG(fexit_rcu_barrier)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
SEC("raw_tp/bpf_test_finish")
int BPF_PROG(collect_lock_syms)
{
diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
index b75ddc6846bb..1fb91a829d54 100644
--- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
+++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
@@ -119,6 +119,8 @@ struct task_struct {
struct signal_struct *signal;
struct sighand_struct *sighand;
struct css_set *cgroups;
+ int rcu_read_lock_nesting;
+ int trc_reader_nesting;
} __attribute__((preserve_access_index));
struct trace_entry {
diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c
index f6e68d740860..37415f7b7f2c 100644
--- a/tools/perf/util/lock-contention.c
+++ b/tools/perf/util/lock-contention.c
@@ -39,6 +39,10 @@ const struct perf_lock_type lock_type_table[] = {
{ LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
/* alias for optimistic spinning only */
{ LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
+ { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" },
+ { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" },
+ { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" },
+ { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" },
};
int parse_call_stack(const struct option *opt __maybe_unused, const char *str,
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index 68e9bd77e8a9..88b2c11244ad 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -128,6 +128,8 @@ struct thread_stat {
#define LCB_F_RT (1U << 3)
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+#define LCB_F_SRCU (1U << 6)
+#define LCB_F_RCU (1U << 7)
struct perf_lock_type {
unsigned int flags;
@@ -140,7 +142,7 @@ struct perf_lock_type {
const char *lock_name;
};
-extern const struct perf_lock_type lock_type_table[14];
+extern const struct perf_lock_type lock_type_table[18];
struct evlist;
struct machine;
--
2.56.0.307.g5fe7bd680c-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* [PATCH v6 11/11] perf lock contention: Extend BPF lock contention to RCU and SRCU
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (9 preceding siblings ...)
2026-10-22 20:30 ` [PATCH v6 10/11] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
@ 2026-10-22 20:30 ` Ian Rogers
2026-10-22 20:43 ` sashiko-bot
10 siblings, 1 reply; 106+ messages in thread
From: Ian Rogers @ 2026-10-22 20:30 UTC (permalink / raw)
To: irogers, acme, namhyung
Cc: adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
Extend 'perf lock contention --use-bpf' to track RCU and SRCU
grace-period and barrier waits:
- Attach fentry/fexit programs to synchronize_rcu(),
synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(),
synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier()
when available in BTF.
- Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu',
'-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct
symbol names).
Example usage:
# perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1
total wait avg wait type caller
35.12 ms 8.78 ms rcu:W cleanup_net+0x284
7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114
# perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1
total wait avg wait address symbol
35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu)
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/tests/shell/lock_contention.sh | 14 ++
tools/perf/util/bpf_lock_contention.c | 174 +++++++++++++-
.../perf/util/bpf_skel/lock_contention.bpf.c | 222 ++++++++++++++----
tools/perf/util/machine.c | 11 +
5 files changed, 366 insertions(+), 62 deletions(-)
diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt
index c17b3e318169..bfeb98c976e3 100644
--- a/tools/perf/Documentation/perf-lock.txt
+++ b/tools/perf/Documentation/perf-lock.txt
@@ -188,11 +188,14 @@ CONTENTION OPTIONS
Show lock contention only for given lock types (comma separated list).
Available values are:
semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W,
- rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem,
- pcpu-sem:R, pcpu-sem:W, mutex
+ rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem,
+ pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W
Note that RW-variant of locks have :R and :W suffix. Names without the
suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W.
+ For RCU and SRCU with --use-bpf, contention tracks grace-period and
+ barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(),
+ srcu_barrier()).
-L::
--lock-filter=<value>::
diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh
index 5df9a0cd48a3..999dcfd1ef26 100755
--- a/tools/perf/tests/shell/lock_contention.sh
+++ b/tools/perf/tests/shell/lock_contention.sh
@@ -181,6 +181,13 @@ test_type_filter()
err=1
exit
fi
+
+ perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result}
+ if [ "$(grep -c -v -w rcu "${result}")" != "0" ]; then
+ echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")"
+ err=1
+ exit
+ fi
}
test_lock_filter()
@@ -224,6 +231,13 @@ test_lock_filter()
err=1
exit
fi
+
+ perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result}
+ if [ "$(grep -c -v -w rcu "${result}")" != "0" ]; then
+ echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")"
+ err=1
+ exit
+ fi
}
test_stack_filter()
diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c
index c20bd075664e..081a1ecc10a6 100644
--- a/tools/perf/util/bpf_lock_contention.c
+++ b/tools/perf/util/bpf_lock_contention.c
@@ -16,6 +16,10 @@
#include <bpf/bpf.h>
#include <bpf/btf.h>
#include <inttypes.h>
+#include <stddef.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
#include "bpf_skel/lock_contention.skel.h"
#include "bpf_skel/lock_data.h"
@@ -180,6 +184,133 @@ static void init_numa_data(struct lock_contention *con)
free(buf);
}
+static bool has_bpf_fentry_support(struct btf *btf)
+{
+ /*
+ * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and
+ * bpf_get_func_arg() helper availability (5.17+) used by
+ * get_lock_addr() in the SRCU fentry/fexit hooks.
+ */
+ struct bpf_insn insns[] = {
+ { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 },
+ { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3,
+ .src_reg = BPF_REG_10 },
+ { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 },
+ { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg },
+ { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 },
+ { .code = BPF_JMP | BPF_EXIT },
+ };
+ LIBBPF_OPTS(bpf_prog_load_opts, opts,
+ .expected_attach_type = BPF_TRACE_FENTRY,
+ );
+ s32 btf_id;
+ int prog_fd, link_fd;
+
+ if (!btf)
+ return false;
+
+ btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC);
+ if (btf_id <= 0)
+ return false;
+
+ opts.attach_btf_id = btf_id;
+ prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL",
+ insns, ARRAY_SIZE(insns), &opts);
+ if (prog_fd < 0)
+ return false;
+
+ link_fd = bpf_raw_tracepoint_open(NULL, prog_fd);
+ if (link_fd >= 0)
+ close(link_fd);
+ close(prog_fd);
+ return link_fd >= 0;
+}
+
+static void configure_rcu_contention_progs(struct lock_contention *con,
+ bool has_rcu_lock)
+{
+ static const struct {
+ size_t prog_offset;
+ const char *func;
+ bool is_srcu;
+ } rcu_progs[] = {
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu),
+ "synchronize_rcu", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu),
+ "synchronize_rcu", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited),
+ "synchronize_rcu_expedited", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited),
+ "synchronize_rcu_expedited", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks),
+ "synchronize_rcu_tasks", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks),
+ "synchronize_rcu_tasks", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier),
+ "rcu_barrier", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier),
+ "rcu_barrier", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu),
+ "synchronize_srcu", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu),
+ "synchronize_srcu", true },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited),
+ "synchronize_srcu_expedited", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited),
+ "synchronize_srcu_expedited", true },
+ { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier),
+ "srcu_barrier", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier),
+ "srcu_barrier", true },
+ };
+ struct symbol *sym;
+ struct map *kmap;
+ bool need_rcu = true, need_srcu = true;
+ bool fentry_ok;
+
+ sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap);
+ if (sym) {
+ /*
+ * Offset by 1 byte so rcu_state_addr is distinct from
+ * &rcu_state.node[0].lock at offset 0 of rcu_state.
+ */
+ skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start) + 1;
+ map__put(kmap);
+ } else {
+ skel->rodata->rcu_state_addr = 1ULL;
+ }
+
+ if (con->filters->nr_types) {
+ need_rcu = false;
+ need_srcu = false;
+ for (int i = 0; i < con->filters->nr_types; i++) {
+ if (con->filters->types[i] & LCB_F_RCU)
+ need_rcu = true;
+ if (con->filters->types[i] & LCB_F_SRCU)
+ need_srcu = true;
+ }
+ }
+
+ if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) {
+ if (!has_rcu_lock && !con->filters->nr_addrs)
+ need_rcu = false;
+ if (!con->filters->nr_addrs && !con->filters->nr_slabs)
+ need_srcu = false;
+ }
+
+ fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf);
+
+ for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) {
+ struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset;
+ bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu;
+ bool enable = fentry_ok && want &&
+ btf__find_by_name_kind(con->btf, rcu_progs[i].func,
+ BTF_KIND_FUNC) > 0;
+
+ bpf_program__set_autoload(*progp, enable);
+ }
+}
+
int lock_contention_prepare(struct lock_contention *con)
{
int i, fd;
@@ -187,6 +318,7 @@ int lock_contention_prepare(struct lock_contention *con)
struct evlist *evlist = con->evlist;
struct target *target = con->target;
bool has_mmap_lock = false;
+ bool has_rcu_lock = false;
/* make sure it loads the kernel map before lookup */
map__load(machine__kernel_map(con->machine));
@@ -200,6 +332,7 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64));
bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries);
bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries);
+ bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries);
if (con->aggr_mode == LOCK_AGGR_TASK)
bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries);
@@ -249,6 +382,10 @@ int lock_contention_prepare(struct lock_contention *con)
has_mmap_lock = true;
continue;
}
+ if (!strcmp(con->filters->syms[i], "rcu")) {
+ has_rcu_lock = true;
+ continue;
+ }
sym = machine__find_kernel_symbol_by_name(con->machine,
con->filters->syms[i],
@@ -269,7 +406,7 @@ int lock_contention_prepare(struct lock_contention *con)
addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start);
con->filters->addrs = addrs;
}
- naddrs = con->filters->nr_addrs ?: has_mmap_lock;
+ naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock);
skel->rodata->has_addr = 1;
}
@@ -305,6 +442,7 @@ int lock_contention_prepare(struct lock_contention *con)
skel->rodata->needs_callstack = con->save_callstack;
skel->rodata->lock_owner = con->owner;
skel->rodata->has_mmap_lock = has_mmap_lock;
+ skel->rodata->has_rcu_lock = has_rcu_lock;
if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) {
if (cgroup_is_v2("perf_event"))
@@ -321,6 +459,7 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map__set_max_entries(skel->maps.slab_filter, nslabs);
init_numa_data(con);
+ configure_rcu_contention_progs(con, has_rcu_lock);
if (lock_contention_bpf__load(skel) < 0) {
pr_err("Failed to load lock-contention BPF skeleton\n");
@@ -503,23 +642,30 @@ static void account_end_timestamp(struct lock_contention *con)
enum lock_aggr_mode aggr_mode = con->aggr_mode;
struct tstamp_data ts_data, *cpu_data;
- /* Iterate per-task tstamp map (key = TID) */
- ts_fd = bpf_map__fd(skel->maps.tstamp);
+ /* Iterate per-task tstamp maps (key = TID) */
+ int ts_fds[] = {
+ bpf_map__fd(skel->maps.tstamp),
+ bpf_map__fd(skel->maps.tstamp_rcu),
+ };
+
stat_fd = bpf_map__fd(skel->maps.lock_stat);
- prev_key = NULL;
- while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) {
- if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) {
- int pid = key;
+ for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) {
+ ts_fd = ts_fds[i];
+ prev_key = NULL;
+ while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) {
+ if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) {
+ int pid = key;
- if (aggr_mode == LOCK_AGGR_TASK && con->owner)
- pid = ts_data.flags;
+ if (aggr_mode == LOCK_AGGR_TASK && con->owner)
+ pid = ts_data.flags;
- update_lock_stat(stat_fd, pid, end_ts, aggr_mode,
- &ts_data);
- }
+ update_lock_stat(stat_fd, pid, end_ts, aggr_mode,
+ &ts_data);
+ }
- prev_key = &key;
+ prev_key = &key;
+ }
}
/* Now it'll check per-cpu tstamp map which doesn't have TID. */
@@ -604,6 +750,8 @@ static const char *lock_contention_get_name(struct lock_contention *con,
return "mmap_lock";
if (flags & LCD_F_SIGHAND_LOCK)
return "siglock";
+ if (flags & LCD_F_RCU_LOCK)
+ return "rcu";
/* global locks with symbols */
sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap);
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index c70c3955e559..8ae3fc889528 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -59,6 +59,18 @@ struct {
__uint(max_entries, MAX_ENTRIES);
} tstamp SEC(".maps");
+/*
+ * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits
+ * so they do not collide with inner sleeping lock contentions (such as
+ * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, int);
+ __type(value, struct tstamp_data);
+ __uint(max_entries, MAX_ENTRIES);
+} tstamp_rcu SEC(".maps");
+
/* maintain per-CPU timestamp at the beginning of contention */
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
@@ -160,6 +172,7 @@ const volatile int has_addr;
const volatile int has_cgroup;
const volatile int has_slab;
const volatile int has_mmap_lock;
+const volatile int has_rcu_lock;
const volatile int needs_callstack;
const volatile int stack_skip;
const volatile int lock_owner;
@@ -170,6 +183,8 @@ const volatile int lock_delay;
/* determine the key of lock stat */
const volatile int aggr_mode;
+const volatile __u64 rcu_state_addr;
+
int enabled;
int perf_subsys_id = -1;
@@ -213,7 +228,7 @@ static inline __u64 get_current_cgroup_id(void)
return BPF_CORE_READ(cgrp, kn, id);
}
-static inline int can_record(u64 *ctx)
+static __always_inline int can_record(__u64 lock, __u32 flags)
{
bool is_addr_ok = false;
@@ -237,7 +252,6 @@ static inline int can_record(u64 *ctx)
if (has_type) {
__u8 *ok;
- __u32 flags = (__u32)ctx[1];
ok = bpf_map_lookup_elem(&type_filter, &flags);
if (!ok)
@@ -246,10 +260,9 @@ static inline int can_record(u64 *ctx)
if (has_addr) {
__u8 *ok;
- __u64 addr = ctx[0];
- ok = bpf_map_lookup_elem(&addr_filter, &addr);
- if (!ok && !has_slab && !has_mmap_lock)
+ ok = bpf_map_lookup_elem(&addr_filter, &lock);
+ if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock)
return 0;
is_addr_ok = !!ok;
@@ -267,26 +280,27 @@ static inline int can_record(u64 *ctx)
if (is_addr_ok)
return 1;
- /* slab and mmap_lock are part of the addr_filter */
+ /* slab, mmap_lock, and rcu_lock are part of the addr_filter */
if (has_slab && bpf_get_kmem_cache) {
__u8 *ok;
- __u64 addr = ctx[0];
long kmem_cache_addr;
- kmem_cache_addr = (long)bpf_get_kmem_cache(addr);
+ kmem_cache_addr = (long)bpf_get_kmem_cache(lock);
ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr);
if (ok)
return 1;
- else if (!has_mmap_lock)
+ else if (!has_mmap_lock && !has_rcu_lock)
return 0;
}
- if (has_mmap_lock) {
- __u64 lock = ctx[0];
- __u32 flag = ctx[1];
+ if (has_mmap_lock || has_rcu_lock) {
+ __u32 ltype = check_lock_type(lock, flags);
- if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK)
- return 0;
+ if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK)
+ return 1;
+ if (has_rcu_lock && ltype == LCD_F_RCU_LOCK)
+ return 1;
+ return 0;
}
return 1;
@@ -346,6 +360,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
{
__u32 pid;
struct tstamp_data *pelem;
+ void *map;
/* Use per-cpu array map for spinlock and rwlock */
if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) {
@@ -358,8 +373,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
return pelem;
}
+ map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp;
pid = bpf_get_current_pid_tgid();
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ pelem = bpf_map_lookup_elem(map, &pid);
/* Do not update the element for nested locks */
if (pelem && pelem->lock)
return NULL;
@@ -367,12 +383,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
if (pelem == NULL) {
struct tstamp_data zero = {};
- if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) {
+ if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) {
__sync_fetch_and_add(&task_fail, 1);
return NULL;
}
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ pelem = bpf_map_lookup_elem(map, &pid);
if (pelem == NULL) {
__sync_fetch_and_add(&task_fail, 1);
return NULL;
@@ -469,21 +485,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags)
}
}
-SEC("tp_btf/contention_begin")
-int contention_begin(u64 *ctx)
+static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags)
{
struct tstamp_data *pelem;
- if (!enabled || !can_record(ctx))
+ if (!enabled || !can_record(lock, flags))
return 0;
- pelem = get_tstamp_elem(ctx[1]);
+ pelem = get_tstamp_elem(flags);
if (pelem == NULL)
return 0;
pelem->timestamp = bpf_ktime_get_ns();
- pelem->lock = (__u64)ctx[0];
- pelem->flags = (__u32)ctx[1];
+ pelem->lock = lock;
+ pelem->flags = flags;
if (aggr_mode == LOCK_AGGR_CGROUP)
pelem->cgroup_id = get_current_cgroup_id();
@@ -554,8 +569,21 @@ int contention_begin(u64 *ctx)
}
}
skip_owner:
- pelem->stack_id = bpf_get_stackid(ctx, &stacks,
- BPF_F_FAST_STACK_CMP | stack_skip);
+ {
+ int skip = stack_skip;
+
+ /*
+ * Unlike tp_btf/contention_begin (which fires inside a
+ * lock slowpath via trace_contention_begin), fentry
+ * RCU/SRCU programs attach directly to the top-level
+ * wait API, which is 2 frames shallower.
+ */
+ if (flags & (LCB_F_RCU | LCB_F_SRCU))
+ skip = stack_skip > 2 ? stack_skip - 2 : 0;
+
+ pelem->stack_id = bpf_get_stackid(ctx, &stacks,
+ BPF_F_FAST_STACK_CMP | skip);
+ }
if (pelem->stack_id < 0)
__sync_fetch_and_add(&stack_fail, 1);
} else if (aggr_mode == LOCK_AGGR_TASK) {
@@ -583,8 +611,7 @@ int contention_begin(u64 *ctx)
return 0;
}
-SEC("tp_btf/contention_end")
-int contention_end(u64 *ctx)
+static __always_inline int do_contention_end(__u64 lock, __s32 ret, bool is_rcu)
{
__u32 pid = 0, idx = 0;
struct tstamp_data *pelem;
@@ -592,30 +619,34 @@ int contention_end(u64 *ctx)
struct contention_data *data;
__u64 timestamp;
__u64 duration;
- bool need_delete = false;
+ void *ts_map = NULL;
if (!enabled)
return 0;
/*
- * For spinlock and rwlock, it needs to get the timestamp for the
- * per-cpu map. However, contention_end does not have the flags
- * so it cannot know whether it reads percpu or hash map.
- *
- * Try per-cpu map first and check if there's active contention.
- * If it is, do not read hash map because it cannot go to sleeping
- * locks before releasing the spinning locks.
+ * RCU/SRCU fexit hooks always use the dedicated per-task tstamp_rcu
+ * map. For tp_btf/contention_end, try the per-cpu map first (used by
+ * spinlocks and rwlocks) and fall back to the per-task tstamp map.
*/
- pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx);
- if (pelem && pelem->lock) {
- if (pelem->lock != ctx[0])
- return 0;
- } else {
+ if (is_rcu) {
pid = bpf_get_current_pid_tgid();
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
- if (!pelem || pelem->lock != ctx[0])
+ pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid);
+ if (!pelem || pelem->lock != lock)
return 0;
- need_delete = true;
+ ts_map = &tstamp_rcu;
+ } else {
+ pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx);
+ if (pelem && pelem->lock) {
+ if (pelem->lock != lock)
+ return 0;
+ } else {
+ pid = bpf_get_current_pid_tgid();
+ pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ if (!pelem || pelem->lock != lock)
+ return 0;
+ ts_map = &tstamp;
+ }
}
timestamp = bpf_ktime_get_ns();
@@ -643,7 +674,6 @@ int contention_end(u64 *ctx)
*/
else {
u32 i = 0;
- s32 ret = (s32)ctx[1];
u64 *buf;
otdata->timestamp = timestamp;
@@ -708,7 +738,7 @@ int contention_end(u64 *ctx)
if (lock_owner)
key.pid = pelem->flags;
else {
- if (!need_delete)
+ if (!ts_map)
pid = bpf_get_current_pid_tgid();
key.pid = pid;
}
@@ -790,11 +820,109 @@ int contention_end(u64 *ctx)
check_lock_delay(pelem->lock);
pelem->lock = 0;
- if (need_delete)
- bpf_map_delete_elem(&tstamp, &pid);
+ if (ts_map)
+ bpf_map_delete_elem(ts_map, &pid);
return 0;
}
+SEC("tp_btf/contention_begin")
+int contention_begin(u64 *ctx)
+{
+ return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]);
+}
+
+SEC("tp_btf/contention_end")
+int contention_end(u64 *ctx)
+{
+ return do_contention_end((__u64)ctx[0], (__s32)ctx[1], false);
+}
+
+/* RCU grace-period synchronize / barrier contention hooks */
+SEC("fentry/synchronize_rcu")
+int BPF_PROG(fentry_synchronize_rcu)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu")
+int BPF_PROG(fexit_synchronize_rcu)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+SEC("fentry/synchronize_rcu_expedited")
+int BPF_PROG(fentry_synchronize_rcu_expedited)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu_expedited")
+int BPF_PROG(fexit_synchronize_rcu_expedited)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+SEC("fentry/synchronize_rcu_tasks")
+int BPF_PROG(fentry_synchronize_rcu_tasks)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu_tasks")
+int BPF_PROG(fexit_synchronize_rcu_tasks)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+SEC("fentry/rcu_barrier")
+int BPF_PROG(fentry_rcu_barrier)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/rcu_barrier")
+int BPF_PROG(fexit_rcu_barrier)
+{
+ return do_contention_end(rcu_state_addr, 0, true);
+}
+
+/* SRCU grace-period synchronize / barrier contention hooks */
+SEC("fentry/synchronize_srcu")
+int BPF_PROG(fentry_synchronize_srcu, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu")
+int BPF_PROG(fexit_synchronize_srcu, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0, true);
+}
+
+SEC("fentry/synchronize_srcu_expedited")
+int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu_expedited")
+int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0, true);
+}
+
+SEC("fentry/srcu_barrier")
+int BPF_PROG(fentry_srcu_barrier, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/srcu_barrier")
+int BPF_PROG(fexit_srcu_barrier, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0, true);
+}
+
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
const volatile int nr_nodes;
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 1d9d3bf57720..0a0f223b3335 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -3379,6 +3379,17 @@ bool machine__is_lock_function(struct machine *machine, u64 addr)
return true;
}
+ {
+ struct symbol *sym = machine__find_kernel_symbol(machine, addr, NULL);
+
+ if (sym && (!strncmp(sym->name, "synchronize_rcu", 15) ||
+ !strncmp(sym->name, "synchronize_srcu", 16) ||
+ !strcmp(sym->name, "__synchronize_srcu") ||
+ !strcmp(sym->name, "rcu_barrier") ||
+ !strcmp(sym->name, "srcu_barrier")))
+ return true;
+ }
+
return false;
}
--
2.56.0.307.g5fe7bd680c-goog
^ permalink raw reply related [flat|nested] 106+ messages in thread
* Re: [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF
2026-10-22 20:29 ` [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers
@ 2026-10-22 20:35 ` sashiko-bot
2026-10-22 22:06 ` Namhyung Kim
1 sibling, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-22 20:35 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> In perf_bpf_filter__prepare(), the unpinned filter path attaches
> skel->progs.perf_sample_filter to each FD(evsel, x, y) using
> bpf_program__attach_perf_event_opts() and ignores the returned
> struct bpf_link *, leaking the allocated struct bpf_link_perf and its
> BPF_PERF_EVENT link fd.
>
> Furthermore, bpf_program__attach_perf_event_opts() is designed for
> callers that transfer ownership of the perf_event fd to the returned
> bpf_link: when bpf_link__destroy() is called, bpf_link_perf_detach()
> unconditionally disables and closes perf_link->perf_event_fd. Because
> evsel->core.fd owns FD(evsel, x, y) and closes them in evlist__close(),
> destroying the returned bpf_link would double-close the perf_event file
> descriptors.
>
> Fix this by attaching the program directly with
> [ ... ]
>
> Reported-by: Namhyung Kim <namhyung@kernel.org>
> Closes: https://lore.kernel.org/linux-perf-users/20261020053226.55936-1-namhyung@kernel.org/
> Fixes: 56ec9457a4a2 ("perf bpf filter: Implement event sample filtering")
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261022203003.1779152-1-irogers@google.com?part=1
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 03/11] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-22 20:29 ` [PATCH v6 03/11] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
@ 2026-10-22 20:36 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-22 20:36 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers
> (get_lock_owner, get_current_mmap_lock, get_current_siglock,
> get_lock_addr, check_lock_type, do_collect_zone_lock,
> do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into
> tools/perf/util/bpf_skel/lock_data.h and tools/perf/util/lock-contention.[ch]
> so they can be reused by BPF sample filtering.
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261022203003.1779152-1-irogers@google.com?part=3
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 08/11] perf bpf-filter: Support slab cache locks (&<name>) in lock_held()
2026-10-22 20:30 ` [PATCH v6 08/11] perf bpf-filter: Support slab cache locks (&<name>) " Ian Rogers
@ 2026-10-22 20:37 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-22 20:37 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Extend the lock_held() BPF sample filter to support dynamically
> allocated kernel locks inside slab caches prefixed with '&' (e.g.
> lock_held(&dentry), lock_held_read(&dentry), lock_held_write(&dentry)).
>
> Reuse do_slab_cache_iter() from lock_data.h via a SEC("iter/kmem_cache")
> program to resolve the requested slab cache name to its kernel
> struct kmem_cache * address, and use the bpf_get_kmem_cache() kfunc in
> record_lock_acquired_owner() and record_lock_released_addr() to match
> held locks allocated from that slab cache.
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261022203003.1779152-1-irogers@google.com?part=8
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 09/11] perf test: Add tests for lock_held() BPF sample filter
2026-10-22 20:30 ` [PATCH v6 09/11] perf test: Add tests for lock_held() BPF sample filter Ian Rogers
@ 2026-10-22 20:37 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-22 20:37 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Add unit tests in parse-events.c for parsing lock_held(),
> lock_held_read(), and lock_held_exclusive() BPF filter expressions
> (standalone predicates, negated predicates, and compound group
> expressions) and extend record_bpf_filter.sh with lock_held(),
> lock_held_read()/lock_held_write(), and ':R'/':W' recording tests.
>
> Example output:
>
> # perf test -v 'Parsing of BPF sample filters'
> 5: Parse event definition strings :
> 5.7: Parsing of BPF sample filters : Ok
>
> # perf test -v 'perf record sample filtering (by BPF) tests'
> 112: perf record sample filtering (by BPF) tests :
> Checking BPF-filter privilege
> Basic bpf-filter test
> Basic bpf-filter test [Success]
> Failing bpf-filter test
> Failing bpf-filter test [Success]
> Group bpf-filter test
> Group bpf-filter test [Success]
> Multiple bpf-filter test
> Multiple bpf-filter test [Success]
> Cgroup bpf-filter test
> Cgroup bpf-filter test [Success]
> Lock-held bpf-filter test
> Lock-held bpf-filter test [Success]
> 112: perf record sample filtering (by BPF) tests : Ok
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261022203003.1779152-1-irogers@google.com?part=9
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 04/11] perf lock contention: Extend get_lock_owner() for rt_mutex and mutex handoff
2026-10-22 20:29 ` [PATCH v6 04/11] perf lock contention: Extend get_lock_owner() for rt_mutex and mutex handoff Ian Rogers
@ 2026-10-22 20:41 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-22 20:41 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Extend get_lock_owner() in tools/perf/util/bpf_skel/lock_data.h to:
> - Support rt_mutex locks (LCB_F_RT) and CONFIG_PREEMPT_RT mutexes via
> struct rt_mutex_base and struct rt_mutex___old, masking out the low
> waiter bit (0x1UL).
> - Return NULL when a mutex has MUTEX_FLAG_HANDOFF (0x4UL) set on
> owner.counter and is in the middle of being handed off rather than
> actively held.
>
> Also add rw_semaphore.count and struct rt_mutex_base to the fallback
> tools/perf/util/bpf_skel/vmlinux/vmlinux.h.
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261022203003.1779152-1-irogers@google.com?part=4
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 11/11] perf lock contention: Extend BPF lock contention to RCU and SRCU
2026-10-22 20:30 ` [PATCH v6 11/11] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
@ 2026-10-22 20:43 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-22 20:43 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Extend 'perf lock contention --use-bpf' to track RCU and SRCU
> grace-period and barrier waits:
>
> - Attach fentry/fexit programs to synchronize_rcu(),
> synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(),
> synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier()
> when available in BTF.
> - Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu',
> '-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct
> symbol names).
>
> Example usage:
>
> # perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1
> total wait avg wait type caller
>
> 35.12 ms 8.78 ms rcu:W cleanup_net+0x284
> 7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114
>
> # perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1
> total wait avg wait address symbol
>
> 35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu)
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261022203003.1779152-1-irogers@google.com?part=11
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 10/11] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
2026-10-22 20:30 ` [PATCH v6 10/11] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
@ 2026-10-22 20:44 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-22 20:44 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks
> Trace RCU, and SRCU locks:
>
> - 'rcu' (supports both read/shared and write/exclusive modes):
> - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)',
> 'lock_held(rcu_read_lock)'): directly inspects
> current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on
> CONFIG_PREEMPT_RCU kernels), accounting for the baseline
> rcu_read_lock() nesting acquired by the perf overflow handler itself
> (bpf_overflow_handler() and do_perf_sw_event()) with zero extra
> fentry/fexit tracing hooks.
> - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'):
> attaches fentry/fexit hooks to synchronize_rcu(),
> synchronize_rcu_expedited(), synchronize_rcu_tasks(), and
> rcu_barrier() to track grace-period and barrier waits.
> [ ... ]
>
> # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1
> # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261022203003.1779152-1-irogers@google.com?part=10
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 05/11] perf bpf-filter: Add lock_held() sample filter for global mutexes
2026-10-22 20:29 ` [PATCH v6 05/11] perf bpf-filter: Add lock_held() sample filter for global mutexes Ian Rogers
@ 2026-10-22 20:46 ` sashiko-bot
2026-10-22 22:55 ` Namhyung Kim
1 sibling, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-22 20:46 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf bpf-filter: fix module symbol matching and _edata bound in find_lock_sym_cb()
--
commit 50246e2587ada8982167d48e27c24df7afceb9ec
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Add lock_held() sample filter for global mutexes
This commit adds BPF sample filter predicates such as lock_held(<lock>) to
record perf event samples only when a specified kernel lock is held or not
held. In this initial step, global and static kernel mutex symbols are
resolved via /proc/kallsyms and inspected in kernel memory via
get_lock_owner().
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 0d3c700d3dcd5..639db6c031379 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
[ ... ]
> @@ -169,9 +178,263 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
[ ... ]
> +struct lock_sym_cb_args {
> + const char *name;
> + u64 addr;
> + bool in_text;
> + bool in_data;
> + bool found;
> +};
> +
> +static int find_lock_sym_cb(void *arg, const char *name, char type, u64 start)
> +{
> + struct lock_sym_cb_args *args = arg;
> +
> + if (!strcmp(name, "_text") || !strcmp(name, "_stext") ||
> + !strcmp(name, "_sinittext")) {
> + args->in_text = true;
> + return 0;
> + }
> + if (!strcmp(name, "_etext") || !strcmp(name, "_einittext")) {
> + args->in_text = false;
> + return 0;
> + }
> + if (!strcmp(name, "_sdata") || !strcmp(name, "__bss_start")) {
> + args->in_text = false;
> + args->in_data = true;
> + return 0;
> + }
> + if (!strcmp(name, "__bss_stop")) {
> + args->in_data = false;
> + return 0;
> + }
> +
> + /*
> + * Ignore kernel text functions, module text functions, and
> + * absolute/undefined symbols. On CONFIG_LTO_CLANG kernels, vmlinux
> + * .data symbols between _sdata and _edata may have kallsyms type
> + * 'T'/'t', so allow 'T'/'t' within vmlinux data/bss (args->in_data).
> + */
> + if (args->in_text || type == 'A' || type == 'a' ||
> + type == 'U' || type == 'u' ||
> + (!args->in_data && kallsyms__is_function(type)))
> + return 0;
> +
> + if (!strcmp(name, args->name)) {
> + args->found = true;
> + args->addr = start;
> + if (start != 0)
> + return 1;
> + }
[Severity: Medium]
When resolving a lock symbol through the call chain:
perf_bpf_filter__prepare()
-> get_filter_entries()
-> resolve_lock_expr()
-> find_kernel_lock_sym()
-> kallsyms__parse(..., find_lock_sym_cb)
there appear to be two symbol resolution issues in find_lock_sym_cb() in
tools/perf/util/bpf-filter.c:
First, in tools/lib/symbol/kallsyms.c, kallsyms__parse() reads each symbol
name up to the newline character:
for (i = 0; i < KSYM_NAME_LEN; i++) {
ch = io__get_char(&io);
if (ch < 0 || ch == '\n')
break;
symbol_name[i] = ch;
}
symbol_name[i] = '\0';
In /proc/kallsyms, module symbols are formatted with a trailing module name
suffix ("\t[<module>]"). Because find_lock_sym_cb() compares names using
!strcmp(name, args->name), will lock symbols residing in loaded kernel
modules always fail to match? Other kallsyms callbacks such as
kern_sym_name_match() in tools/perf/util/auxtrace.c handle this by allowing a
tab delimiter after the symbol name:
static bool kern_sym_name_match(const char *kname, const char *name)
{
size_t n = strlen(name);
return !strcmp(kname, name) ||
(!strncmp(kname, name, n) && kname[n] == '\t');
}
Second, find_lock_sym_cb() sets args->in_data = true at both _sdata and
__bss_start, and the comment notes that .data symbols between _sdata and
_edata may have type 'T' or 't'. However, only __bss_stop resets
args->in_data to false. Should _edata also reset args->in_data = false so
that sections between _edata and __bss_start (such as .exit.text outside
_sinittext.._einittext) are not treated as vmlinux data/bss?
> + return 0;
> +}
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261022203003.1779152-1-irogers@google.com?part=5
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 07/11] perf bpf-filter: Support per-process and semi-global locks in lock_held()
2026-10-22 20:29 ` [PATCH v6 07/11] perf bpf-filter: Support per-process and semi-global locks in lock_held() Ian Rogers
@ 2026-10-22 20:49 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-22 20:49 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf bpf-filter: unhandled zone_lock resolution errors in init_numa_data()
--
commit f38f272294852331e8bd18c104c37ae24d5b109c
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Support per-process and semi-global locks in lock_held()
This commit extends the lock_held() BPF sample filter to support the
per-process locks mmap_lock and siglock as well as the semi-global locks
rq_lock and zone_lock. For rq_lock and zone_lock, it populates the lock_syms
BPF map at filter setup time via a raw_tp/bpf_test_finish program.
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 8797f67698f7d..f542920f914fc 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
> @@ -534,6 +534,54 @@ static bool parse_lock_hex_addr(const char *str, u64 *addr)
> return true;
> }
>
> +static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf)
> +{
> + u64 addr = 0;
> + char *buf = NULL, *p;
> + size_t len;
> + long last = -1;
> + int ret;
> +
> + if (!btf)
> + return;
> +
> + ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT);
> + if (ret < 0)
> + return;
> +
> + ret = btf__resolve_size(btf, ret);
> + if (ret < 0)
> + return;
> +
> + skel->rodata->sizeof_zone = ret;
> +
> + if (find_kernel_lock_sym("contig_page_data", &addr) == 0) {
> + skel->rodata->contig_page_data_addr = addr;
> + return;
> + }
> +
> + if (find_kernel_lock_sym("node_data", &addr) != 0)
> + return;
> +
> + skel->rodata->node_data_addr = addr;
> +
> + if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) {
> + p = buf;
> + while (p && *p) {
> + char *end;
> +
> + last = strtol(p, &end, 0);
> + if (end == p)
> + break;
> + p = end;
> + if (*p == ',' || *p == '-' || *p == '\n')
> + p++;
> + }
> + skel->rodata->nr_nodes = last + 1;
> + free(buf);
> + }
> +}
> +
> static bool btf_has_func(struct btf *btf, const char *name)
[ ... ]
> @@ -657,7 +707,8 @@ static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
> static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
> bool *has_lock_filter,
> unsigned int *prog_groups,
> - unsigned int *required_groups)
> + unsigned int *required_groups,
> + bool *needs_lock_syms)
> {
> char base[128];
> bool has_amp;
> @@ -672,6 +723,29 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
> sizeof(base), &has_amp, &mode) < 0 || has_amp)
> return;
>
> + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
> + /*
> + * Write ownership of mm->mmap_lock is inspected directly via
> + * is_rwsem_writer_owner() without tracing hooks; only reader
> + * tracking requires LOCK_PROG_RWSEM_READ.
> + */
> + if (mode & PBF_LOCK_MODE_READ) {
> + *prog_groups |= LOCK_PROG_RWSEM_READ;
> + *required_groups |= LOCK_PROG_RWSEM_READ;
> + }
> + return;
> + }
> + if (!strcmp(base, "siglock")) {
> + *prog_groups |= LOCK_PROG_SPIN;
> + *required_groups |= LOCK_PROG_SPIN;
> + return;
> + }
> + if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) {
> + *prog_groups |= LOCK_PROG_SPIN;
> + *required_groups |= LOCK_PROG_SPIN;
> + *needs_lock_syms = true;
> + return;
> + }
[ ... ]
> @@ -853,6 +932,53 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf,
[ ... ]
> + if (!strcmp(base, "zone_lock")) {
> + if (!(mode & PBF_LOCK_MODE_WRITE)) {
> + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
> + name);
> + return -EINVAL;
> + }
> + entry->part = PBF_LOCK_CLASS | mode;
> + entry->value = LOCK_CLASS_ZONE_LOCK;
> + if (skel)
> + skel->bss->has_lock_class = 1;
> + return 0;
> + }
[ ... ]
> @@ -1280,12 +1406,16 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> btf = btf__load_vmlinux_btf();
> bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES);
> bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS);
> + if (needs_lock_syms) {
> + bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
> + init_numa_data(skel, btf);
> + }
[Severity: Medium]
In perf_bpf_filter__prepare() -> prescan_lock_filters() ->
prescan_one_lock_expr(), both rq_lock and zone_lock set *needs_lock_syms =
true, and perf_bpf_filter__prepare() unconditionally calls
init_numa_data(skel, btf) whenever needs_lock_syms is set.
When a user only specifies lock_held(rq_lock), which resolves runqueues via
BPF BTF ksym and does not use kallsyms or NUMA zone data, init_numa_data()
still queries kallsyms for contig_page_data and node_data. If /proc/kallsyms
addresses are restricted (for instance under kptr_restrict=2 or when running
with CAP_BPF without CAP_SYSLOG), find_kernel_lock_sym() prints spurious
permission error messages for NUMA symbols that rq_lock does not use. Also,
on a UMA kernel where contig_page_data exists in /proc/kallsyms with address
0, find_kernel_lock_sym("contig_page_data", &addr) prints an error and
returns -EACCES, and init_numa_data() then falls through to query node_data
as well.
Conversely, when a user specifies lock_held(zone_lock), init_numa_data()
returns void and silently ignores failures if BTF lookup or size resolution
for struct zone fails, if both find_kernel_lock_sym("contig_page_data",
&addr) and find_kernel_lock_sym("node_data", &addr) fail, or if
sysfs__read_str("devices/system/node/online", ...) fails (leaving
skel->rodata->nr_nodes as 0).
In resolve_lock_expr(), the zone_lock branch then succeeds without checking
whether skel->rodata->contig_page_data_addr or skel->rodata->node_data_addr
(with skel->rodata->nr_nodes > 0) was populated. As a result,
collect_lock_syms() runs do_collect_zone_lock() with zeroed addresses,
populates zero LOCK_CLASS_ZONE_LOCK entries in lock_syms, and attaches
kernel-wide spinlock fentry/fexit programs for a filter that can never match
any zone_lock.
Should init_numa_data() only be invoked when zone_lock is used, and should
failures to resolve the NUMA zone parameters be propagated so that
perf_bpf_filter__prepare() aborts instead of attaching a non-functional
zone_lock filter?
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261022203003.1779152-1-irogers@google.com?part=7
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types
2026-10-22 20:29 ` [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types Ian Rogers
@ 2026-10-22 20:55 ` sashiko-bot
2026-10-22 22:56 ` Namhyung Kim
1 sibling, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-22 20:55 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] perf bpf-filter: prescan_one_lock_expr() misses percpu_rw_semaphore symbols
- [Medium] perf bpf-filter: '*_sem' rw_semaphore and percpu_rw_semaphore symbols misclassified as counting semaphores
--
commit 088eb8ec579be2bc3ffee09560153843b89f7a25
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Extend lock_held() to other kernel lock types
This commit extends the lock_held() BPF sample filter to support additional
kernel lock types, lock type flags, and hexadecimal lock addresses. It uses
direct owner inspection where possible and attaches fentry/fexit BPF hooks
to track lock acquisitions and releases in an LRU hash map otherwise.
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 639db6c031379..8797f67698f7d 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
[ ... ]
> @@ -301,26 +517,225 @@ static int find_kernel_lock_sym(const char *name, u64 *addr)
[ ... ]
> static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
> - bool *has_lock_filter)
> + bool *has_lock_filter,
> + unsigned int *prog_groups,
> + unsigned int *required_groups)
> {
> + char base[128];
> + bool has_amp;
> + unsigned int mode, groups = 0;
> + u64 dummy;
> +
> if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name)
> return;
>
> *has_lock_filter = true;
> + if (split_lock_name_mode(expr->lock_name, expr->part, base,
> + sizeof(base), &has_amp, &mode) < 0 || has_amp)
> + return;
> +
> + if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) {
> + *prog_groups |= groups;
> + *required_groups |= groups;
> + return;
> + }
> +
> + if (parse_lock_hex_addr(base, &dummy)) {
> + *prog_groups |= LOCK_PROG_ALL;
> + return;
> + }
> +
> + if (str_ends_with(base, "rt_mutex") || str_ends_with(base, "rtmutex") ||
> + str_ends_with(base, "mutex")) {
> + /* Inspected directly via lock->owner; no tracing hooks needed. */
> + return;
> + } else if (str_ends_with(base, "percpu_rwsem") ||
> + str_ends_with(base, "pcpu_rwsem")) {
> + *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
> + *required_groups |= LOCK_PROG_PERCPU_RWSEM;
> + } else if (str_ends_with(base, "rwsem")) {
> + if (mode & PBF_LOCK_MODE_READ) {
> + *prog_groups |= LOCK_PROG_RWSEM_READ;
> + *required_groups |= LOCK_PROG_RWSEM_READ;
> + }
> + if (mode & PBF_LOCK_MODE_WRITE) {
> + *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM;
> + *required_groups |= LOCK_PROG_RWSEM_WRITE;
> + }
> + } else if (str_ends_with(base, "spinlock")) {
> + *prog_groups |= LOCK_PROG_SPIN;
> + *required_groups |= LOCK_PROG_SPIN;
> + } else if (str_ends_with(base, "sem")) {
> + *prog_groups |= LOCK_PROG_SEM;
> + *required_groups |= LOCK_PROG_SEM;
> + } else if (str_ends_with(base, "rwlock")) {
> + if (mode & PBF_LOCK_MODE_READ)
> + groups |= LOCK_PROG_RWLOCK_READ;
> + if (mode & PBF_LOCK_MODE_WRITE)
> + groups |= LOCK_PROG_RWLOCK_WRITE;
> + *prog_groups |= groups;
> + *required_groups |= groups;
> + } else {
> + *prog_groups |= LOCK_PROG_ALL;
> + }
> }
[Severity: Medium]
In perf_bpf_filter__prepare(), prescan_lock_filters() calls
prescan_one_lock_expr() before btf__load_vmlinux_btf() is called:
perf_bpf_filter__prepare()
prescan_lock_filters()
prescan_one_lock_expr()
...
btf = btf__load_vmlinux_btf();
configure_lock_progs(skel, btf, prog_groups, required_groups, ...);
sample_filter_bpf__load(skel);
get_filter_entries(evsel, skel, btf, entry)
resolve_lock_expr(skel, btf, ...)
is_percpu_rwsem_sym(btf, base)
Because prescan_one_lock_expr() only checks whether base ends with
"percpu_rwsem" or "pcpu_rwsem" without calling is_percpu_rwsem_sym(),
won't percpu_rw_semaphore symbols with other suffixes be misclassified
during prescan?
For example, when filtering on dup_mmap_sem or bp_cpuinfo_sem,
prescan_one_lock_expr() matches str_ends_with(base, "sem") and only sets
LOCK_PROG_SEM. Then configure_lock_progs() disables autoload for
fexit_percpu_down_write and fentry_percpu_up_write before the skeleton is
loaded. When resolve_lock_expr() later runs and identifies the symbol via
is_percpu_rwsem_sym(btf, base), the LOCK_PROG_PERCPU_RWSEM programs were
never loaded or attached.
Similarly, for percpu_rw_semaphore symbols ending in "_rwsem" (such as
cgroup_threadgroup_rwsem) or "_lock" (such as cpu_hotplug_lock and
mem_hotplug_lock), wouldn't prescan_one_lock_expr() enable unnecessary
LOCK_PROG_RWSEM or LOCK_PROG_ALL hooks across other lock families?
> @@ -390,14 +805,42 @@ static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr,
> return -EINVAL;
> }
>
> -static int resolve_lock_expr(struct perf_bpf_filter_expr *expr,
> +static bool is_percpu_rwsem_sym(struct btf *btf, const char *name)
> +{
> + char rc_name[160];
> +
> + if (str_ends_with(name, "percpu_rwsem") ||
> + str_ends_with(name, "pcpu_rwsem"))
> + return true;
> +
> + if (!btf)
> + return false;
> +
> + snprintf(rc_name, sizeof(rc_name), "__percpu_rwsem_rc_%s", name);
> + return btf__find_by_name_kind(btf, rc_name, BTF_KIND_VAR) > 0;
> +}
[ ... ]
> @@ -410,29 +853,79 @@ static int resolve_lock_expr(struct perf_bpf_filter_expr *expr,
[ ... ]
> + ret = find_kernel_lock_sym(base, &addr);
> + if (ret == 0) {
> + unsigned int direct_flag = 0;
> + bool is_pcpu_rwsem = is_percpu_rwsem_sym(btf, base);
> +
> + if (str_ends_with(base, "rt_mutex") || str_ends_with(base, "rtmutex"))
> + direct_flag = PBF_LOCK_DIRECT_RTMUTEX;
> + else if (str_ends_with(base, "mutex"))
> + direct_flag = PBF_LOCK_DIRECT_MUTEX;
> + else if (str_ends_with(base, "rwsem") && !is_pcpu_rwsem)
> + direct_flag = PBF_LOCK_DIRECT_RWSEM;
> +
> + if ((direct_flag == PBF_LOCK_DIRECT_MUTEX ||
> + direct_flag == PBF_LOCK_DIRECT_RTMUTEX ||
> + is_pcpu_rwsem ||
> + str_ends_with(base, "spinlock") ||
> + (str_ends_with(base, "sem") &&
> + !str_ends_with(base, "rwsem"))) &&
> + !(mode & PBF_LOCK_MODE_WRITE)) {
> + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
> + name);
> + return -EINVAL;
> + }
[Severity: Medium]
How are global or static rw_semaphore locks whose names end in "_sem"
(such as namespace_sem, uts_sem, pci_bus_sem, or dev_addr_sem, defined
with DECLARE_RWSEM) handled here and in prescan_one_lock_expr()?
Because both functions classify any symbol ending in "sem" (and not
"rwsem") as a counting semaphore:
1. In resolve_lock_expr(), read-mode filters such as
lock_held_read(namespace_sem) or lock_held(uts_sem:R) hit the
str_ends_with(base, "sem") && !str_ends_with(base, "rwsem") check
when !(mode & PBF_LOCK_MODE_WRITE) and are rejected with -EINVAL.
2. For write-mode or any-mode filters such as lock_held(namespace_sem)
or lock_held_write(uts_sem), prescan_one_lock_expr() only enables
LOCK_PROG_SEM (down/up) and resolve_lock_expr() leaves direct_flag as 0,
so neither PBF_LOCK_DIRECT_RWSEM nor the down_read/down_write hooks are
used.
Could BTF type inspection on the symbol (or checking whether the BTF var
type is struct rw_semaphore vs struct semaphore) be used to distinguish
rw_semaphore symbols ending in "_sem" from counting semaphores?
> +
> + entry->part = PBF_LOCK_ADDR | mode | direct_flag;
> + entry->value = addr;
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261022203003.1779152-1-irogers@google.com?part=6
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 02/11] perf bpf-filter: Support BPF sample filters on tracepoint events
2026-10-22 20:29 ` [PATCH v6 02/11] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
@ 2026-10-22 20:55 ` sashiko-bot
0 siblings, 0 replies; 106+ messages in thread
From: sashiko-bot @ 2026-10-22 20:55 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] perf parse-events: is_possible_tp_filter() misroutes generic cpu tp filters
- [Medium] perf parse-events: misrouted tracepoint filters in is_possible_tp_filter()
--
commit dbe9f68d8a043d94d321ad8e9a98e3b1d3114baf
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Support BPF sample filters on tracepoint events
This commit adds a BPF_PROG_TYPE_TRACEPOINT sample filter program to evaluate
context-independent filter terms (uid, gid, tid, pid, cpu, time, cgroup) on
tracepoint events, and updates set_filter() and perf_bpf_filter__prepare() to
route and attach those filters.
> diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
> index 2f3317ce17f08..d6bab89ade5de 100644
> --- a/tools/perf/util/parse-events.c
> +++ b/tools/perf/util/parse-events.c
[ ... ]
> @@ -2593,10 +2595,40 @@ foreach_evsel_in_last_glob(struct evlist *evlist,
> return 0;
> }
>
> +static bool is_filter_word_char(char c)
> +{
> + return isalnum((unsigned char)c) || c == '_' || c == '/' || c == '.';
> +}
> +
> +static bool has_filter_word(const char *str, const char *word)
> +{
> + size_t len = strlen(word);
> + const char *p = str;
> +
> + while ((p = strstr(p, word)) != NULL) {
> + bool start_ok = (p == str) || !is_filter_word_char(p[-1]);
> + bool end_ok = !is_filter_word_char(p[len]);
> +
> + if (start_ok && end_ok)
> + return true;
> + p += len;
> + }
> + return false;
> +}
> +
> /* Will a tracepoint filter work for str or should a BPF filter be used? */
> -static bool is_possible_tp_filter(const char *str)
> +static bool is_possible_tp_filter(struct evsel *evsel, const char *str)
> {
> - return strstr(str, "uid") == NULL;
> + static const char *const bpf_tp_terms[] = {
> + "uid", "gid", "tid", "pid", "cpu", "time", "cgroup",
> + };
[Severity: Medium]
In tools/perf/util/parse-events.c, is_possible_tp_filter() includes "cpu" in
bpf_tp_terms[] and returns false whenever evsel__field(evsel, "cpu") is NULL.
In kernel/trace/trace_events.c, trace_define_generic_fields() registers "cpu"
in ftrace_generic_fields for all tracepoint events:
kernel/trace/trace_events.c:trace_define_generic_fields() {
int ret;
__generic_field(int, CPU, FILTER_CPU);
__generic_field(int, cpu, FILTER_CPU);
__generic_field(int, common_cpu, FILTER_CPU);
While trace_find_event_field() resolves ftrace_generic_fields when applying a
tracepoint filter via PERF_EVENT_IOC_SET_FILTER, f_next() and f_show() only
emit ftrace_common_fields and event-specific fields in the tracefs format
file parsed by evsel__field().
For any tracepoint that does not define its own event-specific "cpu" field
(such as sched:sched_switch or kmem:mm_page_alloc), evsel__field(evsel, "cpu")
returns NULL, so parse_filter() -> set_filter() -> is_possible_tp_filter()
returns false.
Will this cause valid ftrace tracepoint predicate filters referencing the
generic "cpu" field (such as compound filters combining "cpu" with
event-specific fields, "&&", parentheses, or "CPUS{...}" masks) to be
misrouted to perf_bpf_filter__parse() and fail to parse, while forcing
standalone "cpu" tracepoint filters onto the shared
trace_event_call->prog_array BPF filter requiring CAP_BPF?
> +
> + for (size_t i = 0; i < ARRAY_SIZE(bpf_tp_terms); i++) {
> + if (has_filter_word(str, bpf_tp_terms[i]) &&
> + !evsel__field(evsel, bpf_tp_terms[i]))
> + return false;
> + }
> + return true;
> }
[Severity: Medium]
Can parse_filter() -> foreach_evsel_in_last_glob() -> set_filter() ->
is_possible_tp_filter() in tools/perf/util/parse-events.c also misroute
tracepoint filters in the following cases?
First, when perf is built without HAVE_LIBTRACEEVENT (NO_LIBTRACEEVENT=1),
PERF_TYPE_TRACEPOINT events and kernel ftrace filters (applied via
PERF_EVENT_IOC_SET_FILTER in perf_evsel__apply_filter()) are still supported
via tp_pmu__id(), but the stub in tools/perf/util/evsel.h unconditionally
returns NULL:
tools/perf/util/evsel.h:
static inline struct tep_format_field *
evsel__field(struct evsel *evsel __maybe_unused, const char *name __maybe_unused)
{
return NULL;
}
When evsel__field() returns NULL, any tracepoint filter referencing a format
field whose name appears in bpf_tp_terms[] (such as "pid" on
sched:sched_wakeup, sched:sched_process_fork, or signal:signal_generate)
causes is_possible_tp_filter() to return false. A filter like
--filter "pid == 1234" on sched:sched_wakeup will then be routed to
perf_tp_sample_filter and match current's TGID (bpf_get_current_pid_tgid() >>
32) instead of the tracepoint's target "pid" field, while compound filters
like --filter "pid == 1234 && success == 1" will fail to parse in
perf_bpf_filter__parse().
Second, even with HAVE_LIBTRACEEVENT enabled, has_filter_word() scans the raw
filter string with strstr() without skipping quoted string literals, and
is_filter_word_char() treats quotes, hyphens, and glob characters as word
boundaries. On an event without a field of that name (for example,
sched:sched_switch), a valid tracepoint string filter such as
prev_comm == "cpu" or next_comm == "systemd-time" matches inside the quoted
literal, causing is_possible_tp_filter() to return false and
perf_bpf_filter__parse() to reject the filter.
Third, if a filter on an event like sched:sched_wakeup combines a tracepoint
field whose name is in bpf_tp_terms[] with a BPF-only term (such as
"pid == 1234, uid == 0"), is_possible_tp_filter() returns false for the
entire string, causing "pid" to be evaluated as current's TGID in BPF rather
than the tracepoint's "pid" field.
>
> static int set_filter(struct evsel *evsel, const void *arg)
> @@ -2611,7 +2643,7 @@ static int set_filter(struct evsel *evsel, const void *arg)
> return -1;
> }
>
> - if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT && is_possible_tp_filter(str)) {
> + if (evsel->core.attr.type == PERF_TYPE_TRACEPOINT && is_possible_tp_filter(evsel, str)) {
> if (evsel__append_tp_filter(evsel, str) < 0) {
> fprintf(stderr,
> "not enough memory to hold filter string\n");
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261022203003.1779152-1-irogers@google.com?part=2
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF
2026-10-22 20:29 ` [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers
2026-10-22 20:35 ` sashiko-bot
@ 2026-10-22 22:06 ` Namhyung Kim
1 sibling, 0 replies; 106+ messages in thread
From: Namhyung Kim @ 2026-10-22 22:06 UTC (permalink / raw)
To: Ian Rogers
Cc: acme, adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
On Thu, Oct 22, 2026 at 01:29:53PM -0700, Ian Rogers wrote:
> In perf_bpf_filter__prepare(), the unpinned filter path attaches
> skel->progs.perf_sample_filter to each FD(evsel, x, y) using
> bpf_program__attach_perf_event_opts() and ignores the returned
> struct bpf_link *, leaking the allocated struct bpf_link_perf and its
> BPF_PERF_EVENT link fd.
>
> Furthermore, bpf_program__attach_perf_event_opts() is designed for
> callers that transfer ownership of the perf_event fd to the returned
> bpf_link: when bpf_link__destroy() is called, bpf_link_perf_detach()
> unconditionally disables and closes perf_link->perf_event_fd. Because
> evsel->core.fd owns FD(evsel, x, y) and closes them in evlist__close(),
> destroying the returned bpf_link would double-close the perf_event file
> descriptors.
>
> Fix this by attaching the program directly with
> ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd), matching the pinned
> filter path right above it in perf_bpf_filter__prepare(). In the
> kernel, PERF_EVENT_IOC_SET_BPF takes a reference on the BPF program and
> _free_event() -> perf_event_free_bpf_prog() automatically detaches and
> releases it when the perf_event fd is closed. This also avoids enabling
> the event prematurely, allowing pe_opts (.dont_enable = true) and the
> LIBBPF_CURRENT_VERSION_GEQ(1, 7) guard to be dropped.
>
> Before (built with EXTRA_CFLAGS="-fsanitize=address"):
> # perf record -e cycles --filter 'period > 1000' -- true
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.024 MB perf.data ]
>
> =================================================================
> ==51328==ERROR: LeakSanitizer: detected memory leaks
>
> Direct leak of 1568 byte(s) in 28 object(s) allocated from:
> #0 0x7ffbe76ef24f in calloc (/lib64/libasan.so.8+0xef24f)
> #1 0x625df7 in bpf_program__attach_perf_event_opts (tools/perf/perf+0x625df7)
> #2 0xbeddc0 in perf_bpf_filter__prepare util/bpf-filter.c:507
> #3 0x8f3def in evlist__apply_filters util/evlist.c:1421
> #4 0x44bbcf in record__open tools/perf/builtin-record.c:1469
> #5 0x453f9e in __cmd_record tools/perf/builtin-record.c:2594
> #6 0x45dadb in cmd_record tools/perf/builtin-record.c:4427
> #7 0x5df56b in run_builtin tools/perf/perf.c:383
> #8 0x5dfa0e in handle_internal_command tools/perf/perf.c:436
> #9 0x5dfddc in run_argv tools/perf/perf.c:480
> #10 0x5e0333 in main tools/perf/perf.c:581
>
> SUMMARY: AddressSanitizer: 1568 byte(s) leaked in 28 allocation(s).
>
> After (built with EXTRA_CFLAGS="-fsanitize=address"):
> # perf record -e cycles --filter 'period > 1000' -- true
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.024 MB perf.data ]
>
> Reported-by: Namhyung Kim <namhyung@kernel.org>
> Closes: https://lore.kernel.org/linux-perf-users/20261020053226.55936-1-namhyung@kernel.org/
> Fixes: 56ec9457a4a2 ("perf bpf filter: Implement event sample filtering")
> Signed-off-by: Ian Rogers <irogers@google.com>
Tested-by: Namhyung Kim <namhyung@kernel.org>
Thanks,
Namhyung
> ---
> tools/perf/util/bpf-filter.c | 18 +++++-------------
> 1 file changed, 5 insertions(+), 13 deletions(-)
>
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 72e83ac1d6a4..506414e5c407 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
> @@ -40,6 +40,7 @@
> * The 'dropped' map is to keep how many samples it dropped by the filter and
> * it will be reported as lost samples.
> */
> +#include <errno.h>
> #include <stdlib.h>
> #include <fcntl.h>
> #include <sys/ioctl.h>
> @@ -423,16 +424,8 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> {
> int i, x, y, fd, ret;
> struct sample_filter_bpf *skel = NULL;
> - struct bpf_program *prog;
> - struct bpf_link *link;
> struct perf_bpf_filter_entry *entry;
> bool needs_idx_hash = !target__has_cpu(target);
> -#if LIBBPF_CURRENT_VERSION_GEQ(1, 7)
> - DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts,
> - .dont_enable = true);
> -#else
> - DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts);
> -#endif
>
> entry = calloc(MAX_FILTERS, sizeof(*entry));
> if (entry == NULL)
> @@ -501,14 +494,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> goto err;
> }
>
> - prog = skel->progs.perf_sample_filter;
> + fd = bpf_program__fd(skel->progs.perf_sample_filter);
> for (x = 0; x < xyarray__max_x(evsel->core.fd); x++) {
> for (y = 0; y < xyarray__max_y(evsel->core.fd); y++) {
> - link = bpf_program__attach_perf_event_opts(prog, FD(evsel, x, y),
> - &pe_opts);
> - if (IS_ERR(link)) {
> + ret = ioctl(FD(evsel, x, y), PERF_EVENT_IOC_SET_BPF, fd);
> + if (ret < 0) {
> + ret = -errno;
> pr_err("Failed to attach perf sample-filter program\n");
> - ret = PTR_ERR(link);
> goto err;
> }
> }
> --
> 2.56.0.307.g5fe7bd680c-goog
>
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 05/11] perf bpf-filter: Add lock_held() sample filter for global mutexes
2026-10-22 20:29 ` [PATCH v6 05/11] perf bpf-filter: Add lock_held() sample filter for global mutexes Ian Rogers
2026-10-22 20:46 ` sashiko-bot
@ 2026-10-22 22:55 ` Namhyung Kim
2026-10-23 5:16 ` Ian Rogers
1 sibling, 1 reply; 106+ messages in thread
From: Namhyung Kim @ 2026-10-22 22:55 UTC (permalink / raw)
To: Ian Rogers
Cc: acme, adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
On Thu, Oct 22, 2026 at 01:29:57PM -0700, Ian Rogers wrote:
> Add BPF sample filter predicates lock_held(<lock>),
> lock_held_read(<lock>) / lock_held_shared(<lock>), and
> lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R'
> and ':W' suffixes on <lock>) to allow recording perf event samples only
> when a specified kernel lock is held (or not held with '!').
>
> In this initial step, support global/static kernel mutex symbols (e.g.
> lock_held(cgroup_mutex)) resolved via /proc/kallsyms. On sample
> overflow, perf_sample_filter and perf_tp_sample_filter inspect the
> mutex's owner field directly in kernel memory via get_lock_owner()
> without attaching any BPF fentry/fexit hooks.
>
> Subsequent commits extend lock_held() to other kernel lock types,
> per-process and semi-global locks, slab cache locks, and RCU/SRCU.
>
> Signed-off-by: Ian Rogers <irogers@google.com>
> ---
> tools/perf/Documentation/perf-record.txt | 16 +-
> tools/perf/util/bpf-filter.c | 350 ++++++++++++++++++-
> tools/perf/util/bpf-filter.h | 13 +
> tools/perf/util/bpf-filter.l | 31 +-
> tools/perf/util/bpf-filter.y | 47 ++-
> tools/perf/util/bpf_skel/sample-filter.h | 36 ++
> tools/perf/util/bpf_skel/sample_filter.bpf.c | 54 +++
> tools/perf/util/parse-events.c | 2 +
> 8 files changed, 530 insertions(+), 19 deletions(-)
>
> diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
> index fcf1ea62bf1c..445cc98a6c33 100644
> --- a/tools/perf/Documentation/perf-record.txt
> +++ b/tools/perf/Documentation/perf-record.txt
> @@ -210,7 +210,21 @@ OPTIONS
> ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr,
> code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat,
> p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock,
> - mem_dtlb, mem_blk, mem_hops, uid, gid
> + mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>),
> + lock_held_read(<lock>), lock_held_shared(<lock>),
> + lock_held_write(<lock>), lock_held_exclusive(<lock>)
> +
> + The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters
> + samples based on whether the specified kernel lock is currently held
> + by the kernel thread. To differentiate between shared/read holding
> + and exclusive/write holding, use 'lock_held_read(<lock>)' (alias
> + 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' (alias
> + 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to <lock>.
> + <lock> can be a kernel mutex symbol name (e.g.
> + 'lock_held(cgroup_mutex)'). When filtering on a mutex symbol
> + ('*_mutex'), 'perf_sample_filter' inspects the lock's 'owner' field
> + directly in kernel memory on each sample without attaching any BPF
> + fentry/fexit hooks.
>
> The <operator> can be one of:
> ==, !=, >, >=, <, <=, &
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 0d3c700d3dcd..639db6c03137 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
> @@ -43,6 +43,9 @@
> #include <errno.h>
> #include <stdio.h>
> #include <stdlib.h>
> +#include <stdbool.h>
> +#include <stddef.h>
> +#include <string.h>
> #include <fcntl.h>
> #include <unistd.h>
> #include <sys/ioctl.h>
> @@ -50,14 +53,19 @@
>
> #include <bpf/bpf.h>
> #include <linux/err.h>
> +#include <linux/kernel.h>
> #include <linux/list.h>
> +#include <linux/string.h>
> +#include <linux/zalloc.h>
> #include <api/fs/fs.h>
> #include <internal/xyarray.h>
> #include <perf/threadmap.h>
> +#include <symbol/kallsyms.h>
>
> #include "util/cap.h"
> #include "util/debug.h"
> #include "util/evsel.h"
> +#include "util/symbol_conf.h"
> #include "util/target.h"
> #include "util/thread_map.h"
> #include "util/bpf-utils.h"
> @@ -126,7 +134,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
> {
> const struct perf_sample_info *info;
>
> - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
> + if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID ||
> + expr->term == PBF_TERM_LOCK_HELD) {
> /* Not dependent on the sample_type as computed from a BPF helper. */
> return 0;
> }
> @@ -169,9 +178,263 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
> return -1;
> }
>
> +static int split_lock_name_mode(const char *name, int expr_part,
> + char *base_buf, size_t buf_sz,
> + bool *has_amp, unsigned int *mode_out)
> +{
> + const char *bare;
> + size_t len;
> + unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK;
> + unsigned int suffix_mode = 0;
> +
> + if (!name || !*name)
> + return -EINVAL;
> +
> + *has_amp = (name[0] == '&');
> + bare = *has_amp ? name + 1 : name;
> + len = strlen(bare);
> + if (len == 0)
> + return -EINVAL;
> +
> + if (len > 2 && !strcmp(bare + len - 2, ":R")) {
> + suffix_mode = PBF_LOCK_MODE_READ;
> + len -= 2;
> + } else if (len > 2 && !strcmp(bare + len - 2, ":W")) {
> + suffix_mode = PBF_LOCK_MODE_WRITE;
> + len -= 2;
> + }
> +
> + if (len == 0 || len >= buf_sz)
> + return -EINVAL;
> +
> + if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY &&
> + suffix_mode && expr_mode != suffix_mode)
> + return -EINVAL;
> +
> + memcpy(base_buf, bare, len);
> + base_buf[len] = '\0';
> +
> + if (suffix_mode)
> + *mode_out = suffix_mode;
> + else if (expr_mode)
> + *mode_out = expr_mode;
> + else
> + *mode_out = PBF_LOCK_MODE_ANY;
> +
> + return 0;
> +}
> +
> +struct lock_sym_cb_args {
> + const char *name;
> + u64 addr;
> + bool in_text;
> + bool in_data;
> + bool found;
> +};
> +
> +static int find_lock_sym_cb(void *arg, const char *name, char type, u64 start)
> +{
> + struct lock_sym_cb_args *args = arg;
> +
> + if (!strcmp(name, "_text") || !strcmp(name, "_stext") ||
> + !strcmp(name, "_sinittext")) {
> + args->in_text = true;
> + return 0;
> + }
> + if (!strcmp(name, "_etext") || !strcmp(name, "_einittext")) {
> + args->in_text = false;
> + return 0;
> + }
> + if (!strcmp(name, "_sdata") || !strcmp(name, "__bss_start")) {
> + args->in_text = false;
> + args->in_data = true;
> + return 0;
> + }
> + if (!strcmp(name, "__bss_stop")) {
> + args->in_data = false;
> + return 0;
> + }
> +
> + /*
> + * Ignore kernel text functions, module text functions, and
> + * absolute/undefined symbols. On CONFIG_LTO_CLANG kernels, vmlinux
> + * .data symbols between _sdata and _edata may have kallsyms type
> + * 'T'/'t', so allow 'T'/'t' within vmlinux data/bss (args->in_data).
> + */
> + if (args->in_text || type == 'A' || type == 'a' ||
> + type == 'U' || type == 'u' ||
> + (!args->in_data && kallsyms__is_function(type)))
> + return 0;
> +
> + if (!strcmp(name, args->name)) {
> + args->found = true;
> + args->addr = start;
> + if (start != 0)
> + return 1;
> + }
> + return 0;
> +}
> +
> +static int find_kernel_lock_sym(const char *name, u64 *addr)
> +{
> + struct lock_sym_cb_args args = {
> + .name = name,
> + };
> + const char *kallsyms_filename = "/proc/kallsyms";
> +
> + if (symbol_conf.kallsyms_name != NULL)
> + kallsyms_filename = symbol_conf.kallsyms_name;
> +
> + if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0)
> + return -ENOENT;
> +
> + if (!args.found)
> + return -ENOENT;
> +
> + if (args.addr == 0) {
> + pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n",
> + name);
> + return -EACCES;
> + }
> +
> + *addr = args.addr;
> + return 0;
> +}
> +
> +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
> + bool *has_lock_filter)
> +{
> + if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name)
> + return;
> +
> + *has_lock_filter = true;
> +}
> +
> +static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter)
> +{
> + struct perf_bpf_filter_expr *expr, *group;
> +
> + *has_lock_filter = false;
> +
> + list_for_each_entry(expr, &evsel->bpf_filters, list) {
> + prescan_one_lock_expr(expr, has_lock_filter);
> + if (expr->op == PBF_OP_GROUP_BEGIN) {
> + list_for_each_entry(group, &expr->groups, list)
> + prescan_one_lock_expr(group, has_lock_filter);
> + }
> + }
> +}
> +
> +static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr,
> + enum perf_bpf_filter_op *op_out)
> +{
> + switch (expr->op) {
> + case PBF_OP_EQ:
> + if (expr->val == 1) {
> + *op_out = PBF_OP_EQ;
> + return 0;
> + }
> + if (expr->val == 0) {
> + *op_out = PBF_OP_NEQ;
> + return 0;
> + }
> + break;
> + case PBF_OP_NEQ:
> + if (expr->val == 0) {
> + *op_out = PBF_OP_EQ;
> + return 0;
> + }
> + if (expr->val == 1) {
> + *op_out = PBF_OP_NEQ;
> + return 0;
> + }
> + break;
> + case PBF_OP_GT:
> + if (expr->val == 0) {
> + *op_out = PBF_OP_EQ;
> + return 0;
> + }
> + break;
> + case PBF_OP_GE:
> + if (expr->val == 1) {
> + *op_out = PBF_OP_EQ;
> + return 0;
> + }
> + break;
> + case PBF_OP_LT:
> + if (expr->val == 1) {
> + *op_out = PBF_OP_NEQ;
> + return 0;
> + }
> + break;
> + case PBF_OP_LE:
> + if (expr->val == 0) {
> + *op_out = PBF_OP_NEQ;
> + return 0;
> + }
> + break;
> + case PBF_OP_AND:
> + if (expr->val & 1) {
> + *op_out = PBF_OP_EQ;
> + return 0;
> + }
> + break;
> + case PBF_OP_GROUP_BEGIN:
> + case PBF_OP_GROUP_END:
> + case PBF_OP_DONE:
> + default:
> + break;
> + }
> + pr_err("Error: Invalid comparison for boolean lock_held(%s) predicate\n",
> + expr->lock_name ?: "");
> + return -EINVAL;
> +}
> +
> +static int resolve_lock_expr(struct perf_bpf_filter_expr *expr,
> + struct perf_bpf_filter_entry *entry)
> +{
> + const char *name = expr->lock_name;
> + char base[128];
> + bool has_amp;
> + unsigned int mode;
> + u64 addr = 0;
> + int ret;
> +
> + ret = normalize_lock_expr_op(expr, &entry->op);
> + if (ret < 0)
> + return ret;
> +
> + if (split_lock_name_mode(name, expr->part, base, sizeof(base),
> + &has_amp, &mode) < 0 || has_amp) {
> + pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: "");
> + return -EINVAL;
> + }
> +
> + if (str_ends_with(base, "mutex") &&
> + !str_ends_with(base, "rt_mutex") &&
> + !str_ends_with(base, "rtmutex")) {
Is it just checking the symbol name ends with "mutex" or so? It seems
risky and limited, can we check BTF to verify if it's really a mutex?
Thanks,
Namhyung
> + ret = find_kernel_lock_sym(base, &addr);
> + if (ret == 0) {
> + if (!(mode & PBF_LOCK_MODE_WRITE)) {
> + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
> + name);
> + return -EINVAL;
> + }
> + entry->part = PBF_LOCK_ADDR | mode | PBF_LOCK_DIRECT_MUTEX;
> + entry->value = addr;
> + return 0;
> + }
> + if (ret == -EACCES)
> + return ret;
> + }
> +
> + pr_err("Error: Cannot resolve kernel lock '%s'\n", name);
> + return -EINVAL;
> +}
> +
> static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry)
> {
> - int i = 0;
> + int i = 0, ret;
> struct perf_bpf_filter_expr *expr;
>
> list_for_each_entry(expr, &evsel->bpf_filters, list) {
> @@ -185,6 +448,11 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
> entry[i].part = expr->part;
> entry[i].term = expr->term;
> entry[i].value = expr->val;
> + if (expr->term == PBF_TERM_LOCK_HELD) {
> + ret = resolve_lock_expr(expr, &entry[i]);
> + if (ret < 0)
> + return ret;
> + }
> i++;
>
> if (expr->op == PBF_OP_GROUP_BEGIN) {
> @@ -198,6 +466,11 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
> entry[i].part = group->part;
> entry[i].term = group->term;
> entry[i].value = group->val;
> + if (group->term == PBF_TERM_LOCK_HELD) {
> + ret = resolve_lock_expr(group, &entry[i]);
> + if (ret < 0)
> + return ret;
> + }
> i++;
> }
>
> @@ -440,20 +713,24 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> struct perf_bpf_filter_entry *entry;
> bool needs_idx_hash = !target__has_cpu(target);
> bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
> + bool has_lock_filter = false;
>
> entry = calloc(MAX_FILTERS, sizeof(*entry));
> if (entry == NULL)
> return -1;
>
> - ret = get_filter_entries(evsel, entry);
> - if (ret < 0) {
> - pr_err("Failed to process filter entries\n");
> + prescan_lock_filters(evsel, &has_lock_filter);
> +
> + if ((has_lock_filter || is_tracepoint) && geteuid() != 0 &&
> + !perf_cap__capable(CAP_BPF)) {
> + pr_err("Error: BPF filter requires CAP_BPF capability\n");
> + ret = -EPERM;
> goto err;
> }
>
> - if (is_tracepoint && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
> - pr_err("Error: Tracepoint BPF filter requires CAP_BPF capability\n");
> - ret = -EPERM;
> + ret = get_filter_entries(evsel, entry);
> + if (ret < 0) {
> + pr_err("Failed to process filter entries\n");
> goto err;
> }
>
> @@ -513,6 +790,9 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> goto err;
> }
>
> + if (has_lock_filter)
> + skel->bss->lock_enabled = 1;
> +
> i = 0;
> fd = bpf_map__fd(skel->maps.filters);
>
> @@ -571,6 +851,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
> return ret;
> }
>
> +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr)
> +{
> + struct perf_bpf_filter_expr *group, *tmp;
> +
> + if (!expr)
> + return;
> +
> + list_for_each_entry_safe(group, tmp, &expr->groups, list) {
> + list_del(&group->list);
> + perf_bpf_filter_expr__delete(group);
> + }
> + zfree(&expr->lock_name);
> + free(expr);
> +}
> +
> int perf_bpf_filter__destroy(struct evsel *evsel)
> {
> struct perf_bpf_filter_expr *expr, *tmp;
> @@ -578,7 +873,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel)
>
> list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) {
> list_del(&expr->list);
> - free(expr);
> + perf_bpf_filter_expr__delete(expr);
> }
> sample_filter_bpf__destroy(evsel->bpf_skel);
>
> @@ -630,7 +925,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
> {
> struct perf_bpf_filter_expr *expr;
>
> - expr = malloc(sizeof(*expr));
> + expr = zalloc(sizeof(*expr));
> if (expr != NULL) {
> expr->term = term;
> expr->part = part;
> @@ -641,6 +936,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
> return expr;
> }
>
> +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
> + int mode,
> + enum perf_bpf_filter_op op,
> + unsigned long val)
> +{
> + struct perf_bpf_filter_expr *expr;
> +
> + if (!lock_name || !*lock_name)
> + return NULL;
> +
> + expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val);
> + if (!expr)
> + return NULL;
> +
> + expr->lock_name = strdup(lock_name);
> + if (!expr->lock_name) {
> + free(expr);
> + return NULL;
> + }
> + return expr;
> +}
> +
> static bool check_bpf_filter_capable(void)
> {
> int fd;
> @@ -661,14 +978,11 @@ static bool check_bpf_filter_capable(void)
> return false;
> }
>
> -int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
> +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
> {
> YY_BUFFER_STATE buffer;
> int ret;
>
> - if (!check_bpf_filter_capable())
> - return -EPERM;
> -
> buffer = perf_bpf_filter__scan_string(str);
>
> ret = perf_bpf_filter_parse(expr_head);
> @@ -680,6 +994,14 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
> return ret;
> }
>
> +int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
> +{
> + if (!check_bpf_filter_capable())
> + return -EPERM;
> +
> + return __perf_bpf_filter__parse(expr_head, str);
> +}
> +
> /*
> * perf_bpf_filter__pin() and perf_bpf_filter__unpin() only load
> * perf_sample_filter and disable autoload on other programs in the skeleton
> diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h
> index 818c554b91b2..fda95ff226b3 100644
> --- a/tools/perf/util/bpf-filter.h
> +++ b/tools/perf/util/bpf-filter.h
> @@ -14,6 +14,13 @@ struct perf_bpf_filter_expr {
> int part;
> enum perf_bpf_filter_term term;
> unsigned long val;
> + /*
> + * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions
> + * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R").
> + * Resolved into `entry.part` and `entry.target` during
> + * perf_bpf_filter__prepare().
> + */
> + char *lock_name;
> };
>
> struct evsel;
> @@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
> int part,
> enum perf_bpf_filter_op op,
> unsigned long val);
> +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
> + int mode,
> + enum perf_bpf_filter_op op,
> + unsigned long val);
> +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr);
> +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
> int perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
> int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target);
> int perf_bpf_filter__destroy(struct evsel *evsel);
> diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l
> index 1be9df6550fc..b4022110f1e5 100644
> --- a/tools/perf/util/bpf-filter.l
> +++ b/tools/perf/util/bpf-filter.l
> @@ -1,9 +1,12 @@
> %option prefix="perf_bpf_filter_"
> %option noyywrap
>
> +%x LOCK_ARG
> +
> %{
> #include <stdio.h>
> #include <stdlib.h>
> +#include <string.h>
> #include <linux/perf_event.h>
>
> #include "bpf-filter.h"
> @@ -76,7 +79,7 @@ static int path_or_error(void)
> num_dec [0-9]+
> num_hex 0[Xx][0-9a-fA-F]+
> space [ \t]+
> -path [^ \t\n,]+
> +path [^ \t\n,()!=>&|<]+
> ident [_a-zA-Z][_a-zA-Z0-9]+
>
> %%
> @@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); }
> uid { return sample(PBF_TERM_UID); }
> gid { return sample(PBF_TERM_GID); }
> cgroup { return sample_path(PBF_TERM_CGROUP); }
> -
> +lock_held[ \t]*"(" {
> + BEGIN(LOCK_ARG);
> + perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY;
> + return BFT_LOCK_HELD;
> + }
> +(lock_held_read|lock_held_shared)[ \t]*"(" {
> + BEGIN(LOCK_ARG);
> + perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ;
> + return BFT_LOCK_HELD;
> + }
> +(lock_held_write|lock_held_exclusive)[ \t]*"(" {
> + BEGIN(LOCK_ARG);
> + perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE;
> + return BFT_LOCK_HELD;
> + }
> +
> +<LOCK_ARG>{space} { }
> +<LOCK_ARG>[^ \t\n()]+ {
> + perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text);
> + return BFT_LOCK_NAME;
> + }
> +<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; }
> +<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; }
> +
> +"!" { return '!'; }
> "==" { return operator(PBF_OP_EQ); }
> "!=" { return operator(PBF_OP_NEQ); }
> ">" { return operator(PBF_OP_GT); }
> diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y
> index 5a79a8e7a45b..5ae05d4608c6 100644
> --- a/tools/perf/util/bpf-filter.y
> +++ b/tools/perf/util/bpf-filter.y
> @@ -8,9 +8,11 @@
> #endif
>
> #include <stdio.h>
> +#include <stdlib.h>
> #include <string.h>
> #include <linux/compiler.h>
> #include <linux/list.h>
> +#include <linux/zalloc.h>
> #include "bpf-filter.h"
> #include "cgroup.h"
>
> @@ -31,6 +33,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
> {
> unsigned long num;
> char *path;
> + char *lock_name;
> + struct {
> + char *name;
> + int mode;
> + } lock;
> struct {
> enum perf_bpf_filter_term term;
> int part;
> @@ -40,12 +47,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
> }
>
> %token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH
> +%token BFT_LOCK_HELD BFT_LOCK_NAME
> %type <expr> filter_term filter_expr
> -%destructor { free ($$); } <expr>
> +%destructor { perf_bpf_filter_expr__delete($$); } <expr>
> +%destructor { free($$); } <lock_name>
> +%destructor { free($$.name); } <lock>
> %type <sample> BFT_SAMPLE BFT_SAMPLE_PATH
> %type <op> BFT_OP
> -%type <num> BFT_NUM
> +%type <num> BFT_NUM BFT_LOCK_HELD
> %type <path> BFT_PATH
> +%type <lock_name> BFT_LOCK_NAME
> +%type <lock> lock_term
>
> %%
>
> @@ -105,5 +117,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH
> $$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id);
> cgroup__put(cgrp);
> }
> +|
> +lock_term
> +{
> + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1);
> + zfree(&$1.name);
> + if (!$$)
> + YYERROR;
> +}
> +|
> +'!' lock_term
> +{
> + $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0);
> + zfree(&$2.name);
> + if (!$$)
> + YYERROR;
> +}
> +|
> +lock_term BFT_OP BFT_NUM
> +{
> + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3);
> + zfree(&$1.name);
> + if (!$$)
> + YYERROR;
> +}
> +
> +lock_term:
> +BFT_LOCK_HELD BFT_LOCK_NAME ')'
> +{
> + $$.name = $2;
> + $$.mode = $1;
> +}
>
> %%
> diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
> index 683fec85e71e..ca1cf586ed6e 100644
> --- a/tools/perf/util/bpf_skel/sample-filter.h
> +++ b/tools/perf/util/bpf_skel/sample-filter.h
> @@ -53,8 +53,44 @@ enum perf_bpf_filter_term {
> /* Terms computed from BPF helpers. */
> PBF_TERM_UID,
> PBF_TERM_GID,
> + /*
> + * Kernel lock held predicate (`lock_held(...)`). Evaluates whether the
> + * specified kernel lock is currently held by the running task/CPU.
> + * The lock specification kind is stored in `entry.part`
> + * (`enum perf_bpf_lock_target_kind`), `entry.op` is normalized to
> + * `PBF_OP_EQ` (match when held) or `PBF_OP_NEQ` (match when not held),
> + * and the resolved target address is stored in `entry.value`.
> + */
> + PBF_TERM_LOCK_HELD,
> };
>
> +/*
> + * Lock target categories for PBF_TERM_LOCK_HELD filters.
> + * Stored in `perf_bpf_filter_entry.part`.
> + */
> +enum perf_bpf_lock_target_kind {
> + PBF_LOCK_NONE = 0,
> + /*
> + * Specific global or static lock address resolved from kallsyms.
> + * `entry.value` holds the kernel address.
> + */
> + PBF_LOCK_ADDR,
> +};
> +
> +/*
> + * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD:
> + * Bits 0..3: `enum perf_bpf_lock_target_kind`
> + * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding)
> + * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding)
> + * Bit 6: PBF_LOCK_DIRECT_MUTEX (inspect struct mutex owner)
> + */
> +#define PBF_LOCK_KIND_MASK 0x0fU
> +#define PBF_LOCK_MODE_READ (1U << 4)
> +#define PBF_LOCK_MODE_WRITE (1U << 5)
> +#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE)
> +#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY
> +#define PBF_LOCK_DIRECT_MUTEX (1U << 6)
> +
> /* BPF map entry for filtering */
> struct perf_bpf_filter_entry {
> enum perf_bpf_filter_op op;
> diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
> index a190374ed87a..42556ab0e2d5 100644
> --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
> +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
> @@ -5,6 +5,7 @@
> #include <bpf/bpf_tracing.h>
> #include <bpf/bpf_core_read.h>
>
> +#include "lock_data.h"
> #include "sample-filter.h"
>
> /* BPF map that will be filled by user space */
> @@ -45,8 +46,57 @@ struct lost_count {
>
> volatile const int use_idx_hash;
>
> +/* Control flag set by userspace when lock_held(...) filters are active. */
> +int lock_enabled;
> +
> void *bpf_cast_to_kern_ctx(void *) __ksym;
>
> +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __u32 part)
> +{
> + if (!curr)
> + return false;
> +
> + if ((part & PBF_LOCK_DIRECT_MUTEX) &&
> + (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr)
> + return true;
> + return false;
> +}
> +
> +static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry)
> +{
> + __u64 curr = bpf_get_current_task();
> + __u32 kind = entry->part & PBF_LOCK_KIND_MASK;
> + __u32 mode = entry->part & PBF_LOCK_MODE_MASK;
> +
> + if (!mode)
> + mode = PBF_LOCK_MODE_ANY;
> +
> + switch (kind) {
> + case PBF_LOCK_ADDR:
> + if ((mode & PBF_LOCK_MODE_WRITE) &&
> + is_direct_lock_owner(entry->value, curr, entry->part))
> + return true;
> + return false;
> +
> + default:
> + return false;
> + }
> +}
> +
> +static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry)
> +{
> + /*
> + * Pinned filters (use_idx_hash != 0) share a world-writable filters
> + * map and do not support lock_held(). Reject evaluation when pinned
> + * or when lock_enabled is not set so unprivileged users cannot probe
> + * arbitrary kernel addresses via entry->value.
> + */
> + if (use_idx_hash || !lock_enabled)
> + return entry->op == PBF_OP_EQ ? ~entry->value : entry->value;
> +
> + return is_lock_held(entry) ? entry->value : ~entry->value;
> +}
> +
> /* new kernel perf_sample_data definition */
> struct perf_sample_data___new {
> __u64 sample_flags;
> @@ -175,6 +225,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
> return bpf_get_current_uid_gid() & 0xFFFFFFFF;
> case PBF_TERM_GID:
> return bpf_get_current_uid_gid() >> 32;
> + case PBF_TERM_LOCK_HELD:
> + return check_lock_held(entry);
> case PBF_TERM_NONE:
> case __PBF_UNUSED_TERM4:
> case __PBF_UNUSED_TERM5:
> @@ -209,6 +261,8 @@ static __always_inline __u64 perf_get_common_sample(struct perf_bpf_filter_entry
> return bpf_get_current_uid_gid() & 0xFFFFFFFF;
> case PBF_TERM_GID:
> return bpf_get_current_uid_gid() >> 32;
> + case PBF_TERM_LOCK_HELD:
> + return check_lock_held(entry);
> default:
> return 0;
> }
> diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
> index d6bab89ade5d..da687e9d1a67 100644
> --- a/tools/perf/util/parse-events.c
> +++ b/tools/perf/util/parse-events.c
> @@ -2621,6 +2621,8 @@ static bool is_possible_tp_filter(struct evsel *evsel, const char *str)
> {
> static const char *const bpf_tp_terms[] = {
> "uid", "gid", "tid", "pid", "cpu", "time", "cgroup",
> + "lock_held", "lock_held_read", "lock_held_shared",
> + "lock_held_write", "lock_held_exclusive",
> };
>
> for (size_t i = 0; i < ARRAY_SIZE(bpf_tp_terms); i++) {
> --
> 2.56.0.307.g5fe7bd680c-goog
>
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types
2026-10-22 20:29 ` [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types Ian Rogers
2026-10-22 20:55 ` sashiko-bot
@ 2026-10-22 22:56 ` Namhyung Kim
2026-10-23 5:19 ` Ian Rogers
1 sibling, 1 reply; 106+ messages in thread
From: Namhyung Kim @ 2026-10-22 22:56 UTC (permalink / raw)
To: Ian Rogers
Cc: acme, adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
On Thu, Oct 22, 2026 at 01:29:58PM -0700, Ian Rogers wrote:
> Extend the lock_held() BPF sample filter beyond global mutexes to
> support:
> - Other global/static kernel lock symbols (rwsem, spinlock, rwlock,
> rt-mutex, percpu-rwsem, semaphore) and hexadecimal kernel lock
> addresses.
> - Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W,
> rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W,
> semaphore.
>
> For rt-mutexes and write-locked rwsems, perf_sample_filter inspects the
> lock's owner field directly in kernel memory. For locks and modes without
> an authoritative per-task owner pointer (spinlocks, rwlocks, semaphores,
> reader-locked rwsems, percpu-rwsems, and lock type filters), attach BPF
> fentry/fexit hooks on kernel lock acquire and release functions to track
> held locks in the held_locks BPF LRU hash map.
>
> Note that counting semaphores (down*() / up()) do not enforce same-task
> ownership in the kernel: when a semaphore is used as a cross-task
> completion/signaling mechanism, fexit/down and fentry/up will not pair
> on the same task ID. Semaphore hooks are therefore only enabled when
> 'semaphore' or a '*_sem' symbol (such as console_sem) is explicitly
> specified, and held_locks uses BPF_MAP_TYPE_LRU_HASH so any unpaired
> cross-task entries are automatically evicted.
>
> Signed-off-by: Ian Rogers <irogers@google.com>
> ---
> tools/perf/Documentation/perf-record.txt | 53 +-
> tools/perf/util/bpf-filter.c | 596 ++++++++++++-
> tools/perf/util/bpf_skel/sample-filter.h | 17 +-
> tools/perf/util/bpf_skel/sample_filter.bpf.c | 851 ++++++++++++++++++-
> 4 files changed, 1465 insertions(+), 52 deletions(-)
>
> diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
> index 445cc98a6c33..9fb5d15de135 100644
> --- a/tools/perf/Documentation/perf-record.txt
> +++ b/tools/perf/Documentation/perf-record.txt
> @@ -216,15 +216,50 @@ OPTIONS
>
> The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters
> samples based on whether the specified kernel lock is currently held
> - by the kernel thread. To differentiate between shared/read holding
> - and exclusive/write holding, use 'lock_held_read(<lock>)' (alias
> - 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' (alias
> - 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to <lock>.
> - <lock> can be a kernel mutex symbol name (e.g.
> - 'lock_held(cgroup_mutex)'). When filtering on a mutex symbol
> - ('*_mutex'), 'perf_sample_filter' inspects the lock's 'owner' field
> - directly in kernel memory on each sample without attaching any BPF
> - fentry/fexit hooks.
> + by the kernel thread or CPU. To differentiate between shared/read
> + holding and exclusive/write holding, use 'lock_held_read(<lock>)'
> + (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)'
> + (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to
> + <lock> (e.g. 'lock_held(rwsem:R)' or 'lock_held(rwsem:W)').
> + <lock> can be a kernel lock symbol name (e.g.
> + 'lock_held(cgroup_mutex)'), a hexadecimal lock address, or a lock type
> + ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
> + 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
> + 'semaphore'). Both read/shared and write/exclusive tracking are
> + supported for 'rwsem' and 'rwlock'; 'mutex', 'spinlock', 'rt-mutex',
> + 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support
> + write/exclusive tracking only. Note that 'semaphore' tracking
> + ('down*()' / 'up()') assumes binary mutex-like usage where the
> + acquiring task also releases the semaphore (such as 'console_sem'),
> + rather than cross-task completion signaling.
> +
> + For example, 'lock_held()' can be combined with memory allocation or
> + reclaim events to detect allocations or direct memory reclaim occurring
> + while a kernel lock is held (which can stall lock waiters for extended
> + periods):
> +
> + # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \
> + --filter 'lock_held(mutex) || lock_held(rwsem)'
> + # perf record -a -g -e kmem:mm_page_alloc \
> + --filter 'gfp_flags & 0x400' \
> + --filter 'lock_held(mutex) || lock_held(rwsem)'
> + # perf record -a -g -e major-faults \
> + --filter 'lock_held(mutex) || lock_held(rwsem)'
> +
> + When filtering on a specific mutex symbol ('*_mutex') or rt-mutex
> + symbol ('*_rt_mutex'), 'perf_sample_filter' inspects the lock's
> + 'owner' field directly in kernel memory on each sample without attaching
> + any BPF fentry/fexit hooks. For locks that do not maintain an
> + authoritative per-task owner pointer (such as spinlocks, rwlocks,
> + semaphores, reader-locked rwsems, and lock type filters), it attaches
> + BPF fentry/fexit programs to kernel lock acquire and release functions
> + to track held locks in a BPF map. These BPF hooks run on every kernel
> + lock acquisition and release for the attached lock family, even when no
> + perf sample is being taken. To minimize this overhead, only the hooks
> + for the relevant lock family and mode are attached when known, whereas
> + raw hex addresses or symbols without a recognizable lock-type suffix
> + attach hooks across all lock families (including high-frequency
> + spinlocks).
>
> The <operator> can be one of:
> ==, !=, >, >=, <, <=, &
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 639db6c03137..8797f67698f7 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
> @@ -52,6 +52,7 @@
> #include <sys/stat.h>
>
> #include <bpf/bpf.h>
> +#include <bpf/btf.h>
> #include <linux/err.h>
> #include <linux/kernel.h>
> #include <linux/list.h>
> @@ -65,6 +66,7 @@
> #include "util/cap.h"
> #include "util/debug.h"
> #include "util/evsel.h"
> +#include "util/lock-contention.h"
> #include "util/symbol_conf.h"
> #include "util/target.h"
> #include "util/thread_map.h"
> @@ -74,6 +76,7 @@
> #include <util/bpf-filter-flex.h>
> #include <util/bpf-filter-bison.h>
>
> +#include "bpf_skel/lock_data.h"
> #include "bpf_skel/sample-filter.h"
> #include "bpf_skel/sample_filter.skel.h"
>
> @@ -178,6 +181,153 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
> return -1;
> }
>
> +/*
> + * Program groups for `lock_held(...)` BPF tracing hooks.
> + *
> + * When a `lock_held(...)` filter is specified, only the BPF tracing program
> + * groups required for the requested lock(s) are autoloaded and attached.
> + * If no `lock_held(...)` filter is present, none of these programs are loaded.
> + */
> +enum lock_prog_group {
> + LOCK_PROG_MUTEX = 1U << 0,
> + LOCK_PROG_RWSEM_READ = 1U << 1,
> + LOCK_PROG_RWSEM_WRITE = 1U << 2,
> + LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE,
> + LOCK_PROG_SPIN = 1U << 3,
> + LOCK_PROG_RWLOCK_READ = 1U << 4,
> + LOCK_PROG_RWLOCK_WRITE = 1U << 5,
> + LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE,
> + LOCK_PROG_RTMUTEX = 1U << 6,
> + LOCK_PROG_SEM = 1U << 7,
> + LOCK_PROG_PERCPU_RWSEM = 1U << 8,
> + LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM,
> +};
> +
> +static unsigned int lock_flags_to_prog_group(unsigned int flags)
> +{
> + if (flags & LCB_F_MUTEX)
> + return LOCK_PROG_MUTEX;
> + if (flags & LCB_F_RT)
> + return LOCK_PROG_RTMUTEX;
> + if (flags & LCB_F_PERCPU)
> + return LOCK_PROG_PERCPU_RWSEM;
> + if (flags == LCB_F_SPIN)
> + return LOCK_PROG_SPIN;
> + if (flags & LCB_F_SPIN)
> + return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE;
> + if (flags & LCB_F_READ)
> + return LOCK_PROG_RWSEM_READ;
> + if (flags & LCB_F_WRITE)
> + return LOCK_PROG_RWSEM_WRITE;
> + return LOCK_PROG_SEM;
> +}
> +
> +struct lock_prog_desc {
> + size_t prog_offset;
> + const char *btf_func;
> + unsigned int group;
> +};
> +
> +#define LOCK_FPROG(field, func, grp) \
> + { offsetof(struct sample_filter_bpf, progs.field), func, grp }
> +
> +static const struct lock_prog_desc lock_progs[] = {
> + /* Mutex */
> + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
> + LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
I'm not sure if these are still needed.. it can handle mutexes
specially, right?
Thanks,
Namhyung
> + /* RW semaphore */
> + LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ),
> + LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ),
> + LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ),
> + LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ),
> + LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested",
> + LOCK_PROG_RWSEM_READ),
> + LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ),
> + LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE),
> + LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE),
> + LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE),
> + LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE),
> + LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested",
> + LOCK_PROG_RWSEM_WRITE),
> + LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE),
> + LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM),
> + LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ),
> + LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE),
> + /* Spinlock */
> + LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested",
> + LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN),
> + LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN),
> + LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN),
> + LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN),
> + LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable",
> + LOCK_PROG_SPIN),
> + LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore",
> + LOCK_PROG_SPIN),
> + LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN),
> + /* RWlock */
> + LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore",
> + LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ),
> + LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore",
> + LOCK_PROG_RWLOCK_WRITE),
> + LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE),
> + /* RT-Mutex */
> + LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX),
> + LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX),
> + LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX),
> + LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible",
> + LOCK_PROG_RTMUTEX),
> + LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX),
> + LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX),
> + LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX),
> + /* Semaphore */
> + LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM),
> + LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM),
> + LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM),
> + LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM),
> + LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM),
> + LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM),
> + /* Percpu-RWsem */
> + LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
> + LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
> +};
> +
> +#undef LOCK_FPROG
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 05/11] perf bpf-filter: Add lock_held() sample filter for global mutexes
2026-10-22 22:55 ` Namhyung Kim
@ 2026-10-23 5:16 ` Ian Rogers
0 siblings, 0 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-23 5:16 UTC (permalink / raw)
To: Namhyung Kim
Cc: acme, adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
On Thu, Oct 22, 2026 at 3:55 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> On Thu, Oct 22, 2026 at 01:29:57PM -0700, Ian Rogers wrote:
[snip]
> > +static int resolve_lock_expr(struct perf_bpf_filter_expr *expr,
> > + struct perf_bpf_filter_entry *entry)
> > +{
> > + const char *name = expr->lock_name;
> > + char base[128];
> > + bool has_amp;
> > + unsigned int mode;
> > + u64 addr = 0;
> > + int ret;
> > +
> > + ret = normalize_lock_expr_op(expr, &entry->op);
> > + if (ret < 0)
> > + return ret;
> > +
> > + if (split_lock_name_mode(name, expr->part, base, sizeof(base),
> > + &has_amp, &mode) < 0 || has_amp) {
> > + pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: "");
> > + return -EINVAL;
> > + }
> > +
> > + if (str_ends_with(base, "mutex") &&
> > + !str_ends_with(base, "rt_mutex") &&
> > + !str_ends_with(base, "rtmutex")) {
>
> Is it just checking the symbol name ends with "mutex" or so? It seems
> risky and limited, can we check BTF to verify if it's really a mutex?
Agreed, a suffix check is only a heuristic. The catch is that kallsyms
gives an address for a global or static variable but not its type. And
vmlinux BTF normally only has BTF_KIND_VAR entries for per-CPU
variables: scripts/Makefile.btf passes "--btf_features=...,var,..."
but not "global_var". So on most kernels there's no BTF type for
something like cgroup_mutex to check.
Patch 6 in this series adds classify_lock_sym(), which uses BTF where
it can and falls back to the name suffix otherwise. It checks:
1. for a __percpu_rwsem_rc_<name> BTF var. DEFINE_PERCPU_RWSEM()
always emits one, and as a per-CPU variable it's always in BTF;
2. the symbol's BTF_KIND_VAR type (struct mutex, rt_mutex,
rw_semaphore, semaphore, ...), when the kernel was built with global
variables in BTF;
3. otherwise, the name suffix.
In any case a wrong guess is harmless. The direct owner check only
reads lock->owner with bpf_probe_read_kernel(), and only at an address
already confirmed to be in kernel .data/.bss (function symbols are
rejected via kallsyms and BTF). The worst case is that the filter
doesn't match.
Patch 5 adding a suffix check that patch 6 then replaces isn't
intuitive, so in the next version I'll move classify_lock_sym() into
patch 5 and use it from the start.
Thanks,
Ian
> Thanks,
> Namhyung
^ permalink raw reply [flat|nested] 106+ messages in thread
* Re: [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types
2026-10-22 22:56 ` Namhyung Kim
@ 2026-10-23 5:19 ` Ian Rogers
0 siblings, 0 replies; 106+ messages in thread
From: Ian Rogers @ 2026-10-23 5:19 UTC (permalink / raw)
To: Namhyung Kim
Cc: acme, adrian.hunter, ddhariwal, eranian, james.clark, jolsa,
linux-kernel, linux-perf-users, mingo, peterz, rcu
On Thu, Oct 22, 2026 at 3:56 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> On Thu, Oct 22, 2026 at 01:29:58PM -0700, Ian Rogers wrote:
[snip]
> > +static const struct lock_prog_desc lock_progs[] = {
> > + /* Mutex */
> > + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
> > + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
> > + LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX),
> > + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
> > + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
> > + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
> > + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
> > + LOCK_PROG_MUTEX),
> > + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
> > + LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX),
> > + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
> > + LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX),
> > + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
>
> I'm not sure if these are still needed.. it can handle mutexes
> specially, right?
The direct lock->owner check only works when we already know the
mutex's address, i.e. lock_held(<global mutex symbol>). In that case
prescan_one_lock_expr() adds no LOCK_PROG_MUTEX group, and
configure_lock_progs() turns autoload off for all of these programs,
so none of them are loaded.
They're still needed when the address isn't known up front, or the
target matches many mutexes:
- lock type filters, e.g. lock_held(mutex) or "lock_held(mutex) ||
lock_held(rwsem)" for the direct reclaim use case. There we need to
know whether the current task holds any mutex, which means tracking
acquire/release in held_locks;
- slab cache filters (lock_held(&<cache>)), where the lock is
embedded in a dynamically allocated object and we match on the
kmem_cache of the address recorded at acquire time;
- hex addresses and symbols whose type can't be determined, where we
hook every lock family.
So they're only loaded when the filter actually needs them. I'll add a
comment above lock_progs[] in the next version explaining this.
Thanks,
Ian
> Thanks,
> Namhyung
^ permalink raw reply [flat|nested] 106+ messages in thread
end of thread, other threads:[~2026-10-23 5:19 UTC | newest]
Thread overview: 106+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-17 5:52 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers
2026-10-17 6:09 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers
2026-10-17 5:49 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-17 5:56 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-17 6:08 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-17 17:31 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers
2026-10-17 17:35 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers
2026-10-17 17:17 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-17 17:25 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-17 17:30 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers
2026-10-17 17:17 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-18 5:37 ` [PATCH v3 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-18 5:50 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers
2026-10-18 6:12 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers
2026-10-18 5:41 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-18 5:51 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-18 5:48 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers
2026-10-18 5:40 ` sashiko-bot
2026-10-19 7:00 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Namhyung Kim
2026-10-19 19:08 ` Ian Rogers
2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers
2026-10-19 22:09 ` [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat Ian Rogers
2026-10-19 22:14 ` sashiko-bot
2026-10-20 4:34 ` Namhyung Kim
2026-10-21 21:39 ` Arnaldo Carvalho de Melo
2026-10-19 22:09 ` [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
2026-10-19 22:26 ` sashiko-bot
2026-10-20 5:41 ` Namhyung Kim
2026-10-20 15:30 ` Ian Rogers
2026-10-20 22:29 ` Namhyung Kim
2026-10-19 22:09 ` [PATCH v4 3/7] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-19 22:17 ` sashiko-bot
2026-10-19 22:09 ` [PATCH v4 4/7] perf bpf-filter: Add lock_held() sample filter Ian Rogers
2026-10-19 22:28 ` sashiko-bot
2026-10-19 22:09 ` [PATCH v4 5/7] perf test: Add tests for lock_held() BPF " Ian Rogers
2026-10-19 22:12 ` sashiko-bot
2026-10-19 22:09 ` [PATCH v4 6/7] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-19 22:19 ` sashiko-bot
2026-10-19 22:09 ` [PATCH v4 7/7] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-19 22:19 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-20 20:58 ` [PATCH v5 1/8] perf lock contention: Filter events before creating lock_stat Ian Rogers
2026-10-20 22:38 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 2/8] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers
2026-10-20 22:41 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 3/8] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
2026-10-20 22:57 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-20 22:42 ` Namhyung Kim
2026-10-20 23:02 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter Ian Rogers
2026-10-20 22:49 ` Namhyung Kim
2026-10-20 23:17 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 6/8] perf test: Add tests for lock_held() BPF " Ian Rogers
2026-10-20 23:20 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 7/8] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-20 23:29 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 8/8] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-20 23:37 ` sashiko-bot
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-22 20:29 ` [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers
2026-10-22 20:35 ` sashiko-bot
2026-10-22 22:06 ` Namhyung Kim
2026-10-22 20:29 ` [PATCH v6 02/11] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
2026-10-22 20:55 ` sashiko-bot
2026-10-22 20:29 ` [PATCH v6 03/11] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-22 20:36 ` sashiko-bot
2026-10-22 20:29 ` [PATCH v6 04/11] perf lock contention: Extend get_lock_owner() for rt_mutex and mutex handoff Ian Rogers
2026-10-22 20:41 ` sashiko-bot
2026-10-22 20:29 ` [PATCH v6 05/11] perf bpf-filter: Add lock_held() sample filter for global mutexes Ian Rogers
2026-10-22 20:46 ` sashiko-bot
2026-10-22 22:55 ` Namhyung Kim
2026-10-23 5:16 ` Ian Rogers
2026-10-22 20:29 ` [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types Ian Rogers
2026-10-22 20:55 ` sashiko-bot
2026-10-22 22:56 ` Namhyung Kim
2026-10-23 5:19 ` Ian Rogers
2026-10-22 20:29 ` [PATCH v6 07/11] perf bpf-filter: Support per-process and semi-global locks in lock_held() Ian Rogers
2026-10-22 20:49 ` sashiko-bot
2026-10-22 20:30 ` [PATCH v6 08/11] perf bpf-filter: Support slab cache locks (&<name>) " Ian Rogers
2026-10-22 20:37 ` sashiko-bot
2026-10-22 20:30 ` [PATCH v6 09/11] perf test: Add tests for lock_held() BPF sample filter Ian Rogers
2026-10-22 20:37 ` sashiko-bot
2026-10-22 20:30 ` [PATCH v6 10/11] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-22 20:44 ` sashiko-bot
2026-10-22 20:30 ` [PATCH v6 11/11] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-22 20:43 ` sashiko-bot
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.