* [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support
@ 2026-10-17 5:45 Ian Rogers
2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
` (4 more replies)
0 siblings, 5 replies; 11+ messages in thread
From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Namhyung Kim
Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, rcu
This series adds a new BPF sample filter predicate, `lock_held(<lock>)`
(along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`,
`lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W`
mode suffixes), to `perf record --filter`, allowing perf event samples to
be recorded only when a specified kernel lock is held (or not held with
`!`) by the running kernel thread or CPU. It also extends both
`lock_held()` and `perf lock contention --use-bpf` to support RCU and
SRCU.
For example:
# perf record -a -g -e cycles --filter 'lock_held(&cgroup_mutex)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' -- ...
# perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ...
# perf lock contention -a -b -Y rcu,srcu -- sleep 1
Supported `<lock>` specifications match `perf lock contention`:
- Global/static kernel lock symbols (with or without leading `&`, e.g.
`lock_held(&cgroup_mutex)` or `lock_held(cgroup_mutex)`)
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`,
`lock_held_read(&dentry)`)
- Well-known per-process, semi-global, and RCU locks: `mmap_lock`,
`siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`
- Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`,
`rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`,
`percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`,
`rcu:W`, `semaphore`
Limitations and weaknesses in determining whether a lock is held:
- Inlined lock/unlock operations:
- When the kernel is configured to inline spinlock or rwlock operations
(`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`,
`CONFIG_INLINE_WRITE_LOCK`, etc., which are enabled on many
`!CONFIG_PREEMPT_RT` / `!CONFIG_PROVE_LOCKING` builds unless
`CONFIG_UNINLINE_SPIN_UNLOCK` or lockdep/preemption forces them out of
line), `_raw_spin_lock*` / `_raw_spin_unlock*` and
`_raw_read_lock*` / `_raw_write_lock*` are inlined into callers and
have no BTF function entry points for `fentry`/`fexit` to attach to.
- Fast-path `percpu_down_read()` / `percpu_up_read()` and
`srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined
per-CPU counter increments without function calls, so only
`percpu_down_write()` / `percpu_up_write()` and out-of-line
`__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked.
- On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to
`preempt_disable()` and does not maintain
`current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)`
read-side inspection requires `CONFIG_PREEMPT_RCU`.
- Lack of an inspectable owner field on non-exclusive / spinning locks:
- While `struct mutex`, `struct rt_mutex`, and write-locked
`struct rw_semaphore` store the owning `task_struct *` in `lock->owner`
(allowing `perf_sample_filter` to inspect kernel memory directly on
sample overflow when filtering by lock address), spinlocks, rwlocks,
semaphores, SRCU, and reader-locked `rw_semaphore`s (where
`__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`)
do not maintain an authoritative per-task owner pointer.
- Tracking these locks (as well as slab-cache and lock-type filters)
relies on `fexit` (acquire) and `fentry` (release) BPF hooks
maintaining the `held_locks` BPF hash map:
- Any such lock already held before `perf record` attaches its BPF
programs will not be present in `held_locks` until it is released
and re-acquired.
- The attached `fentry`/`fexit` hooks execute on every kernel lock
acquire and release for the hooked lock family across the system,
adding tracing overhead outside of perf sample handling.
Patch overview:
1. "perf lock contention: Share common BPF and lock type definitions in
lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and
`LCB_F_*` flags, CO-RE structs, and common BPF lock helpers
(`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`,
`get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`,
`do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`.
2. "perf bpf-filter: Add lock_held() sample filter": Implements
`lock_held()`, `lock_held_read()`/`lock_held_shared()`,
`lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode
suffixes using direct lock owner inspection where possible (`mutex`,
`rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF map
updated by selectively attached `fentry`/`fexit` hooks.
3. "perf test: Add tests for lock_held() BPF sample filter": Adds unit
tests in `tests/parse-events.c` and shell tests in
`tests/shell/record_bpf_filter.sh`.
4. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU": Supports `rcu` (`rcu:R` via direct
`current->rcu_read_lock_nesting` inspection with zero tracing hooks;
`rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks),
`rcu-trace` (via `current->trc_reader_nesting`), and `srcu`
(`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W`
via `synchronize_srcu*()` / `srcu_barrier()`).
5. "perf lock contention: Extend BPF lock contention to RCU and SRCU":
Extends `perf lock contention --use-bpf` to track RCU and SRCU
grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`).
Assisted-by: Antigravity:gemini-3.1-pro
Ian Rogers (5):
perf lock contention: Share common BPF and lock type definitions in
lock_data.h
perf bpf-filter: Add lock_held() sample filter
perf test: Add tests for lock_held() BPF sample filter
perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU
perf lock contention: Extend BPF lock contention to RCU and SRCU
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/Documentation/perf-record.txt | 51 +-
tools/perf/builtin-lock.c | 27 -
tools/perf/tests/parse-events.c | 121 ++
tools/perf/tests/shell/lock_contention.sh | 14 +
tools/perf/tests/shell/record_bpf_filter.sh | 119 ++
tools/perf/util/bpf-filter.c | 1070 ++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 46 +-
tools/perf/util/bpf_lock_contention.c | 167 ++-
.../perf/util/bpf_skel/lock_contention.bpf.c | 410 +++----
tools/perf/util/bpf_skel/lock_data.h | 360 +++++-
tools/perf/util/bpf_skel/sample-filter.h | 71 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 1051 ++++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 +
tools/perf/util/lock-contention.c | 22 +
tools/perf/util/lock-contention.h | 15 +
tools/perf/util/parse-events.c | 3 +-
19 files changed, 3273 insertions(+), 332 deletions(-)
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
@ 2026-10-17 5:45 ` Ian Rogers
2026-10-17 5:52 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers
` (3 subsequent siblings)
4 siblings, 1 reply; 11+ messages in thread
From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Namhyung Kim
Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, rcu
Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers
(get_lock_owner, get_current_mmap_lock, get_current_siglock,
get_lock_addr, check_lock_type, do_collect_zone_lock,
do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into
tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF
sample filtering.
Also add rw_semaphore.count and struct rt_mutex_base to the minimal
vmlinux.h and extend get_lock_owner() to support rt_mutex and check
writer-ownership and handoff flags.
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/builtin-lock.c | 27 --
.../perf/util/bpf_skel/lock_contention.bpf.c | 265 +--------------
tools/perf/util/bpf_skel/lock_data.h | 316 ++++++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 +
tools/perf/util/lock-contention.c | 18 +
tools/perf/util/lock-contention.h | 13 +
6 files changed, 370 insertions(+), 274 deletions(-)
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index d5c0d55cb82d..feaf1cfd08cf 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -1477,33 +1477,6 @@ static void sort_result(void)
}
}
-static const struct {
- unsigned int flags;
- /*
- * Name of the lock flags (access), with delimeter ':'.
- * For example, rwsem:R of rwsem:W.
- */
- const char *flags_name;
- /* Name of the lock (type), for example, rwlock or rwsem. */
- const char *lock_name;
-} lock_type_table[] = {
- { 0, "semaphore", "semaphore" },
- { LCB_F_SPIN, "spinlock", "spinlock" },
- { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
- { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
- { LCB_F_READ, "rwsem:R", "rwsem" },
- { LCB_F_WRITE, "rwsem:W", "rwsem" },
- { LCB_F_RT, "rt-mutex", "rt-mutex" },
- { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
- { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
- { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
- { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
- { LCB_F_MUTEX, "mutex", "mutex" },
- { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
- /* alias for optimistic spinning only */
- { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
-};
-
static const char *get_type_flags_name(unsigned int flags)
{
flags &= LCB_F_TYPE_MASK;
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index 0d9c6f55050e..0f2118006005 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -8,23 +8,9 @@
#include "lock_data.h"
-/* for collect_lock_syms(). 4096 was rejected by the verifier */
-#define MAX_CPUS 1024
-
-/* for collect_zone_lock(). It should be more than the actual zones. */
-#define MAX_ZONES 10
-
/* for do_lock_delay(). Arbitrarily set to 1 million. */
#define MAX_LOOP (1U << 20)
-/* lock contention flags from include/trace/events/lock.h */
-#define LCB_F_SPIN (1U << 0)
-#define LCB_F_READ (1U << 1)
-#define LCB_F_WRITE (1U << 2)
-#define LCB_F_RT (1U << 3)
-#define LCB_F_PERCPU (1U << 4)
-#define LCB_F_MUTEX (1U << 5)
-
/* callstack storage */
struct {
__uint(type, BPF_MAP_TYPE_STACK_TRACE);
@@ -96,13 +82,6 @@ struct {
__uint(max_entries, MAX_ENTRIES);
} task_data SEC(".maps");
-struct {
- __uint(type, BPF_MAP_TYPE_HASH);
- __uint(key_size, sizeof(__u64));
- __uint(value_size, sizeof(__u32));
- __uint(max_entries, MAX_ENTRIES);
-} lock_syms SEC(".maps");
-
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(key_size, sizeof(__u32));
@@ -138,6 +117,13 @@ struct {
__uint(max_entries, 1);
} cgroup_filter SEC(".maps");
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u32));
+ __uint(max_entries, MAX_ENTRIES);
+} lock_syms SEC(".maps");
+
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(key_size, sizeof(long));
@@ -159,22 +145,6 @@ struct {
__uint(max_entries, 1);
} lock_delays SEC(".maps");
-struct rw_semaphore___old {
- struct task_struct *owner;
-} __attribute__((preserve_access_index));
-
-struct rw_semaphore___new {
- atomic_long_t owner;
-} __attribute__((preserve_access_index));
-
-struct mm_struct___old {
- struct rw_semaphore mmap_sem;
-} __attribute__((preserve_access_index));
-
-struct mm_struct___new {
- struct rw_semaphore mmap_lock;
-} __attribute__((preserve_access_index));
-
struct cas_ctx {
struct contention_data *data;
u64 duration;
@@ -182,8 +152,6 @@ struct cas_ctx {
int min_done;
};
-extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak;
-
/* control flags */
const volatile int has_cpu;
const volatile int has_task;
@@ -202,6 +170,11 @@ const volatile int lock_delay;
/* determine the key of lock stat */
const volatile int aggr_mode;
+const volatile __u64 contig_page_data_addr;
+const volatile __u64 node_data_addr;
+const volatile int nr_nodes;
+const volatile int sizeof_zone;
+
int enabled;
int perf_subsys_id = -1;
@@ -222,8 +195,6 @@ int data_map_full;
struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak;
void bpf_task_release(struct task_struct *p) __ksym __weak;
-static inline __u32 check_lock_type(__u64 lock, __u32 flags);
-
static inline __u64 get_current_cgroup_id(void)
{
struct task_struct *task;
@@ -347,85 +318,6 @@ static inline int update_task_data(struct task_struct *task)
return 0;
}
-#ifndef __has_builtin
-# define __has_builtin(x) 0
-#endif
-
-static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
-{
- struct task_struct *task;
- __u64 owner = 0;
-
- if (flags & LCB_F_MUTEX) {
- struct mutex *mutex = (void *)lock;
- owner = BPF_CORE_READ(mutex, owner.counter);
- } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
- /*
- * Support for the BPF_TYPE_MATCHES argument to the
- * __builtin_preserve_type_info builtin was added at some point during
- * development of clang 15 and it's what is needed for
- * bpf_core_type_matches.
- */
-#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
- if (bpf_core_type_matches(struct rw_semaphore___old)) {
- struct rw_semaphore___old *rwsem = (void *)lock;
- owner = (unsigned long)BPF_CORE_READ(rwsem, owner);
- } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
- struct rw_semaphore___new *rwsem = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
- }
-#else
- /* assume new struct */
- struct rw_semaphore *rwsem = (void *)lock;
- owner = BPF_CORE_READ(rwsem, owner.counter);
-#endif
- }
-
- if (!owner)
- return NULL;
-
- task = (void *)(owner & ~7UL);
- return task;
-}
-
-static inline __u32 check_lock_type(__u64 lock, __u32 flags)
-{
- struct task_struct *curr;
- struct mm_struct___old *mm_old;
- struct mm_struct___new *mm_new;
- struct sighand_struct *sighand;
-
- switch (flags) {
- case LCB_F_READ: /* rwsem */
- case LCB_F_WRITE:
- curr = bpf_get_current_task_btf();
- if (curr->mm == NULL)
- break;
- mm_new = (void *)curr->mm;
- if (bpf_core_field_exists(mm_new->mmap_lock)) {
- if (&mm_new->mmap_lock == (void *)lock)
- return LCD_F_MMAP_LOCK;
- break;
- }
- mm_old = (void *)curr->mm;
- if (bpf_core_field_exists(mm_old->mmap_sem)) {
- if (&mm_old->mmap_sem == (void *)lock)
- return LCD_F_MMAP_LOCK;
- }
- break;
- case LCB_F_SPIN: /* spinlock */
- curr = bpf_get_current_task_btf();
- sighand = curr->sighand;
-
- if (sighand && &sighand->siglock == (void *)lock)
- return LCD_F_SIGHAND_LOCK;
- break;
- default:
- break;
- }
- return 0;
-}
-
static inline long delay_callback(__u64 idx, void *arg)
{
__u64 target = *(__u64 *)arg;
@@ -908,145 +800,24 @@ int contention_end(u64 *ctx)
return 0;
}
-extern struct rq runqueues __ksym;
-
-const volatile __u64 contig_page_data_addr;
-const volatile __u64 node_data_addr;
-const volatile int nr_nodes;
-const volatile int sizeof_zone;
-
-struct rq___old {
- raw_spinlock_t lock;
-} __attribute__((preserve_access_index));
-
-struct rq___new {
- raw_spinlock_t __lock;
-} __attribute__((preserve_access_index));
-
-static void collect_zone_lock(void)
-{
- __u64 nr_zones, zone_off;
- __u64 lock_addr, lock_off;
- __u32 lock_flag = LOCK_CLASS_ZONE_LOCK;
-
- zone_off = offsetof(struct pglist_data, node_zones);
- lock_off = offsetof(struct zone, lock);
-
- if (contig_page_data_addr) {
- struct pglist_data *contig_page_data;
-
- contig_page_data = (void *)(long)contig_page_data_addr;
- nr_zones = BPF_CORE_READ(contig_page_data, nr_zones);
-
- for (int i = 0; i < MAX_ZONES; i++) {
- __u64 zone_addr;
-
- if (i >= nr_zones)
- break;
-
- zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off;
- lock_addr = zone_addr + lock_off;
-
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
- } else if (nr_nodes > 0) {
- struct pglist_data **node_data = (void *)(long)node_data_addr;
-
- for (int i = 0; i < nr_nodes; i++) {
- struct pglist_data *pgdat = NULL;
- int err;
-
- err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
- if (err < 0 || pgdat == NULL)
- break;
-
- nr_zones = BPF_CORE_READ(pgdat, nr_zones);
- for (int k = 0; k < MAX_ZONES; k++) {
- __u64 zone_addr;
-
- if (k >= nr_zones)
- break;
-
- zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off;
- lock_addr = zone_addr + lock_off;
-
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
- }
- }
-}
-
SEC("raw_tp/bpf_test_finish")
-int BPF_PROG(collect_lock_syms)
+int BPF_PROG(end_timestamp)
{
- __u64 lock_addr, lock_off;
- __u32 lock_flag;
-
- if (bpf_core_field_exists(struct rq___new, __lock))
- lock_off = offsetof(struct rq___new, __lock);
- else
- lock_off = offsetof(struct rq___old, lock);
-
- for (int i = 0; i < MAX_CPUS; i++) {
- struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
-
- if (rq == NULL)
- break;
-
- lock_addr = (__u64)(void *)rq + lock_off;
- lock_flag = LOCK_CLASS_RQLOCK;
- bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY);
- }
-
- collect_zone_lock();
-
+ end_ts = bpf_ktime_get_ns();
return 0;
}
SEC("raw_tp/bpf_test_finish")
-int BPF_PROG(end_timestamp)
+int BPF_PROG(collect_lock_syms)
{
- end_ts = bpf_ktime_get_ns();
- return 0;
+ return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
}
-/*
- * bpf_iter__kmem_cache added recently so old kernels don't have it in the
- * vmlinux.h. But we cannot add it here since it will cause a compiler error
- * due to redefinition of the struct on later kernels.
- *
- * So it uses a CO-RE trick to access the member only if it has the type.
- * This will support both old and new kernels without compiler errors.
- */
-struct bpf_iter__kmem_cache___new {
- struct kmem_cache *s;
-} __attribute__((preserve_access_index));
-
SEC("iter/kmem_cache")
int slab_cache_iter(void *ctx)
{
- struct kmem_cache *s = NULL;
- struct slab_cache_data d;
- const char *nameptr;
-
- if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) {
- struct bpf_iter__kmem_cache___new *iter = ctx;
-
- s = iter->s;
- }
-
- if (s == NULL)
- return 0;
-
- nameptr = s->name;
- bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr);
-
- d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT;
- if (d.id >= LCB_F_SLAB_ID_END)
- return 0;
-
- bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST);
- return 0;
+ return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
}
char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index 652e114e6b87..fed445c9d8b4 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -45,6 +45,14 @@ struct contention_task_data {
#define LCB_F_SLAB_ID_END (1U << 26)
#define LCB_F_SLAB_ID_MASK 0x03FF0000U
+/* lock contention flags from include/trace/events/lock.h */
+#define LCB_F_SPIN (1U << 0)
+#define LCB_F_READ (1U << 1)
+#define LCB_F_WRITE (1U << 2)
+#define LCB_F_RT (1U << 3)
+#define LCB_F_PERCPU (1U << 4)
+#define LCB_F_MUTEX (1U << 5)
+
#define LCB_F_TYPE_MAX (1U << 7)
#define LCB_F_TYPE_MASK 0x0000007FU
@@ -76,4 +84,312 @@ struct slab_cache_data {
char name[SLAB_NAME_MAX];
};
+#ifdef __BPF__
+
+/* for collect_lock_syms(). 4096 was rejected by the verifier */
+#define MAX_CPUS 1024
+
+/* for collect_zone_lock(). It should be more than the actual zones. */
+#define MAX_ZONES 10
+
+extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak;
+extern struct rq runqueues __ksym __weak;
+
+#ifndef __has_builtin
+# define __has_builtin(x) 0
+#endif
+
+struct rw_semaphore___old {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
+struct rw_semaphore___new {
+ atomic_long_t count;
+ atomic_long_t owner;
+} __attribute__((preserve_access_index));
+
+struct mm_struct___old {
+ struct rw_semaphore mmap_sem;
+} __attribute__((preserve_access_index));
+
+struct mm_struct___new {
+ struct rw_semaphore mmap_lock;
+} __attribute__((preserve_access_index));
+
+struct rq___old {
+ raw_spinlock_t lock;
+} __attribute__((preserve_access_index));
+
+struct rq___new {
+ raw_spinlock_t __lock;
+} __attribute__((preserve_access_index));
+
+/*
+ * bpf_iter__kmem_cache added recently so old kernels don't have it in the
+ * vmlinux.h. But we cannot add it here since it will cause a compiler error
+ * due to redefinition of the struct on later kernels.
+ *
+ * So it uses a CO-RE trick to access the member only if it has the type.
+ * This will support both old and new kernels without compiler errors.
+ */
+struct bpf_iter__kmem_cache___new {
+ struct kmem_cache *s;
+} __attribute__((preserve_access_index));
+
+static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags)
+{
+ struct task_struct *task;
+ __u64 owner = 0;
+
+ if (flags & LCB_F_MUTEX) {
+ struct mutex *mutex = (void *)lock;
+
+ owner = BPF_CORE_READ(mutex, owner.counter);
+ if (owner & 0x4UL)
+ return NULL;
+ } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) {
+ struct rw_semaphore___new *rwsem = (void *)lock;
+
+ if (flags == LCB_F_WRITE && bpf_core_field_exists(rwsem->count.counter)) {
+ __u64 count = BPF_CORE_READ(rwsem, count.counter);
+
+ /* Must have RWSEM_WRITER_LOCKED (bit 0) set */
+ if (!(count & 1UL))
+ return NULL;
+ }
+ /*
+ * Support for the BPF_TYPE_MATCHES argument to the
+ * __builtin_preserve_type_info builtin was added at some point during
+ * development of clang 15 and it's what is needed for
+ * bpf_core_type_matches.
+ */
+#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15
+ if (bpf_core_type_matches(struct rw_semaphore___old)) {
+ struct rw_semaphore___old *old = (void *)lock;
+
+ owner = (unsigned long)BPF_CORE_READ(old, owner);
+ } else if (bpf_core_type_matches(struct rw_semaphore___new)) {
+ owner = BPF_CORE_READ(rwsem, owner.counter);
+ }
+#else
+ /* assume new struct */
+ owner = BPF_CORE_READ(rwsem, owner.counter);
+#endif
+ /* RWSEM_READER_OWNED is bit 0 of owner */
+ if (flags == LCB_F_WRITE && (owner & 1UL))
+ return NULL;
+ } else if (flags == LCB_F_RT) {
+ struct rt_mutex_base *rtm = (void *)lock;
+
+ if (bpf_core_type_exists(struct rt_mutex_base)) {
+ owner = (__u64)BPF_CORE_READ(rtm, owner);
+ return (void *)(owner & ~1UL);
+ }
+ }
+
+ if (!owner)
+ return NULL;
+
+ task = (void *)(owner & ~7UL);
+ return task;
+}
+
+static __always_inline __u64 get_current_mmap_lock(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+ struct mm_struct___new *mm_new = NULL;
+ struct mm_struct___old *mm_old = NULL;
+ __u64 mm;
+
+ if (!curr)
+ return 0;
+
+ mm = (__u64)BPF_CORE_READ(task, mm);
+ if (!mm)
+ return 0;
+
+ if (bpf_core_field_exists(mm_new->mmap_lock))
+ return mm + bpf_core_field_offset(mm_new->mmap_lock);
+
+ if (bpf_core_field_exists(mm_old->mmap_sem))
+ return mm + bpf_core_field_offset(mm_old->mmap_sem);
+
+ return 0;
+}
+
+static __always_inline __u64 get_current_siglock(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+ struct sighand_struct *sighand = NULL;
+ __u64 sh;
+
+ if (!curr)
+ return 0;
+
+ sh = (__u64)BPF_CORE_READ(task, sighand);
+ if (!sh)
+ return 0;
+
+ return sh + bpf_core_field_offset(sighand->siglock);
+}
+
+/*
+ * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar.
+ *
+ * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the
+ * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed
+ * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such
+ * as `bpf_get_kmem_cache(u64 addr)`.
+ */
+static __always_inline __u64 get_lock_addr(void *ctx)
+{
+ __u64 lock_addr = 0;
+
+ bpf_get_func_arg(ctx, 0, &lock_addr);
+ return lock_addr;
+}
+
+static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags)
+{
+ __u64 curr;
+
+ switch (flags) {
+ case LCB_F_READ: /* rwsem */
+ case LCB_F_WRITE:
+ curr = bpf_get_current_task();
+ if (get_current_mmap_lock(curr) == lock)
+ return LCD_F_MMAP_LOCK;
+ break;
+ case LCB_F_SPIN: /* spinlock */
+ curr = bpf_get_current_task();
+ if (get_current_siglock(curr) == lock)
+ return LCD_F_SIGHAND_LOCK;
+ break;
+ default:
+ break;
+ }
+ return 0;
+}
+
+static __always_inline void do_collect_zone_lock(void *lock_syms_map,
+ __u64 contig_page_data_addr,
+ __u64 node_data_addr,
+ int nr_nodes,
+ int sizeof_zone)
+{
+ __u64 nr_zones, zone_off;
+ __u64 lock_addr, lock_off;
+ __u32 lock_flag = LOCK_CLASS_ZONE_LOCK;
+
+ zone_off = offsetof(struct pglist_data, node_zones);
+ lock_off = offsetof(struct zone, lock);
+
+ if (contig_page_data_addr) {
+ struct pglist_data *contig_page_data;
+
+ contig_page_data = (void *)(long)contig_page_data_addr;
+ nr_zones = BPF_CORE_READ(contig_page_data, nr_zones);
+
+ for (int i = 0; i < MAX_ZONES; i++) {
+ __u64 zone_addr;
+
+ if (i >= nr_zones)
+ break;
+
+ zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off;
+ lock_addr = zone_addr + lock_off;
+
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ } else if (nr_nodes > 0) {
+ struct pglist_data **node_data = (void *)(long)node_data_addr;
+
+ for (int i = 0; i < nr_nodes; i++) {
+ struct pglist_data *pgdat = NULL;
+ int err;
+
+ err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]);
+ if (err < 0)
+ break;
+ if (pgdat == NULL)
+ continue;
+
+ nr_zones = BPF_CORE_READ(pgdat, nr_zones);
+ for (int k = 0; k < MAX_ZONES; k++) {
+ __u64 zone_addr;
+
+ if (k >= nr_zones)
+ break;
+
+ zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off;
+ lock_addr = zone_addr + lock_off;
+
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ }
+ }
+}
+
+static __always_inline int do_collect_lock_syms(void *lock_syms_map,
+ __u64 contig_page_data_addr,
+ __u64 node_data_addr,
+ int nr_nodes,
+ int sizeof_zone)
+{
+ __u64 lock_addr, lock_off;
+ __u32 lock_flag;
+
+ if (bpf_core_field_exists(struct rq___new, __lock))
+ lock_off = offsetof(struct rq___new, __lock);
+ else
+ lock_off = offsetof(struct rq___old, lock);
+
+ if (&runqueues) {
+ for (int i = 0; i < MAX_CPUS; i++) {
+ struct rq *rq = bpf_per_cpu_ptr(&runqueues, i);
+
+ if (rq == NULL)
+ break;
+
+ lock_addr = (__u64)(void *)rq + lock_off;
+ lock_flag = LOCK_CLASS_RQLOCK;
+ bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY);
+ }
+ }
+
+ do_collect_zone_lock(lock_syms_map, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
+
+ return 0;
+}
+
+static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map,
+ __u32 *slab_cache_id)
+{
+ struct kmem_cache *s = NULL;
+ struct slab_cache_data d = {};
+ const char *nameptr;
+
+ if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) {
+ struct bpf_iter__kmem_cache___new *iter = ctx;
+
+ s = iter->s;
+ }
+
+ if (s == NULL)
+ return 0;
+
+ nameptr = s->name;
+ bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr);
+
+ d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT;
+ if (d.id >= LCB_F_SLAB_ID_END)
+ return 0;
+
+ bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST);
+ return 0;
+}
+
+#endif /* __BPF__ */
+
#endif /* UTIL_BPF_SKEL_LOCK_DATA_H */
diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
index 94e42b022ab1..b75ddc6846bb 100644
--- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
+++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
@@ -76,6 +76,7 @@ struct signal_struct {
} __attribute__((preserve_access_index));
struct rw_semaphore {
+ atomic_long_t count;
atomic_long_t owner;
} __attribute__((preserve_access_index));
@@ -83,6 +84,10 @@ struct mutex {
atomic_long_t owner;
} __attribute__((preserve_access_index));
+struct rt_mutex_base {
+ struct task_struct *owner;
+} __attribute__((preserve_access_index));
+
struct kernfs_node {
u64 id;
} __attribute__((preserve_access_index));
diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c
index 119a7206f3cd..a1bb78d6e29c 100644
--- a/tools/perf/util/lock-contention.c
+++ b/tools/perf/util/lock-contention.c
@@ -22,6 +22,24 @@ struct callstack_filter {
static LIST_HEAD(callstack_filters);
struct hlist_head *lockhash_table;
+const struct perf_lock_type lock_type_table[] = {
+ { 0, "semaphore", "semaphore" },
+ { LCB_F_SPIN, "spinlock", "spinlock" },
+ { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" },
+ { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" },
+ { LCB_F_READ, "rwsem:R", "rwsem" },
+ { LCB_F_WRITE, "rwsem:W", "rwsem" },
+ { LCB_F_RT, "rt-mutex", "rt-mutex" },
+ { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" },
+ { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" },
+ { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" },
+ { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" },
+ { LCB_F_MUTEX, "mutex", "mutex" },
+ { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
+ /* alias for optimistic spinning only */
+ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
+};
+
int parse_call_stack(const struct option *opt __maybe_unused, const char *str,
int unset __maybe_unused)
{
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index 59c94190b092..68e9bd77e8a9 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -129,6 +129,19 @@ struct thread_stat {
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+struct perf_lock_type {
+ unsigned int flags;
+ /*
+ * Name of the lock flags (access), with delimiter ':'.
+ * For example, rwsem:R or rwsem:W.
+ */
+ const char *flags_name;
+ /* Name of the lock (type), for example, rwlock or rwsem. */
+ const char *lock_name;
+};
+
+extern const struct perf_lock_type lock_type_table[14];
+
struct evlist;
struct machine;
struct target;
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter
2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
@ 2026-10-17 5:45 ` Ian Rogers
2026-10-17 6:09 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers
` (2 subsequent siblings)
4 siblings, 1 reply; 11+ messages in thread
From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Namhyung Kim
Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, rcu
Add BPF sample filter predicates lock_held(<lock>),
lock_held_read(<lock>) / lock_held_shared(<lock>), and
lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R'
and ':W' suffixes on <lock>) to allow recording perf event samples only
when a specified kernel lock is held (or not held with '!').
Supported <lock> specifications match perf lock contention:
- Global/static kernel lock symbols (with or without '&', e.g.
lock_held(&cgroup_mutex) or lock_held(cgroup_mutex))
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with '&' (e.g. lock_held(&dentry),
lock_held_read(&dentry))
- Per-process and semi-global locks: mmap_lock, siglock, rq_lock,
zone_lock
- Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W,
rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W,
semaphore
Supported lock hold modes by lock family:
- Both read/shared and write/exclusive: rwsem (including mmap_lock)
and rwlock.
- Write/exclusive only: mutex, spinlock (including siglock, rq_lock,
zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write()
/ percpu_up_write(); fast-path percpu_down_read() / percpu_up_read()
are inlined per-CPU counter operations without function entry points).
When evaluating the filter on sample overflow, perf_sample_filter first
inspects the lock's owner field directly in kernel memory where
possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls
back to a held_locks BPF hash map (tracking separate read_count and
write_count) populated by fentry/fexit hooks on kernel lock/unlock
functions. To minimize tracing overhead, only the BPF hooks for the
requested lock family and mode are autoloaded and attached.
Example usage:
# perf record -a -g -e cycles --filter 'lock_held(&mmap_lock)' \
-- perf bench mem mmap -t 2 -l 100
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ]
# perf report --stdio --no-children
# Overhead Command Shared Object Symbol
# ........ ........ ................. ........................
#
18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms
|
---clear_page_erms
get_page_from_freelist
__alloc_pages_noprof
folio_alloc_mpol_noprof
vma_alloc_folio_noprof
do_anonymous_page
__handle_mm_fault
handle_mm_fault
do_user_addr_fault
exc_page_fault
asm_exc_page_fault
worker_thread
Read/shared vs write/exclusive holding can also be filtered separately:
# perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' ...
# perf record -a -g -e cycles --filter 'lock_held_write(&mmap_lock)' ...
# perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' ...
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 41 +-
tools/perf/util/bpf-filter.c | 934 ++++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 46 +-
tools/perf/util/bpf_skel/sample-filter.h | 59 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 920 ++++++++++++++++++
tools/perf/util/parse-events.c | 3 +-
8 files changed, 2021 insertions(+), 26 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 2351f7b4f5fc..246ae085744d 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -204,7 +204,46 @@ OPTIONS
ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr,
code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat,
p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock,
- mem_dtlb, mem_blk, mem_hops, uid, gid
+ mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>),
+ lock_held_read(<lock>), lock_held_shared(<lock>),
+ lock_held_write(<lock>), lock_held_exclusive(<lock>)
+
+ The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters
+ samples based on whether the specified kernel lock is currently held
+ by the kernel thread or CPU. To differentiate between shared/read
+ holding and exclusive/write holding, use 'lock_held_read(<lock>)'
+ (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)'
+ (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to
+ <lock> (e.g. 'lock_held(&mmap_lock:R)' or 'lock_held(&mmap_lock:W)').
+ <lock> can be a kernel lock symbol name (with or without a leading
+ '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'),
+ a hexadecimal lock address, a slab cache name prefixed with '&'
+ (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known
+ lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock
+ type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
+ 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
+ 'semaphore'). Both read/shared and write/exclusive tracking are
+ supported for 'rwsem' (including 'mmap_lock') and 'rwlock';
+ 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'),
+ 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()')
+ support write/exclusive tracking only.
+
+ Note that 'lock_held(<lock>)' introduces overhead outside of sample
+ overflow handling: in addition to checking the lock's owner field on
+ each sample where possible (such as for mutexes, rt-mutexes, and
+ write-locked rwsems), it attaches BPF fentry/fexit programs to kernel
+ lock acquire and release functions to track held locks (such as
+ spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a
+ BPF map. These BPF hooks run on every kernel lock acquisition and
+ release for the attached lock family, even when no perf sample is
+ being taken. To minimize this overhead, only the hooks for the
+ relevant lock family and mode are attached when known (for example,
+ 'lock_held_read(&mmap_lock)' only hooks rwsem read functions,
+ 'lock_held_write(&mmap_lock)' only hooks rwsem write functions,
+ '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks
+ spinlock functions), whereas slab filters ('&<slab>'), raw hex
+ addresses, or symbols without a recognizable lock-type suffix attach
+ hooks across all lock families (including high-frequency spinlocks).
The <operator> can be one of:
==, !=, >, >=, <, <=, &
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 72e83ac1d6a4..35d6f7c63868 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -40,21 +40,35 @@
* The 'dropped' map is to keep how many samples it dropped by the filter and
* it will be reported as lost samples.
*/
+#include <errno.h>
+#include <stdio.h>
#include <stdlib.h>
+#include <stdbool.h>
+#include <stddef.h>
+#include <string.h>
#include <fcntl.h>
+#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/stat.h>
#include <bpf/bpf.h>
+#include <bpf/btf.h>
#include <linux/err.h>
+#include <linux/kernel.h>
#include <linux/list.h>
+#include <linux/string.h>
+#include <linux/zalloc.h>
#include <api/fs/fs.h>
#include <internal/xyarray.h>
#include <perf/threadmap.h>
+#include <symbol/kallsyms.h>
#include "util/cap.h"
#include "util/debug.h"
#include "util/evsel.h"
+#include "util/lock-contention.h"
+#include "util/string2.h"
+#include "util/symbol_conf.h"
#include "util/target.h"
#include "util/thread_map.h"
#include "util/bpf-utils.h"
@@ -63,6 +77,7 @@
#include <util/bpf-filter-flex.h>
#include <util/bpf-filter-bison.h>
+#include "bpf_skel/lock_data.h"
#include "bpf_skel/sample-filter.h"
#include "bpf_skel/sample_filter.skel.h"
@@ -127,7 +142,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
(evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START))))
return 0;
- if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) {
+ if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID ||
+ expr->term == PBF_TERM_LOCK_HELD) {
/* Not dependent on the sample_type as computed from a BPF helper. */
return 0;
}
@@ -155,9 +171,713 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
return -1;
}
-static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry)
+/*
+ * Program groups for `lock_held(...)` BPF tracing hooks.
+ *
+ * When a `lock_held(...)` filter is specified, only the BPF tracing program
+ * groups required for the requested lock(s) are autoloaded and attached.
+ * If no `lock_held(...)` filter is present, none of these programs are loaded.
+ */
+enum lock_prog_group {
+ LOCK_PROG_MUTEX = 1U << 0,
+ LOCK_PROG_RWSEM_READ = 1U << 1,
+ LOCK_PROG_RWSEM_WRITE = 1U << 2,
+ LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE,
+ LOCK_PROG_SPIN = 1U << 3,
+ LOCK_PROG_RWLOCK_READ = 1U << 4,
+ LOCK_PROG_RWLOCK_WRITE = 1U << 5,
+ LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE,
+ LOCK_PROG_RTMUTEX = 1U << 6,
+ LOCK_PROG_SEM = 1U << 7,
+ LOCK_PROG_PERCPU_RWSEM = 1U << 8,
+ LOCK_PROG_ALL = 0x1ffU,
+};
+
+static unsigned int lock_flags_to_prog_group(unsigned int flags)
+{
+ if (flags & LCB_F_MUTEX)
+ return LOCK_PROG_MUTEX;
+ if (flags & LCB_F_RT)
+ return LOCK_PROG_RTMUTEX;
+ if (flags & LCB_F_PERCPU)
+ return LOCK_PROG_PERCPU_RWSEM;
+ if (flags == LCB_F_SPIN)
+ return LOCK_PROG_SPIN;
+ if (flags & LCB_F_SPIN)
+ return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE;
+ if (flags & LCB_F_READ)
+ return LOCK_PROG_RWSEM_READ;
+ if (flags & LCB_F_WRITE)
+ return LOCK_PROG_RWSEM_WRITE;
+ return LOCK_PROG_SEM;
+}
+
+struct lock_prog_desc {
+ size_t prog_offset;
+ const char *btf_func;
+ unsigned int group;
+};
+
+#define LOCK_FPROG(field, func, grp) \
+ { offsetof(struct sample_filter_bpf, progs.field), func, grp }
+
+static const struct lock_prog_desc lock_progs[] = {
+ /* Mutex */
+ LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
+ LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
+ /* RW semaphore */
+ LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested",
+ LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested",
+ LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM),
+ LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE),
+ /* Spinlock */
+ LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN),
+ /* RWlock */
+ LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE),
+ /* RT-Mutex */
+ LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible",
+ LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX),
+ /* Semaphore */
+ LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM),
+ LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM),
+ /* Percpu-RWsem */
+ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
+ LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+};
+
+#undef LOCK_FPROG
+
+static int split_lock_name_mode(const char *name, int expr_part,
+ char *base_buf, size_t buf_sz,
+ bool *has_amp, unsigned int *mode_out)
+{
+ const char *bare;
+ size_t len;
+ unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK;
+ unsigned int suffix_mode = 0;
+
+ if (!name || !*name)
+ return -EINVAL;
+
+ *has_amp = (name[0] == '&');
+ bare = *has_amp ? name + 1 : name;
+ len = strlen(bare);
+ if (len == 0)
+ return -EINVAL;
+
+ if (len > 2 && !strcmp(bare + len - 2, ":R")) {
+ suffix_mode = PBF_LOCK_MODE_READ;
+ len -= 2;
+ } else if (len > 2 && !strcmp(bare + len - 2, ":W")) {
+ suffix_mode = PBF_LOCK_MODE_WRITE;
+ len -= 2;
+ }
+
+ if (len == 0 || len >= buf_sz)
+ return -EINVAL;
+
+ if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY &&
+ suffix_mode && expr_mode != suffix_mode)
+ return -EINVAL;
+
+ memcpy(base_buf, bare, len);
+ base_buf[len] = '\0';
+
+ if (suffix_mode)
+ *mode_out = suffix_mode;
+ else if (expr_mode)
+ *mode_out = expr_mode;
+ else
+ *mode_out = PBF_LOCK_MODE_ANY;
+
+ return 0;
+}
+
+/*
+ * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R",
+ * "spinlock", "percpu-rwsem").
+ *
+ * A lock type name may correspond to up to two flag combinations (e.g.
+ * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed
+ * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is
+ * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`.
+ */
+static bool parse_lock_type_name(const char *name, unsigned int mode,
+ struct sample_filter_bpf *skel,
+ u64 *encoded_pair, unsigned int *prog_groups)
+{
+ bool found = false;
+ u32 flag1 = 0xff, flag2 = 0xff;
+ unsigned int groups = 0;
+ const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name;
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) {
+ u32 f = lock_type_table[i].flags;
+ bool is_read = (f & LCB_F_READ) != 0;
+
+ if (strcmp(lock_type_table[i].lock_name, lookup) &&
+ strcmp(lock_type_table[i].flags_name, name))
+ continue;
+
+ /*
+ * percpu_down_read() / percpu_up_read() fast paths are inlined
+ * per-CPU counter increments without function entry points, so
+ * only writer holding (percpu_down_write / percpu_up_write) can
+ * be tracked for percpu-rwsem. rwlock-rt is also not hooked.
+ */
+ if ((f & LCB_F_PERCPU) && is_read)
+ continue;
+ if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE)))
+ continue;
+
+ if (is_read && !(mode & PBF_LOCK_MODE_READ))
+ continue;
+ if (!is_read && !(mode & PBF_LOCK_MODE_WRITE))
+ continue;
+
+ if (!found)
+ flag1 = f;
+ else if (f != flag1)
+ flag2 = f;
+ if (skel && f < LCB_F_TYPE_MAX) {
+ skel->bss->has_lock_type = 1;
+ skel->bss->lock_type_filter[f] = 1;
+ }
+ groups |= lock_flags_to_prog_group(f);
+ found = true;
+ }
+
+ if (!found)
+ return false;
+
+ if (encoded_pair)
+ *encoded_pair = flag1 | ((u64)flag2 << 8);
+ if (prog_groups)
+ *prog_groups |= groups;
+ return true;
+}
+
+struct lock_sym_cb_args {
+ const char *name;
+ u64 addr;
+ bool found;
+};
+
+static int find_lock_sym_cb(void *arg, const char *name,
+ char type __maybe_unused, u64 start)
+{
+ struct lock_sym_cb_args *args = arg;
+
+ if (!strcmp(name, args->name)) {
+ args->found = true;
+ args->addr = start;
+ if (start != 0)
+ return 1;
+ }
+ return 0;
+}
+
+static int find_kernel_lock_sym(const char *name, u64 *addr)
{
- int i = 0;
+ struct lock_sym_cb_args args = {
+ .name = name,
+ };
+ const char *kallsyms_filename = "/proc/kallsyms";
+
+ if (symbol_conf.kallsyms_name != NULL)
+ kallsyms_filename = symbol_conf.kallsyms_name;
+
+ if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0)
+ return -ENOENT;
+
+ if (!args.found)
+ return -ENOENT;
+
+ if (args.addr == 0) {
+ pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n",
+ name);
+ return -EACCES;
+ }
+
+ *addr = args.addr;
+ return 0;
+}
+
+static bool parse_lock_hex_addr(const char *str, u64 *addr)
+{
+ char *end;
+ unsigned long long val;
+
+ if (!str || !*str)
+ return false;
+
+ errno = 0;
+ val = strtoull(str, &end, 16);
+ if (*end != '\0' || errno != 0 || val == 0)
+ return false;
+
+ *addr = val;
+ return true;
+}
+
+static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf)
+{
+ u64 addr = 0;
+ char *buf = NULL, *p;
+ size_t len;
+ long last = -1;
+ int ret;
+
+ if (!btf)
+ return;
+
+ ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT);
+ if (ret < 0)
+ return;
+
+ ret = btf__resolve_size(btf, ret);
+ if (ret < 0)
+ return;
+
+ skel->rodata->sizeof_zone = ret;
+
+ if (find_kernel_lock_sym("contig_page_data", &addr) == 0) {
+ skel->rodata->contig_page_data_addr = addr;
+ return;
+ }
+
+ if (find_kernel_lock_sym("node_data", &addr) == 0)
+ skel->rodata->node_data_addr = addr;
+
+ if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) {
+ p = buf;
+ while (p && *p) {
+ char *end;
+
+ last = strtol(p, &end, 0);
+ if (end == p)
+ break;
+ p = end;
+ if (*p == ',' || *p == '-' || *p == '\n')
+ p++;
+ }
+ skel->rodata->nr_nodes = last + 1;
+ free(buf);
+ }
+}
+
+/*
+ * Enable or disable autoload for each lock-tracking BPF program in the
+ * skeleton before `sample_filter_bpf__load()` is called.
+ *
+ * Each program is enabled only if its group is in `needed_groups` and its
+ * target kernel function or tracepoint exists in the running kernel's BTF.
+ */
+static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
+ unsigned int needed_groups, bool needs_lock_syms,
+ bool needs_slab_iter)
+{
+ unsigned int enabled_progs = 0;
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) {
+ struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset;
+ struct bpf_program *prog = *progp;
+ bool enable = false;
+
+ if ((needed_groups & lock_progs[i].group) && btf &&
+ lock_progs[i].btf_func) {
+ if (btf__find_by_name_kind(btf, lock_progs[i].btf_func,
+ BTF_KIND_FUNC) > 0)
+ enable = true;
+ }
+
+ if (enable)
+ enabled_progs++;
+ bpf_program__set_autoload(prog, enable);
+ }
+
+ bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms);
+ if (needs_slab_iter && btf &&
+ btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) {
+ bpf_program__set_autoload(skel->progs.slab_cache_iter, true);
+ } else {
+ bpf_program__set_autoload(skel->progs.slab_cache_iter, false);
+ }
+
+ if (needed_groups && !enabled_progs) {
+ pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
+ return -ENOTSUP;
+ }
+ return 0;
+}
+
+/*
+ * Pre-scan a single filter expression prior to loading the BPF skeleton in
+ * order to determine which BPF maps need to be sized and which lock tracing
+ * program groups need to be autoloaded.
+ */
+static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
+ bool *has_lock_filter,
+ unsigned int *prog_groups,
+ bool *needs_lock_syms,
+ bool *needs_slab_iter)
+{
+ char base[128];
+ bool has_amp;
+ unsigned int mode;
+ u64 dummy;
+
+ if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name)
+ return;
+
+ *has_lock_filter = true;
+ if (split_lock_name_mode(expr->lock_name, expr->part, base,
+ sizeof(base), &has_amp, &mode) < 0)
+ return;
+
+ if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
+ if (mode & PBF_LOCK_MODE_READ)
+ *prog_groups |= LOCK_PROG_RWSEM_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ *prog_groups |= LOCK_PROG_RWSEM_WRITE;
+ return;
+ }
+ if (!strcmp(base, "siglock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ return;
+ }
+ if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *needs_lock_syms = true;
+ return;
+ }
+ if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, prog_groups))
+ return;
+
+ if (!has_amp && parse_lock_hex_addr(base, &dummy)) {
+ *prog_groups |= LOCK_PROG_ALL;
+ return;
+ }
+
+ if (has_amp)
+ *needs_slab_iter = true;
+
+ if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) {
+ *prog_groups |= LOCK_PROG_RTMUTEX;
+ } else if (strtailcmp(base, "mutex") == 0) {
+ *prog_groups |= LOCK_PROG_MUTEX;
+ } else if (strtailcmp(base, "percpu_rwsem") == 0 ||
+ strtailcmp(base, "pcpu_rwsem") == 0) {
+ *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
+ } else if (strtailcmp(base, "rwsem") == 0) {
+ if (mode & PBF_LOCK_MODE_READ)
+ *prog_groups |= LOCK_PROG_RWSEM_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ *prog_groups |= LOCK_PROG_RWSEM_WRITE;
+ } else if (strtailcmp(base, "spinlock") == 0) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ } else if (strtailcmp(base, "rwlock") == 0) {
+ if (mode & PBF_LOCK_MODE_READ)
+ *prog_groups |= LOCK_PROG_RWLOCK_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ *prog_groups |= LOCK_PROG_RWLOCK_WRITE;
+ } else {
+ *prog_groups |= LOCK_PROG_ALL;
+ }
+
+ if (!has_amp && find_kernel_lock_sym(base, &dummy) != 0)
+ *needs_slab_iter = true;
+}
+
+static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter,
+ unsigned int *prog_groups,
+ bool *needs_lock_syms,
+ bool *needs_slab_iter)
+{
+ struct perf_bpf_filter_expr *expr, *group;
+
+ *has_lock_filter = false;
+ *prog_groups = 0;
+ *needs_lock_syms = false;
+ *needs_slab_iter = false;
+
+ list_for_each_entry(expr, &evsel->bpf_filters, list) {
+ prescan_one_lock_expr(expr, has_lock_filter, prog_groups,
+ needs_lock_syms, needs_slab_iter);
+ if (expr->op == PBF_OP_GROUP_BEGIN) {
+ list_for_each_entry(group, &expr->groups, list) {
+ prescan_one_lock_expr(group, has_lock_filter,
+ prog_groups, needs_lock_syms,
+ needs_slab_iter);
+ }
+ }
+ }
+}
+
+static void run_slab_cache_iter(struct sample_filter_bpf *skel)
+{
+ struct bpf_link *link;
+ int fd;
+ char buf[256];
+
+ if (!bpf_program__autoload(skel->progs.slab_cache_iter))
+ return;
+
+ link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL);
+ if (!link || IS_ERR(link)) {
+ pr_debug("cannot attach slab cache iter\n");
+ return;
+ }
+
+ fd = bpf_iter_create(bpf_link__fd(link));
+ if (fd >= 0) {
+ while (read(fd, buf, sizeof(buf)) > 0)
+ continue;
+ close(fd);
+ }
+
+ bpf_link__destroy(link);
+}
+
+static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
+ u64 *kmem_cache_addr)
+{
+ int cache_fd;
+ u64 key, *prev_key = NULL;
+
+ if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter))
+ return false;
+
+ cache_fd = bpf_map__fd(skel->maps.slab_caches);
+ while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) {
+ struct slab_cache_data data;
+
+ if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0)
+ break;
+
+ if (!strcmp(data.name, name)) {
+ *kmem_cache_addr = key;
+ return true;
+ }
+ prev_key = &key;
+ }
+ return false;
+}
+
+/*
+ * Resolve a `lock_held(<name>)` filter expression into `entry->part`
+ * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate
+ * the corresponding BPF filter maps and BSS control flags.
+ *
+ * Resolution order:
+ * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
+ * `rq_lock`, `zone_lock` (with or without leading '&').
+ * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`,
+ * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`,
+ * `semaphore`, etc.
+ * 3. Hexadecimal lock address (when no leading '&').
+ * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`).
+ * 5. Global/static kernel lock symbol in kallsyms (with or without leading '&',
+ * e.g. `&cgroup_mutex` or `cgroup_mutex`).
+ * 6. Slab cache name without leading '&' fallback.
+ */
+static int resolve_lock_expr(struct sample_filter_bpf *skel,
+ struct perf_bpf_filter_expr *expr,
+ struct perf_bpf_filter_entry *entry)
+{
+ const char *name = expr->lock_name;
+ char base[128];
+ bool has_amp;
+ unsigned int mode;
+ u64 addr = 0, encoded_pair = 0;
+ u8 one = 1;
+ int ret;
+
+ if (split_lock_name_mode(name, expr->part, base, sizeof(base),
+ &has_amp, &mode) < 0) {
+ pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: "");
+ return -EINVAL;
+ }
+
+ if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) {
+ entry->part = PBF_LOCK_MMAP | mode;
+ entry->target = 0;
+ if (skel)
+ skel->bss->has_mmap_lock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "siglock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_SIGHAND | mode;
+ entry->target = 0;
+ if (skel)
+ skel->bss->has_siglock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "rq_lock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_CLASS | mode;
+ entry->target = LOCK_CLASS_RQLOCK;
+ if (skel)
+ skel->bss->has_lock_class = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "zone_lock")) {
+ if (!(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+ entry->part = PBF_LOCK_CLASS | mode;
+ entry->target = LOCK_CLASS_ZONE_LOCK;
+ if (skel)
+ skel->bss->has_lock_class = 1;
+ return 0;
+ }
+
+ if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
+ entry->part = PBF_LOCK_TYPE | mode;
+ entry->target = encoded_pair;
+ return 0;
+ }
+
+ if (!has_amp && parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) {
+ pr_err("Error: Lock type '%s' does not support the requested read/write mode\n",
+ name);
+ return -EINVAL;
+ }
+
+ if (!has_amp && parse_lock_hex_addr(base, &addr)) {
+ entry->part = PBF_LOCK_ADDR | mode;
+ entry->target = addr;
+ if (skel) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
+ }
+ return 0;
+ }
+
+ if (has_amp && find_slab_cache(skel, base, &addr)) {
+ int fd = bpf_map__fd(skel->maps.slab_filter);
+
+ entry->part = PBF_LOCK_SLAB | mode;
+ entry->target = addr;
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_slab = 1;
+ return 0;
+ }
+
+ ret = find_kernel_lock_sym(base, &addr);
+ if (ret == 0) {
+ entry->part = PBF_LOCK_ADDR | mode;
+ entry->target = addr;
+ if (skel) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
+ }
+ return 0;
+ }
+ if (ret == -EACCES)
+ return ret;
+
+ if (!has_amp && find_slab_cache(skel, base, &addr)) {
+ int fd = bpf_map__fd(skel->maps.slab_filter);
+
+ entry->part = PBF_LOCK_SLAB | mode;
+ entry->target = addr;
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_slab = 1;
+ return 0;
+ }
+
+ pr_err("Error: Cannot resolve kernel lock '%s'\n", name);
+ return -EINVAL;
+}
+
+static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct perf_bpf_filter_entry *entry)
+{
+ int i = 0, ret;
struct perf_bpf_filter_expr *expr;
list_for_each_entry(expr, &evsel->bpf_filters, list) {
@@ -171,6 +891,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].part = expr->part;
entry[i].term = expr->term;
entry[i].value = expr->val;
+ entry[i].target = 0;
+ if (expr->term == PBF_TERM_LOCK_HELD) {
+ ret = resolve_lock_expr(skel, expr, &entry[i]);
+ if (ret < 0)
+ return ret;
+ }
i++;
if (expr->op == PBF_OP_GROUP_BEGIN) {
@@ -184,6 +910,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].part = group->part;
entry[i].term = group->term;
entry[i].value = group->val;
+ entry[i].target = 0;
+ if (group->term == PBF_TERM_LOCK_HELD) {
+ ret = resolve_lock_expr(skel, group, &entry[i]);
+ if (ret < 0)
+ return ret;
+ }
i++;
}
@@ -423,10 +1155,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
{
int i, x, y, fd, ret;
struct sample_filter_bpf *skel = NULL;
+ struct btf *btf = NULL;
struct bpf_program *prog;
struct bpf_link *link;
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
+ bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false;
+ unsigned int prog_groups = 0;
#if LIBBPF_CURRENT_VERSION_GEQ(1, 7)
DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts,
.dont_enable = true);
@@ -438,15 +1173,24 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
if (entry == NULL)
return -1;
- ret = get_filter_entries(evsel, entry);
- if (ret < 0) {
- pr_err("Failed to process filter entries\n");
- goto err;
- }
+ prescan_lock_filters(evsel, &has_lock_filter, &prog_groups,
+ &needs_lock_syms, &needs_slab_iter);
- if (needs_idx_hash && geteuid() != 0) {
+ if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
int zero = 0;
+ if (has_lock_filter) {
+ pr_err("Error: lock_held BPF filter requires CAP_BPF capability\n");
+ ret = -EPERM;
+ goto err;
+ }
+
+ ret = get_filter_entries(evsel, NULL, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
+
/* The filters map is shared among other processes */
ret = create_idx_hash(evsel, entry);
if (ret < 0)
@@ -484,13 +1228,69 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return 0;
}
- skel = sample_filter_bpf__open_and_load();
+ skel = sample_filter_bpf__open();
if (!skel) {
+ ret = -errno;
+ pr_err("Failed to open perf sample-filter BPF skeleton\n");
+ goto err;
+ }
+
+ if (has_lock_filter) {
+ btf = btf__load_vmlinux_btf();
+ bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES);
+ bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS);
+ if (needs_slab_iter) {
+ bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS);
+ bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES);
+ }
+ if (needs_lock_syms) {
+ bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
+ init_numa_data(skel, btf);
+ }
+ ret = configure_lock_progs(skel, btf, prog_groups, needs_lock_syms,
+ needs_slab_iter);
+ if (ret < 0)
+ goto err;
+ } else {
+ configure_lock_progs(skel, NULL, 0, false, false);
+ }
+
+ if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
pr_err("Failed to load perf sample-filter BPF skeleton\n");
goto err;
}
+ if (has_lock_filter && needs_slab_iter)
+ run_slab_cache_iter(skel);
+
+ ret = get_filter_entries(evsel, skel, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
+
+ if (has_lock_filter) {
+ if (needs_lock_syms && skel->bss->has_lock_class) {
+ DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts,
+ .flags = BPF_F_TEST_RUN_ON_CPU,
+ );
+ int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms);
+
+ bpf_prog_test_run_opts(prog_fd, &opts);
+ }
+ bpf_program__set_autoattach(skel->progs.perf_sample_filter, false);
+ bpf_program__set_autoattach(skel->progs.collect_lock_syms, false);
+ bpf_program__set_autoattach(skel->progs.slab_cache_iter, false);
+
+ ret = sample_filter_bpf__attach(skel);
+ if (ret < 0) {
+ pr_err("Failed to attach lock sample-filter BPF programs\n");
+ goto err;
+ }
+ skel->bss->lock_enabled = 1;
+ }
+
i = 0;
fd = bpf_map__fd(skel->maps.filters);
@@ -513,11 +1313,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
}
}
}
+ btf__free(btf);
free(entry);
evsel->bpf_skel = skel;
return 0;
err:
+ btf__free(btf);
free(entry);
if (!list_empty(&pinned_filters)) {
struct pinned_filter_idx *pfi, *tmp;
@@ -532,6 +1334,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
return ret;
}
+void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr)
+{
+ struct perf_bpf_filter_expr *group, *tmp;
+
+ if (!expr)
+ return;
+
+ list_for_each_entry_safe(group, tmp, &expr->groups, list) {
+ list_del(&group->list);
+ perf_bpf_filter_expr__delete(group);
+ }
+ zfree(&expr->lock_name);
+ free(expr);
+}
+
int perf_bpf_filter__destroy(struct evsel *evsel)
{
struct perf_bpf_filter_expr *expr, *tmp;
@@ -539,7 +1356,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel)
list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) {
list_del(&expr->list);
- free(expr);
+ perf_bpf_filter_expr__delete(expr);
}
sample_filter_bpf__destroy(evsel->bpf_skel);
@@ -591,7 +1408,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
{
struct perf_bpf_filter_expr *expr;
- expr = malloc(sizeof(*expr));
+ expr = zalloc(sizeof(*expr));
if (expr != NULL) {
expr->term = term;
expr->part = part;
@@ -602,6 +1419,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
return expr;
}
+struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
+ int mode,
+ enum perf_bpf_filter_op op,
+ unsigned long val)
+{
+ struct perf_bpf_filter_expr *expr;
+
+ if (!lock_name || !*lock_name)
+ return NULL;
+
+ expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val);
+ if (!expr)
+ return NULL;
+
+ expr->lock_name = strdup(lock_name);
+ if (!expr->lock_name) {
+ free(expr);
+ return NULL;
+ }
+ return expr;
+}
+
static bool check_bpf_filter_capable(void)
{
int fd;
@@ -622,14 +1461,11 @@ static bool check_bpf_filter_capable(void)
return false;
}
-int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
+int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
{
YY_BUFFER_STATE buffer;
int ret;
- if (!check_bpf_filter_capable())
- return -EPERM;
-
buffer = perf_bpf_filter__scan_string(str);
ret = perf_bpf_filter_parse(expr_head);
@@ -641,6 +1477,53 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
return ret;
}
+int perf_bpf_filter__parse(struct list_head *expr_head, const char *str)
+{
+ if (!check_bpf_filter_capable())
+ return -EPERM;
+
+ return __perf_bpf_filter__parse(expr_head, str);
+}
+
+static int pin_filter_object(struct sample_filter_bpf *skel, const char *path)
+{
+ char *prog_path = NULL;
+ int ret;
+
+ ret = bpf_object__pin_maps(skel->obj, path);
+ if (ret < 0)
+ return ret;
+
+ if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
+ bpf_object__unpin_maps(skel->obj, path);
+ return -ENOMEM;
+ }
+
+ ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path);
+ if (ret < 0)
+ bpf_object__unpin_maps(skel->obj, path);
+
+ free(prog_path);
+ return ret;
+}
+
+static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path)
+{
+ char *prog_path = NULL;
+ int ret, map_ret;
+
+ if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) {
+ bpf_object__unpin_maps(skel->obj, path);
+ return -ENOMEM;
+ }
+
+ ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path);
+ free(prog_path);
+
+ map_ret = bpf_object__unpin_maps(skel->obj, path);
+ return ret < 0 ? ret : map_ret;
+}
+
int perf_bpf_filter__pin(void)
{
struct sample_filter_bpf *skel;
@@ -654,6 +1537,8 @@ int perf_bpf_filter__pin(void)
goto err;
}
+ configure_lock_progs(skel, NULL, 0, false, false);
+
/* pinned program will use pid-hash */
bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH);
@@ -674,7 +1559,7 @@ int perf_bpf_filter__pin(void)
goto err;
}
- ret = bpf_object__pin(skel->obj, path);
+ ret = pin_filter_object(skel, path);
if (ret < 0) {
pr_err("Failed to pin BPF filter objects\n");
goto err;
@@ -683,7 +1568,7 @@ int perf_bpf_filter__pin(void)
/* setup access permissions for the pinned objects */
dir_fd = open(path, O_PATH);
if (dir_fd < 0) {
- bpf_object__unpin(skel->obj, path);
+ unpin_filter_object(skel, path);
ret = dir_fd;
goto err;
}
@@ -727,6 +1612,8 @@ int perf_bpf_filter__pin(void)
err_close:
close(dir_fd);
+ if (ret < 0)
+ unpin_filter_object(skel, path);
err:
free(path);
@@ -740,13 +1627,20 @@ int perf_bpf_filter__unpin(void)
char *path = NULL;
int ret = -1;
- skel = sample_filter_bpf__open_and_load();
+ skel = sample_filter_bpf__open();
if (!skel) {
ret = -errno;
pr_err("Failed to open perf sample-filter BPF skeleton\n");
goto err;
}
+ configure_lock_progs(skel, NULL, 0, false, false);
+ if (sample_filter_bpf__load(skel) < 0) {
+ ret = -errno;
+ pr_err("Failed to load perf sample-filter BPF skeleton\n");
+ goto err;
+ }
+
if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(),
PERF_BPF_FILTER_PIN_PATH) < 0) {
ret = -errno;
@@ -754,7 +1648,7 @@ int perf_bpf_filter__unpin(void)
goto err;
}
- ret = bpf_object__unpin(skel->obj, path);
+ ret = unpin_filter_object(skel, path);
err:
free(path);
diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h
index 818c554b91b2..fda95ff226b3 100644
--- a/tools/perf/util/bpf-filter.h
+++ b/tools/perf/util/bpf-filter.h
@@ -14,6 +14,13 @@ struct perf_bpf_filter_expr {
int part;
enum perf_bpf_filter_term term;
unsigned long val;
+ /*
+ * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions
+ * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R").
+ * Resolved into `entry.part` and `entry.target` during
+ * perf_bpf_filter__prepare().
+ */
+ char *lock_name;
};
struct evsel;
@@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term
int part,
enum perf_bpf_filter_op op,
unsigned long val);
+struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name,
+ int mode,
+ enum perf_bpf_filter_op op,
+ unsigned long val);
+void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr);
+int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
int perf_bpf_filter__parse(struct list_head *expr_head, const char *str);
int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target);
int perf_bpf_filter__destroy(struct evsel *evsel);
diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l
index 1be9df6550fc..b4022110f1e5 100644
--- a/tools/perf/util/bpf-filter.l
+++ b/tools/perf/util/bpf-filter.l
@@ -1,9 +1,12 @@
%option prefix="perf_bpf_filter_"
%option noyywrap
+%x LOCK_ARG
+
%{
#include <stdio.h>
#include <stdlib.h>
+#include <string.h>
#include <linux/perf_event.h>
#include "bpf-filter.h"
@@ -76,7 +79,7 @@ static int path_or_error(void)
num_dec [0-9]+
num_hex 0[Xx][0-9a-fA-F]+
space [ \t]+
-path [^ \t\n,]+
+path [^ \t\n,()!=>&|<]+
ident [_a-zA-Z][_a-zA-Z0-9]+
%%
@@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); }
uid { return sample(PBF_TERM_UID); }
gid { return sample(PBF_TERM_GID); }
cgroup { return sample_path(PBF_TERM_CGROUP); }
-
+lock_held[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY;
+ return BFT_LOCK_HELD;
+ }
+(lock_held_read|lock_held_shared)[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ;
+ return BFT_LOCK_HELD;
+ }
+(lock_held_write|lock_held_exclusive)[ \t]*"(" {
+ BEGIN(LOCK_ARG);
+ perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE;
+ return BFT_LOCK_HELD;
+ }
+
+<LOCK_ARG>{space} { }
+<LOCK_ARG>[^ \t\n()]+ {
+ perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text);
+ return BFT_LOCK_NAME;
+ }
+<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; }
+<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; }
+
+"!" { return '!'; }
"==" { return operator(PBF_OP_EQ); }
"!=" { return operator(PBF_OP_NEQ); }
">" { return operator(PBF_OP_GT); }
diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y
index 5a79a8e7a45b..6ee6edef13f7 100644
--- a/tools/perf/util/bpf-filter.y
+++ b/tools/perf/util/bpf-filter.y
@@ -11,6 +11,7 @@
#include <string.h>
#include <linux/compiler.h>
#include <linux/list.h>
+#include <linux/zalloc.h>
#include "bpf-filter.h"
#include "cgroup.h"
@@ -31,6 +32,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
{
unsigned long num;
char *path;
+ char *lock_name;
+ struct {
+ char *name;
+ int mode;
+ } lock;
struct {
enum perf_bpf_filter_term term;
int part;
@@ -40,12 +46,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused,
}
%token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH
+%token BFT_LOCK_HELD BFT_LOCK_NAME
%type <expr> filter_term filter_expr
-%destructor { free ($$); } <expr>
+%destructor { perf_bpf_filter_expr__delete($$); } <expr>
+%destructor { free($$); } <lock_name>
+%destructor { free($$.name); } <lock>
%type <sample> BFT_SAMPLE BFT_SAMPLE_PATH
%type <op> BFT_OP
-%type <num> BFT_NUM
+%type <num> BFT_NUM BFT_LOCK_HELD
%type <path> BFT_PATH
+%type <lock_name> BFT_LOCK_NAME
+%type <lock> lock_term
%%
@@ -105,5 +116,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH
$$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id);
cgroup__put(cgrp);
}
+|
+lock_term
+{
+ $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1);
+ zfree(&$1.name);
+ if (!$$)
+ YYERROR;
+}
+|
+'!' lock_term
+{
+ $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0);
+ zfree(&$2.name);
+ if (!$$)
+ YYERROR;
+}
+|
+lock_term BFT_OP BFT_NUM
+{
+ $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3);
+ zfree(&$1.name);
+ if (!$$)
+ YYERROR;
+}
+
+lock_term:
+BFT_LOCK_HELD BFT_LOCK_NAME ')'
+{
+ $$.name = $2;
+ $$.mode = $1;
+}
%%
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index 683fec85e71e..cd711d1d04b6 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -53,14 +53,73 @@ enum perf_bpf_filter_term {
/* Terms computed from BPF helpers. */
PBF_TERM_UID,
PBF_TERM_GID,
+ /*
+ * Kernel lock held predicate (`lock_held(...)`). Evaluates to 1 if the
+ * specified kernel lock is currently held by the running task/CPU, or
+ * 0 otherwise. The lock specification kind is stored in `entry.part`
+ * (`enum perf_bpf_lock_target_kind`) and the resolved target address,
+ * class ID, or type flag encoding is stored in `entry.target`.
+ */
+ PBF_TERM_LOCK_HELD,
};
+/*
+ * Lock target categories for PBF_TERM_LOCK_HELD filters.
+ * Stored in `perf_bpf_filter_entry.part`.
+ */
+enum perf_bpf_lock_target_kind {
+ PBF_LOCK_NONE = 0,
+ /*
+ * Specific global or static lock address (resolved from kallsyms or
+ * given as a hex address). `entry.target` holds the kernel address.
+ */
+ PBF_LOCK_ADDR,
+ /*
+ * Dynamic lock allocated inside a slab cache (`&slab_name`).
+ * `entry.target` holds the kernel `struct kmem_cache *` address.
+ */
+ PBF_LOCK_SLAB,
+ /*
+ * Per-process `current->mm->mmap_lock` (or `mmap_sem`).
+ */
+ PBF_LOCK_MMAP,
+ /*
+ * Per-process `current->sighand->siglock`.
+ */
+ PBF_LOCK_SIGHAND,
+ /*
+ * Semi-global lock class (`rq_lock` or `zone_lock`).
+ * `entry.target` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or
+ * LOCK_CLASS_ZONE_LOCK).
+ */
+ PBF_LOCK_CLASS,
+ /*
+ * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.).
+ * `entry.target` packs up to two `LCB_F_*` flag combinations in the
+ * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
+ */
+ PBF_LOCK_TYPE,
+};
+
+/*
+ * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD:
+ * Bits 0..3: `enum perf_bpf_lock_target_kind`
+ * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding)
+ * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding)
+ */
+#define PBF_LOCK_KIND_MASK 0x0fU
+#define PBF_LOCK_MODE_READ (1U << 4)
+#define PBF_LOCK_MODE_WRITE (1U << 5)
+#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE)
+#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY
+
/* BPF map entry for filtering */
struct perf_bpf_filter_entry {
enum perf_bpf_filter_op op;
__u32 part; /* sub-sample type info when it has multiple values */
enum perf_bpf_filter_term term;
__u64 value;
+ __u64 target; /* lock address, kmem_cache *, class, or type flags */
};
struct idx_hash_key {
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index e5666d4c1722..fbd2e51815d7 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -5,8 +5,47 @@
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
+#include "lock_data.h"
#include "sample-filter.h"
+/*
+ * Key for the `held_locks` BPF hash map tracking currently held kernel locks.
+ *
+ * - `addr_or_id`:
+ * - PBF_LOCK_ADDR: kernel virtual address of the lock instance
+ * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK)
+ * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address
+ * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags
+ * - `owner_id`:
+ * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`),
+ * since the holder may sleep and migrate across CPUs while holding the lock.
+ * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks
+ * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption
+ * is disabled on the holding CPU and spinlocks such as `rq_lock` may be
+ * acquired by one task and released by the next task across `context_switch()`.
+ * - `kind`: `enum perf_bpf_lock_target_kind`.
+ */
+struct held_lock_key {
+ __u64 addr_or_id;
+ __u32 owner_id;
+ __u32 kind;
+};
+
+/*
+ * Value for `held_locks` entries.
+ * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`)
+ * and exclusive/write acquisitions (including nested class/slab/type
+ * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of
+ * the lock so release tracepoints that do not carry flags (such as
+ * `lock:contended_release`) can decrement matching class/slab/type entries.
+ */
+struct held_lock_val {
+ __u32 read_count;
+ __u32 write_count;
+ __u32 flags;
+ __u32 pad;
+};
+
/* BPF map that will be filled by user space */
struct filters {
__uint(type, BPF_MAP_TYPE_HASH);
@@ -43,10 +82,198 @@ struct lost_count {
__uint(max_entries, 1);
} dropped SEC(".maps");
+/* Currently held kernel locks tracked by fentry/fexit and lock tracepoints. */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, struct held_lock_key);
+ __type(value, struct held_lock_val);
+ __uint(max_entries, 1);
+} held_locks SEC(".maps");
+
+/* Target lock addresses for PBF_LOCK_ADDR filters. */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, __u64);
+ __type(value, __u8);
+ __uint(max_entries, 1);
+} lock_addr_filter SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u32));
+ __uint(max_entries, 1);
+} lock_syms SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(__u8));
+ __uint(max_entries, 1);
+} slab_filter SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u64));
+ __uint(value_size, sizeof(struct slab_cache_data));
+ __uint(max_entries, 1);
+} slab_caches SEC(".maps");
+
volatile const int use_idx_hash;
+const volatile __u64 contig_page_data_addr;
+const volatile __u64 node_data_addr;
+const volatile int nr_nodes;
+const volatile int sizeof_zone;
+
+__u32 slab_cache_id;
+
+/* Control flags set by userspace when lock_held(...) filters are active. */
+int lock_enabled;
+int has_lock_addr;
+int has_lock_slab;
+int has_lock_class;
+int has_lock_type;
+int has_mmap_lock;
+int has_siglock;
+__u8 lock_type_filter[LCB_F_TYPE_MAX];
+
void *bpf_cast_to_kern_ctx(void *) __ksym;
+/*
+ * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`.
+ *
+ * For locks that maintain a task_struct pointer in their `owner` field
+ * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`),
+ * `perf_sample_filter` can inspect the lock structure directly in kernel memory
+ * and compare the owner against `current` (`bpf_get_current_task()`).
+ *
+ * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear
+ * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and
+ * spinlocks are checked via the `held_locks` BPF map instead.
+ */
+static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr)
+{
+ if (!curr)
+ return false;
+
+ return (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr ||
+ (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr ||
+ (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr;
+}
+
+static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode)
+{
+ if (!val)
+ return false;
+ if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0)
+ return true;
+ if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0)
+ return true;
+ return false;
+}
+
+static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind,
+ __u32 mode, __u32 pid, __u32 cpu_id)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = pid,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+
+ if (pid) {
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+ }
+
+ key.owner_id = cpu_id;
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+
+ return 0;
+}
+
+static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode,
+ __u32 pid, __u32 cpu_id)
+{
+ __u32 flag1 = target & 0xff;
+ __u32 flag2 = (target >> 8) & 0xff;
+
+ if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ return 0;
+}
+
+static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry)
+{
+ __u64 curr;
+ __u32 pid;
+ __u32 cpu_id;
+ __u32 kind;
+ __u32 mode;
+ __u64 lock_addr;
+
+ /*
+ * Pinned filters (use_idx_hash != 0) share a world-writable filters
+ * map and do not support lock_held(). Reject evaluation when pinned
+ * or when lock_enabled is not set so unprivileged users cannot probe
+ * arbitrary kernel addresses via entry->target.
+ */
+ if (use_idx_hash || !lock_enabled)
+ return 0;
+
+ curr = bpf_get_current_task();
+ pid = (__u32)bpf_get_current_pid_tgid();
+ cpu_id = 0x80000000U | bpf_get_smp_processor_id();
+ kind = entry->part & PBF_LOCK_KIND_MASK;
+ mode = entry->part & PBF_LOCK_MODE_MASK;
+
+ if (!mode)
+ mode = PBF_LOCK_MODE_ANY;
+
+ switch (kind) {
+ case PBF_LOCK_ADDR:
+ lock_addr = entry->target;
+ if ((mode & PBF_LOCK_MODE_WRITE) &&
+ is_direct_lock_owner(lock_addr, curr))
+ return 1;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_MMAP:
+ lock_addr = get_current_mmap_lock(curr);
+ if (!lock_addr)
+ return 0;
+ if ((mode & PBF_LOCK_MODE_WRITE) &&
+ (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr)
+ return 1;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_SIGHAND:
+ lock_addr = get_current_siglock(curr);
+ if (!lock_addr)
+ return 0;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id);
+
+ case PBF_LOCK_CLASS:
+ return is_lock_in_map(entry->target, PBF_LOCK_CLASS, mode, pid, cpu_id);
+
+ case PBF_LOCK_SLAB:
+ return is_lock_in_map(entry->target, PBF_LOCK_SLAB, mode, pid, cpu_id);
+
+ case PBF_LOCK_TYPE:
+ return is_lock_type_in_map(entry->target, mode, pid, cpu_id);
+
+ default:
+ return 0;
+ }
+}
+
/* new kernel perf_sample_data definition */
struct perf_sample_data___new {
__u64 sample_flags;
@@ -175,6 +402,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx,
return bpf_get_current_uid_gid() & 0xFFFFFFFF;
case PBF_TERM_GID:
return bpf_get_current_uid_gid() >> 32;
+ case PBF_TERM_LOCK_HELD:
+ return check_lock_held(entry);
case PBF_TERM_NONE:
case __PBF_UNUSED_TERM4:
case __PBF_UNUSED_TERM5:
@@ -295,4 +524,695 @@ int perf_sample_filter(void *ctx)
return 0;
}
+static __always_inline __u32 get_lock_owner_id(__u32 flags)
+{
+ __u32 owner_id;
+
+ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN)
+ return 0x80000000U | bpf_get_smp_processor_id();
+
+ owner_id = (__u32)bpf_get_current_pid_tgid();
+ if (!owner_id)
+ owner_id = 0x80000000U | bpf_get_smp_processor_id();
+ return owner_id;
+}
+
+static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id,
+ __u32 kind, __u32 flags, bool acquired)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = owner_id,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (acquired) {
+ if (val) {
+ if (is_read)
+ __sync_fetch_and_add(&val->read_count, 1);
+ else
+ __sync_fetch_and_add(&val->write_count, 1);
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags,
+ };
+
+ bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY);
+ }
+ } else if (val) {
+ if (is_read) {
+ if (val->read_count <= 1 && val->write_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->read_count > 0)
+ __sync_fetch_and_add(&val->read_count, -1);
+ } else {
+ if (val->write_count <= 1 && val->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->write_count > 0)
+ __sync_fetch_and_add(&val->write_count, -1);
+ }
+ }
+}
+
+static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id)
+{
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ bool matched = false;
+ bool is_read = (flags & LCB_F_READ) != 0;
+ __u32 class_id = 0;
+ __u64 kmem_cache_addr = 0;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool match_type = false;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
+ matched = true;
+
+ if (has_mmap_lock || has_siglock) {
+ __u32 ltype = check_lock_type(lock_addr, flags);
+
+ if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
+ (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
+ matched = true;
+ }
+
+ if (has_lock_class && flags == LCB_F_SPIN) {
+ __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
+
+ if (cid) {
+ class_id = *cid;
+ matched = true;
+ }
+ }
+
+ if (has_lock_slab && bpf_get_kmem_cache) {
+ kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
+ if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr))
+ matched = true;
+ else
+ kmem_cache_addr = 0;
+ }
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) {
+ match_type = true;
+ matched = true;
+ }
+
+ if (!matched)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (existing) {
+ /* Non-reader locks are not recursive; avoid double-counting */
+ if (!is_read && existing->write_count > 0)
+ return 0;
+ if (is_read)
+ __sync_fetch_and_add(&existing->read_count, 1);
+ else
+ __sync_fetch_and_add(&existing->write_count, 1);
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags,
+ };
+
+ if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0)
+ return 0;
+ }
+
+ if (class_id)
+ update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, flags, true);
+ if (kmem_cache_addr) {
+ update_held_lock_count((__u64)kmem_cache_addr, owner_id,
+ PBF_LOCK_SLAB, flags, true);
+ }
+ if (match_type)
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, true);
+
+ return 0;
+}
+
+static __always_inline int record_lock_acquired(void *ctx, __u32 flags)
+{
+ return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags));
+}
+
+static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags)
+{
+ __u32 owner_id = get_lock_owner_id(flags);
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (!existing)
+ return 0;
+
+ if (is_read) {
+ if (existing->read_count <= 1 && existing->write_count == 0)
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ else if (existing->read_count > 0)
+ __sync_fetch_and_add(&existing->read_count, -1);
+ } else {
+ if (existing->write_count == 0 && existing->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ else if (existing->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ else
+ existing->write_count = 0;
+ }
+
+ if (has_lock_class && flags == LCB_F_SPIN) {
+ __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr);
+
+ if (cid)
+ update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false);
+ }
+
+ if (has_lock_slab && bpf_get_kmem_cache) {
+ __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr);
+
+ if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) {
+ update_held_lock_count(kmem_cache_addr, owner_id,
+ PBF_LOCK_SLAB, flags, false);
+ }
+ }
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag])
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false);
+
+ return 0;
+}
+
+static __always_inline int record_lock_released(void *ctx, __u32 flags)
+{
+ return record_lock_released_addr(get_lock_addr(ctx), flags);
+}
+
+/* Mutex acquire/release hooks */
+SEC("fexit/mutex_lock")
+int BPF_PROG(fexit_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_nested")
+int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io")
+int BPF_PROG(fexit_mutex_lock_io, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io_nested")
+int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_interruptible")
+int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_interruptible_nested")
+int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_killable")
+int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_trylock")
+int BPF_PROG(fexit_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fentry/mutex_unlock")
+int BPF_PROG(fentry_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_MUTEX);
+}
+
+/* RW semaphore acquire/release hooks */
+SEC("fexit/down_read")
+int BPF_PROG(fexit_down_read, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_nested")
+int BPF_PROG(fexit_down_read_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_interruptible")
+int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable")
+int BPF_PROG(fexit_down_read_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable_nested")
+int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_trylock")
+int BPF_PROG(fexit_down_read_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_write")
+int BPF_PROG(fexit_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_nested")
+int BPF_PROG(fexit_down_write_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_killable")
+int BPF_PROG(fexit_down_write_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_killable_nested")
+int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_trylock")
+int BPF_PROG(fexit_down_write_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/downgrade_write")
+int BPF_PROG(fexit_downgrade_write, void *sem)
+{
+ record_lock_acquired(ctx, LCB_F_READ);
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+SEC("fentry/up_read")
+int BPF_PROG(fentry_up_read, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_READ);
+}
+
+SEC("fentry/up_write")
+int BPF_PROG(fentry_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+/* Spinlock acquire/release hooks */
+SEC("fexit/_raw_spin_lock")
+int BPF_PROG(fexit_raw_spin_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_nested")
+int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq")
+int BPF_PROG(fexit_raw_spin_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq_disable")
+int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave")
+int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave_nested")
+int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_bh")
+int BPF_PROG(fexit_raw_spin_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_trylock")
+int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fexit/_raw_spin_trylock_bh")
+int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fentry/_raw_spin_unlock")
+int BPF_PROG(fentry_raw_spin_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq")
+int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq_enable")
+int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irqrestore")
+int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_bh")
+int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+/* RWlock acquire/release hooks */
+SEC("fexit/_raw_read_lock")
+int BPF_PROG(fexit_raw_read_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_lock_irq")
+int BPF_PROG(fexit_raw_read_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_lock_irqsave")
+int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_lock_bh")
+int BPF_PROG(fexit_raw_read_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_trylock")
+int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+ return 0;
+}
+
+SEC("fentry/_raw_read_unlock")
+int BPF_PROG(fentry_raw_read_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irq")
+int BPF_PROG(fentry_raw_read_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irqrestore")
+int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_bh")
+int BPF_PROG(fentry_raw_read_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_write_lock")
+int BPF_PROG(fexit_raw_write_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_nested")
+int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irq")
+int BPF_PROG(fexit_raw_write_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irqsave")
+int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_bh")
+int BPF_PROG(fexit_raw_write_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_trylock")
+int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fentry/_raw_write_unlock")
+int BPF_PROG(fentry_raw_write_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irq")
+int BPF_PROG(fentry_raw_write_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irqrestore")
+int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_bh")
+int BPF_PROG(fentry_raw_write_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+/* RT-Mutex acquire/release hooks */
+SEC("fexit/rt_mutex_lock")
+int BPF_PROG(fexit_rt_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_nested")
+int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_interruptible")
+int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_lock_killable")
+int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_trylock")
+int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fentry/rt_mutex_unlock")
+int BPF_PROG(fentry_rt_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_RT);
+}
+
+/* Semaphore acquire/release hooks */
+SEC("fexit/down")
+int BPF_PROG(fexit_down, void *sem)
+{
+ return record_lock_acquired(ctx, 0);
+}
+
+SEC("fexit/down_interruptible")
+int BPF_PROG(fexit_down_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_killable")
+int BPF_PROG(fexit_down_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_timeout")
+int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_trylock")
+int BPF_PROG(fexit_down_trylock, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fentry/up")
+int BPF_PROG(fentry_up, void *sem)
+{
+ return record_lock_released(ctx, 0);
+}
+
+/* Percpu-rwsem write acquire/release hooks */
+SEC("fexit/percpu_down_write")
+int BPF_PROG(fexit_percpu_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
+SEC("fentry/percpu_up_write")
+int BPF_PROG(fentry_percpu_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
+SEC("raw_tp/bpf_test_finish")
+int BPF_PROG(collect_lock_syms)
+{
+ return do_collect_lock_syms(&lock_syms, contig_page_data_addr,
+ node_data_addr, nr_nodes, sizeof_zone);
+}
+
+SEC("iter/kmem_cache")
+int slab_cache_iter(void *ctx)
+{
+ return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id);
+}
+
char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index 2f3317ce17f0..1fcb424b75c0 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist,
/* Will a tracepoint filter work for str or should a BPF filter be used? */
static bool is_possible_tp_filter(const char *str)
{
- return strstr(str, "uid") == NULL;
+ return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL &&
+ strstr(str, "lock_held") == NULL;
}
static int set_filter(struct evsel *evsel, const void *arg)
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH v1 3/5] perf test: Add tests for lock_held() BPF sample filter
2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers
@ 2026-10-17 5:45 ` Ian Rogers
2026-10-17 5:49 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
4 siblings, 1 reply; 11+ messages in thread
From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Namhyung Kim
Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, rcu
Add unit tests in parse-events.c for parsing lock_held(),
lock_held_read(), and lock_held_exclusive() BPF filter expressions
(standalone predicates, negated predicates, and compound group
expressions) and extend record_bpf_filter.sh with lock_held(),
lock_held_read()/lock_held_write(), and ':R'/':W' recording tests.
Example output:
# perf test -v 'Parsing of BPF sample filters'
5: Parse event definition strings :
5.7: Parsing of BPF sample filters : Ok
# perf test -v 'perf record sample filtering (by BPF) tests'
112: perf record sample filtering (by BPF) tests :
Checking BPF-filter privilege
Basic bpf-filter test
Basic bpf-filter test [Success]
Failing bpf-filter test
Failing bpf-filter test [Success]
Group bpf-filter test
Group bpf-filter test [Success]
Multiple bpf-filter test
Multiple bpf-filter test [Success]
Cgroup bpf-filter test
Cgroup bpf-filter test [Success]
Lock-held bpf-filter test
Lock-held bpf-filter test [Success]
112: perf record sample filtering (by BPF) tests : Ok
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/tests/parse-events.c | 108 ++++++++++++++++++++
tools/perf/tests/shell/record_bpf_filter.sh | 99 ++++++++++++++++++
2 files changed, 207 insertions(+)
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 5f6f95c8a5b4..4cf9192762f2 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -8,9 +8,11 @@
#include "debug.h"
#include "pmu.h"
#include "pmus.h"
+#include "bpf-filter.h"
#include "strbuf.h"
#include <dirent.h>
#include <errno.h>
+#include <string.h>
#include "fncache.h"
#include <sys/types.h>
#include <sys/stat.h>
@@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused,
return ret;
}
+#ifdef HAVE_BPF_SKEL
+static void clear_bpf_filter_exprs(struct list_head *expr_head)
+{
+ struct perf_bpf_filter_expr *expr, *tmp;
+
+ list_for_each_entry_safe(expr, tmp, expr_head, list) {
+ list_del(&expr->list);
+ perf_bpf_filter_expr__delete(expr);
+ }
+}
+
+static int do_test_bpf_filters(struct list_head *expr_head)
+{
+ struct perf_bpf_filter_expr *expr, *group;
+ const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100";
+
+ /* Simple lock_held(&cgroup_mutex) */
+ TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)",
+ __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is &cgroup_mutex",
+ expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /* Negated !lock_held(cgroup_mutex) */
+ TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)",
+ __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 0", expr->val == 0);
+ TEST_ASSERT_VAL("lock_name is cgroup_mutex",
+ expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /* Read/shared and write/exclusive predicates */
+ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)",
+ __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is &mmap_lock",
+ expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock"));
+ clear_bpf_filter_exprs(expr_head);
+
+ TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)",
+ __perf_bpf_filter__parse(expr_head,
+ "!lock_held_exclusive(mmap_lock)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 0", expr->val == 0);
+ TEST_ASSERT_VAL("lock_name is mmap_lock",
+ expr->lock_name && !strcmp(expr->lock_name, "mmap_lock"));
+ clear_bpf_filter_exprs(expr_head);
+
+ /*
+ * Comparison and compound group:
+ * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100
+ */
+ TEST_ASSERT_VAL("parse compound lock_held filter",
+ __perf_bpf_filter__parse(expr_head, compound) == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN);
+ TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups));
+ group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("first group member lock_name",
+ group->lock_name && !strcmp(group->lock_name, "rwsem:R"));
+ TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1);
+ group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("second group member lock_name",
+ group->lock_name && !strcmp(group->lock_name, "&dentry"));
+ TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0);
+ expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100);
+
+ return TEST_OK;
+}
+#endif
+
+static int test__bpf_filters(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
+{
+#ifdef HAVE_BPF_SKEL
+ struct list_head expr_head = LIST_HEAD_INIT(expr_head);
+ int ret = do_test_bpf_filters(&expr_head);
+
+ clear_bpf_filter_exprs(&expr_head);
+ return ret;
+#else
+ return TEST_SKIP;
+#endif
+}
+
static struct test_case tests__parse_events[] = {
TEST_CASE_REASON("Test event parsing",
events2,
@@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = {
"no aliases in sysfs"),
TEST_CASE("Parsing of aliased events", pmu_events_alias2),
TEST_CASE("Parsing of terms (event modifiers)", terms2),
+ TEST_CASE("Parsing of BPF sample filters", bpf_filters),
{ .name = NULL, }
};
diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh
index 383574cb3bd3..7ec75d969bd6 100755
--- a/tools/perf/tests/shell/record_bpf_filter.sh
+++ b/tools/perf/tests/shell/record_bpf_filter.sh
@@ -176,6 +176,101 @@ test_bpf_filter_cgroup() {
echo "Cgroup bpf-filter test [Success]"
}
+test_bpf_filter_lock_held() {
+ echo "Lock-held bpf-filter test"
+ lock_sym="cgroup_mutex"
+
+ if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms
+ then
+ lock_sym="slab_mutex"
+ fi
+
+ if ! perf record -e task-clock --filter "lock_held(&${lock_sym})" \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if [ "$(id -u)" != 0 ]
+ then
+ echo "Lock-held bpf-filter test [Skipped permission]"
+ return
+ fi
+ if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms
+ then
+ echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]"
+ return
+ fi
+ echo "Lock-held bpf-filter test [Failed &${lock_sym}]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \
+ -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed mmap_lock]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(rq_lock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed rq_lock]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(spinlock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock type]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock \
+ --filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read/write(&mmap_lock)]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held(&mmap_lock:R) || lock_held(&mmap_lock:W)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed &mmap_lock:R/:W]"
+ err=1
+ return
+ fi
+
+ if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \
+ -o /dev/null true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]"
+ err=1
+ return
+ fi
+
+ if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \
+ -o /dev/null true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]"
+ err=1
+ return
+ fi
+
+ echo "Lock-held bpf-filter test [Success]"
+}
+
test_bpf_filter_priv
if [ $err = 0 ]; then
@@ -198,5 +293,9 @@ if [ $err = 0 ]; then
test_bpf_filter_cgroup
fi
+if [ $err = 0 ]; then
+ test_bpf_filter_lock_held
+fi
+
cleanup
exit $err
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (2 preceding siblings ...)
2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers
@ 2026-10-17 5:45 ` Ian Rogers
2026-10-17 5:56 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
4 siblings, 1 reply; 11+ messages in thread
From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Namhyung Kim
Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, rcu
Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks
Trace RCU, and SRCU locks:
- 'rcu' (supports both read/shared and write/exclusive modes):
- Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)',
'lock_held(rcu_read_lock)'): directly inspects
current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on
CONFIG_PREEMPT_RCU kernels), accounting for the baseline
rcu_read_lock() nesting acquired by the perf overflow handler itself
(bpf_overflow_handler() and do_perf_sw_event()) with zero extra
fentry/fexit tracing hooks.
- Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'):
attaches fentry/fexit hooks to synchronize_rcu(),
synchronize_rcu_expedited(), synchronize_rcu_tasks(), and
rcu_barrier() to track grace-period and barrier waits.
- 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both
read/shared and write/exclusive modes):
- Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches
fexit/__srcu_read_lock and fentry/__srcu_read_unlock.
- Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'):
attaches fentry/fexit on synchronize_srcu(),
synchronize_srcu_expedited(), and srcu_barrier().
- 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports
read/shared mode only): directly inspects
current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels
with zero extra tracing hooks.
Example usage:
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ]
# perf report --stdio --no-children
# Overhead Command Shared Object Symbol
# ........ ....... ................. ........................
#
14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu
|
---__d_lookup_rcu
lookup_fast
walk_component
link_path_walk
path_lookupat
filename_lookup
vfs_statx
__do_sys_newstat
do_syscall_64
entry_SYSCALL_64_after_hwframe
# perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1
# perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 42 +++--
tools/perf/tests/parse-events.c | 13 ++
tools/perf/tests/shell/record_bpf_filter.sh | 20 +++
tools/perf/util/bpf-filter.c | 158 +++++++++++++++++--
tools/perf/util/bpf_skel/lock_data.h | 44 +++++-
tools/perf/util/bpf_skel/sample-filter.h | 12 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 135 +++++++++++++++-
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 +
tools/perf/util/lock-contention.c | 4 +
tools/perf/util/lock-contention.h | 4 +-
10 files changed, 402 insertions(+), 32 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 246ae085744d..c8ccfb0e0ce6 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -219,28 +219,38 @@ OPTIONS
'&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'),
a hexadecimal lock address, a slab cache name prefixed with '&'
(e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known
- lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock
- type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
- 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
- 'semaphore'). Both read/shared and write/exclusive tracking are
- supported for 'rwsem' (including 'mmap_lock') and 'rwlock';
+ lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock', 'rcu',
+ 'rcu-trace'), or a lock type ('mutex', 'spinlock', 'rwsem', 'rwsem:R',
+ 'rwsem:W', 'rwlock', 'rwlock:R', 'rwlock:W', 'rt-mutex',
+ 'percpu-rwsem', 'pcpu-sem:W', 'srcu', 'srcu:R', 'srcu:W', 'rcu:R',
+ 'rcu:W', 'semaphore'). Both read/shared and write/exclusive tracking
+ are supported for 'rwsem' (including 'mmap_lock'), 'rwlock', 'rcu',
+ and 'srcu' (where read/shared mode matches 'rcu_read_lock()' /
+ 'srcu_read_lock()' critical sections and write/exclusive mode matches
+ grace-period waits 'synchronize_rcu*()', 'synchronize_srcu*()',
+ 'rcu_barrier()', and 'srcu_barrier()'); 'rcu-trace'
+ ('rcu_read_lock_trace()') supports read/shared tracking only; and
'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'),
'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()')
support write/exclusive tracking only.
Note that 'lock_held(<lock>)' introduces overhead outside of sample
overflow handling: in addition to checking the lock's owner field on
- each sample where possible (such as for mutexes, rt-mutexes, and
- write-locked rwsems), it attaches BPF fentry/fexit programs to kernel
- lock acquire and release functions to track held locks (such as
- spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a
- BPF map. These BPF hooks run on every kernel lock acquisition and
- release for the attached lock family, even when no perf sample is
- being taken. To minimize this overhead, only the hooks for the
- relevant lock family and mode are attached when known (for example,
- 'lock_held_read(&mmap_lock)' only hooks rwsem read functions,
- 'lock_held_write(&mmap_lock)' only hooks rwsem write functions,
- '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks
+ each sample where possible (such as for mutexes, rt-mutexes,
+ write-locked rwsems, and 'rcu:R' / 'rcu-trace' which inspect
+ 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting'
+ directly with no extra tracing hooks), it attaches BPF fentry/fexit
+ programs to kernel lock acquire and release functions to track held
+ locks (such as spinlocks, rwlocks, reader-locked rwsems, SRCU locks,
+ RCU synchronize waits, and slab/type filters) in a BPF map. These
+ BPF hooks run on every kernel lock acquisition and release for the
+ attached lock family, even when no perf sample is being taken. To
+ minimize this overhead, only the hooks for the relevant lock family
+ and mode are attached when known (for example, 'lock_held_read(rcu)'
+ and 'rcu-trace' attach no extra hooks, 'lock_held_read(&mmap_lock)'
+ only hooks rwsem read functions, 'lock_held_write(&mmap_lock)' only
+ hooks rwsem write functions, '&cgroup_mutex' only hooks mutex
+ functions, 'srcu' only hooks SRCU functions, and 'rq_lock' only hooks
spinlock functions), whereas slab filters ('&<slab>'), raw hex
addresses, or symbols without a recognizable lock-type suffix attach
hooks across all lock families (including high-frequency spinlocks).
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 4cf9192762f2..730466b7ebf6 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head)
expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex"));
clear_bpf_filter_exprs(expr_head);
+ /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */
+ TEST_ASSERT_VAL("parse lock_held(rcu)",
+ __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0);
+ TEST_ASSERT_VAL("not empty", !list_empty(expr_head));
+ expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list);
+ TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD);
+ TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY);
+ TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ);
+ TEST_ASSERT_VAL("val is 1", expr->val == 1);
+ TEST_ASSERT_VAL("lock_name is rcu",
+ expr->lock_name && !strcmp(expr->lock_name, "rcu"));
+ clear_bpf_filter_exprs(expr_head);
+
/* Read/shared and write/exclusive predicates */
TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)",
__perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0);
diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh
index 7ec75d969bd6..39ba628022db 100755
--- a/tools/perf/tests/shell/record_bpf_filter.sh
+++ b/tools/perf/tests/shell/record_bpf_filter.sh
@@ -235,6 +235,26 @@ test_bpf_filter_lock_held() {
return
fi
+ if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ echo "Lock-held bpf-filter test [Failed rcu:W/srcu]"
+ err=1
+ return
+ fi
+
+ if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \
+ -o "${perfdata}" true 2> /dev/null
+ then
+ if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \
+ -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU
+ then
+ echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]"
+ err=1
+ return
+ fi
+ fi
+
if ! perf record -e task-clock \
--filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \
-o "${perfdata}" true 2> /dev/null
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 35d6f7c63868..d4d7f7a50f18 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -190,11 +190,19 @@ enum lock_prog_group {
LOCK_PROG_RTMUTEX = 1U << 6,
LOCK_PROG_SEM = 1U << 7,
LOCK_PROG_PERCPU_RWSEM = 1U << 8,
- LOCK_PROG_ALL = 0x1ffU,
+ LOCK_PROG_SRCU_READ = 1U << 9,
+ LOCK_PROG_SRCU_WRITE = 1U << 10,
+ LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE,
+ LOCK_PROG_RCU_WRITE = 1U << 11,
+ LOCK_PROG_ALL = 0xfffU,
};
static unsigned int lock_flags_to_prog_group(unsigned int flags)
{
+ if (flags & LCB_F_RCU)
+ return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0;
+ if (flags & LCB_F_SRCU)
+ return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE;
if (flags & LCB_F_MUTEX)
return LOCK_PROG_MUTEX;
if (flags & LCB_F_RT)
@@ -308,6 +316,32 @@ static const struct lock_prog_desc lock_progs[] = {
/* Percpu-RWsem */
LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+ /* SRCU */
+ LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe",
+ LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe",
+ LOCK_PROG_SRCU_READ),
+ LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited",
+ LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited",
+ LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE),
+ LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE),
+ /* RCU grace-period synchronize / barrier */
+ LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited",
+ LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited",
+ LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE),
+ LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE),
};
#undef LOCK_FPROG
@@ -616,6 +650,17 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
*needs_lock_syms = true;
return;
}
+ if (!strcmp(base, "rcu")) {
+ if (mode & PBF_LOCK_MODE_WRITE)
+ *prog_groups |= LOCK_PROG_RCU_WRITE;
+ return;
+ }
+ if (!strcmp(base, "rcu_read_lock") ||
+ !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") ||
+ !strcmp(base, "rcu_read_lock_trace")) {
+ /* Inspected directly on task_struct; no tracing hooks needed. */
+ return;
+ }
if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, prog_groups))
return;
@@ -646,6 +691,11 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
*prog_groups |= LOCK_PROG_RWLOCK_READ;
if (mode & PBF_LOCK_MODE_WRITE)
*prog_groups |= LOCK_PROG_RWLOCK_WRITE;
+ } else if (strtailcmp(base, "srcu") == 0) {
+ if (mode & PBF_LOCK_MODE_READ)
+ *prog_groups |= LOCK_PROG_SRCU_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ *prog_groups |= LOCK_PROG_SRCU_WRITE;
} else {
*prog_groups |= LOCK_PROG_ALL;
}
@@ -729,16 +779,54 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
return false;
}
+static u64 rcu_min_depth(struct evsel *evsel)
+{
+ if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE &&
+ evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK &&
+ evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK)
+ return 2;
+ return 1;
+}
+
+static bool btf_has_task_field(struct btf *btf, const char *field_name)
+{
+ const struct btf_type *t;
+ const struct btf_member *m;
+ s32 id;
+ u16 i, vlen;
+
+ if (!btf)
+ return false;
+
+ id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT);
+ if (id < 0)
+ return false;
+
+ t = btf__type_by_id(btf, id);
+ if (!t)
+ return false;
+
+ m = btf_members(t);
+ vlen = btf_vlen(t);
+ for (i = 0; i < vlen; i++, m++) {
+ const char *name = btf__name_by_offset(btf, m->name_off);
+
+ if (name && !strcmp(name, field_name))
+ return true;
+ }
+ return false;
+}
+
/*
* Resolve a `lock_held(<name>)` filter expression into `entry->part`
* (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate
* the corresponding BPF filter maps and BSS control flags.
*
* Resolution order:
- * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`,
- * `rq_lock`, `zone_lock` (with or without leading '&').
+ * 1. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`,
+ * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` (with or without leading '&').
* 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`,
- * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`,
+ * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, `srcu`,
* `semaphore`, etc.
* 3. Hexadecimal lock address (when no leading '&').
* 4. Slab cache name (when prefixed with '&', e.g. `&dentry`).
@@ -746,8 +834,8 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name,
* e.g. `&cgroup_mutex` or `cgroup_mutex`).
* 6. Slab cache name without leading '&' fallback.
*/
-static int resolve_lock_expr(struct sample_filter_bpf *skel,
- struct perf_bpf_filter_expr *expr,
+static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct btf *btf, struct perf_bpf_filter_expr *expr,
struct perf_bpf_filter_entry *entry)
{
const char *name = expr->lock_name;
@@ -811,6 +899,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel,
return 0;
}
+ if (!strcmp(base, "rcu_read_lock")) {
+ if (mode == PBF_LOCK_MODE_WRITE) {
+ pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
+ return -EINVAL;
+ }
+ if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
+ pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ;
+ entry->target = rcu_min_depth(evsel);
+ return 0;
+ }
+
+ if (!strcmp(base, "rcu")) {
+ if ((mode & PBF_LOCK_MODE_READ) && btf &&
+ !btf_has_task_field(btf, "rcu_read_lock_nesting")) {
+ pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU | mode;
+ entry->target = rcu_min_depth(evsel);
+ if ((mode & PBF_LOCK_MODE_WRITE) && skel)
+ skel->bss->has_rcu_lock = 1;
+ return 0;
+ }
+
+ if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") ||
+ !strcmp(base, "rcu_read_lock_trace")) {
+ if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) {
+ pr_err("Error: Lock '%s' only supports read/shared mode\n", name);
+ return -EINVAL;
+ }
+ if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) {
+ pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n");
+ return -EOPNOTSUPP;
+ }
+ entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ;
+ entry->target = 0;
+ return 0;
+ }
+
if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
entry->part = PBF_LOCK_TYPE | mode;
entry->target = encoded_pair;
@@ -875,7 +1005,7 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel,
}
static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
- struct perf_bpf_filter_entry *entry)
+ struct btf *btf, struct perf_bpf_filter_entry *entry)
{
int i = 0, ret;
struct perf_bpf_filter_expr *expr;
@@ -893,7 +1023,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske
entry[i].value = expr->val;
entry[i].target = 0;
if (expr->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(skel, expr, &entry[i]);
+ ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]);
if (ret < 0)
return ret;
}
@@ -912,7 +1042,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske
entry[i].value = group->val;
entry[i].target = 0;
if (group->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(skel, group, &entry[i]);
+ ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]);
if (ret < 0)
return ret;
}
@@ -1185,7 +1315,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- ret = get_filter_entries(evsel, NULL, entry);
+ ret = get_filter_entries(evsel, NULL, NULL, entry);
if (ret < 0) {
pr_err("Failed to process filter entries\n");
goto err;
@@ -1247,6 +1377,12 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES);
init_numa_data(skel, btf);
}
+ if (prog_groups & LOCK_PROG_RCU_WRITE) {
+ u64 rcu_addr = 1ULL;
+
+ find_kernel_lock_sym("rcu_state", &rcu_addr);
+ skel->rodata->rcu_state_addr = rcu_addr;
+ }
ret = configure_lock_progs(skel, btf, prog_groups, needs_lock_syms,
needs_slab_iter);
if (ret < 0)
@@ -1264,7 +1400,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
if (has_lock_filter && needs_slab_iter)
run_slab_cache_iter(skel);
- ret = get_filter_entries(evsel, skel, entry);
+ ret = get_filter_entries(evsel, skel, btf, entry);
if (ret < 0) {
pr_err("Failed to process filter entries\n");
goto err;
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
index fed445c9d8b4..e32ba230d5f5 100644
--- a/tools/perf/util/bpf_skel/lock_data.h
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -39,6 +39,7 @@ struct contention_task_data {
*/
#define LCD_F_MMAP_LOCK (1U << 31)
#define LCD_F_SIGHAND_LOCK (1U << 30)
+#define LCD_F_RCU_LOCK (1U << 29)
#define LCB_F_SLAB_ID_SHIFT 16
#define LCB_F_SLAB_ID_START (1U << 16)
@@ -52,9 +53,11 @@ struct contention_task_data {
#define LCB_F_RT (1U << 3)
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+#define LCB_F_SRCU (1U << 6)
+#define LCB_F_RCU (1U << 7)
-#define LCB_F_TYPE_MAX (1U << 7)
-#define LCB_F_TYPE_MASK 0x0000007FU
+#define LCB_F_TYPE_MAX (1U << 8)
+#define LCB_F_TYPE_MASK 0x000000FFU
#define SLAB_NAME_MAX 28
@@ -233,6 +236,40 @@ static __always_inline __u64 get_current_siglock(__u64 curr)
return sh + bpf_core_field_offset(sighand->siglock);
}
+/*
+ * Check if `curr` is inside a preemptible RCU read-side critical section
+ * (`rcu_read_lock()`).
+ *
+ * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires
+ * `rcu_read_lock()` before invoking `perf_sample_filter` (and
+ * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for
+ * non-clock software events), `min_depth` is the baseline nesting count
+ * contributed by the perf event overflow handler itself (1 or 2).
+ */
+static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth)
+{
+ struct task_struct *task = (void *)curr;
+
+ if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting))
+ return false;
+
+ return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth;
+}
+
+/*
+ * Check if `curr` is inside a Tasks Trace RCU read-side critical section
+ * (`rcu_read_lock_trace()`).
+ */
+static __always_inline bool is_rcu_trace_lock_held(__u64 curr)
+{
+ struct task_struct *task = (void *)curr;
+
+ if (!curr || !bpf_core_field_exists(task->trc_reader_nesting))
+ return false;
+
+ return BPF_CORE_READ(task, trc_reader_nesting) > 0;
+}
+
/*
* Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar.
*
@@ -265,6 +302,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags)
if (get_current_siglock(curr) == lock)
return LCD_F_SIGHAND_LOCK;
break;
+ case LCB_F_RCU:
+ case LCB_F_RCU | LCB_F_WRITE:
+ return LCD_F_RCU_LOCK;
default:
break;
}
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index cd711d1d04b6..37c108608bbc 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -99,6 +99,18 @@ enum perf_bpf_lock_target_kind {
* low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
*/
PBF_LOCK_TYPE,
+ /*
+ * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`).
+ * Inspected directly via `current->rcu_read_lock_nesting > entry.target`,
+ * where `entry.target` is the baseline RCU nesting depth added by the
+ * kernel perf overflow path.
+ */
+ PBF_LOCK_RCU,
+ /*
+ * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`).
+ * Inspected directly via `current->trc_reader_nesting > 0`.
+ */
+ PBF_LOCK_RCU_TRACE,
};
/*
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index fbd2e51815d7..22d9b01e5851 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -123,6 +123,7 @@ volatile const int use_idx_hash;
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
+const volatile __u64 rcu_state_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
@@ -136,6 +137,7 @@ int has_lock_class;
int has_lock_type;
int has_mmap_lock;
int has_siglock;
+int has_rcu_lock;
__u8 lock_type_filter[LCB_F_TYPE_MAX];
void *bpf_cast_to_kern_ctx(void *) __ksym;
@@ -269,6 +271,20 @@ static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry
case PBF_LOCK_TYPE:
return is_lock_type_in_map(entry->target, mode, pid, cpu_id);
+ case PBF_LOCK_RCU:
+ if ((mode & PBF_LOCK_MODE_READ) &&
+ is_rcu_lock_held(curr, (int)entry->target))
+ return 1;
+ if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr &&
+ is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id))
+ return 1;
+ return 0;
+
+ case PBF_LOCK_RCU_TRACE:
+ if ((mode & PBF_LOCK_MODE_READ) && is_rcu_trace_lock_held(curr))
+ return 1;
+ return 0;
+
default:
return 0;
}
@@ -600,11 +616,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla
if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
matched = true;
- if (has_mmap_lock || has_siglock) {
+ if (has_mmap_lock || has_siglock || has_rcu_lock) {
__u32 ltype = check_lock_type(lock_addr, flags);
if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) ||
- (has_siglock && ltype == LCD_F_SIGHAND_LOCK))
+ (has_siglock && ltype == LCD_F_SIGHAND_LOCK) ||
+ (has_rcu_lock && ltype == LCD_F_RCU_LOCK))
matched = true;
}
@@ -1202,6 +1219,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem)
return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
}
+/* SRCU acquire/release and synchronize hooks */
+SEC("fexit/__srcu_read_lock")
+int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/__srcu_read_unlock")
+int BPF_PROG(fentry_srcu_read_unlock, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fexit/__srcu_read_lock_nmisafe")
+int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/__srcu_read_unlock_nmisafe")
+int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ);
+}
+
+SEC("fentry/synchronize_srcu")
+int BPF_PROG(fentry_synchronize_srcu, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu")
+int BPF_PROG(fexit_synchronize_srcu, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_srcu_expedited")
+int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu_expedited")
+int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fentry/srcu_barrier")
+int BPF_PROG(fentry_srcu_barrier, void *ssp)
+{
+ return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/srcu_barrier")
+int BPF_PROG(fexit_srcu_barrier, void *ssp)
+{
+ return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE);
+}
+
+/* RCU grace-period synchronize / barrier hooks */
+SEC("fentry/synchronize_rcu")
+int BPF_PROG(fentry_synchronize_rcu)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu")
+int BPF_PROG(fexit_synchronize_rcu)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_rcu_expedited")
+int BPF_PROG(fentry_synchronize_rcu_expedited)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu_expedited")
+int BPF_PROG(fexit_synchronize_rcu_expedited)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/synchronize_rcu_tasks")
+int BPF_PROG(fentry_synchronize_rcu_tasks)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/synchronize_rcu_tasks")
+int BPF_PROG(fexit_synchronize_rcu_tasks)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fentry/rcu_barrier")
+int BPF_PROG(fentry_rcu_barrier)
+{
+ return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE,
+ get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE));
+}
+
+SEC("fexit/rcu_barrier")
+int BPF_PROG(fexit_rcu_barrier)
+{
+ return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
SEC("raw_tp/bpf_test_finish")
int BPF_PROG(collect_lock_syms)
{
diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
index b75ddc6846bb..1fb91a829d54 100644
--- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
+++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h
@@ -119,6 +119,8 @@ struct task_struct {
struct signal_struct *signal;
struct sighand_struct *sighand;
struct css_set *cgroups;
+ int rcu_read_lock_nesting;
+ int trc_reader_nesting;
} __attribute__((preserve_access_index));
struct trace_entry {
diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c
index a1bb78d6e29c..b45b075f3346 100644
--- a/tools/perf/util/lock-contention.c
+++ b/tools/perf/util/lock-contention.c
@@ -38,6 +38,10 @@ const struct perf_lock_type lock_type_table[] = {
{ LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" },
/* alias for optimistic spinning only */
{ LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" },
+ { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" },
+ { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" },
+ { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" },
+ { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" },
};
int parse_call_stack(const struct option *opt __maybe_unused, const char *str,
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index 68e9bd77e8a9..88b2c11244ad 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -128,6 +128,8 @@ struct thread_stat {
#define LCB_F_RT (1U << 3)
#define LCB_F_PERCPU (1U << 4)
#define LCB_F_MUTEX (1U << 5)
+#define LCB_F_SRCU (1U << 6)
+#define LCB_F_RCU (1U << 7)
struct perf_lock_type {
unsigned int flags;
@@ -140,7 +142,7 @@ struct perf_lock_type {
const char *lock_name;
};
-extern const struct perf_lock_type lock_type_table[14];
+extern const struct perf_lock_type lock_type_table[18];
struct evlist;
struct machine;
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU and SRCU
2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
` (3 preceding siblings ...)
2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
@ 2026-10-17 5:45 ` Ian Rogers
2026-10-17 6:08 ` sashiko-bot
4 siblings, 1 reply; 11+ messages in thread
From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Namhyung Kim
Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, rcu
Extend 'perf lock contention --use-bpf' to track RCU and SRCU
grace-period and barrier waits:
- Attach fentry/fexit programs to synchronize_rcu(),
synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(),
synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier()
when available in BTF.
- Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu',
'-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct
symbol names).
Example usage:
# perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1
total wait avg wait type caller
35.12 ms 8.78 ms rcu:W cleanup_net+0x284
7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114
# perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1
total wait avg wait address symbol
35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu)
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/tests/shell/lock_contention.sh | 14 ++
tools/perf/util/bpf_lock_contention.c | 167 ++++++++++++++--
.../perf/util/bpf_skel/lock_contention.bpf.c | 183 ++++++++++++++----
4 files changed, 322 insertions(+), 49 deletions(-)
diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt
index c17b3e318169..bfeb98c976e3 100644
--- a/tools/perf/Documentation/perf-lock.txt
+++ b/tools/perf/Documentation/perf-lock.txt
@@ -188,11 +188,14 @@ CONTENTION OPTIONS
Show lock contention only for given lock types (comma separated list).
Available values are:
semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W,
- rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem,
- pcpu-sem:R, pcpu-sem:W, mutex
+ rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem,
+ pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W
Note that RW-variant of locks have :R and :W suffix. Names without the
suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W.
+ For RCU and SRCU with --use-bpf, contention tracks grace-period and
+ barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(),
+ srcu_barrier()).
-L::
--lock-filter=<value>::
diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh
index 5df9a0cd48a3..c6d37c910d4b 100755
--- a/tools/perf/tests/shell/lock_contention.sh
+++ b/tools/perf/tests/shell/lock_contention.sh
@@ -181,6 +181,13 @@ test_type_filter()
err=1
exit
fi
+
+ perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result}
+ if [ "$(grep -c -v rcu "${result}")" != "0" ]; then
+ echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")"
+ err=1
+ exit
+ fi
}
test_lock_filter()
@@ -224,6 +231,13 @@ test_lock_filter()
err=1
exit
fi
+
+ perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result}
+ if [ "$(grep -c -v rcu "${result}")" != "0" ]; then
+ echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")"
+ err=1
+ exit
+ fi
}
test_stack_filter()
diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c
index c20bd075664e..fa7df9187b08 100644
--- a/tools/perf/util/bpf_lock_contention.c
+++ b/tools/perf/util/bpf_lock_contention.c
@@ -16,6 +16,7 @@
#include <bpf/bpf.h>
#include <bpf/btf.h>
#include <inttypes.h>
+#include <unistd.h>
#include "bpf_skel/lock_contention.skel.h"
#include "bpf_skel/lock_data.h"
@@ -180,6 +181,129 @@ static void init_numa_data(struct lock_contention *con)
free(buf);
}
+static bool has_bpf_fentry_support(struct btf *btf)
+{
+ /*
+ * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and
+ * bpf_get_func_arg() helper availability (5.17+) used by
+ * get_lock_addr() in the SRCU fentry/fexit hooks.
+ */
+ struct bpf_insn insns[] = {
+ { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 },
+ { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3,
+ .src_reg = BPF_REG_10 },
+ { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 },
+ { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg },
+ { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 },
+ { .code = BPF_JMP | BPF_EXIT },
+ };
+ LIBBPF_OPTS(bpf_prog_load_opts, opts,
+ .expected_attach_type = BPF_TRACE_FENTRY,
+ );
+ s32 btf_id;
+ int prog_fd, link_fd;
+
+ if (!btf)
+ return false;
+
+ btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC);
+ if (btf_id <= 0)
+ return false;
+
+ opts.attach_btf_id = btf_id;
+ prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL",
+ insns, ARRAY_SIZE(insns), &opts);
+ if (prog_fd < 0)
+ return false;
+
+ link_fd = bpf_raw_tracepoint_open(NULL, prog_fd);
+ if (link_fd >= 0)
+ close(link_fd);
+ close(prog_fd);
+ return link_fd >= 0;
+}
+
+static void configure_rcu_contention_progs(struct lock_contention *con,
+ bool has_rcu_lock)
+{
+ static const struct {
+ size_t prog_offset;
+ const char *func;
+ bool is_srcu;
+ } rcu_progs[] = {
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu),
+ "synchronize_rcu", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu),
+ "synchronize_rcu", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited),
+ "synchronize_rcu_expedited", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited),
+ "synchronize_rcu_expedited", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks),
+ "synchronize_rcu_tasks", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks),
+ "synchronize_rcu_tasks", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier),
+ "rcu_barrier", false },
+ { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier),
+ "rcu_barrier", false },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu),
+ "synchronize_srcu", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu),
+ "synchronize_srcu", true },
+ { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited),
+ "synchronize_srcu_expedited", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited),
+ "synchronize_srcu_expedited", true },
+ { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier),
+ "srcu_barrier", true },
+ { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier),
+ "srcu_barrier", true },
+ };
+ struct symbol *sym;
+ struct map *kmap;
+ bool need_rcu = true, need_srcu = true;
+ bool fentry_ok;
+
+ sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap);
+ if (sym) {
+ skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start);
+ map__put(kmap);
+ } else {
+ skel->rodata->rcu_state_addr = 1ULL;
+ }
+
+ if (con->filters->nr_types) {
+ need_rcu = false;
+ need_srcu = false;
+ for (int i = 0; i < con->filters->nr_types; i++) {
+ if (con->filters->types[i] & LCB_F_RCU)
+ need_rcu = true;
+ if (con->filters->types[i] & LCB_F_SRCU)
+ need_srcu = true;
+ }
+ }
+
+ if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) {
+ if (!has_rcu_lock && !con->filters->nr_addrs)
+ need_rcu = false;
+ if (!con->filters->nr_addrs && !con->filters->nr_slabs)
+ need_srcu = false;
+ }
+
+ fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf);
+
+ for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) {
+ struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset;
+ bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu;
+ bool enable = fentry_ok && want &&
+ btf__find_by_name_kind(con->btf, rcu_progs[i].func,
+ BTF_KIND_FUNC) > 0;
+
+ bpf_program__set_autoload(*progp, enable);
+ }
+}
+
int lock_contention_prepare(struct lock_contention *con)
{
int i, fd;
@@ -187,6 +311,7 @@ int lock_contention_prepare(struct lock_contention *con)
struct evlist *evlist = con->evlist;
struct target *target = con->target;
bool has_mmap_lock = false;
+ bool has_rcu_lock = false;
/* make sure it loads the kernel map before lookup */
map__load(machine__kernel_map(con->machine));
@@ -200,6 +325,7 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64));
bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries);
bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries);
+ bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries);
if (con->aggr_mode == LOCK_AGGR_TASK)
bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries);
@@ -249,6 +375,10 @@ int lock_contention_prepare(struct lock_contention *con)
has_mmap_lock = true;
continue;
}
+ if (!strcmp(con->filters->syms[i], "rcu")) {
+ has_rcu_lock = true;
+ continue;
+ }
sym = machine__find_kernel_symbol_by_name(con->machine,
con->filters->syms[i],
@@ -269,7 +399,7 @@ int lock_contention_prepare(struct lock_contention *con)
addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start);
con->filters->addrs = addrs;
}
- naddrs = con->filters->nr_addrs ?: has_mmap_lock;
+ naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock);
skel->rodata->has_addr = 1;
}
@@ -305,6 +435,7 @@ int lock_contention_prepare(struct lock_contention *con)
skel->rodata->needs_callstack = con->save_callstack;
skel->rodata->lock_owner = con->owner;
skel->rodata->has_mmap_lock = has_mmap_lock;
+ skel->rodata->has_rcu_lock = has_rcu_lock;
if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) {
if (cgroup_is_v2("perf_event"))
@@ -321,6 +452,7 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map__set_max_entries(skel->maps.slab_filter, nslabs);
init_numa_data(con);
+ configure_rcu_contention_progs(con, has_rcu_lock);
if (lock_contention_bpf__load(skel) < 0) {
pr_err("Failed to load lock-contention BPF skeleton\n");
@@ -503,23 +635,30 @@ static void account_end_timestamp(struct lock_contention *con)
enum lock_aggr_mode aggr_mode = con->aggr_mode;
struct tstamp_data ts_data, *cpu_data;
- /* Iterate per-task tstamp map (key = TID) */
- ts_fd = bpf_map__fd(skel->maps.tstamp);
+ /* Iterate per-task tstamp maps (key = TID) */
+ int ts_fds[] = {
+ bpf_map__fd(skel->maps.tstamp),
+ bpf_map__fd(skel->maps.tstamp_rcu),
+ };
+
stat_fd = bpf_map__fd(skel->maps.lock_stat);
- prev_key = NULL;
- while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) {
- if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) {
- int pid = key;
+ for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) {
+ ts_fd = ts_fds[i];
+ prev_key = NULL;
+ while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) {
+ if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) {
+ int pid = key;
- if (aggr_mode == LOCK_AGGR_TASK && con->owner)
- pid = ts_data.flags;
+ if (aggr_mode == LOCK_AGGR_TASK && con->owner)
+ pid = ts_data.flags;
- update_lock_stat(stat_fd, pid, end_ts, aggr_mode,
- &ts_data);
- }
+ update_lock_stat(stat_fd, pid, end_ts, aggr_mode,
+ &ts_data);
+ }
- prev_key = &key;
+ prev_key = &key;
+ }
}
/* Now it'll check per-cpu tstamp map which doesn't have TID. */
@@ -604,6 +743,8 @@ static const char *lock_contention_get_name(struct lock_contention *con,
return "mmap_lock";
if (flags & LCD_F_SIGHAND_LOCK)
return "siglock";
+ if (flags & LCD_F_RCU_LOCK)
+ return "rcu";
/* global locks with symbols */
sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap);
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index 0f2118006005..257e3402e815 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -59,6 +59,18 @@ struct {
__uint(max_entries, MAX_ENTRIES);
} tstamp SEC(".maps");
+/*
+ * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits
+ * so they do not collide with inner sleeping lock contentions (such as
+ * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, int);
+ __type(value, struct tstamp_data);
+ __uint(max_entries, MAX_ENTRIES);
+} tstamp_rcu SEC(".maps");
+
/* maintain per-CPU timestamp at the beginning of contention */
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
@@ -160,6 +172,7 @@ const volatile int has_addr;
const volatile int has_cgroup;
const volatile int has_slab;
const volatile int has_mmap_lock;
+const volatile int has_rcu_lock;
const volatile int needs_callstack;
const volatile int stack_skip;
const volatile int lock_owner;
@@ -172,6 +185,7 @@ const volatile int aggr_mode;
const volatile __u64 contig_page_data_addr;
const volatile __u64 node_data_addr;
+const volatile __u64 rcu_state_addr;
const volatile int nr_nodes;
const volatile int sizeof_zone;
@@ -218,7 +232,7 @@ static inline __u64 get_current_cgroup_id(void)
return BPF_CORE_READ(cgrp, kn, id);
}
-static inline int can_record(u64 *ctx)
+static __always_inline int can_record(__u64 lock, __u32 flags)
{
bool is_addr_ok = false;
@@ -242,7 +256,6 @@ static inline int can_record(u64 *ctx)
if (has_type) {
__u8 *ok;
- __u32 flags = (__u32)ctx[1];
ok = bpf_map_lookup_elem(&type_filter, &flags);
if (!ok)
@@ -251,10 +264,9 @@ static inline int can_record(u64 *ctx)
if (has_addr) {
__u8 *ok;
- __u64 addr = ctx[0];
- ok = bpf_map_lookup_elem(&addr_filter, &addr);
- if (!ok && !has_slab && !has_mmap_lock)
+ ok = bpf_map_lookup_elem(&addr_filter, &lock);
+ if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock)
return 0;
is_addr_ok = !!ok;
@@ -272,26 +284,27 @@ static inline int can_record(u64 *ctx)
if (is_addr_ok)
return 1;
- /* slab and mmap_lock are part of the addr_filter */
+ /* slab, mmap_lock, and rcu_lock are part of the addr_filter */
if (has_slab && bpf_get_kmem_cache) {
__u8 *ok;
- __u64 addr = ctx[0];
long kmem_cache_addr;
- kmem_cache_addr = (long)bpf_get_kmem_cache(addr);
+ kmem_cache_addr = (long)bpf_get_kmem_cache(lock);
ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr);
if (ok)
return 1;
- else if (!has_mmap_lock)
+ else if (!has_mmap_lock && !has_rcu_lock)
return 0;
}
- if (has_mmap_lock) {
- __u64 lock = ctx[0];
- __u32 flag = ctx[1];
+ if (has_mmap_lock || has_rcu_lock) {
+ __u32 ltype = check_lock_type(lock, flags);
- if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK)
- return 0;
+ if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK)
+ return 1;
+ if (has_rcu_lock && ltype == LCD_F_RCU_LOCK)
+ return 1;
+ return 0;
}
return 1;
@@ -351,6 +364,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
{
__u32 pid;
struct tstamp_data *pelem;
+ void *map;
/* Use per-cpu array map for spinlock and rwlock */
if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) {
@@ -363,8 +377,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
return pelem;
}
+ map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp;
pid = bpf_get_current_pid_tgid();
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ pelem = bpf_map_lookup_elem(map, &pid);
/* Do not update the element for nested locks */
if (pelem && pelem->lock)
return NULL;
@@ -372,12 +387,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags)
if (pelem == NULL) {
struct tstamp_data zero = {};
- if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) {
+ if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) {
__sync_fetch_and_add(&task_fail, 1);
return NULL;
}
- pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ pelem = bpf_map_lookup_elem(map, &pid);
if (pelem == NULL) {
__sync_fetch_and_add(&task_fail, 1);
return NULL;
@@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags)
}
}
-SEC("tp_btf/contention_begin")
-int contention_begin(u64 *ctx)
+static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags)
{
struct tstamp_data *pelem;
- if (!enabled || !can_record(ctx))
+ if (!enabled || !can_record(lock, flags))
return 0;
- pelem = get_tstamp_elem(ctx[1]);
+ pelem = get_tstamp_elem(flags);
if (pelem == NULL)
return 0;
pelem->timestamp = bpf_ktime_get_ns();
- pelem->lock = (__u64)ctx[0];
- pelem->flags = (__u32)ctx[1];
+ pelem->lock = lock;
+ pelem->flags = flags;
if (aggr_mode == LOCK_AGGR_CGROUP)
pelem->cgroup_id = get_current_cgroup_id();
@@ -588,8 +602,7 @@ int contention_begin(u64 *ctx)
return 0;
}
-SEC("tp_btf/contention_end")
-int contention_end(u64 *ctx)
+static __always_inline int do_contention_end(__u64 lock, __s32 ret)
{
__u32 pid = 0, idx = 0;
struct tstamp_data *pelem;
@@ -597,7 +610,7 @@ int contention_end(u64 *ctx)
struct contention_data *data;
__u64 timestamp;
__u64 duration;
- bool need_delete = false;
+ void *ts_map = NULL;
if (!enabled)
return 0;
@@ -613,14 +626,19 @@ int contention_end(u64 *ctx)
*/
pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx);
if (pelem && pelem->lock) {
- if (pelem->lock != ctx[0])
+ if (pelem->lock != lock)
return 0;
} else {
pid = bpf_get_current_pid_tgid();
pelem = bpf_map_lookup_elem(&tstamp, &pid);
- if (!pelem || pelem->lock != ctx[0])
- return 0;
- need_delete = true;
+ if (pelem && pelem->lock == lock) {
+ ts_map = &tstamp;
+ } else {
+ pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid);
+ if (!pelem || pelem->lock != lock)
+ return 0;
+ ts_map = &tstamp_rcu;
+ }
}
timestamp = bpf_ktime_get_ns();
@@ -648,7 +666,6 @@ int contention_end(u64 *ctx)
*/
else {
u32 i = 0;
- s32 ret = (s32)ctx[1];
u64 *buf;
otdata->timestamp = timestamp;
@@ -713,7 +730,7 @@ int contention_end(u64 *ctx)
if (lock_owner)
key.pid = pelem->flags;
else {
- if (!need_delete)
+ if (!ts_map)
pid = bpf_get_current_pid_tgid();
key.pid = pid;
}
@@ -795,11 +812,109 @@ int contention_end(u64 *ctx)
check_lock_delay(pelem->lock);
pelem->lock = 0;
- if (need_delete)
- bpf_map_delete_elem(&tstamp, &pid);
+ if (ts_map)
+ bpf_map_delete_elem(ts_map, &pid);
return 0;
}
+SEC("tp_btf/contention_begin")
+int contention_begin(u64 *ctx)
+{
+ return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]);
+}
+
+SEC("tp_btf/contention_end")
+int contention_end(u64 *ctx)
+{
+ return do_contention_end((__u64)ctx[0], (__s32)ctx[1]);
+}
+
+/* RCU grace-period synchronize / barrier contention hooks */
+SEC("fentry/synchronize_rcu")
+int BPF_PROG(fentry_synchronize_rcu)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu")
+int BPF_PROG(fexit_synchronize_rcu)
+{
+ return do_contention_end(rcu_state_addr, 0);
+}
+
+SEC("fentry/synchronize_rcu_expedited")
+int BPF_PROG(fentry_synchronize_rcu_expedited)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu_expedited")
+int BPF_PROG(fexit_synchronize_rcu_expedited)
+{
+ return do_contention_end(rcu_state_addr, 0);
+}
+
+SEC("fentry/synchronize_rcu_tasks")
+int BPF_PROG(fentry_synchronize_rcu_tasks)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_rcu_tasks")
+int BPF_PROG(fexit_synchronize_rcu_tasks)
+{
+ return do_contention_end(rcu_state_addr, 0);
+}
+
+SEC("fentry/rcu_barrier")
+int BPF_PROG(fentry_rcu_barrier)
+{
+ return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
+}
+
+SEC("fexit/rcu_barrier")
+int BPF_PROG(fexit_rcu_barrier)
+{
+ return do_contention_end(rcu_state_addr, 0);
+}
+
+/* SRCU grace-period synchronize / barrier contention hooks */
+SEC("fentry/synchronize_srcu")
+int BPF_PROG(fentry_synchronize_srcu, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu")
+int BPF_PROG(fexit_synchronize_srcu, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0);
+}
+
+SEC("fentry/synchronize_srcu_expedited")
+int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/synchronize_srcu_expedited")
+int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0);
+}
+
+SEC("fentry/srcu_barrier")
+int BPF_PROG(fentry_srcu_barrier, void *ssp)
+{
+ return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE);
+}
+
+SEC("fexit/srcu_barrier")
+int BPF_PROG(fexit_srcu_barrier, void *ssp)
+{
+ return do_contention_end(get_lock_addr(ctx), 0);
+}
+
SEC("raw_tp/bpf_test_finish")
int BPF_PROG(end_timestamp)
{
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH v1 3/5] perf test: Add tests for lock_held() BPF sample filter
2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers
@ 2026-10-17 5:49 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-10-17 5:49 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Add unit tests in parse-events.c for parsing lock_held(),
> lock_held_read(), and lock_held_exclusive() BPF filter expressions
> (standalone predicates, negated predicates, and compound group
> expressions) and extend record_bpf_filter.sh with lock_held(),
> lock_held_read()/lock_held_write(), and ':R'/':W' recording tests.
>
> Example output:
>
> # perf test -v 'Parsing of BPF sample filters'
> 5: Parse event definition strings :
> 5.7: Parsing of BPF sample filters : Ok
>
> # perf test -v 'perf record sample filtering (by BPF) tests'
> 112: perf record sample filtering (by BPF) tests :
> Checking BPF-filter privilege
> Basic bpf-filter test
> Basic bpf-filter test [Success]
> Failing bpf-filter test
> Failing bpf-filter test [Success]
> Group bpf-filter test
> Group bpf-filter test [Success]
> Multiple bpf-filter test
> Multiple bpf-filter test [Success]
> Cgroup bpf-filter test
> Cgroup bpf-filter test [Success]
> Lock-held bpf-filter test
> Lock-held bpf-filter test [Success]
> 112: perf record sample filtering (by BPF) tests : Ok
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=3
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h
2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
@ 2026-10-17 5:52 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-10-17 5:52 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers
> (get_lock_owner, get_current_mmap_lock, get_current_siglock,
> get_lock_addr, check_lock_type, do_collect_zone_lock,
> do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into
> tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF
> sample filtering.
>
> Also add rw_semaphore.count and struct rt_mutex_base to the minimal
> vmlinux.h and extend get_lock_owner() to support rt_mutex and check
> writer-ownership and handoff flags.
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=1
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU
2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
@ 2026-10-17 5:56 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-10-17 5:56 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
> Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks
> Trace RCU, and SRCU locks:
>
> - 'rcu' (supports both read/shared and write/exclusive modes):
> - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)',
> 'lock_held(rcu_read_lock)'): directly inspects
> current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on
> CONFIG_PREEMPT_RCU kernels), accounting for the baseline
> rcu_read_lock() nesting acquired by the perf overflow handler itself
> (bpf_overflow_handler() and do_perf_sw_event()) with zero extra
> fentry/fexit tracing hooks.
> - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'):
> attaches fentry/fexit hooks to synchronize_rcu(),
> synchronize_rcu_expedited(), synchronize_rcu_tasks(), and
> rcu_barrier() to track grace-period and barrier waits.
> [ ... ]
>
> # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1
> # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1
>
> Signed-off-by: Ian Rogers <irogers@google.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=4
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU and SRCU
2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
@ 2026-10-17 6:08 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-10-17 6:08 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf lock contention: wrong caller stack skip in do_contention_begin() for fentry
--
commit 9ca74d6f51b3408e9a151b639dd644cd10562df1
Author: Ian Rogers <irogers@google.com>
perf lock contention: Extend BPF lock contention to RCU and SRCU
This commit extends 'perf lock contention --use-bpf' to track RCU and SRCU
grace-period and barrier waits using fentry/fexit BPF programs and adds
lock type and name filtering support for rcu and srcu.
> diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
> index 0f2118006005a..257e3402e8151 100644
> --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
> +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
[ ... ]
> @@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags)
> }
> }
>
> -SEC("tp_btf/contention_begin")
> -int contention_begin(u64 *ctx)
> +static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags)
> {
> struct tstamp_data *pelem;
>
> - if (!enabled || !can_record(ctx))
> + if (!enabled || !can_record(lock, flags))
> return 0;
>
> - pelem = get_tstamp_elem(ctx[1]);
> + pelem = get_tstamp_elem(flags);
> if (pelem == NULL)
> return 0;
>
> pelem->timestamp = bpf_ktime_get_ns();
> - pelem->lock = (__u64)ctx[0];
> - pelem->flags = (__u32)ctx[1];
> + pelem->lock = lock;
> + pelem->flags = flags;
> if (aggr_mode == LOCK_AGGR_CGROUP)
> pelem->cgroup_id = get_current_cgroup_id();
>
[ ... ]
> @@ -795,11 +812,109 @@ int contention_end(u64 *ctx)
> check_lock_delay(pelem->lock);
>
> pelem->lock = 0;
> - if (need_delete)
> - bpf_map_delete_elem(&tstamp, &pid);
> + if (ts_map)
> + bpf_map_delete_elem(ts_map, &pid);
> return 0;
> }
>
> +SEC("tp_btf/contention_begin")
> +int contention_begin(u64 *ctx)
> +{
> + return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]);
> +}
> +
> +SEC("tp_btf/contention_end")
> +int contention_end(u64 *ctx)
> +{
> + return do_contention_end((__u64)ctx[0], (__s32)ctx[1]);
> +}
> +
> +/* RCU grace-period synchronize / barrier contention hooks */
> +SEC("fentry/synchronize_rcu")
> +int BPF_PROG(fentry_synchronize_rcu)
> +{
> + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE);
> +}
[Severity: Medium]
In do_contention_begin() in tools/perf/util/bpf_skel/lock_contention.bpf.c,
bpf_get_stackid() uses stack_skip (which defaults to CONTENTION_STACK_SKIP =
4) unconditionally for both tp_btf/contention_begin and the new fentry RCU
and SRCU programs:
skip_owner:
pelem->stack_id = bpf_get_stackid(ctx, &stacks,
BPF_F_FAST_STACK_CMP | stack_skip);
For tp_btf/contention_begin, the tracepoint fires inside a lock slowpath via
tracepoint dispatch:
caller -> mutex_lock -> __mutex_lock_slowpath -> __mutex_lock ->
trace_contention_begin -> __bpf_trace_contention_begin -> bpf_trace_run2 ->
bpf_prog
Skipping 4 frames lands inside the locking functions (.sched.text,
.spinlock.text, or trace_contention_begin), which lock_contention_get_name()
in tools/perf/util/bpf_lock_contention.c then skips via
machine__is_lock_function() to reach the direct caller:
/* LOCK_AGGR_CALLER: skip lock internal functions */
while (machine__is_lock_function(machine, stack_trace[idx]) &&
idx < con->max_stack - 1)
idx++;
addr = stack_trace[idx];
In contrast, fentry_synchronize_rcu() (and the other six RCU/SRCU fentry
programs) attach directly to the top-level API functions. In an fentry
program, the callchain captured by bpf_get_stackid_raw_tp() ->
perf_callchain_kernel() only has two frames before the traced function
itself: ip[0] is the BPF prog, ip[1] is the BPF trampoline, ip[2] is the
traced function (e.g., synchronize_rcu), ip[3] is the direct caller, and
ip[4] is the caller's caller.
Won't skipping stack_skip = 4 frames in bpf_get_stackid() skip past ip[3],
dropping the direct caller from the recorded stack trace, attributing RCU
and SRCU contention in LOCK_AGGR_CALLER mode to the caller's caller (ip[4]),
and causing callstack filtering (-S) in match_callstack_filter() to miss the
direct caller?
Also, if the fentry stack skip is adjusted so that the traced RCU/SRCU
function itself can appear at stack_trace[0], machine__is_lock_function() in
tools/perf/util/machine.c only checks .sched.text, .spinlock.text,
__traceiter_contention_begin, and trace_contention_begin. Since
synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(), and srcu_barrier()
are not in .sched.text or .spinlock.text, should machine__is_lock_function()
or the fentry skip depth account for those functions as well?
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=5
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter
2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers
@ 2026-10-17 6:09 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-10-17 6:09 UTC (permalink / raw)
To: Ian Rogers; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [Medium] perf bpf-filter: held_locks leak and count corruption in fexit_downgrade_write()
- [Medium] perf bpf-filter: prescan_one_lock_expr() misses percpu_rw_semaphore symbols
- [Medium] perf bpf-filter: configure_lock_progs() ignores missing inlined spin/rwlock unlocks
- [Medium] perf bpf-filter: missing CONFIG_DEBUG_LOCK_ALLOC mutex hooks in lock_progs
- [Medium] perf bpf-filter: unguarded mutex->owner read in is_direct_lock_owner() on RT
--
commit 362a7c2a8d1d1d0355b0ac1b022fdffe73aae6c0
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Add lock_held() sample filter
This commit adds BPF sample filter predicates lock_held(), lock_held_read(),
lock_held_shared(), lock_held_write(), and lock_held_exclusive() to record
perf event samples only when a specified kernel lock is held. It uses direct
kernel lock owner inspection where possible and falls back to a held_locks
BPF hash map populated by fentry/fexit hooks on kernel lock and unlock
functions.
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 72e83ac1d6a4b..35d6f7c638684 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
[ ... ]
> +static const struct lock_prog_desc lock_progs[] = {
> + /* Mutex */
> + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
> + LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
[Severity: Medium]
On kernels built with CONFIG_DEBUG_LOCK_ALLOC=y, will mutex acquisitions via
mutex_lock_killable(), mutex_trylock(), and mutex_lock_nest_lock() be missed
by the held_locks map?
In include/linux/mutex.h, when CONFIG_DEBUG_LOCK_ALLOC=y, those functions are
macros that expand to different underlying entry points:
#define mutex_lock(lock) mutex_lock_nested(lock, 0)
#define mutex_lock_interruptible(lock) mutex_lock_interruptible_nested(lock, 0)
#define mutex_lock_killable(lock) _mutex_lock_killable(lock, 0, NULL)
#define mutex_lock_io(lock) mutex_lock_io_nested(lock, 0)
#define mutex_lock_nest_lock(lock, nest_lock) _mutex_lock_nest_lock(lock, &(nest_lock)->dep_map)
#define mutex_trylock(lock) _mutex_trylock_nest_lock(lock, NULL)
While lock_progs[] and sample_filter.bpf.c hook mutex_lock_nested,
mutex_lock_interruptible_nested, and mutex_lock_io_nested, they only hook
mutex_lock_killable and mutex_trylock rather than _mutex_lock_killable,
_mutex_trylock_nest_lock, and _mutex_lock_nest_lock.
For filters that rely on held_locks rather than direct owner inspection (such
as PBF_LOCK_SLAB and PBF_LOCK_TYPE), wouldn't mutexes acquired via
mutex_lock_killable() or mutex_trylock() go untracked on lockdep-enabled
kernels?
[ ... ]
> +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
> + unsigned int needed_groups, bool needs_lock_syms,
> + bool needs_slab_iter)
> +{
> + unsigned int enabled_progs = 0;
> +
> + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) {
> + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset;
> + struct bpf_program *prog = *progp;
> + bool enable = false;
> +
> + if ((needed_groups & lock_progs[i].group) && btf &&
> + lock_progs[i].btf_func) {
> + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func,
> + BTF_KIND_FUNC) > 0)
> + enable = true;
> + }
> +
> + if (enable)
> + enabled_progs++;
> + bpf_program__set_autoload(prog, enable);
> + }
> +
> + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms);
> + if (needs_slab_iter && btf &&
> + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) {
> + bpf_program__set_autoload(skel->progs.slab_cache_iter, true);
> + } else {
> + bpf_program__set_autoload(skel->progs.slab_cache_iter, false);
> + }
> +
> + if (needed_groups && !enabled_progs) {
> + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
> + return -ENOTSUP;
> + }
> + return 0;
> +}
[Severity: Medium]
What happens on kernels where spinlock and rwlock unlock functions are
inlined?
In kernel/Kconfig.locks, when !CONFIG_PREEMPTION && !CONFIG_DEBUG_SPINLOCK
(for example CONFIG_PREEMPT_NONE or CONFIG_PREEMPT_VOLUNTARY without
CONFIG_PREEMPT_DYNAMIC), CONFIG_UNINLINE_SPIN_UNLOCK is not set and
CONFIG_INLINE_SPIN_UNLOCK_IRQ, CONFIG_INLINE_READ_UNLOCK,
CONFIG_INLINE_READ_UNLOCK_IRQ, CONFIG_INLINE_WRITE_UNLOCK, and
CONFIG_INLINE_WRITE_UNLOCK_IRQ are enabled.
In that configuration, _raw_spin_unlock, _raw_spin_unlock_irq,
_raw_read_unlock, _raw_read_unlock_irq, _raw_write_unlock, and
_raw_write_unlock_irq are inlined and absent from vmlinux BTF, while the
acquire functions and *_unlock_irqrestore / *_unlock_bh remain out-of-line
in BTF.
Because configure_lock_progs() enables each program individually if its BTF
function exists and only fails when enabled_progs == 0, it will attach the
spinlock and rwlock acquire hooks without the corresponding unlock hooks.
Wouldn't any spinlock or rwlock released via an inlined unlock (including
rq_lock, zone_lock, and siglock) remain permanently recorded as held in
held_locks?
> +/*
> + * Pre-scan a single filter expression prior to loading the BPF skeleton in
> + * order to determine which BPF maps need to be sized and which lock tracing
> + * program groups need to be autoloaded.
> + */
> +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
> + bool *has_lock_filter,
> + unsigned int *prog_groups,
> + bool *needs_lock_syms,
> + bool *needs_slab_iter)
> +{
[ ... ]
> + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) {
> + *prog_groups |= LOCK_PROG_RTMUTEX;
> + } else if (strtailcmp(base, "mutex") == 0) {
> + *prog_groups |= LOCK_PROG_MUTEX;
> + } else if (strtailcmp(base, "percpu_rwsem") == 0 ||
> + strtailcmp(base, "pcpu_rwsem") == 0) {
> + *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
> + } else if (strtailcmp(base, "rwsem") == 0) {
> + if (mode & PBF_LOCK_MODE_READ)
> + *prog_groups |= LOCK_PROG_RWSEM_READ;
> + if (mode & PBF_LOCK_MODE_WRITE)
> + *prog_groups |= LOCK_PROG_RWSEM_WRITE;
[Severity: Medium]
Can prescan_one_lock_expr() misclassify kernel struct percpu_rw_semaphore
symbols as regular rwsems?
Kernel percpu_rw_semaphore locks defined with DEFINE_PERCPU_RWSEM or
DEFINE_STATIC_PERCPU_RWSEM typically end in _rwsem rather than _percpu_rwsem
(for example cgroup_threadgroup_rwsem, file_rwsem, or scx_fork_rwsem).
In prescan_one_lock_expr(), those symbols match strtailcmp(base, "rwsem") == 0
and only enable LOCK_PROG_RWSEM_READ and/or LOCK_PROG_RWSEM_WRITE, leaving
LOCK_PROG_PERCPU_RWSEM (fexit_percpu_down_write and fentry_percpu_up_write)
disabled. Since is_direct_lock_owner() only inspects struct mutex, struct
rw_semaphore, and struct rt_mutex_base, won't filtering on a
percpu_rw_semaphore symbol like cgroup_threadgroup_rwsem always evaluate to 0?
[ ... ]
> diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
> index e5666d4c17228..fbd2e51815d75 100644
> --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
> +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
[ ... ]
> +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr)
> +{
> + if (!curr)
> + return false;
> +
> + return (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr ||
> + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr ||
> + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr;
> +}
[Severity: Medium]
Will calling get_lock_owner(lock_addr, LCB_F_MUTEX) from
is_direct_lock_owner() break loading sample_filter_bpf on CONFIG_PREEMPT_RT=y
kernels?
perf_sample_filter() inlines perf_get_sample() -> check_lock_held() ->
is_direct_lock_owner(), which calls get_lock_owner(lock_addr, LCB_F_MUTEX) in
tools/perf/util/bpf_skel/lock_data.h:
if (flags & LCB_F_MUTEX) {
struct mutex *mutex = (void *)lock;
owner = BPF_CORE_READ(mutex, owner.counter);
if (owner & 0x4UL)
return NULL;
}
On CONFIG_PREEMPT_RT=y kernels, struct mutex wraps struct rt_mutex_base
rtmutex and does not have an owner field. Because get_lock_owner() does not
guard BPF_CORE_READ(mutex, owner.counter) with bpf_core_field_exists(),
wouldn't libbpf poison the instruction during CO-RE relocation and cause the
BPF verifier to reject perf_sample_filter on PREEMPT_RT kernels even when
lock_held() is not used?
[ ... ]
> +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags)
> +{
> + __u32 owner_id = get_lock_owner_id(flags);
> + struct held_lock_key addr_key = {
> + .addr_or_id = lock_addr,
> + .owner_id = owner_id,
> + .kind = PBF_LOCK_ADDR,
> + };
> + struct held_lock_val *existing;
> + __u32 type_flag = flags & LCB_F_TYPE_MASK;
> + bool is_read = (flags & LCB_F_READ) != 0;
> +
> + if (!lock_enabled || !lock_addr)
> + return 0;
> +
> + existing = bpf_map_lookup_elem(&held_locks, &addr_key);
> + if (!existing)
> + return 0;
> +
> + if (is_read) {
> + if (existing->read_count <= 1 && existing->write_count == 0)
> + bpf_map_delete_elem(&held_locks, &addr_key);
> + else if (existing->read_count > 0)
> + __sync_fetch_and_add(&existing->read_count, -1);
> + } else {
> + if (existing->write_count == 0 && existing->read_count == 0)
> + bpf_map_delete_elem(&held_locks, &addr_key);
> + else if (existing->read_count == 0)
> + bpf_map_delete_elem(&held_locks, &addr_key);
> + else
> + existing->write_count = 0;
> + }
[ ... ]
> +SEC("fexit/downgrade_write")
> +int BPF_PROG(fexit_downgrade_write, void *sem)
> +{
> + record_lock_acquired(ctx, LCB_F_READ);
> + return record_lock_released(ctx, LCB_F_WRITE);
> +}
[Severity: Medium]
Can fexit_downgrade_write() leak held_locks map entries under write-only rwsem
filters and corrupt PBF_LOCK_SLAB / PBF_LOCK_TYPE write counts?
First, in lock_progs[], fexit_downgrade_write has group LOCK_PROG_RWSEM
(LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE), while fentry_up_read has group
LOCK_PROG_RWSEM_READ:
LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM),
LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ),
LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE),
When a write-only filter such as lock_held_write(&mmap_lock) is used,
prescan_one_lock_expr() sets only LOCK_PROG_RWSEM_WRITE. Because
(needed_groups & lock_progs[i].group) is non-zero for LOCK_PROG_RWSEM,
configure_lock_progs() enables fexit_downgrade_write while leaving
fentry_up_read disabled.
When the kernel calls downgrade_write() (for example mmap_write_downgrade()),
fexit_downgrade_write() calls record_lock_acquired(ctx, LCB_F_READ), setting
existing->read_count = 1, and then calls record_lock_released(ctx,
LCB_F_WRITE), which sees existing->read_count != 0 and sets
existing->write_count = 0 instead of deleting addr_key. Since fentry_up_read
is not attached, up_read() never decrements read_count, leaking the held_locks
entry (and any PBF_LOCK_SLAB read_count).
Second, because fexit_downgrade_write() calls record_lock_acquired(ctx,
LCB_F_READ) before record_lock_released(ctx, LCB_F_WRITE), if downgrade_write()
is called on an rwsem whose initial down_write() was not recorded in
held_locks (such as under a read-only filter where LOCK_PROG_RWSEM_WRITE is
disabled, or when acquired before lock_enabled = 1), record_lock_acquired(ctx,
LCB_F_READ) creates addr_key with read_count = 1 and write_count = 0. The
subsequent record_lock_released(ctx, LCB_F_WRITE) then finds existing != NULL
even though existing->write_count == 0, and proceeds to decrement write_count
for PBF_LOCK_SLAB and PBF_LOCK_TYPE.
Should fexit_downgrade_write release LCB_F_WRITE before acquiring LCB_F_READ
(and only record LCB_F_READ when LOCK_PROG_RWSEM_READ is active), and should
record_lock_released_addr() verify that the corresponding read_count or
write_count was non-zero before decrementing the aggregate slab/type counts?
[ ... ]
完美！ Let's double check all rules:
- No markdown headers, no backticks, no ALL CAPS shouting (except inside quoted code or config/macro names that are literally in code). Wait: `CONFIG_DEBUG_LOCK_ALLOC=y`, `PBF_LOCK_SLAB`, `PBF_LOCK_TYPE`, `CONFIG_PREEMPT_RT=y`, `LOCK_PROG_RWSEM` are exact code identifiers. Let's make sure there are no non-code ALL CAPS words.
- No line numbers mentioned anywhere.
- Ends with a blank line.
- Severity tags `[Severity: Medium]` on their own line right above each inline comment block.
- All 5 findings included.
Everything is clean and strictly follows instructions. I'm ready to output.完美! Let's emit the final response.commit 362a7c2a8d1d1d0355b0ac1b022fdffe73aae6c0
Author: Ian Rogers <irogers@google.com>
perf bpf-filter: Add lock_held() sample filter
This commit adds BPF sample filter predicates lock_held(), lock_held_read(),
lock_held_shared(), lock_held_write(), and lock_held_exclusive() to record
perf event samples only when a specified kernel lock is held. It uses direct
kernel lock owner inspection where possible and falls back to a held_locks
BPF hash map populated by fentry/fexit hooks on kernel lock and unlock
functions.
> diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
> index 72e83ac1d6a4b..35d6f7c638684 100644
> --- a/tools/perf/util/bpf-filter.c
> +++ b/tools/perf/util/bpf-filter.c
[ ... ]
> +static const struct lock_prog_desc lock_progs[] = {
> + /* Mutex */
> + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
> + LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
> + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
[Severity: Medium]
On kernels built with CONFIG_DEBUG_LOCK_ALLOC=y, will mutex acquisitions via
mutex_lock_killable(), mutex_trylock(), and mutex_lock_nest_lock() be missed
by the held_locks map?
In include/linux/mutex.h, when CONFIG_DEBUG_LOCK_ALLOC=y, those functions are
macros that expand to different underlying entry points:
#define mutex_lock(lock) mutex_lock_nested(lock, 0)
#define mutex_lock_interruptible(lock) mutex_lock_interruptible_nested(lock, 0)
#define mutex_lock_killable(lock) _mutex_lock_killable(lock, 0, NULL)
#define mutex_lock_io(lock) mutex_lock_io_nested(lock, 0)
#define mutex_lock_nest_lock(lock, nest_lock) _mutex_lock_nest_lock(lock, &(nest_lock)->dep_map)
#define mutex_trylock(lock) _mutex_trylock_nest_lock(lock, NULL)
While lock_progs[] and sample_filter.bpf.c hook mutex_lock_nested,
mutex_lock_interruptible_nested, and mutex_lock_io_nested, they only hook
mutex_lock_killable and mutex_trylock rather than _mutex_lock_killable,
_mutex_trylock_nest_lock, and _mutex_lock_nest_lock.
For filters that rely on held_locks rather than direct owner inspection (such
as PBF_LOCK_SLAB and PBF_LOCK_TYPE), wouldn't mutexes acquired via
mutex_lock_killable() or mutex_trylock() go untracked on lockdep-enabled
kernels?
[ ... ]
> +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
> + unsigned int needed_groups, bool needs_lock_syms,
> + bool needs_slab_iter)
> +{
> + unsigned int enabled_progs = 0;
> +
> + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) {
> + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset;
> + struct bpf_program *prog = *progp;
> + bool enable = false;
> +
> + if ((needed_groups & lock_progs[i].group) && btf &&
> + lock_progs[i].btf_func) {
> + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func,
> + BTF_KIND_FUNC) > 0)
> + enable = true;
> + }
> +
> + if (enable)
> + enabled_progs++;
> + bpf_program__set_autoload(prog, enable);
> + }
> +
> + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms);
> + if (needs_slab_iter && btf &&
> + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) {
> + bpf_program__set_autoload(skel->progs.slab_cache_iter, true);
> + } else {
> + bpf_program__set_autoload(skel->progs.slab_cache_iter, false);
> + }
> +
> + if (needed_groups && !enabled_progs) {
> + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
> + return -ENOTSUP;
> + }
> + return 0;
> +}
[Severity: Medium]
What happens on kernels where spinlock and rwlock unlock functions are
inlined?
In kernel/Kconfig.locks, when !CONFIG_PREEMPTION && !CONFIG_DEBUG_SPINLOCK
(for example CONFIG_PREEMPT_NONE or CONFIG_PREEMPT_VOLUNTARY without
CONFIG_PREEMPT_DYNAMIC), CONFIG_UNINLINE_SPIN_UNLOCK is not set and
CONFIG_INLINE_SPIN_UNLOCK_IRQ, CONFIG_INLINE_READ_UNLOCK,
CONFIG_INLINE_READ_UNLOCK_IRQ, CONFIG_INLINE_WRITE_UNLOCK, and
CONFIG_INLINE_WRITE_UNLOCK_IRQ are enabled.
In that configuration, _raw_spin_unlock, _raw_spin_unlock_irq,
_raw_read_unlock, _raw_read_unlock_irq, _raw_write_unlock, and
_raw_write_unlock_irq are inlined and absent from vmlinux BTF, while the
acquire functions and *_unlock_irqrestore / *_unlock_bh remain out-of-line
in BTF.
Because configure_lock_progs() enables each program individually if its BTF
function exists and only fails when enabled_progs == 0, it will attach the
spinlock and rwlock acquire hooks without the corresponding unlock hooks.
Wouldn't any spinlock or rwlock released via an inlined unlock (including
rq_lock, zone_lock, and siglock) remain permanently recorded as held in
held_locks?
> +/*
> + * Pre-scan a single filter expression prior to loading the BPF skeleton in
> + * order to determine which BPF maps need to be sized and which lock tracing
> + * program groups need to be autoloaded.
> + */
> +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
> + bool *has_lock_filter,
> + unsigned int *prog_groups,
> + bool *needs_lock_syms,
> + bool *needs_slab_iter)
> +{
[ ... ]
> + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) {
> + *prog_groups |= LOCK_PROG_RTMUTEX;
> + } else if (strtailcmp(base, "mutex") == 0) {
> + *prog_groups |= LOCK_PROG_MUTEX;
> + } else if (strtailcmp(base, "percpu_rwsem") == 0 ||
> + strtailcmp(base, "pcpu_rwsem") == 0) {
> + *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
> + } else if (strtailcmp(base, "rwsem") == 0) {
> + if (mode & PBF_LOCK_MODE_READ)
> + *prog_groups |= LOCK_PROG_RWSEM_READ;
> + if (mode & PBF_LOCK_MODE_WRITE)
> + *prog_groups |= LOCK_PROG_RWSEM_WRITE;
[Severity: Medium]
Can prescan_one_lock_expr() misclassify kernel struct percpu_rw_semaphore
symbols as regular rwsems?
Kernel percpu_rw_semaphore locks defined with DEFINE_PERCPU_RWSEM or
DEFINE_STATIC_PERCPU_RWSEM typically end in _rwsem rather than _percpu_rwsem
(for example cgroup_threadgroup_rwsem, file_rwsem, or scx_fork_rwsem).
In prescan_one_lock_expr(), those symbols match strtailcmp(base, "rwsem") == 0
and only enable LOCK_PROG_RWSEM_READ and/or LOCK_PROG_RWSEM_WRITE, leaving
LOCK_PROG_PERCPU_RWSEM (fexit_percpu_down_write and fentry_percpu_up_write)
disabled. Since is_direct_lock_owner() only inspects struct mutex, struct
rw_semaphore, and struct rt_mutex_base, won't filtering on a
percpu_rw_semaphore symbol like cgroup_threadgroup_rwsem always evaluate to 0?
[ ... ]
> diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
> index e5666d4c17228..fbd2e51815d75 100644
> --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
> +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
[ ... ]
> +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr)
> +{
> + if (!curr)
> + return false;
> +
> + return (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr ||
> + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr ||
> + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr;
> +}
[Severity: Medium]
Will calling get_lock_owner(lock_addr, LCB_F_MUTEX) from
is_direct_lock_owner() break loading sample_filter_bpf on CONFIG_PREEMPT_RT=y
kernels?
perf_sample_filter() inlines perf_get_sample() -> check_lock_held() ->
is_direct_lock_owner(), which calls get_lock_owner(lock_addr, LCB_F_MUTEX) in
tools/perf/util/bpf_skel/lock_data.h:
if (flags & LCB_F_MUTEX) {
struct mutex *mutex = (void *)lock;
owner = BPF_CORE_READ(mutex, owner.counter);
if (owner & 0x4UL)
return NULL;
}
On CONFIG_PREEMPT_RT=y kernels, struct mutex wraps struct rt_mutex_base
rtmutex and does not have an owner field. Because get_lock_owner() does not
guard BPF_CORE_READ(mutex, owner.counter) with bpf_core_field_exists(),
wouldn't libbpf poison the instruction during CO-RE relocation and cause the
BPF verifier to reject perf_sample_filter on PREEMPT_RT kernels even when
lock_held() is not used?
[ ... ]
> +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags)
> +{
> + __u32 owner_id = get_lock_owner_id(flags);
> + struct held_lock_key addr_key = {
> + .addr_or_id = lock_addr,
> + .owner_id = owner_id,
> + .kind = PBF_LOCK_ADDR,
> + };
> + struct held_lock_val *existing;
> + __u32 type_flag = flags & LCB_F_TYPE_MASK;
> + bool is_read = (flags & LCB_F_READ) != 0;
> +
> + if (!lock_enabled || !lock_addr)
> + return 0;
> +
> + existing = bpf_map_lookup_elem(&held_locks, &addr_key);
> + if (!existing)
> + return 0;
> +
> + if (is_read) {
> + if (existing->read_count <= 1 && existing->write_count == 0)
> + bpf_map_delete_elem(&held_locks, &addr_key);
> + else if (existing->read_count > 0)
> + __sync_fetch_and_add(&existing->read_count, -1);
> + } else {
> + if (existing->write_count == 0 && existing->read_count == 0)
> + bpf_map_delete_elem(&held_locks, &addr_key);
> + else if (existing->read_count == 0)
> + bpf_map_delete_elem(&held_locks, &addr_key);
> + else
> + existing->write_count = 0;
> + }
[ ... ]
> +SEC("fexit/downgrade_write")
> +int BPF_PROG(fexit_downgrade_write, void *sem)
> +{
> + record_lock_acquired(ctx, LCB_F_READ);
> + return record_lock_released(ctx, LCB_F_WRITE);
> +}
[Severity: Medium]
Can fexit_downgrade_write() leak held_locks map entries under write-only rwsem
filters and corrupt PBF_LOCK_SLAB / PBF_LOCK_TYPE write counts?
First, in lock_progs[], fexit_downgrade_write has group LOCK_PROG_RWSEM
(LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE), while fentry_up_read has group
LOCK_PROG_RWSEM_READ:
LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM),
LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ),
LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE),
When a write-only filter such as lock_held_write(&mmap_lock) is used,
prescan_one_lock_expr() sets only LOCK_PROG_RWSEM_WRITE. Because
(needed_groups & lock_progs[i].group) is non-zero for LOCK_PROG_RWSEM,
configure_lock_progs() enables fexit_downgrade_write while leaving
fentry_up_read disabled.
When the kernel calls downgrade_write() (for example mmap_write_downgrade()),
fexit_downgrade_write() calls record_lock_acquired(ctx, LCB_F_READ), setting
existing->read_count = 1, and then calls record_lock_released(ctx,
LCB_F_WRITE), which sees existing->read_count != 0 and sets
existing->write_count = 0 instead of deleting addr_key. Since fentry_up_read
is not attached, up_read() never decrements read_count, leaking the held_locks
entry (and any PBF_LOCK_SLAB read_count).
Second, because fexit_downgrade_write() calls record_lock_acquired(ctx,
LCB_F_READ) before record_lock_released(ctx, LCB_F_WRITE), if downgrade_write()
is called on an rwsem whose initial down_write() was not recorded in
held_locks (such as under a read-only filter where LOCK_PROG_RWSEM_WRITE is
disabled, or when acquired before lock_enabled = 1), record_lock_acquired(ctx,
LCB_F_READ) creates addr_key with read_count = 1 and write_count = 0. The
subsequent record_lock_released(ctx, LCB_F_WRITE) then finds existing != NULL
even though existing->write_count == 0, and proceeds to decrement write_count
for PBF_LOCK_SLAB and PBF_LOCK_TYPE.
Should fexit_downgrade_write release LCB_F_WRITE before acquiring LCB_F_READ
(and only record LCB_F_READ when LOCK_PROG_RWSEM_READ is active), and should
record_lock_released_addr() verify that the corresponding read_count or
write_count was non-zero before decrementing the aggregate slab/type counts?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=2
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2026-10-17 6:09 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-17 5:52 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers
2026-10-17 6:09 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers
2026-10-17 5:49 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-17 5:56 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-17 6:08 ` sashiko-bot
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox