* [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support
@ 2026-10-17 5:45 Ian Rogers
2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
` (5 more replies)
0 siblings, 6 replies; 24+ messages in thread
From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Namhyung Kim
Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter,
James Clark, linux-perf-users, linux-kernel, rcu
This series adds a new BPF sample filter predicate, `lock_held(<lock>)`
(along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`,
`lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W`
mode suffixes), to `perf record --filter`, allowing perf event samples to
be recorded only when a specified kernel lock is held (or not held with
`!`) by the running kernel thread or CPU. It also extends both
`lock_held()` and `perf lock contention --use-bpf` to support RCU and
SRCU.
For example:
# perf record -a -g -e cycles --filter 'lock_held(&cgroup_mutex)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' -- ...
# perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' -- ...
# perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ...
# perf lock contention -a -b -Y rcu,srcu -- sleep 1
Supported `<lock>` specifications match `perf lock contention`:
- Global/static kernel lock symbols (with or without leading `&`, e.g.
`lock_held(&cgroup_mutex)` or `lock_held(cgroup_mutex)`)
- Hexadecimal kernel lock addresses
- Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`,
`lock_held_read(&dentry)`)
- Well-known per-process, semi-global, and RCU locks: `mmap_lock`,
`siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace`
- Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`,
`rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`,
`percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`,
`rcu:W`, `semaphore`
Limitations and weaknesses in determining whether a lock is held:
- Inlined lock/unlock operations:
- When the kernel is configured to inline spinlock or rwlock operations
(`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`,
`CONFIG_INLINE_WRITE_LOCK`, etc., which are enabled on many
`!CONFIG_PREEMPT_RT` / `!CONFIG_PROVE_LOCKING` builds unless
`CONFIG_UNINLINE_SPIN_UNLOCK` or lockdep/preemption forces them out of
line), `_raw_spin_lock*` / `_raw_spin_unlock*` and
`_raw_read_lock*` / `_raw_write_lock*` are inlined into callers and
have no BTF function entry points for `fentry`/`fexit` to attach to.
- Fast-path `percpu_down_read()` / `percpu_up_read()` and
`srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined
per-CPU counter increments without function calls, so only
`percpu_down_write()` / `percpu_up_write()` and out-of-line
`__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked.
- On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to
`preempt_disable()` and does not maintain
`current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)`
read-side inspection requires `CONFIG_PREEMPT_RCU`.
- Lack of an inspectable owner field on non-exclusive / spinning locks:
- While `struct mutex`, `struct rt_mutex`, and write-locked
`struct rw_semaphore` store the owning `task_struct *` in `lock->owner`
(allowing `perf_sample_filter` to inspect kernel memory directly on
sample overflow when filtering by lock address), spinlocks, rwlocks,
semaphores, SRCU, and reader-locked `rw_semaphore`s (where
`__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`)
do not maintain an authoritative per-task owner pointer.
- Tracking these locks (as well as slab-cache and lock-type filters)
relies on `fexit` (acquire) and `fentry` (release) BPF hooks
maintaining the `held_locks` BPF hash map:
- Any such lock already held before `perf record` attaches its BPF
programs will not be present in `held_locks` until it is released
and re-acquired.
- The attached `fentry`/`fexit` hooks execute on every kernel lock
acquire and release for the hooked lock family across the system,
adding tracing overhead outside of perf sample handling.
Patch overview:
1. "perf lock contention: Share common BPF and lock type definitions in
lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and
`LCB_F_*` flags, CO-RE structs, and common BPF lock helpers
(`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`,
`get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`,
`do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`.
2. "perf bpf-filter: Add lock_held() sample filter": Implements
`lock_held()`, `lock_held_read()`/`lock_held_shared()`,
`lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode
suffixes using direct lock owner inspection where possible (`mutex`,
`rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF map
updated by selectively attached `fentry`/`fexit` hooks.
3. "perf test: Add tests for lock_held() BPF sample filter": Adds unit
tests in `tests/parse-events.c` and shell tests in
`tests/shell/record_bpf_filter.sh`.
4. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU": Supports `rcu` (`rcu:R` via direct
`current->rcu_read_lock_nesting` inspection with zero tracing hooks;
`rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks),
`rcu-trace` (via `current->trc_reader_nesting`), and `srcu`
(`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W`
via `synchronize_srcu*()` / `srcu_barrier()`).
5. "perf lock contention: Extend BPF lock contention to RCU and SRCU":
Extends `perf lock contention --use-bpf` to track RCU and SRCU
grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`).
Assisted-by: Antigravity:gemini-3.1-pro
Ian Rogers (5):
perf lock contention: Share common BPF and lock type definitions in
lock_data.h
perf bpf-filter: Add lock_held() sample filter
perf test: Add tests for lock_held() BPF sample filter
perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU,
and SRCU
perf lock contention: Extend BPF lock contention to RCU and SRCU
tools/perf/Documentation/perf-lock.txt | 7 +-
tools/perf/Documentation/perf-record.txt | 51 +-
tools/perf/builtin-lock.c | 27 -
tools/perf/tests/parse-events.c | 121 ++
tools/perf/tests/shell/lock_contention.sh | 14 +
tools/perf/tests/shell/record_bpf_filter.sh | 119 ++
tools/perf/util/bpf-filter.c | 1070 ++++++++++++++++-
tools/perf/util/bpf-filter.h | 13 +
tools/perf/util/bpf-filter.l | 31 +-
tools/perf/util/bpf-filter.y | 46 +-
tools/perf/util/bpf_lock_contention.c | 167 ++-
.../perf/util/bpf_skel/lock_contention.bpf.c | 410 +++----
tools/perf/util/bpf_skel/lock_data.h | 360 +++++-
tools/perf/util/bpf_skel/sample-filter.h | 71 ++
tools/perf/util/bpf_skel/sample_filter.bpf.c | 1051 ++++++++++++++++
tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 +
tools/perf/util/lock-contention.c | 22 +
tools/perf/util/lock-contention.h | 15 +
tools/perf/util/parse-events.c | 3 +-
19 files changed, 3273 insertions(+), 332 deletions(-)
--
2.56.0.212.g129ede20d9-goog
^ permalink raw reply [flat|nested] 24+ messages in thread* [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h 2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers @ 2026-10-17 5:45 ` Ian Rogers 2026-10-17 5:52 ` sashiko-bot 2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers ` (4 subsequent siblings) 5 siblings, 1 reply; 24+ messages in thread From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Namhyung Kim Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark, linux-perf-users, linux-kernel, rcu Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers (get_lock_owner, get_current_mmap_lock, get_current_siglock, get_lock_addr, check_lock_type, do_collect_zone_lock, do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF sample filtering. Also add rw_semaphore.count and struct rt_mutex_base to the minimal vmlinux.h and extend get_lock_owner() to support rt_mutex and check writer-ownership and handoff flags. Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/builtin-lock.c | 27 -- .../perf/util/bpf_skel/lock_contention.bpf.c | 265 +-------------- tools/perf/util/bpf_skel/lock_data.h | 316 ++++++++++++++++++ tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 + tools/perf/util/lock-contention.c | 18 + tools/perf/util/lock-contention.h | 13 + 6 files changed, 370 insertions(+), 274 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index d5c0d55cb82d..feaf1cfd08cf 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -1477,33 +1477,6 @@ static void sort_result(void) } } -static const struct { - unsigned int flags; - /* - * Name of the lock flags (access), with delimeter ':'. - * For example, rwsem:R of rwsem:W. - */ - const char *flags_name; - /* Name of the lock (type), for example, rwlock or rwsem. */ - const char *lock_name; -} lock_type_table[] = { - { 0, "semaphore", "semaphore" }, - { LCB_F_SPIN, "spinlock", "spinlock" }, - { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, - { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, - { LCB_F_READ, "rwsem:R", "rwsem" }, - { LCB_F_WRITE, "rwsem:W", "rwsem" }, - { LCB_F_RT, "rt-mutex", "rt-mutex" }, - { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, - { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, - { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, - { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, - { LCB_F_MUTEX, "mutex", "mutex" }, - { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, - /* alias for optimistic spinning only */ - { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, -}; - static const char *get_type_flags_name(unsigned int flags) { flags &= LCB_F_TYPE_MASK; diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0d9c6f55050e..0f2118006005 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -8,23 +8,9 @@ #include "lock_data.h" -/* for collect_lock_syms(). 4096 was rejected by the verifier */ -#define MAX_CPUS 1024 - -/* for collect_zone_lock(). It should be more than the actual zones. */ -#define MAX_ZONES 10 - /* for do_lock_delay(). Arbitrarily set to 1 million. */ #define MAX_LOOP (1U << 20) -/* lock contention flags from include/trace/events/lock.h */ -#define LCB_F_SPIN (1U << 0) -#define LCB_F_READ (1U << 1) -#define LCB_F_WRITE (1U << 2) -#define LCB_F_RT (1U << 3) -#define LCB_F_PERCPU (1U << 4) -#define LCB_F_MUTEX (1U << 5) - /* callstack storage */ struct { __uint(type, BPF_MAP_TYPE_STACK_TRACE); @@ -96,13 +82,6 @@ struct { __uint(max_entries, MAX_ENTRIES); } task_data SEC(".maps"); -struct { - __uint(type, BPF_MAP_TYPE_HASH); - __uint(key_size, sizeof(__u64)); - __uint(value_size, sizeof(__u32)); - __uint(max_entries, MAX_ENTRIES); -} lock_syms SEC(".maps"); - struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(key_size, sizeof(__u32)); @@ -138,6 +117,13 @@ struct { __uint(max_entries, 1); } cgroup_filter SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, MAX_ENTRIES); +} lock_syms SEC(".maps"); + struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(key_size, sizeof(long)); @@ -159,22 +145,6 @@ struct { __uint(max_entries, 1); } lock_delays SEC(".maps"); -struct rw_semaphore___old { - struct task_struct *owner; -} __attribute__((preserve_access_index)); - -struct rw_semaphore___new { - atomic_long_t owner; -} __attribute__((preserve_access_index)); - -struct mm_struct___old { - struct rw_semaphore mmap_sem; -} __attribute__((preserve_access_index)); - -struct mm_struct___new { - struct rw_semaphore mmap_lock; -} __attribute__((preserve_access_index)); - struct cas_ctx { struct contention_data *data; u64 duration; @@ -182,8 +152,6 @@ struct cas_ctx { int min_done; }; -extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; - /* control flags */ const volatile int has_cpu; const volatile int has_task; @@ -202,6 +170,11 @@ const volatile int lock_delay; /* determine the key of lock stat */ const volatile int aggr_mode; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + int enabled; int perf_subsys_id = -1; @@ -222,8 +195,6 @@ int data_map_full; struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak; void bpf_task_release(struct task_struct *p) __ksym __weak; -static inline __u32 check_lock_type(__u64 lock, __u32 flags); - static inline __u64 get_current_cgroup_id(void) { struct task_struct *task; @@ -347,85 +318,6 @@ static inline int update_task_data(struct task_struct *task) return 0; } -#ifndef __has_builtin -# define __has_builtin(x) 0 -#endif - -static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) -{ - struct task_struct *task; - __u64 owner = 0; - - if (flags & LCB_F_MUTEX) { - struct mutex *mutex = (void *)lock; - owner = BPF_CORE_READ(mutex, owner.counter); - } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { - /* - * Support for the BPF_TYPE_MATCHES argument to the - * __builtin_preserve_type_info builtin was added at some point during - * development of clang 15 and it's what is needed for - * bpf_core_type_matches. - */ -#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 - if (bpf_core_type_matches(struct rw_semaphore___old)) { - struct rw_semaphore___old *rwsem = (void *)lock; - owner = (unsigned long)BPF_CORE_READ(rwsem, owner); - } else if (bpf_core_type_matches(struct rw_semaphore___new)) { - struct rw_semaphore___new *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); - } -#else - /* assume new struct */ - struct rw_semaphore *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); -#endif - } - - if (!owner) - return NULL; - - task = (void *)(owner & ~7UL); - return task; -} - -static inline __u32 check_lock_type(__u64 lock, __u32 flags) -{ - struct task_struct *curr; - struct mm_struct___old *mm_old; - struct mm_struct___new *mm_new; - struct sighand_struct *sighand; - - switch (flags) { - case LCB_F_READ: /* rwsem */ - case LCB_F_WRITE: - curr = bpf_get_current_task_btf(); - if (curr->mm == NULL) - break; - mm_new = (void *)curr->mm; - if (bpf_core_field_exists(mm_new->mmap_lock)) { - if (&mm_new->mmap_lock == (void *)lock) - return LCD_F_MMAP_LOCK; - break; - } - mm_old = (void *)curr->mm; - if (bpf_core_field_exists(mm_old->mmap_sem)) { - if (&mm_old->mmap_sem == (void *)lock) - return LCD_F_MMAP_LOCK; - } - break; - case LCB_F_SPIN: /* spinlock */ - curr = bpf_get_current_task_btf(); - sighand = curr->sighand; - - if (sighand && &sighand->siglock == (void *)lock) - return LCD_F_SIGHAND_LOCK; - break; - default: - break; - } - return 0; -} - static inline long delay_callback(__u64 idx, void *arg) { __u64 target = *(__u64 *)arg; @@ -908,145 +800,24 @@ int contention_end(u64 *ctx) return 0; } -extern struct rq runqueues __ksym; - -const volatile __u64 contig_page_data_addr; -const volatile __u64 node_data_addr; -const volatile int nr_nodes; -const volatile int sizeof_zone; - -struct rq___old { - raw_spinlock_t lock; -} __attribute__((preserve_access_index)); - -struct rq___new { - raw_spinlock_t __lock; -} __attribute__((preserve_access_index)); - -static void collect_zone_lock(void) -{ - __u64 nr_zones, zone_off; - __u64 lock_addr, lock_off; - __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; - - zone_off = offsetof(struct pglist_data, node_zones); - lock_off = offsetof(struct zone, lock); - - if (contig_page_data_addr) { - struct pglist_data *contig_page_data; - - contig_page_data = (void *)(long)contig_page_data_addr; - nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); - - for (int i = 0; i < MAX_ZONES; i++) { - __u64 zone_addr; - - if (i >= nr_zones) - break; - - zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } else if (nr_nodes > 0) { - struct pglist_data **node_data = (void *)(long)node_data_addr; - - for (int i = 0; i < nr_nodes; i++) { - struct pglist_data *pgdat = NULL; - int err; - - err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); - if (err < 0 || pgdat == NULL) - break; - - nr_zones = BPF_CORE_READ(pgdat, nr_zones); - for (int k = 0; k < MAX_ZONES; k++) { - __u64 zone_addr; - - if (k >= nr_zones) - break; - - zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } - } -} - SEC("raw_tp/bpf_test_finish") -int BPF_PROG(collect_lock_syms) +int BPF_PROG(end_timestamp) { - __u64 lock_addr, lock_off; - __u32 lock_flag; - - if (bpf_core_field_exists(struct rq___new, __lock)) - lock_off = offsetof(struct rq___new, __lock); - else - lock_off = offsetof(struct rq___old, lock); - - for (int i = 0; i < MAX_CPUS; i++) { - struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); - - if (rq == NULL) - break; - - lock_addr = (__u64)(void *)rq + lock_off; - lock_flag = LOCK_CLASS_RQLOCK; - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - - collect_zone_lock(); - + end_ts = bpf_ktime_get_ns(); return 0; } SEC("raw_tp/bpf_test_finish") -int BPF_PROG(end_timestamp) +int BPF_PROG(collect_lock_syms) { - end_ts = bpf_ktime_get_ns(); - return 0; + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); } -/* - * bpf_iter__kmem_cache added recently so old kernels don't have it in the - * vmlinux.h. But we cannot add it here since it will cause a compiler error - * due to redefinition of the struct on later kernels. - * - * So it uses a CO-RE trick to access the member only if it has the type. - * This will support both old and new kernels without compiler errors. - */ -struct bpf_iter__kmem_cache___new { - struct kmem_cache *s; -} __attribute__((preserve_access_index)); - SEC("iter/kmem_cache") int slab_cache_iter(void *ctx) { - struct kmem_cache *s = NULL; - struct slab_cache_data d; - const char *nameptr; - - if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { - struct bpf_iter__kmem_cache___new *iter = ctx; - - s = iter->s; - } - - if (s == NULL) - return 0; - - nameptr = s->name; - bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); - - d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT; - if (d.id >= LCB_F_SLAB_ID_END) - return 0; - - bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST); - return 0; + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); } char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index 652e114e6b87..fed445c9d8b4 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -45,6 +45,14 @@ struct contention_task_data { #define LCB_F_SLAB_ID_END (1U << 26) #define LCB_F_SLAB_ID_MASK 0x03FF0000U +/* lock contention flags from include/trace/events/lock.h */ +#define LCB_F_SPIN (1U << 0) +#define LCB_F_READ (1U << 1) +#define LCB_F_WRITE (1U << 2) +#define LCB_F_RT (1U << 3) +#define LCB_F_PERCPU (1U << 4) +#define LCB_F_MUTEX (1U << 5) + #define LCB_F_TYPE_MAX (1U << 7) #define LCB_F_TYPE_MASK 0x0000007FU @@ -76,4 +84,312 @@ struct slab_cache_data { char name[SLAB_NAME_MAX]; }; +#ifdef __BPF__ + +/* for collect_lock_syms(). 4096 was rejected by the verifier */ +#define MAX_CPUS 1024 + +/* for collect_zone_lock(). It should be more than the actual zones. */ +#define MAX_ZONES 10 + +extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; +extern struct rq runqueues __ksym __weak; + +#ifndef __has_builtin +# define __has_builtin(x) 0 +#endif + +struct rw_semaphore___old { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + +struct rw_semaphore___new { + atomic_long_t count; + atomic_long_t owner; +} __attribute__((preserve_access_index)); + +struct mm_struct___old { + struct rw_semaphore mmap_sem; +} __attribute__((preserve_access_index)); + +struct mm_struct___new { + struct rw_semaphore mmap_lock; +} __attribute__((preserve_access_index)); + +struct rq___old { + raw_spinlock_t lock; +} __attribute__((preserve_access_index)); + +struct rq___new { + raw_spinlock_t __lock; +} __attribute__((preserve_access_index)); + +/* + * bpf_iter__kmem_cache added recently so old kernels don't have it in the + * vmlinux.h. But we cannot add it here since it will cause a compiler error + * due to redefinition of the struct on later kernels. + * + * So it uses a CO-RE trick to access the member only if it has the type. + * This will support both old and new kernels without compiler errors. + */ +struct bpf_iter__kmem_cache___new { + struct kmem_cache *s; +} __attribute__((preserve_access_index)); + +static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) +{ + struct task_struct *task; + __u64 owner = 0; + + if (flags & LCB_F_MUTEX) { + struct mutex *mutex = (void *)lock; + + owner = BPF_CORE_READ(mutex, owner.counter); + if (owner & 0x4UL) + return NULL; + } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { + struct rw_semaphore___new *rwsem = (void *)lock; + + if (flags == LCB_F_WRITE && bpf_core_field_exists(rwsem->count.counter)) { + __u64 count = BPF_CORE_READ(rwsem, count.counter); + + /* Must have RWSEM_WRITER_LOCKED (bit 0) set */ + if (!(count & 1UL)) + return NULL; + } + /* + * Support for the BPF_TYPE_MATCHES argument to the + * __builtin_preserve_type_info builtin was added at some point during + * development of clang 15 and it's what is needed for + * bpf_core_type_matches. + */ +#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 + if (bpf_core_type_matches(struct rw_semaphore___old)) { + struct rw_semaphore___old *old = (void *)lock; + + owner = (unsigned long)BPF_CORE_READ(old, owner); + } else if (bpf_core_type_matches(struct rw_semaphore___new)) { + owner = BPF_CORE_READ(rwsem, owner.counter); + } +#else + /* assume new struct */ + owner = BPF_CORE_READ(rwsem, owner.counter); +#endif + /* RWSEM_READER_OWNED is bit 0 of owner */ + if (flags == LCB_F_WRITE && (owner & 1UL)) + return NULL; + } else if (flags == LCB_F_RT) { + struct rt_mutex_base *rtm = (void *)lock; + + if (bpf_core_type_exists(struct rt_mutex_base)) { + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } + } + + if (!owner) + return NULL; + + task = (void *)(owner & ~7UL); + return task; +} + +static __always_inline __u64 get_current_mmap_lock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct mm_struct___new *mm_new = NULL; + struct mm_struct___old *mm_old = NULL; + __u64 mm; + + if (!curr) + return 0; + + mm = (__u64)BPF_CORE_READ(task, mm); + if (!mm) + return 0; + + if (bpf_core_field_exists(mm_new->mmap_lock)) + return mm + bpf_core_field_offset(mm_new->mmap_lock); + + if (bpf_core_field_exists(mm_old->mmap_sem)) + return mm + bpf_core_field_offset(mm_old->mmap_sem); + + return 0; +} + +static __always_inline __u64 get_current_siglock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct sighand_struct *sighand = NULL; + __u64 sh; + + if (!curr) + return 0; + + sh = (__u64)BPF_CORE_READ(task, sighand); + if (!sh) + return 0; + + return sh + bpf_core_field_offset(sighand->siglock); +} + +/* + * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. + * + * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the + * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed + * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such + * as `bpf_get_kmem_cache(u64 addr)`. + */ +static __always_inline __u64 get_lock_addr(void *ctx) +{ + __u64 lock_addr = 0; + + bpf_get_func_arg(ctx, 0, &lock_addr); + return lock_addr; +} + +static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) +{ + __u64 curr; + + switch (flags) { + case LCB_F_READ: /* rwsem */ + case LCB_F_WRITE: + curr = bpf_get_current_task(); + if (get_current_mmap_lock(curr) == lock) + return LCD_F_MMAP_LOCK; + break; + case LCB_F_SPIN: /* spinlock */ + curr = bpf_get_current_task(); + if (get_current_siglock(curr) == lock) + return LCD_F_SIGHAND_LOCK; + break; + default: + break; + } + return 0; +} + +static __always_inline void do_collect_zone_lock(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 nr_zones, zone_off; + __u64 lock_addr, lock_off; + __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; + + zone_off = offsetof(struct pglist_data, node_zones); + lock_off = offsetof(struct zone, lock); + + if (contig_page_data_addr) { + struct pglist_data *contig_page_data; + + contig_page_data = (void *)(long)contig_page_data_addr; + nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); + + for (int i = 0; i < MAX_ZONES; i++) { + __u64 zone_addr; + + if (i >= nr_zones) + break; + + zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } else if (nr_nodes > 0) { + struct pglist_data **node_data = (void *)(long)node_data_addr; + + for (int i = 0; i < nr_nodes; i++) { + struct pglist_data *pgdat = NULL; + int err; + + err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); + if (err < 0) + break; + if (pgdat == NULL) + continue; + + nr_zones = BPF_CORE_READ(pgdat, nr_zones); + for (int k = 0; k < MAX_ZONES; k++) { + __u64 zone_addr; + + if (k >= nr_zones) + break; + + zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + } +} + +static __always_inline int do_collect_lock_syms(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 lock_addr, lock_off; + __u32 lock_flag; + + if (bpf_core_field_exists(struct rq___new, __lock)) + lock_off = offsetof(struct rq___new, __lock); + else + lock_off = offsetof(struct rq___old, lock); + + if (&runqueues) { + for (int i = 0; i < MAX_CPUS; i++) { + struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); + + if (rq == NULL) + break; + + lock_addr = (__u64)(void *)rq + lock_off; + lock_flag = LOCK_CLASS_RQLOCK; + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + + do_collect_zone_lock(lock_syms_map, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); + + return 0; +} + +static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map, + __u32 *slab_cache_id) +{ + struct kmem_cache *s = NULL; + struct slab_cache_data d = {}; + const char *nameptr; + + if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { + struct bpf_iter__kmem_cache___new *iter = ctx; + + s = iter->s; + } + + if (s == NULL) + return 0; + + nameptr = s->name; + bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); + + d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT; + if (d.id >= LCB_F_SLAB_ID_END) + return 0; + + bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST); + return 0; +} + +#endif /* __BPF__ */ + #endif /* UTIL_BPF_SKEL_LOCK_DATA_H */ diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index 94e42b022ab1..b75ddc6846bb 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -76,6 +76,7 @@ struct signal_struct { } __attribute__((preserve_access_index)); struct rw_semaphore { + atomic_long_t count; atomic_long_t owner; } __attribute__((preserve_access_index)); @@ -83,6 +84,10 @@ struct mutex { atomic_long_t owner; } __attribute__((preserve_access_index)); +struct rt_mutex_base { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + struct kernfs_node { u64 id; } __attribute__((preserve_access_index)); diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index 119a7206f3cd..a1bb78d6e29c 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -22,6 +22,24 @@ struct callstack_filter { static LIST_HEAD(callstack_filters); struct hlist_head *lockhash_table; +const struct perf_lock_type lock_type_table[] = { + { 0, "semaphore", "semaphore" }, + { LCB_F_SPIN, "spinlock", "spinlock" }, + { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, + { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, + { LCB_F_READ, "rwsem:R", "rwsem" }, + { LCB_F_WRITE, "rwsem:W", "rwsem" }, + { LCB_F_RT, "rt-mutex", "rt-mutex" }, + { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, + { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, + { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, + { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, + { LCB_F_MUTEX, "mutex", "mutex" }, + { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, + /* alias for optimistic spinning only */ + { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, +}; + int parse_call_stack(const struct option *opt __maybe_unused, const char *str, int unset __maybe_unused) { diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 59c94190b092..68e9bd77e8a9 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -129,6 +129,19 @@ struct thread_stat { #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +struct perf_lock_type { + unsigned int flags; + /* + * Name of the lock flags (access), with delimiter ':'. + * For example, rwsem:R or rwsem:W. + */ + const char *flags_name; + /* Name of the lock (type), for example, rwlock or rwsem. */ + const char *lock_name; +}; + +extern const struct perf_lock_type lock_type_table[14]; + struct evlist; struct machine; struct target; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h 2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers @ 2026-10-17 5:52 ` sashiko-bot 0 siblings, 0 replies; 24+ messages in thread From: sashiko-bot @ 2026-10-17 5:52 UTC (permalink / raw) To: Ian Rogers; +Cc: linux-perf-users > Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers > (get_lock_owner, get_current_mmap_lock, get_current_siglock, > get_lock_addr, check_lock_type, do_collect_zone_lock, > do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into > tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF > sample filtering. > > Also add rw_semaphore.count and struct rt_mutex_base to the minimal > vmlinux.h and extend get_lock_owner() to support rt_mutex and check > writer-ownership and handoff flags. > > Signed-off-by: Ian Rogers <irogers@google.com> Sashiko has reviewed this patch and found no issues. -- Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=1 ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter 2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers @ 2026-10-17 5:45 ` Ian Rogers 2026-10-17 6:09 ` sashiko-bot 2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers ` (3 subsequent siblings) 5 siblings, 1 reply; 24+ messages in thread From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Namhyung Kim Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark, linux-perf-users, linux-kernel, rcu Add BPF sample filter predicates lock_held(<lock>), lock_held_read(<lock>) / lock_held_shared(<lock>), and lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R' and ':W' suffixes on <lock>) to allow recording perf event samples only when a specified kernel lock is held (or not held with '!'). Supported <lock> specifications match perf lock contention: - Global/static kernel lock symbols (with or without '&', e.g. lock_held(&cgroup_mutex) or lock_held(cgroup_mutex)) - Hexadecimal kernel lock addresses - Slab cache names prefixed with '&' (e.g. lock_held(&dentry), lock_held_read(&dentry)) - Per-process and semi-global locks: mmap_lock, siglock, rq_lock, zone_lock - Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W, rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W, semaphore Supported lock hold modes by lock family: - Both read/shared and write/exclusive: rwsem (including mmap_lock) and rwlock. - Write/exclusive only: mutex, spinlock (including siglock, rq_lock, zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write() / percpu_up_write(); fast-path percpu_down_read() / percpu_up_read() are inlined per-CPU counter operations without function entry points). When evaluating the filter on sample overflow, perf_sample_filter first inspects the lock's owner field directly in kernel memory where possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls back to a held_locks BPF hash map (tracking separate read_count and write_count) populated by fentry/fexit hooks on kernel lock/unlock functions. To minimize tracing overhead, only the BPF hooks for the requested lock family and mode are autoloaded and attached. Example usage: # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock)' \ -- perf bench mem mmap -t 2 -l 100 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ........ ................. ........................ # 18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms | ---clear_page_erms get_page_from_freelist __alloc_pages_noprof folio_alloc_mpol_noprof vma_alloc_folio_noprof do_anonymous_page __handle_mm_fault handle_mm_fault do_user_addr_fault exc_page_fault asm_exc_page_fault worker_thread Read/shared vs write/exclusive holding can also be filtered separately: # perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held_write(&mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' ... Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 41 +- tools/perf/util/bpf-filter.c | 934 ++++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 46 +- tools/perf/util/bpf_skel/sample-filter.h | 59 ++ tools/perf/util/bpf_skel/sample_filter.bpf.c | 920 ++++++++++++++++++ tools/perf/util/parse-events.c | 3 +- 8 files changed, 2021 insertions(+), 26 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 2351f7b4f5fc..246ae085744d 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -204,7 +204,46 @@ OPTIONS ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr, code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat, p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock, - mem_dtlb, mem_blk, mem_hops, uid, gid + mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>), + lock_held_read(<lock>), lock_held_shared(<lock>), + lock_held_write(<lock>), lock_held_exclusive(<lock>) + + The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters + samples based on whether the specified kernel lock is currently held + by the kernel thread or CPU. To differentiate between shared/read + holding and exclusive/write holding, use 'lock_held_read(<lock>)' + (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' + (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to + <lock> (e.g. 'lock_held(&mmap_lock:R)' or 'lock_held(&mmap_lock:W)'). + <lock> can be a kernel lock symbol name (with or without a leading + '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'), + a hexadecimal lock address, a slab cache name prefixed with '&' + (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known + lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock + type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', + 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', + 'semaphore'). Both read/shared and write/exclusive tracking are + supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; + 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), + 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') + support write/exclusive tracking only. + + Note that 'lock_held(<lock>)' introduces overhead outside of sample + overflow handling: in addition to checking the lock's owner field on + each sample where possible (such as for mutexes, rt-mutexes, and + write-locked rwsems), it attaches BPF fentry/fexit programs to kernel + lock acquire and release functions to track held locks (such as + spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a + BPF map. These BPF hooks run on every kernel lock acquisition and + release for the attached lock family, even when no perf sample is + being taken. To minimize this overhead, only the hooks for the + relevant lock family and mode are attached when known (for example, + 'lock_held_read(&mmap_lock)' only hooks rwsem read functions, + 'lock_held_write(&mmap_lock)' only hooks rwsem write functions, + '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks + spinlock functions), whereas slab filters ('&<slab>'), raw hex + addresses, or symbols without a recognizable lock-type suffix attach + hooks across all lock families (including high-frequency spinlocks). The <operator> can be one of: ==, !=, >, >=, <, <=, & diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 72e83ac1d6a4..35d6f7c63868 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -40,21 +40,35 @@ * The 'dropped' map is to keep how many samples it dropped by the filter and * it will be reported as lost samples. */ +#include <errno.h> +#include <stdio.h> #include <stdlib.h> +#include <stdbool.h> +#include <stddef.h> +#include <string.h> #include <fcntl.h> +#include <unistd.h> #include <sys/ioctl.h> #include <sys/stat.h> #include <bpf/bpf.h> +#include <bpf/btf.h> #include <linux/err.h> +#include <linux/kernel.h> #include <linux/list.h> +#include <linux/string.h> +#include <linux/zalloc.h> #include <api/fs/fs.h> #include <internal/xyarray.h> #include <perf/threadmap.h> +#include <symbol/kallsyms.h> #include "util/cap.h" #include "util/debug.h" #include "util/evsel.h" +#include "util/lock-contention.h" +#include "util/string2.h" +#include "util/symbol_conf.h" #include "util/target.h" #include "util/thread_map.h" #include "util/bpf-utils.h" @@ -63,6 +77,7 @@ #include <util/bpf-filter-flex.h> #include <util/bpf-filter-bison.h> +#include "bpf_skel/lock_data.h" #include "bpf_skel/sample-filter.h" #include "bpf_skel/sample_filter.skel.h" @@ -127,7 +142,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) return 0; - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { + if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID || + expr->term == PBF_TERM_LOCK_HELD) { /* Not dependent on the sample_type as computed from a BPF helper. */ return 0; } @@ -155,9 +171,713 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * return -1; } -static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry) +/* + * Program groups for `lock_held(...)` BPF tracing hooks. + * + * When a `lock_held(...)` filter is specified, only the BPF tracing program + * groups required for the requested lock(s) are autoloaded and attached. + * If no `lock_held(...)` filter is present, none of these programs are loaded. + */ +enum lock_prog_group { + LOCK_PROG_MUTEX = 1U << 0, + LOCK_PROG_RWSEM_READ = 1U << 1, + LOCK_PROG_RWSEM_WRITE = 1U << 2, + LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE, + LOCK_PROG_SPIN = 1U << 3, + LOCK_PROG_RWLOCK_READ = 1U << 4, + LOCK_PROG_RWLOCK_WRITE = 1U << 5, + LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE, + LOCK_PROG_RTMUTEX = 1U << 6, + LOCK_PROG_SEM = 1U << 7, + LOCK_PROG_PERCPU_RWSEM = 1U << 8, + LOCK_PROG_ALL = 0x1ffU, +}; + +static unsigned int lock_flags_to_prog_group(unsigned int flags) +{ + if (flags & LCB_F_MUTEX) + return LOCK_PROG_MUTEX; + if (flags & LCB_F_RT) + return LOCK_PROG_RTMUTEX; + if (flags & LCB_F_PERCPU) + return LOCK_PROG_PERCPU_RWSEM; + if (flags == LCB_F_SPIN) + return LOCK_PROG_SPIN; + if (flags & LCB_F_SPIN) + return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE; + if (flags & LCB_F_READ) + return LOCK_PROG_RWSEM_READ; + if (flags & LCB_F_WRITE) + return LOCK_PROG_RWSEM_WRITE; + return LOCK_PROG_SEM; +} + +struct lock_prog_desc { + size_t prog_offset; + const char *btf_func; + unsigned int group; +}; + +#define LOCK_FPROG(field, func, grp) \ + { offsetof(struct sample_filter_bpf, progs.field), func, grp } + +static const struct lock_prog_desc lock_progs[] = { + /* Mutex */ + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested", + LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX), + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX), + /* RW semaphore */ + LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested", + LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested", + LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM), + LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE), + /* Spinlock */ + LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested", + LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN), + /* RWlock */ + LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore", + LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore", + LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE), + /* RT-Mutex */ + LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible", + LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX), + /* Semaphore */ + LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM), + LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM), + /* Percpu-RWsem */ + LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), + LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), +}; + +#undef LOCK_FPROG + +static int split_lock_name_mode(const char *name, int expr_part, + char *base_buf, size_t buf_sz, + bool *has_amp, unsigned int *mode_out) +{ + const char *bare; + size_t len; + unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK; + unsigned int suffix_mode = 0; + + if (!name || !*name) + return -EINVAL; + + *has_amp = (name[0] == '&'); + bare = *has_amp ? name + 1 : name; + len = strlen(bare); + if (len == 0) + return -EINVAL; + + if (len > 2 && !strcmp(bare + len - 2, ":R")) { + suffix_mode = PBF_LOCK_MODE_READ; + len -= 2; + } else if (len > 2 && !strcmp(bare + len - 2, ":W")) { + suffix_mode = PBF_LOCK_MODE_WRITE; + len -= 2; + } + + if (len == 0 || len >= buf_sz) + return -EINVAL; + + if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY && + suffix_mode && expr_mode != suffix_mode) + return -EINVAL; + + memcpy(base_buf, bare, len); + base_buf[len] = '\0'; + + if (suffix_mode) + *mode_out = suffix_mode; + else if (expr_mode) + *mode_out = expr_mode; + else + *mode_out = PBF_LOCK_MODE_ANY; + + return 0; +} + +/* + * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R", + * "spinlock", "percpu-rwsem"). + * + * A lock type name may correspond to up to two flag combinations (e.g. + * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed + * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is + * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`. + */ +static bool parse_lock_type_name(const char *name, unsigned int mode, + struct sample_filter_bpf *skel, + u64 *encoded_pair, unsigned int *prog_groups) +{ + bool found = false; + u32 flag1 = 0xff, flag2 = 0xff; + unsigned int groups = 0; + const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name; + + for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) { + u32 f = lock_type_table[i].flags; + bool is_read = (f & LCB_F_READ) != 0; + + if (strcmp(lock_type_table[i].lock_name, lookup) && + strcmp(lock_type_table[i].flags_name, name)) + continue; + + /* + * percpu_down_read() / percpu_up_read() fast paths are inlined + * per-CPU counter increments without function entry points, so + * only writer holding (percpu_down_write / percpu_up_write) can + * be tracked for percpu-rwsem. rwlock-rt is also not hooked. + */ + if ((f & LCB_F_PERCPU) && is_read) + continue; + if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE))) + continue; + + if (is_read && !(mode & PBF_LOCK_MODE_READ)) + continue; + if (!is_read && !(mode & PBF_LOCK_MODE_WRITE)) + continue; + + if (!found) + flag1 = f; + else if (f != flag1) + flag2 = f; + if (skel && f < LCB_F_TYPE_MAX) { + skel->bss->has_lock_type = 1; + skel->bss->lock_type_filter[f] = 1; + } + groups |= lock_flags_to_prog_group(f); + found = true; + } + + if (!found) + return false; + + if (encoded_pair) + *encoded_pair = flag1 | ((u64)flag2 << 8); + if (prog_groups) + *prog_groups |= groups; + return true; +} + +struct lock_sym_cb_args { + const char *name; + u64 addr; + bool found; +}; + +static int find_lock_sym_cb(void *arg, const char *name, + char type __maybe_unused, u64 start) +{ + struct lock_sym_cb_args *args = arg; + + if (!strcmp(name, args->name)) { + args->found = true; + args->addr = start; + if (start != 0) + return 1; + } + return 0; +} + +static int find_kernel_lock_sym(const char *name, u64 *addr) { - int i = 0; + struct lock_sym_cb_args args = { + .name = name, + }; + const char *kallsyms_filename = "/proc/kallsyms"; + + if (symbol_conf.kallsyms_name != NULL) + kallsyms_filename = symbol_conf.kallsyms_name; + + if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0) + return -ENOENT; + + if (!args.found) + return -ENOENT; + + if (args.addr == 0) { + pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n", + name); + return -EACCES; + } + + *addr = args.addr; + return 0; +} + +static bool parse_lock_hex_addr(const char *str, u64 *addr) +{ + char *end; + unsigned long long val; + + if (!str || !*str) + return false; + + errno = 0; + val = strtoull(str, &end, 16); + if (*end != '\0' || errno != 0 || val == 0) + return false; + + *addr = val; + return true; +} + +static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf) +{ + u64 addr = 0; + char *buf = NULL, *p; + size_t len; + long last = -1; + int ret; + + if (!btf) + return; + + ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT); + if (ret < 0) + return; + + ret = btf__resolve_size(btf, ret); + if (ret < 0) + return; + + skel->rodata->sizeof_zone = ret; + + if (find_kernel_lock_sym("contig_page_data", &addr) == 0) { + skel->rodata->contig_page_data_addr = addr; + return; + } + + if (find_kernel_lock_sym("node_data", &addr) == 0) + skel->rodata->node_data_addr = addr; + + if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) { + p = buf; + while (p && *p) { + char *end; + + last = strtol(p, &end, 0); + if (end == p) + break; + p = end; + if (*p == ',' || *p == '-' || *p == '\n') + p++; + } + skel->rodata->nr_nodes = last + 1; + free(buf); + } +} + +/* + * Enable or disable autoload for each lock-tracking BPF program in the + * skeleton before `sample_filter_bpf__load()` is called. + * + * Each program is enabled only if its group is in `needed_groups` and its + * target kernel function or tracepoint exists in the running kernel's BTF. + */ +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, + unsigned int needed_groups, bool needs_lock_syms, + bool needs_slab_iter) +{ + unsigned int enabled_progs = 0; + + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) { + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset; + struct bpf_program *prog = *progp; + bool enable = false; + + if ((needed_groups & lock_progs[i].group) && btf && + lock_progs[i].btf_func) { + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func, + BTF_KIND_FUNC) > 0) + enable = true; + } + + if (enable) + enabled_progs++; + bpf_program__set_autoload(prog, enable); + } + + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms); + if (needs_slab_iter && btf && + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) { + bpf_program__set_autoload(skel->progs.slab_cache_iter, true); + } else { + bpf_program__set_autoload(skel->progs.slab_cache_iter, false); + } + + if (needed_groups && !enabled_progs) { + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n"); + return -ENOTSUP; + } + return 0; +} + +/* + * Pre-scan a single filter expression prior to loading the BPF skeleton in + * order to determine which BPF maps need to be sized and which lock tracing + * program groups need to be autoloaded. + */ +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, + bool *has_lock_filter, + unsigned int *prog_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + char base[128]; + bool has_amp; + unsigned int mode; + u64 dummy; + + if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name) + return; + + *has_lock_filter = true; + if (split_lock_name_mode(expr->lock_name, expr->part, base, + sizeof(base), &has_amp, &mode) < 0) + return; + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + if (mode & PBF_LOCK_MODE_READ) + *prog_groups |= LOCK_PROG_RWSEM_READ; + if (mode & PBF_LOCK_MODE_WRITE) + *prog_groups |= LOCK_PROG_RWSEM_WRITE; + return; + } + if (!strcmp(base, "siglock")) { + *prog_groups |= LOCK_PROG_SPIN; + return; + } + if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) { + *prog_groups |= LOCK_PROG_SPIN; + *needs_lock_syms = true; + return; + } + if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, prog_groups)) + return; + + if (!has_amp && parse_lock_hex_addr(base, &dummy)) { + *prog_groups |= LOCK_PROG_ALL; + return; + } + + if (has_amp) + *needs_slab_iter = true; + + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) { + *prog_groups |= LOCK_PROG_RTMUTEX; + } else if (strtailcmp(base, "mutex") == 0) { + *prog_groups |= LOCK_PROG_MUTEX; + } else if (strtailcmp(base, "percpu_rwsem") == 0 || + strtailcmp(base, "pcpu_rwsem") == 0) { + *prog_groups |= LOCK_PROG_PERCPU_RWSEM; + } else if (strtailcmp(base, "rwsem") == 0) { + if (mode & PBF_LOCK_MODE_READ) + *prog_groups |= LOCK_PROG_RWSEM_READ; + if (mode & PBF_LOCK_MODE_WRITE) + *prog_groups |= LOCK_PROG_RWSEM_WRITE; + } else if (strtailcmp(base, "spinlock") == 0) { + *prog_groups |= LOCK_PROG_SPIN; + } else if (strtailcmp(base, "rwlock") == 0) { + if (mode & PBF_LOCK_MODE_READ) + *prog_groups |= LOCK_PROG_RWLOCK_READ; + if (mode & PBF_LOCK_MODE_WRITE) + *prog_groups |= LOCK_PROG_RWLOCK_WRITE; + } else { + *prog_groups |= LOCK_PROG_ALL; + } + + if (!has_amp && find_kernel_lock_sym(base, &dummy) != 0) + *needs_slab_iter = true; +} + +static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter, + unsigned int *prog_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + struct perf_bpf_filter_expr *expr, *group; + + *has_lock_filter = false; + *prog_groups = 0; + *needs_lock_syms = false; + *needs_slab_iter = false; + + list_for_each_entry(expr, &evsel->bpf_filters, list) { + prescan_one_lock_expr(expr, has_lock_filter, prog_groups, + needs_lock_syms, needs_slab_iter); + if (expr->op == PBF_OP_GROUP_BEGIN) { + list_for_each_entry(group, &expr->groups, list) { + prescan_one_lock_expr(group, has_lock_filter, + prog_groups, needs_lock_syms, + needs_slab_iter); + } + } + } +} + +static void run_slab_cache_iter(struct sample_filter_bpf *skel) +{ + struct bpf_link *link; + int fd; + char buf[256]; + + if (!bpf_program__autoload(skel->progs.slab_cache_iter)) + return; + + link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL); + if (!link || IS_ERR(link)) { + pr_debug("cannot attach slab cache iter\n"); + return; + } + + fd = bpf_iter_create(bpf_link__fd(link)); + if (fd >= 0) { + while (read(fd, buf, sizeof(buf)) > 0) + continue; + close(fd); + } + + bpf_link__destroy(link); +} + +static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, + u64 *kmem_cache_addr) +{ + int cache_fd; + u64 key, *prev_key = NULL; + + if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter)) + return false; + + cache_fd = bpf_map__fd(skel->maps.slab_caches); + while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) { + struct slab_cache_data data; + + if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0) + break; + + if (!strcmp(data.name, name)) { + *kmem_cache_addr = key; + return true; + } + prev_key = &key; + } + return false; +} + +/* + * Resolve a `lock_held(<name>)` filter expression into `entry->part` + * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate + * the corresponding BPF filter maps and BSS control flags. + * + * Resolution order: + * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock` (with or without leading '&'). + * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`, + * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, + * `semaphore`, etc. + * 3. Hexadecimal lock address (when no leading '&'). + * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`). + * 5. Global/static kernel lock symbol in kallsyms (with or without leading '&', + * e.g. `&cgroup_mutex` or `cgroup_mutex`). + * 6. Slab cache name without leading '&' fallback. + */ +static int resolve_lock_expr(struct sample_filter_bpf *skel, + struct perf_bpf_filter_expr *expr, + struct perf_bpf_filter_entry *entry) +{ + const char *name = expr->lock_name; + char base[128]; + bool has_amp; + unsigned int mode; + u64 addr = 0, encoded_pair = 0; + u8 one = 1; + int ret; + + if (split_lock_name_mode(name, expr->part, base, sizeof(base), + &has_amp, &mode) < 0) { + pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: ""); + return -EINVAL; + } + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + entry->part = PBF_LOCK_MMAP | mode; + entry->target = 0; + if (skel) + skel->bss->has_mmap_lock = 1; + return 0; + } + + if (!strcmp(base, "siglock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_SIGHAND | mode; + entry->target = 0; + if (skel) + skel->bss->has_siglock = 1; + return 0; + } + + if (!strcmp(base, "rq_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->target = LOCK_CLASS_RQLOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!strcmp(base, "zone_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->target = LOCK_CLASS_ZONE_LOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { + entry->part = PBF_LOCK_TYPE | mode; + entry->target = encoded_pair; + return 0; + } + + if (!has_amp && parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) { + pr_err("Error: Lock type '%s' does not support the requested read/write mode\n", + name); + return -EINVAL; + } + + if (!has_amp && parse_lock_hex_addr(base, &addr)) { + entry->part = PBF_LOCK_ADDR | mode; + entry->target = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + + if (has_amp && find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->target = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + + ret = find_kernel_lock_sym(base, &addr); + if (ret == 0) { + entry->part = PBF_LOCK_ADDR | mode; + entry->target = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + if (ret == -EACCES) + return ret; + + if (!has_amp && find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->target = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + + pr_err("Error: Cannot resolve kernel lock '%s'\n", name); + return -EINVAL; +} + +static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, + struct perf_bpf_filter_entry *entry) +{ + int i = 0, ret; struct perf_bpf_filter_expr *expr; list_for_each_entry(expr, &evsel->bpf_filters, list) { @@ -171,6 +891,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = expr->part; entry[i].term = expr->term; entry[i].value = expr->val; + entry[i].target = 0; + if (expr->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, expr, &entry[i]); + if (ret < 0) + return ret; + } i++; if (expr->op == PBF_OP_GROUP_BEGIN) { @@ -184,6 +910,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = group->part; entry[i].term = group->term; entry[i].value = group->val; + entry[i].target = 0; + if (group->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, group, &entry[i]); + if (ret < 0) + return ret; + } i++; } @@ -423,10 +1155,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) { int i, x, y, fd, ret; struct sample_filter_bpf *skel = NULL; + struct btf *btf = NULL; struct bpf_program *prog; struct bpf_link *link; struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); + bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false; + unsigned int prog_groups = 0; #if LIBBPF_CURRENT_VERSION_GEQ(1, 7) DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts, .dont_enable = true); @@ -438,15 +1173,24 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (entry == NULL) return -1; - ret = get_filter_entries(evsel, entry); - if (ret < 0) { - pr_err("Failed to process filter entries\n"); - goto err; - } + prescan_lock_filters(evsel, &has_lock_filter, &prog_groups, + &needs_lock_syms, &needs_slab_iter); - if (needs_idx_hash && geteuid() != 0) { + if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { int zero = 0; + if (has_lock_filter) { + pr_err("Error: lock_held BPF filter requires CAP_BPF capability\n"); + ret = -EPERM; + goto err; + } + + ret = get_filter_entries(evsel, NULL, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + /* The filters map is shared among other processes */ ret = create_idx_hash(evsel, entry); if (ret < 0) @@ -484,13 +1228,69 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return 0; } - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { + ret = -errno; + pr_err("Failed to open perf sample-filter BPF skeleton\n"); + goto err; + } + + if (has_lock_filter) { + btf = btf__load_vmlinux_btf(); + bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES); + bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS); + if (needs_slab_iter) { + bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS); + bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES); + } + if (needs_lock_syms) { + bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); + init_numa_data(skel, btf); + } + ret = configure_lock_progs(skel, btf, prog_groups, needs_lock_syms, + needs_slab_iter); + if (ret < 0) + goto err; + } else { + configure_lock_progs(skel, NULL, 0, false, false); + } + + if (sample_filter_bpf__load(skel) < 0) { ret = -errno; pr_err("Failed to load perf sample-filter BPF skeleton\n"); goto err; } + if (has_lock_filter && needs_slab_iter) + run_slab_cache_iter(skel); + + ret = get_filter_entries(evsel, skel, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + + if (has_lock_filter) { + if (needs_lock_syms && skel->bss->has_lock_class) { + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts, + .flags = BPF_F_TEST_RUN_ON_CPU, + ); + int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms); + + bpf_prog_test_run_opts(prog_fd, &opts); + } + bpf_program__set_autoattach(skel->progs.perf_sample_filter, false); + bpf_program__set_autoattach(skel->progs.collect_lock_syms, false); + bpf_program__set_autoattach(skel->progs.slab_cache_iter, false); + + ret = sample_filter_bpf__attach(skel); + if (ret < 0) { + pr_err("Failed to attach lock sample-filter BPF programs\n"); + goto err; + } + skel->bss->lock_enabled = 1; + } + i = 0; fd = bpf_map__fd(skel->maps.filters); @@ -513,11 +1313,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) } } } + btf__free(btf); free(entry); evsel->bpf_skel = skel; return 0; err: + btf__free(btf); free(entry); if (!list_empty(&pinned_filters)) { struct pinned_filter_idx *pfi, *tmp; @@ -532,6 +1334,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return ret; } +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr) +{ + struct perf_bpf_filter_expr *group, *tmp; + + if (!expr) + return; + + list_for_each_entry_safe(group, tmp, &expr->groups, list) { + list_del(&group->list); + perf_bpf_filter_expr__delete(group); + } + zfree(&expr->lock_name); + free(expr); +} + int perf_bpf_filter__destroy(struct evsel *evsel) { struct perf_bpf_filter_expr *expr, *tmp; @@ -539,7 +1356,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel) list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) { list_del(&expr->list); - free(expr); + perf_bpf_filter_expr__delete(expr); } sample_filter_bpf__destroy(evsel->bpf_skel); @@ -591,7 +1408,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term { struct perf_bpf_filter_expr *expr; - expr = malloc(sizeof(*expr)); + expr = zalloc(sizeof(*expr)); if (expr != NULL) { expr->term = term; expr->part = part; @@ -602,6 +1419,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term return expr; } +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val) +{ + struct perf_bpf_filter_expr *expr; + + if (!lock_name || !*lock_name) + return NULL; + + expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val); + if (!expr) + return NULL; + + expr->lock_name = strdup(lock_name); + if (!expr->lock_name) { + free(expr); + return NULL; + } + return expr; +} + static bool check_bpf_filter_capable(void) { int fd; @@ -622,14 +1461,11 @@ static bool check_bpf_filter_capable(void) return false; } -int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str) { YY_BUFFER_STATE buffer; int ret; - if (!check_bpf_filter_capable()) - return -EPERM; - buffer = perf_bpf_filter__scan_string(str); ret = perf_bpf_filter_parse(expr_head); @@ -641,6 +1477,53 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) return ret; } +int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +{ + if (!check_bpf_filter_capable()) + return -EPERM; + + return __perf_bpf_filter__parse(expr_head, str); +} + +static int pin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret; + + ret = bpf_object__pin_maps(skel->obj, path); + if (ret < 0) + return ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path); + if (ret < 0) + bpf_object__unpin_maps(skel->obj, path); + + free(prog_path); + return ret; +} + +static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret, map_ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path); + free(prog_path); + + map_ret = bpf_object__unpin_maps(skel->obj, path); + return ret < 0 ? ret : map_ret; +} + int perf_bpf_filter__pin(void) { struct sample_filter_bpf *skel; @@ -654,6 +1537,8 @@ int perf_bpf_filter__pin(void) goto err; } + configure_lock_progs(skel, NULL, 0, false, false); + /* pinned program will use pid-hash */ bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); @@ -674,7 +1559,7 @@ int perf_bpf_filter__pin(void) goto err; } - ret = bpf_object__pin(skel->obj, path); + ret = pin_filter_object(skel, path); if (ret < 0) { pr_err("Failed to pin BPF filter objects\n"); goto err; @@ -683,7 +1568,7 @@ int perf_bpf_filter__pin(void) /* setup access permissions for the pinned objects */ dir_fd = open(path, O_PATH); if (dir_fd < 0) { - bpf_object__unpin(skel->obj, path); + unpin_filter_object(skel, path); ret = dir_fd; goto err; } @@ -727,6 +1612,8 @@ int perf_bpf_filter__pin(void) err_close: close(dir_fd); + if (ret < 0) + unpin_filter_object(skel, path); err: free(path); @@ -740,13 +1627,20 @@ int perf_bpf_filter__unpin(void) char *path = NULL; int ret = -1; - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { ret = -errno; pr_err("Failed to open perf sample-filter BPF skeleton\n"); goto err; } + configure_lock_progs(skel, NULL, 0, false, false); + if (sample_filter_bpf__load(skel) < 0) { + ret = -errno; + pr_err("Failed to load perf sample-filter BPF skeleton\n"); + goto err; + } + if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(), PERF_BPF_FILTER_PIN_PATH) < 0) { ret = -errno; @@ -754,7 +1648,7 @@ int perf_bpf_filter__unpin(void) goto err; } - ret = bpf_object__unpin(skel->obj, path); + ret = unpin_filter_object(skel, path); err: free(path); diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h index 818c554b91b2..fda95ff226b3 100644 --- a/tools/perf/util/bpf-filter.h +++ b/tools/perf/util/bpf-filter.h @@ -14,6 +14,13 @@ struct perf_bpf_filter_expr { int part; enum perf_bpf_filter_term term; unsigned long val; + /* + * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions + * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R"). + * Resolved into `entry.part` and `entry.target` during + * perf_bpf_filter__prepare(). + */ + char *lock_name; }; struct evsel; @@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term int part, enum perf_bpf_filter_op op, unsigned long val); +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val); +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr); +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target); int perf_bpf_filter__destroy(struct evsel *evsel); diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l index 1be9df6550fc..b4022110f1e5 100644 --- a/tools/perf/util/bpf-filter.l +++ b/tools/perf/util/bpf-filter.l @@ -1,9 +1,12 @@ %option prefix="perf_bpf_filter_" %option noyywrap +%x LOCK_ARG + %{ #include <stdio.h> #include <stdlib.h> +#include <string.h> #include <linux/perf_event.h> #include "bpf-filter.h" @@ -76,7 +79,7 @@ static int path_or_error(void) num_dec [0-9]+ num_hex 0[Xx][0-9a-fA-F]+ space [ \t]+ -path [^ \t\n,]+ +path [^ \t\n,()!=>&|<]+ ident [_a-zA-Z][_a-zA-Z0-9]+ %% @@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); } uid { return sample(PBF_TERM_UID); } gid { return sample(PBF_TERM_GID); } cgroup { return sample_path(PBF_TERM_CGROUP); } - +lock_held[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY; + return BFT_LOCK_HELD; + } +(lock_held_read|lock_held_shared)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ; + return BFT_LOCK_HELD; + } +(lock_held_write|lock_held_exclusive)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE; + return BFT_LOCK_HELD; + } + +<LOCK_ARG>{space} { } +<LOCK_ARG>[^ \t\n()]+ { + perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text); + return BFT_LOCK_NAME; + } +<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; } +<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; } + +"!" { return '!'; } "==" { return operator(PBF_OP_EQ); } "!=" { return operator(PBF_OP_NEQ); } ">" { return operator(PBF_OP_GT); } diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y index 5a79a8e7a45b..6ee6edef13f7 100644 --- a/tools/perf/util/bpf-filter.y +++ b/tools/perf/util/bpf-filter.y @@ -11,6 +11,7 @@ #include <string.h> #include <linux/compiler.h> #include <linux/list.h> +#include <linux/zalloc.h> #include "bpf-filter.h" #include "cgroup.h" @@ -31,6 +32,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, { unsigned long num; char *path; + char *lock_name; + struct { + char *name; + int mode; + } lock; struct { enum perf_bpf_filter_term term; int part; @@ -40,12 +46,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, } %token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH +%token BFT_LOCK_HELD BFT_LOCK_NAME %type <expr> filter_term filter_expr -%destructor { free ($$); } <expr> +%destructor { perf_bpf_filter_expr__delete($$); } <expr> +%destructor { free($$); } <lock_name> +%destructor { free($$.name); } <lock> %type <sample> BFT_SAMPLE BFT_SAMPLE_PATH %type <op> BFT_OP -%type <num> BFT_NUM +%type <num> BFT_NUM BFT_LOCK_HELD %type <path> BFT_PATH +%type <lock_name> BFT_LOCK_NAME +%type <lock> lock_term %% @@ -105,5 +116,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH $$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id); cgroup__put(cgrp); } +| +lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1); + zfree(&$1.name); + if (!$$) + YYERROR; +} +| +'!' lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0); + zfree(&$2.name); + if (!$$) + YYERROR; +} +| +lock_term BFT_OP BFT_NUM +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3); + zfree(&$1.name); + if (!$$) + YYERROR; +} + +lock_term: +BFT_LOCK_HELD BFT_LOCK_NAME ')' +{ + $$.name = $2; + $$.mode = $1; +} %% diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index 683fec85e71e..cd711d1d04b6 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -53,14 +53,73 @@ enum perf_bpf_filter_term { /* Terms computed from BPF helpers. */ PBF_TERM_UID, PBF_TERM_GID, + /* + * Kernel lock held predicate (`lock_held(...)`). Evaluates to 1 if the + * specified kernel lock is currently held by the running task/CPU, or + * 0 otherwise. The lock specification kind is stored in `entry.part` + * (`enum perf_bpf_lock_target_kind`) and the resolved target address, + * class ID, or type flag encoding is stored in `entry.target`. + */ + PBF_TERM_LOCK_HELD, }; +/* + * Lock target categories for PBF_TERM_LOCK_HELD filters. + * Stored in `perf_bpf_filter_entry.part`. + */ +enum perf_bpf_lock_target_kind { + PBF_LOCK_NONE = 0, + /* + * Specific global or static lock address (resolved from kallsyms or + * given as a hex address). `entry.target` holds the kernel address. + */ + PBF_LOCK_ADDR, + /* + * Dynamic lock allocated inside a slab cache (`&slab_name`). + * `entry.target` holds the kernel `struct kmem_cache *` address. + */ + PBF_LOCK_SLAB, + /* + * Per-process `current->mm->mmap_lock` (or `mmap_sem`). + */ + PBF_LOCK_MMAP, + /* + * Per-process `current->sighand->siglock`. + */ + PBF_LOCK_SIGHAND, + /* + * Semi-global lock class (`rq_lock` or `zone_lock`). + * `entry.target` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or + * LOCK_CLASS_ZONE_LOCK). + */ + PBF_LOCK_CLASS, + /* + * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.). + * `entry.target` packs up to two `LCB_F_*` flag combinations in the + * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). + */ + PBF_LOCK_TYPE, +}; + +/* + * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD: + * Bits 0..3: `enum perf_bpf_lock_target_kind` + * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding) + * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding) + */ +#define PBF_LOCK_KIND_MASK 0x0fU +#define PBF_LOCK_MODE_READ (1U << 4) +#define PBF_LOCK_MODE_WRITE (1U << 5) +#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE) +#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY + /* BPF map entry for filtering */ struct perf_bpf_filter_entry { enum perf_bpf_filter_op op; __u32 part; /* sub-sample type info when it has multiple values */ enum perf_bpf_filter_term term; __u64 value; + __u64 target; /* lock address, kmem_cache *, class, or type flags */ }; struct idx_hash_key { diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index e5666d4c1722..fbd2e51815d7 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -5,8 +5,47 @@ #include <bpf/bpf_tracing.h> #include <bpf/bpf_core_read.h> +#include "lock_data.h" #include "sample-filter.h" +/* + * Key for the `held_locks` BPF hash map tracking currently held kernel locks. + * + * - `addr_or_id`: + * - PBF_LOCK_ADDR: kernel virtual address of the lock instance + * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK) + * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address + * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags + * - `owner_id`: + * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`), + * since the holder may sleep and migrate across CPUs while holding the lock. + * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks + * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption + * is disabled on the holding CPU and spinlocks such as `rq_lock` may be + * acquired by one task and released by the next task across `context_switch()`. + * - `kind`: `enum perf_bpf_lock_target_kind`. + */ +struct held_lock_key { + __u64 addr_or_id; + __u32 owner_id; + __u32 kind; +}; + +/* + * Value for `held_locks` entries. + * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`) + * and exclusive/write acquisitions (including nested class/slab/type + * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of + * the lock so release tracepoints that do not carry flags (such as + * `lock:contended_release`) can decrement matching class/slab/type entries. + */ +struct held_lock_val { + __u32 read_count; + __u32 write_count; + __u32 flags; + __u32 pad; +}; + /* BPF map that will be filled by user space */ struct filters { __uint(type, BPF_MAP_TYPE_HASH); @@ -43,10 +82,198 @@ struct lost_count { __uint(max_entries, 1); } dropped SEC(".maps"); +/* Currently held kernel locks tracked by fentry/fexit and lock tracepoints. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, struct held_lock_key); + __type(value, struct held_lock_val); + __uint(max_entries, 1); +} held_locks SEC(".maps"); + +/* Target lock addresses for PBF_LOCK_ADDR filters. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u64); + __type(value, __u8); + __uint(max_entries, 1); +} lock_addr_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, 1); +} lock_syms SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u8)); + __uint(max_entries, 1); +} slab_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(struct slab_cache_data)); + __uint(max_entries, 1); +} slab_caches SEC(".maps"); + volatile const int use_idx_hash; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + +__u32 slab_cache_id; + +/* Control flags set by userspace when lock_held(...) filters are active. */ +int lock_enabled; +int has_lock_addr; +int has_lock_slab; +int has_lock_class; +int has_lock_type; +int has_mmap_lock; +int has_siglock; +__u8 lock_type_filter[LCB_F_TYPE_MAX]; + void *bpf_cast_to_kern_ctx(void *) __ksym; +/* + * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`. + * + * For locks that maintain a task_struct pointer in their `owner` field + * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`), + * `perf_sample_filter` can inspect the lock structure directly in kernel memory + * and compare the owner against `current` (`bpf_get_current_task()`). + * + * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear + * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and + * spinlocks are checked via the `held_locks` BPF map instead. + */ +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr) +{ + if (!curr) + return false; + + return (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr || + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr || + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr; +} + +static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode) +{ + if (!val) + return false; + if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0) + return true; + if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0) + return true; + return false; +} + +static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind, + __u32 mode, __u32 pid, __u32 cpu_id) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = pid, + .kind = kind, + }; + struct held_lock_val *val; + + if (pid) { + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + } + + key.owner_id = cpu_id; + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + + return 0; +} + +static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode, + __u32 pid, __u32 cpu_id) +{ + __u32 flag1 = target & 0xff; + __u32 flag2 = (target >> 8) & 0xff; + + if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + return 0; +} + +static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry) +{ + __u64 curr; + __u32 pid; + __u32 cpu_id; + __u32 kind; + __u32 mode; + __u64 lock_addr; + + /* + * Pinned filters (use_idx_hash != 0) share a world-writable filters + * map and do not support lock_held(). Reject evaluation when pinned + * or when lock_enabled is not set so unprivileged users cannot probe + * arbitrary kernel addresses via entry->target. + */ + if (use_idx_hash || !lock_enabled) + return 0; + + curr = bpf_get_current_task(); + pid = (__u32)bpf_get_current_pid_tgid(); + cpu_id = 0x80000000U | bpf_get_smp_processor_id(); + kind = entry->part & PBF_LOCK_KIND_MASK; + mode = entry->part & PBF_LOCK_MODE_MASK; + + if (!mode) + mode = PBF_LOCK_MODE_ANY; + + switch (kind) { + case PBF_LOCK_ADDR: + lock_addr = entry->target; + if ((mode & PBF_LOCK_MODE_WRITE) && + is_direct_lock_owner(lock_addr, curr)) + return 1; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_MMAP: + lock_addr = get_current_mmap_lock(curr); + if (!lock_addr) + return 0; + if ((mode & PBF_LOCK_MODE_WRITE) && + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr) + return 1; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_SIGHAND: + lock_addr = get_current_siglock(curr); + if (!lock_addr) + return 0; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_CLASS: + return is_lock_in_map(entry->target, PBF_LOCK_CLASS, mode, pid, cpu_id); + + case PBF_LOCK_SLAB: + return is_lock_in_map(entry->target, PBF_LOCK_SLAB, mode, pid, cpu_id); + + case PBF_LOCK_TYPE: + return is_lock_type_in_map(entry->target, mode, pid, cpu_id); + + default: + return 0; + } +} + /* new kernel perf_sample_data definition */ struct perf_sample_data___new { __u64 sample_flags; @@ -175,6 +402,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, return bpf_get_current_uid_gid() & 0xFFFFFFFF; case PBF_TERM_GID: return bpf_get_current_uid_gid() >> 32; + case PBF_TERM_LOCK_HELD: + return check_lock_held(entry); case PBF_TERM_NONE: case __PBF_UNUSED_TERM4: case __PBF_UNUSED_TERM5: @@ -295,4 +524,695 @@ int perf_sample_filter(void *ctx) return 0; } +static __always_inline __u32 get_lock_owner_id(__u32 flags) +{ + __u32 owner_id; + + if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) + return 0x80000000U | bpf_get_smp_processor_id(); + + owner_id = (__u32)bpf_get_current_pid_tgid(); + if (!owner_id) + owner_id = 0x80000000U | bpf_get_smp_processor_id(); + return owner_id; +} + +static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id, + __u32 kind, __u32 flags, bool acquired) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = owner_id, + .kind = kind, + }; + struct held_lock_val *val; + bool is_read = (flags & LCB_F_READ) != 0; + + val = bpf_map_lookup_elem(&held_locks, &key); + if (acquired) { + if (val) { + if (is_read) + __sync_fetch_and_add(&val->read_count, 1); + else + __sync_fetch_and_add(&val->write_count, 1); + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags, + }; + + bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY); + } + } else if (val) { + if (is_read) { + if (val->read_count <= 1 && val->write_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->read_count > 0) + __sync_fetch_and_add(&val->read_count, -1); + } else { + if (val->write_count <= 1 && val->read_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->write_count > 0) + __sync_fetch_and_add(&val->write_count, -1); + } + } +} + +static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id) +{ + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + bool matched = false; + bool is_read = (flags & LCB_F_READ) != 0; + __u32 class_id = 0; + __u64 kmem_cache_addr = 0; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool match_type = false; + + if (!lock_enabled || !lock_addr) + return 0; + + if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) + matched = true; + + if (has_mmap_lock || has_siglock) { + __u32 ltype = check_lock_type(lock_addr, flags); + + if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || + (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + matched = true; + } + + if (has_lock_class && flags == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) { + class_id = *cid; + matched = true; + } + } + + if (has_lock_slab && bpf_get_kmem_cache) { + kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) + matched = true; + else + kmem_cache_addr = 0; + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) { + match_type = true; + matched = true; + } + + if (!matched) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (existing) { + /* Non-reader locks are not recursive; avoid double-counting */ + if (!is_read && existing->write_count > 0) + return 0; + if (is_read) + __sync_fetch_and_add(&existing->read_count, 1); + else + __sync_fetch_and_add(&existing->write_count, 1); + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags, + }; + + if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0) + return 0; + } + + if (class_id) + update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, flags, true); + if (kmem_cache_addr) { + update_held_lock_count((__u64)kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, flags, true); + } + if (match_type) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, true); + + return 0; +} + +static __always_inline int record_lock_acquired(void *ctx, __u32 flags) +{ + return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags)); +} + +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags) +{ + __u32 owner_id = get_lock_owner_id(flags); + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool is_read = (flags & LCB_F_READ) != 0; + + if (!lock_enabled || !lock_addr) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (!existing) + return 0; + + if (is_read) { + if (existing->read_count <= 1 && existing->write_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else if (existing->read_count > 0) + __sync_fetch_and_add(&existing->read_count, -1); + } else { + if (existing->write_count == 0 && existing->read_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else if (existing->read_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else + existing->write_count = 0; + } + + if (has_lock_class && flags == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) + update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false); + } + + if (has_lock_slab && bpf_get_kmem_cache) { + __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) { + update_held_lock_count(kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, flags, false); + } + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false); + + return 0; +} + +static __always_inline int record_lock_released(void *ctx, __u32 flags) +{ + return record_lock_released_addr(get_lock_addr(ctx), flags); +} + +/* Mutex acquire/release hooks */ +SEC("fexit/mutex_lock") +int BPF_PROG(fexit_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_nested") +int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io") +int BPF_PROG(fexit_mutex_lock_io, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io_nested") +int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_interruptible") +int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_interruptible_nested") +int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_killable") +int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_trylock") +int BPF_PROG(fexit_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fentry/mutex_unlock") +int BPF_PROG(fentry_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_MUTEX); +} + +/* RW semaphore acquire/release hooks */ +SEC("fexit/down_read") +int BPF_PROG(fexit_down_read, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_nested") +int BPF_PROG(fexit_down_read_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_interruptible") +int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable") +int BPF_PROG(fexit_down_read_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable_nested") +int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_trylock") +int BPF_PROG(fexit_down_read_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_write") +int BPF_PROG(fexit_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_nested") +int BPF_PROG(fexit_down_write_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_killable") +int BPF_PROG(fexit_down_write_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_killable_nested") +int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_trylock") +int BPF_PROG(fexit_down_write_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/downgrade_write") +int BPF_PROG(fexit_downgrade_write, void *sem) +{ + record_lock_acquired(ctx, LCB_F_READ); + return record_lock_released(ctx, LCB_F_WRITE); +} + +SEC("fentry/up_read") +int BPF_PROG(fentry_up_read, void *sem) +{ + return record_lock_released(ctx, LCB_F_READ); +} + +SEC("fentry/up_write") +int BPF_PROG(fentry_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_WRITE); +} + +/* Spinlock acquire/release hooks */ +SEC("fexit/_raw_spin_lock") +int BPF_PROG(fexit_raw_spin_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nested") +int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq") +int BPF_PROG(fexit_raw_spin_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq_disable") +int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave") +int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave_nested") +int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_bh") +int BPF_PROG(fexit_raw_spin_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_trylock") +int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fexit/_raw_spin_trylock_bh") +int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fentry/_raw_spin_unlock") +int BPF_PROG(fentry_raw_spin_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq") +int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq_enable") +int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irqrestore") +int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_bh") +int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +/* RWlock acquire/release hooks */ +SEC("fexit/_raw_read_lock") +int BPF_PROG(fexit_raw_read_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_irq") +int BPF_PROG(fexit_raw_read_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_irqsave") +int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_bh") +int BPF_PROG(fexit_raw_read_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_trylock") +int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); + return 0; +} + +SEC("fentry/_raw_read_unlock") +int BPF_PROG(fentry_raw_read_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irq") +int BPF_PROG(fentry_raw_read_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irqrestore") +int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_bh") +int BPF_PROG(fentry_raw_read_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_write_lock") +int BPF_PROG(fexit_raw_write_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_nested") +int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irq") +int BPF_PROG(fexit_raw_write_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irqsave") +int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_bh") +int BPF_PROG(fexit_raw_write_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_trylock") +int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); + return 0; +} + +SEC("fentry/_raw_write_unlock") +int BPF_PROG(fentry_raw_write_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irq") +int BPF_PROG(fentry_raw_write_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irqrestore") +int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_bh") +int BPF_PROG(fentry_raw_write_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +/* RT-Mutex acquire/release hooks */ +SEC("fexit/rt_mutex_lock") +int BPF_PROG(fexit_rt_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_nested") +int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_interruptible") +int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_lock_killable") +int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_trylock") +int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fentry/rt_mutex_unlock") +int BPF_PROG(fentry_rt_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_RT); +} + +/* Semaphore acquire/release hooks */ +SEC("fexit/down") +int BPF_PROG(fexit_down, void *sem) +{ + return record_lock_acquired(ctx, 0); +} + +SEC("fexit/down_interruptible") +int BPF_PROG(fexit_down_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_killable") +int BPF_PROG(fexit_down_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_timeout") +int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_trylock") +int BPF_PROG(fexit_down_trylock, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fentry/up") +int BPF_PROG(fentry_up, void *sem) +{ + return record_lock_released(ctx, 0); +} + +/* Percpu-rwsem write acquire/release hooks */ +SEC("fexit/percpu_down_write") +int BPF_PROG(fexit_percpu_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("fentry/percpu_up_write") +int BPF_PROG(fentry_percpu_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("raw_tp/bpf_test_finish") +int BPF_PROG(collect_lock_syms) +{ + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); +} + +SEC("iter/kmem_cache") +int slab_cache_iter(void *ctx) +{ + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c index 2f3317ce17f0..1fcb424b75c0 100644 --- a/tools/perf/util/parse-events.c +++ b/tools/perf/util/parse-events.c @@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist, /* Will a tracepoint filter work for str or should a BPF filter be used? */ static bool is_possible_tp_filter(const char *str) { - return strstr(str, "uid") == NULL; + return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL && + strstr(str, "lock_held") == NULL; } static int set_filter(struct evsel *evsel, const void *arg) -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter 2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers @ 2026-10-17 6:09 ` sashiko-bot 0 siblings, 0 replies; 24+ messages in thread From: sashiko-bot @ 2026-10-17 6:09 UTC (permalink / raw) To: Ian Rogers; +Cc: linux-perf-users Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider: - [Medium] perf bpf-filter: held_locks leak and count corruption in fexit_downgrade_write() - [Medium] perf bpf-filter: prescan_one_lock_expr() misses percpu_rw_semaphore symbols - [Medium] perf bpf-filter: configure_lock_progs() ignores missing inlined spin/rwlock unlocks - [Medium] perf bpf-filter: missing CONFIG_DEBUG_LOCK_ALLOC mutex hooks in lock_progs - [Medium] perf bpf-filter: unguarded mutex->owner read in is_direct_lock_owner() on RT -- commit 362a7c2a8d1d1d0355b0ac1b022fdffe73aae6c0 Author: Ian Rogers <irogers@google.com> perf bpf-filter: Add lock_held() sample filter This commit adds BPF sample filter predicates lock_held(), lock_held_read(), lock_held_shared(), lock_held_write(), and lock_held_exclusive() to record perf event samples only when a specified kernel lock is held. It uses direct kernel lock owner inspection where possible and falls back to a held_locks BPF hash map populated by fentry/fexit hooks on kernel lock and unlock functions. > diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c > index 72e83ac1d6a4b..35d6f7c638684 100644 > --- a/tools/perf/util/bpf-filter.c > +++ b/tools/perf/util/bpf-filter.c [ ... ] > +static const struct lock_prog_desc lock_progs[] = { > + /* Mutex */ > + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested", > + LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX), > + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX), [Severity: Medium] On kernels built with CONFIG_DEBUG_LOCK_ALLOC=y, will mutex acquisitions via mutex_lock_killable(), mutex_trylock(), and mutex_lock_nest_lock() be missed by the held_locks map? In include/linux/mutex.h, when CONFIG_DEBUG_LOCK_ALLOC=y, those functions are macros that expand to different underlying entry points: #define mutex_lock(lock) mutex_lock_nested(lock, 0) #define mutex_lock_interruptible(lock) mutex_lock_interruptible_nested(lock, 0) #define mutex_lock_killable(lock) _mutex_lock_killable(lock, 0, NULL) #define mutex_lock_io(lock) mutex_lock_io_nested(lock, 0) #define mutex_lock_nest_lock(lock, nest_lock) _mutex_lock_nest_lock(lock, &(nest_lock)->dep_map) #define mutex_trylock(lock) _mutex_trylock_nest_lock(lock, NULL) While lock_progs[] and sample_filter.bpf.c hook mutex_lock_nested, mutex_lock_interruptible_nested, and mutex_lock_io_nested, they only hook mutex_lock_killable and mutex_trylock rather than _mutex_lock_killable, _mutex_trylock_nest_lock, and _mutex_lock_nest_lock. For filters that rely on held_locks rather than direct owner inspection (such as PBF_LOCK_SLAB and PBF_LOCK_TYPE), wouldn't mutexes acquired via mutex_lock_killable() or mutex_trylock() go untracked on lockdep-enabled kernels? [ ... ] > +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, > + unsigned int needed_groups, bool needs_lock_syms, > + bool needs_slab_iter) > +{ > + unsigned int enabled_progs = 0; > + > + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) { > + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset; > + struct bpf_program *prog = *progp; > + bool enable = false; > + > + if ((needed_groups & lock_progs[i].group) && btf && > + lock_progs[i].btf_func) { > + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func, > + BTF_KIND_FUNC) > 0) > + enable = true; > + } > + > + if (enable) > + enabled_progs++; > + bpf_program__set_autoload(prog, enable); > + } > + > + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms); > + if (needs_slab_iter && btf && > + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) { > + bpf_program__set_autoload(skel->progs.slab_cache_iter, true); > + } else { > + bpf_program__set_autoload(skel->progs.slab_cache_iter, false); > + } > + > + if (needed_groups && !enabled_progs) { > + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n"); > + return -ENOTSUP; > + } > + return 0; > +} [Severity: Medium] What happens on kernels where spinlock and rwlock unlock functions are inlined? In kernel/Kconfig.locks, when !CONFIG_PREEMPTION && !CONFIG_DEBUG_SPINLOCK (for example CONFIG_PREEMPT_NONE or CONFIG_PREEMPT_VOLUNTARY without CONFIG_PREEMPT_DYNAMIC), CONFIG_UNINLINE_SPIN_UNLOCK is not set and CONFIG_INLINE_SPIN_UNLOCK_IRQ, CONFIG_INLINE_READ_UNLOCK, CONFIG_INLINE_READ_UNLOCK_IRQ, CONFIG_INLINE_WRITE_UNLOCK, and CONFIG_INLINE_WRITE_UNLOCK_IRQ are enabled. In that configuration, _raw_spin_unlock, _raw_spin_unlock_irq, _raw_read_unlock, _raw_read_unlock_irq, _raw_write_unlock, and _raw_write_unlock_irq are inlined and absent from vmlinux BTF, while the acquire functions and *_unlock_irqrestore / *_unlock_bh remain out-of-line in BTF. Because configure_lock_progs() enables each program individually if its BTF function exists and only fails when enabled_progs == 0, it will attach the spinlock and rwlock acquire hooks without the corresponding unlock hooks. Wouldn't any spinlock or rwlock released via an inlined unlock (including rq_lock, zone_lock, and siglock) remain permanently recorded as held in held_locks? > +/* > + * Pre-scan a single filter expression prior to loading the BPF skeleton in > + * order to determine which BPF maps need to be sized and which lock tracing > + * program groups need to be autoloaded. > + */ > +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, > + bool *has_lock_filter, > + unsigned int *prog_groups, > + bool *needs_lock_syms, > + bool *needs_slab_iter) > +{ [ ... ] > + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) { > + *prog_groups |= LOCK_PROG_RTMUTEX; > + } else if (strtailcmp(base, "mutex") == 0) { > + *prog_groups |= LOCK_PROG_MUTEX; > + } else if (strtailcmp(base, "percpu_rwsem") == 0 || > + strtailcmp(base, "pcpu_rwsem") == 0) { > + *prog_groups |= LOCK_PROG_PERCPU_RWSEM; > + } else if (strtailcmp(base, "rwsem") == 0) { > + if (mode & PBF_LOCK_MODE_READ) > + *prog_groups |= LOCK_PROG_RWSEM_READ; > + if (mode & PBF_LOCK_MODE_WRITE) > + *prog_groups |= LOCK_PROG_RWSEM_WRITE; [Severity: Medium] Can prescan_one_lock_expr() misclassify kernel struct percpu_rw_semaphore symbols as regular rwsems? Kernel percpu_rw_semaphore locks defined with DEFINE_PERCPU_RWSEM or DEFINE_STATIC_PERCPU_RWSEM typically end in _rwsem rather than _percpu_rwsem (for example cgroup_threadgroup_rwsem, file_rwsem, or scx_fork_rwsem). In prescan_one_lock_expr(), those symbols match strtailcmp(base, "rwsem") == 0 and only enable LOCK_PROG_RWSEM_READ and/or LOCK_PROG_RWSEM_WRITE, leaving LOCK_PROG_PERCPU_RWSEM (fexit_percpu_down_write and fentry_percpu_up_write) disabled. Since is_direct_lock_owner() only inspects struct mutex, struct rw_semaphore, and struct rt_mutex_base, won't filtering on a percpu_rw_semaphore symbol like cgroup_threadgroup_rwsem always evaluate to 0? [ ... ] > diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c > index e5666d4c17228..fbd2e51815d75 100644 > --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c > +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c [ ... ] > +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr) > +{ > + if (!curr) > + return false; > + > + return (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr || > + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr || > + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr; > +} [Severity: Medium] Will calling get_lock_owner(lock_addr, LCB_F_MUTEX) from is_direct_lock_owner() break loading sample_filter_bpf on CONFIG_PREEMPT_RT=y kernels? perf_sample_filter() inlines perf_get_sample() -> check_lock_held() -> is_direct_lock_owner(), which calls get_lock_owner(lock_addr, LCB_F_MUTEX) in tools/perf/util/bpf_skel/lock_data.h: if (flags & LCB_F_MUTEX) { struct mutex *mutex = (void *)lock; owner = BPF_CORE_READ(mutex, owner.counter); if (owner & 0x4UL) return NULL; } On CONFIG_PREEMPT_RT=y kernels, struct mutex wraps struct rt_mutex_base rtmutex and does not have an owner field. Because get_lock_owner() does not guard BPF_CORE_READ(mutex, owner.counter) with bpf_core_field_exists(), wouldn't libbpf poison the instruction during CO-RE relocation and cause the BPF verifier to reject perf_sample_filter on PREEMPT_RT kernels even when lock_held() is not used? [ ... ] > +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags) > +{ > + __u32 owner_id = get_lock_owner_id(flags); > + struct held_lock_key addr_key = { > + .addr_or_id = lock_addr, > + .owner_id = owner_id, > + .kind = PBF_LOCK_ADDR, > + }; > + struct held_lock_val *existing; > + __u32 type_flag = flags & LCB_F_TYPE_MASK; > + bool is_read = (flags & LCB_F_READ) != 0; > + > + if (!lock_enabled || !lock_addr) > + return 0; > + > + existing = bpf_map_lookup_elem(&held_locks, &addr_key); > + if (!existing) > + return 0; > + > + if (is_read) { > + if (existing->read_count <= 1 && existing->write_count == 0) > + bpf_map_delete_elem(&held_locks, &addr_key); > + else if (existing->read_count > 0) > + __sync_fetch_and_add(&existing->read_count, -1); > + } else { > + if (existing->write_count == 0 && existing->read_count == 0) > + bpf_map_delete_elem(&held_locks, &addr_key); > + else if (existing->read_count == 0) > + bpf_map_delete_elem(&held_locks, &addr_key); > + else > + existing->write_count = 0; > + } [ ... ] > +SEC("fexit/downgrade_write") > +int BPF_PROG(fexit_downgrade_write, void *sem) > +{ > + record_lock_acquired(ctx, LCB_F_READ); > + return record_lock_released(ctx, LCB_F_WRITE); > +} [Severity: Medium] Can fexit_downgrade_write() leak held_locks map entries under write-only rwsem filters and corrupt PBF_LOCK_SLAB / PBF_LOCK_TYPE write counts? First, in lock_progs[], fexit_downgrade_write has group LOCK_PROG_RWSEM (LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE), while fentry_up_read has group LOCK_PROG_RWSEM_READ: LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM), LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ), LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE), When a write-only filter such as lock_held_write(&mmap_lock) is used, prescan_one_lock_expr() sets only LOCK_PROG_RWSEM_WRITE. Because (needed_groups & lock_progs[i].group) is non-zero for LOCK_PROG_RWSEM, configure_lock_progs() enables fexit_downgrade_write while leaving fentry_up_read disabled. When the kernel calls downgrade_write() (for example mmap_write_downgrade()), fexit_downgrade_write() calls record_lock_acquired(ctx, LCB_F_READ), setting existing->read_count = 1, and then calls record_lock_released(ctx, LCB_F_WRITE), which sees existing->read_count != 0 and sets existing->write_count = 0 instead of deleting addr_key. Since fentry_up_read is not attached, up_read() never decrements read_count, leaking the held_locks entry (and any PBF_LOCK_SLAB read_count). Second, because fexit_downgrade_write() calls record_lock_acquired(ctx, LCB_F_READ) before record_lock_released(ctx, LCB_F_WRITE), if downgrade_write() is called on an rwsem whose initial down_write() was not recorded in held_locks (such as under a read-only filter where LOCK_PROG_RWSEM_WRITE is disabled, or when acquired before lock_enabled = 1), record_lock_acquired(ctx, LCB_F_READ) creates addr_key with read_count = 1 and write_count = 0. The subsequent record_lock_released(ctx, LCB_F_WRITE) then finds existing != NULL even though existing->write_count == 0, and proceeds to decrement write_count for PBF_LOCK_SLAB and PBF_LOCK_TYPE. Should fexit_downgrade_write release LCB_F_WRITE before acquiring LCB_F_READ (and only record LCB_F_READ when LOCK_PROG_RWSEM_READ is active), and should record_lock_released_addr() verify that the corresponding read_count or write_count was non-zero before decrementing the aggregate slab/type counts? [ ... ] 完美！ Let's double check all rules: - No markdown headers, no backticks, no ALL CAPS shouting (except inside quoted code or config/macro names that are literally in code). Wait: `CONFIG_DEBUG_LOCK_ALLOC=y`, `PBF_LOCK_SLAB`, `PBF_LOCK_TYPE`, `CONFIG_PREEMPT_RT=y`, `LOCK_PROG_RWSEM` are exact code identifiers. Let's make sure there are no non-code ALL CAPS words. - No line numbers mentioned anywhere. - Ends with a blank line. - Severity tags `[Severity: Medium]` on their own line right above each inline comment block. - All 5 findings included. Everything is clean and strictly follows instructions. I'm ready to output.完美! Let's emit the final response.commit 362a7c2a8d1d1d0355b0ac1b022fdffe73aae6c0 Author: Ian Rogers <irogers@google.com> perf bpf-filter: Add lock_held() sample filter This commit adds BPF sample filter predicates lock_held(), lock_held_read(), lock_held_shared(), lock_held_write(), and lock_held_exclusive() to record perf event samples only when a specified kernel lock is held. It uses direct kernel lock owner inspection where possible and falls back to a held_locks BPF hash map populated by fentry/fexit hooks on kernel lock and unlock functions. > diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c > index 72e83ac1d6a4b..35d6f7c638684 100644 > --- a/tools/perf/util/bpf-filter.c > +++ b/tools/perf/util/bpf-filter.c [ ... ] > +static const struct lock_prog_desc lock_progs[] = { > + /* Mutex */ > + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested", > + LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX), > + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX), > + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX), [Severity: Medium] On kernels built with CONFIG_DEBUG_LOCK_ALLOC=y, will mutex acquisitions via mutex_lock_killable(), mutex_trylock(), and mutex_lock_nest_lock() be missed by the held_locks map? In include/linux/mutex.h, when CONFIG_DEBUG_LOCK_ALLOC=y, those functions are macros that expand to different underlying entry points: #define mutex_lock(lock) mutex_lock_nested(lock, 0) #define mutex_lock_interruptible(lock) mutex_lock_interruptible_nested(lock, 0) #define mutex_lock_killable(lock) _mutex_lock_killable(lock, 0, NULL) #define mutex_lock_io(lock) mutex_lock_io_nested(lock, 0) #define mutex_lock_nest_lock(lock, nest_lock) _mutex_lock_nest_lock(lock, &(nest_lock)->dep_map) #define mutex_trylock(lock) _mutex_trylock_nest_lock(lock, NULL) While lock_progs[] and sample_filter.bpf.c hook mutex_lock_nested, mutex_lock_interruptible_nested, and mutex_lock_io_nested, they only hook mutex_lock_killable and mutex_trylock rather than _mutex_lock_killable, _mutex_trylock_nest_lock, and _mutex_lock_nest_lock. For filters that rely on held_locks rather than direct owner inspection (such as PBF_LOCK_SLAB and PBF_LOCK_TYPE), wouldn't mutexes acquired via mutex_lock_killable() or mutex_trylock() go untracked on lockdep-enabled kernels? [ ... ] > +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, > + unsigned int needed_groups, bool needs_lock_syms, > + bool needs_slab_iter) > +{ > + unsigned int enabled_progs = 0; > + > + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) { > + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset; > + struct bpf_program *prog = *progp; > + bool enable = false; > + > + if ((needed_groups & lock_progs[i].group) && btf && > + lock_progs[i].btf_func) { > + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func, > + BTF_KIND_FUNC) > 0) > + enable = true; > + } > + > + if (enable) > + enabled_progs++; > + bpf_program__set_autoload(prog, enable); > + } > + > + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms); > + if (needs_slab_iter && btf && > + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) { > + bpf_program__set_autoload(skel->progs.slab_cache_iter, true); > + } else { > + bpf_program__set_autoload(skel->progs.slab_cache_iter, false); > + } > + > + if (needed_groups && !enabled_progs) { > + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n"); > + return -ENOTSUP; > + } > + return 0; > +} [Severity: Medium] What happens on kernels where spinlock and rwlock unlock functions are inlined? In kernel/Kconfig.locks, when !CONFIG_PREEMPTION && !CONFIG_DEBUG_SPINLOCK (for example CONFIG_PREEMPT_NONE or CONFIG_PREEMPT_VOLUNTARY without CONFIG_PREEMPT_DYNAMIC), CONFIG_UNINLINE_SPIN_UNLOCK is not set and CONFIG_INLINE_SPIN_UNLOCK_IRQ, CONFIG_INLINE_READ_UNLOCK, CONFIG_INLINE_READ_UNLOCK_IRQ, CONFIG_INLINE_WRITE_UNLOCK, and CONFIG_INLINE_WRITE_UNLOCK_IRQ are enabled. In that configuration, _raw_spin_unlock, _raw_spin_unlock_irq, _raw_read_unlock, _raw_read_unlock_irq, _raw_write_unlock, and _raw_write_unlock_irq are inlined and absent from vmlinux BTF, while the acquire functions and *_unlock_irqrestore / *_unlock_bh remain out-of-line in BTF. Because configure_lock_progs() enables each program individually if its BTF function exists and only fails when enabled_progs == 0, it will attach the spinlock and rwlock acquire hooks without the corresponding unlock hooks. Wouldn't any spinlock or rwlock released via an inlined unlock (including rq_lock, zone_lock, and siglock) remain permanently recorded as held in held_locks? > +/* > + * Pre-scan a single filter expression prior to loading the BPF skeleton in > + * order to determine which BPF maps need to be sized and which lock tracing > + * program groups need to be autoloaded. > + */ > +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, > + bool *has_lock_filter, > + unsigned int *prog_groups, > + bool *needs_lock_syms, > + bool *needs_slab_iter) > +{ [ ... ] > + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) { > + *prog_groups |= LOCK_PROG_RTMUTEX; > + } else if (strtailcmp(base, "mutex") == 0) { > + *prog_groups |= LOCK_PROG_MUTEX; > + } else if (strtailcmp(base, "percpu_rwsem") == 0 || > + strtailcmp(base, "pcpu_rwsem") == 0) { > + *prog_groups |= LOCK_PROG_PERCPU_RWSEM; > + } else if (strtailcmp(base, "rwsem") == 0) { > + if (mode & PBF_LOCK_MODE_READ) > + *prog_groups |= LOCK_PROG_RWSEM_READ; > + if (mode & PBF_LOCK_MODE_WRITE) > + *prog_groups |= LOCK_PROG_RWSEM_WRITE; [Severity: Medium] Can prescan_one_lock_expr() misclassify kernel struct percpu_rw_semaphore symbols as regular rwsems? Kernel percpu_rw_semaphore locks defined with DEFINE_PERCPU_RWSEM or DEFINE_STATIC_PERCPU_RWSEM typically end in _rwsem rather than _percpu_rwsem (for example cgroup_threadgroup_rwsem, file_rwsem, or scx_fork_rwsem). In prescan_one_lock_expr(), those symbols match strtailcmp(base, "rwsem") == 0 and only enable LOCK_PROG_RWSEM_READ and/or LOCK_PROG_RWSEM_WRITE, leaving LOCK_PROG_PERCPU_RWSEM (fexit_percpu_down_write and fentry_percpu_up_write) disabled. Since is_direct_lock_owner() only inspects struct mutex, struct rw_semaphore, and struct rt_mutex_base, won't filtering on a percpu_rw_semaphore symbol like cgroup_threadgroup_rwsem always evaluate to 0? [ ... ] > diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c > index e5666d4c17228..fbd2e51815d75 100644 > --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c > +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c [ ... ] > +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr) > +{ > + if (!curr) > + return false; > + > + return (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr || > + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr || > + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr; > +} [Severity: Medium] Will calling get_lock_owner(lock_addr, LCB_F_MUTEX) from is_direct_lock_owner() break loading sample_filter_bpf on CONFIG_PREEMPT_RT=y kernels? perf_sample_filter() inlines perf_get_sample() -> check_lock_held() -> is_direct_lock_owner(), which calls get_lock_owner(lock_addr, LCB_F_MUTEX) in tools/perf/util/bpf_skel/lock_data.h: if (flags & LCB_F_MUTEX) { struct mutex *mutex = (void *)lock; owner = BPF_CORE_READ(mutex, owner.counter); if (owner & 0x4UL) return NULL; } On CONFIG_PREEMPT_RT=y kernels, struct mutex wraps struct rt_mutex_base rtmutex and does not have an owner field. Because get_lock_owner() does not guard BPF_CORE_READ(mutex, owner.counter) with bpf_core_field_exists(), wouldn't libbpf poison the instruction during CO-RE relocation and cause the BPF verifier to reject perf_sample_filter on PREEMPT_RT kernels even when lock_held() is not used? [ ... ] > +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags) > +{ > + __u32 owner_id = get_lock_owner_id(flags); > + struct held_lock_key addr_key = { > + .addr_or_id = lock_addr, > + .owner_id = owner_id, > + .kind = PBF_LOCK_ADDR, > + }; > + struct held_lock_val *existing; > + __u32 type_flag = flags & LCB_F_TYPE_MASK; > + bool is_read = (flags & LCB_F_READ) != 0; > + > + if (!lock_enabled || !lock_addr) > + return 0; > + > + existing = bpf_map_lookup_elem(&held_locks, &addr_key); > + if (!existing) > + return 0; > + > + if (is_read) { > + if (existing->read_count <= 1 && existing->write_count == 0) > + bpf_map_delete_elem(&held_locks, &addr_key); > + else if (existing->read_count > 0) > + __sync_fetch_and_add(&existing->read_count, -1); > + } else { > + if (existing->write_count == 0 && existing->read_count == 0) > + bpf_map_delete_elem(&held_locks, &addr_key); > + else if (existing->read_count == 0) > + bpf_map_delete_elem(&held_locks, &addr_key); > + else > + existing->write_count = 0; > + } [ ... ] > +SEC("fexit/downgrade_write") > +int BPF_PROG(fexit_downgrade_write, void *sem) > +{ > + record_lock_acquired(ctx, LCB_F_READ); > + return record_lock_released(ctx, LCB_F_WRITE); > +} [Severity: Medium] Can fexit_downgrade_write() leak held_locks map entries under write-only rwsem filters and corrupt PBF_LOCK_SLAB / PBF_LOCK_TYPE write counts? First, in lock_progs[], fexit_downgrade_write has group LOCK_PROG_RWSEM (LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE), while fentry_up_read has group LOCK_PROG_RWSEM_READ: LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM), LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ), LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE), When a write-only filter such as lock_held_write(&mmap_lock) is used, prescan_one_lock_expr() sets only LOCK_PROG_RWSEM_WRITE. Because (needed_groups & lock_progs[i].group) is non-zero for LOCK_PROG_RWSEM, configure_lock_progs() enables fexit_downgrade_write while leaving fentry_up_read disabled. When the kernel calls downgrade_write() (for example mmap_write_downgrade()), fexit_downgrade_write() calls record_lock_acquired(ctx, LCB_F_READ), setting existing->read_count = 1, and then calls record_lock_released(ctx, LCB_F_WRITE), which sees existing->read_count != 0 and sets existing->write_count = 0 instead of deleting addr_key. Since fentry_up_read is not attached, up_read() never decrements read_count, leaking the held_locks entry (and any PBF_LOCK_SLAB read_count). Second, because fexit_downgrade_write() calls record_lock_acquired(ctx, LCB_F_READ) before record_lock_released(ctx, LCB_F_WRITE), if downgrade_write() is called on an rwsem whose initial down_write() was not recorded in held_locks (such as under a read-only filter where LOCK_PROG_RWSEM_WRITE is disabled, or when acquired before lock_enabled = 1), record_lock_acquired(ctx, LCB_F_READ) creates addr_key with read_count = 1 and write_count = 0. The subsequent record_lock_released(ctx, LCB_F_WRITE) then finds existing != NULL even though existing->write_count == 0, and proceeds to decrement write_count for PBF_LOCK_SLAB and PBF_LOCK_TYPE. Should fexit_downgrade_write release LCB_F_WRITE before acquiring LCB_F_READ (and only record LCB_F_READ when LOCK_PROG_RWSEM_READ is active), and should record_lock_released_addr() verify that the corresponding read_count or write_count was non-zero before decrementing the aggregate slab/type counts? -- Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=2 ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH v1 3/5] perf test: Add tests for lock_held() BPF sample filter 2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers 2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers @ 2026-10-17 5:45 ` Ian Rogers 2026-10-17 5:49 ` sashiko-bot 2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers ` (2 subsequent siblings) 5 siblings, 1 reply; 24+ messages in thread From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Namhyung Kim Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark, linux-perf-users, linux-kernel, rcu Add unit tests in parse-events.c for parsing lock_held(), lock_held_read(), and lock_held_exclusive() BPF filter expressions (standalone predicates, negated predicates, and compound group expressions) and extend record_bpf_filter.sh with lock_held(), lock_held_read()/lock_held_write(), and ':R'/':W' recording tests. Example output: # perf test -v 'Parsing of BPF sample filters' 5: Parse event definition strings : 5.7: Parsing of BPF sample filters : Ok # perf test -v 'perf record sample filtering (by BPF) tests' 112: perf record sample filtering (by BPF) tests : Checking BPF-filter privilege Basic bpf-filter test Basic bpf-filter test [Success] Failing bpf-filter test Failing bpf-filter test [Success] Group bpf-filter test Group bpf-filter test [Success] Multiple bpf-filter test Multiple bpf-filter test [Success] Cgroup bpf-filter test Cgroup bpf-filter test [Success] Lock-held bpf-filter test Lock-held bpf-filter test [Success] 112: perf record sample filtering (by BPF) tests : Ok Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/tests/parse-events.c | 108 ++++++++++++++++++++ tools/perf/tests/shell/record_bpf_filter.sh | 99 ++++++++++++++++++ 2 files changed, 207 insertions(+) diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 5f6f95c8a5b4..4cf9192762f2 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -8,9 +8,11 @@ #include "debug.h" #include "pmu.h" #include "pmus.h" +#include "bpf-filter.h" #include "strbuf.h" #include <dirent.h> #include <errno.h> +#include <string.h> #include "fncache.h" #include <sys/types.h> #include <sys/stat.h> @@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused, return ret; } +#ifdef HAVE_BPF_SKEL +static void clear_bpf_filter_exprs(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *tmp; + + list_for_each_entry_safe(expr, tmp, expr_head, list) { + list_del(&expr->list); + perf_bpf_filter_expr__delete(expr); + } +} + +static int do_test_bpf_filters(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *group; + const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100"; + + /* Simple lock_held(&cgroup_mutex) */ + TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Negated !lock_held(cgroup_mutex) */ + TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Read/shared and write/exclusive predicates */ + TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", + __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)", + __perf_bpf_filter__parse(expr_head, + "!lock_held_exclusive(mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + /* + * Comparison and compound group: + * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100 + */ + TEST_ASSERT_VAL("parse compound lock_held filter", + __perf_bpf_filter__parse(expr_head, compound) == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN); + TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups)); + group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("first group member lock_name", + group->lock_name && !strcmp(group->lock_name, "rwsem:R")); + TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1); + group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second group member lock_name", + group->lock_name && !strcmp(group->lock_name, "&dentry")); + TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0); + expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100); + + return TEST_OK; +} +#endif + +static int test__bpf_filters(struct test_suite *test __maybe_unused, + int subtest __maybe_unused) +{ +#ifdef HAVE_BPF_SKEL + struct list_head expr_head = LIST_HEAD_INIT(expr_head); + int ret = do_test_bpf_filters(&expr_head); + + clear_bpf_filter_exprs(&expr_head); + return ret; +#else + return TEST_SKIP; +#endif +} + static struct test_case tests__parse_events[] = { TEST_CASE_REASON("Test event parsing", events2, @@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = { "no aliases in sysfs"), TEST_CASE("Parsing of aliased events", pmu_events_alias2), TEST_CASE("Parsing of terms (event modifiers)", terms2), + TEST_CASE("Parsing of BPF sample filters", bpf_filters), { .name = NULL, } }; diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index 383574cb3bd3..7ec75d969bd6 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -176,6 +176,101 @@ test_bpf_filter_cgroup() { echo "Cgroup bpf-filter test [Success]" } +test_bpf_filter_lock_held() { + echo "Lock-held bpf-filter test" + lock_sym="cgroup_mutex" + + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + lock_sym="slab_mutex" + fi + + if ! perf record -e task-clock --filter "lock_held(&${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + if [ "$(id -u)" != 0 ] + then + echo "Lock-held bpf-filter test [Skipped permission]" + return + fi + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]" + return + fi + echo "Lock-held bpf-filter test [Failed &${lock_sym}]" + err=1 + return + fi + + if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \ + -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed mmap_lock]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(rq_lock)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed rq_lock]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(spinlock)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock type]" + err=1 + return + fi + + if ! perf record -e task-clock \ + --filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read/write(&mmap_lock)]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(&mmap_lock:R) || lock_held(&mmap_lock:W)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed &mmap_lock:R/:W]" + err=1 + return + fi + + if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]" + err=1 + return + fi + + if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]" + err=1 + return + fi + + echo "Lock-held bpf-filter test [Success]" +} + test_bpf_filter_priv if [ $err = 0 ]; then @@ -198,5 +293,9 @@ if [ $err = 0 ]; then test_bpf_filter_cgroup fi +if [ $err = 0 ]; then + test_bpf_filter_lock_held +fi + cleanup exit $err -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH v1 3/5] perf test: Add tests for lock_held() BPF sample filter 2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers @ 2026-10-17 5:49 ` sashiko-bot 0 siblings, 0 replies; 24+ messages in thread From: sashiko-bot @ 2026-10-17 5:49 UTC (permalink / raw) To: Ian Rogers; +Cc: linux-perf-users > Add unit tests in parse-events.c for parsing lock_held(), > lock_held_read(), and lock_held_exclusive() BPF filter expressions > (standalone predicates, negated predicates, and compound group > expressions) and extend record_bpf_filter.sh with lock_held(), > lock_held_read()/lock_held_write(), and ':R'/':W' recording tests. > > Example output: > > # perf test -v 'Parsing of BPF sample filters' > 5: Parse event definition strings : > 5.7: Parsing of BPF sample filters : Ok > > # perf test -v 'perf record sample filtering (by BPF) tests' > 112: perf record sample filtering (by BPF) tests : > Checking BPF-filter privilege > Basic bpf-filter test > Basic bpf-filter test [Success] > Failing bpf-filter test > Failing bpf-filter test [Success] > Group bpf-filter test > Group bpf-filter test [Success] > Multiple bpf-filter test > Multiple bpf-filter test [Success] > Cgroup bpf-filter test > Cgroup bpf-filter test [Success] > Lock-held bpf-filter test > Lock-held bpf-filter test [Success] > 112: perf record sample filtering (by BPF) tests : Ok > > Signed-off-by: Ian Rogers <irogers@google.com> Sashiko has reviewed this patch and found no issues. -- Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=3 ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU 2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (2 preceding siblings ...) 2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers @ 2026-10-17 5:45 ` Ian Rogers 2026-10-17 5:56 ` sashiko-bot 2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 5 siblings, 1 reply; 24+ messages in thread From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Namhyung Kim Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark, linux-perf-users, linux-kernel, rcu Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks Trace RCU, and SRCU locks: - 'rcu' (supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)', 'lock_held(rcu_read_lock)'): directly inspects current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on CONFIG_PREEMPT_RCU kernels), accounting for the baseline rcu_read_lock() nesting acquired by the perf overflow handler itself (bpf_overflow_handler() and do_perf_sw_event()) with zero extra fentry/fexit tracing hooks. - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'): attaches fentry/fexit hooks to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), and rcu_barrier() to track grace-period and barrier waits. - 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches fexit/__srcu_read_lock and fentry/__srcu_read_unlock. - Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'): attaches fentry/fexit on synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier(). - 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports read/shared mode only): directly inspects current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels with zero extra tracing hooks. Example usage: # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ....... ................. ........................ # 14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu | ---__d_lookup_rcu lookup_fast walk_component link_path_walk path_lookupat filename_lookup vfs_statx __do_sys_newstat do_syscall_64 entry_SYSCALL_64_after_hwframe # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1 # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1 Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 42 +++-- tools/perf/tests/parse-events.c | 13 ++ tools/perf/tests/shell/record_bpf_filter.sh | 20 +++ tools/perf/util/bpf-filter.c | 158 +++++++++++++++++-- tools/perf/util/bpf_skel/lock_data.h | 44 +++++- tools/perf/util/bpf_skel/sample-filter.h | 12 ++ tools/perf/util/bpf_skel/sample_filter.bpf.c | 135 +++++++++++++++- tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 + tools/perf/util/lock-contention.c | 4 + tools/perf/util/lock-contention.h | 4 +- 10 files changed, 402 insertions(+), 32 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 246ae085744d..c8ccfb0e0ce6 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -219,28 +219,38 @@ OPTIONS '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache name prefixed with '&' (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known - lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock - type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', - 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', - 'semaphore'). Both read/shared and write/exclusive tracking are - supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; + lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock', 'rcu', + 'rcu-trace'), or a lock type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', + 'rwsem:W', 'rwlock', 'rwlock:R', 'rwlock:W', 'rt-mutex', + 'percpu-rwsem', 'pcpu-sem:W', 'srcu', 'srcu:R', 'srcu:W', 'rcu:R', + 'rcu:W', 'semaphore'). Both read/shared and write/exclusive tracking + are supported for 'rwsem' (including 'mmap_lock'), 'rwlock', 'rcu', + and 'srcu' (where read/shared mode matches 'rcu_read_lock()' / + 'srcu_read_lock()' critical sections and write/exclusive mode matches + grace-period waits 'synchronize_rcu*()', 'synchronize_srcu*()', + 'rcu_barrier()', and 'srcu_barrier()'); 'rcu-trace' + ('rcu_read_lock_trace()') supports read/shared tracking only; and 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support write/exclusive tracking only. Note that 'lock_held(<lock>)' introduces overhead outside of sample overflow handling: in addition to checking the lock's owner field on - each sample where possible (such as for mutexes, rt-mutexes, and - write-locked rwsems), it attaches BPF fentry/fexit programs to kernel - lock acquire and release functions to track held locks (such as - spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a - BPF map. These BPF hooks run on every kernel lock acquisition and - release for the attached lock family, even when no perf sample is - being taken. To minimize this overhead, only the hooks for the - relevant lock family and mode are attached when known (for example, - 'lock_held_read(&mmap_lock)' only hooks rwsem read functions, - 'lock_held_write(&mmap_lock)' only hooks rwsem write functions, - '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks + each sample where possible (such as for mutexes, rt-mutexes, + write-locked rwsems, and 'rcu:R' / 'rcu-trace' which inspect + 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting' + directly with no extra tracing hooks), it attaches BPF fentry/fexit + programs to kernel lock acquire and release functions to track held + locks (such as spinlocks, rwlocks, reader-locked rwsems, SRCU locks, + RCU synchronize waits, and slab/type filters) in a BPF map. These + BPF hooks run on every kernel lock acquisition and release for the + attached lock family, even when no perf sample is being taken. To + minimize this overhead, only the hooks for the relevant lock family + and mode are attached when known (for example, 'lock_held_read(rcu)' + and 'rcu-trace' attach no extra hooks, 'lock_held_read(&mmap_lock)' + only hooks rwsem read functions, 'lock_held_write(&mmap_lock)' only + hooks rwsem write functions, '&cgroup_mutex' only hooks mutex + functions, 'srcu' only hooks SRCU functions, and 'rq_lock' only hooks spinlock functions), whereas slab filters ('&<slab>'), raw hex addresses, or symbols without a recognizable lock-type suffix attach hooks across all lock families (including high-frequency spinlocks). diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 4cf9192762f2..730466b7ebf6 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head) expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); clear_bpf_filter_exprs(expr_head); + /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */ + TEST_ASSERT_VAL("parse lock_held(rcu)", + __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is rcu", + expr->lock_name && !strcmp(expr->lock_name, "rcu")); + clear_bpf_filter_exprs(expr_head); + /* Read/shared and write/exclusive predicates */ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index 7ec75d969bd6..39ba628022db 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -235,6 +235,26 @@ test_bpf_filter_lock_held() { return fi + if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed rcu:W/srcu]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o "${perfdata}" true 2> /dev/null + then + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]" + err=1 + return + fi + fi + if ! perf record -e task-clock \ --filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \ -o "${perfdata}" true 2> /dev/null diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 35d6f7c63868..d4d7f7a50f18 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -190,11 +190,19 @@ enum lock_prog_group { LOCK_PROG_RTMUTEX = 1U << 6, LOCK_PROG_SEM = 1U << 7, LOCK_PROG_PERCPU_RWSEM = 1U << 8, - LOCK_PROG_ALL = 0x1ffU, + LOCK_PROG_SRCU_READ = 1U << 9, + LOCK_PROG_SRCU_WRITE = 1U << 10, + LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE, + LOCK_PROG_RCU_WRITE = 1U << 11, + LOCK_PROG_ALL = 0xfffU, }; static unsigned int lock_flags_to_prog_group(unsigned int flags) { + if (flags & LCB_F_RCU) + return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0; + if (flags & LCB_F_SRCU) + return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE; if (flags & LCB_F_MUTEX) return LOCK_PROG_MUTEX; if (flags & LCB_F_RT) @@ -308,6 +316,32 @@ static const struct lock_prog_desc lock_progs[] = { /* Percpu-RWsem */ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), + /* SRCU */ + LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + /* RCU grace-period synchronize / barrier */ + LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), }; #undef LOCK_FPROG @@ -616,6 +650,17 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, *needs_lock_syms = true; return; } + if (!strcmp(base, "rcu")) { + if (mode & PBF_LOCK_MODE_WRITE) + *prog_groups |= LOCK_PROG_RCU_WRITE; + return; + } + if (!strcmp(base, "rcu_read_lock") || + !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + /* Inspected directly on task_struct; no tracing hooks needed. */ + return; + } if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, prog_groups)) return; @@ -646,6 +691,11 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, *prog_groups |= LOCK_PROG_RWLOCK_READ; if (mode & PBF_LOCK_MODE_WRITE) *prog_groups |= LOCK_PROG_RWLOCK_WRITE; + } else if (strtailcmp(base, "srcu") == 0) { + if (mode & PBF_LOCK_MODE_READ) + *prog_groups |= LOCK_PROG_SRCU_READ; + if (mode & PBF_LOCK_MODE_WRITE) + *prog_groups |= LOCK_PROG_SRCU_WRITE; } else { *prog_groups |= LOCK_PROG_ALL; } @@ -729,16 +779,54 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, return false; } +static u64 rcu_min_depth(struct evsel *evsel) +{ + if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE && + evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK && + evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK) + return 2; + return 1; +} + +static bool btf_has_task_field(struct btf *btf, const char *field_name) +{ + const struct btf_type *t; + const struct btf_member *m; + s32 id; + u16 i, vlen; + + if (!btf) + return false; + + id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT); + if (id < 0) + return false; + + t = btf__type_by_id(btf, id); + if (!t) + return false; + + m = btf_members(t); + vlen = btf_vlen(t); + for (i = 0; i < vlen; i++, m++) { + const char *name = btf__name_by_offset(btf, m->name_off); + + if (name && !strcmp(name, field_name)) + return true; + } + return false; +} + /* * Resolve a `lock_held(<name>)` filter expression into `entry->part` * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate * the corresponding BPF filter maps and BSS control flags. * * Resolution order: - * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, - * `rq_lock`, `zone_lock` (with or without leading '&'). + * 1. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` (with or without leading '&'). * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`, - * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, + * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, `srcu`, * `semaphore`, etc. * 3. Hexadecimal lock address (when no leading '&'). * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`). @@ -746,8 +834,8 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, * e.g. `&cgroup_mutex` or `cgroup_mutex`). * 6. Slab cache name without leading '&' fallback. */ -static int resolve_lock_expr(struct sample_filter_bpf *skel, - struct perf_bpf_filter_expr *expr, +static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel, + struct btf *btf, struct perf_bpf_filter_expr *expr, struct perf_bpf_filter_entry *entry) { const char *name = expr->lock_name; @@ -811,6 +899,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, return 0; } + if (!strcmp(base, "rcu_read_lock")) { + if (mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ; + entry->target = rcu_min_depth(evsel); + return 0; + } + + if (!strcmp(base, "rcu")) { + if ((mode & PBF_LOCK_MODE_READ) && btf && + !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | mode; + entry->target = rcu_min_depth(evsel); + if ((mode & PBF_LOCK_MODE_WRITE) && skel) + skel->bss->has_rcu_lock = 1; + return 0; + } + + if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) { + pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ; + entry->target = 0; + return 0; + } + if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { entry->part = PBF_LOCK_TYPE | mode; entry->target = encoded_pair; @@ -875,7 +1005,7 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, } static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, - struct perf_bpf_filter_entry *entry) + struct btf *btf, struct perf_bpf_filter_entry *entry) { int i = 0, ret; struct perf_bpf_filter_expr *expr; @@ -893,7 +1023,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].value = expr->val; entry[i].target = 0; if (expr->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, expr, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]); if (ret < 0) return ret; } @@ -912,7 +1042,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].value = group->val; entry[i].target = 0; if (group->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, group, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]); if (ret < 0) return ret; } @@ -1185,7 +1315,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - ret = get_filter_entries(evsel, NULL, entry); + ret = get_filter_entries(evsel, NULL, NULL, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); goto err; @@ -1247,6 +1377,12 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); init_numa_data(skel, btf); } + if (prog_groups & LOCK_PROG_RCU_WRITE) { + u64 rcu_addr = 1ULL; + + find_kernel_lock_sym("rcu_state", &rcu_addr); + skel->rodata->rcu_state_addr = rcu_addr; + } ret = configure_lock_progs(skel, btf, prog_groups, needs_lock_syms, needs_slab_iter); if (ret < 0) @@ -1264,7 +1400,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (has_lock_filter && needs_slab_iter) run_slab_cache_iter(skel); - ret = get_filter_entries(evsel, skel, entry); + ret = get_filter_entries(evsel, skel, btf, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); goto err; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index fed445c9d8b4..e32ba230d5f5 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -39,6 +39,7 @@ struct contention_task_data { */ #define LCD_F_MMAP_LOCK (1U << 31) #define LCD_F_SIGHAND_LOCK (1U << 30) +#define LCD_F_RCU_LOCK (1U << 29) #define LCB_F_SLAB_ID_SHIFT 16 #define LCB_F_SLAB_ID_START (1U << 16) @@ -52,9 +53,11 @@ struct contention_task_data { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) -#define LCB_F_TYPE_MAX (1U << 7) -#define LCB_F_TYPE_MASK 0x0000007FU +#define LCB_F_TYPE_MAX (1U << 8) +#define LCB_F_TYPE_MASK 0x000000FFU #define SLAB_NAME_MAX 28 @@ -233,6 +236,40 @@ static __always_inline __u64 get_current_siglock(__u64 curr) return sh + bpf_core_field_offset(sighand->siglock); } +/* + * Check if `curr` is inside a preemptible RCU read-side critical section + * (`rcu_read_lock()`). + * + * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires + * `rcu_read_lock()` before invoking `perf_sample_filter` (and + * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for + * non-clock software events), `min_depth` is the baseline nesting count + * contributed by the perf event overflow handler itself (1 or 2). + */ +static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting)) + return false; + + return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth; +} + +/* + * Check if `curr` is inside a Tasks Trace RCU read-side critical section + * (`rcu_read_lock_trace()`). + */ +static __always_inline bool is_rcu_trace_lock_held(__u64 curr) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->trc_reader_nesting)) + return false; + + return BPF_CORE_READ(task, trc_reader_nesting) > 0; +} + /* * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. * @@ -265,6 +302,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) if (get_current_siglock(curr) == lock) return LCD_F_SIGHAND_LOCK; break; + case LCB_F_RCU: + case LCB_F_RCU | LCB_F_WRITE: + return LCD_F_RCU_LOCK; default: break; } diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index cd711d1d04b6..37c108608bbc 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -99,6 +99,18 @@ enum perf_bpf_lock_target_kind { * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). */ PBF_LOCK_TYPE, + /* + * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`). + * Inspected directly via `current->rcu_read_lock_nesting > entry.target`, + * where `entry.target` is the baseline RCU nesting depth added by the + * kernel perf overflow path. + */ + PBF_LOCK_RCU, + /* + * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`). + * Inspected directly via `current->trc_reader_nesting > 0`. + */ + PBF_LOCK_RCU_TRACE, }; /* diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index fbd2e51815d7..22d9b01e5851 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -123,6 +123,7 @@ volatile const int use_idx_hash; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -136,6 +137,7 @@ int has_lock_class; int has_lock_type; int has_mmap_lock; int has_siglock; +int has_rcu_lock; __u8 lock_type_filter[LCB_F_TYPE_MAX]; void *bpf_cast_to_kern_ctx(void *) __ksym; @@ -269,6 +271,20 @@ static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry case PBF_LOCK_TYPE: return is_lock_type_in_map(entry->target, mode, pid, cpu_id); + case PBF_LOCK_RCU: + if ((mode & PBF_LOCK_MODE_READ) && + is_rcu_lock_held(curr, (int)entry->target)) + return 1; + if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr && + is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id)) + return 1; + return 0; + + case PBF_LOCK_RCU_TRACE: + if ((mode & PBF_LOCK_MODE_READ) && is_rcu_trace_lock_held(curr)) + return 1; + return 0; + default: return 0; } @@ -600,11 +616,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) matched = true; - if (has_mmap_lock || has_siglock) { + if (has_mmap_lock || has_siglock || has_rcu_lock) { __u32 ltype = check_lock_type(lock_addr, flags); if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || - (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + (has_siglock && ltype == LCD_F_SIGHAND_LOCK) || + (has_rcu_lock && ltype == LCD_F_RCU_LOCK)) matched = true; } @@ -1202,6 +1219,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem) return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); } +/* SRCU acquire/release and synchronize hooks */ +SEC("fexit/__srcu_read_lock") +int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock") +int BPF_PROG(fentry_srcu_read_unlock, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fexit/__srcu_read_lock_nmisafe") +int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock_nmisafe") +int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +/* RCU grace-period synchronize / barrier hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(collect_lock_syms) { diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index b75ddc6846bb..1fb91a829d54 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -119,6 +119,8 @@ struct task_struct { struct signal_struct *signal; struct sighand_struct *sighand; struct css_set *cgroups; + int rcu_read_lock_nesting; + int trc_reader_nesting; } __attribute__((preserve_access_index)); struct trace_entry { diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index a1bb78d6e29c..b45b075f3346 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -38,6 +38,10 @@ const struct perf_lock_type lock_type_table[] = { { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, /* alias for optimistic spinning only */ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, + { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" }, + { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" }, + { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" }, + { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" }, }; int parse_call_stack(const struct option *opt __maybe_unused, const char *str, diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 68e9bd77e8a9..88b2c11244ad 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -128,6 +128,8 @@ struct thread_stat { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) struct perf_lock_type { unsigned int flags; @@ -140,7 +142,7 @@ struct perf_lock_type { const char *lock_name; }; -extern const struct perf_lock_type lock_type_table[14]; +extern const struct perf_lock_type lock_type_table[18]; struct evlist; struct machine; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU 2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers @ 2026-10-17 5:56 ` sashiko-bot 0 siblings, 0 replies; 24+ messages in thread From: sashiko-bot @ 2026-10-17 5:56 UTC (permalink / raw) To: Ian Rogers; +Cc: linux-perf-users > Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks > Trace RCU, and SRCU locks: > > - 'rcu' (supports both read/shared and write/exclusive modes): > - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)', > 'lock_held(rcu_read_lock)'): directly inspects > current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on > CONFIG_PREEMPT_RCU kernels), accounting for the baseline > rcu_read_lock() nesting acquired by the perf overflow handler itself > (bpf_overflow_handler() and do_perf_sw_event()) with zero extra > fentry/fexit tracing hooks. > - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'): > attaches fentry/fexit hooks to synchronize_rcu(), > synchronize_rcu_expedited(), synchronize_rcu_tasks(), and > rcu_barrier() to track grace-period and barrier waits. > [ ... ] > > # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1 > # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1 > > Signed-off-by: Ian Rogers <irogers@google.com> Sashiko has reviewed this patch and found no issues. -- Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=4 ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU and SRCU 2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (3 preceding siblings ...) 2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers @ 2026-10-17 5:45 ` Ian Rogers 2026-10-17 6:08 ` sashiko-bot 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 5 siblings, 1 reply; 24+ messages in thread From: Ian Rogers @ 2026-10-17 5:45 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Namhyung Kim Cc: Peter Zijlstra, Ingo Molnar, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark, linux-perf-users, linux-kernel, rcu Extend 'perf lock contention --use-bpf' to track RCU and SRCU grace-period and barrier waits: - Attach fentry/fexit programs to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(), synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier() when available in BTF. - Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu', '-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct symbol names). Example usage: # perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1 total wait avg wait type caller 35.12 ms 8.78 ms rcu:W cleanup_net+0x284 7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114 # perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1 total wait avg wait address symbol 35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu) Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-lock.txt | 7 +- tools/perf/tests/shell/lock_contention.sh | 14 ++ tools/perf/util/bpf_lock_contention.c | 167 ++++++++++++++-- .../perf/util/bpf_skel/lock_contention.bpf.c | 183 ++++++++++++++---- 4 files changed, 322 insertions(+), 49 deletions(-) diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt index c17b3e318169..bfeb98c976e3 100644 --- a/tools/perf/Documentation/perf-lock.txt +++ b/tools/perf/Documentation/perf-lock.txt @@ -188,11 +188,14 @@ CONTENTION OPTIONS Show lock contention only for given lock types (comma separated list). Available values are: semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W, - rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem, - pcpu-sem:R, pcpu-sem:W, mutex + rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem, + pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W Note that RW-variant of locks have :R and :W suffix. Names without the suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W. + For RCU and SRCU with --use-bpf, contention tracks grace-period and + barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(), + srcu_barrier()). -L:: --lock-filter=<value>:: diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh index 5df9a0cd48a3..c6d37c910d4b 100755 --- a/tools/perf/tests/shell/lock_contention.sh +++ b/tools/perf/tests/shell/lock_contention.sh @@ -181,6 +181,13 @@ test_type_filter() err=1 exit fi + + perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_lock_filter() @@ -224,6 +231,13 @@ test_lock_filter() err=1 exit fi + + perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_stack_filter() diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c index c20bd075664e..fa7df9187b08 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -16,6 +16,7 @@ #include <bpf/bpf.h> #include <bpf/btf.h> #include <inttypes.h> +#include <unistd.h> #include "bpf_skel/lock_contention.skel.h" #include "bpf_skel/lock_data.h" @@ -180,6 +181,129 @@ static void init_numa_data(struct lock_contention *con) free(buf); } +static bool has_bpf_fentry_support(struct btf *btf) +{ + /* + * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and + * bpf_get_func_arg() helper availability (5.17+) used by + * get_lock_addr() in the SRCU fentry/fexit hooks. + */ + struct bpf_insn insns[] = { + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 }, + { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3, + .src_reg = BPF_REG_10 }, + { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 }, + { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg }, + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 }, + { .code = BPF_JMP | BPF_EXIT }, + }; + LIBBPF_OPTS(bpf_prog_load_opts, opts, + .expected_attach_type = BPF_TRACE_FENTRY, + ); + s32 btf_id; + int prog_fd, link_fd; + + if (!btf) + return false; + + btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC); + if (btf_id <= 0) + return false; + + opts.attach_btf_id = btf_id; + prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL", + insns, ARRAY_SIZE(insns), &opts); + if (prog_fd < 0) + return false; + + link_fd = bpf_raw_tracepoint_open(NULL, prog_fd); + if (link_fd >= 0) + close(link_fd); + close(prog_fd); + return link_fd >= 0; +} + +static void configure_rcu_contention_progs(struct lock_contention *con, + bool has_rcu_lock) +{ + static const struct { + size_t prog_offset; + const char *func; + bool is_srcu; + } rcu_progs[] = { + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier), + "srcu_barrier", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier), + "srcu_barrier", true }, + }; + struct symbol *sym; + struct map *kmap; + bool need_rcu = true, need_srcu = true; + bool fentry_ok; + + sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap); + if (sym) { + skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start); + map__put(kmap); + } else { + skel->rodata->rcu_state_addr = 1ULL; + } + + if (con->filters->nr_types) { + need_rcu = false; + need_srcu = false; + for (int i = 0; i < con->filters->nr_types; i++) { + if (con->filters->types[i] & LCB_F_RCU) + need_rcu = true; + if (con->filters->types[i] & LCB_F_SRCU) + need_srcu = true; + } + } + + if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) { + if (!has_rcu_lock && !con->filters->nr_addrs) + need_rcu = false; + if (!con->filters->nr_addrs && !con->filters->nr_slabs) + need_srcu = false; + } + + fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf); + + for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) { + struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset; + bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu; + bool enable = fentry_ok && want && + btf__find_by_name_kind(con->btf, rcu_progs[i].func, + BTF_KIND_FUNC) > 0; + + bpf_program__set_autoload(*progp, enable); + } +} + int lock_contention_prepare(struct lock_contention *con) { int i, fd; @@ -187,6 +311,7 @@ int lock_contention_prepare(struct lock_contention *con) struct evlist *evlist = con->evlist; struct target *target = con->target; bool has_mmap_lock = false; + bool has_rcu_lock = false; /* make sure it loads the kernel map before lookup */ map__load(machine__kernel_map(con->machine)); @@ -200,6 +325,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64)); bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries); bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries); + bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries); if (con->aggr_mode == LOCK_AGGR_TASK) bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries); @@ -249,6 +375,10 @@ int lock_contention_prepare(struct lock_contention *con) has_mmap_lock = true; continue; } + if (!strcmp(con->filters->syms[i], "rcu")) { + has_rcu_lock = true; + continue; + } sym = machine__find_kernel_symbol_by_name(con->machine, con->filters->syms[i], @@ -269,7 +399,7 @@ int lock_contention_prepare(struct lock_contention *con) addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start); con->filters->addrs = addrs; } - naddrs = con->filters->nr_addrs ?: has_mmap_lock; + naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock); skel->rodata->has_addr = 1; } @@ -305,6 +435,7 @@ int lock_contention_prepare(struct lock_contention *con) skel->rodata->needs_callstack = con->save_callstack; skel->rodata->lock_owner = con->owner; skel->rodata->has_mmap_lock = has_mmap_lock; + skel->rodata->has_rcu_lock = has_rcu_lock; if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) { if (cgroup_is_v2("perf_event")) @@ -321,6 +452,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_max_entries(skel->maps.slab_filter, nslabs); init_numa_data(con); + configure_rcu_contention_progs(con, has_rcu_lock); if (lock_contention_bpf__load(skel) < 0) { pr_err("Failed to load lock-contention BPF skeleton\n"); @@ -503,23 +635,30 @@ static void account_end_timestamp(struct lock_contention *con) enum lock_aggr_mode aggr_mode = con->aggr_mode; struct tstamp_data ts_data, *cpu_data; - /* Iterate per-task tstamp map (key = TID) */ - ts_fd = bpf_map__fd(skel->maps.tstamp); + /* Iterate per-task tstamp maps (key = TID) */ + int ts_fds[] = { + bpf_map__fd(skel->maps.tstamp), + bpf_map__fd(skel->maps.tstamp_rcu), + }; + stat_fd = bpf_map__fd(skel->maps.lock_stat); - prev_key = NULL; - while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { - if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { - int pid = key; + for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) { + ts_fd = ts_fds[i]; + prev_key = NULL; + while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { + if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { + int pid = key; - if (aggr_mode == LOCK_AGGR_TASK && con->owner) - pid = ts_data.flags; + if (aggr_mode == LOCK_AGGR_TASK && con->owner) + pid = ts_data.flags; - update_lock_stat(stat_fd, pid, end_ts, aggr_mode, - &ts_data); - } + update_lock_stat(stat_fd, pid, end_ts, aggr_mode, + &ts_data); + } - prev_key = &key; + prev_key = &key; + } } /* Now it'll check per-cpu tstamp map which doesn't have TID. */ @@ -604,6 +743,8 @@ static const char *lock_contention_get_name(struct lock_contention *con, return "mmap_lock"; if (flags & LCD_F_SIGHAND_LOCK) return "siglock"; + if (flags & LCD_F_RCU_LOCK) + return "rcu"; /* global locks with symbols */ sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap); diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0f2118006005..257e3402e815 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -59,6 +59,18 @@ struct { __uint(max_entries, MAX_ENTRIES); } tstamp SEC(".maps"); +/* + * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits + * so they do not collide with inner sleeping lock contentions (such as + * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait. + */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, int); + __type(value, struct tstamp_data); + __uint(max_entries, MAX_ENTRIES); +} tstamp_rcu SEC(".maps"); + /* maintain per-CPU timestamp at the beginning of contention */ struct { __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); @@ -160,6 +172,7 @@ const volatile int has_addr; const volatile int has_cgroup; const volatile int has_slab; const volatile int has_mmap_lock; +const volatile int has_rcu_lock; const volatile int needs_callstack; const volatile int stack_skip; const volatile int lock_owner; @@ -172,6 +185,7 @@ const volatile int aggr_mode; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -218,7 +232,7 @@ static inline __u64 get_current_cgroup_id(void) return BPF_CORE_READ(cgrp, kn, id); } -static inline int can_record(u64 *ctx) +static __always_inline int can_record(__u64 lock, __u32 flags) { bool is_addr_ok = false; @@ -242,7 +256,6 @@ static inline int can_record(u64 *ctx) if (has_type) { __u8 *ok; - __u32 flags = (__u32)ctx[1]; ok = bpf_map_lookup_elem(&type_filter, &flags); if (!ok) @@ -251,10 +264,9 @@ static inline int can_record(u64 *ctx) if (has_addr) { __u8 *ok; - __u64 addr = ctx[0]; - ok = bpf_map_lookup_elem(&addr_filter, &addr); - if (!ok && !has_slab && !has_mmap_lock) + ok = bpf_map_lookup_elem(&addr_filter, &lock); + if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock) return 0; is_addr_ok = !!ok; @@ -272,26 +284,27 @@ static inline int can_record(u64 *ctx) if (is_addr_ok) return 1; - /* slab and mmap_lock are part of the addr_filter */ + /* slab, mmap_lock, and rcu_lock are part of the addr_filter */ if (has_slab && bpf_get_kmem_cache) { __u8 *ok; - __u64 addr = ctx[0]; long kmem_cache_addr; - kmem_cache_addr = (long)bpf_get_kmem_cache(addr); + kmem_cache_addr = (long)bpf_get_kmem_cache(lock); ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr); if (ok) return 1; - else if (!has_mmap_lock) + else if (!has_mmap_lock && !has_rcu_lock) return 0; } - if (has_mmap_lock) { - __u64 lock = ctx[0]; - __u32 flag = ctx[1]; + if (has_mmap_lock || has_rcu_lock) { + __u32 ltype = check_lock_type(lock, flags); - if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK) - return 0; + if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK) + return 1; + if (has_rcu_lock && ltype == LCD_F_RCU_LOCK) + return 1; + return 0; } return 1; @@ -351,6 +364,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) { __u32 pid; struct tstamp_data *pelem; + void *map; /* Use per-cpu array map for spinlock and rwlock */ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) { @@ -363,8 +377,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) return pelem; } + map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp; pid = bpf_get_current_pid_tgid(); - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); /* Do not update the element for nested locks */ if (pelem && pelem->lock) return NULL; @@ -372,12 +387,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) if (pelem == NULL) { struct tstamp_data zero = {}; - if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) { + if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) { __sync_fetch_and_add(&task_fail, 1); return NULL; } - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); if (pelem == NULL) { __sync_fetch_and_add(&task_fail, 1); return NULL; @@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags) } } -SEC("tp_btf/contention_begin") -int contention_begin(u64 *ctx) +static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags) { struct tstamp_data *pelem; - if (!enabled || !can_record(ctx)) + if (!enabled || !can_record(lock, flags)) return 0; - pelem = get_tstamp_elem(ctx[1]); + pelem = get_tstamp_elem(flags); if (pelem == NULL) return 0; pelem->timestamp = bpf_ktime_get_ns(); - pelem->lock = (__u64)ctx[0]; - pelem->flags = (__u32)ctx[1]; + pelem->lock = lock; + pelem->flags = flags; if (aggr_mode == LOCK_AGGR_CGROUP) pelem->cgroup_id = get_current_cgroup_id(); @@ -588,8 +602,7 @@ int contention_begin(u64 *ctx) return 0; } -SEC("tp_btf/contention_end") -int contention_end(u64 *ctx) +static __always_inline int do_contention_end(__u64 lock, __s32 ret) { __u32 pid = 0, idx = 0; struct tstamp_data *pelem; @@ -597,7 +610,7 @@ int contention_end(u64 *ctx) struct contention_data *data; __u64 timestamp; __u64 duration; - bool need_delete = false; + void *ts_map = NULL; if (!enabled) return 0; @@ -613,14 +626,19 @@ int contention_end(u64 *ctx) */ pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx); if (pelem && pelem->lock) { - if (pelem->lock != ctx[0]) + if (pelem->lock != lock) return 0; } else { pid = bpf_get_current_pid_tgid(); pelem = bpf_map_lookup_elem(&tstamp, &pid); - if (!pelem || pelem->lock != ctx[0]) - return 0; - need_delete = true; + if (pelem && pelem->lock == lock) { + ts_map = &tstamp; + } else { + pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid); + if (!pelem || pelem->lock != lock) + return 0; + ts_map = &tstamp_rcu; + } } timestamp = bpf_ktime_get_ns(); @@ -648,7 +666,6 @@ int contention_end(u64 *ctx) */ else { u32 i = 0; - s32 ret = (s32)ctx[1]; u64 *buf; otdata->timestamp = timestamp; @@ -713,7 +730,7 @@ int contention_end(u64 *ctx) if (lock_owner) key.pid = pelem->flags; else { - if (!need_delete) + if (!ts_map) pid = bpf_get_current_pid_tgid(); key.pid = pid; } @@ -795,11 +812,109 @@ int contention_end(u64 *ctx) check_lock_delay(pelem->lock); pelem->lock = 0; - if (need_delete) - bpf_map_delete_elem(&tstamp, &pid); + if (ts_map) + bpf_map_delete_elem(ts_map, &pid); return 0; } +SEC("tp_btf/contention_begin") +int contention_begin(u64 *ctx) +{ + return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]); +} + +SEC("tp_btf/contention_end") +int contention_end(u64 *ctx) +{ + return do_contention_end((__u64)ctx[0], (__s32)ctx[1]); +} + +/* RCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return do_contention_end(rcu_state_addr, 0); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return do_contention_end(rcu_state_addr, 0); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return do_contention_end(rcu_state_addr, 0); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return do_contention_end(rcu_state_addr, 0); +} + +/* SRCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(end_timestamp) { -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU and SRCU 2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers @ 2026-10-17 6:08 ` sashiko-bot 0 siblings, 0 replies; 24+ messages in thread From: sashiko-bot @ 2026-10-17 6:08 UTC (permalink / raw) To: Ian Rogers; +Cc: linux-perf-users Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf lock contention: wrong caller stack skip in do_contention_begin() for fentry -- commit 9ca74d6f51b3408e9a151b639dd644cd10562df1 Author: Ian Rogers <irogers@google.com> perf lock contention: Extend BPF lock contention to RCU and SRCU This commit extends 'perf lock contention --use-bpf' to track RCU and SRCU grace-period and barrier waits using fentry/fexit BPF programs and adds lock type and name filtering support for rcu and srcu. > diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c > index 0f2118006005a..257e3402e8151 100644 > --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c > +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c [ ... ] > @@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags) > } > } > > -SEC("tp_btf/contention_begin") > -int contention_begin(u64 *ctx) > +static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags) > { > struct tstamp_data *pelem; > > - if (!enabled || !can_record(ctx)) > + if (!enabled || !can_record(lock, flags)) > return 0; > > - pelem = get_tstamp_elem(ctx[1]); > + pelem = get_tstamp_elem(flags); > if (pelem == NULL) > return 0; > > pelem->timestamp = bpf_ktime_get_ns(); > - pelem->lock = (__u64)ctx[0]; > - pelem->flags = (__u32)ctx[1]; > + pelem->lock = lock; > + pelem->flags = flags; > if (aggr_mode == LOCK_AGGR_CGROUP) > pelem->cgroup_id = get_current_cgroup_id(); > [ ... ] > @@ -795,11 +812,109 @@ int contention_end(u64 *ctx) > check_lock_delay(pelem->lock); > > pelem->lock = 0; > - if (need_delete) > - bpf_map_delete_elem(&tstamp, &pid); > + if (ts_map) > + bpf_map_delete_elem(ts_map, &pid); > return 0; > } > > +SEC("tp_btf/contention_begin") > +int contention_begin(u64 *ctx) > +{ > + return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]); > +} > + > +SEC("tp_btf/contention_end") > +int contention_end(u64 *ctx) > +{ > + return do_contention_end((__u64)ctx[0], (__s32)ctx[1]); > +} > + > +/* RCU grace-period synchronize / barrier contention hooks */ > +SEC("fentry/synchronize_rcu") > +int BPF_PROG(fentry_synchronize_rcu) > +{ > + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); > +} [Severity: Medium] In do_contention_begin() in tools/perf/util/bpf_skel/lock_contention.bpf.c, bpf_get_stackid() uses stack_skip (which defaults to CONTENTION_STACK_SKIP = 4) unconditionally for both tp_btf/contention_begin and the new fentry RCU and SRCU programs: skip_owner: pelem->stack_id = bpf_get_stackid(ctx, &stacks, BPF_F_FAST_STACK_CMP | stack_skip); For tp_btf/contention_begin, the tracepoint fires inside a lock slowpath via tracepoint dispatch: caller -> mutex_lock -> __mutex_lock_slowpath -> __mutex_lock -> trace_contention_begin -> __bpf_trace_contention_begin -> bpf_trace_run2 -> bpf_prog Skipping 4 frames lands inside the locking functions (.sched.text, .spinlock.text, or trace_contention_begin), which lock_contention_get_name() in tools/perf/util/bpf_lock_contention.c then skips via machine__is_lock_function() to reach the direct caller: /* LOCK_AGGR_CALLER: skip lock internal functions */ while (machine__is_lock_function(machine, stack_trace[idx]) && idx < con->max_stack - 1) idx++; addr = stack_trace[idx]; In contrast, fentry_synchronize_rcu() (and the other six RCU/SRCU fentry programs) attach directly to the top-level API functions. In an fentry program, the callchain captured by bpf_get_stackid_raw_tp() -> perf_callchain_kernel() only has two frames before the traced function itself: ip[0] is the BPF prog, ip[1] is the BPF trampoline, ip[2] is the traced function (e.g., synchronize_rcu), ip[3] is the direct caller, and ip[4] is the caller's caller. Won't skipping stack_skip = 4 frames in bpf_get_stackid() skip past ip[3], dropping the direct caller from the recorded stack trace, attributing RCU and SRCU contention in LOCK_AGGR_CALLER mode to the caller's caller (ip[4]), and causing callstack filtering (-S) in match_callstack_filter() to miss the direct caller? Also, if the fentry stack skip is adjusted so that the traced RCU/SRCU function itself can appear at stack_trace[0], machine__is_lock_function() in tools/perf/util/machine.c only checks .sched.text, .spinlock.text, __traceiter_contention_begin, and trace_contention_begin. Since synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(), and srcu_barrier() are not in .sched.text or .spinlock.text, should machine__is_lock_function() or the fentry skip depth account for those functions as well? [ ... ] -- Sashiko AI review · https://sashiko.dev/#/patchset/20261017054555.1655407-1-irogers@google.com?part=5 ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support 2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (4 preceding siblings ...) 2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers @ 2026-10-17 17:14 ` Ian Rogers 2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers ` (5 more replies) 5 siblings, 6 replies; 24+ messages in thread From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu This series adds a new BPF sample filter predicate, `lock_held(<lock>)` (along with `lock_held_read(<lock>)` / `lock_held_shared(<lock>)`, `lock_held_write(<lock>)` / `lock_held_exclusive(<lock>)`, and `:R` / `:W` mode suffixes), to `perf record --filter`, allowing perf event samples to be recorded only when a specified kernel lock is held (or not held with `!`) by the running kernel thread or CPU. It also extends both `lock_held()` and `perf lock contention --use-bpf` to support RCU and SRCU, and fixes a pre-existing `lock_stat` filtering bug in `perf lock contention`. For example: # perf record -a -g -e cycles --filter 'lock_held(&cgroup_mutex)' -- ... # perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' -- ... # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' -- ... # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- ... # perf lock contention -a -b -Y rcu,srcu -- sleep 1 Supported `<lock>` specifications match `perf lock contention`: - Global/static kernel lock symbols (with or without leading `&`, e.g. `lock_held(&cgroup_mutex)` or `lock_held(cgroup_mutex)`) - Hexadecimal kernel lock addresses - Slab cache names prefixed with `&` (e.g. `lock_held(&dentry)`, `lock_held_read(&dentry)`) - Well-known per-process, semi-global, and RCU locks: `mmap_lock`, `siglock`, `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` - Lock types and flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `rwlock:R`, `rwlock:W`, `rt-mutex`, `percpu-rwsem`, `pcpu-sem:W`, `srcu`, `srcu:R`, `srcu:W`, `rcu:R`, `rcu:W`, `semaphore` Limitations and weaknesses in determining whether a lock is held: - Inlined lock/unlock operations: - When the kernel is configured to inline spinlock or rwlock operations (`CONFIG_INLINE_SPIN_LOCK`, `CONFIG_INLINE_READ_LOCK`, `CONFIG_INLINE_WRITE_LOCK`, `!CONFIG_UNINLINE_SPIN_UNLOCK`, `CONFIG_INLINE_*_UNLOCK*`, etc.), `_raw_spin_lock*` / `_raw_spin_unlock*` and `_raw_read_lock*` / `_raw_write_lock*` are inlined into callers and have no BTF function entry points for `fentry`/`fexit` to attach to. `configure_lock_progs()` verifies that both acquire and release entry points exist in BTF before enabling a lock family. - Fast-path `percpu_down_read()` / `percpu_up_read()` and `srcu_read_lock_fast()` / `srcu_read_unlock_fast()` are inlined per-CPU counter increments without function calls, so only `percpu_down_write()` / `percpu_up_write()` and out-of-line `__srcu_read_lock*()` / `__srcu_read_unlock*()` can be hooked. - On `!CONFIG_PREEMPT_RCU` kernels, `rcu_read_lock()` compiles to `preempt_disable()` and does not maintain `current->rcu_read_lock_nesting` in `task_struct`, so `lock_held(rcu)` read-side inspection requires `CONFIG_PREEMPT_RCU`. - Lack of an inspectable owner field on non-exclusive / spinning locks: - While `struct mutex`, `struct rt_mutex`, and write-locked `struct rw_semaphore` store the owning `task_struct *` in `lock->owner` (allowing `perf_sample_filter` to inspect kernel memory directly on sample overflow when filtering by lock address), spinlocks, rwlocks, semaphores, SRCU, and reader-locked `rw_semaphore`s (where `__up_read()` does not clear `sem->owner` when `!CONFIG_DEBUG_RWSEMS`) do not maintain an authoritative per-task owner pointer. - Tracking these locks (as well as slab-cache and lock-type filters) relies on `fexit` (acquire) and `fentry` (release) BPF hooks maintaining the `held_locks` BPF hash map: - Any such lock already held before `perf record` attaches its BPF programs will not be present in `held_locks` until it is released and re-acquired. - The attached `fentry`/`fexit` hooks execute on every kernel lock acquire and release for the hooked lock family across the system, adding tracing overhead outside of perf sample handling. Patch overview: 1. "perf lock contention: Share common BPF and lock type definitions in lock_data.h": Moves `lock_type_table` to `util/lock-contention.c` and `LCB_F_*` flags, CO-RE structs, and common BPF lock helpers (`get_lock_owner`, `get_current_mmap_lock`, `get_current_siglock`, `get_lock_addr`, `check_lock_type`, `do_collect_lock_syms`, `do_slab_cache_iter`) into `tools/perf/util/bpf_skel/lock_data.h`. 2. "perf bpf-filter: Add lock_held() sample filter": Implements `lock_held()`, `lock_held_read()`/`lock_held_shared()`, `lock_held_write()`/`lock_held_exclusive()`, and `:R`/`:W` mode suffixes using direct lock owner inspection where possible (`mutex`, `rt_mutex`, write-locked `rw_semaphore`) and a `held_locks` BPF map updated by selectively attached `fentry`/`fexit` hooks. 3. "perf test: Add tests for lock_held() BPF sample filter": Adds unit tests in `tests/parse-events.c` and shell tests in `tests/shell/record_bpf_filter.sh`. 4. "perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU": Supports `rcu` (`rcu:R` via direct `current->rcu_read_lock_nesting` inspection with zero tracing hooks; `rcu:W` via `synchronize_rcu*()` / `rcu_barrier()` hooks), `rcu-trace` (via `current->trc_reader_nesting`), and `srcu` (`srcu:R` via `__srcu_read_lock*` / `__srcu_read_unlock*`; `srcu:W` via `synchronize_srcu*()` / `srcu_barrier()`). 5. "perf lock contention: Extend BPF lock contention to RCU and SRCU": Extends `perf lock contention --use-bpf` to track RCU and SRCU grace-period and barrier waits (`-Y rcu`, `-Y srcu`, `-L rcu`). 6. "perf lock contention: Filter events before creating lock_stat": Fixes `report_lock_contention_begin_event()` in `builtin-lock.c` to apply `-Y`, `-L`, and `-S` filters before calling `lock_stat_findnew()`. Changes in v2: - Guard `BPF_CORE_READ(mutex, owner.counter)` with `bpf_core_field_exists` and fall back to `struct rt_mutex_base` in `get_lock_owner()` so CO-RE relocation succeeds on `CONFIG_PREEMPT_RT=y` kernels. - Add `CONFIG_DEBUG_LOCK_ALLOC=y` lock entry points (`_mutex_lock_nest_lock`, `_mutex_lock_killable`, `_mutex_trylock_nest_lock`, `_down_write_nest_lock`, `_raw_spin_lock_nest_lock`, `_rt_mutex_lock_nest_lock`) to `sample_filter.bpf.c` and `lock_progs[]`. - Verify in `supported_lock_prog_groups()` that both acquire and all unlock BTF entry points exist for a lock family before enabling its tracing hooks, preventing acquire-only attachment when spinlock/rwlock unlock functions are inlined. - Enable `LOCK_PROG_PERCPU_RWSEM` alongside `LOCK_PROG_RWSEM_WRITE` when a symbol name ends in `rwsem` (such as `cgroup_threadgroup_rwsem`). - Gate `record_lock_acquired(ctx, LCB_F_READ)` in `fexit_downgrade_write` on `track_rwsem_read` and verify non-zero read/write counts in `record_lock_released_addr()` before decrementing aggregate slab/type counts. - Adjust stack skip depth by 2 for `fentry` RCU/SRCU programs in `lock_contention.bpf.c` and recognize `synchronize_rcu*`, `synchronize_srcu*`, `rcu_barrier`, and `srcu_barrier` in `machine__is_lock_function()`. - Add patch 6 fixing `report_lock_contention_begin_event()` to check type, address, and callstack filters before creating `lock_stat` entries. Assisted-by: Antigravity:gemini-3.1-pro Ian Rogers (6): perf lock contention: Share common BPF and lock type definitions in lock_data.h perf bpf-filter: Add lock_held() sample filter perf test: Add tests for lock_held() BPF sample filter perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU perf lock contention: Extend BPF lock contention to RCU and SRCU perf lock contention: Filter events before creating lock_stat tools/perf/Documentation/perf-lock.txt | 7 +- tools/perf/Documentation/perf-record.txt | 51 +- tools/perf/builtin-lock.c | 88 +- tools/perf/tests/parse-events.c | 121 ++ tools/perf/tests/shell/lock_contention.sh | 14 + tools/perf/tests/shell/record_bpf_filter.sh | 123 ++ tools/perf/util/bpf-filter.c | 1188 ++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 46 +- tools/perf/util/bpf_lock_contention.c | 167 ++- .../perf/util/bpf_skel/lock_contention.bpf.c | 427 +++--- tools/perf/util/bpf_skel/lock_data.h | 368 ++++- tools/perf/util/bpf_skel/sample-filter.h | 71 + tools/perf/util/bpf_skel/sample_filter.bpf.c | 1096 +++++++++++++++ tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 7 + tools/perf/util/lock-contention.c | 22 + tools/perf/util/lock-contention.h | 15 + tools/perf/util/machine.c | 11 + tools/perf/util/parse-events.c | 3 +- 20 files changed, 3507 insertions(+), 362 deletions(-) -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers @ 2026-10-17 17:14 ` Ian Rogers 2026-10-17 17:31 ` sashiko-bot 2026-10-17 17:14 ` [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers ` (4 subsequent siblings) 5 siblings, 1 reply; 24+ messages in thread From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Move lock_type_table, LCB_F_* flags, and the common BPF lock helpers (get_lock_owner, get_current_mmap_lock, get_current_siglock, get_lock_addr, check_lock_type, do_collect_zone_lock, do_collect_lock_syms, do_slab_cache_iter) and shared CO-RE structs into tools/perf/util/bpf_skel/lock_data.h so they can be reused by BPF sample filtering. Also add rw_semaphore.count and struct rt_mutex_base to the minimal vmlinux.h and extend get_lock_owner() to support rt_mutex and check writer-ownership and handoff flags. Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/builtin-lock.c | 27 -- .../perf/util/bpf_skel/lock_contention.bpf.c | 265 +------------- tools/perf/util/bpf_skel/lock_data.h | 324 ++++++++++++++++++ tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 5 + tools/perf/util/lock-contention.c | 18 + tools/perf/util/lock-contention.h | 13 + 6 files changed, 378 insertions(+), 274 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index d5c0d55cb82d..feaf1cfd08cf 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -1477,33 +1477,6 @@ static void sort_result(void) } } -static const struct { - unsigned int flags; - /* - * Name of the lock flags (access), with delimeter ':'. - * For example, rwsem:R of rwsem:W. - */ - const char *flags_name; - /* Name of the lock (type), for example, rwlock or rwsem. */ - const char *lock_name; -} lock_type_table[] = { - { 0, "semaphore", "semaphore" }, - { LCB_F_SPIN, "spinlock", "spinlock" }, - { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, - { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, - { LCB_F_READ, "rwsem:R", "rwsem" }, - { LCB_F_WRITE, "rwsem:W", "rwsem" }, - { LCB_F_RT, "rt-mutex", "rt-mutex" }, - { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, - { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, - { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, - { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, - { LCB_F_MUTEX, "mutex", "mutex" }, - { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, - /* alias for optimistic spinning only */ - { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, -}; - static const char *get_type_flags_name(unsigned int flags) { flags &= LCB_F_TYPE_MASK; diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0d9c6f55050e..0f2118006005 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -8,23 +8,9 @@ #include "lock_data.h" -/* for collect_lock_syms(). 4096 was rejected by the verifier */ -#define MAX_CPUS 1024 - -/* for collect_zone_lock(). It should be more than the actual zones. */ -#define MAX_ZONES 10 - /* for do_lock_delay(). Arbitrarily set to 1 million. */ #define MAX_LOOP (1U << 20) -/* lock contention flags from include/trace/events/lock.h */ -#define LCB_F_SPIN (1U << 0) -#define LCB_F_READ (1U << 1) -#define LCB_F_WRITE (1U << 2) -#define LCB_F_RT (1U << 3) -#define LCB_F_PERCPU (1U << 4) -#define LCB_F_MUTEX (1U << 5) - /* callstack storage */ struct { __uint(type, BPF_MAP_TYPE_STACK_TRACE); @@ -96,13 +82,6 @@ struct { __uint(max_entries, MAX_ENTRIES); } task_data SEC(".maps"); -struct { - __uint(type, BPF_MAP_TYPE_HASH); - __uint(key_size, sizeof(__u64)); - __uint(value_size, sizeof(__u32)); - __uint(max_entries, MAX_ENTRIES); -} lock_syms SEC(".maps"); - struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(key_size, sizeof(__u32)); @@ -138,6 +117,13 @@ struct { __uint(max_entries, 1); } cgroup_filter SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, MAX_ENTRIES); +} lock_syms SEC(".maps"); + struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(key_size, sizeof(long)); @@ -159,22 +145,6 @@ struct { __uint(max_entries, 1); } lock_delays SEC(".maps"); -struct rw_semaphore___old { - struct task_struct *owner; -} __attribute__((preserve_access_index)); - -struct rw_semaphore___new { - atomic_long_t owner; -} __attribute__((preserve_access_index)); - -struct mm_struct___old { - struct rw_semaphore mmap_sem; -} __attribute__((preserve_access_index)); - -struct mm_struct___new { - struct rw_semaphore mmap_lock; -} __attribute__((preserve_access_index)); - struct cas_ctx { struct contention_data *data; u64 duration; @@ -182,8 +152,6 @@ struct cas_ctx { int min_done; }; -extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; - /* control flags */ const volatile int has_cpu; const volatile int has_task; @@ -202,6 +170,11 @@ const volatile int lock_delay; /* determine the key of lock stat */ const volatile int aggr_mode; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + int enabled; int perf_subsys_id = -1; @@ -222,8 +195,6 @@ int data_map_full; struct task_struct *bpf_task_from_pid(s32 pid) __ksym __weak; void bpf_task_release(struct task_struct *p) __ksym __weak; -static inline __u32 check_lock_type(__u64 lock, __u32 flags); - static inline __u64 get_current_cgroup_id(void) { struct task_struct *task; @@ -347,85 +318,6 @@ static inline int update_task_data(struct task_struct *task) return 0; } -#ifndef __has_builtin -# define __has_builtin(x) 0 -#endif - -static inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) -{ - struct task_struct *task; - __u64 owner = 0; - - if (flags & LCB_F_MUTEX) { - struct mutex *mutex = (void *)lock; - owner = BPF_CORE_READ(mutex, owner.counter); - } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { - /* - * Support for the BPF_TYPE_MATCHES argument to the - * __builtin_preserve_type_info builtin was added at some point during - * development of clang 15 and it's what is needed for - * bpf_core_type_matches. - */ -#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 - if (bpf_core_type_matches(struct rw_semaphore___old)) { - struct rw_semaphore___old *rwsem = (void *)lock; - owner = (unsigned long)BPF_CORE_READ(rwsem, owner); - } else if (bpf_core_type_matches(struct rw_semaphore___new)) { - struct rw_semaphore___new *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); - } -#else - /* assume new struct */ - struct rw_semaphore *rwsem = (void *)lock; - owner = BPF_CORE_READ(rwsem, owner.counter); -#endif - } - - if (!owner) - return NULL; - - task = (void *)(owner & ~7UL); - return task; -} - -static inline __u32 check_lock_type(__u64 lock, __u32 flags) -{ - struct task_struct *curr; - struct mm_struct___old *mm_old; - struct mm_struct___new *mm_new; - struct sighand_struct *sighand; - - switch (flags) { - case LCB_F_READ: /* rwsem */ - case LCB_F_WRITE: - curr = bpf_get_current_task_btf(); - if (curr->mm == NULL) - break; - mm_new = (void *)curr->mm; - if (bpf_core_field_exists(mm_new->mmap_lock)) { - if (&mm_new->mmap_lock == (void *)lock) - return LCD_F_MMAP_LOCK; - break; - } - mm_old = (void *)curr->mm; - if (bpf_core_field_exists(mm_old->mmap_sem)) { - if (&mm_old->mmap_sem == (void *)lock) - return LCD_F_MMAP_LOCK; - } - break; - case LCB_F_SPIN: /* spinlock */ - curr = bpf_get_current_task_btf(); - sighand = curr->sighand; - - if (sighand && &sighand->siglock == (void *)lock) - return LCD_F_SIGHAND_LOCK; - break; - default: - break; - } - return 0; -} - static inline long delay_callback(__u64 idx, void *arg) { __u64 target = *(__u64 *)arg; @@ -908,145 +800,24 @@ int contention_end(u64 *ctx) return 0; } -extern struct rq runqueues __ksym; - -const volatile __u64 contig_page_data_addr; -const volatile __u64 node_data_addr; -const volatile int nr_nodes; -const volatile int sizeof_zone; - -struct rq___old { - raw_spinlock_t lock; -} __attribute__((preserve_access_index)); - -struct rq___new { - raw_spinlock_t __lock; -} __attribute__((preserve_access_index)); - -static void collect_zone_lock(void) -{ - __u64 nr_zones, zone_off; - __u64 lock_addr, lock_off; - __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; - - zone_off = offsetof(struct pglist_data, node_zones); - lock_off = offsetof(struct zone, lock); - - if (contig_page_data_addr) { - struct pglist_data *contig_page_data; - - contig_page_data = (void *)(long)contig_page_data_addr; - nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); - - for (int i = 0; i < MAX_ZONES; i++) { - __u64 zone_addr; - - if (i >= nr_zones) - break; - - zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } else if (nr_nodes > 0) { - struct pglist_data **node_data = (void *)(long)node_data_addr; - - for (int i = 0; i < nr_nodes; i++) { - struct pglist_data *pgdat = NULL; - int err; - - err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); - if (err < 0 || pgdat == NULL) - break; - - nr_zones = BPF_CORE_READ(pgdat, nr_zones); - for (int k = 0; k < MAX_ZONES; k++) { - __u64 zone_addr; - - if (k >= nr_zones) - break; - - zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; - lock_addr = zone_addr + lock_off; - - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - } - } -} - SEC("raw_tp/bpf_test_finish") -int BPF_PROG(collect_lock_syms) +int BPF_PROG(end_timestamp) { - __u64 lock_addr, lock_off; - __u32 lock_flag; - - if (bpf_core_field_exists(struct rq___new, __lock)) - lock_off = offsetof(struct rq___new, __lock); - else - lock_off = offsetof(struct rq___old, lock); - - for (int i = 0; i < MAX_CPUS; i++) { - struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); - - if (rq == NULL) - break; - - lock_addr = (__u64)(void *)rq + lock_off; - lock_flag = LOCK_CLASS_RQLOCK; - bpf_map_update_elem(&lock_syms, &lock_addr, &lock_flag, BPF_ANY); - } - - collect_zone_lock(); - + end_ts = bpf_ktime_get_ns(); return 0; } SEC("raw_tp/bpf_test_finish") -int BPF_PROG(end_timestamp) +int BPF_PROG(collect_lock_syms) { - end_ts = bpf_ktime_get_ns(); - return 0; + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); } -/* - * bpf_iter__kmem_cache added recently so old kernels don't have it in the - * vmlinux.h. But we cannot add it here since it will cause a compiler error - * due to redefinition of the struct on later kernels. - * - * So it uses a CO-RE trick to access the member only if it has the type. - * This will support both old and new kernels without compiler errors. - */ -struct bpf_iter__kmem_cache___new { - struct kmem_cache *s; -} __attribute__((preserve_access_index)); - SEC("iter/kmem_cache") int slab_cache_iter(void *ctx) { - struct kmem_cache *s = NULL; - struct slab_cache_data d; - const char *nameptr; - - if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { - struct bpf_iter__kmem_cache___new *iter = ctx; - - s = iter->s; - } - - if (s == NULL) - return 0; - - nameptr = s->name; - bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); - - d.id = ++slab_cache_id << LCB_F_SLAB_ID_SHIFT; - if (d.id >= LCB_F_SLAB_ID_END) - return 0; - - bpf_map_update_elem(&slab_caches, &s, &d, BPF_NOEXIST); - return 0; + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); } char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index 652e114e6b87..bf39128e0be1 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -45,6 +45,14 @@ struct contention_task_data { #define LCB_F_SLAB_ID_END (1U << 26) #define LCB_F_SLAB_ID_MASK 0x03FF0000U +/* lock contention flags from include/trace/events/lock.h */ +#define LCB_F_SPIN (1U << 0) +#define LCB_F_READ (1U << 1) +#define LCB_F_WRITE (1U << 2) +#define LCB_F_RT (1U << 3) +#define LCB_F_PERCPU (1U << 4) +#define LCB_F_MUTEX (1U << 5) + #define LCB_F_TYPE_MAX (1U << 7) #define LCB_F_TYPE_MASK 0x0000007FU @@ -76,4 +84,320 @@ struct slab_cache_data { char name[SLAB_NAME_MAX]; }; +#ifdef __BPF__ + +/* for collect_lock_syms(). 4096 was rejected by the verifier */ +#define MAX_CPUS 1024 + +/* for collect_zone_lock(). It should be more than the actual zones. */ +#define MAX_ZONES 10 + +extern struct kmem_cache *bpf_get_kmem_cache(u64 addr) __ksym __weak; +extern struct rq runqueues __ksym __weak; + +#ifndef __has_builtin +# define __has_builtin(x) 0 +#endif + +struct rw_semaphore___old { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + +struct rw_semaphore___new { + atomic_long_t count; + atomic_long_t owner; +} __attribute__((preserve_access_index)); + +struct mm_struct___old { + struct rw_semaphore mmap_sem; +} __attribute__((preserve_access_index)); + +struct mm_struct___new { + struct rw_semaphore mmap_lock; +} __attribute__((preserve_access_index)); + +struct rq___old { + raw_spinlock_t lock; +} __attribute__((preserve_access_index)); + +struct rq___new { + raw_spinlock_t __lock; +} __attribute__((preserve_access_index)); + +/* + * bpf_iter__kmem_cache added recently so old kernels don't have it in the + * vmlinux.h. But we cannot add it here since it will cause a compiler error + * due to redefinition of the struct on later kernels. + * + * So it uses a CO-RE trick to access the member only if it has the type. + * This will support both old and new kernels without compiler errors. + */ +struct bpf_iter__kmem_cache___new { + struct kmem_cache *s; +} __attribute__((preserve_access_index)); + +static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) +{ + struct task_struct *task; + __u64 owner = 0; + + if (flags & LCB_F_MUTEX) { + struct mutex *mutex = (void *)lock; + + if (bpf_core_field_exists(mutex->owner.counter)) { + owner = BPF_CORE_READ(mutex, owner.counter); + if (owner & 0x4UL) + return NULL; + } else if (bpf_core_type_exists(struct rt_mutex_base)) { + struct rt_mutex_base *rtm = (void *)lock; + + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } + } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { + struct rw_semaphore___new *rwsem = (void *)lock; + + if (flags == LCB_F_WRITE && bpf_core_field_exists(rwsem->count.counter)) { + __u64 count = BPF_CORE_READ(rwsem, count.counter); + + /* Must have RWSEM_WRITER_LOCKED (bit 0) set */ + if (!(count & 1UL)) + return NULL; + } + /* + * Support for the BPF_TYPE_MATCHES argument to the + * __builtin_preserve_type_info builtin was added at some point during + * development of clang 15 and it's what is needed for + * bpf_core_type_matches. + */ +#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 + if (bpf_core_type_matches(struct rw_semaphore___old)) { + struct rw_semaphore___old *old = (void *)lock; + + owner = (unsigned long)BPF_CORE_READ(old, owner); + } else if (bpf_core_type_matches(struct rw_semaphore___new)) { + owner = BPF_CORE_READ(rwsem, owner.counter); + } +#else + /* assume new struct */ + if (bpf_core_field_exists(rwsem->owner.counter)) + owner = BPF_CORE_READ(rwsem, owner.counter); +#endif + /* RWSEM_READER_OWNED is bit 0 of owner */ + if (flags == LCB_F_WRITE && (owner & 1UL)) + return NULL; + } else if (flags == LCB_F_RT) { + struct rt_mutex_base *rtm = (void *)lock; + + if (bpf_core_type_exists(struct rt_mutex_base)) { + owner = (__u64)BPF_CORE_READ(rtm, owner); + return (void *)(owner & ~1UL); + } + } + + if (!owner) + return NULL; + + task = (void *)(owner & ~7UL); + return task; +} + +static __always_inline __u64 get_current_mmap_lock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct mm_struct___new *mm_new = NULL; + struct mm_struct___old *mm_old = NULL; + __u64 mm; + + if (!curr) + return 0; + + mm = (__u64)BPF_CORE_READ(task, mm); + if (!mm) + return 0; + + if (bpf_core_field_exists(mm_new->mmap_lock)) + return mm + bpf_core_field_offset(mm_new->mmap_lock); + + if (bpf_core_field_exists(mm_old->mmap_sem)) + return mm + bpf_core_field_offset(mm_old->mmap_sem); + + return 0; +} + +static __always_inline __u64 get_current_siglock(__u64 curr) +{ + struct task_struct *task = (void *)curr; + struct sighand_struct *sighand = NULL; + __u64 sh; + + if (!curr) + return 0; + + sh = (__u64)BPF_CORE_READ(task, sighand); + if (!sh) + return 0; + + return sh + bpf_core_field_offset(sighand->siglock); +} + +/* + * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. + * + * Using `bpf_get_func_arg()` rather than reading `ctx[0]` directly ensures the + * BPF verifier tracks the address as a `SCALAR_VALUE` instead of a typed + * `PTR_TO_BTF_ID`, which is required when passing the address to kfuncs such + * as `bpf_get_kmem_cache(u64 addr)`. + */ +static __always_inline __u64 get_lock_addr(void *ctx) +{ + __u64 lock_addr = 0; + + bpf_get_func_arg(ctx, 0, &lock_addr); + return lock_addr; +} + +static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) +{ + __u64 curr; + + switch (flags) { + case LCB_F_READ: /* rwsem */ + case LCB_F_WRITE: + curr = bpf_get_current_task(); + if (get_current_mmap_lock(curr) == lock) + return LCD_F_MMAP_LOCK; + break; + case LCB_F_SPIN: /* spinlock */ + curr = bpf_get_current_task(); + if (get_current_siglock(curr) == lock) + return LCD_F_SIGHAND_LOCK; + break; + default: + break; + } + return 0; +} + +static __always_inline void do_collect_zone_lock(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 nr_zones, zone_off; + __u64 lock_addr, lock_off; + __u32 lock_flag = LOCK_CLASS_ZONE_LOCK; + + zone_off = offsetof(struct pglist_data, node_zones); + lock_off = offsetof(struct zone, lock); + + if (contig_page_data_addr) { + struct pglist_data *contig_page_data; + + contig_page_data = (void *)(long)contig_page_data_addr; + nr_zones = BPF_CORE_READ(contig_page_data, nr_zones); + + for (int i = 0; i < MAX_ZONES; i++) { + __u64 zone_addr; + + if (i >= nr_zones) + break; + + zone_addr = contig_page_data_addr + (sizeof_zone * i) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } else if (nr_nodes > 0) { + struct pglist_data **node_data = (void *)(long)node_data_addr; + + for (int i = 0; i < nr_nodes; i++) { + struct pglist_data *pgdat = NULL; + int err; + + err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); + if (err < 0) + break; + if (pgdat == NULL) + continue; + + nr_zones = BPF_CORE_READ(pgdat, nr_zones); + for (int k = 0; k < MAX_ZONES; k++) { + __u64 zone_addr; + + if (k >= nr_zones) + break; + + zone_addr = (__u64)(void *)pgdat + (sizeof_zone * k) + zone_off; + lock_addr = zone_addr + lock_off; + + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + } +} + +static __always_inline int do_collect_lock_syms(void *lock_syms_map, + __u64 contig_page_data_addr, + __u64 node_data_addr, + int nr_nodes, + int sizeof_zone) +{ + __u64 lock_addr, lock_off; + __u32 lock_flag; + + if (bpf_core_field_exists(struct rq___new, __lock)) + lock_off = offsetof(struct rq___new, __lock); + else + lock_off = offsetof(struct rq___old, lock); + + if (&runqueues) { + for (int i = 0; i < MAX_CPUS; i++) { + struct rq *rq = bpf_per_cpu_ptr(&runqueues, i); + + if (rq == NULL) + break; + + lock_addr = (__u64)(void *)rq + lock_off; + lock_flag = LOCK_CLASS_RQLOCK; + bpf_map_update_elem(lock_syms_map, &lock_addr, &lock_flag, BPF_ANY); + } + } + + do_collect_zone_lock(lock_syms_map, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); + + return 0; +} + +static __always_inline int do_slab_cache_iter(void *ctx, void *slab_caches_map, + __u32 *slab_cache_id) +{ + struct kmem_cache *s = NULL; + struct slab_cache_data d = {}; + const char *nameptr; + + if (bpf_core_type_exists(struct bpf_iter__kmem_cache)) { + struct bpf_iter__kmem_cache___new *iter = ctx; + + s = iter->s; + } + + if (s == NULL) + return 0; + + nameptr = s->name; + bpf_probe_read_kernel_str(d.name, sizeof(d.name), nameptr); + + d.id = ++(*slab_cache_id) << LCB_F_SLAB_ID_SHIFT; + if (d.id >= LCB_F_SLAB_ID_END) + return 0; + + bpf_map_update_elem(slab_caches_map, &s, &d, BPF_NOEXIST); + return 0; +} + +#endif /* __BPF__ */ + #endif /* UTIL_BPF_SKEL_LOCK_DATA_H */ diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index 94e42b022ab1..b75ddc6846bb 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -76,6 +76,7 @@ struct signal_struct { } __attribute__((preserve_access_index)); struct rw_semaphore { + atomic_long_t count; atomic_long_t owner; } __attribute__((preserve_access_index)); @@ -83,6 +84,10 @@ struct mutex { atomic_long_t owner; } __attribute__((preserve_access_index)); +struct rt_mutex_base { + struct task_struct *owner; +} __attribute__((preserve_access_index)); + struct kernfs_node { u64 id; } __attribute__((preserve_access_index)); diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index 119a7206f3cd..a1bb78d6e29c 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -22,6 +22,24 @@ struct callstack_filter { static LIST_HEAD(callstack_filters); struct hlist_head *lockhash_table; +const struct perf_lock_type lock_type_table[] = { + { 0, "semaphore", "semaphore" }, + { LCB_F_SPIN, "spinlock", "spinlock" }, + { LCB_F_SPIN | LCB_F_READ, "rwlock:R", "rwlock" }, + { LCB_F_SPIN | LCB_F_WRITE, "rwlock:W", "rwlock" }, + { LCB_F_READ, "rwsem:R", "rwsem" }, + { LCB_F_WRITE, "rwsem:W", "rwsem" }, + { LCB_F_RT, "rt-mutex", "rt-mutex" }, + { LCB_F_RT | LCB_F_READ, "rwlock-rt:R", "rwlock-rt" }, + { LCB_F_RT | LCB_F_WRITE, "rwlock-rt:W", "rwlock-rt" }, + { LCB_F_PERCPU | LCB_F_READ, "pcpu-sem:R", "percpu-rwsem" }, + { LCB_F_PERCPU | LCB_F_WRITE, "pcpu-sem:W", "percpu-rwsem" }, + { LCB_F_MUTEX, "mutex", "mutex" }, + { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, + /* alias for optimistic spinning only */ + { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, +}; + int parse_call_stack(const struct option *opt __maybe_unused, const char *str, int unset __maybe_unused) { diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 59c94190b092..68e9bd77e8a9 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -129,6 +129,19 @@ struct thread_stat { #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +struct perf_lock_type { + unsigned int flags; + /* + * Name of the lock flags (access), with delimiter ':'. + * For example, rwsem:R or rwsem:W. + */ + const char *flags_name; + /* Name of the lock (type), for example, rwlock or rwsem. */ + const char *lock_name; +}; + +extern const struct perf_lock_type lock_type_table[14]; + struct evlist; struct machine; struct target; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h 2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers @ 2026-10-17 17:31 ` sashiko-bot 0 siblings, 0 replies; 24+ messages in thread From: sashiko-bot @ 2026-10-17 17:31 UTC (permalink / raw) To: Ian Rogers; +Cc: linux-perf-users Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf lock: get_lock_owner() conflates waiter and owner rwsem flags -- commit bc2b5a2f6b16ec842dcc1aeeebcdc72f657fd11b Author: Ian Rogers <irogers@google.com> perf lock contention: Share common BPF and lock type definitions in lock_data.h This commit moves lock_type_table, LCB_F_* flags, CO-RE structs, and common BPF lock helpers into tools/perf/util/bpf_skel/lock_data.h so they can be shared with BPF sample filtering. It also extends get_lock_owner() to support rt_mutex and check writer-ownership and handoff flags. > diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h > index 652e114e6b877..bf39128e0be15 100644 > --- a/tools/perf/util/bpf_skel/lock_data.h > +++ b/tools/perf/util/bpf_skel/lock_data.h > @@ -76,4 +84,320 @@ struct slab_cache_data { [ ... ] > +static __always_inline struct task_struct *get_lock_owner(__u64 lock, __u32 flags) > +{ > + struct task_struct *task; > + __u64 owner = 0; > + > + if (flags & LCB_F_MUTEX) { > + struct mutex *mutex = (void *)lock; > + > + if (bpf_core_field_exists(mutex->owner.counter)) { > + owner = BPF_CORE_READ(mutex, owner.counter); > + if (owner & 0x4UL) > + return NULL; > + } else if (bpf_core_type_exists(struct rt_mutex_base)) { > + struct rt_mutex_base *rtm = (void *)lock; > + > + owner = (__u64)BPF_CORE_READ(rtm, owner); > + return (void *)(owner & ~1UL); > + } > + } else if (flags == LCB_F_READ || flags == LCB_F_WRITE) { > + struct rw_semaphore___new *rwsem = (void *)lock; > + > + if (flags == LCB_F_WRITE && bpf_core_field_exists(rwsem->count.counter)) { > + __u64 count = BPF_CORE_READ(rwsem, count.counter); > + > + /* Must have RWSEM_WRITER_LOCKED (bit 0) set */ > + if (!(count & 1UL)) > + return NULL; > + } > + /* > + * Support for the BPF_TYPE_MATCHES argument to the > + * __builtin_preserve_type_info builtin was added at some point during > + * development of clang 15 and it's what is needed for > + * bpf_core_type_matches. > + */ > +#if __has_builtin(__builtin_preserve_type_info) && __clang_major__ >= 15 > + if (bpf_core_type_matches(struct rw_semaphore___old)) { > + struct rw_semaphore___old *old = (void *)lock; > + > + owner = (unsigned long)BPF_CORE_READ(old, owner); > + } else if (bpf_core_type_matches(struct rw_semaphore___new)) { > + owner = BPF_CORE_READ(rwsem, owner.counter); > + } > +#else > + /* assume new struct */ > + if (bpf_core_field_exists(rwsem->owner.counter)) > + owner = BPF_CORE_READ(rwsem, owner.counter); > +#endif > + /* RWSEM_READER_OWNED is bit 0 of owner */ > + if (flags == LCB_F_WRITE && (owner & 1UL)) > + return NULL; [Severity: Medium] In tools/perf/util/bpf_skel/lock_contention.bpf.c, contention_begin() calls get_lock_owner(pelem->lock, pelem->flags) for both owner stack tracing and LOCK_AGGR_TASK owner PID attribution: tools/perf/util/bpf_skel/lock_contention.bpf.c:contention_begin() { ... task = get_lock_owner(pelem->lock, pelem->flags); if (!task) goto skip_owner; ... if (lock_owner) { task = get_lock_owner(pelem->lock, pelem->flags); ... } Here pelem->flags comes from trace_contention_begin() and represents the contending waiter's mode (LCB_F_READ in rwsem_down_read_slowpath() or LCB_F_WRITE in rwsem_down_write_slowpath()), rather than the lock owner's mode. When a writer contends on a reader-owned rw_semaphore, pelem->flags is LCB_F_WRITE while rwsem->count does not have RWSEM_WRITER_LOCKED (bit 0) set and rwsem->owner has RWSEM_READER_OWNED (bit 0) set. Conversely, when a reader contends (flags == LCB_F_READ), both checks are bypassed. Will checking flags == LCB_F_WRITE in get_lock_owner() cause it to always return NULL whenever a writer contends on a reader-owned rw_semaphore, dropping owner attribution and owner callstacks in perf lock contention -b -o? [ ... ] -- Sashiko AI review · https://sashiko.dev/#/patchset/20261017171437.2066343-1-irogers@google.com?part=1 ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers @ 2026-10-17 17:14 ` Ian Rogers 2026-10-17 17:35 ` sashiko-bot 2026-10-17 17:14 ` [PATCH v2 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers ` (3 subsequent siblings) 5 siblings, 1 reply; 24+ messages in thread From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Add BPF sample filter predicates lock_held(<lock>), lock_held_read(<lock>) / lock_held_shared(<lock>), and lock_held_write(<lock>) / lock_held_exclusive(<lock>) (as well as ':R' and ':W' suffixes on <lock>) to allow recording perf event samples only when a specified kernel lock is held (or not held with '!'). Supported <lock> specifications match perf lock contention: - Global/static kernel lock symbols (with or without '&', e.g. lock_held(&cgroup_mutex) or lock_held(cgroup_mutex)) - Hexadecimal kernel lock addresses - Slab cache names prefixed with '&' (e.g. lock_held(&dentry), lock_held_read(&dentry)) - Per-process and semi-global locks: mmap_lock, siglock, rq_lock, zone_lock - Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W, rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W, semaphore Supported lock hold modes by lock family: - Both read/shared and write/exclusive: rwsem (including mmap_lock) and rwlock. - Write/exclusive only: mutex, spinlock (including siglock, rq_lock, zone_lock), rt-mutex, semaphore, and percpu-rwsem (percpu_down_write() / percpu_up_write(); fast-path percpu_down_read() / percpu_up_read() are inlined per-CPU counter operations without function entry points). When evaluating the filter on sample overflow, perf_sample_filter first inspects the lock's owner field directly in kernel memory where possible (for mutexes, rt-mutexes, and write-locked rwsems), and falls back to a held_locks BPF hash map (tracking separate read_count and write_count) populated by fentry/fexit hooks on kernel lock/unlock functions. To minimize tracing overhead, only the BPF hooks for the requested lock family and mode are autoloaded and attached. Example usage: # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock)' \ -- perf bench mem mmap -t 2 -l 100 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.342 MB perf.data (318 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ........ ................. ........................ # 18.24% mem-mmap [kernel.kallsyms] [k] clear_page_erms | ---clear_page_erms get_page_from_freelist __alloc_pages_noprof folio_alloc_mpol_noprof vma_alloc_folio_noprof do_anonymous_page __handle_mm_fault handle_mm_fault do_user_addr_fault exc_page_fault asm_exc_page_fault worker_thread Read/shared vs write/exclusive holding can also be filtered separately: # perf record -a -g -e cycles --filter 'lock_held_read(&mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held_write(&mmap_lock)' ... # perf record -a -g -e cycles --filter 'lock_held(&mmap_lock:W)' ... Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-record.txt | 41 +- tools/perf/util/bpf-filter.c | 1036 +++++++++++++++++- tools/perf/util/bpf-filter.h | 13 + tools/perf/util/bpf-filter.l | 31 +- tools/perf/util/bpf-filter.y | 46 +- tools/perf/util/bpf_skel/sample-filter.h | 59 + tools/perf/util/bpf_skel/sample_filter.bpf.c | 965 ++++++++++++++++ tools/perf/util/parse-events.c | 3 +- 8 files changed, 2168 insertions(+), 26 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 2351f7b4f5fc..246ae085744d 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -204,7 +204,46 @@ OPTIONS ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr, code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat, p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock, - mem_dtlb, mem_blk, mem_hops, uid, gid + mem_dtlb, mem_blk, mem_hops, uid, gid, cgroup, lock_held(<lock>), + lock_held_read(<lock>), lock_held_shared(<lock>), + lock_held_write(<lock>), lock_held_exclusive(<lock>) + + The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters + samples based on whether the specified kernel lock is currently held + by the kernel thread or CPU. To differentiate between shared/read + holding and exclusive/write holding, use 'lock_held_read(<lock>)' + (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' + (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to + <lock> (e.g. 'lock_held(&mmap_lock:R)' or 'lock_held(&mmap_lock:W)'). + <lock> can be a kernel lock symbol name (with or without a leading + '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'), + a hexadecimal lock address, a slab cache name prefixed with '&' + (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known + lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock + type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', + 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', + 'semaphore'). Both read/shared and write/exclusive tracking are + supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; + 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), + 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') + support write/exclusive tracking only. + + Note that 'lock_held(<lock>)' introduces overhead outside of sample + overflow handling: in addition to checking the lock's owner field on + each sample where possible (such as for mutexes, rt-mutexes, and + write-locked rwsems), it attaches BPF fentry/fexit programs to kernel + lock acquire and release functions to track held locks (such as + spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a + BPF map. These BPF hooks run on every kernel lock acquisition and + release for the attached lock family, even when no perf sample is + being taken. To minimize this overhead, only the hooks for the + relevant lock family and mode are attached when known (for example, + 'lock_held_read(&mmap_lock)' only hooks rwsem read functions, + 'lock_held_write(&mmap_lock)' only hooks rwsem write functions, + '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks + spinlock functions), whereas slab filters ('&<slab>'), raw hex + addresses, or symbols without a recognizable lock-type suffix attach + hooks across all lock families (including high-frequency spinlocks). The <operator> can be one of: ==, !=, >, >=, <, <=, & diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 72e83ac1d6a4..2f4c6bd85f89 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -40,21 +40,35 @@ * The 'dropped' map is to keep how many samples it dropped by the filter and * it will be reported as lost samples. */ +#include <errno.h> +#include <stdio.h> #include <stdlib.h> +#include <stdbool.h> +#include <stddef.h> +#include <string.h> #include <fcntl.h> +#include <unistd.h> #include <sys/ioctl.h> #include <sys/stat.h> #include <bpf/bpf.h> +#include <bpf/btf.h> #include <linux/err.h> +#include <linux/kernel.h> #include <linux/list.h> +#include <linux/string.h> +#include <linux/zalloc.h> #include <api/fs/fs.h> #include <internal/xyarray.h> #include <perf/threadmap.h> +#include <symbol/kallsyms.h> #include "util/cap.h" #include "util/debug.h" #include "util/evsel.h" +#include "util/lock-contention.h" +#include "util/string2.h" +#include "util/symbol_conf.h" #include "util/target.h" #include "util/thread_map.h" #include "util/bpf-utils.h" @@ -63,6 +77,7 @@ #include <util/bpf-filter-flex.h> #include <util/bpf-filter-bison.h> +#include "bpf_skel/lock_data.h" #include "bpf_skel/sample-filter.h" #include "bpf_skel/sample_filter.skel.h" @@ -127,7 +142,8 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * (evsel->core.attr.sample_type & (1 << (expr->term - PBF_TERM_SAMPLE_START)))) return 0; - if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID) { + if (expr->term == PBF_TERM_UID || expr->term == PBF_TERM_GID || + expr->term == PBF_TERM_LOCK_HELD) { /* Not dependent on the sample_type as computed from a BPF helper. */ return 0; } @@ -155,9 +171,814 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * return -1; } -static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry) +/* + * Program groups for `lock_held(...)` BPF tracing hooks. + * + * When a `lock_held(...)` filter is specified, only the BPF tracing program + * groups required for the requested lock(s) are autoloaded and attached. + * If no `lock_held(...)` filter is present, none of these programs are loaded. + */ +enum lock_prog_group { + LOCK_PROG_MUTEX = 1U << 0, + LOCK_PROG_RWSEM_READ = 1U << 1, + LOCK_PROG_RWSEM_WRITE = 1U << 2, + LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE, + LOCK_PROG_SPIN = 1U << 3, + LOCK_PROG_RWLOCK_READ = 1U << 4, + LOCK_PROG_RWLOCK_WRITE = 1U << 5, + LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE, + LOCK_PROG_RTMUTEX = 1U << 6, + LOCK_PROG_SEM = 1U << 7, + LOCK_PROG_PERCPU_RWSEM = 1U << 8, + LOCK_PROG_ALL = 0x1ffU, +}; + +static unsigned int lock_flags_to_prog_group(unsigned int flags) +{ + if (flags & LCB_F_MUTEX) + return LOCK_PROG_MUTEX; + if (flags & LCB_F_RT) + return LOCK_PROG_RTMUTEX; + if (flags & LCB_F_PERCPU) + return LOCK_PROG_PERCPU_RWSEM; + if (flags == LCB_F_SPIN) + return LOCK_PROG_SPIN; + if (flags & LCB_F_SPIN) + return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE; + if (flags & LCB_F_READ) + return LOCK_PROG_RWSEM_READ; + if (flags & LCB_F_WRITE) + return LOCK_PROG_RWSEM_WRITE; + return LOCK_PROG_SEM; +} + +struct lock_prog_desc { + size_t prog_offset; + const char *btf_func; + unsigned int group; +}; + +#define LOCK_FPROG(field, func, grp) \ + { offsetof(struct sample_filter_bpf, progs.field), func, grp } + +static const struct lock_prog_desc lock_progs[] = { + /* Mutex */ + LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested", + LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX), + LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX), + LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX), + /* RW semaphore */ + LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested", + LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested", + LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE), + LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM), + LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ), + LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE), + /* Spinlock */ + LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested", + LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN), + LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore", + LOCK_PROG_SPIN), + LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN), + /* RWlock */ + LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore", + LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ), + LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore", + LOCK_PROG_RWLOCK_WRITE), + LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE), + /* RT-Mutex */ + LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible", + LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX), + LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX), + /* Semaphore */ + LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM), + LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM), + LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM), + /* Percpu-RWsem */ + LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), + LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), +}; + +#undef LOCK_FPROG + +static int split_lock_name_mode(const char *name, int expr_part, + char *base_buf, size_t buf_sz, + bool *has_amp, unsigned int *mode_out) +{ + const char *bare; + size_t len; + unsigned int expr_mode = expr_part & PBF_LOCK_MODE_MASK; + unsigned int suffix_mode = 0; + + if (!name || !*name) + return -EINVAL; + + *has_amp = (name[0] == '&'); + bare = *has_amp ? name + 1 : name; + len = strlen(bare); + if (len == 0) + return -EINVAL; + + if (len > 2 && !strcmp(bare + len - 2, ":R")) { + suffix_mode = PBF_LOCK_MODE_READ; + len -= 2; + } else if (len > 2 && !strcmp(bare + len - 2, ":W")) { + suffix_mode = PBF_LOCK_MODE_WRITE; + len -= 2; + } + + if (len == 0 || len >= buf_sz) + return -EINVAL; + + if (expr_mode && expr_mode != PBF_LOCK_MODE_ANY && + suffix_mode && expr_mode != suffix_mode) + return -EINVAL; + + memcpy(base_buf, bare, len); + base_buf[len] = '\0'; + + if (suffix_mode) + *mode_out = suffix_mode; + else if (expr_mode) + *mode_out = expr_mode; + else + *mode_out = PBF_LOCK_MODE_ANY; + + return 0; +} + +/* + * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R", + * "spinlock", "percpu-rwsem"). + * + * A lock type name may correspond to up to two flag combinations (e.g. + * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed + * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is + * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`. + */ +static bool parse_lock_type_name(const char *name, unsigned int mode, + struct sample_filter_bpf *skel, + u64 *encoded_pair, unsigned int *prog_groups) +{ + bool found = false; + u32 flag1 = 0xff, flag2 = 0xff; + unsigned int groups = 0; + const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name; + + for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) { + u32 f = lock_type_table[i].flags; + bool is_read = (f & LCB_F_READ) != 0; + + if (strcmp(lock_type_table[i].lock_name, lookup) && + strcmp(lock_type_table[i].flags_name, name)) + continue; + + /* + * percpu_down_read() / percpu_up_read() fast paths are inlined + * per-CPU counter increments without function entry points, so + * only writer holding (percpu_down_write / percpu_up_write) can + * be tracked for percpu-rwsem. rwlock-rt is also not hooked. + */ + if ((f & LCB_F_PERCPU) && is_read) + continue; + if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE))) + continue; + + if (is_read && !(mode & PBF_LOCK_MODE_READ)) + continue; + if (!is_read && !(mode & PBF_LOCK_MODE_WRITE)) + continue; + + if (!found) + flag1 = f; + else if (f != flag1) + flag2 = f; + if (skel && f < LCB_F_TYPE_MAX) { + skel->bss->has_lock_type = 1; + skel->bss->lock_type_filter[f] = 1; + } + groups |= lock_flags_to_prog_group(f); + found = true; + } + + if (!found) + return false; + + if (encoded_pair) + *encoded_pair = flag1 | ((u64)flag2 << 8); + if (prog_groups) + *prog_groups |= groups; + return true; +} + +struct lock_sym_cb_args { + const char *name; + u64 addr; + bool found; +}; + +static int find_lock_sym_cb(void *arg, const char *name, + char type __maybe_unused, u64 start) +{ + struct lock_sym_cb_args *args = arg; + + if (!strcmp(name, args->name)) { + args->found = true; + args->addr = start; + if (start != 0) + return 1; + } + return 0; +} + +static int find_kernel_lock_sym(const char *name, u64 *addr) +{ + struct lock_sym_cb_args args = { + .name = name, + }; + const char *kallsyms_filename = "/proc/kallsyms"; + + if (symbol_conf.kallsyms_name != NULL) + kallsyms_filename = symbol_conf.kallsyms_name; + + if (kallsyms__parse(kallsyms_filename, &args, find_lock_sym_cb) < 0) + return -ENOENT; + + if (!args.found) + return -ENOENT; + + if (args.addr == 0) { + pr_err("Error: Kernel symbol '%s' address is 0 (check permissions / kptr_restrict)\n", + name); + return -EACCES; + } + + *addr = args.addr; + return 0; +} + +static bool parse_lock_hex_addr(const char *str, u64 *addr) +{ + char *end; + unsigned long long val; + + if (!str || !*str) + return false; + + errno = 0; + val = strtoull(str, &end, 16); + if (*end != '\0' || errno != 0 || val == 0) + return false; + + *addr = val; + return true; +} + +static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf) +{ + u64 addr = 0; + char *buf = NULL, *p; + size_t len; + long last = -1; + int ret; + + if (!btf) + return; + + ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT); + if (ret < 0) + return; + + ret = btf__resolve_size(btf, ret); + if (ret < 0) + return; + + skel->rodata->sizeof_zone = ret; + + if (find_kernel_lock_sym("contig_page_data", &addr) == 0) { + skel->rodata->contig_page_data_addr = addr; + return; + } + + if (find_kernel_lock_sym("node_data", &addr) == 0) + skel->rodata->node_data_addr = addr; + + if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) { + p = buf; + while (p && *p) { + char *end; + + last = strtol(p, &end, 0); + if (end == p) + break; + p = end; + if (*p == ',' || *p == '-' || *p == '\n') + p++; + } + skel->rodata->nr_nodes = last + 1; + free(buf); + } +} + +static bool btf_has_func(struct btf *btf, const char *name) +{ + return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0; +} + +static unsigned int supported_lock_prog_groups(struct btf *btf) +{ + unsigned int supported = 0; + + if (!btf) + return 0; + + if ((btf_has_func(btf, "mutex_lock") || btf_has_func(btf, "mutex_lock_nested")) && + btf_has_func(btf, "mutex_unlock")) + supported |= LOCK_PROG_MUTEX; + + if ((btf_has_func(btf, "down_read") || btf_has_func(btf, "down_read_nested")) && + btf_has_func(btf, "up_read")) + supported |= LOCK_PROG_RWSEM_READ; + + if ((btf_has_func(btf, "down_write") || btf_has_func(btf, "down_write_nested")) && + btf_has_func(btf, "up_write")) + supported |= LOCK_PROG_RWSEM_WRITE; + + /* + * Require both acquire and all unlock entry points in BTF so we never + * attach spinlock/rwlock acquire hooks when unlock functions are + * inlined (e.g. !CONFIG_UNINLINE_SPIN_UNLOCK or CONFIG_INLINE_*_UNLOCK). + */ + if ((btf_has_func(btf, "_raw_spin_lock") || btf_has_func(btf, "_raw_spin_lock_nested")) && + btf_has_func(btf, "_raw_spin_unlock") && + btf_has_func(btf, "_raw_spin_unlock_irq") && + btf_has_func(btf, "_raw_spin_unlock_irqrestore") && + btf_has_func(btf, "_raw_spin_unlock_bh")) + supported |= LOCK_PROG_SPIN; + + if (btf_has_func(btf, "_raw_read_lock") && + btf_has_func(btf, "_raw_read_unlock") && + btf_has_func(btf, "_raw_read_unlock_irq") && + btf_has_func(btf, "_raw_read_unlock_irqrestore") && + btf_has_func(btf, "_raw_read_unlock_bh")) + supported |= LOCK_PROG_RWLOCK_READ; + + if ((btf_has_func(btf, "_raw_write_lock") || btf_has_func(btf, "_raw_write_lock_nested")) && + btf_has_func(btf, "_raw_write_unlock") && + btf_has_func(btf, "_raw_write_unlock_irq") && + btf_has_func(btf, "_raw_write_unlock_irqrestore") && + btf_has_func(btf, "_raw_write_unlock_bh")) + supported |= LOCK_PROG_RWLOCK_WRITE; + + if ((btf_has_func(btf, "rt_mutex_lock") || btf_has_func(btf, "rt_mutex_lock_nested")) && + btf_has_func(btf, "rt_mutex_unlock")) + supported |= LOCK_PROG_RTMUTEX; + + if (btf_has_func(btf, "down") && btf_has_func(btf, "up")) + supported |= LOCK_PROG_SEM; + + if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write")) + supported |= LOCK_PROG_PERCPU_RWSEM; + + return supported; +} + +/* + * Enable or disable autoload for each lock-tracking BPF program in the + * skeleton before `sample_filter_bpf__load()` is called. + * + * Each program is enabled only if its group is in `needed_groups` and its + * target kernel function or tracepoint exists in the running kernel's BTF. + */ +static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf, + unsigned int needed_groups, + unsigned int required_groups, + bool needs_lock_syms, bool needs_slab_iter) +{ + unsigned int supported_groups = supported_lock_prog_groups(btf); + unsigned int enabled_progs = 0; + + if ((required_groups & supported_groups) != required_groups) { + pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n"); + return -ENOTSUP; + } + + needed_groups &= supported_groups; + if (skel) + skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ); + + for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) { + struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset; + struct bpf_program *prog = *progp; + bool enable = false; + + if ((needed_groups & lock_progs[i].group) && btf && + lock_progs[i].btf_func) { + if (btf__find_by_name_kind(btf, lock_progs[i].btf_func, + BTF_KIND_FUNC) > 0) + enable = true; + } + + if (enable) + enabled_progs++; + bpf_program__set_autoload(prog, enable); + } + + bpf_program__set_autoload(skel->progs.collect_lock_syms, needs_lock_syms); + if (needs_slab_iter && btf && + btf__find_by_name_kind(btf, "bpf_iter__kmem_cache", BTF_KIND_STRUCT) > 0) { + bpf_program__set_autoload(skel->progs.slab_cache_iter, true); + } else { + bpf_program__set_autoload(skel->progs.slab_cache_iter, false); + } + + if (needed_groups && !enabled_progs) { + pr_err("Error: Kernel BTF lacks required lock tracing entry points\n"); + return -ENOTSUP; + } + return 0; +} + +/* + * Pre-scan a single filter expression prior to loading the BPF skeleton in + * order to determine which BPF maps need to be sized and which lock tracing + * program groups need to be autoloaded. + */ +static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, + bool *has_lock_filter, + unsigned int *prog_groups, + unsigned int *required_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + char base[128]; + bool has_amp; + unsigned int mode, groups = 0; + u64 dummy; + + if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name) + return; + + *has_lock_filter = true; + if (split_lock_name_mode(expr->lock_name, expr->part, base, + sizeof(base), &has_amp, &mode) < 0) + return; + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_RWSEM_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_RWSEM_WRITE; + *prog_groups |= groups; + *required_groups |= groups; + return; + } + if (!strcmp(base, "siglock")) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + return; + } + if (!strcmp(base, "rq_lock") || !strcmp(base, "zone_lock")) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + *needs_lock_syms = true; + return; + } + if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, &groups)) { + *prog_groups |= groups; + *required_groups |= groups; + return; + } + + if (!has_amp && parse_lock_hex_addr(base, &dummy)) { + *prog_groups |= LOCK_PROG_ALL; + return; + } + + if (has_amp) + *needs_slab_iter = true; + + if (strtailcmp(base, "rt_mutex") == 0 || strtailcmp(base, "rtmutex") == 0) { + *prog_groups |= LOCK_PROG_RTMUTEX; + *required_groups |= LOCK_PROG_RTMUTEX; + } else if (strtailcmp(base, "mutex") == 0) { + *prog_groups |= LOCK_PROG_MUTEX; + *required_groups |= LOCK_PROG_MUTEX; + } else if (strtailcmp(base, "percpu_rwsem") == 0 || + strtailcmp(base, "pcpu_rwsem") == 0) { + *prog_groups |= LOCK_PROG_PERCPU_RWSEM; + *required_groups |= LOCK_PROG_PERCPU_RWSEM; + } else if (strtailcmp(base, "rwsem") == 0) { + if (mode & PBF_LOCK_MODE_READ) { + *prog_groups |= LOCK_PROG_RWSEM_READ; + *required_groups |= LOCK_PROG_RWSEM_READ; + } + if (mode & PBF_LOCK_MODE_WRITE) { + *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM; + *required_groups |= LOCK_PROG_RWSEM_WRITE; + } + } else if (strtailcmp(base, "spinlock") == 0) { + *prog_groups |= LOCK_PROG_SPIN; + *required_groups |= LOCK_PROG_SPIN; + } else if (strtailcmp(base, "rwlock") == 0) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_RWLOCK_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_RWLOCK_WRITE; + *prog_groups |= groups; + *required_groups |= groups; + } else { + *prog_groups |= LOCK_PROG_ALL; + } + + if (!has_amp && find_kernel_lock_sym(base, &dummy) != 0) + *needs_slab_iter = true; +} + +static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter, + unsigned int *prog_groups, + unsigned int *required_groups, + bool *needs_lock_syms, + bool *needs_slab_iter) +{ + struct perf_bpf_filter_expr *expr, *group; + + *has_lock_filter = false; + *prog_groups = 0; + *required_groups = 0; + *needs_lock_syms = false; + *needs_slab_iter = false; + + list_for_each_entry(expr, &evsel->bpf_filters, list) { + prescan_one_lock_expr(expr, has_lock_filter, prog_groups, + required_groups, needs_lock_syms, + needs_slab_iter); + if (expr->op == PBF_OP_GROUP_BEGIN) { + list_for_each_entry(group, &expr->groups, list) { + prescan_one_lock_expr(group, has_lock_filter, + prog_groups, required_groups, + needs_lock_syms, needs_slab_iter); + } + } + } +} + +static void run_slab_cache_iter(struct sample_filter_bpf *skel) +{ + struct bpf_link *link; + int fd; + char buf[256]; + + if (!bpf_program__autoload(skel->progs.slab_cache_iter)) + return; + + link = bpf_program__attach_iter(skel->progs.slab_cache_iter, NULL); + if (!link || IS_ERR(link)) { + pr_debug("cannot attach slab cache iter\n"); + return; + } + + fd = bpf_iter_create(bpf_link__fd(link)); + if (fd >= 0) { + while (read(fd, buf, sizeof(buf)) > 0) + continue; + close(fd); + } + + bpf_link__destroy(link); +} + +static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, + u64 *kmem_cache_addr) +{ + int cache_fd; + u64 key, *prev_key = NULL; + + if (!skel || !bpf_program__autoload(skel->progs.slab_cache_iter)) + return false; + + cache_fd = bpf_map__fd(skel->maps.slab_caches); + while (!bpf_map_get_next_key(cache_fd, prev_key, &key)) { + struct slab_cache_data data; + + if (bpf_map_lookup_elem(cache_fd, &key, &data) < 0) + break; + + if (!strcmp(data.name, name)) { + *kmem_cache_addr = key; + return true; + } + prev_key = &key; + } + return false; +} + +/* + * Resolve a `lock_held(<name>)` filter expression into `entry->part` + * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate + * the corresponding BPF filter maps and BSS control flags. + * + * Resolution order: + * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock` (with or without leading '&'). + * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`, + * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, + * `semaphore`, etc. + * 3. Hexadecimal lock address (when no leading '&'). + * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`). + * 5. Global/static kernel lock symbol in kallsyms (with or without leading '&', + * e.g. `&cgroup_mutex` or `cgroup_mutex`). + * 6. Slab cache name without leading '&' fallback. + */ +static int resolve_lock_expr(struct sample_filter_bpf *skel, + struct perf_bpf_filter_expr *expr, + struct perf_bpf_filter_entry *entry) +{ + const char *name = expr->lock_name; + char base[128]; + bool has_amp; + unsigned int mode; + u64 addr = 0, encoded_pair = 0; + u8 one = 1; + int ret; + + if (split_lock_name_mode(name, expr->part, base, sizeof(base), + &has_amp, &mode) < 0) { + pr_err("Error: Invalid kernel lock filter '%s'\n", name ?: ""); + return -EINVAL; + } + + if (!strcmp(base, "mmap_lock") || !strcmp(base, "mmap_sem")) { + entry->part = PBF_LOCK_MMAP | mode; + entry->target = 0; + if (skel) + skel->bss->has_mmap_lock = 1; + return 0; + } + + if (!strcmp(base, "siglock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_SIGHAND | mode; + entry->target = 0; + if (skel) + skel->bss->has_siglock = 1; + return 0; + } + + if (!strcmp(base, "rq_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->target = LOCK_CLASS_RQLOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!strcmp(base, "zone_lock")) { + if (!(mode & PBF_LOCK_MODE_WRITE)) { + pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n", + name); + return -EINVAL; + } + entry->part = PBF_LOCK_CLASS | mode; + entry->target = LOCK_CLASS_ZONE_LOCK; + if (skel) + skel->bss->has_lock_class = 1; + return 0; + } + + if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { + entry->part = PBF_LOCK_TYPE | mode; + entry->target = encoded_pair; + return 0; + } + + if (!has_amp && parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) { + pr_err("Error: Lock type '%s' does not support the requested read/write mode\n", + name); + return -EINVAL; + } + + if (!has_amp && parse_lock_hex_addr(base, &addr)) { + entry->part = PBF_LOCK_ADDR | mode; + entry->target = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + + if (has_amp && find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->target = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + + ret = find_kernel_lock_sym(base, &addr); + if (ret == 0) { + entry->part = PBF_LOCK_ADDR | mode; + entry->target = addr; + if (skel) { + int fd = bpf_map__fd(skel->maps.lock_addr_filter); + + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_addr = 1; + } + return 0; + } + if (ret == -EACCES) + return ret; + + if (!has_amp && find_slab_cache(skel, base, &addr)) { + int fd = bpf_map__fd(skel->maps.slab_filter); + + entry->part = PBF_LOCK_SLAB | mode; + entry->target = addr; + bpf_map_update_elem(fd, &addr, &one, BPF_ANY); + skel->bss->has_lock_slab = 1; + return 0; + } + + pr_err("Error: Cannot resolve kernel lock '%s'\n", name); + return -EINVAL; +} + +static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, + struct perf_bpf_filter_entry *entry) { - int i = 0; + int i = 0, ret; struct perf_bpf_filter_expr *expr; list_for_each_entry(expr, &evsel->bpf_filters, list) { @@ -171,6 +992,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = expr->part; entry[i].term = expr->term; entry[i].value = expr->val; + entry[i].target = 0; + if (expr->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, expr, &entry[i]); + if (ret < 0) + return ret; + } i++; if (expr->op == PBF_OP_GROUP_BEGIN) { @@ -184,6 +1011,12 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry entry[i].part = group->part; entry[i].term = group->term; entry[i].value = group->val; + entry[i].target = 0; + if (group->term == PBF_TERM_LOCK_HELD) { + ret = resolve_lock_expr(skel, group, &entry[i]); + if (ret < 0) + return ret; + } i++; } @@ -423,10 +1256,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) { int i, x, y, fd, ret; struct sample_filter_bpf *skel = NULL; + struct btf *btf = NULL; struct bpf_program *prog; struct bpf_link *link; struct perf_bpf_filter_entry *entry; bool needs_idx_hash = !target__has_cpu(target); + bool has_lock_filter = false, needs_lock_syms = false, needs_slab_iter = false; + unsigned int prog_groups = 0, required_groups = 0; #if LIBBPF_CURRENT_VERSION_GEQ(1, 7) DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts, .dont_enable = true); @@ -438,15 +1274,25 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (entry == NULL) return -1; - ret = get_filter_entries(evsel, entry); - if (ret < 0) { - pr_err("Failed to process filter entries\n"); - goto err; - } + prescan_lock_filters(evsel, &has_lock_filter, &prog_groups, + &required_groups, &needs_lock_syms, + &needs_slab_iter); - if (needs_idx_hash && geteuid() != 0) { + if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) { int zero = 0; + if (has_lock_filter) { + pr_err("Error: lock_held BPF filter requires CAP_BPF capability\n"); + ret = -EPERM; + goto err; + } + + ret = get_filter_entries(evsel, NULL, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + /* The filters map is shared among other processes */ ret = create_idx_hash(evsel, entry); if (ret < 0) @@ -484,13 +1330,69 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return 0; } - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { + ret = -errno; + pr_err("Failed to open perf sample-filter BPF skeleton\n"); + goto err; + } + + if (has_lock_filter) { + btf = btf__load_vmlinux_btf(); + bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES); + bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS); + if (needs_slab_iter) { + bpf_map__set_max_entries(skel->maps.slab_filter, MAX_FILTERS); + bpf_map__set_max_entries(skel->maps.slab_caches, MAX_ENTRIES); + } + if (needs_lock_syms) { + bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); + init_numa_data(skel, btf); + } + ret = configure_lock_progs(skel, btf, prog_groups, required_groups, + needs_lock_syms, needs_slab_iter); + if (ret < 0) + goto err; + } else { + configure_lock_progs(skel, NULL, 0, 0, false, false); + } + + if (sample_filter_bpf__load(skel) < 0) { ret = -errno; pr_err("Failed to load perf sample-filter BPF skeleton\n"); goto err; } + if (has_lock_filter && needs_slab_iter) + run_slab_cache_iter(skel); + + ret = get_filter_entries(evsel, skel, entry); + if (ret < 0) { + pr_err("Failed to process filter entries\n"); + goto err; + } + + if (has_lock_filter) { + if (needs_lock_syms && skel->bss->has_lock_class) { + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts, + .flags = BPF_F_TEST_RUN_ON_CPU, + ); + int prog_fd = bpf_program__fd(skel->progs.collect_lock_syms); + + bpf_prog_test_run_opts(prog_fd, &opts); + } + bpf_program__set_autoattach(skel->progs.perf_sample_filter, false); + bpf_program__set_autoattach(skel->progs.collect_lock_syms, false); + bpf_program__set_autoattach(skel->progs.slab_cache_iter, false); + + ret = sample_filter_bpf__attach(skel); + if (ret < 0) { + pr_err("Failed to attach lock sample-filter BPF programs\n"); + goto err; + } + skel->bss->lock_enabled = 1; + } + i = 0; fd = bpf_map__fd(skel->maps.filters); @@ -513,11 +1415,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) } } } + btf__free(btf); free(entry); evsel->bpf_skel = skel; return 0; err: + btf__free(btf); free(entry); if (!list_empty(&pinned_filters)) { struct pinned_filter_idx *pfi, *tmp; @@ -532,6 +1436,21 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) return ret; } +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr) +{ + struct perf_bpf_filter_expr *group, *tmp; + + if (!expr) + return; + + list_for_each_entry_safe(group, tmp, &expr->groups, list) { + list_del(&group->list); + perf_bpf_filter_expr__delete(group); + } + zfree(&expr->lock_name); + free(expr); +} + int perf_bpf_filter__destroy(struct evsel *evsel) { struct perf_bpf_filter_expr *expr, *tmp; @@ -539,7 +1458,7 @@ int perf_bpf_filter__destroy(struct evsel *evsel) list_for_each_entry_safe(expr, tmp, &evsel->bpf_filters, list) { list_del(&expr->list); - free(expr); + perf_bpf_filter_expr__delete(expr); } sample_filter_bpf__destroy(evsel->bpf_skel); @@ -591,7 +1510,7 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term { struct perf_bpf_filter_expr *expr; - expr = malloc(sizeof(*expr)); + expr = zalloc(sizeof(*expr)); if (expr != NULL) { expr->term = term; expr->part = part; @@ -602,6 +1521,28 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term return expr; } +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val) +{ + struct perf_bpf_filter_expr *expr; + + if (!lock_name || !*lock_name) + return NULL; + + expr = perf_bpf_filter_expr__new(PBF_TERM_LOCK_HELD, mode, op, val); + if (!expr) + return NULL; + + expr->lock_name = strdup(lock_name); + if (!expr->lock_name) { + free(expr); + return NULL; + } + return expr; +} + static bool check_bpf_filter_capable(void) { int fd; @@ -622,14 +1563,11 @@ static bool check_bpf_filter_capable(void) return false; } -int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str) { YY_BUFFER_STATE buffer; int ret; - if (!check_bpf_filter_capable()) - return -EPERM; - buffer = perf_bpf_filter__scan_string(str); ret = perf_bpf_filter_parse(expr_head); @@ -641,6 +1579,53 @@ int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) return ret; } +int perf_bpf_filter__parse(struct list_head *expr_head, const char *str) +{ + if (!check_bpf_filter_capable()) + return -EPERM; + + return __perf_bpf_filter__parse(expr_head, str); +} + +static int pin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret; + + ret = bpf_object__pin_maps(skel->obj, path); + if (ret < 0) + return ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__pin(skel->progs.perf_sample_filter, prog_path); + if (ret < 0) + bpf_object__unpin_maps(skel->obj, path); + + free(prog_path); + return ret; +} + +static int unpin_filter_object(struct sample_filter_bpf *skel, const char *path) +{ + char *prog_path = NULL; + int ret, map_ret; + + if (asprintf(&prog_path, "%s/perf_sample_filter", path) < 0) { + bpf_object__unpin_maps(skel->obj, path); + return -ENOMEM; + } + + ret = bpf_program__unpin(skel->progs.perf_sample_filter, prog_path); + free(prog_path); + + map_ret = bpf_object__unpin_maps(skel->obj, path); + return ret < 0 ? ret : map_ret; +} + int perf_bpf_filter__pin(void) { struct sample_filter_bpf *skel; @@ -654,6 +1639,8 @@ int perf_bpf_filter__pin(void) goto err; } + configure_lock_progs(skel, NULL, 0, 0, false, false); + /* pinned program will use pid-hash */ bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS); bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH); @@ -674,7 +1661,7 @@ int perf_bpf_filter__pin(void) goto err; } - ret = bpf_object__pin(skel->obj, path); + ret = pin_filter_object(skel, path); if (ret < 0) { pr_err("Failed to pin BPF filter objects\n"); goto err; @@ -683,7 +1670,7 @@ int perf_bpf_filter__pin(void) /* setup access permissions for the pinned objects */ dir_fd = open(path, O_PATH); if (dir_fd < 0) { - bpf_object__unpin(skel->obj, path); + unpin_filter_object(skel, path); ret = dir_fd; goto err; } @@ -727,6 +1714,8 @@ int perf_bpf_filter__pin(void) err_close: close(dir_fd); + if (ret < 0) + unpin_filter_object(skel, path); err: free(path); @@ -740,13 +1729,20 @@ int perf_bpf_filter__unpin(void) char *path = NULL; int ret = -1; - skel = sample_filter_bpf__open_and_load(); + skel = sample_filter_bpf__open(); if (!skel) { ret = -errno; pr_err("Failed to open perf sample-filter BPF skeleton\n"); goto err; } + configure_lock_progs(skel, NULL, 0, 0, false, false); + if (sample_filter_bpf__load(skel) < 0) { + ret = -errno; + pr_err("Failed to load perf sample-filter BPF skeleton\n"); + goto err; + } + if (asprintf(&path, "%s/fs/bpf/%s", sysfs__mountpoint(), PERF_BPF_FILTER_PIN_PATH) < 0) { ret = -errno; @@ -754,7 +1750,7 @@ int perf_bpf_filter__unpin(void) goto err; } - ret = bpf_object__unpin(skel->obj, path); + ret = unpin_filter_object(skel, path); err: free(path); diff --git a/tools/perf/util/bpf-filter.h b/tools/perf/util/bpf-filter.h index 818c554b91b2..fda95ff226b3 100644 --- a/tools/perf/util/bpf-filter.h +++ b/tools/perf/util/bpf-filter.h @@ -14,6 +14,13 @@ struct perf_bpf_filter_expr { int part; enum perf_bpf_filter_term term; unsigned long val; + /* + * Raw lock descriptor string for PBF_TERM_LOCK_HELD expressions + * (e.g. "&cgroup_mutex", "mmap_lock", "&dentry", "rwsem:R"). + * Resolved into `entry.part` and `entry.target` during + * perf_bpf_filter__prepare(). + */ + char *lock_name; }; struct evsel; @@ -27,6 +34,12 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term int part, enum perf_bpf_filter_op op, unsigned long val); +struct perf_bpf_filter_expr *perf_bpf_filter_expr__new_lock(const char *lock_name, + int mode, + enum perf_bpf_filter_op op, + unsigned long val); +void perf_bpf_filter_expr__delete(struct perf_bpf_filter_expr *expr); +int __perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__parse(struct list_head *expr_head, const char *str); int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target); int perf_bpf_filter__destroy(struct evsel *evsel); diff --git a/tools/perf/util/bpf-filter.l b/tools/perf/util/bpf-filter.l index 1be9df6550fc..b4022110f1e5 100644 --- a/tools/perf/util/bpf-filter.l +++ b/tools/perf/util/bpf-filter.l @@ -1,9 +1,12 @@ %option prefix="perf_bpf_filter_" %option noyywrap +%x LOCK_ARG + %{ #include <stdio.h> #include <stdlib.h> +#include <string.h> #include <linux/perf_event.h> #include "bpf-filter.h" @@ -76,7 +79,7 @@ static int path_or_error(void) num_dec [0-9]+ num_hex 0[Xx][0-9a-fA-F]+ space [ \t]+ -path [^ \t\n,]+ +path [^ \t\n,()!=>&|<]+ ident [_a-zA-Z][_a-zA-Z0-9]+ %% @@ -116,7 +119,31 @@ mem_hops { return sample_part(PBF_TERM_DATA_SRC, 8); } uid { return sample(PBF_TERM_UID); } gid { return sample(PBF_TERM_GID); } cgroup { return sample_path(PBF_TERM_CGROUP); } - +lock_held[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_ANY; + return BFT_LOCK_HELD; + } +(lock_held_read|lock_held_shared)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_READ; + return BFT_LOCK_HELD; + } +(lock_held_write|lock_held_exclusive)[ \t]*"(" { + BEGIN(LOCK_ARG); + perf_bpf_filter_lval.num = PBF_LOCK_MODE_WRITE; + return BFT_LOCK_HELD; + } + +<LOCK_ARG>{space} { } +<LOCK_ARG>[^ \t\n()]+ { + perf_bpf_filter_lval.lock_name = strdup(perf_bpf_filter_text); + return BFT_LOCK_NAME; + } +<LOCK_ARG>")" { BEGIN(INITIAL); return ')'; } +<LOCK_ARG>.|\n { BEGIN(INITIAL); return BFT_ERROR; } + +"!" { return '!'; } "==" { return operator(PBF_OP_EQ); } "!=" { return operator(PBF_OP_NEQ); } ">" { return operator(PBF_OP_GT); } diff --git a/tools/perf/util/bpf-filter.y b/tools/perf/util/bpf-filter.y index 5a79a8e7a45b..6ee6edef13f7 100644 --- a/tools/perf/util/bpf-filter.y +++ b/tools/perf/util/bpf-filter.y @@ -11,6 +11,7 @@ #include <string.h> #include <linux/compiler.h> #include <linux/list.h> +#include <linux/zalloc.h> #include "bpf-filter.h" #include "cgroup.h" @@ -31,6 +32,11 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, { unsigned long num; char *path; + char *lock_name; + struct { + char *name; + int mode; + } lock; struct { enum perf_bpf_filter_term term; int part; @@ -40,12 +46,17 @@ static void perf_bpf_filter_error(struct list_head *expr __maybe_unused, } %token BFT_SAMPLE BFT_SAMPLE_PATH BFT_OP BFT_ERROR BFT_NUM BFT_LOGICAL_OR BFT_PATH +%token BFT_LOCK_HELD BFT_LOCK_NAME %type <expr> filter_term filter_expr -%destructor { free ($$); } <expr> +%destructor { perf_bpf_filter_expr__delete($$); } <expr> +%destructor { free($$); } <lock_name> +%destructor { free($$.name); } <lock> %type <sample> BFT_SAMPLE BFT_SAMPLE_PATH %type <op> BFT_OP -%type <num> BFT_NUM +%type <num> BFT_NUM BFT_LOCK_HELD %type <path> BFT_PATH +%type <lock_name> BFT_LOCK_NAME +%type <lock> lock_term %% @@ -105,5 +116,36 @@ BFT_SAMPLE_PATH BFT_OP BFT_PATH $$ = perf_bpf_filter_expr__new($1.term, $1.part, $2, cgroup_id); cgroup__put(cgrp); } +| +lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, PBF_OP_EQ, 1); + zfree(&$1.name); + if (!$$) + YYERROR; +} +| +'!' lock_term +{ + $$ = perf_bpf_filter_expr__new_lock($2.name, $2.mode, PBF_OP_EQ, 0); + zfree(&$2.name); + if (!$$) + YYERROR; +} +| +lock_term BFT_OP BFT_NUM +{ + $$ = perf_bpf_filter_expr__new_lock($1.name, $1.mode, $2, $3); + zfree(&$1.name); + if (!$$) + YYERROR; +} + +lock_term: +BFT_LOCK_HELD BFT_LOCK_NAME ')' +{ + $$.name = $2; + $$.mode = $1; +} %% diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index 683fec85e71e..cd711d1d04b6 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -53,14 +53,73 @@ enum perf_bpf_filter_term { /* Terms computed from BPF helpers. */ PBF_TERM_UID, PBF_TERM_GID, + /* + * Kernel lock held predicate (`lock_held(...)`). Evaluates to 1 if the + * specified kernel lock is currently held by the running task/CPU, or + * 0 otherwise. The lock specification kind is stored in `entry.part` + * (`enum perf_bpf_lock_target_kind`) and the resolved target address, + * class ID, or type flag encoding is stored in `entry.target`. + */ + PBF_TERM_LOCK_HELD, }; +/* + * Lock target categories for PBF_TERM_LOCK_HELD filters. + * Stored in `perf_bpf_filter_entry.part`. + */ +enum perf_bpf_lock_target_kind { + PBF_LOCK_NONE = 0, + /* + * Specific global or static lock address (resolved from kallsyms or + * given as a hex address). `entry.target` holds the kernel address. + */ + PBF_LOCK_ADDR, + /* + * Dynamic lock allocated inside a slab cache (`&slab_name`). + * `entry.target` holds the kernel `struct kmem_cache *` address. + */ + PBF_LOCK_SLAB, + /* + * Per-process `current->mm->mmap_lock` (or `mmap_sem`). + */ + PBF_LOCK_MMAP, + /* + * Per-process `current->sighand->siglock`. + */ + PBF_LOCK_SIGHAND, + /* + * Semi-global lock class (`rq_lock` or `zone_lock`). + * `entry.target` holds `enum lock_class_sym` (LOCK_CLASS_RQLOCK or + * LOCK_CLASS_ZONE_LOCK). + */ + PBF_LOCK_CLASS, + /* + * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.). + * `entry.target` packs up to two `LCB_F_*` flag combinations in the + * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). + */ + PBF_LOCK_TYPE, +}; + +/* + * `perf_bpf_filter_entry.part` layout for PBF_TERM_LOCK_HELD: + * Bits 0..3: `enum perf_bpf_lock_target_kind` + * Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding) + * Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding) + */ +#define PBF_LOCK_KIND_MASK 0x0fU +#define PBF_LOCK_MODE_READ (1U << 4) +#define PBF_LOCK_MODE_WRITE (1U << 5) +#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE) +#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY + /* BPF map entry for filtering */ struct perf_bpf_filter_entry { enum perf_bpf_filter_op op; __u32 part; /* sub-sample type info when it has multiple values */ enum perf_bpf_filter_term term; __u64 value; + __u64 target; /* lock address, kmem_cache *, class, or type flags */ }; struct idx_hash_key { diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index e5666d4c1722..480ba4b49793 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -5,8 +5,47 @@ #include <bpf/bpf_tracing.h> #include <bpf/bpf_core_read.h> +#include "lock_data.h" #include "sample-filter.h" +/* + * Key for the `held_locks` BPF hash map tracking currently held kernel locks. + * + * - `addr_or_id`: + * - PBF_LOCK_ADDR: kernel virtual address of the lock instance + * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK) + * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address + * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags + * - `owner_id`: + * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`), + * since the holder may sleep and migrate across CPUs while holding the lock. + * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks + * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption + * is disabled on the holding CPU and spinlocks such as `rq_lock` may be + * acquired by one task and released by the next task across `context_switch()`. + * - `kind`: `enum perf_bpf_lock_target_kind`. + */ +struct held_lock_key { + __u64 addr_or_id; + __u32 owner_id; + __u32 kind; +}; + +/* + * Value for `held_locks` entries. + * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`) + * and exclusive/write acquisitions (including nested class/slab/type + * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of + * the lock so release tracepoints that do not carry flags (such as + * `lock:contended_release`) can decrement matching class/slab/type entries. + */ +struct held_lock_val { + __u32 read_count; + __u32 write_count; + __u32 flags; + __u32 pad; +}; + /* BPF map that will be filled by user space */ struct filters { __uint(type, BPF_MAP_TYPE_HASH); @@ -43,10 +82,199 @@ struct lost_count { __uint(max_entries, 1); } dropped SEC(".maps"); +/* Currently held kernel locks tracked by fentry/fexit and lock tracepoints. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, struct held_lock_key); + __type(value, struct held_lock_val); + __uint(max_entries, 1); +} held_locks SEC(".maps"); + +/* Target lock addresses for PBF_LOCK_ADDR filters. */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u64); + __type(value, __u8); + __uint(max_entries, 1); +} lock_addr_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, 1); +} lock_syms SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(__u8)); + __uint(max_entries, 1); +} slab_filter SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(struct slab_cache_data)); + __uint(max_entries, 1); +} slab_caches SEC(".maps"); + volatile const int use_idx_hash; +const volatile __u64 contig_page_data_addr; +const volatile __u64 node_data_addr; +const volatile int nr_nodes; +const volatile int sizeof_zone; + +__u32 slab_cache_id; + +/* Control flags set by userspace when lock_held(...) filters are active. */ +int lock_enabled; +int has_lock_addr; +int has_lock_slab; +int has_lock_class; +int has_lock_type; +int has_mmap_lock; +int has_siglock; +int track_rwsem_read; +__u8 lock_type_filter[LCB_F_TYPE_MAX]; + void *bpf_cast_to_kern_ctx(void *) __ksym; +/* + * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`. + * + * For locks that maintain a task_struct pointer in their `owner` field + * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`), + * `perf_sample_filter` can inspect the lock structure directly in kernel memory + * and compare the owner against `current` (`bpf_get_current_task()`). + * + * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear + * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and + * spinlocks are checked via the `held_locks` BPF map instead. + */ +static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr) +{ + if (!curr) + return false; + + return (__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr || + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr || + (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr; +} + +static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode) +{ + if (!val) + return false; + if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0) + return true; + if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0) + return true; + return false; +} + +static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind, + __u32 mode, __u32 pid, __u32 cpu_id) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = pid, + .kind = kind, + }; + struct held_lock_val *val; + + if (pid) { + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + } + + key.owner_id = cpu_id; + val = bpf_map_lookup_elem(&held_locks, &key); + if (match_held_val(val, mode)) + return 1; + + return 0; +} + +static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode, + __u32 pid, __u32 cpu_id) +{ + __u32 flag1 = target & 0xff; + __u32 flag2 = (target >> 8) & 0xff; + + if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id)) + return 1; + return 0; +} + +static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry) +{ + __u64 curr; + __u32 pid; + __u32 cpu_id; + __u32 kind; + __u32 mode; + __u64 lock_addr; + + /* + * Pinned filters (use_idx_hash != 0) share a world-writable filters + * map and do not support lock_held(). Reject evaluation when pinned + * or when lock_enabled is not set so unprivileged users cannot probe + * arbitrary kernel addresses via entry->target. + */ + if (use_idx_hash || !lock_enabled) + return 0; + + curr = bpf_get_current_task(); + pid = (__u32)bpf_get_current_pid_tgid(); + cpu_id = 0x80000000U | bpf_get_smp_processor_id(); + kind = entry->part & PBF_LOCK_KIND_MASK; + mode = entry->part & PBF_LOCK_MODE_MASK; + + if (!mode) + mode = PBF_LOCK_MODE_ANY; + + switch (kind) { + case PBF_LOCK_ADDR: + lock_addr = entry->target; + if ((mode & PBF_LOCK_MODE_WRITE) && + is_direct_lock_owner(lock_addr, curr)) + return 1; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_MMAP: + lock_addr = get_current_mmap_lock(curr); + if (!lock_addr) + return 0; + if ((mode & PBF_LOCK_MODE_WRITE) && + (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr) + return 1; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_SIGHAND: + lock_addr = get_current_siglock(curr); + if (!lock_addr) + return 0; + return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id); + + case PBF_LOCK_CLASS: + return is_lock_in_map(entry->target, PBF_LOCK_CLASS, mode, pid, cpu_id); + + case PBF_LOCK_SLAB: + return is_lock_in_map(entry->target, PBF_LOCK_SLAB, mode, pid, cpu_id); + + case PBF_LOCK_TYPE: + return is_lock_type_in_map(entry->target, mode, pid, cpu_id); + + default: + return 0; + } +} + /* new kernel perf_sample_data definition */ struct perf_sample_data___new { __u64 sample_flags; @@ -175,6 +403,8 @@ static inline __u64 perf_get_sample(struct bpf_perf_event_data_kern *kctx, return bpf_get_current_uid_gid() & 0xFFFFFFFF; case PBF_TERM_GID: return bpf_get_current_uid_gid() >> 32; + case PBF_TERM_LOCK_HELD: + return check_lock_held(entry); case PBF_TERM_NONE: case __PBF_UNUSED_TERM4: case __PBF_UNUSED_TERM5: @@ -295,4 +525,739 @@ int perf_sample_filter(void *ctx) return 0; } +static __always_inline __u32 get_lock_owner_id(__u32 flags) +{ + __u32 owner_id; + + if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) + return 0x80000000U | bpf_get_smp_processor_id(); + + owner_id = (__u32)bpf_get_current_pid_tgid(); + if (!owner_id) + owner_id = 0x80000000U | bpf_get_smp_processor_id(); + return owner_id; +} + +static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id, + __u32 kind, __u32 flags, bool acquired) +{ + struct held_lock_key key = { + .addr_or_id = addr_or_id, + .owner_id = owner_id, + .kind = kind, + }; + struct held_lock_val *val; + bool is_read = (flags & LCB_F_READ) != 0; + + val = bpf_map_lookup_elem(&held_locks, &key); + if (acquired) { + if (val) { + if (is_read) + __sync_fetch_and_add(&val->read_count, 1); + else + __sync_fetch_and_add(&val->write_count, 1); + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags, + }; + + bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY); + } + } else if (val) { + if (is_read) { + if (val->read_count <= 1 && val->write_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->read_count > 0) + __sync_fetch_and_add(&val->read_count, -1); + } else { + if (val->write_count <= 1 && val->read_count == 0) + bpf_map_delete_elem(&held_locks, &key); + else if (val->write_count > 0) + __sync_fetch_and_add(&val->write_count, -1); + } + } +} + +static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id) +{ + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + bool matched = false; + bool is_read = (flags & LCB_F_READ) != 0; + __u32 class_id = 0; + __u64 kmem_cache_addr = 0; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool match_type = false; + + if (!lock_enabled || !lock_addr) + return 0; + + if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) + matched = true; + + if (has_mmap_lock || has_siglock) { + __u32 ltype = check_lock_type(lock_addr, flags); + + if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || + (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + matched = true; + } + + if (has_lock_class && flags == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) { + class_id = *cid; + matched = true; + } + } + + if (has_lock_slab && bpf_get_kmem_cache) { + kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) + matched = true; + else + kmem_cache_addr = 0; + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) { + match_type = true; + matched = true; + } + + if (!matched) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (existing) { + /* Non-reader locks are not recursive; avoid double-counting */ + if (!is_read && existing->write_count > 0) + return 0; + if (is_read) + __sync_fetch_and_add(&existing->read_count, 1); + else + __sync_fetch_and_add(&existing->write_count, 1); + } else { + struct held_lock_val first = { + .read_count = is_read ? 1 : 0, + .write_count = is_read ? 0 : 1, + .flags = flags, + }; + + if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0) + return 0; + } + + if (class_id) + update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, flags, true); + if (kmem_cache_addr) { + update_held_lock_count((__u64)kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, flags, true); + } + if (match_type) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, true); + + return 0; +} + +static __always_inline int record_lock_acquired(void *ctx, __u32 flags) +{ + return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags)); +} + +static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags) +{ + __u32 owner_id = get_lock_owner_id(flags); + struct held_lock_key addr_key = { + .addr_or_id = lock_addr, + .owner_id = owner_id, + .kind = PBF_LOCK_ADDR, + }; + struct held_lock_val *existing; + __u32 type_flag = flags & LCB_F_TYPE_MASK; + bool is_read = (flags & LCB_F_READ) != 0; + + if (!lock_enabled || !lock_addr) + return 0; + + existing = bpf_map_lookup_elem(&held_locks, &addr_key); + if (!existing) + return 0; + + if (is_read) { + if (existing->read_count == 0) + return 0; + if (existing->read_count == 1 && existing->write_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else + __sync_fetch_and_add(&existing->read_count, -1); + } else { + if (existing->write_count == 0) + return 0; + if (existing->read_count == 0) + bpf_map_delete_elem(&held_locks, &addr_key); + else + existing->write_count = 0; + } + + if (has_lock_class && flags == LCB_F_SPIN) { + __u32 *cid = bpf_map_lookup_elem(&lock_syms, &lock_addr); + + if (cid) + update_held_lock_count(*cid, owner_id, PBF_LOCK_CLASS, flags, false); + } + + if (has_lock_slab && bpf_get_kmem_cache) { + __u64 kmem_cache_addr = (__u64)bpf_get_kmem_cache(lock_addr); + + if (kmem_cache_addr && bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr)) { + update_held_lock_count(kmem_cache_addr, owner_id, + PBF_LOCK_SLAB, flags, false); + } + } + + if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false); + + return 0; +} + +static __always_inline int record_lock_released(void *ctx, __u32 flags) +{ + return record_lock_released_addr(get_lock_addr(ctx), flags); +} + +/* Mutex acquire/release hooks */ +SEC("fexit/mutex_lock") +int BPF_PROG(fexit_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_nested") +int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/_mutex_lock_nest_lock") +int BPF_PROG(fexit__mutex_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io") +int BPF_PROG(fexit_mutex_lock_io, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_io_nested") +int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_MUTEX); +} + +SEC("fexit/mutex_lock_interruptible") +int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_interruptible_nested") +int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_lock_killable") +int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/_mutex_lock_killable") +int BPF_PROG(fexit__mutex_lock_killable, void *lock, unsigned int subclass, + void *nest_lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/mutex_trylock") +int BPF_PROG(fexit_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fexit/_mutex_trylock_nest_lock") +int BPF_PROG(fexit__mutex_trylock_nest_lock, void *lock, void *nest_lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_MUTEX); + return 0; +} + +SEC("fentry/mutex_unlock") +int BPF_PROG(fentry_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_MUTEX); +} + +/* RW semaphore acquire/release hooks */ +SEC("fexit/down_read") +int BPF_PROG(fexit_down_read, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_nested") +int BPF_PROG(fexit_down_read_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_READ); +} + +SEC("fexit/down_read_interruptible") +int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable") +int BPF_PROG(fexit_down_read_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_killable_nested") +int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_read_trylock") +int BPF_PROG(fexit_down_read_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_READ); + return 0; +} + +SEC("fexit/down_write") +int BPF_PROG(fexit_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_nested") +int BPF_PROG(fexit_down_write_nested, void *sem, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/_down_write_nest_lock") +int BPF_PROG(fexit__down_write_nest_lock, void *sem, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_WRITE); +} + +SEC("fexit/down_write_killable") +int BPF_PROG(fexit_down_write_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_killable_nested") +int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/down_write_trylock") +int BPF_PROG(fexit_down_write_trylock, void *sem, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_WRITE); + return 0; +} + +SEC("fexit/downgrade_write") +int BPF_PROG(fexit_downgrade_write, void *sem) +{ + if (track_rwsem_read) + record_lock_acquired(ctx, LCB_F_READ); + return record_lock_released(ctx, LCB_F_WRITE); +} + +SEC("fentry/up_read") +int BPF_PROG(fentry_up_read, void *sem) +{ + return record_lock_released(ctx, LCB_F_READ); +} + +SEC("fentry/up_write") +int BPF_PROG(fentry_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_WRITE); +} + +/* Spinlock acquire/release hooks */ +SEC("fexit/_raw_spin_lock") +int BPF_PROG(fexit_raw_spin_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nested") +int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_nest_lock") +int BPF_PROG(fexit_raw_spin_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq") +int BPF_PROG(fexit_raw_spin_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irq_disable") +int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave") +int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_irqsave_nested") +int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_lock_bh") +int BPF_PROG(fexit_raw_spin_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN); +} + +SEC("fexit/_raw_spin_trylock") +int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fexit/_raw_spin_trylock_bh") +int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN); + return 0; +} + +SEC("fentry/_raw_spin_unlock") +int BPF_PROG(fentry_raw_spin_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq") +int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irq_enable") +int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_irqrestore") +int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +SEC("fentry/_raw_spin_unlock_bh") +int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN); +} + +/* RWlock acquire/release hooks */ +SEC("fexit/_raw_read_lock") +int BPF_PROG(fexit_raw_read_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_irq") +int BPF_PROG(fexit_raw_read_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_irqsave") +int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_lock_bh") +int BPF_PROG(fexit_raw_read_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_read_trylock") +int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); + return 0; +} + +SEC("fentry/_raw_read_unlock") +int BPF_PROG(fentry_raw_read_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irq") +int BPF_PROG(fentry_raw_read_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_irqrestore") +int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fentry/_raw_read_unlock_bh") +int BPF_PROG(fentry_raw_read_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ); +} + +SEC("fexit/_raw_write_lock") +int BPF_PROG(fexit_raw_write_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_nested") +int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irq") +int BPF_PROG(fexit_raw_write_lock_irq, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_irqsave") +int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_lock_bh") +int BPF_PROG(fexit_raw_write_lock_bh, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fexit/_raw_write_trylock") +int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE); + return 0; +} + +SEC("fentry/_raw_write_unlock") +int BPF_PROG(fentry_raw_write_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irq") +int BPF_PROG(fentry_raw_write_unlock_irq, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_irqrestore") +int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +SEC("fentry/_raw_write_unlock_bh") +int BPF_PROG(fentry_raw_write_unlock_bh, void *lock) +{ + return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE); +} + +/* RT-Mutex acquire/release hooks */ +SEC("fexit/rt_mutex_lock") +int BPF_PROG(fexit_rt_mutex_lock, void *lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_nested") +int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/_rt_mutex_lock_nest_lock") +int BPF_PROG(fexit__rt_mutex_lock_nest_lock, void *lock, void *nest_lock) +{ + return record_lock_acquired(ctx, LCB_F_RT); +} + +SEC("fexit/rt_mutex_lock_interruptible") +int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_lock_killable") +int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fexit/rt_mutex_trylock") +int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret) +{ + if (ret) + record_lock_acquired(ctx, LCB_F_RT); + return 0; +} + +SEC("fentry/rt_mutex_unlock") +int BPF_PROG(fentry_rt_mutex_unlock, void *lock) +{ + return record_lock_released(ctx, LCB_F_RT); +} + +/* Semaphore acquire/release hooks */ +SEC("fexit/down") +int BPF_PROG(fexit_down, void *sem) +{ + return record_lock_acquired(ctx, 0); +} + +SEC("fexit/down_interruptible") +int BPF_PROG(fexit_down_interruptible, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_killable") +int BPF_PROG(fexit_down_killable, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_timeout") +int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fexit/down_trylock") +int BPF_PROG(fexit_down_trylock, void *sem, int ret) +{ + if (ret == 0) + record_lock_acquired(ctx, 0); + return 0; +} + +SEC("fentry/up") +int BPF_PROG(fentry_up, void *sem) +{ + return record_lock_released(ctx, 0); +} + +/* Percpu-rwsem write acquire/release hooks */ +SEC("fexit/percpu_down_write") +int BPF_PROG(fexit_percpu_down_write, void *sem) +{ + return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("fentry/percpu_up_write") +int BPF_PROG(fentry_percpu_up_write, void *sem) +{ + return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); +} + +SEC("raw_tp/bpf_test_finish") +int BPF_PROG(collect_lock_syms) +{ + return do_collect_lock_syms(&lock_syms, contig_page_data_addr, + node_data_addr, nr_nodes, sizeof_zone); +} + +SEC("iter/kmem_cache") +int slab_cache_iter(void *ctx) +{ + return do_slab_cache_iter(ctx, &slab_caches, &slab_cache_id); +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c index 2f3317ce17f0..1fcb424b75c0 100644 --- a/tools/perf/util/parse-events.c +++ b/tools/perf/util/parse-events.c @@ -2596,7 +2596,8 @@ foreach_evsel_in_last_glob(struct evlist *evlist, /* Will a tracepoint filter work for str or should a BPF filter be used? */ static bool is_possible_tp_filter(const char *str) { - return strstr(str, "uid") == NULL; + return strstr(str, "uid") == NULL && strstr(str, "gid") == NULL && + strstr(str, "lock_held") == NULL; } static int set_filter(struct evsel *evsel, const void *arg) -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter 2026-10-17 17:14 ` [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers @ 2026-10-17 17:35 ` sashiko-bot 0 siblings, 0 replies; 24+ messages in thread From: sashiko-bot @ 2026-10-17 17:35 UTC (permalink / raw) To: Ian Rogers; +Cc: linux-perf-users Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] perf bpf-filter: rwlock read_count leak in record_lock_acquired_owner() - [Medium] perf bpf-filter: missing error return on node_data lookup in init_numa_data() -- commit 782c7a3aaec8d457ca5bf045258b33f6bbfbcae8 Author: Ian Rogers <irogers@google.com> perf bpf-filter: Add lock_held() sample filter This commit adds BPF sample filter predicates lock_held(<lock>), lock_held_read(<lock>) / lock_held_shared(<lock>), and lock_held_write(<lock>) / lock_held_exclusive(<lock>) to record perf event samples only when a specified kernel lock is held. It evaluates lock ownership directly in kernel memory where possible and falls back to a held_locks BPF hash map populated by fentry/fexit hooks on kernel lock/unlock functions. > diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c > index 72e83ac1d6a4b..2f4c6bd85f89f 100644 > --- a/tools/perf/util/bpf-filter.c > +++ b/tools/perf/util/bpf-filter.c [ ... ] > @@ -155,9 +171,814 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr * [ ... ] > +static void init_numa_data(struct sample_filter_bpf *skel, struct btf *btf) > +{ > + u64 addr = 0; > + char *buf = NULL, *p; > + size_t len; > + long last = -1; > + int ret; > + > + if (!btf) > + return; > + > + ret = btf__find_by_name_kind(btf, "zone", BTF_KIND_STRUCT); > + if (ret < 0) > + return; > + > + ret = btf__resolve_size(btf, ret); > + if (ret < 0) > + return; > + > + skel->rodata->sizeof_zone = ret; > + > + if (find_kernel_lock_sym("contig_page_data", &addr) == 0) { > + skel->rodata->contig_page_data_addr = addr; > + return; > + } > + > + if (find_kernel_lock_sym("node_data", &addr) == 0) > + skel->rodata->node_data_addr = addr; > + > + if (sysfs__read_str("devices/system/node/online", &buf, &len) >= 0) { > + p = buf; > + while (p && *p) { > + char *end; > + > + last = strtol(p, &end, 0); > + if (end == p) > + break; > + p = end; > + if (*p == ',' || *p == '-' || *p == '\n') > + p++; > + } > + skel->rodata->nr_nodes = last + 1; > + free(buf); > + } > +} [Severity: Medium] What happens in init_numa_data() when find_kernel_lock_sym("node_data", &addr) fails (for example, when /proc/kallsyms zeroes symbol addresses under kptr_restrict for a non-root user with CAP_BPF)? Unlike init_numa_data() in tools/perf/util/bpf_lock_contention.c, this does not return early on failure, so skel->rodata->node_data_addr remains 0 while skel->rodata->nr_nodes is populated from devices/system/node/online. When collect_lock_syms later calls do_collect_lock_syms() -> do_collect_zone_lock() in tools/perf/util/bpf_skel/lock_data.h: } else if (nr_nodes > 0) { struct pglist_data **node_data = (void *)(long)node_data_addr; for (int i = 0; i < nr_nodes; i++) { struct pglist_data *pgdat = NULL; int err; err = bpf_core_read(&pgdat, sizeof(pgdat), &node_data[i]); will this attempt to read from kernel address 0x0 and leave lock_syms unpopulated without reporting an error for lock_held(zone_lock)? [ ... ] > diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c > index e5666d4c17228..480ba4b49793d 100644 > --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c > +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c [ ... ] > @@ -295,4 +525,739 @@ int perf_sample_filter(void *ctx) [ ... ] > +static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id) > +{ [ ... ] > + existing = bpf_map_lookup_elem(&held_locks, &addr_key); > + if (existing) { > + /* Non-reader locks are not recursive; avoid double-counting */ > + if (!is_read && existing->write_count > 0) > + return 0; > + if (is_read) > + __sync_fetch_and_add(&existing->read_count, 1); > + else > + __sync_fetch_and_add(&existing->write_count, 1); > + } else { > + struct held_lock_val first = { > + .read_count = is_read ? 1 : 0, > + .write_count = is_read ? 0 : 1, > + .flags = flags, > + }; > + > + if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0) > + return 0; > + } > + > + if (class_id) > + update_held_lock_count(class_id, owner_id, PBF_LOCK_CLASS, flags, true); > + if (kmem_cache_addr) { > + update_held_lock_count((__u64)kmem_cache_addr, owner_id, > + PBF_LOCK_SLAB, flags, true); > + } > + if (match_type) > + update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, true); > + > + return 0; > +} [Severity: Medium] Can record_lock_acquired_owner() leak read_count in the held_locks map when rwlocks are acquired via read_lock_irq() or read_lock_bh() on kernels with CONFIG_GENERIC_LOCKBREAK=y and CONFIG_DEBUG_LOCK_ALLOC=n? In kernel/locking/spinlock.c, BUILD_LOCK_OPS(read, rwlock, __acquires_shared) implements __raw_read_lock_irq() and __raw_read_lock_bh() by calling the noinline function _raw_read_lock_irqsave(): static void __lockfunc __raw_##op##_lock_irq(locktype##_t *lock) \ lock_ctx_op(lock) \ { \ _raw_##op##_lock_irqsave(lock); \ } \ \ static void __lockfunc __raw_##op##_lock_bh(locktype##_t *lock) \ lock_ctx_op(lock) \ { \ unsigned long flags; \ flags = _raw_##op##_lock_irqsave(lock); \ local_bh_disable(); \ local_irq_restore(flags); \ } Since LOCK_PROG_RWLOCK_READ attaches fexit hooks to _raw_read_lock_irq, _raw_read_lock_bh, and _raw_read_lock_irqsave, a single read_lock_irq() or read_lock_bh() call fires both the inner fexit_raw_read_lock_irqsave hook and the outer fexit_raw_read_lock_irq / fexit_raw_read_lock_bh hook. Because is_read is true for LCB_F_SPIN | LCB_F_READ, the nested-call check (!is_read && existing->write_count > 0) is bypassed and read_count is incremented twice (for PBF_LOCK_ADDR as well as any active PBF_LOCK_SLAB or PBF_LOCK_TYPE entries). On release, _raw_read_unlock_irq() and _raw_read_unlock_bh() do not call _raw_read_unlock_irqrestore(), so record_lock_released() only runs once, leaving read_count permanently elevated in held_locks. [ ... ] > +/* RWlock acquire/release hooks */ > +SEC("fexit/_raw_read_lock") > +int BPF_PROG(fexit_raw_read_lock, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); > +} > + > +SEC("fexit/_raw_read_lock_irq") > +int BPF_PROG(fexit_raw_read_lock_irq, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); > +} > + > +SEC("fexit/_raw_read_lock_irqsave") > +int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); > +} > + > +SEC("fexit/_raw_read_lock_bh") > +int BPF_PROG(fexit_raw_read_lock_bh, void *lock) > +{ > + return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ); > +} [ ... ] -- Sashiko AI review · https://sashiko.dev/#/patchset/20261017171437.2066343-1-irogers@google.com?part=2 ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH v2 3/6] perf test: Add tests for lock_held() BPF sample filter 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers 2026-10-17 17:14 ` [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers @ 2026-10-17 17:14 ` Ian Rogers 2026-10-17 17:17 ` sashiko-bot 2026-10-17 17:14 ` [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers ` (2 subsequent siblings) 5 siblings, 1 reply; 24+ messages in thread From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Add unit tests in parse-events.c for parsing lock_held(), lock_held_read(), and lock_held_exclusive() BPF filter expressions (standalone predicates, negated predicates, and compound group expressions) and extend record_bpf_filter.sh with lock_held(), lock_held_read()/lock_held_write(), and ':R'/':W' recording tests. Example output: # perf test -v 'Parsing of BPF sample filters' 5: Parse event definition strings : 5.7: Parsing of BPF sample filters : Ok # perf test -v 'perf record sample filtering (by BPF) tests' 112: perf record sample filtering (by BPF) tests : Checking BPF-filter privilege Basic bpf-filter test Basic bpf-filter test [Success] Failing bpf-filter test Failing bpf-filter test [Success] Group bpf-filter test Group bpf-filter test [Success] Multiple bpf-filter test Multiple bpf-filter test [Success] Cgroup bpf-filter test Cgroup bpf-filter test [Success] Lock-held bpf-filter test Lock-held bpf-filter test [Success] 112: perf record sample filtering (by BPF) tests : Ok Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/tests/parse-events.c | 108 ++++++++++++++++++++ tools/perf/tests/shell/record_bpf_filter.sh | 103 +++++++++++++++++++ 2 files changed, 211 insertions(+) diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 5f6f95c8a5b4..4cf9192762f2 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -8,9 +8,11 @@ #include "debug.h" #include "pmu.h" #include "pmus.h" +#include "bpf-filter.h" #include "strbuf.h" #include <dirent.h> #include <errno.h> +#include <string.h> #include "fncache.h" #include <sys/types.h> #include <sys/stat.h> @@ -2938,6 +2940,111 @@ static int test__pmu_events_alias2(struct test_suite *test __maybe_unused, return ret; } +#ifdef HAVE_BPF_SKEL +static void clear_bpf_filter_exprs(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *tmp; + + list_for_each_entry_safe(expr, tmp, expr_head, list) { + list_del(&expr->list); + perf_bpf_filter_expr__delete(expr); + } +} + +static int do_test_bpf_filters(struct list_head *expr_head) +{ + struct perf_bpf_filter_expr *expr, *group; + const char *compound = "lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100"; + + /* Simple lock_held(&cgroup_mutex) */ + TEST_ASSERT_VAL("parse lock_held(&cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "lock_held(&cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "&cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Negated !lock_held(cgroup_mutex) */ + TEST_ASSERT_VAL("parse !lock_held(cgroup_mutex)", + __perf_bpf_filter__parse(expr_head, "!lock_held(cgroup_mutex)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is cgroup_mutex", + expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); + clear_bpf_filter_exprs(expr_head); + + /* Read/shared and write/exclusive predicates */ + TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", + __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_READ", expr->part == PBF_LOCK_MODE_READ); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is &mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "&mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + TEST_ASSERT_VAL("parse !lock_held_exclusive(mmap_lock)", + __perf_bpf_filter__parse(expr_head, + "!lock_held_exclusive(mmap_lock)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_WRITE", expr->part == PBF_LOCK_MODE_WRITE); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 0", expr->val == 0); + TEST_ASSERT_VAL("lock_name is mmap_lock", + expr->lock_name && !strcmp(expr->lock_name, "mmap_lock")); + clear_bpf_filter_exprs(expr_head); + + /* + * Comparison and compound group: + * lock_held(rwsem:R) == 1 || lock_held(&dentry) != 0, period > 100 + */ + TEST_ASSERT_VAL("parse compound lock_held filter", + __perf_bpf_filter__parse(expr_head, compound) == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("group op", expr->op == PBF_OP_GROUP_BEGIN); + TEST_ASSERT_VAL("groups not empty", !list_empty(&expr->groups)); + group = list_first_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("first group member lock_name", + group->lock_name && !strcmp(group->lock_name, "rwsem:R")); + TEST_ASSERT_VAL("first group member op", group->op == PBF_OP_EQ && group->val == 1); + group = list_last_entry(&expr->groups, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second group member lock_name", + group->lock_name && !strcmp(group->lock_name, "&dentry")); + TEST_ASSERT_VAL("second group member op", group->op == PBF_OP_NEQ && group->val == 0); + expr = list_last_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("second term is period", expr->term == PBF_TERM_PERIOD && expr->val == 100); + + return TEST_OK; +} +#endif + +static int test__bpf_filters(struct test_suite *test __maybe_unused, + int subtest __maybe_unused) +{ +#ifdef HAVE_BPF_SKEL + struct list_head expr_head = LIST_HEAD_INIT(expr_head); + int ret = do_test_bpf_filters(&expr_head); + + clear_bpf_filter_exprs(&expr_head); + return ret; +#else + return TEST_SKIP; +#endif +} + static struct test_case tests__parse_events[] = { TEST_CASE_REASON("Test event parsing", events2, @@ -2952,6 +3059,7 @@ static struct test_case tests__parse_events[] = { "no aliases in sysfs"), TEST_CASE("Parsing of aliased events", pmu_events_alias2), TEST_CASE("Parsing of terms (event modifiers)", terms2), + TEST_CASE("Parsing of BPF sample filters", bpf_filters), { .name = NULL, } }; diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index 383574cb3bd3..adaa5363d0b7 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -176,6 +176,105 @@ test_bpf_filter_cgroup() { echo "Cgroup bpf-filter test [Success]" } +test_bpf_filter_lock_held() { + echo "Lock-held bpf-filter test" + lock_sym="cgroup_mutex" + + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + lock_sym="slab_mutex" + fi + + if ! perf record -e task-clock --filter "lock_held(&${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + if [ "$(id -u)" != 0 ] + then + echo "Lock-held bpf-filter test [Skipped permission]" + return + fi + if [ -r /proc/kallsyms ] && ! grep -qw "${lock_sym}" /proc/kallsyms + then + echo "Lock-held bpf-filter test [Skipped missing ${lock_sym}]" + return + fi + echo "Lock-held bpf-filter test [Failed &${lock_sym}]" + err=1 + return + fi + + if ! perf record -e task-clock --filter "!lock_held(${lock_sym})" \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed !lock_held(${lock_sym})]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(mmap_lock)' \ + -o "${perfdata}" perf bench mem mmap -t 2 -l 10 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed mmap_lock]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \ + -o "${perfdata}" true 2> /dev/null + then + if ! perf record -e task-clock --filter 'lock_held(rq_lock) || lock_held(spinlock)' \ + -o /dev/null true 2>&1 | grep -q "Kernel BTF lacks required" + then + echo "Lock-held bpf-filter test [Failed rq_lock/spinlock]" + err=1 + return + fi + fi + + if ! perf record -e task-clock --filter 'lock_held(mutex) || lock_held(rwsem)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock type]" + err=1 + return + fi + + if ! perf record -e task-clock \ + --filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read/write(&mmap_lock)]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held(&mmap_lock:R) || lock_held(&mmap_lock:W)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed &mmap_lock:R/:W]" + err=1 + return + fi + + if perf record -e task-clock --filter 'lock_held_read(rq_lock)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rq_lock) should fail]" + err=1 + return + fi + + if perf record -e task-clock --filter 'lock_held(nonexistent_lock_xyz)' \ + -o /dev/null true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed nonexistent lock should fail]" + err=1 + return + fi + + echo "Lock-held bpf-filter test [Success]" +} + test_bpf_filter_priv if [ $err = 0 ]; then @@ -198,5 +297,9 @@ if [ $err = 0 ]; then test_bpf_filter_cgroup fi +if [ $err = 0 ]; then + test_bpf_filter_lock_held +fi + cleanup exit $err -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH v2 3/6] perf test: Add tests for lock_held() BPF sample filter 2026-10-17 17:14 ` [PATCH v2 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers @ 2026-10-17 17:17 ` sashiko-bot 0 siblings, 0 replies; 24+ messages in thread From: sashiko-bot @ 2026-10-17 17:17 UTC (permalink / raw) To: Ian Rogers; +Cc: linux-perf-users > Add unit tests in parse-events.c for parsing lock_held(), > lock_held_read(), and lock_held_exclusive() BPF filter expressions > (standalone predicates, negated predicates, and compound group > expressions) and extend record_bpf_filter.sh with lock_held(), > lock_held_read()/lock_held_write(), and ':R'/':W' recording tests. > > Example output: > > # perf test -v 'Parsing of BPF sample filters' > 5: Parse event definition strings : > 5.7: Parsing of BPF sample filters : Ok > > # perf test -v 'perf record sample filtering (by BPF) tests' > 112: perf record sample filtering (by BPF) tests : > Checking BPF-filter privilege > Basic bpf-filter test > Basic bpf-filter test [Success] > Failing bpf-filter test > Failing bpf-filter test [Success] > Group bpf-filter test > Group bpf-filter test [Success] > Multiple bpf-filter test > Multiple bpf-filter test [Success] > Cgroup bpf-filter test > Cgroup bpf-filter test [Success] > Lock-held bpf-filter test > Lock-held bpf-filter test [Success] > 112: perf record sample filtering (by BPF) tests : Ok > > Signed-off-by: Ian Rogers <irogers@google.com> Sashiko has reviewed this patch and found no issues. -- Sashiko AI review · https://sashiko.dev/#/patchset/20261017171437.2066343-1-irogers@google.com?part=3 ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (2 preceding siblings ...) 2026-10-17 17:14 ` [PATCH v2 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers @ 2026-10-17 17:14 ` Ian Rogers 2026-10-17 17:25 ` sashiko-bot 2026-10-17 17:14 ` [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers 2026-10-17 17:14 ` [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers 5 siblings, 1 reply; 24+ messages in thread From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks Trace RCU, and SRCU locks: - 'rcu' (supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)', 'lock_held(rcu_read_lock)'): directly inspects current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on CONFIG_PREEMPT_RCU kernels), accounting for the baseline rcu_read_lock() nesting acquired by the perf overflow handler itself (bpf_overflow_handler() and do_perf_sw_event()) with zero extra fentry/fexit tracing hooks. - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'): attaches fentry/fexit hooks to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), and rcu_barrier() to track grace-period and barrier waits. - 'srcu' and specific SRCU symbols (e.g. '&srcu_symbol', supports both read/shared and write/exclusive modes): - Read/shared ('lock_held_read(srcu)', 'lock_held(srcu:R)'): attaches fexit/__srcu_read_lock and fentry/__srcu_read_unlock. - Write/exclusive ('lock_held_write(srcu)', 'lock_held(srcu:W)'): attaches fentry/fexit on synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier(). - 'rcu-trace' / 'rcu_trace' / 'rcu_read_lock_trace' (supports read/shared mode only): directly inspects current->trc_reader_nesting > 0 on CONFIG_TASKS_TRACE_RCU kernels with zero extra tracing hooks. Example usage: # perf record -a -g -e cycles --filter 'lock_held_read(rcu)' -- sleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 1.482 MB perf.data (412 samples) ] # perf report --stdio --no-children # Overhead Command Shared Object Symbol # ........ ....... ................. ........................ # 14.32% perf [kernel.kallsyms] [k] __d_lookup_rcu | ---__d_lookup_rcu lookup_fast walk_component link_path_walk path_lookupat filename_lookup vfs_statx __do_sys_newstat do_syscall_64 entry_SYSCALL_64_after_hwframe # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1 # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1 Signed-off-by: Ian Rogers <irogers@google.com> # Conflicts: # tools/perf/util/bpf-filter.c # tools/perf/util/bpf_skel/sample_filter.bpf.c --- tools/perf/Documentation/perf-record.txt | 42 +++-- tools/perf/tests/parse-events.c | 13 ++ tools/perf/tests/shell/record_bpf_filter.sh | 20 +++ tools/perf/util/bpf-filter.c | 174 +++++++++++++++++-- tools/perf/util/bpf_skel/lock_data.h | 44 ++++- tools/perf/util/bpf_skel/sample-filter.h | 12 ++ tools/perf/util/bpf_skel/sample_filter.bpf.c | 135 +++++++++++++- tools/perf/util/bpf_skel/vmlinux/vmlinux.h | 2 + tools/perf/util/lock-contention.c | 4 + tools/perf/util/lock-contention.h | 4 +- 10 files changed, 418 insertions(+), 32 deletions(-) diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 246ae085744d..c8ccfb0e0ce6 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -219,28 +219,38 @@ OPTIONS '&', e.g. 'lock_held(&cgroup_mutex)' or 'lock_held(cgroup_mutex)'), a hexadecimal lock address, a slab cache name prefixed with '&' (e.g. 'lock_held(&dentry)', 'lock_held_read(&dentry)'), a well-known - lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock'), or a lock - type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock', - 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W', - 'semaphore'). Both read/shared and write/exclusive tracking are - supported for 'rwsem' (including 'mmap_lock') and 'rwlock'; + lock name ('mmap_lock', 'siglock', 'rq_lock', 'zone_lock', 'rcu', + 'rcu-trace'), or a lock type ('mutex', 'spinlock', 'rwsem', 'rwsem:R', + 'rwsem:W', 'rwlock', 'rwlock:R', 'rwlock:W', 'rt-mutex', + 'percpu-rwsem', 'pcpu-sem:W', 'srcu', 'srcu:R', 'srcu:W', 'rcu:R', + 'rcu:W', 'semaphore'). Both read/shared and write/exclusive tracking + are supported for 'rwsem' (including 'mmap_lock'), 'rwlock', 'rcu', + and 'srcu' (where read/shared mode matches 'rcu_read_lock()' / + 'srcu_read_lock()' critical sections and write/exclusive mode matches + grace-period waits 'synchronize_rcu*()', 'synchronize_srcu*()', + 'rcu_barrier()', and 'srcu_barrier()'); 'rcu-trace' + ('rcu_read_lock_trace()') supports read/shared tracking only; and 'mutex', 'spinlock' (including 'siglock', 'rq_lock', 'zone_lock'), 'rt-mutex', 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support write/exclusive tracking only. Note that 'lock_held(<lock>)' introduces overhead outside of sample overflow handling: in addition to checking the lock's owner field on - each sample where possible (such as for mutexes, rt-mutexes, and - write-locked rwsems), it attaches BPF fentry/fexit programs to kernel - lock acquire and release functions to track held locks (such as - spinlocks, rwlocks, reader-locked rwsems, and slab/type filters) in a - BPF map. These BPF hooks run on every kernel lock acquisition and - release for the attached lock family, even when no perf sample is - being taken. To minimize this overhead, only the hooks for the - relevant lock family and mode are attached when known (for example, - 'lock_held_read(&mmap_lock)' only hooks rwsem read functions, - 'lock_held_write(&mmap_lock)' only hooks rwsem write functions, - '&cgroup_mutex' only hooks mutex functions, and 'rq_lock' only hooks + each sample where possible (such as for mutexes, rt-mutexes, + write-locked rwsems, and 'rcu:R' / 'rcu-trace' which inspect + 'current->rcu_read_lock_nesting' / 'current->trc_reader_nesting' + directly with no extra tracing hooks), it attaches BPF fentry/fexit + programs to kernel lock acquire and release functions to track held + locks (such as spinlocks, rwlocks, reader-locked rwsems, SRCU locks, + RCU synchronize waits, and slab/type filters) in a BPF map. These + BPF hooks run on every kernel lock acquisition and release for the + attached lock family, even when no perf sample is being taken. To + minimize this overhead, only the hooks for the relevant lock family + and mode are attached when known (for example, 'lock_held_read(rcu)' + and 'rcu-trace' attach no extra hooks, 'lock_held_read(&mmap_lock)' + only hooks rwsem read functions, 'lock_held_write(&mmap_lock)' only + hooks rwsem write functions, '&cgroup_mutex' only hooks mutex + functions, 'srcu' only hooks SRCU functions, and 'rq_lock' only hooks spinlock functions), whereas slab filters ('&<slab>'), raw hex addresses, or symbols without a recognizable lock-type suffix attach hooks across all lock families (including high-frequency spinlocks). diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c index 4cf9192762f2..730466b7ebf6 100644 --- a/tools/perf/tests/parse-events.c +++ b/tools/perf/tests/parse-events.c @@ -2980,6 +2980,19 @@ static int do_test_bpf_filters(struct list_head *expr_head) expr->lock_name && !strcmp(expr->lock_name, "cgroup_mutex")); clear_bpf_filter_exprs(expr_head); + /* RCU and SRCU lock_held(rcu) || lock_held(srcu) */ + TEST_ASSERT_VAL("parse lock_held(rcu)", + __perf_bpf_filter__parse(expr_head, "lock_held(rcu)") == 0); + TEST_ASSERT_VAL("not empty", !list_empty(expr_head)); + expr = list_first_entry(expr_head, struct perf_bpf_filter_expr, list); + TEST_ASSERT_VAL("term is PBF_TERM_LOCK_HELD", expr->term == PBF_TERM_LOCK_HELD); + TEST_ASSERT_VAL("part is PBF_LOCK_MODE_ANY", expr->part == PBF_LOCK_MODE_ANY); + TEST_ASSERT_VAL("op is PBF_OP_EQ", expr->op == PBF_OP_EQ); + TEST_ASSERT_VAL("val is 1", expr->val == 1); + TEST_ASSERT_VAL("lock_name is rcu", + expr->lock_name && !strcmp(expr->lock_name, "rcu")); + clear_bpf_filter_exprs(expr_head); + /* Read/shared and write/exclusive predicates */ TEST_ASSERT_VAL("parse lock_held_read(&mmap_lock)", __perf_bpf_filter__parse(expr_head, "lock_held_read(&mmap_lock)") == 0); diff --git a/tools/perf/tests/shell/record_bpf_filter.sh b/tools/perf/tests/shell/record_bpf_filter.sh index adaa5363d0b7..dc1c1a02eaad 100755 --- a/tools/perf/tests/shell/record_bpf_filter.sh +++ b/tools/perf/tests/shell/record_bpf_filter.sh @@ -239,6 +239,26 @@ test_bpf_filter_lock_held() { return fi + if ! perf record -e task-clock --filter 'lock_held_write(rcu) || lock_held(srcu)' \ + -o "${perfdata}" true 2> /dev/null + then + echo "Lock-held bpf-filter test [Failed rcu:W/srcu]" + err=1 + return + fi + + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o "${perfdata}" true 2> /dev/null + then + if ! perf record -e task-clock --filter 'lock_held_read(rcu)' \ + -o /dev/null true 2>&1 | grep -q CONFIG_PREEMPT_RCU + then + echo "Lock-held bpf-filter test [Failed lock_held_read(rcu)]" + err=1 + return + fi + fi + if ! perf record -e task-clock \ --filter 'lock_held_read(&mmap_lock) || lock_held_write(&mmap_lock)' \ -o "${perfdata}" true 2> /dev/null diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 2f4c6bd85f89..6261a79b1455 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -190,11 +190,19 @@ enum lock_prog_group { LOCK_PROG_RTMUTEX = 1U << 6, LOCK_PROG_SEM = 1U << 7, LOCK_PROG_PERCPU_RWSEM = 1U << 8, - LOCK_PROG_ALL = 0x1ffU, + LOCK_PROG_SRCU_READ = 1U << 9, + LOCK_PROG_SRCU_WRITE = 1U << 10, + LOCK_PROG_SRCU = LOCK_PROG_SRCU_READ | LOCK_PROG_SRCU_WRITE, + LOCK_PROG_RCU_WRITE = 1U << 11, + LOCK_PROG_ALL = 0xfffU, }; static unsigned int lock_flags_to_prog_group(unsigned int flags) { + if (flags & LCB_F_RCU) + return (flags & LCB_F_WRITE) ? LOCK_PROG_RCU_WRITE : 0; + if (flags & LCB_F_SRCU) + return (flags & LCB_F_READ) ? LOCK_PROG_SRCU_READ : LOCK_PROG_SRCU_WRITE; if (flags & LCB_F_MUTEX) return LOCK_PROG_MUTEX; if (flags & LCB_F_RT) @@ -314,6 +322,32 @@ static const struct lock_prog_desc lock_progs[] = { /* Percpu-RWsem */ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM), LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM), + /* SRCU */ + LOCK_FPROG(fexit_srcu_read_lock, "__srcu_read_lock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock, "__srcu_read_unlock", LOCK_PROG_SRCU_READ), + LOCK_FPROG(fexit_srcu_read_lock_nmisafe, "__srcu_read_lock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_srcu_read_unlock_nmisafe, "__srcu_read_unlock_nmisafe", + LOCK_PROG_SRCU_READ), + LOCK_FPROG(fentry_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu, "synchronize_srcu", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_synchronize_srcu_expedited, "synchronize_srcu_expedited", + LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fentry_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + LOCK_FPROG(fexit_srcu_barrier, "srcu_barrier", LOCK_PROG_SRCU_WRITE), + /* RCU grace-period synchronize / barrier */ + LOCK_FPROG(fentry_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu, "synchronize_rcu", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_expedited, "synchronize_rcu_expedited", + LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_synchronize_rcu_tasks, "synchronize_rcu_tasks", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fentry_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), + LOCK_FPROG(fexit_rcu_barrier, "rcu_barrier", LOCK_PROG_RCU_WRITE), }; #undef LOCK_FPROG @@ -597,6 +631,18 @@ static unsigned int supported_lock_prog_groups(struct btf *btf) if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write")) supported |= LOCK_PROG_PERCPU_RWSEM; + if ((btf_has_func(btf, "__srcu_read_lock") || + btf_has_func(btf, "__srcu_read_lock_nmisafe")) && + (btf_has_func(btf, "__srcu_read_unlock") || + btf_has_func(btf, "__srcu_read_unlock_nmisafe"))) + supported |= LOCK_PROG_SRCU_READ; + + if (btf_has_func(btf, "synchronize_srcu")) + supported |= LOCK_PROG_SRCU_WRITE; + + if (btf_has_func(btf, "synchronize_rcu")) + supported |= LOCK_PROG_RCU_WRITE; + return supported; } @@ -701,6 +747,19 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, *needs_lock_syms = true; return; } + if (!strcmp(base, "rcu")) { + if (mode & PBF_LOCK_MODE_WRITE) { + *prog_groups |= LOCK_PROG_RCU_WRITE; + *required_groups |= LOCK_PROG_RCU_WRITE; + } + return; + } + if (!strcmp(base, "rcu_read_lock") || + !strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + /* Inspected directly on task_struct; no tracing hooks needed. */ + return; + } if (!has_amp && parse_lock_type_name(base, mode, NULL, NULL, &groups)) { *prog_groups |= groups; *required_groups |= groups; @@ -744,6 +803,13 @@ static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr, groups |= LOCK_PROG_RWLOCK_WRITE; *prog_groups |= groups; *required_groups |= groups; + } else if (strtailcmp(base, "srcu") == 0) { + if (mode & PBF_LOCK_MODE_READ) + groups |= LOCK_PROG_SRCU_READ; + if (mode & PBF_LOCK_MODE_WRITE) + groups |= LOCK_PROG_SRCU_WRITE; + *prog_groups |= groups; + *required_groups |= groups; } else { *prog_groups |= LOCK_PROG_ALL; } @@ -830,16 +896,54 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, return false; } +static u64 rcu_min_depth(struct evsel *evsel) +{ + if (evsel && evsel->core.attr.type == PERF_TYPE_SOFTWARE && + evsel->core.attr.config != PERF_COUNT_SW_CPU_CLOCK && + evsel->core.attr.config != PERF_COUNT_SW_TASK_CLOCK) + return 2; + return 1; +} + +static bool btf_has_task_field(struct btf *btf, const char *field_name) +{ + const struct btf_type *t; + const struct btf_member *m; + s32 id; + u16 i, vlen; + + if (!btf) + return false; + + id = btf__find_by_name_kind(btf, "task_struct", BTF_KIND_STRUCT); + if (id < 0) + return false; + + t = btf__type_by_id(btf, id); + if (!t) + return false; + + m = btf_members(t); + vlen = btf_vlen(t); + for (i = 0; i < vlen; i++, m++) { + const char *name = btf__name_by_offset(btf, m->name_off); + + if (name && !strcmp(name, field_name)) + return true; + } + return false; +} + /* * Resolve a `lock_held(<name>)` filter expression into `entry->part` * (`enum perf_bpf_lock_target_kind`) and `entry->target`, and populate * the corresponding BPF filter maps and BSS control flags. * * Resolution order: - * 1. Well-known per-process / semi-global locks: `mmap_lock`, `siglock`, - * `rq_lock`, `zone_lock` (with or without leading '&'). + * 1. Well-known per-process / semi-global / RCU locks: `mmap_lock`, `siglock`, + * `rq_lock`, `zone_lock`, `rcu`, `rcu-trace` (with or without leading '&'). * 2. Lock types / flags (when no leading '&'): `mutex`, `spinlock`, - * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, + * `rwsem`, `rwsem:R`, `rwsem:W`, `rwlock`, `percpu-rwsem`, `srcu`, * `semaphore`, etc. * 3. Hexadecimal lock address (when no leading '&'). * 4. Slab cache name (when prefixed with '&', e.g. `&dentry`). @@ -847,8 +951,8 @@ static bool find_slab_cache(struct sample_filter_bpf *skel, const char *name, * e.g. `&cgroup_mutex` or `cgroup_mutex`). * 6. Slab cache name without leading '&' fallback. */ -static int resolve_lock_expr(struct sample_filter_bpf *skel, - struct perf_bpf_filter_expr *expr, +static int resolve_lock_expr(struct evsel *evsel, struct sample_filter_bpf *skel, + struct btf *btf, struct perf_bpf_filter_expr *expr, struct perf_bpf_filter_entry *entry) { const char *name = expr->lock_name; @@ -912,6 +1016,48 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, return 0; } + if (!strcmp(base, "rcu_read_lock")) { + if (mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | PBF_LOCK_MODE_READ; + entry->target = rcu_min_depth(evsel); + return 0; + } + + if (!strcmp(base, "rcu")) { + if ((mode & PBF_LOCK_MODE_READ) && btf && + !btf_has_task_field(btf, "rcu_read_lock_nesting")) { + pr_err("Error: lock_held(rcu) requires CONFIG_PREEMPT_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU | mode; + entry->target = rcu_min_depth(evsel); + if ((mode & PBF_LOCK_MODE_WRITE) && skel) + skel->bss->has_rcu_lock = 1; + return 0; + } + + if (!strcmp(base, "rcu-trace") || !strcmp(base, "rcu_trace") || + !strcmp(base, "rcu_read_lock_trace")) { + if (!(mode & PBF_LOCK_MODE_READ) || mode == PBF_LOCK_MODE_WRITE) { + pr_err("Error: Lock '%s' only supports read/shared mode\n", name); + return -EINVAL; + } + if (btf && !btf_has_task_field(btf, "trc_reader_nesting")) { + pr_err("Error: lock_held(rcu-trace) requires CONFIG_TASKS_TRACE_RCU\n"); + return -EOPNOTSUPP; + } + entry->part = PBF_LOCK_RCU_TRACE | PBF_LOCK_MODE_READ; + entry->target = 0; + return 0; + } + if (!has_amp && parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) { entry->part = PBF_LOCK_TYPE | mode; entry->target = encoded_pair; @@ -976,7 +1122,7 @@ static int resolve_lock_expr(struct sample_filter_bpf *skel, } static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel, - struct perf_bpf_filter_entry *entry) + struct btf *btf, struct perf_bpf_filter_entry *entry) { int i = 0, ret; struct perf_bpf_filter_expr *expr; @@ -994,7 +1140,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].value = expr->val; entry[i].target = 0; if (expr->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, expr, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, expr, &entry[i]); if (ret < 0) return ret; } @@ -1013,7 +1159,7 @@ static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *ske entry[i].value = group->val; entry[i].target = 0; if (group->term == PBF_TERM_LOCK_HELD) { - ret = resolve_lock_expr(skel, group, &entry[i]); + ret = resolve_lock_expr(evsel, skel, btf, group, &entry[i]); if (ret < 0) return ret; } @@ -1287,7 +1433,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) goto err; } - ret = get_filter_entries(evsel, NULL, entry); + ret = get_filter_entries(evsel, NULL, NULL, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); goto err; @@ -1349,6 +1495,12 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) bpf_map__set_max_entries(skel->maps.lock_syms, MAX_ENTRIES); init_numa_data(skel, btf); } + if (prog_groups & LOCK_PROG_RCU_WRITE) { + u64 rcu_addr = 1ULL; + + find_kernel_lock_sym("rcu_state", &rcu_addr); + skel->rodata->rcu_state_addr = rcu_addr; + } ret = configure_lock_progs(skel, btf, prog_groups, required_groups, needs_lock_syms, needs_slab_iter); if (ret < 0) @@ -1366,7 +1518,7 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target) if (has_lock_filter && needs_slab_iter) run_slab_cache_iter(skel); - ret = get_filter_entries(evsel, skel, entry); + ret = get_filter_entries(evsel, skel, btf, entry); if (ret < 0) { pr_err("Failed to process filter entries\n"); goto err; diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index bf39128e0be1..85570798baae 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -39,6 +39,7 @@ struct contention_task_data { */ #define LCD_F_MMAP_LOCK (1U << 31) #define LCD_F_SIGHAND_LOCK (1U << 30) +#define LCD_F_RCU_LOCK (1U << 29) #define LCB_F_SLAB_ID_SHIFT 16 #define LCB_F_SLAB_ID_START (1U << 16) @@ -52,9 +53,11 @@ struct contention_task_data { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) -#define LCB_F_TYPE_MAX (1U << 7) -#define LCB_F_TYPE_MASK 0x0000007FU +#define LCB_F_TYPE_MAX (1U << 8) +#define LCB_F_TYPE_MASK 0x000000FFU #define SLAB_NAME_MAX 28 @@ -241,6 +244,40 @@ static __always_inline __u64 get_current_siglock(__u64 curr) return sh + bpf_core_field_offset(sighand->siglock); } +/* + * Check if `curr` is inside a preemptible RCU read-side critical section + * (`rcu_read_lock()`). + * + * Because `bpf_overflow_handler()` in `kernel/events/core.c` acquires + * `rcu_read_lock()` before invoking `perf_sample_filter` (and + * `do_perf_sw_event()` acquires an additional `rcu_read_lock()` for + * non-clock software events), `min_depth` is the baseline nesting count + * contributed by the perf event overflow handler itself (1 or 2). + */ +static __always_inline bool is_rcu_lock_held(__u64 curr, int min_depth) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->rcu_read_lock_nesting)) + return false; + + return BPF_CORE_READ(task, rcu_read_lock_nesting) > min_depth; +} + +/* + * Check if `curr` is inside a Tasks Trace RCU read-side critical section + * (`rcu_read_lock_trace()`). + */ +static __always_inline bool is_rcu_trace_lock_held(__u64 curr) +{ + struct task_struct *task = (void *)curr; + + if (!curr || !bpf_core_field_exists(task->trc_reader_nesting)) + return false; + + return BPF_CORE_READ(task, trc_reader_nesting) > 0; +} + /* * Read the 1st function argument (`lock` / `sem` / `ssp`) as a 64-bit scalar. * @@ -273,6 +310,9 @@ static __always_inline __u32 check_lock_type(__u64 lock, __u32 flags) if (get_current_siglock(curr) == lock) return LCD_F_SIGHAND_LOCK; break; + case LCB_F_RCU: + case LCB_F_RCU | LCB_F_WRITE: + return LCD_F_RCU_LOCK; default: break; } diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h index cd711d1d04b6..37c108608bbc 100644 --- a/tools/perf/util/bpf_skel/sample-filter.h +++ b/tools/perf/util/bpf_skel/sample-filter.h @@ -99,6 +99,18 @@ enum perf_bpf_lock_target_kind { * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused). */ PBF_LOCK_TYPE, + /* + * Preemptible RCU read-side lock (`rcu` / `rcu_read_lock`). + * Inspected directly via `current->rcu_read_lock_nesting > entry.target`, + * where `entry.target` is the baseline RCU nesting depth added by the + * kernel perf overflow path. + */ + PBF_LOCK_RCU, + /* + * Tasks Trace RCU read-side lock (`rcu-trace` / `rcu_read_lock_trace`). + * Inspected directly via `current->trc_reader_nesting > 0`. + */ + PBF_LOCK_RCU_TRACE, }; /* diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c index 480ba4b49793..6f3adbbe0289 100644 --- a/tools/perf/util/bpf_skel/sample_filter.bpf.c +++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c @@ -123,6 +123,7 @@ volatile const int use_idx_hash; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -136,6 +137,7 @@ int has_lock_class; int has_lock_type; int has_mmap_lock; int has_siglock; +int has_rcu_lock; int track_rwsem_read; __u8 lock_type_filter[LCB_F_TYPE_MAX]; @@ -270,6 +272,20 @@ static __always_inline __u64 check_lock_held(struct perf_bpf_filter_entry *entry case PBF_LOCK_TYPE: return is_lock_type_in_map(entry->target, mode, pid, cpu_id); + case PBF_LOCK_RCU: + if ((mode & PBF_LOCK_MODE_READ) && + is_rcu_lock_held(curr, (int)entry->target)) + return 1; + if ((mode & PBF_LOCK_MODE_WRITE) && rcu_state_addr && + is_lock_in_map(rcu_state_addr, PBF_LOCK_ADDR, PBF_LOCK_MODE_WRITE, pid, cpu_id)) + return 1; + return 0; + + case PBF_LOCK_RCU_TRACE: + if ((mode & PBF_LOCK_MODE_READ) && is_rcu_trace_lock_held(curr)) + return 1; + return 0; + default: return 0; } @@ -601,11 +617,12 @@ static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 fla if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr)) matched = true; - if (has_mmap_lock || has_siglock) { + if (has_mmap_lock || has_siglock || has_rcu_lock) { __u32 ltype = check_lock_type(lock_addr, flags); if ((has_mmap_lock && ltype == LCD_F_MMAP_LOCK) || - (has_siglock && ltype == LCD_F_SIGHAND_LOCK)) + (has_siglock && ltype == LCD_F_SIGHAND_LOCK) || + (has_rcu_lock && ltype == LCD_F_RCU_LOCK)) matched = true; } @@ -1247,6 +1264,120 @@ int BPF_PROG(fentry_percpu_up_write, void *sem) return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE); } +/* SRCU acquire/release and synchronize hooks */ +SEC("fexit/__srcu_read_lock") +int BPF_PROG(fexit_srcu_read_lock, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock") +int BPF_PROG(fentry_srcu_read_unlock, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fexit/__srcu_read_lock_nmisafe") +int BPF_PROG(fexit_srcu_read_lock_nmisafe, void *ssp, int ret) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/__srcu_read_unlock_nmisafe") +int BPF_PROG(fentry_srcu_read_unlock_nmisafe, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_READ); +} + +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return record_lock_acquired(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return record_lock_released(ctx, LCB_F_SRCU | LCB_F_WRITE); +} + +/* RCU grace-period synchronize / barrier hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return record_lock_acquired_owner(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE, + get_lock_owner_id(LCB_F_RCU | LCB_F_WRITE)); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return record_lock_released_addr(rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(collect_lock_syms) { diff --git a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h index b75ddc6846bb..1fb91a829d54 100644 --- a/tools/perf/util/bpf_skel/vmlinux/vmlinux.h +++ b/tools/perf/util/bpf_skel/vmlinux/vmlinux.h @@ -119,6 +119,8 @@ struct task_struct { struct signal_struct *signal; struct sighand_struct *sighand; struct css_set *cgroups; + int rcu_read_lock_nesting; + int trc_reader_nesting; } __attribute__((preserve_access_index)); struct trace_entry { diff --git a/tools/perf/util/lock-contention.c b/tools/perf/util/lock-contention.c index a1bb78d6e29c..b45b075f3346 100644 --- a/tools/perf/util/lock-contention.c +++ b/tools/perf/util/lock-contention.c @@ -38,6 +38,10 @@ const struct perf_lock_type lock_type_table[] = { { LCB_F_MUTEX | LCB_F_SPIN, "mutex", "mutex" }, /* alias for optimistic spinning only */ { LCB_F_MUTEX | LCB_F_SPIN, "mutex:spin", "mutex-spin" }, + { LCB_F_SRCU | LCB_F_READ, "srcu:R", "srcu" }, + { LCB_F_SRCU | LCB_F_WRITE, "srcu:W", "srcu" }, + { LCB_F_RCU | LCB_F_READ, "rcu:R", "rcu" }, + { LCB_F_RCU | LCB_F_WRITE, "rcu:W", "rcu" }, }; int parse_call_stack(const struct option *opt __maybe_unused, const char *str, diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 68e9bd77e8a9..88b2c11244ad 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -128,6 +128,8 @@ struct thread_stat { #define LCB_F_RT (1U << 3) #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) +#define LCB_F_SRCU (1U << 6) +#define LCB_F_RCU (1U << 7) struct perf_lock_type { unsigned int flags; @@ -140,7 +142,7 @@ struct perf_lock_type { const char *lock_name; }; -extern const struct perf_lock_type lock_type_table[14]; +extern const struct perf_lock_type lock_type_table[18]; struct evlist; struct machine; -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU 2026-10-17 17:14 ` [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers @ 2026-10-17 17:25 ` sashiko-bot 0 siblings, 0 replies; 24+ messages in thread From: sashiko-bot @ 2026-10-17 17:25 UTC (permalink / raw) To: Ian Rogers; +Cc: linux-perf-users > Extend the lock_held(<lock>) BPF sample filter to support RCU, Tasks > Trace RCU, and SRCU locks: > > - 'rcu' (supports both read/shared and write/exclusive modes): > - Read/shared ('lock_held_read(rcu)', 'lock_held(rcu:R)', > 'lock_held(rcu_read_lock)'): directly inspects > current->rcu_read_lock_nesting in perf_sample_filter via CO-RE (on > CONFIG_PREEMPT_RCU kernels), accounting for the baseline > rcu_read_lock() nesting acquired by the perf overflow handler itself > (bpf_overflow_handler() and do_perf_sw_event()) with zero extra > fentry/fexit tracing hooks. > - Write/exclusive ('lock_held_write(rcu)', 'lock_held(rcu:W)'): > attaches fentry/fexit hooks to synchronize_rcu(), > synchronize_rcu_expedited(), synchronize_rcu_tasks(), and > rcu_barrier() to track grace-period and barrier waits. > [ ... ] > > # perf record -a -g -e task-clock --filter 'lock_held(rcu:W)' -- sleep 1 > # perf record -a -g -e cycles --filter 'lock_held(srcu)' -- sleep 1 > > Signed-off-by: Ian Rogers <irogers@google.com> Sashiko has reviewed this patch and found no issues. -- Sashiko AI review · https://sashiko.dev/#/patchset/20261017171437.2066343-1-irogers@google.com?part=4 ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU and SRCU 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (3 preceding siblings ...) 2026-10-17 17:14 ` [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers @ 2026-10-17 17:14 ` Ian Rogers 2026-10-17 17:30 ` sashiko-bot 2026-10-17 17:14 ` [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers 5 siblings, 1 reply; 24+ messages in thread From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu Extend 'perf lock contention --use-bpf' to track RCU and SRCU grace-period and barrier waits: - Attach fentry/fexit programs to synchronize_rcu(), synchronize_rcu_expedited(), synchronize_rcu_tasks(), rcu_barrier(), synchronize_srcu(), synchronize_srcu_expedited(), and srcu_barrier() when available in BTF. - Support filtering by lock type ('-Y rcu', '-Y rcu:W', '-Y srcu', '-Y srcu:W') and by lock name ('-L rcu' or specific srcu_struct symbol names). Example usage: # perf lock con -a -b -Y rcu,srcu -F wait_total,avg_wait -- sleep 1 total wait avg wait type caller 35.12 ms 8.78 ms rcu:W cleanup_net+0x284 7.84 ms 3.92 ms srcu:W __fsnotify_parent+0x114 # perf lock con -a -b -l -L rcu -F wait_total,avg_wait -- sleep 1 total wait avg wait address symbol 35.12 ms 8.78 ms ffffffff83754dc0 rcu (rcu) Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/Documentation/perf-lock.txt | 7 +- tools/perf/tests/shell/lock_contention.sh | 14 ++ tools/perf/util/bpf_lock_contention.c | 167 +++++++++++++-- .../perf/util/bpf_skel/lock_contention.bpf.c | 200 ++++++++++++++---- tools/perf/util/machine.c | 11 + 5 files changed, 348 insertions(+), 51 deletions(-) diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt index c17b3e318169..bfeb98c976e3 100644 --- a/tools/perf/Documentation/perf-lock.txt +++ b/tools/perf/Documentation/perf-lock.txt @@ -188,11 +188,14 @@ CONTENTION OPTIONS Show lock contention only for given lock types (comma separated list). Available values are: semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W, - rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwmem, pcpu-sem, - pcpu-sem:R, pcpu-sem:W, mutex + rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, percpu-rwsem, pcpu-sem, + pcpu-sem:R, pcpu-sem:W, mutex, srcu, srcu:W, rcu, rcu:W Note that RW-variant of locks have :R and :W suffix. Names without the suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W. + For RCU and SRCU with --use-bpf, contention tracks grace-period and + barrier waits (synchronize_rcu*(), synchronize_srcu*(), rcu_barrier(), + srcu_barrier()). -L:: --lock-filter=<value>:: diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh index 5df9a0cd48a3..c6d37c910d4b 100755 --- a/tools/perf/tests/shell/lock_contention.sh +++ b/tools/perf/tests/shell/lock_contention.sh @@ -181,6 +181,13 @@ test_type_filter() err=1 exit fi + + perf lock con -a -b -Y rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_lock_filter() @@ -224,6 +231,13 @@ test_lock_filter() err=1 exit fi + + perf lock con -a -b -L rcu -q -- true > /dev/null 2> ${result} + if [ "$(grep -c -v rcu "${result}")" != "0" ]; then + echo "[Fail] BPF result should not have non-rcu locks:" "$(cat "${result}")" + err=1 + exit + fi } test_stack_filter() diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c index c20bd075664e..fa7df9187b08 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -16,6 +16,7 @@ #include <bpf/bpf.h> #include <bpf/btf.h> #include <inttypes.h> +#include <unistd.h> #include "bpf_skel/lock_contention.skel.h" #include "bpf_skel/lock_data.h" @@ -180,6 +181,129 @@ static void init_numa_data(struct lock_contention *con) free(buf); } +static bool has_bpf_fentry_support(struct btf *btf) +{ + /* + * Verify both BPF_TRACE_FENTRY trampoline support (5.5+) and + * bpf_get_func_arg() helper availability (5.17+) used by + * get_lock_addr() in the SRCU fentry/fexit hooks. + */ + struct bpf_insn insns[] = { + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_2, .imm = 0 }, + { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_3, + .src_reg = BPF_REG_10 }, + { .code = BPF_ALU64 | BPF_ADD | BPF_K, .dst_reg = BPF_REG_3, .imm = -8 }, + { .code = BPF_JMP | BPF_CALL, .imm = BPF_FUNC_get_func_arg }, + { .code = BPF_ALU64 | BPF_MOV | BPF_K, .dst_reg = BPF_REG_0, .imm = 0 }, + { .code = BPF_JMP | BPF_EXIT }, + }; + LIBBPF_OPTS(bpf_prog_load_opts, opts, + .expected_attach_type = BPF_TRACE_FENTRY, + ); + s32 btf_id; + int prog_fd, link_fd; + + if (!btf) + return false; + + btf_id = btf__find_by_name_kind(btf, "synchronize_rcu", BTF_KIND_FUNC); + if (btf_id <= 0) + return false; + + opts.attach_btf_id = btf_id; + prog_fd = bpf_prog_load(BPF_PROG_TYPE_TRACING, NULL, "GPL", + insns, ARRAY_SIZE(insns), &opts); + if (prog_fd < 0) + return false; + + link_fd = bpf_raw_tracepoint_open(NULL, prog_fd); + if (link_fd >= 0) + close(link_fd); + close(prog_fd); + return link_fd >= 0; +} + +static void configure_rcu_contention_progs(struct lock_contention *con, + bool has_rcu_lock) +{ + static const struct { + size_t prog_offset; + const char *func; + bool is_srcu; + } rcu_progs[] = { + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu), + "synchronize_rcu", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_expedited), + "synchronize_rcu_expedited", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_rcu_tasks), + "synchronize_rcu_tasks", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fexit_rcu_barrier), + "rcu_barrier", false }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu), + "synchronize_srcu", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_synchronize_srcu_expedited), + "synchronize_srcu_expedited", true }, + { offsetof(struct lock_contention_bpf, progs.fentry_srcu_barrier), + "srcu_barrier", true }, + { offsetof(struct lock_contention_bpf, progs.fexit_srcu_barrier), + "srcu_barrier", true }, + }; + struct symbol *sym; + struct map *kmap; + bool need_rcu = true, need_srcu = true; + bool fentry_ok; + + sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap); + if (sym) { + skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start); + map__put(kmap); + } else { + skel->rodata->rcu_state_addr = 1ULL; + } + + if (con->filters->nr_types) { + need_rcu = false; + need_srcu = false; + for (int i = 0; i < con->filters->nr_types; i++) { + if (con->filters->types[i] & LCB_F_RCU) + need_rcu = true; + if (con->filters->types[i] & LCB_F_SRCU) + need_srcu = true; + } + } + + if (con->filters->nr_syms || con->filters->nr_addrs || con->filters->nr_slabs) { + if (!has_rcu_lock && !con->filters->nr_addrs) + need_rcu = false; + if (!con->filters->nr_addrs && !con->filters->nr_slabs) + need_srcu = false; + } + + fentry_ok = (need_rcu || need_srcu) && has_bpf_fentry_support(con->btf); + + for (size_t i = 0; i < ARRAY_SIZE(rcu_progs); i++) { + struct bpf_program **progp = (void *)skel + rcu_progs[i].prog_offset; + bool want = rcu_progs[i].is_srcu ? need_srcu : need_rcu; + bool enable = fentry_ok && want && + btf__find_by_name_kind(con->btf, rcu_progs[i].func, + BTF_KIND_FUNC) > 0; + + bpf_program__set_autoload(*progp, enable); + } +} + int lock_contention_prepare(struct lock_contention *con) { int i, fd; @@ -187,6 +311,7 @@ int lock_contention_prepare(struct lock_contention *con) struct evlist *evlist = con->evlist; struct target *target = con->target; bool has_mmap_lock = false; + bool has_rcu_lock = false; /* make sure it loads the kernel map before lookup */ map__load(machine__kernel_map(con->machine)); @@ -200,6 +325,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64)); bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries); bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries); + bpf_map__set_max_entries(skel->maps.tstamp_rcu, con->map_nr_entries); if (con->aggr_mode == LOCK_AGGR_TASK) bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries); @@ -249,6 +375,10 @@ int lock_contention_prepare(struct lock_contention *con) has_mmap_lock = true; continue; } + if (!strcmp(con->filters->syms[i], "rcu")) { + has_rcu_lock = true; + continue; + } sym = machine__find_kernel_symbol_by_name(con->machine, con->filters->syms[i], @@ -269,7 +399,7 @@ int lock_contention_prepare(struct lock_contention *con) addrs[con->filters->nr_addrs++] = map__unmap_ip(kmap, sym->start); con->filters->addrs = addrs; } - naddrs = con->filters->nr_addrs ?: has_mmap_lock; + naddrs = con->filters->nr_addrs ?: (has_mmap_lock || has_rcu_lock); skel->rodata->has_addr = 1; } @@ -305,6 +435,7 @@ int lock_contention_prepare(struct lock_contention *con) skel->rodata->needs_callstack = con->save_callstack; skel->rodata->lock_owner = con->owner; skel->rodata->has_mmap_lock = has_mmap_lock; + skel->rodata->has_rcu_lock = has_rcu_lock; if (con->aggr_mode == LOCK_AGGR_CGROUP || con->filters->nr_cgrps) { if (cgroup_is_v2("perf_event")) @@ -321,6 +452,7 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map__set_max_entries(skel->maps.slab_filter, nslabs); init_numa_data(con); + configure_rcu_contention_progs(con, has_rcu_lock); if (lock_contention_bpf__load(skel) < 0) { pr_err("Failed to load lock-contention BPF skeleton\n"); @@ -503,23 +635,30 @@ static void account_end_timestamp(struct lock_contention *con) enum lock_aggr_mode aggr_mode = con->aggr_mode; struct tstamp_data ts_data, *cpu_data; - /* Iterate per-task tstamp map (key = TID) */ - ts_fd = bpf_map__fd(skel->maps.tstamp); + /* Iterate per-task tstamp maps (key = TID) */ + int ts_fds[] = { + bpf_map__fd(skel->maps.tstamp), + bpf_map__fd(skel->maps.tstamp_rcu), + }; + stat_fd = bpf_map__fd(skel->maps.lock_stat); - prev_key = NULL; - while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { - if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { - int pid = key; + for (size_t i = 0; i < ARRAY_SIZE(ts_fds); i++) { + ts_fd = ts_fds[i]; + prev_key = NULL; + while (!bpf_map_get_next_key(ts_fd, prev_key, &key)) { + if (bpf_map_lookup_elem(ts_fd, &key, &ts_data) == 0) { + int pid = key; - if (aggr_mode == LOCK_AGGR_TASK && con->owner) - pid = ts_data.flags; + if (aggr_mode == LOCK_AGGR_TASK && con->owner) + pid = ts_data.flags; - update_lock_stat(stat_fd, pid, end_ts, aggr_mode, - &ts_data); - } + update_lock_stat(stat_fd, pid, end_ts, aggr_mode, + &ts_data); + } - prev_key = &key; + prev_key = &key; + } } /* Now it'll check per-cpu tstamp map which doesn't have TID. */ @@ -604,6 +743,8 @@ static const char *lock_contention_get_name(struct lock_contention *con, return "mmap_lock"; if (flags & LCD_F_SIGHAND_LOCK) return "siglock"; + if (flags & LCD_F_RCU_LOCK) + return "rcu"; /* global locks with symbols */ sym = machine__find_kernel_symbol(machine, key->lock_addr_or_cgroup, &kmap); diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 0f2118006005..58d45c43150b 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -59,6 +59,18 @@ struct { __uint(max_entries, MAX_ENTRIES); } tstamp SEC(".maps"); +/* + * Separate per-task timestamp map for RCU/SRCU grace-period and barrier waits + * so they do not collide with inner sleeping lock contentions (such as + * rcu_barrier_mutex or srcu_cb_mutex) acquired during the wait. + */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, int); + __type(value, struct tstamp_data); + __uint(max_entries, MAX_ENTRIES); +} tstamp_rcu SEC(".maps"); + /* maintain per-CPU timestamp at the beginning of contention */ struct { __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); @@ -160,6 +172,7 @@ const volatile int has_addr; const volatile int has_cgroup; const volatile int has_slab; const volatile int has_mmap_lock; +const volatile int has_rcu_lock; const volatile int needs_callstack; const volatile int stack_skip; const volatile int lock_owner; @@ -172,6 +185,7 @@ const volatile int aggr_mode; const volatile __u64 contig_page_data_addr; const volatile __u64 node_data_addr; +const volatile __u64 rcu_state_addr; const volatile int nr_nodes; const volatile int sizeof_zone; @@ -218,7 +232,7 @@ static inline __u64 get_current_cgroup_id(void) return BPF_CORE_READ(cgrp, kn, id); } -static inline int can_record(u64 *ctx) +static __always_inline int can_record(__u64 lock, __u32 flags) { bool is_addr_ok = false; @@ -242,7 +256,6 @@ static inline int can_record(u64 *ctx) if (has_type) { __u8 *ok; - __u32 flags = (__u32)ctx[1]; ok = bpf_map_lookup_elem(&type_filter, &flags); if (!ok) @@ -251,10 +264,9 @@ static inline int can_record(u64 *ctx) if (has_addr) { __u8 *ok; - __u64 addr = ctx[0]; - ok = bpf_map_lookup_elem(&addr_filter, &addr); - if (!ok && !has_slab && !has_mmap_lock) + ok = bpf_map_lookup_elem(&addr_filter, &lock); + if (!ok && !has_slab && !has_mmap_lock && !has_rcu_lock) return 0; is_addr_ok = !!ok; @@ -272,26 +284,27 @@ static inline int can_record(u64 *ctx) if (is_addr_ok) return 1; - /* slab and mmap_lock are part of the addr_filter */ + /* slab, mmap_lock, and rcu_lock are part of the addr_filter */ if (has_slab && bpf_get_kmem_cache) { __u8 *ok; - __u64 addr = ctx[0]; long kmem_cache_addr; - kmem_cache_addr = (long)bpf_get_kmem_cache(addr); + kmem_cache_addr = (long)bpf_get_kmem_cache(lock); ok = bpf_map_lookup_elem(&slab_filter, &kmem_cache_addr); if (ok) return 1; - else if (!has_mmap_lock) + else if (!has_mmap_lock && !has_rcu_lock) return 0; } - if (has_mmap_lock) { - __u64 lock = ctx[0]; - __u32 flag = ctx[1]; + if (has_mmap_lock || has_rcu_lock) { + __u32 ltype = check_lock_type(lock, flags); - if (check_lock_type(lock, flag) != LCD_F_MMAP_LOCK) - return 0; + if (has_mmap_lock && ltype == LCD_F_MMAP_LOCK) + return 1; + if (has_rcu_lock && ltype == LCD_F_RCU_LOCK) + return 1; + return 0; } return 1; @@ -351,6 +364,7 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) { __u32 pid; struct tstamp_data *pelem; + void *map; /* Use per-cpu array map for spinlock and rwlock */ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN) { @@ -363,8 +377,9 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) return pelem; } + map = (flags & (LCB_F_RCU | LCB_F_SRCU)) ? (void *)&tstamp_rcu : (void *)&tstamp; pid = bpf_get_current_pid_tgid(); - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); /* Do not update the element for nested locks */ if (pelem && pelem->lock) return NULL; @@ -372,12 +387,12 @@ static inline struct tstamp_data *get_tstamp_elem(__u32 flags) if (pelem == NULL) { struct tstamp_data zero = {}; - if (bpf_map_update_elem(&tstamp, &pid, &zero, BPF_NOEXIST) < 0) { + if (bpf_map_update_elem(map, &pid, &zero, BPF_NOEXIST) < 0) { __sync_fetch_and_add(&task_fail, 1); return NULL; } - pelem = bpf_map_lookup_elem(&tstamp, &pid); + pelem = bpf_map_lookup_elem(map, &pid); if (pelem == NULL) { __sync_fetch_and_add(&task_fail, 1); return NULL; @@ -474,21 +489,20 @@ static inline void update_owner_stat(u32 id, u64 duration, u32 flags) } } -SEC("tp_btf/contention_begin") -int contention_begin(u64 *ctx) +static __always_inline int do_contention_begin(void *ctx, __u64 lock, __u32 flags) { struct tstamp_data *pelem; - if (!enabled || !can_record(ctx)) + if (!enabled || !can_record(lock, flags)) return 0; - pelem = get_tstamp_elem(ctx[1]); + pelem = get_tstamp_elem(flags); if (pelem == NULL) return 0; pelem->timestamp = bpf_ktime_get_ns(); - pelem->lock = (__u64)ctx[0]; - pelem->flags = (__u32)ctx[1]; + pelem->lock = lock; + pelem->flags = flags; if (aggr_mode == LOCK_AGGR_CGROUP) pelem->cgroup_id = get_current_cgroup_id(); @@ -559,8 +573,21 @@ int contention_begin(u64 *ctx) } } skip_owner: - pelem->stack_id = bpf_get_stackid(ctx, &stacks, - BPF_F_FAST_STACK_CMP | stack_skip); + { + int skip = stack_skip; + + /* + * Unlike tp_btf/contention_begin (which fires inside a + * lock slowpath via trace_contention_begin), fentry + * RCU/SRCU programs attach directly to the top-level + * wait API, which is 2 frames shallower. + */ + if (flags & (LCB_F_RCU | LCB_F_SRCU)) + skip = stack_skip > 2 ? stack_skip - 2 : 0; + + pelem->stack_id = bpf_get_stackid(ctx, &stacks, + BPF_F_FAST_STACK_CMP | skip); + } if (pelem->stack_id < 0) __sync_fetch_and_add(&stack_fail, 1); } else if (aggr_mode == LOCK_AGGR_TASK) { @@ -588,8 +615,7 @@ int contention_begin(u64 *ctx) return 0; } -SEC("tp_btf/contention_end") -int contention_end(u64 *ctx) +static __always_inline int do_contention_end(__u64 lock, __s32 ret) { __u32 pid = 0, idx = 0; struct tstamp_data *pelem; @@ -597,7 +623,7 @@ int contention_end(u64 *ctx) struct contention_data *data; __u64 timestamp; __u64 duration; - bool need_delete = false; + void *ts_map = NULL; if (!enabled) return 0; @@ -613,14 +639,19 @@ int contention_end(u64 *ctx) */ pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx); if (pelem && pelem->lock) { - if (pelem->lock != ctx[0]) + if (pelem->lock != lock) return 0; } else { pid = bpf_get_current_pid_tgid(); pelem = bpf_map_lookup_elem(&tstamp, &pid); - if (!pelem || pelem->lock != ctx[0]) - return 0; - need_delete = true; + if (pelem && pelem->lock == lock) { + ts_map = &tstamp; + } else { + pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid); + if (!pelem || pelem->lock != lock) + return 0; + ts_map = &tstamp_rcu; + } } timestamp = bpf_ktime_get_ns(); @@ -648,7 +679,6 @@ int contention_end(u64 *ctx) */ else { u32 i = 0; - s32 ret = (s32)ctx[1]; u64 *buf; otdata->timestamp = timestamp; @@ -713,7 +743,7 @@ int contention_end(u64 *ctx) if (lock_owner) key.pid = pelem->flags; else { - if (!need_delete) + if (!ts_map) pid = bpf_get_current_pid_tgid(); key.pid = pid; } @@ -795,11 +825,109 @@ int contention_end(u64 *ctx) check_lock_delay(pelem->lock); pelem->lock = 0; - if (need_delete) - bpf_map_delete_elem(&tstamp, &pid); + if (ts_map) + bpf_map_delete_elem(ts_map, &pid); return 0; } +SEC("tp_btf/contention_begin") +int contention_begin(u64 *ctx) +{ + return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]); +} + +SEC("tp_btf/contention_end") +int contention_end(u64 *ctx) +{ + return do_contention_end((__u64)ctx[0], (__s32)ctx[1]); +} + +/* RCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_rcu") +int BPF_PROG(fentry_synchronize_rcu) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu") +int BPF_PROG(fexit_synchronize_rcu) +{ + return do_contention_end(rcu_state_addr, 0); +} + +SEC("fentry/synchronize_rcu_expedited") +int BPF_PROG(fentry_synchronize_rcu_expedited) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_expedited") +int BPF_PROG(fexit_synchronize_rcu_expedited) +{ + return do_contention_end(rcu_state_addr, 0); +} + +SEC("fentry/synchronize_rcu_tasks") +int BPF_PROG(fentry_synchronize_rcu_tasks) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_rcu_tasks") +int BPF_PROG(fexit_synchronize_rcu_tasks) +{ + return do_contention_end(rcu_state_addr, 0); +} + +SEC("fentry/rcu_barrier") +int BPF_PROG(fentry_rcu_barrier) +{ + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); +} + +SEC("fexit/rcu_barrier") +int BPF_PROG(fexit_rcu_barrier) +{ + return do_contention_end(rcu_state_addr, 0); +} + +/* SRCU grace-period synchronize / barrier contention hooks */ +SEC("fentry/synchronize_srcu") +int BPF_PROG(fentry_synchronize_srcu, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu") +int BPF_PROG(fexit_synchronize_srcu, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0); +} + +SEC("fentry/synchronize_srcu_expedited") +int BPF_PROG(fentry_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/synchronize_srcu_expedited") +int BPF_PROG(fexit_synchronize_srcu_expedited, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0); +} + +SEC("fentry/srcu_barrier") +int BPF_PROG(fentry_srcu_barrier, void *ssp) +{ + return do_contention_begin(ctx, get_lock_addr(ctx), LCB_F_SRCU | LCB_F_WRITE); +} + +SEC("fexit/srcu_barrier") +int BPF_PROG(fexit_srcu_barrier, void *ssp) +{ + return do_contention_end(get_lock_addr(ctx), 0); +} + SEC("raw_tp/bpf_test_finish") int BPF_PROG(end_timestamp) { diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c index 1d9d3bf57720..0a0f223b3335 100644 --- a/tools/perf/util/machine.c +++ b/tools/perf/util/machine.c @@ -3379,6 +3379,17 @@ bool machine__is_lock_function(struct machine *machine, u64 addr) return true; } + { + struct symbol *sym = machine__find_kernel_symbol(machine, addr, NULL); + + if (sym && (!strncmp(sym->name, "synchronize_rcu", 15) || + !strncmp(sym->name, "synchronize_srcu", 16) || + !strcmp(sym->name, "__synchronize_srcu") || + !strcmp(sym->name, "rcu_barrier") || + !strcmp(sym->name, "srcu_barrier"))) + return true; + } + return false; } -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU and SRCU 2026-10-17 17:14 ` [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers @ 2026-10-17 17:30 ` sashiko-bot 0 siblings, 0 replies; 24+ messages in thread From: sashiko-bot @ 2026-10-17 17:30 UTC (permalink / raw) To: Ian Rogers; +Cc: linux-perf-users Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf lock: rcu_state_addr collides with rcu_state.node[0].lock in BPF -- commit 66aefea7c5ca7c7c5ee421770f454d75f5c24b84 Author: Ian Rogers <irogers@google.com> perf lock contention: Extend BPF lock contention to RCU and SRCU This commit extends 'perf lock contention --use-bpf' to track RCU and SRCU grace-period and barrier waits using fentry/fexit BPF programs and adds support for filtering by RCU and SRCU lock types and names. > diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c > index c20bd075664e1..fa7df9187b08b 100644 > --- a/tools/perf/util/bpf_lock_contention.c > +++ b/tools/perf/util/bpf_lock_contention.c [ ... ] > @@ -180,6 +181,129 @@ static void init_numa_data(struct lock_contention *con) [ ... ] > +static void configure_rcu_contention_progs(struct lock_contention *con, > + bool has_rcu_lock) > +{ [ ... ] > + sym = machine__find_kernel_symbol_by_name(con->machine, "rcu_state", &kmap); > + if (sym) { > + skel->rodata->rcu_state_addr = map__unmap_ip(kmap, sym->start); > + map__put(kmap); > + } else { > + skel->rodata->rcu_state_addr = 1ULL; > + } [ ... ] > diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c > index 0f2118006005a..58d45c43150b5 100644 > --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c > +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c [ ... ] > @@ -613,14 +639,19 @@ int contention_end(u64 *ctx) > */ > pelem = bpf_map_lookup_elem(&tstamp_cpu, &idx); > if (pelem && pelem->lock) { > - if (pelem->lock != ctx[0]) > + if (pelem->lock != lock) > return 0; > } else { > pid = bpf_get_current_pid_tgid(); > pelem = bpf_map_lookup_elem(&tstamp, &pid); > - if (!pelem || pelem->lock != ctx[0]) > - return 0; > - need_delete = true; > + if (pelem && pelem->lock == lock) { > + ts_map = &tstamp; > + } else { > + pelem = bpf_map_lookup_elem(&tstamp_rcu, &pid); > + if (!pelem || pelem->lock != lock) > + return 0; > + ts_map = &tstamp_rcu; > + } > } [Severity: Medium] Can setting rcu_state_addr to the start address of rcu_state and sharing do_contention_end() between tp_btf/contention_end and the RCU/SRCU fexit hooks cause lock address collisions and corrupted wait tracking? In kernel/rcu/tree.h, struct rcu_state begins at offset 0 with node[NUM_RCU_NODES], and struct rcu_node begins at offset 0 with its raw_spinlock_t lock: kernel/rcu/tree.h: struct rcu_node { raw_spinlock_t __private lock; ... }; struct rcu_state { struct rcu_node node[NUM_RCU_NODES]; ... }; Because of this layout, &rcu_state.node[0].lock has the exact same address as rcu_state (rcu_state_addr). Together with do_contention_end() not distinguishing whether its caller is tp_btf/contention_end or an RCU/SRCU fexit program, three issues can occur: 1. When filtering for RCU (for example with '-Y rcu', '-Y rcu:W', or '-L rcu'), a task executing synchronize_rcu(), synchronize_rcu_expedited(), or rcu_barrier() records an active entry in tstamp_rcu[pid] with pelem->lock == rcu_state_addr. If that task contends on &rcu_state.node[0].lock during grace-period initialization or waiting, tp_btf/contention_begin(lock = rcu_state_addr, flags = LCB_F_SPIN) is rejected by can_record(), leaving tstamp_cpu empty. When the spinlock is acquired and tp_btf/contention_end(lock = rcu_state_addr) fires, do_contention_end() falls through tstamp_cpu and tstamp into tstamp_rcu[pid], matches pelem->lock == rcu_state_addr, records the short spinlock wait duration as the RCU wait, and deletes tstamp_rcu[pid] before the fexit hook runs. 2. If an RCU or SRCU fexit program runs on a CPU where tstamp_cpu happens to have an active spinlock entry (pelem->lock != 0 and pelem->lock != lock), do_contention_end() returns 0 at the tstamp_cpu check without looking up or deleting tstamp_rcu[pid]. That leaves a stale entry in tstamp_rcu[pid] that blocks subsequent RCU/SRCU waits for that task in get_tstamp_elem() and charges end_ts - pelem->timestamp in account_end_timestamp(). 3. In LOCK_AGGR_ADDR mode ('perf lock con -a -b -l'), both do_contention_end() and lock_contention_read() key lock_stat entries by lock address: tools/perf/util/bpf_skel/lock_contention.bpf.c:do_contention_end() case LOCK_AGGR_ADDR: key.lock_addr_or_cgroup = pelem->lock; Because &rcu_state.node[0].lock and rcu_state_addr share the same address, root rcu_node spinlock contentions (LCB_F_SPIN) and RCU grace-period waits (LCB_F_RCU | LCB_F_WRITE) merge into a single lock_stat entry under whichever lock type was recorded first. Would it make sense to pass a distinct synthetic address or offset for RCU waits (or disambiguate the lock key), and pass an explicit flag to do_contention_end() so that RCU/SRCU fexit hooks only inspect tstamp_rcu while tp_btf/contention_end only inspects tstamp_cpu and tstamp? [ ... ] > @@ -795,11 +825,109 @@ int contention_end(u64 *ctx) > check_lock_delay(pelem->lock); > > pelem->lock = 0; > - if (need_delete) > - bpf_map_delete_elem(&tstamp, &pid); > + if (ts_map) > + bpf_map_delete_elem(ts_map, &pid); > return 0; > } > > +SEC("tp_btf/contention_begin") > +int contention_begin(u64 *ctx) > +{ > + return do_contention_begin(ctx, (__u64)ctx[0], (__u32)ctx[1]); > +} > + > +SEC("tp_btf/contention_end") > +int contention_end(u64 *ctx) > +{ > + return do_contention_end((__u64)ctx[0], (__s32)ctx[1]); > +} > + > +/* RCU grace-period synchronize / barrier contention hooks */ > +SEC("fentry/synchronize_rcu") > +int BPF_PROG(fentry_synchronize_rcu) > +{ > + return do_contention_begin(ctx, rcu_state_addr, LCB_F_RCU | LCB_F_WRITE); > +} > + > +SEC("fexit/synchronize_rcu") > +int BPF_PROG(fexit_synchronize_rcu) > +{ > + return do_contention_end(rcu_state_addr, 0); > +} [ ... ] -- Sashiko AI review · https://sashiko.dev/#/patchset/20261017171437.2066343-1-irogers@google.com?part=5 ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers ` (4 preceding siblings ...) 2026-10-17 17:14 ` [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers @ 2026-10-17 17:14 ` Ian Rogers 2026-10-17 17:17 ` sashiko-bot 5 siblings, 1 reply; 24+ messages in thread From: Ian Rogers @ 2026-10-17 17:14 UTC (permalink / raw) To: irogers, acme, namhyung Cc: adrian.hunter, james.clark, jolsa, linux-kernel, linux-perf-users, mingo, peterz, rcu In report_lock_contention_begin_event(), lock_stat_findnew() was called before checking filters.nr_types (-Y), filters.nr_addrs (-L), and match_callstack_filter() (-S). When a filtered-out lock event (such as a mutex contention when running with -Y spinlock) shared the same aggregation key as a subsequent matching event (for example, contention on the mutex's internal wait_lock spinlock from the same caller), the first event created the lock_stat entry with the filtered-out lock's flags and the second event reused that entry, causing non-spinlock entries to appear in -Y spinlock output. Check the type, address, and callstack filters before looking up or creating the lock_stat entry. Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option") Assisted-by: Antigravity:gemini-3.1-pro Signed-off-by: Ian Rogers <irogers@google.com> --- tools/perf/builtin-lock.c | 61 +++++++++++++++++++++------------------ 1 file changed, 33 insertions(+), 28 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index feaf1cfd08cf..a031709a2cc2 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -975,6 +975,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) struct machine *machine = &session->machines.host; struct map *kmap; struct symbol *sym; + u64 *callstack = NULL; ret = get_key_by_aggr_mode(&key, addr, sample); if (ret < 0) @@ -1010,33 +1011,6 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) } } - ls = lock_stat_find(key); - if (!ls) { - char buf[128]; - const char *name = ""; - - switch (aggr_mode) { - case LOCK_AGGR_ADDR: - sym = machine__find_kernel_symbol(machine, key, &kmap); - if (sym) - name = sym->name; - break; - case LOCK_AGGR_CALLER: - name = buf; - if (lock_contention_caller(sample, buf, sizeof(buf)) < 0) - name = "Unknown"; - break; - case LOCK_AGGR_CGROUP: - case LOCK_AGGR_TASK: - default: - break; - } - - ls = lock_stat_findnew(key, name, flags); - if (!ls) - return -ENOMEM; - } - if (filters.nr_types) { bool found = false; @@ -1066,7 +1040,7 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) } if (needs_callstack()) { - u64 *callstack = get_callstack(sample, max_stack_depth); + callstack = get_callstack(sample, max_stack_depth); if (callstack == NULL) return 0; @@ -1074,7 +1048,38 @@ static int report_lock_contention_begin_event(struct perf_sample *sample) free(callstack); return 0; } + } + + ls = lock_stat_find(key); + if (!ls) { + char buf[128]; + const char *name = ""; + + switch (aggr_mode) { + case LOCK_AGGR_ADDR: + sym = machine__find_kernel_symbol(machine, key, &kmap); + if (sym) + name = sym->name; + break; + case LOCK_AGGR_CALLER: + name = buf; + if (lock_contention_caller(sample, buf, sizeof(buf)) < 0) + name = "Unknown"; + break; + case LOCK_AGGR_CGROUP: + case LOCK_AGGR_TASK: + default: + break; + } + + ls = lock_stat_findnew(key, name, flags); + if (!ls) { + free(callstack); + return -ENOMEM; + } + } + if (callstack) { if (ls->callstack == NULL) ls->callstack = callstack; else -- 2.56.0.212.g129ede20d9-goog ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat 2026-10-17 17:14 ` [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers @ 2026-10-17 17:17 ` sashiko-bot 0 siblings, 0 replies; 24+ messages in thread From: sashiko-bot @ 2026-10-17 17:17 UTC (permalink / raw) To: Ian Rogers; +Cc: linux-perf-users > In report_lock_contention_begin_event(), lock_stat_findnew() was > called before checking filters.nr_types (-Y), filters.nr_addrs (-L), > and match_callstack_filter() (-S). > > When a filtered-out lock event (such as a mutex contention when > running with -Y spinlock) shared the same aggregation key as a > subsequent matching event (for example, contention on the mutex's > internal wait_lock spinlock from the same caller), the first event > created the lock_stat entry with the filtered-out lock's flags and the > second event reused that entry, causing non-spinlock entries to appear > in -Y spinlock output. > > Check the type, address, and callstack filters before looking up or > creating the lock_stat entry. > > Fixes: b4a7eff93c39 ("perf lock contention: Add -Y/--type-filter option") > Assisted-by: Antigravity:gemini-3.1-pro > Signed-off-by: Ian Rogers <irogers@google.com> Sashiko has reviewed this patch and found no issues. -- Sashiko AI review · https://sashiko.dev/#/patchset/20261017171437.2066343-1-irogers@google.com?part=6 ^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2026-10-17 17:35 UTC | newest] Thread overview: 24+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers 2026-10-17 5:52 ` sashiko-bot 2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers 2026-10-17 6:09 ` sashiko-bot 2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers 2026-10-17 5:49 ` sashiko-bot 2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers 2026-10-17 5:56 ` sashiko-bot 2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers 2026-10-17 6:08 ` sashiko-bot 2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers 2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers 2026-10-17 17:31 ` sashiko-bot 2026-10-17 17:14 ` [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers 2026-10-17 17:35 ` sashiko-bot 2026-10-17 17:14 ` [PATCH v2 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers 2026-10-17 17:17 ` sashiko-bot 2026-10-17 17:14 ` [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers 2026-10-17 17:25 ` sashiko-bot 2026-10-17 17:14 ` [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers 2026-10-17 17:30 ` sashiko-bot 2026-10-17 17:14 ` [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers 2026-10-17 17:17 ` sashiko-bot
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox