From: Ian Rogers <irogers@google.com>
To: irogers@google.com, acme@kernel.org, namhyung@kernel.org
Cc: adrian.hunter@intel.com, ddhariwal@google.com,
eranian@google.com, james.clark@linaro.org, jolsa@kernel.org,
linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org,
mingo@redhat.com, peterz@infradead.org, rcu@vger.kernel.org
Subject: [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types
Date: Thu, 22 Oct 2026 13:29:58 -0700 [thread overview]
Message-ID: <20261022203003.1779152-7-irogers@google.com> (raw)
In-Reply-To: <20261022203003.1779152-1-irogers@google.com>
Extend the lock_held() BPF sample filter beyond global mutexes to
support:
- Other global/static kernel lock symbols (rwsem, spinlock, rwlock,
rt-mutex, percpu-rwsem, semaphore) and hexadecimal kernel lock
addresses.
- Lock types and flags: mutex, spinlock, rwsem, rwsem:R, rwsem:W,
rwlock, rwlock:R, rwlock:W, rt-mutex, percpu-rwsem, pcpu-sem:W,
semaphore.
For rt-mutexes and write-locked rwsems, perf_sample_filter inspects the
lock's owner field directly in kernel memory. For locks and modes without
an authoritative per-task owner pointer (spinlocks, rwlocks, semaphores,
reader-locked rwsems, percpu-rwsems, and lock type filters), attach BPF
fentry/fexit hooks on kernel lock acquire and release functions to track
held locks in the held_locks BPF LRU hash map.
Note that counting semaphores (down*() / up()) do not enforce same-task
ownership in the kernel: when a semaphore is used as a cross-task
completion/signaling mechanism, fexit/down and fentry/up will not pair
on the same task ID. Semaphore hooks are therefore only enabled when
'semaphore' or a '*_sem' symbol (such as console_sem) is explicitly
specified, and held_locks uses BPF_MAP_TYPE_LRU_HASH so any unpaired
cross-task entries are automatically evicted.
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/Documentation/perf-record.txt | 53 +-
tools/perf/util/bpf-filter.c | 596 ++++++++++++-
tools/perf/util/bpf_skel/sample-filter.h | 17 +-
tools/perf/util/bpf_skel/sample_filter.bpf.c | 851 ++++++++++++++++++-
4 files changed, 1465 insertions(+), 52 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 445cc98a6c33..9fb5d15de135 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -216,15 +216,50 @@ OPTIONS
The 'lock_held(<lock>)' predicate (or '!lock_held(<lock>)') filters
samples based on whether the specified kernel lock is currently held
- by the kernel thread. To differentiate between shared/read holding
- and exclusive/write holding, use 'lock_held_read(<lock>)' (alias
- 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)' (alias
- 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to <lock>.
- <lock> can be a kernel mutex symbol name (e.g.
- 'lock_held(cgroup_mutex)'). When filtering on a mutex symbol
- ('*_mutex'), 'perf_sample_filter' inspects the lock's 'owner' field
- directly in kernel memory on each sample without attaching any BPF
- fentry/fexit hooks.
+ by the kernel thread or CPU. To differentiate between shared/read
+ holding and exclusive/write holding, use 'lock_held_read(<lock>)'
+ (alias 'lock_held_shared(<lock>)') or 'lock_held_write(<lock>)'
+ (alias 'lock_held_exclusive(<lock>)'), or append ':R' or ':W' to
+ <lock> (e.g. 'lock_held(rwsem:R)' or 'lock_held(rwsem:W)').
+ <lock> can be a kernel lock symbol name (e.g.
+ 'lock_held(cgroup_mutex)'), a hexadecimal lock address, or a lock type
+ ('mutex', 'spinlock', 'rwsem', 'rwsem:R', 'rwsem:W', 'rwlock',
+ 'rwlock:R', 'rwlock:W', 'rt-mutex', 'percpu-rwsem', 'pcpu-sem:W',
+ 'semaphore'). Both read/shared and write/exclusive tracking are
+ supported for 'rwsem' and 'rwlock'; 'mutex', 'spinlock', 'rt-mutex',
+ 'semaphore', and 'percpu-rwsem' ('percpu_down_write()') support
+ write/exclusive tracking only. Note that 'semaphore' tracking
+ ('down*()' / 'up()') assumes binary mutex-like usage where the
+ acquiring task also releases the semaphore (such as 'console_sem'),
+ rather than cross-task completion signaling.
+
+ For example, 'lock_held()' can be combined with memory allocation or
+ reclaim events to detect allocations or direct memory reclaim occurring
+ while a kernel lock is held (which can stall lock waiters for extended
+ periods):
+
+ # perf record -a -g -e vmscan:mm_vmscan_direct_reclaim_begin \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+ # perf record -a -g -e kmem:mm_page_alloc \
+ --filter 'gfp_flags & 0x400' \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+ # perf record -a -g -e major-faults \
+ --filter 'lock_held(mutex) || lock_held(rwsem)'
+
+ When filtering on a specific mutex symbol ('*_mutex') or rt-mutex
+ symbol ('*_rt_mutex'), 'perf_sample_filter' inspects the lock's
+ 'owner' field directly in kernel memory on each sample without attaching
+ any BPF fentry/fexit hooks. For locks that do not maintain an
+ authoritative per-task owner pointer (such as spinlocks, rwlocks,
+ semaphores, reader-locked rwsems, and lock type filters), it attaches
+ BPF fentry/fexit programs to kernel lock acquire and release functions
+ to track held locks in a BPF map. These BPF hooks run on every kernel
+ lock acquisition and release for the attached lock family, even when no
+ perf sample is being taken. To minimize this overhead, only the hooks
+ for the relevant lock family and mode are attached when known, whereas
+ raw hex addresses or symbols without a recognizable lock-type suffix
+ attach hooks across all lock families (including high-frequency
+ spinlocks).
The <operator> can be one of:
==, !=, >, >=, <, <=, &
diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c
index 639db6c03137..8797f67698f7 100644
--- a/tools/perf/util/bpf-filter.c
+++ b/tools/perf/util/bpf-filter.c
@@ -52,6 +52,7 @@
#include <sys/stat.h>
#include <bpf/bpf.h>
+#include <bpf/btf.h>
#include <linux/err.h>
#include <linux/kernel.h>
#include <linux/list.h>
@@ -65,6 +66,7 @@
#include "util/cap.h"
#include "util/debug.h"
#include "util/evsel.h"
+#include "util/lock-contention.h"
#include "util/symbol_conf.h"
#include "util/target.h"
#include "util/thread_map.h"
@@ -74,6 +76,7 @@
#include <util/bpf-filter-flex.h>
#include <util/bpf-filter-bison.h>
+#include "bpf_skel/lock_data.h"
#include "bpf_skel/sample-filter.h"
#include "bpf_skel/sample_filter.skel.h"
@@ -178,6 +181,153 @@ static int check_sample_flags(struct evsel *evsel, struct perf_bpf_filter_expr *
return -1;
}
+/*
+ * Program groups for `lock_held(...)` BPF tracing hooks.
+ *
+ * When a `lock_held(...)` filter is specified, only the BPF tracing program
+ * groups required for the requested lock(s) are autoloaded and attached.
+ * If no `lock_held(...)` filter is present, none of these programs are loaded.
+ */
+enum lock_prog_group {
+ LOCK_PROG_MUTEX = 1U << 0,
+ LOCK_PROG_RWSEM_READ = 1U << 1,
+ LOCK_PROG_RWSEM_WRITE = 1U << 2,
+ LOCK_PROG_RWSEM = LOCK_PROG_RWSEM_READ | LOCK_PROG_RWSEM_WRITE,
+ LOCK_PROG_SPIN = 1U << 3,
+ LOCK_PROG_RWLOCK_READ = 1U << 4,
+ LOCK_PROG_RWLOCK_WRITE = 1U << 5,
+ LOCK_PROG_RWLOCK = LOCK_PROG_RWLOCK_READ | LOCK_PROG_RWLOCK_WRITE,
+ LOCK_PROG_RTMUTEX = 1U << 6,
+ LOCK_PROG_SEM = 1U << 7,
+ LOCK_PROG_PERCPU_RWSEM = 1U << 8,
+ LOCK_PROG_ALL = 0x1ffU & ~LOCK_PROG_SEM,
+};
+
+static unsigned int lock_flags_to_prog_group(unsigned int flags)
+{
+ if (flags & LCB_F_MUTEX)
+ return LOCK_PROG_MUTEX;
+ if (flags & LCB_F_RT)
+ return LOCK_PROG_RTMUTEX;
+ if (flags & LCB_F_PERCPU)
+ return LOCK_PROG_PERCPU_RWSEM;
+ if (flags == LCB_F_SPIN)
+ return LOCK_PROG_SPIN;
+ if (flags & LCB_F_SPIN)
+ return (flags & LCB_F_READ) ? LOCK_PROG_RWLOCK_READ : LOCK_PROG_RWLOCK_WRITE;
+ if (flags & LCB_F_READ)
+ return LOCK_PROG_RWSEM_READ;
+ if (flags & LCB_F_WRITE)
+ return LOCK_PROG_RWSEM_WRITE;
+ return LOCK_PROG_SEM;
+}
+
+struct lock_prog_desc {
+ size_t prog_offset;
+ const char *btf_func;
+ unsigned int group;
+};
+
+#define LOCK_FPROG(field, func, grp) \
+ { offsetof(struct sample_filter_bpf, progs.field), func, grp }
+
+static const struct lock_prog_desc lock_progs[] = {
+ /* Mutex */
+ LOCK_FPROG(fexit_mutex_lock, "mutex_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_nested, "mutex_lock_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_lock_nest_lock, "_mutex_lock_nest_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io, "mutex_lock_io", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_io_nested, "mutex_lock_io_nested", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible, "mutex_lock_interruptible", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_interruptible_nested, "mutex_lock_interruptible_nested",
+ LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_lock_killable, "mutex_lock_killable", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_lock_killable, "_mutex_lock_killable", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit_mutex_trylock, "mutex_trylock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fexit__mutex_trylock_nest_lock, "_mutex_trylock_nest_lock", LOCK_PROG_MUTEX),
+ LOCK_FPROG(fentry_mutex_unlock, "mutex_unlock", LOCK_PROG_MUTEX),
+ /* RW semaphore */
+ LOCK_FPROG(fexit_down_read, "down_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_nested, "down_read_nested", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_interruptible, "down_read_interruptible", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable, "down_read_killable", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_killable_nested, "down_read_killable_nested",
+ LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_read_trylock, "down_read_trylock", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fexit_down_write, "down_write", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_nested, "down_write_nested", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit__down_write_nest_lock, "_down_write_nest_lock", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable, "down_write_killable", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_killable_nested, "down_write_killable_nested",
+ LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_down_write_trylock, "down_write_trylock", LOCK_PROG_RWSEM_WRITE),
+ LOCK_FPROG(fexit_downgrade_write, "downgrade_write", LOCK_PROG_RWSEM),
+ LOCK_FPROG(fentry_up_read, "up_read", LOCK_PROG_RWSEM_READ),
+ LOCK_FPROG(fentry_up_write, "up_write", LOCK_PROG_RWSEM_WRITE),
+ /* Spinlock */
+ LOCK_FPROG(fexit_raw_spin_lock, "_raw_spin_lock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_nested, "_raw_spin_lock_nested", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_nest_lock, "_raw_spin_lock_nest_lock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq, "_raw_spin_lock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irq_disable, "_raw_spin_lock_irq_disable", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave, "_raw_spin_lock_irqsave", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_irqsave_nested, "_raw_spin_lock_irqsave_nested",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_lock_bh, "_raw_spin_lock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock, "_raw_spin_trylock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fexit_raw_spin_trylock_bh, "_raw_spin_trylock_bh", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock, "_raw_spin_unlock", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq, "_raw_spin_unlock_irq", LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irq_enable, "_raw_spin_unlock_irq_enable",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_irqrestore, "_raw_spin_unlock_irqrestore",
+ LOCK_PROG_SPIN),
+ LOCK_FPROG(fentry_raw_spin_unlock_bh, "_raw_spin_unlock_bh", LOCK_PROG_SPIN),
+ /* RWlock */
+ LOCK_FPROG(fexit_raw_read_lock, "_raw_read_lock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irq, "_raw_read_lock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_irqsave, "_raw_read_lock_irqsave", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_lock_bh, "_raw_read_lock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_read_trylock, "_raw_read_trylock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock, "_raw_read_unlock", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irq, "_raw_read_unlock_irq", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_irqrestore, "_raw_read_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fentry_raw_read_unlock_bh, "_raw_read_unlock_bh", LOCK_PROG_RWLOCK_READ),
+ LOCK_FPROG(fexit_raw_write_lock, "_raw_write_lock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_nested, "_raw_write_lock_nested", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irq, "_raw_write_lock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_irqsave, "_raw_write_lock_irqsave", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_lock_bh, "_raw_write_lock_bh", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fexit_raw_write_trylock, "_raw_write_trylock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock, "_raw_write_unlock", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irq, "_raw_write_unlock_irq", LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_irqrestore, "_raw_write_unlock_irqrestore",
+ LOCK_PROG_RWLOCK_WRITE),
+ LOCK_FPROG(fentry_raw_write_unlock_bh, "_raw_write_unlock_bh", LOCK_PROG_RWLOCK_WRITE),
+ /* RT-Mutex */
+ LOCK_FPROG(fexit_rt_mutex_lock, "rt_mutex_lock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_nested, "rt_mutex_lock_nested", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit__rt_mutex_lock_nest_lock, "_rt_mutex_lock_nest_lock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_interruptible, "rt_mutex_lock_interruptible",
+ LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_lock_killable, "rt_mutex_lock_killable", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fexit_rt_mutex_trylock, "rt_mutex_trylock", LOCK_PROG_RTMUTEX),
+ LOCK_FPROG(fentry_rt_mutex_unlock, "rt_mutex_unlock", LOCK_PROG_RTMUTEX),
+ /* Semaphore */
+ LOCK_FPROG(fexit_down, "down", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_interruptible, "down_interruptible", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_killable, "down_killable", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_timeout, "down_timeout", LOCK_PROG_SEM),
+ LOCK_FPROG(fexit_down_trylock, "down_trylock", LOCK_PROG_SEM),
+ LOCK_FPROG(fentry_up, "up", LOCK_PROG_SEM),
+ /* Percpu-RWsem */
+ LOCK_FPROG(fexit_percpu_down_write, "percpu_down_write", LOCK_PROG_PERCPU_RWSEM),
+ LOCK_FPROG(fentry_percpu_up_write, "percpu_up_write", LOCK_PROG_PERCPU_RWSEM),
+};
+
+#undef LOCK_FPROG
+
static int split_lock_name_mode(const char *name, int expr_part,
char *base_buf, size_t buf_sz,
bool *has_amp, unsigned int *mode_out)
@@ -224,6 +374,72 @@ static int split_lock_name_mode(const char *name, int expr_part,
return 0;
}
+/*
+ * Parse a lock type name or flags name (e.g. "mutex", "rwsem", "rwsem:R",
+ * "spinlock", "percpu-rwsem").
+ *
+ * A lock type name may correspond to up to two flag combinations (e.g.
+ * "rwsem" matches both `LCB_F_READ` and `LCB_F_WRITE`). These are packed
+ * into `*encoded_pair` as `flag1 | (flag2 << 8)` (with 0xff if `flag2` is
+ * unused) for O(1) lookup in BPF, and accumulated as a bitmask in `*type_mask`.
+ */
+static bool parse_lock_type_name(const char *name, unsigned int mode,
+ struct sample_filter_bpf *skel,
+ u64 *encoded_pair, unsigned int *prog_groups)
+{
+ bool found = false;
+ u32 flag1 = 0xff, flag2 = 0xff;
+ unsigned int groups = 0;
+ const char *lookup = !strcmp(name, "pcpu-sem") ? "percpu-rwsem" : name;
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_type_table); i++) {
+ u32 f = lock_type_table[i].flags;
+ bool is_read = (f & LCB_F_READ) != 0;
+
+ if (strcmp(lock_type_table[i].lock_name, lookup) &&
+ strcmp(lock_type_table[i].flags_name, name))
+ continue;
+
+ /*
+ * percpu_down_read() / percpu_up_read() fast paths are inlined
+ * per-CPU counter increments without function entry points, so
+ * only writer holding (percpu_down_write / percpu_up_write) can
+ * be tracked for percpu-rwsem. rwlock-rt is also not hooked.
+ */
+ if ((f & LCB_F_PERCPU) && is_read)
+ continue;
+ if ((f & LCB_F_RT) && (f & (LCB_F_READ | LCB_F_WRITE)))
+ continue;
+ if ((f & LCB_F_MUTEX) && (f & LCB_F_SPIN))
+ continue;
+
+ if (is_read && !(mode & PBF_LOCK_MODE_READ))
+ continue;
+ if (!is_read && !(mode & PBF_LOCK_MODE_WRITE))
+ continue;
+
+ if (!found)
+ flag1 = f;
+ else if (f != flag1)
+ flag2 = f;
+ if (skel && f < LCB_F_TYPE_MAX) {
+ skel->bss->has_lock_type = 1;
+ skel->bss->lock_type_filter[f] = 1;
+ }
+ groups |= lock_flags_to_prog_group(f);
+ found = true;
+ }
+
+ if (!found)
+ return false;
+
+ if (encoded_pair)
+ *encoded_pair = flag1 | ((u64)flag2 << 8);
+ if (prog_groups)
+ *prog_groups |= groups;
+ return true;
+}
+
struct lock_sym_cb_args {
const char *name;
u64 addr;
@@ -301,26 +517,225 @@ static int find_kernel_lock_sym(const char *name, u64 *addr)
return 0;
}
+static bool parse_lock_hex_addr(const char *str, u64 *addr)
+{
+ char *end;
+ unsigned long long val;
+
+ if (!str || !*str)
+ return false;
+
+ errno = 0;
+ val = strtoull(str, &end, 16);
+ if (*end != '\0' || errno != 0 || val == 0)
+ return false;
+
+ *addr = val;
+ return true;
+}
+
+static bool btf_has_func(struct btf *btf, const char *name)
+{
+ return btf && name && btf__find_by_name_kind(btf, name, BTF_KIND_FUNC) > 0;
+}
+
+static unsigned int supported_lock_prog_groups(struct btf *btf)
+{
+ unsigned int supported = 0;
+
+ if (!btf)
+ return 0;
+
+ if ((btf_has_func(btf, "mutex_lock") || btf_has_func(btf, "mutex_lock_nested")) &&
+ btf_has_func(btf, "mutex_unlock"))
+ supported |= LOCK_PROG_MUTEX;
+
+ if ((btf_has_func(btf, "down_read") || btf_has_func(btf, "down_read_nested")) &&
+ btf_has_func(btf, "up_read"))
+ supported |= LOCK_PROG_RWSEM_READ;
+
+ if ((btf_has_func(btf, "down_write") || btf_has_func(btf, "down_write_nested")) &&
+ btf_has_func(btf, "up_write"))
+ supported |= LOCK_PROG_RWSEM_WRITE;
+
+ /*
+ * Require both acquire and all unlock entry points in BTF so we never
+ * attach spinlock/rwlock acquire hooks when unlock functions are
+ * inlined (e.g. !CONFIG_UNINLINE_SPIN_UNLOCK or CONFIG_INLINE_*_UNLOCK).
+ */
+ if ((btf_has_func(btf, "_raw_spin_lock") || btf_has_func(btf, "_raw_spin_lock_nested")) &&
+ btf_has_func(btf, "_raw_spin_unlock") &&
+ btf_has_func(btf, "_raw_spin_unlock_irq") &&
+ btf_has_func(btf, "_raw_spin_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_spin_unlock_bh"))
+ supported |= LOCK_PROG_SPIN;
+
+ if (btf_has_func(btf, "_raw_read_lock") &&
+ btf_has_func(btf, "_raw_read_unlock") &&
+ btf_has_func(btf, "_raw_read_unlock_irq") &&
+ btf_has_func(btf, "_raw_read_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_read_unlock_bh"))
+ supported |= LOCK_PROG_RWLOCK_READ;
+
+ if ((btf_has_func(btf, "_raw_write_lock") || btf_has_func(btf, "_raw_write_lock_nested")) &&
+ btf_has_func(btf, "_raw_write_unlock") &&
+ btf_has_func(btf, "_raw_write_unlock_irq") &&
+ btf_has_func(btf, "_raw_write_unlock_irqrestore") &&
+ btf_has_func(btf, "_raw_write_unlock_bh"))
+ supported |= LOCK_PROG_RWLOCK_WRITE;
+
+ if ((btf_has_func(btf, "rt_mutex_lock") || btf_has_func(btf, "rt_mutex_lock_nested")) &&
+ btf_has_func(btf, "rt_mutex_unlock"))
+ supported |= LOCK_PROG_RTMUTEX;
+
+ if (btf_has_func(btf, "down") && btf_has_func(btf, "up"))
+ supported |= LOCK_PROG_SEM;
+
+ if (btf_has_func(btf, "percpu_down_write") && btf_has_func(btf, "percpu_up_write"))
+ supported |= LOCK_PROG_PERCPU_RWSEM;
+
+ return supported;
+}
+
+/*
+ * Enable or disable autoload for each lock-tracking BPF program in the
+ * skeleton before `sample_filter_bpf__load()` is called.
+ *
+ * Each program is enabled only if its group is in `needed_groups` and its
+ * target kernel function or tracepoint exists in the running kernel's BTF.
+ */
+static int configure_lock_progs(struct sample_filter_bpf *skel, struct btf *btf,
+ unsigned int needed_groups,
+ unsigned int required_groups,
+ bool is_tracepoint)
+{
+ unsigned int supported_groups = supported_lock_prog_groups(btf);
+ unsigned int enabled_progs = 0;
+
+ bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
+ bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
+
+ if ((required_groups & supported_groups) != required_groups ||
+ (needed_groups && !(needed_groups & supported_groups))) {
+ pr_err("Error: Kernel BTF lacks required lock/unlock tracing entry points\n");
+ return -ENOTSUP;
+ }
+
+ needed_groups &= supported_groups;
+ if (skel)
+ skel->bss->track_rwsem_read = !!(needed_groups & LOCK_PROG_RWSEM_READ);
+
+ for (size_t i = 0; i < ARRAY_SIZE(lock_progs); i++) {
+ struct bpf_program **progp = (void *)skel + lock_progs[i].prog_offset;
+ struct bpf_program *prog = *progp;
+ bool enable = false;
+
+ if ((needed_groups & lock_progs[i].group) && btf &&
+ lock_progs[i].btf_func) {
+ if (btf__find_by_name_kind(btf, lock_progs[i].btf_func,
+ BTF_KIND_FUNC) > 0)
+ enable = true;
+ }
+
+ if (enable)
+ enabled_progs++;
+ bpf_program__set_autoload(prog, enable);
+ }
+
+ if (needed_groups && !enabled_progs) {
+ pr_err("Error: Kernel BTF lacks required lock tracing entry points\n");
+ return -ENOTSUP;
+ }
+ return 0;
+}
+
+/*
+ * Pre-scan a single filter expression prior to loading the BPF skeleton in
+ * order to determine which BPF maps need to be sized and which lock tracing
+ * program groups need to be autoloaded.
+ */
static void prescan_one_lock_expr(struct perf_bpf_filter_expr *expr,
- bool *has_lock_filter)
+ bool *has_lock_filter,
+ unsigned int *prog_groups,
+ unsigned int *required_groups)
{
+ char base[128];
+ bool has_amp;
+ unsigned int mode, groups = 0;
+ u64 dummy;
+
if (expr->term != PBF_TERM_LOCK_HELD || !expr->lock_name)
return;
*has_lock_filter = true;
+ if (split_lock_name_mode(expr->lock_name, expr->part, base,
+ sizeof(base), &has_amp, &mode) < 0 || has_amp)
+ return;
+
+ if (parse_lock_type_name(base, mode, NULL, NULL, &groups)) {
+ *prog_groups |= groups;
+ *required_groups |= groups;
+ return;
+ }
+
+ if (parse_lock_hex_addr(base, &dummy)) {
+ *prog_groups |= LOCK_PROG_ALL;
+ return;
+ }
+
+ if (str_ends_with(base, "rt_mutex") || str_ends_with(base, "rtmutex") ||
+ str_ends_with(base, "mutex")) {
+ /* Inspected directly via lock->owner; no tracing hooks needed. */
+ return;
+ } else if (str_ends_with(base, "percpu_rwsem") ||
+ str_ends_with(base, "pcpu_rwsem")) {
+ *prog_groups |= LOCK_PROG_PERCPU_RWSEM;
+ *required_groups |= LOCK_PROG_PERCPU_RWSEM;
+ } else if (str_ends_with(base, "rwsem")) {
+ if (mode & PBF_LOCK_MODE_READ) {
+ *prog_groups |= LOCK_PROG_RWSEM_READ;
+ *required_groups |= LOCK_PROG_RWSEM_READ;
+ }
+ if (mode & PBF_LOCK_MODE_WRITE) {
+ *prog_groups |= LOCK_PROG_RWSEM_WRITE | LOCK_PROG_PERCPU_RWSEM;
+ *required_groups |= LOCK_PROG_RWSEM_WRITE;
+ }
+ } else if (str_ends_with(base, "spinlock")) {
+ *prog_groups |= LOCK_PROG_SPIN;
+ *required_groups |= LOCK_PROG_SPIN;
+ } else if (str_ends_with(base, "sem")) {
+ *prog_groups |= LOCK_PROG_SEM;
+ *required_groups |= LOCK_PROG_SEM;
+ } else if (str_ends_with(base, "rwlock")) {
+ if (mode & PBF_LOCK_MODE_READ)
+ groups |= LOCK_PROG_RWLOCK_READ;
+ if (mode & PBF_LOCK_MODE_WRITE)
+ groups |= LOCK_PROG_RWLOCK_WRITE;
+ *prog_groups |= groups;
+ *required_groups |= groups;
+ } else {
+ *prog_groups |= LOCK_PROG_ALL;
+ }
}
-static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter)
+static void prescan_lock_filters(struct evsel *evsel, bool *has_lock_filter,
+ unsigned int *prog_groups,
+ unsigned int *required_groups)
{
struct perf_bpf_filter_expr *expr, *group;
*has_lock_filter = false;
+ *prog_groups = 0;
+ *required_groups = 0;
list_for_each_entry(expr, &evsel->bpf_filters, list) {
- prescan_one_lock_expr(expr, has_lock_filter);
+ prescan_one_lock_expr(expr, has_lock_filter, prog_groups,
+ required_groups);
if (expr->op == PBF_OP_GROUP_BEGIN) {
- list_for_each_entry(group, &expr->groups, list)
- prescan_one_lock_expr(group, has_lock_filter);
+ list_for_each_entry(group, &expr->groups, list) {
+ prescan_one_lock_expr(group, has_lock_filter,
+ prog_groups, required_groups);
+ }
}
}
}
@@ -390,14 +805,42 @@ static int normalize_lock_expr_op(const struct perf_bpf_filter_expr *expr,
return -EINVAL;
}
-static int resolve_lock_expr(struct perf_bpf_filter_expr *expr,
+static bool is_percpu_rwsem_sym(struct btf *btf, const char *name)
+{
+ char rc_name[160];
+
+ if (str_ends_with(name, "percpu_rwsem") ||
+ str_ends_with(name, "pcpu_rwsem"))
+ return true;
+
+ if (!btf)
+ return false;
+
+ snprintf(rc_name, sizeof(rc_name), "__percpu_rwsem_rc_%s", name);
+ return btf__find_by_name_kind(btf, rc_name, BTF_KIND_VAR) > 0;
+}
+
+/*
+ * Resolve a `lock_held(<name>)` filter expression into `entry->op`,
+ * `entry->part` (`enum perf_bpf_lock_target_kind`), and `entry->value`,
+ * and populate the corresponding BPF filter maps and BSS control flags.
+ *
+ * Resolution order:
+ * 1. Lock types / flags: `mutex`, `spinlock`, `rwsem`, `rwsem:R`, `rwsem:W`,
+ * `rwlock`, `percpu-rwsem`, `semaphore`, etc.
+ * 2. Hexadecimal lock address.
+ * 3. Global/static kernel lock symbol in kallsyms (e.g. `cgroup_mutex`).
+ */
+static int resolve_lock_expr(struct sample_filter_bpf *skel, struct btf *btf,
+ struct perf_bpf_filter_expr *expr,
struct perf_bpf_filter_entry *entry)
{
const char *name = expr->lock_name;
char base[128];
bool has_amp;
unsigned int mode;
- u64 addr = 0;
+ u64 addr = 0, encoded_pair = 0;
+ u8 one = 1;
int ret;
ret = normalize_lock_expr_op(expr, &entry->op);
@@ -410,29 +853,79 @@ static int resolve_lock_expr(struct perf_bpf_filter_expr *expr,
return -EINVAL;
}
- if (str_ends_with(base, "mutex") &&
- !str_ends_with(base, "rt_mutex") &&
- !str_ends_with(base, "rtmutex")) {
- ret = find_kernel_lock_sym(base, &addr);
- if (ret == 0) {
- if (!(mode & PBF_LOCK_MODE_WRITE)) {
- pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
- name);
- return -EINVAL;
- }
- entry->part = PBF_LOCK_ADDR | mode | PBF_LOCK_DIRECT_MUTEX;
- entry->value = addr;
- return 0;
+ if (parse_lock_type_name(base, mode, skel, &encoded_pair, NULL)) {
+ entry->part = PBF_LOCK_TYPE | mode;
+ entry->value = encoded_pair;
+ return 0;
+ }
+
+ if (parse_lock_type_name(base, PBF_LOCK_MODE_ANY, NULL, NULL, NULL)) {
+ pr_err("Error: Lock type '%s' does not support the requested read/write mode\n",
+ name);
+ return -EINVAL;
+ }
+
+ if (parse_lock_hex_addr(base, &addr)) {
+ entry->part = PBF_LOCK_ADDR | mode;
+ entry->value = addr;
+ if (skel) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
}
- if (ret == -EACCES)
- return ret;
+ return 0;
}
+ if (btf_has_func(btf, base)) {
+ pr_err("Error: '%s' is a kernel function, not a lock\n", name);
+ return -EINVAL;
+ }
+
+ ret = find_kernel_lock_sym(base, &addr);
+ if (ret == 0) {
+ unsigned int direct_flag = 0;
+ bool is_pcpu_rwsem = is_percpu_rwsem_sym(btf, base);
+
+ if (str_ends_with(base, "rt_mutex") || str_ends_with(base, "rtmutex"))
+ direct_flag = PBF_LOCK_DIRECT_RTMUTEX;
+ else if (str_ends_with(base, "mutex"))
+ direct_flag = PBF_LOCK_DIRECT_MUTEX;
+ else if (str_ends_with(base, "rwsem") && !is_pcpu_rwsem)
+ direct_flag = PBF_LOCK_DIRECT_RWSEM;
+
+ if ((direct_flag == PBF_LOCK_DIRECT_MUTEX ||
+ direct_flag == PBF_LOCK_DIRECT_RTMUTEX ||
+ is_pcpu_rwsem ||
+ str_ends_with(base, "spinlock") ||
+ (str_ends_with(base, "sem") &&
+ !str_ends_with(base, "rwsem"))) &&
+ !(mode & PBF_LOCK_MODE_WRITE)) {
+ pr_err("Error: Lock '%s' is exclusive and does not support read/shared mode\n",
+ name);
+ return -EINVAL;
+ }
+
+ entry->part = PBF_LOCK_ADDR | mode | direct_flag;
+ entry->value = addr;
+ if (skel && direct_flag != PBF_LOCK_DIRECT_MUTEX &&
+ direct_flag != PBF_LOCK_DIRECT_RTMUTEX) {
+ int fd = bpf_map__fd(skel->maps.lock_addr_filter);
+
+ bpf_map_update_elem(fd, &addr, &one, BPF_ANY);
+ skel->bss->has_lock_addr = 1;
+ }
+ return 0;
+ }
+ if (ret == -EACCES)
+ return ret;
+
pr_err("Error: Cannot resolve kernel lock '%s'\n", name);
return -EINVAL;
}
-static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry *entry)
+static int get_filter_entries(struct evsel *evsel, struct sample_filter_bpf *skel,
+ struct btf *btf, struct perf_bpf_filter_entry *entry)
{
int i = 0, ret;
struct perf_bpf_filter_expr *expr;
@@ -449,7 +942,7 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].term = expr->term;
entry[i].value = expr->val;
if (expr->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(expr, &entry[i]);
+ ret = resolve_lock_expr(skel, btf, expr, &entry[i]);
if (ret < 0)
return ret;
}
@@ -467,7 +960,7 @@ static int get_filter_entries(struct evsel *evsel, struct perf_bpf_filter_entry
entry[i].term = group->term;
entry[i].value = group->val;
if (group->term == PBF_TERM_LOCK_HELD) {
- ret = resolve_lock_expr(group, &entry[i]);
+ ret = resolve_lock_expr(skel, btf, group, &entry[i]);
if (ret < 0)
return ret;
}
@@ -710,16 +1203,19 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
{
int i, x, y, fd, ret;
struct sample_filter_bpf *skel = NULL;
+ struct btf *btf = NULL;
struct perf_bpf_filter_entry *entry;
bool needs_idx_hash = !target__has_cpu(target);
bool is_tracepoint = (evsel->core.attr.type == PERF_TYPE_TRACEPOINT);
bool has_lock_filter = false;
+ unsigned int prog_groups = 0, required_groups = 0;
entry = calloc(MAX_FILTERS, sizeof(*entry));
if (entry == NULL)
return -1;
- prescan_lock_filters(evsel, &has_lock_filter);
+ prescan_lock_filters(evsel, &has_lock_filter, &prog_groups,
+ &required_groups);
if ((has_lock_filter || is_tracepoint) && geteuid() != 0 &&
!perf_cap__capable(CAP_BPF)) {
@@ -728,15 +1224,14 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- ret = get_filter_entries(evsel, entry);
- if (ret < 0) {
- pr_err("Failed to process filter entries\n");
- goto err;
- }
-
if (needs_idx_hash && geteuid() != 0 && !perf_cap__capable(CAP_BPF)) {
int zero = 0;
+ ret = get_filter_entries(evsel, NULL, NULL, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
/* The filters map is shared among other processes */
ret = create_idx_hash(evsel, entry);
if (ret < 0)
@@ -781,8 +1276,17 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- bpf_program__set_autoload(skel->progs.perf_sample_filter, !is_tracepoint);
- bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, is_tracepoint);
+ if (has_lock_filter) {
+ btf = btf__load_vmlinux_btf();
+ bpf_map__set_max_entries(skel->maps.held_locks, MAX_ENTRIES);
+ bpf_map__set_max_entries(skel->maps.lock_addr_filter, MAX_FILTERS);
+ ret = configure_lock_progs(skel, btf, prog_groups, required_groups,
+ is_tracepoint);
+ if (ret < 0)
+ goto err;
+ } else {
+ configure_lock_progs(skel, NULL, 0, 0, is_tracepoint);
+ }
if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
@@ -790,8 +1294,23 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
goto err;
}
- if (has_lock_filter)
+ ret = get_filter_entries(evsel, skel, btf, entry);
+ if (ret < 0) {
+ pr_err("Failed to process filter entries\n");
+ goto err;
+ }
+
+ if (has_lock_filter) {
+ bpf_program__set_autoattach(skel->progs.perf_sample_filter, false);
+ bpf_program__set_autoattach(skel->progs.perf_tp_sample_filter, false);
+
+ ret = sample_filter_bpf__attach(skel);
+ if (ret < 0) {
+ pr_err("Failed to attach lock sample-filter BPF programs\n");
+ goto err;
+ }
skel->bss->lock_enabled = 1;
+ }
i = 0;
fd = bpf_map__fd(skel->maps.filters);
@@ -832,11 +1351,13 @@ int perf_bpf_filter__prepare(struct evsel *evsel, struct target *target)
}
}
}
+ btf__free(btf);
free(entry);
evsel->bpf_skel = skel;
return 0;
err:
+ btf__free(btf);
free(entry);
if (!list_empty(&pinned_filters)) {
struct pinned_filter_idx *pfi, *tmp;
@@ -1064,8 +1585,9 @@ int perf_bpf_filter__pin(void)
goto err;
}
+ configure_lock_progs(skel, NULL, 0, 0, false);
+
/* pinned program will use pid-hash */
- bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
bpf_map__set_max_entries(skel->maps.filters, MAX_FILTERS);
bpf_map__set_max_entries(skel->maps.event_hash, MAX_EVT_HASH);
bpf_map__set_max_entries(skel->maps.idx_hash, MAX_IDX_HASH);
@@ -1160,7 +1682,7 @@ int perf_bpf_filter__unpin(void)
goto err;
}
- bpf_program__set_autoload(skel->progs.perf_tp_sample_filter, false);
+ configure_lock_progs(skel, NULL, 0, 0, false);
if (sample_filter_bpf__load(skel) < 0) {
ret = -errno;
pr_err("Failed to load perf sample-filter BPF skeleton\n");
diff --git a/tools/perf/util/bpf_skel/sample-filter.h b/tools/perf/util/bpf_skel/sample-filter.h
index ca1cf586ed6e..c240eee782f8 100644
--- a/tools/perf/util/bpf_skel/sample-filter.h
+++ b/tools/perf/util/bpf_skel/sample-filter.h
@@ -59,7 +59,8 @@ enum perf_bpf_filter_term {
* The lock specification kind is stored in `entry.part`
* (`enum perf_bpf_lock_target_kind`), `entry.op` is normalized to
* `PBF_OP_EQ` (match when held) or `PBF_OP_NEQ` (match when not held),
- * and the resolved target address is stored in `entry.value`.
+ * and the resolved target address or type flag encoding is stored in
+ * `entry.value`.
*/
PBF_TERM_LOCK_HELD,
};
@@ -71,10 +72,16 @@ enum perf_bpf_filter_term {
enum perf_bpf_lock_target_kind {
PBF_LOCK_NONE = 0,
/*
- * Specific global or static lock address resolved from kallsyms.
- * `entry.value` holds the kernel address.
+ * Specific global or static lock address (resolved from kallsyms or
+ * given as a hex address). `entry.value` holds the kernel address.
*/
PBF_LOCK_ADDR,
+ /*
+ * Lock type filter (`mutex`, `spinlock`, `rwsem`, `rwsem:R`, etc.).
+ * `entry.value` packs up to two `LCB_F_*` flag combinations in the
+ * low two bytes: `flag1 | (flag2 << 8)` (with 0xff indicating unused).
+ */
+ PBF_LOCK_TYPE,
};
/*
@@ -83,6 +90,8 @@ enum perf_bpf_lock_target_kind {
* Bit 4: PBF_LOCK_MODE_READ (match read/shared lock holding)
* Bit 5: PBF_LOCK_MODE_WRITE (match write/exclusive lock holding)
* Bit 6: PBF_LOCK_DIRECT_MUTEX (inspect struct mutex owner)
+ * Bit 7: PBF_LOCK_DIRECT_RWSEM (inspect struct rw_semaphore writer owner)
+ * Bit 8: PBF_LOCK_DIRECT_RTMUTEX (inspect struct rt_mutex_base / rt_mutex owner)
*/
#define PBF_LOCK_KIND_MASK 0x0fU
#define PBF_LOCK_MODE_READ (1U << 4)
@@ -90,6 +99,8 @@ enum perf_bpf_lock_target_kind {
#define PBF_LOCK_MODE_ANY (PBF_LOCK_MODE_READ | PBF_LOCK_MODE_WRITE)
#define PBF_LOCK_MODE_MASK PBF_LOCK_MODE_ANY
#define PBF_LOCK_DIRECT_MUTEX (1U << 6)
+#define PBF_LOCK_DIRECT_RWSEM (1U << 7)
+#define PBF_LOCK_DIRECT_RTMUTEX (1U << 8)
/* BPF map entry for filtering */
struct perf_bpf_filter_entry {
diff --git a/tools/perf/util/bpf_skel/sample_filter.bpf.c b/tools/perf/util/bpf_skel/sample_filter.bpf.c
index 42556ab0e2d5..e527fbaf8a46 100644
--- a/tools/perf/util/bpf_skel/sample_filter.bpf.c
+++ b/tools/perf/util/bpf_skel/sample_filter.bpf.c
@@ -8,6 +8,44 @@
#include "lock_data.h"
#include "sample-filter.h"
+/*
+ * Key for the `held_locks` BPF hash map tracking currently held kernel locks.
+ *
+ * - `addr_or_id`:
+ * - PBF_LOCK_ADDR: kernel virtual address of the lock instance
+ * - PBF_LOCK_CLASS: `enum lock_class_sym` (LOCK_CLASS_RQLOCK / LOCK_CLASS_ZONE_LOCK)
+ * - PBF_LOCK_SLAB: kernel `struct kmem_cache *` address
+ * - PBF_LOCK_TYPE: `LCB_F_*` lock type flags
+ * - `owner_id`:
+ * - For sleeping locks: kernel task TID (`(__u32)bpf_get_current_pid_tgid()`),
+ * since the holder may sleep and migrate across CPUs while holding the lock.
+ * - For spinning locks (`LCB_F_SPIN` without `LCB_F_MUTEX`) or idle tasks
+ * (`pid == 0`): `0x80000000U | bpf_get_smp_processor_id()`, since preemption
+ * is disabled on the holding CPU and spinlocks such as `rq_lock` may be
+ * acquired by one task and released by the next task across `context_switch()`.
+ * - `kind`: `enum perf_bpf_lock_target_kind`.
+ */
+struct held_lock_key {
+ __u64 addr_or_id;
+ __u32 owner_id;
+ __u32 kind;
+};
+
+/*
+ * Value for `held_locks` entries.
+ * `read_count` and `write_count` separately track shared/read (`LCB_F_READ`)
+ * and exclusive/write acquisitions (including nested class/slab/type
+ * acquisitions by the same owner). `flags` stores the `LCB_F_*` type flags of
+ * the lock so release tracepoints that do not carry flags (such as
+ * `lock:contended_release`) can decrement matching class/slab/type entries.
+ */
+struct held_lock_val {
+ __u32 read_count;
+ __u32 write_count;
+ __u32 flags;
+ __u32 pad;
+};
+
/* BPF map that will be filled by user space */
struct filters {
__uint(type, BPF_MAP_TYPE_HASH);
@@ -44,13 +82,65 @@ struct lost_count {
__uint(max_entries, 1);
} dropped SEC(".maps");
+/*
+ * Currently held kernel locks tracked by fentry/fexit hooks.
+ * Uses BPF_MAP_TYPE_LRU_HASH so that if a counting semaphore (down/up) or
+ * non-owner rwsem is acquired in one task and released in another task or
+ * interrupt handler (cross-context signaling), any orphaned entry is
+ * automatically evicted rather than exhausting the map.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_LRU_HASH);
+ __type(key, struct held_lock_key);
+ __type(value, struct held_lock_val);
+ __uint(max_entries, 1);
+} held_locks SEC(".maps");
+
+/* Target lock addresses for PBF_LOCK_ADDR filters. */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, __u64);
+ __type(value, __u8);
+ __uint(max_entries, 1);
+} lock_addr_filter SEC(".maps");
+
volatile const int use_idx_hash;
-/* Control flag set by userspace when lock_held(...) filters are active. */
+/* Control flags set by userspace when lock_held(...) filters are active. */
int lock_enabled;
+int has_lock_addr;
+int has_lock_type;
+int track_rwsem_read;
+__u8 lock_type_filter[LCB_F_TYPE_MAX];
void *bpf_cast_to_kern_ctx(void *) __ksym;
+/*
+ * Direct lock owner inspection helper using `get_lock_owner()` from `lock_data.h`.
+ *
+ * For locks that maintain a task_struct pointer in their `owner` field
+ * (`struct mutex`, write-locked `struct rw_semaphore`, and `struct rt_mutex`),
+ * `perf_sample_filter` can inspect the lock structure directly in kernel memory
+ * and compare the owner against `current` (`bpf_get_current_task()`).
+ *
+ * Note that for reader-locked `rw_semaphore`s, `__up_read()` does not clear
+ * `sem->owner` when `CONFIG_DEBUG_RWSEMS` is disabled, so reader locks and
+ * spinlocks are checked via the `held_locks` BPF map instead.
+ */
+static __always_inline bool is_rwsem_writer_owner(__u64 lock_addr, __u64 curr)
+{
+ struct rw_semaphore___new *rwsem = (void *)lock_addr;
+
+ if (bpf_core_field_exists(rwsem->count.counter) &&
+ !(BPF_CORE_READ(rwsem, count.counter) & 1UL))
+ return false;
+ if (bpf_core_field_exists(rwsem->owner.counter) &&
+ (BPF_CORE_READ(rwsem, owner.counter) & 1UL))
+ return false;
+
+ return (__u64)get_lock_owner(lock_addr, LCB_F_WRITE) == curr;
+}
+
static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __u32 part)
{
if (!curr)
@@ -59,24 +149,85 @@ static __always_inline bool is_direct_lock_owner(__u64 lock_addr, __u64 curr, __
if ((part & PBF_LOCK_DIRECT_MUTEX) &&
(__u64)get_lock_owner(lock_addr, LCB_F_MUTEX) == curr)
return true;
+ if ((part & PBF_LOCK_DIRECT_RWSEM) &&
+ is_rwsem_writer_owner(lock_addr, curr))
+ return true;
+ if ((part & PBF_LOCK_DIRECT_RTMUTEX) &&
+ (__u64)get_lock_owner(lock_addr, LCB_F_RT) == curr)
+ return true;
return false;
}
+static __always_inline bool match_held_val(struct held_lock_val *val, __u32 mode)
+{
+ if (!val)
+ return false;
+ if ((mode & PBF_LOCK_MODE_READ) && val->read_count > 0)
+ return true;
+ if ((mode & PBF_LOCK_MODE_WRITE) && val->write_count > 0)
+ return true;
+ return false;
+}
+
+static __always_inline __u64 is_lock_in_map(__u64 addr_or_id, __u32 kind,
+ __u32 mode, __u32 pid, __u32 cpu_id)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = pid,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+
+ if (pid) {
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+ }
+
+ key.owner_id = cpu_id;
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (match_held_val(val, mode))
+ return 1;
+
+ return 0;
+}
+
+static __always_inline __u64 is_lock_type_in_map(__u64 target, __u32 mode,
+ __u32 pid, __u32 cpu_id)
+{
+ __u32 flag1 = target & 0xff;
+ __u32 flag2 = (target >> 8) & 0xff;
+
+ if (is_lock_in_map(flag1, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ if (flag2 != 0xff && is_lock_in_map(flag2, PBF_LOCK_TYPE, mode, pid, cpu_id))
+ return 1;
+ return 0;
+}
+
static __always_inline bool is_lock_held(struct perf_bpf_filter_entry *entry)
{
__u64 curr = bpf_get_current_task();
+ __u32 pid = (__u32)bpf_get_current_pid_tgid();
+ __u32 cpu_id = 0x80000000U | bpf_get_smp_processor_id();
__u32 kind = entry->part & PBF_LOCK_KIND_MASK;
__u32 mode = entry->part & PBF_LOCK_MODE_MASK;
+ __u64 lock_addr;
if (!mode)
mode = PBF_LOCK_MODE_ANY;
switch (kind) {
case PBF_LOCK_ADDR:
+ lock_addr = entry->value;
if ((mode & PBF_LOCK_MODE_WRITE) &&
- is_direct_lock_owner(entry->value, curr, entry->part))
+ is_direct_lock_owner(lock_addr, curr, entry->part))
return true;
- return false;
+ return is_lock_in_map(lock_addr, PBF_LOCK_ADDR, mode, pid, cpu_id) != 0;
+
+ case PBF_LOCK_TYPE:
+ return is_lock_type_in_map(entry->value, mode, pid, cpu_id) != 0;
default:
return false;
@@ -444,4 +595,698 @@ int perf_tp_sample_filter(void *ctx)
return 0;
}
+static __always_inline __u32 get_lock_owner_id(__u32 flags)
+{
+ __u32 owner_id;
+
+ if ((flags & (LCB_F_SPIN | LCB_F_MUTEX)) == LCB_F_SPIN)
+ return 0x80000000U | bpf_get_smp_processor_id();
+
+ owner_id = (__u32)bpf_get_current_pid_tgid();
+ if (!owner_id)
+ owner_id = 0x80000000U | bpf_get_smp_processor_id();
+ return owner_id;
+}
+
+static __always_inline void update_held_lock_count(__u64 addr_or_id, __u32 owner_id,
+ __u32 kind, __u32 flags, bool acquired)
+{
+ struct held_lock_key key = {
+ .addr_or_id = addr_or_id,
+ .owner_id = owner_id,
+ .kind = kind,
+ };
+ struct held_lock_val *val;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ val = bpf_map_lookup_elem(&held_locks, &key);
+ if (acquired) {
+ if (val) {
+ if (is_read)
+ __sync_fetch_and_add(&val->read_count, 1);
+ else
+ __sync_fetch_and_add(&val->write_count, 1);
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags,
+ };
+
+ bpf_map_update_elem(&held_locks, &key, &first, BPF_ANY);
+ }
+ } else if (val) {
+ if (is_read) {
+ if (val->read_count <= 1 && val->write_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->read_count > 0)
+ __sync_fetch_and_add(&val->read_count, -1);
+ } else {
+ if (val->write_count <= 1 && val->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &key);
+ else if (val->write_count > 0)
+ __sync_fetch_and_add(&val->write_count, -1);
+ }
+ }
+}
+
+#define LCB_F_READ_IRQSAVE (1U << 30)
+#define LCB_F_READ_IRQ_BH (1U << 31)
+
+static __always_inline int record_lock_acquired_owner(__u64 lock_addr, __u32 flags, __u32 owner_id)
+{
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ bool matched = false;
+ bool is_read = (flags & LCB_F_READ) != 0;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool match_type = false;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ if (has_lock_addr && bpf_map_lookup_elem(&lock_addr_filter, &lock_addr))
+ matched = true;
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag]) {
+ match_type = true;
+ matched = true;
+ }
+
+ if (!matched)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (existing) {
+ /*
+ * On CONFIG_GENERIC_LOCKBREAK=y && !CONFIG_DEBUG_LOCK_ALLOC
+ * kernels, _raw_read_lock_irq() and _raw_read_lock_bh() call
+ * the out-of-line _raw_read_lock_irqsave(). Clear the marker
+ * and skip the duplicate outer fexit increment.
+ */
+ if ((flags & LCB_F_READ_IRQ_BH) &&
+ (existing->flags & LCB_F_READ_IRQSAVE)) {
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ return 0;
+ }
+ /* Non-reader locks are not recursive; avoid double-counting */
+ if (!is_read && existing->write_count > 0)
+ return 0;
+ if (is_read) {
+ __sync_fetch_and_add(&existing->read_count, 1);
+ if (flags & LCB_F_READ_IRQSAVE)
+ existing->flags |= LCB_F_READ_IRQSAVE;
+ else
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ } else {
+ __sync_fetch_and_add(&existing->write_count, 1);
+ }
+ } else {
+ struct held_lock_val first = {
+ .read_count = is_read ? 1 : 0,
+ .write_count = is_read ? 0 : 1,
+ .flags = flags & (LCB_F_TYPE_MASK | LCB_F_READ_IRQSAVE),
+ };
+
+ if (bpf_map_update_elem(&held_locks, &addr_key, &first, BPF_ANY) < 0)
+ return 0;
+ }
+
+ if (match_type)
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, type_flag, true);
+
+ return 0;
+}
+
+static __always_inline int record_lock_acquired(void *ctx, __u32 flags)
+{
+ return record_lock_acquired_owner(get_lock_addr(ctx), flags, get_lock_owner_id(flags));
+}
+
+static __always_inline int record_lock_released_addr(__u64 lock_addr, __u32 flags)
+{
+ __u32 owner_id = get_lock_owner_id(flags);
+ struct held_lock_key addr_key = {
+ .addr_or_id = lock_addr,
+ .owner_id = owner_id,
+ .kind = PBF_LOCK_ADDR,
+ };
+ struct held_lock_val *existing;
+ __u32 type_flag = flags & LCB_F_TYPE_MASK;
+ bool is_read = (flags & LCB_F_READ) != 0;
+
+ if (!lock_enabled || !lock_addr)
+ return 0;
+
+ existing = bpf_map_lookup_elem(&held_locks, &addr_key);
+ if (!existing)
+ return 0;
+
+ if (is_read) {
+ if (existing->read_count == 0)
+ return 0;
+ if (existing->read_count == 1 && existing->write_count == 0) {
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ } else {
+ existing->flags &= ~LCB_F_READ_IRQSAVE;
+ __sync_fetch_and_add(&existing->read_count, -1);
+ }
+ } else {
+ if (existing->write_count == 0)
+ return 0;
+ if (existing->read_count == 0)
+ bpf_map_delete_elem(&held_locks, &addr_key);
+ else
+ existing->write_count = 0;
+ }
+
+ if (has_lock_type && type_flag < LCB_F_TYPE_MAX && lock_type_filter[type_flag])
+ update_held_lock_count(type_flag, owner_id, PBF_LOCK_TYPE, flags, false);
+
+ return 0;
+}
+
+static __always_inline int record_lock_released(void *ctx, __u32 flags)
+{
+ return record_lock_released_addr(get_lock_addr(ctx), flags);
+}
+
+/* Mutex acquire/release hooks */
+SEC("fexit/mutex_lock")
+int BPF_PROG(fexit_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_nested")
+int BPF_PROG(fexit_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/_mutex_lock_nest_lock")
+int BPF_PROG(fexit__mutex_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io")
+int BPF_PROG(fexit_mutex_lock_io, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_io_nested")
+int BPF_PROG(fexit_mutex_lock_io_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_MUTEX);
+}
+
+SEC("fexit/mutex_lock_interruptible")
+int BPF_PROG(fexit_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_interruptible_nested")
+int BPF_PROG(fexit_mutex_lock_interruptible_nested, void *lock, unsigned int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_lock_killable")
+int BPF_PROG(fexit_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/_mutex_lock_killable")
+int BPF_PROG(fexit__mutex_lock_killable, void *lock, unsigned int subclass,
+ void *nest_lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/mutex_trylock")
+int BPF_PROG(fexit_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fexit/_mutex_trylock_nest_lock")
+int BPF_PROG(fexit__mutex_trylock_nest_lock, void *lock, void *nest_lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_MUTEX);
+ return 0;
+}
+
+SEC("fentry/mutex_unlock")
+int BPF_PROG(fentry_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_MUTEX);
+}
+
+/* RW semaphore acquire/release hooks */
+SEC("fexit/down_read")
+int BPF_PROG(fexit_down_read, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_nested")
+int BPF_PROG(fexit_down_read_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_READ);
+}
+
+SEC("fexit/down_read_interruptible")
+int BPF_PROG(fexit_down_read_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable")
+int BPF_PROG(fexit_down_read_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_killable_nested")
+int BPF_PROG(fexit_down_read_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_read_trylock")
+int BPF_PROG(fexit_down_read_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return 0;
+}
+
+SEC("fexit/down_write")
+int BPF_PROG(fexit_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_nested")
+int BPF_PROG(fexit_down_write_nested, void *sem, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/_down_write_nest_lock")
+int BPF_PROG(fexit__down_write_nest_lock, void *sem, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_WRITE);
+}
+
+SEC("fexit/down_write_killable")
+int BPF_PROG(fexit_down_write_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_killable_nested")
+int BPF_PROG(fexit_down_write_killable_nested, void *sem, int subclass, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/down_write_trylock")
+int BPF_PROG(fexit_down_write_trylock, void *sem, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fexit/downgrade_write")
+int BPF_PROG(fexit_downgrade_write, void *sem)
+{
+ if (track_rwsem_read)
+ record_lock_acquired(ctx, LCB_F_READ);
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+SEC("fentry/up_read")
+int BPF_PROG(fentry_up_read, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_READ);
+}
+
+SEC("fentry/up_write")
+int BPF_PROG(fentry_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_WRITE);
+}
+
+/* Spinlock acquire/release hooks */
+SEC("fexit/_raw_spin_lock")
+int BPF_PROG(fexit_raw_spin_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_nested")
+int BPF_PROG(fexit_raw_spin_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_nest_lock")
+int BPF_PROG(fexit_raw_spin_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq")
+int BPF_PROG(fexit_raw_spin_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irq_disable")
+int BPF_PROG(fexit_raw_spin_lock_irq_disable, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave")
+int BPF_PROG(fexit_raw_spin_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_irqsave_nested")
+int BPF_PROG(fexit_raw_spin_lock_irqsave_nested, void *lock, int subclass, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_lock_bh")
+int BPF_PROG(fexit_raw_spin_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN);
+}
+
+SEC("fexit/_raw_spin_trylock")
+int BPF_PROG(fexit_raw_spin_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fexit/_raw_spin_trylock_bh")
+int BPF_PROG(fexit_raw_spin_trylock_bh, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN);
+ return 0;
+}
+
+SEC("fentry/_raw_spin_unlock")
+int BPF_PROG(fentry_raw_spin_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq")
+int BPF_PROG(fentry_raw_spin_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irq_enable")
+int BPF_PROG(fentry_raw_spin_unlock_irq_enable, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_irqrestore")
+int BPF_PROG(fentry_raw_spin_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+SEC("fentry/_raw_spin_unlock_bh")
+int BPF_PROG(fentry_raw_spin_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN);
+}
+
+/* RWlock acquire/release hooks */
+SEC("fexit/_raw_read_lock")
+int BPF_PROG(fexit_raw_read_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_read_lock_irq")
+int BPF_PROG(fexit_raw_read_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH);
+}
+
+SEC("fexit/_raw_read_lock_irqsave")
+int BPF_PROG(fexit_raw_read_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQSAVE);
+}
+
+SEC("fexit/_raw_read_lock_bh")
+int BPF_PROG(fexit_raw_read_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ | LCB_F_READ_IRQ_BH);
+}
+
+SEC("fexit/_raw_read_trylock")
+int BPF_PROG(fexit_raw_read_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_READ);
+ return 0;
+}
+
+SEC("fentry/_raw_read_unlock")
+int BPF_PROG(fentry_raw_read_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irq")
+int BPF_PROG(fentry_raw_read_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_irqrestore")
+int BPF_PROG(fentry_raw_read_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fentry/_raw_read_unlock_bh")
+int BPF_PROG(fentry_raw_read_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_READ);
+}
+
+SEC("fexit/_raw_write_lock")
+int BPF_PROG(fexit_raw_write_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_nested")
+int BPF_PROG(fexit_raw_write_lock_nested, void *lock, int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irq")
+int BPF_PROG(fexit_raw_write_lock_irq, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_irqsave")
+int BPF_PROG(fexit_raw_write_lock_irqsave, void *lock, unsigned long flags)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_lock_bh")
+int BPF_PROG(fexit_raw_write_lock_bh, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fexit/_raw_write_trylock")
+int BPF_PROG(fexit_raw_write_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_SPIN | LCB_F_WRITE);
+ return 0;
+}
+
+SEC("fentry/_raw_write_unlock")
+int BPF_PROG(fentry_raw_write_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irq")
+int BPF_PROG(fentry_raw_write_unlock_irq, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_irqrestore")
+int BPF_PROG(fentry_raw_write_unlock_irqrestore, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+SEC("fentry/_raw_write_unlock_bh")
+int BPF_PROG(fentry_raw_write_unlock_bh, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_SPIN | LCB_F_WRITE);
+}
+
+/* RT-Mutex acquire/release hooks */
+SEC("fexit/rt_mutex_lock")
+int BPF_PROG(fexit_rt_mutex_lock, void *lock)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_nested")
+int BPF_PROG(fexit_rt_mutex_lock_nested, void *lock, unsigned int subclass)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/_rt_mutex_lock_nest_lock")
+int BPF_PROG(fexit__rt_mutex_lock_nest_lock, void *lock, void *nest_lock)
+{
+ return record_lock_acquired(ctx, LCB_F_RT);
+}
+
+SEC("fexit/rt_mutex_lock_interruptible")
+int BPF_PROG(fexit_rt_mutex_lock_interruptible, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_lock_killable")
+int BPF_PROG(fexit_rt_mutex_lock_killable, void *lock, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fexit/rt_mutex_trylock")
+int BPF_PROG(fexit_rt_mutex_trylock, void *lock, int ret)
+{
+ if (ret)
+ record_lock_acquired(ctx, LCB_F_RT);
+ return 0;
+}
+
+SEC("fentry/rt_mutex_unlock")
+int BPF_PROG(fentry_rt_mutex_unlock, void *lock)
+{
+ return record_lock_released(ctx, LCB_F_RT);
+}
+
+/* Semaphore acquire/release hooks */
+SEC("fexit/down")
+int BPF_PROG(fexit_down, void *sem)
+{
+ return record_lock_acquired(ctx, 0);
+}
+
+SEC("fexit/down_interruptible")
+int BPF_PROG(fexit_down_interruptible, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_killable")
+int BPF_PROG(fexit_down_killable, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_timeout")
+int BPF_PROG(fexit_down_timeout, void *sem, long timeout, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fexit/down_trylock")
+int BPF_PROG(fexit_down_trylock, void *sem, int ret)
+{
+ if (ret == 0)
+ record_lock_acquired(ctx, 0);
+ return 0;
+}
+
+SEC("fentry/up")
+int BPF_PROG(fentry_up, void *sem)
+{
+ return record_lock_released(ctx, 0);
+}
+
+/* Percpu-rwsem write acquire/release hooks */
+SEC("fexit/percpu_down_write")
+int BPF_PROG(fexit_percpu_down_write, void *sem)
+{
+ return record_lock_acquired(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
+SEC("fentry/percpu_up_write")
+int BPF_PROG(fentry_percpu_up_write, void *sem)
+{
+ return record_lock_released(ctx, LCB_F_PERCPU | LCB_F_WRITE);
+}
+
char LICENSE[] SEC("license") = "Dual BSD/GPL";
--
2.56.0.307.g5fe7bd680c-goog
next prev parent reply other threads:[~2026-10-22 20:30 UTC|newest]
Thread overview: 104+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-17 5:45 [PATCH v1 0/5] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-17 5:45 ` [PATCH v1 1/5] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-17 5:52 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 2/5] perf bpf-filter: Add lock_held() sample filter Ian Rogers
2026-10-17 6:09 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 3/5] perf test: Add tests for lock_held() BPF " Ian Rogers
2026-10-17 5:49 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 4/5] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-17 5:56 ` sashiko-bot
2026-10-17 5:45 ` [PATCH v1 5/5] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-17 6:08 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-17 17:14 ` [PATCH v2 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-17 17:31 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers
2026-10-17 17:35 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers
2026-10-17 17:17 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-17 17:25 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-17 17:30 ` sashiko-bot
2026-10-17 17:14 ` [PATCH v2 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers
2026-10-17 17:17 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-18 5:37 ` [PATCH v3 1/6] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-18 5:50 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 2/6] perf bpf-filter: Add lock_held() sample filter Ian Rogers
2026-10-18 6:12 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 3/6] perf test: Add tests for lock_held() BPF " Ian Rogers
2026-10-18 5:41 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 4/6] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-18 5:51 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 5/6] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-18 5:48 ` sashiko-bot
2026-10-18 5:37 ` [PATCH v3 6/6] perf lock contention: Filter events before creating lock_stat Ian Rogers
2026-10-18 5:40 ` sashiko-bot
2026-10-19 7:00 ` [PATCH v3 0/6] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Namhyung Kim
2026-10-19 19:08 ` Ian Rogers
2026-10-19 22:09 ` [PATCH v4 0/7] " Ian Rogers
2026-10-19 22:09 ` [PATCH v4 1/7] perf lock contention: Filter events before creating lock_stat Ian Rogers
2026-10-19 22:14 ` sashiko-bot
2026-10-20 4:34 ` Namhyung Kim
2026-10-21 21:39 ` Arnaldo Carvalho de Melo
2026-10-19 22:09 ` [PATCH v4 2/7] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
2026-10-19 22:26 ` sashiko-bot
2026-10-20 5:41 ` Namhyung Kim
2026-10-20 15:30 ` Ian Rogers
2026-10-20 22:29 ` Namhyung Kim
2026-10-19 22:09 ` [PATCH v4 3/7] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-19 22:17 ` sashiko-bot
2026-10-19 22:09 ` [PATCH v4 4/7] perf bpf-filter: Add lock_held() sample filter Ian Rogers
2026-10-19 22:28 ` sashiko-bot
2026-10-19 22:09 ` [PATCH v4 5/7] perf test: Add tests for lock_held() BPF " Ian Rogers
2026-10-19 22:12 ` sashiko-bot
2026-10-19 22:09 ` [PATCH v4 6/7] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-19 22:19 ` sashiko-bot
2026-10-19 22:09 ` [PATCH v4 7/7] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-19 22:19 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 0/8] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-20 20:58 ` [PATCH v5 1/8] perf lock contention: Filter events before creating lock_stat Ian Rogers
2026-10-20 22:38 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 2/8] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers
2026-10-20 22:41 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 3/8] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
2026-10-20 22:57 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 4/8] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-20 22:42 ` Namhyung Kim
2026-10-20 23:02 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 5/8] perf bpf-filter: Add lock_held() sample filter Ian Rogers
2026-10-20 22:49 ` Namhyung Kim
2026-10-20 23:17 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 6/8] perf test: Add tests for lock_held() BPF " Ian Rogers
2026-10-20 23:20 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 7/8] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-20 23:29 ` sashiko-bot
2026-10-20 20:58 ` [PATCH v5 8/8] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-20 23:37 ` sashiko-bot
2026-10-22 20:29 ` [PATCH v6 00/11] perf bpf-filter / lock contention: Add lock_held() sample filter and RCU/SRCU support Ian Rogers
2026-10-22 20:29 ` [PATCH v6 01/11] perf bpf-filter: Fix bpf_link leak by using PERF_EVENT_IOC_SET_BPF Ian Rogers
2026-10-22 20:35 ` sashiko-bot
2026-10-22 22:06 ` Namhyung Kim
2026-10-22 20:29 ` [PATCH v6 02/11] perf bpf-filter: Support BPF sample filters on tracepoint events Ian Rogers
2026-10-22 20:55 ` sashiko-bot
2026-10-22 20:29 ` [PATCH v6 03/11] perf lock contention: Share common BPF and lock type definitions in lock_data.h Ian Rogers
2026-10-22 20:36 ` sashiko-bot
2026-10-22 20:29 ` [PATCH v6 04/11] perf lock contention: Extend get_lock_owner() for rt_mutex and mutex handoff Ian Rogers
2026-10-22 20:41 ` sashiko-bot
2026-10-22 20:29 ` [PATCH v6 05/11] perf bpf-filter: Add lock_held() sample filter for global mutexes Ian Rogers
2026-10-22 20:46 ` sashiko-bot
2026-10-22 22:55 ` Namhyung Kim
2026-10-22 20:29 ` Ian Rogers [this message]
2026-10-22 20:55 ` [PATCH v6 06/11] perf bpf-filter: Extend lock_held() to other kernel lock types sashiko-bot
2026-10-22 22:56 ` Namhyung Kim
2026-10-22 20:29 ` [PATCH v6 07/11] perf bpf-filter: Support per-process and semi-global locks in lock_held() Ian Rogers
2026-10-22 20:49 ` sashiko-bot
2026-10-22 20:30 ` [PATCH v6 08/11] perf bpf-filter: Support slab cache locks (&<name>) " Ian Rogers
2026-10-22 20:37 ` sashiko-bot
2026-10-22 20:30 ` [PATCH v6 09/11] perf test: Add tests for lock_held() BPF sample filter Ian Rogers
2026-10-22 20:37 ` sashiko-bot
2026-10-22 20:30 ` [PATCH v6 10/11] perf bpf-filter: Extend lock_held() to support RCU, Tasks Trace RCU, and SRCU Ian Rogers
2026-10-22 20:44 ` sashiko-bot
2026-10-22 20:30 ` [PATCH v6 11/11] perf lock contention: Extend BPF lock contention to RCU " Ian Rogers
2026-10-22 20:43 ` sashiko-bot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261022203003.1779152-7-irogers@google.com \
--to=irogers@google.com \
--cc=acme@kernel.org \
--cc=adrian.hunter@intel.com \
--cc=ddhariwal@google.com \
--cc=eranian@google.com \
--cc=james.clark@linaro.org \
--cc=jolsa@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
--cc=rcu@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.