* [PATCH bpf-next 0/2] uprobes: Switch uretprobes_srcu to SRCU-fast-updown @ 2026-07-06 17:27 Puranjay Mohan 2026-07-06 17:27 ` [PATCH bpf-next 1/2] srcu: Add lock guard for srcu_fast_updown flavor Puranjay Mohan 2026-07-06 17:27 ` [PATCH bpf-next 2/2] uprobes: Switch uretprobes_srcu to SRCU-fast-updown Puranjay Mohan 0 siblings, 2 replies; 5+ messages in thread From: Puranjay Mohan @ 2026-07-06 17:27 UTC (permalink / raw) To: Lai Jiangshan, Paul E. McKenney, Josh Triplett, Masami Hiramatsu, Oleg Nesterov, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Alexei Starovoitov, Andrii Nakryiko Cc: Puranjay Mohan, Steven Rostedt, Mathieu Desnoyers, Mark Rutland, Alexander Shishkin, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark, rcu, linux-kernel, linux-trace-kernel, linux-perf-users, bpf uretprobes_srcu currently uses normal SRCU. Normal SRCU issues an smp_mb() on both srcu_read_lock() and srcu_read_unlock(), i.e. two full memory barriers per read-side critical section. For uretprobes this cost is paid on every uretprobe invocation: prepare_uretprobe() takes the read lock that is later dropped on the normal return path (hprobe_finalize()) or from ri_timer()/dup_utask() (hprobe_expire()). Switch uretprobes_srcu to the SRCU-fast-updown flavor. SRCU-fast moves the read-side ordering off the reader and onto the (rare) grace-period side: synchronize_srcu() rides on synchronize_rcu() instead of relying on reader-side smp_mb(). This is a good trade for uretprobes, where reader-side hits vastly outnumber grace periods (uprobe unregistration). The updown variant (rather than plain SRCU-fast) is required because the read lock is acquired in prepare_uretprobe() on the way out to user space and is released only once the return instance is finalized -- from a different context than it was taken: the normal return path (uprobe_handle_trampoline() -> hprobe_finalize()), the timer callback, or the fork path (ri_timer()/dup_utask() -> hprobe_expire()). srcu_down_read_fast()/srcu_up_read_fast() are designed for this semaphore-like, cross-context pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry lockdep read-side tracking that would warn on it -- which is why the old code had to use the raw __srcu_read_lock() here. For the short, same-context sections in ri_timer() and dup_utask(), guard(srcu_fast_updown) is used instead, giving proper lockdep coverage. Patch 1 adds the guard(srcu_fast_updown) definition, following the existing guard(srcu)/guard(srcu_fast) pattern. Patch 2 does the uretprobes_srcu conversion. Note ---- Only uretprobes_srcu is converted; the main uprobe readers (RB-tree lookup and consumer-list iteration) are deliberately left on RCU Tasks Trace. RCU Tasks Trace is already implemented on top of srcu_read_lock_fast(), so the reader-side cost is identical, and it has a nesting fast path that the uprobe -> sleepable-BPF-program call chain relies on (the BPF trampoline takes rcu_read_lock_trace() while uprobes already holds it; the nested acquire is just a counter bump). Converting those readers to a separate srcu_struct would turn one real + one nested lock into two real locks and lose that optimization for no reader-side gain. uretprobes_srcu is different: it uses normal SRCU (not Tasks Trace), its readers are long-lived and cross-context, and it genuinely benefits from dropping the per-reader barriers. Puranjay Mohan (2): srcu: Add lock guard for srcu_fast_updown flavor uprobes: Switch uretprobes_srcu to SRCU-fast-updown include/linux/srcu.h | 7 +++++++ include/linux/uprobes.h | 5 +++-- kernel/events/uprobes.c | 29 +++++++++++++++++------------ 3 files changed, 27 insertions(+), 14 deletions(-) base-commit: 87bfe634b1193db90e5170e1ddbad04a63ef4501 -- 2.53.0-Meta ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH bpf-next 1/2] srcu: Add lock guard for srcu_fast_updown flavor 2026-07-06 17:27 [PATCH bpf-next 0/2] uprobes: Switch uretprobes_srcu to SRCU-fast-updown Puranjay Mohan @ 2026-07-06 17:27 ` Puranjay Mohan 2026-07-06 17:36 ` Paul E. McKenney 2026-07-06 17:27 ` [PATCH bpf-next 2/2] uprobes: Switch uretprobes_srcu to SRCU-fast-updown Puranjay Mohan 1 sibling, 1 reply; 5+ messages in thread From: Puranjay Mohan @ 2026-07-06 17:27 UTC (permalink / raw) To: Lai Jiangshan, Paul E. McKenney, Josh Triplett, Masami Hiramatsu, Oleg Nesterov, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Alexei Starovoitov, Andrii Nakryiko Cc: Puranjay Mohan, Steven Rostedt, Mathieu Desnoyers, Mark Rutland, Alexander Shishkin, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark, rcu, linux-kernel, linux-trace-kernel, linux-perf-users, bpf Add a guard(srcu_fast_updown) definition for scoped SRCU-fast-updown read-side critical sections, following the existing pattern of guard(srcu) and guard(srcu_fast). Signed-off-by: Puranjay Mohan <puranjay@kernel.org> --- include/linux/srcu.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index a54ce9e808b92..72c86d3b23f2e 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -638,4 +638,11 @@ DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct, DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T)) #define class_srcu_fast_notrace_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, _T) +DEFINE_LOCK_GUARD_1(srcu_fast_updown, struct srcu_struct, + _T->scp = srcu_read_lock_fast_updown(_T->lock), + srcu_read_unlock_fast_updown(_T->lock, _T->scp), + struct srcu_ctr __percpu *scp) +DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_updown, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T)) +#define class_srcu_fast_updown_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_updown, _T) + #endif -- 2.53.0-Meta ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH bpf-next 1/2] srcu: Add lock guard for srcu_fast_updown flavor 2026-07-06 17:27 ` [PATCH bpf-next 1/2] srcu: Add lock guard for srcu_fast_updown flavor Puranjay Mohan @ 2026-07-06 17:36 ` Paul E. McKenney 0 siblings, 0 replies; 5+ messages in thread From: Paul E. McKenney @ 2026-07-06 17:36 UTC (permalink / raw) To: Puranjay Mohan Cc: Lai Jiangshan, Josh Triplett, Masami Hiramatsu, Oleg Nesterov, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Alexei Starovoitov, Andrii Nakryiko, Steven Rostedt, Mathieu Desnoyers, Mark Rutland, Alexander Shishkin, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark, rcu, linux-kernel, linux-trace-kernel, linux-perf-users, bpf On Mon, Jul 06, 2026 at 10:27:41AM -0700, Puranjay Mohan wrote: > Add a guard(srcu_fast_updown) definition for scoped > SRCU-fast-updown read-side critical sections, following the > existing pattern of guard(srcu) and guard(srcu_fast). > > Signed-off-by: Puranjay Mohan <puranjay@kernel.org> Reviewed-by: Paul E. McKenney <paulmck@kernel.org> (Or I can take it if you would prefer, but it might be easier and faster for you to send it along with the next patch.) > --- > include/linux/srcu.h | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/include/linux/srcu.h b/include/linux/srcu.h > index a54ce9e808b92..72c86d3b23f2e 100644 > --- a/include/linux/srcu.h > +++ b/include/linux/srcu.h > @@ -638,4 +638,11 @@ DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct, > DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T)) > #define class_srcu_fast_notrace_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, _T) > > +DEFINE_LOCK_GUARD_1(srcu_fast_updown, struct srcu_struct, > + _T->scp = srcu_read_lock_fast_updown(_T->lock), > + srcu_read_unlock_fast_updown(_T->lock, _T->scp), > + struct srcu_ctr __percpu *scp) > +DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_updown, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T)) > +#define class_srcu_fast_updown_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_updown, _T) > + > #endif > -- > 2.53.0-Meta > ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH bpf-next 2/2] uprobes: Switch uretprobes_srcu to SRCU-fast-updown 2026-07-06 17:27 [PATCH bpf-next 0/2] uprobes: Switch uretprobes_srcu to SRCU-fast-updown Puranjay Mohan 2026-07-06 17:27 ` [PATCH bpf-next 1/2] srcu: Add lock guard for srcu_fast_updown flavor Puranjay Mohan @ 2026-07-06 17:27 ` Puranjay Mohan 2026-07-06 17:42 ` sashiko-bot 1 sibling, 1 reply; 5+ messages in thread From: Puranjay Mohan @ 2026-07-06 17:27 UTC (permalink / raw) To: Lai Jiangshan, Paul E. McKenney, Josh Triplett, Masami Hiramatsu, Oleg Nesterov, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim, Alexei Starovoitov, Andrii Nakryiko Cc: Puranjay Mohan, Steven Rostedt, Mathieu Desnoyers, Mark Rutland, Alexander Shishkin, Jiri Olsa, Ian Rogers, Adrian Hunter, James Clark, rcu, linux-kernel, linux-trace-kernel, linux-perf-users, bpf uretprobes_srcu currently uses normal SRCU, which issues two smp_mb() per read lock/unlock pair. This overhead is paid on every uretprobe hit. Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (synchronize_rcu() instead of smp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits. The updown flavor is required because the SRCU read lock is taken in prepare_uretprobe() when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path via uprobe_handle_trampoline() -> hprobe_finalize(), or from ri_timer() (expiry) or dup_utask() (fork) via hprobe_expire(). srcu_down_read_fast() / srcu_up_read_fast() are designed for this acquire-here / release-elsewhere pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry the lockdep read-side tracking that would warn on it. The short, same-context SRCU sections in ri_timer() and dup_utask() (which guard the uprobe against reuse across the hprobe_expire() cmpxchg) instead use guard(srcu_fast_updown) for proper lockdep coverage. Signed-off-by: Puranjay Mohan <puranjay@kernel.org> --- include/linux/uprobes.h | 5 +++-- kernel/events/uprobes.c | 29 +++++++++++++++++------------ 2 files changed, 20 insertions(+), 14 deletions(-) diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index f548fea2adec8..f3b07753c2f3d 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -25,6 +25,7 @@ struct mm_struct; struct inode; struct notifier_block; struct page; +struct srcu_ctr; /* * Allowed return values from uprobe consumer's handler callback @@ -106,7 +107,7 @@ enum hprobe_state { * underlying uprobe is not guaranteed anymore. __UPROBE_DEAD is just an * internal marker and is handled transparently by hprobe_fetch() helper. * - * When uprobe is SRCU-protected, we also record srcu_idx value, necessary for + * When uprobe is SRCU-protected, we also record srcu_scp value, necessary for * SRCU unlocking. * * See hprobe_expire() and hprobe_fetch() for details of race-free uprobe @@ -115,7 +116,7 @@ enum hprobe_state { */ struct hprobe { enum hprobe_state state; - int srcu_idx; + struct srcu_ctr __percpu *srcu_scp; struct uprobe *uprobe; }; diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 4084e926e2844..afa491b0bd3f9 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -54,7 +54,7 @@ static struct mutex uprobes_mmap_mutex[UPROBES_HASH_SZ]; DEFINE_STATIC_PERCPU_RWSEM(dup_mmap_sem); /* Covers return_instance's uprobe lifetime. */ -DEFINE_STATIC_SRCU(uretprobes_srcu); +DEFINE_STATIC_SRCU_FAST_UPDOWN(uretprobes_srcu); /* Have a copy of original instruction */ #define UPROBE_COPY_INSN 0 @@ -707,12 +707,13 @@ static void put_uprobe(struct uprobe *uprobe) } /* Initialize hprobe as SRCU-protected "leased" uprobe */ -static void hprobe_init_leased(struct hprobe *hprobe, struct uprobe *uprobe, int srcu_idx) +static void hprobe_init_leased(struct hprobe *hprobe, struct uprobe *uprobe, + struct srcu_ctr __percpu *srcu_scp) { WARN_ON(!uprobe); hprobe->state = HPROBE_LEASED; hprobe->uprobe = uprobe; - hprobe->srcu_idx = srcu_idx; + hprobe->srcu_scp = srcu_scp; } /* Initialize hprobe as refcounted ("stable") uprobe (uprobe can be NULL). */ @@ -720,7 +721,7 @@ static void hprobe_init_stable(struct hprobe *hprobe, struct uprobe *uprobe) { hprobe->state = uprobe ? HPROBE_STABLE : HPROBE_GONE; hprobe->uprobe = uprobe; - hprobe->srcu_idx = -1; + hprobe->srcu_scp = NULL; } /* @@ -757,7 +758,7 @@ static void hprobe_finalize(struct hprobe *hprobe, enum hprobe_state hstate) { switch (hstate) { case HPROBE_LEASED: - __srcu_read_unlock(&uretprobes_srcu, hprobe->srcu_idx); + srcu_up_read_fast(&uretprobes_srcu, hprobe->srcu_scp); break; case HPROBE_STABLE: put_uprobe(hprobe->uprobe); @@ -829,7 +830,7 @@ static struct uprobe *hprobe_expire(struct hprobe *hprobe, bool get) */ if (try_cmpxchg(&hprobe->state, &hstate, uprobe ? HPROBE_STABLE : HPROBE_GONE)) { /* We won the race, we are the ones to unlock SRCU */ - __srcu_read_unlock(&uretprobes_srcu, hprobe->srcu_idx); + srcu_up_read_fast(&uretprobes_srcu, hprobe->srcu_scp); return get ? get_uprobe(uprobe) : uprobe; } @@ -2045,7 +2046,7 @@ static void ri_timer(struct timer_list *timer) struct return_instance *ri; /* SRCU protects uprobe from reuse for the cmpxchg() inside hprobe_expire(). */ - guard(srcu)(&uretprobes_srcu); + guard(srcu_fast_updown)(&uretprobes_srcu); /* RCU protects return_instance from freeing. */ guard(rcu)(); @@ -2142,7 +2143,7 @@ static int dup_utask(struct task_struct *t, struct uprobe_task *o_utask) t->utask = n_utask; /* protect uprobes from freeing, we'll need try_get_uprobe() them */ - guard(srcu)(&uretprobes_srcu); + guard(srcu_fast_updown)(&uretprobes_srcu); p = &n_utask->return_instances; for (o = o_utask->return_instances; o; o = o->next) { @@ -2254,8 +2255,8 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs, { struct uprobe_task *utask = current->utask; unsigned long orig_ret_vaddr, trampoline_vaddr; + struct srcu_ctr __percpu *srcu_scp; bool chained; - int srcu_idx; if (!get_xol_area()) goto free; @@ -2293,8 +2294,12 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs, orig_ret_vaddr = utask->return_instances->orig_ret_vaddr; } - /* __srcu_read_lock() because SRCU lock survives switch to user space */ - srcu_idx = __srcu_read_lock(&uretprobes_srcu); + /* + * Use srcu_down_read_fast() because the SRCU lock survives a switch to + * user space and can be unlocked from a different context by ri_timer() + * or dup_utask(). + */ + srcu_scp = srcu_down_read_fast(&uretprobes_srcu); ri->func = instruction_pointer(regs); ri->stack = user_stack_pointer(regs); @@ -2303,7 +2308,7 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs, utask->depth++; - hprobe_init_leased(&ri->hprobe, uprobe, srcu_idx); + hprobe_init_leased(&ri->hprobe, uprobe, srcu_scp); ri->next = utask->return_instances; rcu_assign_pointer(utask->return_instances, ri); -- 2.53.0-Meta ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH bpf-next 2/2] uprobes: Switch uretprobes_srcu to SRCU-fast-updown 2026-07-06 17:27 ` [PATCH bpf-next 2/2] uprobes: Switch uretprobes_srcu to SRCU-fast-updown Puranjay Mohan @ 2026-07-06 17:42 ` sashiko-bot 0 siblings, 0 replies; 5+ messages in thread From: sashiko-bot @ 2026-07-06 17:42 UTC (permalink / raw) To: Puranjay Mohan; +Cc: linux-perf-users, bpf Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: Pre-existing issues: - [High] Concurrent hprobe_expire() and hprobe_consume() can cause a use-after-free on uprobe due to misunderstood SRCU semantics. -- commit 86c857f0a70ec453b9e38c36e230b36b91bef460 Author: Puranjay Mohan <puranjay@kernel.org> uprobes: Switch uretprobes_srcu to SRCU-fast-updown uretprobes_srcu currently uses normal SRCU, which issues two smp_mb() per read lock/unlock pair. This overhead is paid on every uretprobe hit. Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side. > diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c > index 4084e926e2844..afa491b0bd3f9 100644 > --- a/kernel/events/uprobes.c > +++ b/kernel/events/uprobes.c [ ... ] > @@ -2045,7 +2046,7 @@ static void ri_timer(struct timer_list *timer) > struct return_instance *ri; > > /* SRCU protects uprobe from reuse for the cmpxchg() inside hprobe_expire(). */ > - guard(srcu)(&uretprobes_srcu); > + guard(srcu_fast_updown)(&uretprobes_srcu); [Severity: High] This isn't a bug introduced by this patch, but does the new SRCU read lock acquired here actually protect the uprobe from being freed? If a uprobe is being unregistered, uprobe_free_deferred() calls call_srcu(&uretprobes_srcu, ...) to free the uprobe. This starts an SRCU grace period that waits only for pre-existing SRCU readers to finish. Concurrently, if ri_timer() or dup_utask() takes this new SRCU read lock, the grace period will not wait for it because the new lock started after the grace period began. If the traced task concurrently executes uprobe_handle_trampoline(), calls hprobe_consume() and hprobe_finalize(), and releases the original SRCU read lock, the grace period can complete and free the uprobe (via call_rcu_tasks_trace()). When ri_timer() then proceeds to call hprobe_expire(), which does: kernel/events/uprobes.c:hprobe_expire() { ... case HPROBE_LEASED: { struct uprobe *uprobe = try_get_uprobe(hprobe->uprobe); ... } Could this lead to a use-after-free on the uprobe? The comment in hprobe_expire() seems to assume the new lock prevents the free: /* * Even if hprobe_consume() or another hprobe_expire() wins * the state update race and unlocks SRCU from under us, we * still have a guarantee that underyling uprobe won't be * freed due to ongoing caller's SRCU lock region, so we can * return it regardless. */ Is this analysis accurate? > /* RCU protects return_instance from freeing. */ > guard(rcu)(); > -- Sashiko AI review · https://sashiko.dev/#/patchset/20260706172744.3920417-1-puranjay@kernel.org?part=2 ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-07-06 17:42 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-07-06 17:27 [PATCH bpf-next 0/2] uprobes: Switch uretprobes_srcu to SRCU-fast-updown Puranjay Mohan 2026-07-06 17:27 ` [PATCH bpf-next 1/2] srcu: Add lock guard for srcu_fast_updown flavor Puranjay Mohan 2026-07-06 17:36 ` Paul E. McKenney 2026-07-06 17:27 ` [PATCH bpf-next 2/2] uprobes: Switch uretprobes_srcu to SRCU-fast-updown Puranjay Mohan 2026-07-06 17:42 ` sashiko-bot
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox