From: Steven Rostedt <rostedt@kernel.org>
To: linux-kernel@vger.kernel.org
Cc: Masami Hiramatsu <mhiramat@kernel.org>,
Mark Rutland <mark.rutland@arm.com>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Andrew Morton <akpm@linux-foundation.org>,
Peter Zijlstra <peterz@infradead.org>,
Vineet Gupta <vineet.gupta@linux.dev>,
stable@vger.kernel.org, Andrii Nakryiko <andrii@kernel.org>
Subject: [for-linus][PATCH 13/32] fgraph: Allocate shadow stacks inline with GFP_NOWAIT
Date: Fri, 20 Nov 2026 12:39:05 -0500 [thread overview]
Message-ID: <20261120174116.504279783@kernel.org> (raw)
In-Reply-To: 20261120173852.687189370@kernel.org
From: Vineet Gupta <vineet.gupta@linux.dev>
When ftrace graphing is turned on, all tasks in the system missing
return stack page are assigned one. This is done in a simplistic
multi-sweep loop of FTRACE_RETSTACK_ALLOC_SIZE (currently 32) tasks
at a time as follows:
start_graph_tracing()
do {
alloc_retstack_tasklist
} while (-EAGAIN);
alloc_retstack_tasklist()
alloc x32 # GFP_KERNEL, may sleep
rcu_read_lock() # preempt off
for_each_process_thread walk N_total, no cond_resched
t->ret_stack = new_page
rcu_read_unlock() # preempt enable but no explicit yield
Each successive iteration of loop invokes for_each_process_thread()
which doesn't support cursor based resume and always restarts from the
init_task. Thus each successive loop needs to skip the tasks assigned
ret_stack in prior sweeps and thus take longer and longer to find the
candidate 32 tasks. And while the loop end calls rcu unlock,
and re-enables preemption briefly, there is no explicit yield.
The total number of iterations turn out to be
N_total * N_null / (2 * FTRACE_RETSTACK_ALLOC_SIZE)
where N_total is the number of threads on the system and N_null the number
of those whose ret_stack is still NULL. After boot with no fgraph user that
is every thread.
This is not a tracing-only path. Since commit 4346ba160409 ("fprobe:
Rewrite fprobe on function-graph tracer") fprobe is built on fgraph,
so an ordinary kprobe_multi BPF attach that flips ftrace_graph_active
from 0 to 1 pays the whole cost inside one bpf() syscall. Commit
2c67dc457bc6 ("tracing: fprobe: optimization for entry only case")
narrows that to return and session probes but does not remove it.
A host running hundreds of thousands of threads, with the current batch
size of 32, turns the ftrace_graph_active 0 -> 1 transition into a
multi-second, and eventually multi-minute, operation which showed up as
RCU stall and softlockup_panic on heavily loaded Meta Fleet machines
with preemption disabled.
rcu: INFO: rcu_sched self-detected stall on CPU
rcu: 0-....: (20718 ticks this GP) (t=21000 jiffies g=913 q=8 ncpus=8)
CPU: 0 UID: 0 PID: 160141 Comm: bpftrace
___slab_alloc+0x549/0xa20
kmem_cache_alloc_noprof+0x16c/0x340
register_ftrace_graph+0x3d7/0x6c0
register_fprobe_ips+0x2d5/0x310
bpf_kprobe_multi_link_attach+0x218/0x7e0
__sys_bpf+0x267a/0x27a0
Fix this by allocating inline with GFP_NOWAIT instead of pre-allocating
a fixed batch. GFP_NOWAIT does not sleep, so it is legal inside the RCU
read-side critical section.
A single sweep now serves every task, so the walk becomes O(N) rather
than O(N_total * N_null), and the retry loop is no longer the normal
path.
The pre-allocated array remains as the fallback when GFP_NOWAIT fails.
The runs below never needed it, so it was exercised with failslab
injection: at a 100% failure rate every task falls back and -EAGAIN
retries ~1560 times for 50000 threads, with no stall or splat.
smp_store_release() is a drop-in replacement for the smp_wmb() and
plain store it replaces.
Measured on a 60-core Sapphire Rapids machine running a PREEMPT_LAZY
kernel (CONFIG_PREEMPT_DYNAMIC=y, mode lazy), timing the write that
drives the ftrace_graph_active 0 -> 1 transition. Threads are spawned
while fgraph is inactive so every one of them has ret_stack == NULL,
and the loop was instrumented to count sweeps:
threads before after
sweeps time sweeps time
100000 3126 15.3 s 1 0.019 s
200000 6253 59.1 s 1 0.047 s
400000 12503 227.8 s 1 0.092 s
The sweep count is 1 throughout. Time now scales linearly with the
thread count at roughly 230 ns per task, which is the allocation and
initialisation of one shadow stack.
Fixes: f201ae2356c7 ("tracing/function-return-tracer: store return stack into task_struct and allocate it dynamically")
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Vineet Gupta <vineet.gupta@linux.dev>
Tested-by: Vineet Gupta <vineet.gupta@linux.dev>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20261021174406.2891598-2-vineet.gupta@linux.dev
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
kernel/trace/fgraph.c | 39 ++++++++++++++++++++++++---------------
1 file changed, 24 insertions(+), 15 deletions(-)
diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c
index 6d87143b3ddb..59c217b06695 100644
--- a/kernel/trace/fgraph.c
+++ b/kernel/trace/fgraph.c
@@ -1033,13 +1033,16 @@ void ftrace_stub_graph(struct ftrace_graph_ret *trace, struct fgraph_ops *gops,
trace_func_graph_ret_t ftrace_graph_return = ftrace_stub_graph;
trace_func_graph_ent_t ftrace_graph_entry = ftrace_graph_entry_stub;
-/* Try to assign a return stack array on FTRACE_RETSTACK_ALLOC_SIZE tasks. */
+/*
+ * Loop thru all the tasks in the system and assign ret_stack.
+ * The @ret_stack_list is a small pre-allocated fallback, in case
+ * the GFP_NOWAIT allocations here fail.
+ */
static int alloc_retstack_tasklist(unsigned long **ret_stack_list)
{
- int i;
- int ret = 0;
int start = 0, end = FTRACE_RETSTACK_ALLOC_SIZE;
struct task_struct *g, *t;
+ int i, ret = 0;
if (WARN_ON_ONCE(!fgraph_stack_cachep))
return -ENOMEM;
@@ -1056,20 +1059,26 @@ static int alloc_retstack_tasklist(unsigned long **ret_stack_list)
rcu_read_lock();
for_each_process_thread(g, t) {
- if (start == end) {
- ret = -EAGAIN;
- goto unlock;
- }
+ unsigned long *rs;
- if (t->ret_stack == NULL) {
- atomic_set(&t->trace_overrun, 0);
- ret_stack_init_task_vars(ret_stack_list[start]);
- t->curr_ret_stack = 0;
- t->curr_ret_depth = -1;
- /* Make sure the tasks see the 0 first: */
- smp_wmb();
- t->ret_stack = ret_stack_list[start++];
+ if (t->ret_stack)
+ continue;
+
+ rs = kmem_cache_alloc(fgraph_stack_cachep, GFP_NOWAIT);
+ if (!rs) {
+ if (start == end) {
+ ret = -EAGAIN;
+ goto unlock;
+ }
+ rs = ret_stack_list[start++];
}
+
+ atomic_set(&t->trace_overrun, 0);
+ ret_stack_init_task_vars(rs);
+ t->curr_ret_stack = 0;
+ t->curr_ret_depth = -1;
+ /* Make sure the tasks see the 0 first: */
+ smp_store_release(&t->ret_stack, rs);
}
unlock:
--
2.53.0
next prev parent reply other threads:[~2026-11-20 17:41 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 17:38 Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 01/32] ring-buffer: Do not WARN when trace_rb_cpu_prepare() fails to allocate Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 02/32] tracing: Keep copying markers when one instance is stopped Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 03/32] tracing: Restrict perf filters that read kernel strings Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 04/32] tracing/filters: Treat zero-length dynamic strings as empty Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 05/32] tracing: Define common event fields before enabling boot events Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 06/32] tracing: Drop the synthetic event ref when clearing hist triggers Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 07/32] tracing: Avoid undefined shifts for zero histogram map size Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 08/32] tracing/filters: Fix MATCH_END_ONLY regex matching for fixed-size strings Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 09/32] tracing/filters: Handle out-of-range CPU for OP_NE in cpumask comparison Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 10/32] tracing/filters: Check for missing operand after operator in parse_pred() Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 11/32] tracing/filters: Fix false success return in process_system_preds() Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 12/32] tracing: Free tracer resources on instance creation failure Steven Rostedt
2026-11-20 17:39 ` Steven Rostedt [this message]
2026-11-20 17:39 ` [for-linus][PATCH 14/32] fgraph: Use scoped_guard() for the task list walk Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 15/32] fgraph: Read the task->ret_stack with a smp_load_acquire() Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 16/32] tracing/osnoise: Dont start a kthread in timerlat user-space mode Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 17/32] tracing/user_events: Dont free an event that a dynamic event still references Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 18/32] tracing: Wait for marker writers when copy_trace_marker is cleared Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 19/32] tracing: Validate user addresses in faultable buffer copies Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 20/32] fgraph: Do not mark index 0 in every shadow stack frame Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 21/32] ring-buffer: Skip remote reader page swap when on tail Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 22/32] ring-buffer: Check remote reader page commit before meta entries Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 23/32] ring-buffer: Order remote page commit with data and timestamp Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 24/32] ring-buffer: Increment pages_read on remote ring buffer page swap Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 25/32] tracing/hwlat: Do the latency calculations in nanoseconds Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 26/32] tracing: Properly match strings that start with numbers Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 27/32] tracing: Add tests for globbing number-based strings Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 28/32] tracing: Take trace_access_lock() in tracing_buffers_ioctl() Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 29/32] tracing: Do not allow user events to add a system name Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 30/32] ring-buffer: Do not check si_mem_available() before deferred init completes Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 31/32] tracing: Add notrace to event functions that hook to ftrace Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 32/32] tracing: Ignore .function on the function trace event Steven Rostedt
2026-11-20 18:30 ` [PATCH 0/32] tracing: Fixes for v7.4 Steven Rostedt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120174116.504279783@kernel.org \
--to=rostedt@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=andrii@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mathieu.desnoyers@efficios.com \
--cc=mhiramat@kernel.org \
--cc=peterz@infradead.org \
--cc=stable@vger.kernel.org \
--cc=vineet.gupta@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.