All of lore.kernel.org
 help / color / mirror / Atom feed
From: Steven Rostedt <rostedt@kernel.org>
To: linux-kernel@vger.kernel.org
Cc: Masami Hiramatsu <mhiramat@kernel.org>,
	Mark Rutland <mark.rutland@arm.com>,
	Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Vineet Gupta <vineet.gupta@linux.dev>,
	stable@vger.kernel.org, Andrii Nakryiko <andrii@kernel.org>
Subject: [for-linus][PATCH 13/32] fgraph: Allocate shadow stacks inline with GFP_NOWAIT
Date: Fri, 20 Nov 2026 12:39:05 -0500	[thread overview]
Message-ID: <20261120174116.504279783@kernel.org> (raw)
In-Reply-To: 20261120173852.687189370@kernel.org

From: Vineet Gupta <vineet.gupta@linux.dev>

When ftrace graphing is turned on, all tasks in the system missing
return stack page are assigned one. This is done in a simplistic
multi-sweep loop of FTRACE_RETSTACK_ALLOC_SIZE (currently 32) tasks
at a time as follows:

   start_graph_tracing()
	do {
		alloc_retstack_tasklist
	} while (-EAGAIN);

   alloc_retstack_tasklist()
	alloc x32           # GFP_KERNEL, may sleep
	rcu_read_lock()     # preempt off
		for_each_process_thread    walk N_total, no cond_resched
                     t->ret_stack = new_page
	rcu_read_unlock()   # preempt enable but no explicit yield

Each successive iteration of loop invokes for_each_process_thread()
which doesn't support cursor based resume and always restarts from the
init_task. Thus each successive loop needs to skip the tasks assigned
ret_stack in prior sweeps and thus take longer and longer to find the
candidate 32 tasks. And while the loop end calls rcu unlock,
and re-enables preemption briefly, there is no explicit yield.

The total number of iterations turn out to be

	N_total * N_null / (2 * FTRACE_RETSTACK_ALLOC_SIZE)

where N_total is the number of threads on the system and N_null the number
of those whose ret_stack is still NULL. After boot with no fgraph user that
is every thread.

This is not a tracing-only path. Since commit 4346ba160409 ("fprobe:
Rewrite fprobe on function-graph tracer") fprobe is built on fgraph,
so an ordinary kprobe_multi BPF attach that flips ftrace_graph_active
from 0 to 1 pays the whole cost inside one bpf() syscall. Commit
2c67dc457bc6 ("tracing: fprobe: optimization for entry only case")
narrows that to return and session probes but does not remove it.

A host running hundreds of thousands of threads, with the current batch
size of 32, turns the ftrace_graph_active 0 -> 1 transition into a
multi-second, and eventually multi-minute, operation which showed up as
RCU stall and softlockup_panic on heavily loaded Meta Fleet machines
with preemption disabled.

    rcu: INFO: rcu_sched self-detected stall on CPU
    rcu: 0-....: (20718 ticks this GP) (t=21000 jiffies g=913 q=8 ncpus=8)
    CPU: 0 UID: 0 PID: 160141 Comm: bpftrace
     ___slab_alloc+0x549/0xa20
     kmem_cache_alloc_noprof+0x16c/0x340
     register_ftrace_graph+0x3d7/0x6c0
     register_fprobe_ips+0x2d5/0x310
     bpf_kprobe_multi_link_attach+0x218/0x7e0
     __sys_bpf+0x267a/0x27a0

Fix this by allocating inline with GFP_NOWAIT instead of pre-allocating
a fixed batch. GFP_NOWAIT does not sleep, so it is legal inside the RCU
read-side critical section.

A single sweep now serves every task, so the walk becomes O(N) rather
than O(N_total * N_null), and the retry loop is no longer the normal
path.

The pre-allocated array remains as the fallback when GFP_NOWAIT fails.
The runs below never needed it, so it was exercised with failslab
injection: at a 100% failure rate every task falls back and -EAGAIN
retries ~1560 times for 50000 threads, with no stall or splat.

smp_store_release() is a drop-in replacement for the smp_wmb() and
plain store it replaces.

Measured on a 60-core Sapphire Rapids machine running a PREEMPT_LAZY
kernel (CONFIG_PREEMPT_DYNAMIC=y, mode lazy), timing the write that
drives the ftrace_graph_active 0 -> 1 transition. Threads are spawned
while fgraph is inactive so every one of them has ret_stack == NULL,
and the loop was instrumented to count sweeps:

	threads   before              after
	          sweeps   time       sweeps   time
	100000     3126     15.3 s      1       0.019 s
	200000     6253     59.1 s      1       0.047 s
	400000    12503    227.8 s      1       0.092 s

The sweep count is 1 throughout. Time now scales linearly with the
thread count at roughly 230 ns per task, which is the allocation and
initialisation of one shadow stack.

Fixes: f201ae2356c7 ("tracing/function-return-tracer: store return stack into task_struct and allocate it dynamically")
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Vineet Gupta <vineet.gupta@linux.dev>
Tested-by: Vineet Gupta <vineet.gupta@linux.dev>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20261021174406.2891598-2-vineet.gupta@linux.dev
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 kernel/trace/fgraph.c | 39 ++++++++++++++++++++++++---------------
 1 file changed, 24 insertions(+), 15 deletions(-)

diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c
index 6d87143b3ddb..59c217b06695 100644
--- a/kernel/trace/fgraph.c
+++ b/kernel/trace/fgraph.c
@@ -1033,13 +1033,16 @@ void ftrace_stub_graph(struct ftrace_graph_ret *trace, struct fgraph_ops *gops,
 trace_func_graph_ret_t ftrace_graph_return = ftrace_stub_graph;
 trace_func_graph_ent_t ftrace_graph_entry = ftrace_graph_entry_stub;
 
-/* Try to assign a return stack array on FTRACE_RETSTACK_ALLOC_SIZE tasks. */
+/*
+ * Loop thru all the tasks in the system and assign ret_stack.
+ * The @ret_stack_list is a small pre-allocated fallback, in case
+ * the GFP_NOWAIT allocations here fail.
+ */
 static int alloc_retstack_tasklist(unsigned long **ret_stack_list)
 {
-	int i;
-	int ret = 0;
 	int start = 0, end = FTRACE_RETSTACK_ALLOC_SIZE;
 	struct task_struct *g, *t;
+	int i, ret = 0;
 
 	if (WARN_ON_ONCE(!fgraph_stack_cachep))
 		return -ENOMEM;
@@ -1056,20 +1059,26 @@ static int alloc_retstack_tasklist(unsigned long **ret_stack_list)
 
 	rcu_read_lock();
 	for_each_process_thread(g, t) {
-		if (start == end) {
-			ret = -EAGAIN;
-			goto unlock;
-		}
+		unsigned long *rs;
 
-		if (t->ret_stack == NULL) {
-			atomic_set(&t->trace_overrun, 0);
-			ret_stack_init_task_vars(ret_stack_list[start]);
-			t->curr_ret_stack = 0;
-			t->curr_ret_depth = -1;
-			/* Make sure the tasks see the 0 first: */
-			smp_wmb();
-			t->ret_stack = ret_stack_list[start++];
+		if (t->ret_stack)
+			continue;
+
+		rs = kmem_cache_alloc(fgraph_stack_cachep, GFP_NOWAIT);
+		if (!rs) {
+			if (start == end) {
+				ret = -EAGAIN;
+				goto unlock;
+			}
+			rs = ret_stack_list[start++];
 		}
+
+		atomic_set(&t->trace_overrun, 0);
+		ret_stack_init_task_vars(rs);
+		t->curr_ret_stack = 0;
+		t->curr_ret_depth = -1;
+		/* Make sure the tasks see the 0 first: */
+		smp_store_release(&t->ret_stack, rs);
 	}
 
 unlock:
-- 
2.53.0

  parent reply	other threads:[~2026-11-20 17:41 UTC|newest]

Thread overview: 34+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20 17:38 Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 01/32] ring-buffer: Do not WARN when trace_rb_cpu_prepare() fails to allocate Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 02/32] tracing: Keep copying markers when one instance is stopped Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 03/32] tracing: Restrict perf filters that read kernel strings Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 04/32] tracing/filters: Treat zero-length dynamic strings as empty Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 05/32] tracing: Define common event fields before enabling boot events Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 06/32] tracing: Drop the synthetic event ref when clearing hist triggers Steven Rostedt
2026-11-20 17:38 ` [for-linus][PATCH 07/32] tracing: Avoid undefined shifts for zero histogram map size Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 08/32] tracing/filters: Fix MATCH_END_ONLY regex matching for fixed-size strings Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 09/32] tracing/filters: Handle out-of-range CPU for OP_NE in cpumask comparison Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 10/32] tracing/filters: Check for missing operand after operator in parse_pred() Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 11/32] tracing/filters: Fix false success return in process_system_preds() Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 12/32] tracing: Free tracer resources on instance creation failure Steven Rostedt
2026-11-20 17:39 ` Steven Rostedt [this message]
2026-11-20 17:39 ` [for-linus][PATCH 14/32] fgraph: Use scoped_guard() for the task list walk Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 15/32] fgraph: Read the task->ret_stack with a smp_load_acquire() Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 16/32] tracing/osnoise: Dont start a kthread in timerlat user-space mode Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 17/32] tracing/user_events: Dont free an event that a dynamic event still references Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 18/32] tracing: Wait for marker writers when copy_trace_marker is cleared Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 19/32] tracing: Validate user addresses in faultable buffer copies Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 20/32] fgraph: Do not mark index 0 in every shadow stack frame Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 21/32] ring-buffer: Skip remote reader page swap when on tail Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 22/32] ring-buffer: Check remote reader page commit before meta entries Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 23/32] ring-buffer: Order remote page commit with data and timestamp Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 24/32] ring-buffer: Increment pages_read on remote ring buffer page swap Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 25/32] tracing/hwlat: Do the latency calculations in nanoseconds Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 26/32] tracing: Properly match strings that start with numbers Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 27/32] tracing: Add tests for globbing number-based strings Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 28/32] tracing: Take trace_access_lock() in tracing_buffers_ioctl() Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 29/32] tracing: Do not allow user events to add a system name Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 30/32] ring-buffer: Do not check si_mem_available() before deferred init completes Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 31/32] tracing: Add notrace to event functions that hook to ftrace Steven Rostedt
2026-11-20 17:39 ` [for-linus][PATCH 32/32] tracing: Ignore .function on the function trace event Steven Rostedt
2026-11-20 18:30 ` [PATCH 0/32] tracing: Fixes for v7.4 Steven Rostedt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120174116.504279783@kernel.org \
    --to=rostedt@kernel.org \
    --cc=akpm@linux-foundation.org \
    --cc=andrii@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mark.rutland@arm.com \
    --cc=mathieu.desnoyers@efficios.com \
    --cc=mhiramat@kernel.org \
    --cc=peterz@infradead.org \
    --cc=stable@vger.kernel.org \
    --cc=vineet.gupta@linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.