From: Vineet Gupta <vineet.gupta@linux.dev>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: mhiramat@kernel.org, mark.rutland@arm.com,
mathieu.desnoyers@efficios.com, andrii@kernel.org,
peterz@infradead.org, linux-trace-kernel@vger.kernel.org,
linux-kernel@vger.kernel.org, bpf@vger.kernel.org,
kernel-team@meta.com
Subject: Re: [PATCH 0/2] tracing: fgraph: cut the cost of the shadow stack retry loop
Date: Thu, 24 Sep 2026 19:31:53 -0700 [thread overview]
Message-ID: <1cffebd0-57b3-4e59-8595-37a0e157b2a2@linux.dev> (raw)
In-Reply-To: <20260924164512.451a5d97@fedora>
On 9/24/26 13:45, Steven Rostedt wrote:
> Just an FYI. My wife flew into Europe today and I'll be touring Europe
> with her until Oct 3rd. I may get time to look at patches, but it will
> be very limited. And then LPC starts and who knows when I'll get a
> chance.
>
> Thus I may not be very responsive for the next couple of weeks.
Thx for the reply Steve. I understand it is that time of the year with
travels and such.
No rush, just wanted to push it out of mailbox, to yours :-)
Besides it might all be moot as Peter's solution is much more elegant
and efficient (as always).
I will reply to that but don't feel pressed to respond.
Thx,
-Vineet
>
> -- Steve
>
>
> On Tue, 22 Sep 2026 15:55:24 -0700
> Vineet Gupta <vineet.gupta@linux.dev> wrote:
>
>> Turning on function-graph tracing, or attaching a kprobe_multi return
>> probe, hands every thread a shadow stack. alloc_retstack_tasklist()
>> does that 32 tasks at a time, and since for_each_process_thread() has
>> no cursor, every sweep restarts from init_task and re-walks the tasks
>> already served. Total work is quadratic O(N^2) on thread count.
>>
>> On a 60-core Sapphire Rapids machine with 400000 idle threads the 0 -> 1
>> transition takes 227 s, inside a single bpf() syscall for the kprobe_multi
>> case. On Meta fleet this showed up as RCU stalls and softlockup panics.
>>
>> Patch 1 raises the batch to 1024, dividing the sweeps by 32:
>> 227 s -> 7.2 s at 400000 threads. It helps on every preemption model.
>>
>> Patch 2 adds a cond_resched() between sweeps. It is supplementary and
>> separable: a no-op on current x86 and arm64, but on !CONFIG_PREEMPTION
>> builds it takes soft lockups from 3-of-3 runs to 0-of-3. Dropping it
>> leaves patch 1 intact.
>>
>> Neither changes the O(N^2) shape; a cursor-based walk would, but
>> task_struct lifetime makes that considerably more involved.
>>
>> Vineet Gupta (2):
>> tracing: fgraph: Raise FTRACE_RETSTACK_ALLOC_SIZE to 1024
>> tracing: fgraph: Add a cond_resched() to the shadow stack retry loop
>>
>> include/linux/ftrace.h | 7 ++++++-
>> kernel/trace/fgraph.c | 8 ++++++++
>> 2 files changed, 14 insertions(+), 1 deletion(-)
>>
prev parent reply other threads:[~2026-09-25 2:31 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-09-22 22:55 [PATCH 0/2] tracing: fgraph: cut the cost of the shadow stack retry loop Vineet Gupta
2026-09-22 22:55 ` [PATCH 1/2] tracing: fgraph: Raise FTRACE_RETSTACK_ALLOC_SIZE to 1024 Vineet Gupta
2026-09-24 10:00 ` Peter Zijlstra
2026-09-25 19:32 ` Vineet Gupta
2026-09-26 12:46 ` Steven Rostedt
2026-09-22 22:55 ` [PATCH 2/2] tracing: fgraph: Add a cond_resched() to the shadow stack retry loop Vineet Gupta
2026-09-23 8:47 ` Steven Rostedt
2026-09-24 20:45 ` [PATCH 0/2] tracing: fgraph: cut the cost of " Steven Rostedt
2026-09-25 2:31 ` Vineet Gupta [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1cffebd0-57b3-4e59-8595-37a0e157b2a2@linux.dev \
--to=vineet.gupta@linux.dev \
--cc=andrii@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=kernel-team@meta.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-trace-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mathieu.desnoyers@efficios.com \
--cc=mhiramat@kernel.org \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.