From: Steven Rostedt <rostedt@goodmis.org>
To: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
Cc: bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org,
linux-kselftest@vger.kernel.org, ast@kernel.org,
daniel@iogearbox.net, andrii@kernel.org, song@kernel.org,
jolsa@kernel.org, naveen@kernel.org, davem@davemloft.net,
mhiramat@kernel.org, stable@vger.kernel.org,
linux-open-source@crowdstrike.com, mbenes@suse.cz,
Linus Torvalds <torvalds@linux-foundation.org>
Subject: Re: [RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent
Date: Wed, 29 Jul 2026 11:31:36 -0400 [thread overview]
Message-ID: <20260729113136.32af2a4d@gandalf.local.home> (raw)
In-Reply-To: <CAOu3gNgc0XHScgEf+yWF5Kr8sAm8Fsa4RBuwL0YE2EscSiBPrA@mail.gmail.com>
[ Adding Linus in case he cares ]
On Wed, 29 Jul 2026 11:00:31 -0400
Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:
> ,
>
> On Wed, Jul 29, 2026 at 10:40 AM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > On Tue, 28 Jul 2026 20:59:56 -0400
> > Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:
> >
> > > This fixes a long-standing issue: kernel.ftrace_enabled=0 silently
> > > disables BPF trampolines (fentry/fexit) and ftrace-based
> > > kprobes/kretprobes. The write succeeds, the hook stops firing with no
> > > error, and re-enabling silently restores it. Livepatch already solved
> > > this for itself via FTRACE_OPS_FL_PERMANENT, which refuses to disable
> > > ftrace while a permanent ops is registered and refuses to register
> > > one while ftrace is disabled[1].
> >
> > The /proc/sys/kernel/ftrace_enabled was added as a "safety kill switch"
> > back when ftrace was first added to the kernel. It's addition was namely
> > there because live runtime modification of kernel code was new and we were
> > worried about how stable it could be.
> >
> > Honestly, I would love to get rid of it as today ftrace has proven to be
> > rather stable. But as it is a user space ABI, I'm not sure what will break
> > if we do. I wonder if we just make it a nop, and print a message to dmesg
> > saying:
> >
> > "ftrace_enabled no longer does anything. Please report if you need it to."
> >
> > ?
>
> Thanks Steven, if you and/or other community members would approve such
> approach I would happily do this instead for a few reasons -
It's actually if Linus is OK with this approach.
>
> 1) Seems like the more correct course of action if idneed this switch
> effectively became obsolete in usage instead of piling extra logic to work
> around it.
>
> 2) As I mentioned in the end of the cover letter, I believe for some cases
> such as fsessions/ksessions and multi opts retprobes, this patchset is
> not effective and the issue will persist, requiring a more complicated solution
> for them anyway if we want watertight resilience.
>
> If you approve this - I can try what you suggested, this would include
> making the
> knob a NOP with a message, dropping the FTRACE_OPS_FL_PERMANENT
> flag and reverting/cleaning what's possible from the original livepatch patchset
> that dealt with this issue [1]
>
> One question is what the correct set of tests to run in such a case
> would be to verify
> we didn't break anything. For this patchset I ran BPF and livepatch
Hah! That's the key question isn't it? It's a user space visible knob. I
don't have anything that uses it, but I have no idea of there's a tool out
there that does. Linus's rule is to not break user space. If nothing uses
it, then it's OK to get rid of (as getting rid of it will not break user
space). But, if something uses it for a good reason, then we are stuck with
it. We really will not know until we disable it and somebody complains.
-- Steve
> selftest suites, but
> I wonder what else would need to be run.
>
> CC Miroslav for Livepatch.
>
> Thanks,
> Andrey
>
> [1] - https://lore.kernel.org/all/20191016113316.13415-1-mbenes@suse.cz/T/#u
>
> >
> > >
> > > For trampolines this restores a historical property: from 2019-2022
> > > they shared one global direct_ops, marked permanent the same
> > > way[2].It was later lost as a side effect of the 2022
> > > per-trampoline-ops split (patch 1's Fixes tag) and never
> > > restored.[3][4] Kprobes never carried this protection at all, so
> > > for them that is long-standing issue rather than a regression.
> > >
> > > Patch 1: trampolines. Patch 2: classic kprobes/kretprobes. Patch 3:
> > > a selftest covering both directions for all four hook types.
> > >
> > > P.S
> > > I initially implemented a per-record opt-in flag[5], but dropped it
> > > as over-engineering once I saw the original blanket restriction.
> > >
> > > P.P.S
> > > Open question: kprobe.multi/kretprobe.multi/kprobe.session
> > > (fprobe-backed) aren't covered -- return-capturing fprobes share the
> > > function-graph tracer's subops manager with unrelated tracers, so
> > > marking it permanent needs a different, per-record approach. Perhaps
> > > something along the lines of [5].
> >
> > Honestly, I would like to start deprecating that kill switch.
> >
> > For the series:
> >
> > Acked-by: Steven Rostedt <rostedt@goodmis.org>
> >
> > -- Steve
next prev parent reply other threads:[~2026-07-29 15:31 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-29 0:59 [RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent Andrey Grodzovsky
2026-07-29 0:59 ` [RFC PATCH bpf-next 1/3] bpf: mark trampoline " Andrey Grodzovsky
2026-07-29 13:29 ` Jiri Olsa
2026-07-29 0:59 ` [RFC PATCH bpf-next 2/3] kprobes: mark ftrace-based kprobe ops permanent Andrey Grodzovsky
2026-07-29 0:59 ` [RFC PATCH bpf-next 3/3] selftests/bpf: add ftrace_permanent test Andrey Grodzovsky
2026-07-29 13:23 ` Jiri Olsa
2026-07-29 17:16 ` Andrey Grodzovsky
2026-07-29 17:38 ` Steven Rostedt
2026-07-29 14:41 ` [RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent Steven Rostedt
2026-07-29 15:00 ` Andrey Grodzovsky
2026-07-29 15:31 ` Steven Rostedt [this message]
2026-07-29 15:47 ` Linus Torvalds
2026-07-29 17:32 ` Steven Rostedt
2026-07-29 17:53 ` Steven Rostedt
[not found] ` <CAOu3gNg6kNYp-Sc9Eme6yW7Nu7wV6hZ9=CG0eptROfhQ-H8nFA@mail.gmail.com>
2026-07-29 19:45 ` Steven Rostedt
2026-07-29 19:48 ` Andrey Grodzovsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260729113136.32af2a4d@gandalf.local.home \
--to=rostedt@goodmis.org \
--cc=andrey.grodzovsky@crowdstrike.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=jolsa@kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-open-source@crowdstrike.com \
--cc=linux-trace-kernel@vger.kernel.org \
--cc=mbenes@suse.cz \
--cc=mhiramat@kernel.org \
--cc=naveen@kernel.org \
--cc=song@kernel.org \
--cc=stable@vger.kernel.org \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox