[RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent

[RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent

[Andrey Grodzovsky]

This fixes a long-standing issue: kernel.ftrace_enabled=0 silently
disables BPF trampolines (fentry/fexit) and ftrace-based
kprobes/kretprobes. The write succeeds, the hook stops firing with no
error, and re-enabling silently restores it. Livepatch already solved
this for itself via FTRACE_OPS_FL_PERMANENT, which refuses to disable
ftrace while a permanent ops is registered and refuses to register
one while ftrace is disabled[1].

For trampolines this restores a historical property: from 2019-2022
they shared one global direct_ops, marked permanent the same
way[2].It was later lost as a side effect of the 2022
per-trampoline-ops split (patch 1's Fixes tag) and never
restored.[3][4] Kprobes never carried this protection at all, so
for them that is long-standing issue rather than a regression.

Patch 1: trampolines. Patch 2: classic kprobes/kretprobes. Patch 3:
a selftest covering both directions for all four hook types.

P.S
I initially implemented a per-record opt-in flag[5], but dropped it
as over-engineering once I saw the original blanket restriction.

P.P.S
Open question: kprobe.multi/kretprobe.multi/kprobe.session
(fprobe-backed) aren't covered -- return-capturing fprobes share the
function-graph tracer's subops manager with unrelated tracers, so
marking it permanent needs a different, per-record approach. Perhaps
something along the lines of [5].

[1] - https://lore.kernel.org/all/20191016113316.13415-1-mbenes@suse.cz/T/#u
[2] - https://lore.kernel.org/all/20191108213450.032003836@goodmis.org/
[3] - https://lore.kernel.org/all/20220602193706.2607681-1-song@kernel.org/
[4] - https://lore.kernel.org/bpf/20251230145010.103439-1-jolsa@kernel.org/
[5] - https://github.com/kernel-patches/bpf/compare/bpf-next_base...andrey-grodzovsky:bpf:ftrace-permanent-per-record

Andrey Grodzovsky (3):
  bpf: mark trampoline ftrace_ops permanent
  kprobes: mark ftrace-based kprobe ops permanent
  selftests/bpf: add ftrace_permanent test

 kernel/bpf/trampoline.c                       |   4 +
 kernel/kprobes.c                              |   5 +-
 .../bpf/prog_tests/ftrace_permanent.c         | 144 ++++++++++++++++++
 .../selftests/bpf/progs/ftrace_permanent.c    |  43 ++++++
 4 files changed, 194 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c
 create mode 100644 tools/testing/selftests/bpf/progs/ftrace_permanent.c

-- 
2.34.1

[RFC PATCH bpf-next 1/3] bpf: mark trampoline ftrace_ops permanent

[Andrey Grodzovsky]

kernel.ftrace_enabled=0 silently kills BPF trampolines (fentry/fexit):
attach still succeeds, the hook stops firing with no error, and
re-enabling silently restores it with no indication anything happened.

This is a regression, not a new gap: BPF trampolines were already
protected against this the same way livepatch protects itself, via
FTRACE_OPS_FL_PERMANENT on the shared trampoline ftrace_ops. That
protection was silently dropped during a later refactoring and never
restored, so this has been broken for years.

Both the shared direct_ops (CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
arches) and the per-trampoline ops allocated on other arches (arm64,
s390) are BPF-trampoline-private, not shared with any unrelated
subsystem, so restoring the same unconditional flag is safe and needs
no kernel/trace/ftrace.c changes -- its existing generic sysctl gate
and attach-time refusal already scan the ops list and pick this up
for free.

Fixes: 00963a2e75a8 ("bpf: Support bpf_trampoline on functions with IPMODIFY (e.g. livepatch)")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-sonnet-5
Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
---
 kernel/bpf/trampoline.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index 129d07db117e..3d5069091db7 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -224,6 +224,8 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym)
  */
 struct ftrace_ops direct_ops = {
 	.ops_func = bpf_tramp_ftrace_ops_func,
+	/* Same protection livepatch gives its own ftrace_ops. */
+	.flags = FTRACE_OPS_FL_PERMANENT,
 };
 
 static int direct_ops_alloc(struct bpf_trampoline *tr)
@@ -303,6 +305,8 @@ static int direct_ops_alloc(struct bpf_trampoline *tr)
 		return -ENOMEM;
 	tr->fops->private = tr;
 	tr->fops->ops_func = bpf_tramp_ftrace_ops_func;
+	/* See the direct_ops initializer above for why. */
+	tr->fops->flags |= FTRACE_OPS_FL_PERMANENT;
 	return 0;
 }
 
-- 
2.34.1

[RFC PATCH bpf-next 2/3] kprobes: mark ftrace-based kprobe ops permanent

[Andrey Grodzovsky]

kernel.ftrace_enabled=0 silently kills ftrace-based kprobes/kretprobes
the same way it does BPF trampolines: arm succeeds, the probe stops
firing with no error, and re-enabling silently restores it. Unlike
trampolines this is not a regression -- ftrace-based kprobes have
never carried this protection, so the bug is long-standing.

kprobe_ftrace_ops/kprobe_ipmodify_ops are shared only among
ftrace-based kprobe attachers, not with any other tracer, so the same
unconditional FTRACE_OPS_FL_PERMANENT livepatch already uses applies
cleanly here too, with no kernel/trace/ftrace.c changes needed.

Known limitation: kprobe.multi/kretprobe.multi/kprobe.session
(fprobe-backed, kernel/trace/fprobe.c) are not covered. Entry-only
fprobes could take the same fix, but return-capturing fprobes route
through the function-graph tracer's shared subops manager, which is
also used by unrelated tracers (function_graph, irqsoff, wakeup
latency, function profiler) -- marking it permanent would block
ftrace_enabled=0 for those too. Left for a follow-up.

Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-sonnet-5
Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
---
 kernel/kprobes.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index bfc89083daa9..5a54511eee79 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1122,14 +1122,15 @@ static struct kprobe *alloc_aggr_kprobe(struct kprobe *p)
 #endif /* CONFIG_OPTPROBES */
 
 #ifdef CONFIG_KPROBES_ON_FTRACE
+/* Same protection livepatch gives its own ftrace_ops. */
 static struct ftrace_ops kprobe_ftrace_ops __read_mostly = {
 	.func = kprobe_ftrace_handler,
-	.flags = FTRACE_OPS_FL_SAVE_REGS,
+	.flags = FTRACE_OPS_FL_SAVE_REGS | FTRACE_OPS_FL_PERMANENT,
 };
 
 static struct ftrace_ops kprobe_ipmodify_ops __read_mostly = {
 	.func = kprobe_ftrace_handler,
-	.flags = FTRACE_OPS_FL_SAVE_REGS | FTRACE_OPS_FL_IPMODIFY,
+	.flags = FTRACE_OPS_FL_SAVE_REGS | FTRACE_OPS_FL_IPMODIFY | FTRACE_OPS_FL_PERMANENT,
 };
 
 static int kprobe_ipmodify_enabled;
-- 
2.34.1

[RFC PATCH bpf-next 3/3] selftests/bpf: add ftrace_permanent test

[Andrey Grodzovsky]

Cover the FTRACE_OPS_FL_PERMANENT fix on BPF trampolines and
ftrace-based kprobes/kretprobes: attach a fentry, fexit, kprobe, and
kretprobe program in turn and confirm kernel.ftrace_enabled=0 is
refused with EBUSY while attached, then succeeds once detached. Also
confirm a new fentry/kprobe attach is itself refused while
ftrace_enabled=0 is already in effect.

kprobe.multi/kretprobe.multi/kprobe.session are intentionally not
covered, matching the production change's scope.

Skip the kprobe/kretprobe subtests when CONFIG_KPROBES_ON_FTRACE is
off (e.g. arm64), read back via skel->kconfig->CONFIG_KPROBES_ON_FTRACE
(same __kconfig idiom as progs/test_fill_link_info.c).

Assisted-by: Claude:claude-sonnet-5
Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
---
 .../bpf/prog_tests/ftrace_permanent.c         | 144 ++++++++++++++++++
 .../selftests/bpf/progs/ftrace_permanent.c    |  43 ++++++
 2 files changed, 187 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c
 create mode 100644 tools/testing/selftests/bpf/progs/ftrace_permanent.c

diff --git a/tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c b/tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c
new file mode 100644
index 000000000000..dbe78009a291
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c
@@ -0,0 +1,144 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 CrowdStrike */
+#include <test_progs.h>
+#include "ftrace_permanent.skel.h"
+
+/*
+ * fentry/fexit/kprobe/kretprobe now carry FTRACE_OPS_FL_PERMANENT, so
+ * kernel.ftrace_enabled=0 must be refused with EBUSY while any of them
+ * is attached, and allowed again once detached.
+ *
+ * kprobe.multi/kretprobe.multi/kprobe.session are out of scope.
+ * kprobe/kretprobe subtests need CONFIG_KPROBES_ON_FTRACE (e.g. not on
+ * arm64) and are skipped otherwise.
+ */
+
+#define FTRACE_ENABLED_PATH "/proc/sys/kernel/ftrace_enabled"
+
+static int read_ftrace_enabled(int *val)
+{
+	char buf[16] = {};
+	int fd, n;
+
+	fd = open(FTRACE_ENABLED_PATH, O_RDONLY);
+	if (fd < 0)
+		return -errno;
+	n = read(fd, buf, sizeof(buf) - 1);
+	close(fd);
+	if (n <= 0)
+		return -EIO;
+	*val = atoi(buf);
+	return 0;
+}
+
+/* Returns 0 on success, or -errno on write failure. */
+static int write_ftrace_enabled(int val)
+{
+	char buf[4];
+	int fd, n, len, err = 0;
+
+	fd = open(FTRACE_ENABLED_PATH, O_WRONLY);
+	if (fd < 0)
+		return -errno;
+	len = snprintf(buf, sizeof(buf), "%d", val);
+	n = write(fd, buf, len);
+	if (n < 0)
+		err = -errno;
+	close(fd);
+	return err;
+}
+
+/*
+ * Attach @prog, assert kernel.ftrace_enabled=0 is refused while attached
+ * and stays at 1, then detach and assert the disable now succeeds.
+ */
+static void check_blocks_disable(struct bpf_program *prog, const char *name)
+{
+	struct bpf_link *link;
+	int val, err;
+
+	link = bpf_program__attach(prog);
+	if (!ASSERT_OK_PTR(link, name))
+		return;
+
+	err = write_ftrace_enabled(0);
+	ASSERT_EQ(err, -EBUSY, "disable_refused");
+	if (!ASSERT_OK(read_ftrace_enabled(&val), "read_back"))
+		goto detach;
+	ASSERT_EQ(val, 1, "still_enabled");
+
+detach:
+	bpf_link__destroy(link);
+
+	ASSERT_OK(write_ftrace_enabled(0), "disable_after_detach");
+	ASSERT_OK(write_ftrace_enabled(1), "reenable");
+}
+
+/* Attach @prog while ftrace_enabled=0 and assert it is refused. */
+static void check_attach_while_disabled_refused(struct bpf_program *prog, const char *name)
+{
+	struct bpf_link *link;
+
+	if (!ASSERT_OK(write_ftrace_enabled(0), "disable"))
+		return;
+
+	link = bpf_program__attach(prog);
+	if (!ASSERT_ERR_PTR(link, name))
+		bpf_link__destroy(link);
+
+	ASSERT_OK(write_ftrace_enabled(1), "reenable");
+}
+
+void test_ftrace_permanent(void)
+{
+	struct ftrace_permanent *skel;
+	bool kprobes_on_ftrace;
+	int orig = 1;
+
+	/* Save and always restore ftrace_enabled. */
+	if (read_ftrace_enabled(&orig)) {
+		test__skip();
+		return;
+	}
+
+	skel = ftrace_permanent__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "skel_open_and_load"))
+		goto restore;
+
+	kprobes_on_ftrace = skel->kconfig->CONFIG_KPROBES_ON_FTRACE;
+
+	if (test__start_subtest("fentry_blocks_disable"))
+		check_blocks_disable(skel->progs.test_fentry, "attach_fentry");
+
+	if (test__start_subtest("fexit_blocks_disable"))
+		check_blocks_disable(skel->progs.test_fexit, "attach_fexit");
+
+	if (test__start_subtest("kprobe_blocks_disable")) {
+		if (kprobes_on_ftrace)
+			check_blocks_disable(skel->progs.test_kprobe, "attach_kprobe");
+		else
+			test__skip();
+	}
+
+	if (test__start_subtest("kretprobe_blocks_disable")) {
+		if (kprobes_on_ftrace)
+			check_blocks_disable(skel->progs.test_kretprobe, "attach_kretprobe");
+		else
+			test__skip();
+	}
+
+	if (test__start_subtest("fentry_attach_while_disabled_refused"))
+		check_attach_while_disabled_refused(skel->progs.test_fentry, "attach_fentry");
+
+	if (test__start_subtest("kprobe_attach_while_disabled_refused")) {
+		if (kprobes_on_ftrace)
+			check_attach_while_disabled_refused(skel->progs.test_kprobe,
+							    "attach_kprobe");
+		else
+			test__skip();
+	}
+
+	ftrace_permanent__destroy(skel);
+restore:
+	write_ftrace_enabled(orig);
+}
diff --git a/tools/testing/selftests/bpf/progs/ftrace_permanent.c b/tools/testing/selftests/bpf/progs/ftrace_permanent.c
new file mode 100644
index 000000000000..ca706d8edc23
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/ftrace_permanent.c
@@ -0,0 +1,43 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 CrowdStrike */
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <stdbool.h>
+
+char _license[] SEC("license") = "GPL";
+
+extern bool CONFIG_KPROBES_ON_FTRACE __kconfig __weak;
+
+/* This function is here to have CONFIG_KPROBES_ON_FTRACE used and
+ * added to object BTF, so the userspace side can read it back via
+ * skel->kconfig->CONFIG_KPROBES_ON_FTRACE.
+ */
+int unused(void)
+{
+	return CONFIG_KPROBES_ON_FTRACE ? 0 : 1;
+}
+
+SEC("fentry/bpf_fentry_test1")
+int BPF_PROG(test_fentry, int a)
+{
+	return 0;
+}
+
+SEC("fexit/bpf_fentry_test2")
+int BPF_PROG(test_fexit, int a, __u64 b)
+{
+	return 0;
+}
+
+SEC("kprobe/bpf_fentry_test3")
+int test_kprobe(struct pt_regs *ctx)
+{
+	return 0;
+}
+
+SEC("kretprobe/bpf_fentry_test4")
+int BPF_KRETPROBE(test_kretprobe)
+{
+	return 0;
+}
-- 
2.34.1

Re: [RFC PATCH bpf-next 3/3] selftests/bpf: add ftrace_permanent test

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Medium] The test dynamically alters a global system-wide setting (`kernel.ftrace_enabled`) but is not marked as a serial test, which will cause flaky test failures during default parallel execution.
- [Medium] The test does not explicitly ensure `kernel.ftrace_enabled = 1` after reading its original state, causing guaranteed test failures if the testing environment starts with ftrace disabled.
- [Low] A multi-line comment violates the BPF subsystem's mandatory comment style.
--

--- Patch [3]: [RFC PATCH bpf-next 3/3] selftests/bpf: add ftrace_permanent test ---
commit 16e4b9c3d2d63207f5f4817fc1dedd224a76749e
Author: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>

selftests/bpf: add ftrace_permanent test

This patch introduces a selftest to verify that kernel.ftrace_enabled=0 is 
refused with EBUSY when fentry, fexit, kprobe, or kretprobe BPF programs are 
attached. It confirms that the sysctl is allowed again once the programs are 
detached, and skips kprobes testing on architectures without 
CONFIG_KPROBES_ON_FTRACE.

> diff --git a/tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c b/tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c
> index 0000000000000..dbe78009a2913
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c

[ ... ]

> +void test_ftrace_permanent(void)
> +{

[Severity: Medium]
Since this test dynamically alters a global system-wide setting 
(kernel.ftrace_enabled), could this cause flaky failures for other concurrently 
running BPF selftests? 

Should this function be named serial_test_ftrace_permanent to ensure the test 
runner executes it exclusively?

> +	struct ftrace_permanent *skel;
> +	bool kprobes_on_ftrace;
> +	int orig = 1;
> +
> +	/* Save and always restore ftrace_enabled. */
> +	if (read_ftrace_enabled(&orig)) {
> +		test__skip();
> +		return;
> +	}
> +

[Severity: Medium]
If the testing environment happens to start with ftrace disabled (orig is 0), 
will check_blocks_disable() unconditionally fail when it tries to attach 
programs? 

Does the test need to explicitly set kernel.ftrace_enabled = 1 here before 
proceeding?

> +	skel = ftrace_permanent__open_and_load();

[ ... ]

> diff --git a/tools/testing/selftests/bpf/progs/ftrace_permanent.c b/tools/testing/selftests/bpf/progs/ftrace_permanent.c
> index 0000000000000..ca706d8edc23b
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/ftrace_permanent.c

[ ... ]

> +extern bool CONFIG_KPROBES_ON_FTRACE __kconfig __weak;
> +
> +/* This function is here to have CONFIG_KPROBES_ON_FTRACE used and
> + * added to object BTF, so the userspace side can read it back via
> + * skel->kconfig->CONFIG_KPROBES_ON_FTRACE.
> + */

[Severity: Low]
This isn't a bug, but this multi-line comment has text on the same line as the 
opening /*, which conflicts with the BPF subsystem's required comment style.

> +int unused(void)
> +{
> +	return CONFIG_KPROBES_ON_FTRACE ? 0 : 1;
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729005959.3853865-1-andrey.grodzovsky@crowdstrike.com?part=3

Re: [RFC PATCH bpf-next 2/3] kprobes: mark ftrace-based kprobe ops permanent

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Missing error propagation in `__register_kprobe()` causes silent failures when `arm_kprobe()` fails, returning 0 (success) despite the kprobe failing to arm and being unhashed.
--

--- Patch [2]: [RFC PATCH bpf-next 2/3] kprobes: mark ftrace-based kprobe ops permanent ---
commit 97fc5419ead104f2c26b158adf549fba313680cd
Author: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
Subject: kprobes: mark ftrace-based kprobe ops permanent

This commit adds the FTRACE_OPS_FL_PERMANENT flag to ftrace-based kprobe
operations. It prevents kernel.ftrace_enabled=0 from silently killing
ftrace-based kprobes by allowing them to function independently of the
global ftrace state.

> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
> index bfc89083daa93..5a54511eee791 100644
> --- a/kernel/kprobes.c
> +++ b/kernel/kprobes.c
> @@ -1122,14 +1122,15 @@ static struct kprobe *alloc_aggr_kprobe(struct kprobe *p)
>  #endif /* CONFIG_OPTPROBES */
>  
>  #ifdef CONFIG_KPROBES_ON_FTRACE
> +/* Same protection livepatch gives its own ftrace_ops. */
>  static struct ftrace_ops kprobe_ftrace_ops __read_mostly = {
>  	.func = kprobe_ftrace_handler,
> -	.flags = FTRACE_OPS_FL_SAVE_REGS,
> +	.flags = FTRACE_OPS_FL_SAVE_REGS | FTRACE_OPS_FL_PERMANENT,
>  };

[Severity: High]
This is a pre-existing issue, but does adding this flag now cause a silent 
failure during kprobe registration if ftrace is disabled?

With FTRACE_OPS_FL_PERMANENT, register_ftrace_function() (and therefore
arm_kprobe()) will reliably fail with -EBUSY when ftrace_enabled=0.

However, in __register_kprobe(), the error code from arm_kprobe() is
discarded instead of being propagated to the caller:

kernel/kprobes.c:__register_kprobe() {
	if (!kprobes_all_disarmed && !kprobe_disabled(p)) {
		ret = arm_kprobe(p);
		if (ret) {
			hlist_del_rcu(&p->hlist);
			synchronize_rcu();
		}
	}

	/* Try to optimize kprobe */
	try_to_optimize_kprobe(p);
	return 0;
}

Could this cause the registration API to return 0 (success) even though the 
kprobe fails to arm and is unhashed? 

If the caller later calls unregister_kprobe(), it might fail internally with
-EINVAL because the kprobe is no longer in the hash table, skipping
__unregister_kprobe_bottom() and potentially leaking architecture-specific
resources.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729005959.3853865-1-andrey.grodzovsky@crowdstrike.com?part=2