[RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent

[RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent

[Andrey Grodzovsky]

This fixes a long-standing issue: kernel.ftrace_enabled=0 silently
disables BPF trampolines (fentry/fexit) and ftrace-based
kprobes/kretprobes. The write succeeds, the hook stops firing with no
error, and re-enabling silently restores it. Livepatch already solved
this for itself via FTRACE_OPS_FL_PERMANENT, which refuses to disable
ftrace while a permanent ops is registered and refuses to register
one while ftrace is disabled[1].

For trampolines this restores a historical property: from 2019-2022
they shared one global direct_ops, marked permanent the same
way[2].It was later lost as a side effect of the 2022
per-trampoline-ops split (patch 1's Fixes tag) and never
restored.[3][4] Kprobes never carried this protection at all, so
for them that is long-standing issue rather than a regression.

Patch 1: trampolines. Patch 2: classic kprobes/kretprobes. Patch 3:
a selftest covering both directions for all four hook types.

P.S
I initially implemented a per-record opt-in flag[5], but dropped it
as over-engineering once I saw the original blanket restriction.

P.P.S
Open question: kprobe.multi/kretprobe.multi/kprobe.session
(fprobe-backed) aren't covered -- return-capturing fprobes share the
function-graph tracer's subops manager with unrelated tracers, so
marking it permanent needs a different, per-record approach. Perhaps
something along the lines of [5].

[1] - https://lore.kernel.org/all/20191016113316.13415-1-mbenes@suse.cz/T/#u
[2] - https://lore.kernel.org/all/20191108213450.032003836@goodmis.org/
[3] - https://lore.kernel.org/all/20220602193706.2607681-1-song@kernel.org/
[4] - https://lore.kernel.org/bpf/20251230145010.103439-1-jolsa@kernel.org/
[5] - https://github.com/kernel-patches/bpf/compare/bpf-next_base...andrey-grodzovsky:bpf:ftrace-permanent-per-record

Andrey Grodzovsky (3):
  bpf: mark trampoline ftrace_ops permanent
  kprobes: mark ftrace-based kprobe ops permanent
  selftests/bpf: add ftrace_permanent test

 kernel/bpf/trampoline.c                       |   4 +
 kernel/kprobes.c                              |   5 +-
 .../bpf/prog_tests/ftrace_permanent.c         | 144 ++++++++++++++++++
 .../selftests/bpf/progs/ftrace_permanent.c    |  43 ++++++
 4 files changed, 194 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c
 create mode 100644 tools/testing/selftests/bpf/progs/ftrace_permanent.c

-- 
2.34.1

[RFC PATCH bpf-next 1/3] bpf: mark trampoline ftrace_ops permanent

[Andrey Grodzovsky]

kernel.ftrace_enabled=0 silently kills BPF trampolines (fentry/fexit):
attach still succeeds, the hook stops firing with no error, and
re-enabling silently restores it with no indication anything happened.

This is a regression, not a new gap: BPF trampolines were already
protected against this the same way livepatch protects itself, via
FTRACE_OPS_FL_PERMANENT on the shared trampoline ftrace_ops. That
protection was silently dropped during a later refactoring and never
restored, so this has been broken for years.

Both the shared direct_ops (CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
arches) and the per-trampoline ops allocated on other arches (arm64,
s390) are BPF-trampoline-private, not shared with any unrelated
subsystem, so restoring the same unconditional flag is safe and needs
no kernel/trace/ftrace.c changes -- its existing generic sysctl gate
and attach-time refusal already scan the ops list and pick this up
for free.

Fixes: 00963a2e75a8 ("bpf: Support bpf_trampoline on functions with IPMODIFY (e.g. livepatch)")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-sonnet-5
Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
---
 kernel/bpf/trampoline.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index 129d07db117e..3d5069091db7 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -224,6 +224,8 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym)
  */
 struct ftrace_ops direct_ops = {
 	.ops_func = bpf_tramp_ftrace_ops_func,
+	/* Same protection livepatch gives its own ftrace_ops. */
+	.flags = FTRACE_OPS_FL_PERMANENT,
 };
 
 static int direct_ops_alloc(struct bpf_trampoline *tr)
@@ -303,6 +305,8 @@ static int direct_ops_alloc(struct bpf_trampoline *tr)
 		return -ENOMEM;
 	tr->fops->private = tr;
 	tr->fops->ops_func = bpf_tramp_ftrace_ops_func;
+	/* See the direct_ops initializer above for why. */
+	tr->fops->flags |= FTRACE_OPS_FL_PERMANENT;
 	return 0;
 }
 
-- 
2.34.1

Re: [RFC PATCH bpf-next 1/3] bpf: mark trampoline ftrace_ops permanent

[Jiri Olsa]

On Tue, Jul 28, 2026 at 08:59:57PM -0400, Andrey Grodzovsky wrote:
> kernel.ftrace_enabled=0 silently kills BPF trampolines (fentry/fexit):
> attach still succeeds, the hook stops firing with no error, and
> re-enabling silently restores it with no indication anything happened.
> 
> This is a regression, not a new gap: BPF trampolines were already
> protected against this the same way livepatch protects itself, via
> FTRACE_OPS_FL_PERMANENT on the shared trampoline ftrace_ops. That
> protection was silently dropped during a later refactoring and never
> restored, so this has been broken for years.
> 
> Both the shared direct_ops (CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
> arches) and the per-trampoline ops allocated on other arches (arm64,
> s390) are BPF-trampoline-private, not shared with any unrelated
> subsystem, so restoring the same unconditional flag is safe and needs
> no kernel/trace/ftrace.c changes -- its existing generic sysctl gate
> and attach-time refusal already scan the ops list and pick this up
> for free.
> 
> Fixes: 00963a2e75a8 ("bpf: Support bpf_trampoline on functions with IPMODIFY (e.g. livepatch)")

hum, we did use it before this commit, I completely missed that, thanks

Acked-by: Jiri Olsa <jolsa@kernel.org>

jirka


> Cc: stable@vger.kernel.org
> Assisted-by: Claude:claude-sonnet-5
> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>

> ---
>  kernel/bpf/trampoline.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
> index 129d07db117e..3d5069091db7 100644
> --- a/kernel/bpf/trampoline.c
> +++ b/kernel/bpf/trampoline.c
> @@ -224,6 +224,8 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym)
>   */
>  struct ftrace_ops direct_ops = {
>  	.ops_func = bpf_tramp_ftrace_ops_func,
> +	/* Same protection livepatch gives its own ftrace_ops. */
> +	.flags = FTRACE_OPS_FL_PERMANENT,
>  };
>  
>  static int direct_ops_alloc(struct bpf_trampoline *tr)
> @@ -303,6 +305,8 @@ static int direct_ops_alloc(struct bpf_trampoline *tr)
>  		return -ENOMEM;
>  	tr->fops->private = tr;
>  	tr->fops->ops_func = bpf_tramp_ftrace_ops_func;
> +	/* See the direct_ops initializer above for why. */
> +	tr->fops->flags |= FTRACE_OPS_FL_PERMANENT;
>  	return 0;
>  }
>  
> -- 
> 2.34.1
> 

[RFC PATCH bpf-next 2/3] kprobes: mark ftrace-based kprobe ops permanent

[Andrey Grodzovsky]

kernel.ftrace_enabled=0 silently kills ftrace-based kprobes/kretprobes
the same way it does BPF trampolines: arm succeeds, the probe stops
firing with no error, and re-enabling silently restores it. Unlike
trampolines this is not a regression -- ftrace-based kprobes have
never carried this protection, so the bug is long-standing.

kprobe_ftrace_ops/kprobe_ipmodify_ops are shared only among
ftrace-based kprobe attachers, not with any other tracer, so the same
unconditional FTRACE_OPS_FL_PERMANENT livepatch already uses applies
cleanly here too, with no kernel/trace/ftrace.c changes needed.

Known limitation: kprobe.multi/kretprobe.multi/kprobe.session
(fprobe-backed, kernel/trace/fprobe.c) are not covered. Entry-only
fprobes could take the same fix, but return-capturing fprobes route
through the function-graph tracer's shared subops manager, which is
also used by unrelated tracers (function_graph, irqsoff, wakeup
latency, function profiler) -- marking it permanent would block
ftrace_enabled=0 for those too. Left for a follow-up.

Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-sonnet-5
Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
---
 kernel/kprobes.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index bfc89083daa9..5a54511eee79 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1122,14 +1122,15 @@ static struct kprobe *alloc_aggr_kprobe(struct kprobe *p)
 #endif /* CONFIG_OPTPROBES */
 
 #ifdef CONFIG_KPROBES_ON_FTRACE
+/* Same protection livepatch gives its own ftrace_ops. */
 static struct ftrace_ops kprobe_ftrace_ops __read_mostly = {
 	.func = kprobe_ftrace_handler,
-	.flags = FTRACE_OPS_FL_SAVE_REGS,
+	.flags = FTRACE_OPS_FL_SAVE_REGS | FTRACE_OPS_FL_PERMANENT,
 };
 
 static struct ftrace_ops kprobe_ipmodify_ops __read_mostly = {
 	.func = kprobe_ftrace_handler,
-	.flags = FTRACE_OPS_FL_SAVE_REGS | FTRACE_OPS_FL_IPMODIFY,
+	.flags = FTRACE_OPS_FL_SAVE_REGS | FTRACE_OPS_FL_IPMODIFY | FTRACE_OPS_FL_PERMANENT,
 };
 
 static int kprobe_ipmodify_enabled;
-- 
2.34.1

[RFC PATCH bpf-next 3/3] selftests/bpf: add ftrace_permanent test

[Andrey Grodzovsky]

Cover the FTRACE_OPS_FL_PERMANENT fix on BPF trampolines and
ftrace-based kprobes/kretprobes: attach a fentry, fexit, kprobe, and
kretprobe program in turn and confirm kernel.ftrace_enabled=0 is
refused with EBUSY while attached, then succeeds once detached. Also
confirm a new fentry/kprobe attach is itself refused while
ftrace_enabled=0 is already in effect.

kprobe.multi/kretprobe.multi/kprobe.session are intentionally not
covered, matching the production change's scope.

Skip the kprobe/kretprobe subtests when CONFIG_KPROBES_ON_FTRACE is
off (e.g. arm64), read back via skel->kconfig->CONFIG_KPROBES_ON_FTRACE
(same __kconfig idiom as progs/test_fill_link_info.c).

Assisted-by: Claude:claude-sonnet-5
Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
---
 .../bpf/prog_tests/ftrace_permanent.c         | 144 ++++++++++++++++++
 .../selftests/bpf/progs/ftrace_permanent.c    |  43 ++++++
 2 files changed, 187 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c
 create mode 100644 tools/testing/selftests/bpf/progs/ftrace_permanent.c

diff --git a/tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c b/tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c
new file mode 100644
index 000000000000..dbe78009a291
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c
@@ -0,0 +1,144 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 CrowdStrike */
+#include <test_progs.h>
+#include "ftrace_permanent.skel.h"
+
+/*
+ * fentry/fexit/kprobe/kretprobe now carry FTRACE_OPS_FL_PERMANENT, so
+ * kernel.ftrace_enabled=0 must be refused with EBUSY while any of them
+ * is attached, and allowed again once detached.
+ *
+ * kprobe.multi/kretprobe.multi/kprobe.session are out of scope.
+ * kprobe/kretprobe subtests need CONFIG_KPROBES_ON_FTRACE (e.g. not on
+ * arm64) and are skipped otherwise.
+ */
+
+#define FTRACE_ENABLED_PATH "/proc/sys/kernel/ftrace_enabled"
+
+static int read_ftrace_enabled(int *val)
+{
+	char buf[16] = {};
+	int fd, n;
+
+	fd = open(FTRACE_ENABLED_PATH, O_RDONLY);
+	if (fd < 0)
+		return -errno;
+	n = read(fd, buf, sizeof(buf) - 1);
+	close(fd);
+	if (n <= 0)
+		return -EIO;
+	*val = atoi(buf);
+	return 0;
+}
+
+/* Returns 0 on success, or -errno on write failure. */
+static int write_ftrace_enabled(int val)
+{
+	char buf[4];
+	int fd, n, len, err = 0;
+
+	fd = open(FTRACE_ENABLED_PATH, O_WRONLY);
+	if (fd < 0)
+		return -errno;
+	len = snprintf(buf, sizeof(buf), "%d", val);
+	n = write(fd, buf, len);
+	if (n < 0)
+		err = -errno;
+	close(fd);
+	return err;
+}
+
+/*
+ * Attach @prog, assert kernel.ftrace_enabled=0 is refused while attached
+ * and stays at 1, then detach and assert the disable now succeeds.
+ */
+static void check_blocks_disable(struct bpf_program *prog, const char *name)
+{
+	struct bpf_link *link;
+	int val, err;
+
+	link = bpf_program__attach(prog);
+	if (!ASSERT_OK_PTR(link, name))
+		return;
+
+	err = write_ftrace_enabled(0);
+	ASSERT_EQ(err, -EBUSY, "disable_refused");
+	if (!ASSERT_OK(read_ftrace_enabled(&val), "read_back"))
+		goto detach;
+	ASSERT_EQ(val, 1, "still_enabled");
+
+detach:
+	bpf_link__destroy(link);
+
+	ASSERT_OK(write_ftrace_enabled(0), "disable_after_detach");
+	ASSERT_OK(write_ftrace_enabled(1), "reenable");
+}
+
+/* Attach @prog while ftrace_enabled=0 and assert it is refused. */
+static void check_attach_while_disabled_refused(struct bpf_program *prog, const char *name)
+{
+	struct bpf_link *link;
+
+	if (!ASSERT_OK(write_ftrace_enabled(0), "disable"))
+		return;
+
+	link = bpf_program__attach(prog);
+	if (!ASSERT_ERR_PTR(link, name))
+		bpf_link__destroy(link);
+
+	ASSERT_OK(write_ftrace_enabled(1), "reenable");
+}
+
+void test_ftrace_permanent(void)
+{
+	struct ftrace_permanent *skel;
+	bool kprobes_on_ftrace;
+	int orig = 1;
+
+	/* Save and always restore ftrace_enabled. */
+	if (read_ftrace_enabled(&orig)) {
+		test__skip();
+		return;
+	}
+
+	skel = ftrace_permanent__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "skel_open_and_load"))
+		goto restore;
+
+	kprobes_on_ftrace = skel->kconfig->CONFIG_KPROBES_ON_FTRACE;
+
+	if (test__start_subtest("fentry_blocks_disable"))
+		check_blocks_disable(skel->progs.test_fentry, "attach_fentry");
+
+	if (test__start_subtest("fexit_blocks_disable"))
+		check_blocks_disable(skel->progs.test_fexit, "attach_fexit");
+
+	if (test__start_subtest("kprobe_blocks_disable")) {
+		if (kprobes_on_ftrace)
+			check_blocks_disable(skel->progs.test_kprobe, "attach_kprobe");
+		else
+			test__skip();
+	}
+
+	if (test__start_subtest("kretprobe_blocks_disable")) {
+		if (kprobes_on_ftrace)
+			check_blocks_disable(skel->progs.test_kretprobe, "attach_kretprobe");
+		else
+			test__skip();
+	}
+
+	if (test__start_subtest("fentry_attach_while_disabled_refused"))
+		check_attach_while_disabled_refused(skel->progs.test_fentry, "attach_fentry");
+
+	if (test__start_subtest("kprobe_attach_while_disabled_refused")) {
+		if (kprobes_on_ftrace)
+			check_attach_while_disabled_refused(skel->progs.test_kprobe,
+							    "attach_kprobe");
+		else
+			test__skip();
+	}
+
+	ftrace_permanent__destroy(skel);
+restore:
+	write_ftrace_enabled(orig);
+}
diff --git a/tools/testing/selftests/bpf/progs/ftrace_permanent.c b/tools/testing/selftests/bpf/progs/ftrace_permanent.c
new file mode 100644
index 000000000000..ca706d8edc23
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/ftrace_permanent.c
@@ -0,0 +1,43 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 CrowdStrike */
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <stdbool.h>
+
+char _license[] SEC("license") = "GPL";
+
+extern bool CONFIG_KPROBES_ON_FTRACE __kconfig __weak;
+
+/* This function is here to have CONFIG_KPROBES_ON_FTRACE used and
+ * added to object BTF, so the userspace side can read it back via
+ * skel->kconfig->CONFIG_KPROBES_ON_FTRACE.
+ */
+int unused(void)
+{
+	return CONFIG_KPROBES_ON_FTRACE ? 0 : 1;
+}
+
+SEC("fentry/bpf_fentry_test1")
+int BPF_PROG(test_fentry, int a)
+{
+	return 0;
+}
+
+SEC("fexit/bpf_fentry_test2")
+int BPF_PROG(test_fexit, int a, __u64 b)
+{
+	return 0;
+}
+
+SEC("kprobe/bpf_fentry_test3")
+int test_kprobe(struct pt_regs *ctx)
+{
+	return 0;
+}
+
+SEC("kretprobe/bpf_fentry_test4")
+int BPF_KRETPROBE(test_kretprobe)
+{
+	return 0;
+}
-- 
2.34.1

Re: [RFC PATCH bpf-next 3/3] selftests/bpf: add ftrace_permanent test

[Jiri Olsa]

On Tue, Jul 28, 2026 at 08:59:59PM -0400, Andrey Grodzovsky wrote:
> Cover the FTRACE_OPS_FL_PERMANENT fix on BPF trampolines and
> ftrace-based kprobes/kretprobes: attach a fentry, fexit, kprobe, and
> kretprobe program in turn and confirm kernel.ftrace_enabled=0 is
> refused with EBUSY while attached, then succeeds once detached. Also
> confirm a new fentry/kprobe attach is itself refused while
> ftrace_enabled=0 is already in effect.
> 
> kprobe.multi/kretprobe.multi/kprobe.session are intentionally not
> covered, matching the production change's scope.
> 
> Skip the kprobe/kretprobe subtests when CONFIG_KPROBES_ON_FTRACE is
> off (e.g. arm64), read back via skel->kconfig->CONFIG_KPROBES_ON_FTRACE
> (same __kconfig idiom as progs/test_fill_link_info.c).
> 
> Assisted-by: Claude:claude-sonnet-5
> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
> ---

hi,
I have some links active already after boot which makes test to fail:

  [root@qemu-1 bpf]# echo 0 > /proc/sys/kernel/ftrace_enabled
  bash: echo: write error: Device or resource busy

  [root@qemu-1 bpf]# ./test_progs -t ftrace_permanent -v
  bpf_testmod.ko is already unloaded.
  Loading bpf_testmod.ko...
  Successfully loaded bpf_testmod.ko.
  test_ftrace_permanent:PASS:skel_open_and_load 0 nsec
  check_blocks_disable:PASS:attach_fentry 0 nsec
  check_blocks_disable:PASS:disable_refused 0 nsec
  check_blocks_disable:PASS:read_back 0 nsec
  check_blocks_disable:PASS:still_enabled 0 nsec
  check_blocks_disable:FAIL:disable_after_detach unexpected error: -16 (errno 16)
  check_blocks_disable:PASS:reenable 0 nsec
  #145/1   ftrace_permanent/fentry_blocks_disable:FAIL
  check_blocks_disable:PASS:attach_fexit 0 nsec
  check_blocks_disable:PASS:disable_refused 0 nsec
  check_blocks_disable:PASS:read_back 0 nsec
  check_blocks_disable:PASS:still_enabled 0 nsec
  check_blocks_disable:FAIL:disable_after_detach unexpected error: -16 (errno 16)
  check_blocks_disable:PASS:reenable 0 nsec
  #145/2   ftrace_permanent/fexit_blocks_disable:FAIL
  check_blocks_disable:PASS:attach_kprobe 0 nsec
  check_blocks_disable:PASS:disable_refused 0 nsec
  check_blocks_disable:PASS:read_back 0 nsec
  check_blocks_disable:PASS:still_enabled 0 nsec
  check_blocks_disable:FAIL:disable_after_detach unexpected error: -16 (errno 16)
  check_blocks_disable:PASS:reenable 0 nsec
  #145/3   ftrace_permanent/kprobe_blocks_disable:FAIL
  check_blocks_disable:PASS:attach_kretprobe 0 nsec
  check_blocks_disable:PASS:disable_refused 0 nsec
  check_blocks_disable:PASS:read_back 0 nsec
  check_blocks_disable:PASS:still_enabled 0 nsec
  check_blocks_disable:FAIL:disable_after_detach unexpected error: -16 (errno 16)
  check_blocks_disable:PASS:reenable 0 nsec
  #145/4   ftrace_permanent/kretprobe_blocks_disable:FAIL
  check_attach_while_disabled_refused:FAIL:disable unexpected error: -16 (errno 16)
  #145/5   ftrace_permanent/fentry_attach_while_disabled_refused:FAIL
  check_attach_while_disabled_refused:FAIL:disable unexpected error: -16 (errno 16)
  #145/6   ftrace_permanent/kprobe_attach_while_disabled_refused:FAIL
  #145     ftrace_permanent:FAIL
  Summary: 0/0 PASSED, 0 SKIPPED, 1 FAILED
  Successfully unloaded bpf_testmod.ko.

jirka


>  .../bpf/prog_tests/ftrace_permanent.c         | 144 ++++++++++++++++++
>  .../selftests/bpf/progs/ftrace_permanent.c    |  43 ++++++
>  2 files changed, 187 insertions(+)
>  create mode 100644 tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c
>  create mode 100644 tools/testing/selftests/bpf/progs/ftrace_permanent.c
> 
> diff --git a/tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c b/tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c
> new file mode 100644
> index 000000000000..dbe78009a291
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/ftrace_permanent.c
> @@ -0,0 +1,144 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/* Copyright (c) 2026 CrowdStrike */
> +#include <test_progs.h>
> +#include "ftrace_permanent.skel.h"
> +
> +/*
> + * fentry/fexit/kprobe/kretprobe now carry FTRACE_OPS_FL_PERMANENT, so
> + * kernel.ftrace_enabled=0 must be refused with EBUSY while any of them
> + * is attached, and allowed again once detached.
> + *
> + * kprobe.multi/kretprobe.multi/kprobe.session are out of scope.
> + * kprobe/kretprobe subtests need CONFIG_KPROBES_ON_FTRACE (e.g. not on
> + * arm64) and are skipped otherwise.
> + */
> +
> +#define FTRACE_ENABLED_PATH "/proc/sys/kernel/ftrace_enabled"
> +
> +static int read_ftrace_enabled(int *val)
> +{
> +	char buf[16] = {};
> +	int fd, n;
> +
> +	fd = open(FTRACE_ENABLED_PATH, O_RDONLY);
> +	if (fd < 0)
> +		return -errno;
> +	n = read(fd, buf, sizeof(buf) - 1);
> +	close(fd);
> +	if (n <= 0)
> +		return -EIO;
> +	*val = atoi(buf);
> +	return 0;
> +}
> +
> +/* Returns 0 on success, or -errno on write failure. */
> +static int write_ftrace_enabled(int val)
> +{
> +	char buf[4];
> +	int fd, n, len, err = 0;
> +
> +	fd = open(FTRACE_ENABLED_PATH, O_WRONLY);
> +	if (fd < 0)
> +		return -errno;
> +	len = snprintf(buf, sizeof(buf), "%d", val);
> +	n = write(fd, buf, len);
> +	if (n < 0)
> +		err = -errno;
> +	close(fd);
> +	return err;
> +}
> +
> +/*
> + * Attach @prog, assert kernel.ftrace_enabled=0 is refused while attached
> + * and stays at 1, then detach and assert the disable now succeeds.
> + */
> +static void check_blocks_disable(struct bpf_program *prog, const char *name)
> +{
> +	struct bpf_link *link;
> +	int val, err;
> +
> +	link = bpf_program__attach(prog);
> +	if (!ASSERT_OK_PTR(link, name))
> +		return;
> +
> +	err = write_ftrace_enabled(0);
> +	ASSERT_EQ(err, -EBUSY, "disable_refused");
> +	if (!ASSERT_OK(read_ftrace_enabled(&val), "read_back"))
> +		goto detach;
> +	ASSERT_EQ(val, 1, "still_enabled");
> +
> +detach:
> +	bpf_link__destroy(link);
> +
> +	ASSERT_OK(write_ftrace_enabled(0), "disable_after_detach");
> +	ASSERT_OK(write_ftrace_enabled(1), "reenable");
> +}
> +
> +/* Attach @prog while ftrace_enabled=0 and assert it is refused. */
> +static void check_attach_while_disabled_refused(struct bpf_program *prog, const char *name)
> +{
> +	struct bpf_link *link;
> +
> +	if (!ASSERT_OK(write_ftrace_enabled(0), "disable"))
> +		return;
> +
> +	link = bpf_program__attach(prog);
> +	if (!ASSERT_ERR_PTR(link, name))
> +		bpf_link__destroy(link);
> +
> +	ASSERT_OK(write_ftrace_enabled(1), "reenable");
> +}
> +
> +void test_ftrace_permanent(void)
> +{
> +	struct ftrace_permanent *skel;
> +	bool kprobes_on_ftrace;
> +	int orig = 1;
> +
> +	/* Save and always restore ftrace_enabled. */
> +	if (read_ftrace_enabled(&orig)) {
> +		test__skip();
> +		return;
> +	}
> +
> +	skel = ftrace_permanent__open_and_load();
> +	if (!ASSERT_OK_PTR(skel, "skel_open_and_load"))
> +		goto restore;
> +
> +	kprobes_on_ftrace = skel->kconfig->CONFIG_KPROBES_ON_FTRACE;
> +
> +	if (test__start_subtest("fentry_blocks_disable"))
> +		check_blocks_disable(skel->progs.test_fentry, "attach_fentry");
> +
> +	if (test__start_subtest("fexit_blocks_disable"))
> +		check_blocks_disable(skel->progs.test_fexit, "attach_fexit");
> +
> +	if (test__start_subtest("kprobe_blocks_disable")) {
> +		if (kprobes_on_ftrace)
> +			check_blocks_disable(skel->progs.test_kprobe, "attach_kprobe");
> +		else
> +			test__skip();
> +	}
> +
> +	if (test__start_subtest("kretprobe_blocks_disable")) {
> +		if (kprobes_on_ftrace)
> +			check_blocks_disable(skel->progs.test_kretprobe, "attach_kretprobe");
> +		else
> +			test__skip();
> +	}
> +
> +	if (test__start_subtest("fentry_attach_while_disabled_refused"))
> +		check_attach_while_disabled_refused(skel->progs.test_fentry, "attach_fentry");
> +
> +	if (test__start_subtest("kprobe_attach_while_disabled_refused")) {
> +		if (kprobes_on_ftrace)
> +			check_attach_while_disabled_refused(skel->progs.test_kprobe,
> +							    "attach_kprobe");
> +		else
> +			test__skip();
> +	}
> +
> +	ftrace_permanent__destroy(skel);
> +restore:
> +	write_ftrace_enabled(orig);
> +}
> diff --git a/tools/testing/selftests/bpf/progs/ftrace_permanent.c b/tools/testing/selftests/bpf/progs/ftrace_permanent.c
> new file mode 100644
> index 000000000000..ca706d8edc23
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/ftrace_permanent.c
> @@ -0,0 +1,43 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/* Copyright (c) 2026 CrowdStrike */
> +#include "vmlinux.h"
> +#include <bpf/bpf_helpers.h>
> +#include <bpf/bpf_tracing.h>
> +#include <stdbool.h>
> +
> +char _license[] SEC("license") = "GPL";
> +
> +extern bool CONFIG_KPROBES_ON_FTRACE __kconfig __weak;
> +
> +/* This function is here to have CONFIG_KPROBES_ON_FTRACE used and
> + * added to object BTF, so the userspace side can read it back via
> + * skel->kconfig->CONFIG_KPROBES_ON_FTRACE.
> + */
> +int unused(void)
> +{
> +	return CONFIG_KPROBES_ON_FTRACE ? 0 : 1;
> +}
> +
> +SEC("fentry/bpf_fentry_test1")
> +int BPF_PROG(test_fentry, int a)
> +{
> +	return 0;
> +}
> +
> +SEC("fexit/bpf_fentry_test2")
> +int BPF_PROG(test_fexit, int a, __u64 b)
> +{
> +	return 0;
> +}
> +
> +SEC("kprobe/bpf_fentry_test3")
> +int test_kprobe(struct pt_regs *ctx)
> +{
> +	return 0;
> +}
> +
> +SEC("kretprobe/bpf_fentry_test4")
> +int BPF_KRETPROBE(test_kretprobe)
> +{
> +	return 0;
> +}
> -- 
> 2.34.1
> 

Re: [RFC PATCH bpf-next 3/3] selftests/bpf: add ftrace_permanent test

[Andrey Grodzovsky]

On Wed, Jul 29, 2026 at 9:23 AM Jiri Olsa <olsajiri@gmail.com> wrote:
>
> On Tue, Jul 28, 2026 at 08:59:59PM -0400, Andrey Grodzovsky wrote:
> > Cover the FTRACE_OPS_FL_PERMANENT fix on BPF trampolines and
> > ftrace-based kprobes/kretprobes: attach a fentry, fexit, kprobe, and
> > kretprobe program in turn and confirm kernel.ftrace_enabled=0 is
> > refused with EBUSY while attached, then succeeds once detached. Also
> > confirm a new fentry/kprobe attach is itself refused while
> > ftrace_enabled=0 is already in effect.
> >
> > kprobe.multi/kretprobe.multi/kprobe.session are intentionally not
> > covered, matching the production change's scope.
> >
> > Skip the kprobe/kretprobe subtests when CONFIG_KPROBES_ON_FTRACE is
> > off (e.g. arm64), read back via skel->kconfig->CONFIG_KPROBES_ON_FTRACE
> > (same __kconfig idiom as progs/test_fill_link_info.c).
> >
> > Assisted-by: Claude:claude-sonnet-5
> > Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
> > ---
>
> hi,
> I have some links active already after boot which makes test to fail:
>
>   [root@qemu-1 bpf]# echo 0 > /proc/sys/kernel/ftrace_enabled
>   bash: echo: write error: Device or resource busy
>
>   [root@qemu-1 bpf]# ./test_progs -t ftrace_permanent -v
>   bpf_testmod.ko is already unloaded.
>   Loading bpf_testmod.ko...
>   Successfully loaded bpf_testmod.ko.
>   test_ftrace_permanent:PASS:skel_open_and_load 0 nsec
>   check_blocks_disable:PASS:attach_fentry 0 nsec
>   check_blocks_disable:PASS:disable_refused 0 nsec
>   check_blocks_disable:PASS:read_back 0 nsec
>   check_blocks_disable:PASS:still_enabled 0 nsec
>   check_blocks_disable:FAIL:disable_after_detach unexpected error: -16 (errno 16)
>   check_blocks_disable:PASS:reenable 0 nsec
>   #145/1   ftrace_permanent/fentry_blocks_disable:FAIL
>   check_blocks_disable:PASS:attach_fexit 0 nsec
>   check_blocks_disable:PASS:disable_refused 0 nsec
>   check_blocks_disable:PASS:read_back 0 nsec
>   check_blocks_disable:PASS:still_enabled 0 nsec
>   check_blocks_disable:FAIL:disable_after_detach unexpected error: -16 (errno 16)
>   check_blocks_disable:PASS:reenable 0 nsec
>   #145/2   ftrace_permanent/fexit_blocks_disable:FAIL
>   check_blocks_disable:PASS:attach_kprobe 0 nsec
>   check_blocks_disable:PASS:disable_refused 0 nsec
>   check_blocks_disable:PASS:read_back 0 nsec
>   check_blocks_disable:PASS:still_enabled 0 nsec
>   check_blocks_disable:FAIL:disable_after_detach unexpected error: -16 (errno 16)
>   check_blocks_disable:PASS:reenable 0 nsec
>   #145/3   ftrace_permanent/kprobe_blocks_disable:FAIL
>   check_blocks_disable:PASS:attach_kretprobe 0 nsec
>   check_blocks_disable:PASS:disable_refused 0 nsec
>   check_blocks_disable:PASS:read_back 0 nsec
>   check_blocks_disable:PASS:still_enabled 0 nsec
>   check_blocks_disable:FAIL:disable_after_detach unexpected error: -16 (errno 16)
>   check_blocks_disable:PASS:reenable 0 nsec
>   #145/4   ftrace_permanent/kretprobe_blocks_disable:FAIL
>   check_attach_while_disabled_refused:FAIL:disable unexpected error: -16 (errno 16)
>   #145/5   ftrace_permanent/fentry_attach_while_disabled_refused:FAIL
>   check_attach_while_disabled_refused:FAIL:disable unexpected error: -16 (errno 16)
>   #145/6   ftrace_permanent/kprobe_attach_while_disabled_refused:FAIL
>   #145     ftrace_permanent:FAIL
>   Summary: 0/0 PASSED, 0 SKIPPED, 1 FAILED
>   Successfully unloaded bpf_testmod.ko.
>
> jirka
>
>

Thank you, Jiri. I will address this once the decision on whether to
proceed with this
route or, just make ftrace_enabled obsolete, is made.

Andrey

Re: [RFC PATCH bpf-next 3/3] selftests/bpf: add ftrace_permanent test

[Steven Rostedt]

On Wed, 29 Jul 2026 13:16:01 -0400
Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:

> Thank you, Jiri. I will address this once the decision on whether to
> proceed with this
> route or, just make ftrace_enabled obsolete, is made.

Note, the ftrace_enabled becoming obsolete *can* be reverted if it breaks
somebody's tool. So still be ready with this patch set.

Also note, I have had success in getting changes that do break user space
by fixing that user space so it's not broken again. But then I have to wait
until all distros (including Debian Stable) have that version of the tool
before I can add the change that broke it.

-- Steve

Re: [RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent

[Steven Rostedt]

On Tue, 28 Jul 2026 20:59:56 -0400
Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:

> This fixes a long-standing issue: kernel.ftrace_enabled=0 silently
> disables BPF trampolines (fentry/fexit) and ftrace-based
> kprobes/kretprobes. The write succeeds, the hook stops firing with no
> error, and re-enabling silently restores it. Livepatch already solved
> this for itself via FTRACE_OPS_FL_PERMANENT, which refuses to disable
> ftrace while a permanent ops is registered and refuses to register
> one while ftrace is disabled[1].

The /proc/sys/kernel/ftrace_enabled was added as a "safety kill switch"
back when ftrace was first added to the kernel. It's addition was namely
there because live runtime modification of kernel code was new and we were
worried about how stable it could be.

Honestly, I would love to get rid of it as today ftrace has proven to be
rather stable. But as it is a user space ABI, I'm not sure what will break
if we do. I wonder if we just make it a nop, and print a message to dmesg
saying:

  "ftrace_enabled no longer does anything. Please report if you need it to."

?

> 
> For trampolines this restores a historical property: from 2019-2022
> they shared one global direct_ops, marked permanent the same
> way[2].It was later lost as a side effect of the 2022
> per-trampoline-ops split (patch 1's Fixes tag) and never
> restored.[3][4] Kprobes never carried this protection at all, so
> for them that is long-standing issue rather than a regression.
> 
> Patch 1: trampolines. Patch 2: classic kprobes/kretprobes. Patch 3:
> a selftest covering both directions for all four hook types.
> 
> P.S
> I initially implemented a per-record opt-in flag[5], but dropped it
> as over-engineering once I saw the original blanket restriction.
> 
> P.P.S
> Open question: kprobe.multi/kretprobe.multi/kprobe.session
> (fprobe-backed) aren't covered -- return-capturing fprobes share the
> function-graph tracer's subops manager with unrelated tracers, so
> marking it permanent needs a different, per-record approach. Perhaps
> something along the lines of [5].

Honestly, I would like to start deprecating that kill switch.

For the series:

Acked-by: Steven Rostedt <rostedt@goodmis.org>

-- Steve

Re: [RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent

[Andrey Grodzovsky]

,

On Wed, Jul 29, 2026 at 10:40 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Tue, 28 Jul 2026 20:59:56 -0400
> Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:
>
> > This fixes a long-standing issue: kernel.ftrace_enabled=0 silently
> > disables BPF trampolines (fentry/fexit) and ftrace-based
> > kprobes/kretprobes. The write succeeds, the hook stops firing with no
> > error, and re-enabling silently restores it. Livepatch already solved
> > this for itself via FTRACE_OPS_FL_PERMANENT, which refuses to disable
> > ftrace while a permanent ops is registered and refuses to register
> > one while ftrace is disabled[1].
>
> The /proc/sys/kernel/ftrace_enabled was added as a "safety kill switch"
> back when ftrace was first added to the kernel. It's addition was namely
> there because live runtime modification of kernel code was new and we were
> worried about how stable it could be.
>
> Honestly, I would love to get rid of it as today ftrace has proven to be
> rather stable. But as it is a user space ABI, I'm not sure what will break
> if we do. I wonder if we just make it a nop, and print a message to dmesg
> saying:
>
>   "ftrace_enabled no longer does anything. Please report if you need it to."
>
> ?

Thanks Steven, if you and/or other community members would approve such
approach I would happily do this instead for a few reasons -

1) Seems like the more correct course of action if idneed this switch
effectively became obsolete in usage instead of piling extra logic to work
around it.

2) As I mentioned in the end of the cover letter, I believe for some cases
such as fsessions/ksessions and multi opts retprobes, this patchset is
not effective and the issue will persist, requiring a more complicated solution
for them anyway if we want watertight resilience.

If you approve this - I can try what you suggested, this would include
making the
knob a NOP with a message, dropping the FTRACE_OPS_FL_PERMANENT
flag and reverting/cleaning what's possible from the original livepatch patchset
that dealt with this issue [1]

One question is what the correct set of tests to run in such a case
would be to verify
we didn't break anything. For this patchset I ran BPF and livepatch
selftest suites, but
I wonder what else would need to be run.

CC Miroslav for Livepatch.

Thanks,
Andrey

[1] - https://lore.kernel.org/all/20191016113316.13415-1-mbenes@suse.cz/T/#u

>
> >
> > For trampolines this restores a historical property: from 2019-2022
> > they shared one global direct_ops, marked permanent the same
> > way[2].It was later lost as a side effect of the 2022
> > per-trampoline-ops split (patch 1's Fixes tag) and never
> > restored.[3][4] Kprobes never carried this protection at all, so
> > for them that is long-standing issue rather than a regression.
> >
> > Patch 1: trampolines. Patch 2: classic kprobes/kretprobes. Patch 3:
> > a selftest covering both directions for all four hook types.
> >
> > P.S
> > I initially implemented a per-record opt-in flag[5], but dropped it
> > as over-engineering once I saw the original blanket restriction.
> >
> > P.P.S
> > Open question: kprobe.multi/kretprobe.multi/kprobe.session
> > (fprobe-backed) aren't covered -- return-capturing fprobes share the
> > function-graph tracer's subops manager with unrelated tracers, so
> > marking it permanent needs a different, per-record approach. Perhaps
> > something along the lines of [5].
>
> Honestly, I would like to start deprecating that kill switch.
>
> For the series:
>
> Acked-by: Steven Rostedt <rostedt@goodmis.org>
>
> -- Steve

Re: [RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent

[Steven Rostedt]

[ Adding Linus in case he cares ]

On Wed, 29 Jul 2026 11:00:31 -0400
Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:

> ,
> 
> On Wed, Jul 29, 2026 at 10:40 AM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > On Tue, 28 Jul 2026 20:59:56 -0400
> > Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:
> >  
> > > This fixes a long-standing issue: kernel.ftrace_enabled=0 silently
> > > disables BPF trampolines (fentry/fexit) and ftrace-based
> > > kprobes/kretprobes. The write succeeds, the hook stops firing with no
> > > error, and re-enabling silently restores it. Livepatch already solved
> > > this for itself via FTRACE_OPS_FL_PERMANENT, which refuses to disable
> > > ftrace while a permanent ops is registered and refuses to register
> > > one while ftrace is disabled[1].  
> >
> > The /proc/sys/kernel/ftrace_enabled was added as a "safety kill switch"
> > back when ftrace was first added to the kernel. It's addition was namely
> > there because live runtime modification of kernel code was new and we were
> > worried about how stable it could be.
> >
> > Honestly, I would love to get rid of it as today ftrace has proven to be
> > rather stable. But as it is a user space ABI, I'm not sure what will break
> > if we do. I wonder if we just make it a nop, and print a message to dmesg
> > saying:
> >
> >   "ftrace_enabled no longer does anything. Please report if you need it to."
> >
> > ?  
> 
> Thanks Steven, if you and/or other community members would approve such
> approach I would happily do this instead for a few reasons -

It's actually if Linus is OK with this approach.

> 
> 1) Seems like the more correct course of action if idneed this switch
> effectively became obsolete in usage instead of piling extra logic to work
> around it.
> 
> 2) As I mentioned in the end of the cover letter, I believe for some cases
> such as fsessions/ksessions and multi opts retprobes, this patchset is
> not effective and the issue will persist, requiring a more complicated solution
> for them anyway if we want watertight resilience.
> 
> If you approve this - I can try what you suggested, this would include
> making the
> knob a NOP with a message, dropping the FTRACE_OPS_FL_PERMANENT
> flag and reverting/cleaning what's possible from the original livepatch patchset
> that dealt with this issue [1]
> 
> One question is what the correct set of tests to run in such a case
> would be to verify
> we didn't break anything. For this patchset I ran BPF and livepatch

Hah! That's the key question isn't it? It's a user space visible knob. I
don't have anything that uses it, but I have no idea of there's a tool out
there that does. Linus's rule is to not break user space. If nothing uses
it, then it's OK to get rid of (as getting rid of it will not break user
space). But, if something uses it for a good reason, then we are stuck with
it. We really will not know until we disable it and somebody complains.

-- Steve



> selftest suites, but
> I wonder what else would need to be run.
> 
> CC Miroslav for Livepatch.
> 
> Thanks,
> Andrey
> 
> [1] - https://lore.kernel.org/all/20191016113316.13415-1-mbenes@suse.cz/T/#u
> 
> >  
> > >
> > > For trampolines this restores a historical property: from 2019-2022
> > > they shared one global direct_ops, marked permanent the same
> > > way[2].It was later lost as a side effect of the 2022
> > > per-trampoline-ops split (patch 1's Fixes tag) and never
> > > restored.[3][4] Kprobes never carried this protection at all, so
> > > for them that is long-standing issue rather than a regression.
> > >
> > > Patch 1: trampolines. Patch 2: classic kprobes/kretprobes. Patch 3:
> > > a selftest covering both directions for all four hook types.
> > >
> > > P.S
> > > I initially implemented a per-record opt-in flag[5], but dropped it
> > > as over-engineering once I saw the original blanket restriction.
> > >
> > > P.P.S
> > > Open question: kprobe.multi/kretprobe.multi/kprobe.session
> > > (fprobe-backed) aren't covered -- return-capturing fprobes share the
> > > function-graph tracer's subops manager with unrelated tracers, so
> > > marking it permanent needs a different, per-record approach. Perhaps
> > > something along the lines of [5].  
> >
> > Honestly, I would like to start deprecating that kill switch.
> >
> > For the series:
> >
> > Acked-by: Steven Rostedt <rostedt@goodmis.org>
> >
> > -- Steve  

Re: [RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent

[Linus Torvalds]

On Wed, 29 Jul 2026 at 08:31, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> > > Honestly, I would love to get rid of it as today ftrace has proven to be
> > > rather stable. But as it is a user space ABI, I'm not sure what will break
> > > if we do. I wonder if we just make it a nop, and print a message to dmesg
> > > saying:
> > >
> > >   "ftrace_enabled no longer does anything. Please report if you need it to."
> > >
> > > ?
> >
> > Thanks Steven, if you and/or other community members would approve such
> > approach I would happily do this instead for a few reasons -
>
> It's actually if Linus is OK with this approach.

The rule is always the same, and I don't understand why there's any
discussion about it because it's been stated a million times: we can
try to make any ABI changes at all, and what matters is whether it
breaks somebodys setup.

If people rely on old behavior, it can't change. And if people don't,
nobody cares. It really is that simple. That is literally the
DEFINITION of "don't break user space".

Whether something is user-*VISIBLE* in theory is entirely immaterial
except in the sense of "be careful, this needs testing and may have to
be reverted". So user-visibility isn't a "don't do it", it's a "do it
carefully, and knowing that you may have to revert immediately if
somebody complains".

So changing user-visible things is potentially a lot of wasted effort,
and needs loving care and follow-up.

So everybody should typically avoid changing user-visible things
because it's potentially painful and developer-intensive, but not
because it can't be done.

All that matters is whether real user workloads regress.

And please make sure that any changed behaviour is really obvious, so
that people don't spend lots of time chasing down *why* something
broke. A kernel printk is generally way too subtle - and no, the
tracing "make the printk extra ugly with crazy ASCII barfics" model is
not an improvement.

It might be better to actually return an error to user space so that
users can see that setting ftrace_enabled to zero failed and it would
show up in user logs. kernel developers may think that kernel logs are
important, but users seldom even think about them. Of course, that can
then cause even worse regressions, so there's a balance here.

End result: don't do user visible changes unless you really really
care. And then you spend the extra effort.

               Linus

Re: [RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent

[Steven Rostedt]

On Wed, 29 Jul 2026 08:47:33 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Wed, 29 Jul 2026 at 08:31, Steven Rostedt <rostedt@goodmis.org> wrote:
> >  
> > > > Honestly, I would love to get rid of it as today ftrace has proven to be
> > > > rather stable. But as it is a user space ABI, I'm not sure what will break
> > > > if we do. I wonder if we just make it a nop, and print a message to dmesg
> > > > saying:
> > > >
> > > >   "ftrace_enabled no longer does anything. Please report if you need it to."
> > > >
> > > > ?  
> > >
> > > Thanks Steven, if you and/or other community members would approve such
> > > approach I would happily do this instead for a few reasons -  
> >
> > It's actually if Linus is OK with this approach.  
> 
> The rule is always the same, and I don't understand why there's any
> discussion about it because it's been stated a million times: we can
> try to make any ABI changes at all, and what matters is whether it
> breaks somebodys setup.
> 
> If people rely on old behavior, it can't change. And if people don't,
> nobody cares. It really is that simple. That is literally the
> DEFINITION of "don't break user space".

That's basically what I said. I was just restating it and not asking if I
was correct. I added you more for the below information you added.

> 
> Whether something is user-*VISIBLE* in theory is entirely immaterial
> except in the sense of "be careful, this needs testing and may have to
> be reverted". So user-visibility isn't a "don't do it", it's a "do it
> carefully, and knowing that you may have to revert immediately if
> somebody complains".

Totally agree.

> 
> So changing user-visible things is potentially a lot of wasted effort,
> and needs loving care and follow-up.
> 
> So everybody should typically avoid changing user-visible things
> because it's potentially painful and developer-intensive, but not
> because it can't be done.
> 
> All that matters is whether real user workloads regress.
> 
> And please make sure that any changed behaviour is really obvious, so
> that people don't spend lots of time chasing down *why* something
> broke. A kernel printk is generally way too subtle - and no, the
> tracing "make the printk extra ugly with crazy ASCII barfics" model is
> not an improvement.
> 
> It might be better to actually return an error to user space so that
> users can see that setting ftrace_enabled to zero failed and it would
> show up in user logs. kernel developers may think that kernel logs are
> important, but users seldom even think about them. Of course, that can
> then cause even worse regressions, so there's a balance here.
> 
> End result: don't do user visible changes unless you really really
> care. And then you spend the extra effort.

Honestly, I don't know of any tooling that would use this kill-switch or
any reason to do so. For live kernel patching and for BPF, it doesn't even
work. Hasn't for some time. It's now a "kill some ftrace but not all".

Hence, again, the switch itself is rather useless.

It now is starting to cause a burden, as there's internal code depending on
ftrace functionality to always be working and can cause some nasty
undefined behavior if someone were to set that kill switch to zero. The
solution is to either play whack-a-mole and make sure all those use cases
have the PERMANENT flag set so the kill-switch does not affect them. Or we
simply disable the kill-switch.

But I like your idea about it returning -ENOSUP if someone tries to set it
to zero. Doing that and the printk should let people know why it didn't
work if they do care.

And yes, we need to be ready to revert it if someone does come back and say
their tooling broke because of it.

Andrey, care to make a patch to do that?

Have setting it to zero return -ENOSUP and print a message saying:

 "The ftrace_enabled file is deprecated and no longer disables ftrace"

-- Steve

Re: [RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent

[Steven Rostedt]

On Wed, 29 Jul 2026 13:32:32 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> Honestly, I don't know of any tooling that would use this kill-switch or
> any reason to do so. For live kernel patching and for BPF, it doesn't even
> work. Hasn't for some time. It's now a "kill some ftrace but not all".
> 
> Hence, again, the switch itself is rather useless.

Also, since the BPF direct trampolines originally had the PERMANENT flag
set, but due to some code updates lost the flag, and if someone were to
turn off ftrace, it would cause bugs with the BPF programs that were using
it, it makes me feel more confident that nothing is using that file to
disable ftrace.

The bug causing BPF programs being disabled by it has been around since 6.0
and there hasn't been any complaints about those BPF programs breaking. It
makes me think there hasn't been tooling that set ftrace_disabled to zero!

[ Distros now use BPF with ftrace
 cat /sys/kernel/tracing/enabled_functions to see if yours is too
]

-- Steve

Re: [RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent

[Andrey Grodzovsky]

On Wed, Jul 29, 2026 at 1:32 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Wed, 29 Jul 2026 08:47:33 -0700
> Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
> > On Wed, 29 Jul 2026 at 08:31, Steven Rostedt <rostedt@goodmis.org> wrote:
> > >
> > > > > Honestly, I would love to get rid of it as today ftrace has proven to be
> > > > > rather stable. But as it is a user space ABI, I'm not sure what will break
> > > > > if we do. I wonder if we just make it a nop, and print a message to dmesg
> > > > > saying:
> > > > >
> > > > >   "ftrace_enabled no longer does anything. Please report if you need it to."
> > > > >
> > > > > ?
> > > >
> > > > Thanks Steven, if you and/or other community members would approve such
> > > > approach I would happily do this instead for a few reasons -
> > >
> > > It's actually if Linus is OK with this approach.
> >
> > The rule is always the same, and I don't understand why there's any
> > discussion about it because it's been stated a million times: we can
> > try to make any ABI changes at all, and what matters is whether it
> > breaks somebodys setup.
> >
> > If people rely on old behavior, it can't change. And if people don't,
> > nobody cares. It really is that simple. That is literally the
> > DEFINITION of "don't break user space".
>
> That's basically what I said. I was just restating it and not asking if I
> was correct. I added you more for the below information you added.
>
> >
> > Whether something is user-*VISIBLE* in theory is entirely immaterial
> > except in the sense of "be careful, this needs testing and may have to
> > be reverted". So user-visibility isn't a "don't do it", it's a "do it
> > carefully, and knowing that you may have to revert immediately if
> > somebody complains".
>
> Totally agree.
>
> >
> > So changing user-visible things is potentially a lot of wasted effort,
> > and needs loving care and follow-up.
> >
> > So everybody should typically avoid changing user-visible things
> > because it's potentially painful and developer-intensive, but not
> > because it can't be done.
> >
> > All that matters is whether real user workloads regress.
> >
> > And please make sure that any changed behaviour is really obvious, so
> > that people don't spend lots of time chasing down *why* something
> > broke. A kernel printk is generally way too subtle - and no, the
> > tracing "make the printk extra ugly with crazy ASCII barfics" model is
> > not an improvement.
> >
> > It might be better to actually return an error to user space so that
> > users can see that setting ftrace_enabled to zero failed and it would
> > show up in user logs. kernel developers may think that kernel logs are
> > important, but users seldom even think about them. Of course, that can
> > then cause even worse regressions, so there's a balance here.
> >
> > End result: don't do user visible changes unless you really really
> > care. And then you spend the extra effort.
>
> Honestly, I don't know of any tooling that would use this kill-switch or
> any reason to do so. For live kernel patching and for BPF, it doesn't even
> work. Hasn't for some time. It's now a "kill some ftrace but not all".
>
> Hence, again, the switch itself is rather useless.
>
> It now is starting to cause a burden, as there's internal code depending on
> ftrace functionality to always be working and can cause some nasty
> undefined behavior if someone were to set that kill switch to zero. The
> solution is to either play whack-a-mole and make sure all those use cases
> have the PERMANENT flag set so the kill-switch does not affect them. Or we
> simply disable the kill-switch.
>
> But I like your idea about it returning -ENOSUP if someone tries to set it
> to zero. Doing that and the printk should let people know why it didn't
> work if they do care.
>
> And yes, we need to be ready to revert it if someone does come back and say
> their tooling broke because of it.
>
> Andrey, care to make a patch to do that?
>
> Have setting it to zero return -ENOSUP and print a message saying:
>
>  "The ftrace_enabled file is deprecated and no longer disables ftrace"
>
> -- Steve

Gladly! As BPF users we can attest that this switch causes pain and
handling complications.

Should I also clean up the entire PERMANENT flag logic we now have and
drop the flag itself?

Thanks,
Andrey

Re: [RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent

[Steven Rostedt]

On Wed, 29 Jul 2026 14:20:17 -0400
Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:

> Should I also clean up the entire PERMANENT flag logic we now have and
> drop the flag itself?

No, because we may need it in the unlikely case that we need to revert the
disabling of ftrace_enabled code.

After so many years, we can drop that flag if nobody complains.

-- Steve

Re: [RFC PATCH bpf-next 0/3] ftrace, kprobes, bpf: mark trampoline/kprobe ftrace_ops permanent

[Andrey Grodzovsky]

On Wed, Jul 29, 2026 at 3:45 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Wed, 29 Jul 2026 14:20:17 -0400
> Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com> wrote:
>
> > Should I also clean up the entire PERMANENT flag logic we now have and
> > drop the flag itself?
>
> No, because we may need it in the unlikely case that we need to revert the
> disabling of ftrace_enabled code.
>
> After so many years, we can drop that flag if nobody complains.
>
> -- Steve

Got you.
Will do.

Andrey