[PATCH bpf v3 1/2] x86/ftrace: Mark jmp mode by clearing bit 63 of trampoline address

[PATCH bpf v3 1/2] x86/ftrace: Mark jmp mode by clearing bit 63 of trampoline address

[Daniel Borkmann]

Since commit 0c3772a8db1f ("x86/ftrace: Implement DYNAMIC_FTRACE_WITH_JMP")
ftrace_call_replace() emits a jmp instead of a call when bit 0 of the
target is set, the mark of a direct trampoline that is entered by a jmp.
ftrace_update_ftrace_func() and arch_ftrace_update_trampoline() use it
too, for the call of the ftrace callback in ftrace_caller, ftrace_regs_caller
and the ftrace trampolines, but a callback is a C function and can start
at an odd address. It does with gcc13 and CONFIG_CC_OPTIMIZE_FOR_SIZE=y:
gcc13 has no -fmin-function-alignment, so CONFIG_CC_HAS_SANE_FUNCTION_ALIGNMENT
is not set, and the -falign-functions=16 used in its place does not apply
to functions optimized for size. Here ftrace_ops_list_func() starts at:

  ffffffff810e0147 T ftrace_ops_list_func

ftrace_regs_call then becomes a jmp to the byte before it:

  ftrace_regs_call ffffffff810374f3: e9 4e 8c 0a 00

That byte is the last one of the preceding function's tail jump, and with
the first two of ftrace_ops_list_func() it decodes as an inc DWORD PTR
[rcx+0x57], which swallows the push %r15 of the prologue. Entered without
a return address and one push short, the list function returns through
the pt_regs that ftrace_regs_caller saved, here to the saved R14:

  BUG: unable to handle page fault for address: ffff888116c29f40
  #PF: supervisor instruction fetch in kernel mode
  RIP: 0010:0xffff888116c29f40
  R13: 0000000000000000 R14: ffff888116c29f40 R15: 0000000000000000
  Call Trace:
   ? bpf_trampoline_6442499918+0x1/0x149
   ? bpf_lsm_file_open+0x5/0x8
   ? bpf_lsm_file_open+0x5/0x8
   ? security_file_open+0x19/0x46
   ? do_dentry_open+0x16c/0x32a

The list function is installed whenever two ftrace_ops are registered,
and a function that two of them trace calls it. A kprobe next to a BPF
trampoline is one such case, and with the single ftrace_ops of the BPF
trampolines every update_ftrace_direct_mod() is another, as it registers
a stub ops for the sites it modifies. Here, detaching one of several BPF
LSM programs from a busy hook crashed the tasks that ran the hook during
the update.

Bit 0 is free only in the direct trampolines that a dyn_ftrace record can
point to, but bit 63 is set in every kernel text address on x86_64. Thus,
mark jmp mode by clearing bit 63 instead, so that ftrace_is_jmp() is false
for any callback, whatever its alignment, and ftrace_regs_caller sets the
bit again where it cleared bit 0 before. A marked address is not canonical,
so a return or indirect branch to one that missed ftrace_jmp_get() faults
instead of running from a byte before its target.

Fixes: 0c3772a8db1f ("x86/ftrace: Implement DYNAMIC_FTRACE_WITH_JMP")
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Cc: Menglong Dong <dongml2@chinatelecom.cn>
---
 v2 -> v3:
   - Removed Kconfig text, rest as-is
 v1 -> v2:
   - Steven suggested as an alternative/more generic fix to go
     the route via bit 63
   - BPF selftest kept as-is

 arch/x86/kernel/ftrace_64.S | 6 +++---
 include/linux/ftrace.h      | 8 +++++---
 2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S
index 62c1c93aa1c6..4b20673a3395 100644
--- a/arch/x86/kernel/ftrace_64.S
+++ b/arch/x86/kernel/ftrace_64.S
@@ -286,9 +286,9 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL)
 	RET
 
 1:
-	testb	$1, %al
-	jz	2f
-	andq $0xfffffffffffffffe, %rax
+	/* See ftrace_jmp_set(), jmp to the trampoline */
+	btsq	$63, %rax
+	jc	2f
 	movq %rax, MCOUNT_REG_SIZE+8(%rsp)
 	restore_mcount_regs
 	/* Restore flags */
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index b041d9142d2b..1c2056a1a68b 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -618,19 +618,21 @@ static inline void arch_ftrace_set_direct_caller(struct ftrace_regs *fregs,
 #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */
 
 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_JMP
+#define FTRACE_JMP_BIT	BIT(BITS_PER_LONG - 1)
+
 static inline bool ftrace_is_jmp(unsigned long addr)
 {
-	return addr & 1;
+	return !(addr & FTRACE_JMP_BIT);
 }
 
 static inline unsigned long ftrace_jmp_set(unsigned long addr)
 {
-	return addr | 1UL;
+	return addr & ~FTRACE_JMP_BIT;
 }
 
 static inline unsigned long ftrace_jmp_get(unsigned long addr)
 {
-	return addr & ~1UL;
+	return addr | FTRACE_JMP_BIT;
 }
 #else
 static inline bool ftrace_is_jmp(unsigned long addr)
-- 
2.43.0

[PATCH bpf v3 2/2] selftests/bpf: Test kprobe and fentry/fexit programs on one function

[Daniel Borkmann]

A kprobe next to a BPF trampoline makes the ftrace site of the function
call ftrace_regs_caller and from there the list of ftrace callbacks,
the path that crashed while ftrace_ops_list_func() started at an odd
address. No test covers a function traced by both. Thus, attach an
fentry program and a kprobe to bpf_fentry_test1(), then a second fentry
program, which modifies the trampoline of the shared site, then an fexit
program, after which ftrace_regs_caller enters the trampoline by a jmp,
then detach the kprobe, after which the site jumps to it directly. After
each step, check that every attached program ran once for the call and
saw its argument, and the fexit program its return value.

Without the previous fix the test crashes the kernel, but only where
ftrace_ops_list_func() is at an odd address (in my test with gcc +
CONFIG_CC_OPTIMIZE_FOR_SIZE=y). With the fix it passes:

  # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t fentry_kprobe_shared
  [...]
  #147     fentry_kprobe_shared:OK
  Summary: 1/0 PASSED, 0 SKIPPED, 0/0 FAILED

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Cc: Menglong Dong <dongml2@chinatelecom.cn>
Cc: Steven Rostedt <rostedt@goodmis.org>
---
 .../bpf/prog_tests/fentry_kprobe_shared.c     | 65 +++++++++++++++++++
 .../bpf/progs/fentry_kprobe_shared.c          | 49 ++++++++++++++
 2 files changed, 114 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/fentry_kprobe_shared.c
 create mode 100644 tools/testing/selftests/bpf/progs/fentry_kprobe_shared.c

diff --git a/tools/testing/selftests/bpf/prog_tests/fentry_kprobe_shared.c b/tools/testing/selftests/bpf/prog_tests/fentry_kprobe_shared.c
new file mode 100644
index 000000000000..751aa49d6106
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/fentry_kprobe_shared.c
@@ -0,0 +1,65 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "fentry_kprobe_shared.skel.h"
+
+static void trigger(struct fentry_kprobe_shared *skel)
+{
+	LIBBPF_OPTS(bpf_test_run_opts, topts);
+	int err;
+
+	err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.fentry1), &topts);
+	ASSERT_OK(err, "test_run");
+}
+
+/*
+ * With a kprobe next to the trampoline, the ftrace site of the function
+ * reaches the trampoline through the list of ftrace callbacks, and so does
+ * the site of a trampoline that is modified to add or remove a program.
+ * With the fexit program, the trampoline is entered by a jmp.
+ */
+void test_fentry_kprobe_shared(void)
+{
+	struct fentry_kprobe_shared *skel;
+
+	skel = fentry_kprobe_shared__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "open_and_load"))
+		return;
+	skel->bss->pid = getpid();
+
+	skel->links.fentry1 = bpf_program__attach(skel->progs.fentry1);
+	if (!ASSERT_OK_PTR(skel->links.fentry1, "attach_fentry1"))
+		goto out;
+	skel->links.kprobe1 = bpf_program__attach(skel->progs.kprobe1);
+	if (!ASSERT_OK_PTR(skel->links.kprobe1, "attach_kprobe1"))
+		goto out;
+	trigger(skel);
+	ASSERT_EQ(skel->bss->fentry1_hits, 1, "fentry1_hits");
+	ASSERT_EQ(skel->bss->kprobe_hits, 1, "kprobe_hits");
+
+	skel->links.fentry2 = bpf_program__attach(skel->progs.fentry2);
+	if (!ASSERT_OK_PTR(skel->links.fentry2, "attach_fentry2"))
+		goto out;
+	trigger(skel);
+	ASSERT_EQ(skel->bss->fentry1_hits, 2, "fentry1_hits");
+	ASSERT_EQ(skel->bss->fentry2_hits, 1, "fentry2_hits");
+	ASSERT_EQ(skel->bss->kprobe_hits, 2, "kprobe_hits");
+
+	skel->links.fexit1 = bpf_program__attach(skel->progs.fexit1);
+	if (!ASSERT_OK_PTR(skel->links.fexit1, "attach_fexit1"))
+		goto out;
+	trigger(skel);
+	ASSERT_EQ(skel->bss->fentry1_hits, 3, "fentry1_hits");
+	ASSERT_EQ(skel->bss->fentry2_hits, 2, "fentry2_hits");
+	ASSERT_EQ(skel->bss->fexit_hits, 1, "fexit_hits");
+	ASSERT_EQ(skel->bss->kprobe_hits, 3, "kprobe_hits");
+
+	bpf_link__destroy(skel->links.kprobe1);
+	skel->links.kprobe1 = NULL;
+	trigger(skel);
+	ASSERT_EQ(skel->bss->fentry1_hits, 4, "fentry1_hits");
+	ASSERT_EQ(skel->bss->fentry2_hits, 3, "fentry2_hits");
+	ASSERT_EQ(skel->bss->fexit_hits, 2, "fexit_hits");
+	ASSERT_EQ(skel->bss->kprobe_hits, 3, "kprobe_hits");
+out:
+	fentry_kprobe_shared__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/fentry_kprobe_shared.c b/tools/testing/selftests/bpf/progs/fentry_kprobe_shared.c
new file mode 100644
index 000000000000..3d16773aada8
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/fentry_kprobe_shared.c
@@ -0,0 +1,49 @@
+// SPDX-License-Identifier: GPL-2.0
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+char _license[] SEC("license") = "GPL";
+
+int pid;
+int fentry1_hits;
+int fentry2_hits;
+int fexit_hits;
+int kprobe_hits;
+
+static bool from_test(int a)
+{
+	return (bpf_get_current_pid_tgid() >> 32) == pid && a == 1;
+}
+
+SEC("fentry/bpf_fentry_test1")
+int BPF_PROG(fentry1, int a)
+{
+	if (from_test(a))
+		fentry1_hits++;
+	return 0;
+}
+
+SEC("fentry/bpf_fentry_test1")
+int BPF_PROG(fentry2, int a)
+{
+	if (from_test(a))
+		fentry2_hits++;
+	return 0;
+}
+
+SEC("fexit/bpf_fentry_test1")
+int BPF_PROG(fexit1, int a, int ret)
+{
+	if (from_test(a) && ret == 2)
+		fexit_hits++;
+	return 0;
+}
+
+SEC("kprobe/bpf_fentry_test1")
+int BPF_KPROBE(kprobe1, int a)
+{
+	if (from_test(a))
+		kprobe_hits++;
+	return 0;
+}
-- 
2.43.0

Re: [PATCH bpf v3 2/2] selftests/bpf: Test kprobe and fentry/fexit programs on one function

[Jiri Olsa]

On Wed, Oct 28, 2026 at 09:23:45PM +0100, Daniel Borkmann wrote:
> A kprobe next to a BPF trampoline makes the ftrace site of the function
> call ftrace_regs_caller and from there the list of ftrace callbacks,
> the path that crashed while ftrace_ops_list_func() started at an odd
> address. No test covers a function traced by both. Thus, attach an
> fentry program and a kprobe to bpf_fentry_test1(), then a second fentry
> program, which modifies the trampoline of the shared site, then an fexit
> program, after which ftrace_regs_caller enters the trampoline by a jmp,
> then detach the kprobe, after which the site jumps to it directly. After
> each step, check that every attached program ran once for the call and
> saw its argument, and the fexit program its return value.
> 
> Without the previous fix the test crashes the kernel, but only where
> ftrace_ops_list_func() is at an odd address (in my test with gcc +
> CONFIG_CC_OPTIMIZE_FOR_SIZE=y). With the fix it passes:
> 
>   # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t fentry_kprobe_shared
>   [...]
>   #147     fentry_kprobe_shared:OK
>   Summary: 1/0 PASSED, 0 SKIPPED, 0/0 FAILED
> 
> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
> Cc: Menglong Dong <dongml2@chinatelecom.cn>
> Cc: Steven Rostedt <rostedt@goodmis.org>

Acked-by: Jiri Olsa <jolsa@kernel.org>

jirka


> ---
>  .../bpf/prog_tests/fentry_kprobe_shared.c     | 65 +++++++++++++++++++
>  .../bpf/progs/fentry_kprobe_shared.c          | 49 ++++++++++++++
>  2 files changed, 114 insertions(+)
>  create mode 100644 tools/testing/selftests/bpf/prog_tests/fentry_kprobe_shared.c
>  create mode 100644 tools/testing/selftests/bpf/progs/fentry_kprobe_shared.c
> 
> diff --git a/tools/testing/selftests/bpf/prog_tests/fentry_kprobe_shared.c b/tools/testing/selftests/bpf/prog_tests/fentry_kprobe_shared.c
> new file mode 100644
> index 000000000000..751aa49d6106
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/fentry_kprobe_shared.c
> @@ -0,0 +1,65 @@
> +// SPDX-License-Identifier: GPL-2.0
> +#include <test_progs.h>
> +#include "fentry_kprobe_shared.skel.h"
> +
> +static void trigger(struct fentry_kprobe_shared *skel)
> +{
> +	LIBBPF_OPTS(bpf_test_run_opts, topts);
> +	int err;
> +
> +	err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.fentry1), &topts);
> +	ASSERT_OK(err, "test_run");
> +}
> +
> +/*
> + * With a kprobe next to the trampoline, the ftrace site of the function
> + * reaches the trampoline through the list of ftrace callbacks, and so does
> + * the site of a trampoline that is modified to add or remove a program.
> + * With the fexit program, the trampoline is entered by a jmp.
> + */
> +void test_fentry_kprobe_shared(void)
> +{
> +	struct fentry_kprobe_shared *skel;
> +
> +	skel = fentry_kprobe_shared__open_and_load();
> +	if (!ASSERT_OK_PTR(skel, "open_and_load"))
> +		return;
> +	skel->bss->pid = getpid();
> +
> +	skel->links.fentry1 = bpf_program__attach(skel->progs.fentry1);
> +	if (!ASSERT_OK_PTR(skel->links.fentry1, "attach_fentry1"))
> +		goto out;
> +	skel->links.kprobe1 = bpf_program__attach(skel->progs.kprobe1);
> +	if (!ASSERT_OK_PTR(skel->links.kprobe1, "attach_kprobe1"))
> +		goto out;
> +	trigger(skel);
> +	ASSERT_EQ(skel->bss->fentry1_hits, 1, "fentry1_hits");
> +	ASSERT_EQ(skel->bss->kprobe_hits, 1, "kprobe_hits");
> +
> +	skel->links.fentry2 = bpf_program__attach(skel->progs.fentry2);
> +	if (!ASSERT_OK_PTR(skel->links.fentry2, "attach_fentry2"))
> +		goto out;
> +	trigger(skel);
> +	ASSERT_EQ(skel->bss->fentry1_hits, 2, "fentry1_hits");
> +	ASSERT_EQ(skel->bss->fentry2_hits, 1, "fentry2_hits");
> +	ASSERT_EQ(skel->bss->kprobe_hits, 2, "kprobe_hits");
> +
> +	skel->links.fexit1 = bpf_program__attach(skel->progs.fexit1);
> +	if (!ASSERT_OK_PTR(skel->links.fexit1, "attach_fexit1"))
> +		goto out;
> +	trigger(skel);
> +	ASSERT_EQ(skel->bss->fentry1_hits, 3, "fentry1_hits");
> +	ASSERT_EQ(skel->bss->fentry2_hits, 2, "fentry2_hits");
> +	ASSERT_EQ(skel->bss->fexit_hits, 1, "fexit_hits");
> +	ASSERT_EQ(skel->bss->kprobe_hits, 3, "kprobe_hits");
> +
> +	bpf_link__destroy(skel->links.kprobe1);
> +	skel->links.kprobe1 = NULL;
> +	trigger(skel);
> +	ASSERT_EQ(skel->bss->fentry1_hits, 4, "fentry1_hits");
> +	ASSERT_EQ(skel->bss->fentry2_hits, 3, "fentry2_hits");
> +	ASSERT_EQ(skel->bss->fexit_hits, 2, "fexit_hits");
> +	ASSERT_EQ(skel->bss->kprobe_hits, 3, "kprobe_hits");
> +out:
> +	fentry_kprobe_shared__destroy(skel);
> +}
> diff --git a/tools/testing/selftests/bpf/progs/fentry_kprobe_shared.c b/tools/testing/selftests/bpf/progs/fentry_kprobe_shared.c
> new file mode 100644
> index 000000000000..3d16773aada8
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/fentry_kprobe_shared.c
> @@ -0,0 +1,49 @@
> +// SPDX-License-Identifier: GPL-2.0
> +#include "vmlinux.h"
> +#include <bpf/bpf_helpers.h>
> +#include <bpf/bpf_tracing.h>
> +
> +char _license[] SEC("license") = "GPL";
> +
> +int pid;
> +int fentry1_hits;
> +int fentry2_hits;
> +int fexit_hits;
> +int kprobe_hits;
> +
> +static bool from_test(int a)
> +{
> +	return (bpf_get_current_pid_tgid() >> 32) == pid && a == 1;
> +}
> +
> +SEC("fentry/bpf_fentry_test1")
> +int BPF_PROG(fentry1, int a)
> +{
> +	if (from_test(a))
> +		fentry1_hits++;
> +	return 0;
> +}
> +
> +SEC("fentry/bpf_fentry_test1")
> +int BPF_PROG(fentry2, int a)
> +{
> +	if (from_test(a))
> +		fentry2_hits++;
> +	return 0;
> +}
> +
> +SEC("fexit/bpf_fentry_test1")
> +int BPF_PROG(fexit1, int a, int ret)
> +{
> +	if (from_test(a) && ret == 2)
> +		fexit_hits++;
> +	return 0;
> +}
> +
> +SEC("kprobe/bpf_fentry_test1")
> +int BPF_KPROBE(kprobe1, int a)
> +{
> +	if (from_test(a))
> +		kprobe_hits++;
> +	return 0;
> +}
> -- 
> 2.43.0
> 

Re: [PATCH bpf v3 1/2] x86/ftrace: Mark jmp mode by clearing bit 63 of trampoline address

[Jiri Olsa]

On Wed, Oct 28, 2026 at 09:23:44PM +0100, Daniel Borkmann wrote:
> Since commit 0c3772a8db1f ("x86/ftrace: Implement DYNAMIC_FTRACE_WITH_JMP")
> ftrace_call_replace() emits a jmp instead of a call when bit 0 of the
> target is set, the mark of a direct trampoline that is entered by a jmp.
> ftrace_update_ftrace_func() and arch_ftrace_update_trampoline() use it
> too, for the call of the ftrace callback in ftrace_caller, ftrace_regs_caller
> and the ftrace trampolines, but a callback is a C function and can start
> at an odd address. It does with gcc13 and CONFIG_CC_OPTIMIZE_FOR_SIZE=y:
> gcc13 has no -fmin-function-alignment, so CONFIG_CC_HAS_SANE_FUNCTION_ALIGNMENT
> is not set, and the -falign-functions=16 used in its place does not apply
> to functions optimized for size. Here ftrace_ops_list_func() starts at:
> 
>   ffffffff810e0147 T ftrace_ops_list_func
> 
> ftrace_regs_call then becomes a jmp to the byte before it:
> 
>   ftrace_regs_call ffffffff810374f3: e9 4e 8c 0a 00
> 
> That byte is the last one of the preceding function's tail jump, and with
> the first two of ftrace_ops_list_func() it decodes as an inc DWORD PTR
> [rcx+0x57], which swallows the push %r15 of the prologue. Entered without
> a return address and one push short, the list function returns through
> the pt_regs that ftrace_regs_caller saved, here to the saved R14:
> 
>   BUG: unable to handle page fault for address: ffff888116c29f40
>   #PF: supervisor instruction fetch in kernel mode
>   RIP: 0010:0xffff888116c29f40
>   R13: 0000000000000000 R14: ffff888116c29f40 R15: 0000000000000000
>   Call Trace:
>    ? bpf_trampoline_6442499918+0x1/0x149
>    ? bpf_lsm_file_open+0x5/0x8
>    ? bpf_lsm_file_open+0x5/0x8
>    ? security_file_open+0x19/0x46
>    ? do_dentry_open+0x16c/0x32a
> 
> The list function is installed whenever two ftrace_ops are registered,
> and a function that two of them trace calls it. A kprobe next to a BPF
> trampoline is one such case, and with the single ftrace_ops of the BPF
> trampolines every update_ftrace_direct_mod() is another, as it registers
> a stub ops for the sites it modifies. Here, detaching one of several BPF
> LSM programs from a busy hook crashed the tasks that ran the hook during
> the update.
> 
> Bit 0 is free only in the direct trampolines that a dyn_ftrace record can
> point to, but bit 63 is set in every kernel text address on x86_64. Thus,
> mark jmp mode by clearing bit 63 instead, so that ftrace_is_jmp() is false
> for any callback, whatever its alignment, and ftrace_regs_caller sets the
> bit again where it cleared bit 0 before. A marked address is not canonical,
> so a return or indirect branch to one that missed ftrace_jmp_get() faults
> instead of running from a byte before its target.
> 
> Fixes: 0c3772a8db1f ("x86/ftrace: Implement DYNAMIC_FTRACE_WITH_JMP")
> Suggested-by: Steven Rostedt <rostedt@goodmis.org>
> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
> Cc: Menglong Dong <dongml2@chinatelecom.cn>

Reviewed-by: Jiri Olsa <jolsa@kernel.org>

jirka


> ---
>  v2 -> v3:
>    - Removed Kconfig text, rest as-is
>  v1 -> v2:
>    - Steven suggested as an alternative/more generic fix to go
>      the route via bit 63
>    - BPF selftest kept as-is
> 
>  arch/x86/kernel/ftrace_64.S | 6 +++---
>  include/linux/ftrace.h      | 8 +++++---
>  2 files changed, 8 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S
> index 62c1c93aa1c6..4b20673a3395 100644
> --- a/arch/x86/kernel/ftrace_64.S
> +++ b/arch/x86/kernel/ftrace_64.S
> @@ -286,9 +286,9 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL)
>  	RET
>  
>  1:
> -	testb	$1, %al
> -	jz	2f
> -	andq $0xfffffffffffffffe, %rax
> +	/* See ftrace_jmp_set(), jmp to the trampoline */
> +	btsq	$63, %rax
> +	jc	2f
>  	movq %rax, MCOUNT_REG_SIZE+8(%rsp)
>  	restore_mcount_regs
>  	/* Restore flags */
> diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
> index b041d9142d2b..1c2056a1a68b 100644
> --- a/include/linux/ftrace.h
> +++ b/include/linux/ftrace.h
> @@ -618,19 +618,21 @@ static inline void arch_ftrace_set_direct_caller(struct ftrace_regs *fregs,
>  #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */
>  
>  #ifdef CONFIG_DYNAMIC_FTRACE_WITH_JMP
> +#define FTRACE_JMP_BIT	BIT(BITS_PER_LONG - 1)
> +
>  static inline bool ftrace_is_jmp(unsigned long addr)
>  {
> -	return addr & 1;
> +	return !(addr & FTRACE_JMP_BIT);
>  }
>  
>  static inline unsigned long ftrace_jmp_set(unsigned long addr)
>  {
> -	return addr | 1UL;
> +	return addr & ~FTRACE_JMP_BIT;
>  }
>  
>  static inline unsigned long ftrace_jmp_get(unsigned long addr)
>  {
> -	return addr & ~1UL;
> +	return addr | FTRACE_JMP_BIT;
>  }
>  #else
>  static inline bool ftrace_is_jmp(unsigned long addr)
> -- 
> 2.43.0
>