Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Andrey Grodzovsky]

Dear Upstream Livepatch  team and Ubuntu Kernel team - I included you both in this since the issue lies on the boundary between Ubuntu kernel and upstream. 

According to official kernel documentation  - https://docs.kernel.org/livepatch/livepatch.html#livepatch, section 7, Limitations - 
1 - Kretprobes using the ftrace framework conflict with the patched functions.
2 - Kprobes in the original function are ignored when the code is redirected to the new implementation.

Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable kernel) machine, I have no problem applying Livepatch and then setting krpobes and kretprobes on a patched function using bpftrace (or directly by coding a BPF program with kprobe/kretprobe attachment)and can confirm both execute without issues. Also the opposite works fine, running my krpobe and kretprobe hooks doesn't prevent from livepatch to be applied successfully. 

fentry/fexit probes do fail in in both directions - but this is expected according to my understanding as coexistence of livepatching and ftrace based BPF hooks are mutually exclusive until 6.0 based kernels 

libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16


Please help me understand this contradiction about kprobes - is this because the KPROBES are FTRACE based  or any other reason ?


Bellow a few prints to prove my point - the kernel function i am using to test is begin_new_exec

ubuntu@ip-xxxxx:~$ sudo canonical-livepatch status
last check: 19 minutes ago
kernel: 5.15.0-1005.7-aws
server check-in: succeeded
kernel state: ✗ Livepatch coverage ended 2023-05-20; please upgrade the kernel and reboot
patch state: ✓ all applicable livepatch modules inserted
patch version: 94.1
tier: stable
machine id: xxxxx

ubuntu@ip-xxxxxx:~$ sudo cat /proc/kallsyms | grep begin_new_exec
ffffffff8af7bdc0 T begin_new_exec
ffffffff8c335694 r __ksymtab_begin_new_exec
ffffffff8c369830 r __kstrtabns_begin_new_exec
ffffffff8c36cf3d r __kstrtab_begin_new_exec
ffffffffc28c1650 t begin_new_exec	[lkp_Ubuntu_5_15_0_1005_7_aws_94]
ffffffffc1a9d388 t bpf_prog_211a1dc71de113a4_begin_new_exec_kprobe	[bpf]
ffffffffc1a9f588 t bpf_prog_5522bacfbb7628bd_begin_new_exec_kretprobe	[bpf]

ubuntu@ip-xxxxx:~$ sudo cat /sys/kernel/debug/kprobes/list  | grep begin_new_exec
ffffffff8af7bdc0  r  begin_new_exec+0x0    [FTRACE]
ffffffff8af7bdc0  k  begin_new_exec+0x0    [FTRACE]


root@ip-xxxxx:~# echo 102400 > /sys/kernel/debug/tracing/buffer_size_kb && echo 1 > /sys/kernel/debug/tracing/events/bpf_trace/enable && echo 1 > /sys/kernel/debug/tracing/tracing_on && cat 
/sys/kernel/debug/tracing/trace_pipe
             cat-10406   [000] d...1 433341.003035: bpf_trace_printk: kprobe: begin_new_exec called with bprm=000000005304a36e
             cat-10406   [000] d...1 433341.003163: bpf_trace_printk: kretprobe: begin_new_exec returned 0

Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Petr Mladek]

On Tue 2025-10-14 21:37:49, Andrey Grodzovsky wrote:
> Dear Upstream Livepatch  team and Ubuntu Kernel team - I included you both in this since the issue lies on the boundary between Ubuntu kernel and upstream. 
> 
> According to official kernel documentation  - https://docs.kernel.org/livepatch/livepatch.html#livepatch, section 7, Limitations - 
> 1 - Kretprobes using the ftrace framework conflict with the patched functions.
> 2 - Kprobes in the original function are ignored when the code is redirected to the new implementation.
> 
> Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable kernel) machine, I have no problem applying Livepatch and then setting krpobes and kretprobes on a patched function using bpftrace (or directly by coding a BPF program with kprobe/kretprobe attachment)and can confirm both execute without issues. Also the opposite works fine, running my krpobe and kretprobe hooks doesn't prevent from livepatch to be applied successfully. 
> 
> fentry/fexit probes do fail in in both directions - but this is expected according to my understanding as coexistence of livepatching and ftrace based BPF hooks are mutually exclusive until 6.0 based kernels 
> 
> libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
> libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16
> 
> 
> Please help me understand this contradiction about kprobes - is this because the KPROBES are FTRACE based  or any other reason ?

Heh, it seems that we have discussed this 10 years ago and I already
forgot most details.

Yes, the conflict is detected when KPROBES are using FTRACE
infrastructure. But it happens only when the KPROBE needs to redirect
the function call, namely when it needs to modify IP address which will be used
when all attached ftrace callbacks are proceed.

It is related to the FTRACE_OPS_FL_IPMODIFY flag.

More details can be found in the discussion at
https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/#re746846b6b16c49a55c89b4c63b7995fe5972111

I seems that I made some analyze when it worked and it did not work,
see https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/#mffd8c8bf4325b473d89876f2805f42f1af7c82d7
But I am not 100% sure that it was correct. Also it was before the
BPF-boom.

Also you might look at the selftest in the todays Linus' tree:

  + tools/testing/selftests/livepatch/test-kprobe.sh
  + tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c
  + tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c

The parallel load fails when the Kprobe is using a post_handler.

Sigh, we should fix the livepatch documentation. The kretprobes
obviously work. register_kretprobe() even explicitely sets:

	rp->kp.post_handler = NULL;

It seems that .post_handler is called after all ftrace handlers.
And it sets IP after the NOPs, see kprobe_ftrace_handler().
I am not sure about the use case.

Best Regards,
Petr

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Andrey Grodzovsky]

On 10/15/25 07:53, Petr Mladek wrote:
> On Tue 2025-10-14 21:37:49, Andrey Grodzovsky wrote:
>> Dear Upstream Livepatch  team and Ubuntu Kernel team - I included you both in this since the issue lies on the boundary between Ubuntu kernel and upstream.
>>
>> According to official kernel documentation  - https://urldefense.com/v3/__https://docs.kernel.org/livepatch/livepatch.html*livepatch__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5TeFxUxQ$ , section 7, Limitations -
>> 1 - Kretprobes using the ftrace framework conflict with the patched functions.
>> 2 - Kprobes in the original function are ignored when the code is redirected to the new implementation.
>>
>> Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable kernel) machine, I have no problem applying Livepatch and then setting krpobes and kretprobes on a patched function using bpftrace (or directly by coding a BPF program with kprobe/kretprobe attachment)and can confirm both execute without issues. Also the opposite works fine, running my krpobe and kretprobe hooks doesn't prevent from livepatch to be applied successfully.
>>
>> fentry/fexit probes do fail in in both directions - but this is expected according to my understanding as coexistence of livepatching and ftrace based BPF hooks are mutually exclusive until 6.0 based kernels
>>
>> libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
>> libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16
>>
>>
>> Please help me understand this contradiction about kprobes - is this because the KPROBES are FTRACE based  or any other reason ?
> Heh, it seems that we have discussed this 10 years ago and I already
> forgot most details.
>
> Yes, the conflict is detected when KPROBES are using FTRACE
> infrastructure. But it happens only when the KPROBE needs to redirect
> the function call, namely when it needs to modify IP address which will be used
> when all attached ftrace callbacks are proceed.
>
> It is related to the FTRACE_OPS_FL_IPMODIFY flag.


I see, that explains my case as my probes are simple, print only probes 
that defently don't that the ip pointer.

But i still don't understand limitation 2 above doesn't show itself - 
how my kprobes and especially kretprobes, continue execute even after 
livepatch applied to the function  i am attached to ? The code execution 
is redirected to a different function then to which i attached my probes...

Also - can you please confirm that as far as i checked, starting with 
kernel 6.0 fentry/fexit on x86 should not have any conflicts with 
livepatch per merge of this RFC - 
https://lkml.iu.edu/hypermail/linux/kernel/2207.2/00858.html ?

Thanks,
Andrey

>
> More details can be found in the discussion at
> https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*re746846b6b16c49a55c89b4c63b7995fe5972111__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu6pjuIgig$
>
> I seems that I made some analyze when it worked and it did not work,
> see https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*mffd8c8bf4325b473d89876f2805f42f1af7c82d7__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5xbeoulA$
> But I am not 100% sure that it was correct. Also it was before the
> BPF-boom.
>
> Also you might look at the selftest in the todays Linus' tree:
>
>    + tools/testing/selftests/livepatch/https://urldefense.com/v3/__http://test-kprobe.sh__;!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5RXF-AnA$
>    + tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c
>    + tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c
>
> The parallel load fails when the Kprobe is using a post_handler.
>
> Sigh, we should fix the livepatch documentation. The kretprobes
> obviously work. register_kretprobe() even explicitely sets:
>
> 	rp->kp.post_handler = NULL;
>
> It seems that .post_handler is called after all ftrace handlers.
> And it sets IP after the NOPs, see kprobe_ftrace_handler().
> I am not sure about the use case.
>
> Best Regards,
> Petr

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Petr Mladek]

Added Song Liu and Steven into Cc because they are more familiar with
the ftrace trampolines.

On Wed 2025-10-15 17:11:31, Andrey Grodzovsky wrote:
> On 10/15/25 07:53, Petr Mladek wrote:
> > On Tue 2025-10-14 21:37:49, Andrey Grodzovsky wrote:
> > > Dear Upstream Livepatch  team and Ubuntu Kernel team - I included you both in this since the issue lies on the boundary between Ubuntu kernel and upstream.
> > > 
> > > According to official kernel documentation  - https://urldefense.com/v3/__https://docs.kernel.org/livepatch/livepatch.html*livepatch__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5TeFxUxQ$ , section 7, Limitations -
> > > 1 - Kretprobes using the ftrace framework conflict with the patched functions.
> > > 2 - Kprobes in the original function are ignored when the code is redirected to the new implementation.
> > > 
> > > Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable kernel) machine, I have no problem applying Livepatch and then setting krpobes and kretprobes on a patched function using bpftrace (or directly by coding a BPF program with kprobe/kretprobe attachment)and can confirm both execute without issues. Also the opposite works fine, running my krpobe and kretprobe hooks doesn't prevent from livepatch to be applied successfully.
> > > 
> > > fentry/fexit probes do fail in in both directions - but this is expected according to my understanding as coexistence of livepatching and ftrace based BPF hooks are mutually exclusive until 6.0 based kernels
> > > 
> > > libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
> > > libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16
> > > 
> > > 
> > > Please help me understand this contradiction about kprobes - is this because the KPROBES are FTRACE based  or any other reason ?
> > Heh, it seems that we have discussed this 10 years ago and I already
> > forgot most details.
> > 
> > Yes, the conflict is detected when KPROBES are using FTRACE
> > infrastructure. But it happens only when the KPROBE needs to redirect
> > the function call, namely when it needs to modify IP address which will be used
> > when all attached ftrace callbacks are proceed.
> > 
> > It is related to the FTRACE_OPS_FL_IPMODIFY flag.
> 
> 
> I see, that explains my case as my probes are simple, print only probes that
> defently don't that the ip pointer.
> 
> But i still don't understand limitation 2 above doesn't show itself - how my
> kprobes and especially kretprobes, continue execute even after livepatch
> applied to the function  i am attached to ? The code execution is redirected
> to a different function then to which i attached my probes...

The code is rather complicated and it is not obvious to me.

But I expect that kretprobes modify return addresses which are
stored on the stack. They replace them
with a helper function (trampoline) which would process
the kretprobe callback and jump to the original return address.

It works even with livepatching. The only requirement is that
the function returns using a "ret" instruction which reads
the return address from the stack.

> Also - can you please confirm that as far as i checked, starting with kernel
> 6.0 fentry/fexit on x86 should not have any conflicts with livepatch per
> merge of this RFC -
> https://lkml.iu.edu/hypermail/linux/kernel/2207.2/00858.html ?

My understanding is that the above mentioned patchset allowed to use
livepatching and BPF on the same function at the same time. BPF used
to redirect the function to a BPF trampoline. And the patchset allowed
to call the BPF trampoline/callback directly from the ftrace callback.

Best Regards,
Petr

> > 
> > More details can be found in the discussion at
> > https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*re746846b6b16c49a55c89b4c63b7995fe5972111__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu6pjuIgig$
> > 
> > I seems that I made some analyze when it worked and it did not work,
> > see https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*mffd8c8bf4325b473d89876f2805f42f1af7c82d7__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5xbeoulA$
> > But I am not 100% sure that it was correct. Also it was before the
> > BPF-boom.
> > 
> > Also you might look at the selftest in the todays Linus' tree:
> > 
> >    + tools/testing/selftests/livepatch/https://urldefense.com/v3/__http://test-kprobe.sh__;!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5RXF-AnA$
> >    + tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c
> >    + tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c
> > 
> > The parallel load fails when the Kprobe is using a post_handler.
> > 
> > Sigh, we should fix the livepatch documentation. The kretprobes
> > obviously work. register_kretprobe() even explicitely sets:
> > 
> > 	rp->kp.post_handler = NULL;
> > 
> > It seems that .post_handler is called after all ftrace handlers.
> > And it sets IP after the NOPs, see kprobe_ftrace_handler().
> > I am not sure about the use case.
> > 
> > Best Regards,
> > Petr
> 
> 

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Song Liu]

On Thu, Oct 16, 2025 at 3:56 AM Petr Mladek <pmladek@suse.com> wrote:
>
> Added Song Liu and Steven into Cc because they are more familiar with
> the ftrace trampolines.
>
> On Wed 2025-10-15 17:11:31, Andrey Grodzovsky wrote:
> > On 10/15/25 07:53, Petr Mladek wrote:
> > > On Tue 2025-10-14 21:37:49, Andrey Grodzovsky wrote:
> > > > Dear Upstream Livepatch  team and Ubuntu Kernel team - I included you both in this since the issue lies on the boundary between Ubuntu kernel and upstream.
> > > >
> > > > According to official kernel documentation  - https://urldefense.com/v3/__https://docs.kernel.org/livepatch/livepatch.html*livepatch__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5TeFxUxQ$ , section 7, Limitations -
> > > > 1 - Kretprobes using the ftrace framework conflict with the patched functions.
> > > > 2 - Kprobes in the original function are ignored when the code is redirected to the new implementation.
> > > >
> > > > Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable kernel) machine, I have no problem applying Livepatch and then setting krpobes and kretprobes on a patched function using bpftrace (or directly by coding a BPF program with kprobe/kretprobe attachment)and can confirm both execute without issues. Also the opposite works fine, running my krpobe and kretprobe hooks doesn't prevent from livepatch to be applied successfully.
> > > >
> > > > fentry/fexit probes do fail in in both directions - but this is expected according to my understanding as coexistence of livepatching and ftrace based BPF hooks are mutually exclusive until 6.0 based kernels
> > > >
> > > > libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
> > > > libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16
> > > >
> > > >
> > > > Please help me understand this contradiction about kprobes - is this because the KPROBES are FTRACE based  or any other reason ?
> > > Heh, it seems that we have discussed this 10 years ago and I already
> > > forgot most details.
> > >
> > > Yes, the conflict is detected when KPROBES are using FTRACE
> > > infrastructure. But it happens only when the KPROBE needs to redirect
> > > the function call, namely when it needs to modify IP address which will be used
> > > when all attached ftrace callbacks are proceed.
> > >
> > > It is related to the FTRACE_OPS_FL_IPMODIFY flag.
> >
> >
> > I see, that explains my case as my probes are simple, print only probes that
> > defently don't that the ip pointer.
> >
> > But i still don't understand limitation 2 above doesn't show itself - how my
> > kprobes and especially kretprobes, continue execute even after livepatch
> > applied to the function  i am attached to ? The code execution is redirected
> > to a different function then to which i attached my probes...
>
> The code is rather complicated and it is not obvious to me.
>
> But I expect that kretprobes modify return addresses which are
> stored on the stack. They replace them
> with a helper function (trampoline) which would process
> the kretprobe callback and jump to the original return address.
>
> It works even with livepatching. The only requirement is that
> the function returns using a "ret" instruction which reads
> the return address from the stack.
>
> > Also - can you please confirm that as far as i checked, starting with kernel
> > 6.0 fentry/fexit on x86 should not have any conflicts with livepatch per
> > merge of this RFC -
> > https://lkml.iu.edu/hypermail/linux/kernel/2207.2/00858.html ?
>
> My understanding is that the above mentioned patchset allowed to use
> livepatching and BPF on the same function at the same time. BPF used
> to redirect the function to a BPF trampoline. And the patchset allowed
> to call the BPF trampoline/callback directly from the ftrace callback.

Yes, the patch set above allows BPF trampoline to work on the same
function at the same time with livepatch. After this patchset, BPF
trampoline no longer sets FTRACE_OPS_FL_IPMODIFY.

Thanks,
Song



> Best Regards,
> Petr
>
> > >
> > > More details can be found in the discussion at
> > > https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*re746846b6b16c49a55c89b4c63b7995fe5972111__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu6pjuIgig$
> > >
> > > I seems that I made some analyze when it worked and it did not work,
> > > see https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*mffd8c8bf4325b473d89876f2805f42f1af7c82d7__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5xbeoulA$
> > > But I am not 100% sure that it was correct. Also it was before the
> > > BPF-boom.
> > >
> > > Also you might look at the selftest in the todays Linus' tree:
> > >
> > >    + tools/testing/selftests/livepatch/https://urldefense.com/v3/__http://test-kprobe.sh__;!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5RXF-AnA$
> > >    + tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c
> > >    + tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c
> > >
> > > The parallel load fails when the Kprobe is using a post_handler.
> > >
> > > Sigh, we should fix the livepatch documentation. The kretprobes
> > > obviously work. register_kretprobe() even explicitely sets:
> > >
> > >     rp->kp.post_handler = NULL;
> > >
> > > It seems that .post_handler is called after all ftrace handlers.
> > > And it sets IP after the NOPs, see kprobe_ftrace_handler().
> > > I am not sure about the use case.
> > >
> > > Best Regards,
> > > Petr
> >
> >

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Andrey Grodzovsky]

On 10/16/25 17:32, Song Liu wrote:
> On Thu, Oct 16, 2025 at 3:56 AM Petr Mladek <pmladek@suse.com> wrote:
>> Added Song Liu and Steven into Cc because they are more familiar with
>> the ftrace trampolines.
>>
>> On Wed 2025-10-15 17:11:31, Andrey Grodzovsky wrote:
>>> On 10/15/25 07:53, Petr Mladek wrote:
>>>> On Tue 2025-10-14 21:37:49, Andrey Grodzovsky wrote:
>>>>> Dear Upstream Livepatch  team and Ubuntu Kernel team - I included you both in this since the issue lies on the boundary between Ubuntu kernel and upstream.
>>>>>
>>>>> According to official kernel documentation  - https://urldefense.com/v3/__https://docs.kernel.org/livepatch/livepatch.html*livepatch__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5TeFxUxQ$ , section 7, Limitations -
>>>>> 1 - Kretprobes using the ftrace framework conflict with the patched functions.
>>>>> 2 - Kprobes in the original function are ignored when the code is redirected to the new implementation.
>>>>>
>>>>> Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable kernel) machine, I have no problem applying Livepatch and then setting krpobes and kretprobes on a patched function using bpftrace (or directly by coding a BPF program with kprobe/kretprobe attachment)and can confirm both execute without issues. Also the opposite works fine, running my krpobe and kretprobe hooks doesn't prevent from livepatch to be applied successfully.
>>>>>
>>>>> fentry/fexit probes do fail in in both directions - but this is expected according to my understanding as coexistence of livepatching and ftrace based BPF hooks are mutually exclusive until 6.0 based kernels
>>>>>
>>>>> libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
>>>>> libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16
>>>>>
>>>>>
>>>>> Please help me understand this contradiction about kprobes - is this because the KPROBES are FTRACE based  or any other reason ?
>>>> Heh, it seems that we have discussed this 10 years ago and I already
>>>> forgot most details.
>>>>
>>>> Yes, the conflict is detected when KPROBES are using FTRACE
>>>> infrastructure. But it happens only when the KPROBE needs to redirect
>>>> the function call, namely when it needs to modify IP address which will be used
>>>> when all attached ftrace callbacks are proceed.
>>>>
>>>> It is related to the FTRACE_OPS_FL_IPMODIFY flag.
>>>
>>> I see, that explains my case as my probes are simple, print only probes that
>>> defently don't that the ip pointer.
>>>
>>> But i still don't understand limitation 2 above doesn't show itself - how my
>>> kprobes and especially kretprobes, continue execute even after livepatch
>>> applied to the function  i am attached to ? The code execution is redirected
>>> to a different function then to which i attached my probes...
>> The code is rather complicated and it is not obvious to me.
>>
>> But I expect that kretprobes modify return addresses which are
>> stored on the stack. They replace them
>> with a helper function (trampoline) which would process
>> the kretprobe callback and jump to the original return address.
>>
>> It works even with livepatching. The only requirement is that
>> the function returns using a "ret" instruction which reads
>> the return address from the stack.
>>
>>> Also - can you please confirm that as far as i checked, starting with kernel
>>> 6.0 fentry/fexit on x86 should not have any conflicts with livepatch per
>>> merge of this RFC -
>>> https://urldefense.com/v3/__https://lkml.iu.edu/hypermail/linux/kernel/2207.2/00858.html__;!!BmdzS3_lV9HdKG8!wlwzQrMFIQcM6TTsDw6Eso8D6cFjSA1lZeeRI7xStg-c4RFuGmqrg7B6GeRXLQ8M1l6HgitfnBCfMdSeleudJcNY$  ?
>> My understanding is that the above mentioned patchset allowed to use
>> livepatching and BPF on the same function at the same time. BPF used
>> to redirect the function to a BPF trampoline. And the patchset allowed
>> to call the BPF trampoline/callback directly from the ftrace callback.
> Yes, the patch set above allows BPF trampoline to work on the same
> function at the same time with livepatch. After this patchset, BPF
> trampoline no longer sets FTRACE_OPS_FL_IPMODIFY.
>
> Thanks,
> Song


I tested this bellow on 6.1.0 based Ubuntu kernel using bpftrace - in 
the end as seen bellow, I wasn't able to attach using fentry/fexit (or 
krpboe/kretprobe) as long as livepatch was present

ubuntu@ip-10-10-114-204:~$ uname -r
6.8.0-1008-aws
ubuntu@ip-10-10-114-204:~$ cat /proc/version_signature
Ubuntu 6.8.0-1008.8-aws 6.8.1

[AG] - See bellow the patched function I am to attach to -

ubuntu@ip-10-10-114-204:~$ sudo cat /proc/kallsyms | grep begin_new_exec
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name 
resolution
ffffffff81ee0420 T __pfx_begin_new_exec
ffffffff81ee0430 T begin_new_exec
ffffffff8374b3d8 r __ksymtab_begin_new_exec
ffffffffc07c9390 t begin_new_exec [lkp_Ubuntu_6_8_0_1008_8_aws_111]
ffffffffc07c9380 t __pfx_begin_new_exec [lkp_Ubuntu_6_8_0_1008_8_aws_111]

[AG] - Attaching to begin_new_exec

ubuntu@ip-10-10-114-204:~$ sudo bpftool btf dump file 
/sys/kernel/btf/vmlinux | grep begin_new_exec
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name 
resolution
[68920] FUNC 'begin_new_exec' type_id=48063 linkage=static

[AG] - , trying to attach to fentry and fexit and first error asks me to 
explicitly resolve the ambiguity - which i guess make sense

ubuntu@ip-10-10-114-204:~$
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -v -e 'fentry:begin_new_exec { 
@start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n", 
pid, comm); } fexit:begin_new_exec { $latency = nsecs - @start[tid]; 
delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s, 
Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
stdin:1:1-22: ERROR: ambiguous attach point, please specify module 
containing the function 'begin_new_exec'
fentry:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START 
(fentry): PID %d, Comm %s\n", pid, comm); } fexit:begin_new_exec { 
$latency = nsecs - @start[tid]; delete(@start[tid]); printf("<- EXEC END 
(fexit): PID %d, Comm %s, Retval %d, Latency %d us\n", pid, comm, 
retval, $latency / 1000); }
~~~~~~~~~~~~~~~~~~~~~
stdin:1:111-132: ERROR: ambiguous attach point, please specify module 
containing the function 'begin_new_exec'
fentry:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START 
(fentry): PID %d, Comm %s\n", pid, comm); } fexit:begin_new_exec { 
$latency = nsecs - @start[tid]; delete(@start[tid]); printf("<- EXEC END 
(fexit): PID %d, Comm %s, Retval %d, Latency %d us\n", pid, comm, 
retval, $latency / 1000); }
                                             ~~~~~~~~~~~~~~~~~~~~~
[AG] - Trying first to point him at the original  function - but he 
fails on the fexit I assume  - which is strange, I assumed fexit 
(kretfunc) and livepatch can coexist ?

ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e 
'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC 
START (fentry): PID %d, Comm %s\n", pid, comm); } 
fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid]; 
delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s, 
Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
Attaching 2 probes...
ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec

[AG] - Trying to skip the fexit and only do fentry - he still rejects
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e 
'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC 
START (fentry): PID %d, Comm %s\n", pid, comm); }'
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name 
resolution
INFO: node count: 12
Attaching 1 probe...

Program ID: 295

The verifier log:
processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3 
peak_states 3 mark_read 1

Attaching kfunc:vmlinux:begin_new_exec
ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec


[AG] - Now trying to point him at the livepatch function but here he 
complains that Uubntu doesn't have BTF info for the livepatch module

ubuntu@ip-10-10-114-204:~$
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -v -e 
'fentry:lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec { @start[tid] = 
nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n", pid, comm); } 
fexit:lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec { $latency = nsecs 
- @start[tid]; delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, 
Comm %s, Retval %d, Latency %d us\n", pid, comm, retval, $latency / 
1000); }'
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name 
resolution
INFO: node count: 33
Attaching 2 probes...
ERROR: No BTF found for lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec

[AG] - Disabling livepatch, rebooting to verify all livepatch symbols 
are out of kalsym table and now everything fine

ubuntu@ip-10-10-114-204:~$   sudo bpftrace -v -e 'fentry:begin_new_exec 
{ @start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm 
%s\n", pid, comm]; delete(@start[tid]); printf("<- EXEC END (fexit): PID 
%d, Comm %s, Retval %d, Latency %d us\n", pid, comm, retval, $latency / 
1000); }'
INFO: node count: 33
Attaching 2 probes...
Program ID: 33
The verifier log:
processed 76 insns (limit 1000000) max_states_per_insn 0 total_states 5 
peak_states 5 mark_read 3
Attaching kretfunc:vmlinux:begin_new_exec
Program ID: 34
The verifier log:
processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3 
peak_states 3 mark_read 1

Attaching kfunc:vmlinux:begin_new_exec
-> EXEC START (fentry): PID 1714, Comm bash
<- EXEC END (fexit): PID 1714, Comm sudo, Retval 0, Latency 99 us


ubuntu@ip-10-10-114-204:~$ sudo bpftool link
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name 
resolution
1: tracing  prog 2
     prog_type tracing  attach_type modify_return
     target_obj_id 1  target_btf_id 119714
5: tracing  prog 33
     prog_type tracing  attach_type trace_fexit
     target_obj_id 1  target_btf_id 68920
6: tracing  prog 34
     prog_type tracing  attach_type trace_fentry
     target_obj_id 1  target_btf_id 68920
ubuntu@ip-10-10-114-204:~$

Thanks,
Andrey


>
>
>
>> Best Regards,
>> Petr
>>
>>>> More details can be found in the discussion at
>>>> https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*re746846b6b16c49a55c89b4c63b7995fe5972111__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu6pjuIgig$
>>>>
>>>> I seems that I made some analyze when it worked and it did not work,
>>>> see https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*mffd8c8bf4325b473d89876f2805f42f1af7c82d7__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5xbeoulA$
>>>> But I am not 100% sure that it was correct. Also it was before the
>>>> BPF-boom.
>>>>
>>>> Also you might look at the selftest in the todays Linus' tree:
>>>>
>>>>     + tools/testing/selftests/livepatch/https://urldefense.com/v3/__http://test-kprobe.sh__;!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5RXF-AnA$
>>>>     + tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c
>>>>     + tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c
>>>>
>>>> The parallel load fails when the Kprobe is using a post_handler.
>>>>
>>>> Sigh, we should fix the livepatch documentation. The kretprobes
>>>> obviously work. register_kretprobe() even explicitely sets:
>>>>
>>>>      rp->kp.post_handler = NULL;
>>>>
>>>> It seems that .post_handler is called after all ftrace handlers.
>>>> And it sets IP after the NOPs, see kprobe_ftrace_handler().
>>>> I am not sure about the use case.
>>>>
>>>> Best Regards,
>>>> Petr
>>>

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Song Liu]

On Thu, Oct 16, 2025 at 2:55 PM Andrey Grodzovsky
<andrey.grodzovsky@crowdstrike.com> wrote:
[...]
> [AG] - Trying first to point him at the original  function - but he
> fails on the fexit I assume  - which is strange, I assumed fexit
> (kretfunc) and livepatch can coexist ?
>
> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e
> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
> START (fentry): PID %d, Comm %s\n", pid, comm); }
> fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid];
> delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s,
> Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
> Attaching 2 probes...
> ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec
>
> [AG] - Trying to skip the fexit and only do fentry - he still rejects
> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e
> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
> START (fentry): PID %d, Comm %s\n", pid, comm); }'
> sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
> resolution
> INFO: node count: 12
> Attaching 1 probe...
>
> Program ID: 295
>
> The verifier log:
> processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3
> peak_states 3 mark_read 1
>
> Attaching kfunc:vmlinux:begin_new_exec
> ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec

OK, I could reproduce this issue and found the issue. In my test,
fexit+livepatch works on some older kernel, but fails on some newer
kernel. I haven't bisected to the commit that broke it.

Something like the following make it work:

diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
index 2e113f8b13a2..4277b4f33eb8 100644
--- i/kernel/trace/ftrace.c
+++ w/kernel/trace/ftrace.c
@@ -5985,6 +5985,8 @@ int register_ftrace_direct(struct ftrace_ops
*ops, unsigned long addr)
        ops->direct_call = addr;

        err = register_ftrace_function_nolock(ops);
+       if (err)
+               remove_direct_functions_hash(direct_functions, addr);

  out_unlock:
        mutex_unlock(&direct_mutex);

Andrey, could you also test this change?

Thanks,
Song

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Song Liu]

[-- Attachment #1: Type: text/plain, Size: 2422 bytes --]

On Fri, Oct 17, 2025 at 9:58 AM Song Liu <song@kernel.org> wrote:
>
> On Thu, Oct 16, 2025 at 2:55 PM Andrey Grodzovsky
> <andrey.grodzovsky@crowdstrike.com> wrote:
> [...]
> > [AG] - Trying first to point him at the original  function - but he
> > fails on the fexit I assume  - which is strange, I assumed fexit
> > (kretfunc) and livepatch can coexist ?
> >
> > ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e
> > 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
> > START (fentry): PID %d, Comm %s\n", pid, comm); }
> > fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid];
> > delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s,
> > Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
> > Attaching 2 probes...
> > ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec
> >
> > [AG] - Trying to skip the fexit and only do fentry - he still rejects
> > ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e
> > 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
> > START (fentry): PID %d, Comm %s\n", pid, comm); }'
> > sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
> > resolution
> > INFO: node count: 12
> > Attaching 1 probe...
> >
> > Program ID: 295
> >
> > The verifier log:
> > processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3
> > peak_states 3 mark_read 1
> >
> > Attaching kfunc:vmlinux:begin_new_exec
> > ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec
>
> OK, I could reproduce this issue and found the issue. In my test,
> fexit+livepatch works on some older kernel, but fails on some newer
> kernel. I haven't bisected to the commit that broke it.
>
> Something like the following make it work:
>
> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
> index 2e113f8b13a2..4277b4f33eb8 100644
> --- i/kernel/trace/ftrace.c
> +++ w/kernel/trace/ftrace.c
> @@ -5985,6 +5985,8 @@ int register_ftrace_direct(struct ftrace_ops
> *ops, unsigned long addr)
>         ops->direct_call = addr;
>
>         err = register_ftrace_function_nolock(ops);
> +       if (err)
> +               remove_direct_functions_hash(direct_functions, addr);
>
>   out_unlock:
>         mutex_unlock(&direct_mutex);
>
> Andrey, could you also test this change?

Attached is a better version of the fix.

Thanks,
Song

[-- Attachment #2: 0001-ftrace-Fix-BPF-fexit-with-livepatch.patch --]
[-- Type: application/octet-stream, Size: 2100 bytes --]

From d4aa9c0058c794d6259248187bd5896cb40eea29 Mon Sep 17 00:00:00 2001
From: Song Liu <song@kernel.org>
Date: Fri, 17 Oct 2025 11:49:45 -0700
Subject: [PATCH] ftrace: Fix BPF fexit with livepatch

When livepatch is attached to the same function as bpf trampoline with
a fexit program, bpf trampoline code calls register_ftrace_direct()
twice. The first time will fail with -EAGAIN, and the second time it
will succeed. This requires register_ftrace_direct() to unregister
the address on the first attempt. Otherwise, the bpf trampoline cannot
attach. Here is an easy way to reproduce this issue:

  insmod samples/livepatch/livepatch-sample.ko
  bpftrace -e 'fexit:cmdline_proc_show {}'
  ERROR: Unable to attach probe: fexit:vmlinux:cmdline_proc_show...

Fix this by updating direct_functions hash after
register_ftrace_function_nolock succeeds.

Fixes: d05cb470663a ("ftrace: Fix modification of direct_function hash while in use")
Cc: stable@vger.kernel.org # v6.6+
Reported-by: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
Closes: https://lore.kernel.org/live-patching/c5058315a39d4615b333e485893345be@crowdstrike.com/
Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Song Liu <song@kernel.org>
---
 kernel/trace/ftrace.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index a69067367c29..82a06b7d0268 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -6038,16 +6038,18 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
 		}
 	}
 
-	free_hash = direct_functions;
-	rcu_assign_pointer(direct_functions, new_hash);
-	new_hash = NULL;
-
 	ops->func = call_direct_funcs;
 	ops->flags = MULTI_FLAGS;
 	ops->trampoline = FTRACE_REGS_ADDR;
 	ops->direct_call = addr;
 
 	err = register_ftrace_function_nolock(ops);
+	if (err)
+		goto out_unlock;
+
+	free_hash = direct_functions;
+	rcu_assign_pointer(direct_functions, new_hash);
+	new_hash = NULL;
 
  out_unlock:
 	mutex_unlock(&direct_mutex);
-- 
2.47.3

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Andrey Grodzovsky]

On 10/17/25 15:07, Song Liu wrote:
> On Fri, Oct 17, 2025 at 9:58 AM Song Liu <song@kernel.org> wrote:
>> On Thu, Oct 16, 2025 at 2:55 PM Andrey Grodzovsky
>> <andrey.grodzovsky@crowdstrike.com> wrote:
>> [...]
>>> [AG] - Trying first to point him at the original  function - but he
>>> fails on the fexit I assume  - which is strange, I assumed fexit
>>> (kretfunc) and livepatch can coexist ?
>>>
>>> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e
>>> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
>>> START (fentry): PID %d, Comm %s\n", pid, comm); }
>>> fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid];
>>> delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s,
>>> Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
>>> Attaching 2 probes...
>>> ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec
>>>
>>> [AG] - Trying to skip the fexit and only do fentry - he still rejects
>>> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e
>>> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
>>> START (fentry): PID %d, Comm %s\n", pid, comm); }'
>>> sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
>>> resolution
>>> INFO: node count: 12
>>> Attaching 1 probe...
>>>
>>> Program ID: 295
>>>
>>> The verifier log:
>>> processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3
>>> peak_states 3 mark_read 1
>>>
>>> Attaching kfunc:vmlinux:begin_new_exec
>>> ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec
>> OK, I could reproduce this issue and found the issue. In my test,
>> fexit+livepatch works on some older kernel, but fails on some newer
>> kernel. I haven't bisected to the commit that broke it.
>>
>> Something like the following make it work:
>>
>> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
>> index 2e113f8b13a2..4277b4f33eb8 100644
>> --- i/kernel/trace/ftrace.c
>> +++ w/kernel/trace/ftrace.c
>> @@ -5985,6 +5985,8 @@ int register_ftrace_direct(struct ftrace_ops
>> *ops, unsigned long addr)
>>          ops->direct_call = addr;
>>
>>          err = register_ftrace_function_nolock(ops);
>> +       if (err)
>> +               remove_direct_functions_hash(direct_functions, addr);
>>
>>    out_unlock:
>>          mutex_unlock(&direct_mutex);
>>
>> Andrey, could you also test this change?
> Attached is a better version of the fix.
>
> Thanks,
> Song

Thank you Song!

So, with this You say both fentry and fexit will work no issues ?

So juts to understand, as i am not familiar with live-patch generation, 
I get the sources for my Ubuntu kernel, I apply your patch, I also 
generate manually livepatch module that makes a dummy patching to my 
test function (begin_new_exec), and apply this patch to my running 
costum kernel ? Because i don't think the stadard ubuntu livepatching 
will agree to apply his livepatch CVEs to my ostum kenel, it will 
probably recognize it's not stock ubuntu kernel but amanully built one.

Thanks,
Andrey

So i need to build the stable

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Song Liu]

On Fri, Oct 17, 2025 at 12:48 PM Andrey Grodzovsky
<andrey.grodzovsky@crowdstrike.com> wrote:
>
> On 10/17/25 15:07, Song Liu wrote:
> > On Fri, Oct 17, 2025 at 9:58 AM Song Liu <song@kernel.org> wrote:
> >> On Thu, Oct 16, 2025 at 2:55 PM Andrey Grodzovsky
> >> <andrey.grodzovsky@crowdstrike.com> wrote:
> >> [...]
> >>> [AG] - Trying first to point him at the original  function - but he
> >>> fails on the fexit I assume  - which is strange, I assumed fexit
> >>> (kretfunc) and livepatch can coexist ?
> >>>
> >>> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e
> >>> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
> >>> START (fentry): PID %d, Comm %s\n", pid, comm); }
> >>> fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid];
> >>> delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s,
> >>> Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
> >>> Attaching 2 probes...
> >>> ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec
> >>>
> >>> [AG] - Trying to skip the fexit and only do fentry - he still rejects
> >>> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e
> >>> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
> >>> START (fentry): PID %d, Comm %s\n", pid, comm); }'
> >>> sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
> >>> resolution
> >>> INFO: node count: 12
> >>> Attaching 1 probe...
> >>>
> >>> Program ID: 295
> >>>
> >>> The verifier log:
> >>> processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3
> >>> peak_states 3 mark_read 1
> >>>
> >>> Attaching kfunc:vmlinux:begin_new_exec
> >>> ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec
> >> OK, I could reproduce this issue and found the issue. In my test,
> >> fexit+livepatch works on some older kernel, but fails on some newer
> >> kernel. I haven't bisected to the commit that broke it.
> >>
> >> Something like the following make it work:
> >>
> >> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
> >> index 2e113f8b13a2..4277b4f33eb8 100644
> >> --- i/kernel/trace/ftrace.c
> >> +++ w/kernel/trace/ftrace.c
> >> @@ -5985,6 +5985,8 @@ int register_ftrace_direct(struct ftrace_ops
> >> *ops, unsigned long addr)
> >>          ops->direct_call = addr;
> >>
> >>          err = register_ftrace_function_nolock(ops);
> >> +       if (err)
> >> +               remove_direct_functions_hash(direct_functions, addr);
> >>
> >>    out_unlock:
> >>          mutex_unlock(&direct_mutex);
> >>
> >> Andrey, could you also test this change?
> > Attached is a better version of the fix.
> >
> > Thanks,
> > Song
>
> Thank you Song!
>
> So, with this You say both fentry and fexit will work no issues ?

Yes, fentry and fexit should both work.

> So juts to understand, as i am not familiar with live-patch generation,
> I get the sources for my Ubuntu kernel, I apply your patch, I also
> generate manually livepatch module that makes a dummy patching to my
> test function (begin_new_exec), and apply this patch to my running
> costum kernel ? Because i don't think the stadard ubuntu livepatching
> will agree to apply his livepatch CVEs to my ostum kenel, it will
> probably recognize it's not stock ubuntu kernel but amanully built one.

livepatch is a kernel module. Therefore, unless the two kernels are almost
identical, livepatch built for one kernel cannot be used on the other.

If you build the kernel from source code, there are some samples in
samples/livepatch that you can use for testing. PS: You need to enable

  CONFIG_SAMPLE_LIVEPATCH=m

I hope this helps.

Thanks,
Song

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Andrey Grodzovsky]

On 10/17/25 17:11, Song Liu wrote:
> On Fri, Oct 17, 2025 at 12:48 PM Andrey Grodzovsky
> <andrey.grodzovsky@crowdstrike.com> wrote:
>> On 10/17/25 15:07, Song Liu wrote:
>>> On Fri, Oct 17, 2025 at 9:58 AM Song Liu <song@kernel.org> wrote:
>>>> On Thu, Oct 16, 2025 at 2:55 PM Andrey Grodzovsky
>>>> <andrey.grodzovsky@crowdstrike.com> wrote:
>>>> [...]
>>>>> [AG] - Trying first to point him at the original  function - but he
>>>>> fails on the fexit I assume  - which is strange, I assumed fexit
>>>>> (kretfunc) and livepatch can coexist ?
>>>>>
>>>>> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e
>>>>> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
>>>>> START (fentry): PID %d, Comm %s\n", pid, comm); }
>>>>> fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid];
>>>>> delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s,
>>>>> Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
>>>>> Attaching 2 probes...
>>>>> ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec
>>>>>
>>>>> [AG] - Trying to skip the fexit and only do fentry - he still rejects
>>>>> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e
>>>>> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
>>>>> START (fentry): PID %d, Comm %s\n", pid, comm); }'
>>>>> sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
>>>>> resolution
>>>>> INFO: node count: 12
>>>>> Attaching 1 probe...
>>>>>
>>>>> Program ID: 295
>>>>>
>>>>> The verifier log:
>>>>> processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3
>>>>> peak_states 3 mark_read 1
>>>>>
>>>>> Attaching kfunc:vmlinux:begin_new_exec
>>>>> ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec
>>>> OK, I could reproduce this issue and found the issue. In my test,
>>>> fexit+livepatch works on some older kernel, but fails on some newer
>>>> kernel. I haven't bisected to the commit that broke it.
>>>>
>>>> Something like the following make it work:
>>>>
>>>> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
>>>> index 2e113f8b13a2..4277b4f33eb8 100644
>>>> --- i/kernel/trace/ftrace.c
>>>> +++ w/kernel/trace/ftrace.c
>>>> @@ -5985,6 +5985,8 @@ int register_ftrace_direct(struct ftrace_ops
>>>> *ops, unsigned long addr)
>>>>           ops->direct_call = addr;
>>>>
>>>>           err = register_ftrace_function_nolock(ops);
>>>> +       if (err)
>>>> +               remove_direct_functions_hash(direct_functions, addr);
>>>>
>>>>     out_unlock:
>>>>           mutex_unlock(&direct_mutex);
>>>>
>>>> Andrey, could you also test this change?
>>> Attached is a better version of the fix.
>>>
>>> Thanks,
>>> Song
>> Thank you Song!
>>
>> So, with this You say both fentry and fexit will work no issues ?
> Yes, fentry and fexit should both work.
>
>> So juts to understand, as i am not familiar with live-patch generation,
>> I get the sources for my Ubuntu kernel, I apply your patch, I also
>> generate manually livepatch module that makes a dummy patching to my
>> test function (begin_new_exec), and apply this patch to my running
>> costum kernel ? Because i don't think the stadard ubuntu livepatching
>> will agree to apply his livepatch CVEs to my ostum kenel, it will
>> probably recognize it's not stock ubuntu kernel but amanully built one.
> livepatch is a kernel module. Therefore, unless the two kernels are almost
> identical, livepatch built for one kernel cannot be used on the other.
>
> If you build the kernel from source code, there are some samples in
> samples/livepatch that you can use for testing. PS: You need to enable
>
>    CONFIG_SAMPLE_LIVEPATCH=m
>
> I hope this helps.

Thanks Song, working on repro, kernel rebuilt, test module is loading 
but, bpftrace is refusing to attach now to fentries/fexits claiming the 
costum kernel is not supporting it. It did
attach in the case of stock AWS kernel i copied the .config from. So 
just trying to figure out now if some Kcofnig flags are missing or 
different . Let me know in case you manage to confirm yourself in the 
meanwhile the fix works for
you.

Thanks,
Andrey


>
> Thanks,
> Song

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Song Liu]

On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
<andrey.grodzovsky@crowdstrike.com> wrote:
[...]
> > If you build the kernel from source code, there are some samples in
> > samples/livepatch that you can use for testing. PS: You need to enable
> >
> >    CONFIG_SAMPLE_LIVEPATCH=m
> >
> > I hope this helps.
>
> Thanks Song, working on repro, kernel rebuilt, test module is loading
> but, bpftrace is refusing to attach now to fentries/fexits claiming the
> costum kernel is not supporting it. It did
> attach in the case of stock AWS kernel i copied the .config from. So
> just trying to figure out now if some Kcofnig flags are missing or
> different . Let me know in case you manage to confirm yourself in the
> meanwhile the fix works for
> you.

Yes, it worked in my tests.

[root@(none) /]# kpatch load linux/samples/livepatch/livepatch-sample.ko
loading patch module: linux/samples/livepatch/livepatch-sample.ko
[root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show
{printf("fexit\n");}' &
[1] 388
[root@(none) /]# Attached 1 probe
[root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show
{printf("fentry\n");}' &
[2] 397
[root@(none) /]# Attached 1 probe

[root@(none) /]# cat /proc/cmdline
this has been live patched
fentry
fexit

Thanks,
Song

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Andrey Grodzovsky]

[-- Attachment #1: Type: text/plain, Size: 4982 bytes --]

On 10/20/25 12:03, Song Liu wrote:
> On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
> <andrey.grodzovsky@crowdstrike.com> wrote:
> [...]
>>> If you build the kernel from source code, there are some samples in
>>> samples/livepatch that you can use for testing. PS: You need to enable
>>>
>>>     CONFIG_SAMPLE_LIVEPATCH=m
>>>
>>> I hope this helps.
>> Thanks Song, working on repro, kernel rebuilt, test module is loading
>> but, bpftrace is refusing to attach now to fentries/fexits claiming the
>> costum kernel is not supporting it. It did
>> attach in the case of stock AWS kernel i copied the .config from. So
>> just trying to figure out now if some Kcofnig flags are missing or
>> different . Let me know in case you manage to confirm yourself in the
>> meanwhile the fix works for
>> you.
> Yes, it worked in my tests.
>
> [root@(none) /]# kpatch load linux/samples/livepatch/livepatch-sample.ko
> loading patch module: linux/samples/livepatch/livepatch-sample.ko
> [root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show
> {printf("fexit\n");}' &
> [1] 388
> [root@(none) /]# Attached 1 probe
> [root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show
> {printf("fentry\n");}' &
> [2] 397
> [root@(none) /]# Attached 1 probe
>
> [root@(none) /]# cat /proc/cmdline
> this has been live patched
> fentry
> fexit
>
> Thanks,
> Song
>

Verified the failures I observe when trying to attach with BPF trace are 
only in presence of patch you provided.
Please see attached dmesg for failures. Initial warning on boot. 
Subsequebt warnings and errors at the point i try to run
sudo bpftrace -e "fexit:cmdline_proc_show { printf(\"fexit hit\\n\"); 
exit(); }"

sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name 
resolution
stdin:1:1-25: ERROR: kfunc/kretfunc not available for your kernel version.

ubuntu@ip-10-10-115-238:~/linux-6.8.1$ sudo cat 
/sys/kernel/debug/tracing/available_filter_functions | grep 
cmdline_proc_show
sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name 
resolution
cat: /sys/kernel/debug/tracing/available_filter_functions: No such device

After reboot and before trying to attacg with bpftrace, 
/sys/kernel/debug/tracing/available_filter_functions is available and 
shows all function.

Using stable kernel from 
https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.8.1.tar.gz for build.
FTRACE related KCONFIGs bellow

ubuntu@ip-10-10-115-238:~$ cat /boot/config-6.8.1 | grep TRACE
CONFIG_TASKS_TRACE_RCU=y
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_TRACEPOINTS=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_PM_TRACE=y
CONFIG_PM_TRACE_RTC=y
CONFIG_KPROBES_ON_FTRACE=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_ARCH_CORRECT_STACKTRACE_ON_KRETPROBE=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_TRACE_IRQFLAGS_NMI_SUPPORT=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_RELIABLE_STACKTRACE=y
CONFIG_NETFILTER_XT_TARGET_TRACE=m
# CONFIG_DMA_FENCE_TRACE is not set
CONFIG_SCSI_IPR_TRACE=y
# CONFIG_ATH5K_TRACER is not set
# CONFIG_DRM_I915_LOW_LEVEL_TRACEPOINTS is not set
CONFIG_STM_SOURCE_FTRACE=m
# CONFIG_PSTORE_FTRACE is not set
# CONFIG_STACKTRACE_BUILD_ID is not set
CONFIG_STACKTRACE=y
# CONFIG_RCU_TRACE is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS=y
CONFIG_HAVE_DYNAMIC_FTRACE_NO_PATCHABLE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACE_CLOCK=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_GENERIC_TRACER=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
CONFIG_DYNAMIC_FTRACE_WITH_ARGS=y
CONFIG_STACK_TRACER=y
# CONFIG_IRQSOFF_TRACER is not set
CONFIG_SCHED_TRACER=y
CONFIG_HWLAT_TRACER=y
CONFIG_OSNOISE_TRACER=y
CONFIG_TIMERLAT_TRACER=y
CONFIG_MMIOTRACE=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_TRACER_SNAPSHOT=y
# CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP is not set
CONFIG_BLK_DEV_IO_TRACE=y
# CONFIG_KPROBE_EVENTS_ON_NOTRACE is not set
CONFIG_FTRACE_MCOUNT_RECORD=y
CONFIG_FTRACE_MCOUNT_USE_CC=y
CONFIG_TRACE_EVENT_INJECT=y
# CONFIG_TRACEPOINT_BENCHMARK is not set
# CONFIG_TRACE_EVAL_MAP_FILE is not set
# CONFIG_FTRACE_RECORD_RECURSION is not set
# CONFIG_FTRACE_STARTUP_TEST is not set
CONFIG_FTRACE_SORT_STARTUP_TEST=y
# CONFIG_MMIOTRACE_TEST is not set
# CONFIG_SAMPLE_TRACE_EVENTS is not set
# CONFIG_SAMPLE_TRACE_CUSTOM_EVENTS is not set
CONFIG_SAMPLE_TRACE_PRINTK=m
CONFIG_SAMPLE_FTRACE_DIRECT=m
# CONFIG_SAMPLE_FTRACE_DIRECT_MULTI is not set
# CONFIG_SAMPLE_FTRACE_OPS is not set
CONFIG_SAMPLE_TRACE_ARRAY=m
CONFIG_HAVE_SAMPLE_FTRACE_DIRECT=y
CONFIG_HAVE_SAMPLE_FTRACE_DIRECT_MULTI=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_CC_HAS_SANCOV_TRACE_PC=y
# CONFIG_BACKTRACE_SELF_TEST is not set


Andrey




[-- Attachment #2: dmesg --]
[-- Type: text/plain, Size: 28141 bytes --]

[    0.958579] NET: Registered PF_INET6 protocol family
[    0.960794] Segment Routing with IPv6
[    0.961819] In-situ OAM (IOAM) with IPv6
[    0.962927] NET: Registered PF_PACKET protocol family
[    0.964393] Key type dns_resolver registered
[    0.968676] No MBM correction factor available
[    0.969933] IPI shorthand broadcast: enabled
[    0.972554] sched_clock: Marking stable (673007499, 299279483)->(1155822036, -183535054)
[    0.974892] registered taskstats version 1
[    0.977043] Loading compiled-in X.509 certificates
[    0.979081] Loaded X.509 cert 'Build time autogenerated kernel key: 774d7f8f61628ad4e8d329abbb98ef067d7b43a9'
[    0.985242] Key type .fscrypt registered
[    0.986488] Key type fscrypt-provisioning registered
[    0.988673] Key type encrypted registered
[    0.989923] AppArmor: AppArmor sha256 policy hashing enabled
[    0.991934] ima: No TPM chip found, activating TPM-bypass!
[    0.993570] Loading compiled-in module X.509 certificates
[    0.996021] Loaded X.509 cert 'Build time autogenerated kernel key: 774d7f8f61628ad4e8d329abbb98ef067d7b43a9'
[    0.999303] ima: Allocated hash algorithm: sha256
[    1.000769] ima: No architecture policies found
[    1.002184] evm: Initialising EVM extended attributes:
[    1.003740] evm: security.selinux
[    1.004829] evm: security.SMACK64
[    1.005916] evm: security.SMACK64EXEC
[    1.007085] evm: security.SMACK64TRANSMUTE
[    1.008389] evm: security.SMACK64MMAP
[    1.009567] evm: security.apparmor
[    1.010663] evm: security.ima
[    1.011651] evm: security.capability
[    1.012788] evm: HMAC attrs: 0x1
[    1.014248] PM:   Magic number: 13:837:285
[    1.019035] ------------[ cut here ]------------
[    1.020163] WARNING: CPU: 10 PID: 1 at kernel/trace/ftrace.c:2589 ftrace_get_addr_new+0x10c/0x120
[    1.022417] Modules linked in:
[    1.023417] CPU: 10 PID: 1 Comm: swapper/0 Not tainted 6.8.1 #1
[    1.025148] Hardware name: Amazon EC2 c5.4xlarge/, BIOS 1.0 10/16/2017
[    1.027039] RIP: 0010:ftrace_get_addr_new+0x10c/0x120
[    1.028559] Code: 00 48 85 c0 74 9d 5b 41 5c 41 5d 5d 31 d2 31 c9 31 f6 31 ff c3 cc cc cc cc 48 8b 3f e8 5d fe ff ff 48 85 c0 0f 85 38 ff ff ff <0f> 0b 48 8b 43 08 e9 0f ff ff ff 66 0f 1f 84 00 00 00 00 00 90 90
[    1.033947] RSP: 0000:ffffa8bbc0027808 EFLAGS: 00010246
[    1.035521] RAX: 0000000000000000 RBX: ffff88bf800ec540 RCX: 0000000000000000
[    1.037587] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[    1.039648] RBP: ffffa8bbc0027820 R08: 0000000000000000 R09: 0000000000000000
[    1.041713] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88bf800ec540
[    1.043773] R13: 0000000000000001 R14: ffffffffa39aef08 R15: ffffffffa2206cca
[    1.045836] FS:  0000000000000000(0000) GS:ffff88c665900000(0000) knlGS:0000000000000000
[    1.048143] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.049851] CR2: 0000000000000000 CR3: 000000037203c001 CR4: 00000000007706f0
[    1.051926] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    1.053992] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    1.056061] PKRU: 55555554
[    1.056978] Call Trace:
[    1.057833]  <TASK>
[    1.058584]  ? show_regs+0x6d/0x80
[    1.059676]  ? __warn+0x89/0x150
[    1.060714]  ? ftrace_get_addr_new+0x10c/0x120
[    1.062073]  ? report_bug+0x16a/0x190
[    1.063219]  ? handle_bug+0x51/0xa0
[    1.064351]  ? exc_invalid_op+0x18/0x80
[    1.065547]  ? asm_exc_invalid_op+0x1b/0x20
[    1.066840]  ? ftrace_get_addr_new+0x10c/0x120
[    1.068207]  ? ftrace_get_addr_new+0x103/0x120
[    1.069576]  ftrace_replace_code+0x11d/0x1b0
[    1.070894]  ftrace_modify_all_code+0x15e/0x1a0
[    1.072317]  arch_ftrace_update_code+0x9/0x20
[    1.073666]  ftrace_startup+0xcc/0x180
[    1.074855]  register_ftrace_function_nolock+0x24/0x70
[    1.076438]  register_ftrace_direct+0x2ed/0x370
[    1.077838]  ? 0xffffffffc010ca00
[    1.078923]  ? __pfx___hid_bpf_tail_call+0x10/0x10
[    1.080404]  bpf_trampoline_update+0x469/0x650
[    1.081775]  ? 0xffffffffc010ca00
[    1.082833]  ? 0xffffffffc010ca00
[    1.083881]  __bpf_trampoline_link_prog+0x10d/0x2d0
[    1.085359]  bpf_trampoline_link_prog+0x2b/0x50
[    1.086700]  bpf_tracing_prog_attach+0x3c2/0x5f0
[    1.088073]  bpf_raw_tp_link_attach+0x206/0x2d0
[    1.089430]  __sys_bpf+0x1120/0x2500
[    1.090531]  kern_sys_bpf+0x61/0x1e0
[    1.091635]  ? close_fd+0x58/0x80
[    1.092664]  skel_raw_tracepoint_open.constprop.0+0x43/0x70
[    1.094297]  hid_bpf_preload_skel+0x111/0x180
[    1.095596]  ? __pfx_hid_bpf_init+0x10/0x10
[    1.096849]  hid_bpf_init+0x1e/0xc0
[    1.097930]  do_one_initcall+0x5e/0x310
[    1.099092]  kernel_init_freeable+0x359/0x500
[    1.100403]  ? __pfx_kernel_init+0x10/0x10
[    1.101628]  kernel_init+0x1b/0x200
[    1.102695]  ret_from_fork+0x47/0x70
[    1.103790]  ? __pfx_kernel_init+0x10/0x10
[    1.105011]  ret_from_fork_asm+0x1b/0x30
[    1.106207]  </TASK>
[    1.106931] ---[ end trace 0000000000000000 ]---
[    1.109385] RAS: Correctable Errors collector initialized.
[    1.111113] clk: Disabling unused clocks
[    1.416156] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
[    1.418725] md: Waiting for all devices to be available before autodetect
[    1.419892] md: If you don't use raid, use raid=noautodetect
[    1.420894] md: Autodetecting RAID arrays.
[    1.421656] md: autorun ...
[    1.422215] md: ... autorun DONE.
[    1.429543] EXT4-fs (nvme0n1p1): mounted filesystem 6387caa1-122f-48bc-b447-9f1386d06e06 ro with ordered data mode. Quota mode: none.
[    1.431922] VFS: Mounted root (ext4 filesystem) readonly on device 259:1.
[    1.434051] devtmpfs: mounted
[    1.435734] Freeing unused decrypted memory: 2028K
[    1.437643] Freeing unused kernel image (initmem) memory: 4984K
[    1.439121] Write protecting the kernel read-only data: 34816k
[    1.440674] Freeing unused kernel image (rodata/data gap) memory: 872K
[    1.486425] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    1.487559] x86/mm: Checking user space page tables
[    1.531530] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    1.532674] Run /sbin/init as init process
[    1.533460]   with arguments:
[    1.533462]     /sbin/init
[    1.533463]   with environment:
[    1.533464]     HOME=/
[    1.533465]     TERM=linux
[    1.533465]     BOOT_IMAGE=/vmlinuz-6.8.1
[    1.691160] systemd[1]: Inserted module 'autofs4'
[    1.739692] systemd[1]: systemd 255.4-1ubuntu8 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    1.745330] systemd[1]: Detected virtualization amazon.
[    1.746574] systemd[1]: Detected architecture x86-64.
[    1.751410] systemd[1]: Hostname set to <ip-10-10-115-238>.
[    2.144939] systemd[1]: Queued start job for default target graphical.target.
[    2.169782] systemd[1]: Created slice system-modprobe.slice - Slice /system/modprobe.
[    2.173549] systemd[1]: Created slice system-serial\x2dgetty.slice - Slice /system/serial-getty.
[    2.177997] systemd[1]: Created slice system-systemd\x2dfsck.slice - Slice /system/systemd-fsck.
[    2.181785] systemd[1]: Created slice user.slice - User and Session Slice.
[    2.184612] systemd[1]: Started ntpsec-systemd-netif.path.
[    2.187009] systemd[1]: Started systemd-ask-password-wall.path - Forward Password Requests to Wall Directory Watch.
[    2.191530] systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.
[    2.196152] systemd[1]: Expecting device dev-disk-by\x2dlabel-BOOT.device - /dev/disk/by-label/BOOT...
[    2.200018] systemd[1]: Expecting device dev-disk-by\x2dlabel-UEFI.device - /dev/disk/by-label/UEFI...
[    2.203876] systemd[1]: Expecting device dev-ttyS0.device - /dev/ttyS0...
[    2.206699] systemd[1]: Reached target integritysetup.target - Local Integrity Protected Volumes.
[    2.210142] systemd[1]: Reached target slices.target - Slice Units.
[    2.213182] systemd[1]: Reached target swap.target - Swaps.
[    2.215621] systemd[1]: Reached target veritysetup.target - Local Verity Protected Volumes.
[    2.219064] systemd[1]: Listening on dm-event.socket - Device-mapper event daemon FIFOs.
[    2.222434] systemd[1]: Listening on lvm2-lvmpolld.socket - LVM2 poll daemon socket.
[    2.226036] systemd[1]: Listening on multipathd.socket - multipathd control socket.
[    2.242149] systemd[1]: Listening on rpcbind.socket - RPCbind Server Activation Socket.
[    2.246074] systemd[1]: Listening on syslog.socket - Syslog Socket.
[    2.249145] systemd[1]: Listening on systemd-fsckd.socket - fsck to fsckd communication Socket.
[    2.252891] systemd[1]: Listening on systemd-initctl.socket - initctl Compatibility Named Pipe.
[    2.256663] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log).
[    2.260465] systemd[1]: Listening on systemd-journald.socket - Journal Socket.
[    2.263756] systemd[1]: Listening on systemd-networkd.socket - Network Service Netlink Socket.
[    2.267430] systemd[1]: systemd-pcrextend.socket - TPM2 PCR Extension (Varlink) was skipped because of an unmet condition check (ConditionSecurity=measured-uki).
[    2.270426] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socket.
[    2.273935] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket.
[    2.291382] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    2.295171] systemd[1]: Mounting dev-mqueue.mount - POSIX Message Queue File System...
[    2.299230] systemd[1]: Mounting sys-kernel-debug.mount - Kernel Debug File System...
[    2.303059] systemd[1]: Mounting sys-kernel-tracing.mount - Kernel Trace File System...
[    2.306079] systemd[1]: auth-rpcgss-module.service - Kernel Module supporting RPCSEC_GSS was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab).
[    2.310195] systemd[1]: Starting keyboard-setup.service - Set the console keyboard layout...
[    2.314585] systemd[1]: Starting kmod-static-nodes.service - Create List of Static Device Nodes...
[    2.319415] systemd[1]: Starting lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[    2.324999] systemd[1]: Starting modprobe@configfs.service - Load Kernel Module configfs...
[    2.328934] systemd[1]: Starting modprobe@dm_mod.service - Load Kernel Module dm_mod...
[    2.332977] systemd[1]: Starting modprobe@drm.service - Load Kernel Module drm...
[    2.337481] systemd[1]: Starting modprobe@efi_pstore.service - Load Kernel Module efi_pstore...
[    2.341518] systemd[1]: Starting modprobe@fuse.service - Load Kernel Module fuse...
[    2.345226] systemd[1]: Starting modprobe@loop.service - Load Kernel Module loop...
[    2.348111] systemd[1]: netplan-ovs-cleanup.service - OpenVSwitch configuration for cleanup was skipped because of an unmet condition check (ConditionFileIsExecutable=/usr/bin/ovs-vsctl).
[    2.351100] pstore: Using crash dump compression: deflate
[    2.352436] systemd[1]: Starting systemd-fsck-root.service - File System Check on Root Device...
[    2.356026] pstore: Registered efi_pstore as persistent store backend
[    2.358847] systemd[1]: Starting systemd-journald.service - Journal Service...
[    2.364360] systemd[1]: Starting systemd-modules-load.service - Load Kernel Modules...
[    2.366617] systemd[1]: systemd-pcrmachine.service - TPM2 PCR Machine ID Measurement was skipped because of an unmet condition check (ConditionSecurity=measured-uki).
[    2.370028] systemd[1]: systemd-tpm2-setup-early.service - TPM2 SRK Setup (Early) was skipped because of an unmet condition check (ConditionSecurity=measured-uki).
[    2.373850] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...
[    2.377223] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    2.380691] systemd[1]: Mounted dev-mqueue.mount - POSIX Message Queue File System.
[    2.383791] systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.
[    2.385162] systemd-journald[255]: Collecting audit messages is disabled.
[    2.387968] systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.
[    2.391368] systemd[1]: Finished keyboard-setup.service - Set the console keyboard layout.
[    2.394953] systemd[1]: Finished kmod-static-nodes.service - Create List of Static Device Nodes.
[    2.398483] systemd[1]: modprobe@configfs.service: Deactivated successfully.
[    2.399910] systemd[1]: Finished modprobe@configfs.service - Load Kernel Module configfs.
[    2.403776] systemd[1]: modprobe@dm_mod.service: Deactivated successfully.
[    2.405468] systemd[1]: Finished modprobe@dm_mod.service - Load Kernel Module dm_mod.
[    2.408695] systemd[1]: Finished lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
[    2.412929] systemd[1]: modprobe@drm.service: Deactivated successfully.
[    2.414661] systemd[1]: Finished modprobe@drm.service - Load Kernel Module drm.
[    2.417973] systemd[1]: Started systemd-journald.service - Journal Service.
[    2.545456] EXT4-fs (nvme0n1p1): re-mounted 6387caa1-122f-48bc-b447-9f1386d06e06 r/w. Quota mode: none.
[    2.572734] systemd-journald[255]: Received client request to flush runtime journal.
[    2.594301] systemd-journald[255]: /var/log/journal/ec220e94df30bda465b15689b7378e3a/system.journal: Journal file uses a different sequence number ID, rotating.
[    2.594307] systemd-journald[255]: Rotating system journal.
[    2.761565] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
[    2.786745] ena 0000:00:05.0: ENA device version: 0.10
[    2.786750] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    2.802370] parport_pc 00:03: reported by Plug and Play ACPI
[    2.841917] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0510000, mac addr 12:67:59:84:01:a5
[    2.852675] ppdev: user-space parallel port driver
[    2.853640] ena 0000:00:05.0 ens5: renamed from eth0
[    2.860372] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer
[    2.866183] cryptd: max_cpu_qlen set to 1000
[    2.899090] AVX2 version of gcm_enc/dec engaged.
[    2.899202] AES CTR mode by8 optimization enabled
[    2.984184] EXT4-fs (nvme0n1p16): mounted filesystem 9d39f3ff-b465-4c20-9ec1-06182226356c r/w with ordered data mode. Quota mode: none.
[    3.084656] audit: type=1400 audit(1760973474.672:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="busybox" pid=708 comm="apparmor_parser"
[    3.084663] audit: type=1400 audit(1760973474.672:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="crun" pid=714 comm="apparmor_parser"
[    3.084665] audit: type=1400 audit(1760973474.672:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="ch-checkns" pid=710 comm="apparmor_parser"
[    3.084668] audit: type=1400 audit(1760973474.672:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="devhelp" pid=715 comm="apparmor_parser"
[    3.084671] audit: type=1400 audit(1760973474.672:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="Discord" pid=703 comm="apparmor_parser"
[    3.084697] audit: type=1400 audit(1760973474.672:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="brave" pid=706 comm="apparmor_parser"
[    3.084727] audit: type=1400 audit(1760973474.672:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="vscode" pid=713 comm="apparmor_parser"
[    3.084763] audit: type=1400 audit(1760973474.672:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="ch-run" pid=711 comm="apparmor_parser"
[    3.084794] audit: type=1400 audit(1760973474.672:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="element-desktop" pid=716 comm="apparmor_parser"
[    3.502649] RPC: Registered named UNIX socket transport module.
[    3.502652] RPC: Registered udp transport module.
[    3.502653] RPC: Registered tcp transport module.
[    3.502654] RPC: Registered tcp-with-tls transport module.
[    3.502655] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    3.674659] input: ImPS/2 Generic Wheel Mouse as /devices/platform/i8042/serio1/input/input4
[    4.013037] 8021q: 802.1Q VLAN Support v1.8
[    4.140597] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[    4.140856] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[    4.140992] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600'
[    4.141693] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[    4.141696] cfg80211: failed to load regulatory.db
[    9.730872] kauditd_printk_skb: 94 callbacks suppressed
[    9.730875] audit: type=1400 audit(1760973481.318:105): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="rsyslogd" pid=1152 comm="apparmor_parser"
[    9.906210] NET: Registered PF_QIPCRTR protocol family
[   10.657618] audit: type=1400 audit(1760973482.245:106): apparmor="STATUS" operation="profile_load" profile="unconfined" name="docker-default" pid=1353 comm="apparmor_parser"
[   10.743033] evm: overlay not supported
[   10.843306] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   10.852812] Bridge firewalling registered
[   11.008539] Initializing XFRM netlink socket
[   49.644664] systemd-journald[255]: /var/log/journal/ec220e94df30bda465b15689b7378e3a/user-1000.journal: Journal file uses a different sequence number ID, rotating.
[ 1237.397558] ------------[ cut here ]------------
[ 1237.397562] WARNING: CPU: 12 PID: 1988 at arch/x86/kernel/ftrace.c:99 ftrace_verify_code+0x4a/0x90
[ 1237.397569] Modules linked in: xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables libcrc32c br_netfilter bridge qrtr overlay cfg80211 8021q garp mrp stp llc sunrpc binfmt_misc intel_rapl_msr intel_rapl_common nls_iso8859_1 intel_uncore_frequency_common isst_if_common nfit crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd rapl ppdev parport_pc psmouse ena parport i2c_piix4 input_leds mac_hid serio_raw dm_multipath msr efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4
[ 1237.397610] CPU: 12 PID: 1988 Comm: bpftrace Tainted: G        W          6.8.1 #1
[ 1237.397613] Hardware name: Amazon EC2 c5.4xlarge/, BIOS 1.0 10/16/2017
[ 1237.397614] RIP: 0010:ftrace_verify_code+0x4a/0x90
[ 1237.397616] Code: 00 48 89 45 f0 31 c0 c7 45 eb 00 00 00 00 c6 45 ef 00 e8 09 9c 2f 00 48 85 c0 75 41 8b 03 39 45 eb 74 2d 48 89 1d 26 01 be 02 <0f> 0b b8 ea ff ff ff 48 8b 55 f0 65 48 2b 14 25 28 00 00 00 75 26
[ 1237.397618] RSP: 0018:ffffa8bbc1d7fbd8 EFLAGS: 00010286
[ 1237.397621] RAX: 000000001d8d3be8 RBX: ffffffffa39aef08 RCX: 0000000000000000
[ 1237.397622] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 1237.397623] RBP: ffffa8bbc1d7fbf0 R08: 0000000000000000 R09: 0000000000000000
[ 1237.397625] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa39aef08
[ 1237.397626] R13: 0000000000000001 R14: ffff88bf80114d20 R15: ffffffffa39aef08
[ 1237.397627] FS:  00007fa3bcbd0780(0000) GS:ffff88c665a00000(0000) knlGS:0000000000000000
[ 1237.397629] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1237.397630] CR2: 00007ffe71155124 CR3: 0000000117032001 CR4: 00000000007706f0
[ 1237.397636] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1237.397637] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1237.397639] PKRU: 55555554
[ 1237.397640] Call Trace:
[ 1237.397641]  <TASK>
[ 1237.397643]  ? show_regs+0x6d/0x80
[ 1237.397649]  ? __warn+0x89/0x150
[ 1237.397653]  ? ftrace_verify_code+0x4a/0x90
[ 1237.397655]  ? report_bug+0x16a/0x190
[ 1237.397659]  ? handle_bug+0x51/0xa0
[ 1237.397662]  ? exc_invalid_op+0x18/0x80
[ 1237.397664]  ? asm_exc_invalid_op+0x1b/0x20
[ 1237.397668]  ? ftrace_verify_code+0x4a/0x90
[ 1237.397669]  ? ftrace_verify_code+0x37/0x90
[ 1237.397670]  ftrace_replace_code+0x71/0x1b0
[ 1237.397673]  ftrace_modify_all_code+0x125/0x1a0
[ 1237.397675]  ? 0xffffffffc01206c0
[ 1237.397684]  arch_ftrace_update_code+0x9/0x20
[ 1237.397686]  ftrace_shutdown.part.0+0x114/0x1e0
[ 1237.397688]  ? 0xffffffffc01206c0
[ 1237.397690]  ? 0xffffffffc01206c0
[ 1237.397691]  unregister_ftrace_function+0x47/0x170
[ 1237.397693]  ? 0xffffffffc01206c0
[ 1237.397694]  ? 0xffffffffc01206c0
[ 1237.397696]  unregister_ftrace_direct+0x60/0x1e0
[ 1237.397697]  ? kmalloc_trace+0x136/0x310
[ 1237.397700]  ? bpf_trampoline_update+0x68/0x650
[ 1237.397703]  ? 0xffffffffc01206c0
[ 1237.397704]  ? __pfx_sched_fork+0x10/0x10
[ 1237.397707]  bpf_trampoline_update+0x4f7/0x650
[ 1237.397709]  bpf_trampoline_unlink_prog+0x9d/0x130
[ 1237.397711]  bpf_tracing_link_release+0x16/0x50
[ 1237.397715]  bpf_link_free+0x55/0x80
[ 1237.397717]  bpf_link_release+0x26/0x40
[ 1237.397719]  __fput+0xa1/0x2e0
[ 1237.397723]  __fput_sync+0x1c/0x30
[ 1237.397725]  __x64_sys_close+0x3e/0x90
[ 1237.397728]  do_syscall_64+0x85/0x180
[ 1237.397732]  ? syscall_exit_to_user_mode+0x86/0x260
[ 1237.397736]  ? do_syscall_64+0x92/0x180
[ 1237.397738]  ? do_syscall_64+0x92/0x180
[ 1237.397740]  ? do_syscall_64+0x92/0x180
[ 1237.397743]  ? irqentry_exit+0x21/0x40
[ 1237.397745]  ? exc_page_fault+0x94/0x190
[ 1237.397748]  entry_SYSCALL_64_after_hwframe+0x6e/0x76
[ 1237.397752] RIP: 0033:0x7fa3b4f166f4
[ 1237.397755] Code: 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 80 3d 55 49 0f 00 00 74 13 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 3c c3 0f 1f 00 55 48 89 e5 48 83 ec 10 89 7d
[ 1237.397756] RSP: 002b:00007ffe7113d028 EFLAGS: 00000202 ORIG_RAX: 0000000000000003
[ 1237.397758] RAX: ffffffffffffffda RBX: 0000555e10456800 RCX: 00007fa3b4f166f4
[ 1237.397760] RDX: 0000000000000010 RSI: 00007ffe7113cf60 RDI: 0000000000000004
[ 1237.397761] RBP: 00007ffe7113d080 R08: 0000555e0f7fb86c R09: 0000000000000000
[ 1237.397762] R10: 00007ffe7113ba00 R11: 0000000000000202 R12: 0000000000000004
[ 1237.397763] R13: 0000000000000003 R14: 0000555e0f7faeac R15: 00007ffe7113d750
[ 1237.397765]  </TASK>
[ 1237.397766] ---[ end trace 0000000000000000 ]---
[ 1237.397767] ------------[ ftrace bug ]------------
[ 1237.397768] ftrace failed to modify 
[ 1237.397769] [<ffffffffa0f47980>] sched_fork+0x0/0x260
[ 1237.397771]  actual:   e8:db:38:f6:ff
[ 1237.397773]  expected: e8:3b:8d:1d:1f
[ 1237.397775] Setting ftrace call site to NOP
[ 1237.397775] ftrace record flags: e8980000
[ 1237.397776]  (0) R  	tramp: ERROR!
                expected tramp: ffffffffc01206c0
[ 1237.397787] ------------[ cut here ]------------
[ 1237.397788] WARNING: CPU: 12 PID: 1988 at kernel/trace/ftrace.c:2179 ftrace_bug+0x10d/0x2d0
[ 1237.397790] Modules linked in: xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables libcrc32c br_netfilter bridge qrtr overlay cfg80211 8021q garp mrp stp llc sunrpc binfmt_misc intel_rapl_msr intel_rapl_common nls_iso8859_1 intel_uncore_frequency_common isst_if_common nfit crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd rapl ppdev parport_pc psmouse ena parport i2c_piix4 input_leds mac_hid serio_raw dm_multipath msr efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4
[ 1237.397819] CPU: 12 PID: 1988 Comm: bpftrace Tainted: G        W          6.8.1 #1
[ 1237.397821] Hardware name: Amazon EC2 c5.4xlarge/, BIOS 1.0 10/16/2017
[ 1237.397821] RIP: 0010:ftrace_bug+0x10d/0x2d0
[ 1237.397823] Code: 07 00 e8 46 df f2 ff 41 f6 44 24 0b 08 0f 85 18 01 00 00 4c 89 e7 e8 a2 fc ff ff 48 c7 c7 7d 51 77 a2 48 89 c6 e8 23 df f2 ff <0f> 0b c7 05 af cf 3d 02 01 00 00 00 5b c7 05 b4 cf 3d 02 00 00 00
[ 1237.397824] RSP: 0018:ffffa8bbc1d7fbd8 EFLAGS: 00010246
[ 1237.397826] RAX: 0000000000000022 RBX: ffffffffa344a640 RCX: 0000000000000000
[ 1237.397827] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 1237.397828] RBP: ffffa8bbc1d7fbf0 R08: 0000000000000000 R09: 0000000000000000
[ 1237.397829] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88bf80114d20
[ 1237.397830] R13: ffffffffa0f47980 R14: ffff88bf80114d20 R15: ffffffffa39aef08
[ 1237.397831] FS:  00007fa3bcbd0780(0000) GS:ffff88c665a00000(0000) knlGS:0000000000000000
[ 1237.397833] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1237.397834] CR2: 00007ffe71155124 CR3: 0000000117032001 CR4: 00000000007706f0
[ 1237.397839] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1237.397840] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1237.397841] PKRU: 55555554
[ 1237.397842] Call Trace:
[ 1237.397843]  <TASK>
[ 1237.397844]  ? show_regs+0x6d/0x80
[ 1237.397846]  ? __warn+0x89/0x150
[ 1237.397849]  ? ftrace_bug+0x10d/0x2d0
[ 1237.397850]  ? report_bug+0x16a/0x190
[ 1237.397853]  ? handle_bug+0x51/0xa0
[ 1237.397854]  ? exc_invalid_op+0x18/0x80
[ 1237.397856]  ? asm_exc_invalid_op+0x1b/0x20
[ 1237.397858]  ? __pfx_sched_fork+0x10/0x10
[ 1237.397859]  ? ftrace_bug+0x10d/0x2d0
[ 1237.397861]  ? ftrace_bug+0x10d/0x2d0
[ 1237.397863]  ftrace_replace_code+0xc7/0x1b0
[ 1237.397864]  ftrace_modify_all_code+0x125/0x1a0
[ 1237.397866]  ? 0xffffffffc01206c0
[ 1237.397868]  arch_ftrace_update_code+0x9/0x20
[ 1237.397870]  ftrace_shutdown.part.0+0x114/0x1e0
[ 1237.397871]  ? 0xffffffffc01206c0
[ 1237.397873]  ? 0xffffffffc01206c0
[ 1237.397874]  unregister_ftrace_function+0x47/0x170
[ 1237.397876]  ? 0xffffffffc01206c0
[ 1237.397877]  ? 0xffffffffc01206c0
[ 1237.397878]  unregister_ftrace_direct+0x60/0x1e0
[ 1237.397880]  ? kmalloc_trace+0x136/0x310
[ 1237.397882]  ? bpf_trampoline_update+0x68/0x650
[ 1237.397884]  ? 0xffffffffc01206c0
[ 1237.397885]  ? __pfx_sched_fork+0x10/0x10
[ 1237.397886]  bpf_trampoline_update+0x4f7/0x650
[ 1237.397889]  bpf_trampoline_unlink_prog+0x9d/0x130
[ 1237.397891]  bpf_tracing_link_release+0x16/0x50
[ 1237.397893]  bpf_link_free+0x55/0x80
[ 1237.397895]  bpf_link_release+0x26/0x40
[ 1237.397897]  __fput+0xa1/0x2e0
[ 1237.397899]  __fput_sync+0x1c/0x30
[ 1237.397901]  __x64_sys_close+0x3e/0x90
[ 1237.397903]  do_syscall_64+0x85/0x180
[ 1237.397906]  ? syscall_exit_to_user_mode+0x86/0x260
[ 1237.397908]  ? do_syscall_64+0x92/0x180
[ 1237.397911]  ? do_syscall_64+0x92/0x180
[ 1237.397913]  ? do_syscall_64+0x92/0x180
[ 1237.397915]  ? irqentry_exit+0x21/0x40
[ 1237.397917]  ? exc_page_fault+0x94/0x190
[ 1237.397920]  entry_SYSCALL_64_after_hwframe+0x6e/0x76
[ 1237.397922] RIP: 0033:0x7fa3b4f166f4
[ 1237.397924] Code: 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 80 3d 55 49 0f 00 00 74 13 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 3c c3 0f 1f 00 55 48 89 e5 48 83 ec 10 89 7d
[ 1237.397925] RSP: 002b:00007ffe7113d028 EFLAGS: 00000202 ORIG_RAX: 0000000000000003
[ 1237.397927] RAX: ffffffffffffffda RBX: 0000555e10456800 RCX: 00007fa3b4f166f4
[ 1237.397928] RDX: 0000000000000010 RSI: 00007ffe7113cf60 RDI: 0000000000000004
[ 1237.397929] RBP: 00007ffe7113d080 R08: 0000555e0f7fb86c R09: 0000000000000000
[ 1237.397930] R10: 00007ffe7113ba00 R11: 0000000000000202 R12: 0000000000000004
[ 1237.397931] R13: 0000000000000003 R14: 0000555e0f7faeac R15: 00007ffe7113d750
[ 1237.397933]  </TASK>
[ 1237.397934] ---[ end trace 0000000000000000 ]---

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Song Liu]

On Mon, Oct 20, 2025 at 9:45 AM Andrey Grodzovsky
<andrey.grodzovsky@crowdstrike.com> wrote:
>
> On 10/20/25 12:03, Song Liu wrote:
> > On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
> > <andrey.grodzovsky@crowdstrike.com> wrote:
> > [...]
> >>> If you build the kernel from source code, there are some samples in
> >>> samples/livepatch that you can use for testing. PS: You need to enable
> >>>
> >>>     CONFIG_SAMPLE_LIVEPATCH=m
> >>>
> >>> I hope this helps.
> >> Thanks Song, working on repro, kernel rebuilt, test module is loading
> >> but, bpftrace is refusing to attach now to fentries/fexits claiming the
> >> costum kernel is not supporting it. It did
> >> attach in the case of stock AWS kernel i copied the .config from. So
> >> just trying to figure out now if some Kcofnig flags are missing or
> >> different . Let me know in case you manage to confirm yourself in the
> >> meanwhile the fix works for
> >> you.
> > Yes, it worked in my tests.
> >
> > [root@(none) /]# kpatch load linux/samples/livepatch/livepatch-sample.ko
> > loading patch module: linux/samples/livepatch/livepatch-sample.ko
> > [root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show
> > {printf("fexit\n");}' &
> > [1] 388
> > [root@(none) /]# Attached 1 probe
> > [root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show
> > {printf("fentry\n");}' &
> > [2] 397
> > [root@(none) /]# Attached 1 probe
> >
> > [root@(none) /]# cat /proc/cmdline
> > this has been live patched
> > fentry
> > fexit
> >
> > Thanks,
> > Song
> >
>
> Verified the failures I observe when trying to attach with BPF trace are
> only in presence of patch you provided.
> Please see attached dmesg for failures. Initial warning on boot.
> Subsequebt warnings and errors at the point i try to run
> sudo bpftrace -e "fexit:cmdline_proc_show { printf(\"fexit hit\\n\");
> exit(); }"
>
> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name
> resolution
> stdin:1:1-25: ERROR: kfunc/kretfunc not available for your kernel version.
>
> ubuntu@ip-10-10-115-238:~/linux-6.8.1$ sudo cat
> /sys/kernel/debug/tracing/available_filter_functions | grep
> cmdline_proc_show
> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name
> resolution
> cat: /sys/kernel/debug/tracing/available_filter_functions: No such device
>
> After reboot and before trying to attacg with bpftrace,
> /sys/kernel/debug/tracing/available_filter_functions is available and
> shows all function.
>
> Using stable kernel from
> https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.8.1.tar.gz for build.
> FTRACE related KCONFIGs bellow

I can see the similar issue with the upstream kernel. I was testing on
stable 6.17 before just know because of another issue with upstream
kernel, and somehow 6.17 kernel doesn't seem to have the issue.

To fix this, I think we should land a fix similar to the earlier diff:

diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
index 42bd2ba68a82..8f320df0ac52 100644
--- i/kernel/trace/ftrace.c
+++ w/kernel/trace/ftrace.c
@@ -6049,6 +6049,9 @@ int register_ftrace_direct(struct ftrace_ops
*ops, unsigned long addr)

        err = register_ftrace_function_nolock(ops);

+       if (err)
+               remove_direct_functions_hash(hash, addr);
+
  out_unlock:
        mutex_unlock(&direct_mutex);


Steven,

Does this change look good to you?

Thanks,
Song

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Andrey Grodzovsky]

On 10/20/25 14:53, Song Liu wrote:
> On Mon, Oct 20, 2025 at 9:45 AM Andrey Grodzovsky
> <andrey.grodzovsky@crowdstrike.com> wrote:
>> On 10/20/25 12:03, Song Liu wrote:
>>> On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
>>> <andrey.grodzovsky@crowdstrike.com> wrote:
>>> [...]
>>>>> If you build the kernel from source code, there are some samples in
>>>>> samples/livepatch that you can use for testing. PS: You need to enable
>>>>>
>>>>>      CONFIG_SAMPLE_LIVEPATCH=m
>>>>>
>>>>> I hope this helps.
>>>> Thanks Song, working on repro, kernel rebuilt, test module is loading
>>>> but, bpftrace is refusing to attach now to fentries/fexits claiming the
>>>> costum kernel is not supporting it. It did
>>>> attach in the case of stock AWS kernel i copied the .config from. So
>>>> just trying to figure out now if some Kcofnig flags are missing or
>>>> different . Let me know in case you manage to confirm yourself in the
>>>> meanwhile the fix works for
>>>> you.
>>> Yes, it worked in my tests.
>>>
>>> [root@(none) /]# kpatch load linux/samples/livepatch/livepatch-sample.ko
>>> loading patch module: linux/samples/livepatch/livepatch-sample.ko
>>> [root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show
>>> {printf("fexit\n");}' &
>>> [1] 388
>>> [root@(none) /]# Attached 1 probe
>>> [root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show
>>> {printf("fentry\n");}' &
>>> [2] 397
>>> [root@(none) /]# Attached 1 probe
>>>
>>> [root@(none) /]# cat /proc/cmdline
>>> this has been live patched
>>> fentry
>>> fexit
>>>
>>> Thanks,
>>> Song
>>>
>> Verified the failures I observe when trying to attach with BPF trace are
>> only in presence of patch you provided.
>> Please see attached dmesg for failures. Initial warning on boot.
>> Subsequebt warnings and errors at the point i try to run
>> sudo bpftrace -e "fexit:cmdline_proc_show { printf(\"fexit hit\\n\");
>> exit(); }"
>>
>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name
>> resolution
>> stdin:1:1-25: ERROR: kfunc/kretfunc not available for your kernel version.
>>
>> ubuntu@ip-10-10-115-238:~/linux-6.8.1$ sudo cat
>> /sys/kernel/debug/tracing/available_filter_functions | grep
>> cmdline_proc_show
>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name
>> resolution
>> cat: /sys/kernel/debug/tracing/available_filter_functions: No such device
>>
>> After reboot and before trying to attacg with bpftrace,
>> /sys/kernel/debug/tracing/available_filter_functions is available and
>> shows all function.
>>
>> Using stable kernel from
>> https://urldefense.com/v3/__https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.8.1.tar.gz__;!!BmdzS3_lV9HdKG8!1ZJe4jY49_xIzp4h4i4AbqpkLKoAqrXLFX2wDxhoSUDg2kSeTjy3COy9MngNDRlZhJ1oUKgf1yPqmnTY9-Y50TkA$  for build.
>> FTRACE related KCONFIGs bellow
> I can see the similar issue with the upstream kernel. I was testing on
> stable 6.17 before just know because of another issue with upstream
> kernel, and somehow 6.17 kernel doesn't seem to have the issue.
>
> To fix this, I think we should land a fix similar to the earlier diff:
>
> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
> index 42bd2ba68a82..8f320df0ac52 100644
> --- i/kernel/trace/ftrace.c
> +++ w/kernel/trace/ftrace.c
> @@ -6049,6 +6049,9 @@ int register_ftrace_direct(struct ftrace_ops
> *ops, unsigned long addr)
>
>          err = register_ftrace_function_nolock(ops);
>
> +       if (err)
> +               remove_direct_functions_hash(hash, addr);
> +
>    out_unlock:
>          mutex_unlock(&direct_mutex);
>
>
> Steven,
>
> Does this change look good to you?
>
>

Seems reasonable to me, we are simply cleaning the entry on failure so 
we don't encounter it late anymore.
So I will apply this patch ONLY and retest - correct ?

Another question - it seems you found where it broke ? I saw 'Cc: 
stable@vger.kernel.org # v6.6+' in your prev. patch.'
If so , can you please point me to the offending patch so I add this to 
my records of my discovery work of bpf coexistence
livepatching ?

Thanks,

Andrey

>
> Thanks,
> Song

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Song Liu]

On Mon, Oct 20, 2025 at 12:10 PM Andrey Grodzovsky
<andrey.grodzovsky@crowdstrike.com> wrote:
[...]
> Seems reasonable to me, we are simply cleaning the entry on failure so
> we don't encounter it late anymore.
> So I will apply this patch ONLY and retest - correct ?

Right, we only need this small change.

> Another question - it seems you found where it broke ? I saw 'Cc:
> stable@vger.kernel.org # v6.6+' in your prev. patch.'
> If so , can you please point me to the offending patch so I add this to
> my records of my discovery work of bpf coexistence
> livepatching ?

The commit that causing the issue was in the Fixes tag:

Fixes: d05cb470663a ("ftrace: Fix modification of direct_function hash
while in use")

Thanks,
Song

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Andrey Grodzovsky]

On 10/20/25 15:10, Andrey Grodzovsky wrote:
> On 10/20/25 14:53, Song Liu wrote:
>> On Mon, Oct 20, 2025 at 9:45 AM Andrey Grodzovsky
>> <andrey.grodzovsky@crowdstrike.com> wrote:
>>> On 10/20/25 12:03, Song Liu wrote:
>>>> On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
>>>> <andrey.grodzovsky@crowdstrike.com> wrote:
>>>> [...]
>>>>>> If you build the kernel from source code, there are some samples in
>>>>>> samples/livepatch that you can use for testing. PS: You need to 
>>>>>> enable
>>>>>>
>>>>>>      CONFIG_SAMPLE_LIVEPATCH=m
>>>>>>
>>>>>> I hope this helps.
>>>>> Thanks Song, working on repro, kernel rebuilt, test module is loading
>>>>> but, bpftrace is refusing to attach now to fentries/fexits 
>>>>> claiming the
>>>>> costum kernel is not supporting it. It did
>>>>> attach in the case of stock AWS kernel i copied the .config from. So
>>>>> just trying to figure out now if some Kcofnig flags are missing or
>>>>> different . Let me know in case you manage to confirm yourself in the
>>>>> meanwhile the fix works for
>>>>> you.
>>>> Yes, it worked in my tests.
>>>>
>>>> [root@(none) /]# kpatch load 
>>>> linux/samples/livepatch/livepatch-sample.ko
>>>> loading patch module: linux/samples/livepatch/livepatch-sample.ko
>>>> [root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show
>>>> {printf("fexit\n");}' &
>>>> [1] 388
>>>> [root@(none) /]# Attached 1 probe
>>>> [root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show
>>>> {printf("fentry\n");}' &
>>>> [2] 397
>>>> [root@(none) /]# Attached 1 probe
>>>>
>>>> [root@(none) /]# cat /proc/cmdline
>>>> this has been live patched
>>>> fentry
>>>> fexit
>>>>
>>>> Thanks,
>>>> Song
>>>>
>>> Verified the failures I observe when trying to attach with BPF trace 
>>> are
>>> only in presence of patch you provided.
>>> Please see attached dmesg for failures. Initial warning on boot.
>>> Subsequebt warnings and errors at the point i try to run
>>> sudo bpftrace -e "fexit:cmdline_proc_show { printf(\"fexit hit\\n\");
>>> exit(); }"
>>>
>>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in 
>>> name
>>> resolution
>>> stdin:1:1-25: ERROR: kfunc/kretfunc not available for your kernel 
>>> version.
>>>
>>> ubuntu@ip-10-10-115-238:~/linux-6.8.1$ sudo cat
>>> /sys/kernel/debug/tracing/available_filter_functions | grep
>>> cmdline_proc_show
>>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in 
>>> name
>>> resolution
>>> cat: /sys/kernel/debug/tracing/available_filter_functions: No such 
>>> device
>>>
>>> After reboot and before trying to attacg with bpftrace,
>>> /sys/kernel/debug/tracing/available_filter_functions is available and
>>> shows all function.
>>>
>>> Using stable kernel from
>>> https://urldefense.com/v3/__https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.8.1.tar.gz__;!!BmdzS3_lV9HdKG8!1ZJe4jY49_xIzp4h4i4AbqpkLKoAqrXLFX2wDxhoSUDg2kSeTjy3COy9MngNDRlZhJ1oUKgf1yPqmnTY9-Y50TkA$ 
>>> for build.
>>> FTRACE related KCONFIGs bellow
>> I can see the similar issue with the upstream kernel. I was testing on
>> stable 6.17 before just know because of another issue with upstream
>> kernel, and somehow 6.17 kernel doesn't seem to have the issue.
>>
>> To fix this, I think we should land a fix similar to the earlier diff:
>>
>> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
>> index 42bd2ba68a82..8f320df0ac52 100644
>> --- i/kernel/trace/ftrace.c
>> +++ w/kernel/trace/ftrace.c
>> @@ -6049,6 +6049,9 @@ int register_ftrace_direct(struct ftrace_ops
>> *ops, unsigned long addr)
>>
>>          err = register_ftrace_function_nolock(ops);
>>
>> +       if (err)
>> +               remove_direct_functions_hash(hash, addr);
>> +
>>    out_unlock:
>>          mutex_unlock(&direct_mutex);
>>
>>
>> Steven,
>>
>> Does this change look good to you?
>>
>>
>
> Seems reasonable to me, we are simply cleaning the entry on failure so 
> we don't encounter it late anymore.
> So I will apply this patch ONLY and retest - correct ?
>
> Another question - it seems you found where it broke ? I saw 'Cc: 
> stable@vger.kernel.org # v6.6+' in your prev. patch.'
> If so , can you please point me to the offending patch so I add this 
> to my records of my discovery work of bpf coexistence
> livepatching ?
>
> Thanks,
>
> Andrey


Update - with latest fix work find, both after loading the livepatch 
.ko  no conflicts and hooks work and,
before loading it, pre exsisting hooks keep working after the patch is 
loaded

You can add Acked-and-tested-by: Andrey Grodzovsky 
<andrey.grodzovsky@crowdstrike.com>

Once again, in case you now the exact commit that broke it, please let 
me know.

Thanks,
Andrey

>
>>
>> Thanks,
>> Song
>
>

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Andrey Grodzovsky]

On 10/20/25 15:53, Andrey Grodzovsky wrote:
> On 10/20/25 15:10, Andrey Grodzovsky wrote:
>> On 10/20/25 14:53, Song Liu wrote:
>>> On Mon, Oct 20, 2025 at 9:45 AM Andrey Grodzovsky
>>> <andrey.grodzovsky@crowdstrike.com> wrote:
>>>> On 10/20/25 12:03, Song Liu wrote:
>>>>> On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
>>>>> <andrey.grodzovsky@crowdstrike.com> wrote:
>>>>> [...]
>>>>>>> If you build the kernel from source code, there are some samples in
>>>>>>> samples/livepatch that you can use for testing. PS: You need to 
>>>>>>> enable
>>>>>>>
>>>>>>>      CONFIG_SAMPLE_LIVEPATCH=m
>>>>>>>
>>>>>>> I hope this helps.
>>>>>> Thanks Song, working on repro, kernel rebuilt, test module is 
>>>>>> loading
>>>>>> but, bpftrace is refusing to attach now to fentries/fexits 
>>>>>> claiming the
>>>>>> costum kernel is not supporting it. It did
>>>>>> attach in the case of stock AWS kernel i copied the .config from. So
>>>>>> just trying to figure out now if some Kcofnig flags are missing or
>>>>>> different . Let me know in case you manage to confirm yourself in 
>>>>>> the
>>>>>> meanwhile the fix works for
>>>>>> you.
>>>>> Yes, it worked in my tests.
>>>>>
>>>>> [root@(none) /]# kpatch load 
>>>>> linux/samples/livepatch/livepatch-sample.ko
>>>>> loading patch module: linux/samples/livepatch/livepatch-sample.ko
>>>>> [root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show
>>>>> {printf("fexit\n");}' &
>>>>> [1] 388
>>>>> [root@(none) /]# Attached 1 probe
>>>>> [root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show
>>>>> {printf("fentry\n");}' &
>>>>> [2] 397
>>>>> [root@(none) /]# Attached 1 probe
>>>>>
>>>>> [root@(none) /]# cat /proc/cmdline
>>>>> this has been live patched
>>>>> fentry
>>>>> fexit
>>>>>
>>>>> Thanks,
>>>>> Song
>>>>>
>>>> Verified the failures I observe when trying to attach with BPF 
>>>> trace are
>>>> only in presence of patch you provided.
>>>> Please see attached dmesg for failures. Initial warning on boot.
>>>> Subsequebt warnings and errors at the point i try to run
>>>> sudo bpftrace -e "fexit:cmdline_proc_show { printf(\"fexit hit\\n\");
>>>> exit(); }"
>>>>
>>>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in 
>>>> name
>>>> resolution
>>>> stdin:1:1-25: ERROR: kfunc/kretfunc not available for your kernel 
>>>> version.
>>>>
>>>> ubuntu@ip-10-10-115-238:~/linux-6.8.1$ sudo cat
>>>> /sys/kernel/debug/tracing/available_filter_functions | grep
>>>> cmdline_proc_show
>>>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in 
>>>> name
>>>> resolution
>>>> cat: /sys/kernel/debug/tracing/available_filter_functions: No such 
>>>> device
>>>>
>>>> After reboot and before trying to attacg with bpftrace,
>>>> /sys/kernel/debug/tracing/available_filter_functions is available and
>>>> shows all function.
>>>>
>>>> Using stable kernel from
>>>> https://urldefense.com/v3/__https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.8.1.tar.gz__;!!BmdzS3_lV9HdKG8!1ZJe4jY49_xIzp4h4i4AbqpkLKoAqrXLFX2wDxhoSUDg2kSeTjy3COy9MngNDRlZhJ1oUKgf1yPqmnTY9-Y50TkA$ 
>>>> for build.
>>>> FTRACE related KCONFIGs bellow
>>> I can see the similar issue with the upstream kernel. I was testing on
>>> stable 6.17 before just know because of another issue with upstream
>>> kernel, and somehow 6.17 kernel doesn't seem to have the issue.
>>>
>>> To fix this, I think we should land a fix similar to the earlier diff:
>>>
>>> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
>>> index 42bd2ba68a82..8f320df0ac52 100644
>>> --- i/kernel/trace/ftrace.c
>>> +++ w/kernel/trace/ftrace.c
>>> @@ -6049,6 +6049,9 @@ int register_ftrace_direct(struct ftrace_ops
>>> *ops, unsigned long addr)
>>>
>>>          err = register_ftrace_function_nolock(ops);
>>>
>>> +       if (err)
>>> +               remove_direct_functions_hash(hash, addr);
>>> +
>>>    out_unlock:
>>>          mutex_unlock(&direct_mutex);
>>>
>>>
>>> Steven,
>>>
>>> Does this change look good to you?
>>>
>>>
>>
>> Seems reasonable to me, we are simply cleaning the entry on failure 
>> so we don't encounter it late anymore.
>> So I will apply this patch ONLY and retest - correct ?
>>
>> Another question - it seems you found where it broke ? I saw 'Cc: 
>> stable@vger.kernel.org # v6.6+' in your prev. patch.'
>> If so , can you please point me to the offending patch so I add this 
>> to my records of my discovery work of bpf coexistence
>> livepatching ?
>>
>> Thanks,
>>
>> Andrey
>
>
> Update - with latest fix work find, both after loading the livepatch 
> .ko  no conflicts and hooks work and,
> before loading it, pre exsisting hooks keep working after the patch is 
> loaded
>
> You can add Acked-and-tested-by: Andrey Grodzovsky 
> <andrey.grodzovsky@crowdstrike.com>
>
> Once again, in case you now the exact commit that broke it, please let 
> me know.
>
> Thanks,
> Andrey


Song, I identified another issue in pre 6.6 kernel, building 
~/linux-6.5/samples/livepatch/livepatch-sample.c as ko,
before insmoding it, bpftrace fentry/fexit fires as expected, after 
insmod, while no errors reported on attachments,
the hooks stop firing, both if attaching before insmod and if attaching 
after insmod. If i rrmod the ko, existing hooks
resume working.

ubuntu@ip-10-10-115-238:~$ cat /proc/version_signature
Ubuntu 6.5.0-1008.8-aws 6.5.3
Source obtained to build the test module for the AWS kernel from the 
related stable branch - 
https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz

Let me know what you think.

Thanks,
Andrey



>
>>
>>>
>>> Thanks,
>>> Song
>>
>>
>

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Song Liu]

On Mon, Oct 20, 2025 at 2:31 PM Andrey Grodzovsky
<andrey.grodzovsky@crowdstrike.com> wrote:
[...]
> Song, I identified another issue in pre 6.6 kernel, building
> ~/linux-6.5/samples/livepatch/livepatch-sample.c as ko,
> before insmoding it, bpftrace fentry/fexit fires as expected, after
> insmod, while no errors reported on attachments,
> the hooks stop firing, both if attaching before insmod and if attaching
> after insmod. If i rrmod the ko, existing hooks
> resume working.
>
> ubuntu@ip-10-10-115-238:~$ cat /proc/version_signature
> Ubuntu 6.5.0-1008.8-aws 6.5.3
> Source obtained to build the test module for the AWS kernel from the
> related stable branch -
> https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz
>
> Let me know what you think.

I tested various stable kernels. I got:

With livepatch, fentry and fexit work on 6.3 kernels.

On 6.4 and 6.5 kernels, the combination stops working since this commit:

commit 60c8971899f3b34ad24857913c0784dab08962f0
Author: Florent Revest <revest@chromium.org>
Date:   2 years, 7 months ago

    ftrace: Make DIRECT_CALLS work WITH_ARGS and !WITH_REGS


On 6.5 kernels, it got fixed by the following two commits:

commit a8b9cf62ade1bf17261a979fc97e40c2d7842353
Author: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Date: 1 year, 9 months ago
ftrace: Fix DIRECT_CALLS to use SAVE_REGS by default

commit bdbddb109c75365d22ec4826f480c5e75869e1cb
Author: Petr Pavlu <petr.pavlu@suse.com>
Date:   1 year, 8 months ago

    tracing: Fix HAVE_DYNAMIC_FTRACE_WITH_REGS ifdef

I tried to cherry-pick 60c8971899f3b34ad24857913c0784dab08962f0
and a8b9cf62ade1bf17261a979fc97e40c2d7842353, on top of 6.5.13
kernel. Then, fentry and fexit both work with livepatch.

Thanks,
Song

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Steven Rostedt]

On Mon, 20 Oct 2025 23:07:26 -0700
Song Liu <song@kernel.org> wrote:

> commit a8b9cf62ade1bf17261a979fc97e40c2d7842353
> Author: Masami Hiramatsu (Google) <mhiramat@kernel.org>
> Date: 1 year, 9 months ago
> ftrace: Fix DIRECT_CALLS to use SAVE_REGS by default

Hmm, this is a work around. I wonder if we can make this work with ARGS as
well? Hmm. I'll have to take a look when I get a chance.

-- Steve

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Song Liu]

On Tue, Oct 21, 2025 at 7:09 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Mon, 20 Oct 2025 23:07:26 -0700
> Song Liu <song@kernel.org> wrote:
>
> > commit a8b9cf62ade1bf17261a979fc97e40c2d7842353
> > Author: Masami Hiramatsu (Google) <mhiramat@kernel.org>
> > Date: 1 year, 9 months ago
> > ftrace: Fix DIRECT_CALLS to use SAVE_REGS by default
>
> Hmm, this is a work around. I wonder if we can make this work with ARGS as
> well? Hmm. I'll have to take a look when I get a chance.

Given both ftrace_caller and arch_prepare_bpf_trampoline are created
per arch, I think it is possible. But I guess it may require quite some work
to update and test each architecture?

Thanks,
Song

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Andrey Grodzovsky]

On 10/21/25 02:07, Song Liu wrote:
> On Mon, Oct 20, 2025 at 2:31 PM Andrey Grodzovsky
> <andrey.grodzovsky@crowdstrike.com> wrote:
> [...]
>> Song, I identified another issue in pre 6.6 kernel, building
>> ~/linux-6.5/samples/livepatch/livepatch-sample.c as ko,
>> before insmoding it, bpftrace fentry/fexit fires as expected, after
>> insmod, while no errors reported on attachments,
>> the hooks stop firing, both if attaching before insmod and if attaching
>> after insmod. If i rrmod the ko, existing hooks
>> resume working.
>>
>> ubuntu@ip-10-10-115-238:~$ cat /proc/version_signature
>> Ubuntu 6.5.0-1008.8-aws 6.5.3
>> Source obtained to build the test module for the AWS kernel from the
>> related stable branch -
>> https://urldefense.com/v3/__https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz__;!!BmdzS3_lV9HdKG8!xLmCb7PCwCj7vM8JjKx_n7ZUVjW0Oj8Ih9T3YqU4I-JoGy7evTsc7U17emt3nnDmdXdchXxcKHi_6mVBt5QbKzj2$
>>
>> Let me know what you think.
> I tested various stable kernels. I got:
>
> With livepatch, fentry and fexit work on 6.3 kernels.
>
> On 6.4 and 6.5 kernels, the combination stops working since this commit:
>
> commit 60c8971899f3b34ad24857913c0784dab08962f0
> Author: Florent Revest <revest@chromium.org>
> Date:   2 years, 7 months ago
>
>      ftrace: Make DIRECT_CALLS work WITH_ARGS and !WITH_REGS
>
>
> On 6.5 kernels, it got fixed by the following two commits:
>
> commit a8b9cf62ade1bf17261a979fc97e40c2d7842353
> Author: Masami Hiramatsu (Google) <mhiramat@kernel.org>
> Date: 1 year, 9 months ago
> ftrace: Fix DIRECT_CALLS to use SAVE_REGS by default
>
> commit bdbddb109c75365d22ec4826f480c5e75869e1cb
> Author: Petr Pavlu <petr.pavlu@suse.com>
> Date:   1 year, 8 months ago
>
>      tracing: Fix HAVE_DYNAMIC_FTRACE_WITH_REGS ifdef
>
> I tried to cherry-pick 60c8971899f3b34ad24857913c0784dab08962f0
> and a8b9cf62ade1bf17261a979fc97e40c2d7842353, on top of 6.5.13
> kernel. Then, fentry and fexit both work with livepatch.


I see, thanks for testing! Is the reason it breaks so often is because 
this combination of having BPF
and llivepatch together on a system with intersection on same functions 
as relatively   rate event and
so regressions go easily unnoticed ? Isn't there any relevant automated 
testing in upstream that checks for
those types of breaks ?

Thanks,
Andrey

>
> Thanks,
> Song

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Song Liu]

On Tue, Oct 21, 2025 at 7:15 AM Andrey Grodzovsky
<andrey.grodzovsky@crowdstrike.com> wrote:
[...]
> > commit a8b9cf62ade1bf17261a979fc97e40c2d7842353
> > Author: Masami Hiramatsu (Google) <mhiramat@kernel.org>
> > Date: 1 year, 9 months ago
> > ftrace: Fix DIRECT_CALLS to use SAVE_REGS by default
> >
> > commit bdbddb109c75365d22ec4826f480c5e75869e1cb
> > Author: Petr Pavlu <petr.pavlu@suse.com>
> > Date:   1 year, 8 months ago
> >
> >      tracing: Fix HAVE_DYNAMIC_FTRACE_WITH_REGS ifdef
> >
> > I tried to cherry-pick 60c8971899f3b34ad24857913c0784dab08962f0
> > and a8b9cf62ade1bf17261a979fc97e40c2d7842353, on top of 6.5.13
> > kernel. Then, fentry and fexit both work with livepatch.
>
>
> I see, thanks for testing! Is the reason it breaks so often is because
> this combination of having BPF
> and llivepatch together on a system with intersection on same functions
> as relatively   rate event and
> so regressions go easily unnoticed ? Isn't there any relevant automated
> testing in upstream that checks for
> those types of breaks ?

This case is not being covered because it is the intersection of tracing
and livepatch. I am thinking about adding a BPF selftest to cover this
case.

Thanks,
Song

Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Steven Rostedt]

On Mon, 20 Oct 2025 11:53:34 -0700
Song Liu <song@kernel.org> wrote:

> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
> index 42bd2ba68a82..8f320df0ac52 100644
> --- i/kernel/trace/ftrace.c
> +++ w/kernel/trace/ftrace.c
> @@ -6049,6 +6049,9 @@ int register_ftrace_direct(struct ftrace_ops
> *ops, unsigned long addr)
> 
>         err = register_ftrace_function_nolock(ops);
> 
> +       if (err)
> +               remove_direct_functions_hash(hash, addr);
> +
>   out_unlock:
>         mutex_unlock(&direct_mutex);
> 
> 
> Steven,
> 
> Does this change look good to you?

With the small nit that you don't need a space between the err = and the if (err).

But yeah, looks fine to me.

-- Steve