Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>]

On 10/16/25 17:32, Song Liu wrote:
> On Thu, Oct 16, 2025 at 3:56 AM Petr Mladek <pmladek@suse.com> wrote:
>> Added Song Liu and Steven into Cc because they are more familiar with
>> the ftrace trampolines.
>>
>> On Wed 2025-10-15 17:11:31, Andrey Grodzovsky wrote:
>>> On 10/15/25 07:53, Petr Mladek wrote:
>>>> On Tue 2025-10-14 21:37:49, Andrey Grodzovsky wrote:
>>>>> Dear Upstream Livepatch  team and Ubuntu Kernel team - I included you both in this since the issue lies on the boundary between Ubuntu kernel and upstream.
>>>>>
>>>>> According to official kernel documentation  - https://urldefense.com/v3/__https://docs.kernel.org/livepatch/livepatch.html*livepatch__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5TeFxUxQ$ , section 7, Limitations -
>>>>> 1 - Kretprobes using the ftrace framework conflict with the patched functions.
>>>>> 2 - Kprobes in the original function are ignored when the code is redirected to the new implementation.
>>>>>
>>>>> Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable kernel) machine, I have no problem applying Livepatch and then setting krpobes and kretprobes on a patched function using bpftrace (or directly by coding a BPF program with kprobe/kretprobe attachment)and can confirm both execute without issues. Also the opposite works fine, running my krpobe and kretprobe hooks doesn't prevent from livepatch to be applied successfully.
>>>>>
>>>>> fentry/fexit probes do fail in in both directions - but this is expected according to my understanding as coexistence of livepatching and ftrace based BPF hooks are mutually exclusive until 6.0 based kernels
>>>>>
>>>>> libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
>>>>> libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16
>>>>>
>>>>>
>>>>> Please help me understand this contradiction about kprobes - is this because the KPROBES are FTRACE based  or any other reason ?
>>>> Heh, it seems that we have discussed this 10 years ago and I already
>>>> forgot most details.
>>>>
>>>> Yes, the conflict is detected when KPROBES are using FTRACE
>>>> infrastructure. But it happens only when the KPROBE needs to redirect
>>>> the function call, namely when it needs to modify IP address which will be used
>>>> when all attached ftrace callbacks are proceed.
>>>>
>>>> It is related to the FTRACE_OPS_FL_IPMODIFY flag.
>>>
>>> I see, that explains my case as my probes are simple, print only probes that
>>> defently don't that the ip pointer.
>>>
>>> But i still don't understand limitation 2 above doesn't show itself - how my
>>> kprobes and especially kretprobes, continue execute even after livepatch
>>> applied to the function  i am attached to ? The code execution is redirected
>>> to a different function then to which i attached my probes...
>> The code is rather complicated and it is not obvious to me.
>>
>> But I expect that kretprobes modify return addresses which are
>> stored on the stack. They replace them
>> with a helper function (trampoline) which would process
>> the kretprobe callback and jump to the original return address.
>>
>> It works even with livepatching. The only requirement is that
>> the function returns using a "ret" instruction which reads
>> the return address from the stack.
>>
>>> Also - can you please confirm that as far as i checked, starting with kernel
>>> 6.0 fentry/fexit on x86 should not have any conflicts with livepatch per
>>> merge of this RFC -
>>> https://urldefense.com/v3/__https://lkml.iu.edu/hypermail/linux/kernel/2207.2/00858.html__;!!BmdzS3_lV9HdKG8!wlwzQrMFIQcM6TTsDw6Eso8D6cFjSA1lZeeRI7xStg-c4RFuGmqrg7B6GeRXLQ8M1l6HgitfnBCfMdSeleudJcNY$  ?
>> My understanding is that the above mentioned patchset allowed to use
>> livepatching and BPF on the same function at the same time. BPF used
>> to redirect the function to a BPF trampoline. And the patchset allowed
>> to call the BPF trampoline/callback directly from the ftrace callback.
> Yes, the patch set above allows BPF trampoline to work on the same
> function at the same time with livepatch. After this patchset, BPF
> trampoline no longer sets FTRACE_OPS_FL_IPMODIFY.
>
> Thanks,
> Song


I tested this bellow on 6.1.0 based Ubuntu kernel using bpftrace - in 
the end as seen bellow, I wasn't able to attach using fentry/fexit (or 
krpboe/kretprobe) as long as livepatch was present

ubuntu@ip-10-10-114-204:~$ uname -r
6.8.0-1008-aws
ubuntu@ip-10-10-114-204:~$ cat /proc/version_signature
Ubuntu 6.8.0-1008.8-aws 6.8.1

[AG] - See bellow the patched function I am to attach to -

ubuntu@ip-10-10-114-204:~$ sudo cat /proc/kallsyms | grep begin_new_exec
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name 
resolution
ffffffff81ee0420 T __pfx_begin_new_exec
ffffffff81ee0430 T begin_new_exec
ffffffff8374b3d8 r __ksymtab_begin_new_exec
ffffffffc07c9390 t begin_new_exec [lkp_Ubuntu_6_8_0_1008_8_aws_111]
ffffffffc07c9380 t __pfx_begin_new_exec [lkp_Ubuntu_6_8_0_1008_8_aws_111]

[AG] - Attaching to begin_new_exec

ubuntu@ip-10-10-114-204:~$ sudo bpftool btf dump file 
/sys/kernel/btf/vmlinux | grep begin_new_exec
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name 
resolution
[68920] FUNC 'begin_new_exec' type_id=48063 linkage=static

[AG] - , trying to attach to fentry and fexit and first error asks me to 
explicitly resolve the ambiguity - which i guess make sense

ubuntu@ip-10-10-114-204:~$
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -v -e 'fentry:begin_new_exec { 
@start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n", 
pid, comm); } fexit:begin_new_exec { $latency = nsecs - @start[tid]; 
delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s, 
Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
stdin:1:1-22: ERROR: ambiguous attach point, please specify module 
containing the function 'begin_new_exec'
fentry:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START 
(fentry): PID %d, Comm %s\n", pid, comm); } fexit:begin_new_exec { 
$latency = nsecs - @start[tid]; delete(@start[tid]); printf("<- EXEC END 
(fexit): PID %d, Comm %s, Retval %d, Latency %d us\n", pid, comm, 
retval, $latency / 1000); }
~~~~~~~~~~~~~~~~~~~~~
stdin:1:111-132: ERROR: ambiguous attach point, please specify module 
containing the function 'begin_new_exec'
fentry:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START 
(fentry): PID %d, Comm %s\n", pid, comm); } fexit:begin_new_exec { 
$latency = nsecs - @start[tid]; delete(@start[tid]); printf("<- EXEC END 
(fexit): PID %d, Comm %s, Retval %d, Latency %d us\n", pid, comm, 
retval, $latency / 1000); }
                                             ~~~~~~~~~~~~~~~~~~~~~
[AG] - Trying first to point him at the original  function - but he 
fails on the fexit I assume  - which is strange, I assumed fexit 
(kretfunc) and livepatch can coexist ?

ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e 
'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC 
START (fentry): PID %d, Comm %s\n", pid, comm); } 
fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid]; 
delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s, 
Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
Attaching 2 probes...
ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec

[AG] - Trying to skip the fexit and only do fentry - he still rejects
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e 
'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC 
START (fentry): PID %d, Comm %s\n", pid, comm); }'
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name 
resolution
INFO: node count: 12
Attaching 1 probe...

Program ID: 295

The verifier log:
processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3 
peak_states 3 mark_read 1

Attaching kfunc:vmlinux:begin_new_exec
ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec


[AG] - Now trying to point him at the livepatch function but here he 
complains that Uubntu doesn't have BTF info for the livepatch module

ubuntu@ip-10-10-114-204:~$
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -v -e 
'fentry:lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec { @start[tid] = 
nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n", pid, comm); } 
fexit:lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec { $latency = nsecs 
- @start[tid]; delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, 
Comm %s, Retval %d, Latency %d us\n", pid, comm, retval, $latency / 
1000); }'
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name 
resolution
INFO: node count: 33
Attaching 2 probes...
ERROR: No BTF found for lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec

[AG] - Disabling livepatch, rebooting to verify all livepatch symbols 
are out of kalsym table and now everything fine

ubuntu@ip-10-10-114-204:~$   sudo bpftrace -v -e 'fentry:begin_new_exec 
{ @start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm 
%s\n", pid, comm]; delete(@start[tid]); printf("<- EXEC END (fexit): PID 
%d, Comm %s, Retval %d, Latency %d us\n", pid, comm, retval, $latency / 
1000); }'
INFO: node count: 33
Attaching 2 probes...
Program ID: 33
The verifier log:
processed 76 insns (limit 1000000) max_states_per_insn 0 total_states 5 
peak_states 5 mark_read 3
Attaching kretfunc:vmlinux:begin_new_exec
Program ID: 34
The verifier log:
processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3 
peak_states 3 mark_read 1

Attaching kfunc:vmlinux:begin_new_exec
-> EXEC START (fentry): PID 1714, Comm bash
<- EXEC END (fexit): PID 1714, Comm sudo, Retval 0, Latency 99 us


ubuntu@ip-10-10-114-204:~$ sudo bpftool link
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name 
resolution
1: tracing  prog 2
     prog_type tracing  attach_type modify_return
     target_obj_id 1  target_btf_id 119714
5: tracing  prog 33
     prog_type tracing  attach_type trace_fexit
     target_obj_id 1  target_btf_id 68920
6: tracing  prog 34
     prog_type tracing  attach_type trace_fentry
     target_obj_id 1  target_btf_id 68920
ubuntu@ip-10-10-114-204:~$

Thanks,
Andrey


>
>
>
>> Best Regards,
>> Petr
>>
>>>> More details can be found in the discussion at
>>>> https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*re746846b6b16c49a55c89b4c63b7995fe5972111__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu6pjuIgig$
>>>>
>>>> I seems that I made some analyze when it worked and it did not work,
>>>> see https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*mffd8c8bf4325b473d89876f2805f42f1af7c82d7__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5xbeoulA$
>>>> But I am not 100% sure that it was correct. Also it was before the
>>>> BPF-boom.
>>>>
>>>> Also you might look at the selftest in the todays Linus' tree:
>>>>
>>>>     + tools/testing/selftests/livepatch/https://urldefense.com/v3/__http://test-kprobe.sh__;!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5RXF-AnA$
>>>>     + tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c
>>>>     + tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c
>>>>
>>>> The parallel load fails when the Kprobe is using a post_handler.
>>>>
>>>> Sigh, we should fix the livepatch documentation. The kretprobes
>>>> obviously work. register_kretprobe() even explicitely sets:
>>>>
>>>>      rp->kp.post_handler = NULL;
>>>>
>>>> It seems that .post_handler is called after all ftrace handlers.
>>>> And it sets IP after the NOPs, see kprobe_ftrace_handler().
>>>> I am not sure about the use case.
>>>>
>>>> Best Regards,
>>>> Petr
>>>