Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Petr Mladek <pmladek@suse.com>]

Added Song Liu and Steven into Cc because they are more familiar with
the ftrace trampolines.

On Wed 2025-10-15 17:11:31, Andrey Grodzovsky wrote:
> On 10/15/25 07:53, Petr Mladek wrote:
> > On Tue 2025-10-14 21:37:49, Andrey Grodzovsky wrote:
> > > Dear Upstream Livepatch  team and Ubuntu Kernel team - I included you both in this since the issue lies on the boundary between Ubuntu kernel and upstream.
> > > 
> > > According to official kernel documentation  - https://urldefense.com/v3/__https://docs.kernel.org/livepatch/livepatch.html*livepatch__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5TeFxUxQ$ , section 7, Limitations -
> > > 1 - Kretprobes using the ftrace framework conflict with the patched functions.
> > > 2 - Kprobes in the original function are ignored when the code is redirected to the new implementation.
> > > 
> > > Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable kernel) machine, I have no problem applying Livepatch and then setting krpobes and kretprobes on a patched function using bpftrace (or directly by coding a BPF program with kprobe/kretprobe attachment)and can confirm both execute without issues. Also the opposite works fine, running my krpobe and kretprobe hooks doesn't prevent from livepatch to be applied successfully.
> > > 
> > > fentry/fexit probes do fail in in both directions - but this is expected according to my understanding as coexistence of livepatching and ftrace based BPF hooks are mutually exclusive until 6.0 based kernels
> > > 
> > > libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
> > > libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16
> > > 
> > > 
> > > Please help me understand this contradiction about kprobes - is this because the KPROBES are FTRACE based  or any other reason ?
> > Heh, it seems that we have discussed this 10 years ago and I already
> > forgot most details.
> > 
> > Yes, the conflict is detected when KPROBES are using FTRACE
> > infrastructure. But it happens only when the KPROBE needs to redirect
> > the function call, namely when it needs to modify IP address which will be used
> > when all attached ftrace callbacks are proceed.
> > 
> > It is related to the FTRACE_OPS_FL_IPMODIFY flag.
> 
> 
> I see, that explains my case as my probes are simple, print only probes that
> defently don't that the ip pointer.
> 
> But i still don't understand limitation 2 above doesn't show itself - how my
> kprobes and especially kretprobes, continue execute even after livepatch
> applied to the function  i am attached to ? The code execution is redirected
> to a different function then to which i attached my probes...

The code is rather complicated and it is not obvious to me.

But I expect that kretprobes modify return addresses which are
stored on the stack. They replace them
with a helper function (trampoline) which would process
the kretprobe callback and jump to the original return address.

It works even with livepatching. The only requirement is that
the function returns using a "ret" instruction which reads
the return address from the stack.

> Also - can you please confirm that as far as i checked, starting with kernel
> 6.0 fentry/fexit on x86 should not have any conflicts with livepatch per
> merge of this RFC -
> https://lkml.iu.edu/hypermail/linux/kernel/2207.2/00858.html ?

My understanding is that the above mentioned patchset allowed to use
livepatching and BPF on the same function at the same time. BPF used
to redirect the function to a BPF trampoline. And the patchset allowed
to call the BPF trampoline/callback directly from the ftrace callback.

Best Regards,
Petr

> > 
> > More details can be found in the discussion at
> > https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*re746846b6b16c49a55c89b4c63b7995fe5972111__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu6pjuIgig$
> > 
> > I seems that I made some analyze when it worked and it did not work,
> > see https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*mffd8c8bf4325b473d89876f2805f42f1af7c82d7__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5xbeoulA$
> > But I am not 100% sure that it was correct. Also it was before the
> > BPF-boom.
> > 
> > Also you might look at the selftest in the todays Linus' tree:
> > 
> >    + tools/testing/selftests/livepatch/https://urldefense.com/v3/__http://test-kprobe.sh__;!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5RXF-AnA$
> >    + tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c
> >    + tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c
> > 
> > The parallel load fails when the Kprobe is using a post_handler.
> > 
> > Sigh, we should fix the livepatch documentation. The kretprobes
> > obviously work. register_kretprobe() even explicitely sets:
> > 
> > 	rp->kp.post_handler = NULL;
> > 
> > It seems that .post_handler is called after all ftrace handlers.
> > And it sets IP after the NOPs, see kprobe_ftrace_handler().
> > I am not sure about the use case.
> > 
> > Best Regards,
> > Petr
> 
>