* Re: [PATCH v2] livepatch: Match old_sympos 0 and 1 in klp_find_func()
From: Petr Mladek @ 2025-10-15 13:10 UTC (permalink / raw)
To: Song Liu
Cc: live-patching, jpoimboe, jikos, mbenes, joe.lawrence, kernel-team
In-Reply-To: <20251013173019.990707-1-song@kernel.org>
On Mon 2025-10-13 10:30:19, Song Liu wrote:
> When there is only one function of the same name, old_sympos of 0 and 1
> are logically identical. Match them in klp_find_func().
>
> This is to avoid a corner case with different toolchain behavior.
>
> In this specific issue, issue two versions of kpatch-build were used to
s/issue, issue/issue/
> build livepatch for the same kernel. One assigns old_sympos == 0 for
> unique local functions, the other assigns old_sympos == 1 for unique
> local functions. Both versions work fine by themselves. (PS: This
> behavior change was introduced in a downstream version of kpatch-build.
> This change does not exist in upstream kpatch-build.)
>
> However, during livepatch upgrade (with the replace flag set) from a
> patch built with one version of kpatch-build to the same fix built with
> the other version of kpatch-build, livepatching fails with errors like:
>
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -88,8 +88,14 @@ static struct klp_func *klp_find_func(struct klp_object *obj,
> struct klp_func *func;
>
> klp_for_each_func(obj, func) {
> + /*
> + * Besides identical old_sympos, also condiser old_sympos
As Josh already pointed out:
s/condiser/consider/
> + * of 0 and 1 are identical.
> + */
> if ((strcmp(old_func->old_name, func->old_name) == 0) &&
> - (old_func->old_sympos == func->old_sympos)) {
> + ((old_func->old_sympos == func->old_sympos) ||
> + (old_func->old_sympos == 0 && func->old_sympos == 1) ||
> + (old_func->old_sympos == 1 && func->old_sympos == 0))) {
> return func;
> }
> }
With the two above fixes:
Reviewed-by: Petr Mladek <pmladek@suse.com>
Tested-by: Petr Mladek <pmladek@suse.com>
I have fixed both problems when committing.
And the patch has been committed into livepatching.git,
branch for-6.19/trivial.
Best Regards,
Petr
^ permalink raw reply
* Re: [PATCH] selftests: livepatch: use canonical ftrace path
From: Petr Mladek @ 2025-10-15 13:11 UTC (permalink / raw)
To: Fushuai Wang
Cc: jpoimboe, jikos, mbenes, joe.lawrence, shuah, live-patching,
linux-kselftest, linux-kernel
In-Reply-To: <20251010120727.20631-1-wangfushuai@baidu.com>
On Fri 2025-10-10 20:07:27, Fushuai Wang wrote:
> Since v4.1 kernel, a new interface for ftrace called "tracefs" was
> introduced, which is usually mounted in /sys/kernel/tracing. Therefore,
> tracing files can now be accessed via either the legacy path
> /sys/kernel/debug/tracing or the newer path /sys/kernel/tracing.
>
> Signed-off-by: Fushuai Wang <wangfushuai@baidu.com>
JFYI, the patch has been comitted into livepatching.git,
branch for-6.19/trivial.
Best Regards,
Petr
^ permalink raw reply
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Andrey Grodzovsky @ 2025-10-15 21:11 UTC (permalink / raw)
To: Petr Mladek; +Cc: kernel-team@lists.ubuntu.com, live-patching@vger.kernel.org
In-Reply-To: <aO-LMaY-os44cEJP@pathway.suse.cz>
On 10/15/25 07:53, Petr Mladek wrote:
> On Tue 2025-10-14 21:37:49, Andrey Grodzovsky wrote:
>> Dear Upstream Livepatch team and Ubuntu Kernel team - I included you both in this since the issue lies on the boundary between Ubuntu kernel and upstream.
>>
>> According to official kernel documentation - https://urldefense.com/v3/__https://docs.kernel.org/livepatch/livepatch.html*livepatch__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5TeFxUxQ$ , section 7, Limitations -
>> 1 - Kretprobes using the ftrace framework conflict with the patched functions.
>> 2 - Kprobes in the original function are ignored when the code is redirected to the new implementation.
>>
>> Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable kernel) machine, I have no problem applying Livepatch and then setting krpobes and kretprobes on a patched function using bpftrace (or directly by coding a BPF program with kprobe/kretprobe attachment)and can confirm both execute without issues. Also the opposite works fine, running my krpobe and kretprobe hooks doesn't prevent from livepatch to be applied successfully.
>>
>> fentry/fexit probes do fail in in both directions - but this is expected according to my understanding as coexistence of livepatching and ftrace based BPF hooks are mutually exclusive until 6.0 based kernels
>>
>> libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
>> libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16
>>
>>
>> Please help me understand this contradiction about kprobes - is this because the KPROBES are FTRACE based or any other reason ?
> Heh, it seems that we have discussed this 10 years ago and I already
> forgot most details.
>
> Yes, the conflict is detected when KPROBES are using FTRACE
> infrastructure. But it happens only when the KPROBE needs to redirect
> the function call, namely when it needs to modify IP address which will be used
> when all attached ftrace callbacks are proceed.
>
> It is related to the FTRACE_OPS_FL_IPMODIFY flag.
I see, that explains my case as my probes are simple, print only probes
that defently don't that the ip pointer.
But i still don't understand limitation 2 above doesn't show itself -
how my kprobes and especially kretprobes, continue execute even after
livepatch applied to the function i am attached to ? The code execution
is redirected to a different function then to which i attached my probes...
Also - can you please confirm that as far as i checked, starting with
kernel 6.0 fentry/fexit on x86 should not have any conflicts with
livepatch per merge of this RFC -
https://lkml.iu.edu/hypermail/linux/kernel/2207.2/00858.html ?
Thanks,
Andrey
>
> More details can be found in the discussion at
> https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*re746846b6b16c49a55c89b4c63b7995fe5972111__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu6pjuIgig$
>
> I seems that I made some analyze when it worked and it did not work,
> see https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*mffd8c8bf4325b473d89876f2805f42f1af7c82d7__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5xbeoulA$
> But I am not 100% sure that it was correct. Also it was before the
> BPF-boom.
>
> Also you might look at the selftest in the todays Linus' tree:
>
> + tools/testing/selftests/livepatch/https://urldefense.com/v3/__http://test-kprobe.sh__;!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5RXF-AnA$
> + tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c
> + tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c
>
> The parallel load fails when the Kprobe is using a post_handler.
>
> Sigh, we should fix the livepatch documentation. The kretprobes
> obviously work. register_kretprobe() even explicitely sets:
>
> rp->kp.post_handler = NULL;
>
> It seems that .post_handler is called after all ftrace handlers.
> And it sets IP after the NOPs, see kprobe_ftrace_handler().
> I am not sure about the use case.
>
> Best Regards,
> Petr
^ permalink raw reply
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Petr Mladek @ 2025-10-16 10:56 UTC (permalink / raw)
To: Andrey Grodzovsky
Cc: kernel-team@lists.ubuntu.com, live-patching@vger.kernel.org,
Song Liu, Steven Rostedt
In-Reply-To: <eb176565-6e13-4f98-8c9a-cacf7fc42f3a@crowdstrike.com>
Added Song Liu and Steven into Cc because they are more familiar with
the ftrace trampolines.
On Wed 2025-10-15 17:11:31, Andrey Grodzovsky wrote:
> On 10/15/25 07:53, Petr Mladek wrote:
> > On Tue 2025-10-14 21:37:49, Andrey Grodzovsky wrote:
> > > Dear Upstream Livepatch team and Ubuntu Kernel team - I included you both in this since the issue lies on the boundary between Ubuntu kernel and upstream.
> > >
> > > According to official kernel documentation - https://urldefense.com/v3/__https://docs.kernel.org/livepatch/livepatch.html*livepatch__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5TeFxUxQ$ , section 7, Limitations -
> > > 1 - Kretprobes using the ftrace framework conflict with the patched functions.
> > > 2 - Kprobes in the original function are ignored when the code is redirected to the new implementation.
> > >
> > > Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable kernel) machine, I have no problem applying Livepatch and then setting krpobes and kretprobes on a patched function using bpftrace (or directly by coding a BPF program with kprobe/kretprobe attachment)and can confirm both execute without issues. Also the opposite works fine, running my krpobe and kretprobe hooks doesn't prevent from livepatch to be applied successfully.
> > >
> > > fentry/fexit probes do fail in in both directions - but this is expected according to my understanding as coexistence of livepatching and ftrace based BPF hooks are mutually exclusive until 6.0 based kernels
> > >
> > > libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
> > > libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16
> > >
> > >
> > > Please help me understand this contradiction about kprobes - is this because the KPROBES are FTRACE based or any other reason ?
> > Heh, it seems that we have discussed this 10 years ago and I already
> > forgot most details.
> >
> > Yes, the conflict is detected when KPROBES are using FTRACE
> > infrastructure. But it happens only when the KPROBE needs to redirect
> > the function call, namely when it needs to modify IP address which will be used
> > when all attached ftrace callbacks are proceed.
> >
> > It is related to the FTRACE_OPS_FL_IPMODIFY flag.
>
>
> I see, that explains my case as my probes are simple, print only probes that
> defently don't that the ip pointer.
>
> But i still don't understand limitation 2 above doesn't show itself - how my
> kprobes and especially kretprobes, continue execute even after livepatch
> applied to the function i am attached to ? The code execution is redirected
> to a different function then to which i attached my probes...
The code is rather complicated and it is not obvious to me.
But I expect that kretprobes modify return addresses which are
stored on the stack. They replace them
with a helper function (trampoline) which would process
the kretprobe callback and jump to the original return address.
It works even with livepatching. The only requirement is that
the function returns using a "ret" instruction which reads
the return address from the stack.
> Also - can you please confirm that as far as i checked, starting with kernel
> 6.0 fentry/fexit on x86 should not have any conflicts with livepatch per
> merge of this RFC -
> https://lkml.iu.edu/hypermail/linux/kernel/2207.2/00858.html ?
My understanding is that the above mentioned patchset allowed to use
livepatching and BPF on the same function at the same time. BPF used
to redirect the function to a BPF trampoline. And the patchset allowed
to call the BPF trampoline/callback directly from the ftrace callback.
Best Regards,
Petr
> >
> > More details can be found in the discussion at
> > https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*re746846b6b16c49a55c89b4c63b7995fe5972111__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu6pjuIgig$
> >
> > I seems that I made some analyze when it worked and it did not work,
> > see https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*mffd8c8bf4325b473d89876f2805f42f1af7c82d7__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5xbeoulA$
> > But I am not 100% sure that it was correct. Also it was before the
> > BPF-boom.
> >
> > Also you might look at the selftest in the todays Linus' tree:
> >
> > + tools/testing/selftests/livepatch/https://urldefense.com/v3/__http://test-kprobe.sh__;!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5RXF-AnA$
> > + tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c
> > + tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c
> >
> > The parallel load fails when the Kprobe is using a post_handler.
> >
> > Sigh, we should fix the livepatch documentation. The kretprobes
> > obviously work. register_kretprobe() even explicitely sets:
> >
> > rp->kp.post_handler = NULL;
> >
> > It seems that .post_handler is called after all ftrace handlers.
> > And it sets IP after the NOPs, see kprobe_ftrace_handler().
> > I am not sure about the use case.
> >
> > Best Regards,
> > Petr
>
>
^ permalink raw reply
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Song Liu @ 2025-10-16 21:32 UTC (permalink / raw)
To: Petr Mladek
Cc: Andrey Grodzovsky, kernel-team@lists.ubuntu.com,
live-patching@vger.kernel.org, Song Liu, Steven Rostedt
In-Reply-To: <aPDPYIA4_mpo-OZS@pathway.suse.cz>
On Thu, Oct 16, 2025 at 3:56 AM Petr Mladek <pmladek@suse.com> wrote:
>
> Added Song Liu and Steven into Cc because they are more familiar with
> the ftrace trampolines.
>
> On Wed 2025-10-15 17:11:31, Andrey Grodzovsky wrote:
> > On 10/15/25 07:53, Petr Mladek wrote:
> > > On Tue 2025-10-14 21:37:49, Andrey Grodzovsky wrote:
> > > > Dear Upstream Livepatch team and Ubuntu Kernel team - I included you both in this since the issue lies on the boundary between Ubuntu kernel and upstream.
> > > >
> > > > According to official kernel documentation - https://urldefense.com/v3/__https://docs.kernel.org/livepatch/livepatch.html*livepatch__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5TeFxUxQ$ , section 7, Limitations -
> > > > 1 - Kretprobes using the ftrace framework conflict with the patched functions.
> > > > 2 - Kprobes in the original function are ignored when the code is redirected to the new implementation.
> > > >
> > > > Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable kernel) machine, I have no problem applying Livepatch and then setting krpobes and kretprobes on a patched function using bpftrace (or directly by coding a BPF program with kprobe/kretprobe attachment)and can confirm both execute without issues. Also the opposite works fine, running my krpobe and kretprobe hooks doesn't prevent from livepatch to be applied successfully.
> > > >
> > > > fentry/fexit probes do fail in in both directions - but this is expected according to my understanding as coexistence of livepatching and ftrace based BPF hooks are mutually exclusive until 6.0 based kernels
> > > >
> > > > libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
> > > > libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16
> > > >
> > > >
> > > > Please help me understand this contradiction about kprobes - is this because the KPROBES are FTRACE based or any other reason ?
> > > Heh, it seems that we have discussed this 10 years ago and I already
> > > forgot most details.
> > >
> > > Yes, the conflict is detected when KPROBES are using FTRACE
> > > infrastructure. But it happens only when the KPROBE needs to redirect
> > > the function call, namely when it needs to modify IP address which will be used
> > > when all attached ftrace callbacks are proceed.
> > >
> > > It is related to the FTRACE_OPS_FL_IPMODIFY flag.
> >
> >
> > I see, that explains my case as my probes are simple, print only probes that
> > defently don't that the ip pointer.
> >
> > But i still don't understand limitation 2 above doesn't show itself - how my
> > kprobes and especially kretprobes, continue execute even after livepatch
> > applied to the function i am attached to ? The code execution is redirected
> > to a different function then to which i attached my probes...
>
> The code is rather complicated and it is not obvious to me.
>
> But I expect that kretprobes modify return addresses which are
> stored on the stack. They replace them
> with a helper function (trampoline) which would process
> the kretprobe callback and jump to the original return address.
>
> It works even with livepatching. The only requirement is that
> the function returns using a "ret" instruction which reads
> the return address from the stack.
>
> > Also - can you please confirm that as far as i checked, starting with kernel
> > 6.0 fentry/fexit on x86 should not have any conflicts with livepatch per
> > merge of this RFC -
> > https://lkml.iu.edu/hypermail/linux/kernel/2207.2/00858.html ?
>
> My understanding is that the above mentioned patchset allowed to use
> livepatching and BPF on the same function at the same time. BPF used
> to redirect the function to a BPF trampoline. And the patchset allowed
> to call the BPF trampoline/callback directly from the ftrace callback.
Yes, the patch set above allows BPF trampoline to work on the same
function at the same time with livepatch. After this patchset, BPF
trampoline no longer sets FTRACE_OPS_FL_IPMODIFY.
Thanks,
Song
> Best Regards,
> Petr
>
> > >
> > > More details can be found in the discussion at
> > > https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*re746846b6b16c49a55c89b4c63b7995fe5972111__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu6pjuIgig$
> > >
> > > I seems that I made some analyze when it worked and it did not work,
> > > see https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*mffd8c8bf4325b473d89876f2805f42f1af7c82d7__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5xbeoulA$
> > > But I am not 100% sure that it was correct. Also it was before the
> > > BPF-boom.
> > >
> > > Also you might look at the selftest in the todays Linus' tree:
> > >
> > > + tools/testing/selftests/livepatch/https://urldefense.com/v3/__http://test-kprobe.sh__;!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5RXF-AnA$
> > > + tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c
> > > + tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c
> > >
> > > The parallel load fails when the Kprobe is using a post_handler.
> > >
> > > Sigh, we should fix the livepatch documentation. The kretprobes
> > > obviously work. register_kretprobe() even explicitely sets:
> > >
> > > rp->kp.post_handler = NULL;
> > >
> > > It seems that .post_handler is called after all ftrace handlers.
> > > And it sets IP after the NOPs, see kprobe_ftrace_handler().
> > > I am not sure about the use case.
> > >
> > > Best Regards,
> > > Petr
> >
> >
^ permalink raw reply
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Andrey Grodzovsky @ 2025-10-16 21:55 UTC (permalink / raw)
To: Song Liu, Petr Mladek
Cc: kernel-team@lists.ubuntu.com, live-patching@vger.kernel.org,
Steven Rostedt
In-Reply-To: <CAHzjS_v2HfpH1Oof3BWawN51WVM_1V1uXro4MSC=0YmMiqVWcg@mail.gmail.com>
On 10/16/25 17:32, Song Liu wrote:
> On Thu, Oct 16, 2025 at 3:56 AM Petr Mladek <pmladek@suse.com> wrote:
>> Added Song Liu and Steven into Cc because they are more familiar with
>> the ftrace trampolines.
>>
>> On Wed 2025-10-15 17:11:31, Andrey Grodzovsky wrote:
>>> On 10/15/25 07:53, Petr Mladek wrote:
>>>> On Tue 2025-10-14 21:37:49, Andrey Grodzovsky wrote:
>>>>> Dear Upstream Livepatch team and Ubuntu Kernel team - I included you both in this since the issue lies on the boundary between Ubuntu kernel and upstream.
>>>>>
>>>>> According to official kernel documentation - https://urldefense.com/v3/__https://docs.kernel.org/livepatch/livepatch.html*livepatch__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5TeFxUxQ$ , section 7, Limitations -
>>>>> 1 - Kretprobes using the ftrace framework conflict with the patched functions.
>>>>> 2 - Kprobes in the original function are ignored when the code is redirected to the new implementation.
>>>>>
>>>>> Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable kernel) machine, I have no problem applying Livepatch and then setting krpobes and kretprobes on a patched function using bpftrace (or directly by coding a BPF program with kprobe/kretprobe attachment)and can confirm both execute without issues. Also the opposite works fine, running my krpobe and kretprobe hooks doesn't prevent from livepatch to be applied successfully.
>>>>>
>>>>> fentry/fexit probes do fail in in both directions - but this is expected according to my understanding as coexistence of livepatching and ftrace based BPF hooks are mutually exclusive until 6.0 based kernels
>>>>>
>>>>> libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
>>>>> libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16
>>>>>
>>>>>
>>>>> Please help me understand this contradiction about kprobes - is this because the KPROBES are FTRACE based or any other reason ?
>>>> Heh, it seems that we have discussed this 10 years ago and I already
>>>> forgot most details.
>>>>
>>>> Yes, the conflict is detected when KPROBES are using FTRACE
>>>> infrastructure. But it happens only when the KPROBE needs to redirect
>>>> the function call, namely when it needs to modify IP address which will be used
>>>> when all attached ftrace callbacks are proceed.
>>>>
>>>> It is related to the FTRACE_OPS_FL_IPMODIFY flag.
>>>
>>> I see, that explains my case as my probes are simple, print only probes that
>>> defently don't that the ip pointer.
>>>
>>> But i still don't understand limitation 2 above doesn't show itself - how my
>>> kprobes and especially kretprobes, continue execute even after livepatch
>>> applied to the function i am attached to ? The code execution is redirected
>>> to a different function then to which i attached my probes...
>> The code is rather complicated and it is not obvious to me.
>>
>> But I expect that kretprobes modify return addresses which are
>> stored on the stack. They replace them
>> with a helper function (trampoline) which would process
>> the kretprobe callback and jump to the original return address.
>>
>> It works even with livepatching. The only requirement is that
>> the function returns using a "ret" instruction which reads
>> the return address from the stack.
>>
>>> Also - can you please confirm that as far as i checked, starting with kernel
>>> 6.0 fentry/fexit on x86 should not have any conflicts with livepatch per
>>> merge of this RFC -
>>> https://urldefense.com/v3/__https://lkml.iu.edu/hypermail/linux/kernel/2207.2/00858.html__;!!BmdzS3_lV9HdKG8!wlwzQrMFIQcM6TTsDw6Eso8D6cFjSA1lZeeRI7xStg-c4RFuGmqrg7B6GeRXLQ8M1l6HgitfnBCfMdSeleudJcNY$ ?
>> My understanding is that the above mentioned patchset allowed to use
>> livepatching and BPF on the same function at the same time. BPF used
>> to redirect the function to a BPF trampoline. And the patchset allowed
>> to call the BPF trampoline/callback directly from the ftrace callback.
> Yes, the patch set above allows BPF trampoline to work on the same
> function at the same time with livepatch. After this patchset, BPF
> trampoline no longer sets FTRACE_OPS_FL_IPMODIFY.
>
> Thanks,
> Song
I tested this bellow on 6.1.0 based Ubuntu kernel using bpftrace - in
the end as seen bellow, I wasn't able to attach using fentry/fexit (or
krpboe/kretprobe) as long as livepatch was present
ubuntu@ip-10-10-114-204:~$ uname -r
6.8.0-1008-aws
ubuntu@ip-10-10-114-204:~$ cat /proc/version_signature
Ubuntu 6.8.0-1008.8-aws 6.8.1
[AG] - See bellow the patched function I am to attach to -
ubuntu@ip-10-10-114-204:~$ sudo cat /proc/kallsyms | grep begin_new_exec
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
resolution
ffffffff81ee0420 T __pfx_begin_new_exec
ffffffff81ee0430 T begin_new_exec
ffffffff8374b3d8 r __ksymtab_begin_new_exec
ffffffffc07c9390 t begin_new_exec [lkp_Ubuntu_6_8_0_1008_8_aws_111]
ffffffffc07c9380 t __pfx_begin_new_exec [lkp_Ubuntu_6_8_0_1008_8_aws_111]
[AG] - Attaching to begin_new_exec
ubuntu@ip-10-10-114-204:~$ sudo bpftool btf dump file
/sys/kernel/btf/vmlinux | grep begin_new_exec
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
resolution
[68920] FUNC 'begin_new_exec' type_id=48063 linkage=static
[AG] - , trying to attach to fentry and fexit and first error asks me to
explicitly resolve the ambiguity - which i guess make sense
ubuntu@ip-10-10-114-204:~$
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -v -e 'fentry:begin_new_exec {
@start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n",
pid, comm); } fexit:begin_new_exec { $latency = nsecs - @start[tid];
delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s,
Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
stdin:1:1-22: ERROR: ambiguous attach point, please specify module
containing the function 'begin_new_exec'
fentry:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START
(fentry): PID %d, Comm %s\n", pid, comm); } fexit:begin_new_exec {
$latency = nsecs - @start[tid]; delete(@start[tid]); printf("<- EXEC END
(fexit): PID %d, Comm %s, Retval %d, Latency %d us\n", pid, comm,
retval, $latency / 1000); }
~~~~~~~~~~~~~~~~~~~~~
stdin:1:111-132: ERROR: ambiguous attach point, please specify module
containing the function 'begin_new_exec'
fentry:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START
(fentry): PID %d, Comm %s\n", pid, comm); } fexit:begin_new_exec {
$latency = nsecs - @start[tid]; delete(@start[tid]); printf("<- EXEC END
(fexit): PID %d, Comm %s, Retval %d, Latency %d us\n", pid, comm,
retval, $latency / 1000); }
~~~~~~~~~~~~~~~~~~~~~
[AG] - Trying first to point him at the original function - but he
fails on the fexit I assume - which is strange, I assumed fexit
(kretfunc) and livepatch can coexist ?
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e
'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
START (fentry): PID %d, Comm %s\n", pid, comm); }
fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid];
delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s,
Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
Attaching 2 probes...
ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec
[AG] - Trying to skip the fexit and only do fentry - he still rejects
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e
'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
START (fentry): PID %d, Comm %s\n", pid, comm); }'
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
resolution
INFO: node count: 12
Attaching 1 probe...
Program ID: 295
The verifier log:
processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3
peak_states 3 mark_read 1
Attaching kfunc:vmlinux:begin_new_exec
ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec
[AG] - Now trying to point him at the livepatch function but here he
complains that Uubntu doesn't have BTF info for the livepatch module
ubuntu@ip-10-10-114-204:~$
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -v -e
'fentry:lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec { @start[tid] =
nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n", pid, comm); }
fexit:lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec { $latency = nsecs
- @start[tid]; delete(@start[tid]); printf("<- EXEC END (fexit): PID %d,
Comm %s, Retval %d, Latency %d us\n", pid, comm, retval, $latency /
1000); }'
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
resolution
INFO: node count: 33
Attaching 2 probes...
ERROR: No BTF found for lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec
[AG] - Disabling livepatch, rebooting to verify all livepatch symbols
are out of kalsym table and now everything fine
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -v -e 'fentry:begin_new_exec
{ @start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm
%s\n", pid, comm]; delete(@start[tid]); printf("<- EXEC END (fexit): PID
%d, Comm %s, Retval %d, Latency %d us\n", pid, comm, retval, $latency /
1000); }'
INFO: node count: 33
Attaching 2 probes...
Program ID: 33
The verifier log:
processed 76 insns (limit 1000000) max_states_per_insn 0 total_states 5
peak_states 5 mark_read 3
Attaching kretfunc:vmlinux:begin_new_exec
Program ID: 34
The verifier log:
processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3
peak_states 3 mark_read 1
Attaching kfunc:vmlinux:begin_new_exec
-> EXEC START (fentry): PID 1714, Comm bash
<- EXEC END (fexit): PID 1714, Comm sudo, Retval 0, Latency 99 us
ubuntu@ip-10-10-114-204:~$ sudo bpftool link
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
resolution
1: tracing prog 2
prog_type tracing attach_type modify_return
target_obj_id 1 target_btf_id 119714
5: tracing prog 33
prog_type tracing attach_type trace_fexit
target_obj_id 1 target_btf_id 68920
6: tracing prog 34
prog_type tracing attach_type trace_fentry
target_obj_id 1 target_btf_id 68920
ubuntu@ip-10-10-114-204:~$
Thanks,
Andrey
>
>
>
>> Best Regards,
>> Petr
>>
>>>> More details can be found in the discussion at
>>>> https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*re746846b6b16c49a55c89b4c63b7995fe5972111__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu6pjuIgig$
>>>>
>>>> I seems that I made some analyze when it worked and it did not work,
>>>> see https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*mffd8c8bf4325b473d89876f2805f42f1af7c82d7__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5xbeoulA$
>>>> But I am not 100% sure that it was correct. Also it was before the
>>>> BPF-boom.
>>>>
>>>> Also you might look at the selftest in the todays Linus' tree:
>>>>
>>>> + tools/testing/selftests/livepatch/https://urldefense.com/v3/__http://test-kprobe.sh__;!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5RXF-AnA$
>>>> + tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c
>>>> + tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c
>>>>
>>>> The parallel load fails when the Kprobe is using a post_handler.
>>>>
>>>> Sigh, we should fix the livepatch documentation. The kretprobes
>>>> obviously work. register_kretprobe() even explicitely sets:
>>>>
>>>> rp->kp.post_handler = NULL;
>>>>
>>>> It seems that .post_handler is called after all ftrace handlers.
>>>> And it sets IP after the NOPs, see kprobe_ftrace_handler().
>>>> I am not sure about the use case.
>>>>
>>>> Best Regards,
>>>> Petr
>>>
^ permalink raw reply
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Song Liu @ 2025-10-17 16:58 UTC (permalink / raw)
To: Andrey Grodzovsky
Cc: Song Liu, Petr Mladek, kernel-team@lists.ubuntu.com,
live-patching@vger.kernel.org, Steven Rostedt
In-Reply-To: <82eaaada-f3fc-44f7-826d-8de47ce9fd39@crowdstrike.com>
On Thu, Oct 16, 2025 at 2:55 PM Andrey Grodzovsky
<andrey.grodzovsky@crowdstrike.com> wrote:
[...]
> [AG] - Trying first to point him at the original function - but he
> fails on the fexit I assume - which is strange, I assumed fexit
> (kretfunc) and livepatch can coexist ?
>
> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e
> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
> START (fentry): PID %d, Comm %s\n", pid, comm); }
> fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid];
> delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s,
> Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
> Attaching 2 probes...
> ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec
>
> [AG] - Trying to skip the fexit and only do fentry - he still rejects
> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e
> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
> START (fentry): PID %d, Comm %s\n", pid, comm); }'
> sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
> resolution
> INFO: node count: 12
> Attaching 1 probe...
>
> Program ID: 295
>
> The verifier log:
> processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3
> peak_states 3 mark_read 1
>
> Attaching kfunc:vmlinux:begin_new_exec
> ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec
OK, I could reproduce this issue and found the issue. In my test,
fexit+livepatch works on some older kernel, but fails on some newer
kernel. I haven't bisected to the commit that broke it.
Something like the following make it work:
diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
index 2e113f8b13a2..4277b4f33eb8 100644
--- i/kernel/trace/ftrace.c
+++ w/kernel/trace/ftrace.c
@@ -5985,6 +5985,8 @@ int register_ftrace_direct(struct ftrace_ops
*ops, unsigned long addr)
ops->direct_call = addr;
err = register_ftrace_function_nolock(ops);
+ if (err)
+ remove_direct_functions_hash(direct_functions, addr);
out_unlock:
mutex_unlock(&direct_mutex);
Andrey, could you also test this change?
Thanks,
Song
^ permalink raw reply related
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Song Liu @ 2025-10-17 19:07 UTC (permalink / raw)
To: Song Liu
Cc: Andrey Grodzovsky, Petr Mladek, kernel-team@lists.ubuntu.com,
live-patching@vger.kernel.org, Steven Rostedt
In-Reply-To: <CAHzjS_s2RhM3_H9CCedud3zkGUWW2xkmvxvPLR1qujLZRhgL1A@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2422 bytes --]
On Fri, Oct 17, 2025 at 9:58 AM Song Liu <song@kernel.org> wrote:
>
> On Thu, Oct 16, 2025 at 2:55 PM Andrey Grodzovsky
> <andrey.grodzovsky@crowdstrike.com> wrote:
> [...]
> > [AG] - Trying first to point him at the original function - but he
> > fails on the fexit I assume - which is strange, I assumed fexit
> > (kretfunc) and livepatch can coexist ?
> >
> > ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e
> > 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
> > START (fentry): PID %d, Comm %s\n", pid, comm); }
> > fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid];
> > delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s,
> > Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
> > Attaching 2 probes...
> > ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec
> >
> > [AG] - Trying to skip the fexit and only do fentry - he still rejects
> > ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e
> > 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
> > START (fentry): PID %d, Comm %s\n", pid, comm); }'
> > sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
> > resolution
> > INFO: node count: 12
> > Attaching 1 probe...
> >
> > Program ID: 295
> >
> > The verifier log:
> > processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3
> > peak_states 3 mark_read 1
> >
> > Attaching kfunc:vmlinux:begin_new_exec
> > ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec
>
> OK, I could reproduce this issue and found the issue. In my test,
> fexit+livepatch works on some older kernel, but fails on some newer
> kernel. I haven't bisected to the commit that broke it.
>
> Something like the following make it work:
>
> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
> index 2e113f8b13a2..4277b4f33eb8 100644
> --- i/kernel/trace/ftrace.c
> +++ w/kernel/trace/ftrace.c
> @@ -5985,6 +5985,8 @@ int register_ftrace_direct(struct ftrace_ops
> *ops, unsigned long addr)
> ops->direct_call = addr;
>
> err = register_ftrace_function_nolock(ops);
> + if (err)
> + remove_direct_functions_hash(direct_functions, addr);
>
> out_unlock:
> mutex_unlock(&direct_mutex);
>
> Andrey, could you also test this change?
Attached is a better version of the fix.
Thanks,
Song
[-- Attachment #2: 0001-ftrace-Fix-BPF-fexit-with-livepatch.patch --]
[-- Type: application/octet-stream, Size: 2100 bytes --]
From d4aa9c0058c794d6259248187bd5896cb40eea29 Mon Sep 17 00:00:00 2001
From: Song Liu <song@kernel.org>
Date: Fri, 17 Oct 2025 11:49:45 -0700
Subject: [PATCH] ftrace: Fix BPF fexit with livepatch
When livepatch is attached to the same function as bpf trampoline with
a fexit program, bpf trampoline code calls register_ftrace_direct()
twice. The first time will fail with -EAGAIN, and the second time it
will succeed. This requires register_ftrace_direct() to unregister
the address on the first attempt. Otherwise, the bpf trampoline cannot
attach. Here is an easy way to reproduce this issue:
insmod samples/livepatch/livepatch-sample.ko
bpftrace -e 'fexit:cmdline_proc_show {}'
ERROR: Unable to attach probe: fexit:vmlinux:cmdline_proc_show...
Fix this by updating direct_functions hash after
register_ftrace_function_nolock succeeds.
Fixes: d05cb470663a ("ftrace: Fix modification of direct_function hash while in use")
Cc: stable@vger.kernel.org # v6.6+
Reported-by: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
Closes: https://lore.kernel.org/live-patching/c5058315a39d4615b333e485893345be@crowdstrike.com/
Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Song Liu <song@kernel.org>
---
kernel/trace/ftrace.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index a69067367c29..82a06b7d0268 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -6038,16 +6038,18 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
}
}
- free_hash = direct_functions;
- rcu_assign_pointer(direct_functions, new_hash);
- new_hash = NULL;
-
ops->func = call_direct_funcs;
ops->flags = MULTI_FLAGS;
ops->trampoline = FTRACE_REGS_ADDR;
ops->direct_call = addr;
err = register_ftrace_function_nolock(ops);
+ if (err)
+ goto out_unlock;
+
+ free_hash = direct_functions;
+ rcu_assign_pointer(direct_functions, new_hash);
+ new_hash = NULL;
out_unlock:
mutex_unlock(&direct_mutex);
--
2.47.3
^ permalink raw reply related
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Andrey Grodzovsky @ 2025-10-17 19:48 UTC (permalink / raw)
To: Song Liu
Cc: Petr Mladek, kernel-team@lists.ubuntu.com,
live-patching@vger.kernel.org, Steven Rostedt
In-Reply-To: <CAHzjS_sQQaTZpxC2drGx8=7zCMAKQN_CNjYFcNzxZEGhd+yXPA@mail.gmail.com>
On 10/17/25 15:07, Song Liu wrote:
> On Fri, Oct 17, 2025 at 9:58 AM Song Liu <song@kernel.org> wrote:
>> On Thu, Oct 16, 2025 at 2:55 PM Andrey Grodzovsky
>> <andrey.grodzovsky@crowdstrike.com> wrote:
>> [...]
>>> [AG] - Trying first to point him at the original function - but he
>>> fails on the fexit I assume - which is strange, I assumed fexit
>>> (kretfunc) and livepatch can coexist ?
>>>
>>> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e
>>> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
>>> START (fentry): PID %d, Comm %s\n", pid, comm); }
>>> fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid];
>>> delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s,
>>> Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
>>> Attaching 2 probes...
>>> ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec
>>>
>>> [AG] - Trying to skip the fexit and only do fentry - he still rejects
>>> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e
>>> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
>>> START (fentry): PID %d, Comm %s\n", pid, comm); }'
>>> sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
>>> resolution
>>> INFO: node count: 12
>>> Attaching 1 probe...
>>>
>>> Program ID: 295
>>>
>>> The verifier log:
>>> processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3
>>> peak_states 3 mark_read 1
>>>
>>> Attaching kfunc:vmlinux:begin_new_exec
>>> ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec
>> OK, I could reproduce this issue and found the issue. In my test,
>> fexit+livepatch works on some older kernel, but fails on some newer
>> kernel. I haven't bisected to the commit that broke it.
>>
>> Something like the following make it work:
>>
>> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
>> index 2e113f8b13a2..4277b4f33eb8 100644
>> --- i/kernel/trace/ftrace.c
>> +++ w/kernel/trace/ftrace.c
>> @@ -5985,6 +5985,8 @@ int register_ftrace_direct(struct ftrace_ops
>> *ops, unsigned long addr)
>> ops->direct_call = addr;
>>
>> err = register_ftrace_function_nolock(ops);
>> + if (err)
>> + remove_direct_functions_hash(direct_functions, addr);
>>
>> out_unlock:
>> mutex_unlock(&direct_mutex);
>>
>> Andrey, could you also test this change?
> Attached is a better version of the fix.
>
> Thanks,
> Song
Thank you Song!
So, with this You say both fentry and fexit will work no issues ?
So juts to understand, as i am not familiar with live-patch generation,
I get the sources for my Ubuntu kernel, I apply your patch, I also
generate manually livepatch module that makes a dummy patching to my
test function (begin_new_exec), and apply this patch to my running
costum kernel ? Because i don't think the stadard ubuntu livepatching
will agree to apply his livepatch CVEs to my ostum kenel, it will
probably recognize it's not stock ubuntu kernel but amanully built one.
Thanks,
Andrey
So i need to build the stable
^ permalink raw reply
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Song Liu @ 2025-10-17 21:11 UTC (permalink / raw)
To: Andrey Grodzovsky
Cc: Song Liu, Petr Mladek, kernel-team@lists.ubuntu.com,
live-patching@vger.kernel.org, Steven Rostedt
In-Reply-To: <69339fb8-04a6-4c28-bb71-d9522ebd7282@crowdstrike.com>
On Fri, Oct 17, 2025 at 12:48 PM Andrey Grodzovsky
<andrey.grodzovsky@crowdstrike.com> wrote:
>
> On 10/17/25 15:07, Song Liu wrote:
> > On Fri, Oct 17, 2025 at 9:58 AM Song Liu <song@kernel.org> wrote:
> >> On Thu, Oct 16, 2025 at 2:55 PM Andrey Grodzovsky
> >> <andrey.grodzovsky@crowdstrike.com> wrote:
> >> [...]
> >>> [AG] - Trying first to point him at the original function - but he
> >>> fails on the fexit I assume - which is strange, I assumed fexit
> >>> (kretfunc) and livepatch can coexist ?
> >>>
> >>> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e
> >>> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
> >>> START (fentry): PID %d, Comm %s\n", pid, comm); }
> >>> fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid];
> >>> delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s,
> >>> Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
> >>> Attaching 2 probes...
> >>> ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec
> >>>
> >>> [AG] - Trying to skip the fexit and only do fentry - he still rejects
> >>> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e
> >>> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
> >>> START (fentry): PID %d, Comm %s\n", pid, comm); }'
> >>> sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
> >>> resolution
> >>> INFO: node count: 12
> >>> Attaching 1 probe...
> >>>
> >>> Program ID: 295
> >>>
> >>> The verifier log:
> >>> processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3
> >>> peak_states 3 mark_read 1
> >>>
> >>> Attaching kfunc:vmlinux:begin_new_exec
> >>> ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec
> >> OK, I could reproduce this issue and found the issue. In my test,
> >> fexit+livepatch works on some older kernel, but fails on some newer
> >> kernel. I haven't bisected to the commit that broke it.
> >>
> >> Something like the following make it work:
> >>
> >> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
> >> index 2e113f8b13a2..4277b4f33eb8 100644
> >> --- i/kernel/trace/ftrace.c
> >> +++ w/kernel/trace/ftrace.c
> >> @@ -5985,6 +5985,8 @@ int register_ftrace_direct(struct ftrace_ops
> >> *ops, unsigned long addr)
> >> ops->direct_call = addr;
> >>
> >> err = register_ftrace_function_nolock(ops);
> >> + if (err)
> >> + remove_direct_functions_hash(direct_functions, addr);
> >>
> >> out_unlock:
> >> mutex_unlock(&direct_mutex);
> >>
> >> Andrey, could you also test this change?
> > Attached is a better version of the fix.
> >
> > Thanks,
> > Song
>
> Thank you Song!
>
> So, with this You say both fentry and fexit will work no issues ?
Yes, fentry and fexit should both work.
> So juts to understand, as i am not familiar with live-patch generation,
> I get the sources for my Ubuntu kernel, I apply your patch, I also
> generate manually livepatch module that makes a dummy patching to my
> test function (begin_new_exec), and apply this patch to my running
> costum kernel ? Because i don't think the stadard ubuntu livepatching
> will agree to apply his livepatch CVEs to my ostum kenel, it will
> probably recognize it's not stock ubuntu kernel but amanully built one.
livepatch is a kernel module. Therefore, unless the two kernels are almost
identical, livepatch built for one kernel cannot be used on the other.
If you build the kernel from source code, there are some samples in
samples/livepatch that you can use for testing. PS: You need to enable
CONFIG_SAMPLE_LIVEPATCH=m
I hope this helps.
Thanks,
Song
^ permalink raw reply
* Re: [PATCH v4 08/63] kbuild: Remove 'kmod_' prefix from __KBUILD_MODNAME
From: Alexander Stein @ 2025-10-20 12:20 UTC (permalink / raw)
To: x86, Josh Poimboeuf
Cc: linux-kernel, Petr Mladek, Miroslav Benes, Joe Lawrence,
live-patching, Song Liu, laokz, Jiri Kosina,
Marcos Paulo de Souza, Weinan Liu, Fazla Mehrab, Chen Zhongjin,
Puranjay Mohan, Dylan Hatch, Peter Zijlstra, Masahiro Yamada
In-Reply-To: <f382dddad4b7c8079ce3dd91e5eaea921b03af72.1758067942.git.jpoimboe@kernel.org>
Hi,
Am Mittwoch, 17. September 2025, 18:03:16 CEST schrieb Josh Poimboeuf:
> In preparation for the objtool klp diff subcommand, remove the arbitrary
> 'kmod_' prefix from __KBUILD_MODNAME and instead add it explicitly in
> the __initcall_id() macro.
>
> This change supports the standardization of "unique" symbol naming by
> ensuring the non-unique portion of the name comes before the unique
> part. That will enable objtool to properly correlate symbols across
> builds.
>
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Starting with this commit 6717e8f91db71 ("kbuild: Remove 'kmod_' prefix
from __KBUILD_MODNAME") in next-20251020 I don't get any
module aliases anymore.
modinfo spi-fsl-dspi.ko returns:
> filename: /work/repo/linux/build_arm64/drivers/spi/spi-fsl-dspi.ko
> alias: platform:fsl-dspi
> license: GPL
> description: Freescale DSPI Controller Driver
> depends:
> intree: Y
> name: spi_fsl_dspi
> vermagic: 6.18.0-rc1+ SMP preempt mod_unload modversions aarch64
but it should be like this:
> filename: /work/repo/linux/build_arm64/drivers/spi/spi-fsl-dspi.ko
> alias: platform:fsl-dspi
> license: GPL
> description: Freescale DSPI Controller Driver
> alias: of:N*T*Cnxp,s32g2-dspiC*
> alias: of:N*T*Cnxp,s32g2-dspi
> alias: of:N*T*Cfsl,lx2160a-dspiC*
> alias: of:N*T*Cfsl,lx2160a-dspi
> alias: of:N*T*Cfsl,ls2085a-dspiC*
> alias: of:N*T*Cfsl,ls2085a-dspi
> alias: of:N*T*Cfsl,ls2080a-dspiC*
> alias: of:N*T*Cfsl,ls2080a-dspi
> alias: of:N*T*Cfsl,ls1046a-dspiC*
> alias: of:N*T*Cfsl,ls1046a-dspi
> alias: of:N*T*Cfsl,ls1043a-dspiC*
> alias: of:N*T*Cfsl,ls1043a-dspi
> alias: of:N*T*Cfsl,ls1028a-dspiC*
> alias: of:N*T*Cfsl,ls1028a-dspi
> alias: of:N*T*Cfsl,ls1012a-dspiC*
> alias: of:N*T*Cfsl,ls1012a-dspi
> alias: of:N*T*Cfsl,ls1021a-v1.0-dspiC*
> alias: of:N*T*Cfsl,ls1021a-v1.0-dspi
> alias: of:N*T*Cfsl,vf610-dspiC*
> alias: of:N*T*Cfsl,vf610-dspi
> depends:
> intree: Y
> name: spi_fsl_dspi
> vermagic: 6.18.0-rc1+ SMP preempt mod_unload modversions aarch64
Reverting does not work on all building platforms, it works on one but fails
on another one.
Best regards
Alexander
> ---
> include/linux/init.h | 3 ++-
> scripts/Makefile.lib | 2 +-
> 2 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/include/linux/init.h b/include/linux/init.h
> index 17c1bc712e234..40331923b9f4a 100644
> --- a/include/linux/init.h
> +++ b/include/linux/init.h
> @@ -200,12 +200,13 @@ extern struct module __this_module;
>
> /* Format: <modname>__<counter>_<line>_<fn> */
> #define __initcall_id(fn) \
> + __PASTE(kmod_, \
> __PASTE(__KBUILD_MODNAME, \
> __PASTE(__, \
> __PASTE(__COUNTER__, \
> __PASTE(_, \
> __PASTE(__LINE__, \
> - __PASTE(_, fn))))))
> + __PASTE(_, fn)))))))
>
> /* Format: __<prefix>__<iid><id> */
> #define __initcall_name(prefix, __iid, id) \
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index 1d581ba5df66f..b955602661240 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -20,7 +20,7 @@ name-fix-token = $(subst $(comma),_,$(subst -,_,$1))
> name-fix = $(call stringify,$(call name-fix-token,$1))
> basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
> modname_flags = -DKBUILD_MODNAME=$(call name-fix,$(modname)) \
> - -D__KBUILD_MODNAME=kmod_$(call name-fix-token,$(modname))
> + -D__KBUILD_MODNAME=$(call name-fix-token,$(modname))
> modfile_flags = -DKBUILD_MODFILE=$(call stringify,$(modfile))
>
> _c_flags = $(filter-out $(CFLAGS_REMOVE_$(target-stem).o), \
>
--
TQ-Systems GmbH | Mühlstraße 2, Gut Delling | 82229 Seefeld, Germany
Amtsgericht München, HRB 105018
Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider
http://www.tq-group.com/
^ permalink raw reply
* Re: [PATCH v4 08/63] kbuild: Remove 'kmod_' prefix from __KBUILD_MODNAME
From: Marek Szyprowski @ 2025-10-20 12:59 UTC (permalink / raw)
To: Josh Poimboeuf, x86
Cc: linux-kernel, Petr Mladek, Miroslav Benes, Joe Lawrence,
live-patching, Song Liu, laokz, Jiri Kosina,
Marcos Paulo de Souza, Weinan Liu, Fazla Mehrab, Chen Zhongjin,
Puranjay Mohan, Dylan Hatch, Peter Zijlstra, Masahiro Yamada
In-Reply-To: <f382dddad4b7c8079ce3dd91e5eaea921b03af72.1758067942.git.jpoimboe@kernel.org>
On 17.09.2025 18:03, Josh Poimboeuf wrote:
> In preparation for the objtool klp diff subcommand, remove the arbitrary
> 'kmod_' prefix from __KBUILD_MODNAME and instead add it explicitly in
> the __initcall_id() macro.
>
> This change supports the standardization of "unique" symbol naming by
> ensuring the non-unique portion of the name comes before the unique
> part. That will enable objtool to properly correlate symbols across
> builds.
>
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
This patch landed recently in linux-next as commit 6717e8f91db7
("kbuild: Remove 'kmod_' prefix from __KBUILD_MODNAME"). In my tests I
found that it completely breaks automatic modules loading on all tested
boards (ARM 32bit, ARM 64bit and RiscV64 based), what looks like some
kind of a generic issue. Reverting it on top of current linux-next fixes
this issue.
> ---
> include/linux/init.h | 3 ++-
> scripts/Makefile.lib | 2 +-
> 2 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/include/linux/init.h b/include/linux/init.h
> index 17c1bc712e234..40331923b9f4a 100644
> --- a/include/linux/init.h
> +++ b/include/linux/init.h
> @@ -200,12 +200,13 @@ extern struct module __this_module;
>
> /* Format: <modname>__<counter>_<line>_<fn> */
> #define __initcall_id(fn) \
> + __PASTE(kmod_, \
> __PASTE(__KBUILD_MODNAME, \
> __PASTE(__, \
> __PASTE(__COUNTER__, \
> __PASTE(_, \
> __PASTE(__LINE__, \
> - __PASTE(_, fn))))))
> + __PASTE(_, fn)))))))
>
> /* Format: __<prefix>__<iid><id> */
> #define __initcall_name(prefix, __iid, id) \
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index 1d581ba5df66f..b955602661240 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -20,7 +20,7 @@ name-fix-token = $(subst $(comma),_,$(subst -,_,$1))
> name-fix = $(call stringify,$(call name-fix-token,$1))
> basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
> modname_flags = -DKBUILD_MODNAME=$(call name-fix,$(modname)) \
> - -D__KBUILD_MODNAME=kmod_$(call name-fix-token,$(modname))
> + -D__KBUILD_MODNAME=$(call name-fix-token,$(modname))
> modfile_flags = -DKBUILD_MODFILE=$(call stringify,$(modfile))
>
> _c_flags = $(filter-out $(CFLAGS_REMOVE_$(target-stem).o), \
Best regards
--
Marek Szyprowski, PhD
Samsung R&D Institute Poland
^ permalink raw reply
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Andrey Grodzovsky @ 2025-10-20 14:56 UTC (permalink / raw)
To: Song Liu
Cc: Petr Mladek, kernel-team@lists.ubuntu.com,
live-patching@vger.kernel.org, Steven Rostedt
In-Reply-To: <CAHzjS_tf0KeBnzA6psjHSCuiXn--hK=owDPhCPUB0=jnLDBk=A@mail.gmail.com>
On 10/17/25 17:11, Song Liu wrote:
> On Fri, Oct 17, 2025 at 12:48 PM Andrey Grodzovsky
> <andrey.grodzovsky@crowdstrike.com> wrote:
>> On 10/17/25 15:07, Song Liu wrote:
>>> On Fri, Oct 17, 2025 at 9:58 AM Song Liu <song@kernel.org> wrote:
>>>> On Thu, Oct 16, 2025 at 2:55 PM Andrey Grodzovsky
>>>> <andrey.grodzovsky@crowdstrike.com> wrote:
>>>> [...]
>>>>> [AG] - Trying first to point him at the original function - but he
>>>>> fails on the fexit I assume - which is strange, I assumed fexit
>>>>> (kretfunc) and livepatch can coexist ?
>>>>>
>>>>> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e
>>>>> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
>>>>> START (fentry): PID %d, Comm %s\n", pid, comm); }
>>>>> fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid];
>>>>> delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s,
>>>>> Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
>>>>> Attaching 2 probes...
>>>>> ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec
>>>>>
>>>>> [AG] - Trying to skip the fexit and only do fentry - he still rejects
>>>>> ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e
>>>>> 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
>>>>> START (fentry): PID %d, Comm %s\n", pid, comm); }'
>>>>> sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
>>>>> resolution
>>>>> INFO: node count: 12
>>>>> Attaching 1 probe...
>>>>>
>>>>> Program ID: 295
>>>>>
>>>>> The verifier log:
>>>>> processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3
>>>>> peak_states 3 mark_read 1
>>>>>
>>>>> Attaching kfunc:vmlinux:begin_new_exec
>>>>> ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec
>>>> OK, I could reproduce this issue and found the issue. In my test,
>>>> fexit+livepatch works on some older kernel, but fails on some newer
>>>> kernel. I haven't bisected to the commit that broke it.
>>>>
>>>> Something like the following make it work:
>>>>
>>>> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
>>>> index 2e113f8b13a2..4277b4f33eb8 100644
>>>> --- i/kernel/trace/ftrace.c
>>>> +++ w/kernel/trace/ftrace.c
>>>> @@ -5985,6 +5985,8 @@ int register_ftrace_direct(struct ftrace_ops
>>>> *ops, unsigned long addr)
>>>> ops->direct_call = addr;
>>>>
>>>> err = register_ftrace_function_nolock(ops);
>>>> + if (err)
>>>> + remove_direct_functions_hash(direct_functions, addr);
>>>>
>>>> out_unlock:
>>>> mutex_unlock(&direct_mutex);
>>>>
>>>> Andrey, could you also test this change?
>>> Attached is a better version of the fix.
>>>
>>> Thanks,
>>> Song
>> Thank you Song!
>>
>> So, with this You say both fentry and fexit will work no issues ?
> Yes, fentry and fexit should both work.
>
>> So juts to understand, as i am not familiar with live-patch generation,
>> I get the sources for my Ubuntu kernel, I apply your patch, I also
>> generate manually livepatch module that makes a dummy patching to my
>> test function (begin_new_exec), and apply this patch to my running
>> costum kernel ? Because i don't think the stadard ubuntu livepatching
>> will agree to apply his livepatch CVEs to my ostum kenel, it will
>> probably recognize it's not stock ubuntu kernel but amanully built one.
> livepatch is a kernel module. Therefore, unless the two kernels are almost
> identical, livepatch built for one kernel cannot be used on the other.
>
> If you build the kernel from source code, there are some samples in
> samples/livepatch that you can use for testing. PS: You need to enable
>
> CONFIG_SAMPLE_LIVEPATCH=m
>
> I hope this helps.
Thanks Song, working on repro, kernel rebuilt, test module is loading
but, bpftrace is refusing to attach now to fentries/fexits claiming the
costum kernel is not supporting it. It did
attach in the case of stock AWS kernel i copied the .config from. So
just trying to figure out now if some Kcofnig flags are missing or
different . Let me know in case you manage to confirm yourself in the
meanwhile the fix works for
you.
Thanks,
Andrey
>
> Thanks,
> Song
^ permalink raw reply
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Song Liu @ 2025-10-20 16:03 UTC (permalink / raw)
To: Andrey Grodzovsky
Cc: Song Liu, Petr Mladek, kernel-team@lists.ubuntu.com,
live-patching@vger.kernel.org, Steven Rostedt
In-Reply-To: <4cc825e6-fdf8-4fc1-8ccd-9bad456c2131@crowdstrike.com>
On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
<andrey.grodzovsky@crowdstrike.com> wrote:
[...]
> > If you build the kernel from source code, there are some samples in
> > samples/livepatch that you can use for testing. PS: You need to enable
> >
> > CONFIG_SAMPLE_LIVEPATCH=m
> >
> > I hope this helps.
>
> Thanks Song, working on repro, kernel rebuilt, test module is loading
> but, bpftrace is refusing to attach now to fentries/fexits claiming the
> costum kernel is not supporting it. It did
> attach in the case of stock AWS kernel i copied the .config from. So
> just trying to figure out now if some Kcofnig flags are missing or
> different . Let me know in case you manage to confirm yourself in the
> meanwhile the fix works for
> you.
Yes, it worked in my tests.
[root@(none) /]# kpatch load linux/samples/livepatch/livepatch-sample.ko
loading patch module: linux/samples/livepatch/livepatch-sample.ko
[root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show
{printf("fexit\n");}' &
[1] 388
[root@(none) /]# Attached 1 probe
[root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show
{printf("fentry\n");}' &
[2] 397
[root@(none) /]# Attached 1 probe
[root@(none) /]# cat /proc/cmdline
this has been live patched
fentry
fexit
Thanks,
Song
^ permalink raw reply
* Re: [PATCH v4 08/63] kbuild: Remove 'kmod_' prefix from __KBUILD_MODNAME
From: Mark Brown @ 2025-10-20 16:34 UTC (permalink / raw)
To: Josh Poimboeuf
Cc: x86, linux-kernel, Petr Mladek, Miroslav Benes, Joe Lawrence,
live-patching, Song Liu, laokz, Jiri Kosina,
Marcos Paulo de Souza, Weinan Liu, Fazla Mehrab, Chen Zhongjin,
Puranjay Mohan, Dylan Hatch, Peter Zijlstra, Masahiro Yamada,
Aishwarya.TCV
In-Reply-To: <f382dddad4b7c8079ce3dd91e5eaea921b03af72.1758067942.git.jpoimboe@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 23333 bytes --]
On Wed, Sep 17, 2025 at 09:03:16AM -0700, Josh Poimboeuf wrote:
> In preparation for the objtool klp diff subcommand, remove the arbitrary
> 'kmod_' prefix from __KBUILD_MODNAME and instead add it explicitly in
> the __initcall_id() macro.
I'll add my name to the list of people reporting that this is causing
extensive breakage in -next, anything that tries to load modules has a
bad time which takes out a lot of tests.
Sample bisect log:
# bad: [606da5bb165594c052ee11de79bf05bc38bc1aa6] Add linux-next specific files for 20251020
# good: [4d7f6adc7b72e14c0f2646a4abb79bb6cc676ada] Merge branch 'for-linux-next-fixes' of https://gitlab.freedesktop.org/drm/misc/kernel.git
# good: [211ddde0823f1442e4ad052a2f30f050145ccada] Linux 6.18-rc2
# good: [0cdb2b1b7edaefb54773d790c7b5c2e4ac7db60d] spi: airoha: driver fixes & improvements
# good: [0743acf746a81e0460a56fd5ff847d97fa7eb370] spi: airoha: buffer must be 0xff-ed before writing
# good: [661856ca131c8bf6724905966e02149805660abe] spi: airoha: remove unnecessary restriction length
# good: [70eec454f2d6cdfab547c262781acd38328e11a1] spi: airoha: avoid setting of page/oob sizes in REG_SPI_NFI_PAGEFMT
# good: [d1ff30df1d9a4eb4c067795abb5e2a66910fd108] spi: airoha: reduce the number of modification of REG_SPI_NFI_CNFG and REG_SPI_NFI_SECCUS_SIZE registers
# good: [fb81b5cecb8553e3ca2b45288cf340d43c9c2991] spi: airoha: set custom sector size equal to flash page size
# good: [902c0ea18a97b1a6eeee5799cb1fd9a79ef9208e] spi: airoha: avoid reading flash page settings from SNFI registers during driver startup
# good: [80b09137aeab27e59004383058f8cc696a9ee048] spi: airoha: support of dualio/quadio flash reading commands
# good: [0cc08c8130ac8f74419f99fe707dc193b7f79d86] spi: aspeed: Fix an IS_ERR() vs NULL bug in probe()
# good: [93b2838c6e79bc263e6129d88c5ab043dd793d28] Add Audio Support for Kaanapali MTP Boards
# good: [233a22687411ea053a4b169c07324ee6aa33bf38] spi: airoha: unify dirmap read/write code
# good: [7350f8dc15bfbb7abf1ce4babea6fcace1c574c5] spi: airoha: remove unnecessary switch to non-dma mode
# good: [5263cd81578f99a00b2dd7de1da2b570b96a1b7c] rpmh-regulators: Update rpmh-regulator driver and
# good: [281c97376cfcfc8cef4f5ed5dd961a1b39f5a25e] ASoC: codecs: va-macro: Rework version checking
# good: [d77daa49085b067137d0adbe3263f75a7ee13a1b] spi: aspeed: fix spelling mistake "triming" -> "trimming"
# good: [4673dbe9837e3eb2fecdd12f0953006c31636aac] ASoC: qcom: sc8280xp: Add support for Kaanapali
# good: [367ca0688e4218e51c3d4dfdf3ef5657a62cf88d] ASoC: dt-bindings: qcom,sm8250: Add kaanapali sound card
# good: [15afe57a874eaf104bfbb61ec598fa31627f7b19] ASoC: dt-bindings: qcom: Add Kaanapali LPASS macro codecs
# good: [1e570e77392f43a3cdab2849d1f81535f8a033e2] ASoC: mxs-saif: support usage with simple-audio-card
# good: [1356c98ef911e14ccfaf374800840ce5bdcb3bbd] regulator: dt-bindings: rpmh-regulator: Update pmic-id DT prop info for new CMD-DB
# good: [fb25114cd760c13cf177d9ac37837fafcc9657b5] regulator: sy7636a: add gpios and input regulator
# good: [835dfb12fc389f36eb007657f163bd1c539dcd45] regulator: dt-bindings: rpmh-regulator : Add compatibles for PMH01XX & PMCX0102
# good: [65efe5404d151767653c7b7dd39bd2e7ad532c2d] regulator: rpmh-regulator: Add RPMH regulator support for Glymur
# good: [6621b0f118d500092f5f3d72ddddb22aeeb3c3a0] ASoC: codecs: rt5670: use SOC_VALUE_ENUM_SINGLE_DECL for DAC2 L/R MX-1B
# good: [6a8cdef7dc2a4c0dbde3f7d7100b3d99712a766b] regulator: rpmh-regulator: Add support for new resource name format
# good: [cee2c8396d9c8e834fe28929bc1d8153d7e9897f] ASoC: use sof_sdw as default Intel SOF SDW machine
# good: [7d9c2924f61dcabcbc5868bec6054ab4f4de01d1] spi: aspeed: Improve clock, timing and address
# good: [79c36ecfc8994011ab0a973d3b4148aa5d9e0c91] ASoC: use sof_sdw as default Intel SOF SDW machine
# good: [433e294c3c5b5d2020085a0e36c1cb47b694690a] regulator: core: forward undervoltage events downstream by default
# good: [0b0eb7702a9fa410755e86124b4b7cd36e7d1cb4] ASoC: replace use of system_wq with system_dfl_wq
# good: [92a42edd347c3b5a9045bb137a33204c6ddc0803] Add target mode support for the DesignWare SPI
# good: [c1afb0350069c3be137b5692923ad13d69648970] Add tegra264 audio device tree support
# good: [a758314f71ba90cca2a5813bbf96c4954a15b613] spi: offload: Add offset parameter
# good: [506cbe36a2ac7b504a2362476dc53cd548b7a29e] ASoC: soc_sdw_utils: export asoc_sdw_get_dai_type
# good: [225d70b8074502acee3943bf0c2e839e867cd38c] ASoC: SOF: don't check the existence of dummy topology
# good: [55f8b5a96597a7b88c323a7de7228f9eae8c9943] ASoC: Intel: export sof_sdw_get_tplg_files
# good: [7e7e2c6e2a1cb250f8d03bb99eed01f6d982d5dd] ASoC: sof-function-topology-lib: escalate the log when missing function topoplogy
# good: [bb940b13998c40d55e186f0cf5d65c592ea1677a] ASoC: SOF: Don't print the monolithic topology name if function topology may be used
# good: [aa1ee85ce3576defd29f2a389d7508d2036af977] ASoC: soc_sdw_utils: add name_prefix to asoc_sdw_codec_info struct
# good: [7f47685b150dbc20f881d029a7366a81b1d66322] ASoC: SOF: Intel: use sof_sdw as default SDW machine driver
# good: [3180c7b1575d635851f0ceab6bdb176bb15e69dd] ASoC: soc-acpi: make some variables of acpi adr and link adr non-const
# good: [0d202ae0256e8e7dcea862ead5904fa27cf4ce6a] ASoC: SOF: add platform name into sof_intel_dsp_desc
# good: [2b92b98cc4765fbb0748742e7e0dd94d15d6f178] ASoC: SOF: Don't print the monolithic topology name if function topology may be used
# good: [6277a486a7faaa6c87f4bf1d59a2de233a093248] regulator: dt-bindings: Convert Dialog DA9211 Regulators to DT schema
# good: [454cd43a283f7697297c52981c7a499a16725656] spi: dt-bindings: spi-qpic-snand: Add IPQ5424 compatible
# good: [6937ff42f28a13ffdbe2d1f5b9a51a35f626e93a] ASoC: SOF: add platform name into sof_intel_dsp_desc
# good: [b926b15547d29a88932de3c24a05c12826fc1dbc] spi: dw: rename the spi controller to ctlr
# good: [7196fc4e482928a276da853e2687f31cd8ea2611] ASoC: Intel: export sof_sdw_get_tplg_files
# good: [d25de16477657f9eddd4be9abd409515edcc3b9e] ASoC: soc-acpi: make some variables of acpi adr and link adr non-const
# good: [630a185fd06109193574d10f38b29812986c21de] spi: aspeed: Force default address decoding range assignment for each CS
# good: [5226d19d4cae5398caeb93a6052bfb614e0099c7] ASoC: SOF: Intel: use sof_sdw as default SDW machine driver
# good: [31dcc7e1f8a9377d8fd9f967f84c121c5ba8f89c] spi: aspeed: Update clock selection strategy
# good: [9797329220a2c6622411eb9ecf6a35b24ce09d04] ASoC: sof-function-topology-lib: escalate the log when missing function topoplogy
# good: [ea97713903784286ef1ce45456f404ed288f19b1] ASoC: soc_sdw_utils: add name_prefix to asoc_sdw_codec_info struct
# good: [4ebe64f507ca921c5109eb37eae6058b77413d93] ASoC: tas2781: Add TAS5802, TAS5815, and TAS5828
# good: [5ed60e45c59d66e61586a10433e2b5527d4d72b5] ASoC: soc_sdw_utils: export asoc_sdw_get_dai_type
# good: [b83fb1b14c06bdd765903ac852ba20a14e24f227] spi: offload: Add offset parameter
# good: [4d410ba9aa275e7990a270f63ce436990ace1bea] dt-bindings: sound: Update ADMAIF bindings for tegra264
# good: [29fa213c6ab00c6749db47b47e384cab760c109e] ASoC: dt-bindings: ti,tas2781: Add TAS5802, TAS5815, and TAS5828
# good: [99c159279c6dfa2c4867c7f76875f58263f8f43b] ASoC: SOF: don't check the existence of dummy topology
# good: [0586b53d4a0c7c5a132629f99da934cc674ea4cd] spi: aspeed: Add per-platform adjust_window callback for decoding range
# good: [3c89238ca35bfe176ba34bc688541f90f6fa7bdb] ASoc: tas2783A: Remove unneeded variable assignment
# good: [5e537031f322d55315cd384398b726a9a0748d47] ASoC: codecs: Fix the error of excessive semicolons
# good: [4412ab501677606436e5c49e41151a1e6eac7ac0] spi: dt-bindings: spi-qpic-snand: Add IPQ5332 compatible
# good: [efb79de36e947d136517bac14c139d494fcc72fa] spi: aspeed: Improve timing calibration algorithm for AST2600 platform
# good: [fe8cc44dd173cde5788ab4e3730ac61f3d316d9c] spi: dw: add target mode support
# good: [64d87ccfae3326a9561fe41dc6073064a083e0df] spi: aspeed: Only map necessary address window region
# good: [be2ba2fef1676861b295053c2a567b057e9031b9] bugs/sh: Concatenate 'cond_str' with '__FILE__' in __WARN_FLAGS(), to extend WARN_ON/BUG_ON output
# good: [6c177775dcc5e70a64ddf4ee842c66af498f2c7c] Merge branch 'next/drivers' into for-next
# good: [da15636d84a812275f785057451c0e7b15b39012] Merge branch 'v6.18/arm64-dt' into for-next
# good: [06dd3eda0e958cdae48ca755eb5047484f678d78] Merge branch 'vfs-6.18.rust' into vfs.all
# good: [6dde339a3df80a57ac3d780d8cfc14d9262e2acd] landlock: Minor comments improvements
# good: [447c98c1ca4a4b0d43be99f76c558c09956484f3] tools/power turbostat: Add idle governor statistics reporting
# good: [3778dcb2dcc1ce03a2b25e5d951fe9912e843d1e] random: use offstack cpumask when necessary
# good: [6684aba0780da9f505c202f27e68ee6d18c0aa66] XArray: Add extra debugging check to xas_lock and friends
git bisect start '606da5bb165594c052ee11de79bf05bc38bc1aa6' '4d7f6adc7b72e14c0f2646a4abb79bb6cc676ada' '211ddde0823f1442e4ad052a2f30f050145ccada' '0cdb2b1b7edaefb54773d790c7b5c2e4ac7db60d' '0743acf746a81e0460a56fd5ff847d97fa7eb370' '661856ca131c8bf6724905966e02149805660abe' '70eec454f2d6cdfab547c262781acd38328e11a1' 'd1ff30df1d9a4eb4c067795abb5e2a66910fd108' 'fb81b5cecb8553e3ca2b45288cf340d43c9c2991' '902c0ea18a97b1a6eeee5799cb1fd9a79ef9208e' '80b09137aeab27e59004383058f8cc696a9ee048' '0cc08c8130ac8f74419f99fe707dc193b7f79d86' '93b2838c6e79bc263e6129d88c5ab043dd793d28' '233a22687411ea053a4b169c07324ee6aa33bf38' '7350f8dc15bfbb7abf1ce4babea6fcace1c574c5' '5263cd81578f99a00b2dd7de1da2b570b96a1b7c' '281c97376cfcfc8cef4f5ed5dd961a1b39f5a25e' 'd77daa49085b067137d0adbe3263f75a7ee13a1b' '4673dbe9837e3eb2fecdd12f0953006c31636aac' '367ca0688e4218e51c3d4dfdf3ef5657a62cf88d' '15afe57a874eaf104bfbb61ec598fa31627f7b19' '1e570e77392f43a3cdab2849d1f81535f8a033e2' '1356c98ef911e14ccfaf374800840ce5bdcb3bbd' 'fb25114cd760c13cf177d9ac37837fafcc9657b5' '835dfb12fc389f36eb007657f163bd1c539dcd45' '65efe5404d151767653c7b7dd39bd2e7ad532c2d' '6621b0f118d500092f5f3d72ddddb22aeeb3c3a0' '6a8cdef7dc2a4c0dbde3f7d7100b3d99712a766b' 'cee2c8396d9c8e834fe28929bc1d8153d7e9897f' '7d9c2924f61dcabcbc5868bec6054ab4f4de01d1' '79c36ecfc8994011ab0a973d3b4148aa5d9e0c91' '433e294c3c5b5d2020085a0e36c1cb47b694690a' '0b0eb7702a9fa410755e86124b4b7cd36e7d1cb4' '92a42edd347c3b5a9045bb137a33204c6ddc0803' 'c1afb0350069c3be137b5692923ad13d69648970' 'a758314f71ba90cca2a5813bbf96c4954a15b613' '506cbe36a2ac7b504a2362476dc53cd548b7a29e' '225d70b8074502acee3943bf0c2e839e867cd38c' '55f8b5a96597a7b88c323a7de7228f9eae8c9943' '7e7e2c6e2a1cb250f8d03bb99eed01f6d982d5dd' 'bb940b13998c40d55e186f0cf5d65c592ea1677a' 'aa1ee85ce3576defd29f2a389d7508d2036af977' '7f47685b150dbc20f881d029a7366a81b1d66322' '3180c7b1575d635851f0ceab6bdb176bb15e69dd' '0d202ae0256e8e7dcea862ead5904fa27cf4ce6a' '2b92b98cc4765fbb0748742e7e0dd94d15d6f178' '6277a486a7faaa6c87f4bf1d59a2de233a093248' '454cd43a283f7697297c52981c7a499a16725656' '6937ff42f28a13ffdbe2d1f5b9a51a35f626e93a' 'b926b15547d29a88932de3c24a05c12826fc1dbc' '7196fc4e482928a276da853e2687f31cd8ea2611' 'd25de16477657f9eddd4be9abd409515edcc3b9e' '630a185fd06109193574d10f38b29812986c21de' '5226d19d4cae5398caeb93a6052bfb614e0099c7' '31dcc7e1f8a9377d8fd9f967f84c121c5ba8f89c' '9797329220a2c6622411eb9ecf6a35b24ce09d04' 'ea97713903784286ef1ce45456f404ed288f19b1' '4ebe64f507ca921c5109eb37eae6058b77413d93' '5ed60e45c59d66e61586a10433e2b5527d4d72b5' 'b83fb1b14c06bdd765903ac852ba20a14e24f227' '4d410ba9aa275e7990a270f63ce436990ace1bea' '29fa213c6ab00c6749db47b47e384cab760c109e' '99c159279c6dfa2c4867c7f76875f58263f8f43b' '0586b53d4a0c7c5a132629f99da934cc674ea4cd' '3c89238ca35bfe176ba34bc688541f90f6fa7bdb' '5e537031f322d55315cd384398b726a9a0748d47' '4412ab501677606436e5c49e41151a1e6eac7ac0' 'efb79de36e947d136517bac14c139d494fcc72fa' 'fe8cc44dd173cde5788ab4e3730ac61f3d316d9c' '64d87ccfae3326a9561fe41dc6073064a083e0df' 'be2ba2fef1676861b295053c2a567b057e9031b9' '6c177775dcc5e70a64ddf4ee842c66af498f2c7c' 'da15636d84a812275f785057451c0e7b15b39012' '06dd3eda0e958cdae48ca755eb5047484f678d78' '6dde339a3df80a57ac3d780d8cfc14d9262e2acd' '447c98c1ca4a4b0d43be99f76c558c09956484f3' '3778dcb2dcc1ce03a2b25e5d951fe9912e843d1e' '6684aba0780da9f505c202f27e68ee6d18c0aa66'
# test job: [211ddde0823f1442e4ad052a2f30f050145ccada] https://lava.sirena.org.uk/scheduler/job/1976562
# test job: [0cdb2b1b7edaefb54773d790c7b5c2e4ac7db60d] https://lava.sirena.org.uk/scheduler/job/1965930
# test job: [0743acf746a81e0460a56fd5ff847d97fa7eb370] https://lava.sirena.org.uk/scheduler/job/1964854
# test job: [661856ca131c8bf6724905966e02149805660abe] https://lava.sirena.org.uk/scheduler/job/1965121
# test job: [70eec454f2d6cdfab547c262781acd38328e11a1] https://lava.sirena.org.uk/scheduler/job/1964899
# test job: [d1ff30df1d9a4eb4c067795abb5e2a66910fd108] https://lava.sirena.org.uk/scheduler/job/1965054
# test job: [fb81b5cecb8553e3ca2b45288cf340d43c9c2991] https://lava.sirena.org.uk/scheduler/job/1965035
# test job: [902c0ea18a97b1a6eeee5799cb1fd9a79ef9208e] https://lava.sirena.org.uk/scheduler/job/1964990
# test job: [80b09137aeab27e59004383058f8cc696a9ee048] https://lava.sirena.org.uk/scheduler/job/1965075
# test job: [0cc08c8130ac8f74419f99fe707dc193b7f79d86] https://lava.sirena.org.uk/scheduler/job/1965700
# test job: [93b2838c6e79bc263e6129d88c5ab043dd793d28] https://lava.sirena.org.uk/scheduler/job/1964758
# test job: [233a22687411ea053a4b169c07324ee6aa33bf38] https://lava.sirena.org.uk/scheduler/job/1965101
# test job: [7350f8dc15bfbb7abf1ce4babea6fcace1c574c5] https://lava.sirena.org.uk/scheduler/job/1965137
# test job: [5263cd81578f99a00b2dd7de1da2b570b96a1b7c] https://lava.sirena.org.uk/scheduler/job/1964744
# test job: [281c97376cfcfc8cef4f5ed5dd961a1b39f5a25e] https://lava.sirena.org.uk/scheduler/job/1962783
# test job: [d77daa49085b067137d0adbe3263f75a7ee13a1b] https://lava.sirena.org.uk/scheduler/job/1962822
# test job: [4673dbe9837e3eb2fecdd12f0953006c31636aac] https://lava.sirena.org.uk/scheduler/job/1962868
# test job: [367ca0688e4218e51c3d4dfdf3ef5657a62cf88d] https://lava.sirena.org.uk/scheduler/job/1962854
# test job: [15afe57a874eaf104bfbb61ec598fa31627f7b19] https://lava.sirena.org.uk/scheduler/job/1962908
# test job: [1e570e77392f43a3cdab2849d1f81535f8a033e2] https://lava.sirena.org.uk/scheduler/job/1962270
# test job: [1356c98ef911e14ccfaf374800840ce5bdcb3bbd] https://lava.sirena.org.uk/scheduler/job/1959898
# test job: [fb25114cd760c13cf177d9ac37837fafcc9657b5] https://lava.sirena.org.uk/scheduler/job/1960171
# test job: [835dfb12fc389f36eb007657f163bd1c539dcd45] https://lava.sirena.org.uk/scheduler/job/1959997
# test job: [65efe5404d151767653c7b7dd39bd2e7ad532c2d] https://lava.sirena.org.uk/scheduler/job/1959947
# test job: [6621b0f118d500092f5f3d72ddddb22aeeb3c3a0] https://lava.sirena.org.uk/scheduler/job/1959730
# test job: [6a8cdef7dc2a4c0dbde3f7d7100b3d99712a766b] https://lava.sirena.org.uk/scheduler/job/1959863
# test job: [cee2c8396d9c8e834fe28929bc1d8153d7e9897f] https://lava.sirena.org.uk/scheduler/job/1959137
# test job: [7d9c2924f61dcabcbc5868bec6054ab4f4de01d1] https://lava.sirena.org.uk/scheduler/job/1957764
# test job: [79c36ecfc8994011ab0a973d3b4148aa5d9e0c91] https://lava.sirena.org.uk/scheduler/job/1959155
# test job: [433e294c3c5b5d2020085a0e36c1cb47b694690a] https://lava.sirena.org.uk/scheduler/job/1957349
# test job: [0b0eb7702a9fa410755e86124b4b7cd36e7d1cb4] https://lava.sirena.org.uk/scheduler/job/1957419
# test job: [92a42edd347c3b5a9045bb137a33204c6ddc0803] https://lava.sirena.org.uk/scheduler/job/1957776
# test job: [c1afb0350069c3be137b5692923ad13d69648970] https://lava.sirena.org.uk/scheduler/job/1959166
# test job: [a758314f71ba90cca2a5813bbf96c4954a15b613] https://lava.sirena.org.uk/scheduler/job/1957731
# test job: [506cbe36a2ac7b504a2362476dc53cd548b7a29e] https://lava.sirena.org.uk/scheduler/job/1954111
# test job: [225d70b8074502acee3943bf0c2e839e867cd38c] https://lava.sirena.org.uk/scheduler/job/1954369
# test job: [55f8b5a96597a7b88c323a7de7228f9eae8c9943] https://lava.sirena.org.uk/scheduler/job/1954364
# test job: [7e7e2c6e2a1cb250f8d03bb99eed01f6d982d5dd] https://lava.sirena.org.uk/scheduler/job/1954346
# test job: [bb940b13998c40d55e186f0cf5d65c592ea1677a] https://lava.sirena.org.uk/scheduler/job/1954225
# test job: [aa1ee85ce3576defd29f2a389d7508d2036af977] https://lava.sirena.org.uk/scheduler/job/1954318
# test job: [7f47685b150dbc20f881d029a7366a81b1d66322] https://lava.sirena.org.uk/scheduler/job/1954198
# test job: [3180c7b1575d635851f0ceab6bdb176bb15e69dd] https://lava.sirena.org.uk/scheduler/job/1954125
# test job: [0d202ae0256e8e7dcea862ead5904fa27cf4ce6a] https://lava.sirena.org.uk/scheduler/job/1954349
# test job: [2b92b98cc4765fbb0748742e7e0dd94d15d6f178] https://lava.sirena.org.uk/scheduler/job/1947182
# test job: [6277a486a7faaa6c87f4bf1d59a2de233a093248] https://lava.sirena.org.uk/scheduler/job/1947002
# test job: [454cd43a283f7697297c52981c7a499a16725656] https://lava.sirena.org.uk/scheduler/job/1946525
# test job: [6937ff42f28a13ffdbe2d1f5b9a51a35f626e93a] https://lava.sirena.org.uk/scheduler/job/1947463
# test job: [b926b15547d29a88932de3c24a05c12826fc1dbc] https://lava.sirena.org.uk/scheduler/job/1947155
# test job: [7196fc4e482928a276da853e2687f31cd8ea2611] https://lava.sirena.org.uk/scheduler/job/1947424
# test job: [d25de16477657f9eddd4be9abd409515edcc3b9e] https://lava.sirena.org.uk/scheduler/job/1947441
# test job: [630a185fd06109193574d10f38b29812986c21de] https://lava.sirena.org.uk/scheduler/job/1947408
# test job: [5226d19d4cae5398caeb93a6052bfb614e0099c7] https://lava.sirena.org.uk/scheduler/job/1947447
# test job: [31dcc7e1f8a9377d8fd9f967f84c121c5ba8f89c] https://lava.sirena.org.uk/scheduler/job/1947173
# test job: [9797329220a2c6622411eb9ecf6a35b24ce09d04] https://lava.sirena.org.uk/scheduler/job/1947403
# test job: [ea97713903784286ef1ce45456f404ed288f19b1] https://lava.sirena.org.uk/scheduler/job/1947261
# test job: [4ebe64f507ca921c5109eb37eae6058b77413d93] https://lava.sirena.org.uk/scheduler/job/1946707
# test job: [5ed60e45c59d66e61586a10433e2b5527d4d72b5] https://lava.sirena.org.uk/scheduler/job/1947418
# test job: [b83fb1b14c06bdd765903ac852ba20a14e24f227] https://lava.sirena.org.uk/scheduler/job/1946831
# test job: [4d410ba9aa275e7990a270f63ce436990ace1bea] https://lava.sirena.org.uk/scheduler/job/1947842
# test job: [29fa213c6ab00c6749db47b47e384cab760c109e] https://lava.sirena.org.uk/scheduler/job/1946529
# test job: [99c159279c6dfa2c4867c7f76875f58263f8f43b] https://lava.sirena.org.uk/scheduler/job/1947475
# test job: [0586b53d4a0c7c5a132629f99da934cc674ea4cd] https://lava.sirena.org.uk/scheduler/job/1947341
# test job: [3c89238ca35bfe176ba34bc688541f90f6fa7bdb] https://lava.sirena.org.uk/scheduler/job/1946592
# test job: [5e537031f322d55315cd384398b726a9a0748d47] https://lava.sirena.org.uk/scheduler/job/1946678
# test job: [4412ab501677606436e5c49e41151a1e6eac7ac0] https://lava.sirena.org.uk/scheduler/job/1946586
# test job: [efb79de36e947d136517bac14c139d494fcc72fa] https://lava.sirena.org.uk/scheduler/job/1947142
# test job: [fe8cc44dd173cde5788ab4e3730ac61f3d316d9c] https://lava.sirena.org.uk/scheduler/job/1947055
# test job: [64d87ccfae3326a9561fe41dc6073064a083e0df] https://lava.sirena.org.uk/scheduler/job/1947243
# test job: [be2ba2fef1676861b295053c2a567b057e9031b9] https://lava.sirena.org.uk/scheduler/job/1738858
# test job: [6c177775dcc5e70a64ddf4ee842c66af498f2c7c] https://lava.sirena.org.uk/scheduler/job/1774070
# test job: [da15636d84a812275f785057451c0e7b15b39012] https://lava.sirena.org.uk/scheduler/job/1855168
# test job: [06dd3eda0e958cdae48ca755eb5047484f678d78] https://lava.sirena.org.uk/scheduler/job/1831740
# test job: [6dde339a3df80a57ac3d780d8cfc14d9262e2acd] https://lava.sirena.org.uk/scheduler/job/1647204
# test job: [447c98c1ca4a4b0d43be99f76c558c09956484f3] https://lava.sirena.org.uk/scheduler/job/1647256
# test job: [3778dcb2dcc1ce03a2b25e5d951fe9912e843d1e] https://lava.sirena.org.uk/scheduler/job/1676804
# test job: [6684aba0780da9f505c202f27e68ee6d18c0aa66] https://lava.sirena.org.uk/scheduler/job/1738724
# test job: [606da5bb165594c052ee11de79bf05bc38bc1aa6] https://lava.sirena.org.uk/scheduler/job/1976482
# bad: [606da5bb165594c052ee11de79bf05bc38bc1aa6] Add linux-next specific files for 20251020
git bisect bad 606da5bb165594c052ee11de79bf05bc38bc1aa6
# skip: [9a8ae3ba7a0d4467067d039a3eba8c0b0bd1c153] Merge branch 'libcrypto-next' of https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git
git bisect skip 9a8ae3ba7a0d4467067d039a3eba8c0b0bd1c153
# test job: [9a4bbd20a879b66dddb563698345b0ae24d810a6] https://lava.sirena.org.uk/scheduler/job/1978205
# good: [9a4bbd20a879b66dddb563698345b0ae24d810a6] hwmon: (tmp108) Drop mutex.h include
git bisect good 9a4bbd20a879b66dddb563698345b0ae24d810a6
# test job: [42277a1f86a14172c673a249bdc9e757065ca5a1] https://lava.sirena.org.uk/scheduler/job/1978413
# good: [42277a1f86a14172c673a249bdc9e757065ca5a1] Merge branch 'for-6.19' into for-next
git bisect good 42277a1f86a14172c673a249bdc9e757065ca5a1
# test job: [dd590d4d57ebeeb826823c288741f2ed20f452af] https://lava.sirena.org.uk/scheduler/job/1978495
# bad: [dd590d4d57ebeeb826823c288741f2ed20f452af] objtool/klp: Introduce klp diff subcommand for diffing object files
git bisect bad dd590d4d57ebeeb826823c288741f2ed20f452af
# test job: [3e4b5f66cf1a7879a081f5044ff1796aa33cb999] https://lava.sirena.org.uk/scheduler/job/1978783
# bad: [3e4b5f66cf1a7879a081f5044ff1796aa33cb999] objtool: Check for missing annotation entries in read_annotate()
git bisect bad 3e4b5f66cf1a7879a081f5044ff1796aa33cb999
# test job: [4109043bff31f95d3da9ace33eb3c1925fd62cbd] https://lava.sirena.org.uk/scheduler/job/1978959
# bad: [4109043bff31f95d3da9ace33eb3c1925fd62cbd] modpost: Ignore unresolved section bounds symbols
git bisect bad 4109043bff31f95d3da9ace33eb3c1925fd62cbd
# test job: [1ba9f8979426590367406c70c1c821f5b943f993] https://lava.sirena.org.uk/scheduler/job/1979168
# good: [1ba9f8979426590367406c70c1c821f5b943f993] vmlinux.lds: Unify TEXT_MAIN, DATA_MAIN, and related macros
git bisect good 1ba9f8979426590367406c70c1c821f5b943f993
# test job: [9f14f1f91883aa2bfd6663161d2002c8ce937c43] https://lava.sirena.org.uk/scheduler/job/1979231
# good: [9f14f1f91883aa2bfd6663161d2002c8ce937c43] compiler.h: Make addressable symbols less of an eyesore
git bisect good 9f14f1f91883aa2bfd6663161d2002c8ce937c43
# test job: [6717e8f91db71641cb52855ed14c7900972ed0bc] https://lava.sirena.org.uk/scheduler/job/1979182
# bad: [6717e8f91db71641cb52855ed14c7900972ed0bc] kbuild: Remove 'kmod_' prefix from __KBUILD_MODNAME
git bisect bad 6717e8f91db71641cb52855ed14c7900972ed0bc
# test job: [c2d420796a427dda71a2400909864e7f8e037fd4] https://lava.sirena.org.uk/scheduler/job/1979307
# good: [c2d420796a427dda71a2400909864e7f8e037fd4] elfnote: Change ELFNOTE() to use __UNIQUE_ID()
git bisect good c2d420796a427dda71a2400909864e7f8e037fd4
# first bad commit: [6717e8f91db71641cb52855ed14c7900972ed0bc] kbuild: Remove 'kmod_' prefix from __KBUILD_MODNAME
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* Re: [PATCH v4 08/63] kbuild: Remove 'kmod_' prefix from __KBUILD_MODNAME
From: Cosmin Tanislav @ 2025-10-20 16:37 UTC (permalink / raw)
To: Josh Poimboeuf, x86
Cc: linux-kernel, Petr Mladek, Miroslav Benes, Joe Lawrence,
live-patching, Song Liu, laokz, Jiri Kosina,
Marcos Paulo de Souza, Weinan Liu, Fazla Mehrab, Chen Zhongjin,
Puranjay Mohan, Dylan Hatch, Peter Zijlstra, Masahiro Yamada
In-Reply-To: <f382dddad4b7c8079ce3dd91e5eaea921b03af72.1758067942.git.jpoimboe@kernel.org>
On 9/17/25 7:03 PM, Josh Poimboeuf wrote:
> In preparation for the objtool klp diff subcommand, remove the arbitrary
> 'kmod_' prefix from __KBUILD_MODNAME and instead add it explicitly in
> the __initcall_id() macro.
>
> This change supports the standardization of "unique" symbol naming by
> ensuring the non-unique portion of the name comes before the unique
> part. That will enable objtool to properly correlate symbols across
> builds.
>
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
> ---
> include/linux/init.h | 3 ++-
> scripts/Makefile.lib | 2 +-
> 2 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/include/linux/init.h b/include/linux/init.h
> index 17c1bc712e234..40331923b9f4a 100644
> --- a/include/linux/init.h
> +++ b/include/linux/init.h
> @@ -200,12 +200,13 @@ extern struct module __this_module;
>
> /* Format: <modname>__<counter>_<line>_<fn> */
> #define __initcall_id(fn) \
> + __PASTE(kmod_, \
> __PASTE(__KBUILD_MODNAME, \
> __PASTE(__, \
> __PASTE(__COUNTER__, \
> __PASTE(_, \
> __PASTE(__LINE__, \
> - __PASTE(_, fn))))))
> + __PASTE(_, fn)))))))
>
> /* Format: __<prefix>__<iid><id> */
> #define __initcall_name(prefix, __iid, id) \
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index 1d581ba5df66f..b955602661240 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -20,7 +20,7 @@ name-fix-token = $(subst $(comma),_,$(subst -,_,$1))
> name-fix = $(call stringify,$(call name-fix-token,$1))
> basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
> modname_flags = -DKBUILD_MODNAME=$(call name-fix,$(modname)) \
> - -D__KBUILD_MODNAME=kmod_$(call name-fix-token,$(modname))
> + -D__KBUILD_MODNAME=$(call name-fix-token,$(modname))
As others have mentioned, this breaks modules.alias generation.
The following diff seems to fix it, although in introduces a slight
functional change if symbols do not actually follow the naming scheme.
diff --git a/scripts/mod/file2alias.c b/scripts/mod/file2alias.c
index b3333560b95e..c3c06b944c69 100644
--- a/scripts/mod/file2alias.c
+++ b/scripts/mod/file2alias.c
@@ -1489,14 +1489,11 @@ void handle_moddevtable(struct module *mod,
struct elf_info *info,
if (ELF_ST_TYPE(sym->st_info) != STT_OBJECT)
return;
- /* All our symbols are of form
__mod_device_table__kmod_<modname>__<type>__<name>. */
+ /* All our symbols are of form
__mod_device_table__<modname>__<type>__<name>. */
if (!strstarts(symname, prefix))
return;
- modname = strstr(symname, "__kmod_");
- if (!modname)
- return;
- modname += strlen("__kmod_");
+ modname = symname + strlen(prefix);
type = strstr(modname, "__");
if (!type)
It would seem like rust generated symbols don't follow it exactly?
See module_device_table macro in rust/kernel/device_id.rs.
> modfile_flags = -DKBUILD_MODFILE=$(call stringify,$(modfile))
>
> _c_flags = $(filter-out $(CFLAGS_REMOVE_$(target-stem).o), \
^ permalink raw reply related
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Andrey Grodzovsky @ 2025-10-20 16:45 UTC (permalink / raw)
To: Song Liu
Cc: Petr Mladek, kernel-team@lists.ubuntu.com,
live-patching@vger.kernel.org, Steven Rostedt
In-Reply-To: <CAHzjS_soRQwKKP24DObNKBnOtiNsVZHOM-NnY_34w5GwGhC9rw@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 4982 bytes --]
On 10/20/25 12:03, Song Liu wrote:
> On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
> <andrey.grodzovsky@crowdstrike.com> wrote:
> [...]
>>> If you build the kernel from source code, there are some samples in
>>> samples/livepatch that you can use for testing. PS: You need to enable
>>>
>>> CONFIG_SAMPLE_LIVEPATCH=m
>>>
>>> I hope this helps.
>> Thanks Song, working on repro, kernel rebuilt, test module is loading
>> but, bpftrace is refusing to attach now to fentries/fexits claiming the
>> costum kernel is not supporting it. It did
>> attach in the case of stock AWS kernel i copied the .config from. So
>> just trying to figure out now if some Kcofnig flags are missing or
>> different . Let me know in case you manage to confirm yourself in the
>> meanwhile the fix works for
>> you.
> Yes, it worked in my tests.
>
> [root@(none) /]# kpatch load linux/samples/livepatch/livepatch-sample.ko
> loading patch module: linux/samples/livepatch/livepatch-sample.ko
> [root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show
> {printf("fexit\n");}' &
> [1] 388
> [root@(none) /]# Attached 1 probe
> [root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show
> {printf("fentry\n");}' &
> [2] 397
> [root@(none) /]# Attached 1 probe
>
> [root@(none) /]# cat /proc/cmdline
> this has been live patched
> fentry
> fexit
>
> Thanks,
> Song
>
Verified the failures I observe when trying to attach with BPF trace are
only in presence of patch you provided.
Please see attached dmesg for failures. Initial warning on boot.
Subsequebt warnings and errors at the point i try to run
sudo bpftrace -e "fexit:cmdline_proc_show { printf(\"fexit hit\\n\");
exit(); }"
sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name
resolution
stdin:1:1-25: ERROR: kfunc/kretfunc not available for your kernel version.
ubuntu@ip-10-10-115-238:~/linux-6.8.1$ sudo cat
/sys/kernel/debug/tracing/available_filter_functions | grep
cmdline_proc_show
sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name
resolution
cat: /sys/kernel/debug/tracing/available_filter_functions: No such device
After reboot and before trying to attacg with bpftrace,
/sys/kernel/debug/tracing/available_filter_functions is available and
shows all function.
Using stable kernel from
https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.8.1.tar.gz for build.
FTRACE related KCONFIGs bellow
ubuntu@ip-10-10-115-238:~$ cat /boot/config-6.8.1 | grep TRACE
CONFIG_TASKS_TRACE_RCU=y
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_TRACEPOINTS=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_PM_TRACE=y
CONFIG_PM_TRACE_RTC=y
CONFIG_KPROBES_ON_FTRACE=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_ARCH_CORRECT_STACKTRACE_ON_KRETPROBE=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_TRACE_IRQFLAGS_NMI_SUPPORT=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_RELIABLE_STACKTRACE=y
CONFIG_NETFILTER_XT_TARGET_TRACE=m
# CONFIG_DMA_FENCE_TRACE is not set
CONFIG_SCSI_IPR_TRACE=y
# CONFIG_ATH5K_TRACER is not set
# CONFIG_DRM_I915_LOW_LEVEL_TRACEPOINTS is not set
CONFIG_STM_SOURCE_FTRACE=m
# CONFIG_PSTORE_FTRACE is not set
# CONFIG_STACKTRACE_BUILD_ID is not set
CONFIG_STACKTRACE=y
# CONFIG_RCU_TRACE is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS=y
CONFIG_HAVE_DYNAMIC_FTRACE_NO_PATCHABLE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACE_CLOCK=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_GENERIC_TRACER=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
CONFIG_DYNAMIC_FTRACE_WITH_ARGS=y
CONFIG_STACK_TRACER=y
# CONFIG_IRQSOFF_TRACER is not set
CONFIG_SCHED_TRACER=y
CONFIG_HWLAT_TRACER=y
CONFIG_OSNOISE_TRACER=y
CONFIG_TIMERLAT_TRACER=y
CONFIG_MMIOTRACE=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_TRACER_SNAPSHOT=y
# CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP is not set
CONFIG_BLK_DEV_IO_TRACE=y
# CONFIG_KPROBE_EVENTS_ON_NOTRACE is not set
CONFIG_FTRACE_MCOUNT_RECORD=y
CONFIG_FTRACE_MCOUNT_USE_CC=y
CONFIG_TRACE_EVENT_INJECT=y
# CONFIG_TRACEPOINT_BENCHMARK is not set
# CONFIG_TRACE_EVAL_MAP_FILE is not set
# CONFIG_FTRACE_RECORD_RECURSION is not set
# CONFIG_FTRACE_STARTUP_TEST is not set
CONFIG_FTRACE_SORT_STARTUP_TEST=y
# CONFIG_MMIOTRACE_TEST is not set
# CONFIG_SAMPLE_TRACE_EVENTS is not set
# CONFIG_SAMPLE_TRACE_CUSTOM_EVENTS is not set
CONFIG_SAMPLE_TRACE_PRINTK=m
CONFIG_SAMPLE_FTRACE_DIRECT=m
# CONFIG_SAMPLE_FTRACE_DIRECT_MULTI is not set
# CONFIG_SAMPLE_FTRACE_OPS is not set
CONFIG_SAMPLE_TRACE_ARRAY=m
CONFIG_HAVE_SAMPLE_FTRACE_DIRECT=y
CONFIG_HAVE_SAMPLE_FTRACE_DIRECT_MULTI=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_CC_HAS_SANCOV_TRACE_PC=y
# CONFIG_BACKTRACE_SELF_TEST is not set
Andrey
[-- Attachment #2: dmesg --]
[-- Type: text/plain, Size: 28141 bytes --]
[ 0.958579] NET: Registered PF_INET6 protocol family
[ 0.960794] Segment Routing with IPv6
[ 0.961819] In-situ OAM (IOAM) with IPv6
[ 0.962927] NET: Registered PF_PACKET protocol family
[ 0.964393] Key type dns_resolver registered
[ 0.968676] No MBM correction factor available
[ 0.969933] IPI shorthand broadcast: enabled
[ 0.972554] sched_clock: Marking stable (673007499, 299279483)->(1155822036, -183535054)
[ 0.974892] registered taskstats version 1
[ 0.977043] Loading compiled-in X.509 certificates
[ 0.979081] Loaded X.509 cert 'Build time autogenerated kernel key: 774d7f8f61628ad4e8d329abbb98ef067d7b43a9'
[ 0.985242] Key type .fscrypt registered
[ 0.986488] Key type fscrypt-provisioning registered
[ 0.988673] Key type encrypted registered
[ 0.989923] AppArmor: AppArmor sha256 policy hashing enabled
[ 0.991934] ima: No TPM chip found, activating TPM-bypass!
[ 0.993570] Loading compiled-in module X.509 certificates
[ 0.996021] Loaded X.509 cert 'Build time autogenerated kernel key: 774d7f8f61628ad4e8d329abbb98ef067d7b43a9'
[ 0.999303] ima: Allocated hash algorithm: sha256
[ 1.000769] ima: No architecture policies found
[ 1.002184] evm: Initialising EVM extended attributes:
[ 1.003740] evm: security.selinux
[ 1.004829] evm: security.SMACK64
[ 1.005916] evm: security.SMACK64EXEC
[ 1.007085] evm: security.SMACK64TRANSMUTE
[ 1.008389] evm: security.SMACK64MMAP
[ 1.009567] evm: security.apparmor
[ 1.010663] evm: security.ima
[ 1.011651] evm: security.capability
[ 1.012788] evm: HMAC attrs: 0x1
[ 1.014248] PM: Magic number: 13:837:285
[ 1.019035] ------------[ cut here ]------------
[ 1.020163] WARNING: CPU: 10 PID: 1 at kernel/trace/ftrace.c:2589 ftrace_get_addr_new+0x10c/0x120
[ 1.022417] Modules linked in:
[ 1.023417] CPU: 10 PID: 1 Comm: swapper/0 Not tainted 6.8.1 #1
[ 1.025148] Hardware name: Amazon EC2 c5.4xlarge/, BIOS 1.0 10/16/2017
[ 1.027039] RIP: 0010:ftrace_get_addr_new+0x10c/0x120
[ 1.028559] Code: 00 48 85 c0 74 9d 5b 41 5c 41 5d 5d 31 d2 31 c9 31 f6 31 ff c3 cc cc cc cc 48 8b 3f e8 5d fe ff ff 48 85 c0 0f 85 38 ff ff ff <0f> 0b 48 8b 43 08 e9 0f ff ff ff 66 0f 1f 84 00 00 00 00 00 90 90
[ 1.033947] RSP: 0000:ffffa8bbc0027808 EFLAGS: 00010246
[ 1.035521] RAX: 0000000000000000 RBX: ffff88bf800ec540 RCX: 0000000000000000
[ 1.037587] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 1.039648] RBP: ffffa8bbc0027820 R08: 0000000000000000 R09: 0000000000000000
[ 1.041713] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88bf800ec540
[ 1.043773] R13: 0000000000000001 R14: ffffffffa39aef08 R15: ffffffffa2206cca
[ 1.045836] FS: 0000000000000000(0000) GS:ffff88c665900000(0000) knlGS:0000000000000000
[ 1.048143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.049851] CR2: 0000000000000000 CR3: 000000037203c001 CR4: 00000000007706f0
[ 1.051926] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1.053992] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1.056061] PKRU: 55555554
[ 1.056978] Call Trace:
[ 1.057833] <TASK>
[ 1.058584] ? show_regs+0x6d/0x80
[ 1.059676] ? __warn+0x89/0x150
[ 1.060714] ? ftrace_get_addr_new+0x10c/0x120
[ 1.062073] ? report_bug+0x16a/0x190
[ 1.063219] ? handle_bug+0x51/0xa0
[ 1.064351] ? exc_invalid_op+0x18/0x80
[ 1.065547] ? asm_exc_invalid_op+0x1b/0x20
[ 1.066840] ? ftrace_get_addr_new+0x10c/0x120
[ 1.068207] ? ftrace_get_addr_new+0x103/0x120
[ 1.069576] ftrace_replace_code+0x11d/0x1b0
[ 1.070894] ftrace_modify_all_code+0x15e/0x1a0
[ 1.072317] arch_ftrace_update_code+0x9/0x20
[ 1.073666] ftrace_startup+0xcc/0x180
[ 1.074855] register_ftrace_function_nolock+0x24/0x70
[ 1.076438] register_ftrace_direct+0x2ed/0x370
[ 1.077838] ? 0xffffffffc010ca00
[ 1.078923] ? __pfx___hid_bpf_tail_call+0x10/0x10
[ 1.080404] bpf_trampoline_update+0x469/0x650
[ 1.081775] ? 0xffffffffc010ca00
[ 1.082833] ? 0xffffffffc010ca00
[ 1.083881] __bpf_trampoline_link_prog+0x10d/0x2d0
[ 1.085359] bpf_trampoline_link_prog+0x2b/0x50
[ 1.086700] bpf_tracing_prog_attach+0x3c2/0x5f0
[ 1.088073] bpf_raw_tp_link_attach+0x206/0x2d0
[ 1.089430] __sys_bpf+0x1120/0x2500
[ 1.090531] kern_sys_bpf+0x61/0x1e0
[ 1.091635] ? close_fd+0x58/0x80
[ 1.092664] skel_raw_tracepoint_open.constprop.0+0x43/0x70
[ 1.094297] hid_bpf_preload_skel+0x111/0x180
[ 1.095596] ? __pfx_hid_bpf_init+0x10/0x10
[ 1.096849] hid_bpf_init+0x1e/0xc0
[ 1.097930] do_one_initcall+0x5e/0x310
[ 1.099092] kernel_init_freeable+0x359/0x500
[ 1.100403] ? __pfx_kernel_init+0x10/0x10
[ 1.101628] kernel_init+0x1b/0x200
[ 1.102695] ret_from_fork+0x47/0x70
[ 1.103790] ? __pfx_kernel_init+0x10/0x10
[ 1.105011] ret_from_fork_asm+0x1b/0x30
[ 1.106207] </TASK>
[ 1.106931] ---[ end trace 0000000000000000 ]---
[ 1.109385] RAS: Correctable Errors collector initialized.
[ 1.111113] clk: Disabling unused clocks
[ 1.416156] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
[ 1.418725] md: Waiting for all devices to be available before autodetect
[ 1.419892] md: If you don't use raid, use raid=noautodetect
[ 1.420894] md: Autodetecting RAID arrays.
[ 1.421656] md: autorun ...
[ 1.422215] md: ... autorun DONE.
[ 1.429543] EXT4-fs (nvme0n1p1): mounted filesystem 6387caa1-122f-48bc-b447-9f1386d06e06 ro with ordered data mode. Quota mode: none.
[ 1.431922] VFS: Mounted root (ext4 filesystem) readonly on device 259:1.
[ 1.434051] devtmpfs: mounted
[ 1.435734] Freeing unused decrypted memory: 2028K
[ 1.437643] Freeing unused kernel image (initmem) memory: 4984K
[ 1.439121] Write protecting the kernel read-only data: 34816k
[ 1.440674] Freeing unused kernel image (rodata/data gap) memory: 872K
[ 1.486425] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[ 1.487559] x86/mm: Checking user space page tables
[ 1.531530] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[ 1.532674] Run /sbin/init as init process
[ 1.533460] with arguments:
[ 1.533462] /sbin/init
[ 1.533463] with environment:
[ 1.533464] HOME=/
[ 1.533465] TERM=linux
[ 1.533465] BOOT_IMAGE=/vmlinuz-6.8.1
[ 1.691160] systemd[1]: Inserted module 'autofs4'
[ 1.739692] systemd[1]: systemd 255.4-1ubuntu8 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[ 1.745330] systemd[1]: Detected virtualization amazon.
[ 1.746574] systemd[1]: Detected architecture x86-64.
[ 1.751410] systemd[1]: Hostname set to <ip-10-10-115-238>.
[ 2.144939] systemd[1]: Queued start job for default target graphical.target.
[ 2.169782] systemd[1]: Created slice system-modprobe.slice - Slice /system/modprobe.
[ 2.173549] systemd[1]: Created slice system-serial\x2dgetty.slice - Slice /system/serial-getty.
[ 2.177997] systemd[1]: Created slice system-systemd\x2dfsck.slice - Slice /system/systemd-fsck.
[ 2.181785] systemd[1]: Created slice user.slice - User and Session Slice.
[ 2.184612] systemd[1]: Started ntpsec-systemd-netif.path.
[ 2.187009] systemd[1]: Started systemd-ask-password-wall.path - Forward Password Requests to Wall Directory Watch.
[ 2.191530] systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.
[ 2.196152] systemd[1]: Expecting device dev-disk-by\x2dlabel-BOOT.device - /dev/disk/by-label/BOOT...
[ 2.200018] systemd[1]: Expecting device dev-disk-by\x2dlabel-UEFI.device - /dev/disk/by-label/UEFI...
[ 2.203876] systemd[1]: Expecting device dev-ttyS0.device - /dev/ttyS0...
[ 2.206699] systemd[1]: Reached target integritysetup.target - Local Integrity Protected Volumes.
[ 2.210142] systemd[1]: Reached target slices.target - Slice Units.
[ 2.213182] systemd[1]: Reached target swap.target - Swaps.
[ 2.215621] systemd[1]: Reached target veritysetup.target - Local Verity Protected Volumes.
[ 2.219064] systemd[1]: Listening on dm-event.socket - Device-mapper event daemon FIFOs.
[ 2.222434] systemd[1]: Listening on lvm2-lvmpolld.socket - LVM2 poll daemon socket.
[ 2.226036] systemd[1]: Listening on multipathd.socket - multipathd control socket.
[ 2.242149] systemd[1]: Listening on rpcbind.socket - RPCbind Server Activation Socket.
[ 2.246074] systemd[1]: Listening on syslog.socket - Syslog Socket.
[ 2.249145] systemd[1]: Listening on systemd-fsckd.socket - fsck to fsckd communication Socket.
[ 2.252891] systemd[1]: Listening on systemd-initctl.socket - initctl Compatibility Named Pipe.
[ 2.256663] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log).
[ 2.260465] systemd[1]: Listening on systemd-journald.socket - Journal Socket.
[ 2.263756] systemd[1]: Listening on systemd-networkd.socket - Network Service Netlink Socket.
[ 2.267430] systemd[1]: systemd-pcrextend.socket - TPM2 PCR Extension (Varlink) was skipped because of an unmet condition check (ConditionSecurity=measured-uki).
[ 2.270426] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socket.
[ 2.273935] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket.
[ 2.291382] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[ 2.295171] systemd[1]: Mounting dev-mqueue.mount - POSIX Message Queue File System...
[ 2.299230] systemd[1]: Mounting sys-kernel-debug.mount - Kernel Debug File System...
[ 2.303059] systemd[1]: Mounting sys-kernel-tracing.mount - Kernel Trace File System...
[ 2.306079] systemd[1]: auth-rpcgss-module.service - Kernel Module supporting RPCSEC_GSS was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab).
[ 2.310195] systemd[1]: Starting keyboard-setup.service - Set the console keyboard layout...
[ 2.314585] systemd[1]: Starting kmod-static-nodes.service - Create List of Static Device Nodes...
[ 2.319415] systemd[1]: Starting lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[ 2.324999] systemd[1]: Starting modprobe@configfs.service - Load Kernel Module configfs...
[ 2.328934] systemd[1]: Starting modprobe@dm_mod.service - Load Kernel Module dm_mod...
[ 2.332977] systemd[1]: Starting modprobe@drm.service - Load Kernel Module drm...
[ 2.337481] systemd[1]: Starting modprobe@efi_pstore.service - Load Kernel Module efi_pstore...
[ 2.341518] systemd[1]: Starting modprobe@fuse.service - Load Kernel Module fuse...
[ 2.345226] systemd[1]: Starting modprobe@loop.service - Load Kernel Module loop...
[ 2.348111] systemd[1]: netplan-ovs-cleanup.service - OpenVSwitch configuration for cleanup was skipped because of an unmet condition check (ConditionFileIsExecutable=/usr/bin/ovs-vsctl).
[ 2.351100] pstore: Using crash dump compression: deflate
[ 2.352436] systemd[1]: Starting systemd-fsck-root.service - File System Check on Root Device...
[ 2.356026] pstore: Registered efi_pstore as persistent store backend
[ 2.358847] systemd[1]: Starting systemd-journald.service - Journal Service...
[ 2.364360] systemd[1]: Starting systemd-modules-load.service - Load Kernel Modules...
[ 2.366617] systemd[1]: systemd-pcrmachine.service - TPM2 PCR Machine ID Measurement was skipped because of an unmet condition check (ConditionSecurity=measured-uki).
[ 2.370028] systemd[1]: systemd-tpm2-setup-early.service - TPM2 SRK Setup (Early) was skipped because of an unmet condition check (ConditionSecurity=measured-uki).
[ 2.373850] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...
[ 2.377223] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[ 2.380691] systemd[1]: Mounted dev-mqueue.mount - POSIX Message Queue File System.
[ 2.383791] systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.
[ 2.385162] systemd-journald[255]: Collecting audit messages is disabled.
[ 2.387968] systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.
[ 2.391368] systemd[1]: Finished keyboard-setup.service - Set the console keyboard layout.
[ 2.394953] systemd[1]: Finished kmod-static-nodes.service - Create List of Static Device Nodes.
[ 2.398483] systemd[1]: modprobe@configfs.service: Deactivated successfully.
[ 2.399910] systemd[1]: Finished modprobe@configfs.service - Load Kernel Module configfs.
[ 2.403776] systemd[1]: modprobe@dm_mod.service: Deactivated successfully.
[ 2.405468] systemd[1]: Finished modprobe@dm_mod.service - Load Kernel Module dm_mod.
[ 2.408695] systemd[1]: Finished lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
[ 2.412929] systemd[1]: modprobe@drm.service: Deactivated successfully.
[ 2.414661] systemd[1]: Finished modprobe@drm.service - Load Kernel Module drm.
[ 2.417973] systemd[1]: Started systemd-journald.service - Journal Service.
[ 2.545456] EXT4-fs (nvme0n1p1): re-mounted 6387caa1-122f-48bc-b447-9f1386d06e06 r/w. Quota mode: none.
[ 2.572734] systemd-journald[255]: Received client request to flush runtime journal.
[ 2.594301] systemd-journald[255]: /var/log/journal/ec220e94df30bda465b15689b7378e3a/system.journal: Journal file uses a different sequence number ID, rotating.
[ 2.594307] systemd-journald[255]: Rotating system journal.
[ 2.761565] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
[ 2.786745] ena 0000:00:05.0: ENA device version: 0.10
[ 2.786750] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[ 2.802370] parport_pc 00:03: reported by Plug and Play ACPI
[ 2.841917] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0510000, mac addr 12:67:59:84:01:a5
[ 2.852675] ppdev: user-space parallel port driver
[ 2.853640] ena 0000:00:05.0 ens5: renamed from eth0
[ 2.860372] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer
[ 2.866183] cryptd: max_cpu_qlen set to 1000
[ 2.899090] AVX2 version of gcm_enc/dec engaged.
[ 2.899202] AES CTR mode by8 optimization enabled
[ 2.984184] EXT4-fs (nvme0n1p16): mounted filesystem 9d39f3ff-b465-4c20-9ec1-06182226356c r/w with ordered data mode. Quota mode: none.
[ 3.084656] audit: type=1400 audit(1760973474.672:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="busybox" pid=708 comm="apparmor_parser"
[ 3.084663] audit: type=1400 audit(1760973474.672:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="crun" pid=714 comm="apparmor_parser"
[ 3.084665] audit: type=1400 audit(1760973474.672:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="ch-checkns" pid=710 comm="apparmor_parser"
[ 3.084668] audit: type=1400 audit(1760973474.672:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="devhelp" pid=715 comm="apparmor_parser"
[ 3.084671] audit: type=1400 audit(1760973474.672:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="Discord" pid=703 comm="apparmor_parser"
[ 3.084697] audit: type=1400 audit(1760973474.672:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="brave" pid=706 comm="apparmor_parser"
[ 3.084727] audit: type=1400 audit(1760973474.672:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="vscode" pid=713 comm="apparmor_parser"
[ 3.084763] audit: type=1400 audit(1760973474.672:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="ch-run" pid=711 comm="apparmor_parser"
[ 3.084794] audit: type=1400 audit(1760973474.672:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="element-desktop" pid=716 comm="apparmor_parser"
[ 3.502649] RPC: Registered named UNIX socket transport module.
[ 3.502652] RPC: Registered udp transport module.
[ 3.502653] RPC: Registered tcp transport module.
[ 3.502654] RPC: Registered tcp-with-tls transport module.
[ 3.502655] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 3.674659] input: ImPS/2 Generic Wheel Mouse as /devices/platform/i8042/serio1/input/input4
[ 4.013037] 8021q: 802.1Q VLAN Support v1.8
[ 4.140597] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[ 4.140856] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[ 4.140992] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600'
[ 4.141693] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[ 4.141696] cfg80211: failed to load regulatory.db
[ 9.730872] kauditd_printk_skb: 94 callbacks suppressed
[ 9.730875] audit: type=1400 audit(1760973481.318:105): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="rsyslogd" pid=1152 comm="apparmor_parser"
[ 9.906210] NET: Registered PF_QIPCRTR protocol family
[ 10.657618] audit: type=1400 audit(1760973482.245:106): apparmor="STATUS" operation="profile_load" profile="unconfined" name="docker-default" pid=1353 comm="apparmor_parser"
[ 10.743033] evm: overlay not supported
[ 10.843306] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 10.852812] Bridge firewalling registered
[ 11.008539] Initializing XFRM netlink socket
[ 49.644664] systemd-journald[255]: /var/log/journal/ec220e94df30bda465b15689b7378e3a/user-1000.journal: Journal file uses a different sequence number ID, rotating.
[ 1237.397558] ------------[ cut here ]------------
[ 1237.397562] WARNING: CPU: 12 PID: 1988 at arch/x86/kernel/ftrace.c:99 ftrace_verify_code+0x4a/0x90
[ 1237.397569] Modules linked in: xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables libcrc32c br_netfilter bridge qrtr overlay cfg80211 8021q garp mrp stp llc sunrpc binfmt_misc intel_rapl_msr intel_rapl_common nls_iso8859_1 intel_uncore_frequency_common isst_if_common nfit crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd rapl ppdev parport_pc psmouse ena parport i2c_piix4 input_leds mac_hid serio_raw dm_multipath msr efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4
[ 1237.397610] CPU: 12 PID: 1988 Comm: bpftrace Tainted: G W 6.8.1 #1
[ 1237.397613] Hardware name: Amazon EC2 c5.4xlarge/, BIOS 1.0 10/16/2017
[ 1237.397614] RIP: 0010:ftrace_verify_code+0x4a/0x90
[ 1237.397616] Code: 00 48 89 45 f0 31 c0 c7 45 eb 00 00 00 00 c6 45 ef 00 e8 09 9c 2f 00 48 85 c0 75 41 8b 03 39 45 eb 74 2d 48 89 1d 26 01 be 02 <0f> 0b b8 ea ff ff ff 48 8b 55 f0 65 48 2b 14 25 28 00 00 00 75 26
[ 1237.397618] RSP: 0018:ffffa8bbc1d7fbd8 EFLAGS: 00010286
[ 1237.397621] RAX: 000000001d8d3be8 RBX: ffffffffa39aef08 RCX: 0000000000000000
[ 1237.397622] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 1237.397623] RBP: ffffa8bbc1d7fbf0 R08: 0000000000000000 R09: 0000000000000000
[ 1237.397625] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa39aef08
[ 1237.397626] R13: 0000000000000001 R14: ffff88bf80114d20 R15: ffffffffa39aef08
[ 1237.397627] FS: 00007fa3bcbd0780(0000) GS:ffff88c665a00000(0000) knlGS:0000000000000000
[ 1237.397629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1237.397630] CR2: 00007ffe71155124 CR3: 0000000117032001 CR4: 00000000007706f0
[ 1237.397636] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1237.397637] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1237.397639] PKRU: 55555554
[ 1237.397640] Call Trace:
[ 1237.397641] <TASK>
[ 1237.397643] ? show_regs+0x6d/0x80
[ 1237.397649] ? __warn+0x89/0x150
[ 1237.397653] ? ftrace_verify_code+0x4a/0x90
[ 1237.397655] ? report_bug+0x16a/0x190
[ 1237.397659] ? handle_bug+0x51/0xa0
[ 1237.397662] ? exc_invalid_op+0x18/0x80
[ 1237.397664] ? asm_exc_invalid_op+0x1b/0x20
[ 1237.397668] ? ftrace_verify_code+0x4a/0x90
[ 1237.397669] ? ftrace_verify_code+0x37/0x90
[ 1237.397670] ftrace_replace_code+0x71/0x1b0
[ 1237.397673] ftrace_modify_all_code+0x125/0x1a0
[ 1237.397675] ? 0xffffffffc01206c0
[ 1237.397684] arch_ftrace_update_code+0x9/0x20
[ 1237.397686] ftrace_shutdown.part.0+0x114/0x1e0
[ 1237.397688] ? 0xffffffffc01206c0
[ 1237.397690] ? 0xffffffffc01206c0
[ 1237.397691] unregister_ftrace_function+0x47/0x170
[ 1237.397693] ? 0xffffffffc01206c0
[ 1237.397694] ? 0xffffffffc01206c0
[ 1237.397696] unregister_ftrace_direct+0x60/0x1e0
[ 1237.397697] ? kmalloc_trace+0x136/0x310
[ 1237.397700] ? bpf_trampoline_update+0x68/0x650
[ 1237.397703] ? 0xffffffffc01206c0
[ 1237.397704] ? __pfx_sched_fork+0x10/0x10
[ 1237.397707] bpf_trampoline_update+0x4f7/0x650
[ 1237.397709] bpf_trampoline_unlink_prog+0x9d/0x130
[ 1237.397711] bpf_tracing_link_release+0x16/0x50
[ 1237.397715] bpf_link_free+0x55/0x80
[ 1237.397717] bpf_link_release+0x26/0x40
[ 1237.397719] __fput+0xa1/0x2e0
[ 1237.397723] __fput_sync+0x1c/0x30
[ 1237.397725] __x64_sys_close+0x3e/0x90
[ 1237.397728] do_syscall_64+0x85/0x180
[ 1237.397732] ? syscall_exit_to_user_mode+0x86/0x260
[ 1237.397736] ? do_syscall_64+0x92/0x180
[ 1237.397738] ? do_syscall_64+0x92/0x180
[ 1237.397740] ? do_syscall_64+0x92/0x180
[ 1237.397743] ? irqentry_exit+0x21/0x40
[ 1237.397745] ? exc_page_fault+0x94/0x190
[ 1237.397748] entry_SYSCALL_64_after_hwframe+0x6e/0x76
[ 1237.397752] RIP: 0033:0x7fa3b4f166f4
[ 1237.397755] Code: 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 80 3d 55 49 0f 00 00 74 13 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 3c c3 0f 1f 00 55 48 89 e5 48 83 ec 10 89 7d
[ 1237.397756] RSP: 002b:00007ffe7113d028 EFLAGS: 00000202 ORIG_RAX: 0000000000000003
[ 1237.397758] RAX: ffffffffffffffda RBX: 0000555e10456800 RCX: 00007fa3b4f166f4
[ 1237.397760] RDX: 0000000000000010 RSI: 00007ffe7113cf60 RDI: 0000000000000004
[ 1237.397761] RBP: 00007ffe7113d080 R08: 0000555e0f7fb86c R09: 0000000000000000
[ 1237.397762] R10: 00007ffe7113ba00 R11: 0000000000000202 R12: 0000000000000004
[ 1237.397763] R13: 0000000000000003 R14: 0000555e0f7faeac R15: 00007ffe7113d750
[ 1237.397765] </TASK>
[ 1237.397766] ---[ end trace 0000000000000000 ]---
[ 1237.397767] ------------[ ftrace bug ]------------
[ 1237.397768] ftrace failed to modify
[ 1237.397769] [<ffffffffa0f47980>] sched_fork+0x0/0x260
[ 1237.397771] actual: e8:db:38:f6:ff
[ 1237.397773] expected: e8:3b:8d:1d:1f
[ 1237.397775] Setting ftrace call site to NOP
[ 1237.397775] ftrace record flags: e8980000
[ 1237.397776] (0) R tramp: ERROR!
expected tramp: ffffffffc01206c0
[ 1237.397787] ------------[ cut here ]------------
[ 1237.397788] WARNING: CPU: 12 PID: 1988 at kernel/trace/ftrace.c:2179 ftrace_bug+0x10d/0x2d0
[ 1237.397790] Modules linked in: xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables libcrc32c br_netfilter bridge qrtr overlay cfg80211 8021q garp mrp stp llc sunrpc binfmt_misc intel_rapl_msr intel_rapl_common nls_iso8859_1 intel_uncore_frequency_common isst_if_common nfit crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd rapl ppdev parport_pc psmouse ena parport i2c_piix4 input_leds mac_hid serio_raw dm_multipath msr efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4
[ 1237.397819] CPU: 12 PID: 1988 Comm: bpftrace Tainted: G W 6.8.1 #1
[ 1237.397821] Hardware name: Amazon EC2 c5.4xlarge/, BIOS 1.0 10/16/2017
[ 1237.397821] RIP: 0010:ftrace_bug+0x10d/0x2d0
[ 1237.397823] Code: 07 00 e8 46 df f2 ff 41 f6 44 24 0b 08 0f 85 18 01 00 00 4c 89 e7 e8 a2 fc ff ff 48 c7 c7 7d 51 77 a2 48 89 c6 e8 23 df f2 ff <0f> 0b c7 05 af cf 3d 02 01 00 00 00 5b c7 05 b4 cf 3d 02 00 00 00
[ 1237.397824] RSP: 0018:ffffa8bbc1d7fbd8 EFLAGS: 00010246
[ 1237.397826] RAX: 0000000000000022 RBX: ffffffffa344a640 RCX: 0000000000000000
[ 1237.397827] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 1237.397828] RBP: ffffa8bbc1d7fbf0 R08: 0000000000000000 R09: 0000000000000000
[ 1237.397829] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88bf80114d20
[ 1237.397830] R13: ffffffffa0f47980 R14: ffff88bf80114d20 R15: ffffffffa39aef08
[ 1237.397831] FS: 00007fa3bcbd0780(0000) GS:ffff88c665a00000(0000) knlGS:0000000000000000
[ 1237.397833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1237.397834] CR2: 00007ffe71155124 CR3: 0000000117032001 CR4: 00000000007706f0
[ 1237.397839] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1237.397840] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1237.397841] PKRU: 55555554
[ 1237.397842] Call Trace:
[ 1237.397843] <TASK>
[ 1237.397844] ? show_regs+0x6d/0x80
[ 1237.397846] ? __warn+0x89/0x150
[ 1237.397849] ? ftrace_bug+0x10d/0x2d0
[ 1237.397850] ? report_bug+0x16a/0x190
[ 1237.397853] ? handle_bug+0x51/0xa0
[ 1237.397854] ? exc_invalid_op+0x18/0x80
[ 1237.397856] ? asm_exc_invalid_op+0x1b/0x20
[ 1237.397858] ? __pfx_sched_fork+0x10/0x10
[ 1237.397859] ? ftrace_bug+0x10d/0x2d0
[ 1237.397861] ? ftrace_bug+0x10d/0x2d0
[ 1237.397863] ftrace_replace_code+0xc7/0x1b0
[ 1237.397864] ftrace_modify_all_code+0x125/0x1a0
[ 1237.397866] ? 0xffffffffc01206c0
[ 1237.397868] arch_ftrace_update_code+0x9/0x20
[ 1237.397870] ftrace_shutdown.part.0+0x114/0x1e0
[ 1237.397871] ? 0xffffffffc01206c0
[ 1237.397873] ? 0xffffffffc01206c0
[ 1237.397874] unregister_ftrace_function+0x47/0x170
[ 1237.397876] ? 0xffffffffc01206c0
[ 1237.397877] ? 0xffffffffc01206c0
[ 1237.397878] unregister_ftrace_direct+0x60/0x1e0
[ 1237.397880] ? kmalloc_trace+0x136/0x310
[ 1237.397882] ? bpf_trampoline_update+0x68/0x650
[ 1237.397884] ? 0xffffffffc01206c0
[ 1237.397885] ? __pfx_sched_fork+0x10/0x10
[ 1237.397886] bpf_trampoline_update+0x4f7/0x650
[ 1237.397889] bpf_trampoline_unlink_prog+0x9d/0x130
[ 1237.397891] bpf_tracing_link_release+0x16/0x50
[ 1237.397893] bpf_link_free+0x55/0x80
[ 1237.397895] bpf_link_release+0x26/0x40
[ 1237.397897] __fput+0xa1/0x2e0
[ 1237.397899] __fput_sync+0x1c/0x30
[ 1237.397901] __x64_sys_close+0x3e/0x90
[ 1237.397903] do_syscall_64+0x85/0x180
[ 1237.397906] ? syscall_exit_to_user_mode+0x86/0x260
[ 1237.397908] ? do_syscall_64+0x92/0x180
[ 1237.397911] ? do_syscall_64+0x92/0x180
[ 1237.397913] ? do_syscall_64+0x92/0x180
[ 1237.397915] ? irqentry_exit+0x21/0x40
[ 1237.397917] ? exc_page_fault+0x94/0x190
[ 1237.397920] entry_SYSCALL_64_after_hwframe+0x6e/0x76
[ 1237.397922] RIP: 0033:0x7fa3b4f166f4
[ 1237.397924] Code: 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 80 3d 55 49 0f 00 00 74 13 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 3c c3 0f 1f 00 55 48 89 e5 48 83 ec 10 89 7d
[ 1237.397925] RSP: 002b:00007ffe7113d028 EFLAGS: 00000202 ORIG_RAX: 0000000000000003
[ 1237.397927] RAX: ffffffffffffffda RBX: 0000555e10456800 RCX: 00007fa3b4f166f4
[ 1237.397928] RDX: 0000000000000010 RSI: 00007ffe7113cf60 RDI: 0000000000000004
[ 1237.397929] RBP: 00007ffe7113d080 R08: 0000555e0f7fb86c R09: 0000000000000000
[ 1237.397930] R10: 00007ffe7113ba00 R11: 0000000000000202 R12: 0000000000000004
[ 1237.397931] R13: 0000000000000003 R14: 0000555e0f7faeac R15: 00007ffe7113d750
[ 1237.397933] </TASK>
[ 1237.397934] ---[ end trace 0000000000000000 ]---
^ permalink raw reply
* Re: [PATCH v4 08/63] kbuild: Remove 'kmod_' prefix from __KBUILD_MODNAME
From: Josh Poimboeuf @ 2025-10-20 17:22 UTC (permalink / raw)
To: Alexander Stein
Cc: x86, linux-kernel, Petr Mladek, Miroslav Benes, Joe Lawrence,
live-patching, Song Liu, laokz, Jiri Kosina,
Marcos Paulo de Souza, Weinan Liu, Fazla Mehrab, Chen Zhongjin,
Puranjay Mohan, Dylan Hatch, Peter Zijlstra, Masahiro Yamada,
Marek Szyprowski, Mark Brown, Cosmin Tanislav
In-Reply-To: <5936475.DvuYhMxLoT@steina-w>
On Mon, Oct 20, 2025 at 02:20:35PM +0200, Alexander Stein wrote:
> Hi,
>
> Am Mittwoch, 17. September 2025, 18:03:16 CEST schrieb Josh Poimboeuf:
> > In preparation for the objtool klp diff subcommand, remove the arbitrary
> > 'kmod_' prefix from __KBUILD_MODNAME and instead add it explicitly in
> > the __initcall_id() macro.
> >
> > This change supports the standardization of "unique" symbol naming by
> > ensuring the non-unique portion of the name comes before the unique
> > part. That will enable objtool to properly correlate symbols across
> > builds.
> >
> > Cc: Masahiro Yamada <masahiroy@kernel.org>
> > Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
>
> Starting with this commit 6717e8f91db71 ("kbuild: Remove 'kmod_' prefix
> from __KBUILD_MODNAME") in next-20251020 I don't get any
> module aliases anymore.
> modinfo spi-fsl-dspi.ko returns:
> > filename: /work/repo/linux/build_arm64/drivers/spi/spi-fsl-dspi.ko
> > alias: platform:fsl-dspi
> > license: GPL
> > description: Freescale DSPI Controller Driver
> > depends:
> > intree: Y
> > name: spi_fsl_dspi
> > vermagic: 6.18.0-rc1+ SMP preempt mod_unload modversions aarch64
>
> but it should be like this:
> > filename: /work/repo/linux/build_arm64/drivers/spi/spi-fsl-dspi.ko
> > alias: platform:fsl-dspi
> > license: GPL
> > description: Freescale DSPI Controller Driver
> > alias: of:N*T*Cnxp,s32g2-dspiC*
Thanks, this patch broke the MODULE_DEVICE_TABLE() macro, as it no
longer produces the format expected by scripts/mod/file2alias.c.
I didn't see this in x86 testing since it doesn't have device tree.
I will post the following fix shortly:
diff --git a/include/linux/module.h b/include/linux/module.h
index e135cc79aceea..d80c3ea574726 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -251,10 +251,11 @@ struct module_kobject *lookup_or_create_module_kobject(const char *name);
*/
#define __mod_device_table(type, name) \
__PASTE(__mod_device_table__, \
+ __PASTE(kmod_, \
__PASTE(__KBUILD_MODNAME, \
__PASTE(__, \
__PASTE(type, \
- __PASTE(__, name)))))
+ __PASTE(__, name))))))
/* Creates an alias so file2alias.c can find device table. */
#define MODULE_DEVICE_TABLE(type, name) \
^ permalink raw reply related
* [PATCH] module: Fix device table module aliases
From: Josh Poimboeuf @ 2025-10-20 17:53 UTC (permalink / raw)
To: x86
Cc: linux-kernel, Petr Mladek, Miroslav Benes, Joe Lawrence,
live-patching, Song Liu, laokz, Jiri Kosina,
Marcos Paulo de Souza, Weinan Liu, Fazla Mehrab, Chen Zhongjin,
Puranjay Mohan, Dylan Hatch, Peter Zijlstra, Alexander Stein,
Marek Szyprowski, Mark Brown, Cosmin Tanislav
Commit 6717e8f91db7 ("kbuild: Remove 'kmod_' prefix from
__KBUILD_MODNAME") inadvertently broke module alias generation for
modules which rely on MODULE_DEVICE_TABLE().
It removed the "kmod_" prefix from __KBUILD_MODNAME, which caused
MODULE_DEVICE_TABLE() to generate a symbol name which no longer matched
the format expected by handle_moddevtable() in scripts/mod/file2alias.c.
As a result, modpost failed to find the device tables, leading to
missing module aliases.
Fix this by explicitly adding the "kmod_" string within the
MODULE_DEVICE_TABLE() macro itself, restoring the symbol name to the
format expected by file2alias.c.
Fixes: 6717e8f91db7 ("kbuild: Remove 'kmod_' prefix from __KBUILD_MODNAME")
Reported-by: Alexander Stein <alexander.stein@ew.tq-group.com>
Reported-by: Marek Szyprowski <m.szyprowski@samsung.com>
Reported-by: Mark Brown <broonie@kernel.org>
Reported-by: Cosmin Tanislav <demonsingur@gmail.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
include/linux/module.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/include/linux/module.h b/include/linux/module.h
index e135cc79aceea..d80c3ea574726 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -251,10 +251,11 @@ struct module_kobject *lookup_or_create_module_kobject(const char *name);
*/
#define __mod_device_table(type, name) \
__PASTE(__mod_device_table__, \
+ __PASTE(kmod_, \
__PASTE(__KBUILD_MODNAME, \
__PASTE(__, \
__PASTE(type, \
- __PASTE(__, name)))))
+ __PASTE(__, name))))))
/* Creates an alias so file2alias.c can find device table. */
#define MODULE_DEVICE_TABLE(type, name) \
--
2.51.0
^ permalink raw reply related
* Re: [PATCH] module: Fix device table module aliases
From: Cosmin Tanislav @ 2025-10-20 18:07 UTC (permalink / raw)
To: Josh Poimboeuf, x86
Cc: linux-kernel, Petr Mladek, Miroslav Benes, Joe Lawrence,
live-patching, Song Liu, laokz, Jiri Kosina,
Marcos Paulo de Souza, Weinan Liu, Fazla Mehrab, Chen Zhongjin,
Puranjay Mohan, Dylan Hatch, Peter Zijlstra, Alexander Stein,
Marek Szyprowski, Mark Brown
In-Reply-To: <e52ee3edf32874da645a9e037a7d77c69893a22a.1760982784.git.jpoimboe@kernel.org>
On 10/20/25 8:53 PM, Josh Poimboeuf wrote:
> Commit 6717e8f91db7 ("kbuild: Remove 'kmod_' prefix from
> __KBUILD_MODNAME") inadvertently broke module alias generation for
> modules which rely on MODULE_DEVICE_TABLE().
>
> It removed the "kmod_" prefix from __KBUILD_MODNAME, which caused
> MODULE_DEVICE_TABLE() to generate a symbol name which no longer matched
> the format expected by handle_moddevtable() in scripts/mod/file2alias.c.
>
> As a result, modpost failed to find the device tables, leading to
> missing module aliases.
>
> Fix this by explicitly adding the "kmod_" string within the
> MODULE_DEVICE_TABLE() macro itself, restoring the symbol name to the
> format expected by file2alias.c.
>
> Fixes: 6717e8f91db7 ("kbuild: Remove 'kmod_' prefix from __KBUILD_MODNAME")
> Reported-by: Alexander Stein <alexander.stein@ew.tq-group.com>
> Reported-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Reported-by: Mark Brown <broonie@kernel.org>
> Reported-by: Cosmin Tanislav <demonsingur@gmail.com>
> Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Tested-by: Cosmin Tanislav <demonsingur@gmail.com>
> ---
> include/linux/module.h | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/include/linux/module.h b/include/linux/module.h
> index e135cc79aceea..d80c3ea574726 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -251,10 +251,11 @@ struct module_kobject *lookup_or_create_module_kobject(const char *name);
> */
> #define __mod_device_table(type, name) \
> __PASTE(__mod_device_table__, \
> + __PASTE(kmod_, \
> __PASTE(__KBUILD_MODNAME, \
> __PASTE(__, \
> __PASTE(type, \
> - __PASTE(__, name)))))
> + __PASTE(__, name))))))
>
> /* Creates an alias so file2alias.c can find device table. */
> #define MODULE_DEVICE_TABLE(type, name) \
^ permalink raw reply
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Song Liu @ 2025-10-20 18:53 UTC (permalink / raw)
To: Andrey Grodzovsky
Cc: Song Liu, Petr Mladek, kernel-team@lists.ubuntu.com,
live-patching@vger.kernel.org, Steven Rostedt
In-Reply-To: <5477a73a-1dce-4b9e-b389-e757ef5536c4@crowdstrike.com>
On Mon, Oct 20, 2025 at 9:45 AM Andrey Grodzovsky
<andrey.grodzovsky@crowdstrike.com> wrote:
>
> On 10/20/25 12:03, Song Liu wrote:
> > On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
> > <andrey.grodzovsky@crowdstrike.com> wrote:
> > [...]
> >>> If you build the kernel from source code, there are some samples in
> >>> samples/livepatch that you can use for testing. PS: You need to enable
> >>>
> >>> CONFIG_SAMPLE_LIVEPATCH=m
> >>>
> >>> I hope this helps.
> >> Thanks Song, working on repro, kernel rebuilt, test module is loading
> >> but, bpftrace is refusing to attach now to fentries/fexits claiming the
> >> costum kernel is not supporting it. It did
> >> attach in the case of stock AWS kernel i copied the .config from. So
> >> just trying to figure out now if some Kcofnig flags are missing or
> >> different . Let me know in case you manage to confirm yourself in the
> >> meanwhile the fix works for
> >> you.
> > Yes, it worked in my tests.
> >
> > [root@(none) /]# kpatch load linux/samples/livepatch/livepatch-sample.ko
> > loading patch module: linux/samples/livepatch/livepatch-sample.ko
> > [root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show
> > {printf("fexit\n");}' &
> > [1] 388
> > [root@(none) /]# Attached 1 probe
> > [root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show
> > {printf("fentry\n");}' &
> > [2] 397
> > [root@(none) /]# Attached 1 probe
> >
> > [root@(none) /]# cat /proc/cmdline
> > this has been live patched
> > fentry
> > fexit
> >
> > Thanks,
> > Song
> >
>
> Verified the failures I observe when trying to attach with BPF trace are
> only in presence of patch you provided.
> Please see attached dmesg for failures. Initial warning on boot.
> Subsequebt warnings and errors at the point i try to run
> sudo bpftrace -e "fexit:cmdline_proc_show { printf(\"fexit hit\\n\");
> exit(); }"
>
> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name
> resolution
> stdin:1:1-25: ERROR: kfunc/kretfunc not available for your kernel version.
>
> ubuntu@ip-10-10-115-238:~/linux-6.8.1$ sudo cat
> /sys/kernel/debug/tracing/available_filter_functions | grep
> cmdline_proc_show
> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name
> resolution
> cat: /sys/kernel/debug/tracing/available_filter_functions: No such device
>
> After reboot and before trying to attacg with bpftrace,
> /sys/kernel/debug/tracing/available_filter_functions is available and
> shows all function.
>
> Using stable kernel from
> https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.8.1.tar.gz for build.
> FTRACE related KCONFIGs bellow
I can see the similar issue with the upstream kernel. I was testing on
stable 6.17 before just know because of another issue with upstream
kernel, and somehow 6.17 kernel doesn't seem to have the issue.
To fix this, I think we should land a fix similar to the earlier diff:
diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
index 42bd2ba68a82..8f320df0ac52 100644
--- i/kernel/trace/ftrace.c
+++ w/kernel/trace/ftrace.c
@@ -6049,6 +6049,9 @@ int register_ftrace_direct(struct ftrace_ops
*ops, unsigned long addr)
err = register_ftrace_function_nolock(ops);
+ if (err)
+ remove_direct_functions_hash(hash, addr);
+
out_unlock:
mutex_unlock(&direct_mutex);
Steven,
Does this change look good to you?
Thanks,
Song
^ permalink raw reply related
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Andrey Grodzovsky @ 2025-10-20 19:10 UTC (permalink / raw)
To: Song Liu
Cc: Petr Mladek, kernel-team@lists.ubuntu.com,
live-patching@vger.kernel.org, Steven Rostedt
In-Reply-To: <CAHzjS_tuotYQQ0HmncVp=oFOfcyxmYqCds0MDBMOr5FC5KzhSA@mail.gmail.com>
On 10/20/25 14:53, Song Liu wrote:
> On Mon, Oct 20, 2025 at 9:45 AM Andrey Grodzovsky
> <andrey.grodzovsky@crowdstrike.com> wrote:
>> On 10/20/25 12:03, Song Liu wrote:
>>> On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
>>> <andrey.grodzovsky@crowdstrike.com> wrote:
>>> [...]
>>>>> If you build the kernel from source code, there are some samples in
>>>>> samples/livepatch that you can use for testing. PS: You need to enable
>>>>>
>>>>> CONFIG_SAMPLE_LIVEPATCH=m
>>>>>
>>>>> I hope this helps.
>>>> Thanks Song, working on repro, kernel rebuilt, test module is loading
>>>> but, bpftrace is refusing to attach now to fentries/fexits claiming the
>>>> costum kernel is not supporting it. It did
>>>> attach in the case of stock AWS kernel i copied the .config from. So
>>>> just trying to figure out now if some Kcofnig flags are missing or
>>>> different . Let me know in case you manage to confirm yourself in the
>>>> meanwhile the fix works for
>>>> you.
>>> Yes, it worked in my tests.
>>>
>>> [root@(none) /]# kpatch load linux/samples/livepatch/livepatch-sample.ko
>>> loading patch module: linux/samples/livepatch/livepatch-sample.ko
>>> [root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show
>>> {printf("fexit\n");}' &
>>> [1] 388
>>> [root@(none) /]# Attached 1 probe
>>> [root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show
>>> {printf("fentry\n");}' &
>>> [2] 397
>>> [root@(none) /]# Attached 1 probe
>>>
>>> [root@(none) /]# cat /proc/cmdline
>>> this has been live patched
>>> fentry
>>> fexit
>>>
>>> Thanks,
>>> Song
>>>
>> Verified the failures I observe when trying to attach with BPF trace are
>> only in presence of patch you provided.
>> Please see attached dmesg for failures. Initial warning on boot.
>> Subsequebt warnings and errors at the point i try to run
>> sudo bpftrace -e "fexit:cmdline_proc_show { printf(\"fexit hit\\n\");
>> exit(); }"
>>
>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name
>> resolution
>> stdin:1:1-25: ERROR: kfunc/kretfunc not available for your kernel version.
>>
>> ubuntu@ip-10-10-115-238:~/linux-6.8.1$ sudo cat
>> /sys/kernel/debug/tracing/available_filter_functions | grep
>> cmdline_proc_show
>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name
>> resolution
>> cat: /sys/kernel/debug/tracing/available_filter_functions: No such device
>>
>> After reboot and before trying to attacg with bpftrace,
>> /sys/kernel/debug/tracing/available_filter_functions is available and
>> shows all function.
>>
>> Using stable kernel from
>> https://urldefense.com/v3/__https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.8.1.tar.gz__;!!BmdzS3_lV9HdKG8!1ZJe4jY49_xIzp4h4i4AbqpkLKoAqrXLFX2wDxhoSUDg2kSeTjy3COy9MngNDRlZhJ1oUKgf1yPqmnTY9-Y50TkA$ for build.
>> FTRACE related KCONFIGs bellow
> I can see the similar issue with the upstream kernel. I was testing on
> stable 6.17 before just know because of another issue with upstream
> kernel, and somehow 6.17 kernel doesn't seem to have the issue.
>
> To fix this, I think we should land a fix similar to the earlier diff:
>
> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
> index 42bd2ba68a82..8f320df0ac52 100644
> --- i/kernel/trace/ftrace.c
> +++ w/kernel/trace/ftrace.c
> @@ -6049,6 +6049,9 @@ int register_ftrace_direct(struct ftrace_ops
> *ops, unsigned long addr)
>
> err = register_ftrace_function_nolock(ops);
>
> + if (err)
> + remove_direct_functions_hash(hash, addr);
> +
> out_unlock:
> mutex_unlock(&direct_mutex);
>
>
> Steven,
>
> Does this change look good to you?
>
>
Seems reasonable to me, we are simply cleaning the entry on failure so
we don't encounter it late anymore.
So I will apply this patch ONLY and retest - correct ?
Another question - it seems you found where it broke ? I saw 'Cc:
stable@vger.kernel.org # v6.6+' in your prev. patch.'
If so , can you please point me to the offending patch so I add this to
my records of my discovery work of bpf coexistence
livepatching ?
Thanks,
Andrey
>
> Thanks,
> Song
^ permalink raw reply
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Song Liu @ 2025-10-20 19:44 UTC (permalink / raw)
To: Andrey Grodzovsky
Cc: Song Liu, Petr Mladek, kernel-team@lists.ubuntu.com,
live-patching@vger.kernel.org, Steven Rostedt
In-Reply-To: <7e6886ab-b168-422e-9adf-8297b88643d1@crowdstrike.com>
On Mon, Oct 20, 2025 at 12:10 PM Andrey Grodzovsky
<andrey.grodzovsky@crowdstrike.com> wrote:
[...]
> Seems reasonable to me, we are simply cleaning the entry on failure so
> we don't encounter it late anymore.
> So I will apply this patch ONLY and retest - correct ?
Right, we only need this small change.
> Another question - it seems you found where it broke ? I saw 'Cc:
> stable@vger.kernel.org # v6.6+' in your prev. patch.'
> If so , can you please point me to the offending patch so I add this to
> my records of my discovery work of bpf coexistence
> livepatching ?
The commit that causing the issue was in the Fixes tag:
Fixes: d05cb470663a ("ftrace: Fix modification of direct_function hash
while in use")
Thanks,
Song
^ permalink raw reply
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Andrey Grodzovsky @ 2025-10-20 19:53 UTC (permalink / raw)
To: Song Liu
Cc: Petr Mladek, kernel-team@lists.ubuntu.com,
live-patching@vger.kernel.org, Steven Rostedt
In-Reply-To: <7e6886ab-b168-422e-9adf-8297b88643d1@crowdstrike.com>
On 10/20/25 15:10, Andrey Grodzovsky wrote:
> On 10/20/25 14:53, Song Liu wrote:
>> On Mon, Oct 20, 2025 at 9:45 AM Andrey Grodzovsky
>> <andrey.grodzovsky@crowdstrike.com> wrote:
>>> On 10/20/25 12:03, Song Liu wrote:
>>>> On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
>>>> <andrey.grodzovsky@crowdstrike.com> wrote:
>>>> [...]
>>>>>> If you build the kernel from source code, there are some samples in
>>>>>> samples/livepatch that you can use for testing. PS: You need to
>>>>>> enable
>>>>>>
>>>>>> CONFIG_SAMPLE_LIVEPATCH=m
>>>>>>
>>>>>> I hope this helps.
>>>>> Thanks Song, working on repro, kernel rebuilt, test module is loading
>>>>> but, bpftrace is refusing to attach now to fentries/fexits
>>>>> claiming the
>>>>> costum kernel is not supporting it. It did
>>>>> attach in the case of stock AWS kernel i copied the .config from. So
>>>>> just trying to figure out now if some Kcofnig flags are missing or
>>>>> different . Let me know in case you manage to confirm yourself in the
>>>>> meanwhile the fix works for
>>>>> you.
>>>> Yes, it worked in my tests.
>>>>
>>>> [root@(none) /]# kpatch load
>>>> linux/samples/livepatch/livepatch-sample.ko
>>>> loading patch module: linux/samples/livepatch/livepatch-sample.ko
>>>> [root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show
>>>> {printf("fexit\n");}' &
>>>> [1] 388
>>>> [root@(none) /]# Attached 1 probe
>>>> [root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show
>>>> {printf("fentry\n");}' &
>>>> [2] 397
>>>> [root@(none) /]# Attached 1 probe
>>>>
>>>> [root@(none) /]# cat /proc/cmdline
>>>> this has been live patched
>>>> fentry
>>>> fexit
>>>>
>>>> Thanks,
>>>> Song
>>>>
>>> Verified the failures I observe when trying to attach with BPF trace
>>> are
>>> only in presence of patch you provided.
>>> Please see attached dmesg for failures. Initial warning on boot.
>>> Subsequebt warnings and errors at the point i try to run
>>> sudo bpftrace -e "fexit:cmdline_proc_show { printf(\"fexit hit\\n\");
>>> exit(); }"
>>>
>>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in
>>> name
>>> resolution
>>> stdin:1:1-25: ERROR: kfunc/kretfunc not available for your kernel
>>> version.
>>>
>>> ubuntu@ip-10-10-115-238:~/linux-6.8.1$ sudo cat
>>> /sys/kernel/debug/tracing/available_filter_functions | grep
>>> cmdline_proc_show
>>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in
>>> name
>>> resolution
>>> cat: /sys/kernel/debug/tracing/available_filter_functions: No such
>>> device
>>>
>>> After reboot and before trying to attacg with bpftrace,
>>> /sys/kernel/debug/tracing/available_filter_functions is available and
>>> shows all function.
>>>
>>> Using stable kernel from
>>> https://urldefense.com/v3/__https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.8.1.tar.gz__;!!BmdzS3_lV9HdKG8!1ZJe4jY49_xIzp4h4i4AbqpkLKoAqrXLFX2wDxhoSUDg2kSeTjy3COy9MngNDRlZhJ1oUKgf1yPqmnTY9-Y50TkA$
>>> for build.
>>> FTRACE related KCONFIGs bellow
>> I can see the similar issue with the upstream kernel. I was testing on
>> stable 6.17 before just know because of another issue with upstream
>> kernel, and somehow 6.17 kernel doesn't seem to have the issue.
>>
>> To fix this, I think we should land a fix similar to the earlier diff:
>>
>> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
>> index 42bd2ba68a82..8f320df0ac52 100644
>> --- i/kernel/trace/ftrace.c
>> +++ w/kernel/trace/ftrace.c
>> @@ -6049,6 +6049,9 @@ int register_ftrace_direct(struct ftrace_ops
>> *ops, unsigned long addr)
>>
>> err = register_ftrace_function_nolock(ops);
>>
>> + if (err)
>> + remove_direct_functions_hash(hash, addr);
>> +
>> out_unlock:
>> mutex_unlock(&direct_mutex);
>>
>>
>> Steven,
>>
>> Does this change look good to you?
>>
>>
>
> Seems reasonable to me, we are simply cleaning the entry on failure so
> we don't encounter it late anymore.
> So I will apply this patch ONLY and retest - correct ?
>
> Another question - it seems you found where it broke ? I saw 'Cc:
> stable@vger.kernel.org # v6.6+' in your prev. patch.'
> If so , can you please point me to the offending patch so I add this
> to my records of my discovery work of bpf coexistence
> livepatching ?
>
> Thanks,
>
> Andrey
Update - with latest fix work find, both after loading the livepatch
.ko no conflicts and hooks work and,
before loading it, pre exsisting hooks keep working after the patch is
loaded
You can add Acked-and-tested-by: Andrey Grodzovsky
<andrey.grodzovsky@crowdstrike.com>
Once again, in case you now the exact commit that broke it, please let
me know.
Thanks,
Andrey
>
>>
>> Thanks,
>> Song
>
>
^ permalink raw reply
* Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)
From: Steven Rostedt @ 2025-10-20 20:31 UTC (permalink / raw)
To: Song Liu
Cc: Andrey Grodzovsky, Petr Mladek, kernel-team@lists.ubuntu.com,
live-patching@vger.kernel.org
In-Reply-To: <CAHzjS_tuotYQQ0HmncVp=oFOfcyxmYqCds0MDBMOr5FC5KzhSA@mail.gmail.com>
On Mon, 20 Oct 2025 11:53:34 -0700
Song Liu <song@kernel.org> wrote:
> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
> index 42bd2ba68a82..8f320df0ac52 100644
> --- i/kernel/trace/ftrace.c
> +++ w/kernel/trace/ftrace.c
> @@ -6049,6 +6049,9 @@ int register_ftrace_direct(struct ftrace_ops
> *ops, unsigned long addr)
>
> err = register_ftrace_function_nolock(ops);
>
> + if (err)
> + remove_direct_functions_hash(hash, addr);
> +
> out_unlock:
> mutex_unlock(&direct_mutex);
>
>
> Steven,
>
> Does this change look good to you?
With the small nit that you don't need a space between the err = and the if (err).
But yeah, looks fine to me.
-- Steve
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox