Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>]

On 10/20/25 15:53, Andrey Grodzovsky wrote:
> On 10/20/25 15:10, Andrey Grodzovsky wrote:
>> On 10/20/25 14:53, Song Liu wrote:
>>> On Mon, Oct 20, 2025 at 9:45 AM Andrey Grodzovsky
>>> <andrey.grodzovsky@crowdstrike.com> wrote:
>>>> On 10/20/25 12:03, Song Liu wrote:
>>>>> On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
>>>>> <andrey.grodzovsky@crowdstrike.com> wrote:
>>>>> [...]
>>>>>>> If you build the kernel from source code, there are some samples in
>>>>>>> samples/livepatch that you can use for testing. PS: You need to 
>>>>>>> enable
>>>>>>>
>>>>>>>      CONFIG_SAMPLE_LIVEPATCH=m
>>>>>>>
>>>>>>> I hope this helps.
>>>>>> Thanks Song, working on repro, kernel rebuilt, test module is 
>>>>>> loading
>>>>>> but, bpftrace is refusing to attach now to fentries/fexits 
>>>>>> claiming the
>>>>>> costum kernel is not supporting it. It did
>>>>>> attach in the case of stock AWS kernel i copied the .config from. So
>>>>>> just trying to figure out now if some Kcofnig flags are missing or
>>>>>> different . Let me know in case you manage to confirm yourself in 
>>>>>> the
>>>>>> meanwhile the fix works for
>>>>>> you.
>>>>> Yes, it worked in my tests.
>>>>>
>>>>> [root@(none) /]# kpatch load 
>>>>> linux/samples/livepatch/livepatch-sample.ko
>>>>> loading patch module: linux/samples/livepatch/livepatch-sample.ko
>>>>> [root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show
>>>>> {printf("fexit\n");}' &
>>>>> [1] 388
>>>>> [root@(none) /]# Attached 1 probe
>>>>> [root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show
>>>>> {printf("fentry\n");}' &
>>>>> [2] 397
>>>>> [root@(none) /]# Attached 1 probe
>>>>>
>>>>> [root@(none) /]# cat /proc/cmdline
>>>>> this has been live patched
>>>>> fentry
>>>>> fexit
>>>>>
>>>>> Thanks,
>>>>> Song
>>>>>
>>>> Verified the failures I observe when trying to attach with BPF 
>>>> trace are
>>>> only in presence of patch you provided.
>>>> Please see attached dmesg for failures. Initial warning on boot.
>>>> Subsequebt warnings and errors at the point i try to run
>>>> sudo bpftrace -e "fexit:cmdline_proc_show { printf(\"fexit hit\\n\");
>>>> exit(); }"
>>>>
>>>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in 
>>>> name
>>>> resolution
>>>> stdin:1:1-25: ERROR: kfunc/kretfunc not available for your kernel 
>>>> version.
>>>>
>>>> ubuntu@ip-10-10-115-238:~/linux-6.8.1$ sudo cat
>>>> /sys/kernel/debug/tracing/available_filter_functions | grep
>>>> cmdline_proc_show
>>>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in 
>>>> name
>>>> resolution
>>>> cat: /sys/kernel/debug/tracing/available_filter_functions: No such 
>>>> device
>>>>
>>>> After reboot and before trying to attacg with bpftrace,
>>>> /sys/kernel/debug/tracing/available_filter_functions is available and
>>>> shows all function.
>>>>
>>>> Using stable kernel from
>>>> https://urldefense.com/v3/__https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.8.1.tar.gz__;!!BmdzS3_lV9HdKG8!1ZJe4jY49_xIzp4h4i4AbqpkLKoAqrXLFX2wDxhoSUDg2kSeTjy3COy9MngNDRlZhJ1oUKgf1yPqmnTY9-Y50TkA$ 
>>>> for build.
>>>> FTRACE related KCONFIGs bellow
>>> I can see the similar issue with the upstream kernel. I was testing on
>>> stable 6.17 before just know because of another issue with upstream
>>> kernel, and somehow 6.17 kernel doesn't seem to have the issue.
>>>
>>> To fix this, I think we should land a fix similar to the earlier diff:
>>>
>>> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
>>> index 42bd2ba68a82..8f320df0ac52 100644
>>> --- i/kernel/trace/ftrace.c
>>> +++ w/kernel/trace/ftrace.c
>>> @@ -6049,6 +6049,9 @@ int register_ftrace_direct(struct ftrace_ops
>>> *ops, unsigned long addr)
>>>
>>>          err = register_ftrace_function_nolock(ops);
>>>
>>> +       if (err)
>>> +               remove_direct_functions_hash(hash, addr);
>>> +
>>>    out_unlock:
>>>          mutex_unlock(&direct_mutex);
>>>
>>>
>>> Steven,
>>>
>>> Does this change look good to you?
>>>
>>>
>>
>> Seems reasonable to me, we are simply cleaning the entry on failure 
>> so we don't encounter it late anymore.
>> So I will apply this patch ONLY and retest - correct ?
>>
>> Another question - it seems you found where it broke ? I saw 'Cc: 
>> stable@vger.kernel.org # v6.6+' in your prev. patch.'
>> If so , can you please point me to the offending patch so I add this 
>> to my records of my discovery work of bpf coexistence
>> livepatching ?
>>
>> Thanks,
>>
>> Andrey
>
>
> Update - with latest fix work find, both after loading the livepatch 
> .ko  no conflicts and hooks work and,
> before loading it, pre exsisting hooks keep working after the patch is 
> loaded
>
> You can add Acked-and-tested-by: Andrey Grodzovsky 
> <andrey.grodzovsky@crowdstrike.com>
>
> Once again, in case you now the exact commit that broke it, please let 
> me know.
>
> Thanks,
> Andrey


Song, I identified another issue in pre 6.6 kernel, building 
~/linux-6.5/samples/livepatch/livepatch-sample.c as ko,
before insmoding it, bpftrace fentry/fexit fires as expected, after 
insmod, while no errors reported on attachments,
the hooks stop firing, both if attaching before insmod and if attaching 
after insmod. If i rrmod the ko, existing hooks
resume working.

ubuntu@ip-10-10-115-238:~$ cat /proc/version_signature
Ubuntu 6.5.0-1008.8-aws 6.5.3
Source obtained to build the test module for the AWS kernel from the 
related stable branch - 
https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz

Let me know what you think.

Thanks,
Andrey



>
>>
>>>
>>> Thanks,
>>> Song
>>
>>
>