Re: [External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>]

On 10/20/25 14:53, Song Liu wrote:
> On Mon, Oct 20, 2025 at 9:45 AM Andrey Grodzovsky
> <andrey.grodzovsky@crowdstrike.com> wrote:
>> On 10/20/25 12:03, Song Liu wrote:
>>> On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
>>> <andrey.grodzovsky@crowdstrike.com> wrote:
>>> [...]
>>>>> If you build the kernel from source code, there are some samples in
>>>>> samples/livepatch that you can use for testing. PS: You need to enable
>>>>>
>>>>>      CONFIG_SAMPLE_LIVEPATCH=m
>>>>>
>>>>> I hope this helps.
>>>> Thanks Song, working on repro, kernel rebuilt, test module is loading
>>>> but, bpftrace is refusing to attach now to fentries/fexits claiming the
>>>> costum kernel is not supporting it. It did
>>>> attach in the case of stock AWS kernel i copied the .config from. So
>>>> just trying to figure out now if some Kcofnig flags are missing or
>>>> different . Let me know in case you manage to confirm yourself in the
>>>> meanwhile the fix works for
>>>> you.
>>> Yes, it worked in my tests.
>>>
>>> [root@(none) /]# kpatch load linux/samples/livepatch/livepatch-sample.ko
>>> loading patch module: linux/samples/livepatch/livepatch-sample.ko
>>> [root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show
>>> {printf("fexit\n");}' &
>>> [1] 388
>>> [root@(none) /]# Attached 1 probe
>>> [root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show
>>> {printf("fentry\n");}' &
>>> [2] 397
>>> [root@(none) /]# Attached 1 probe
>>>
>>> [root@(none) /]# cat /proc/cmdline
>>> this has been live patched
>>> fentry
>>> fexit
>>>
>>> Thanks,
>>> Song
>>>
>> Verified the failures I observe when trying to attach with BPF trace are
>> only in presence of patch you provided.
>> Please see attached dmesg for failures. Initial warning on boot.
>> Subsequebt warnings and errors at the point i try to run
>> sudo bpftrace -e "fexit:cmdline_proc_show { printf(\"fexit hit\\n\");
>> exit(); }"
>>
>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name
>> resolution
>> stdin:1:1-25: ERROR: kfunc/kretfunc not available for your kernel version.
>>
>> ubuntu@ip-10-10-115-238:~/linux-6.8.1$ sudo cat
>> /sys/kernel/debug/tracing/available_filter_functions | grep
>> cmdline_proc_show
>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name
>> resolution
>> cat: /sys/kernel/debug/tracing/available_filter_functions: No such device
>>
>> After reboot and before trying to attacg with bpftrace,
>> /sys/kernel/debug/tracing/available_filter_functions is available and
>> shows all function.
>>
>> Using stable kernel from
>> https://urldefense.com/v3/__https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.8.1.tar.gz__;!!BmdzS3_lV9HdKG8!1ZJe4jY49_xIzp4h4i4AbqpkLKoAqrXLFX2wDxhoSUDg2kSeTjy3COy9MngNDRlZhJ1oUKgf1yPqmnTY9-Y50TkA$  for build.
>> FTRACE related KCONFIGs bellow
> I can see the similar issue with the upstream kernel. I was testing on
> stable 6.17 before just know because of another issue with upstream
> kernel, and somehow 6.17 kernel doesn't seem to have the issue.
>
> To fix this, I think we should land a fix similar to the earlier diff:
>
> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
> index 42bd2ba68a82..8f320df0ac52 100644
> --- i/kernel/trace/ftrace.c
> +++ w/kernel/trace/ftrace.c
> @@ -6049,6 +6049,9 @@ int register_ftrace_direct(struct ftrace_ops
> *ops, unsigned long addr)
>
>          err = register_ftrace_function_nolock(ops);
>
> +       if (err)
> +               remove_direct_functions_hash(hash, addr);
> +
>    out_unlock:
>          mutex_unlock(&direct_mutex);
>
>
> Steven,
>
> Does this change look good to you?
>
>

Seems reasonable to me, we are simply cleaning the entry on failure so 
we don't encounter it late anymore.
So I will apply this patch ONLY and retest - correct ?

Another question - it seems you found where it broke ? I saw 'Cc: 
stable@vger.kernel.org # v6.6+' in your prev. patch.'
If so , can you please point me to the offending patch so I add this to 
my records of my discovery work of bpf coexistence
livepatching ?

Thanks,

Andrey

>
> Thanks,
> Song