Re: Question: fentry on kernel func optimized by compiler

[Jiri Olsa <olsajiri@gmail.com>]

On Tue, Apr 15, 2025 at 08:10:09PM +0800, Tao Chen wrote:
> 在 2025/3/31 18:13, Alan Maguire 写道:
> > On 28/03/2025 17:21, Andrii Nakryiko wrote:
> > > On Thu, Mar 27, 2025 at 9:03 AM Tao Chen <chen.dylane@linux.dev> wrote:
> > > > 
> > > > Hi,

hi, sorry for late reply

> > > > 
> > > > I recently encountered a problem when using fentry to trace kernel
> > > > functions optimized by compiler, the specific situation is as follows:
> > > > https://github.com/bpftrace/bpftrace/issues/3940
> > > > 
> > > > Simply put, some functions have been optimized by the compiler. The
> > > > original function names are found through BTF, but the optimized
> > > > functions are the ones that exist in kallsyms_lookup_name. Therefore,
> > > > the two do not match.

hum, would you have example BTF/kernel/config with such case? I'd think
if function is in BTF you should find it through kallsyms, the other way
around is not garanteed

> > > > 
> > > >           func_proto = btf_type_by_id(desc_btf, func->type);
> > > >           if (!func_proto || !btf_type_is_func_proto(func_proto)) {
> > > >                   verbose(env, "kernel function btf_id %u does not have a
> > > > valid func_proto\n",
> > > >                           func_id);
> > > >                   return -EINVAL;
> > > >           }
> > > > 
> > > >           func_name = btf_name_by_offset(desc_btf, func->name_off);
> > > >           addr = kallsyms_lookup_name(func_name);
> > > >           if (!addr) {
> > > >                   verbose(env, "cannot find address for kernel function
> > > > %s\n",
> > > >                           func_name);
> > > >                   return -EINVAL;
> > > >           }
> > > > 
> > > > I have made a simple statistics and there are approximately more than
> > > > 2,000 functions in Ubuntu 24.04.
> > > > 
> > > > dylane@2404:~$ cat /proc/kallsyms | grep isra | wc -l
> > > > 2324
> > > > 
> > > > So can we add a judgment from libbpf. If it is an optimized function,
> > > 
> > > No, we cannot. It's a different function at that point and libbpf
> > > isn't going to be in the business of guessing on behalf of the user
> > > whether it's ok to do or not.
> > > 
> > > But the user can use multi-kprobe with `prefix*` naming, if they
> > > encountered (or are anticipating) this situation and think it's fine
> > > for them.
> > > 
> > > As for fentry/fexit, you need to have the correct BTF ID associated
> > > with that function anyways, so I'm not sure that currently you can
> > > attach fentry/fexit to such compiler-optimized functions at all
> > > (pahole won't produce BTF for such functions, right?).
> > > 
> > 
> > Yep, BTF will not be there for all cases, but ever since we've had the
> > "optimized_func" BTF feature, we've have encoded BTF for suffixed
> > functions as long as their parameters are not optimized away and as long
> > as we don't have multiple inconsistent representations associated with a
> > function (say two differing function signatures for the same name).
> > Optimization away of parameters happens quite frequently, but not always
> > for .isra.0 functions so they are potentially sometimes safe for fentry.
> > 
> > The complication here is that - by design - the function name in BTF
> > will be the prefix; i.e. "foo" not "foo.isra.0". So how we match up the
> > BTF with the right suffixed function is an issue; a single function
> > prefix can have ".isra.0" and ".cold.0" suffixes associated for example.
> > The latter isn't really a function entry point (in the C code at least);
> > it's just a split of the function into common path and less common path
> > for better code locality for the more commonly-executed code.
> > 
> > Yonghong and I talked about this a bit last year in Plumbers, but it did
> > occur to me that there are conditions where we could match up the prefix
> > from BTF with a guaranteed fentry point for the function using info we
> > have today.
> > 
> > /sys/kernel/tracing/available_filter_functions_addr has similar info to
> > /proc/kallysyms but as far as I understand it we are also guaranteed
> > that the associated addresses correspond to real function entry points.

available_filter_functions_addr contains exactly the same functions as
available_filter_functions (all functions managed by ftrace), it just
adds address value for each function as a hint to user space

it's used in kprobe_multi bench test to get all traceable addresses,
and passed to kprobe_multi link, which uses following fprobe interface
to attach:

        err = register_fprobe_ips(&link->fp, addrs, cnt);

> > So because the BTF representation currently ensures consistency _and_
> > available function parameters, I think we could use
> > available_filter_functions_addr to carry out the match and provide the
> > right function address for the BTF representation.

where would you do the match to available_filter_functions_addr?

also not sure how it's connected to the original issue, that seems
to be related to pahole eliminating unsafe functions in BTF?

jirka


> > 
> 
> Hi, Alan
> Sorry for not replying in time. As you said，it seems much simpler when use
> the func addr from available_filter_functions_addr. It seems to be a bit
> similar to the way of passing function addresses in kprobe_multi. @Andrii
> @Jiri what do you think?
> 
> > In the future, the hope is we can handle inconsistent representations
> > too in BTF, but the above represents a possible approach we could
> > implement today I think, though I may be missing something. Thanks!
> > 
> > Alan
> 
> 
> -- 
> Best Regards
> Tao Chen
>