Re: [RFC bpf-next 00/15] support inline tracing with BTF

[Jiri Olsa <olsajiri@gmail.com>]

On Wed, Oct 15, 2025 at 04:19:29PM +0100, Alan Maguire wrote:
> On 15/10/2025 15:17, Jiri Olsa wrote:
> > On Tue, Oct 14, 2025 at 03:55:53PM +0100, Alan Maguire wrote:
> >> On 14/10/2025 12:52, Jiri Olsa wrote:
> >>> On Mon, Oct 13, 2025 at 05:12:45PM -0700, Alexei Starovoitov wrote:
> >>>> On Mon, Oct 13, 2025 at 12:38 AM Alan Maguire <alan.maguire@oracle.com> wrote:
> >>>>>
> >>>>>
> >>>>> I was trying to avoid being specific about inlines since the same
> >>>>> approach works for function sites with optimized-out parameters and they
> >>>>> could be easily added to the representation (and probably should be in a
> >>>>> future version of this series). Another "extra" source of info
> >>>>> potentially is the (non per-cpu) global variables that Stephen sent
> >>>>> patches for a while back and the feeling was it was too big to add to
> >>>>> vmlinux BTF proper.
> >>>>>
> >>>>> But extra is a terrible name. .BTF.aux for auxiliary info perhaps?
> >>>>
> >>>> aux is too abstract and doesn't convey any meaning.
> >>>> How about "BTF.func_info" ? It will cover inlined and optimized funcs.
> >>>>
> >>>> Thinking more about reuse of struct btf_type for these...
> >>>> After sleeping on it it feels a bit awkward today, since if they're
> >>>> types they suppose to be in one table with other types,
> >>>> searchable and so on, but we actually don't want them there.
> >>>> btf_find_*() isn't fast and people are trying to optimize it.
> >>>> Also if we teach the kernel to use these loc-s they probably
> >>>> should be in a separate table.
> >>>>
> >>>> global non per-cpu vars fit into current BTF's datasec concept,
> >>>> so they can be another kernel module with a different name.
> >>>>
> >>>> I guess one can argue that LOCSEC is similar to DATASEC.
> >>>> Both need their own search tables separate from the main type table.
> >>>>
> >>>>>
> >>>>>> The partially inlined functions were the biggest footgun so far.
> >>>>>> Missing fully inlined is painful, but it's not a footgun.
> >>>>>> So I think doing "kloc" and usdt-like bpf_loc_arg() completely in
> >>>>>> user space is not enough. It's great and, probably, can be supported,
> >>>>>> but the kernel should use this "BTF.inline_info" as well to
> >>>>>> preserve "backward compatibility" for functions that were
> >>>>>> not-inlined in an older kernel and got partially inlined in a new kernel.
> >>>>>>
> >>>>>
> >>>>> That would be great; we'd need to teach the kernel to handle multi-split
> >>>>> BTF but I would hope that wouldn't be too tricky.
> >>>>>
> >>>>>> If we could use kprobe-multi then usdt-like bpf_loc_arg() would
> >>>>>> make a lot of sense, but since libbpf has to attach a bunch
> >>>>>> of regular kprobes it seems to me the kernel support is more appropriate
> >>>>>> for the whole thing.
> >>>>>
> >>>>> I'm happy with either a userspace or kernel-based approach; the main aim
> >>>>> is to provide this functionality in as straightforward a form as
> >>>>> possible to tracers/libbpf. I have to confess I didn't follow the whole
> >>>>> kprobe multi progress, but at one stage that was more kprobe-based
> >>>>> right? Would there be any value in exploring a flavour of kprobe-multi
> >>>>> that didn't use fprobe and might work for this sort of use case? As you
> >>>>> say if we had that keeping a user-space based approach might be more
> >>>>> attractive as an option.
> >>>>
> >>>> Agree.
> >>>>
> >>>> Jiri,
> >>>> how hard would it be to make multi-kprobe work on arbitrary IPs ?
> >>>
> >>> multi-kprobe uses fprobe which uses ftrace/fgraph fast api to attach,
> >>> but it can do that only on the entry of ftrace-able functions which
> >>> have nop5 hooks at the entry
> >>>
> >>> attaching anywhere else requires standard kprobe and the attach time
> >>> (and execution time) will be bad
> >>>
> >>> would be great if inlined functions kept the nop5/fentry hooks ;-)
> >>> but that's probably not that simple
> >>>
> >>
> >> Yeah, if it was doable - and with metadata about inline sites it
> >> certainly _seems_ possible - it does seem to work against the reason we
> >> inline stuff (saving overheads). Steve mentioned this as a possibility
> >> at GNU cauldron too if I remember, so worth discussing of course!
> >>
> >> I was thinking about something simpler to be honest; a flavour of kprobe
> >> multi that used kprobes under the hood in kernel to be suitable for
> >> inline sites without any tweaking of the sites. So there is a kprobe
> >> performance penalty if you're tracing, but none otherwise.
> > 
> > so you mean we'd still use kprobe_multi api and its code would use fprobe
> > for ftrace-able functions and standard kprobe for the rest?
> > 
> > jirka
> 
> Yeah, if possible. For the kernel inline sites we'd be dealing in raw
> addresses rather than function names so that in itself might be enough
> of a hint that it's not an fprobe site, so I guess it could be framed as
> an extension of kprobe multi to support a mix of fprobe-able and
> non-fprobe-able sites. Not sure how feasible that is though.

that seems doable, kprobe-multi api already supports both symbols and addresses,
and because ftrace keeps track of each ftrace-able function we can tell which
is which via ftrace_location call

looks like there's also register_kprobes call that registers kprobes for multiple
addresses

I wonder what's the standard kprobe attach slowdow nowaday, it was substantial few
years back, will check to have an idea

jirka