Re: [RFC bpf-next 00/15] support inline tracing with BTF

[Masami Hiramatsu (Google) <mhiramat@kernel.org>]

On Tue, 14 Oct 2025 15:55:53 +0100
Alan Maguire <alan.maguire@oracle.com> wrote:

> On 14/10/2025 12:52, Jiri Olsa wrote:
> > On Mon, Oct 13, 2025 at 05:12:45PM -0700, Alexei Starovoitov wrote:
> >> On Mon, Oct 13, 2025 at 12:38 AM Alan Maguire <alan.maguire@oracle.com> wrote:
> >>>
> >>>
> >>> I was trying to avoid being specific about inlines since the same
> >>> approach works for function sites with optimized-out parameters and they
> >>> could be easily added to the representation (and probably should be in a
> >>> future version of this series). Another "extra" source of info
> >>> potentially is the (non per-cpu) global variables that Stephen sent
> >>> patches for a while back and the feeling was it was too big to add to
> >>> vmlinux BTF proper.
> >>>
> >>> But extra is a terrible name. .BTF.aux for auxiliary info perhaps?
> >>
> >> aux is too abstract and doesn't convey any meaning.
> >> How about "BTF.func_info" ? It will cover inlined and optimized funcs.
> >>
> >> Thinking more about reuse of struct btf_type for these...
> >> After sleeping on it it feels a bit awkward today, since if they're
> >> types they suppose to be in one table with other types,
> >> searchable and so on, but we actually don't want them there.
> >> btf_find_*() isn't fast and people are trying to optimize it.
> >> Also if we teach the kernel to use these loc-s they probably
> >> should be in a separate table.
> >>
> >> global non per-cpu vars fit into current BTF's datasec concept,
> >> so they can be another kernel module with a different name.
> >>
> >> I guess one can argue that LOCSEC is similar to DATASEC.
> >> Both need their own search tables separate from the main type table.
> >>
> >>>
> >>>> The partially inlined functions were the biggest footgun so far.
> >>>> Missing fully inlined is painful, but it's not a footgun.
> >>>> So I think doing "kloc" and usdt-like bpf_loc_arg() completely in
> >>>> user space is not enough. It's great and, probably, can be supported,
> >>>> but the kernel should use this "BTF.inline_info" as well to
> >>>> preserve "backward compatibility" for functions that were
> >>>> not-inlined in an older kernel and got partially inlined in a new kernel.
> >>>>
> >>>
> >>> That would be great; we'd need to teach the kernel to handle multi-split
> >>> BTF but I would hope that wouldn't be too tricky.
> >>>
> >>>> If we could use kprobe-multi then usdt-like bpf_loc_arg() would
> >>>> make a lot of sense, but since libbpf has to attach a bunch
> >>>> of regular kprobes it seems to me the kernel support is more appropriate
> >>>> for the whole thing.
> >>>
> >>> I'm happy with either a userspace or kernel-based approach; the main aim
> >>> is to provide this functionality in as straightforward a form as
> >>> possible to tracers/libbpf. I have to confess I didn't follow the whole
> >>> kprobe multi progress, but at one stage that was more kprobe-based
> >>> right? Would there be any value in exploring a flavour of kprobe-multi
> >>> that didn't use fprobe and might work for this sort of use case? As you
> >>> say if we had that keeping a user-space based approach might be more
> >>> attractive as an option.
> >>
> >> Agree.
> >>
> >> Jiri,
> >> how hard would it be to make multi-kprobe work on arbitrary IPs ?
> > 
> > multi-kprobe uses fprobe which uses ftrace/fgraph fast api to attach,
> > but it can do that only on the entry of ftrace-able functions which
> > have nop5 hooks at the entry
> > 
> > attaching anywhere else requires standard kprobe and the attach time
> > (and execution time) will be bad
> > 
> > would be great if inlined functions kept the nop5/fentry hooks ;-)
> > but that's probably not that simple
> >
> 
> Yeah, if it was doable - and with metadata about inline sites it
> certainly _seems_ possible - it does seem to work against the reason we
> inline stuff (saving overheads). Steve mentioned this as a possibility
> at GNU cauldron too if I remember, so worth discussing of course!

IMHO, it may be hard to insert nop (actually it is mcount call) to the
inlined function, because the inlined function code is optimized with
the caller code. In this case, it is hard to find where is the original
entry code. For example, a specific entry code can be skipped even if
a part of its body is used. Thus we need to put mcount calls for each
(but there can be a code which calls both entry and the body).

But if we can work with compilers, since it knows how it optimizes the
code, it may be possible.

> 
> I was thinking about something simpler to be honest; a flavour of kprobe
> multi that used kprobes under the hood in kernel to be suitable for
> inline sites without any tweaking of the sites. So there is a kprobe
> performance penalty if you're tracing, but none otherwise.

It is possible if we can find the actual entry points. Of course using
kprobes means it will take a time (overhead) to insert the sw
breakpoints and to sync the code.

Thank you,

> 
> Thanks!
> 
> Alan


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>