Re: [RFC bpf-next 00/15] support inline tracing with BTF

[Alan Maguire <alan.maguire@oracle.com>]

On 15/10/2025 15:17, Jiri Olsa wrote:
> On Tue, Oct 14, 2025 at 03:55:53PM +0100, Alan Maguire wrote:
>> On 14/10/2025 12:52, Jiri Olsa wrote:
>>> On Mon, Oct 13, 2025 at 05:12:45PM -0700, Alexei Starovoitov wrote:
>>>> On Mon, Oct 13, 2025 at 12:38 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>>>>>
>>>>>
>>>>> I was trying to avoid being specific about inlines since the same
>>>>> approach works for function sites with optimized-out parameters and they
>>>>> could be easily added to the representation (and probably should be in a
>>>>> future version of this series). Another "extra" source of info
>>>>> potentially is the (non per-cpu) global variables that Stephen sent
>>>>> patches for a while back and the feeling was it was too big to add to
>>>>> vmlinux BTF proper.
>>>>>
>>>>> But extra is a terrible name. .BTF.aux for auxiliary info perhaps?
>>>>
>>>> aux is too abstract and doesn't convey any meaning.
>>>> How about "BTF.func_info" ? It will cover inlined and optimized funcs.
>>>>
>>>> Thinking more about reuse of struct btf_type for these...
>>>> After sleeping on it it feels a bit awkward today, since if they're
>>>> types they suppose to be in one table with other types,
>>>> searchable and so on, but we actually don't want them there.
>>>> btf_find_*() isn't fast and people are trying to optimize it.
>>>> Also if we teach the kernel to use these loc-s they probably
>>>> should be in a separate table.
>>>>
>>>> global non per-cpu vars fit into current BTF's datasec concept,
>>>> so they can be another kernel module with a different name.
>>>>
>>>> I guess one can argue that LOCSEC is similar to DATASEC.
>>>> Both need their own search tables separate from the main type table.
>>>>
>>>>>
>>>>>> The partially inlined functions were the biggest footgun so far.
>>>>>> Missing fully inlined is painful, but it's not a footgun.
>>>>>> So I think doing "kloc" and usdt-like bpf_loc_arg() completely in
>>>>>> user space is not enough. It's great and, probably, can be supported,
>>>>>> but the kernel should use this "BTF.inline_info" as well to
>>>>>> preserve "backward compatibility" for functions that were
>>>>>> not-inlined in an older kernel and got partially inlined in a new kernel.
>>>>>>
>>>>>
>>>>> That would be great; we'd need to teach the kernel to handle multi-split
>>>>> BTF but I would hope that wouldn't be too tricky.
>>>>>
>>>>>> If we could use kprobe-multi then usdt-like bpf_loc_arg() would
>>>>>> make a lot of sense, but since libbpf has to attach a bunch
>>>>>> of regular kprobes it seems to me the kernel support is more appropriate
>>>>>> for the whole thing.
>>>>>
>>>>> I'm happy with either a userspace or kernel-based approach; the main aim
>>>>> is to provide this functionality in as straightforward a form as
>>>>> possible to tracers/libbpf. I have to confess I didn't follow the whole
>>>>> kprobe multi progress, but at one stage that was more kprobe-based
>>>>> right? Would there be any value in exploring a flavour of kprobe-multi
>>>>> that didn't use fprobe and might work for this sort of use case? As you
>>>>> say if we had that keeping a user-space based approach might be more
>>>>> attractive as an option.
>>>>
>>>> Agree.
>>>>
>>>> Jiri,
>>>> how hard would it be to make multi-kprobe work on arbitrary IPs ?
>>>
>>> multi-kprobe uses fprobe which uses ftrace/fgraph fast api to attach,
>>> but it can do that only on the entry of ftrace-able functions which
>>> have nop5 hooks at the entry
>>>
>>> attaching anywhere else requires standard kprobe and the attach time
>>> (and execution time) will be bad
>>>
>>> would be great if inlined functions kept the nop5/fentry hooks ;-)
>>> but that's probably not that simple
>>>
>>
>> Yeah, if it was doable - and with metadata about inline sites it
>> certainly _seems_ possible - it does seem to work against the reason we
>> inline stuff (saving overheads). Steve mentioned this as a possibility
>> at GNU cauldron too if I remember, so worth discussing of course!
>>
>> I was thinking about something simpler to be honest; a flavour of kprobe
>> multi that used kprobes under the hood in kernel to be suitable for
>> inline sites without any tweaking of the sites. So there is a kprobe
>> performance penalty if you're tracing, but none otherwise.
> 
> so you mean we'd still use kprobe_multi api and its code would use fprobe
> for ftrace-able functions and standard kprobe for the rest?
> 
> jirka

Yeah, if possible. For the kernel inline sites we'd be dealing in raw
addresses rather than function names so that in itself might be enough
of a hint that it's not an fprobe site, so I guess it could be framed as
an extension of kprobe multi to support a mix of fprobe-able and
non-fprobe-able sites. Not sure how feasible that is though.

Alan