Re: [RFC bpf-next 00/15] support inline tracing with BTF

[Alan Maguire <alan.maguire@oracle.com>]

On 23/10/2025 17:16, Andrii Nakryiko wrote:
> On Thu, Oct 23, 2025 at 7:37 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>>
>> On 16/10/2025 19:36, Andrii Nakryiko wrote:
>>> On Tue, Oct 14, 2025 at 2:58 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>>>>
>>>> On 14/10/2025 01:12, Alexei Starovoitov wrote:
>>>>> On Mon, Oct 13, 2025 at 12:38 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>>>>>>
>>>>>>
>>>>>> I was trying to avoid being specific about inlines since the same
>>>>>> approach works for function sites with optimized-out parameters and they
>>>>>> could be easily added to the representation (and probably should be in a
>>>>>> future version of this series). Another "extra" source of info
>>>>>> potentially is the (non per-cpu) global variables that Stephen sent
>>>>>> patches for a while back and the feeling was it was too big to add to
>>>>>> vmlinux BTF proper.
>>>>>>
>>>>>> But extra is a terrible name. .BTF.aux for auxiliary info perhaps?
>>>>>
>>>>> aux is too abstract and doesn't convey any meaning.
>>>>> How about "BTF.func_info" ? It will cover inlined and optimized funcs.
>>>>>
>>>>
>>>> Sure, works for me.
>>>>
>>>>> Thinking more about reuse of struct btf_type for these...
>>>>> After sleeping on it it feels a bit awkward today, since if they're
>>>>> types they suppose to be in one table with other types,
>>>>> searchable and so on, but we actually don't want them there.
>>>>> btf_find_*() isn't fast and people are trying to optimize it.
>>>>> Also if we teach the kernel to use these loc-s they probably
>>>>> should be in a separate table.
>>>>>
>>>>
>>>> The BTF with location info is a separate split BTF, so it won't regress
>>>> search times of vmlinux/module BTF. Searching by name isn't really a
>>>> need for the non-LOCSEC cases; None of the FUNC_PROTO, LOC_PROTO and
>>>> LOC_PARAM have names, so the searching that will be done to deal with
>>>> inlines will all be within the LOCSEC representations for the inlines,
>>>> and from there it'll just be id-based lookup.
>>>>
>>>> Currently the LOCSECs are sorted internally by address, but we could
>>>> change that to be by name given that name-based lookup is the much more
>>>> likely search mode.
>>>>
>>>> One limitation we hit is that the max BTF vlen number is not sufficient
>>>> to represent all the inlines in one LOCSEC; we max out at specifying a
>>>> vlen of 65535, and need over 400000 LOCSEC entries. So we add multiple
>>>
>>> We have this, currently:
>>>
>>>
>>> /* Max # of struct/union/enum members or func args */
>>> #define BTF_MAX_VLEN    0xffff
>>>
>>> struct btf_type {
>>>         __u32 name_off;
>>>         /* "info" bits arrangement
>>>          * bits  0-15: vlen (e.g. # of struct's members)
>>>          * bits 16-23: unused
>>>          * bits 24-28: kind (e.g. int, ptr, array...etc)
>>>          * bits 29-30: unused
>>>          * bit     31: kind_flag, currently used by
>>>          *             struct, union, enum, fwd, enum64,
>>>          *             decl_tag and type_tag
>>>          */
>>>
>>>
>>> Note those unused 16-23 bits. We can use them to extend vlen up to 8
>>> million, which should hopefully be good enough? This split by name
>>> prefix sounds unnecessarily convoluted, tbh.
>>>
>>
>> That would be great! Do you have a preference for how libbpf might
>> handle this? Currently we have
>>
>>
>> static inline __u16 btf_vlen(const struct btf_type *t)
>> {
>>         return BTF_INFO_VLEN(t->info);
>> }
>>
>> As a result many consumers (in libbpf and elsewhere) use a __u16 for the
>> vlen value.  Would it make sense to add
>>
>> static inline __u32 btf_extended_vlen(const struct btf_type *t)
>> {
>>         return BTF_INFO_VLEN(t->info);
>> }
>>
>> perhaps?
> 
> just update btf_vlen() to return __u32 and use more bits. Those bits
> should be all zeroes today, so all this should be backwards
> compatible.
> 
>>
>>
>>>
>>>
>>>> LOCSECs. That was just a workaround before, but for faster name-based
>>>> lookup we could perhaps make use of the multiple LOCSECs by grouping
>>>> them by sorted function names. So if the first LOCSEC was called
>>>> inline.a and the next LOCSEC inline.c or whatever we'd know locations
>>>> named a*, b* are in that first LOCSEC and then do a binary search within
>>>> it. We could limit the number of LOCSECs to some reasonable upper bound
>>>> like 1024 and this would mean we'd binary search between ~400 LOCSECs
>>>> first and then - once we'd found the right one - within it to optimize
>>>> lookup time.
>>>>
>>>>> global non per-cpu vars fit into current BTF's datasec concept,
>>>>> so they can be another kernel module with a different name.
>>>>>
>>>>> I guess one can argue that LOCSEC is similar to DATASEC.
>>>>> Both need their own search tables separate from the main type table.
>>>>>
>>>>
>>>> Right though we could use a hybrid approach of using the LOCSEC name +
>>>> multiple LOCSECs (which we need anyway) to speed things up.
>>>>>>
>>>>>>> The partially inlined functions were the biggest footgun so far.
>>>>>>> Missing fully inlined is painful, but it's not a footgun.
>>>>>>> So I think doing "kloc" and usdt-like bpf_loc_arg() completely in
>>>>>>> user space is not enough. It's great and, probably, can be supported,
>>>>>>> but the kernel should use this "BTF.inline_info" as well to
>>>>>>> preserve "backward compatibility" for functions that were
>>>>>>> not-inlined in an older kernel and got partially inlined in a new kernel.
>>>>>>>
>>>>>>
>>>>>> That would be great; we'd need to teach the kernel to handle multi-split
>>>>>> BTF but I would hope that wouldn't be too tricky.
>>>>>>
>>>>>>> If we could use kprobe-multi then usdt-like bpf_loc_arg() would
>>>>>>> make a lot of sense, but since libbpf has to attach a bunch
>>>>>>> of regular kprobes it seems to me the kernel support is more appropriate
>>>>>>> for the whole thing.
>>>>>>
>>>>>> I'm happy with either a userspace or kernel-based approach; the main aim
>>>>>> is to provide this functionality in as straightforward a form as
>>>>>> possible to tracers/libbpf. I have to confess I didn't follow the whole
>>>>>> kprobe multi progress, but at one stage that was more kprobe-based
>>>>>> right? Would there be any value in exploring a flavour of kprobe-multi
>>>>>> that didn't use fprobe and might work for this sort of use case? As you
>>>>>> say if we had that keeping a user-space based approach might be more
>>>>>> attractive as an option.
>>>>>
>>>>> Agree.
>>>>>
>>>>> Jiri,
>>>>> how hard would it be to make multi-kprobe work on arbitrary IPs ?
>>>>>
>>>>>>
>>>>>>> I mean when the kernel processes SEC("fentry/foo") into partially
>>>>>>> inlined function "foo" it should use fentry for "foo" and
>>>>>>> automatically add kprobe into inlined callsites and automatically
>>>>>>> generated code that collects arguments from appropriate registers
>>>>>>> and make "fentry/foo" behave like "foo" was not inlined at all.
>>>>>>> Arguably, we can use a new attach type.
>>>>>>> If we teach the kernel to do that then doing bpf_loc_arg() and a bunch
>>>>>>> of regular kprobes from libbpf is unnecessary.
>>>>>>> The kernel can do the same transparently and prepare the args
>>>>>>> depending on location.
>>>>>>> If some of the callsites are missing args it can fail the whole operation.
>>>>>>
>>>>>> There's a few options here but I think having attach modes which are
>>>>>> selectable - either best effort or all-or-none would both be needed I
>>>>>> think.
>>>>>
>>>>> Exactly. For partially inlined we would need all-or-none,
>>>>> but I see a case where somebody would want to say:
>>>>> "pls attach to all places where foo() is called and since
>>>>> it's inlined the actual entry point may not be accurate and it's ok".
>>>>>
>>>>> The latter would probably need a flag in tracing tools like bpftrace.
>>>>> I think all-or-none is a better default.
>>>>>
>>>>
>>>> Yep, agree.
>>>>
>>>>>>> Of course, doing the whole thing from libbpf feels good,
>>>>>>> since we're burdening the kernel with extra complexity,
>>>>>>> but lack of kprobe-multi changes the way to think about this trade off.
>>>>>>>
>>>>>>> Whether we decide that the kernel should do it or stay with bpf_loc_arg()
>>>>>>> the first few patches and pahole support can/should be landed first.
>>>>>>>
>>>>>>
>>>>>> Sounds great! Having patches 1-10 would be useful as that would allow us
>>>>>> in turn to update pahole's libbpf submodule commit to generate location
>>>>>> data, which would then allow us to update kbuild and start using it for
>>>>>> attach. So we can focus on generating the inline info first, and then
>>>>>> think about how we want to present that info to consumers.
>>>>>
>>>>> Yep. Please post pahole patches for review. I doubt folks
>>>>> will look into your git tree ;)
>>>>>
>>>>
>>>
>>> BTW, what happened to the self-described BTF patches? With these
>>> additions we are going to break all the BTF-based tooling one more
>>> time. Let's add minimal amount of changes to BTF to allow tools to
>>> skip unknown BTF types and dump the rest? I don't remember all the
>>> details by now, was there any major blocker last time? I feel like
>>> that minimal approach of fixed size + vlen * vlen_size would still
>>> work even for all these newly added types (even with the alternative
>>> for LOC_PARAM I mention in the corresponding patch).
>>>
>>>
>>
>> Yep that scheme would still work. The reason I didn't prioritize it here
>> is that the BTF with new LOC kinds is separate from the BTF that legacy
>> tools would be looking at, but I'd be happy to revive it if it'd help.
> 
> We are coming up on another big BTF update, so I think it's time to
> add this minimal self-describing info and teach bpftool and other
> tools to understand this, so that going forward we can add new types
> without breaking anything. So yeah, I think we should revive and land
> it roughly in the same time frame.
>

Ok sounds good, I'll work on reviving that series as a prerequisite for
the location stuff ASAP.

One other BTF UAPI issue maybe we should look at; should we steal one
more bit for BTF kind representation; we currently have room for 32 and
are using 19, with 3 more for the location stuff. Feels like we should
move to supporting 64 kinds by stealing a bit there too, what do you
think? That would still leave us with one unused bit in "info".

Alan