Re: [RFC bpf-next 00/15] support inline tracing with BTF

[Alan Maguire <alan.maguire@oracle.com>]

On 16/10/2025 19:36, Andrii Nakryiko wrote:
> On Tue, Oct 14, 2025 at 2:58 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>>
>> On 14/10/2025 01:12, Alexei Starovoitov wrote:
>>> On Mon, Oct 13, 2025 at 12:38 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>>>>
>>>>
>>>> I was trying to avoid being specific about inlines since the same
>>>> approach works for function sites with optimized-out parameters and they
>>>> could be easily added to the representation (and probably should be in a
>>>> future version of this series). Another "extra" source of info
>>>> potentially is the (non per-cpu) global variables that Stephen sent
>>>> patches for a while back and the feeling was it was too big to add to
>>>> vmlinux BTF proper.
>>>>
>>>> But extra is a terrible name. .BTF.aux for auxiliary info perhaps?
>>>
>>> aux is too abstract and doesn't convey any meaning.
>>> How about "BTF.func_info" ? It will cover inlined and optimized funcs.
>>>
>>
>> Sure, works for me.
>>
>>> Thinking more about reuse of struct btf_type for these...
>>> After sleeping on it it feels a bit awkward today, since if they're
>>> types they suppose to be in one table with other types,
>>> searchable and so on, but we actually don't want them there.
>>> btf_find_*() isn't fast and people are trying to optimize it.
>>> Also if we teach the kernel to use these loc-s they probably
>>> should be in a separate table.
>>>
>>
>> The BTF with location info is a separate split BTF, so it won't regress
>> search times of vmlinux/module BTF. Searching by name isn't really a
>> need for the non-LOCSEC cases; None of the FUNC_PROTO, LOC_PROTO and
>> LOC_PARAM have names, so the searching that will be done to deal with
>> inlines will all be within the LOCSEC representations for the inlines,
>> and from there it'll just be id-based lookup.
>>
>> Currently the LOCSECs are sorted internally by address, but we could
>> change that to be by name given that name-based lookup is the much more
>> likely search mode.
>>
>> One limitation we hit is that the max BTF vlen number is not sufficient
>> to represent all the inlines in one LOCSEC; we max out at specifying a
>> vlen of 65535, and need over 400000 LOCSEC entries. So we add multiple
> 
> We have this, currently:
> 
> 
> /* Max # of struct/union/enum members or func args */
> #define BTF_MAX_VLEN    0xffff
> 
> struct btf_type {
>         __u32 name_off;
>         /* "info" bits arrangement
>          * bits  0-15: vlen (e.g. # of struct's members)
>          * bits 16-23: unused
>          * bits 24-28: kind (e.g. int, ptr, array...etc)
>          * bits 29-30: unused
>          * bit     31: kind_flag, currently used by
>          *             struct, union, enum, fwd, enum64,
>          *             decl_tag and type_tag
>          */
> 
> 
> Note those unused 16-23 bits. We can use them to extend vlen up to 8
> million, which should hopefully be good enough? This split by name
> prefix sounds unnecessarily convoluted, tbh.
>

That would be great! Do you have a preference for how libbpf might
handle this? Currently we have


static inline __u16 btf_vlen(const struct btf_type *t)
{
        return BTF_INFO_VLEN(t->info);
}

As a result many consumers (in libbpf and elsewhere) use a __u16 for the
vlen value.  Would it make sense to add

static inline __u32 btf_extended_vlen(const struct btf_type *t)
{
        return BTF_INFO_VLEN(t->info);
}

perhaps?


> 
> 
>> LOCSECs. That was just a workaround before, but for faster name-based
>> lookup we could perhaps make use of the multiple LOCSECs by grouping
>> them by sorted function names. So if the first LOCSEC was called
>> inline.a and the next LOCSEC inline.c or whatever we'd know locations
>> named a*, b* are in that first LOCSEC and then do a binary search within
>> it. We could limit the number of LOCSECs to some reasonable upper bound
>> like 1024 and this would mean we'd binary search between ~400 LOCSECs
>> first and then - once we'd found the right one - within it to optimize
>> lookup time.
>>
>>> global non per-cpu vars fit into current BTF's datasec concept,
>>> so they can be another kernel module with a different name.
>>>
>>> I guess one can argue that LOCSEC is similar to DATASEC.
>>> Both need their own search tables separate from the main type table.
>>>
>>
>> Right though we could use a hybrid approach of using the LOCSEC name +
>> multiple LOCSECs (which we need anyway) to speed things up.
>>>>
>>>>> The partially inlined functions were the biggest footgun so far.
>>>>> Missing fully inlined is painful, but it's not a footgun.
>>>>> So I think doing "kloc" and usdt-like bpf_loc_arg() completely in
>>>>> user space is not enough. It's great and, probably, can be supported,
>>>>> but the kernel should use this "BTF.inline_info" as well to
>>>>> preserve "backward compatibility" for functions that were
>>>>> not-inlined in an older kernel and got partially inlined in a new kernel.
>>>>>
>>>>
>>>> That would be great; we'd need to teach the kernel to handle multi-split
>>>> BTF but I would hope that wouldn't be too tricky.
>>>>
>>>>> If we could use kprobe-multi then usdt-like bpf_loc_arg() would
>>>>> make a lot of sense, but since libbpf has to attach a bunch
>>>>> of regular kprobes it seems to me the kernel support is more appropriate
>>>>> for the whole thing.
>>>>
>>>> I'm happy with either a userspace or kernel-based approach; the main aim
>>>> is to provide this functionality in as straightforward a form as
>>>> possible to tracers/libbpf. I have to confess I didn't follow the whole
>>>> kprobe multi progress, but at one stage that was more kprobe-based
>>>> right? Would there be any value in exploring a flavour of kprobe-multi
>>>> that didn't use fprobe and might work for this sort of use case? As you
>>>> say if we had that keeping a user-space based approach might be more
>>>> attractive as an option.
>>>
>>> Agree.
>>>
>>> Jiri,
>>> how hard would it be to make multi-kprobe work on arbitrary IPs ?
>>>
>>>>
>>>>> I mean when the kernel processes SEC("fentry/foo") into partially
>>>>> inlined function "foo" it should use fentry for "foo" and
>>>>> automatically add kprobe into inlined callsites and automatically
>>>>> generated code that collects arguments from appropriate registers
>>>>> and make "fentry/foo" behave like "foo" was not inlined at all.
>>>>> Arguably, we can use a new attach type.
>>>>> If we teach the kernel to do that then doing bpf_loc_arg() and a bunch
>>>>> of regular kprobes from libbpf is unnecessary.
>>>>> The kernel can do the same transparently and prepare the args
>>>>> depending on location.
>>>>> If some of the callsites are missing args it can fail the whole operation.
>>>>
>>>> There's a few options here but I think having attach modes which are
>>>> selectable - either best effort or all-or-none would both be needed I
>>>> think.
>>>
>>> Exactly. For partially inlined we would need all-or-none,
>>> but I see a case where somebody would want to say:
>>> "pls attach to all places where foo() is called and since
>>> it's inlined the actual entry point may not be accurate and it's ok".
>>>
>>> The latter would probably need a flag in tracing tools like bpftrace.
>>> I think all-or-none is a better default.
>>>
>>
>> Yep, agree.
>>
>>>>> Of course, doing the whole thing from libbpf feels good,
>>>>> since we're burdening the kernel with extra complexity,
>>>>> but lack of kprobe-multi changes the way to think about this trade off.
>>>>>
>>>>> Whether we decide that the kernel should do it or stay with bpf_loc_arg()
>>>>> the first few patches and pahole support can/should be landed first.
>>>>>
>>>>
>>>> Sounds great! Having patches 1-10 would be useful as that would allow us
>>>> in turn to update pahole's libbpf submodule commit to generate location
>>>> data, which would then allow us to update kbuild and start using it for
>>>> attach. So we can focus on generating the inline info first, and then
>>>> think about how we want to present that info to consumers.
>>>
>>> Yep. Please post pahole patches for review. I doubt folks
>>> will look into your git tree ;)
>>>
>>
> 
> BTW, what happened to the self-described BTF patches? With these
> additions we are going to break all the BTF-based tooling one more
> time. Let's add minimal amount of changes to BTF to allow tools to
> skip unknown BTF types and dump the rest? I don't remember all the
> details by now, was there any major blocker last time? I feel like
> that minimal approach of fixed size + vlen * vlen_size would still
> work even for all these newly added types (even with the alternative
> for LOC_PARAM I mention in the corresponding patch).
> 
>

Yep that scheme would still work. The reason I didn't prioritize it here
is that the BTF with new LOC kinds is separate from the BTF that legacy
tools would be looking at, but I'd be happy to revive it if it'd help.

Thanks!

Alan