Re: [PATCH bpf-next v4] bpftool: add support for split BTF to gen min_core_btf

[Alan Maguire <alan.maguire@oracle.com>]

On 02/02/2024 22:16, Andrii Nakryiko wrote:
> On Wed, Jan 31, 2024 at 10:47 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>>
>> On 30/01/2024 23:05, Bryce Kahle wrote:
>>> From: Bryce Kahle <bryce.kahle@datadoghq.com>
>>>
>>> Enables a user to generate minimized kernel module BTF.
>>>
>>> If an eBPF program probes a function within a kernel module or uses
>>> types that come from a kernel module, split BTF is required. The split
>>> module BTF contains only the BTF types that are unique to the module.
>>> It will reference the base/vmlinux BTF types and always starts its type
>>> IDs at X+1 where X is the largest type ID in the base BTF.
>>>
>>> Minimization allows a user to ship only the types necessary to do
>>> relocations for the program(s) in the provided eBPF object file(s). A
>>> minimized module BTF will still not contain vmlinux BTF types, so you
>>> should always minimize the vmlinux file first, and then minimize the
>>> kernel module file.
>>>
>>> Example:
>>>
>>> bpftool gen min_core_btf vmlinux.btf vm-min.btf prog.bpf.o
>>> bpftool -B vm-min.btf gen min_core_btf mod.btf mod-min.btf prog.bpf.o
>>
>> This is great! I've been working on a somewhat related problem involving
>> split BTF for modules, and I'm trying to figure out if there's overlap
>> with what you've done here that can help in either direction. I'll try
>> and describe what I'm doing. Sorry if this is a bit of a diversion,
>> but I just want to check if there are potential ways your changes could
>> facilitate other scenarios in the future.
>>
>> The problem I'm trying to tackle is to enable split BTF module
>> generation to be more resilient to underlying kernel BTF changes;
>> this would allow for example a module that is not built with the kernel
>> to generate BTF and have it work even if small changes in vmlinux occur.
>> Even a small change in BTF ids in base BTF is enough to invalidate the
>> associated split BTF, so the question is how to make this a bit less
>> brittle. This won't be needed for modules built along with the kernel,
>> but more for cases like a package delivering a kernel module.
>>
>> The way this is done is similar to what you're doing - generating
>> minimal base vmlinux BTF along with the module BTF. In my case however
>> the minimization is not driven by CO-RE relocations; rather it is driven
>> by only adding types that are referenced by module BTF and any other
>> associated types needed. We end up with minimal base BTF that is carried
>> along with the module BTF (in a .BTF.base_minimal section) and this
>> minimal BTF will be used to later reconcile module BTF with the running
>> kernel BTF when the module is loaded; it essentially provides the
>> additional information needed to map to current vmlinux types.
>>
>> In this approach, minimal vmlinux BTF is generated via an additional
>> option to pahole which adds an extra phase to BTF deduplication between
>> module and kernel. Once we have found the candidate mappings for
>> deduplication, we can look at all base BTF references from module BTF
>> and recursively add associated types to the base minimal BTF. Finally we
>> reparent the split BTF to this minimal base BTF. Experiments show most
>> modules wind up with base minimal BTF of around 4000 types, so the
>> minimization seems to work well. But it's complex.
>>
>> So what I've been trying to work out is if this dedup complexity can be
>> eliminated with your changes, but from what I can see, the membership in
>> the minimal base BTF in your case is driven by the CO-RE relocations
>> used in the BPF program. Would there do you think be a future where we
>> would look at doing base minimal BTF generation by other criteria (like
>> references from the module BTF)? Thanks!
> 
> Hm... I might be misremembering or missing something, but the problem
> you are solving doesn't seem to be related to BTF minimization. I also
> forgot why you need BTF deduplication, I vaguely remember we needed to
> remember "expectations" of types that module BTF references in vmlinux
> BTF, but I fail to remember why we needed dedup... Perhaps we need a
> BPF office hours session to go over details again?
>

Yeah, that would be great! I've put

Making split BTF more resilient

..on the agenda for 02-15.

The reason BTF minimization comes into the picture is this - the
expectations split BTF can have of base BTF can be quite complex, and in
figuring out ways to represent them, it occurred that BTF itself - in
the form of the minimal BTF needed to represent those split BTF
references - made sense. Consider cases like a split BTF struct that
contains a base BTF struct embedded in it. If we have a minimal base BTF
which contains such needed base types, we are in a position to use it to
later reconcile the base BTF worlds at encoding time and use time (for
example vmlinux BTF at module build time versus current vmlinux BTF).

Further, a natural time to construct that minimal base BTF presents
itself when we do deduplication between split and base BTF.  The phase
after we have mapped split types to canonical types is the ideal time to
handle this; the algorithm is basically

- foreach reference from split -> base BTF
 - add it to base minimal BTF
This is controlled by a new dedup option - gen_base_btf_minimal - which
would be enabled via  a ---btf_features option to pahole for users who
wanted to generate minimal base BTF. pahole places the new minimized
base BTF in .BTF.base_minimal section, with the split BTF referring to
it in the usual .BTF section. Later this base minimal BTF is used to
reconcile the split BTF expectations with current base BTF.

The kinds of minimizations I see are pretty reasonable for kernel
modules; I tried a number of in-tree modules (which wouldn't use this
feature in practice, just wanted to have something to test with), and
around 4000 types were observed in base minimal BTF.

It's possible we could adapt this minimization process to be guided
by CO-RE relocations (rather than split->base BTF references), if that
would help Bryce's case.

Alan