Re: [PATCH v4 bpf-next 00/17] Add kind layout, CRCs to BTF

[Alan Maguire <alan.maguire@oracle.com>]

On 12/11/2023 12:48, Alan Maguire wrote:
> Update struct btf_header to add a new "kind_layout" section containing
> a description of how to parse the BTF kinds known about at BTF
> encoding time.  This provides the opportunity for tools that might
> not know all of these kinds - as is the case when older tools run
> on more newly-generated BTF - to still parse the BTF provided,
> even if it cannot all be used.
> 
> Also add CRCs for the BTF and base BTF (if needed) from which it was
> created.  CRCs provide a few useful features:
> 
> - the base CRC allows us to explicitly identify when the split and
>   base BTF are not matched
> - absence of a base BTF CRC can indicate that BTF is standalone;
>   i.e. not defined relative to base BTF
> 
> The former case can be used to explicitly reject mismatched
> module/kernel BTF rather than assuming it is matched until an
> unexpected type is encountered.
> 
> The latter case is useful for modules that are not built as
> frequently as the kernel; in such cases, the module can be built
> standalone by specifying an empty BTF base:
> 
>  make BTF_BASE= M=path/2/module
> 
> If CRCs are not present (as will be the case for pahole versions
> prior to the proposed v1.26 which will support CRC generation),
> standalone BTF can still be identified by a slower fallback
> method of examining BTF type ids to ensure that BTF is
> self-referential only.
> 
> To ensure existing tooling can handle standalone BTF for kernel
> modules,  we remap the type ids to start after the vmlinux
> BTF ids, to make it appear to be split BTF.  This allows tools
> (and the kernel) that assume split BTF for modules to operate normally.
> 

hi folks

I wanted to capture feedback received on the approach described here for
BTF module generation at my talk at LPC [1].

Stepping back, the aim is to provide a way to generate BTF for a module
such that it is somewhat resilient to minor changes in underlying BTF,
so it does not have to be rebuilt every time vmlinux is built.  The
module references to vmlinux BTF ids are currently very brittle, and
even for the same kernel we get different vmlinux BTF ids if the BTF is
rebuilt.  So the aim is to support a more robust method of module BTF
generation.  Note that the approach described here is not needed for
modules that are built at the same time as the kernel, so it's unlikely
any in-tree modules will need this, but it will be useful for cases such
as where modules are delivered via a package and want to make use
of BTF such that it will not be invalidated.

Turning to the talk, the general consensus - I think - was that the
standalone BTF approach described in this series was problematic.
Consider kfuncs, if we have, for example, our own definition of a
structure in  standalone module BTF, the BTF id of the local structure
will not match that of the core kernel, which has the potential to
confuse the verifier.

A similar problem exists for tracing; we would trace an sk_buff in
the module via the module's view of struct sk_buff, but we have no
guarantees that the module's view is still consistent with the vmlinux
representation (which actually allocated it).

Hopefully I've characterized this correctly; let me know if I missed
something here.

So we need some means to both remap BTF ids in the module BTF that refer
to the vmlinux BTF so they point at the right types, _and_ to check the
consistency of the representation of a vmlinux type between module BTF
build time and when it is loaded into the kernel.

With this in mind, I think a good way forward might be something like
the following:

For cases where we want more change-independent module BTF - which
is resilient to things like reshuffling of vmlinux BTF ids, and small
changes that don't invalidate structure use completely - we add
a "relocatable" option to the --btf_features list of features for pahole
encoding of module BTF.

This option would not be needed for modules built at the same time as
the kernel, since the BTF ids and the types they refer to are consistent.

When used however, it would tell BTF dedup in pahole to add reocation
information as well as generating usual split BTF at the time of module
BTF generation. This relocation information would consist of
descriptions of the BTF types that the module refers to in base BTF and
their dependents. By providing such descriptions, we can then reconcile
the views of types between module and kernel, or if such reconciliation
is impossible, we can refuse to use the BTF. The amount of information
needed for a module will need to be determined, but I'm hopeful in most
cases it would be a small subset of the type information
required for vmlinux as a whole.

The process of reconciling module and vmlinux BTF at module load time
would then be

1. Remap all the split BTF ids representing module-specific types
   and functions to start at last_vmlinux_id + 1. Since the current
   vmlinux may have a different number of types than the vmlinux
   at time of encoding, this remapping is necessary.

2. For each vmlinux type in our list of relocations, check its
   compatibility with the associated vmlinux type.  This is
   somewhat akin to the CO-RE compatibility checks.  Exact rules
   would need to be ironed out, but a somewhat loose approach
   would be ideal such that a few minor changes in a struct
   somewhere do not totally invalidate module BTF. Unlike CO-RE
   though, field offset changes are _not_ good since they imply the
   module has an incorrect view of the structure and might
   start using fields incorrectly.

   Note that this is a bit easier than BTF deduplication, because
   the deduplication process that happened at module encoding time
   has already done the dependency checking for us; we just need
   to do a type-by-type, 1-to-1 comparison between our relocation
   types and current vmlinux types.

3. If all types are consistent, BTF is loaded and we remap the
   module's vmlinux BTF id references to the corresponding
   vmlinux BTF ids of the current vmlinux.

I _think_ this gets us what we want; more resilient module BTF,
but with safety checks to ensure compatible representations.
There were some suggestions of using a hashing method, but I think
such a method presupposes we want exact type matches, which I suspect
would be unlikely to be useful in practice as with most stable-based
distros, small changes in types can be made due to fixes etc.

There were also a suggestion of doing a full dedup, but I think the
consensus in the room (which I agree with) is that would be hard
to do in-kernel.  So the above approach is a compropmise I think;
it gets actual dedup at BTF creation time to create the list of
references and dependents, and we later check them one-by-one on module
load for compatibility.

Anyway I just wanted to try and capture the feedback received, and
lay out a possible direction. Any further thoughts or suggestions
would be much appreciated. Thanks!

Alan

[1] https://lpc.events/event/17/contributions/1576/