Re: [PATCH 2/2] btf: Add the option to include global variable types

[Alan Maguire <alan.maguire@oracle.com>]

On 07/02/2025 23:50, Alexei Starovoitov wrote:
> On Thu, Feb 6, 2025 at 5:21 PM Stephen Brennan
> <stephen.s.brennan@oracle.com> wrote:
>> When the feature was implemented in pahole, my measurements indicated
>> that vmlinux BTF size increased by about 25.8%, and module BTF size
>> increased by 53.2%. Due to these increases, the feature is implemented
>> behind a new config option, allowing users sensitive to increased memory
>> usage to disable it.
>>
> 
> ...
>> +config DEBUG_INFO_BTF_GLOBAL_VARS
>> +       bool "Generate BTF type information for all global variables"
>> +       default y
>> +       depends on DEBUG_INFO_BTF && PAHOLE_VERSION >= 128
>> +       help
>> +         Include type information for all global variables in the BTF. This
>> +         increases the size of the BTF information, which increases memory
>> +         usage at runtime. With global variable types available, runtime
>> +         debugging and tracers may be able to provide more detail.
> 
> This is not a solution.
> Even if it's changed to 'default n' distros will enable it
> like they enable everything and will suffer a regression.
> 
> We need to add a new module like vmlinux_btf.ko that will contain
> this additional BTF data. For global vars and everything else we might need.
> 

In this area, I've been exploring adding support for
CONFIG_DEBUG_INFO_BTF=m , so that the BTF info for vmlinux is delivered
via a module. From the consumer side, everything looks identical
(/sys/kernel/btf/vmlinux is there etc), it is just that the .BTF section
is delivered via btf_vmlinux.ko instead. The original need for this was
that embedded folks noted that because in the current situation BTF data
is in vmlinux, they cannot enable BTF because such small-footprint
systems do not support a large vmlinux binary. However they could
potentially use kernel BTF if it was delivered via a module. The other
nice thing about module delivery in the general case is we can make use
of module compression. In experiments I see a 5.8Mb vmlinux BTF reduce
to a 1.8Mb btf_vmlinux.ko.gz module on-disk.

The challenge in delivering vmlinux BTF in a module is that on module
load during boot other modules expect vmlinux BTF to be there when
adding their own BTF to /sys/kernel/btf. And kfunc registration from
kernel and modules expects this also. So support for deferred BTF module
load/kfunc registration is required too. I've implemented the former and
now am working on the latter. Hope to have some RFC patches ready soon,
but it looks feasible at this point.

Assuming such an option was available to small-footprint systems, should
we consider adding global variables to core vmlinux BTF along with
per-cpu variables? Then vmlinux BTF extras could be used for some of the
additional optional representations like function site-specific data
(inlines etc)? Or are there other factors other than on-disk footprint
that we need to consider? Thanks!

Alan

> pw-bot: cr
>