Re: [PATCH 2/2] btf: Add the option to include global variable types

[Tony Ambardar <tony.ambardar@gmail.com>]

On Tue, Feb 25, 2025 at 10:01:27AM +0000, Alan Maguire wrote:
> On 07/02/2025 23:50, Alexei Starovoitov wrote:
> > On Thu, Feb 6, 2025 at 5:21 PM Stephen Brennan
> > <stephen.s.brennan@oracle.com> wrote:
> >> When the feature was implemented in pahole, my measurements indicated
> >> that vmlinux BTF size increased by about 25.8%, and module BTF size
> >> increased by 53.2%. Due to these increases, the feature is implemented
> >> behind a new config option, allowing users sensitive to increased memory
> >> usage to disable it.
> >>
> > 
> > ...
> >> +config DEBUG_INFO_BTF_GLOBAL_VARS
> >> +       bool "Generate BTF type information for all global variables"
> >> +       default y
> >> +       depends on DEBUG_INFO_BTF && PAHOLE_VERSION >= 128
> >> +       help
> >> +         Include type information for all global variables in the BTF. This
> >> +         increases the size of the BTF information, which increases memory
> >> +         usage at runtime. With global variable types available, runtime
> >> +         debugging and tracers may be able to provide more detail.
> > 
> > This is not a solution.
> > Even if it's changed to 'default n' distros will enable it
> > like they enable everything and will suffer a regression.
> > 
> > We need to add a new module like vmlinux_btf.ko that will contain
> > this additional BTF data. For global vars and everything else we might need.
> > 
>

Hi Alan,

> In this area, I've been exploring adding support for
> CONFIG_DEBUG_INFO_BTF=m , so that the BTF info for vmlinux is delivered
> via a module. From the consumer side, everything looks identical
> (/sys/kernel/btf/vmlinux is there etc), it is just that the .BTF section
> is delivered via btf_vmlinux.ko instead. The original need for this was
> that embedded folks noted that because in the current situation BTF data
> is in vmlinux, they cannot enable BTF because such small-footprint
> systems do not support a large vmlinux binary. However they could
> potentially use kernel BTF if it was delivered via a module. The other
> nice thing about module delivery in the general case is we can make use
> of module compression. In experiments I see a 5.8Mb vmlinux BTF reduce
> to a 1.8Mb btf_vmlinux.ko.gz module on-disk.
> 

Thank you very much for working on this. I was keen to see this since you
first mentioned it a few years back [1], and have been meaning to ping
you on where things stand. Your summary of motivations above is spot on,
and I can add some context w.r.t. OpenWrt, often used on small consumer
Linux routers to: improve security after support ends, expand
functionality, and increase lifetime/reduce e-waste.

This lifetime is already constrained by the limited kernel binary storage
of some devices and ever increasing kernel sizes. The biggest mitigation
is heavy use of loadable modules to avoid using kernel storage and also
reduce the kernel BTF.  Even so, the (compressed) kernel BTF is ~400 KB,
and over the years I've seen kernel sizes grow by ~200 KB per annual LTS
release.

These rates can amount to penalizing BTF usage with _two years of reduced
lifetime_, which is a key obstacle to enabling BTF by default on such
small systems IMO. Having a module-based kernel BTF would be a huge
improvement!

> The challenge in delivering vmlinux BTF in a module is that on module
> load during boot other modules expect vmlinux BTF to be there when
> adding their own BTF to /sys/kernel/btf. And kfunc registration from
> kernel and modules expects this also. So support for deferred BTF module
> load/kfunc registration is required too. I've implemented the former and
> now am working on the latter. Hope to have some RFC patches ready soon,
> but it looks feasible at this point.
> 

That sounds great. I'm looking forward to seeing and trying this out. If
there's anything you can share at this time please let me know.

Thanks,
Tony
 
1: https://lore.kernel.org/bpf/43fd3775-e796-6802-17f0-5c9fdbf368f5@oracle.com/ 

> Assuming such an option was available to small-footprint systems, should
> we consider adding global variables to core vmlinux BTF along with
> per-cpu variables? Then vmlinux BTF extras could be used for some of the
> additional optional representations like function site-specific data
> (inlines etc)? Or are there other factors other than on-disk footprint
> that we need to consider? Thanks!
> 
> Alan
> 
> > pw-bot: cr
> > 
>