Re: [PATCH 2/2] btf: Add the option to include global variable types

[Alan Maguire <alan.maguire@oracle.com>]

On 25/02/2025 21:52, Andrii Nakryiko wrote:
> On Tue, Feb 25, 2025 at 2:02 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>>
>> On 07/02/2025 23:50, Alexei Starovoitov wrote:
>>> On Thu, Feb 6, 2025 at 5:21 PM Stephen Brennan
>>> <stephen.s.brennan@oracle.com> wrote:
>>>> When the feature was implemented in pahole, my measurements indicated
>>>> that vmlinux BTF size increased by about 25.8%, and module BTF size
>>>> increased by 53.2%. Due to these increases, the feature is implemented
>>>> behind a new config option, allowing users sensitive to increased memory
>>>> usage to disable it.
>>>>
>>>
>>> ...
>>>> +config DEBUG_INFO_BTF_GLOBAL_VARS
>>>> +       bool "Generate BTF type information for all global variables"
>>>> +       default y
>>>> +       depends on DEBUG_INFO_BTF && PAHOLE_VERSION >= 128
>>>> +       help
>>>> +         Include type information for all global variables in the BTF. This
>>>> +         increases the size of the BTF information, which increases memory
>>>> +         usage at runtime. With global variable types available, runtime
>>>> +         debugging and tracers may be able to provide more detail.
>>>
>>> This is not a solution.
>>> Even if it's changed to 'default n' distros will enable it
>>> like they enable everything and will suffer a regression.
>>>
>>> We need to add a new module like vmlinux_btf.ko that will contain
>>> this additional BTF data. For global vars and everything else we might need.
>>>
>>
>> In this area, I've been exploring adding support for
>> CONFIG_DEBUG_INFO_BTF=m , so that the BTF info for vmlinux is delivered
>> via a module. From the consumer side, everything looks identical
>> (/sys/kernel/btf/vmlinux is there etc), it is just that the .BTF section
>> is delivered via btf_vmlinux.ko instead. The original need for this was
>> that embedded folks noted that because in the current situation BTF data
>> is in vmlinux, they cannot enable BTF because such small-footprint
>> systems do not support a large vmlinux binary. However they could
>> potentially use kernel BTF if it was delivered via a module. The other
>> nice thing about module delivery in the general case is we can make use
>> of module compression. In experiments I see a 5.8Mb vmlinux BTF reduce
>> to a 1.8Mb btf_vmlinux.ko.gz module on-disk.
>>
>> The challenge in delivering vmlinux BTF in a module is that on module
>> load during boot other modules expect vmlinux BTF to be there when
>> adding their own BTF to /sys/kernel/btf. And kfunc registration from
>> kernel and modules expects this also. So support for deferred BTF module
>> load/kfunc registration is required too. I've implemented the former and
>> now am working on the latter. Hope to have some RFC patches ready soon,
>> but it looks feasible at this point.
> 
> Lazy btf_vmlinux.ko loading when BTF is actually needed (i.e., when
> user reads /sys/kernel/btf/vmlinux for the first time; or when BPF
> program is validated and needs kernel BTF) would be great. Curious too
> see how all that fits together!
> 
>>
>> Assuming such an option was available to small-footprint systems, should
>> we consider adding global variables to core vmlinux BTF along with
>> per-cpu variables? Then vmlinux BTF extras could be used for some of the
>> additional optional representations like function site-specific data
>> (inlines etc)? Or are there other factors other than on-disk footprint
>> that we need to consider? Thanks!
> 
> I'd keep BTF for variables separate from "core" vmlinux BTF. We can
> have /sys/kernel/btf/vmlinux.vars, which would depend on
> /sys/kernel/btf/vmlinux as a base BTF. Separately, we could eventually
> have /sys/kernel/btf/vmlinux.inlines which would also have
> /sys/kernel/btf/vmlinux as base BTF. If no one needs vmlinux.vars on
> the system, we won't need to waste memory on it. Seems more modular
> and extensible.
>

Sounds good. So thinking about how this fits with
CONFIG_DEBUG_INFO_BTF=m, perhaps the approach would be to use
btf_vmlinux.ko for all such extensible /sys/kernel/btf/vmlinux.vars,
vmlinux.inlines etc. Each of these is derived from .BTF.vars ,
.BTF.inlines sections in btf_vmlinux.ko. These are optionally included
via CONFIG_DEBUG_INFO_BTF_EXTRAS list. If CONFIG_DEBUG_INFO_BTF=y the
core vmlinux section stays in vmlinux itself and the extras are
delivered via btf_vmlinux.ko, but if CONFIG_DEBUG_INFO_BTF=m, the
vmlinux .BTF section is delivered in btf_vmlinux.ko too.

If this makes sense, I'll try and put together the
CONFIG_DEBUG_INFO_BTF=m support first, and that will give us a
btf_vmlinux.ko to work with for delivery of extras. Thanks!

Alan

>>
>> Alan
>>
>>> pw-bot: cr
>>>
>>