Questions about the state of some BTF features

Questions about the state of some BTF features

[Andrii Nakryiko]

Hey Alan,

There were a few BTF-related features you've been working on, and I
realized recently that I don't remember exactly where we ended up with
them and whether there is anything blocking those features. So instead
of going on a mailing list archeology trip, I decided to lazily ask
you directly :)

Basically, at some point we were discussing and reviewing BTF
extensions to have a minimal description of BTF types sizes (fixed and
per-item length). What happened to it? Did we decide it's not
necessary, or is it still in the works?

Also, distilled BTF stuff. We landed libbpf-side API (and I believe
the kernel-side changes went in as well, right?), but I don't think we
enabled this functionality for kernel builds, is that right? What's
missing to have relocatable BTF inside kernel modules? Pahole changes?
Has that landed?

Thanks!

-- Andrii

Re: Questions about the state of some BTF features

[Alan Maguire]

hey Andrii

On 23/10/2024 01:08, Andrii Nakryiko wrote:
> Hey Alan,
> 
> There were a few BTF-related features you've been working on, and I
> realized recently that I don't remember exactly where we ended up with
> them and whether there is anything blocking those features. So instead
> of going on a mailing list archeology trip, I decided to lazily ask
> you directly :)
> 
> Basically, at some point we were discussing and reviewing BTF
> extensions to have a minimal description of BTF types sizes (fixed and
> per-item length). What happened to it? Did we decide it's not
> necessary, or is it still in the works?

Yeah, it's still in the works; more on that below..

> 
> Also, distilled BTF stuff. We landed libbpf-side API (and I believe
> the kernel-side changes went in as well, right?), but I don't think we
> enabled this functionality for kernel builds, is that right? What's
> missing to have relocatable BTF inside kernel modules? Pahole changes?
> Has that landed?
> 

The pahole changes are in, and will be available in the imminent v1.28
release. Distilled BTF will however only be generated for out-of-tree
module builds, since it's not needed for kernels where vmlinux + module
are built at the same time.

Here's the set of BTF things I think we've discussed and folks have
talked about wanting. I've tried to order them based upon dependencies,
but in most cases a different ordering is possible.

1. Build vmlinux BTF as a module (support CONFIG_DEBUG_INFO_BTF=m). This
one helps the embedded folks as modules can be on a separate partition,
and a very large vmlinux is a problem in that environment apparently.
Plus we can do module compression, and I did some measurements and
vmlinux BTF shrinks from ~7Mb to ~1.5Mb when gzip-compressed. This is
sort of a dependency for

2. all global variables in BTF. Stephen Brennan added support to pahole,
but we haven't switched the feature on yet in Makefile.btf. Needs more
testing and for some folks the growth in vmlinux BTF (~1.5Mb) may be an
issue, hence a soft dependency on 1.

3. BTF header modifications to support kind layout. I've been waiting
for the need for a new BTF kind to add this, but that's not strictly
needed. But that brings us on to

4. Augmenting BTF representations to support site-specific info
(including function addresses). We talked about this a bit with Yonghong
at plumbers. Will probably require new kind(s) so 3 should likely be
done first. May also need some special handling so as not to expose
function addresses to unprivileged users.

So I think 1 is possibly needed before 2, and I'm working on an RFC for
1 which I hope to get sent out next week (been a bit delayed working on
the pahole release). 3 would need to be done before 4, or ideally any
other series that introduced new BTF kinds.

So that's the set of things I'm aware of - there may be other needs of
course - but the order 1-4 was roughly how I was thinking we could
attack it. 1 and 2 don't require core BTF changes, so are less
disruptive. We'd got pretty far down the road with an earlier version of
3, so if anyone needed it sooner than I get to it, I'd be happy to help
of course.

Thanks!

Alan

Re: Questions about the state of some BTF features

[Andrii Nakryiko]

On Thu, Oct 24, 2024 at 7:10 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>
> hey Andrii
>
> On 23/10/2024 01:08, Andrii Nakryiko wrote:
> > Hey Alan,
> >
> > There were a few BTF-related features you've been working on, and I
> > realized recently that I don't remember exactly where we ended up with
> > them and whether there is anything blocking those features. So instead
> > of going on a mailing list archeology trip, I decided to lazily ask
> > you directly :)
> >
> > Basically, at some point we were discussing and reviewing BTF
> > extensions to have a minimal description of BTF types sizes (fixed and
> > per-item length). What happened to it? Did we decide it's not
> > necessary, or is it still in the works?
>
> Yeah, it's still in the works; more on that below..
>
> >
> > Also, distilled BTF stuff. We landed libbpf-side API (and I believe
> > the kernel-side changes went in as well, right?), but I don't think we
> > enabled this functionality for kernel builds, is that right? What's
> > missing to have relocatable BTF inside kernel modules? Pahole changes?
> > Has that landed?
> >
>
> The pahole changes are in, and will be available in the imminent v1.28
> release. Distilled BTF will however only be generated for out-of-tree
> module builds, since it's not needed for kernels where vmlinux + module
> are built at the same time.

It's not, strictly speaking, needed, but it might be a good thing to
do this anyways to avoid unnecessary rebuilding of kernel modules
(always a good thing).

But at the very least we should enable it for bpf_testmod* in BPF
selftests. Can we start with that?

>
> Here's the set of BTF things I think we've discussed and folks have
> talked about wanting. I've tried to order them based upon dependencies,
> but in most cases a different ordering is possible.
>
> 1. Build vmlinux BTF as a module (support CONFIG_DEBUG_INFO_BTF=m). This
> one helps the embedded folks as modules can be on a separate partition,
> and a very large vmlinux is a problem in that environment apparently.
> Plus we can do module compression, and I did some measurements and
> vmlinux BTF shrinks from ~7Mb to ~1.5Mb when gzip-compressed. This is
> sort of a dependency for
>
> 2. all global variables in BTF. Stephen Brennan added support to pahole,
> but we haven't switched the feature on yet in Makefile.btf. Needs more
> testing and for some folks the growth in vmlinux BTF (~1.5Mb) may be an
> issue, hence a soft dependency on 1.
>
> 3. BTF header modifications to support kind layout. I've been waiting
> for the need for a new BTF kind to add this, but that's not strictly
> needed. But that brings us on to
>
> 4. Augmenting BTF representations to support site-specific info
> (including function addresses). We talked about this a bit with Yonghong
> at plumbers. Will probably require new kind(s) so 3 should likely be
> done first. May also need some special handling so as not to expose
> function addresses to unprivileged users.
>
> So I think 1 is possibly needed before 2, and I'm working on an RFC for
> 1 which I hope to get sent out next week (been a bit delayed working on
> the pahole release). 3 would need to be done before 4, or ideally any
> other series that introduced new BTF kinds.
>
> So that's the set of things I'm aware of - there may be other needs of
> course - but the order 1-4 was roughly how I was thinking we could
> attack it. 1 and 2 don't require core BTF changes, so are less
> disruptive. We'd got pretty far down the road with an earlier version of
> 3, so if anyone needed it sooner than I get to it, I'd be happy to help
> of course.

Thanks, Alan, for the list.

I think we should prioritize 3 (and 1, of course), as you said, any
BTF extension would be blocked on this (as far as I'm concerned at
least). I wouldn't delay until we actually add a new BTF kind to land
BTF header modifications, that would just delay future work
unnecessarily.

>
> Thanks!
>
> Alan
>

Re: Questions about the state of some BTF features

[Alan Maguire]

On 24/10/2024 17:53, Andrii Nakryiko wrote:
> On Thu, Oct 24, 2024 at 7:10 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>>
>> hey Andrii
>>
>> On 23/10/2024 01:08, Andrii Nakryiko wrote:
>>> Hey Alan,
>>>
>>> There were a few BTF-related features you've been working on, and I
>>> realized recently that I don't remember exactly where we ended up with
>>> them and whether there is anything blocking those features. So instead
>>> of going on a mailing list archeology trip, I decided to lazily ask
>>> you directly :)
>>>
>>> Basically, at some point we were discussing and reviewing BTF
>>> extensions to have a minimal description of BTF types sizes (fixed and
>>> per-item length). What happened to it? Did we decide it's not
>>> necessary, or is it still in the works?
>>
>> Yeah, it's still in the works; more on that below..
>>
>>>
>>> Also, distilled BTF stuff. We landed libbpf-side API (and I believe
>>> the kernel-side changes went in as well, right?), but I don't think we
>>> enabled this functionality for kernel builds, is that right? What's
>>> missing to have relocatable BTF inside kernel modules? Pahole changes?
>>> Has that landed?
>>>
>>
>> The pahole changes are in, and will be available in the imminent v1.28
>> release. Distilled BTF will however only be generated for out-of-tree
>> module builds, since it's not needed for kernels where vmlinux + module
>> are built at the same time.
> 
> It's not, strictly speaking, needed, but it might be a good thing to
> do this anyways to avoid unnecessary rebuilding of kernel modules
> (always a good thing).
> 
> But at the very least we should enable it for bpf_testmod* in BPF
> selftests. Can we start with that?
>

The good news is that already happens, provided you have the updated
pahole to handle distilled base generation. After building selftests I see

$ objdump -h bpf_testmod.ko |grep BTF
  7 .BTF_ids      000001c8  0000000000000000  0000000000000000  00002c50
 2**0
 50 .BTF          000036f4  0000000000000000  0000000000000000  0006e048
 2**0
 51 .BTF.base     000004cc  0000000000000000  0000000000000000  0007173c
 2**0

Given that these changes are in the master branch of dwarves, I _think_
we should be testing with this in CI already, or will be imminently at
least. I'll do some retesting at my end to ensure no regressions are
observed in test results when using distilled base BTF.

One thing I neglected to do was to send a patch that describe .BTF.base
in Documentation/bpf/btf.rst ; we discuss .BTF_ids there so I think it'd
be good to mention .BTF.base there too?

>>
>> Here's the set of BTF things I think we've discussed and folks have
>> talked about wanting. I've tried to order them based upon dependencies,
>> but in most cases a different ordering is possible.
>>
>> 1. Build vmlinux BTF as a module (support CONFIG_DEBUG_INFO_BTF=m). This
>> one helps the embedded folks as modules can be on a separate partition,
>> and a very large vmlinux is a problem in that environment apparently.
>> Plus we can do module compression, and I did some measurements and
>> vmlinux BTF shrinks from ~7Mb to ~1.5Mb when gzip-compressed. This is
>> sort of a dependency for
>>
>> 2. all global variables in BTF. Stephen Brennan added support to pahole,
>> but we haven't switched the feature on yet in Makefile.btf. Needs more
>> testing and for some folks the growth in vmlinux BTF (~1.5Mb) may be an
>> issue, hence a soft dependency on 1.
>>
>> 3. BTF header modifications to support kind layout. I've been waiting
>> for the need for a new BTF kind to add this, but that's not strictly
>> needed. But that brings us on to
>>
>> 4. Augmenting BTF representations to support site-specific info
>> (including function addresses). We talked about this a bit with Yonghong
>> at plumbers. Will probably require new kind(s) so 3 should likely be
>> done first. May also need some special handling so as not to expose
>> function addresses to unprivileged users.
>>
>> So I think 1 is possibly needed before 2, and I'm working on an RFC for
>> 1 which I hope to get sent out next week (been a bit delayed working on
>> the pahole release). 3 would need to be done before 4, or ideally any
>> other series that introduced new BTF kinds.
>>
>> So that's the set of things I'm aware of - there may be other needs of
>> course - but the order 1-4 was roughly how I was thinking we could
>> attack it. 1 and 2 don't require core BTF changes, so are less
>> disruptive. We'd got pretty far down the road with an earlier version of
>> 3, so if anyone needed it sooner than I get to it, I'd be happy to help
>> of course.
> 
> Thanks, Alan, for the list.
> 
> I think we should prioritize 3 (and 1, of course), as you said, any
> BTF extension would be blocked on this (as far as I'm concerned at
> least). I wouldn't delay until we actually add a new BTF kind to land
> BTF header modifications, that would just delay future work
> unnecessarily.
>

Sounds good! I'll prioritize 3; it was pretty close last time we
discussed it I think.

Alan

Re: Questions about the state of some BTF features

[Andrii Nakryiko]

On Thu, Oct 24, 2024 at 10:21 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>
> On 24/10/2024 17:53, Andrii Nakryiko wrote:
> > On Thu, Oct 24, 2024 at 7:10 AM Alan Maguire <alan.maguire@oracle.com> wrote:
> >>
> >> hey Andrii
> >>
> >> On 23/10/2024 01:08, Andrii Nakryiko wrote:
> >>> Hey Alan,
> >>>
> >>> There were a few BTF-related features you've been working on, and I
> >>> realized recently that I don't remember exactly where we ended up with
> >>> them and whether there is anything blocking those features. So instead
> >>> of going on a mailing list archeology trip, I decided to lazily ask
> >>> you directly :)
> >>>
> >>> Basically, at some point we were discussing and reviewing BTF
> >>> extensions to have a minimal description of BTF types sizes (fixed and
> >>> per-item length). What happened to it? Did we decide it's not
> >>> necessary, or is it still in the works?
> >>
> >> Yeah, it's still in the works; more on that below..
> >>
> >>>
> >>> Also, distilled BTF stuff. We landed libbpf-side API (and I believe
> >>> the kernel-side changes went in as well, right?), but I don't think we
> >>> enabled this functionality for kernel builds, is that right? What's
> >>> missing to have relocatable BTF inside kernel modules? Pahole changes?
> >>> Has that landed?
> >>>
> >>
> >> The pahole changes are in, and will be available in the imminent v1.28
> >> release. Distilled BTF will however only be generated for out-of-tree
> >> module builds, since it's not needed for kernels where vmlinux + module
> >> are built at the same time.
> >
> > It's not, strictly speaking, needed, but it might be a good thing to
> > do this anyways to avoid unnecessary rebuilding of kernel modules
> > (always a good thing).
> >
> > But at the very least we should enable it for bpf_testmod* in BPF
> > selftests. Can we start with that?
> >
>
> The good news is that already happens, provided you have the updated
> pahole to handle distilled base generation. After building selftests I see
>
> $ objdump -h bpf_testmod.ko |grep BTF
>   7 .BTF_ids      000001c8  0000000000000000  0000000000000000  00002c50
>  2**0
>  50 .BTF          000036f4  0000000000000000  0000000000000000  0006e048
>  2**0
>  51 .BTF.base     000004cc  0000000000000000  0000000000000000  0007173c
>  2**0
>

Indeed, after updating to the latest pahole master now I get
.BTF.base, very nice.

> Given that these changes are in the master branch of dwarves, I _think_
> we should be testing with this in CI already, or will be imminently at
> least. I'll do some retesting at my end to ensure no regressions are
> observed in test results when using distilled base BTF.
>
> One thing I neglected to do was to send a patch that describe .BTF.base
> in Documentation/bpf/btf.rst ; we discuss .BTF_ids there so I think it'd
> be good to mention .BTF.base there too?

Makes sense, please send the patch.

>
> >>
> >> Here's the set of BTF things I think we've discussed and folks have
> >> talked about wanting. I've tried to order them based upon dependencies,
> >> but in most cases a different ordering is possible.
> >>
> >> 1. Build vmlinux BTF as a module (support CONFIG_DEBUG_INFO_BTF=m). This
> >> one helps the embedded folks as modules can be on a separate partition,
> >> and a very large vmlinux is a problem in that environment apparently.
> >> Plus we can do module compression, and I did some measurements and
> >> vmlinux BTF shrinks from ~7Mb to ~1.5Mb when gzip-compressed. This is
> >> sort of a dependency for
> >>
> >> 2. all global variables in BTF. Stephen Brennan added support to pahole,
> >> but we haven't switched the feature on yet in Makefile.btf. Needs more
> >> testing and for some folks the growth in vmlinux BTF (~1.5Mb) may be an
> >> issue, hence a soft dependency on 1.
> >>
> >> 3. BTF header modifications to support kind layout. I've been waiting
> >> for the need for a new BTF kind to add this, but that's not strictly
> >> needed. But that brings us on to
> >>
> >> 4. Augmenting BTF representations to support site-specific info
> >> (including function addresses). We talked about this a bit with Yonghong
> >> at plumbers. Will probably require new kind(s) so 3 should likely be
> >> done first. May also need some special handling so as not to expose
> >> function addresses to unprivileged users.
> >>
> >> So I think 1 is possibly needed before 2, and I'm working on an RFC for
> >> 1 which I hope to get sent out next week (been a bit delayed working on
> >> the pahole release). 3 would need to be done before 4, or ideally any
> >> other series that introduced new BTF kinds.
> >>
> >> So that's the set of things I'm aware of - there may be other needs of
> >> course - but the order 1-4 was roughly how I was thinking we could
> >> attack it. 1 and 2 don't require core BTF changes, so are less
> >> disruptive. We'd got pretty far down the road with an earlier version of
> >> 3, so if anyone needed it sooner than I get to it, I'd be happy to help
> >> of course.
> >
> > Thanks, Alan, for the list.
> >
> > I think we should prioritize 3 (and 1, of course), as you said, any
> > BTF extension would be blocked on this (as far as I'm concerned at
> > least). I wouldn't delay until we actually add a new BTF kind to land
> > BTF header modifications, that would just delay future work
> > unnecessarily.
> >
>
> Sounds good! I'll prioritize 3; it was pretty close last time we
> discussed it I think.
>
> Alan

Re: Questions about the state of some BTF features

[Alexei Starovoitov]

On Thu, Oct 24, 2024 at 4:26 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> >
> > The good news is that already happens, provided you have the updated
> > pahole to handle distilled base generation. After building selftests I see
> >
> > $ objdump -h bpf_testmod.ko |grep BTF
> >   7 .BTF_ids      000001c8  0000000000000000  0000000000000000  00002c50
> >  2**0
> >  50 .BTF          000036f4  0000000000000000  0000000000000000  0006e048
> >  2**0
> >  51 .BTF.base     000004cc  0000000000000000  0000000000000000  0007173c
> >  2**0
> >
>
> Indeed, after updating to the latest pahole master now I get
> .BTF.base, very nice.

I pulled the latest pahole, rebuilt everything,
but still cannot get it to generate BTF.base.

Any special trick needed?

Re: Questions about the state of some BTF features

[Alan Maguire]

On 25/10/2024 17:09, Alexei Starovoitov wrote:
> On Thu, Oct 24, 2024 at 4:26 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
>>
>>>
>>> The good news is that already happens, provided you have the updated
>>> pahole to handle distilled base generation. After building selftests I see
>>>
>>> $ objdump -h bpf_testmod.ko |grep BTF
>>>   7 .BTF_ids      000001c8  0000000000000000  0000000000000000  00002c50
>>>  2**0
>>>  50 .BTF          000036f4  0000000000000000  0000000000000000  0006e048
>>>  2**0
>>>  51 .BTF.base     000004cc  0000000000000000  0000000000000000  0007173c
>>>  2**0
>>>
>>
>> Indeed, after updating to the latest pahole master now I get
>> .BTF.base, very nice.
> 
> I pulled the latest pahole, rebuilt everything,
> but still cannot get it to generate BTF.base.
> 
> Any special trick needed?

Hmm, should just work for bpf_testmod.ko as long as "pahole
--supported_btf_features" reports "distilled_base" among the set of
features. scripts/Makefile.btf should add that feature if KBUILD_EXTMOD
is set, as it should be in the case of building bpf_testmod.ko. I'll
double-check at my end with latest bpf-next, but it was working
yesterday for me.

Alan

Re: Questions about the state of some BTF features

[Alexei Starovoitov]

On Fri, Oct 25, 2024 at 9:15 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>
> On 25/10/2024 17:09, Alexei Starovoitov wrote:
> > On Thu, Oct 24, 2024 at 4:26 PM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> >>
> >>>
> >>> The good news is that already happens, provided you have the updated
> >>> pahole to handle distilled base generation. After building selftests I see
> >>>
> >>> $ objdump -h bpf_testmod.ko |grep BTF
> >>>   7 .BTF_ids      000001c8  0000000000000000  0000000000000000  00002c50
> >>>  2**0
> >>>  50 .BTF          000036f4  0000000000000000  0000000000000000  0006e048
> >>>  2**0
> >>>  51 .BTF.base     000004cc  0000000000000000  0000000000000000  0007173c
> >>>  2**0
> >>>
> >>
> >> Indeed, after updating to the latest pahole master now I get
> >> .BTF.base, very nice.
> >
> > I pulled the latest pahole, rebuilt everything,
> > but still cannot get it to generate BTF.base.
> >
> > Any special trick needed?
>
> Hmm, should just work for bpf_testmod.ko as long as "pahole
> --supported_btf_features" reports "distilled_base" among the set of
> features. scripts/Makefile.btf should add that feature if KBUILD_EXTMOD
> is set, as it should be in the case of building bpf_testmod.ko. I'll
> double-check at my end with latest bpf-next, but it was working
> yesterday for me.

There must be something else necessary:

pahole -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs
--lang_exclude=rust --btf_features=distilled_base --btf_base vmlinux
.../bpf/bpf_testmod/bpf_testmod.ko; .../resolve_btfids -b vmlinux
.../selftests/bpf/bpf_testmod/bpf_testmod.ko;

objdump -h .../testing/selftests/bpf/bpf_testmod/bpf_testmod.ko|grep BTF
  7 .BTF_ids      000001c8  0000000000000000  0000000000000000  00001d94  2**0
 50 .BTF          00002ea7  0000000000000000  0000000000000000  00062e30  2**0

Re: Questions about the state of some BTF features

[Alan Maguire]

On 25/10/2024 17:19, Alexei Starovoitov wrote:
> On Fri, Oct 25, 2024 at 9:15 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>>
>> On 25/10/2024 17:09, Alexei Starovoitov wrote:
>>> On Thu, Oct 24, 2024 at 4:26 PM Andrii Nakryiko
>>> <andrii.nakryiko@gmail.com> wrote:
>>>>
>>>>>
>>>>> The good news is that already happens, provided you have the updated
>>>>> pahole to handle distilled base generation. After building selftests I see
>>>>>
>>>>> $ objdump -h bpf_testmod.ko |grep BTF
>>>>>   7 .BTF_ids      000001c8  0000000000000000  0000000000000000  00002c50
>>>>>  2**0
>>>>>  50 .BTF          000036f4  0000000000000000  0000000000000000  0006e048
>>>>>  2**0
>>>>>  51 .BTF.base     000004cc  0000000000000000  0000000000000000  0007173c
>>>>>  2**0
>>>>>
>>>>
>>>> Indeed, after updating to the latest pahole master now I get
>>>> .BTF.base, very nice.
>>>
>>> I pulled the latest pahole, rebuilt everything,
>>> but still cannot get it to generate BTF.base.
>>>
>>> Any special trick needed?
>>
>> Hmm, should just work for bpf_testmod.ko as long as "pahole
>> --supported_btf_features" reports "distilled_base" among the set of
>> features. scripts/Makefile.btf should add that feature if KBUILD_EXTMOD
>> is set, as it should be in the case of building bpf_testmod.ko. I'll
>> double-check at my end with latest bpf-next, but it was working
>> yesterday for me.
> 
> There must be something else necessary:
> 
> pahole -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs
> --lang_exclude=rust --btf_features=distilled_base --btf_base vmlinux
> .../bpf/bpf_testmod/bpf_testmod.ko; .../resolve_btfids -b vmlinux
> .../selftests/bpf/bpf_testmod/bpf_testmod.ko;
> 
> objdump -h .../testing/selftests/bpf/bpf_testmod/bpf_testmod.ko|grep BTF
>   7 .BTF_ids      000001c8  0000000000000000  0000000000000000  00001d94  2**0
>  50 .BTF          00002ea7  0000000000000000  0000000000000000  00062e30  2**0
> 

Not sure what's going on for you here to be honest. I just tried pulling
latest bpf-next and dwarves master branch, rebuilding pahole and
selftests. I see .BTF.base sections for each .ko in selftests/bpf.
Can you provide the output of

pahole --supported_btf_features

? If it contains distilled_base things _should_ be working. The only
other reason I can think of that you might not get .BTF.base sections is
if dwarves was built against a local libbpf (rather than the git
submodule) or local libbpf headers _and_ that libbpf did not include
distilled base BTF support; in that case we skip .BTF.base generation
since the APIs are not available.

Re: Questions about the state of some BTF features

[Alexei Starovoitov]

On Fri, Oct 25, 2024 at 9:38 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>
> On 25/10/2024 17:19, Alexei Starovoitov wrote:
> > On Fri, Oct 25, 2024 at 9:15 AM Alan Maguire <alan.maguire@oracle.com> wrote:
> >>
> >> On 25/10/2024 17:09, Alexei Starovoitov wrote:
> >>> On Thu, Oct 24, 2024 at 4:26 PM Andrii Nakryiko
> >>> <andrii.nakryiko@gmail.com> wrote:
> >>>>
> >>>>>
> >>>>> The good news is that already happens, provided you have the updated
> >>>>> pahole to handle distilled base generation. After building selftests I see
> >>>>>
> >>>>> $ objdump -h bpf_testmod.ko |grep BTF
> >>>>>   7 .BTF_ids      000001c8  0000000000000000  0000000000000000  00002c50
> >>>>>  2**0
> >>>>>  50 .BTF          000036f4  0000000000000000  0000000000000000  0006e048
> >>>>>  2**0
> >>>>>  51 .BTF.base     000004cc  0000000000000000  0000000000000000  0007173c
> >>>>>  2**0
> >>>>>
> >>>>
> >>>> Indeed, after updating to the latest pahole master now I get
> >>>> .BTF.base, very nice.
> >>>
> >>> I pulled the latest pahole, rebuilt everything,
> >>> but still cannot get it to generate BTF.base.
> >>>
> >>> Any special trick needed?
> >>
> >> Hmm, should just work for bpf_testmod.ko as long as "pahole
> >> --supported_btf_features" reports "distilled_base" among the set of
> >> features. scripts/Makefile.btf should add that feature if KBUILD_EXTMOD
> >> is set, as it should be in the case of building bpf_testmod.ko. I'll
> >> double-check at my end with latest bpf-next, but it was working
> >> yesterday for me.
> >
> > There must be something else necessary:
> >
> > pahole -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs
> > --lang_exclude=rust --btf_features=distilled_base --btf_base vmlinux
> > .../bpf/bpf_testmod/bpf_testmod.ko; .../resolve_btfids -b vmlinux
> > .../selftests/bpf/bpf_testmod/bpf_testmod.ko;
> >
> > objdump -h .../testing/selftests/bpf/bpf_testmod/bpf_testmod.ko|grep BTF
> >   7 .BTF_ids      000001c8  0000000000000000  0000000000000000  00001d94  2**0
> >  50 .BTF          00002ea7  0000000000000000  0000000000000000  00062e30  2**0
> >
>
> Not sure what's going on for you here to be honest. I just tried pulling
> latest bpf-next and dwarves master branch, rebuilding pahole and
> selftests. I see .BTF.base sections for each .ko in selftests/bpf.
> Can you provide the output of
>
> pahole --supported_btf_features
>
> ? If it contains distilled_base things _should_ be working. The only
> other reason I can think of that you might not get .BTF.base sections is
> if dwarves was built against a local libbpf (rather than the git
> submodule)

That was it.
I did 'git pull' in pahole instead of 'git pull --recurse-submodules'.

Thanks for the tips. Now I see .BTF.base section.

Re: Questions about the state of some BTF features

[Alexei Starovoitov]

On Fri, Oct 25, 2024 at 9:49 AM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Fri, Oct 25, 2024 at 9:38 AM Alan Maguire <alan.maguire@oracle.com> wrote:
> >
> > On 25/10/2024 17:19, Alexei Starovoitov wrote:
> > > On Fri, Oct 25, 2024 at 9:15 AM Alan Maguire <alan.maguire@oracle.com> wrote:
> > >>
> > >> On 25/10/2024 17:09, Alexei Starovoitov wrote:
> > >>> On Thu, Oct 24, 2024 at 4:26 PM Andrii Nakryiko
> > >>> <andrii.nakryiko@gmail.com> wrote:
> > >>>>
> > >>>>>
> > >>>>> The good news is that already happens, provided you have the updated
> > >>>>> pahole to handle distilled base generation. After building selftests I see
> > >>>>>
> > >>>>> $ objdump -h bpf_testmod.ko |grep BTF
> > >>>>>   7 .BTF_ids      000001c8  0000000000000000  0000000000000000  00002c50
> > >>>>>  2**0
> > >>>>>  50 .BTF          000036f4  0000000000000000  0000000000000000  0006e048
> > >>>>>  2**0
> > >>>>>  51 .BTF.base     000004cc  0000000000000000  0000000000000000  0007173c
> > >>>>>  2**0
> > >>>>>
> > >>>>
> > >>>> Indeed, after updating to the latest pahole master now I get
> > >>>> .BTF.base, very nice.
> > >>>
> > >>> I pulled the latest pahole, rebuilt everything,
> > >>> but still cannot get it to generate BTF.base.
> > >>>
> > >>> Any special trick needed?
> > >>
> > >> Hmm, should just work for bpf_testmod.ko as long as "pahole
> > >> --supported_btf_features" reports "distilled_base" among the set of
> > >> features. scripts/Makefile.btf should add that feature if KBUILD_EXTMOD
> > >> is set, as it should be in the case of building bpf_testmod.ko. I'll
> > >> double-check at my end with latest bpf-next, but it was working
> > >> yesterday for me.
> > >
> > > There must be something else necessary:
> > >
> > > pahole -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs
> > > --lang_exclude=rust --btf_features=distilled_base --btf_base vmlinux
> > > .../bpf/bpf_testmod/bpf_testmod.ko; .../resolve_btfids -b vmlinux
> > > .../selftests/bpf/bpf_testmod/bpf_testmod.ko;
> > >
> > > objdump -h .../testing/selftests/bpf/bpf_testmod/bpf_testmod.ko|grep BTF
> > >   7 .BTF_ids      000001c8  0000000000000000  0000000000000000  00001d94  2**0
> > >  50 .BTF          00002ea7  0000000000000000  0000000000000000  00062e30  2**0
> > >
> >
> > Not sure what's going on for you here to be honest. I just tried pulling
> > latest bpf-next and dwarves master branch, rebuilding pahole and
> > selftests. I see .BTF.base sections for each .ko in selftests/bpf.
> > Can you provide the output of
> >
> > pahole --supported_btf_features
> >
> > ? If it contains distilled_base things _should_ be working. The only
> > other reason I can think of that you might not get .BTF.base sections is
> > if dwarves was built against a local libbpf (rather than the git
> > submodule)
>
> That was it.
> I did 'git pull' in pahole instead of 'git pull --recurse-submodules'.
>
> Thanks for the tips. Now I see .BTF.base section.

Forgot to add that even in this broken configuration:
before:
$ pahole --supported_btf_features
encode_force,var,float,decl_tag,type_tag,enum64,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build,distilled_base,global_var
after:
$ pahole --supported_btf_features
encode_force,var,float,decl_tag,type_tag,enum64,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build,distilled_base,global_var

No difference :(
Both claim that distilled_base is working, but it didn't.
Before had older libbpf submodule.
Maybe it should print libbpf version or something ?

Re: Questions about the state of some BTF features

[Alan Maguire]

On 25/10/2024 17:51, Alexei Starovoitov wrote:
> On Fri, Oct 25, 2024 at 9:49 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
>>
>> On Fri, Oct 25, 2024 at 9:38 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>>>
>>> On 25/10/2024 17:19, Alexei Starovoitov wrote:
>>>> On Fri, Oct 25, 2024 at 9:15 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>>>>>
>>>>> On 25/10/2024 17:09, Alexei Starovoitov wrote:
>>>>>> On Thu, Oct 24, 2024 at 4:26 PM Andrii Nakryiko
>>>>>> <andrii.nakryiko@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> The good news is that already happens, provided you have the updated
>>>>>>>> pahole to handle distilled base generation. After building selftests I see
>>>>>>>>
>>>>>>>> $ objdump -h bpf_testmod.ko |grep BTF
>>>>>>>>   7 .BTF_ids      000001c8  0000000000000000  0000000000000000  00002c50
>>>>>>>>  2**0
>>>>>>>>  50 .BTF          000036f4  0000000000000000  0000000000000000  0006e048
>>>>>>>>  2**0
>>>>>>>>  51 .BTF.base     000004cc  0000000000000000  0000000000000000  0007173c
>>>>>>>>  2**0
>>>>>>>>
>>>>>>>
>>>>>>> Indeed, after updating to the latest pahole master now I get
>>>>>>> .BTF.base, very nice.
>>>>>>
>>>>>> I pulled the latest pahole, rebuilt everything,
>>>>>> but still cannot get it to generate BTF.base.
>>>>>>
>>>>>> Any special trick needed?
>>>>>
>>>>> Hmm, should just work for bpf_testmod.ko as long as "pahole
>>>>> --supported_btf_features" reports "distilled_base" among the set of
>>>>> features. scripts/Makefile.btf should add that feature if KBUILD_EXTMOD
>>>>> is set, as it should be in the case of building bpf_testmod.ko. I'll
>>>>> double-check at my end with latest bpf-next, but it was working
>>>>> yesterday for me.
>>>>
>>>> There must be something else necessary:
>>>>
>>>> pahole -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs
>>>> --lang_exclude=rust --btf_features=distilled_base --btf_base vmlinux
>>>> .../bpf/bpf_testmod/bpf_testmod.ko; .../resolve_btfids -b vmlinux
>>>> .../selftests/bpf/bpf_testmod/bpf_testmod.ko;
>>>>
>>>> objdump -h .../testing/selftests/bpf/bpf_testmod/bpf_testmod.ko|grep BTF
>>>>   7 .BTF_ids      000001c8  0000000000000000  0000000000000000  00001d94  2**0
>>>>  50 .BTF          00002ea7  0000000000000000  0000000000000000  00062e30  2**0
>>>>
>>>
>>> Not sure what's going on for you here to be honest. I just tried pulling
>>> latest bpf-next and dwarves master branch, rebuilding pahole and
>>> selftests. I see .BTF.base sections for each .ko in selftests/bpf.
>>> Can you provide the output of
>>>
>>> pahole --supported_btf_features
>>>
>>> ? If it contains distilled_base things _should_ be working. The only
>>> other reason I can think of that you might not get .BTF.base sections is
>>> if dwarves was built against a local libbpf (rather than the git
>>> submodule)
>>
>> That was it.
>> I did 'git pull' in pahole instead of 'git pull --recurse-submodules'.
>>
>> Thanks for the tips. Now I see .BTF.base section.
> 
> Forgot to add that even in this broken configuration:
> before:
> $ pahole --supported_btf_features
> encode_force,var,float,decl_tag,type_tag,enum64,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build,distilled_base,global_var
> after:
> $ pahole --supported_btf_features
> encode_force,var,float,decl_tag,type_tag,enum64,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build,distilled_base,global_var
> 
> No difference :(
> Both claim that distilled_base is working, but it didn't.
> Before had older libbpf submodule.
> Maybe it should print libbpf version or something ?

Good point, this should be clearer. I'll send a patch for pahole which
will ensure the feature isn't reported if libbpf is too old to support
it; at least that way it would have been more obvious for you that
distilled_base wasn't present when using an older libbpf to build pahole.

An extended version description including the libbpf version might be
helpful too, though we shouldn't mess with the format of the basic
pahole version string since it is used so extensively.

Thanks for catching this!

Alan