[Question] attributes encoding in BTF

[Question] attributes encoding in BTF

[Alexis Lothoré]

Hi all,
a simpler version of this series has been merged, and so I am now taking a
look at the issue I have put aside in the merged version: dealing with more
specific data layout for arguments passed on stack. For example, a function
can pass small structs on stack, but this need special care when generating
the corresponding bpf trampolines. Those structs have specific alignment
specified by the target ABI, but it can also be altered with attributes
packing the structure or modifying the alignment.

Some platforms already support structs on stack (see
tracing_struct_many_args test), but as discussed earlier, those may suffer
from the same kind of issue mentioned above.

On Wed Apr 23, 2025 at 9:24 PM CEST, Alexis Lothoré wrote:
> Hi Andrii,
>
> On Wed Apr 23, 2025 at 7:15 PM CEST, Andrii Nakryiko wrote:
>> On Thu, Apr 17, 2025 at 12:14 AM Alexis Lothoré
>> <alexis.lothore@bootlin.com> wrote:
>>>
>>> Hi Andrii,
>>>
>>> On Wed Apr 16, 2025 at 11:24 PM CEST, Andrii Nakryiko wrote:

[...]

>> I'd suggest looking at btf__align_of() in libbpf (tools/lib/bpf/btf.c)
>> to see how we calculate alignment there. It seems to work decently
>> enough. It won't cover any arch-specific extra rules like double
>> needing 16-byte alignment (I vaguely remember something like that for
>> some architectures, but I might be misremembering), or anything
>> similar. It also won't detect (I don't think it's possible without
>> DWARF) artificially increased alignment with attribute((aligned(N))).
>
> Thanks for the pointer, I'll take a look at it. The more we discuss this
> series, the less member size sounds relevant for what I'm trying to achieve
> here.
>
> Following Xu's comments, I have been thinking about how I could detect the
> custom alignments and packing on structures, and I was wondering if I could
> somehow benefit from __attribute__ encoding in BTF info ([1]). But
> following your hint, I also see some btf_is_struct_packed() in
> tools/lib/bpf/btf_dump.c that could help. I'll dig this further and see if
> I can manage to make something work with all of this.

Andrii's comment above illustrates well my current issue: when functions
pass arguments on stack, we are missing info for some of them to correctly
build trampolines, especially for struct, which can have attributes like
__attribute__((packed)) or __attribute__((align(x))). [1] seems to be a
recent solution implemented for BTF to cover this need. IIUC it encodes any
arbitratry attribute affecting a data type or function, so if I have some
struct like this one in my kernel or a module:

struct foo {
    short b
    int a;
} __packed;

I would expect the corresponding BTF data to have some BTF_KIND_DECL_TAG
describing the "packed" attribute for the corresponding structure, but I
fail to find any of those when running:

$ bpftool btf dump file vmlinux format raw

In there I see some DECL_TAG but those are mostly 'bpf_kfunc', I see not
arbitrary attribute like 'packed' or 'aligned(x)'.

What I really need to do in the end is to be able to parse those alignments
attributes info in the kernel at runtime when generating trampolines, but I
hoped to be able to see them with bpftool first to validate the concept.

I started taking a look further at this and stumbled upon [2] in which Alan
gives many more details about the feature, so I did the following checks:
- kernel version 6.15.0-rc4 from bpf-next_base: it contains the updated
  Makefile.btf calling pahole with `--btf_features=attributes`
- pahole v1.30
  $ pahole --supported_btf_features
  encode_force,var,float,decl_tag,type_tag,enum64,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build,distilled_base,global_var,attributes
  This pahole comes from my distro pkg manager, but I have also done the
  same test with a freshly built pahole, while taking care of pulling the
  libbpf submodule.
- bpftool v7.6.0
    bpftool v7.6.0
    using libbpf v1.6
    features: llvm, skeletons

Could I be missing something obvious ? Or did I misunderstand the actual
attribute encoding feature ?

Thanks,

Alexis

[1] https://lore.kernel.org/bpf/20250130201239.1429648-1-ihor.solodrai@linux.dev/
[2] https://lore.kernel.org/all/CA+icZUW31vpS=R3zM6G4FMkzuiQovqtd+e-8ihwsK_A-QtSSYg@mail.gmail.com/

-- 
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [Question] attributes encoding in BTF

[Ihor Solodrai]

On 6/4/25 2:02 AM, Alexis Lothoré wrote:
> Hi all,
> a simpler version of this series has been merged, and so I am now taking a
> look at the issue I have put aside in the merged version: dealing with more
> specific data layout for arguments passed on stack. For example, a function
> can pass small structs on stack, but this need special care when generating
> the corresponding bpf trampolines. Those structs have specific alignment
> specified by the target ABI, but it can also be altered with attributes
> packing the structure or modifying the alignment.
> 
> Some platforms already support structs on stack (see
> tracing_struct_many_args test), but as discussed earlier, those may suffer
> from the same kind of issue mentioned above.
> 
> On Wed Apr 23, 2025 at 9:24 PM CEST, Alexis Lothoré wrote:
>> Hi Andrii,
>>
>> On Wed Apr 23, 2025 at 7:15 PM CEST, Andrii Nakryiko wrote:
>>> On Thu, Apr 17, 2025 at 12:14 AM Alexis Lothoré
>>> <alexis.lothore@bootlin.com> wrote:
>>>>
>>>> Hi Andrii,
>>>>
>>>> On Wed Apr 16, 2025 at 11:24 PM CEST, Andrii Nakryiko wrote:
> 
> [...]
> 
>>> I'd suggest looking at btf__align_of() in libbpf (tools/lib/bpf/btf.c)
>>> to see how we calculate alignment there. It seems to work decently
>>> enough. It won't cover any arch-specific extra rules like double
>>> needing 16-byte alignment (I vaguely remember something like that for
>>> some architectures, but I might be misremembering), or anything
>>> similar. It also won't detect (I don't think it's possible without
>>> DWARF) artificially increased alignment with attribute((aligned(N))).
>>
>> Thanks for the pointer, I'll take a look at it. The more we discuss this
>> series, the less member size sounds relevant for what I'm trying to achieve
>> here.
>>
>> Following Xu's comments, I have been thinking about how I could detect the
>> custom alignments and packing on structures, and I was wondering if I could
>> somehow benefit from __attribute__ encoding in BTF info ([1]). But
>> following your hint, I also see some btf_is_struct_packed() in
>> tools/lib/bpf/btf_dump.c that could help. I'll dig this further and see if
>> I can manage to make something work with all of this.
> 
> Andrii's comment above illustrates well my current issue: when functions
> pass arguments on stack, we are missing info for some of them to correctly
> build trampolines, especially for struct, which can have attributes like
> __attribute__((packed)) or __attribute__((align(x))). [1] seems to be a
> recent solution implemented for BTF to cover this need. IIUC it encodes any
> arbitratry attribute affecting a data type or function, so if I have some
> struct like this one in my kernel or a module:
> 
> struct foo {
>      short b
>      int a;
> } __packed;
> 
> I would expect the corresponding BTF data to have some BTF_KIND_DECL_TAG
> describing the "packed" attribute for the corresponding structure, but I
> fail to find any of those when running:
> 
> $ bpftool btf dump file vmlinux format raw
> 
> In there I see some DECL_TAG but those are mostly 'bpf_kfunc', I see not
> arbitrary attribute like 'packed' or 'aligned(x)'.
> 
> What I really need to do in the end is to be able to parse those alignments
> attributes info in the kernel at runtime when generating trampolines, but I
> hoped to be able to see them with bpftool first to validate the concept.
> 
> I started taking a look further at this and stumbled upon [2] in which Alan
> gives many more details about the feature, so I did the following checks:
> - kernel version 6.15.0-rc4 from bpf-next_base: it contains the updated
>    Makefile.btf calling pahole with `--btf_features=attributes`
> - pahole v1.30
>    $ pahole --supported_btf_features
>    encode_force,var,float,decl_tag,type_tag,enum64,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build,distilled_base,global_var,attributes
>    This pahole comes from my distro pkg manager, but I have also done the
>    same test with a freshly built pahole, while taking care of pulling the
>    libbpf submodule.
> - bpftool v7.6.0
>      bpftool v7.6.0
>      using libbpf v1.6
>      features: llvm, skeletons
> 
> Could I be missing something obvious ? Or did I misunderstand the actual
> attribute encoding feature ?

Hi Alexis.

The changes recently landed in pahole and libbpf re attributes had a 
very narrow goal: passing through particular attributes for some BPF 
kfuncs from the kernel source to vmlinux.h

BTF now has a way of encoding any attribute (as opposed to only bpf 
type/decl tags) by setting type/decl tag kind flag [1]. So it is 
possible to represent attributes like packed and aligned in BTF.

However, the BTF tags need to be generated by something, in case of 
vmlinux by pahole. Pahole generates BTF by parsing DWARF. And, as far as 
I understand, attributes are not (can not be?) represented in DWARF in a 
generic way, it really depends on specifics of the attribute.

In order to support packed/aligned, pahole needs to know how to figure 
them out from DWARF input and add the tags to BTF. And this does not 
happen right now, which is why you don't see anything in bpftool output.

[1] 
https://lore.kernel.org/bpf/20250130201239.1429648-1-ihor.solodrai@linux.dev/

> 
> Thanks,
> 
> Alexis
> 
> [1] https://lore.kernel.org/bpf/20250130201239.1429648-1-ihor.solodrai@linux.dev/
> [2] https://lore.kernel.org/all/CA+icZUW31vpS=R3zM6G4FMkzuiQovqtd+e-8ihwsK_A-QtSSYg@mail.gmail.com/
> 

Re: [Question] attributes encoding in BTF

[Alexis Lothoré]

Hi Ihor,

On Wed Jun 4, 2025 at 7:31 PM CEST, Ihor Solodrai wrote:
> On 6/4/25 2:02 AM, Alexis Lothoré wrote:

[...]

>> Could I be missing something obvious ? Or did I misunderstand the actual
>> attribute encoding feature ?
>
> Hi Alexis.
>
> The changes recently landed in pahole and libbpf re attributes had a 
> very narrow goal: passing through particular attributes for some BPF 
> kfuncs from the kernel source to vmlinux.h
>
> BTF now has a way of encoding any attribute (as opposed to only bpf 
> type/decl tags) by setting type/decl tag kind flag [1]. So it is 
> possible to represent attributes like packed and aligned in BTF.
>
> However, the BTF tags need to be generated by something, in case of 
> vmlinux by pahole. Pahole generates BTF by parsing DWARF. And, as far as 
> I understand, attributes are not (can not be?) represented in DWARF in a 
> generic way, it really depends on specifics of the attribute.
>
> In order to support packed/aligned, pahole needs to know how to figure 
> them out from DWARF input and add the tags to BTF. And this does not 
> happen right now, which is why you don't see anything in bpftool output.
>
> [1] 
> https://lore.kernel.org/bpf/20250130201239.1429648-1-ihor.solodrai@linux.dev/

Thanks for the details ! I have missed this possibility, as I have been
assuming that DWARF info was exposing the needed info. I'll take a look at
it, but if those attributes can not be represented by DWARF, I'll have to
find another way of getting those packing/alignment modifications on data
type (eg: re-use/share btf__align_of from libbpf, as suggested by Andrii,
but it may not able to cover all cases).

Thanks,

Alexis

-- 
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [Question] attributes encoding in BTF

[Alexei Starovoitov]

On Thu, Jun 5, 2025 at 12:35 AM Alexis Lothoré
<alexis.lothore@bootlin.com> wrote:
>
> Hi Ihor,
>
> On Wed Jun 4, 2025 at 7:31 PM CEST, Ihor Solodrai wrote:
> > On 6/4/25 2:02 AM, Alexis Lothoré wrote:
>
> [...]
>
> >> Could I be missing something obvious ? Or did I misunderstand the actual
> >> attribute encoding feature ?
> >
> > Hi Alexis.
> >
> > The changes recently landed in pahole and libbpf re attributes had a
> > very narrow goal: passing through particular attributes for some BPF
> > kfuncs from the kernel source to vmlinux.h
> >
> > BTF now has a way of encoding any attribute (as opposed to only bpf
> > type/decl tags) by setting type/decl tag kind flag [1]. So it is
> > possible to represent attributes like packed and aligned in BTF.
> >
> > However, the BTF tags need to be generated by something, in case of
> > vmlinux by pahole. Pahole generates BTF by parsing DWARF. And, as far as
> > I understand, attributes are not (can not be?) represented in DWARF in a
> > generic way, it really depends on specifics of the attribute.
> >
> > In order to support packed/aligned, pahole needs to know how to figure
> > them out from DWARF input and add the tags to BTF. And this does not
> > happen right now, which is why you don't see anything in bpftool output.
> >
> > [1]
> > https://lore.kernel.org/bpf/20250130201239.1429648-1-ihor.solodrai@linux.dev/
>
> Thanks for the details ! I have missed this possibility, as I have been
> assuming that DWARF info was exposing the needed info. I'll take a look at
> it, but if those attributes can not be represented by DWARF, I'll have to
> find another way of getting those packing/alignment modifications on data
> type (eg: re-use/share btf__align_of from libbpf, as suggested by Andrii,
> but it may not able to cover all cases).

Not sure all the trouble is worth it.
I feel it's a corner case. Something we don't need to fix.

Re: [Question] attributes encoding in BTF

[Alexis Lothoré]

Hi Alexei,

On Thu Jun 5, 2025 at 6:09 PM CEST, Alexei Starovoitov wrote:
> On Thu, Jun 5, 2025 at 12:35 AM Alexis Lothoré
> <alexis.lothore@bootlin.com> wrote:
>>
>> Hi Ihor,
>>
>> On Wed Jun 4, 2025 at 7:31 PM CEST, Ihor Solodrai wrote:
>> > On 6/4/25 2:02 AM, Alexis Lothoré wrote:

[...]

>> Thanks for the details ! I have missed this possibility, as I have been
>> assuming that DWARF info was exposing the needed info. I'll take a look at
>> it, but if those attributes can not be represented by DWARF, I'll have to
>> find another way of getting those packing/alignment modifications on data
>> type (eg: re-use/share btf__align_of from libbpf, as suggested by Andrii,
>> but it may not able to cover all cases).
>
> Not sure all the trouble is worth it.
> I feel it's a corner case. Something we don't need to fix.

TBH I don't own any specific use case really needing this handling, so if
it does not feel worth the trouble, I'm fine with not trying to support
this. My effort is rather motivated by the goal of aligning the ARM64
features with other platform, and so of getting rid of
tools/testing/selftests/bpf/DENYLIST.aarch64.

For the record, this effort also showed that the same kind of issue affects
other platforms already supporting many args + structs passed by value ([1])
- structs alignment with specific alignment constraints are not
  specifically handled (eg: a struct with an __int128 as a top-level
  member, leading to a 16 byte alignment requirement)
- packing and custom alignment is not handled

From there, I could do two different things:
1. do nothing, keep ARM64 as-is with the current version which has been
  recently merged: ARM64 then denies attachment to any function trying to
  pass a struct by value on stack. We keep the tracing_struct tests denied
  for ARM64. Other platforms still allow to attach such functions, but may
  be parsing wrongly arguments in those specific cases.
2. add the constraint applied on ARM64 (refusing attachment when structs are
  passed through stack) to other JIT compilers. Then update the
  tracing_struct test to ensure this specific case is properly denied on
  all platforms to avoid risking reading wrongly arguments passed through
  stack when structs or large types are involved.

I tend to think 2. is better, but let me know if you have a different
opinion here.

Thanks,

Alexis

-- 
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [Question] attributes encoding in BTF

[Alexei Starovoitov]

On Fri, Jun 6, 2025 at 12:45 AM Alexis Lothoré
<alexis.lothore@bootlin.com> wrote:
>
> Hi Alexei,
>
> On Thu Jun 5, 2025 at 6:09 PM CEST, Alexei Starovoitov wrote:
> > On Thu, Jun 5, 2025 at 12:35 AM Alexis Lothoré
> > <alexis.lothore@bootlin.com> wrote:
> >>
> >> Hi Ihor,
> >>
> >> On Wed Jun 4, 2025 at 7:31 PM CEST, Ihor Solodrai wrote:
> >> > On 6/4/25 2:02 AM, Alexis Lothoré wrote:
>
> [...]
>
> >> Thanks for the details ! I have missed this possibility, as I have been
> >> assuming that DWARF info was exposing the needed info. I'll take a look at
> >> it, but if those attributes can not be represented by DWARF, I'll have to
> >> find another way of getting those packing/alignment modifications on data
> >> type (eg: re-use/share btf__align_of from libbpf, as suggested by Andrii,
> >> but it may not able to cover all cases).
> >
> > Not sure all the trouble is worth it.
> > I feel it's a corner case. Something we don't need to fix.
>
> TBH I don't own any specific use case really needing this handling, so if
> it does not feel worth the trouble, I'm fine with not trying to support
> this. My effort is rather motivated by the goal of aligning the ARM64
> features with other platform, and so of getting rid of
> tools/testing/selftests/bpf/DENYLIST.aarch64.
>
> For the record, this effort also showed that the same kind of issue affects
> other platforms already supporting many args + structs passed by value ([1])
> - structs alignment with specific alignment constraints are not
>   specifically handled (eg: a struct with an __int128 as a top-level
>   member, leading to a 16 byte alignment requirement)
> - packing and custom alignment is not handled
>
> From there, I could do two different things:
> 1. do nothing, keep ARM64 as-is with the current version which has been
>   recently merged: ARM64 then denies attachment to any function trying to
>   pass a struct by value on stack. We keep the tracing_struct tests denied
>   for ARM64. Other platforms still allow to attach such functions, but may
>   be parsing wrongly arguments in those specific cases.
> 2. add the constraint applied on ARM64 (refusing attachment when structs are
>   passed through stack) to other JIT compilers. Then update the
>   tracing_struct test to ensure this specific case is properly denied on
>   all platforms to avoid risking reading wrongly arguments passed through
>   stack when structs or large types are involved.
>
> I tend to think 2. is better, but let me know if you have a different
> opinion here.

Agree. tracing_struct_many_args is working on x86, but assumptions
about BTF being able to express everything about calling convention
were not correct, so let's roll back.