Pahole/BTF issue with __int128

Pahole/BTF issue with __int128

[Alexis Lothoré]

Hello,

I am working on some ebpf feature for ARM64 (improving trampolines to
attach tracing programs to functions with more arguments than the current
limit), and I am facing an issue with the generated BTF information when
playing with large int types like __int128 (I need to use those large types
to properly test some architecture-specific alignment expectations). I
suspect the issue to be in pahole, but I would like to get some opinions on
my observations, and maybe some guidance on where to look at to go further.

I would like to attach some fentry/fexit programs to the following kind of
function, which is currently defined in a kernel module (bpf_testmod.ko in
bpf selftests):

  struct bpf_testmod_struct_arg_7 {
  	_int128 a;
  };
  
  noinline int bpf_testmod_test_struct_arg_11(
  	struct bpf_testmod_struct_arg_7 a,
  	struct bpf_testmod_struct_arg_7 b,
  	struct bpf_testmod_struct_arg_7 c,
  	struct bpf_testmod_struct_arg_7 d,
  	short e,
  	struct bpf_testmod_struct_arg_7 f)
  {
  	[...]
  }

This one works well (let's call it case 1), I am able to attach
fentry/fexit programs to such function through libbpf.

However, if, in a case 2, I change the bpf_testmod_test_struct_arg_11
prototype to use __in128 arguments instead of struct arguments, like the
following one:

  noinline int bpf_testmod_test_struct_arg_11(
  	__int128 a,
  	__int128 b,
  	__int128 c,
  	__int128 d,
  	short e,
  	__int128 f)
  {
  	[...]
  }

and rebuild the module/run my test, this does not work anymore, and libbpf
complains with the following error:
  libbpf: prog 'test_struct_many_args_9': failed to find kernel BTF type ID
  of 'bpf_testmod_test_struct_arg_11': -ESRCH

Inspecting the generated BTF information in bpf_testmod.ko file with bpftool, I
indeed find some BTF info related to my target func in case 1 but not in
case 2:

  [...]
  [118] STRUCT 'bpf_testmod_struct_arg_7' size=16 vlen=1
          'a' type_id=10 bits_offset=0
  [...]
  [371] FUNC_PROTO '(anon)' ret_type_id=6 vlen=6
          'a' type_id=118
          'b' type_id=118
          'c' type_id=118
          'd' type_id=118
          'e' type_id=5
          'f' type_id=118
  [372] FUNC 'bpf_testmod_test_struct_arg_11' type_id=371 linkage=static
  [...]

I checked the command executed by the kernel build system to generate BTF
info for the module, and got the following one:
  pahole -J -j\
  --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs\
  --btf_features=attributes --lang_exclude=rust\
  --btf_features=distilled_base --btf_base vmlinux\
  tools/testing/selftests/bpf/bpf_testmod.ko

I ran the same command before/after switching the struct arguments to
__int128, and made the same observation (I am running pahole 1.30). I then
took a look at available DWARF info available in bpf_testmod.ko for pahole
to generate BTF info, and AFAICT, it looks ok (to be confirmed ?) in both
cases (I am using an aarch64-linux-gcc toolchain, v13.2.0 from
https://toolchains.bootlin.com/)

Case 1:

  [...]
  <1><262>: Abbrev Number: 106 (DW_TAG_base_type)
     <263>   DW_AT_byte_size   : 16
     <264>   DW_AT_encoding    : 5       (signed)
     <265>   DW_AT_name        : (indirect string, offset: 0x193bc): __int128
  [...]
  <1><23429>: Abbrev Number: 11 (DW_TAG_structure_type)
     <2342a>   DW_AT_name        : (indirect string, offset: 0xe98d): bpf_testmod_struct_arg_7
     <2342e>   DW_AT_byte_size   : 16
     <2342f>   DW_AT_decl_file   : 1
     <23430>   DW_AT_decl_line   : 70
     <23431>   DW_AT_decl_column : 8
     <23432>   DW_AT_sibling     : <0x23442>
  <2><23436>: Abbrev Number: 12 (DW_TAG_member)
     <23437>   DW_AT_name        : a
     <23439>   DW_AT_decl_file   : 1
     <2343a>   DW_AT_decl_line   : 71
     <2343b>   DW_AT_decl_column : 11
     <2343c>   DW_AT_type        : <0x262>
     <23440>   DW_AT_data_member_location: 0
  [...]
  <1><295c1>: Abbrev Number: 99 (DW_TAG_subprogram)
     <295c2>   DW_AT_external    : 1
     <295c2>   DW_AT_name        : (indirect string, offset: 0x5e20): bpf_testmod_test_struct_arg_11
     <295c6>   DW_AT_decl_file   : 1
     <295c7>   DW_AT_decl_line   : 152
     <295c8>   DW_AT_decl_column : 14
     <295c9>   DW_AT_prototyped  : 1
     <295c9>   DW_AT_type        : <0xdd>
     <295cd>   DW_AT_low_pc      : 0x1380
     <295d5>   DW_AT_high_pc     : 0x34
     <295dd>   DW_AT_frame_base  : 1 byte block: 9c      (DW_OP_call_frame_cfa)
     <295df>   DW_AT_GNU_all_call_sites: 1
     <295df>   DW_AT_sibling     : <0x2964a>
  <2><295e3>: Abbrev Number: 45 (DW_TAG_formal_parameter)
     <295e4>   DW_AT_name        : a
     <295e6>   DW_AT_decl_file   : 1
     <295e7>   DW_AT_decl_line   : 152
     <295e8>   DW_AT_decl_column : 77
     <295e9>   DW_AT_type        : <0x23429>
     <295ed>   DW_AT_location    : 0x6196 (location list)
     <295f1>   DW_AT_GNU_locviews: 0x6194
  [...]

Case 2:

  [...]
  <1><262>: Abbrev Number: 106 (DW_TAG_base_type)
     <263>   DW_AT_byte_size   : 16
     <264>   DW_AT_encoding    : 5       (signed)
     <265>   DW_AT_name        : (indirect string, offset: 0x1935d): __int128
  [...]
   <1><29552>: Abbrev Number: 98 (DW_TAG_subprogram)
      <29553>   DW_AT_external    : 1
      <29553>   DW_AT_name        : (indirect string, offset: 0x5e20): bpf_testmod_test_struct_arg_11
      <29557>   DW_AT_decl_file   : 1
      <29558>   DW_AT_decl_line   : 148
      <29559>   DW_AT_decl_column : 14
      <2955a>   DW_AT_prototyped  : 1
      <2955a>   DW_AT_type        : <0xdd>
      <2955e>   DW_AT_low_pc      : 0x1380
      <29566>   DW_AT_high_pc     : 0x34
      <2956e>   DW_AT_frame_base  : 1 byte block: 9c      (DW_OP_call_frame_cfa)
      <29570>   DW_AT_GNU_all_call_sites: 1
      <29570>   DW_AT_sibling     : <0x295d6>
   <2><29574>: Abbrev Number: 46 (DW_TAG_formal_parameter)
      <29575>   DW_AT_name        : a
      <29577>   DW_AT_decl_file   : 1
      <29578>   DW_AT_decl_line   : 148
      <29579>   DW_AT_decl_column : 54
      <2957a>   DW_AT_type        : <0x262>
      <2957e>   DW_AT_location    : 0x6158 (location list)
      <29582>   DW_AT_GNU_locviews: 0x6154
  [...]

Am I missing some constraint or limitation that would prevent the case 2
function from being described with BTF info ? If not, any advice about how
to debug this further ?

Thanks,

Alexis

-- 
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: Pahole/BTF issue with __int128

[Tony Ambardar]

On Wed, May 07, 2025 at 10:02:51PM +0200, Alexis Lothoré wrote:
> Hello,
> 
> I am working on some ebpf feature for ARM64 (improving trampolines to
> attach tracing programs to functions with more arguments than the current
> limit), and I am facing an issue with the generated BTF information when
> playing with large int types like __int128 (I need to use those large types
> to properly test some architecture-specific alignment expectations). I
> suspect the issue to be in pahole, but I would like to get some opinions on
> my observations, and maybe some guidance on where to look at to go further.
> 
> I would like to attach some fentry/fexit programs to the following kind of
> function, which is currently defined in a kernel module (bpf_testmod.ko in
> bpf selftests):
> 
>   struct bpf_testmod_struct_arg_7 {
>   	_int128 a;
>   };
>   
>   noinline int bpf_testmod_test_struct_arg_11(
>   	struct bpf_testmod_struct_arg_7 a,
>   	struct bpf_testmod_struct_arg_7 b,
>   	struct bpf_testmod_struct_arg_7 c,
>   	struct bpf_testmod_struct_arg_7 d,
>   	short e,
>   	struct bpf_testmod_struct_arg_7 f)
>   {
>   	[...]
>   }
> 
> This one works well (let's call it case 1), I am able to attach
> fentry/fexit programs to such function through libbpf.
> 
> However, if, in a case 2, I change the bpf_testmod_test_struct_arg_11
> prototype to use __in128 arguments instead of struct arguments, like the
> following one:
> 
>   noinline int bpf_testmod_test_struct_arg_11(
>   	__int128 a,
>   	__int128 b,
>   	__int128 c,
>   	__int128 d,
>   	short e,
>   	__int128 f)
>   {
>   	[...]
>   }
> 
> and rebuild the module/run my test, this does not work anymore, and libbpf
> complains with the following error:
>   libbpf: prog 'test_struct_many_args_9': failed to find kernel BTF type ID
>   of 'bpf_testmod_test_struct_arg_11': -ESRCH
> 
> Inspecting the generated BTF information in bpf_testmod.ko file with bpftool, I
> indeed find some BTF info related to my target func in case 1 but not in
> case 2:
> 
>   [...]
>   [118] STRUCT 'bpf_testmod_struct_arg_7' size=16 vlen=1
>           'a' type_id=10 bits_offset=0
>   [...]
>   [371] FUNC_PROTO '(anon)' ret_type_id=6 vlen=6
>           'a' type_id=118
>           'b' type_id=118
>           'c' type_id=118
>           'd' type_id=118
>           'e' type_id=5
>           'f' type_id=118
>   [372] FUNC 'bpf_testmod_test_struct_arg_11' type_id=371 linkage=static
>   [...]
> 
> I checked the command executed by the kernel build system to generate BTF
> info for the module, and got the following one:
>   pahole -J -j\
>   --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs\
>   --btf_features=attributes --lang_exclude=rust\
>   --btf_features=distilled_base --btf_base vmlinux\
>   tools/testing/selftests/bpf/bpf_testmod.ko
> 
> I ran the same command before/after switching the struct arguments to
> __int128, and made the same observation (I am running pahole 1.30). I then
> took a look at available DWARF info available in bpf_testmod.ko for pahole
> to generate BTF info, and AFAICT, it looks ok (to be confirmed ?) in both
> cases (I am using an aarch64-linux-gcc toolchain, v13.2.0 from
> https://toolchains.bootlin.com/)
> 
> Case 1:
> 
>   [...]
>   <1><262>: Abbrev Number: 106 (DW_TAG_base_type)
>      <263>   DW_AT_byte_size   : 16
>      <264>   DW_AT_encoding    : 5       (signed)
>      <265>   DW_AT_name        : (indirect string, offset: 0x193bc): __int128
>   [...]
>   <1><23429>: Abbrev Number: 11 (DW_TAG_structure_type)
>      <2342a>   DW_AT_name        : (indirect string, offset: 0xe98d): bpf_testmod_struct_arg_7
>      <2342e>   DW_AT_byte_size   : 16
>      <2342f>   DW_AT_decl_file   : 1
>      <23430>   DW_AT_decl_line   : 70
>      <23431>   DW_AT_decl_column : 8
>      <23432>   DW_AT_sibling     : <0x23442>
>   <2><23436>: Abbrev Number: 12 (DW_TAG_member)
>      <23437>   DW_AT_name        : a
>      <23439>   DW_AT_decl_file   : 1
>      <2343a>   DW_AT_decl_line   : 71
>      <2343b>   DW_AT_decl_column : 11
>      <2343c>   DW_AT_type        : <0x262>
>      <23440>   DW_AT_data_member_location: 0
>   [...]
>   <1><295c1>: Abbrev Number: 99 (DW_TAG_subprogram)
>      <295c2>   DW_AT_external    : 1
>      <295c2>   DW_AT_name        : (indirect string, offset: 0x5e20): bpf_testmod_test_struct_arg_11
>      <295c6>   DW_AT_decl_file   : 1
>      <295c7>   DW_AT_decl_line   : 152
>      <295c8>   DW_AT_decl_column : 14
>      <295c9>   DW_AT_prototyped  : 1
>      <295c9>   DW_AT_type        : <0xdd>
>      <295cd>   DW_AT_low_pc      : 0x1380
>      <295d5>   DW_AT_high_pc     : 0x34
>      <295dd>   DW_AT_frame_base  : 1 byte block: 9c      (DW_OP_call_frame_cfa)
>      <295df>   DW_AT_GNU_all_call_sites: 1
>      <295df>   DW_AT_sibling     : <0x2964a>
>   <2><295e3>: Abbrev Number: 45 (DW_TAG_formal_parameter)
>      <295e4>   DW_AT_name        : a
>      <295e6>   DW_AT_decl_file   : 1
>      <295e7>   DW_AT_decl_line   : 152
>      <295e8>   DW_AT_decl_column : 77
>      <295e9>   DW_AT_type        : <0x23429>
>      <295ed>   DW_AT_location    : 0x6196 (location list)
>      <295f1>   DW_AT_GNU_locviews: 0x6194
>   [...]
> 
> Case 2:
> 
>   [...]
>   <1><262>: Abbrev Number: 106 (DW_TAG_base_type)
>      <263>   DW_AT_byte_size   : 16
>      <264>   DW_AT_encoding    : 5       (signed)
>      <265>   DW_AT_name        : (indirect string, offset: 0x1935d): __int128
>   [...]
>    <1><29552>: Abbrev Number: 98 (DW_TAG_subprogram)
>       <29553>   DW_AT_external    : 1
>       <29553>   DW_AT_name        : (indirect string, offset: 0x5e20): bpf_testmod_test_struct_arg_11
>       <29557>   DW_AT_decl_file   : 1
>       <29558>   DW_AT_decl_line   : 148
>       <29559>   DW_AT_decl_column : 14
>       <2955a>   DW_AT_prototyped  : 1
>       <2955a>   DW_AT_type        : <0xdd>
>       <2955e>   DW_AT_low_pc      : 0x1380
>       <29566>   DW_AT_high_pc     : 0x34
>       <2956e>   DW_AT_frame_base  : 1 byte block: 9c      (DW_OP_call_frame_cfa)
>       <29570>   DW_AT_GNU_all_call_sites: 1
>       <29570>   DW_AT_sibling     : <0x295d6>
>    <2><29574>: Abbrev Number: 46 (DW_TAG_formal_parameter)
>       <29575>   DW_AT_name        : a
>       <29577>   DW_AT_decl_file   : 1
>       <29578>   DW_AT_decl_line   : 148
>       <29579>   DW_AT_decl_column : 54
>       <2957a>   DW_AT_type        : <0x262>
>       <2957e>   DW_AT_location    : 0x6158 (location list)
>       <29582>   DW_AT_GNU_locviews: 0x6154
>   [...]
> 

Hi Alexis,

> Am I missing some constraint or limitation that would prevent the case 2
> function from being described with BTF info ? If not, any advice about how
> to debug this further ?
> 

I suspect this might be related to an issue I ran into where pahole may
mis-encode types larger than register-size [1]. Out of curiosity, could
you try rebuilding and using a pahole with my latest patch [2]?

1: https://lore.kernel.org/dwarves/20250410083359.198724-1-tony.ambardar@gmail.com/
2: https://lore.kernel.org/dwarves/20250502070318.1561924-1-tony.ambardar@gmail.com/

Cheers,
Tony

> Thanks,
> 
> Alexis
> 
> -- 
> Alexis Lothoré, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Re: Pahole/BTF issue with __int128

[Alexis Lothoré]

Hi Tony,
thanks a lot for the prompt answer !

On Wed May 7, 2025 at 10:39 PM CEST, Tony Ambardar wrote:
> On Wed, May 07, 2025 at 10:02:51PM +0200, Alexis Lothoré wrote:
>> Hello,

[...]

> Hi Alexis,
>
>> Am I missing some constraint or limitation that would prevent the case 2
>> function from being described with BTF info ? If not, any advice about how
>> to debug this further ?
>> 
>
> I suspect this might be related to an issue I ran into where pahole may
> mis-encode types larger than register-size [1]. Out of curiosity, could
> you try rebuilding and using a pahole with my latest patch [2]?
>
> 1: https://lore.kernel.org/dwarves/20250410083359.198724-1-tony.ambardar@gmail.com/
> 2: https://lore.kernel.org/dwarves/20250502070318.1561924-1-tony.ambardar@gmail.com/

I gave a try to your patch on top of pahole 1.30, and indeed it seems that
my issue is fixed with your solution. I now have some BTF info for my
bpf_testmod_test_struct_arg_11 func:

  [...]
  [370] FUNC_PROTO '(anon)' ret_type_id=6 vlen=6
          'a' type_id=10
          'b' type_id=10
          'c' type_id=10
          'd' type_id=10
          'e' type_id=5
          'f' type_id=10
  [371] FUNC 'bpf_testmod_test_struct_arg_11' type_id=370 linkage=static
  [...]

I also did some quick tests around Alan's request in your series, I'll
report to your series' thread.

Thanks for the help !

Alexis


>
> Cheers,
> Tony
>
>> Thanks,
>> 
>> Alexis
>> 
>> -- 
>> Alexis Lothoré, Bootlin
>> Embedded Linux and Kernel engineering
>> https://bootlin.com




-- 
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com