HEADS UP: pahole 1.26 to be released soon

HEADS UP: pahole 1.26 to be released soon

[Arnaldo Carvalho de Melo]

Hi,

	I'm prepping up pahole 1.26, the noticeable thing for BTF
encoding are Alan's patches (--btf_features, --btf_features_strict and
--supported_btf_features).

	There was a fix for BTF loading, namely the signedness of
enumerations, but that isn't used when building the kernel, just when
pretty printing from BTF.

	I also have a few fixes and new features in the pretty printing
code, but again, that isn't involved when building the kernel.

	If you have time, please do some tests and report results.

	You can find it in the usual places:

git://git.kernel.org/pub/scm/devel/pahole/pahole.git
https://github.com/acmel/dwarves.git

	In the master branch.

Thanks a lot!

- Arnaldo

Re: HEADS UP: pahole 1.26 to be released soon

[Arnaldo Carvalho de Melo]

On Tue, Feb 27, 2024 at 09:59:17AM -0300, Arnaldo Carvalho de Melo wrote:
> Hi,
> 
> 	I'm prepping up pahole 1.26, the noticeable thing for BTF
> encoding are Alan's patches (--btf_features, --btf_features_strict and
> --supported_btf_features).
> 
> 	There was a fix for BTF loading, namely the signedness of
> enumerations, but that isn't used when building the kernel, just when
> pretty printing from BTF.
> 
> 	I also have a few fixes and new features in the pretty printing
> code, but again, that isn't involved when building the kernel.

The current 'changes-1.26' file:


pahole:

- When expanding types using 'pahole -E' do it for union and struct typedefs and for enums too.

  E.g: that 'state' field in 'struct module':

    $ pahole module | head
    struct module {
            enum module_state          state;                /*     0     4 */
    
            /* XXX 4 bytes hole, try to pack */
    
            struct list_head           list;                 /*     8    16 */
            char                       name[56];             /*    24    56 */
            /* --- cacheline 1 boundary (64 bytes) was 16 bytes ago --- */
            struct module_kobject      mkobj;                /*    80    96 */
            /* --- cacheline 2 boundary (128 bytes) was 48 bytes ago --- */
    $

  now gets expanded:

    $ pahole -E module | head
    struct module {
            enum module_state {
                    MODULE_STATE_LIVE     = 0,
                    MODULE_STATE_COMING   = 1,
                    MODULE_STATE_GOING    = 2,
                    MODULE_STATE_UNFORMED = 3,
            } state; /*     0     4 */
    
            /* XXX 4 bytes hole, try to pack */
    
    $

- Print number of holes, bit holes and bit paddings in class member types.

  Doing this recursively to show how much waste a complex data structure has
  is something that still needs to be done, there were the low hanging fruits
  on the path to having that feature.

  For instance, for 'struct task_struct' in the Linux kernel we get this
  extra info:
  
    --- task_struct.before.c      2024-02-09 11:38:39.249638750 -0300
    +++ task_struct.after.c       2024-02-09 16:19:34.221134835 -0300
    @@ -29,6 +29,12 @@
  
          /* --- cacheline 2 boundary (128 bytes) --- */
          struct sched_entity        se;                   /*   128   256 */
    +
    +     /* XXX last struct has 3 holes */
    +
          /* --- cacheline 6 boundary (384 bytes) --- */
          struct sched_rt_entity     rt;                   /*   384    48 */
          struct sched_dl_entity     dl;                   /*   432   224 */
    +
    +       /* XXX last struct has 1 bit hole */
    +
          /* --- cacheline 10 boundary (640 bytes) was 16 bytes ago --- */
          const struct sched_class  * sched_class;         /*   656     8 */
          struct rb_node             core_node;            /*   664    24 */
    @@ -100,6 +103,9 @@
          /* --- cacheline 35 boundary (2240 bytes) was 16 bytes ago --- */
          struct list_head           tasks;                /*  2256    16 */
          struct plist_node          pushable_tasks;       /*  2272    40 */
    +
    +     /* XXX last struct has 1 hole */
    +
          /* --- cacheline 36 boundary (2304 bytes) was 8 bytes ago --- */
          struct rb_node             pushable_dl_tasks;    /*  2312    24 */
          struct mm_struct *         mm;                   /*  2336     8 */
    @@ -172,6 +178,9 @@
          /* XXX last struct has 4 bytes of padding */
  
          struct vtime               vtime;                /*  2744    48 */
    +
    +     /* XXX last struct has 1 hole */
    +
          /* --- cacheline 43 boundary (2752 bytes) was 40 bytes ago --- */
          atomic_t                   tick_dep_mask;        /*  2792     4 */
  
    @@ -396,9 +405,12 @@
          /* --- cacheline 145 boundary (9280 bytes) --- */
          struct thread_struct       thread __attribute__((__aligned__(64))); /*  9280  4416 */
  
    +       /* XXX last struct has 1 hole, 1 bit hole */
    +
          /* size: 13696, cachelines: 214, members: 262 */
          /* sum members: 13518, holes: 21, sum holes: 162 */
          /* sum bitfield members: 82 bits, bit holes: 2, sum bit holes: 46 bits */
          /* member types with holes: 4, total: 6, bit holes: 2, total: 2 */
          /* paddings: 6, sum paddings: 49 */
          /* forced alignments: 2, forced holes: 2, sum forced holes: 88 */
     };

- Introduce --contains_enumerator=ENUMERATOR_NAME:

  E.g.:

      $ pahole --contains_enumerator S_VERSION
      enum file_time_flags {
             S_ATIME   = 1,
             S_MTIME   = 2,
             S_CTIME   = 4,
             S_VERSION = 8,
      }
      $

  The shorter form --contains_enum is also accepted.

- Fix pretty printing when using DWARF, where sometimes the class (-C) and a specified "type_enum",
  may not be present on the same CU, so wait till both are found.

  Now this example that reads the 'struct perf_event_header' and 'enum perf_event_type'
  from the DWARF info in ~/bin/perf to pretty print records in the perf.data file works
  just like when using type info from BTF in ~/bin/perf:

      $ pahole -F dwarf -V ~/bin/perf \
                --header=perf_file_header \
                --seek_bytes '$header.data.offset' \
                --size_bytes='$header.data.size' \
                -C 'perf_event_header(sizeof,type,type_enum=perf_event_type,filter=type==PERF_RECORD_MMAP2)' \
                --prettify perf.data --count 1
      pahole: sizeof_operator for 'perf_event_header' is 'size'
      pahole: type member for 'perf_event_header' is 'type'
      pahole: type enum for 'perf_event_header' is 'perf_event_type'
      pahole: filter for 'perf_event_header' is 'type==PERF_RECORD_MMAP2'
      pahole: seek bytes evaluated from --seek_bytes=$header.data.offset is 0x3f0
      pahole: size bytes evaluated from --size_bytes=$header.data.size is 0xd10
      // type=perf_event_header, offset=0xc20, sizeof=8, real_sizeof=112
      {
            .header = {
                    .type = PERF_RECORD_MMAP2,
                    .misc = 2,
                    .size = 112,
            },
            .pid = 1533617,
            .tid = 1533617,
            .start = 94667542700032,
            .len = 90112,
            .pgoff = 16384,{
                    .maj = 0,
                    .min = 33,
                    .ino = 35914923,
                    .ino_generation = 26870,
            },{
                    .build_id_size = 0,
                    .__reserved_1 = 0,
                    .__reserved_2 = 0,
                    .build_id = { 33, 0, 0, 0, -85, 4, 36, 2, 0, 0, 0, 0, -10, 104, 0, 0, 0, 0, 0, 0 },
            },
            .prot = 5,
            .flags = 2,
            .filename = "/usr/bin/ls",
      },
      $

DWARF loader:

- Add support for DW_TAG_constant, first seen in Go DWARF.

- Fix loading DW_TAG_subroutine_type generated by the Go compiler, where it may
  have a DW_AT_byte_size. Go DWARF. And pretty print it as if
  it was from C, this helped in writing BPF programs to attach to Go binaries, using
  uprobes.

BTF loader:

- Fix loading of 32-bit signed enums.

BTF encoder:

- Add 'pahole --btf_features' to allow consumers to specify an opt-in set of
  features they want to use in BTF encoding.

  Supported features are a comma-separated combination of
  
          encode_force    Ignore invalid symbols when encoding BTF.
          var             Encode variables using BTF_KIND_VAR in BTF.
          float           Encode floating-point types in BTF.
          decl_tag        Encode declaration tags using BTF_KIND_DECL_TAG.
          type_tag        Encode type tags using BTF_KIND_TYPE_TAG.
          enum64          Encode enum64 values with BTF_KIND_ENUM64.
          optimized_func  Encode representations of optimized functions
                          with suffixes like ".isra.0" etc
          consistent_func Avoid encoding inconsistent static functions.
                          These occur when a parameter is optimized out
                          in some CUs and not others, or when the same
                          function name has inconsistent BTF descriptions
                          in different CUs.
  
  Specifying "--btf_features=all" is the equivalent to setting all of the
  above.  If pahole does not know about a feature specified in
  --btf_features it silently ignores it.
  
  The --btf_features can either be specified via a single comma-separated
  list
          --btf_features=enum64,float
  
  ...or via multiple --btf_features values
  
          --btf_features=enum64 --btf_features=float
  
  These properties allow us to use the --btf_features option in the kernel
  scripts/pahole_flags.sh script to specify the desired set of BTF
  features.
  
  If a feature named in --btf_features is not present in the version of
  pahole used, BTF encoding will not complain.  This is desired because it
  means we no longer have to tie new features to a specific pahole
  version.

  Use --btf_features_strict to change that behaviour and bail out if one of
  the requested features isn't present.

  To see the supported features, use:

    $ pahole --supported_btf_features
    encode_force,var,float,decl_tag,type_tag,enum64,optimized_func,consistent_func
    $

btfdiff:

- Parallelize loading BTF and DWARF, speeding up a bit.

- Do type expansion to cover "private" types and enumerations.

Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

Re: HEADS UP: pahole 1.26 to be released soon

[Jiri Olsa]

On Tue, Feb 27, 2024 at 09:59:14AM -0300, Arnaldo Carvalho de Melo wrote:
> Hi,
> 
> 	I'm prepping up pahole 1.26, the noticeable thing for BTF
> encoding are Alan's patches (--btf_features, --btf_features_strict and
> --supported_btf_features).
> 
> 	There was a fix for BTF loading, namely the signedness of
> enumerations, but that isn't used when building the kernel, just when
> pretty printing from BTF.
> 
> 	I also have a few fixes and new features in the pretty printing
> code, but again, that isn't involved when building the kernel.
> 
> 	If you have time, please do some tests and report results.
> 
> 	You can find it in the usual places:
> 
> git://git.kernel.org/pub/scm/devel/pahole/pahole.git
> https://github.com/acmel/dwarves.git
> 
> 	In the master branch.

hi,
tested on top of my setup and it looks good

I hoped the kfuncs decl tag would get in, but AFAICS there was
no new version of the patches.. I'll ping Daniel about that

thanks,
jirka

> 
> Thanks a lot!
> 
> - Arnaldo

Re: HEADS UP: pahole 1.26 to be released soon

[Arnaldo Carvalho de Melo]

On Wed, Feb 28, 2024 at 09:30:31AM +0100, Jiri Olsa wrote:
> On Tue, Feb 27, 2024 at 09:59:14AM -0300, Arnaldo Carvalho de Melo wrote:
> > 	If you have time, please do some tests and report results.
> > 	You can find it in the usual places:
> > git://git.kernel.org/pub/scm/devel/pahole/pahole.git
> > https://github.com/acmel/dwarves.git
> > 	In the master branch.
 
> hi,
> tested on top of my setup and it looks good

Thanks!
 
> I hoped the kfuncs decl tag would get in, but AFAICS there was
> no new version of the patches.. I'll ping Daniel about that

There is this global variables case that is being discussed, the kfuncs
and I hope to do some work on .dwz, merging what I have at:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=alt_dwarf

So I think I'll go ahead and release 1.26 today and work on merging
these other features to have 1.27 in the near future.

- Arnaldo

Re: HEADS UP: pahole 1.26 to be released soon

[Alan Maguire]

On 28/02/2024 12:34, Arnaldo Carvalho de Melo wrote:
> On Wed, Feb 28, 2024 at 09:30:31AM +0100, Jiri Olsa wrote:
>> On Tue, Feb 27, 2024 at 09:59:14AM -0300, Arnaldo Carvalho de Melo wrote:
>>> 	If you have time, please do some tests and report results.
>>> 	You can find it in the usual places:
>>> git://git.kernel.org/pub/scm/devel/pahole/pahole.git
>>> https://github.com/acmel/dwarves.git
>>> 	In the master branch.
>  
>> hi,
>> tested on top of my setup and it looks good
> 
> Thanks!
>  

Ditto here; tested on x86_64 and aarch64, all looks good.

>> I hoped the kfuncs decl tag would get in, but AFAICS there was
>> no new version of the patches.. I'll ping Daniel about that
> 
> There is this global variables case that is being discussed, the kfuncs
> and I hope to do some work on .dwz, merging what I have at:
> 
> https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=alt_dwarf
> 
> So I think I'll go ahead and release 1.26 today and work on merging
> these other features to have 1.27 in the near future.
> 

Sounds great, thanks! I've been looking into reducing the memory
footprint during pahole operation, but don't have anything ready yet I'm
afraid. I can post a bpf-next patch to switch to using --btf_features
for v1.26 and later; might make Daniel's job a bit easier when adding
the kfunc stuff.

Alan
> - Arnaldo
>