Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Tony Ambardar <tony.ambardar@gmail.com>]

On Mon, Jun 03, 2024 at 08:47:24PM -0700, Tony Ambardar wrote:
> Hi Mark,
> 
> On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
> > On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
> > > Couldn't find a way to ask eu-readelf for more verbose output, where we
> > > could perhaps get some clue as to why it produces nothing while binutils
> > > readelf manages to grok it, Mark, do you know some other way to ask
> > > eu-readelf to produce more debug output?
> > > 
> > > I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
> > > that then made eu-readelf to not be able to process it while pahole,
> > > that uses eltuils' libraries, was able to process the first two CUs for
> > > a kernel module and all the CUs for the vmlinux file :-\
> > > 
> > > Mark, the whole thread is available at:
> > > 
> > > https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
> > 
> > I haven't looked at the vmlinux file. But for the .ko file the issue
> > is that the elfutils MIPS backend isn't complete. Specifically MIPS
> > relocations aren't recognized (and so cannot be applied). There are
> > some pending patches which try to fix that:
> > 
> > https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> 
> Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
> work for MIPS, and I locally rebuilt elfutils and then pahole from their
> respective next/main branches. For elfutils, main (935ee131cf7c) includes
> 
>   e259f126 Support Mips architecture
>   f2acb069 stack: Fix stack unwind failure on mips
>   db33cb0c backends: Add register_info, return_value_location, core_note mips
> 
> which partially applies the patchwork series but leaves out the support for
> readelf, strip, and elflint.
> 
> I believe this means the vmlinux and .ko files I shared are OK, or is there
> more backend work needed for MIPS?
> 
> The bits missing in eu-readelf would explain the blank output both Arnaldo
> and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
> patchwork readelf patch locally but ran into merge conflicts.
> 
> CCing Ying Huang for any more insight.
> 
> Thanks,
> Tony

Hello all,

A short update, starting with answering my own question.

No, apparently the above commits *do not* complete the backend work. Ying
Huang submitted additional related patches since March 5: [1][2]

    strip: Adapt src/strip -o -f on mips
    readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
    elflint: adapt src/elflint --gnu src/nm on mips
    test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh

Despite the titles, these patches do include core backend changes for MIPS.
I resolved the various merge conflicts [3], rebuilt elfutils, and retested
kernel builds to now find:

  - pahole is able to read DWARF[45] info and create .BTF for modules
  - resolve_btfids can successfully patch .BTF_ids in modules
  - kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)

Huzzah!


Ying:

Thank you for developing these MIPS patches. In your view, are the MIPS
changes now complete, or do you plan further updates that might improve or
impact parsing DWARF debug/reloc info in apps like pahole?


Mark:

Given that BTF usage on Linux/MIPS is basically broken without these
patches, could I request some of your review time for them to be merged? If
it's helpful, my branch [3] includes all patches with conflicts fixed, and
I also successfully ran the elfutils self-tests (including MIPS from Ying).
Please feel free to add for these patches:

    Tested-by: Tony Ambardar <Tony.Ambardar@gmail.com>


Arnaldo:

Your stepping through DWARF/reloc diagnostics earlier was helpful. Thanks!
I reran your tests with the updated elfutils and latest pahole (pre-1.27),
and then found:

  - everything that worked before, still works
  - your observations from "btfdiff vmlinux" and 'struct dma_chan' persist
  - we now see expected output from "eu-readelf -winfo netdevsim.ko"

Regarding pahole, DWARF parsing and BTF generation now works:
(with no more die__process: error messages seen)

    kodidev:~/linux$ pahole -F dwarf netdevsim.ko |wc -l
    14504

but strangely pahole still doesn't read its own generated BTF:

    kodidev:~/linux$ pahole -F btf netdevsim.ko
    libbpf: Invalid BTF string section
    pahole: file 'netdevsim.ko' has no btf type information.

Poking inside a little further:
    
    kodidev:~/linux$ ltrace -S pahole -F btf netdevsim.ko
    [...]
    argp_parse(0x563d47da42a0, 4, 0x7ffd5e552698, 0 <unfinished ...>
    SYS_318(0x7fc385bf84d8, 8, 1, 0x7fc385ce9908)    = 8
    SYS_brk(0)                                       = 0x563d47e37000
    SYS_brk(0x563d47e58000)                          = 0x563d47e58000
    <... argp_parse resumed> )                       = 0
    dwarves__init(0x20000, 0, -4096, 213)            = 0
    dwarves__resolve_cacheline_size(0x563d47da40c0, 0, 24, 213) = 64
    cus__new(5, 0, 64, 213)                          = 0x563d47e372a0
    memset(0x563d47da44c0, ' ', 127)                 = 0x563d47da44c0
    strlen("/sys/kernel/btf/")                       = 16
    strncmp("netdevsim.ko", "/sys/kernel/btf/", 16)  = 63
    cus__load_files(0x563d47e372a0, 0x563d47da40c0, 0x7ffd5e5526b0,
    0x563d47da40c0 <unfinished ...>
    SYS_openat(0xffffff9c, 0x7ffd5e554603, 0x80000, 0) = 3
    SYS_newfstatat(3, 0x7fc385baf44f, 0x7ffd5e552210, 4096) = 0
    SYS_read(3, "\177ELF\002\001\001", 4096)         = 4096
    SYS_close(3)                                     = 0
    SYS_openat(0xffffff9c, 0x7ffd5e554603, 0x80000, 0) = 3
    SYS_fcntl(3, 1, 0, 0x7fc3859de2f0)               = 1
    SYS_newfstatat(3, 0x7fc385baf44f, 0x7ffd5e5521f0, 4096) = 0
    SYS_pread(3, 0x7ffd5e5521f0, 64, 0)              = 64
    SYS_pread(3, 0x563d47e3e470, 3520, 0x4f60c8)     = 3520
    SYS_pread(3, 0x563d47e3f240, 525, 0x4f5eb8)      = 525
    SYS_pread(3, 0x563d47e3f460, 0x45dd, 0x2b1fb6)   = 0x45dd
    SYS_write(2, "libbpf: Invalid BTF string secti"..., 35libbpf: Invalid BTF
    string section
    ) = 35
    SYS_close(3)                                     = 0
    <... cus__load_files resumed> )                  = 0xffffffff
    access("netdevsim.ko", 4 <unfinished ...>
    SYS_access("netdevsim.ko", 04)                   = 0
    <... access resumed> )                           = 0
    fprintf(0x7fc385bf26a0, "pahole: file '%s' has no %s type"...,
    "netdevsim.ko", "btf" <unfinished ...>
    SYS_write(2, "pahole: file 'netdevsim.ko' has "..., 57pahole: file
    'netdevsim.ko' has no btf type information.
    ) = 57
    <... fprintf resumed> )                          = 57
    SYS_exit_group(1 <no return ...>
    +++ exited (status 1) +++

Could you help investigate this further? Maybe a libbpf issue? For the
record, I also tried building pahole with embedded libbpf 1.4.3 without
any change. (side note: please make pahole --version also cover libbpf)


Many thanks everyone for your help,
Tony

[1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
[2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
[3]:
https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/