Re: [RFC 0/4] BTF archive with unmodified pahole+toolchain

[Nick Alcock <nick.alcock@oracle.com>]

On 27 Aug 2025, Alexei Starovoitov stated:

> On Thu, Aug 21, 2025 at 2:35 PM Nick Alcock <nick.alcock@oracle.com> wrote:
>>
>> >>I'd like to second Alexei's question.
>> >>In the cover letter Arnaldo points out that un-deduplicated BTF
>> >>amounts for 325Mb, while total DWARF size is 365Mb.
>>
>> That very much depends on the kernels you build. In my tests of
>> enterprise kernels (including modules) with the GCC+btfarchive toolchain
>> (not feeding it to pahole yet), I found total DWARF of 11.2GiB,
>> undeduplicated BTF of 550MiB (counting raw .o compiler output alone),
>> and a final dedupicated BTF size (including all modules) of about 38MiB
>> (which I'm sure I can reduce).
>
> 11.2G doesn't match Arnaldo's 365Mb.
> Frankly I've never seen such huge dwarf objects.

I have, but... it was a while back. I shouldn't have worked from memory.

Regenerating with a more recent toolchain, summing up all written
section sizes (so, undeduplicated .BTF compiler output *and* all the
deduplicated module intermediate links) I usually see DWARF sizes about
two to three times that of the .BTF (e.g. the BTF selftest is about
800MiB versus about 400MiB of BTF: the final BTF size from both
btfarchive and pahole is on the order of 2MiB).

Using a random enterprise kernel config (so 2900+ modules, etc), I see
4072236343 bytes of DWARF, 2199803264 bytes of undeduplicated .BTF
sections: so, again, about 50% reduction.

(toolchain-level dedup on this one takes two minutes and peaks at 5GiB
memory usage, producing a 40MiB BTF archive: I know this output can be
greatly reduced by a fix I'm planning shortly. :) )

> I'm guessing you're using some ultra verbose dwarf compilation
> mode. If so, it's not a realistic comparison, since typical
> kernel build is what Arnaldo reported.
> That's what I observe as well.
>
>> >>The size of DWARF sections in the final vmlinux is comparable to yours: 307Mb.
>> >>The total size of the generated binaries is 905Mb.

Ditto, now. I dont know what weirdo config I was using before (I suspect
it was just an older GCC with a different default DWARF version, and
this is simply DWARF 2/3 versus 5). It's still a nontrivial saving.

>> GNU ld), despite being single-threaded and doing things like ambiguous
>> type detection as well, used 12GiB and took 19 minutes. (Multithreading
>> it is in progress, too). allyesconfig is faster. Anything sane is faster
>> yet. Enterprise kernels take about four minutes, which is not too
>> different from pahole.
>>
>> I was shocked by this: I thought libctf would be slower than pahole, and
>> instead it turned out to be faster, sometimes much faster. I suspect
>> much of this frankly ridiculous difference was DWARF conversion, and so
>> would be improved by doing it in parallel (as here), but... still. Not
>> having to generate and consume all that DWARF is bound to help! It's
>> like 95% less work...
>
> Something doesn't add up here.
> Everyone is using pahole and lots of people doing allmodconfig builds
> with pahole. Noone reported that pahole consumes 70G and runs for hours.
> Something is really not right in your setup.

Well... yeah, that would be the make allmodconfig / allyesconfig
configuration options. pahole takes more reasonable times with more
reasonable configurations, but still ten minutes or more is fairly
routine for me.

> Pls use typical kernel build configs then we can have apple to apple
> comparison and reason about libctf pros/cons.

I'm not sure there is such a thing as typical, really. I hope random
enterprise configs will do, but they probably have more modules than
"normal" and God knows the BTF test configs have fewer :)

-- 
NULL && (void)