Re: Proof-of-concept GCC/GNU ld toolchain-driven BTF generation/deduplication

[Nick Alcock <nick.alcock@oracle.com>]

On 8 Jun 2025, Sam James outgrape:

> Thanks for this, I'm really excited to see it. The disk space (and time
> needed) for DWARF has been a problem for us especially as a source-based
> distribution because we can't assume all builds are done on some beefy
> centralised builder ;)

That probably affects userspace, mostly, since most packages are
userspace packages -- but then, almost all the remaining differences
between BTF and CTFv4 are driven by the needs of userspace
usage.

Ideally I want it to be fast enough that it can be used for real-time
ABI checking in places like ld.so (I have an improvement coming up
suggested by Dodji that might reduce the time needed for such checks of
public symbol type-compat to nearly unit time with minimal space
reqirements).

> Have you got a list of what's left to do and also what you need from the
> community? Or is this a preview of what you're working on and not
> expecting much discussion yet?

Oh, discussion is welcome! If this whole thing is a disaster that nobody
will ever accept, I want to know *now* before I spend more time on it.


My todo list before I try for upstreaming this immensity is here:
<https://sourceware.org/binutils/wiki/CTF/Todo/CTFv4>.  Most of the
really big stuff on that list is done, but there's some things that must
be done for robustness, some that are known regressions from CTFv3, one
that is critical for anything at all (back-compatibility reading) and
some things that we should do anyway as long as we're changing the
format and libctf API and ABI like this. The API breaks are small, but
they're there...

(None of these changes affect anything in the BTF subset, so all the
stuff the kernel build process needs should just keep working
unchanged.)

It's probably three weeks' to a month's work, I guess.

What I need from the community?

Once back-compat reading is in place and the libctf testsuite starts
working properly again, I'll have to start seeing about getting people
to review the libctf series before upstreaming into binutils, and that's
a big thing to ask of anyone. It's cut into smallish pieces but it's
still 12,000 lines or so... not quite ready for that yet though.

A smaller but perhaps harder and more important job: the public API as a
whole could do with review -- it's possible there are changes that don't
require existing users to make more than small mods, but that *also*
make it much simpler and/or more expressive... also, we've added a lot
of new functions in this series, and I'd like to make sure they make
some sort of coherent sense.  API review from unbiased external API
users (or want-to-be-users) is probably the most important thing people
could do to help here, really.  The public API of the new release is
here:

https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=include/ctf-api.h;;hb=refs/heads/users/nalcock/road-to-ctfv4

It's big, but much of the size is CTF/BTF dict creation or linking stuff
that can be ignored if you're only a consumer. However, the query side
is still quite big, and some of the function names around things like
symbol lookup are... not good (speaking as the guy who made them
up). Improved names and/or better ways to do this by combining some of
these functions into one would be nice :)

An .org file briefly listing all the changes is here:

https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=libctf/api.org;;hb=refs/heads/users/nalcock/road-to-ctfv4