[PATCH dwarves v3] pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set

[PATCH dwarves v3] pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set

[Dominique Martinet]

The SOURCE_DATE_EPOCH environment variable is used in reproducible
builds to set build timestamps and is generally synonymous with
reproducible builds: instead of making users figure out how to pass
pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume
that the user wants a reproducible build if this variable is set,
printing a debug message in verbose mode.

The impact on build time appears to be minimal enough even if we
incorrectly make this decision (on a Ryzen 7 7735HS):
$ hyperfine 'sh pahole-test -j --reproducible_build' 'sh pahole-test -j1' 'sh pahole-test -j' -p 'cp .tmp_vmlinux.orig .tmp_vmlinux.btf' -m 4
Benchmark 1: sh pahole-test -j --reproducible_build
  Time (mean ± σ):      3.991 s ±  0.045 s    [User: 7.223 s, System: 3.741 s]
  Range (min … max):    3.950 s …  4.042 s    4 runs

Benchmark 2: sh pahole-test -j1
  Time (mean ± σ):      7.083 s ±  0.095 s    [User: 4.805 s, System: 2.242 s]
  Range (min … max):    6.964 s …  7.191 s    4 runs

Benchmark 3: sh pahole-test -j
  Time (mean ± σ):      3.858 s ±  0.089 s    [User: 13.447 s, System: 7.078 s]
  Range (min … max):    3.763 s …  3.978 s    4 runs

Summary
  sh pahole-test -j ran
    1.03 ± 0.03 times faster than sh pahole-test -j --reproducible_build
    1.84 ± 0.05 times faster than sh pahole-test -j1

Where pahole-test is the pahole command of the linux build (minus -j):
LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats --lang_exclude=rust --skip_encoding_btf_inconsistent_proto --btf_gen_optimized .tmp_vmlinux.btf "$@"

And .tmp_vmlinux.orig was the .tmp_vmlinux.btf binary before being
processed:
ld -m elf_x86_64 -z noexecstack --no-warn-rwx-segments --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn --script=./arch/x86/kernel/vmlinux.lds -o .tmp_vmlinux.btf --whole-archive vmlinux.a .vmlinux.export.o init/version-timestamp.o --no-whole-archive --start-group --end-group

Signed-off-by: Dominique Martinet <asmadeus@codewreck.org>
---
v2 -> v3:
 - added message about the env var to man page.
 - fixed typo / style of comment in code

v1 -> v2:
 - added message in verbose mode if setting reproducible build through
   this.

 man-pages/pahole.1 |  3 +++
 pahole.c           | 11 +++++++++++
 2 files changed, 14 insertions(+)

diff --git a/man-pages/pahole.1 b/man-pages/pahole.1
index f0605935a9f1..d9c2c1ac6912 100644
--- a/man-pages/pahole.1
+++ b/man-pages/pahole.1
@@ -327,6 +327,9 @@ If one wants to add an extra feature to the set of standard ones, the '+' prefix
 \-\-btf_features=+reproducible_build will add all default features plus the 'reproducible_build' extra
 feature.
 
+Alternatively, reproducible_build will also be enabled if the SOURCE_DATE_EPOCH environment variable
+is set, as that is commonly set in build systems that aim for reproducible builds.
+
 .TP
 .B \-\-btf_features_strict
 Identical to \-\-btf_features above, but pahole will exit if it encounters an unrecognized feature.
diff --git a/pahole.c b/pahole.c
index c866095c8be6..7724980ff106 100644
--- a/pahole.c
+++ b/pahole.c
@@ -3705,6 +3705,17 @@ int main(int argc, char *argv[])
 		goto out;
 	}
 
+	/*
+	 * SOURCE_DATE_EPOCH being set means whoever called us likely tries
+	 * to do a reproducible build. The setting only impacts btf_encode code
+	 * so only check in this case to avoid needless messages.
+	 */
+	if (btf_encode && getenv("SOURCE_DATE_EPOCH")) {
+		if (!conf_load.reproducible_build && global_verbose)
+			printf("pahole: auto-enabling reproducible build (SOURCE_DATE_EPOCH set)\n");
+		conf_load.reproducible_build = true;
+	}
+
 	if (languages.str && parse_languages())
 		return rc;
 
-- 
2.45.1

Re: [PATCH dwarves v3] pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set

[Dominique Martinet]

Thanks for the reply!

This can wait until you're back to work, no worry about latency.

Arnaldo Carvalho de Melo wrote on Sun, Jul 07, 2024 at 11:12:06AM -0300:
> I'm in vacation, but it occurred to me that a
> 
> --set_from_env SOURCE_DATE_EPOCH=reproducible_build
> 
> Meaning: if variable SOURCE_DATE_EPOCH
>  Is set, set the boolean command line reproducible_build.

(I understood this as a new command line option to set any btf feature
from env, if the above option is passed to pahole)

From my position there are two faces to this:
- for new kernels, a change in Kbuild to add som flag would definitely
work.
The patch to set --btf_features=reproducible_build to pahole from Kbuild
if KBUILD_BUILD_TIMESTAMP is set got shot down by Masahiro Yamada, but
hopefully something will be worked out eventually...

- for nixos/old kernels, the current pahole version produces
reproducible output if SOURCE_DATE_EPOCH is set (because someone patched
it to default to -j1 in this case in our tree), and now the reproducible
build option exists I'd like to upstream something more appropriate
that'll "just work" without worrying about old kernel trees and adding
new explicit pahole arguments there.

So adding a new switch to set a flag from env probably won't help me
much, but a new env var like PAHOLE_BTF_FEATURES (PAHOLE_FLAGS is
used/overwritten by Kbuild...) that would just call into the
--btf-feature parsing code, e.g. exporting
PAHOLE_BTF_FEATURES=+reproducible_build would toggle the option) would
probably work for me.
(Have to be careful that this is not the same as --reproducible-build,
so perhaps something else might be better... but regardless it'll
probably work out for linux kernel builds)

It's less convenient than having SOURCE_DATE_EPOCH just "make it work",
but at least there'll be a single place to edit that'll work on all
older kernels for a given distro.
(and the "just work" side of things might come for newer kernels at some
point for distros that care less...)

What do you think?
-- 
Dominique Martinet | Asmadeus