* [PATCH dwarves] pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set
@ 2024-06-26 3:22 Dominique Martinet
2024-06-26 14:29 ` Alan Maguire
0 siblings, 1 reply; 9+ messages in thread
From: Dominique Martinet @ 2024-06-26 3:22 UTC (permalink / raw)
To: acme
Cc: andrii.nakryiko, dwarves, jolsa, williams, kcarcia, dxu, eddyz87,
Dominique Martinet
The SOURCE_DATE_EPOCH environment variable is used in reproducible
builds to set build timestamps and is generally synonymous with
reproducible builds: instead of making users figure out how to pass
pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume
that the user wants a reproducible build if this variable is set.
The impact on build time appears to be minimal enough even if we
incorrectly make this decision (on a Ryzen 7 7735HS):
$ hyperfine 'sh pahole-test -j --reproducible_build' 'sh pahole-test -j1' 'sh pahole-test -j' -p 'cp .tmp_vmlinux.orig .tmp_vmlinux.btf' -m 4
Benchmark 1: sh pahole-test -j --reproducible_build
Time (mean ± σ): 3.991 s ± 0.045 s [User: 7.223 s, System: 3.741 s]
Range (min … max): 3.950 s … 4.042 s 4 runs
Benchmark 2: sh pahole-test -j1
Time (mean ± σ): 7.083 s ± 0.095 s [User: 4.805 s, System: 2.242 s]
Range (min … max): 6.964 s … 7.191 s 4 runs
Benchmark 3: sh pahole-test -j
Time (mean ± σ): 3.858 s ± 0.089 s [User: 13.447 s, System: 7.078 s]
Range (min … max): 3.763 s … 3.978 s 4 runs
Summary
sh pahole-test -j ran
1.03 ± 0.03 times faster than sh pahole-test -j --reproducible_build
1.84 ± 0.05 times faster than sh pahole-test -j1
Where pahole-test is the pahole command of the linux build (minus -j):
LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats --lang_exclude=rust --skip_encoding_btf_inconsistent_proto --btf_gen_optimized .tmp_vmlinux.btf "$@"
And .tmp_vmlinux.orig was the .tmp_vmlinux.btf binary before being
processed:
ld -m elf_x86_64 -z noexecstack --no-warn-rwx-segments --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn --script=./arch/x86/kernel/vmlinux.lds -o .tmp_vmlinux.btf --whole-archive vmlinux.a .vmlinux.export.o init/version-timestamp.o --no-whole-archive --start-group --end-group
---
I opened this as https://github.com/acmel/dwarves/pull/51 a couple of
weeks ago but I guess sending by e-mail is probably more appropriate.
(I didn't find a MAINTAINERS list or similar so sending to recipients of
a random patch I found, please tell me if I missed someone)
(Note that because --btf_feature init is lazy setting something like
--btf_feature=default could override this if we set it too early, so I'm
only checking the env var after argp parsing)
pahole.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/pahole.c b/pahole.c
index 954498d2ad4f..2b010658330c 100644
--- a/pahole.c
+++ b/pahole.c
@@ -3705,6 +3705,10 @@ int main(int argc, char *argv[])
goto out;
}
+ /* This being set means whoever called us tries to do a reproducible build */
+ if (getenv("SOURCE_DATE_EPOCH"))
+ conf_load.reproducible_build = true;
+
if (languages.str && parse_languages())
return rc;
--
2.44.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH dwarves] pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set 2024-06-26 3:22 [PATCH dwarves] pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set Dominique Martinet @ 2024-06-26 14:29 ` Alan Maguire 2024-06-26 23:01 ` Dominique Martinet 0 siblings, 1 reply; 9+ messages in thread From: Alan Maguire @ 2024-06-26 14:29 UTC (permalink / raw) To: Dominique Martinet, acme Cc: andrii.nakryiko, dwarves, jolsa, williams, kcarcia, dxu, eddyz87 On 26/06/2024 04:22, Dominique Martinet wrote: > The SOURCE_DATE_EPOCH environment variable is used in reproducible > builds to set build timestamps and is generally synonymous with > reproducible builds: instead of making users figure out how to pass > pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume > that the user wants a reproducible build if this variable is set. > From the kernel perspective it looks like as a result of [1] the setting of SOURCE_DATE_EPOCH will be converted into KBUILD_BUILD_TIMESTAMP; I thought the plan was to add the BTF feature flag explicitly if KBUILD_BUILD_TIMESTAMP was set? Are you planning for use-cases outside of kernel BTF generation? [1] https://patchwork.kernel.org/project/linux-kbuild/patch/1443741332.2730.75.camel@decadent.org.uk/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH dwarves] pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set 2024-06-26 14:29 ` Alan Maguire @ 2024-06-26 23:01 ` Dominique Martinet 2024-06-27 17:25 ` Alan Maguire 0 siblings, 1 reply; 9+ messages in thread From: Dominique Martinet @ 2024-06-26 23:01 UTC (permalink / raw) To: Alan Maguire Cc: acme, andrii.nakryiko, dwarves, jolsa, williams, kcarcia, dxu, eddyz87 Thanks for the reply! Alan Maguire wrote on Wed, Jun 26, 2024 at 03:29:37PM +0100: > On 26/06/2024 04:22, Dominique Martinet wrote: > > The SOURCE_DATE_EPOCH environment variable is used in reproducible > > builds to set build timestamps and is generally synonymous with > > reproducible builds: instead of making users figure out how to pass > > pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume > > that the user wants a reproducible build if this variable is set. > > From the kernel perspective it looks like as a result of [1] > [1] https://patchwork.kernel.org/project/linux-kbuild/patch/1443741332.2730.75.camel@decadent.org.ukOB/ > the setting of SOURCE_DATE_EPOCH will be converted into > KBUILD_BUILD_TIMESTAMP; I thought the plan was to add the > BTF feature flag explicitly if KBUILD_BUILD_TIMESTAMP was set? Hmm, that patch (2015) never got in apparently?... And I don't see anythiing that'd make pahole output reproducible if either variables are set in the kernel either right now; we still build plenty of older kernels so even if such a patch gets in I'd rather this is checked in pahole than kbuild. For context, the patch I'm replacing right now is this one in nixpkgs (2023): https://github.com/NixOS/nixpkgs/commit/e1daacf987d6064bd7dc65a30e7715990bc1c011 (Where we used to force -j1 if SOURCE_DATE_EPOCH is set, and the new feature makes it more appropriate for upstreaming so I decided to send the patch) > Are you planning for use-cases outside of kernel BTF generation? From a quick check the only users of pahole in nixpkgs and alpine are the linux kernel, so as far as I'm concerned if the linux build is reproducible I'm happy. OTOH I'm not sure we're actually properly setting KBUILD_BUILD_TIMESTAMP properly everywhere (for example external modules), whereas SOURCE_DATE_EPOCH is now universally set by the upper build system -- back in 2015 when that patch was sent the reproducible build efforts just started and SOURCE_DATE_EPOCH wasn't well supported, but now I can't imagine it not being set. So I'd rather keep checking on SOURCE_DATE_EPOCH, but if you're sure there won't ever be any user other than the kernel for pahole's btf generation then I can change the patch to KBUILD_BUILD_TIMESTAMP and we can go hunt users not setting it explicitly our our side. Thanks, -- Dominique Martinet | Asmadeus ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH dwarves] pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set 2024-06-26 23:01 ` Dominique Martinet @ 2024-06-27 17:25 ` Alan Maguire 2024-06-28 8:31 ` Alan Maguire 0 siblings, 1 reply; 9+ messages in thread From: Alan Maguire @ 2024-06-27 17:25 UTC (permalink / raw) To: Dominique Martinet Cc: acme, andrii.nakryiko, dwarves, jolsa, williams, kcarcia, dxu, eddyz87 On 27/06/2024 00:01, Dominique Martinet wrote: > Thanks for the reply! > > Alan Maguire wrote on Wed, Jun 26, 2024 at 03:29:37PM +0100: >> On 26/06/2024 04:22, Dominique Martinet wrote: >>> The SOURCE_DATE_EPOCH environment variable is used in reproducible >>> builds to set build timestamps and is generally synonymous with >>> reproducible builds: instead of making users figure out how to pass >>> pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume >>> that the user wants a reproducible build if this variable is set. >> >> From the kernel perspective it looks like as a result of [1] >> [1] https://patchwork.kernel.org/project/linux-kbuild/patch/1443741332.2730.75.camel@decadent.org.ukOB/ >> the setting of SOURCE_DATE_EPOCH will be converted into >> KBUILD_BUILD_TIMESTAMP; I thought the plan was to add the >> BTF feature flag explicitly if KBUILD_BUILD_TIMESTAMP was set? > > Hmm, that patch (2015) never got in apparently?... > And I don't see anythiing that'd make pahole output reproducible if > either variables are set in the kernel either right now; we still build I can send a patch for this to bpf-next if it's needed. But it'd be good to figure out the right approach first, more below... > plenty of older kernels so even if such a patch gets in I'd rather this > is checked in pahole than kbuild. > I'm not sure that's the right answer though - there's a bunch of existing infrastructure in the kernel tree that relies on KBUILD_BUILD_TIMESTAMP, so to integrate properly with the existing kbuild infrastructure around reproducible builds feels like the right direction to me. It's a bit more work, but not too much. Arnaldo may disagree of course, and he's the maintainer so it's his decision - this is just my personal take. We've tried to explicitly enable flags in pahole Makefile.btf so that it's obvious from pahole invocations which features are wanted, and to get that support on older kernels it would just be a matter of backporting the if > 1.26 use --btf_features ...patch, and a patch for adding the reproducible build option to stable kernels. Similar has been done in the past for other pahole features. All that's needed here for bpf-next tree is I think is the below 3-line addition: diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf index b75f09f3f424..40bb72662967 100644 --- a/scripts/Makefile.btf +++ b/scripts/Makefile.btf @@ -21,6 +21,10 @@ else # Switch to using --btf_features for v1.26 and later. pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs +ifneq ($(KBUILD_BUILD_TIMESTAMP),) +pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=reproducible_build +endif + ifneq ($(KBUILD_EXTMOD),) module-pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=distilled_base endif Can you test at your end if that does the right thing? Thanks! Alan ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH dwarves] pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set 2024-06-27 17:25 ` Alan Maguire @ 2024-06-28 8:31 ` Alan Maguire 2024-06-28 13:59 ` Arnaldo Carvalho de Melo 2024-06-28 19:44 ` Dominique Martinet 0 siblings, 2 replies; 9+ messages in thread From: Alan Maguire @ 2024-06-28 8:31 UTC (permalink / raw) To: Dominique Martinet Cc: acme, andrii.nakryiko, dwarves, jolsa, williams, kcarcia, dxu, eddyz87 On 27/06/2024 18:25, Alan Maguire wrote: > On 27/06/2024 00:01, Dominique Martinet wrote: >> Thanks for the reply! >> >> Alan Maguire wrote on Wed, Jun 26, 2024 at 03:29:37PM +0100: >>> On 26/06/2024 04:22, Dominique Martinet wrote: >>>> The SOURCE_DATE_EPOCH environment variable is used in reproducible >>>> builds to set build timestamps and is generally synonymous with >>>> reproducible builds: instead of making users figure out how to pass >>>> pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume >>>> that the user wants a reproducible build if this variable is set. >>> >>> From the kernel perspective it looks like as a result of [1] >>> [1] https://patchwork.kernel.org/project/linux-kbuild/patch/1443741332.2730.75.camel@decadent.org.ukOB/ >>> the setting of SOURCE_DATE_EPOCH will be converted into >>> KBUILD_BUILD_TIMESTAMP; I thought the plan was to add the >>> BTF feature flag explicitly if KBUILD_BUILD_TIMESTAMP was set? >> >> Hmm, that patch (2015) never got in apparently?... >> And I don't see anythiing that'd make pahole output reproducible if >> either variables are set in the kernel either right now; we still build > > I can send a patch for this to bpf-next if it's needed. But it'd be good > to figure out the right approach first, more below... > >> plenty of older kernels so even if such a patch gets in I'd rather this >> is checked in pahole than kbuild. >> > > I'm not sure that's the right answer though - there's a bunch of > existing infrastructure in the kernel tree that relies on > KBUILD_BUILD_TIMESTAMP, so to integrate properly with the existing > kbuild infrastructure around reproducible builds feels like the right > direction to me. It's a bit more work, but not too much. Arnaldo may > disagree of course, and he's the maintainer so it's his decision - this > is just my personal take. > > We've tried to explicitly enable flags in pahole Makefile.btf so that > it's obvious from pahole invocations which features are wanted, and to > get that support on older kernels it would just be a matter of > backporting the > > if > 1.26 > use --btf_features > > ...patch, and a patch for adding the reproducible build option to stable > kernels. Similar has been done in the past for other pahole features. > > All that's needed here for bpf-next tree is I think is the below 3-line > addition: > > diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf > index b75f09f3f424..40bb72662967 100644 > --- a/scripts/Makefile.btf > +++ b/scripts/Makefile.btf > @@ -21,6 +21,10 @@ else > # Switch to using --btf_features for v1.26 and later. > pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j > --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs > > +ifneq ($(KBUILD_BUILD_TIMESTAMP),) > +pahole-flags-$(call test-ge, $(pahole-ver), 126) += > --btf_features=reproducible_build > +endif > + > ifneq ($(KBUILD_EXTMOD),) > module-pahole-flags-$(call test-ge, $(pahole-ver), 126) += > --btf_features=distilled_base > endif > > > Can you test at your end if that does the right thing? Thanks! > > Alan > FWIW I verified that with the above change, and export KBUILD_BUILD_TIMESTAMP=$(date -I) ...the feature flag is added to pahole and as a result we get reproducible BTF in vmlinux; .btf.vmlinux.bin.o is identical across multiple kernel builds. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH dwarves] pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set 2024-06-28 8:31 ` Alan Maguire @ 2024-06-28 13:59 ` Arnaldo Carvalho de Melo 2024-06-28 19:44 ` Dominique Martinet 1 sibling, 0 replies; 9+ messages in thread From: Arnaldo Carvalho de Melo @ 2024-06-28 13:59 UTC (permalink / raw) To: Alan Maguire Cc: Dominique Martinet, andrii.nakryiko, dwarves, jolsa, williams, kcarcia, dxu, eddyz87 On Fri, Jun 28, 2024 at 09:31:17AM +0100, Alan Maguire wrote: > On 27/06/2024 18:25, Alan Maguire wrote: > > On 27/06/2024 00:01, Dominique Martinet wrote: > >> Alan Maguire wrote on Wed, Jun 26, 2024 at 03:29:37PM +0100: > >>> On 26/06/2024 04:22, Dominique Martinet wrote: > >>>> The SOURCE_DATE_EPOCH environment variable is used in reproducible > >>>> builds to set build timestamps and is generally synonymous with > >>>> reproducible builds: instead of making users figure out how to pass > >>>> pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume > >>>> that the user wants a reproducible build if this variable is set. > >>> From the kernel perspective it looks like as a result of [1] > >>> [1] https://patchwork.kernel.org/project/linux-kbuild/patch/1443741332.2730.75.camel@decadent.org.ukOB/ > >>> the setting of SOURCE_DATE_EPOCH will be converted into > >>> KBUILD_BUILD_TIMESTAMP; I thought the plan was to add the > >>> BTF feature flag explicitly if KBUILD_BUILD_TIMESTAMP was set? > >> Hmm, that patch (2015) never got in apparently?... > >> And I don't see anythiing that'd make pahole output reproducible if > >> either variables are set in the kernel either right now; we still build > > > > I can send a patch for this to bpf-next if it's needed. But it'd be good > > to figure out the right approach first, more below... > > > >> plenty of older kernels so even if such a patch gets in I'd rather this > >> is checked in pahole than kbuild. > > I'm not sure that's the right answer though - there's a bunch of > > existing infrastructure in the kernel tree that relies on > > KBUILD_BUILD_TIMESTAMP, so to integrate properly with the existing > > kbuild infrastructure around reproducible builds feels like the right > > direction to me. It's a bit more work, but not too much. Arnaldo may > > disagree of course, and he's the maintainer so it's his decision - this > > is just my personal take. > > > > We've tried to explicitly enable flags in pahole Makefile.btf so that > > it's obvious from pahole invocations which features are wanted, and to > > get that support on older kernels it would just be a matter of > > backporting the > > > > if > 1.26 > > use --btf_features > > > > ...patch, and a patch for adding the reproducible build option to stable > > kernels. Similar has been done in the past for other pahole features. > > > > All that's needed here for bpf-next tree is I think is the below 3-line > > addition: > > > > diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf > > index b75f09f3f424..40bb72662967 100644 > > --- a/scripts/Makefile.btf > > +++ b/scripts/Makefile.btf > > @@ -21,6 +21,10 @@ else > > # Switch to using --btf_features for v1.26 and later. > > pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j > > --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs > > > > +ifneq ($(KBUILD_BUILD_TIMESTAMP),) > > +pahole-flags-$(call test-ge, $(pahole-ver), 126) += > > --btf_features=reproducible_build > > +endif > > + > > ifneq ($(KBUILD_EXTMOD),) > > module-pahole-flags-$(call test-ge, $(pahole-ver), 126) += > > --btf_features=distilled_base > > endif > > > > > > Can you test at your end if that does the right thing? Thanks! > > FWIW I verified that with the above change, and > > export KBUILD_BUILD_TIMESTAMP=$(date -I) > > ...the feature flag is added to pahole and as a result we get > reproducible BTF in vmlinux; .btf.vmlinux.bin.o is identical across > multiple kernel builds. This sounds reasonable, pahole doesn't have to look at something so specific for a particular use case (the kernel) while using what pahole would look (KBUILD_BUILD_TIMESTAMP) but in the kernel build scripts, that in turn uses the prefered, self-documenting way of asking for some specific BTF feature (--btf_features). - Arnaldo ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH dwarves] pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set 2024-06-28 8:31 ` Alan Maguire 2024-06-28 13:59 ` Arnaldo Carvalho de Melo @ 2024-06-28 19:44 ` Dominique Martinet 2024-07-01 16:44 ` Alan Maguire 1 sibling, 1 reply; 9+ messages in thread From: Dominique Martinet @ 2024-06-28 19:44 UTC (permalink / raw) To: Alan Maguire Cc: acme, andrii.nakryiko, dwarves, jolsa, williams, kcarcia, dxu, eddyz87 Alan Maguire wrote on Fri, Jun 28, 2024 at 09:31:17AM +0100: > FWIW I verified that with the above change, and > > export KBUILD_BUILD_TIMESTAMP=$(date -I) > > ...the feature flag is added to pahole and as a result we get > reproducible BTF in vmlinux; .btf.vmlinux.bin.o is identical across > multiple kernel builds. Sorry for the lack of reply, I didn't have time to test until now, and I have no idea how to override the kernel config in nixos so the checking build through nix build took all night. I've now also confirmed this works including for modules BTF & linux "package" build on nixos on master. However from there I built a random kernel module and have also confirmed they don't set KBUILD_BUILD_TIMESTAMP for external modules -- from a quick look it's not part of the default flags defined as buildFlags here[1] because each modules' SOURCE_DATE_EPOCH isn't available at this level (and I have no idea how to call a command at this level), the configurePhase where it's set only applies to the kernel itself and not the gazillion of modules. [1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/kernel/manual-config.nix Add to that that nixos is supporting a dozen of kernels -- all mainlines stable versions but also a dozen of variants (rt, whatever xanmod and zen are, and if you look for things like mobile-nixos you'll also get much older and not update sunxi and other embedded trees) I definitely won't put the effort to backport equivalent patches to all these kernels I wanted the patch for alpine as well but I honestly don't have the time to do the same confirmation work there; if they have any external modules I don't see the flag being set either, at least they don't have so many kernels. (but unlike nixos the kernel build "framework" code isn't shared and I know of at least one out of tree embedded kernel that doesn't set KBUILD_BUILD_TIMESTAMP and only has SOURCE_DATE_EPOCH...) So: - in theory, sure; this patch works. I say we can do this independantly of any other effort. The propere way of setting reproducible build in the kernel is not SOURCE_DATE_EPOCH but KBUILD_BUILD_TIMESTAMP and handling it at kernel makefile level is sound. - in practice, (nixos hat) some modules won't set it but will only have SOURCE_DATE_EPOCH available, and there are trees that won't be updated for years (and that's being optimistic), so our local patch in pahole is here to stay for a while and the benefits far outweight the rebasing work. - (alpine hat) alpine doesn't seem to care as much about reproducibility (the nixos patch wasn't in and nobody complained so far), undecided at this point. Makefile.btf patch is probably good enough, but since I'm maintaining the patch anyway it's no extra work to add it there, undecided at this point. Thanks, -- Dominique Martinet | Asmadeus ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH dwarves] pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set 2024-06-28 19:44 ` Dominique Martinet @ 2024-07-01 16:44 ` Alan Maguire 2024-07-01 23:35 ` Dominique Martinet 0 siblings, 1 reply; 9+ messages in thread From: Alan Maguire @ 2024-07-01 16:44 UTC (permalink / raw) To: Dominique Martinet Cc: acme, andrii.nakryiko, dwarves, jolsa, williams, kcarcia, dxu, eddyz87 On 28/06/2024 20:44, Dominique Martinet wrote: > Alan Maguire wrote on Fri, Jun 28, 2024 at 09:31:17AM +0100: >> FWIW I verified that with the above change, and >> >> export KBUILD_BUILD_TIMESTAMP=$(date -I) >> >> ...the feature flag is added to pahole and as a result we get >> reproducible BTF in vmlinux; .btf.vmlinux.bin.o is identical across >> multiple kernel builds. > > Sorry for the lack of reply, I didn't have time to test until now, and I > have no idea how to override the kernel config in nixos so the checking > build through nix build took all night. > I've now also confirmed this works including for modules BTF & linux > "package" build on nixos on master. > > However from there I built a random kernel module and have also > confirmed they don't set KBUILD_BUILD_TIMESTAMP for external modules -- > from a quick look it's not part of the default flags defined as > buildFlags here[1] because each modules' SOURCE_DATE_EPOCH isn't > available at this level (and I have no idea how to call a command at > this level), the configurePhase where it's set only applies to the > kernel itself and not the gazillion of modules. > [1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/kernel/manual-config.nix > > Add to that that nixos is supporting a dozen of kernels -- all mainlines > stable versions but also a dozen of variants (rt, whatever xanmod and > zen are, and if you look for things like mobile-nixos you'll also get > much older and not update sunxi and other embedded trees) I definitely > won't put the effort to backport equivalent patches to all these kernels > > > I wanted the patch for alpine as well but I honestly don't have the time > to do the same confirmation work there; if they have any external > modules I don't see the flag being set either, at least they don't have > so many kernels. > (but unlike nixos the kernel build "framework" code isn't shared and I > know of at least one out of tree embedded kernel that doesn't set > KBUILD_BUILD_TIMESTAMP and only has SOURCE_DATE_EPOCH...) > > > So: > - in theory, sure; this patch works. I say we can do this independantly > of any other effort. The propere way of setting reproducible build in > the kernel is not SOURCE_DATE_EPOCH but KBUILD_BUILD_TIMESTAMP and > handling it at kernel makefile level is sound. Okay, so I propose we try and get the KBUILD_BUILD_TIMESTAMP-based change upstream. I'll send that later today. That doesn't mean we can't add the envvar-based method too, it just means we have something upstream that handles reproducible builds more explicitly. > - in practice, (nixos hat) some modules won't set it but will only have > SOURCE_DATE_EPOCH available, and there are trees that won't be updated > for years (and that's being optimistic), so our local patch in pahole is > here to stay for a while and the benefits far outweight the rebasing > work. > - (alpine hat) alpine doesn't seem to care as much about > reproducibility (the nixos patch wasn't in and nobody complained so far), > undecided at this point. Makefile.btf patch is probably good enough, but > since I'm maintaining the patch anyway it's no extra work to add it > there, undecided at this point. > > One thing I think would be worth doing in your patch would be adding debug logging (if in verbose mode) that we are enabling reproducible builds due to the presence of the SOURCE_DATE_EPOCH envvar. I don't think we can do this in the non-debug case because build logs would just be flooded with a few thousand of these messages for each module we add BTF to using pahole. With that small addition I'd be happy with the change. Can you send a v2 with that added? Thanks! Alan > Thanks, ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH dwarves] pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set 2024-07-01 16:44 ` Alan Maguire @ 2024-07-01 23:35 ` Dominique Martinet 0 siblings, 0 replies; 9+ messages in thread From: Dominique Martinet @ 2024-07-01 23:35 UTC (permalink / raw) To: Alan Maguire Cc: acme, andrii.nakryiko, dwarves, jolsa, williams, kcarcia, dxu, eddyz87 Alan Maguire wrote on Mon, Jul 01, 2024 at 05:44:15PM +0100: > > - in theory, sure; this patch works. I say we can do this independantly > > of any other effort. The propere way of setting reproducible build in > > the kernel is not SOURCE_DATE_EPOCH but KBUILD_BUILD_TIMESTAMP and > > handling it at kernel makefile level is sound. > > Okay, so I propose we try and get the KBUILD_BUILD_TIMESTAMP-based > change upstream. I'll send that later today. That doesn't mean we can't > add the envvar-based method too, it just means we have something > upstream that handles reproducible builds more explicitly. Thanks. > > - in practice, (nixos hat) some modules won't set it but will only have > > SOURCE_DATE_EPOCH available, and there are trees that won't be updated > > for years (and that's being optimistic), so our local patch in pahole is > > here to stay for a while and the benefits far outweight the rebasing > > work. > > - (alpine hat) alpine doesn't seem to care as much about > > reproducibility (the nixos patch wasn't in and nobody complained so far), > > undecided at this point. Makefile.btf patch is probably good enough, but > > since I'm maintaining the patch anyway it's no extra work to add it > > there, undecided at this point. > > > > > > One thing I think would be worth doing in your patch would be adding > debug logging (if in verbose mode) that we are enabling reproducible > builds due to the presence of the SOURCE_DATE_EPOCH envvar. I don't > think we can do this in the non-debug case because build logs would just > be flooded with a few thousand of these messages for each module we add > BTF to using pahole. With that small addition I'd be happy with the > change. Can you send a v2 with that added? Thanks! That makes sense, I'll send a v2 with a debug log message tonight/tomorrow -- Dominique Martinet | Asmadeus ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2024-07-01 23:36 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-06-26 3:22 [PATCH dwarves] pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set Dominique Martinet 2024-06-26 14:29 ` Alan Maguire 2024-06-26 23:01 ` Dominique Martinet 2024-06-27 17:25 ` Alan Maguire 2024-06-28 8:31 ` Alan Maguire 2024-06-28 13:59 ` Arnaldo Carvalho de Melo 2024-06-28 19:44 ` Dominique Martinet 2024-07-01 16:44 ` Alan Maguire 2024-07-01 23:35 ` Dominique Martinet
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.