pahole -J non-determinism and reproducible builds

pahole -J non-determinism and reproducible builds

[Giuliano Procida]

Hi.

(edited repost due to unfortunate HTML email).

Until the reproducible BTF changes land in a new pahole version, and for
anyone stuck on older versions, pahole -J -j typically produces
non-deterministic BTF.

For Android, we are converging to hermetic, reproducible builds. .BTF
sections are a significant difference we see. I have trivial patches to
pass -j1 instead of -j for the older versions we are using (1.23 and 1.25).

Trivial patches:
https://android-review.googlesource.com/q/Ibd72ac638faa1826f6655b336cc7001591ea70f1

Gaining determinism for the older versions means giving up some performance.
About a factor of 2.1 in our case (or an extra 10s on vmlinux).

Do others want performance or determinism, should I send patches to LKML
and the stable list?

Off-topic: pahole -J -j9 is faster than -j on my machine with 36
cores, by another
factor of 2. This is v1.25. -j1 18.7s, -j9 4.5s, -j 8.9s.

Regards,
Giuliano.

Re: pahole -J non-determinism and reproducible builds

[Arnaldo Carvalho de Melo]

On Thu, May 23, 2024 at 08:25:17PM +0100, Giuliano Procida wrote:
> Hi.
> 
> (edited repost due to unfortunate HTML email).
> 
> Until the reproducible BTF changes land in a new pahole version, and for
> anyone stuck on older versions, pahole -J -j typically produces
> non-deterministic BTF.
> 
> For Android, we are converging to hermetic, reproducible builds. .BTF
> sections are a significant difference we see. I have trivial patches to
> pass -j1 instead of -j for the older versions we are using (1.23 and 1.25).
> 
> Trivial patches:
> https://android-review.googlesource.com/q/Ibd72ac638faa1826f6655b336cc7001591ea70f1

That should be at least configurable thru a CONFIG_REPRODUCIBLE_BUILD
that then would check the pahole version and if less than 1.27 then
would use -j1

So it continues to use -j from the version where it was introduced,
1.22, but for people wanting reproducible builds it plain don't use -j
or if you really want to be explicit, uses -j1 as you did.

For >= 1.27 we will have 'reproducible_build' added to the
--btf_features= line if CONFIG_REPRODUCIBLE_BUILD is set.
 
> Gaining determinism for the older versions means giving up some performance.
> About a factor of 2.1 in our case (or an extra 10s on vmlinux).
 
> Do others want performance or determinism, should I send patches to LKML
> and the stable list?

But that is with older paholes, for 1.27, to be released next week, one
will just use '--btf_features=+reproducible_build and continue to have
parallel DWARF loading with serialized BTF encoding, which should be
close to the non-reproducible build numbers.
 
> Off-topic: pahole -J -j9 is faster than -j on my machine with 36
> cores, by another
> factor of 2. This is v1.25. -j1 18.7s, -j9 4.5s, -j 8.9s.

I'll try to profile that on a 5950X (16 cores, 32 threads) and a 14700K
(8 performance cores (16 threads) + 12 efficiency cores (12 threads),
total 28 threads) to try and see if I can get a better heuristic that
takes into account these factors on the most recent codebase that will
become 1.27 next week.

- Arnaldo

Re: pahole -J non-determinism and reproducible builds

[Alan Maguire]

On 23/05/2024 21:41, Arnaldo Carvalho de Melo wrote:
> On Thu, May 23, 2024 at 08:25:17PM +0100, Giuliano Procida wrote:
>> Hi.
>>
>> (edited repost due to unfortunate HTML email).
>>
>> Until the reproducible BTF changes land in a new pahole version, and for
>> anyone stuck on older versions, pahole -J -j typically produces
>> non-deterministic BTF.
>>
>> For Android, we are converging to hermetic, reproducible builds. .BTF
>> sections are a significant difference we see. I have trivial patches to
>> pass -j1 instead of -j for the older versions we are using (1.23 and 1.25).
>>
>> Trivial patches:
>> https://android-review.googlesource.com/q/Ibd72ac638faa1826f6655b336cc7001591ea70f1
> 
> That should be at least configurable thru a CONFIG_REPRODUCIBLE_BUILD
> that then would check the pahole version and if less than 1.27 then
> would use -j1
> 
> So it continues to use -j from the version where it was introduced,
> 1.22, but for people wanting reproducible builds it plain don't use -j
> or if you really want to be explicit, uses -j1 as you did.
> 
> For >= 1.27 we will have 'reproducible_build' added to the
> --btf_features= line if CONFIG_REPRODUCIBLE_BUILD is set.
>  
>> Gaining determinism for the older versions means giving up some performance.
>> About a factor of 2.1 in our case (or an extra 10s on vmlinux).
>  

From the original thread [1] looks like we could also use
KBUILD_BUILD_TIMESTAMP as an indication a reproducible build is needed,
though an explicit CONFIG variable might be clearer. Only thing is it
might be harder to backport with a new CONFIG value; not sure what the
stable backport perspective is on introducing new CONFIG vars.

[1] https://lore.kernel.org/bpf/Zf2_fgs0cqW82zEu@x1/

>> Do others want performance or determinism, should I send patches to LKML
>> and the stable list?
> 
> But that is with older paholes, for 1.27, to be released next week, one
> will just use '--btf_features=+reproducible_build and continue to have
> parallel DWARF loading with serialized BTF encoding, which should be
> close to the non-reproducible build numbers.
>  
>> Off-topic: pahole -J -j9 is faster than -j on my machine with 36
>> cores, by another
>> factor of 2. This is v1.25. -j1 18.7s, -j9 4.5s, -j 8.9s.
> 
> I'll try to profile that on a 5950X (16 cores, 32 threads) and a 14700K
> (8 performance cores (16 threads) + 12 efficiency cores (12 threads),
> total 28 threads) to try and see if I can get a better heuristic that
> takes into account these factors on the most recent codebase that will
> become 1.27 next week.
>
 > - Arnaldo

Re: pahole -J non-determinism and reproducible builds

[Tony Ambardar]

On Sat, May 25, 2024 at 03:42:50PM +0100, Alan Maguire wrote:
> On 23/05/2024 21:41, Arnaldo Carvalho de Melo wrote:
> > On Thu, May 23, 2024 at 08:25:17PM +0100, Giuliano Procida wrote:
> >> Hi.
> >>
> >> (edited repost due to unfortunate HTML email).
> >>
> >> Until the reproducible BTF changes land in a new pahole version, and for
> >> anyone stuck on older versions, pahole -J -j typically produces
> >> non-deterministic BTF.
> >>
> >> For Android, we are converging to hermetic, reproducible builds. .BTF
> >> sections are a significant difference we see. I have trivial patches to
> >> pass -j1 instead of -j for the older versions we are using (1.23 and 1.25).
> >>
> >> Trivial patches:
> >> https://android-review.googlesource.com/q/Ibd72ac638faa1826f6655b336cc7001591ea70f1
> > 
> > That should be at least configurable thru a CONFIG_REPRODUCIBLE_BUILD
> > that then would check the pahole version and if less than 1.27 then
> > would use -j1
> > 
> > So it continues to use -j from the version where it was introduced,
> > 1.22, but for people wanting reproducible builds it plain don't use -j
> > or if you really want to be explicit, uses -j1 as you did.
> > 
> > For >= 1.27 we will have 'reproducible_build' added to the
> > --btf_features= line if CONFIG_REPRODUCIBLE_BUILD is set.
> >  
> >> Gaining determinism for the older versions means giving up some performance.
> >> About a factor of 2.1 in our case (or an extra 10s on vmlinux).
> >  
> 
> From the original thread [1] looks like we could also use
> KBUILD_BUILD_TIMESTAMP as an indication a reproducible build is needed,
> though an explicit CONFIG variable might be clearer. Only thing is it
> might be harder to backport with a new CONFIG value; not sure what the
> stable backport perspective is on introducing new CONFIG vars.
> 
> [1] https://lore.kernel.org/bpf/Zf2_fgs0cqW82zEu@x1/
> 

Hi Alan,

Are there plans to implement this? It would be nice to see environment
based checks to enable reproducible builds, and in the past I've come
across distro patches also using SOURCE_DATE_EPOCH to force passing -j1.
With version 1.27, would something like the following be reasonable?

--- a/pahole.c
+++ b/pahole.c
@@ -3723,6 +3723,9 @@ int main(int argc, char *argv[])
 		goto out;
 	}
 
+	if (getenv("SOURCE_DATE_EPOCH") || getenv("KBUILD_BUILD_TIMESTAMP"))
+		conf_load.reproducible_build = true;
+
 	if (dwarves__init()) {
 		fputs("pahole: insufficient memory\n", stderr);
 		goto out;


Thanks,
Tony

> >> Do others want performance or determinism, should I send patches to LKML
> >> and the stable list?
> > 
> > But that is with older paholes, for 1.27, to be released next week, one
> > will just use '--btf_features=+reproducible_build and continue to have
> > parallel DWARF loading with serialized BTF encoding, which should be
> > close to the non-reproducible build numbers.
> >  
> >> Off-topic: pahole -J -j9 is faster than -j on my machine with 36
> >> cores, by another
> >> factor of 2. This is v1.25. -j1 18.7s, -j9 4.5s, -j 8.9s.
> > 
> > I'll try to profile that on a 5950X (16 cores, 32 threads) and a 14700K
> > (8 performance cores (16 threads) + 12 efficiency cores (12 threads),
> > total 28 threads) to try and see if I can get a better heuristic that
> > takes into account these factors on the most recent codebase that will
> > become 1.27 next week.
> >
>  > - Arnaldo