Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[Adrian Bunk <bunk@stusta.de>]

On Mon, Feb 24, 2020 at 05:32:29PM +0000, Richard Purdie wrote:
> On Mon, 2020-02-24 at 17:14 +0000, André Draszik wrote:
>...
> > I don't want to penalise myself or anybody else by generally forcing
> > a low number of threads either.
> 
> We didn't have resource issues on the autobuilder with xz but others
> did report it and its why the memlimit was set. It would be better to
> remove the memlimit for determinism really as far as I can tell and if
> necessary throttle the threads count.
> 
> Maybe setting an upper limit of say 10 threads and a minimum of 2 might
> given us what we need.

On a Threadripper with 128 cores and 256 GB RAM it would not be a
problem to use all cores.

A laptop with 8 cores and 8 GB RAM is problematic.

> Perhaps we just allow the number of xz threads
> to be set independently?

dpkg manually reduces the number of threads until less than half
of the RAM is used:
https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574

In a script it would be possible to use --no-adjust to achieve the same:
$ xz -9 --memlimit=50% --no-adjust -T32 /dev/null 
xz: Memory usage limit is too low for the given filter setup.
xz: 39,972 MiB of memory is required. The limit is 32,051 MiB.
$

> > > FWIW we haven't seen variation on the autobuilder due to this as
> > > far as
> > > I know.
> > 
> > BTW, pigz and pbzip should have a similar thread related problem,
> > according to the man pages, if you read between the lines.
> 
> Agreed. We have less of an issue with these as their use is much less
> frequent. xz is used by deb and ipk for packaging.

Good point, dpkg does parallel xz compression and I am not aware of any
reproducibility problems this causes.

> Cheers,
> 
> Richard

cu
Adrian