Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[Adrian Bunk <bunk@stusta.de>]

On Tue, Feb 25, 2020 at 09:16:20AM +0000, André Draszik wrote:
> On Tue, 2020-02-25 at 00:00 +0200, Adrian Bunk wrote:
>...
> > > Perhaps we just allow the number of xz threads
> > > to be set independently?
> > 
> > dpkg manually reduces the number of threads until less than half
> > of the RAM is used:
> > https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574
> > 
> > In a script it would be possible to use --no-adjust to achieve the same:
> > $ xz -9 --memlimit=50% --no-adjust -T32 /dev/null 
> > xz: Memory usage limit is too low for the given filter setup.
> > xz: 39,972 MiB of memory is required. The limit is 32,051 MiB.
> > $
> 
> The problem with --no-adjust is that it also prevents xz from reducing the
> number of threads. It will do that in preference to changing compression
> parameters, so as long as you have more than 2499MiB of memory to support at
> least 2 threads (with default block size), compression will be identical.
>...

5 GB due to the 50%.

But I had something different in mind:

Similar to what dpkg is doing, you could once loop from 
@oe.utils.cpu_count() down to 2 until "--memlimit=50% --no-adjust -T$(i)"
succeeds.

Then run the actual compression without --memlimit.

I would not consider anything with less than 4 GB RAM reasonable for 
building Yocto (g++ also likes to use more than 2 GB).

> Cheers,
> Andre'

cu
Adrian