Re: [OE-core] Automatic cap on BB_NUMBER_THREADS and PARALLEL_MAKE

["Richard Purdie" <richard.purdie@linuxfoundation.org>]

On Thu, 2020-12-03 at 17:48 +0000, Ross Burton wrote:
> Hi,
> 
> Currently, BB_NUMBER_THREADS and PARALLEL_MAKE use the number of
> cores
> available unless told otherwise.  This was a good idea six years
> ago[1] but some modern machines are moving to very large core counts.
> 
> For example, 88 core dual Xeons are fairly common. A ThunderX2 has
> 256
> cores (2 sockets, 4 hyperthreads per physical core). The Ampere Altra
> is dual socket 2*80=160 cores.
> 
> At this level of parallelisation the sheer amount of I/O from the
> unpack storm is quite excessive.  As a strawman argument, I propose a
> hard cap to the default BB_NUMBER_THREADS of -- and I'm literally
> making up numbers here -- 32.  Maybe 64.  Comments?

I've had no issues on 88 core systems. I'm not sure there is an
"automatic" value we can guess at here and it is mentioned in
local.conf for the user to customise...

What might make more sense is to have site.conf from a user's homedir
working better, although that does have issues of its own.

I do agree with Alex that having bitbake throttle on system load would
be nicer.

Cheers,

Richard