Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Ulf Samuelsson <openembedded-core@emagii.com>]

2014-01-28 11:08, Koen Kooi skrev:
> Op 27 jan. 2014, om 15:39 heeft Richard Purdie <richard.purdie@linuxfoundation.org> het volgende geschreven:
>
>> Its rather sad that people don't appear to read local.conf and then complain
>> about slow builds when they're just using a single thread. Most systems have
>> more than one core now so we might as well use a more automatic default
>> for these values. This may lead to better experiences for new users.
>>
>> [YOCTO #2528]
>>
>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>> ---
>> diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
>> index 71856b8..36d33e1 100644
>> --- a/meta/conf/local.conf.sample
>> +++ b/meta/conf/local.conf.sample
>> @@ -18,12 +18,18 @@
>> # option determines how many tasks bitbake should run in parallel:
>> #
>> #BB_NUMBER_THREADS ?= "4"
>> +#
>> +# Default to setting automatically based on cpu count
>> +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
> I've noticed that after 4 threads IO becomes a big bottleneck when you have things like webkit, qt, asio etc in the buildqueue. Combine that with issues like every make -j thread taking >2GB ram with asio and webkit this default seems a bit high. I'd use 0.5*numcpu with a lower bound of 2.
>
> regards,
We discussed this 2.3 months ago.
Did some studies on my dual hex-core machine (24 H/W treads) while 
building a cloud9-gnome-image derivative.
This did about 7500 tasks.

Enabled the CPU supervisors in the panel.

Everything seems to be ok with BB_NUMBER_THREADS = "24" for about 4-4500 
tasks.

Then the CPUs are mostly inactive and only 1-2 running for ~500 tasks.
Then parallellism  is resumed until about task 7000, and again
only a few CPUs are active.

I believe that some tools use "make" within the Makefile,
and they are written badly, and do not use "-j <n>" for
that part of the build.

Got my build down to 83 minutes.

Since I have 96 GB of RAM, I tried creating an 80 GB tmpfs for the build,
and copied the download and the recipes to the ram.

That shaved only 2 monutes from the build, and some stuff,
still built using only a single CPU.

BR
Ulf Samuelsson


> Koen
>
>> #
>> # The second option controls how many processes make should run in parallel when
>> # running compile tasks:
>> #
>> #PARALLEL_MAKE ?= "-j 4"
>> #
>> +# Default to setting automatically based on cpu count
>> +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
>> +#
>> # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
>> # be appropriate for example.
>>
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core


-- 
Best Regards
Ulf Samuelsson
eMagii