Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Ulf Samuelsson <ulf@emagii.com>]

2014-01-29 13:56, Richard Purdie skrev:
> On Wed, 2014-01-29 at 13:09 +0100, Ulf Samuelsson wrote:
>> We discussed this 2.3 months ago.
>> Did some studies on my dual hex-core machine (24 H/W treads) while
>> building a cloud9-gnome-image derivative.
>> This did about 7500 tasks.
>>
>> Enabled the CPU supervisors in the panel.
>>
>> Everything seems to be ok with BB_NUMBER_THREADS = "24" for about 4-4500
>> tasks.
>>
>> Then the CPUs are mostly inactive and only 1-2 running for ~500 tasks.
>> Then parallellism  is resumed until about task 7000, and again
>> only a few CPUs are active.
> This is likely whilst the lib and toolchain is getting built.
>
>> I believe that some tools use "make" within the Makefile,
>> and they are written badly, and do not use "-j <n>" for
>> that part of the build.
> Which recipes were building at this point? It would be interesting to
> track them down.
I think eglibc, node[-js] (native) and a few others

Think I remember mentioning them when it was discussed.

webkit, node are also not running in parallell in later part of the build

>
>> Got my build down to 83 minutes.
>>
>> Since I have 96 GB of RAM, I tried creating an 80 GB tmpfs for the build,
>> and copied the download and the recipes to the ram.
>>
>> That shaved only 2 monutes from the build, and some stuff,
>> still built using only a single CPU.
> There are certainly dependency bottlenecks in the build such as the
> toolchain, compiler, gettext, gtk+ and so where large numbers of things
> need those dependencies to get built before they can proceed. Not sure
> what we can do to help this though.
In order to find out more, perhaps it should be possible to log info 
about the build.
1. How many makes are in progress when you start a task
2. How long does it take to build a recipe.
3. CPU activity (frequency scaling) when the recipe is build.

A recipe which takes very long time to build but with low CPU activity
should be analyzed to find out if more parallellism can be introduced.

BR
Ulf
>
> Cheers,
>
> Richard
>


-- 
Best Regards
Ulf Samuelsson
ulf@emagii.com
+46 722 427437