Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Steffen Sledz <sledz@dresearch-fe.de>]

On 29.01.2014 11:22, Paul Eggleton wrote:
> On Wednesday 29 January 2014 11:14:47 Steffen Sledz wrote:
>> On 27.01.2014 15:39, Richard Purdie wrote:
>>> Its rather sad that people don't appear to read local.conf and then
>>> complain about slow builds when they're just using a single thread. Most
>>> systems have more than one core now so we might as well use a more
>>> automatic default for these values. This may lead to better experiences
>>> for new users.
>>>
>>> [YOCTO #2528]
>>>
>>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>> ---
>>> diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
>>> index 71856b8..36d33e1 100644
>>> --- a/meta/conf/local.conf.sample
>>> +++ b/meta/conf/local.conf.sample
>>> @@ -18,12 +18,18 @@
>>>
>>>  # option determines how many tasks bitbake should run in parallel:
>>>  #
>>>  #BB_NUMBER_THREADS ?= "4"
>>>
>>> +#
>>> +# Default to setting automatically based on cpu count
>>> +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
>>>
>>>  #
>>>  # The second option controls how many processes make should run in
>>>  parallel when # running compile tasks:
>>>  #
>>>  #PARALLEL_MAKE ?= "-j 4"
>>>  #
>>>
>>> +# Default to setting automatically based on cpu count
>>> +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
>>> +#
>>>
>>>  # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j
>>>  4" would # be appropriate for example.
>>
>> On our Fedora-18 build host this change leads to the following exception. :(
>>
>> ----------------------> snip <-------------------------
>> Loading cache...done.
>> Loaded 2167 entries from dependency cache.
>>
>> Parsing recipes...done.
>> Parsing of 1729 .bb files complete (1727 cached, 2 parsed). 2168 targets, 85
>> skipped, 1 masked, 0 errors.
>>
>> NOTE: Resolving any missing task queue dependencies
>>
>> Build Configuration:
>> BB_VERSION        = "1.20.0"
>> BUILD_SYS         = "x86_64-linux"
>> NATIVELSBSTRING   = "Fedora-18"
>> TARGET_SYS        = "arm-angstrom-linux-gnueabi"
>> MACHINE           = "hikirk"
>> DISTRO            = "angstrom"
>> DISTRO_VERSION    = "v2013.12"
>> TUNE_FEATURES     = "armv5 thumb dsp"
>> TARGET_FPU        = "soft"
>> meta-hipos        = "(nobranch):6f2ad6754c9c9ddd6f1c5505e584e0eb62cfcf83"
>> meta-hipos-kirkwood = "(nobranch):1a43f75e217fea3c6b4493536b3b66945ce53921"
>> meta-hipos-fsl    = "(nobranch):ad3beec4157a96d0c506969a29d14f6354e7406f"
>> meta-fsl-arm      = "(nobranch):af392c22bf6b563525ede4a81b6755ff1dd2c1c6"
>> meta-fsl-arm-extra = "(nobranch):07ad83db0fb67c5023bd627a61efb7f474c52622"
>> meta-java         = "(nobranch):9dfb7ecb87e67052e94641d6e02907c582264217"
>> meta-angstrom     = "(nobranch):8fcd781b777a7a94f2b97514c9691a2b5c74eaf2"
>> meta-oe
>> meta-systemd
>> meta-multimedia
>> meta-networking
>> meta-efl
>> meta-gnome        = "(nobranch):eb4563b83be0a57ede4269ab19688af6baa62cd2"
>> meta              = "(nobranch):e86622a932bbd0acdea67ecfb15c8b06c27353d8"
>>
>> NOTE: Preparing runqueue
>> ERROR: An uncaught exception occured in runqueue, please see the failure
>> below: ERROR: Running idle function
>> Traceback (most recent call last):
>>
>>   File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/runqueue.py", line 
> 1111, in RunQueueExecuteScenequeue.__init__(rq=<bb.runqueue.RunQueue instance 
> at 0x1534b2d8>):
>>     >        self.number_tasks =
>>     >        int(self.cfgData.getVar("BB_NUMBER_THREADS", True) or 1)
>>              self.scheduler = self.cfgData.getVar("BB_SCHEDULER", True) or
>> "speed" File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py",
>> line 516, in DataSmart.getVar(var='BB_NUMBER_THREADS', expand=True,
>> noweakdefault=False):
>>          def getVar(self, var, expand=False, noweakdefault=False):
>>     >        return self.getVarFlag(var, "_content", expand, noweakdefault)
>>
>>   File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line
>> 606, in DataSmart.getVarFlag(var='BB_NUMBER_THREADS', flag='_content',
>> expand=True, noweakdefault=False): cachename = var + "[" + flag + "]"
>>
>>     >            value = self.expand(value, cachename)
>>
>>              if value is not None and flag == "_content" and local_var is
>> not None and "_removeactive" in local_var: File
>> "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 344, in
>> DataSmart.expand(s='${@oe.utils.cpu_count()}',
>> varname='BB_NUMBER_THREADS'):
>>          def expand(self, s, varname = None):
>>     >        return self.expandWithRefs(s, varname).value
>>
>>   File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line
>> 334, in DataSmart.expandWithRefs(s='${@oe.utils.cpu_count()}',
>> varname='BB_NUMBER_THREADS'):
>>                  except Exception as exc:
>>     >                raise ExpansionError(varname, s, exc)
>>
>> ExpansionError: Failure expanding variable BB_NUMBER_THREADS, expression was
>> ${@oe.utils.cpu_count()} which triggered exception AttributeError: 'module'
>> object has no attribute 'cpu_count' ----------------------> snip
>> <-------------------------
>>
>> Any ideas? A missing python package at the build host? ...?
> 
> Looks like you're missing Richard's other posted patch that adds this 
> function.

Hmmm? Exactly the same revisions work well on another (openSUSE-13.1) build host.

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058