Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Steffen Sledz <sledz@dresearch-fe.de>]

On 27.01.2014 15:39, Richard Purdie wrote:
> Its rather sad that people don't appear to read local.conf and then complain
> about slow builds when they're just using a single thread. Most systems have
> more than one core now so we might as well use a more automatic default
> for these values. This may lead to better experiences for new users.
> 
> [YOCTO #2528]
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
> index 71856b8..36d33e1 100644
> --- a/meta/conf/local.conf.sample
> +++ b/meta/conf/local.conf.sample
> @@ -18,12 +18,18 @@
>  # option determines how many tasks bitbake should run in parallel:
>  #
>  #BB_NUMBER_THREADS ?= "4"
> +#
> +# Default to setting automatically based on cpu count
> +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
>  # 
>  # The second option controls how many processes make should run in parallel when
>  # running compile tasks:
>  #
>  #PARALLEL_MAKE ?= "-j 4"
>  #
> +# Default to setting automatically based on cpu count
> +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
> +#
>  # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
>  # be appropriate for example.

On our Fedora-18 build host this change leads to the following exception. :(

----------------------> snip <-------------------------
Loading cache...done.
Loaded 2167 entries from dependency cache.

Parsing recipes...done.
Parsing of 1729 .bb files complete (1727 cached, 2 parsed). 2168 targets, 85 skipped, 1 masked, 0 errors.

NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION        = "1.20.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "Fedora-18"
TARGET_SYS        = "arm-angstrom-linux-gnueabi"
MACHINE           = "hikirk"
DISTRO            = "angstrom"
DISTRO_VERSION    = "v2013.12"
TUNE_FEATURES     = "armv5 thumb dsp"
TARGET_FPU        = "soft"
meta-hipos        = "(nobranch):6f2ad6754c9c9ddd6f1c5505e584e0eb62cfcf83"
meta-hipos-kirkwood = "(nobranch):1a43f75e217fea3c6b4493536b3b66945ce53921"
meta-hipos-fsl    = "(nobranch):ad3beec4157a96d0c506969a29d14f6354e7406f"
meta-fsl-arm      = "(nobranch):af392c22bf6b563525ede4a81b6755ff1dd2c1c6"
meta-fsl-arm-extra = "(nobranch):07ad83db0fb67c5023bd627a61efb7f474c52622"
meta-java         = "(nobranch):9dfb7ecb87e67052e94641d6e02907c582264217"
meta-angstrom     = "(nobranch):8fcd781b777a7a94f2b97514c9691a2b5c74eaf2"
meta-oe
meta-systemd
meta-multimedia
meta-networking
meta-efl
meta-gnome        = "(nobranch):eb4563b83be0a57ede4269ab19688af6baa62cd2"
meta              = "(nobranch):e86622a932bbd0acdea67ecfb15c8b06c27353d8"

NOTE: Preparing runqueue
ERROR: An uncaught exception occured in runqueue, please see the failure below:
ERROR: Running idle function
Traceback (most recent call last):
  File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/runqueue.py", line 1111, in RunQueueExecuteScenequeue.__init__(rq=<bb.runqueue.RunQueue instance at 0x1534b2d8>):

    >        self.number_tasks = int(self.cfgData.getVar("BB_NUMBER_THREADS", True) or 1)
             self.scheduler = self.cfgData.getVar("BB_SCHEDULER", True) or "speed"
  File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 516, in DataSmart.getVar(var='BB_NUMBER_THREADS', expand=True, noweakdefault=False):
         def getVar(self, var, expand=False, noweakdefault=False):
    >        return self.getVarFlag(var, "_content", expand, noweakdefault)

  File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 606, in DataSmart.getVarFlag(var='BB_NUMBER_THREADS', flag='_content', expand=True, noweakdefault=False):
                     cachename = var + "[" + flag + "]"
    >            value = self.expand(value, cachename)
             if value is not None and flag == "_content" and local_var is not None and "_removeactive" in local_var:
  File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 344, in DataSmart.expand(s='${@oe.utils.cpu_count()}', varname='BB_NUMBER_THREADS'):
         def expand(self, s, varname = None):
    >        return self.expandWithRefs(s, varname).value

  File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 334, in DataSmart.expandWithRefs(s='${@oe.utils.cpu_count()}', varname='BB_NUMBER_THREADS'):
                 except Exception as exc:
    >                raise ExpansionError(varname, s, exc)

ExpansionError: Failure expanding variable BB_NUMBER_THREADS, expression was ${@oe.utils.cpu_count()} which triggered exception AttributeError: 'module' object has no attribute 'cpu_count'
----------------------> snip <-------------------------

Any ideas? A missing python package at the build host? ...?

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058