RE: [yocto] Parallel builds under yocto

["Monsees, Steven C (US)" <steven.monsees@baesystems.us>]

So, to be clear, in this example:

There are 12 cores total on a board ...
BB_NUMBER_THREADS="8", this tells bitbake to generate 8 tasks 

And, these tells do_make/do_install to each generate 2 tasks:

PARALLEL_MAKE="-j 2"
PARALLEL_MAKEINST="-j 2”

All 12 cores are available to be used yocto build.

Correct ?

Thanks again...

Steve

PARALLEL_MAKE="-j 2"
PARALLEL_MAKEINST="-j 2”
-----Original Message-----
From: Khem Raj <raj.khem@gmail.com> 
Sent: Wednesday, August 14, 2024 3:25 PM
To: yocto@lists.yoctoproject.org; Monsees, Steven C (US) <steven.monsees@baesystems.us>
Subject: Re: [yocto] Parallel builds under yocto

External Email Alert

This email has been sent from an account outside of the BAE Systems network.

Please treat the email with caution, especially if you are requested to click on a link, decrypt/open an attachment, or enable macros.  For further information on how to spot phishing, access “Cybersecurity OneSpace Page” and report phishing by clicking the button “Report Phishing” on the Outlook toolbar.


On Wed, Aug 14, 2024 at 12:15 PM Monsees, Steven C (US) via lists.yoctoproject.org <steven.monsees=baesystems.us@lists.yoctoproject.org> wrote:
>
> Hello:
>
> I am looking for some clarity with regards too parallel builds under 
> Yocto…
>
> I think I had a false assumption on setting PARALEL_MAKE…
>
> To increase thread usage: You add following to your local.conf inside the build/conf directory. Replace x and y with your wanted configuration.
>
> PARALLEL_MAKE = "-j x"
>
>
>
> BB_NUMBER_THREADS = "y"
>
>
>
> Is the following true ?
>
>
>
> PARALLEL_MAKE defines how many threads should be used/dedicated when using make -j command during do_compile.
>
>

seems correct.

>
> BB_NUMBER_THREADS defines number of threads dedicated for bitbake.

It means how many jobs/tasks can bitbake spawn in parallel, its higher order than parallel make.

>
>
>
> I set up like so:
>
>
>
> There are 12 cores my board
>
>
>
> Yocto supporters say threads should be 2 times the number of cores, but don’t go above 20…, or just the number cores, also under 20… depending on what you read.
>
>
>
> They also say, increasing PARALLEL_MAKE to higher values, increases your chance for a race condition.
>
>
>
> Is the following true ?
>
>
>
> PARALLEL_MAKE = "-j x"  # For do_compile
>
>
>
> BB_NUMBER_THREADS = "y"  # For bitbake
>
>
>
> PARALLEL_MAKEINST = "-j x"  # For do_install
>
>
>
> If so, does this mean for 12 cores:
>
>
>
> BB_NUMBER_THREADS="8", will this dedicate 8 cores just for bitbake to use ?

No, it will not reserve 8 cores, rather it tells bitbake to launch 8 task executions in parallel if bitbake deems them not dependent upon each other, e.g. every recipe has several tasks during build do_confgure, do_compile, do_install etc. and there are several recipes in build queue, then bitbake's task manager can find out non-dependent tasks and execute them in parallel, now be aware that in some cases it could be executing 8 tasks which could be all do_compile tasks for different recipes, in this case your parallelism will become -j 8*8 systemwide.

>
>
>
> And does this dedicate the remaining 4 for do_compile and do_install ?
>
>
>
> PARALLEL_MAKE="-j 2"
>
> PARALLEL_MAKEINST="-j 2”
>
>
>
> Thank you…
>
>
>
> Steve
>
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> You automatically follow any topics you start or reply to.
> View/Reply Online (#63699): 
> https://lists.yoctoproject.org/g/yocto/message/63699
> Mute This Topic: https://lists.yoctoproject.org/mt/107901352/1997914
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub 
> [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>