RE: [yocto] Parallel builds under yocto

["Monsees, Steven C (US)" <steven.monsees@baesystems.us>]

Thank you for your responses, I am trying to track down a race condition running yocto in a podman container on RHEL 8 systems 20 + cores...

-----Original Message-----
From: yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org> On Behalf Of Alexander Kanavin
Sent: Thursday, August 15, 2024 6:25 AM
To: yocto@lists.yoctoproject.org; Monsees, Steven C (US) <steven.monsees@baesystems.us>
Cc: Khem Raj <raj.khem@gmail.com>
Subject: Re: [yocto] Parallel builds under yocto

External Email Alert

This email has been sent from an account outside of the BAE Systems network.

Please treat the email with caution, especially if you are requested to click on a link, decrypt/open an attachment, or enable macros.  For further information on how to spot phishing, access “Cybersecurity OneSpace Page” and report phishing by clicking the button “Report Phishing” on the Outlook toolbar.


On Thu, 15 Aug 2024 at 12:18, Monsees, Steven C (US) via lists.yoctoproject.org <steven.monsees=baesystems.us@lists.yoctoproject.org> wrote:
>
>
> So, to be clear, in this example:
>
> There are 12 cores total on a board ...
> BB_NUMBER_THREADS="8", this tells bitbake to generate 8 tasks
>
> And, these tells do_make/do_install to each generate 2 tasks:
>
> PARALLEL_MAKE="-j 2"
> PARALLEL_MAKEINST="-j 2”
>
> All 12 cores are available to be used yocto build.
>
> Correct ?

No. This merely instructs bitbake to start a maximum of 8 tasks (it can run less if the task dependency graph doesn't allow for more until some task on the critical path completes and 'unlocks' further tasks), and each of those tasks, if it's a make-driven one, will run on a maximum of two cores. So you can end up anywhere from 8*2=16 processes to just one process, and on average you're likely to under-utilize the CPU with these settings, and have your build take longer than it could be.

Please do stick with the defaults, they are picked to maximize CPU utilization at small risk of running out of RAM.

Alex