Re: [yocto] Parallel builds under yocto

Re: [yocto] Parallel builds under yocto

[Khem Raj]

On Wed, Aug 14, 2024 at 12:15 PM Monsees, Steven C (US) via
lists.yoctoproject.org
<steven.monsees=baesystems.us@lists.yoctoproject.org> wrote:
>
> Hello:
>
> I am looking for some clarity with regards too parallel builds under Yocto…
>
> I think I had a false assumption on setting PARALEL_MAKE…
>
> To increase thread usage: You add following to your local.conf inside the build/conf directory. Replace x and y with your wanted configuration.
>
> PARALLEL_MAKE = "-j x"
>
>
>
> BB_NUMBER_THREADS = "y"
>
>
>
> Is the following true ?
>
>
>
> PARALLEL_MAKE defines how many threads should be used/dedicated when using make -j command during do_compile.
>
>

seems correct.

>
> BB_NUMBER_THREADS defines number of threads dedicated for bitbake.

It means how many jobs/tasks can bitbake spawn in parallel, its higher
order than parallel make.

>
>
>
> I set up like so:
>
>
>
> There are 12 cores my board
>
>
>
> Yocto supporters say threads should be 2 times the number of cores, but don’t go above 20…, or just the number cores, also under 20… depending on what you read.
>
>
>
> They also say, increasing PARALLEL_MAKE to higher values, increases your chance for a race condition.
>
>
>
> Is the following true ?
>
>
>
> PARALLEL_MAKE = "-j x"  # For do_compile
>
>
>
> BB_NUMBER_THREADS = "y"  # For bitbake
>
>
>
> PARALLEL_MAKEINST = "-j x"  # For do_install
>
>
>
> If so, does this mean for 12 cores:
>
>
>
> BB_NUMBER_THREADS="8", will this dedicate 8 cores just for bitbake to use ?

No, it will not reserve 8 cores, rather it tells bitbake to launch 8
task executions in parallel
if bitbake deems them not dependent upon each other, e.g. every recipe
has several tasks
during build do_confgure, do_compile, do_install etc. and there are
several recipes in build
queue, then bitbake's task manager can find out non-dependent tasks
and execute them
in parallel, now be aware that in some cases it could be executing 8
tasks which could be
all do_compile tasks for different recipes, in this case your
parallelism will become -j 8*8
systemwide.

>
>
>
> And does this dedicate the remaining 4 for do_compile and do_install ?
>
>
>
> PARALLEL_MAKE="-j 2"
>
> PARALLEL_MAKEINST="-j 2”
>
>
>
> Thank you…
>
>
>
> Steve
>
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> You automatically follow any topics you start or reply to.
> View/Reply Online (#63699): https://lists.yoctoproject.org/g/yocto/message/63699
> Mute This Topic: https://lists.yoctoproject.org/mt/107901352/1997914
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [yocto] Parallel builds under yocto

[Alexander Kanavin]

Also there are sensible defaults for both, which use autodetection for
how many cores a system cpu has. Generally you need to start tweaking
these settings only when defaults aren't working well (e.g. if the
system doesn't have enough RAM to support that many instances of gcc).

Alex

On Wed, 14 Aug 2024 at 21:25, Khem Raj via lists.yoctoproject.org
<raj.khem=gmail.com@lists.yoctoproject.org> wrote:
>
> On Wed, Aug 14, 2024 at 12:15 PM Monsees, Steven C (US) via
> lists.yoctoproject.org
> <steven.monsees=baesystems.us@lists.yoctoproject.org> wrote:
> >
> > Hello:
> >
> > I am looking for some clarity with regards too parallel builds under Yocto…
> >
> > I think I had a false assumption on setting PARALEL_MAKE…
> >
> > To increase thread usage: You add following to your local.conf inside the build/conf directory. Replace x and y with your wanted configuration.
> >
> > PARALLEL_MAKE = "-j x"
> >
> >
> >
> > BB_NUMBER_THREADS = "y"
> >
> >
> >
> > Is the following true ?
> >
> >
> >
> > PARALLEL_MAKE defines how many threads should be used/dedicated when using make -j command during do_compile.
> >
> >
>
> seems correct.
>
> >
> > BB_NUMBER_THREADS defines number of threads dedicated for bitbake.
>
> It means how many jobs/tasks can bitbake spawn in parallel, its higher
> order than parallel make.
>
> >
> >
> >
> > I set up like so:
> >
> >
> >
> > There are 12 cores my board
> >
> >
> >
> > Yocto supporters say threads should be 2 times the number of cores, but don’t go above 20…, or just the number cores, also under 20… depending on what you read.
> >
> >
> >
> > They also say, increasing PARALLEL_MAKE to higher values, increases your chance for a race condition.
> >
> >
> >
> > Is the following true ?
> >
> >
> >
> > PARALLEL_MAKE = "-j x"  # For do_compile
> >
> >
> >
> > BB_NUMBER_THREADS = "y"  # For bitbake
> >
> >
> >
> > PARALLEL_MAKEINST = "-j x"  # For do_install
> >
> >
> >
> > If so, does this mean for 12 cores:
> >
> >
> >
> > BB_NUMBER_THREADS="8", will this dedicate 8 cores just for bitbake to use ?
>
> No, it will not reserve 8 cores, rather it tells bitbake to launch 8
> task executions in parallel
> if bitbake deems them not dependent upon each other, e.g. every recipe
> has several tasks
> during build do_confgure, do_compile, do_install etc. and there are
> several recipes in build
> queue, then bitbake's task manager can find out non-dependent tasks
> and execute them
> in parallel, now be aware that in some cases it could be executing 8
> tasks which could be
> all do_compile tasks for different recipes, in this case your
> parallelism will become -j 8*8
> systemwide.
>
> >
> >
> >
> > And does this dedicate the remaining 4 for do_compile and do_install ?
> >
> >
> >
> > PARALLEL_MAKE="-j 2"
> >
> > PARALLEL_MAKEINST="-j 2”
> >
> >
> >
> > Thank you…
> >
> >
> >
> > Steve
> >
> >
> >
> >
> >
> >
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#63700): https://lists.yoctoproject.org/g/yocto/message/63700
> Mute This Topic: https://lists.yoctoproject.org/mt/107901352/1686489
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [yocto] Parallel builds under yocto

[Mikko Rapeli]

Hi,

On Wed, Aug 14, 2024 at 09:36:05PM +0200, Alexander Kanavin via lists.yoctoproject.org wrote:
> Also there are sensible defaults for both, which use autodetection for
> how many cores a system cpu has. Generally you need to start tweaking
> these settings only when defaults aren't working well (e.g. if the
> system doesn't have enough RAM to support that many instances of gcc).

And when RAM becomes the limiting factor, one rule of thumb is to follow
CPU thread count until each thread has at least 2 Gb of physical RAM available.

On complex builds with lots of C++ (qt and friends) this seems to be a good
upper limit.

Less than 2 Gb per CPU thread results in OOM killer actions during building/linking
steps.

If one tracks builds CPU, memory, local disk and networking load with Performance Co-Pilot
or similar tools, it can be deduced that builds are CPU bound only some of the time
and a lot of time they are actually memory and local disk IO bound. Thus reducing
disk writes helps.

http://events17.linuxfoundation.org/sites/events/files/slides/elce-2016-mario-goulart-mikko-rapeli.pdf
and slides 49 and 50, for one setup.

Cheers,

-Mikko

RE: [yocto] Parallel builds under yocto

[Monsees, Steven C (US)]

So, to be clear, in this example:

There are 12 cores total on a board ...
BB_NUMBER_THREADS="8", this tells bitbake to generate 8 tasks 

And, these tells do_make/do_install to each generate 2 tasks:

PARALLEL_MAKE="-j 2"
PARALLEL_MAKEINST="-j 2”

All 12 cores are available to be used yocto build.

Correct ?

Thanks again...

Steve

PARALLEL_MAKE="-j 2"
PARALLEL_MAKEINST="-j 2”
-----Original Message-----
From: Khem Raj <raj.khem@gmail.com> 
Sent: Wednesday, August 14, 2024 3:25 PM
To: yocto@lists.yoctoproject.org; Monsees, Steven C (US) <steven.monsees@baesystems.us>
Subject: Re: [yocto] Parallel builds under yocto

External Email Alert

This email has been sent from an account outside of the BAE Systems network.

Please treat the email with caution, especially if you are requested to click on a link, decrypt/open an attachment, or enable macros.  For further information on how to spot phishing, access “Cybersecurity OneSpace Page” and report phishing by clicking the button “Report Phishing” on the Outlook toolbar.


On Wed, Aug 14, 2024 at 12:15 PM Monsees, Steven C (US) via lists.yoctoproject.org <steven.monsees=baesystems.us@lists.yoctoproject.org> wrote:
>
> Hello:
>
> I am looking for some clarity with regards too parallel builds under 
> Yocto…
>
> I think I had a false assumption on setting PARALEL_MAKE…
>
> To increase thread usage: You add following to your local.conf inside the build/conf directory. Replace x and y with your wanted configuration.
>
> PARALLEL_MAKE = "-j x"
>
>
>
> BB_NUMBER_THREADS = "y"
>
>
>
> Is the following true ?
>
>
>
> PARALLEL_MAKE defines how many threads should be used/dedicated when using make -j command during do_compile.
>
>

seems correct.

>
> BB_NUMBER_THREADS defines number of threads dedicated for bitbake.

It means how many jobs/tasks can bitbake spawn in parallel, its higher order than parallel make.

>
>
>
> I set up like so:
>
>
>
> There are 12 cores my board
>
>
>
> Yocto supporters say threads should be 2 times the number of cores, but don’t go above 20…, or just the number cores, also under 20… depending on what you read.
>
>
>
> They also say, increasing PARALLEL_MAKE to higher values, increases your chance for a race condition.
>
>
>
> Is the following true ?
>
>
>
> PARALLEL_MAKE = "-j x"  # For do_compile
>
>
>
> BB_NUMBER_THREADS = "y"  # For bitbake
>
>
>
> PARALLEL_MAKEINST = "-j x"  # For do_install
>
>
>
> If so, does this mean for 12 cores:
>
>
>
> BB_NUMBER_THREADS="8", will this dedicate 8 cores just for bitbake to use ?

No, it will not reserve 8 cores, rather it tells bitbake to launch 8 task executions in parallel if bitbake deems them not dependent upon each other, e.g. every recipe has several tasks during build do_confgure, do_compile, do_install etc. and there are several recipes in build queue, then bitbake's task manager can find out non-dependent tasks and execute them in parallel, now be aware that in some cases it could be executing 8 tasks which could be all do_compile tasks for different recipes, in this case your parallelism will become -j 8*8 systemwide.

>
>
>
> And does this dedicate the remaining 4 for do_compile and do_install ?
>
>
>
> PARALLEL_MAKE="-j 2"
>
> PARALLEL_MAKEINST="-j 2”
>
>
>
> Thank you…
>
>
>
> Steve
>
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> You automatically follow any topics you start or reply to.
> View/Reply Online (#63699): 
> https://lists.yoctoproject.org/g/yocto/message/63699
> Mute This Topic: https://lists.yoctoproject.org/mt/107901352/1997914
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub 
> [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [yocto] Parallel builds under yocto

[Alexander Kanavin]

On Thu, 15 Aug 2024 at 12:18, Monsees, Steven C (US) via
lists.yoctoproject.org
<steven.monsees=baesystems.us@lists.yoctoproject.org> wrote:
>
>
> So, to be clear, in this example:
>
> There are 12 cores total on a board ...
> BB_NUMBER_THREADS="8", this tells bitbake to generate 8 tasks
>
> And, these tells do_make/do_install to each generate 2 tasks:
>
> PARALLEL_MAKE="-j 2"
> PARALLEL_MAKEINST="-j 2”
>
> All 12 cores are available to be used yocto build.
>
> Correct ?

No. This merely instructs bitbake to start a maximum of 8 tasks (it
can run less if the task dependency graph doesn't allow for more until
some task on the critical path completes and 'unlocks' further tasks),
and each of those tasks, if it's a make-driven one, will run on a
maximum of two cores. So you can end up anywhere from 8*2=16 processes
to just one process, and on average you're likely to under-utilize the
CPU with these settings, and have your build take longer than it could
be.

Please do stick with the defaults, they are picked to maximize CPU
utilization at small risk of running out of RAM.

Alex

RE: [yocto] Parallel builds under yocto

[Monsees, Steven C (US)]

Thank you for your responses, I am trying to track down a race condition running yocto in a podman container on RHEL 8 systems 20 + cores...

-----Original Message-----
From: yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org> On Behalf Of Alexander Kanavin
Sent: Thursday, August 15, 2024 6:25 AM
To: yocto@lists.yoctoproject.org; Monsees, Steven C (US) <steven.monsees@baesystems.us>
Cc: Khem Raj <raj.khem@gmail.com>
Subject: Re: [yocto] Parallel builds under yocto

External Email Alert

This email has been sent from an account outside of the BAE Systems network.

Please treat the email with caution, especially if you are requested to click on a link, decrypt/open an attachment, or enable macros.  For further information on how to spot phishing, access “Cybersecurity OneSpace Page” and report phishing by clicking the button “Report Phishing” on the Outlook toolbar.


On Thu, 15 Aug 2024 at 12:18, Monsees, Steven C (US) via lists.yoctoproject.org <steven.monsees=baesystems.us@lists.yoctoproject.org> wrote:
>
>
> So, to be clear, in this example:
>
> There are 12 cores total on a board ...
> BB_NUMBER_THREADS="8", this tells bitbake to generate 8 tasks
>
> And, these tells do_make/do_install to each generate 2 tasks:
>
> PARALLEL_MAKE="-j 2"
> PARALLEL_MAKEINST="-j 2”
>
> All 12 cores are available to be used yocto build.
>
> Correct ?

No. This merely instructs bitbake to start a maximum of 8 tasks (it can run less if the task dependency graph doesn't allow for more until some task on the critical path completes and 'unlocks' further tasks), and each of those tasks, if it's a make-driven one, will run on a maximum of two cores. So you can end up anywhere from 8*2=16 processes to just one process, and on average you're likely to under-utilize the CPU with these settings, and have your build take longer than it could be.

Please do stick with the defaults, they are picked to maximize CPU utilization at small risk of running out of RAM.

Alex