Re: [docs] [PATCH] Add a document on limiting host resources

["Antonin Godard" <antonin.godard@bootlin.com>]

On Tue Jun 24, 2025 at 5:25 PM CEST, Quentin Schulz wrote:
> Hi Antonin,
>
> On 6/24/25 4:42 PM, Antonin Godard via lists.yoctoproject.org wrote:
>> Add a "Limiting the Host Resources Usage" document to share the
>> different techniques that can be used to limit the host resources usage.
>> We do have a document to document how to speed up a build, so this
>> document comes right after.
>> 
>> [YOCTO #15111]
>> 
>> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
>> ---
>>   documentation/dev-manual/index.rst              |  1 +
>>   documentation/dev-manual/limiting-resources.rst | 85 +++++++++++++++++++++++++
>>   2 files changed, 86 insertions(+)
>> 
>> diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
>> index 63736e0ab..4abfa59e9 100644
>> --- a/documentation/dev-manual/index.rst
>> +++ b/documentation/dev-manual/index.rst
>> @@ -22,6 +22,7 @@ Yocto Project Development Tasks Manual
>>      building
>>      multiconfig
>>      speeding-up-build
>> +   limiting-resources
>>      libraries
>>      prebuilt-libraries
>>      devtool
>> diff --git a/documentation/dev-manual/limiting-resources.rst b/documentation/dev-manual/limiting-resources.rst
>> new file mode 100644
>> index 000000000..df8208bdc
>> --- /dev/null
>> +++ b/documentation/dev-manual/limiting-resources.rst
>> @@ -0,0 +1,85 @@
>> +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
>> +
>> +Limiting the Host Resources Usage
>> +*********************************
>> +
>> +While you sometimes need to :doc:`speed up a build
>> +</dev-manual/speeding-up-build>`, you may also need to limit the resources used
>> +by the :term:`OpenEmbedded Build System`, especially on shared infrastructures
>> +where multiple users start heavy-load builds.
>> +
>
> We could also say "or building on low-power machines".
>
>> +This document aims at giving the different configuration variables available to
>> +limit the resources used by the build system. These variables should be set from
>> +a :term:`configuration file` and thus take effect over the entire build environment.
>> +For each variable, also see the variable description in the glossary for more
>> +details.
>> +
>> +-  :term:`BB_NUMBER_THREADS`:
>> +
>> +   This sets a hard limit on the number of threads :term:`BitBake` can run at the
>> +   same time. Lowering this value will set a limit to the number of> +   :term:`BitBake` threads, but will not prevent a single task from 
> starting more
>
> I was convinced this was the number of tasks BitBake can run in parallel 
> (which is what we say here and there in different places in the doc as 
> far as I could tell), but I also see that we have a bunch of calls to 
> bb.compress.zstd.open in bbclasses that pass this variable as the number 
> of threads to use. Sooo not sure what to trust anymore :)

Shouldn't they use oe.utils.parallel_make() instead? I also assumed
BB_NUMBER_THREADS was limited to Bitbake tasks.

>> +   compilation threads (see :term:`PARALLEL_MAKE`).
>> +
>> +-  :term:`BB_NUMBER_PARSE_THREADS`:
>> +
>> +   Like :term:`BB_NUMBER_THREADS`, but this variable sets a limit to the number
>
> s/to/on/ ?
>
>> +   of threads using during parsing of the environment (before executing tasks).
>
> s/using// ?
>
> +the parsing?
>
>> +
>> +-  :term:`PARALLEL_MAKE`:
>> +
>> +   This variable should be set in the form of ``-jN``, where ``N`` is a positive
>> +   integer. This integer controls the number of threads used when starting
>> +   ``make``. Note that this variable is not limited to the usage of ``make``,
>> +   but extends to the compilation commands defined by the
>> +   :ref:`ref-classes-meson`, :ref:`ref-classes-cmake` and such classes.
>> +
>
> I think it'd be better to say this applies to configure (?) and 
> compilation steps, and typically for make, meson, cmake, you name it, 
> build systems.
>
>> +   If you have identified a recipe that you want to limit separately from the
>> +   rest, it is also possible to achieve with the following syntax::
>
> """
> you want to have a different limit from the rest of the build
> """
>
> ?
>
> Please specify where to put this variable (in any configuration file, 
> but most likely local.conf I assume).
>
>> +
>> +      PARALLEL_MAKE:pn-linux-yocto = "-j4"
>> +
>> +   The above example will limit the number of threads using by ``make`` for the
>
> s/using/used/
>
>> +   ``linux-yocto`` recipe to 4.
>> +
>> +-  :term:`PARALLEL_MAKEINST`:
>> +
>> +   Like :term:`PARALLEL_MAKE`, but this variable controls the number of threads
>> +   used during the :ref:`ref-tasks-install` task.
>> +
>> +-  :term:`BB_PRESSURE_MAX_CPU`, :term:`BB_PRESSURE_MAX_IO` and
>> +   :term:`BB_PRESSURE_MAX_MEMORY`:
>> +
>> +   This variable controls the limit of pressure (as defined by
>
> s/This variable controls/These variables control/
>
> I think we need to state that this requires the host machine's kernel to 
> support pressure reporting/handling. I vaguely remember this was 
> discussed back when this was implemented.

Yes, true, also the last variable kind of implies that but I should also mention
it here.

>> +   https://docs.kernel.org/accounting/psi.html) on the system, and will
>> +   limit the number of :term:`BitBake` threads dynamically depending on the
>
> Is it threads or tasks here? I also don't entirely remember if this can 
> also impact already running tasks (e.g. dynamically reducing the number 
> of threads in the available pool during a make call in a recipe which 
> started with the limit not reached but is running while the limit is 
> reached).

It is only tasks as I understand it. I'm just getting confused by
BB_NUMBER_THREADS which controls the number of tasks, so mixing up words a bit
here.

>> +   current pressure of the system.
>> +
>> +   These variable take a positive integer between 1 (extremely low limit) and
>
> s/variable/variables/
>
>> +   1000000 (value unlikely ever reached). Setting an extremely low value, such
>> +   as 2, is not desirable as it will result in :term:`BitBake` limiting the
>> +   number of threads to 1 most of the time.
>> +
>> +   However, having a low value at the beginning makes sense to get a sense of a
>
> beginning of what?

An "initial low value" would be better maybe, but I wanted to convey that this
variable can/should be adjusted over time.

>> +   reasonable value to set for your system. Indeed, :term:`BitBake` will print
>> +   message on the console in the following format each time the current pressure
>
> s/message/messages/ ?
>
>> +   exceeds of the limit set by the above variables::
>
> s/of// ? or s/of/that of/?
>
>> +
>> +      Pressure status changed to CPU: True, IO: False, Mem: False (CPU: 1105.9/2.0, IO: 0.0/2.0, Mem: 0.0/2.0) - using 1/64 bitbake threads
>> +
>> +   Take a look at the value between parenthesis: ``CPU: 1105.9/2.0, IO: 0.0/2.0,
>
> s/value/values/
>
>> +   Mem: 0.0/2.0``. They correspond to the current pressure value for the CPU, IO
>> +   and memory respectively. Monitoring these values during a build should tell
>> +   you what a reasonable value for your host system is.
>> +
>
> Mmmmm... I'm perplexed. If we set the value very low (e.g. 2) won't we 
> simply limit the number of threads to something very low which actually 
> could mean we're not necessarily nearing the maxing out of the 
> CPU/IO/Memory?
>
> It's like running your 4-stroke motor on only one piston: "I see I can 
> reach 60kph so this is a safe limit for my motor when not choked up"... 
> which yes is true, but now you're not going on the highway because 
> you're limiting yourself to 60kph and taking the long way home. Safe 
> yes, but not sure we can derive much from this test except a very low 
> safe value?

Hmm.. true, so maybe the best option would just to monitor /proc/pressure
without any setting, and then set the value.

>> +-  :term:`BB_LOADFACTOR_MAX`:
>> +
>> +   This variable will limit the number of threads :term:`BitBake` will start
>> +   by monitoring the current load of the host system. :term:`BitBake` will print
>
> I assume we're talking about CPU load here?

Yes, I'll add that.

>> +   the following when the limit set by :term:`BB_LOADFACTOR_MAX` is reached::
>> +
>> +      Load average limiting set to True as load average: 0.7188262939453125 - using 37/64 bitbake threads
>> +
>> +   This variable has no effect when any of :term:`BB_PRESSURE_MAX_CPU`,
>> +   :term:`BB_PRESSURE_MAX_IO` or :term:`BB_PRESSURE_MAX_MEMORY` is set, as it
>> +   was designed for system that do not have pressure information available.
>
> /system/systems/
>
> Hopefully the glossary and this document stay in sync :)

What I can do is revise the current glossary definitions to be minimal and
redirect to this document which would act as a better overall guide on how to
achieve resource limiting. What do you think?

Thanks,
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com