[PATCH] Add a document on limiting host resources

[PATCH] Add a document on limiting host resources

[Antonin Godard]

Add a "Limiting the Host Resources Usage" document to share the
different techniques that can be used to limit the host resources usage.
We do have a document to document how to speed up a build, so this
document comes right after.

[YOCTO #15111]

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
---
 documentation/dev-manual/index.rst              |  1 +
 documentation/dev-manual/limiting-resources.rst | 85 +++++++++++++++++++++++++
 2 files changed, 86 insertions(+)

diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
index 63736e0ab..4abfa59e9 100644
--- a/documentation/dev-manual/index.rst
+++ b/documentation/dev-manual/index.rst
@@ -22,6 +22,7 @@ Yocto Project Development Tasks Manual
    building
    multiconfig
    speeding-up-build
+   limiting-resources
    libraries
    prebuilt-libraries
    devtool
diff --git a/documentation/dev-manual/limiting-resources.rst b/documentation/dev-manual/limiting-resources.rst
new file mode 100644
index 000000000..df8208bdc
--- /dev/null
+++ b/documentation/dev-manual/limiting-resources.rst
@@ -0,0 +1,85 @@
+.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
+
+Limiting the Host Resources Usage
+*********************************
+
+While you sometimes need to :doc:`speed up a build
+</dev-manual/speeding-up-build>`, you may also need to limit the resources used
+by the :term:`OpenEmbedded Build System`, especially on shared infrastructures
+where multiple users start heavy-load builds.
+
+This document aims at giving the different configuration variables available to
+limit the resources used by the build system. These variables should be set from
+a :term:`configuration file` and thus take effect over the entire build environment.
+For each variable, also see the variable description in the glossary for more
+details.
+
+-  :term:`BB_NUMBER_THREADS`:
+
+   This sets a hard limit on the number of threads :term:`BitBake` can run at the
+   same time. Lowering this value will set a limit to the number of
+   :term:`BitBake` threads, but will not prevent a single task from starting more
+   compilation threads (see :term:`PARALLEL_MAKE`).
+
+-  :term:`BB_NUMBER_PARSE_THREADS`:
+
+   Like :term:`BB_NUMBER_THREADS`, but this variable sets a limit to the number
+   of threads using during parsing of the environment (before executing tasks).
+
+-  :term:`PARALLEL_MAKE`:
+
+   This variable should be set in the form of ``-jN``, where ``N`` is a positive
+   integer. This integer controls the number of threads used when starting
+   ``make``. Note that this variable is not limited to the usage of ``make``,
+   but extends to the compilation commands defined by the
+   :ref:`ref-classes-meson`, :ref:`ref-classes-cmake` and such classes.
+
+   If you have identified a recipe that you want to limit separately from the
+   rest, it is also possible to achieve with the following syntax::
+
+      PARALLEL_MAKE:pn-linux-yocto = "-j4"
+
+   The above example will limit the number of threads using by ``make`` for the
+   ``linux-yocto`` recipe to 4.
+
+-  :term:`PARALLEL_MAKEINST`:
+
+   Like :term:`PARALLEL_MAKE`, but this variable controls the number of threads
+   used during the :ref:`ref-tasks-install` task.
+
+-  :term:`BB_PRESSURE_MAX_CPU`, :term:`BB_PRESSURE_MAX_IO` and
+   :term:`BB_PRESSURE_MAX_MEMORY`:
+
+   This variable controls the limit of pressure (as defined by
+   https://docs.kernel.org/accounting/psi.html) on the system, and will
+   limit the number of :term:`BitBake` threads dynamically depending on the
+   current pressure of the system.
+
+   These variable take a positive integer between 1 (extremely low limit) and
+   1000000 (value unlikely ever reached). Setting an extremely low value, such
+   as 2, is not desirable as it will result in :term:`BitBake` limiting the
+   number of threads to 1 most of the time.
+
+   However, having a low value at the beginning makes sense to get a sense of a
+   reasonable value to set for your system. Indeed, :term:`BitBake` will print
+   message on the console in the following format each time the current pressure
+   exceeds of the limit set by the above variables::
+
+      Pressure status changed to CPU: True, IO: False, Mem: False (CPU: 1105.9/2.0, IO: 0.0/2.0, Mem: 0.0/2.0) - using 1/64 bitbake threads
+
+   Take a look at the value between parenthesis: ``CPU: 1105.9/2.0, IO: 0.0/2.0,
+   Mem: 0.0/2.0``. They correspond to the current pressure value for the CPU, IO
+   and memory respectively. Monitoring these values during a build should tell
+   you what a reasonable value for your host system is.
+
+-  :term:`BB_LOADFACTOR_MAX`:
+
+   This variable will limit the number of threads :term:`BitBake` will start
+   by monitoring the current load of the host system. :term:`BitBake` will print
+   the following when the limit set by :term:`BB_LOADFACTOR_MAX` is reached::
+
+      Load average limiting set to True as load average: 0.7188262939453125 - using 37/64 bitbake threads
+
+   This variable has no effect when any of :term:`BB_PRESSURE_MAX_CPU`,
+   :term:`BB_PRESSURE_MAX_IO` or :term:`BB_PRESSURE_MAX_MEMORY` is set, as it
+   was designed for system that do not have pressure information available.

---
base-commit: 85e58c4fc5d1be9a2ea9ed6b813d0168e3162dab
change-id: 20250417-resource-limiting-12fe772af904

Best regards,
--  
Antonin Godard <antonin.godard@bootlin.com>

Re: [docs] [PATCH] Add a document on limiting host resources

[Quentin Schulz]

Hi Antonin,

On 6/24/25 4:42 PM, Antonin Godard via lists.yoctoproject.org wrote:
> Add a "Limiting the Host Resources Usage" document to share the
> different techniques that can be used to limit the host resources usage.
> We do have a document to document how to speed up a build, so this
> document comes right after.
> 
> [YOCTO #15111]
> 
> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
> ---
>   documentation/dev-manual/index.rst              |  1 +
>   documentation/dev-manual/limiting-resources.rst | 85 +++++++++++++++++++++++++
>   2 files changed, 86 insertions(+)
> 
> diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
> index 63736e0ab..4abfa59e9 100644
> --- a/documentation/dev-manual/index.rst
> +++ b/documentation/dev-manual/index.rst
> @@ -22,6 +22,7 @@ Yocto Project Development Tasks Manual
>      building
>      multiconfig
>      speeding-up-build
> +   limiting-resources
>      libraries
>      prebuilt-libraries
>      devtool
> diff --git a/documentation/dev-manual/limiting-resources.rst b/documentation/dev-manual/limiting-resources.rst
> new file mode 100644
> index 000000000..df8208bdc
> --- /dev/null
> +++ b/documentation/dev-manual/limiting-resources.rst
> @@ -0,0 +1,85 @@
> +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
> +
> +Limiting the Host Resources Usage
> +*********************************
> +
> +While you sometimes need to :doc:`speed up a build
> +</dev-manual/speeding-up-build>`, you may also need to limit the resources used
> +by the :term:`OpenEmbedded Build System`, especially on shared infrastructures
> +where multiple users start heavy-load builds.
> +

We could also say "or building on low-power machines".

> +This document aims at giving the different configuration variables available to
> +limit the resources used by the build system. These variables should be set from
> +a :term:`configuration file` and thus take effect over the entire build environment.
> +For each variable, also see the variable description in the glossary for more
> +details.
> +
> +-  :term:`BB_NUMBER_THREADS`:
> +
> +   This sets a hard limit on the number of threads :term:`BitBake` can run at the
> +   same time. Lowering this value will set a limit to the number of> +   :term:`BitBake` threads, but will not prevent a single task from 
starting more

I was convinced this was the number of tasks BitBake can run in parallel 
(which is what we say here and there in different places in the doc as 
far as I could tell), but I also see that we have a bunch of calls to 
bb.compress.zstd.open in bbclasses that pass this variable as the number 
of threads to use. Sooo not sure what to trust anymore :)

> +   compilation threads (see :term:`PARALLEL_MAKE`).
> +
> +-  :term:`BB_NUMBER_PARSE_THREADS`:
> +
> +   Like :term:`BB_NUMBER_THREADS`, but this variable sets a limit to the number

s/to/on/ ?

> +   of threads using during parsing of the environment (before executing tasks).

s/using// ?

+the parsing?

> +
> +-  :term:`PARALLEL_MAKE`:
> +
> +   This variable should be set in the form of ``-jN``, where ``N`` is a positive
> +   integer. This integer controls the number of threads used when starting
> +   ``make``. Note that this variable is not limited to the usage of ``make``,
> +   but extends to the compilation commands defined by the
> +   :ref:`ref-classes-meson`, :ref:`ref-classes-cmake` and such classes.
> +

I think it'd be better to say this applies to configure (?) and 
compilation steps, and typically for make, meson, cmake, you name it, 
build systems.

> +   If you have identified a recipe that you want to limit separately from the
> +   rest, it is also possible to achieve with the following syntax::

"""
you want to have a different limit from the rest of the build
"""

?

Please specify where to put this variable (in any configuration file, 
but most likely local.conf I assume).

> +
> +      PARALLEL_MAKE:pn-linux-yocto = "-j4"
> +
> +   The above example will limit the number of threads using by ``make`` for the

s/using/used/

> +   ``linux-yocto`` recipe to 4.
> +
> +-  :term:`PARALLEL_MAKEINST`:
> +
> +   Like :term:`PARALLEL_MAKE`, but this variable controls the number of threads
> +   used during the :ref:`ref-tasks-install` task.
> +
> +-  :term:`BB_PRESSURE_MAX_CPU`, :term:`BB_PRESSURE_MAX_IO` and
> +   :term:`BB_PRESSURE_MAX_MEMORY`:
> +
> +   This variable controls the limit of pressure (as defined by

s/This variable controls/These variables control/

I think we need to state that this requires the host machine's kernel to 
support pressure reporting/handling. I vaguely remember this was 
discussed back when this was implemented.

> +   https://docs.kernel.org/accounting/psi.html) on the system, and will
> +   limit the number of :term:`BitBake` threads dynamically depending on the

Is it threads or tasks here? I also don't entirely remember if this can 
also impact already running tasks (e.g. dynamically reducing the number 
of threads in the available pool during a make call in a recipe which 
started with the limit not reached but is running while the limit is 
reached).

> +   current pressure of the system.
> +
> +   These variable take a positive integer between 1 (extremely low limit) and

s/variable/variables/

> +   1000000 (value unlikely ever reached). Setting an extremely low value, such
> +   as 2, is not desirable as it will result in :term:`BitBake` limiting the
> +   number of threads to 1 most of the time.
> +
> +   However, having a low value at the beginning makes sense to get a sense of a

beginning of what?

> +   reasonable value to set for your system. Indeed, :term:`BitBake` will print
> +   message on the console in the following format each time the current pressure

s/message/messages/ ?

> +   exceeds of the limit set by the above variables::

s/of// ? or s/of/that of/?

> +
> +      Pressure status changed to CPU: True, IO: False, Mem: False (CPU: 1105.9/2.0, IO: 0.0/2.0, Mem: 0.0/2.0) - using 1/64 bitbake threads
> +
> +   Take a look at the value between parenthesis: ``CPU: 1105.9/2.0, IO: 0.0/2.0,

s/value/values/

> +   Mem: 0.0/2.0``. They correspond to the current pressure value for the CPU, IO
> +   and memory respectively. Monitoring these values during a build should tell
> +   you what a reasonable value for your host system is.
> +

Mmmmm... I'm perplexed. If we set the value very low (e.g. 2) won't we 
simply limit the number of threads to something very low which actually 
could mean we're not necessarily nearing the maxing out of the 
CPU/IO/Memory?

It's like running your 4-stroke motor on only one piston: "I see I can 
reach 60kph so this is a safe limit for my motor when not choked up"... 
which yes is true, but now you're not going on the highway because 
you're limiting yourself to 60kph and taking the long way home. Safe 
yes, but not sure we can derive much from this test except a very low 
safe value?

> +-  :term:`BB_LOADFACTOR_MAX`:
> +
> +   This variable will limit the number of threads :term:`BitBake` will start
> +   by monitoring the current load of the host system. :term:`BitBake` will print

I assume we're talking about CPU load here?

> +   the following when the limit set by :term:`BB_LOADFACTOR_MAX` is reached::
> +
> +      Load average limiting set to True as load average: 0.7188262939453125 - using 37/64 bitbake threads
> +
> +   This variable has no effect when any of :term:`BB_PRESSURE_MAX_CPU`,
> +   :term:`BB_PRESSURE_MAX_IO` or :term:`BB_PRESSURE_MAX_MEMORY` is set, as it
> +   was designed for system that do not have pressure information available.

/system/systems/

Hopefully the glossary and this document stay in sync :)

Cheers,
Quentin

Re: [docs] [PATCH] Add a document on limiting host resources

[Antonin Godard]

On Tue Jun 24, 2025 at 5:25 PM CEST, Quentin Schulz wrote:
> Hi Antonin,
>
> On 6/24/25 4:42 PM, Antonin Godard via lists.yoctoproject.org wrote:
>> Add a "Limiting the Host Resources Usage" document to share the
>> different techniques that can be used to limit the host resources usage.
>> We do have a document to document how to speed up a build, so this
>> document comes right after.
>> 
>> [YOCTO #15111]
>> 
>> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
>> ---
>>   documentation/dev-manual/index.rst              |  1 +
>>   documentation/dev-manual/limiting-resources.rst | 85 +++++++++++++++++++++++++
>>   2 files changed, 86 insertions(+)
>> 
>> diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
>> index 63736e0ab..4abfa59e9 100644
>> --- a/documentation/dev-manual/index.rst
>> +++ b/documentation/dev-manual/index.rst
>> @@ -22,6 +22,7 @@ Yocto Project Development Tasks Manual
>>      building
>>      multiconfig
>>      speeding-up-build
>> +   limiting-resources
>>      libraries
>>      prebuilt-libraries
>>      devtool
>> diff --git a/documentation/dev-manual/limiting-resources.rst b/documentation/dev-manual/limiting-resources.rst
>> new file mode 100644
>> index 000000000..df8208bdc
>> --- /dev/null
>> +++ b/documentation/dev-manual/limiting-resources.rst
>> @@ -0,0 +1,85 @@
>> +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
>> +
>> +Limiting the Host Resources Usage
>> +*********************************
>> +
>> +While you sometimes need to :doc:`speed up a build
>> +</dev-manual/speeding-up-build>`, you may also need to limit the resources used
>> +by the :term:`OpenEmbedded Build System`, especially on shared infrastructures
>> +where multiple users start heavy-load builds.
>> +
>
> We could also say "or building on low-power machines".
>
>> +This document aims at giving the different configuration variables available to
>> +limit the resources used by the build system. These variables should be set from
>> +a :term:`configuration file` and thus take effect over the entire build environment.
>> +For each variable, also see the variable description in the glossary for more
>> +details.
>> +
>> +-  :term:`BB_NUMBER_THREADS`:
>> +
>> +   This sets a hard limit on the number of threads :term:`BitBake` can run at the
>> +   same time. Lowering this value will set a limit to the number of> +   :term:`BitBake` threads, but will not prevent a single task from 
> starting more
>
> I was convinced this was the number of tasks BitBake can run in parallel 
> (which is what we say here and there in different places in the doc as 
> far as I could tell), but I also see that we have a bunch of calls to 
> bb.compress.zstd.open in bbclasses that pass this variable as the number 
> of threads to use. Sooo not sure what to trust anymore :)

Shouldn't they use oe.utils.parallel_make() instead? I also assumed
BB_NUMBER_THREADS was limited to Bitbake tasks.

>> +   compilation threads (see :term:`PARALLEL_MAKE`).
>> +
>> +-  :term:`BB_NUMBER_PARSE_THREADS`:
>> +
>> +   Like :term:`BB_NUMBER_THREADS`, but this variable sets a limit to the number
>
> s/to/on/ ?
>
>> +   of threads using during parsing of the environment (before executing tasks).
>
> s/using// ?
>
> +the parsing?
>
>> +
>> +-  :term:`PARALLEL_MAKE`:
>> +
>> +   This variable should be set in the form of ``-jN``, where ``N`` is a positive
>> +   integer. This integer controls the number of threads used when starting
>> +   ``make``. Note that this variable is not limited to the usage of ``make``,
>> +   but extends to the compilation commands defined by the
>> +   :ref:`ref-classes-meson`, :ref:`ref-classes-cmake` and such classes.
>> +
>
> I think it'd be better to say this applies to configure (?) and 
> compilation steps, and typically for make, meson, cmake, you name it, 
> build systems.
>
>> +   If you have identified a recipe that you want to limit separately from the
>> +   rest, it is also possible to achieve with the following syntax::
>
> """
> you want to have a different limit from the rest of the build
> """
>
> ?
>
> Please specify where to put this variable (in any configuration file, 
> but most likely local.conf I assume).
>
>> +
>> +      PARALLEL_MAKE:pn-linux-yocto = "-j4"
>> +
>> +   The above example will limit the number of threads using by ``make`` for the
>
> s/using/used/
>
>> +   ``linux-yocto`` recipe to 4.
>> +
>> +-  :term:`PARALLEL_MAKEINST`:
>> +
>> +   Like :term:`PARALLEL_MAKE`, but this variable controls the number of threads
>> +   used during the :ref:`ref-tasks-install` task.
>> +
>> +-  :term:`BB_PRESSURE_MAX_CPU`, :term:`BB_PRESSURE_MAX_IO` and
>> +   :term:`BB_PRESSURE_MAX_MEMORY`:
>> +
>> +   This variable controls the limit of pressure (as defined by
>
> s/This variable controls/These variables control/
>
> I think we need to state that this requires the host machine's kernel to 
> support pressure reporting/handling. I vaguely remember this was 
> discussed back when this was implemented.

Yes, true, also the last variable kind of implies that but I should also mention
it here.

>> +   https://docs.kernel.org/accounting/psi.html) on the system, and will
>> +   limit the number of :term:`BitBake` threads dynamically depending on the
>
> Is it threads or tasks here? I also don't entirely remember if this can 
> also impact already running tasks (e.g. dynamically reducing the number 
> of threads in the available pool during a make call in a recipe which 
> started with the limit not reached but is running while the limit is 
> reached).

It is only tasks as I understand it. I'm just getting confused by
BB_NUMBER_THREADS which controls the number of tasks, so mixing up words a bit
here.

>> +   current pressure of the system.
>> +
>> +   These variable take a positive integer between 1 (extremely low limit) and
>
> s/variable/variables/
>
>> +   1000000 (value unlikely ever reached). Setting an extremely low value, such
>> +   as 2, is not desirable as it will result in :term:`BitBake` limiting the
>> +   number of threads to 1 most of the time.
>> +
>> +   However, having a low value at the beginning makes sense to get a sense of a
>
> beginning of what?

An "initial low value" would be better maybe, but I wanted to convey that this
variable can/should be adjusted over time.

>> +   reasonable value to set for your system. Indeed, :term:`BitBake` will print
>> +   message on the console in the following format each time the current pressure
>
> s/message/messages/ ?
>
>> +   exceeds of the limit set by the above variables::
>
> s/of// ? or s/of/that of/?
>
>> +
>> +      Pressure status changed to CPU: True, IO: False, Mem: False (CPU: 1105.9/2.0, IO: 0.0/2.0, Mem: 0.0/2.0) - using 1/64 bitbake threads
>> +
>> +   Take a look at the value between parenthesis: ``CPU: 1105.9/2.0, IO: 0.0/2.0,
>
> s/value/values/
>
>> +   Mem: 0.0/2.0``. They correspond to the current pressure value for the CPU, IO
>> +   and memory respectively. Monitoring these values during a build should tell
>> +   you what a reasonable value for your host system is.
>> +
>
> Mmmmm... I'm perplexed. If we set the value very low (e.g. 2) won't we 
> simply limit the number of threads to something very low which actually 
> could mean we're not necessarily nearing the maxing out of the 
> CPU/IO/Memory?
>
> It's like running your 4-stroke motor on only one piston: "I see I can 
> reach 60kph so this is a safe limit for my motor when not choked up"... 
> which yes is true, but now you're not going on the highway because 
> you're limiting yourself to 60kph and taking the long way home. Safe 
> yes, but not sure we can derive much from this test except a very low 
> safe value?

Hmm.. true, so maybe the best option would just to monitor /proc/pressure
without any setting, and then set the value.

>> +-  :term:`BB_LOADFACTOR_MAX`:
>> +
>> +   This variable will limit the number of threads :term:`BitBake` will start
>> +   by monitoring the current load of the host system. :term:`BitBake` will print
>
> I assume we're talking about CPU load here?

Yes, I'll add that.

>> +   the following when the limit set by :term:`BB_LOADFACTOR_MAX` is reached::
>> +
>> +      Load average limiting set to True as load average: 0.7188262939453125 - using 37/64 bitbake threads
>> +
>> +   This variable has no effect when any of :term:`BB_PRESSURE_MAX_CPU`,
>> +   :term:`BB_PRESSURE_MAX_IO` or :term:`BB_PRESSURE_MAX_MEMORY` is set, as it
>> +   was designed for system that do not have pressure information available.
>
> /system/systems/
>
> Hopefully the glossary and this document stay in sync :)

What I can do is revise the current glossary definitions to be minimal and
redirect to this document which would act as a better overall guide on how to
achieve resource limiting. What do you think?

Thanks,
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH] Add a document on limiting host resources

[Quentin Schulz]

Hi Antonin,

On 6/25/25 9:23 AM, Antonin Godard wrote:
> On Tue Jun 24, 2025 at 5:25 PM CEST, Quentin Schulz wrote:
>> Hi Antonin,
>>
>> On 6/24/25 4:42 PM, Antonin Godard via lists.yoctoproject.org wrote:
>>> Add a "Limiting the Host Resources Usage" document to share the
>>> different techniques that can be used to limit the host resources usage.
>>> We do have a document to document how to speed up a build, so this
>>> document comes right after.
>>>
>>> [YOCTO #15111]
>>>
>>> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
>>> ---
>>>    documentation/dev-manual/index.rst              |  1 +
>>>    documentation/dev-manual/limiting-resources.rst | 85 +++++++++++++++++++++++++
>>>    2 files changed, 86 insertions(+)
>>>
>>> diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
>>> index 63736e0ab..4abfa59e9 100644
>>> --- a/documentation/dev-manual/index.rst
>>> +++ b/documentation/dev-manual/index.rst
>>> @@ -22,6 +22,7 @@ Yocto Project Development Tasks Manual
>>>       building
>>>       multiconfig
>>>       speeding-up-build
>>> +   limiting-resources
>>>       libraries
>>>       prebuilt-libraries
>>>       devtool
>>> diff --git a/documentation/dev-manual/limiting-resources.rst b/documentation/dev-manual/limiting-resources.rst
>>> new file mode 100644
>>> index 000000000..df8208bdc
>>> --- /dev/null
>>> +++ b/documentation/dev-manual/limiting-resources.rst
>>> @@ -0,0 +1,85 @@
>>> +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
>>> +
>>> +Limiting the Host Resources Usage
>>> +*********************************
>>> +
>>> +While you sometimes need to :doc:`speed up a build
>>> +</dev-manual/speeding-up-build>`, you may also need to limit the resources used
>>> +by the :term:`OpenEmbedded Build System`, especially on shared infrastructures
>>> +where multiple users start heavy-load builds.
>>> +
>>
>> We could also say "or building on low-power machines".
>>
>>> +This document aims at giving the different configuration variables available to
>>> +limit the resources used by the build system. These variables should be set from
>>> +a :term:`configuration file` and thus take effect over the entire build environment.
>>> +For each variable, also see the variable description in the glossary for more
>>> +details.
>>> +
>>> +-  :term:`BB_NUMBER_THREADS`:
>>> +
>>> +   This sets a hard limit on the number of threads :term:`BitBake` can run at the
>>> +   same time. Lowering this value will set a limit to the number of> +   :term:`BitBake` threads, but will not prevent a single task from
>> starting more
>>
>> I was convinced this was the number of tasks BitBake can run in parallel
>> (which is what we say here and there in different places in the doc as
>> far as I could tell), but I also see that we have a bunch of calls to
>> bb.compress.zstd.open in bbclasses that pass this variable as the number
>> of threads to use. Sooo not sure what to trust anymore :)
> 
> Shouldn't they use oe.utils.parallel_make() instead? I also assumed
> BB_NUMBER_THREADS was limited to Bitbake tasks.
> 

I assume they could?

But maybe it should be ZSTD_THREADS instead?

[...]

>>> +   https://docs.kernel.org/accounting/psi.html) on the system, and will
>>> +   limit the number of :term:`BitBake` threads dynamically depending on the
>>
>> Is it threads or tasks here? I also don't entirely remember if this can
>> also impact already running tasks (e.g. dynamically reducing the number
>> of threads in the available pool during a make call in a recipe which
>> started with the limit not reached but is running while the limit is
>> reached).
> 
> It is only tasks as I understand it. I'm just getting confused by
> BB_NUMBER_THREADS which controls the number of tasks, so mixing up words a bit
> here.
> 

Naming things really is hard :)

>>> +   current pressure of the system.
>>> +
>>> +   These variable take a positive integer between 1 (extremely low limit) and
>>
>> s/variable/variables/
>>
>>> +   1000000 (value unlikely ever reached). Setting an extremely low value, such
>>> +   as 2, is not desirable as it will result in :term:`BitBake` limiting the
>>> +   number of threads to 1 most of the time.
>>> +
>>> +   However, having a low value at the beginning makes sense to get a sense of a
>>
>> beginning of what?
> 
> An "initial low value" would be better maybe, but I wanted to convey that this
> variable can/should be adjusted over time.
> 

I glanced at the BitBake code handling that and some other articles 
(https://unixism.net/2019/08/linux-pressure-stall-information-psi-by-example/ 
and https://lwn.net/Articles/763629/) and this is all still very 
confusing to me. How would I know what to put there? My understanding is 
that total= in "some" line represents the time a thread was starved. In 
BitBake it seems the values in BB_PRESSURE_* variables is about the 
**diff** between the previous total and the current total. We 
essentially allow a given increase in pressure over a specific amount of 
time. My understanding is that this doesn't prevent being OOM killed as 
if the limit is 500 and we keep having increases of 499, we'll increase 
the actual pressure by 499 every second and not trip the pressure 
threshold. I am not entirely sure I understood this correctly or if this 
is actually doing things properly.

Regardless of the above, how should I know what pressure is allowed? It 
very much seems like an experimental value since we essentially want the 
CPU/IO/memory to be under constant load (no cycle wasted) but not too 
much as to trip OOM or do context switches so often it's hurting the 
performance. If the threshold is too low, we'll likely end up not using 
the full CPU/IO/Memory because the load in threads isn't constant and 
could prevent us from scheduling more threads that would make use of 
100% of the resources all the time.

>>> +   reasonable value to set for your system. Indeed, :term:`BitBake` will print
>>> +   message on the console in the following format each time the current pressure
>>
>> s/message/messages/ ?
>>
>>> +   exceeds of the limit set by the above variables::
>>
>> s/of// ? or s/of/that of/?
>>
>>> +
>>> +      Pressure status changed to CPU: True, IO: False, Mem: False (CPU: 1105.9/2.0, IO: 0.0/2.0, Mem: 0.0/2.0) - using 1/64 bitbake threads
>>> +
>>> +   Take a look at the value between parenthesis: ``CPU: 1105.9/2.0, IO: 0.0/2.0,
>>
>> s/value/values/
>>
>>> +   Mem: 0.0/2.0``. They correspond to the current pressure value for the CPU, IO
>>> +   and memory respectively. Monitoring these values during a build should tell
>>> +   you what a reasonable value for your host system is.
>>> +
>>
>> Mmmmm... I'm perplexed. If we set the value very low (e.g. 2) won't we
>> simply limit the number of threads to something very low which actually
>> could mean we're not necessarily nearing the maxing out of the
>> CPU/IO/Memory?
>>
>> It's like running your 4-stroke motor on only one piston: "I see I can
>> reach 60kph so this is a safe limit for my motor when not choked up"...
>> which yes is true, but now you're not going on the highway because
>> you're limiting yourself to 60kph and taking the long way home. Safe
>> yes, but not sure we can derive much from this test except a very low
>> safe value?
> 
> Hmm.. true, so maybe the best option would just to monitor /proc/pressure
> without any setting, and then set the value.
> 

That would work if the build finishes successfully indeed, you would 
know that most likely limiting to this pressure would result in a 
successful build later on as well. If it doesn't though, I guess you 
just need to check this pressure and lower it until the build passes? 
Like bisecting which value actually work.

>>> +-  :term:`BB_LOADFACTOR_MAX`:
>>> +
>>> +   This variable will limit the number of threads :term:`BitBake` will start
>>> +   by monitoring the current load of the host system. :term:`BitBake` will print
>>
>> I assume we're talking about CPU load here?
> 
> Yes, I'll add that.
> 
>>> +   the following when the limit set by :term:`BB_LOADFACTOR_MAX` is reached::
>>> +
>>> +      Load average limiting set to True as load average: 0.7188262939453125 - using 37/64 bitbake threads
>>> +
>>> +   This variable has no effect when any of :term:`BB_PRESSURE_MAX_CPU`,
>>> +   :term:`BB_PRESSURE_MAX_IO` or :term:`BB_PRESSURE_MAX_MEMORY` is set, as it
>>> +   was designed for system that do not have pressure information available.
>>
>> /system/systems/
>>
>> Hopefully the glossary and this document stay in sync :)
> 
> What I can do is revise the current glossary definitions to be minimal and
> redirect to this document which would act as a better overall guide on how to
> achieve resource limiting. What do you think?
> 

They are defined in the BB glossary and not YP and this patch here is 
for YP and not BB docs (maybe it should be in BB docs? or also in BB 
docs?). But could be an idea yes.

Cheers,
Quentin

Re: [docs] [PATCH] Add a document on limiting host resources

[Mikko Rapeli]

Hi,

On Tue, Jun 24, 2025 at 04:42:15PM +0200, Antonin Godard via lists.yoctoproject.org wrote:
> Add a "Limiting the Host Resources Usage" document to share the
> different techniques that can be used to limit the host resources usage.
> We do have a document to document how to speed up a build, so this
> document comes right after.
> 
> [YOCTO #15111]
> 
> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
> ---
>  documentation/dev-manual/index.rst              |  1 +
>  documentation/dev-manual/limiting-resources.rst | 85 +++++++++++++++++++++++++
>  2 files changed, 86 insertions(+)
> 
> diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
> index 63736e0ab..4abfa59e9 100644
> --- a/documentation/dev-manual/index.rst
> +++ b/documentation/dev-manual/index.rst
> @@ -22,6 +22,7 @@ Yocto Project Development Tasks Manual
>     building
>     multiconfig
>     speeding-up-build
> +   limiting-resources
>     libraries
>     prebuilt-libraries
>     devtool
> diff --git a/documentation/dev-manual/limiting-resources.rst b/documentation/dev-manual/limiting-resources.rst
> new file mode 100644
> index 000000000..df8208bdc
> --- /dev/null
> +++ b/documentation/dev-manual/limiting-resources.rst
> @@ -0,0 +1,85 @@
> +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
> +
> +Limiting the Host Resources Usage
> +*********************************
> +
> +While you sometimes need to :doc:`speed up a build
> +</dev-manual/speeding-up-build>`, you may also need to limit the resources used
> +by the :term:`OpenEmbedded Build System`, especially on shared infrastructures
> +where multiple users start heavy-load builds.
> +
> +This document aims at giving the different configuration variables available to
> +limit the resources used by the build system. These variables should be set from
> +a :term:`configuration file` and thus take effect over the entire build environment.
> +For each variable, also see the variable description in the glossary for more
> +details.
> +
> +-  :term:`BB_NUMBER_THREADS`:
> +
> +   This sets a hard limit on the number of threads :term:`BitBake` can run at the
> +   same time. Lowering this value will set a limit to the number of
> +   :term:`BitBake` threads, but will not prevent a single task from starting more
> +   compilation threads (see :term:`PARALLEL_MAKE`).
> +

I would add here or to intro that confusingly, CPU load caused by too much parallellism
may not be the problem but running out of physical RAM which trigger kernel
out-of-memory killer to kill processes related to builds causing very
odd looking failures in build logs. The oom killer actions are visible in build host
kernel dmesg logs, but only there. Thus if machines has low physical RAM
to CPU thread ratio, say less than 2 Gb, then limiting both BB_NUMBER_THREADS
and PARALLEL_MAKE can help.

Cheers,

-Mikko

> +-  :term:`BB_NUMBER_PARSE_THREADS`:
> +
> +   Like :term:`BB_NUMBER_THREADS`, but this variable sets a limit to the number
> +   of threads using during parsing of the environment (before executing tasks).
> +
> +-  :term:`PARALLEL_MAKE`:
> +
> +   This variable should be set in the form of ``-jN``, where ``N`` is a positive
> +   integer. This integer controls the number of threads used when starting
> +   ``make``. Note that this variable is not limited to the usage of ``make``,
> +   but extends to the compilation commands defined by the
> +   :ref:`ref-classes-meson`, :ref:`ref-classes-cmake` and such classes.
> +
> +   If you have identified a recipe that you want to limit separately from the
> +   rest, it is also possible to achieve with the following syntax::
> +
> +      PARALLEL_MAKE:pn-linux-yocto = "-j4"
> +
> +   The above example will limit the number of threads using by ``make`` for the
> +   ``linux-yocto`` recipe to 4.
> +
> +-  :term:`PARALLEL_MAKEINST`:
> +
> +   Like :term:`PARALLEL_MAKE`, but this variable controls the number of threads
> +   used during the :ref:`ref-tasks-install` task.
> +
> +-  :term:`BB_PRESSURE_MAX_CPU`, :term:`BB_PRESSURE_MAX_IO` and
> +   :term:`BB_PRESSURE_MAX_MEMORY`:
> +
> +   This variable controls the limit of pressure (as defined by
> +   https://docs.kernel.org/accounting/psi.html) on the system, and will
> +   limit the number of :term:`BitBake` threads dynamically depending on the
> +   current pressure of the system.
> +
> +   These variable take a positive integer between 1 (extremely low limit) and
> +   1000000 (value unlikely ever reached). Setting an extremely low value, such
> +   as 2, is not desirable as it will result in :term:`BitBake` limiting the
> +   number of threads to 1 most of the time.
> +
> +   However, having a low value at the beginning makes sense to get a sense of a
> +   reasonable value to set for your system. Indeed, :term:`BitBake` will print
> +   message on the console in the following format each time the current pressure
> +   exceeds of the limit set by the above variables::
> +
> +      Pressure status changed to CPU: True, IO: False, Mem: False (CPU: 1105.9/2.0, IO: 0.0/2.0, Mem: 0.0/2.0) - using 1/64 bitbake threads
> +
> +   Take a look at the value between parenthesis: ``CPU: 1105.9/2.0, IO: 0.0/2.0,
> +   Mem: 0.0/2.0``. They correspond to the current pressure value for the CPU, IO
> +   and memory respectively. Monitoring these values during a build should tell
> +   you what a reasonable value for your host system is.
> +
> +-  :term:`BB_LOADFACTOR_MAX`:
> +
> +   This variable will limit the number of threads :term:`BitBake` will start
> +   by monitoring the current load of the host system. :term:`BitBake` will print
> +   the following when the limit set by :term:`BB_LOADFACTOR_MAX` is reached::
> +
> +      Load average limiting set to True as load average: 0.7188262939453125 - using 37/64 bitbake threads
> +
> +   This variable has no effect when any of :term:`BB_PRESSURE_MAX_CPU`,
> +   :term:`BB_PRESSURE_MAX_IO` or :term:`BB_PRESSURE_MAX_MEMORY` is set, as it
> +   was designed for system that do not have pressure information available.
> 
> ---
> base-commit: 85e58c4fc5d1be9a2ea9ed6b813d0168e3162dab
> change-id: 20250417-resource-limiting-12fe772af904
> 
> Best regards,
> --  
> Antonin Godard <antonin.godard@bootlin.com>
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#7215): https://lists.yoctoproject.org/g/docs/message/7215
> Mute This Topic: https://lists.yoctoproject.org/mt/113808179/7159507
> Group Owner: docs+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/docs/unsub [mikko.rapeli@linaro.org]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [docs] [PATCH] Add a document on limiting host resources

[Antonin Godard]

On Wed Jun 25, 2025 at 8:10 AM CEST, Mikko Rapeli wrote:
> Hi,
>
> On Tue, Jun 24, 2025 at 04:42:15PM +0200, Antonin Godard via lists.yoctoproject.org wrote:
>> Add a "Limiting the Host Resources Usage" document to share the
>> different techniques that can be used to limit the host resources usage.
>> We do have a document to document how to speed up a build, so this
>> document comes right after.
>> 
>> [YOCTO #15111]
>> 
>> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
>> ---
>>  documentation/dev-manual/index.rst              |  1 +
>>  documentation/dev-manual/limiting-resources.rst | 85 +++++++++++++++++++++++++
>>  2 files changed, 86 insertions(+)
>> 
>> diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
>> index 63736e0ab..4abfa59e9 100644
>> --- a/documentation/dev-manual/index.rst
>> +++ b/documentation/dev-manual/index.rst
>> @@ -22,6 +22,7 @@ Yocto Project Development Tasks Manual
>>     building
>>     multiconfig
>>     speeding-up-build
>> +   limiting-resources
>>     libraries
>>     prebuilt-libraries
>>     devtool
>> diff --git a/documentation/dev-manual/limiting-resources.rst b/documentation/dev-manual/limiting-resources.rst
>> new file mode 100644
>> index 000000000..df8208bdc
>> --- /dev/null
>> +++ b/documentation/dev-manual/limiting-resources.rst
>> @@ -0,0 +1,85 @@
>> +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
>> +
>> +Limiting the Host Resources Usage
>> +*********************************
>> +
>> +While you sometimes need to :doc:`speed up a build
>> +</dev-manual/speeding-up-build>`, you may also need to limit the resources used
>> +by the :term:`OpenEmbedded Build System`, especially on shared infrastructures
>> +where multiple users start heavy-load builds.
>> +
>> +This document aims at giving the different configuration variables available to
>> +limit the resources used by the build system. These variables should be set from
>> +a :term:`configuration file` and thus take effect over the entire build environment.
>> +For each variable, also see the variable description in the glossary for more
>> +details.
>> +
>> +-  :term:`BB_NUMBER_THREADS`:
>> +
>> +   This sets a hard limit on the number of threads :term:`BitBake` can run at the
>> +   same time. Lowering this value will set a limit to the number of
>> +   :term:`BitBake` threads, but will not prevent a single task from starting more
>> +   compilation threads (see :term:`PARALLEL_MAKE`).
>> +
>
> I would add here or to intro that confusingly, CPU load caused by too much parallellism
> may not be the problem but running out of physical RAM which trigger kernel
> out-of-memory killer to kill processes related to builds causing very
> odd looking failures in build logs. The oom killer actions are visible in build host
> kernel dmesg logs, but only there. Thus if machines has low physical RAM
> to CPU thread ratio, say less than 2 Gb, then limiting both BB_NUMBER_THREADS
> and PARALLEL_MAKE can help.
>
> Cheers,
>
> -Mikko

That's a good idea for the v2, thank you. I recollect having issues in the
linking stage of some RAM-hungry recipes:

  collect2: error: ld returned 1 exit status

But no real error messages above - so looking at the kernel dmesg is a good tip
here indeed.

Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com