Re: [docs] [PATCH] Add a document on limiting host resources

["Antonin Godard" <antonin.godard@bootlin.com>]

On Wed Jun 25, 2025 at 8:10 AM CEST, Mikko Rapeli wrote:
> Hi,
>
> On Tue, Jun 24, 2025 at 04:42:15PM +0200, Antonin Godard via lists.yoctoproject.org wrote:
>> Add a "Limiting the Host Resources Usage" document to share the
>> different techniques that can be used to limit the host resources usage.
>> We do have a document to document how to speed up a build, so this
>> document comes right after.
>> 
>> [YOCTO #15111]
>> 
>> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
>> ---
>>  documentation/dev-manual/index.rst              |  1 +
>>  documentation/dev-manual/limiting-resources.rst | 85 +++++++++++++++++++++++++
>>  2 files changed, 86 insertions(+)
>> 
>> diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
>> index 63736e0ab..4abfa59e9 100644
>> --- a/documentation/dev-manual/index.rst
>> +++ b/documentation/dev-manual/index.rst
>> @@ -22,6 +22,7 @@ Yocto Project Development Tasks Manual
>>     building
>>     multiconfig
>>     speeding-up-build
>> +   limiting-resources
>>     libraries
>>     prebuilt-libraries
>>     devtool
>> diff --git a/documentation/dev-manual/limiting-resources.rst b/documentation/dev-manual/limiting-resources.rst
>> new file mode 100644
>> index 000000000..df8208bdc
>> --- /dev/null
>> +++ b/documentation/dev-manual/limiting-resources.rst
>> @@ -0,0 +1,85 @@
>> +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
>> +
>> +Limiting the Host Resources Usage
>> +*********************************
>> +
>> +While you sometimes need to :doc:`speed up a build
>> +</dev-manual/speeding-up-build>`, you may also need to limit the resources used
>> +by the :term:`OpenEmbedded Build System`, especially on shared infrastructures
>> +where multiple users start heavy-load builds.
>> +
>> +This document aims at giving the different configuration variables available to
>> +limit the resources used by the build system. These variables should be set from
>> +a :term:`configuration file` and thus take effect over the entire build environment.
>> +For each variable, also see the variable description in the glossary for more
>> +details.
>> +
>> +-  :term:`BB_NUMBER_THREADS`:
>> +
>> +   This sets a hard limit on the number of threads :term:`BitBake` can run at the
>> +   same time. Lowering this value will set a limit to the number of
>> +   :term:`BitBake` threads, but will not prevent a single task from starting more
>> +   compilation threads (see :term:`PARALLEL_MAKE`).
>> +
>
> I would add here or to intro that confusingly, CPU load caused by too much parallellism
> may not be the problem but running out of physical RAM which trigger kernel
> out-of-memory killer to kill processes related to builds causing very
> odd looking failures in build logs. The oom killer actions are visible in build host
> kernel dmesg logs, but only there. Thus if machines has low physical RAM
> to CPU thread ratio, say less than 2 Gb, then limiting both BB_NUMBER_THREADS
> and PARALLEL_MAKE can help.
>
> Cheers,
>
> -Mikko

That's a good idea for the v2, thank you. I recollect having issues in the
linking stage of some RAM-hungry recipes:

  collect2: error: ld returned 1 exit status

But no real error messages above - so looking at the kernel dmesg is a good tip
here indeed.

Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com