Re: bitbake controlling memory use

["Gmane Admin" <gley-yocto@m.gmane-mx.org>]

Op 05-06-2021 om 15:35 schreef Gmane Admin:
> Op 14-04-2021 om 06:59 schreef Richard Purdie:
>> On Tue, 2021-04-13 at 21:14 -0400, Randy MacLeod wrote:
>>> On 2021-04-11 12:19 p.m., Alexander Kanavin wrote:
>>>> make already has -l option for limiting new instances if load 
>>>> average is
>>>> too high, so it's only natural to add a RAM limiter too.
>>>>
>>>>     -l [N], --load-average[=N], --max-load[=N]
>>>>                                 Don't start multiple jobs unless 
>>>> load is
>>>> below N.
>>>>
>>>> In any case, patches welcome :)
>>>
>>> During today's Yocto technical call (1),
>>> we talked about approaches to limiting the system load and avoiding
>>> swap and/or OOM events. Here's what (little!) i recall from the
>>> discussion, 9 busy hours later.
>>>
>>> In the short run, instead of independently maintaining changes to
>>> configurations to limit parallelism or xz memory usage, etc, we
>>> could develop an optional common include file where such limits
>>> are shared across the community.
>>>
>>> In the longer run, changes to how bitbake schedules work may be needed.
>>>
>>> Richard says that there was a make/build server idea and maybe even a
>>> patch from a while ago. It may be in one of his poky-contrib branches.
>>> I took a few minutes to look but nothing popped up. A set of keywords to
>>> search for might help me find it.
>>
>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wipqueue4&id=d66a327fb6189db5de8bc489859235dcba306237 

This patch resolves a starvation of a particular resource (execution 
cores), which is good.
However, the problem I am facing is starvation of another resource (memory).

>> Cheers,
>>
>> Richard
>>
>>
> I like the idea. Unfortunately the patch doesn't apply to Gatesgarth, so 
> I couldn't test it. Any chance you would be doing a refresh?

Ok so I refreshed this patch my self and it seems to be working nicely 
(3000 out of 4000 tasks complete), except for one thing: do_configure 
for cmake-native fails and I don't see why. From the log:

loading initial cache file 
xxxxxx/out/linux64/build/tmp/work/x86_64-linux/cmake-native/3.18.2-r0/build/Bootstrap.cmk/InitialCacheFlags.cmake
-- The C compiler identification is GNU 10.3.0
-- The CXX compiler identification is GNU 10.3.0
-- Detecting C compiler ABI info
CMake Error: Generator: execution of make failed. Make command was: 
xxxxxx/out/linux64/poky/scripts/make-intercept/make cmTC_68352/fast &&
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: xxxxxx/out/linux64/build/tmp/hosttools/gcc
CMake Error: Generator: execution of make failed. Make command was: 
xxxxxx/out/linux64/poky/scripts/make-intercept/make cmTC_f23a0/fast &&
-- Check for working C compiler: 
xxxxxx/out/linux64/build/tmp/hosttools/gcc - broken
CMake Error at Modules/CMakeTestCCompiler.cmake:66 (message):
   The C compiler

     "xxxxxx/out/linux64/build/tmp/hosttools/gcc"

   is not able to compile a simple test program.

   It fails with the following output:

     Change Dir: 
xxxxxx/tmp/work/x86_64-linux/cmake-native/3.18.2-r0/build/CMakeFiles/CMakeTmp

     Run Build 
Command(s):xxxxxx/out/linux64/poky/scripts/make-intercept/make 
cmTC_f23a0/fast && Permission denied
     Generator: execution of make failed. Make command was: 
xxxxxx/out/linux64/poky/scripts/make-intercept/make cmTC_f23a0/fast &&

   CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
   CMakeLists.txt:7 (project)

Crazy. I don't see why making a complete recipe works fine, while making 
a test program during configure fails. Ideas?