Re: [yocto] bitbake controlling memory use

[Ferry Toth <fntoth@gmail.com>]

Hi,

Op 10-01-2023 om 10:24 schreef Ferry Toth:
> Hi,
>
> On 10-01-2023 02:57, Randy MacLeod wrote:
>> Add Zheng.
>> Switch to Trevor's gmail since he might still be interested in this.
>>
>> On 2023-01-09 05:49, Ferry Toth via lists.yoctoproject.org wrote:
>>> Hi
>>>
>>> On 09-01-2023 11:39, Alexander Kanavin wrote:
>>>> On Sun, 8 Jan 2023 at 23:13, Ferry Toth <fntoth@gmail.com> wrote:
>>>>
>>>>> Now it works I'm not sure what to do. Richard marked the original 
>>>>> patch
>>>>> as WIP. Maybe it's not appropriate for including into poky? If so I
>>>>> wouldn't mind carrying it in meta-intel-edison.
>>>>>
>>>>> Or may be with a little work (from me) and a run on the CI servers we
>>>>> could make it go in?
>>>> I'd rather teach make/ninja upstream to watch memory consumption
>>>> and/or host pressure directly (e.g. similar to how it handles -l). And
>>>> offer any resulting patches to upstream first.
>>
As there is already a clone of ninja available with jobserver support 
available I added another patch to update to that version and make an 
intercept to actually pass the named pipe to ninja.

Find it here: https://github.com/htot/poky/commits/kirkstone

This makes both make and ninja threads running from a shared thread 
pool. As cmake relies on ninja recipies are covered as well.

I guess this takes care of the majority of recipes.

In my case this new patch shaves no more then 1 minutes from my image 
build time (1h46) as the recipes built using ninja are not consuming 
that much memory (like nodejs). I hope others will benefit more.

Todo: location of the pipe as mentioned by Richard

>> Zheng and I *started* on that for make over the Xmas holiday.
>> See the (poorly formatted) thread:
>>   Add support for limiting CPU pressure, contrib, 2022/12/20
>> in:
>> https://lists.gnu.org/archive/html/bug-make/2022-12/threads.html
>>
>> There were mixed reviews upstream with the maintainer, Paul Smith,
>> seeming to prefer that we investigate the job server approach and the 
>> current
>> load averaging. I'll happily try to find time to play with Ferry's 
>> job server work.
>>
> The work is actually Richard's. I only fixed what broke over time.
>> I've been thinking about the various workflows and as Richard said, 
>> it seems
>> that for many people who only do one build at a time, the job server 
>> and maybe
>> a little linker restraint, would be all that's needed. For activities 
>> such
>> as YP AB, we could teach the main job server to also look at 
>> /proc/pressure
>> as a way to limit the pool size we could make a meta-jobserver for 
>> those who don't
>> want/need to worry about non-compile tasks such as tests and build 
>> clean-up.
>>
>>
>> Note that cargo has the notion of a job server:
>>    https://github.com/rust-lang/cargo/issues/1744
>>    https://github.com/rust-lang/cargo/pull/4110
>>
>>
>> and ninja has an open issue:
>>    https://github.com/ninja-build/ninja/issues/1139
>> and an initial implementation:
>>    https://github.com/stefanb2/ninja/tree/topic-jobserver-fifo
>>
>> What other build tools are in need of regulation and/or job server 
>> patches?
>>
> What I read, gcc has already -flto=jobserver.
>
> Other (single threaded CPU intensive) might just be started from 
> jobclient?
>
>> ../Randy
>>
>>
>>>>
>>>> Alex
>>>
>>> Yeah, I'd rather teach the kernel to consider thrashing when 
>>> scheduling jobs. As is now any user process can slow down any other 
>>> users process and even the kernel itself to a standstill.
>>>
>>> But kernel developers consider those issues "orthogonal" (i.e. they 
>>> don't want to make the scheduler aware of io).
>>>
>>>
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-
>>> Links: You receive all messages sent to this group.
>>> View/Reply Online (#58943): 
>>> https://lists.yoctoproject.org/g/yocto/message/58943
>>> Mute This Topic: https://lists.yoctoproject.org/mt/82015730/3616765
>>> Group Owner: yocto+owner@lists.yoctoproject.org
>>> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub 
>>> [randy.macleod@windriver.com]
>>> -=-=-=-=-=-=-=-=-=-=-=-
>>>
>>