Re: [OE-core] [PATCH v2 3/5] ninja: build modified version with GNU Make jobserver support -> (jobserver, loadfactor, pressure and all that!)

[Randy MacLeod <randy.macleod@windriver.com>]

[-- Attachment #1: Type: text/plain, Size: 3503 bytes --]

On 2024-12-02 3:07 p.m., Martin Hundebøll wrote:
> Hi Randy,
>
> On Sun, 2024-11-17 at 21:29 -0500, Randy MacLeod wrote:
>> On 2024-08-02 6:41 a.m., Alexandre Belloni wrote:
>>> On 02/08/2024 12:24:06+0200, Martin Hundeb?ll wrote:
>>>> Hi Alexandre,
>>>>
>>>> I see you keep carrying these patches. I'm still working with
>>>> ninja
>>>> upstream to add jobserver support there[1]. I was planning to
>>>> redo
>>>> these patches once ninja merges that. But I can do a respin with
>>>> my
>>>> patches added to ninja if you like?
>>>>
>>>> [1]https://github.com/ninja-build/ninja/pull/2450
>>> I carry them so they get some testing, I can drop until you submit
>>> the new version.
>>>    
>> Hi Martin, et. al.
>>   
>> I'm going to do some experimenting with the job server patches for a
>>   talk that somehow got approved for the 2024.12 YP summit!
>>     https://pretalx.com/yocto-project-summit-2024-12/talk/WGPUTX/
>>
> Thanks for picking this up. I got assigned to other projects, so this
> was put on hold for now.

Hi Martin et. al.,

Errr, some things came up and I didn't feel that I had enough new to say so
I'm not presenting this week.


Hopefully as you say, the ninja patch merges and I can get time to do 
more evaluation
and comparison and one of us can get a patch merged to master.

If all goes well, there will be a presentation at the next YP Summit!

Thanks,

../Randy



>
>> As expected, the patch doesn't apply with all that has changed since
>> April.
>>
>> Do you have a a branch that you are keeping updated.
>> If so can you same me some time otherwise, I can rebase and fix
>> things up.
> I have no branch, no. But I did try to get jobserver client support
> into ninja. My PR got replaced by two other PRs, and I assume you are
> following the progress on the latest one:
> https://github.com/ninja-build/ninja/pull/2506
>
> It looks like that PR will go into ninja eventually.
>
> The qemu patches are no longer needed, so just the ninja patches, the
> jobserver bbclass, and the python server script that are needed.
>
>> I'll of course give you all the credit for the real work.
>> I plan to mention the performance/behaviour improvements that you
>> mentioned in your cover letter:
>>   
>> ---
>> On build machines shared by multiple users, a single jobserver can be
>> shared between multiple builds (using the JOBSERVER_FIFO variable).
>> Running the above build in two different build directories at the
>> same time gives a ~12% improvement (43:17 -> 37:55).
>>
>> Finally, the memory pressure from e.g. compiling multiple c++ based
>> projects is also reduced. In our case, a cloud based build machine
>> (with 32 cores and 32GB RAM) fails to compile llvm-rust-native (in
>> parallel to nodejs) without the jobserver due to a lack of memory.
>> ---
>>   
>> Is there anything else you'd like people to know about?
> Feel free to include my comment wherever you see fit. The credit is
> fine, but not important to me. Again, do what you think is appropriate.
>
>> Does anyone else have jobserver, BB_PRESSURE_*, BB_LOADFACTOR_MAX use
>> cases or benchmarks that they'd like to share with the YP communtiy?
>> All input is welcome and appreciated.
> There is a comment on the github PR that mentions yocto / bitbake:
> https://github.com/ninja-build/ninja/pull/2506#issuecomment-2437267474
>
> Good luck :)
>
> // Martin


-- 
# Randy MacLeod
# Wind River Linux

[-- Attachment #2: Type: text/html, Size: 5483 bytes --]