Re: [OE-core] [PATCH v2 3/5] ninja: build modified version with GNU Make jobserver support -> (jobserver, loadfactor, pressure and all that!)

["Martin Hundebøll" <martin@geanix.com>]

Hi Randy,

On Sun, 2024-11-17 at 21:29 -0500, Randy MacLeod wrote:
> On 2024-08-02 6:41 a.m., Alexandre Belloni wrote:
> > On 02/08/2024 12:24:06+0200, Martin Hundeb?ll wrote:
> > > Hi Alexandre,
> > > 
> > > I see you keep carrying these patches. I'm still working with
> > > ninja
> > > upstream to add jobserver support there[1]. I was planning to
> > > redo
> > > these patches once ninja merges that. But I can do a respin with
> > > my
> > > patches added to ninja if you like?
> > > 
> > > [1] https://github.com/ninja-build/ninja/pull/2450
> > 
> > I carry them so they get some testing, I can drop until you submit
> > the new version.
> >   
> Hi Martin, et. al.
>  
> I'm going to do some experimenting with the job server patches for a 
>  talk that somehow got approved for the 2024.12 YP summit! 
>    https://pretalx.com/yocto-project-summit-2024-12/talk/WGPUTX/
> 


Thanks for picking this up. I got assigned to other projects, so this
was put on hold for now.

> As expected, the patch doesn't apply with all that has changed since
> April.
> 
> Do you have a a branch that you are keeping updated.
> If so can you same me some time otherwise, I can rebase and fix
> things up.

I have no branch, no. But I did try to get jobserver client support
into ninja. My PR got replaced by two other PRs, and I assume you are
following the progress on the latest one:
https://github.com/ninja-build/ninja/pull/2506

It looks like that PR will go into ninja eventually.

The qemu patches are no longer needed, so just the ninja patches, the
jobserver bbclass, and the python server script that are needed.

> I'll of course give you all the credit for the real work.
> I plan to mention the performance/behaviour improvements that you
> mentioned in your cover letter:
>  
> ---   
> On build machines shared by multiple users, a single jobserver can be
> shared between multiple builds (using the JOBSERVER_FIFO variable).
> Running the above build in two different build directories at the
> same time gives a ~12% improvement (43:17 -> 37:55).
> 
> Finally, the memory pressure from e.g. compiling multiple c++ based
> projects is also reduced. In our case, a cloud based build machine
> (with 32 cores and 32GB RAM) fails to compile llvm-rust-native (in
> parallel to nodejs) without the jobserver due to a lack of memory.
> ---
>  
> Is there anything else you'd like people to know about?

Feel free to include my comment wherever you see fit. The credit is
fine, but not important to me. Again, do what you think is appropriate.

> Does anyone else have jobserver, BB_PRESSURE_*, BB_LOADFACTOR_MAX use
> cases or benchmarks that they'd like to share with the YP communtiy?
> All input is welcome and appreciated.

There is a comment on the github PR that mentions yocto / bitbake:
https://github.com/ninja-build/ninja/pull/2506#issuecomment-2437267474

Good luck :)

// Martin