Re: [OE-core] [PATCH v2 3/5] ninja: build modified version with GNU Make jobserver support -> (jobserver, loadfactor, pressure and all that!)

["Martin Hundebøll" <martin@geanix.com>]

On Mon, 2024-12-02 at 15:25 -0500, Randy MacLeod wrote:
> On 2024-12-02 3:07 p.m., Martin Hundebøll wrote: 
> > Hi Randy,
> > On Sun, 2024-11-17 at 21:29 -0500, Randy MacLeod wrote: 
> > > On 2024-08-02 6:41 a.m., Alexandre Belloni wrote:
> > > > On 02/08/2024 12:24:06+0200, Martin Hundeb?ll wrote:
> > > > > Hi Alexandre,
> > > > > 
> > > > > I see you keep carrying these patches. I'm still working with
> > > > > ninja
> > > > > upstream to add jobserver support there[1]. I was planning to
> > > > > redo
> > > > > these patches once ninja merges that. But I can do a respin
> > > > > with
> > > > > my
> > > > > patches added to ninja if you like?
> > > > > 
> > > > > [1] https://github.com/ninja-build/ninja/pull/2450
> > > > >  
> > > > I carry them so they get some testing, I can drop until you
> > > > submit
> > > > the new version.
> > > >   
> > > Hi Martin, et. al.
> > >  
> > > I'm going to do some experimenting with the job server patches
> > > for a 
> > >  talk that somehow got approved for the 2024.12 YP summit! 
> > >    https://pretalx.com/yocto-project-summit-2024-12/talk/WGPUTX/
> > >  
> > Thanks for picking this up. I got assigned to other projects, so
> > this was put on hold for now.
> >  
> Hi Martin et. al., 
> 
> Errr, some things came up and I didn't feel that I had enough new to
> say so I'm not presenting this week. 
> 
> Hopefully as you say, the ninja patch merges and I can get time to do
> more evaluation  and comparison and one of us can get a patch merged
> to master.
> 
> If all goes well, there will be a presentation at the next YP Summit!

Hi Randy,

No problem. We'll see if/when one of us get the time to look into this
again.

// Martin
> 
> > >  
> > > As expected, the patch doesn't apply with all that has changed
> > > since
> > > April.
> > > 
> > > Do you have a a branch that you are keeping updated.
> > > If so can you same me some time otherwise, I can rebase and fix
> > > things up.
> > >  
> >  
> > I have no branch, no. But I did try to get jobserver client support
> > into ninja. My PR got replaced by two other PRs, and I assume you
> > are
> > following the progress on the latest one:
> > https://github.com/ninja-build/ninja/pull/2506
> > 
> > It looks like that PR will go into ninja eventually.
> > 
> > The qemu patches are no longer needed, so just the ninja patches,
> > the
> > jobserver bbclass, and the python server script that are needed.
> > 
> >  
> > >  
> > > I'll of course give you all the credit for the real work.
> > > I plan to mention the performance/behaviour improvements that you
> > > mentioned in your cover letter:
> > >  
> > > ---   
> > > On build machines shared by multiple users, a single jobserver
> > > can be
> > > shared between multiple builds (using the JOBSERVER_FIFO
> > > variable).
> > > Running the above build in two different build directories at the
> > > same time gives a ~12% improvement (43:17 -> 37:55).
> > > 
> > > Finally, the memory pressure from e.g. compiling multiple c++
> > > based
> > > projects is also reduced. In our case, a cloud based build
> > > machine
> > > (with 32 cores and 32GB RAM) fails to compile llvm-rust-native
> > > (in
> > > parallel to nodejs) without the jobserver due to a lack of
> > > memory.
> > > ---
> > >  
> > > Is there anything else you'd like people to know about?
> > >  
> >  
> > Feel free to include my comment wherever you see fit. The credit is
> > fine, but not important to me. Again, do what you think is
> > appropriate.
> > 
> >  
> > >  
> > > Does anyone else have jobserver, BB_PRESSURE_*, BB_LOADFACTOR_MAX
> > > use
> > > cases or benchmarks that they'd like to share with the YP
> > > communtiy?
> > > All input is welcome and appreciated.
> > >  
> >  
> > There is a comment on the github PR that mentions yocto / bitbake:
> > https://github.com/ninja-build/ninja/pull/2506#issuecomment-2437267474
> > 
> > Good luck :)
> > 
> > // Martin
> >  
>  
> 
>  
>