Re: [PATCH 1/1] sstate.bbclass: fix parallel building issue

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Fri, 2013-08-16 at 17:05 +0800, Rongqing Li wrote:
> 
> On 08/16/2013 04:25 PM, Rongqing Li wrote:
> >>> Do we need some kind of a read/write lock on accessing those files.
> >>> (Is this
> >>> even something that we can do easily though the existing mechanisms?)
> >>
> >> It would kill performance for no good reason, been there, looked at
> >> it...
> >>
> >> Cheers,
> >>
> >> Richard
> >>
> >
> > I think reverting the below optimization maybe better than using lock
> >
> > commit 8c5544c2311b080bb212efb7f6b804db63e125f5
> > Author: Richard Purdie <richard.purdie@linuxfoundation.org>
> > Date:   Thu Oct 11 13:36:53 2012 +0100
> >
> >      scripts/cp-noerror: Try and use hardlinks if possible
> >
> >      Since we generally have lots of copies of the directories created
> > using this tool, use
> >      hardlinks where possible. This should save a little disk space and
> > improve performance
> >      slightly.
> >
> >      (From OE-Core rev: bfa11c028c2da093f7b4e6b7b1d611da90ae052f)
> >
> >      Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> >
> >
> > -Roy
> >
> >
> 
> I think the upper commit saves lots of space, but the saved time maybe 
> be ignored.

Its more that we have less files bouncing around the kernel so the disk
IO queues can be used for more useful stuff. You wouldn't hit blocked IO
in your tests above. The commit says space savings were the primary
benefit, the speed/IO is just a nice bonus and can't hurt.

I think Phil is right, we should look at fixing aclocal so file
disappeared errors are just handled gracefully, then this whole mess can
go away and we'll get even better performance.

I like the idea of iterating DEPENDS and figuring out which ones to add
but that is quite a bit more work. Ultimately it would be worthwhile
though and the same logic could then be used for a sysroot per workdir
type logic.

Cheers,

Richard