Re: [RFC 0/4] socketed based data connection

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Wed, 2010-12-22 at 00:49 +0800, Qing He wrote:
> This patch set enables remote data access in the exec'd task process.
> 
> Some test data:
> I've conducted a poky-image-minimal build on two different machines
> with and without the patch, the result are as follow:
> 
> 2 core weybridge: worker count=4, make -j4
>   without patch
>     real: 161min
>     user: 237min
>     sys:  56min
>   with patch
>     real: 138min
>     user: 196min
>     sys:  45min
> 4 core nehalem: worker count=4, make -j4
>   without patch
>     real: 88min
>     user: 218min
>     sys:  37min
>   with patch
>     real: 83min
>     user: 160min
>     sys:  29min

For comparison what numbers does master give? I'm assuming these numbers
were with master using exec() so I'd be interested in the comparisons
with fork().

> the user+sys has an improvement of around 20%~30%, however the
> real runtime improvement is less significant. It's probably
> because of the thread workload unbalance on multicore, or in another
> word, some CPUs are not fully loaded during the build. e.g. when
> do_rootfs_rpm is run, there is probably no other tasks running.
> 
> Haven't done any test against single thread running, but it should
> have a much more considerable impact.
> 
> 
> Known problems:
> 1. fetcher is not working at the moment, maybe bitbake needs an explicit
> InheritFromOS(), but that produces some inconsistency in PATH in my previous
> run
> 2. do_rootfs_rpm has some problem in the first run
> 
> 
> Next step and possible future usage:
> 1. organize remote data to a separate class
> 2. although we have reverted to fork for task running, a remote data mechanism
> would work for a more separate build (e.g. distributed build)

It could also be useful for UI interaction with the datastore.

> 3. the added code in bitbake-runtask is to run necessary initializations for a
> working environment, this is part of cooker. The current cooker seems too heavy
> and may be broken down into several pieces according to functionalitiies

I certainly agree cooker could use some serious refactoring in some
form.

Cheers,

Richard