Re: [RFC 0/4] socketed based data connection

[Qing He <qing.he@intel.com>]

On Wed, 2010-12-22 at 00:49 +0800, He, Qing wrote:
> This patch set enables remote data access in the exec'd task process.

It's intended to be proof of concept to an alternative approach
to work around the slowness introduced by exec bitbake-runtask.
The patch is still WIP.

http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock

Thanks,
Qing

> 
> Some test data:
> I've conducted a poky-image-minimal build on two different machines
> with and without the patch, the result are as follow:
> 
> 2 core weybridge: worker count=4, make -j4
>   without patch
>     real: 161min
>     user: 237min
>     sys:  56min
>   with patch
>     real: 138min
>     user: 196min
>     sys:  45min
> 4 core nehalem: worker count=4, make -j4
>   without patch
>     real: 88min
>     user: 218min
>     sys:  37min
>   with patch
>     real: 83min
>     user: 160min
>     sys:  29min
> 
> the user+sys has an improvement of around 20%~30%, however the
> real runtime improvement is less significant. It's probably
> because of the thread workload unbalance on multicore, or in another
> word, some CPUs are not fully loaded during the build. e.g. when
> do_rootfs_rpm is run, there is probably no other tasks running.
> 
> Haven't done any test against single thread running, but it should
> have a much more considerable impact.
> 
> 
> Known problems:
> 1. fetcher is not working at the moment, maybe bitbake needs an explicit
> InheritFromOS(), but that produces some inconsistency in PATH in my previous
> run
> 2. do_rootfs_rpm has some problem in the first run
> 
> 
> Next step and possible future usage:
> 1. organize remote data to a separate class
> 2. although we have reverted to fork for task running, a remote data mechanism
> would work for a more separate build (e.g. distributed build)
> 3. the added code in bitbake-runtask is to run necessary initializations for a
> working environment, this is part of cooker. The current cooker seems too heavy
> and may be broken down into several pieces according to functionalitiies
> 
> Thanks,
> Qing
> 
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: qhe/datasock
>   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock
> 
> Thanks,
>     Qing He
> ---
> 
> 
> Qing He (4):
>   python method fix
>   sstate.bbclass: workaround
>   data_smart: remote data
>   socket based worker data
> 
>  bitbake/bin/bitbake-runtask  |   80 +++++++++++++++++++++++++++++++++--------
>  bitbake/lib/bb/data_smart.py |   25 ++++++++++++-
>  bitbake/lib/bb/methodpool.py |    6 +++
>  bitbake/lib/bb/parse/ast.py  |    2 +-
>  bitbake/lib/bb/runqueue.py   |   68 ++++++++++++++++++++++++++++++++----
>  meta/classes/sstate.bbclass  |    2 +-
>  6 files changed, 157 insertions(+), 26 deletions(-)
>