[RFC 0/4] socketed based data connection

[RFC 0/4] socketed based data connection

[Qing He]

This patch set enables remote data access in the exec'd task process.

Some test data:
I've conducted a poky-image-minimal build on two different machines
with and without the patch, the result are as follow:

2 core weybridge: worker count=4, make -j4
  without patch
    real: 161min
    user: 237min
    sys:  56min
  with patch
    real: 138min
    user: 196min
    sys:  45min
4 core nehalem: worker count=4, make -j4
  without patch
    real: 88min
    user: 218min
    sys:  37min
  with patch
    real: 83min
    user: 160min
    sys:  29min

the user+sys has an improvement of around 20%~30%, however the
real runtime improvement is less significant. It's probably
because of the thread workload unbalance on multicore, or in another
word, some CPUs are not fully loaded during the build. e.g. when
do_rootfs_rpm is run, there is probably no other tasks running.

Haven't done any test against single thread running, but it should
have a much more considerable impact.


Known problems:
1. fetcher is not working at the moment, maybe bitbake needs an explicit
InheritFromOS(), but that produces some inconsistency in PATH in my previous
run
2. do_rootfs_rpm has some problem in the first run


Next step and possible future usage:
1. organize remote data to a separate class
2. although we have reverted to fork for task running, a remote data mechanism
would work for a more separate build (e.g. distributed build)
3. the added code in bitbake-runtask is to run necessary initializations for a
working environment, this is part of cooker. The current cooker seems too heavy
and may be broken down into several pieces according to functionalitiies

Thanks,
Qing


Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: qhe/datasock
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock

Thanks,
    Qing He
---


Qing He (4):
  python method fix
  sstate.bbclass: workaround
  data_smart: remote data
  socket based worker data

 bitbake/bin/bitbake-runtask  |   80 +++++++++++++++++++++++++++++++++--------
 bitbake/lib/bb/data_smart.py |   25 ++++++++++++-
 bitbake/lib/bb/methodpool.py |    6 +++
 bitbake/lib/bb/parse/ast.py  |    2 +-
 bitbake/lib/bb/runqueue.py   |   68 ++++++++++++++++++++++++++++++++----
 meta/classes/sstate.bbclass  |    2 +-
 6 files changed, 157 insertions(+), 26 deletions(-)

[PATCH 1/4] python method fix

[Qing He]

1. substitute '_' to '-' to avoid misrepresentation by smart data
2. export _parsed_fns

Signed-off-by: Qing He <qing.he@intel.com>
---
 bitbake/lib/bb/methodpool.py |    6 ++++++
 bitbake/lib/bb/parse/ast.py  |    2 +-
 2 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/bitbake/lib/bb/methodpool.py b/bitbake/lib/bb/methodpool.py
index 1485b13..299c54d 100644
--- a/bitbake/lib/bb/methodpool.py
+++ b/bitbake/lib/bb/methodpool.py
@@ -82,3 +82,9 @@ def get_parsed_dict():
     shortcut
     """
     return _parsed_methods
+
+def get_parsed_fns():
+    """
+    used by environment restruct
+    """
+    return _parsed_fns
diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py
index 870ae65..dc56f21 100644
--- a/bitbake/lib/bb/parse/ast.py
+++ b/bitbake/lib/bb/parse/ast.py
@@ -138,7 +138,7 @@ class MethodNode:
 
 class PythonMethodNode(AstNode):
     def __init__(self, funcname, root, body, fn):
-        self.func_name = funcname
+        self.func_name = re.sub('_', '-', funcname)
         self.root = root
         self.body = body
         self.fn = fn
-- 
1.7.0

[PATCH 2/4] sstate.bbclass: workaround

[Qing He]

Avoid calculating task dependencies in task
This inhibit the generation of .siginfo

Signed-off-by: Qing He <qing.he@intel.com>
---
 meta/classes/sstate.bbclass |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 3696a8c..cc5e672 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -287,7 +287,7 @@ def sstate_package(ss, d):
     bb.data.setVar('SSTATE_PKG', sstatepkg, d)
     bb.build.exec_func('sstate_create_package', d)
     
-    bb.siggen.dump_this_task(sstatepkg + ".siginfo", d)
+    #bb.siggen.dump_this_task(sstatepkg + ".siginfo", d)
 
     return
 
-- 
1.7.0

[PATCH 3/4] data_smart: remote data

[Qing He]

ad hoc version of remote data for data_smart
the data.dict["_remote_data"]["content"] is supposed to be the socket
used for data connection

Signed-off-by: Qing He <qing.he@intel.com>
---
 bitbake/lib/bb/data_smart.py |   25 ++++++++++++++++++++++++-
 1 files changed, 24 insertions(+), 1 deletions(-)

diff --git a/bitbake/lib/bb/data_smart.py b/bitbake/lib/bb/data_smart.py
index c8cd8f8..9b630eb 100644
--- a/bitbake/lib/bb/data_smart.py
+++ b/bitbake/lib/bb/data_smart.py
@@ -32,6 +32,13 @@ import copy, re, sys
 import bb
 from bb   import utils
 from bb.COW  import COWDictBase
+import socket
+
+try:
+    import cPickle as pickle
+except ImportError:
+    import pickle
+    bb.msg.note(1, bb.msg.domain.Cache, "Importing cPickle failed. Falling back to a very slow implementation.")
 
 
 __setvar_keyword__ = ["_append", "_prepend"]
@@ -189,6 +196,12 @@ class DataSmart:
             if var in dest:
                 return dest[var]
 
+            if "_remote_data" in dest:
+                s = dest["_remote_data"]["content"]
+                s.send(var)
+                l = int(s.recv(4), 16)
+                return pickle.loads(s.recv(l, socket.MSG_WAITALL))
+
             if "_data" not in dest:
                 break
             dest = dest["_data"]
@@ -351,8 +364,18 @@ class DataSmart:
                 _keys(d["_data"], mykey)
 
             for key in d.keys():
-                if key != "_data":
+                if key != "_data" and key != "_remote_data":
                     mykey[key] = None
+
+            if "_remote_data" in d:
+                s = d["_remote_data"]["content"]
+                s.send("_cmd_keys")
+                l = int(s.recv(4), 16)
+                rk = pickle.loads(s.recv(l, socket.MSG_WAITALL))
+                for key in rk:
+                    if key != "_data":
+                        mykey[key] = None
+
         keytab = {}
         _keys(self.dict, keytab)
         return keytab.keys()
-- 
1.7.0

[PATCH 4/4] socket based worker data

[Qing He]

This patch tries to enable exec'd process to access bitbake data from
a UNIX socket.

It's a PoC version.

Signed-off-by: Qing He <qing.he@intel.com>
---
 bitbake/bin/bitbake-runtask |   80 ++++++++++++++++++++++++++++++++++--------
 bitbake/lib/bb/runqueue.py  |   68 ++++++++++++++++++++++++++++++++----
 2 files changed, 125 insertions(+), 23 deletions(-)

diff --git a/bitbake/bin/bitbake-runtask b/bitbake/bin/bitbake-runtask
index bee0f42..4699490 100755
--- a/bitbake/bin/bitbake-runtask
+++ b/bitbake/bin/bitbake-runtask
@@ -3,6 +3,7 @@
 import os
 import sys
 import warnings
+import re
 sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
 
 try:
@@ -71,8 +72,7 @@ bb.event.useStdout = False
 hashfile = sys.argv[1]
 buildfile = sys.argv[2]
 taskname = sys.argv[3]
-
-import bb.cooker
+socketname = sys.argv[5] or None
 
 p = pickle.Unpickler(file(hashfile, "rb"))
 hashdata = p.load()
@@ -83,25 +83,68 @@ verbose = hashdata["verbose"]
 
 bb.utils.init_logger(bb.msg, verbose, debug, debug_domains)
 
-cooker = bb.cooker.BBCooker(BBConfiguration(debug, debug_domains), None)
-cooker.parseConfiguration()
+if socketname:
+    import bb, bb.data, bb.parse, socket
+    import time
+
+    fn = buildfile
+
+    configuration = BBConfiguration(debug, debug_domains)
+
+    try:
+        the_data = bb.data.init()
+        datasock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+        datasock.connect(socketname)
+        bb.data.setVar("_remote_data", datasock, the_data)
+    except:
+        socketname = ""
+
+if socketname:
+    bb.codeparser.parser_cache_init(the_data)
+    bb.parse.init_parser(the_data, configuration.dump_signatures)
+
+    datasock.send("_cmd_parsed_fns")
+    l = int(datasock.recv(4), 16)
+    parsed = pickle.loads(datasock.recv(l))
+    for key in parsed.keys():
+        funcname = re.sub('_', '-', key)
+        f = bb.data.getVar(funcname, the_data, False)
+        if f:
+            bb.methodpool.check_insert_method(key, f, fn)
+
+    for var in bb.data.getVar('__BBHANDLERS', the_data) or []:
+        bb.event.register(var, bb.data.getVar(var, the_data))
 
-cooker.bb_cache = bb.cache.init(cooker)
-cooker.status = bb.cache.CacheData()
+    if not bb.data.getVar("BUILDNAME", the_data):
+        bb.data.setVar("BUILDNAME", time.strftime('%Y%m%d%H%M'), the_data)
+    bb.data.setVar("BUILDSTART", time.strftime('%m/%d/%Y %H:%M:%S', time.gmtime()), the_data)
 
-(fn, cls) = cooker.bb_cache.virtualfn2realfn(buildfile)
-buildfile = cooker.matchFile(fn)
-fn = cooker.bb_cache.realfn2virtual(buildfile, cls)
 
-cooker.buildSetVars()
+    bb.data.setVar("BB_WORKERCONTEXT", "1", the_data)
+    bb.event.fire(bb.event.ConfigParsed(), the_data)
 
-# Load data into the cache for fn and parse the loaded cache data
-the_data = cooker.bb_cache.loadDataFull(fn, cooker.get_file_appends(fn), cooker.configuration.data)
-cooker.bb_cache.setData(fn, buildfile, the_data)
-cooker.bb_cache.handle_data(fn, cooker.status)
+else:
+    import bb.cooker
 
-#exportlist = bb.utils.preserved_envvars_export_list()
-#bb.utils.filter_environment(exportlist)
+    cooker = bb.cooker.BBCooker(BBConfiguration(debug, debug_domains), None)
+    cooker.parseConfiguration()
+
+    cooker.bb_cache = bb.cache.init(cooker)
+    cooker.status = bb.cache.CacheData()
+
+    (fn, cls) = cooker.bb_cache.virtualfn2realfn(buildfile)
+    buildfile = cooker.matchFile(fn)
+    fn = cooker.bb_cache.realfn2virtual(buildfile, cls)
+
+    cooker.buildSetVars()
+
+    # Load data into the cache for fn and parse the loaded cache data
+    the_data = cooker.bb_cache.loadDataFull(fn, cooker.get_file_appends(fn), cooker.configuration.data)
+    cooker.bb_cache.setData(fn, buildfile, the_data)
+    cooker.bb_cache.handle_data(fn, cooker.status)
+
+    #exportlist = bb.utils.preserved_envvars_export_list()
+    #bb.utils.filter_environment(exportlist)
 
 if taskname.endswith("_setscene"):
     the_data.setVarFlag(taskname, "quieterrors", "1")
@@ -116,5 +159,10 @@ for h in hashdata["deps"]:
 ret = 0
 if sys.argv[4] != "True":
     ret = bb.build.exec_task(fn, taskname, the_data)
+
+if socketname:
+    datasock.send("_cmd_end")
+    datasock.close()
+
 sys.exit(ret)
 
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index e26aa4e..d520d1f 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -24,7 +24,7 @@ Handles preparation and execution of a queue of tasks
 
 import bb, os, sys
 import subprocess
-from bb import msg, data, event
+from bb import msg, data, event, parse
 import signal
 import stat
 import fcntl
@@ -1007,6 +1007,8 @@ class RunQueueExecute:
         self.build_pids = {}
         self.build_pipes = {}
         self.build_procs = {}
+        self.build_sockpids = {}
+        self.build_sockfiles = {}
         self.failed_fnids = []
 
     def runqueue_process_waitpid(self):
@@ -1019,10 +1021,16 @@ class RunQueueExecute:
             proc.poll()
             if proc.returncode is not None:
                 task = self.build_pids[pid]
+                sockfile = self.build_sockfiles[pid]
+                sockpid = self.build_sockpids[pid]
+                os.waitpid(sockpid, 0)
+                os.unlink(sockfile)
                 del self.build_pids[pid]
                 self.build_pipes[pid].close()
                 del self.build_pipes[pid]
                 del self.build_procs[pid]
+                del self.build_sockpids[pid]
+                del self.build_sockfiles[pid]
                 if proc.returncode != 0:
                     self.task_fail(task, proc.returncode)
                 else:
@@ -1058,9 +1066,51 @@ class RunQueueExecute:
         return
 
     def fork_off_task(self, fn, task, taskname):
-        try:
-            the_data = self.cooker.bb_cache.loadDataFull(fn, self.cooker.get_file_appends(fn), self.cooker.configuration.data)
+        the_data = self.cooker.bb_cache.loadDataFull(fn, self.cooker.get_file_appends(fn), self.cooker.configuration.data)
+
+        sockfile = os.tempnam("/tmp", "sock-")
+        sockpid = os.fork()
+        if sockpid == 0:
+            import socket
+
+            try:
+                #si = file("/dev/null", 'r')
+                #os.dup2(si.fileno( ), sys.stdin.fileno( ))
+                #so = file("/dev/null", 'w')
+                #os.dup2(so.fileno( ), sys.stdout.fileno( ))
+                #os.dup2(so.fileno( ), sys.stderr.fileno( ))
+
+                os.setpgrp()
+                signal.signal(signal.SIGPIPE, signal.SIG_DFL)
+                signal.signal(signal.SIGINT, signal.SIG_DFL)
+
+                bb.event.worker_pid = 0
+
+                s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+                s.bind(sockfile)
+                s.listen(1)
+                conn, _ = s.accept()
+
+                while 1:
+                    k = conn.recv(1024)
+                    if k == "_cmd_end":
+                        conn.flush()
+                        conn.close()
+                        os._exit(os.EX_OK)
+                    elif k == "_cmd_parsed_fns":
+                        dump = pickle.dumps(bb.methodpool.get_parsed_fns())
+                    elif k == "_cmd_keys":
+                        dump = pickle.dumps(the_data.keys())
+                    else:
+                        dump = pickle.dumps(the_data._findVar(k))
+
+                    conn.send("%04x" % len(dump))
+                    conn.sendall(dump)
+            except:
+                pass
+            os._exit(os.EX_OK)
 
+        try:
             env = bb.data.export_vars(the_data)
             env = bb.data.export_envvars(env, the_data)
 
@@ -1082,14 +1132,14 @@ class RunQueueExecute:
             sys.stderr.flush()
 
             runtask = the_data.getVar("BB_RUNTASK", True) or "bitbake-runtask"
-            proc = subprocess.Popen([runtask, self.rqdata.hashfile, fn, taskname, str(self.cooker.configuration.dry_run)], env=env, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
+            proc = subprocess.Popen([runtask, self.rqdata.hashfile, fn, taskname, str(self.cooker.configuration.dry_run), sockfile], env=env, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
             pipein = proc.stdout
             pipeout = proc.stdin
             pid = proc.pid
         except OSError as e: 
             bb.msg.fatal(bb.msg.domain.RunQueue, "fork failed: %d (%s)" % (e.errno, e.strerror))
 
-        return proc
+        return proc, sockpid, sockfile
 
 class RunQueueExecuteDummy(RunQueueExecute):
     def __init__(self, rq):
@@ -1234,11 +1284,13 @@ class RunQueueExecuteTasks(RunQueueExecute):
                                                                 task,
                                                                 self.rqdata.get_user_idstring(task)))
 
-            proc = self.fork_off_task(fn, task, taskname)
+            proc, sockpid, sockfile = self.fork_off_task(fn, task, taskname)
 
             self.build_pids[proc.pid] = task
             self.build_procs[proc.pid] = proc
             self.build_pipes[proc.pid] = runQueuePipe(proc.stdout, proc.stdin, self.cfgData)
+            self.build_sockpids[proc.pid] = sockpid
+            self.build_sockfiles[proc.pid] = sockfile
             self.runq_running[task] = 1
             self.stats.taskActive()
             if self.stats.active < self.number_tasks:
@@ -1476,11 +1528,13 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
                         "Running setscene task %d of %d (%s:%s)" % (self.stats.completed + self.stats.active + self.stats.failed + 1,
                                                                          self.stats.total, fn, taskname))
 
-            proc = self.fork_off_task(fn, realtask, taskname)
+            proc, sockpid, sockfile = self.fork_off_task(fn, realtask, taskname)
 
             self.build_pids[proc.pid] = task
             self.build_procs[proc.pid] = proc
             self.build_pipes[proc.pid] = runQueuePipe(proc.stdout, proc.stdin, self.cfgData)
+            self.build_sockpids[proc.pid] = sockpid
+            self.build_sockfiles[proc.pid] = sockfile
             self.runq_running[task] = 1
             self.stats.taskActive()
             if self.stats.active < self.number_tasks:
-- 
1.7.0

Re: [RFC 0/4] socketed based data connection

[Qing He]

On Wed, 2010-12-22 at 00:49 +0800, He, Qing wrote:
> This patch set enables remote data access in the exec'd task process.

It's intended to be proof of concept to an alternative approach
to work around the slowness introduced by exec bitbake-runtask.
The patch is still WIP.

http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock

Thanks,
Qing

> 
> Some test data:
> I've conducted a poky-image-minimal build on two different machines
> with and without the patch, the result are as follow:
> 
> 2 core weybridge: worker count=4, make -j4
>   without patch
>     real: 161min
>     user: 237min
>     sys:  56min
>   with patch
>     real: 138min
>     user: 196min
>     sys:  45min
> 4 core nehalem: worker count=4, make -j4
>   without patch
>     real: 88min
>     user: 218min
>     sys:  37min
>   with patch
>     real: 83min
>     user: 160min
>     sys:  29min
> 
> the user+sys has an improvement of around 20%~30%, however the
> real runtime improvement is less significant. It's probably
> because of the thread workload unbalance on multicore, or in another
> word, some CPUs are not fully loaded during the build. e.g. when
> do_rootfs_rpm is run, there is probably no other tasks running.
> 
> Haven't done any test against single thread running, but it should
> have a much more considerable impact.
> 
> 
> Known problems:
> 1. fetcher is not working at the moment, maybe bitbake needs an explicit
> InheritFromOS(), but that produces some inconsistency in PATH in my previous
> run
> 2. do_rootfs_rpm has some problem in the first run
> 
> 
> Next step and possible future usage:
> 1. organize remote data to a separate class
> 2. although we have reverted to fork for task running, a remote data mechanism
> would work for a more separate build (e.g. distributed build)
> 3. the added code in bitbake-runtask is to run necessary initializations for a
> working environment, this is part of cooker. The current cooker seems too heavy
> and may be broken down into several pieces according to functionalitiies
> 
> Thanks,
> Qing
> 
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: qhe/datasock
>   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock
> 
> Thanks,
>     Qing He
> ---
> 
> 
> Qing He (4):
>   python method fix
>   sstate.bbclass: workaround
>   data_smart: remote data
>   socket based worker data
> 
>  bitbake/bin/bitbake-runtask  |   80 +++++++++++++++++++++++++++++++++--------
>  bitbake/lib/bb/data_smart.py |   25 ++++++++++++-
>  bitbake/lib/bb/methodpool.py |    6 +++
>  bitbake/lib/bb/parse/ast.py  |    2 +-
>  bitbake/lib/bb/runqueue.py   |   68 ++++++++++++++++++++++++++++++++----
>  meta/classes/sstate.bbclass  |    2 +-
>  6 files changed, 157 insertions(+), 26 deletions(-)
> 

Re: [RFC 0/4] socketed based data connection

[Tian, Kevin]

>From: Qing He
>Sent: Wednesday, December 22, 2010 1:25 AM
>
>On Wed, 2010-12-22 at 00:49 +0800, He, Qing wrote:
>> This patch set enables remote data access in the exec'd task process.
>
>It's intended to be proof of concept to an alternative approach
>to work around the slowness introduced by exec bitbake-runtask.
>The patch is still WIP.
>
>http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock

looks the change is not intrusive as I originally thought. :-)

>
>Thanks,
>Qing
>
>>
>> Some test data:
>> I've conducted a poky-image-minimal build on two different machines
>> with and without the patch, the result are as follow:
>>
>> 2 core weybridge: worker count=4, make -j4
>>   without patch
>>     real: 161min
>>     user: 237min
>>     sys:  56min
>>   with patch
>>     real: 138min
>>     user: 196min
>>     sys:  45min
>> 4 core nehalem: worker count=4, make -j4
>>   without patch
>>     real: 88min
>>     user: 218min
>>     sys:  37min
>>   with patch
>>     real: 83min
>>     user: 160min
>>     sys:  29min
>>
>> the user+sys has an improvement of around 20%~30%, however the
>> real runtime improvement is less significant. It's probably
>> because of the thread workload unbalance on multicore, or in another
>> word, some CPUs are not fully loaded during the build. e.g. when
>> do_rootfs_rpm is run, there is probably no other tasks running.
>>
>> Haven't done any test against single thread running, but it should
>> have a much more considerable impact.
>>
>>
>> Known problems:
>> 1. fetcher is not working at the moment, maybe bitbake needs an explicit
>> InheritFromOS(), but that produces some inconsistency in PATH in my previous
>> run
>> 2. do_rootfs_rpm has some problem in the first run
>>
>>
>> Next step and possible future usage:
>> 1. organize remote data to a separate class
>> 2. although we have reverted to fork for task running, a remote data mechanism
>> would work for a more separate build (e.g. distributed build)
>> 3. the added code in bitbake-runtask is to run necessary initializations for a
>> working environment, this is part of cooker. The current cooker seems too heavy
>> and may be broken down into several pieces according to functionalitiies
>>
>> Thanks,
>> Qing
>>
>>
>> Pull URL: git://git.pokylinux.org/poky-contrib.git
>>   Branch: qhe/datasock
>>   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock
>>
>> Thanks,
>>     Qing He
>> ---
>>
>>
>> Qing He (4):
>>   python method fix
>>   sstate.bbclass: workaround
>>   data_smart: remote data
>>   socket based worker data
>>
>>  bitbake/bin/bitbake-runtask  |   80
>+++++++++++++++++++++++++++++++++--------
>>  bitbake/lib/bb/data_smart.py |   25 ++++++++++++-
>>  bitbake/lib/bb/methodpool.py |    6 +++
>>  bitbake/lib/bb/parse/ast.py  |    2 +-
>>  bitbake/lib/bb/runqueue.py   |   68
>++++++++++++++++++++++++++++++++----
>>  meta/classes/sstate.bbclass  |    2 +-
>>  6 files changed, 157 insertions(+), 26 deletions(-)
>>
>_______________________________________________
>poky mailing list
>poky@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/poky

Re: [RFC 0/4] socketed based data connection

[Richard Purdie]

On Wed, 2010-12-22 at 00:49 +0800, Qing He wrote:
> This patch set enables remote data access in the exec'd task process.
> 
> Some test data:
> I've conducted a poky-image-minimal build on two different machines
> with and without the patch, the result are as follow:
> 
> 2 core weybridge: worker count=4, make -j4
>   without patch
>     real: 161min
>     user: 237min
>     sys:  56min
>   with patch
>     real: 138min
>     user: 196min
>     sys:  45min
> 4 core nehalem: worker count=4, make -j4
>   without patch
>     real: 88min
>     user: 218min
>     sys:  37min
>   with patch
>     real: 83min
>     user: 160min
>     sys:  29min

For comparison what numbers does master give? I'm assuming these numbers
were with master using exec() so I'd be interested in the comparisons
with fork().

> the user+sys has an improvement of around 20%~30%, however the
> real runtime improvement is less significant. It's probably
> because of the thread workload unbalance on multicore, or in another
> word, some CPUs are not fully loaded during the build. e.g. when
> do_rootfs_rpm is run, there is probably no other tasks running.
> 
> Haven't done any test against single thread running, but it should
> have a much more considerable impact.
> 
> 
> Known problems:
> 1. fetcher is not working at the moment, maybe bitbake needs an explicit
> InheritFromOS(), but that produces some inconsistency in PATH in my previous
> run
> 2. do_rootfs_rpm has some problem in the first run
> 
> 
> Next step and possible future usage:
> 1. organize remote data to a separate class
> 2. although we have reverted to fork for task running, a remote data mechanism
> would work for a more separate build (e.g. distributed build)

It could also be useful for UI interaction with the datastore.

> 3. the added code in bitbake-runtask is to run necessary initializations for a
> working environment, this is part of cooker. The current cooker seems too heavy
> and may be broken down into several pieces according to functionalitiies

I certainly agree cooker could use some serious refactoring in some
form.

Cheers,

Richard