[PATCH 9/9] runqueue: fire sceneQueueTaskStarted event when a setscene queue starts

[PATCH 9/9] runqueue: fire sceneQueueTaskStarted event when a setscene queue starts

[Dongxiao Xu]

The current code prints a log when a setscene task starts, therefore
the progressbar in hob will not receive it. Use a sceneQueueTaskStarted
event instead.

Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 lib/bb/runqueue.py |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 0e83d05..0ba414f 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1619,6 +1619,9 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
                 self.task_skip(task)
                 return True
 
+            startevent = sceneQueueTaskStarted(task, self.stats, self.rq)
+            bb.event.fire(startevent, self.cfgData)
+
             pid, pipein, pipeout = self.fork_off_task(fn, realtask, taskname)
 
             self.build_pids[pid] = task
@@ -1686,6 +1689,13 @@ class runQueueTaskStarted(runQueueEvent):
         runQueueEvent.__init__(self, task, stats, rq)
         self.noexec = noexec
 
+class sceneQueueTaskStarted(runQueueTaskStarted):
+    """
+    Event notifing a setscene task was started
+    """
+    def __init__(self, task, stats, rq, noexec=False):
+        runQueueTaskStarted.__init__(self, task, stats, rq, noexec)
+
 class runQueueTaskFailed(runQueueEvent):
     """
     Event notifing a task failed
-- 
1.7.0.4

[PATCH 0/9][PULL] Hob2: bitbake related changes

[Dongxiao Xu]

Hi Richard,

This is the hob2 related bitbake changes. Please help to review and pull.

Compared with previous version:
 - cache.py: remove the DATE field and value in calculating self.configuration's hash value,
   which causes the cache invalid after one day.

Thanks,
Dongxiao

The following changes since commit 95a599067650902727ecb4a39d6dd003c5cfedf3:

  bitbake/ui/knotty: fix incorrect exit code (2012-02-22 21:30:18 +0000)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib dxu4/hob2-bitbake-changes
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/hob2-bitbake-changes

Dongxiao Xu (7):
  cache: Use configuration's hash value to validate cache
  command.py: add resolve option for generateTargetsTree API
  event.py: Add new events RequestPackageInfo and PackageInfo
  bitbake: add -B option to bind with interface
  bitbake: Add client socket info for BitBakeServerConnection
  command.py: Add a new API triggerEvent()
  runqueue: fire sceneQueueTaskStarted event when a setscene queue
    starts

Shane Wang (2):
  bitbake: change for adding progress bar in Hob2.
  command.py: add new API to get the cpu count on the server

 bin/bitbake             |   17 +++++++-
 lib/bb/cache.py         |   34 ++++++----------
 lib/bb/command.py       |   23 ++++++++++-
 lib/bb/cooker.py        |   74 ++++++++++++++++++++++++++++--------
 lib/bb/data_smart.py    |   21 ++++++++++
 lib/bb/event.py         |   96 +++++++++++++++++++++++++++++++++++-----------
 lib/bb/runqueue.py      |   10 +++++
 lib/bb/server/xmlrpc.py |   10 ++--
 lib/bb/ui/knotty.py     |    5 ++-
 lib/bb/ui/uievent.py    |    7 ++-
 lib/bb/utils.py         |    4 ++
 11 files changed, 227 insertions(+), 74 deletions(-)

-- 
1.7.4.1

[PATCH 1/9] cache: Use configuration's hash value to validate cache

[Dongxiao Xu]

Previously we use the file time stamp to judge if a cache is valid.
Here this commit introduce a new method, which calculates the total
hash value for a certain configuration's key/value paris, and tag
it into cache filename, for example, bb_cache.dat.xxxyyyzzz.

This mechanism also ensures the cache's correctness if user
dynamically setting variables from some frontend GUI, like HOB.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 lib/bb/cache.py      |   32 ++++++++++++--------------------
 lib/bb/cooker.py     |    4 +++-
 lib/bb/data_smart.py |   21 +++++++++++++++++++++
 3 files changed, 36 insertions(+), 21 deletions(-)

diff --git a/lib/bb/cache.py b/lib/bb/cache.py
index 99e0f34..3d89435 100644
--- a/lib/bb/cache.py
+++ b/lib/bb/cache.py
@@ -42,10 +42,10 @@ except ImportError:
     logger.info("Importing cPickle failed. "
                 "Falling back to a very slow implementation.")
 
-__cache_version__ = "142"
+__cache_version__ = "143"
 
-def getCacheFile(path, filename):
-    return os.path.join(path, filename)
+def getCacheFile(path, filename, data_hash):
+    return os.path.join(path, filename + "." + data_hash)
 
 # RecipeInfoCommon defines common data retrieving methods
 # from meta data for caches. CoreRecipeInfo as well as other
@@ -245,7 +245,7 @@ class Cache(object):
     BitBake Cache implementation
     """
 
-    def __init__(self, data, caches_array):
+    def __init__(self, data, data_hash, caches_array):
         # Pass caches_array information into Cache Constructor
         # It will be used in later for deciding whether we 
         # need extra cache file dump/load support 
@@ -257,6 +257,7 @@ class Cache(object):
         self.data = None
         self.data_fn = None
         self.cacheclean = True
+        self.data_hash = data_hash
 
         if self.cachedir in [None, '']:
             self.has_cache = False
@@ -265,26 +266,17 @@ class Cache(object):
             return
 
         self.has_cache = True
-        self.cachefile = getCacheFile(self.cachedir, "bb_cache.dat")
+        self.cachefile = getCacheFile(self.cachedir, "bb_cache.dat", self.data_hash)
 
         logger.debug(1, "Using cache in '%s'", self.cachedir)
         bb.utils.mkdirhier(self.cachedir)
 
-        # If any of configuration.data's dependencies are newer than the
-        # cache there isn't even any point in loading it...
-        newest_mtime = 0
-        deps = data.getVar("__base_depends")
-
-        old_mtimes = [old_mtime for _, old_mtime in deps]
-        old_mtimes.append(newest_mtime)
-        newest_mtime = max(old_mtimes)
-
         cache_ok = True
         if self.caches_array:
             for cache_class in self.caches_array:
                 if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
-                    cachefile = getCacheFile(self.cachedir, cache_class.cachefile)
-                    cache_ok = cache_ok and (bb.parse.cached_mtime_noerror(cachefile) >= newest_mtime)
+                    cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
+                    cache_ok = cache_ok and os.path.exists(cachefile)
                     cache_class.init_cacheData(self)
         if cache_ok:
             self.load_cachefile()
@@ -318,7 +310,7 @@ class Cache(object):
         # Calculate the correct cachesize of all those cache files
         for cache_class in self.caches_array:
             if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
-                cachefile = getCacheFile(self.cachedir, cache_class.cachefile)
+                cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
                 with open(cachefile, "rb") as cachefile:
                     cachesize += os.fstat(cachefile.fileno()).st_size
 
@@ -326,7 +318,7 @@ class Cache(object):
         
         for cache_class in self.caches_array:
             if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
-                cachefile = getCacheFile(self.cachedir, cache_class.cachefile)
+                cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
                 with open(cachefile, "rb") as cachefile:
                     pickled = pickle.Unpickler(cachefile)                    
                     while cachefile:
@@ -579,7 +571,7 @@ class Cache(object):
         for cache_class in self.caches_array:
             if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
                 cache_class_name = cache_class.__name__
-                cachefile = getCacheFile(self.cachedir, cache_class.cachefile)
+                cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
                 file_dict[cache_class_name] = open(cachefile, "wb")
                 pickler_dict[cache_class_name] =  pickle.Pickler(file_dict[cache_class_name], pickle.HIGHEST_PROTOCOL)
                    
@@ -684,7 +676,7 @@ def init(cooker):
     Files causing parsing errors are evicted from the cache.
 
     """
-    return Cache(cooker.configuration.data)
+    return Cache(cooker.configuration.data, cooker.configuration.data_hash)
 
 
 class CacheData(object):
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index f0778e5..af91178 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -849,6 +849,7 @@ class BBCooker:
         bb.event.fire(bb.event.ConfigParsed(), data)
         bb.parse.init_parser(data)
         self.configuration.data = data
+        self.configuration.data_hash = data.get_hash()
 
     def handleCollections( self, collections ):
         """Handle collections"""
@@ -1494,6 +1495,7 @@ class CookerParser(object):
         self.filelist = filelist
         self.cooker = cooker
         self.cfgdata = cooker.configuration.data
+        self.cfghash = cooker.configuration.data_hash
 
         # Accounting statistics
         self.parsed = 0
@@ -1509,7 +1511,7 @@ class CookerParser(object):
         self.num_processes = int(self.cfgdata.getVar("BB_NUMBER_PARSE_THREADS", True) or
                                  multiprocessing.cpu_count())
 
-        self.bb_cache = bb.cache.Cache(self.cfgdata, cooker.caches_array)
+        self.bb_cache = bb.cache.Cache(self.cfgdata, self.cfghash, cooker.caches_array)
         self.fromcache = []
         self.willparse = []
         for filename in self.filelist:
diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index ea13478..24c7a8f 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -31,6 +31,7 @@ BitBake build tools.
 import copy, re
 from collections import MutableMapping
 import logging
+import hashlib
 import bb, bb.codeparser
 from bb   import utils
 from bb.COW  import COWDictBase
@@ -459,3 +460,23 @@ class DataSmart(MutableMapping):
 
     def __delitem__(self, var):
         self.delVar(var)
+
+    def get_hash(self):
+        data = ""
+        keys = iter(self)
+        for key in keys:
+            if key in ["TIME", "DATE"]:
+                continue
+            if key == "__depends":
+                deps = list(self.getVar(key, False))
+                deps.sort()
+                value = [deps[i][0] for i in range(len(deps))]
+            elif key == "PATH":
+                path = list(set(self.getVar(key, False).split(':')))
+                path.sort()
+                value = " ".join(path)
+            else:
+                value = self.getVar(key, False) or ""
+            data = data + key + ': ' + str(value) + '\n'
+
+        return hashlib.md5(data).hexdigest()
-- 
1.7.4.1

[PATCH 2/9] command.py: add resolve option for generateTargetsTree API

[Dongxiao Xu]

Currently we have generateTargetsTree API, which is used to get
dependency information. However in that tree, there will be
"virtual/xxx" in depends fields. Therefore we add the resolve option
to replace it with its real providers.

Besides, for packages that provided by multiple recipes, we will find
their preverred provider.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 lib/bb/command.py |   11 ++++++++-
 lib/bb/cooker.py  |   58 ++++++++++++++++++++++++++++++++++++++++++----------
 2 files changed, 56 insertions(+), 13 deletions(-)

diff --git a/lib/bb/command.py b/lib/bb/command.py
index 2a3a3af..05555c5 100644
--- a/lib/bb/command.py
+++ b/lib/bb/command.py
@@ -230,14 +230,21 @@ class CommandsAsync:
         included in the package list.
         If pkg_list provided use that list (plus any extras brought in by
         klass) rather than generating a tree for all packages.
+
+        Add a new option "resolve" to indicate if we need to resolve the
+        replacement for "virtual/xxx" like pn.
         """
         klass = params[0]
-        if len(params) > 1:
+        resolve = False
+        if len(params) > 2:
+            pkg_list = params[1]
+            resolve = params[2]
+        elif len(params) > 1:
             pkg_list = params[1]
         else:
             pkg_list = []
 
-        command.cooker.generateTargetsTree(klass, pkg_list)
+        command.cooker.generateTargetsTree(klass, pkg_list, resolve)
         command.finishAsyncCommand()
     generateTargetsTree.needcache = True
 
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index af91178..85e391b 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -429,7 +429,20 @@ class BBCooker:
 
         return depend_tree
 
-    def generatePkgDepTreeData(self, pkgs_to_build, task):
+    def append_package(self, taskdata, depend_tree_package, package):
+        if package not in depend_tree_package:
+            targetid = taskdata.getrun_id(package)
+            if targetid in taskdata.run_targets and taskdata.run_targets[targetid]:
+                fnid = taskdata.run_targets[targetid][0]
+                fn = taskdata.fn_index[fnid]
+                pn = self.status.pkg_fn[fn]
+                version  = "%s:%s-%s" % self.status.pkg_pepvpr[fn]
+                depend_tree_package[package] = {}
+                depend_tree_package[package]["pn"] = pn
+                depend_tree_package[package]["filename"] = fn
+                depend_tree_package[package]["version"] = version
+
+    def generatePkgDepTreeData(self, pkgs_to_build, task, resolve=False):
         """
         Create a dependency tree of pkgs_to_build, returning the data.
         """
@@ -446,6 +459,7 @@ class BBCooker:
         depend_tree["rdepends-pn"] = {}
         depend_tree["packages"] = {}
         depend_tree["rdepends-pkg"] = {}
+        depend_tree["rrecs-pkg"] = {}
 
         for task in xrange(len(tasks_fnid)):
             fnid = tasks_fnid[task]
@@ -456,6 +470,8 @@ class BBCooker:
             lic = self.status.license[fn]
             section = self.status.section[fn]
             description = self.status.description[fn]
+            rdepends = self.status.rundeps[fn]
+            rrecs = self.status.runrecs[fn]
             if pn not in depend_tree["pn"]:
                 depend_tree["pn"][pn] = {}
                 depend_tree["pn"][pn]["filename"] = fn
@@ -464,6 +480,7 @@ class BBCooker:
                 depend_tree["pn"][pn]["license"] = lic
                 depend_tree["pn"][pn]["section"] = section
                 depend_tree["pn"][pn]["description"] = description
+                depend_tree["pn"][pn]["packages"] = rdepends.keys()
 
             if fnid not in seen_fnids:
                 seen_fnids.append(fnid)
@@ -471,25 +488,44 @@ class BBCooker:
 
                 depend_tree["depends"][pn] = []
                 for dep in taskdata.depids[fnid]:
-                    depend_tree["depends"][pn].append(taskdata.build_names_index[dep])
+                    if resolve:
+                        item = taskdata.build_names_index[dep]
+                        pn_provider = ""
+                        targetid = taskdata.getbuild_id(item)
+                        if targetid in taskdata.build_targets and taskdata.build_targets[targetid]:
+                            fnid = taskdata.build_targets[targetid][0]
+                            fn_provider = taskdata.fn_index[fnid]
+                            pn_provider = self.status.pkg_fn[fn_provider]
+                        else:
+                            pn_provider = item
+                        depend_tree["depends"][pn].append(pn_provider)
+                    else:
+                        depend_tree["depends"][pn].append(taskdata.build_names_index[dep])
 
                 depend_tree["rdepends-pn"][pn] = []
                 for rdep in taskdata.rdepids[fnid]:
                     depend_tree["rdepends-pn"][pn].append(taskdata.run_names_index[rdep])
 
-                rdepends = self.status.rundeps[fn]
                 for package in rdepends:
                     depend_tree["rdepends-pkg"][package] = []
                     for rdepend in rdepends[package]:
                         depend_tree["rdepends-pkg"][package].append(rdepend)
-                    packages.append(package)
+                        if resolve:
+                            self.append_package(taskdata, depend_tree["packages"], rdepend)
+                    if not package in packages:
+                        packages.append(package)
+
+                for package in rrecs:
+                    depend_tree["rrecs-pkg"][package] = []
+                    for rrec in rrecs[package]:
+                        depend_tree["rrecs-pkg"][package].append(rrec)
+                        if resolve:
+                            self.append_package(taskdata, depend_tree["packages"], rrec)
+                    if not package in packages:
+                        packages.append(package)
 
                 for package in packages:
-                    if package not in depend_tree["packages"]:
-                        depend_tree["packages"][package] = {}
-                        depend_tree["packages"][package]["pn"] = pn
-                        depend_tree["packages"][package]["filename"] = fn
-                        depend_tree["packages"][package]["version"] = version
+                    self.append_package(taskdata, depend_tree["packages"], package)
 
         return depend_tree
 
@@ -735,7 +771,7 @@ class BBCooker:
 
         return pkg_list
 
-    def generateTargetsTree(self, klass=None, pkgs=[]):
+    def generateTargetsTree(self, klass=None, pkgs=[], resolve=False):
         """
         Generate a dependency tree of buildable targets
         Generate an event with the result
@@ -750,7 +786,7 @@ class BBCooker:
             pkgs = pkgs + extra_pkgs
 
         # generate a dependency tree for all our packages
-        tree = self.generatePkgDepTreeData(pkgs, 'build')
+        tree = self.generatePkgDepTreeData(pkgs, 'build', resolve)
         bb.event.fire(bb.event.TargetsTreeGenerated(tree), self.configuration.data)
 
     def buildWorldTargetList(self):
-- 
1.7.4.1

[PATCH 3/9] event.py: Add new events RequestPackageInfo and PackageInfo

[Dongxiao Xu]

RequestPackageInfo is triggered by GUI client to request the available
package information.

PackageInfo event is to pass package information back to GUI.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 lib/bb/event.py |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)

diff --git a/lib/bb/event.py b/lib/bb/event.py
index 8d7f941..10036c0 100644
--- a/lib/bb/event.py
+++ b/lib/bb/event.py
@@ -467,3 +467,16 @@ class LogHandler(logging.Handler):
     def filter(self, record):
         record.taskpid = worker_pid
         return True
+
+class RequestPackageInfo(Event):
+    """
+    Event to request package information
+    """
+
+class PackageInfo(Event):
+    """
+    Package information for GUI
+    """
+    def __init__(self, pkginfolist):
+        Event.__init__(self)
+        self._pkginfolist = pkginfolist
-- 
1.7.4.1

[PATCH 4/9] bitbake: change for adding progress bar in Hob2.

[Dongxiao Xu]

From: Shane Wang <shane.wang@intel.com>

The changes include:
- Clean some events in event.py
- Fire essential events for Hob2 to handle with more information.
- knotty changes

Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 lib/bb/cache.py     |    2 +-
 lib/bb/cooker.py    |   12 +++++--
 lib/bb/event.py     |   83 ++++++++++++++++++++++++++++++++++++--------------
 lib/bb/ui/knotty.py |    5 ++-
 4 files changed, 73 insertions(+), 29 deletions(-)

diff --git a/lib/bb/cache.py b/lib/bb/cache.py
index 3d89435..47e814b 100644
--- a/lib/bb/cache.py
+++ b/lib/bb/cache.py
@@ -336,7 +336,7 @@ class Cache(object):
                         current_percent = 100 * current_progress / cachesize
                         if current_percent > previous_percent:
                             previous_percent = current_percent
-                            bb.event.fire(bb.event.CacheLoadProgress(current_progress),
+                            bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize),
                                           self.data)
 
                     previous_progress += current_progress
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 85e391b..ebe95af 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -334,6 +334,7 @@ class BBCooker:
         """
         Prepare a runqueue and taskdata object for iteration over pkgs_to_build
         """
+        bb.event.fire(bb.event.TreeDataPreparationStarted(), self.configuration.data)
         # Need files parsed
         self.updateCache()
         # If we are told to do the None task then query the default task
@@ -350,11 +351,14 @@ class BBCooker:
         taskdata = bb.taskdata.TaskData(False, skiplist=self.skiplist)
 
         runlist = []
+        current = 0
         for k in pkgs_to_build:
             taskdata.add_provider(localdata, self.status, k)
             runlist.append([k, "do_%s" % task])
+            current += 1
+            bb.event.fire(bb.event.TreeDataPreparationProgress(current, len(pkgs_to_build)), self.configuration.data)
         taskdata.add_unresolved(localdata, self.status)
-
+        bb.event.fire(bb.event.TreeDataPreparationCompleted(len(pkgs_to_build)), self.configuration.data)
         return runlist, taskdata
 
     def generateTaskDepTreeData(self, pkgs_to_build, task):
@@ -1100,7 +1104,7 @@ class BBCooker:
                 return False
 
             if not retval:
-                bb.event.fire(bb.event.BuildCompleted(buildname, item, failures), self.configuration.event_data)
+                bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runq_fnid), buildname, item, failures), self.configuration.event_data)
                 self.command.finishAsyncCommand()
                 return False
             if retval is True:
@@ -1140,7 +1144,7 @@ class BBCooker:
                 return False
 
             if not retval:
-                bb.event.fire(bb.event.BuildCompleted(buildname, targets, failures), self.configuration.data)
+                bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runq_fnid), buildname, targets, failures), self.configuration.data)
                 self.command.finishAsyncCommand()
                 return False
             if retval is True:
@@ -1665,7 +1669,7 @@ class CookerParser(object):
         if parsed:
             self.parsed += 1
             if self.parsed % self.progress_chunk == 0:
-                bb.event.fire(bb.event.ParseProgress(self.parsed),
+                bb.event.fire(bb.event.ParseProgress(self.parsed, self.toparse),
                               self.cfgdata)
         else:
             self.cached += 1
diff --git a/lib/bb/event.py b/lib/bb/event.py
index 10036c0..bbece58 100644
--- a/lib/bb/event.py
+++ b/lib/bb/event.py
@@ -204,6 +204,27 @@ def getName(e):
     else:
         return e.__name__
 
+class OperationStarted(Event):
+    """An operation has begun"""
+    def __init__(self, msg = "Operation Started"):
+        Event.__init__(self)
+        self.msg = msg
+
+class OperationCompleted(Event):
+    """An operation has completed"""
+    def __init__(self, total, msg = "Operation Completed"):
+        Event.__init__(self)
+        self.total = total
+        self.msg = msg
+
+class OperationProgress(Event):
+    """An operation is in progress"""
+    def __init__(self, current, total, msg = "Operation in Progress"):
+        Event.__init__(self)
+        self.current = current
+        self.total = total
+        self.msg = msg + ": %s/%s" % (current, total);
+
 class ConfigParsed(Event):
     """Configuration Parsing Complete"""
 
@@ -276,14 +297,20 @@ class BuildBase(Event):
 
 
 
-class BuildStarted(BuildBase):
+class BuildStarted(BuildBase, OperationStarted):
     """bbmake build run started"""
+    def __init__(self, n, p, failures = 0):
+        OperationStarted.__init__(self, "Building Started")
+        BuildBase.__init__(self, n, p, failures)
 
-
-class BuildCompleted(BuildBase):
+class BuildCompleted(BuildBase, OperationCompleted):
     """bbmake build run completed"""
-
-
+    def __init__(self, total, n, p, failures = 0):
+        if not failures:
+            OperationCompleted.__init__(self, total, "Building Succeeded")
+        else:
+            OperationCompleted.__init__(self, total, "Building Failed")
+        BuildBase.__init__(self, n, p, failures)
 
 
 class NoProvider(Event):
@@ -329,17 +356,16 @@ class MultipleProviders(Event):
         """
         return self._candidates
 
-class ParseStarted(Event):
+class ParseStarted(OperationStarted):
     """Recipe parsing for the runqueue has begun"""
     def __init__(self, total):
-        Event.__init__(self)
+        OperationStarted.__init__(self, "Recipe parsing Started")
         self.total = total
 
-class ParseCompleted(Event):
+class ParseCompleted(OperationCompleted):
     """Recipe parsing for the runqueue has completed"""
-
     def __init__(self, cached, parsed, skipped, masked, virtuals, errors, total):
-        Event.__init__(self)
+        OperationCompleted.__init__(self, total, "Recipe parsing Completed")
         self.cached = cached
         self.parsed = parsed
         self.skipped = skipped
@@ -347,33 +373,44 @@ class ParseCompleted(Event):
         self.masked = masked
         self.errors = errors
         self.sofar = cached + parsed
-        self.total = total
 
-class ParseProgress(Event):
+class ParseProgress(OperationProgress):
     """Recipe parsing progress"""
+    def __init__(self, current, total):
+        OperationProgress.__init__(self, current, total, "Recipe parsing")
 
-    def __init__(self, current):
-        self.current = current
 
-class CacheLoadStarted(Event):
+class CacheLoadStarted(OperationStarted):
     """Loading of the dependency cache has begun"""
     def __init__(self, total):
-        Event.__init__(self)
+        OperationStarted.__init__(self, "Loading cache Started")
         self.total = total
 
-class CacheLoadProgress(Event):
+class CacheLoadProgress(OperationProgress):
     """Cache loading progress"""
-    def __init__(self, current):
-        Event.__init__(self)
-        self.current = current
+    def __init__(self, current, total):
+        OperationProgress.__init__(self, current, total, "Loading cache")
 
-class CacheLoadCompleted(Event):
+class CacheLoadCompleted(OperationCompleted):
     """Cache loading is complete"""
     def __init__(self, total, num_entries):
-        Event.__init__(self)
-        self.total = total
+        OperationCompleted.__init__(self, total, "Loading cache Completed")
         self.num_entries = num_entries
 
+class TreeDataPreparationStarted(OperationStarted):
+    """Tree data preparation started"""
+    def __init__(self):
+        OperationStarted.__init__(self, "Preparing tree data Started")
+
+class TreeDataPreparationProgress(OperationProgress):
+    """Tree data preparation is in progress"""
+    def __init__(self, current, total):
+        OperationProgress.__init__(self, current, total, "Preparing tree data")
+
+class TreeDataPreparationCompleted(OperationCompleted):
+    """Tree data preparation completed"""
+    def __init__(self, total):
+        OperationCompleted.__init__(self, total, "Preparing tree data Completed")
 
 class DepTreeGenerated(Event):
     """
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 158a132..e2e6ac3 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -263,7 +263,10 @@ def main(server, eventHandler):
                                   bb.event.RecipeParsed,
                                   bb.event.RecipePreFinalise,
                                   bb.runqueue.runQueueEvent,
-                                  bb.runqueue.runQueueExitWait)):
+                                  bb.runqueue.runQueueExitWait,
+                                  bb.event.OperationStarted,
+                                  bb.event.OperationCompleted,
+                                  bb.event.OperationProgress)):
                 continue
 
             logger.error("Unknown event: %s", event)
-- 
1.7.4.1

[PATCH 5/9] bitbake: add -B option to bind with interface

[Dongxiao Xu]

When start bitbake as a server only process, we need to assign certain
interface to it.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 bin/bitbake             |   17 ++++++++++++++---
 lib/bb/server/xmlrpc.py |    6 +++---
 2 files changed, 17 insertions(+), 6 deletions(-)

diff --git a/bin/bitbake b/bin/bitbake
index 6da4980..c06d4e8 100755
--- a/bin/bitbake
+++ b/bin/bitbake
@@ -168,6 +168,8 @@ Default BBFILES are the .bb files in the current directory.""")
     parser.add_option("", "--server-only", help = "Run bitbake without UI,  the frontend can connect with bitbake server itself",
                action = "store_true", dest = "server_only", default = False)
 
+    parser.add_option("-B", "--bind", help = "The name/address for the bitbake server to bind to",
+               action = "store", dest = "bind", default = False)
     options, args = parser.parse_args(sys.argv)
 
     configuration = BBConfiguration(options)
@@ -189,8 +191,14 @@ Default BBFILES are the .bb files in the current directory.""")
         sys.exit("FATAL: Invalid server type '%s' specified.\n"
                  "Valid interfaces: xmlrpc, process [default], none." % servertype)
 
-    if configuration.server_only and configuration.servertype != "xmlrpc":
-        sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n")
+    if configuration.server_only:
+        if configuration.servertype != "xmlrpc":
+            sys.exit("FATAL: If '--server-only' is defined, we must set the servertype as 'xmlrpc'.\n")
+        if not configuration.bind:
+            sys.exit("FATAL: The '--server-only' option requires a name/address to bind to with the -B option.\n")
+
+    if configuration.bind and configuration.servertype != "xmlrpc":
+        sys.exit("FATAL: If '-B' or '--bind' is defined, we must set the servertype as 'xmlrpc'.\n")
 
     # Save a logfile for cooker into the current working directory. When the
     # server is daemonized this logfile will be truncated.
@@ -212,8 +220,11 @@ Default BBFILES are the .bb files in the current directory.""")
     bb.utils.clean_environment()
 
     server = server.BitBakeServer()
+    if configuration.bind:
+        server.initServer((configuration.bind, 0))
+    else:
+        server.initServer()
 
-    server.initServer()
     idle = server.getServerIdleCB()
 
     cooker = bb.cooker.BBCooker(configuration, idle, initialenv)
diff --git a/lib/bb/server/xmlrpc.py b/lib/bb/server/xmlrpc.py
index b5980c6..c53cee4 100644
--- a/lib/bb/server/xmlrpc.py
+++ b/lib/bb/server/xmlrpc.py
@@ -163,7 +163,7 @@ class BitBakeXMLRPCServer(SimpleXMLRPCServer):
     # remove this when you're done with debugging
     # allow_reuse_address = True
 
-    def __init__(self, interface = ("localhost", 0)):
+    def __init__(self, interface):
         """
         Constructor
         """
@@ -267,8 +267,8 @@ class BitBakeServerConnection():
             pass
 
 class BitBakeServer(object):
-    def initServer(self):
-        self.server = BitBakeXMLRPCServer()
+    def initServer(self, interface = ("localhost", 0)):
+        self.server = BitBakeXMLRPCServer(interface)
 
     def addcooker(self, cooker):
         self.cooker = cooker
-- 
1.7.4.1

[PATCH 6/9] bitbake: Add client socket info for BitBakeServerConnection

[Dongxiao Xu]

In server/client split model, the client will bind to a specific address
and port. We need to pass the values to BitBakeServerConnection().

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 lib/bb/server/xmlrpc.py |    4 ++--
 lib/bb/ui/uievent.py    |    7 ++++---
 2 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/lib/bb/server/xmlrpc.py b/lib/bb/server/xmlrpc.py
index c53cee4..eff8009 100644
--- a/lib/bb/server/xmlrpc.py
+++ b/lib/bb/server/xmlrpc.py
@@ -247,9 +247,9 @@ class BitbakeServerInfo():
         self.port = port
 
 class BitBakeServerConnection():
-    def __init__(self, serverinfo):
+    def __init__(self, serverinfo, clientinfo=("localhost", 0)):
         self.connection = _create_server(serverinfo.host, serverinfo.port)
-        self.events = uievent.BBUIEventQueue(self.connection)
+        self.events = uievent.BBUIEventQueue(self.connection, clientinfo)
         for event in bb.event.ui_queue:
             self.events.queue_event(event)
 
diff --git a/lib/bb/ui/uievent.py b/lib/bb/ui/uievent.py
index 0e73817..28817a2 100644
--- a/lib/bb/ui/uievent.py
+++ b/lib/bb/ui/uievent.py
@@ -28,13 +28,14 @@ import socket, threading, pickle
 from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
 
 class BBUIEventQueue:
-    def __init__(self, BBServer):
+    def __init__(self, BBServer, clientinfo=("localhost, 0")):
 
         self.eventQueue = []
         self.eventQueueLock = threading.Lock()
         self.eventQueueNotify = threading.Event()
 
         self.BBServer = BBServer
+        self.clientinfo = clientinfo
 
         self.t = threading.Thread()
         self.t.setDaemon(True)
@@ -72,7 +73,7 @@ class BBUIEventQueue:
 
     def startCallbackHandler(self):
 
-        server = UIXMLRPCServer()
+        server = UIXMLRPCServer(self.clientinfo)
         self.host, self.port = server.socket.getsockname()
 
         server.register_function( self.system_quit, "event.quit" )
@@ -98,7 +99,7 @@ class BBUIEventQueue:
 
 class UIXMLRPCServer (SimpleXMLRPCServer):
 
-    def __init__( self, interface = ("localhost", 0) ):
+    def __init__( self, interface ):
         self.quit = False
         SimpleXMLRPCServer.__init__( self,
                                     interface,
-- 
1.7.4.1

[PATCH 7/9] command.py: add new API to get the cpu count on the server

[Dongxiao Xu]

From: Shane Wang <shane.wang@intel.com>

Add a new API in command.py to get the cpu count in order to set the appropriate default BB_NUMBER_THREADS and PARALLEL_MAKE variables.

Signed-off-by: Shane Wang <shane.wang@intel.com>
---
 lib/bb/command.py |    6 ++++++
 lib/bb/utils.py   |    4 ++++
 2 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/lib/bb/command.py b/lib/bb/command.py
index 05555c5..43875f7 100644
--- a/lib/bb/command.py
+++ b/lib/bb/command.py
@@ -173,6 +173,12 @@ class CommandsSync:
         """
         command.cooker.reset()
 
+    def getCpuCount(self, command, params):
+        """
+        Get the CPU count on the bitbake server
+        """
+        return bb.utils.cpu_count()
+
 
 class CommandsAsync:
     """
diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index a26635a..d7cefb2 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -25,6 +25,7 @@ import errno
 import logging
 import bb
 import bb.msg
+import multiprocessing
 from commands import getstatusoutput
 from contextlib import contextmanager
 
@@ -862,3 +863,6 @@ def contains(variable, checkvalues, truevalue, falsevalue, d):
     if checkvalues.issubset(val): 
         return truevalue
     return falsevalue
+
+def cpu_count():
+    return multiprocessing.cpu_count()
-- 
1.7.4.1

[PATCH 8/9] command.py: Add a new API triggerEvent()

[Dongxiao Xu]

This functions enables the client to request triggering specific event
from bitbake server.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 lib/bb/command.py |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/lib/bb/command.py b/lib/bb/command.py
index 43875f7..06e8869 100644
--- a/lib/bb/command.py
+++ b/lib/bb/command.py
@@ -179,6 +179,12 @@ class CommandsSync:
         """
         return bb.utils.cpu_count()
 
+    def triggerEvent(self, command, params):
+        """
+        Trigger a certain event
+        """
+        event = params[0]
+        bb.event.fire(eval(event), command.cooker.configuration.data)
 
 class CommandsAsync:
     """
-- 
1.7.4.1

[PATCH 9/9] runqueue: fire sceneQueueTaskStarted event when a setscene queue starts

[Dongxiao Xu]

The current code prints a log when a setscene task starts, therefore
the progressbar in hob will not receive it. Use a sceneQueueTaskStarted
event instead.

Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 lib/bb/runqueue.py |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index d1d9ad9..ef28415 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1653,6 +1653,9 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
                 self.task_skip(task)
                 return True
 
+            startevent = sceneQueueTaskStarted(task, self.stats, self.rq)
+            bb.event.fire(startevent, self.cfgData)
+
             pid, pipein, pipeout = self.fork_off_task(fn, realtask, taskname)
 
             self.build_pids[pid] = task
@@ -1720,6 +1723,13 @@ class runQueueTaskStarted(runQueueEvent):
         runQueueEvent.__init__(self, task, stats, rq)
         self.noexec = noexec
 
+class sceneQueueTaskStarted(runQueueTaskStarted):
+    """
+    Event notifing a setscene task was started
+    """
+    def __init__(self, task, stats, rq, noexec=False):
+        runQueueTaskStarted.__init__(self, task, stats, rq, noexec)
+
 class runQueueTaskFailed(runQueueEvent):
     """
     Event notifing a task failed
-- 
1.7.4.1

Re: [PATCH 9/9] runqueue: fire sceneQueueTaskStarted event when a setscene queue starts

[Richard Purdie]

On Thu, 2012-02-23 at 21:47 +0800, Dongxiao Xu wrote:
> The current code prints a log when a setscene task starts, therefore
> the progressbar in hob will not receive it. Use a sceneQueueTaskStarted
> event instead.
> 
> Signed-off-by: Shane Wang <shane.wang@intel.com>
> Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> ---
>  lib/bb/runqueue.py |   10 ++++++++++
>  1 files changed, 10 insertions(+), 0 deletions(-)
> 
> diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
> index d1d9ad9..ef28415 100644
> --- a/lib/bb/runqueue.py
> +++ b/lib/bb/runqueue.py
> @@ -1653,6 +1653,9 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
>                  self.task_skip(task)
>                  return True
>  
> +            startevent = sceneQueueTaskStarted(task, self.stats, self.rq)
> +            bb.event.fire(startevent, self.cfgData)
> +
>              pid, pipein, pipeout = self.fork_off_task(fn, realtask, taskname)
>  
>              self.build_pids[pid] = task
> @@ -1720,6 +1723,13 @@ class runQueueTaskStarted(runQueueEvent):
>          runQueueEvent.__init__(self, task, stats, rq)
>          self.noexec = noexec
>  
> +class sceneQueueTaskStarted(runQueueTaskStarted):
> +    """
> +    Event notifing a setscene task was started
> +    """
> +    def __init__(self, task, stats, rq, noexec=False):
> +        runQueueTaskStarted.__init__(self, task, stats, rq, noexec)
> +

This one has me a little worried since it inherits runQueueTaskStarted
and a UI might confuse this with real runQueueTaskStarted() events. Is
there something else we can inherit here?

Cheers,

Richard

Re: [PATCH 0/9][PULL] Hob2: bitbake related changes

[Richard Purdie]

On Thu, 2012-02-23 at 21:47 +0800, Dongxiao Xu wrote:
> Hi Richard,
> 
> This is the hob2 related bitbake changes. Please help to review and pull.
> 
> Compared with previous version:
>  - cache.py: remove the DATE field and value in calculating self.configuration's hash value,
>    which causes the cache invalid after one day.
> 
> Thanks,
> Dongxiao
> 
> The following changes since commit 95a599067650902727ecb4a39d6dd003c5cfedf3:
> 
>   bitbake/ui/knotty: fix incorrect exit code (2012-02-22 21:30:18 +0000)
> 
> are available in the git repository at:
>   git://git.pokylinux.org/poky-contrib dxu4/hob2-bitbake-changes
>   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/hob2-bitbake-changes
> 
> Dongxiao Xu (7):
>   cache: Use configuration's hash value to validate cache
>   command.py: add resolve option for generateTargetsTree API
>   event.py: Add new events RequestPackageInfo and PackageInfo
>   bitbake: add -B option to bind with interface
>   bitbake: Add client socket info for BitBakeServerConnection
>   command.py: Add a new API triggerEvent()
>   runqueue: fire sceneQueueTaskStarted event when a setscene queue
>     starts
> 
> Shane Wang (2):
>   bitbake: change for adding progress bar in Hob2.
>   command.py: add new API to get the cpu count on the server

The series had things split into good logical chunks which really helped
making review easier, thanks! Most of these changes have been on the
list before so I've merged these with the exception of the
sceneQueueTaskStarted which might need a small change.

Cheers,

Richard

Re: [PATCH 9/9] runqueue: fire sceneQueueTaskStarted event when a setscene queue starts

[Xu, Dongxiao]

On Thu, 2012-02-23 at 22:51 +0000, Richard Purdie wrote:
> On Thu, 2012-02-23 at 21:47 +0800, Dongxiao Xu wrote:
> > The current code prints a log when a setscene task starts, therefore
> > the progressbar in hob will not receive it. Use a sceneQueueTaskStarted
> > event instead.
> > 
> > Signed-off-by: Shane Wang <shane.wang@intel.com>
> > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> > ---
> >  lib/bb/runqueue.py |   10 ++++++++++
> >  1 files changed, 10 insertions(+), 0 deletions(-)
> > 
> > diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
> > index d1d9ad9..ef28415 100644
> > --- a/lib/bb/runqueue.py
> > +++ b/lib/bb/runqueue.py
> > @@ -1653,6 +1653,9 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
> >                  self.task_skip(task)
> >                  return True
> >  
> > +            startevent = sceneQueueTaskStarted(task, self.stats, self.rq)
> > +            bb.event.fire(startevent, self.cfgData)
> > +
> >              pid, pipein, pipeout = self.fork_off_task(fn, realtask, taskname)
> >  
> >              self.build_pids[pid] = task
> > @@ -1720,6 +1723,13 @@ class runQueueTaskStarted(runQueueEvent):
> >          runQueueEvent.__init__(self, task, stats, rq)
> >          self.noexec = noexec
> >  
> > +class sceneQueueTaskStarted(runQueueTaskStarted):
> > +    """
> > +    Event notifing a setscene task was started
> > +    """
> > +    def __init__(self, task, stats, rq, noexec=False):
> > +        runQueueTaskStarted.__init__(self, task, stats, rq, noexec)
> > +
> 
> This one has me a little worried since it inherits runQueueTaskStarted
> and a UI might confuse this with real runQueueTaskStarted() events. Is
> there something else we can inherit here?

Hi Richard,

Here we inherit runQueueTaskStarted event is because we saw the
sceneQueueTaskFailed event inherits runQueueTaskFailed in existing code.

What about change both of sceneQueueTaskxxx directly inherit
runQueueEvent?

Thanks,
Dongxiao

> 
> Cheers,
> 
> Richard
> 
> 

Re: [PATCH 9/9] runqueue: fire sceneQueueTaskStarted event when a setscene queue starts

[Richard Purdie]

On Fri, 2012-02-24 at 08:34 +0800, Xu, Dongxiao wrote:
> On Thu, 2012-02-23 at 22:51 +0000, Richard Purdie wrote:
> > On Thu, 2012-02-23 at 21:47 +0800, Dongxiao Xu wrote:
> > > The current code prints a log when a setscene task starts, therefore
> > > the progressbar in hob will not receive it. Use a sceneQueueTaskStarted
> > > event instead.
> > > 
> > > Signed-off-by: Shane Wang <shane.wang@intel.com>
> > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> > > ---
> > >  lib/bb/runqueue.py |   10 ++++++++++
> > >  1 files changed, 10 insertions(+), 0 deletions(-)
> > > 
> > > diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
> > > index d1d9ad9..ef28415 100644
> > > --- a/lib/bb/runqueue.py
> > > +++ b/lib/bb/runqueue.py
> > > @@ -1653,6 +1653,9 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
> > >                  self.task_skip(task)
> > >                  return True
> > >  
> > > +            startevent = sceneQueueTaskStarted(task, self.stats, self.rq)
> > > +            bb.event.fire(startevent, self.cfgData)
> > > +
> > >              pid, pipein, pipeout = self.fork_off_task(fn, realtask, taskname)
> > >  
> > >              self.build_pids[pid] = task
> > > @@ -1720,6 +1723,13 @@ class runQueueTaskStarted(runQueueEvent):
> > >          runQueueEvent.__init__(self, task, stats, rq)
> > >          self.noexec = noexec
> > >  
> > > +class sceneQueueTaskStarted(runQueueTaskStarted):
> > > +    """
> > > +    Event notifing a setscene task was started
> > > +    """
> > > +    def __init__(self, task, stats, rq, noexec=False):
> > > +        runQueueTaskStarted.__init__(self, task, stats, rq, noexec)
> > > +
> > 
> > This one has me a little worried since it inherits runQueueTaskStarted
> > and a UI might confuse this with real runQueueTaskStarted() events. Is
> > there something else we can inherit here?
> 
> Hi Richard,
> 
> Here we inherit runQueueTaskStarted event is because we saw the
> sceneQueueTaskFailed event inherits runQueueTaskFailed in existing code.
> 
> What about change both of sceneQueueTaskxxx directly inherit
> runQueueEvent?

That sounds reasonable to me. sceneQueueTaskFailed sounds like a bug and
might explain an error I keep seeing in knotty but haven't looked
into/reported yet! You might fix a bug by changing that :)

Cheers,

Richard