[PATCH 0/4] Put extra requested fields into different cache files

[PATCH 0/4] Put extra requested fields into different cache files

[Liping Ke]

From: Liping Ke <liping.ke@intel.com>

Below four patches are for putting extra requested fields inito different
cache files instead of using only on bb_cache.dat. Now image creator need
extra three fields. And in the future, there might be more similar requests.
For each extra requestor, we will save the requested fields data into a
separate cache files so that those who don't need it will not be impacted
with larger fields and large data files.

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: lke/cache_impl
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/cache_impl

Thanks,
    Liping Ke <liping.ke@intel.com>
---


Liping Ke (4):
  Introduce new param bitbake_mode into Cache.py
  Split Cache Data Retrieve method from Data Fields
  Introduce Extra required fields for image creator
  Implement independent cache for Extra Cache Fields Request

 bitbake/lib/bb/cache.py  |  189 +++++++++++++++++++++++++++++++++++++---------
 bitbake/lib/bb/cooker.py |   55 ++++++++++----
 2 files changed, 193 insertions(+), 51 deletions(-)

[PATCH 1/4] Put extra requested fields into different cache files

[Liping Ke]

From: Liping Ke <liping.ke@intel.com>

This patch introduce new param bitbake_mode into Cache.py.
When using ui mode, we will save ui_mode required extra cache
fields into a separate cache file. This patch firstly introduce
this bitbake_mode parameter. It will be used in the following
patches.

Signed-off-by: Liping Ke <liping.ke@intel.com>
---
 bitbake/lib/bb/cache.py  |   19 ++++++++++++++++---
 bitbake/lib/bb/cooker.py |   11 ++++++++---
 2 files changed, 24 insertions(+), 6 deletions(-)

diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index d083c51..c86aa14 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -187,7 +187,10 @@ class Cache(object):
     BitBake Cache implementation
     """
 
-    def __init__(self, data):
+    def __init__(self, data, bitbake_mode=None):       
+        # Pass bitbake_mode information into Cache constructor
+        # It will be used in later for deciding whether
+        # we need separate cache file dump/load support
         self.cachedir = bb.data.getVar("CACHE", data, True)
         self.clean = set()
         self.checked = set()
@@ -196,6 +199,11 @@ class Cache(object):
         self.data_fn = None
         self.cacheclean = True
 
+        # Below three variables are used for loading extra cache
+        self.extra_depends_cache = {}        
+        self.bitbake_mode = bitbake_mode
+        self.extra_cachefile = None
+
         if self.cachedir in [None, '']:
             self.has_cache = False
             logger.info("Not using a cache. "
@@ -205,6 +213,11 @@ class Cache(object):
         self.has_cache = True
         self.cachefile = os.path.join(self.cachedir, "bb_cache.dat")
 
+        # if bitbake_mode is not None, we need to load extra cache files
+        if self.bitbake_mode:
+            self.extra_cachefile = os.path.join(self.cachedir,
+                        "bb_extracache_%s.dat" %self.bitbake_mode)
+
         logger.debug(1, "Using cache in '%s'", self.cachedir)
         bb.utils.mkdirhier(self.cachedir)
 
@@ -302,7 +315,7 @@ class Cache(object):
         return bb_data[virtual]
 
     @classmethod
-    def parse(cls, filename, appends, configdata):
+    def parse(cls, filename, appends, configdata, bitbake_mode=None):
         """Parse the specified filename, returning the recipe information"""
         infos = []
         datastores = cls.load_bbfile(filename, appends, configdata)
@@ -565,7 +578,7 @@ class CacheData(object):
     The data structures we compile from the cached data
     """
 
-    def __init__(self):
+    def __init__(self, bitbake_mode):
         # Direct cache variables
         self.providers = defaultdict(list)
         self.rproviders = defaultdict(list)
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index a1cd4d7..32ad96d 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -71,6 +71,10 @@ class BBCooker:
             self.server = server.BitBakeServer(self)
 
         self.configuration = configuration
+        # Cooker can get the bitbake mode information from configuration,
+        # we need to log down this information and pass it to Cache for
+        # extra cache file implementation
+        self.bitbake_mode = configuration.ui
 
         self.configuration.data = bb.data.init()
 
@@ -713,9 +717,10 @@ class BBCooker:
 
         self.buildSetVars()
 
-        self.status = bb.cache.CacheData()
+        self.status = bb.cache.CacheData(self.bitbake_mode)
         infos = bb.cache.Cache.parse(fn, self.get_file_appends(fn), \
-                                     self.configuration.data)
+                                     self.configuration.data, \
+                                     self.bitbake_mode)
         infos = dict(infos)
 
         fn = bb.cache.Cache.realfn2virtual(buildfile, cls)
@@ -859,7 +864,7 @@ class BBCooker:
                 else:
                     collectlog.info("You have disabled Psyco. This decreases performance.")
 
-            self.status = bb.cache.CacheData()
+            self.status = bb.cache.CacheData(self.bitbake_mode)
 
             ignore = bb.data.getVar("ASSUME_PROVIDED", self.configuration.data, 1) or ""
             self.status.ignored_dependencies = set(ignore.split())
-- 
1.7.0.4

[PATCH 2/4] Put extra requested fields into different cache files

[Liping Ke]

From: Liping Ke <liping.ke@intel.com>

This patch splits Cache Data Retrieve method from Data Fields.
Data Retrive methods will be reused by Extra Cache Data fields.
It is independent Class methods for data retrieving.

Signed-off-by: Liping Ke <liping.ke@intel.com>
---
 bitbake/lib/bb/cache.py |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index c86aa14..dec8bdd 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -83,7 +83,7 @@ recipe_fields = (
 )
 
 
-class RecipeInfo(namedtuple('RecipeInfo', recipe_fields)):
+class RecipeRetrieve():
     __slots__ = ()
 
     @classmethod
@@ -117,6 +117,7 @@ class RecipeInfo(namedtuple('RecipeInfo', recipe_fields)):
     def getvar(cls, var, metadata):
         return metadata.getVar(var, True) or ''
 
+class RecipeInfo(namedtuple('RecipeInfo', recipe_fields), RecipeRetrieve):
     @classmethod
     def make_optional(cls, default=None, **kwargs):
         """Construct the namedtuple from the specified keyword arguments,
-- 
1.7.0.4

[PATCH 3/4] Put extra requested fields into different cache files

[Liping Ke]

From: Liping Ke <liping.ke@intel.com>

This patch introduces Extra required fields for image creator.
We introduce extra required fields as well as namedtuple recipeinfo
class here. We also define the data retrieving methods for this
extra recipeinfo class. This factory methods can be easily extended
for furthing expanding. The extended Hob(Image Creator) class will
be used in the following patches.

Signed-off-by: Liping Ke <Liping.ke@intel.com>
---
 bitbake/lib/bb/cache.py |   29 +++++++++++++++++++++++++++++
 1 files changed, 29 insertions(+), 0 deletions(-)

diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index dec8bdd..dc112ff 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -117,7 +117,36 @@ class RecipeRetrieve():
     def getvar(cls, var, metadata):
         return metadata.getVar(var, True) or ''
 
+# This three fields are only requested by Image Creator
+# hob_recipe_fields, HobRecipeInfo Definition are both
+# introduced by Image Creator. If a user want to introduce
+# more fields, we need to to the same thing as below:
+# 1. introduce extra recipe fields
+# 2. Define your extra named tuple class
+# 3. Add extra data loading methods in the else part of
+#    the Factory Class.
+hob_extra_recipe_fields = (
+    'summary',
+    'license',
+    'section'
+)
+
+HobExtraRecipeInfo = namedtuple('HobExtraRecipeInfo', hob_extra_recipe_fields)
+    
+class ExtraRecipeInfoFactory(RecipeRetrieve):
+    @classmethod
+    def from_metadata(cls, bitbake_mode, metadata):
+        if bitbake_mode == "hob":
+            return HobExtraRecipeInfo(
+                summary          = cls.getvar('SUMMARY', metadata),
+                license          = cls.getvar('LICENSE', metadata),
+                section          = cls.getvar('SECTION', metadata),
+            )
+
+
 class RecipeInfo(namedtuple('RecipeInfo', recipe_fields), RecipeRetrieve):
+    # Please note: fields are the static class member
+    # in namedtuple class RecipeInfo
     @classmethod
     def make_optional(cls, default=None, **kwargs):
         """Construct the namedtuple from the specified keyword arguments,
-- 
1.7.0.4

[PATCH 4/4] Put extra requested fields into different cache files

[Liping Ke]

From: Liping Ke <liping.ke@intel.com>

This patch implements independent cache for Extra Cache Fields Request
Since Image Creator need extra cache fields which are not used
by bitbake, we create the extra cache file to load this extra
cache fields for making it more extensible. In the future, we
can handle similar request. This implementation does not touch
the base recipe info path. Extra fields are dealt with separately.

Signed-off-by: Liping Ke <liping.ke@intel.com>
---
 bitbake/lib/bb/cache.py  |  138 +++++++++++++++++++++++++++++++++++----------
 bitbake/lib/bb/cooker.py |   44 ++++++++++----
 2 files changed, 138 insertions(+), 44 deletions(-)

diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index dc112ff..aefd26b 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -75,9 +75,6 @@ recipe_fields = (
     'basetaskhashes',
     'hashfilename',
     'inherits',
-    'summary',
-    'license',
-    'section',
     'fakerootenv',
     'fakerootdirs'
 )
@@ -204,9 +201,6 @@ class RecipeInfo(namedtuple('RecipeInfo', recipe_fields), RecipeRetrieve):
             rdepends_pkg     = cls.pkgvar('RDEPENDS', packages, metadata),
             rrecommends_pkg  = cls.pkgvar('RRECOMMENDS', packages, metadata),
             inherits         = cls.getvar('__inherit_cache', metadata),
-            summary          = cls.getvar('SUMMARY', metadata),
-            license          = cls.getvar('LICENSE', metadata),
-            section          = cls.getvar('SECTION', metadata),
             fakerootenv      = cls.getvar('FAKEROOTENV', metadata),
             fakerootdirs     = cls.getvar('FAKEROOTDIRS', metadata),
         )
@@ -259,11 +253,21 @@ class Cache(object):
         old_mtimes = [old_mtime for _, old_mtime in deps]
         old_mtimes.append(newest_mtime)
         newest_mtime = max(old_mtimes)
+        
+        # We need load extra cache
+        if bitbake_mode:
+            if bb.parse.cached_mtime_noerror(self.cachefile) >= newest_mtime \
+                    and bb.parse.cached_mtime_noerror(self.extra_cachefile) \
+                            >= newest_mtime:
+                self.load_cachefile()
+            elif os.path.isfile(self.cachefile):
+                logger.info("Out of date cache found, rebuilding...")
+        else:
+            if bb.parse.cached_mtime_noerror(self.cachefile) >= newest_mtime:
+                self.load_cachefile()
+            elif os.path.isfile(self.cachefile):
+                logger.info("Out of date cache found, rebuilding...") 
 
-        if bb.parse.cached_mtime_noerror(self.cachefile) >= newest_mtime:
-            self.load_cachefile()
-        elif os.path.isfile(self.cachefile):
-            logger.info("Out of date cache found, rebuilding...")
 
     def load_cachefile(self):
         with open(self.cachefile, "rb") as cachefile:
@@ -283,9 +287,36 @@ class Cache(object):
                 return
 
             cachesize = os.fstat(cachefile.fileno()).st_size
-            bb.event.fire(bb.event.CacheLoadStarted(cachesize), self.data)
 
             previous_percent = 0
+            current_progress = 0
+            extra_current_progress = 0
+
+            if self.bitbake_mode:
+                with open(self.extra_cachefile, "rb") as extra_cachefile:
+                    cachesize += os.fstat(extra_cachefile.fileno()).st_size
+                    pickled_extra = pickle.Unpickler(extra_cachefile)
+                    bb.event.fire(
+                            bb.event.CacheLoadStarted(cachesize), self.data)
+                    # Refresh Progress Bar                    
+                    while extra_cachefile:
+                        try:
+                            key = pickled_extra.load()
+                            value = pickled_extra.load()
+                        except Exception:
+                            break
+                        self.extra_depends_cache[key] = value
+                        current_progress = extra_cachefile.tell()
+                        current_percent = 100 * current_progress / cachesize
+                        if current_percent > previous_percent:
+                            previous_percent = current_percent
+                            bb.event.fire(
+                                bb.event.CacheLoadProgress(current_progress),
+                                          self.data)
+                extra_current_progress = current_progress
+            else:
+                bb.event.fire(bb.event.CacheLoadStarted(cachesize), self.data)
+
             while cachefile:
                 try:
                     key = pickled.load()
@@ -296,7 +327,7 @@ class Cache(object):
                 self.depends_cache[key] = value
 
                 # only fire events on even percentage boundaries
-                current_progress = cachefile.tell()
+                current_progress = cachefile.tell() + extra_current_progress
                 current_percent = 100 * current_progress / cachesize
                 if current_percent > previous_percent:
                     previous_percent = current_percent
@@ -304,8 +335,8 @@ class Cache(object):
                                   self.data)
 
             bb.event.fire(bb.event.CacheLoadCompleted(cachesize,
-                                                      len(self.depends_cache)),
-                          self.data)
+                    len(self.depends_cache) + len(self.extra_depends_cache)),
+                            self.data)
 
     @staticmethod
     def virtualfn2realfn(virtualfn):
@@ -348,6 +379,7 @@ class Cache(object):
     def parse(cls, filename, appends, configdata, bitbake_mode=None):
         """Parse the specified filename, returning the recipe information"""
         infos = []
+        extra_infos = []
         datastores = cls.load_bbfile(filename, appends, configdata)
         depends = set()
         for variant, data in sorted(datastores.iteritems(),
@@ -359,7 +391,11 @@ class Cache(object):
                 data.setVar("__depends", depends)
             info = RecipeInfo.from_metadata(filename, data)
             infos.append((virtualfn, info))
-        return infos
+            if bitbake_mode:
+                extra_info = \
+                    ExtraRecipeInfoFactory.from_metadata(bitbake_mode, data)
+                extra_infos.append((virtualfn, extra_info))
+        return infos, extra_infos
 
     def load(self, filename, appends, configdata):
         """Obtain the recipe information for the specified filename,
@@ -372,15 +408,20 @@ class Cache(object):
         cached = self.cacheValid(filename)
         if cached:
             infos = []
+            extra_infos= []
             info = self.depends_cache[filename]
-            for variant in info.variants:
+            for i in range(0, len(info.variants)):
+                variant = info.variants[i]
                 virtualfn = self.realfn2virtual(filename, variant)
                 infos.append((virtualfn, self.depends_cache[virtualfn]))
+                if self.bitbake_mode:
+                    extra_infos.append((virtualfn, \
+                            self.extra_depends_cache[virtualfn]))
         else:
             logger.debug(1, "Parsing %s", filename)
-            return self.parse(filename, appends, configdata)
+            return self.parse(filename, appends, configdata, self.bitbake_mode)
 
-        return cached, infos
+        return cached, infos, extra_infos
 
     def loadData(self, fn, appends, cfgData, cacheData):
         """Load the recipe info for the specified filename,
@@ -388,13 +429,17 @@ class Cache(object):
         the recipe information to the supplied CacheData instance."""
         skipped, virtuals = 0, 0
 
-        cached, infos = self.load(fn, appends, cfgData)
-        for virtualfn, info in infos:
+        cached, infos, extra_infos = self.load(fn, appends, cfgData)
+        for i in range (0, len(infos)):
+            (virtualfn, info) = infos[i]
+            if self.bitbake_mode:
+                extra_info = extra_infos[i]
             if info.skipped:
                 logger.debug(1, "Skipping %s", virtualfn)
                 skipped += 1
             else:
-                self.add_info(virtualfn, info, cacheData, not cached)
+                self.add_info(virtualfn, info, extra_info, \
+                            cacheData, not cached)
                 virtuals += 1
 
         return cached, skipped, virtuals
@@ -426,7 +471,8 @@ class Cache(object):
         self.checked.add(fn)
 
         # File isn't in depends_cache
-        if not fn in self.depends_cache:
+        if (not fn in self.depends_cache) or (self.bitbake_mode \
+                       and (not fn in self.extra_depends_cache)):
             logger.debug(2, "Cache: %s is not cached", fn)
             return False
 
@@ -467,7 +513,9 @@ class Cache(object):
         for cls in info.variants:
             virtualfn = self.realfn2virtual(fn, cls)
             self.clean.add(virtualfn)
-            if virtualfn not in self.depends_cache:
+            if (virtualfn not in self.depends_cache) or \
+                         (self.bitbake_mode \
+                         and virtualfn not in self.extra_depends_cache):
                 logger.debug(2, "Cache: %s is not cached", virtualfn)
                 invalid = True
 
@@ -494,6 +542,12 @@ class Cache(object):
         if fn in self.depends_cache:
             logger.debug(1, "Removing %s from cache", fn)
             del self.depends_cache[fn]
+        # when deleting, we need to maintain the consistency
+        # of the two independent cache
+        if self.bitbake_mode and fn in self.extra_depends_cache:
+            logger.debug(1, "Removing %s from ui_extra_cache", fn)
+            del self.extra_depends_cache[fn]
+
         if fn in self.clean:
             logger.debug(1, "Marking %s as unclean", fn)
             self.clean.remove(fn)
@@ -519,15 +573,26 @@ class Cache(object):
                 pickler.dump(key)
                 pickler.dump(value)
 
+        # Sync back the extra cache fields into the separate cache file
+        if self.bitbake_mode:
+            with open(self.extra_cachefile, "wb") as extra_cachefile:
+                extra_pickler = pickle.Pickler(extra_cachefile, \
+                                               pickle.HIGHEST_PROTOCOL)
+                for key, value in self.extra_depends_cache.iteritems():
+                    extra_pickler.dump(key)
+                    extra_pickler.dump(value)
+
         del self.depends_cache
+        del self.extra_depends_cache
 
     @staticmethod
     def mtime(cachefile):
         return bb.parse.cached_mtime_noerror(cachefile)
 
-    def add_info(self, filename, info, cacheData, parsed=None):
+    def add_info(self, filename, info, extra_info, cacheData, parsed=None):
         if not info.skipped:
             cacheData.add_from_recipeinfo(filename, info)
+            cacheData.add_from_extra_recipeinfo(filename, extra_info)
 
         if not self.has_cache:
             return
@@ -536,15 +601,20 @@ class Cache(object):
             if parsed:
                 self.cacheclean = False
             self.depends_cache[filename] = info
+            if self.bitbake_mode:
+                self.extra_depends_cache[filename] = extra_info
 
     def add(self, file_name, data, cacheData, parsed=None):
         """
         Save data we need into the cache
         """
-
         realfn = self.virtualfn2realfn(file_name)[0]
+        extra_info = None
         info = RecipeInfo.from_metadata(realfn, data)
-        self.add_info(file_name, info, cacheData, parsed)
+        if self.bitbake_mode:
+            extra_info = \
+                ExtraRecipeInfoFactory.from_metadata(self.bitbake_mode, data)
+        self.add_info(file_name, info, extra_info, cacheData, parsed)
 
     @staticmethod
     def load_bbfile(bbfile, appends, config):
@@ -610,6 +680,7 @@ class CacheData(object):
 
     def __init__(self, bitbake_mode):
         # Direct cache variables
+        self.bitbake_mode = bitbake_mode
         self.providers = defaultdict(list)
         self.rproviders = defaultdict(list)
         self.packages = defaultdict(list)
@@ -635,11 +706,13 @@ class CacheData(object):
         self.basetaskhash = {}
         self.hashfn = {}
         self.inherits = {}
+        self.fakerootenv = {}
+        self.fakerootdirs = {}
+
+# Extra cache fields
         self.summary = {}
         self.license = {}
         self.section = {}
-        self.fakerootenv = {}
-        self.fakerootdirs = {}
 
         # Indirect Cache variables (set elsewhere)
         self.ignored_dependencies = []
@@ -647,6 +720,12 @@ class CacheData(object):
         self.bbfile_priority = {}
         self.bbfile_config_priorities = []
 
+    def add_from_extra_recipeinfo(self, fn, extra_info):
+        if self.bitbake_mode:
+            self.summary[fn] = extra_info.summary
+            self.license[fn] = extra_info.license
+            self.section[fn] = extra_info.section
+
     def add_from_recipeinfo(self, fn, info):
         self.task_deps[fn] = info.task_deps
         self.pkg_fn[fn] = info.pn
@@ -705,8 +784,5 @@ class CacheData(object):
             self.basetaskhash[identifier] = taskhash
 
         self.inherits[fn] = info.inherits
-        self.summary[fn] = info.summary
-        self.license[fn] = info.license
-        self.section[fn] = info.section
         self.fakerootenv[fn] = info.fakerootenv
         self.fakerootdirs[fn] = info.fakerootdirs
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 32ad96d..0590004 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -718,7 +718,7 @@ class BBCooker:
         self.buildSetVars()
 
         self.status = bb.cache.CacheData(self.bitbake_mode)
-        infos = bb.cache.Cache.parse(fn, self.get_file_appends(fn), \
+        infos, extra_infos = bb.cache.Cache.parse(fn, self.get_file_appends(fn), \
                                      self.configuration.data, \
                                      self.bitbake_mode)
         infos = dict(infos)
@@ -730,6 +730,13 @@ class BBCooker:
             bb.fatal("%s does not exist" % fn)
         self.status.add_from_recipeinfo(fn, maininfo)
 
+        if self.bitbake_mode:
+            try:
+                extra_info = extra_infos[fn]
+            except KeyError:
+                bb.fatal("%s does not exist" % fn)
+            self.status.add_from_extra_recipeinfo(fn, extra_info)
+
         # Tweak some variables
         item = maininfo.pn
         self.status.ignored_dependencies = set()
@@ -1071,9 +1078,9 @@ class ParsingFailure(Exception):
         self.args = (realexception, recipe)
 
 def parse_file(task):
-    filename, appends = task
+    filename, appends, bitbake_mode = task
     try:
-        return True, bb.cache.Cache.parse(filename, appends, parse_file.cfg)
+        return True, bb.cache.Cache.parse(filename, appends, parse_file.cfg, bitbake_mode)
     except Exception, exc:
         exc.recipe = filename
         raise exc
@@ -1087,6 +1094,7 @@ class CookerParser(object):
     def __init__(self, cooker, filelist, masked):
         self.filelist = filelist
         self.cooker = cooker
+        self.bitbake_mode = cooker.configuration.ui
         self.cfgdata = cooker.configuration.data
 
         # Accounting statistics
@@ -1103,13 +1111,13 @@ class CookerParser(object):
         self.num_processes = int(self.cfgdata.getVar("BB_NUMBER_PARSE_THREADS", True) or
                                  multiprocessing.cpu_count())
 
-        self.bb_cache = bb.cache.Cache(self.cfgdata)
+        self.bb_cache = bb.cache.Cache(self.cfgdata,self.bitbake_mode)
         self.fromcache = []
         self.willparse = []
         for filename in self.filelist:
             appends = self.cooker.get_file_appends(filename)
             if not self.bb_cache.cacheValid(filename):
-                self.willparse.append((filename, appends))
+                self.willparse.append((filename, appends, self.bitbake_mode))
             else:
                 self.fromcache.append((filename, appends))
         self.toparse = self.total - len(self.fromcache)
@@ -1148,12 +1156,12 @@ class CookerParser(object):
 
     def load_cached(self):
         for filename, appends in self.fromcache:
-            cached, infos = self.bb_cache.load(filename, appends, self.cfgdata)
-            yield not cached, infos
+            cached, infos, ui_infos = self.bb_cache.load(filename, appends, self.cfgdata)
+            yield not cached, (infos, ui_infos)
 
     def parse_next(self):
         try:
-            parsed, result = self.results.next()
+            parsed, (result, extra_result) = self.results.next()
         except StopIteration:
             self.shutdown()
             return False
@@ -1174,16 +1182,26 @@ class CookerParser(object):
         else:
             self.cached += 1
 
-        for virtualfn, info in result:
+        extra_info = None
+        for i in range (0, len(result)):
+            (virtualfn, info) = result[i]
+            if self.bitbake_mode:
+                (extra_virtualfn, extra_info) = extra_result[i]
+                if (virtualfn != extra_virtualfn):
+                    raise Exception("Inconsistancy happens for extra cache!")
             if info.skipped:
                 self.skipped += 1
-            self.bb_cache.add_info(virtualfn, info, self.cooker.status,
-                                        parsed=parsed)
+            self.bb_cache.add_info(virtualfn, info, extra_info, self.cooker.status,
+                                         parsed=parsed)
         return True
 
     def reparse(self, filename):
-        infos = self.bb_cache.parse(filename,
+        infos, extra_infos = self.bb_cache.parse(filename,
                                     self.cooker.get_file_appends(filename),
-                                    self.cfgdata)
+                                    self.cfgdata, self.bitbake_mode)
         for vfn, info in infos:
             self.cooker.status.add_from_recipeinfo(vfn, info)
+        if self.bitbake_mode:
+            for vfn,extra_info in extra_infos:
+                self.status.add_from_extra_recipeinfo(vfn, extra_info)
+            
-- 
1.7.0.4

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Ke, Liping]

Hi, all

Send twice with the modified subjects since those patches are introduced for image creator backend.

Thanks a lot!
criping

> -----Original Message-----
> From: poky-bounces@yoctoproject.org
> [mailto:poky-bounces@yoctoproject.org] On Behalf Of Liping Ke
> Sent: Monday, May 09, 2011 2:40 PM
> To: poky@yoctoproject.org
> Subject: [poky] [PATCH 0/4][Image Creator]Put extra requested fields into
> different cache files
> 
> From: Liping Ke <liping.ke@intel.com>
> 
> Below four patches are for putting extra requested fields inito different
> cache files instead of using only on bb_cache.dat. Now image creator need
> extra three fields. And in the future, there might be more similar requests.
> For each extra requestor, we will save the requested fields data into a
> separate cache files so that those who don't need it will not be impacted
> with larger fields and large data files.
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: lke/cache_impl
>   Browse:
> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/cache_impl
> 
> Thanks,
>     Liping Ke <liping.ke@intel.com>
> ---
> 
> 
> Liping Ke (4):
>   Introduce new param bitbake_mode into Cache.py
>   Split Cache Data Retrieve method from Data Fields
>   Introduce Extra required fields for image creator
>   Implement independent cache for Extra Cache Fields Request
> 
>  bitbake/lib/bb/cache.py  |  189
> +++++++++++++++++++++++++++++++++++++---------
>  bitbake/lib/bb/cooker.py |   55 ++++++++++----
>  2 files changed, 193 insertions(+), 51 deletions(-)
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Ke, Liping]

Hi, Josh

Those patches are rebased with latest poky master today. I noticed Richard
has changed the code of cache, making sure skipped info is saved for avoiding
unnecessary parsing, so that the cache file will remain the same size each time.
( I found before this change, cache file kept increased).
I verified that our new introduced patch working with this latest cache changes.

Thanks a lot!
criping

> -----Original Message-----
> From: Ke, Liping
> Sent: Monday, May 09, 2011 2:53 PM
> To: Ke, Liping; poky@yoctoproject.org
> Subject: RE: [poky] [PATCH 0/4][Image Creator]Put extra requested fields into
> different cache files
> 
> Hi, all
> 
> Send twice with the modified subjects since those patches are introduced for
> image creator backend.
> 
> Thanks a lot!
> criping
> 
> > -----Original Message-----
> > From: poky-bounces@yoctoproject.org
> > [mailto:poky-bounces@yoctoproject.org] On Behalf Of Liping Ke
> > Sent: Monday, May 09, 2011 2:40 PM
> > To: poky@yoctoproject.org
> > Subject: [poky] [PATCH 0/4][Image Creator]Put extra requested fields into
> > different cache files
> >
> > From: Liping Ke <liping.ke@intel.com>
> >
> > Below four patches are for putting extra requested fields inito different
> > cache files instead of using only on bb_cache.dat. Now image creator need
> > extra three fields. And in the future, there might be more similar requests.
> > For each extra requestor, we will save the requested fields data into a
> > separate cache files so that those who don't need it will not be impacted
> > with larger fields and large data files.
> >
> > Pull URL: git://git.pokylinux.org/poky-contrib.git
> >   Branch: lke/cache_impl
> >   Browse:
> > http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/cache_impl
> >
> > Thanks,
> >     Liping Ke <liping.ke@intel.com>
> > ---
> >
> >
> > Liping Ke (4):
> >   Introduce new param bitbake_mode into Cache.py
> >   Split Cache Data Retrieve method from Data Fields
> >   Introduce Extra required fields for image creator
> >   Implement independent cache for Extra Cache Fields Request
> >
> >  bitbake/lib/bb/cache.py  |  189
> > +++++++++++++++++++++++++++++++++++++---------
> >  bitbake/lib/bb/cooker.py |   55 ++++++++++----
> >  2 files changed, 193 insertions(+), 51 deletions(-)
> >
> > _______________________________________________
> > poky mailing list
> > poky@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/poky

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Richard Purdie]

Hi Liping,

On Mon, 2011-05-09 at 14:39 +0800, Liping Ke wrote:
> Below four patches are for putting extra requested fields inito different
> cache files instead of using only on bb_cache.dat. Now image creator need
> extra three fields. And in the future, there might be more similar requests.
> For each extra requestor, we will save the requested fields data into a
> separate cache files so that those who don't need it will not be impacted
> with larger fields and large data files.
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: lke/cache_impl
>   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/cache_impl
> 
> Thanks,
>     Liping Ke <liping.ke@intel.com>

These patches are good and I like the ultimate result but the patches
themselves could use some tweaks in the order they make changes. Ideally
changes need to be made in a way that each patch can stand on its own so
I could apply patches 1 and 2, leave 3 and 4 out and the code should
still work.

In this sense, the series should really be a set of patches which:

a) Adds "bitbake_mode" and allows UIs to define cache data
b) Adds the cache data to hob
c) Drops the cache data used only by hob from the core

I've also the following comments on the code in general:

a) Could we move the hob specific cache data from cache.py to hob.py. If
not, I'd like to understand why not and maybe address those issues. My
point is you shouldn't need to change cache.py to add extra cache data.

b) Could we rename "bitbake_mode" to "extracaches" or something?
"bitbake_mode" is a bit too generic and meaningless.

c) I was also wondering how hard it would be to make this an array and
allow more than one to be loaded? I'm thinking of the use case where we
may have more than one "UI" in future and configure things so the user
can switch between them without re-parsing.

Cheers,

Richard

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Ke, Liping]

Hi, Richard,

Thanks for your feedback!!! please see below answers.

criping

> These patches are good and I like the ultimate result but the patches
> themselves could use some tweaks in the order they make changes. Ideally
> changes need to be made in a way that each patch can stand on its own so
> I could apply patches 1 and 2, leave 3 and 4 out and the code should
> still work.
> 
> In this sense, the series should really be a set of patches which:
> 
> a) Adds "bitbake_mode" and allows UIs to define cache data
> b) Adds the cache data to hob
> c) Drops the cache data used only by hob from the core
Yes, in general I want to do the same thing. Each patch in the serial will
not break the current cs. Patch[1/4], patch[2/4]& patch [3/4] are for a) and b).
But for c), because of the implantation algorithm I adopted, it's merged with
extra cache data implementation (the biggest patch).

 
> I've also the following comments on the code in general:
> 
> a) Could we move the hob specific cache data from cache.py to hob.py. If
> not, I'd like to understand why not and maybe address those issues. My
> point is you shouldn't need to change cache.py to add extra cache data.

you mean that we will have a separate ExtraCache (I guess hob or adt-installer should have the similar logic?)
implementation file, which has similar functionality with cache.py? 
Oh, I did not think of this implementation at all when implement. In my mind, change less code is the principle since
this part of the code is the core of bitbake:)  If split, many codes in cache could be reused by both cache.py and hob.py?

And also, if you think it is a must, could we split the task, for this phase, keep it in the same file. 
And then refactory them in another patch later? Seems there're many tasks for image creator? Seems the schedule is
tight?

> b) Could we rename "bitbake_mode" to "extracaches" or something?
> "bitbake_mode" is a bit too generic and meaningless.
Sure. It's nice.

> c) I was also wondering how hard it would be to make this an array and
> allow more than one to be loaded? I'm thinking of the use case where we
> may have more than one "UI" in future and configure things so the user
> can switch between them without re-parsing.
> 
Hi, Richard, actually user could have more than one data file. If we have autobuilder extra requests,
we can easily extend with few codes: (just like patch 3)
1)define autobuilder_extra_fields,
2) declare autobuilder namedtuple class
3) add else in ExtraClassFactory for defining from_metadata.
Then the bb_cacheautobuilder.dat will be loaded and processed. 
Then there would be more than two cache data files (bb_cache.dat, bb_cachehob.dat, bb_cacheautobuilder.dat, etc).
But each time when bitbake is running, only two files will be loaded/saved. (bb_cache.dat, bb_cachehob.dat)
or (bb_cache.dat, bb_cacheautobuilder.dat). bitbake_mode are used for passing the parameters, 'hob' or 'autobuilder'?
Is this what you mean?

> Cheers,
> 
> Richard

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Richard Purdie]

On Tue, 2011-05-10 at 09:26 +0800, Ke, Liping wrote:
> > These patches are good and I like the ultimate result but the patches
> > themselves could use some tweaks in the order they make changes. Ideally
> > changes need to be made in a way that each patch can stand on its own so
> > I could apply patches 1 and 2, leave 3 and 4 out and the code should
> > still work.
> > 
> > In this sense, the series should really be a set of patches which:
> > 
> > a) Adds "bitbake_mode" and allows UIs to define cache data
> > b) Adds the cache data to hob
> > c) Drops the cache data used only by hob from the core
> Yes, in general I want to do the same thing. Each patch in the serial will
> not break the current cs. Patch[1/4], patch[2/4]& patch [3/4] are for a) and b).
> But for c), because of the implantation algorithm I adopted, it's merged with
> extra cache data implementation (the biggest patch).

That's fine, your original series didn't seem to work quite like this
though and looked like I'd get failures if I only applied the early
patches from what I remember.

> > I've also the following comments on the code in general:
> > 
> > a) Could we move the hob specific cache data from cache.py to hob.py. If
> > not, I'd like to understand why not and maybe address those issues. My
> > point is you shouldn't need to change cache.py to add extra cache data.
> 
> you mean that we will have a separate ExtraCache (I guess hob or adt-installer should have the similar logic?)
> implementation file, which has similar functionality with cache.py? 
> Oh, I did not think of this implementation at all when implement. In my mind, change less code is the principle since
> this part of the code is the core of bitbake:)  If split, many codes in cache could be reused by both cache.py and hob.py?

I'm thinking that cache.py defines the class but the hob specific
definition using that class to extend the cache data should be part of
the hob codebase. In the future there may be external UIs calling into
this code and needing this functionality and I'd prefer them not to each
need to change cache.py.

> And also, if you think it is a must, could we split the task, for this phase, keep it in the same file. 
> And then refactory them in another patch later? Seems there're many tasks for image creator? Seems the schedule is
> tight?

If you don't think this is realistic in the time available, I might be
able to take a look at it.

> > b) Could we rename "bitbake_mode" to "extracaches" or something?
> > "bitbake_mode" is a bit too generic and meaningless.
> Sure. It's nice.
> 
> > c) I was also wondering how hard it would be to make this an array and
> > allow more than one to be loaded? I'm thinking of the use case where we
> > may have more than one "UI" in future and configure things so the user
> > can switch between them without re-parsing.
> > 
> Hi, Richard, actually user could have more than one data file. If we have autobuilder extra requests,
> we can easily extend with few codes: (just like patch 3)
> 1)define autobuilder_extra_fields,
> 2) declare autobuilder namedtuple class
> 3) add else in ExtraClassFactory for defining from_metadata.
> Then the bb_cacheautobuilder.dat will be loaded and processed. 
> Then there would be more than two cache data files (bb_cache.dat, bb_cachehob.dat, bb_cacheautobuilder.dat, etc).
> But each time when bitbake is running, only two files will be loaded/saved. (bb_cache.dat, bb_cachehob.dat)
> or (bb_cache.dat, bb_cacheautobuilder.dat). bitbake_mode are used for passing the parameters, 'hob' or 'autobuilder'?
> Is this what you mean?

No, I mean can we could pass:

bitbake_mode = ['hob', 'autobuilder']

and it then uses three cache files so bitbake_mode would become a list
rather than a single item.

Cheers,

Richard

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Ke, Liping]

Hi, Richard

> > >
> > > a) Adds "bitbake_mode" and allows UIs to define cache data
> > > b) Adds the cache data to hob
> > > c) Drops the cache data used only by hob from the core
> > Yes, in general I want to do the same thing. Each patch in the serial will
> > not break the current cs. Patch[1/4], patch[2/4]& patch [3/4] are for a) and
> b).
> > But for c), because of the implantation algorithm I adopted, it's merged with
> > extra cache data implementation (the biggest patch).
> 
> That's fine, your original series didn't seem to work quite like this
> though and looked like I'd get failures if I only applied the early
> patches from what I remember.

sure, I understand this. I will make sure that the series of patches works independently
when resending the patch.

> 
> > > I've also the following comments on the code in general:
> > >
> > > a) Could we move the hob specific cache data from cache.py to hob.py. If
> > > not, I'd like to understand why not and maybe address those issues. My
> > > point is you shouldn't need to change cache.py to add extra cache data.
> >
> > you mean that we will have a separate ExtraCache (I guess hob or
> adt-installer should have the similar logic?)
> > implementation file, which has similar functionality with cache.py?
> > Oh, I did not think of this implementation at all when implement. In my mind,
> change less code is the principle since
> > this part of the code is the core of bitbake:)  If split, many codes in cache
> could be reused by both cache.py and hob.py?
> 
> I'm thinking that cache.py defines the class but the hob specific
> definition using that class to extend the cache data should be part of
> the hob codebase. In the future there may be external UIs calling into
> this code and needing this functionality and I'd prefer them not to each
> need to change cache.py.

Hi, Richard, just a question here. You mean ext-hob-cache data will be a sub-class
of cachy data? Inheritance here? We originally consider this, yet just failed to find
a good way. Namedtuple has the static dict fields. So namedtuple class can't be extended.
For inheritance, I must change the original cache.py implantation. 
Just want to hear your suggestion here...

> 
> > And also, if you think it is a must, could we split the task, for this phase, keep
> it in the same file.
> > And then refactory them in another patch later? Seems there're many tasks
> for image creator? Seems the schedule is
> > tight?
> 
> If you don't think this is realistic in the time available, I might be
> able to take a look at it.

Oh, I guess Jessica will re-arrange the schedule today... Seems for the task, there're
many different understanding about the requirement at the very beginning... I will try to
understand all the details before writing the code -:)
 

> No, I mean can we could pass:
> 
> bitbake_mode = ['hob', 'autobuilder']
> 
> and it then uses three cache files so bitbake_mode would become a list
> rather than a single item.
> 
OK, I understand this requirement now.
After we sync up today, I should have a clearer understanding about the requirement and I will redo the patch!

Thanks a lot for your help!
criping


> Cheers,
> 
> Richard