[PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Liping Ke]

From: Liping Ke <liping.ke@intel.com>

Below four patches are for putting extra requested fields inito different
cache files instead of using only on bb_cache.dat. Now image creator need
extra three fields. And in the future, there might be more similar requests.
For each extra requestor, we will save the requested fields data into a
separate cache files so that those who don't need it will not be impacted
with larger fields and large data files.

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: lke/cache_impl
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/cache_impl

Thanks,
    Liping Ke <liping.ke@intel.com>
---


Liping Ke (4):
  This patch introduce new param bitbake_mode into Cache.py.
  This patch splits Cache Data Retrieve method from Data Fields.
  This patch introduces Extra required fields for image creator.
  This patch implements independent cache for Extra Cache Fields
    Request

 bitbake/lib/bb/cache.py       |  161 ++++++++++++++++++++++++++++++++---------
 bitbake/lib/bb/cooker.py      |   55 ++++++++++----
 bitbake/lib/bb/extra_cache.py |   29 ++++++++
 3 files changed, 196 insertions(+), 49 deletions(-)
 create mode 100644 bitbake/lib/bb/extra_cache.py

[PATCH 1/4][Image Creator]Put extra requested fields into different cache files

[Liping Ke]

From: Liping Ke <liping.ke@intel.com>

When using ui mode, we will save ui_mode required extra cache
fields into a separate cache file. This patch firstly introduce
this extracaches parameter. It will be used in the following
patches.

Signed-off-by: Liping Ke <liping.ke@intel.com>
---
 bitbake/lib/bb/cache.py  |   20 +++++++++++++++++---
 bitbake/lib/bb/cooker.py |   11 ++++++++---
 2 files changed, 25 insertions(+), 6 deletions(-)

diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index d083c51..f9d2e5f 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -187,7 +187,10 @@ class Cache(object):
     BitBake Cache implementation
     """
 
-    def __init__(self, data):
+    def __init__(self, data, extracaches=None):       
+        # Pass extracaches information into Cache constructor
+        # It will be used in later for deciding whether
+        # we need separate cache file dump/load support
         self.cachedir = bb.data.getVar("CACHE", data, True)
         self.clean = set()
         self.checked = set()
@@ -196,6 +199,11 @@ class Cache(object):
         self.data_fn = None
         self.cacheclean = True
 
+        # Below three variables are used for loading extra cache
+        self.extra_depends_cache = {}        
+        self.extracaches = extracaches
+        self.extra_cachefile = None
+
         if self.cachedir in [None, '']:
             self.has_cache = False
             logger.info("Not using a cache. "
@@ -205,6 +213,11 @@ class Cache(object):
         self.has_cache = True
         self.cachefile = os.path.join(self.cachedir, "bb_cache.dat")
 
+        # if extracaches is not None, we need to load extra cache files
+        if self.extracaches:
+            self.extra_cachefile = os.path.join(self.cachedir,
+                        "bb_extracache_%s.dat" %self.extracaches)
+
         logger.debug(1, "Using cache in '%s'", self.cachedir)
         bb.utils.mkdirhier(self.cachedir)
 
@@ -302,7 +315,7 @@ class Cache(object):
         return bb_data[virtual]
 
     @classmethod
-    def parse(cls, filename, appends, configdata):
+    def parse(cls, filename, appends, configdata, extracaches=None):
         """Parse the specified filename, returning the recipe information"""
         infos = []
         datastores = cls.load_bbfile(filename, appends, configdata)
@@ -565,8 +578,9 @@ class CacheData(object):
     The data structures we compile from the cached data
     """
 
-    def __init__(self):
+    def __init__(self, extracaches):
         # Direct cache variables
+        self.extracaches = extracaches
         self.providers = defaultdict(list)
         self.rproviders = defaultdict(list)
         self.packages = defaultdict(list)
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index a1cd4d7..eaf5923 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -71,6 +71,10 @@ class BBCooker:
             self.server = server.BitBakeServer(self)
 
         self.configuration = configuration
+        # Cooker can get the bitbake mode information from configuration,
+        # we need to log down this information and pass it to Cache for
+        # extra cache file implementation
+        self.extracaches = configuration.ui
 
         self.configuration.data = bb.data.init()
 
@@ -713,9 +717,10 @@ class BBCooker:
 
         self.buildSetVars()
 
-        self.status = bb.cache.CacheData()
+        self.status = bb.cache.CacheData(self.extracaches)
         infos = bb.cache.Cache.parse(fn, self.get_file_appends(fn), \
-                                     self.configuration.data)
+                                     self.configuration.data, \
+                                     self.extracaches)
         infos = dict(infos)
 
         fn = bb.cache.Cache.realfn2virtual(buildfile, cls)
@@ -859,7 +864,7 @@ class BBCooker:
                 else:
                     collectlog.info("You have disabled Psyco. This decreases performance.")
 
-            self.status = bb.cache.CacheData()
+            self.status = bb.cache.CacheData(self.extracaches)
 
             ignore = bb.data.getVar("ASSUME_PROVIDED", self.configuration.data, 1) or ""
             self.status.ignored_dependencies = set(ignore.split())
-- 
1.7.0.4

[PATCH 2/4][Image Creator]Put extra requested fields into different cache files

[Liping Ke]

From: Liping Ke <liping.ke@intel.com>

Data Retrive methods will be reused by Extra Cache Data fields.
It is independent Class methods for data retrieving.

Signed-off-by: Liping Ke <liping.ke@intel.com>
---
 bitbake/lib/bb/cache.py |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index f9d2e5f..82712ec 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -83,7 +83,7 @@ recipe_fields = (
 )
 
 
-class RecipeInfo(namedtuple('RecipeInfo', recipe_fields)):
+class RecipeRetrieve():
     __slots__ = ()
 
     @classmethod
@@ -117,6 +117,7 @@ class RecipeInfo(namedtuple('RecipeInfo', recipe_fields)):
     def getvar(cls, var, metadata):
         return metadata.getVar(var, True) or ''
 
+class RecipeInfo(namedtuple('RecipeInfo', recipe_fields), RecipeRetrieve):
     @classmethod
     def make_optional(cls, default=None, **kwargs):
         """Construct the namedtuple from the specified keyword arguments,
-- 
1.7.0.4

[PATCH 3/4][Image Creator]Put extra requested fields into different cache files

[Liping Ke]

From: Liping Ke <liping.ke@intel.com>

We introduce extra required fields as well as namedtuple recipeinfo
class here. We also define the data retrieving methods for this
extra recipeinfo class. This factory methods can be easily extended
for furthing expanding. The extended Hob(Image Creator) class will
be used in the following patches.

Signed-off-by: Liping Ke <Liping.ke@intel.com>
---
 bitbake/lib/bb/cache.py       |    2 ++
 bitbake/lib/bb/extra_cache.py |   29 +++++++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 0 deletions(-)
 create mode 100644 bitbake/lib/bb/extra_cache.py

diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index 82712ec..7dc518f 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -118,6 +118,8 @@ class RecipeRetrieve():
         return metadata.getVar(var, True) or ''
 
 class RecipeInfo(namedtuple('RecipeInfo', recipe_fields), RecipeRetrieve):
+    # Please note: fields are the static class member
+    # in namedtuple class RecipeInfo
     @classmethod
     def make_optional(cls, default=None, **kwargs):
         """Construct the namedtuple from the specified keyword arguments,
diff --git a/bitbake/lib/bb/extra_cache.py b/bitbake/lib/bb/extra_cache.py
new file mode 100644
index 0000000..11bb493
--- /dev/null
+++ b/bitbake/lib/bb/extra_cache.py
@@ -0,0 +1,29 @@
+from collections import namedtuple
+from bb.cache import RecipeRetrieve
+
+# This three fields are only requested by Image Creator
+# hob_recipe_fields, HobRecipeInfo Definition are both
+# introduced by Image Creator. If a user want to introduce
+# more fields, we need to to the same thing as below:
+# 1. introduce extra recipe fields
+# 2. Define your extra named tuple class
+# 3. Add extra data loading methods in the else part of
+#    the Factory Class.
+hob_extra_recipe_fields = (
+    'summary',
+    'license',
+    'section'
+)
+
+HobExtraRecipeInfo = namedtuple('HobExtraRecipeInfo', hob_extra_recipe_fields)
+    
+class ExtraRecipeInfoFactory(RecipeRetrieve):
+    @classmethod
+    def from_metadata(cls, extracaches, metadata):
+        if extracaches == "hob":
+            return HobExtraRecipeInfo(
+                summary          = cls.getvar('SUMMARY', metadata),
+                license          = cls.getvar('LICENSE', metadata),
+                section          = cls.getvar('SECTION', metadata),
+            )
+
-- 
1.7.0.4

[PATCH 4/4][Image Creator]Put extra requested fields into different cache files

[Liping Ke]

From: Liping Ke <liping.ke@intel.com>

Since Image Creator need extra cache fields which are not used
by bitbake, we create the extra cache file to load this extra
cache fields for making it more extensible. In the future, we
can handle similar request. This implementation does not touch
the base recipe info path. Extra fields are dealt with separately.

Signed-off-by: Liping Ke <liping.ke@intel.com>
---
 bitbake/lib/bb/cache.py  |  136 ++++++++++++++++++++++++++++++++++++----------
 bitbake/lib/bb/cooker.py |   44 +++++++++++-----
 2 files changed, 138 insertions(+), 42 deletions(-)

diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index 7dc518f..f25bafb 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -75,9 +75,6 @@ recipe_fields = (
     'basetaskhashes',
     'hashfilename',
     'inherits',
-    'summary',
-    'license',
-    'section',
     'fakerootenv',
     'fakerootdirs'
 )
@@ -117,6 +114,7 @@ class RecipeRetrieve():
     def getvar(cls, var, metadata):
         return metadata.getVar(var, True) or ''
 
+from bb.extra_cache import ExtraRecipeInfoFactory
 class RecipeInfo(namedtuple('RecipeInfo', recipe_fields), RecipeRetrieve):
     # Please note: fields are the static class member
     # in namedtuple class RecipeInfo
@@ -177,9 +175,6 @@ class RecipeInfo(namedtuple('RecipeInfo', recipe_fields), RecipeRetrieve):
             rdepends_pkg     = cls.pkgvar('RDEPENDS', packages, metadata),
             rrecommends_pkg  = cls.pkgvar('RRECOMMENDS', packages, metadata),
             inherits         = cls.getvar('__inherit_cache', metadata),
-            summary          = cls.getvar('SUMMARY', metadata),
-            license          = cls.getvar('LICENSE', metadata),
-            section          = cls.getvar('SECTION', metadata),
             fakerootenv      = cls.getvar('FAKEROOTENV', metadata),
             fakerootdirs     = cls.getvar('FAKEROOTDIRS', metadata),
         )
@@ -232,11 +227,21 @@ class Cache(object):
         old_mtimes = [old_mtime for _, old_mtime in deps]
         old_mtimes.append(newest_mtime)
         newest_mtime = max(old_mtimes)
+        
+        # We need load extra cache
+        if extracaches:
+            if bb.parse.cached_mtime_noerror(self.cachefile) >= newest_mtime \
+                    and bb.parse.cached_mtime_noerror(self.extra_cachefile) \
+                            >= newest_mtime:
+                self.load_cachefile()
+            elif os.path.isfile(self.cachefile):
+                logger.info("Out of date cache found, rebuilding...")
+        else:
+            if bb.parse.cached_mtime_noerror(self.cachefile) >= newest_mtime:
+                self.load_cachefile()
+            elif os.path.isfile(self.cachefile):
+                logger.info("Out of date cache found, rebuilding...") 
 
-        if bb.parse.cached_mtime_noerror(self.cachefile) >= newest_mtime:
-            self.load_cachefile()
-        elif os.path.isfile(self.cachefile):
-            logger.info("Out of date cache found, rebuilding...")
 
     def load_cachefile(self):
         with open(self.cachefile, "rb") as cachefile:
@@ -256,9 +261,36 @@ class Cache(object):
                 return
 
             cachesize = os.fstat(cachefile.fileno()).st_size
-            bb.event.fire(bb.event.CacheLoadStarted(cachesize), self.data)
 
             previous_percent = 0
+            current_progress = 0
+            extra_current_progress = 0
+
+            if self.extracaches:
+                with open(self.extra_cachefile, "rb") as extra_cachefile:
+                    cachesize += os.fstat(extra_cachefile.fileno()).st_size
+                    pickled_extra = pickle.Unpickler(extra_cachefile)
+                    bb.event.fire(
+                            bb.event.CacheLoadStarted(cachesize), self.data)
+                    # Refresh Progress Bar                    
+                    while extra_cachefile:
+                        try:
+                            key = pickled_extra.load()
+                            value = pickled_extra.load()
+                        except Exception:
+                            break
+                        self.extra_depends_cache[key] = value
+                        current_progress = extra_cachefile.tell()
+                        current_percent = 100 * current_progress / cachesize
+                        if current_percent > previous_percent:
+                            previous_percent = current_percent
+                            bb.event.fire(
+                                bb.event.CacheLoadProgress(current_progress),
+                                          self.data)
+                extra_current_progress = current_progress
+            else:
+                bb.event.fire(bb.event.CacheLoadStarted(cachesize), self.data)
+
             while cachefile:
                 try:
                     key = pickled.load()
@@ -269,13 +301,15 @@ class Cache(object):
                 self.depends_cache[key] = value
 
                 # only fire events on even percentage boundaries
-                current_progress = cachefile.tell()
+                current_progress = cachefile.tell() + extra_current_progress
                 current_percent = 100 * current_progress / cachesize
                 if current_percent > previous_percent:
                     previous_percent = current_percent
                     bb.event.fire(bb.event.CacheLoadProgress(current_progress),
                                   self.data)
 
+            # printed cache loaded entry size need no increase since extra fields
+            # belong to the same one entry
             bb.event.fire(bb.event.CacheLoadCompleted(cachesize,
                                                       len(self.depends_cache)),
                           self.data)
@@ -321,6 +355,7 @@ class Cache(object):
     def parse(cls, filename, appends, configdata, extracaches=None):
         """Parse the specified filename, returning the recipe information"""
         infos = []
+        extra_infos = []
         datastores = cls.load_bbfile(filename, appends, configdata)
         depends = set()
         for variant, data in sorted(datastores.iteritems(),
@@ -332,7 +367,11 @@ class Cache(object):
                 data.setVar("__depends", depends)
             info = RecipeInfo.from_metadata(filename, data)
             infos.append((virtualfn, info))
-        return infos
+            if extracaches:
+                extra_info = \
+                    ExtraRecipeInfoFactory.from_metadata(extracaches, data)
+                extra_infos.append((virtualfn, extra_info))
+        return infos, extra_infos
 
     def load(self, filename, appends, configdata):
         """Obtain the recipe information for the specified filename,
@@ -345,15 +384,20 @@ class Cache(object):
         cached = self.cacheValid(filename)
         if cached:
             infos = []
+            extra_infos= []
             info = self.depends_cache[filename]
-            for variant in info.variants:
+            for i in range(0, len(info.variants)):
+                variant = info.variants[i]
                 virtualfn = self.realfn2virtual(filename, variant)
                 infos.append((virtualfn, self.depends_cache[virtualfn]))
+                if self.extracaches:
+                    extra_infos.append((virtualfn, \
+                            self.extra_depends_cache[virtualfn]))
         else:
             logger.debug(1, "Parsing %s", filename)
-            return self.parse(filename, appends, configdata)
+            return self.parse(filename, appends, configdata, self.extracaches)
 
-        return cached, infos
+        return cached, infos, extra_infos
 
     def loadData(self, fn, appends, cfgData, cacheData):
         """Load the recipe info for the specified filename,
@@ -361,13 +405,17 @@ class Cache(object):
         the recipe information to the supplied CacheData instance."""
         skipped, virtuals = 0, 0
 
-        cached, infos = self.load(fn, appends, cfgData)
-        for virtualfn, info in infos:
+        cached, infos, extra_infos = self.load(fn, appends, cfgData)
+        for i in range (0, len(infos)):
+            (virtualfn, info) = infos[i]
+            if self.extracaches:
+                extra_info = extra_infos[i]
             if info.skipped:
                 logger.debug(1, "Skipping %s", virtualfn)
                 skipped += 1
             else:
-                self.add_info(virtualfn, info, cacheData, not cached)
+                self.add_info(virtualfn, info, extra_info, \
+                            cacheData, not cached)
                 virtuals += 1
 
         return cached, skipped, virtuals
@@ -399,7 +447,8 @@ class Cache(object):
         self.checked.add(fn)
 
         # File isn't in depends_cache
-        if not fn in self.depends_cache:
+        if (not fn in self.depends_cache) or (self.extracaches \
+                       and (not fn in self.extra_depends_cache)):
             logger.debug(2, "Cache: %s is not cached", fn)
             return False
 
@@ -440,7 +489,9 @@ class Cache(object):
         for cls in info.variants:
             virtualfn = self.realfn2virtual(fn, cls)
             self.clean.add(virtualfn)
-            if virtualfn not in self.depends_cache:
+            if (virtualfn not in self.depends_cache) or \
+                         (self.extracaches \
+                         and virtualfn not in self.extra_depends_cache):
                 logger.debug(2, "Cache: %s is not cached", virtualfn)
                 invalid = True
 
@@ -467,6 +518,12 @@ class Cache(object):
         if fn in self.depends_cache:
             logger.debug(1, "Removing %s from cache", fn)
             del self.depends_cache[fn]
+        # when deleting, we need to maintain the consistency
+        # of the two independent cache
+        if self.extracaches and fn in self.extra_depends_cache:
+            logger.debug(1, "Removing %s from ui_extra_cache", fn)
+            del self.extra_depends_cache[fn]
+
         if fn in self.clean:
             logger.debug(1, "Marking %s as unclean", fn)
             self.clean.remove(fn)
@@ -492,15 +549,26 @@ class Cache(object):
                 pickler.dump(key)
                 pickler.dump(value)
 
+        # Sync back the extra cache fields into the separate cache file
+        if self.extracaches:
+            with open(self.extra_cachefile, "wb") as extra_cachefile:
+                extra_pickler = pickle.Pickler(extra_cachefile, \
+                                               pickle.HIGHEST_PROTOCOL)
+                for key, value in self.extra_depends_cache.iteritems():
+                    extra_pickler.dump(key)
+                    extra_pickler.dump(value)
+
         del self.depends_cache
+        del self.extra_depends_cache
 
     @staticmethod
     def mtime(cachefile):
         return bb.parse.cached_mtime_noerror(cachefile)
 
-    def add_info(self, filename, info, cacheData, parsed=None):
+    def add_info(self, filename, info, extra_info, cacheData, parsed=None):
         if not info.skipped:
             cacheData.add_from_recipeinfo(filename, info)
+            cacheData.add_from_extra_recipeinfo(filename, extra_info)
 
         if not self.has_cache:
             return
@@ -509,15 +577,20 @@ class Cache(object):
             if parsed:
                 self.cacheclean = False
             self.depends_cache[filename] = info
+            if self.extracaches:
+                self.extra_depends_cache[filename] = extra_info
 
     def add(self, file_name, data, cacheData, parsed=None):
         """
         Save data we need into the cache
         """
-
         realfn = self.virtualfn2realfn(file_name)[0]
+        extra_info = None
         info = RecipeInfo.from_metadata(realfn, data)
-        self.add_info(file_name, info, cacheData, parsed)
+        if self.extracaches:
+            extra_info = \
+                ExtraRecipeInfoFactory.from_metadata(self.extracaches, data)
+        self.add_info(file_name, info, extra_info, cacheData, parsed)
 
     @staticmethod
     def load_bbfile(bbfile, appends, config):
@@ -609,11 +682,13 @@ class CacheData(object):
         self.basetaskhash = {}
         self.hashfn = {}
         self.inherits = {}
+        self.fakerootenv = {}
+        self.fakerootdirs = {}
+
+# Extra cache fields
         self.summary = {}
         self.license = {}
         self.section = {}
-        self.fakerootenv = {}
-        self.fakerootdirs = {}
 
         # Indirect Cache variables (set elsewhere)
         self.ignored_dependencies = []
@@ -621,6 +696,12 @@ class CacheData(object):
         self.bbfile_priority = {}
         self.bbfile_config_priorities = []
 
+    def add_from_extra_recipeinfo(self, fn, extra_info):
+        if self.extracaches:
+            self.summary[fn] = extra_info.summary
+            self.license[fn] = extra_info.license
+            self.section[fn] = extra_info.section
+
     def add_from_recipeinfo(self, fn, info):
         self.task_deps[fn] = info.task_deps
         self.pkg_fn[fn] = info.pn
@@ -679,8 +760,5 @@ class CacheData(object):
             self.basetaskhash[identifier] = taskhash
 
         self.inherits[fn] = info.inherits
-        self.summary[fn] = info.summary
-        self.license[fn] = info.license
-        self.section[fn] = info.section
         self.fakerootenv[fn] = info.fakerootenv
         self.fakerootdirs[fn] = info.fakerootdirs
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index eaf5923..58d0e82 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -718,7 +718,7 @@ class BBCooker:
         self.buildSetVars()
 
         self.status = bb.cache.CacheData(self.extracaches)
-        infos = bb.cache.Cache.parse(fn, self.get_file_appends(fn), \
+        infos, extra_infos = bb.cache.Cache.parse(fn, self.get_file_appends(fn), \
                                      self.configuration.data, \
                                      self.extracaches)
         infos = dict(infos)
@@ -730,6 +730,13 @@ class BBCooker:
             bb.fatal("%s does not exist" % fn)
         self.status.add_from_recipeinfo(fn, maininfo)
 
+        if self.extracaches:
+            try:
+                extra_info = extra_infos[fn]
+            except KeyError:
+                bb.fatal("%s does not exist" % fn)
+            self.status.add_from_extra_recipeinfo(fn, extra_info)
+
         # Tweak some variables
         item = maininfo.pn
         self.status.ignored_dependencies = set()
@@ -1071,9 +1078,9 @@ class ParsingFailure(Exception):
         self.args = (realexception, recipe)
 
 def parse_file(task):
-    filename, appends = task
+    filename, appends, extracaches = task
     try:
-        return True, bb.cache.Cache.parse(filename, appends, parse_file.cfg)
+        return True, bb.cache.Cache.parse(filename, appends, parse_file.cfg, extracaches)
     except Exception, exc:
         exc.recipe = filename
         raise exc
@@ -1087,6 +1094,7 @@ class CookerParser(object):
     def __init__(self, cooker, filelist, masked):
         self.filelist = filelist
         self.cooker = cooker
+        self.extracaches = cooker.configuration.ui
         self.cfgdata = cooker.configuration.data
 
         # Accounting statistics
@@ -1103,13 +1111,13 @@ class CookerParser(object):
         self.num_processes = int(self.cfgdata.getVar("BB_NUMBER_PARSE_THREADS", True) or
                                  multiprocessing.cpu_count())
 
-        self.bb_cache = bb.cache.Cache(self.cfgdata)
+        self.bb_cache = bb.cache.Cache(self.cfgdata,self.extracaches)
         self.fromcache = []
         self.willparse = []
         for filename in self.filelist:
             appends = self.cooker.get_file_appends(filename)
             if not self.bb_cache.cacheValid(filename):
-                self.willparse.append((filename, appends))
+                self.willparse.append((filename, appends, self.extracaches))
             else:
                 self.fromcache.append((filename, appends))
         self.toparse = self.total - len(self.fromcache)
@@ -1148,12 +1156,12 @@ class CookerParser(object):
 
     def load_cached(self):
         for filename, appends in self.fromcache:
-            cached, infos = self.bb_cache.load(filename, appends, self.cfgdata)
-            yield not cached, infos
+            cached, infos, ui_infos = self.bb_cache.load(filename, appends, self.cfgdata)
+            yield not cached, (infos, ui_infos)
 
     def parse_next(self):
         try:
-            parsed, result = self.results.next()
+            parsed, (result, extra_result) = self.results.next()
         except StopIteration:
             self.shutdown()
             return False
@@ -1174,16 +1182,26 @@ class CookerParser(object):
         else:
             self.cached += 1
 
-        for virtualfn, info in result:
+        extra_info = None
+        for i in range (0, len(result)):
+            (virtualfn, info) = result[i]
+            if self.extracaches:
+                (extra_virtualfn, extra_info) = extra_result[i]
+                if (virtualfn != extra_virtualfn):
+                    raise Exception("Inconsistancy happens for extra cache!")
             if info.skipped:
                 self.skipped += 1
-            self.bb_cache.add_info(virtualfn, info, self.cooker.status,
-                                        parsed=parsed)
+            self.bb_cache.add_info(virtualfn, info, extra_info, self.cooker.status,
+                                         parsed=parsed)
         return True
 
     def reparse(self, filename):
-        infos = self.bb_cache.parse(filename,
+        infos, extra_infos = self.bb_cache.parse(filename,
                                     self.cooker.get_file_appends(filename),
-                                    self.cfgdata)
+                                    self.cfgdata, self.extracaches)
         for vfn, info in infos:
             self.cooker.status.add_from_recipeinfo(vfn, info)
+        if self.extracaches:
+            for vfn,extra_info in extra_infos:
+                self.status.add_from_extra_recipeinfo(vfn, extra_info)
+            
-- 
1.7.0.4

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Ke, Liping]

Hi, Richard
I resend the patch and corrected some problems including:

1) name bitbake_mode -> extracaches
2) move extra cache definition to a different files

And I test against cs (Add files missing from previous change, 	bc367526f85ec11f82826fa5ce6607b9e73f9273)

And I did meet the problem of EOF of codeparser.py. But I found you have a new changeset for it, so it should be ok.

And I found the mail sequence is not correct (patch 2/4 is the first one...). 
I verified patch 1~4 be applied sequentially, each patch will not break any previous code base:)

As for multiple bitbake_mode support, if you have any input or suggestions, just let me know. I will make the patch for it.
The reason I did not make it right away is that it will cause extra_cache array related code expanding(time stamping judging, several place etc).
I want to have a confirmation for the current patches and then make further improvement.

Thanks a lot for your help!
criping


> -----Original Message-----
> From: yocto-bounces@yoctoproject.org
> [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Liping Ke
> Sent: Friday, May 13, 2011 9:50 AM
> To: yocto@yoctoproject.org
> Subject: [yocto] [PATCH 0/4][Image Creator]Put extra requested fields into
> different cache files
> 
> From: Liping Ke <liping.ke@intel.com>
> 
> Below four patches are for putting extra requested fields inito different
> cache files instead of using only on bb_cache.dat. Now image creator need
> extra three fields. And in the future, there might be more similar requests.
> For each extra requestor, we will save the requested fields data into a
> separate cache files so that those who don't need it will not be impacted
> with larger fields and large data files.
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: lke/cache_impl
>   Browse:
> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/cache_impl
> 
> Thanks,
>     Liping Ke <liping.ke@intel.com>
> ---
> 
> 
> Liping Ke (4):
>   This patch introduce new param bitbake_mode into Cache.py.
>   This patch splits Cache Data Retrieve method from Data Fields.
>   This patch introduces Extra required fields for image creator.
>   This patch implements independent cache for Extra Cache Fields
>     Request
> 
>  bitbake/lib/bb/cache.py       |  161
> ++++++++++++++++++++++++++++++++---------
>  bitbake/lib/bb/cooker.py      |   55 ++++++++++----
>  bitbake/lib/bb/extra_cache.py |   29 ++++++++
>  3 files changed, 196 insertions(+), 49 deletions(-)
>  create mode 100644 bitbake/lib/bb/extra_cache.py
> 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Richard Purdie]

Hi Liping,

On Fri, 2011-05-13 at 09:50 +0800, Liping Ke wrote:
> From: Liping Ke <liping.ke@intel.com>
> 
> Below four patches are for putting extra requested fields inito different
> cache files instead of using only on bb_cache.dat. Now image creator need
> extra three fields. And in the future, there might be more similar requests.
> For each extra requestor, we will save the requested fields data into a
> separate cache files so that those who don't need it will not be impacted
> with larger fields and large data files.
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: lke/cache_impl
>   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/cache_impl
> 
> Thanks,
>     Liping Ke <liping.ke@intel.com>
> ---
> 
> 
> Liping Ke (4):
>   This patch introduce new param bitbake_mode into Cache.py.
>   This patch splits Cache Data Retrieve method from Data Fields.
>   This patch introduces Extra required fields for image creator.
>   This patch implements independent cache for Extra Cache Fields
>     Request

These patches are good and the feedback and explanation here is to
illustrate the bigger picture rather than any criticism of these
patches. I'm hoping to explain the alterations I think this codebase
needs in order to allow bitbake to expand and grow which is our main
objective.

From an overall design standpoint, the place I want to get to is where
we can arbitrarily extend and add bitbake UIs without needing to alter
the core cache code or functionality. To do this we need a clean split
between the cache handling code and the data contained within it. I know
Chris has made changes here moving us toward that but we need to
complete them and remove the detail which I know bugs both Chris and
myself which is that the data is referenced in two places within this
code (RecipeInfo and CacheData).

The place I want to get to is therefore where we have something like an
overall set of cache definitions as classes like:

BaseRecipeInfo
HobRecipeInfo
NewUIRecipeInfo

and each would have a name which would trigger their use (e.g. "base",
"hob" and "newui") through the extracaches variable as in your code.

Your code actually follows the model we need to adopt but I'd make the
following name changes:

RecipeInfo -> BaseRecipeInfo
RecipeRetrieve -> RecipeInfo
HobExtraRecipeInfo -> HobRecipeInfo

RecipeInfo would be a base class which BaseRecipeInfo, HobRecipeInfo and
NewUIRecipeInfo would derive from. The definition would be something
like the existing RecipeInfo but there would be some entries in the
class that any user would be expected to provide. I tried to provide an
example of this and I ran into issues due to the use of namedtuple. I've
talked to Chris and if that usage is going to hold us from cleaning some
of this up, we agreed that we could use something else like a dict. The
trouble is the field list of named tuple needs to be determined in
advance of the class creation where as we need something we can change
within the class itself easily.

RecipeInfo would look something like:

class RecipeInfo(object):

    name = "Override me!"
    cachefile = "bb_extracache_" + self.name + ".dat"

    @classmethod
    def listvar(cls, var, metadata):
        return cls.getvar(var, metadata).split()

    @classmethod
    def intvar(cls, var, metadata):
        return int(cls.getvar(var, metadata) or 0)

    @classmethod
    def depvar(cls, var, metadata):
        return bb.utils.explode_deps(cls.getvar(var, metadata))

    @classmethod
    def pkgvar(cls, var, packages, metadata):
        return dict((pkg, cls.depvar("%s_%s" % (var, pkg), metadata))
                    for pkg in packages)

    @classmethod
    def taskvar(cls, var, tasks, metadata):
        return dict((task, cls.getvar("%s_task-%s" % (var, task), metadata))
                    for task in tasks)

    @classmethod
    def flaglist(cls, flag, varlist, metadata):
        return dict((var, metadata.getVarFlag(var, flag, True))
                    for var in varlist)

BaseRecipeInfo would look something like:

class BaseRecipeInfo(RecipeInfo):

    name = "base"
    cachefile = "bb_cache.dat"

    @classmethod
    def cachedata_init(cls, cachedata):
        cachedata.task_deps = {}
        cachedata.pkg_pn = defaultdict(list)
        cachedata.pkg_fn = {}
        cachedata.pkg_pepvpr = {}
        cachedata.pkg_dp = {}
        [...]

    def __init__(self, filename, metadata):

        self.file_depends = metadata.getVar('__depends', False)
        self.timestamp    = bb.parse.cached_mtime(filename)
        self.variants     = self.listvar('__VARIANTS', metadata) + ['']
        self.nocache      = self.getvar('__BB_DONT_CACHE', metadata)

        if self.getvar('__SKIPPED', metadata):
            self.skipped = True
            return

        self.tasks = metadata.getVar('__BBTASKS', False)

        self.pn = self.getvar('PN', metadata)
        packages = self.listvar('PACKAGES', metadata)
        if not self.pn in packages:
            packages.append(self.pn)

        self.basetaskhashes   = self.taskvar('BB_BASEHASH', self.tasks, metadata)
        self.hashfilename     = self.getvar('BB_HASHFILENAME', metadata)

        self.file_depends     = metadata.getVar('__depends', False)
        self.task_deps        = metadata.getVar('_task_deps', False) or {'tasks': [], 'parents': {}}
        self.variants         = self.listvar('__VARIANTS', metadata) + ['']

        self.skipped          = False
        self.timestamp        = bb.parse.cached_mtime(filename)
        self.packages         = self.listvar('PACKAGES', metadata)

        self.pe               = self.getvar('PE', metadata)
        self.pv               = self.getvar('PV', metadata)
        self.pr               = self.getvar('PR', metadata)
        self.defaultpref      = self.intvar('DEFAULT_PREFERENCE', metadata)
        self.broken           = self.getvar('BROKEN', metadata)
        self.not_world        = self.getvar('EXCLUDE_FROM_WORLD', metadata)
        self.stamp            = self.getvar('STAMP', metadata)
        self.stamp_extrainfo  = self.flaglist('stamp-extra-info', self.tasks, metadata)
        self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC', metadata)
        self.depends          = self.depvar('DEPENDS', metadata)
        self.provides         = self.depvar('PROVIDES', metadata)
        self.rdepends         = self.depvar('RDEPENDS', metadata)
        self.rprovides        = self.depvar('RPROVIDES', metadata)
        self.rrecommends      = self.depvar('RRECOMMENDS', metadata)
        self.rprovides_pkg    = self.pkgvar('RPROVIDES', packages, metadata)
        self.rdepends_pkg     = self.pkgvar('RDEPENDS', packages, metadata)
        self.rrecommends_pkg  = self.pkgvar('RRECOMMENDS', packages, metadata)
        self.inherits         = self.getvar('__inherit_cache', metadata)
        self.summary          = self.getvar('SUMMARY', metadata)
        self.license          = self.getvar('LICENSE', metadata)
        self.section          = self.getvar('SECTION', metadata)
        self.fakerootenv      = self.getvar('FAKEROOTENV', metadata)
        self.fakerootdirs     = self.getvar('FAKEROOTDIRS', metadata)

    def add_cachedata(self, cachedata, fn):
        cachedata.task_deps[fn] = self.task_deps
        cachedata.pkg_fn[fn] = self.pn
        cachedata.pkg_pn[self.pn].append(fn)
        cachedata.pkg_pepvpr[fn] = (self.pe, self.pv, self.pr)
        [...]

HobRecipeInfo would be similar but with its specific field requirements.

Once we've made this split, the code that handles the caches can become
simpler. In each place RecipeInfo is used, we can turn it into an
iterator which iterates over a list of caches. In the default case this
list would just be one (BaseRecipeInfo). With Hob enabled it would be
two (BaseRecipeInfo and HobRecipeInfo). It would equally handle ten
different entries, it would just iterate over them. We'd always require
the BaseRecipeInfo to be present in this list.

In CacheData, we'd take the list of caches and call into the
cachedata_init() and add_cachedata() hooks. We'd need a namespace
convention to ensure that whilst they write to the same CacheData object
there is a prefix to ensure the data is contained.

How do we turn a list of names of caches passed as extracaches into a
list of objects? We can use code like in runqueue.py for handling the
schedulers:

        schedulers = set(obj for obj in globals().values()
                             if type(obj) is type and
                                issubclass(obj, RunQueueScheduler))

so by checking for anything subclassing RecipeInfo we'd have a list of
cache classes to choose from.


Liping: Does this make sense? Is there anything I need to clarify? Do
you want to work on this further from here or do you want me to take
this further?

Cheers,

Richard

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Ke, Liping]

Hi, Richard

Thanks for the detailed design guide. I will sync up with Jessica and redo the patch.

And thanks for advice of the namedtuple usage. During the implementation,
we did tried several ways of inheritance refactory, but all failed with nametuple static attributes.
And when trying to change it to dict, I found too much code need to be modified, and I was very upset and gave up...

I am now basically clear about your comments. If meeting detailed problem when implementation, I will let you know.

And Jessica, so could you make some schedule adjustment? I guess I need some time to finish this refactory.

Thanks& Regards,
criping


> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> Sent: Thursday, May 19, 2011 8:10 AM
> To: Ke, Liping
> Cc: yocto@yoctoproject.org
> Subject: Re: [yocto] [PATCH 0/4][Image Creator]Put extra requested fields into
> different cache files
> 
> Hi Liping,
> 
> On Fri, 2011-05-13 at 09:50 +0800, Liping Ke wrote:
> > From: Liping Ke <liping.ke@intel.com>
> >
> > Below four patches are for putting extra requested fields inito different
> > cache files instead of using only on bb_cache.dat. Now image creator need
> > extra three fields. And in the future, there might be more similar requests.
> > For each extra requestor, we will save the requested fields data into a
> > separate cache files so that those who don't need it will not be impacted
> > with larger fields and large data files.
> >
> > Pull URL: git://git.pokylinux.org/poky-contrib.git
> >   Branch: lke/cache_impl
> >   Browse:
> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/cache_impl
> >
> > Thanks,
> >     Liping Ke <liping.ke@intel.com>
> > ---
> >
> >
> > Liping Ke (4):
> >   This patch introduce new param bitbake_mode into Cache.py.
> >   This patch splits Cache Data Retrieve method from Data Fields.
> >   This patch introduces Extra required fields for image creator.
> >   This patch implements independent cache for Extra Cache Fields
> >     Request
> 
> These patches are good and the feedback and explanation here is to
> illustrate the bigger picture rather than any criticism of these
> patches. I'm hoping to explain the alterations I think this codebase
> needs in order to allow bitbake to expand and grow which is our main
> objective.
> 
> From an overall design standpoint, the place I want to get to is where
> we can arbitrarily extend and add bitbake UIs without needing to alter
> the core cache code or functionality. To do this we need a clean split
> between the cache handling code and the data contained within it. I know
> Chris has made changes here moving us toward that but we need to
> complete them and remove the detail which I know bugs both Chris and
> myself which is that the data is referenced in two places within this
> code (RecipeInfo and CacheData).
> 
> The place I want to get to is therefore where we have something like an
> overall set of cache definitions as classes like:
> 
> BaseRecipeInfo
> HobRecipeInfo
> NewUIRecipeInfo
> 
> and each would have a name which would trigger their use (e.g. "base",
> "hob" and "newui") through the extracaches variable as in your code.
> 
> Your code actually follows the model we need to adopt but I'd make the
> following name changes:
> 
> RecipeInfo -> BaseRecipeInfo
> RecipeRetrieve -> RecipeInfo
> HobExtraRecipeInfo -> HobRecipeInfo
> 
> RecipeInfo would be a base class which BaseRecipeInfo, HobRecipeInfo and
> NewUIRecipeInfo would derive from. The definition would be something
> like the existing RecipeInfo but there would be some entries in the
> class that any user would be expected to provide. I tried to provide an
> example of this and I ran into issues due to the use of namedtuple. I've
> talked to Chris and if that usage is going to hold us from cleaning some
> of this up, we agreed that we could use something else like a dict. The
> trouble is the field list of named tuple needs to be determined in
> advance of the class creation where as we need something we can change
> within the class itself easily.
> 
> RecipeInfo would look something like:
> 
> class RecipeInfo(object):
> 
>     name = "Override me!"
>     cachefile = "bb_extracache_" + self.name + ".dat"
> 
>     @classmethod
>     def listvar(cls, var, metadata):
>         return cls.getvar(var, metadata).split()
> 
>     @classmethod
>     def intvar(cls, var, metadata):
>         return int(cls.getvar(var, metadata) or 0)
> 
>     @classmethod
>     def depvar(cls, var, metadata):
>         return bb.utils.explode_deps(cls.getvar(var, metadata))
> 
>     @classmethod
>     def pkgvar(cls, var, packages, metadata):
>         return dict((pkg, cls.depvar("%s_%s" % (var, pkg), metadata))
>                     for pkg in packages)
> 
>     @classmethod
>     def taskvar(cls, var, tasks, metadata):
>         return dict((task, cls.getvar("%s_task-%s" % (var, task), metadata))
>                     for task in tasks)
> 
>     @classmethod
>     def flaglist(cls, flag, varlist, metadata):
>         return dict((var, metadata.getVarFlag(var, flag, True))
>                     for var in varlist)
> 
> BaseRecipeInfo would look something like:
> 
> class BaseRecipeInfo(RecipeInfo):
> 
>     name = "base"
>     cachefile = "bb_cache.dat"
> 
>     @classmethod
>     def cachedata_init(cls, cachedata):
>         cachedata.task_deps = {}
>         cachedata.pkg_pn = defaultdict(list)
>         cachedata.pkg_fn = {}
>         cachedata.pkg_pepvpr = {}
>         cachedata.pkg_dp = {}
>         [...]
> 
>     def __init__(self, filename, metadata):
> 
>         self.file_depends = metadata.getVar('__depends', False)
>         self.timestamp    = bb.parse.cached_mtime(filename)
>         self.variants     = self.listvar('__VARIANTS', metadata) + ['']
>         self.nocache      = self.getvar('__BB_DONT_CACHE', metadata)
> 
>         if self.getvar('__SKIPPED', metadata):
>             self.skipped = True
>             return
> 
>         self.tasks = metadata.getVar('__BBTASKS', False)
> 
>         self.pn = self.getvar('PN', metadata)
>         packages = self.listvar('PACKAGES', metadata)
>         if not self.pn in packages:
>             packages.append(self.pn)
> 
>         self.basetaskhashes   = self.taskvar('BB_BASEHASH', self.tasks,
> metadata)
>         self.hashfilename     = self.getvar('BB_HASHFILENAME',
> metadata)
> 
>         self.file_depends     = metadata.getVar('__depends', False)
>         self.task_deps        = metadata.getVar('_task_deps', False) or
> {'tasks': [], 'parents': {}}
>         self.variants         = self.listvar('__VARIANTS', metadata) + ['']
> 
>         self.skipped          = False
>         self.timestamp        = bb.parse.cached_mtime(filename)
>         self.packages         = self.listvar('PACKAGES', metadata)
> 
>         self.pe               = self.getvar('PE', metadata)
>         self.pv               = self.getvar('PV', metadata)
>         self.pr               = self.getvar('PR', metadata)
>         self.defaultpref      = self.intvar('DEFAULT_PREFERENCE',
> metadata)
>         self.broken           = self.getvar('BROKEN', metadata)
>         self.not_world        = self.getvar('EXCLUDE_FROM_WORLD',
> metadata)
>         self.stamp            = self.getvar('STAMP', metadata)
>         self.stamp_extrainfo  = self.flaglist('stamp-extra-info', self.tasks,
> metadata)
>         self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC',
> metadata)
>         self.depends          = self.depvar('DEPENDS', metadata)
>         self.provides         = self.depvar('PROVIDES', metadata)
>         self.rdepends         = self.depvar('RDEPENDS', metadata)
>         self.rprovides        = self.depvar('RPROVIDES', metadata)
>         self.rrecommends      = self.depvar('RRECOMMENDS', metadata)
>         self.rprovides_pkg    = self.pkgvar('RPROVIDES', packages,
> metadata)
>         self.rdepends_pkg     = self.pkgvar('RDEPENDS', packages,
> metadata)
>         self.rrecommends_pkg  = self.pkgvar('RRECOMMENDS', packages,
> metadata)
>         self.inherits         = self.getvar('__inherit_cache', metadata)
>         self.summary          = self.getvar('SUMMARY', metadata)
>         self.license          = self.getvar('LICENSE', metadata)
>         self.section          = self.getvar('SECTION', metadata)
>         self.fakerootenv      = self.getvar('FAKEROOTENV', metadata)
>         self.fakerootdirs     = self.getvar('FAKEROOTDIRS', metadata)
> 
>     def add_cachedata(self, cachedata, fn):
>         cachedata.task_deps[fn] = self.task_deps
>         cachedata.pkg_fn[fn] = self.pn
>         cachedata.pkg_pn[self.pn].append(fn)
>         cachedata.pkg_pepvpr[fn] = (self.pe, self.pv, self.pr)
>         [...]
> 
> HobRecipeInfo would be similar but with its specific field requirements.
> 
> Once we've made this split, the code that handles the caches can become
> simpler. In each place RecipeInfo is used, we can turn it into an
> iterator which iterates over a list of caches. In the default case this
> list would just be one (BaseRecipeInfo). With Hob enabled it would be
> two (BaseRecipeInfo and HobRecipeInfo). It would equally handle ten
> different entries, it would just iterate over them. We'd always require
> the BaseRecipeInfo to be present in this list.
> 
> In CacheData, we'd take the list of caches and call into the
> cachedata_init() and add_cachedata() hooks. We'd need a namespace
> convention to ensure that whilst they write to the same CacheData object
> there is a prefix to ensure the data is contained.
> 
> How do we turn a list of names of caches passed as extracaches into a
> list of objects? We can use code like in runqueue.py for handling the
> schedulers:
> 
>         schedulers = set(obj for obj in globals().values()
>                              if type(obj) is type and
>                                 issubclass(obj, RunQueueScheduler))
> 
> so by checking for anything subclassing RecipeInfo we'd have a list of
> cache classes to choose from.
> 
> 
> Liping: Does this make sense? Is there anything I need to clarify? Do
> you want to work on this further from here or do you want me to take
> this further?
> 
> Cheers,
> 
> Richard
> 
> 
> 

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Richard Purdie]

On Thu, 2011-05-19 at 09:40 +0800, Ke, Liping wrote:
> Thanks for the detailed design guide. I will sync up with Jessica and redo the patch.
> 
> And thanks for advice of the namedtuple usage. During the implementation,
> we did tried several ways of inheritance refactory, but all failed with nametuple static attributes.
> And when trying to change it to dict, I found too much code need to be modified, and I was very upset and gave up...

For reference its taken me a lot longer than I expected to be able to
reply to this with code ideas that I'm happy should work for this
implementation and let this code function in a way I'm happy about from
an overall architecture perspective. Its a tricky problem.

> I am now basically clear about your comments. If meeting detailed problem when implementation, I will let you know.
> 
> And Jessica, so could you make some schedule adjustment? I guess I need some time to finish this refactory.

One other thing we need to keep in mind memory usage. The previous
implementation was designed to be light on memory use and we need to try
and ensure this one is too. I've not looked at numbers for this at all
at this point. This is particularly true for OpenEmbedded which has a
lot more recipes that we have in OE-Core.

Cheers,

Richard

[PATCH 0/4] Put extra requested fields into different cache files

[Liping Ke]

From: Liping Ke <liping.ke@intel.com>

Below four patches are for putting extra requested fields inito different
cache files instead of using only on bb_cache.dat. Now image creator need
extra three fields. And in the future, there might be more similar requests.
For each extra requestor, we will save the requested fields data into a
separate cache files so that those who don't need it will not be impacted
with larger fields and large data files.

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: lke/cache_impl
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/cache_impl

Thanks,
    Liping Ke <liping.ke@intel.com>
---


Liping Ke (4):
  Introduce new param bitbake_mode into Cache.py
  Split Cache Data Retrieve method from Data Fields
  Introduce Extra required fields for image creator
  Implement independent cache for Extra Cache Fields Request

 bitbake/lib/bb/cache.py  |  189 +++++++++++++++++++++++++++++++++++++---------
 bitbake/lib/bb/cooker.py |   55 ++++++++++----
 2 files changed, 193 insertions(+), 51 deletions(-)

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Ke, Liping]

Hi, all

Send twice with the modified subjects since those patches are introduced for image creator backend.

Thanks a lot!
criping

> -----Original Message-----
> From: poky-bounces@yoctoproject.org
> [mailto:poky-bounces@yoctoproject.org] On Behalf Of Liping Ke
> Sent: Monday, May 09, 2011 2:40 PM
> To: poky@yoctoproject.org
> Subject: [poky] [PATCH 0/4][Image Creator]Put extra requested fields into
> different cache files
> 
> From: Liping Ke <liping.ke@intel.com>
> 
> Below four patches are for putting extra requested fields inito different
> cache files instead of using only on bb_cache.dat. Now image creator need
> extra three fields. And in the future, there might be more similar requests.
> For each extra requestor, we will save the requested fields data into a
> separate cache files so that those who don't need it will not be impacted
> with larger fields and large data files.
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: lke/cache_impl
>   Browse:
> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/cache_impl
> 
> Thanks,
>     Liping Ke <liping.ke@intel.com>
> ---
> 
> 
> Liping Ke (4):
>   Introduce new param bitbake_mode into Cache.py
>   Split Cache Data Retrieve method from Data Fields
>   Introduce Extra required fields for image creator
>   Implement independent cache for Extra Cache Fields Request
> 
>  bitbake/lib/bb/cache.py  |  189
> +++++++++++++++++++++++++++++++++++++---------
>  bitbake/lib/bb/cooker.py |   55 ++++++++++----
>  2 files changed, 193 insertions(+), 51 deletions(-)
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Ke, Liping]

Hi, Josh

Those patches are rebased with latest poky master today. I noticed Richard
has changed the code of cache, making sure skipped info is saved for avoiding
unnecessary parsing, so that the cache file will remain the same size each time.
( I found before this change, cache file kept increased).
I verified that our new introduced patch working with this latest cache changes.

Thanks a lot!
criping

> -----Original Message-----
> From: Ke, Liping
> Sent: Monday, May 09, 2011 2:53 PM
> To: Ke, Liping; poky@yoctoproject.org
> Subject: RE: [poky] [PATCH 0/4][Image Creator]Put extra requested fields into
> different cache files
> 
> Hi, all
> 
> Send twice with the modified subjects since those patches are introduced for
> image creator backend.
> 
> Thanks a lot!
> criping
> 
> > -----Original Message-----
> > From: poky-bounces@yoctoproject.org
> > [mailto:poky-bounces@yoctoproject.org] On Behalf Of Liping Ke
> > Sent: Monday, May 09, 2011 2:40 PM
> > To: poky@yoctoproject.org
> > Subject: [poky] [PATCH 0/4][Image Creator]Put extra requested fields into
> > different cache files
> >
> > From: Liping Ke <liping.ke@intel.com>
> >
> > Below four patches are for putting extra requested fields inito different
> > cache files instead of using only on bb_cache.dat. Now image creator need
> > extra three fields. And in the future, there might be more similar requests.
> > For each extra requestor, we will save the requested fields data into a
> > separate cache files so that those who don't need it will not be impacted
> > with larger fields and large data files.
> >
> > Pull URL: git://git.pokylinux.org/poky-contrib.git
> >   Branch: lke/cache_impl
> >   Browse:
> > http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/cache_impl
> >
> > Thanks,
> >     Liping Ke <liping.ke@intel.com>
> > ---
> >
> >
> > Liping Ke (4):
> >   Introduce new param bitbake_mode into Cache.py
> >   Split Cache Data Retrieve method from Data Fields
> >   Introduce Extra required fields for image creator
> >   Implement independent cache for Extra Cache Fields Request
> >
> >  bitbake/lib/bb/cache.py  |  189
> > +++++++++++++++++++++++++++++++++++++---------
> >  bitbake/lib/bb/cooker.py |   55 ++++++++++----
> >  2 files changed, 193 insertions(+), 51 deletions(-)
> >
> > _______________________________________________
> > poky mailing list
> > poky@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/poky

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Richard Purdie]

Hi Liping,

On Mon, 2011-05-09 at 14:39 +0800, Liping Ke wrote:
> Below four patches are for putting extra requested fields inito different
> cache files instead of using only on bb_cache.dat. Now image creator need
> extra three fields. And in the future, there might be more similar requests.
> For each extra requestor, we will save the requested fields data into a
> separate cache files so that those who don't need it will not be impacted
> with larger fields and large data files.
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: lke/cache_impl
>   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/cache_impl
> 
> Thanks,
>     Liping Ke <liping.ke@intel.com>

These patches are good and I like the ultimate result but the patches
themselves could use some tweaks in the order they make changes. Ideally
changes need to be made in a way that each patch can stand on its own so
I could apply patches 1 and 2, leave 3 and 4 out and the code should
still work.

In this sense, the series should really be a set of patches which:

a) Adds "bitbake_mode" and allows UIs to define cache data
b) Adds the cache data to hob
c) Drops the cache data used only by hob from the core

I've also the following comments on the code in general:

a) Could we move the hob specific cache data from cache.py to hob.py. If
not, I'd like to understand why not and maybe address those issues. My
point is you shouldn't need to change cache.py to add extra cache data.

b) Could we rename "bitbake_mode" to "extracaches" or something?
"bitbake_mode" is a bit too generic and meaningless.

c) I was also wondering how hard it would be to make this an array and
allow more than one to be loaded? I'm thinking of the use case where we
may have more than one "UI" in future and configure things so the user
can switch between them without re-parsing.

Cheers,

Richard

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Ke, Liping]

Hi, Richard,

Thanks for your feedback!!! please see below answers.

criping

> These patches are good and I like the ultimate result but the patches
> themselves could use some tweaks in the order they make changes. Ideally
> changes need to be made in a way that each patch can stand on its own so
> I could apply patches 1 and 2, leave 3 and 4 out and the code should
> still work.
> 
> In this sense, the series should really be a set of patches which:
> 
> a) Adds "bitbake_mode" and allows UIs to define cache data
> b) Adds the cache data to hob
> c) Drops the cache data used only by hob from the core
Yes, in general I want to do the same thing. Each patch in the serial will
not break the current cs. Patch[1/4], patch[2/4]& patch [3/4] are for a) and b).
But for c), because of the implantation algorithm I adopted, it's merged with
extra cache data implementation (the biggest patch).

 
> I've also the following comments on the code in general:
> 
> a) Could we move the hob specific cache data from cache.py to hob.py. If
> not, I'd like to understand why not and maybe address those issues. My
> point is you shouldn't need to change cache.py to add extra cache data.

you mean that we will have a separate ExtraCache (I guess hob or adt-installer should have the similar logic?)
implementation file, which has similar functionality with cache.py? 
Oh, I did not think of this implementation at all when implement. In my mind, change less code is the principle since
this part of the code is the core of bitbake:)  If split, many codes in cache could be reused by both cache.py and hob.py?

And also, if you think it is a must, could we split the task, for this phase, keep it in the same file. 
And then refactory them in another patch later? Seems there're many tasks for image creator? Seems the schedule is
tight?

> b) Could we rename "bitbake_mode" to "extracaches" or something?
> "bitbake_mode" is a bit too generic and meaningless.
Sure. It's nice.

> c) I was also wondering how hard it would be to make this an array and
> allow more than one to be loaded? I'm thinking of the use case where we
> may have more than one "UI" in future and configure things so the user
> can switch between them without re-parsing.
> 
Hi, Richard, actually user could have more than one data file. If we have autobuilder extra requests,
we can easily extend with few codes: (just like patch 3)
1)define autobuilder_extra_fields,
2) declare autobuilder namedtuple class
3) add else in ExtraClassFactory for defining from_metadata.
Then the bb_cacheautobuilder.dat will be loaded and processed. 
Then there would be more than two cache data files (bb_cache.dat, bb_cachehob.dat, bb_cacheautobuilder.dat, etc).
But each time when bitbake is running, only two files will be loaded/saved. (bb_cache.dat, bb_cachehob.dat)
or (bb_cache.dat, bb_cacheautobuilder.dat). bitbake_mode are used for passing the parameters, 'hob' or 'autobuilder'?
Is this what you mean?

> Cheers,
> 
> Richard

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Richard Purdie]

On Tue, 2011-05-10 at 09:26 +0800, Ke, Liping wrote:
> > These patches are good and I like the ultimate result but the patches
> > themselves could use some tweaks in the order they make changes. Ideally
> > changes need to be made in a way that each patch can stand on its own so
> > I could apply patches 1 and 2, leave 3 and 4 out and the code should
> > still work.
> > 
> > In this sense, the series should really be a set of patches which:
> > 
> > a) Adds "bitbake_mode" and allows UIs to define cache data
> > b) Adds the cache data to hob
> > c) Drops the cache data used only by hob from the core
> Yes, in general I want to do the same thing. Each patch in the serial will
> not break the current cs. Patch[1/4], patch[2/4]& patch [3/4] are for a) and b).
> But for c), because of the implantation algorithm I adopted, it's merged with
> extra cache data implementation (the biggest patch).

That's fine, your original series didn't seem to work quite like this
though and looked like I'd get failures if I only applied the early
patches from what I remember.

> > I've also the following comments on the code in general:
> > 
> > a) Could we move the hob specific cache data from cache.py to hob.py. If
> > not, I'd like to understand why not and maybe address those issues. My
> > point is you shouldn't need to change cache.py to add extra cache data.
> 
> you mean that we will have a separate ExtraCache (I guess hob or adt-installer should have the similar logic?)
> implementation file, which has similar functionality with cache.py? 
> Oh, I did not think of this implementation at all when implement. In my mind, change less code is the principle since
> this part of the code is the core of bitbake:)  If split, many codes in cache could be reused by both cache.py and hob.py?

I'm thinking that cache.py defines the class but the hob specific
definition using that class to extend the cache data should be part of
the hob codebase. In the future there may be external UIs calling into
this code and needing this functionality and I'd prefer them not to each
need to change cache.py.

> And also, if you think it is a must, could we split the task, for this phase, keep it in the same file. 
> And then refactory them in another patch later? Seems there're many tasks for image creator? Seems the schedule is
> tight?

If you don't think this is realistic in the time available, I might be
able to take a look at it.

> > b) Could we rename "bitbake_mode" to "extracaches" or something?
> > "bitbake_mode" is a bit too generic and meaningless.
> Sure. It's nice.
> 
> > c) I was also wondering how hard it would be to make this an array and
> > allow more than one to be loaded? I'm thinking of the use case where we
> > may have more than one "UI" in future and configure things so the user
> > can switch between them without re-parsing.
> > 
> Hi, Richard, actually user could have more than one data file. If we have autobuilder extra requests,
> we can easily extend with few codes: (just like patch 3)
> 1)define autobuilder_extra_fields,
> 2) declare autobuilder namedtuple class
> 3) add else in ExtraClassFactory for defining from_metadata.
> Then the bb_cacheautobuilder.dat will be loaded and processed. 
> Then there would be more than two cache data files (bb_cache.dat, bb_cachehob.dat, bb_cacheautobuilder.dat, etc).
> But each time when bitbake is running, only two files will be loaded/saved. (bb_cache.dat, bb_cachehob.dat)
> or (bb_cache.dat, bb_cacheautobuilder.dat). bitbake_mode are used for passing the parameters, 'hob' or 'autobuilder'?
> Is this what you mean?

No, I mean can we could pass:

bitbake_mode = ['hob', 'autobuilder']

and it then uses three cache files so bitbake_mode would become a list
rather than a single item.

Cheers,

Richard

Re: [PATCH 0/4][Image Creator]Put extra requested fields into different cache files

[Ke, Liping]

Hi, Richard

> > >
> > > a) Adds "bitbake_mode" and allows UIs to define cache data
> > > b) Adds the cache data to hob
> > > c) Drops the cache data used only by hob from the core
> > Yes, in general I want to do the same thing. Each patch in the serial will
> > not break the current cs. Patch[1/4], patch[2/4]& patch [3/4] are for a) and
> b).
> > But for c), because of the implantation algorithm I adopted, it's merged with
> > extra cache data implementation (the biggest patch).
> 
> That's fine, your original series didn't seem to work quite like this
> though and looked like I'd get failures if I only applied the early
> patches from what I remember.

sure, I understand this. I will make sure that the series of patches works independently
when resending the patch.

> 
> > > I've also the following comments on the code in general:
> > >
> > > a) Could we move the hob specific cache data from cache.py to hob.py. If
> > > not, I'd like to understand why not and maybe address those issues. My
> > > point is you shouldn't need to change cache.py to add extra cache data.
> >
> > you mean that we will have a separate ExtraCache (I guess hob or
> adt-installer should have the similar logic?)
> > implementation file, which has similar functionality with cache.py?
> > Oh, I did not think of this implementation at all when implement. In my mind,
> change less code is the principle since
> > this part of the code is the core of bitbake:)  If split, many codes in cache
> could be reused by both cache.py and hob.py?
> 
> I'm thinking that cache.py defines the class but the hob specific
> definition using that class to extend the cache data should be part of
> the hob codebase. In the future there may be external UIs calling into
> this code and needing this functionality and I'd prefer them not to each
> need to change cache.py.

Hi, Richard, just a question here. You mean ext-hob-cache data will be a sub-class
of cachy data? Inheritance here? We originally consider this, yet just failed to find
a good way. Namedtuple has the static dict fields. So namedtuple class can't be extended.
For inheritance, I must change the original cache.py implantation. 
Just want to hear your suggestion here...

> 
> > And also, if you think it is a must, could we split the task, for this phase, keep
> it in the same file.
> > And then refactory them in another patch later? Seems there're many tasks
> for image creator? Seems the schedule is
> > tight?
> 
> If you don't think this is realistic in the time available, I might be
> able to take a look at it.

Oh, I guess Jessica will re-arrange the schedule today... Seems for the task, there're
many different understanding about the requirement at the very beginning... I will try to
understand all the details before writing the code -:)
 

> No, I mean can we could pass:
> 
> bitbake_mode = ['hob', 'autobuilder']
> 
> and it then uses three cache files so bitbake_mode would become a list
> rather than a single item.
> 
OK, I understand this requirement now.
After we sync up today, I should have a clearer understanding about the requirement and I will redo the patch!

Thanks a lot for your help!
criping


> Cheers,
> 
> Richard