[PATCH 0/4] Add reparseFile command and adopt the saved environment

[PATCH 0/4] Add reparseFile command and adopt the saved environment

[Joshua Lock]

This series adapts the cooker to use the saved OS environment instead of the
current os.environ and adds the reparseFiles command which triggered the
changes.

The following changes since commit f0b5d16426b983a67c51c47f3542162108bd4156:

  Update version to 1.13.3 (2011-07-25 13:50:38 +0100)

are available in the git repository at:
  git://github.com/incandescant/bitbake hob
  https://github.com/incandescant/bitbake/tree/hob

Joshua Lock (4):
  data|cooker: use saved environment variables when calling
    inheritFromOS
  bitbake-layers: adapt to cooker change for saved environment
  command|cooker: Add reparseFiles command
  cooker: only append files once

 bin/bitbake-layers |    7 ++++---
 lib/bb/command.py  |    8 ++++++++
 lib/bb/cooker.py   |   51 +++++++++++++++++++++++++++++++--------------------
 lib/bb/data.py     |    8 ++++----
 4 files changed, 47 insertions(+), 27 deletions(-)

-- 
1.7.6

[PATCH 1/4] data|cooker: use saved environment variables when calling inheritFromOS

[Joshua Lock]

Now that we have a pristine copy of the variables available at launch time
we can use them when looking to inherit the OS's environment.

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 lib/bb/cooker.py |    2 +-
 lib/bb/data.py   |    8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index d8535a7..1b5101d 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -132,7 +132,7 @@ class BBCooker:
 
         self.configuration.data = bb.data.init()
 
-        bb.data.inheritFromOS(self.configuration.data)
+        bb.data.inheritFromOS(self.configuration.data, self.savedenv)
 
         try:
             self.parseConfigurationFiles(self.configuration.prefile,
diff --git a/lib/bb/data.py b/lib/bb/data.py
index 2269f9d..65144bf 100644
--- a/lib/bb/data.py
+++ b/lib/bb/data.py
@@ -159,12 +159,12 @@ def expandKeys(alterdata, readdata = None):
         ekey = todolist[key]
         renameVar(key, ekey, alterdata)
 
-def inheritFromOS(d):
-    """Inherit variables from the environment."""
+def inheritFromOS(d, savedenv):
+    """Inherit variables from the initial environment."""
     exportlist = bb.utils.preserved_envvars_exported()
-    for s in os.environ.keys():
+    for s in savedenv.keys():
         try:
-            setVar(s, os.environ[s], d)
+            setVar(s, getVar(s, savedenv, True), d)
             if s in exportlist:
                 setVarFlag(s, "export", True, d)
         except TypeError:
-- 
1.7.6

[PATCH 2/4] bitbake-layers: adapt to cooker change for saved environment

[Joshua Lock]

The Cooker requires a copy of the environment mapping, modify
bitbake-layers to take one and pass it to the cooker.

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 bin/bitbake-layers |    7 ++++---
 1 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/bin/bitbake-layers b/bin/bitbake-layers
index 3d563b5..d9f95d1 100755
--- a/bin/bitbake-layers
+++ b/bin/bitbake-layers
@@ -8,7 +8,7 @@
 
 import cmd
 import logging
-import os.path
+import os
 import sys
 
 bindir = os.path.dirname(__file__)
@@ -41,11 +41,12 @@ def main(args):
 class Commands(cmd.Cmd):
     def __init__(self):
         cmd.Cmd.__init__(self)
-
+        initialenv = os.environ.copy()
         self.returncode = 0
         self.config = Config(parse_only=True)
         self.cooker = bb.cooker.BBCooker(self.config,
-                                         self.register_idle_function)
+                                         self.register_idle_function,
+                                         initialenv)
         self.config_data = self.cooker.configuration.data
         bb.providers.logger.setLevel(logging.ERROR)
         self.cooker_data = None
-- 
1.7.6

[PATCH 3/4] command|cooker: Add reparseFiles command

[Joshua Lock]

Add command reparseFiles to reparse bb files in a running cooker instance.

Fixes [YOCTO #1249]

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 lib/bb/command.py |    8 ++++++++
 lib/bb/cooker.py  |   48 +++++++++++++++++++++++++++++-------------------
 2 files changed, 37 insertions(+), 19 deletions(-)

diff --git a/lib/bb/command.py b/lib/bb/command.py
index a902da2..893a6d9 100644
--- a/lib/bb/command.py
+++ b/lib/bb/command.py
@@ -311,6 +311,14 @@ class CommandsAsync:
         command.finishAsyncCommand()
     parseFiles.needcache = True
 
+    def reparseFiles(self, command, params):
+        """
+        Reparse .bb files
+        """
+        command.cooker.reparseFiles()
+        command.finishAsyncCommand()
+    reparseFiles.needcache = True
+
     def compareRevisions(self, command, params):
         """
         Parse the .bb files
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 1b5101d..94e8d09 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -130,21 +130,8 @@ class BBCooker:
                 logger.critical("Unable to import extra RecipeInfo '%s' from '%s': %s" % (cache_name, module_name, exc))
                 sys.exit("FATAL: Failed to import extra cache class '%s'." % cache_name)
 
-        self.configuration.data = bb.data.init()
-
-        bb.data.inheritFromOS(self.configuration.data, self.savedenv)
-
-        try:
-            self.parseConfigurationFiles(self.configuration.prefile,
-                                         self.configuration.postfile)
-        except SyntaxError:
-            sys.exit(1)
-        except Exception:
-            logger.exception("Error parsing configuration files")
-            sys.exit(1)
-
-        if not self.configuration.cmd:
-            self.configuration.cmd = bb.data.getVar("BB_DEFAULT_TASK", self.configuration.data, True) or "build"
+        self.configuration.data = None
+        self.loadConfigurationData()
 
         bbpkgs = bb.data.getVar('BBPKGS', self.configuration.data, True)
         if bbpkgs and len(self.configuration.pkgs_to_build) == 0:
@@ -171,6 +158,23 @@ class BBCooker:
 
         self.parser = None
 
+    def loadConfigurationData(self):
+        self.configuration.data = bb.data.init()
+
+        bb.data.inheritFromOS(self.configuration.data, self.savedenv)
+
+        try:
+            self.parseConfigurationFiles(self.configuration.prefile,
+                                         self.configuration.postfile)
+        except SyntaxError:
+            sys.exit(1)
+        except Exception:
+            logger.exception("Error parsing configuration files")
+            sys.exit(1)
+
+        if not self.configuration.cmd:
+            self.configuration.cmd = bb.data.getVar("BB_DEFAULT_TASK", self.configuration.data, True) or "build"
+
     def parseConfiguration(self):
 
 
@@ -563,7 +567,7 @@ class BBCooker:
         bb.data.expandKeys(localdata)
 
         # Handle PREFERRED_PROVIDERS
-        for p in (bb.data.getVar('PREFERRED_PROVIDERS', localdata, 1) or "").split():
+        for p in (bb.data.getVar('PREFERRED_PROVIDERS', localdata, True) or "").split():
             try:
                 (providee, provider) = p.split(':')
             except:
@@ -1058,8 +1062,8 @@ class BBCooker:
 
         self.server_registration_cb(buildTargetsIdle, rq)
 
-    def updateCache(self):
-        if self.state == state.running:
+    def updateCache(self, force=False):
+        if self.state == state.running and not force:
             return
 
         if self.state in (state.shutdown, state.stop):
@@ -1069,6 +1073,8 @@ class BBCooker:
         if self.state != state.parsing:
             self.parseConfiguration ()
 
+            if self.status:
+                del self.status
             self.status = bb.cache.CacheData(self.caches_array)
 
             ignore = bb.data.getVar("ASSUME_PROVIDED", self.configuration.data, 1) or ""
@@ -1142,7 +1148,7 @@ class BBCooker:
 
         collectlog.debug(1, "collecting .bb files")
 
-        files = (data.getVar( "BBFILES", self.configuration.data, 1 ) or "").split()
+        files = (data.getVar( "BBFILES", self.configuration.data, True) or "").split()
         data.setVar("BBFILES", " ".join(files), self.configuration.data)
 
         # Sort files by priority
@@ -1240,6 +1246,10 @@ class BBCooker:
     def stop(self):
         self.state = state.stop
 
+    def reparseFiles(self):
+        self.loadConfigurationData()
+        self.updateCache(force=True)
+
 def server_main(cooker, func, *args):
     cooker.pre_serve()
 
-- 
1.7.6

[PATCH 4/4] cooker: only append files once

[Joshua Lock]

A list can contain the same value twice, therefore only append to the
appendlist for a recipe if the append file is not already in the list.

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 lib/bb/cooker.py |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 94e8d09..1f305d9 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -1205,7 +1205,8 @@ class BBCooker:
             base = os.path.basename(f).replace('.bbappend', '.bb')
             if not base in self.appendlist:
                self.appendlist[base] = []
-            self.appendlist[base].append(f)
+            if f not in self.appendlist[base]:
+                self.appendlist[base].append(f)
 
         # Find overlayed recipes
         # bbfiles will be in priority order which makes this easy
-- 
1.7.6

Re: [PATCH 0/4] Add reparseFile command and adopt the saved environment

[Richard Purdie]

On Mon, 2011-07-25 at 11:56 -0700, Joshua Lock wrote:
> This series adapts the cooker to use the saved OS environment instead of the
> current os.environ and adds the reparseFiles command which triggered the
> changes.
> 
> The following changes since commit f0b5d16426b983a67c51c47f3542162108bd4156:
> 
>   Update version to 1.13.3 (2011-07-25 13:50:38 +0100)
> 
> are available in the git repository at:
>   git://github.com/incandescant/bitbake hob
>   https://github.com/incandescant/bitbake/tree/hob
> 
> Joshua Lock (4):
>   data|cooker: use saved environment variables when calling
>     inheritFromOS
>   bitbake-layers: adapt to cooker change for saved environment
>   command|cooker: Add reparseFiles command
>   cooker: only append files once

Merged to master, thanks.

Richard