[PATCH 0/4] BitBake memory-resident mode fixes

[PATCH 0/4] BitBake memory-resident mode fixes

[Paul Eggleton]

Three fixes for memres and one optimisation that I noticed while working
on these.

The following changes since commit 705ab252e631903e6d2e46202b419a9e8adcd861:

  bitbake-layers: check layer dependencies before adding (2017-06-05 13:33:14 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib paule/bb-memres-fixes
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-memres-fixes

Paul Eggleton (4):
  cookerdata: fix variable history not showing in bitbake -e with memres
  cooker: fix file changes not triggering task re-execution with memres
  cooker: set config valid flags in initConfigurationData()
  cooker: add BB_CMDLINE to enable access to UI command line with memres

 lib/bb/command.py    |  3 ++-
 lib/bb/cooker.py     | 29 +++++++++++++++++++++--------
 lib/bb/cookerdata.py |  4 ++--
 3 files changed, 25 insertions(+), 11 deletions(-)

-- 
2.9.4

[PATCH 1/4] cookerdata: fix variable history not showing in bitbake -e with memres

[Paul Eggleton]

CookerConfiguration sets the "tracking" option to True when -e is
specified in order to have history tracking enabled in the datastore
so that we can show variable history (which isn't enabled by default for
performance reasons), however in memory resident mode this wasn't doing
anything because it was happening on the UI side only. We do have a
mechanism for updating the cooker configuration in the server, but the
tracking option wasn't being included in the list of options we updated,
so we just need to add this option to fix the issue.

Fixes [YOCTO #10730].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/cookerdata.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
index 722d860..6b850d3 100644
--- a/lib/bb/cookerdata.py
+++ b/lib/bb/cookerdata.py
@@ -76,7 +76,7 @@ class ConfigParameters(object):
         for o in ["abort", "tryaltconfigs", "force", "invalidate_stamp", 
                   "verbose", "debug", "dry_run", "dump_signatures", 
                   "debug_domains", "extra_assume_provided", "profile",
-                  "prefile", "postfile"]:
+                  "prefile", "postfile", "tracking"]:
             options[o] = getattr(self.options, o)
 
         ret, error = server.runCommand(["updateConfig", options, environment])
-- 
2.9.4

[PATCH 2/4] cooker: fix file changes not triggering task re-execution with memres

[Paul Eggleton]

If you build a recipe, modify one of the files referred to in SRC_URI,
then build it again, it should re-run do_fetch and the tasks that depend
upon it to incorporate the modified file. With memory resident mode this
was not working unless you restarted the server, because the mtime of
the file was cached and thus we never checked the actual file. Clear out
the mtime caches in buildTargets() to ensure this isn't an issue. I did
a minor refactoring to break this out to its own function since we were
already clearing out the build mtime cache here.

(The test I used for this was . ./oe-init-build-env-memres,
bitbake -c fetch mdadm, modify the comments at the top of one of the
patches referred to in the recipe, and then bitbake -c fetch mdadm again
and it should re-execute the fetch task ("... 0 didn't need to be
re-run".)

Fixes [YOCTO #10732].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/cooker.py | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 8846690..651b9c0 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -1313,6 +1313,18 @@ class BBCooker:
         self.data.setVar("DATE", time.strftime('%Y%m%d', t))
         self.data.setVar("TIME", time.strftime('%H%M%S', t))
 
+    def reset_mtime_caches(self):
+        """
+        Reset mtime caches - this is particularly important when memory resident as something
+        which is cached is not unlikely to have changed since the last invocation (e.g. a
+        file associated with a recipe might have been modified by the user).
+        """
+        build.reset_cache()
+        bb.fetch._checksum_cache.mtime_cache.clear()
+        siggen_cache = getattr(bb.parse.siggen, 'checksum_cache', None)
+        if siggen_cache:
+            bb.parse.siggen.checksum_cache.mtime_cache.clear()
+
     def matchFiles(self, bf):
         """
         Find the .bb files which match the expression in 'buildfile'.
@@ -1490,7 +1502,7 @@ class BBCooker:
                 return True
             return retval
 
-        build.reset_cache()
+        self.reset_mtime_caches()
         self.buildSetVars()
 
         # If we are told to do the None task then query the default task
-- 
2.9.4

[PATCH 3/4] cooker: set config valid flags in initConfigurationData()

[Paul Eggleton]

If we set these flags here then we can not only tidy up some of the code
that calls initConfigurationData() (and reset() which simply calls the
former), we also avoid calling initConfigurationData() an extra time
unnecessarily during startup (since setting baseconfig_valid = False in
updateConfigOpts() also resulted in initConfigurationData() being
called from updateCacheSync() at the end of the command.)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/cooker.py | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 651b9c0..1283136 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -213,9 +213,6 @@ class BBCooker:
 
         self.configuration.server_register_idlecallback(_process_inotify_updates, [self.confignotifier, self.notifier])
 
-        self.baseconfig_valid = True
-        self.parsecache_valid = False
-
         # Take a lock so only one copy of bitbake can run against a given build
         # directory at a time
         if not self.lockBitbake():
@@ -375,6 +372,8 @@ class BBCooker:
         self.data.renameVar("__depends", "__base_depends")
         self.add_filewatch(self.data.getVar("__base_depends", False), self.configwatcher)
 
+        self.baseconfig_valid = True
+        self.parsecache_valid = False
 
     def enableDataTracking(self):
         self.configuration.tracking = True
@@ -568,7 +567,6 @@ class BBCooker:
                 clean = False
         if not clean:
             logger.debug(1, "Base environment change, triggering reparse")
-            self.baseconfig_valid = False        
             self.reset()
 
     def runCommands(self, server, data, abort):
@@ -1623,8 +1621,6 @@ class BBCooker:
         if not self.baseconfig_valid:
             logger.debug(1, "Reloading base configuration data")
             self.initConfigurationData()
-            self.baseconfig_valid = True
-            self.parsecache_valid = False
 
     # This is called for all async commands when self.state != running
     def updateCache(self):
-- 
2.9.4

[PATCH 4/4] cooker: add BB_CMDLINE to enable access to UI command line with memres

[Paul Eggleton]

In OpenEmbedded's buildhistory class we want access to the bitbake
command line that launched the build, and up to now we were simply using
sys.argv from within the event handler to get that. Unfortunately that
doesn't work in memory resident mode, since the event handler is
naturally executing within the server and thus will give you the command
that launched the bitbake server which is much less interesting. Add a
dynamic variable BB_CMDLINE to provide access to this, set from sys.argv
within the UI process in updateToServer().

(Note that BB_CMDLINE isn't currently passed through to the worker, so
this is only really readable from event handlers plus any explicit
getVariable calls - in theory an observe-only UI could read it for
example.)

Part of the fix for [YOCTO #11634].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/command.py    | 3 ++-
 lib/bb/cooker.py     | 7 ++++++-
 lib/bb/cookerdata.py | 2 +-
 3 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/lib/bb/command.py b/lib/bb/command.py
index ab51d8e..a919f58 100644
--- a/lib/bb/command.py
+++ b/lib/bb/command.py
@@ -310,7 +310,8 @@ class CommandsSync:
     def updateConfig(self, command, params):
         options = params[0]
         environment = params[1]
-        command.cooker.updateConfigOpts(options, environment)
+        cmdline = params[2]
+        command.cooker.updateConfigOpts(options, environment, cmdline)
     updateConfig.needconfig = False
 
     def parseConfiguration(self, command, params):
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 1283136..45b5a61 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -192,6 +192,8 @@ class BBCooker:
         bb.parse.__mtime_cache = {}
         bb.parse.BBHandler.cached_statements = {}
 
+        self.ui_cmdline = None
+
         self.initConfigurationData()
 
         # we log all events to a file if so directed
@@ -360,6 +362,8 @@ class BBCooker:
         if consolelog:
             self.data.setVar("BB_CONSOLELOG", consolelog)
 
+        self.data.setVar('BB_CMDLINE', self.ui_cmdline)
+
         #
         # Copy of the data store which has been expanded.
         # Used for firing events and accessing variables where expansion needs to be accounted for
@@ -539,7 +543,8 @@ class BBCooker:
 
         self.handleCollections(self.data.getVar("BBFILE_COLLECTIONS"))
 
-    def updateConfigOpts(self, options, environment):
+    def updateConfigOpts(self, options, environment, cmdline):
+        self.ui_cmdline = cmdline
         clean = True
         for o in options:
             if o in ['prefile', 'postfile']:
diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
index 6b850d3..62aa055 100644
--- a/lib/bb/cookerdata.py
+++ b/lib/bb/cookerdata.py
@@ -79,7 +79,7 @@ class ConfigParameters(object):
                   "prefile", "postfile", "tracking"]:
             options[o] = getattr(self.options, o)
 
-        ret, error = server.runCommand(["updateConfig", options, environment])
+        ret, error = server.runCommand(["updateConfig", options, environment, sys.argv])
         if error:
                 raise Exception("Unable to update the server configuration with local parameters: %s" % error)
 
-- 
2.9.4