[PATCH 0/3] reworked environment variable whitelisting fixes

[PATCH 0/3] reworked environment variable whitelisting fixes

[Paul Eggleton]

Adjustments to variable whitelisting to fix sstate package hashing. This supersedes
the paule/sstate contrib branch.

The bitbake changes have been discussed briefly with Richard Purdie & Chris Larson.

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: paule/sstate2
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=paule/sstate2

Thanks,
    Paul Eggleton <paul.eggleton@linux.intel.com>
---


Paul Eggleton (3):
  bitbake: allow excluding dependencies of variables
  bitbake: add optional expansion to getVarFlag()
  poky.conf: adjust variable whitelisting to fix sstate checksums

 bitbake/lib/bb/data.py       |    3 ++-
 bitbake/lib/bb/data_smart.py |    9 ++++++---
 meta/conf/distro/poky.conf   |    3 ++-
 3 files changed, 10 insertions(+), 5 deletions(-)

[PATCH 1/3] bitbake: allow excluding dependencies of variables

[Paul Eggleton]

Adds a vardepsexclude flag that can be used to exclude a dependency of
a variable (the opposite of vardeps). This will allow the exclusion of
variables from the hash generation much more selectively than blanket
whitelisting using BB_HASHBASE_WHITELIST.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 bitbake/lib/bb/data.py |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
index d4d43fd..0c95ebb 100644
--- a/bitbake/lib/bb/data.py
+++ b/bitbake/lib/bb/data.py
@@ -297,6 +297,7 @@ def build_dependencies(key, keys, shelldeps, d):
             deps |= parser.references
             deps = deps | (keys & parser.execs)
         deps |= set((d.getVarFlag(key, "vardeps") or "").split())
+        deps -= set((d.getVarFlag(key, "vardepsexclude") or "").split())
     except:
         bb.note("Error expanding variable %s" % key) 
         raise
-- 
1.7.1

[PATCH 2/3] bitbake: add optional expansion to getVarFlag()

[Paul Eggleton]

Add a parameter to getVarFlag() to auto-expand the value of the flag. This
makes getVarFlag() more consistent with getVar(), and allows expansion of
vardeps and vardepsexclude (which has been done in this commit).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 bitbake/lib/bb/data.py       |    4 ++--
 bitbake/lib/bb/data_smart.py |    9 ++++++---
 2 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
index 0c95ebb..0aa8b40 100644
--- a/bitbake/lib/bb/data.py
+++ b/bitbake/lib/bb/data.py
@@ -296,8 +296,8 @@ def build_dependencies(key, keys, shelldeps, d):
             parser = d.expandWithRefs(d.getVar(key, False), key)
             deps |= parser.references
             deps = deps | (keys & parser.execs)
-        deps |= set((d.getVarFlag(key, "vardeps") or "").split())
-        deps -= set((d.getVarFlag(key, "vardepsexclude") or "").split())
+        deps |= set((d.getVarFlag(key, "vardeps", True) or "").split())
+        deps -= set((d.getVarFlag(key, "vardepsexclude", True) or "").split())
     except:
         bb.note("Error expanding variable %s" % key) 
         raise
diff --git a/bitbake/lib/bb/data_smart.py b/bitbake/lib/bb/data_smart.py
index c8cd8f8..30f9cbc 100644
--- a/bitbake/lib/bb/data_smart.py
+++ b/bitbake/lib/bb/data_smart.py
@@ -277,12 +277,15 @@ class DataSmart:
             self._makeShadowCopy(var)
         self.dict[var][flag] = flagvalue
 
-    def getVarFlag(self, var, flag):
+    def getVarFlag(self, var, flag, exp = False):
         local_var = self._findVar(var)
+        value = None
         if local_var:
             if flag in local_var:
-                return copy.copy(local_var[flag])
-        return None
+                value = copy.copy(local_var[flag])
+        if exp and value:
+            value = self.expand(value, None)
+        return value
 
     def delVarFlag(self, var, flag):
         local_var = self._findVar(var)
-- 
1.7.1

[PATCH 3/3] poky.conf: adjust variable whitelisting to fix sstate checksums

[Paul Eggleton]

After some analysis of task dependencies I have adjusted the global whitelist
used to filter out variables that would otherwise cause unwanted changes in
task checksums for sstate packages.

The following changes have been made to BB_HASHBASE_WHITELIST:

 * Fix typo: FILESEXTRAPATHS not FILESEXTRPATHS
 * Add variables FILE_DIRNAME HOME LOGNAME SHELL TERM USER
 * Remove variables DATE and TIME (these are referred to by a number of tasks
   where they should affect the checksum, e.g. because they influence PV)
 * Remove variable _ (not found in dependencies)

Additionally DATE and SRCDATE are excluded but only explicitly for
patch_do_patch, which uses these only to provide a warning if patches are
out of date.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/conf/distro/poky.conf |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/meta/conf/distro/poky.conf b/meta/conf/distro/poky.conf
index 10b3165..95ca097 100644
--- a/meta/conf/distro/poky.conf
+++ b/meta/conf/distro/poky.conf
@@ -143,4 +143,5 @@ require conf/distro/include/world-broken.inc
 # Setup our hash policy
 BB_SIGNATURE_HANDLER = "basic"
 BB_HASHTASK_WHITELIST = "(.*-cross$|.*-native$|.*-cross-initial$|.*-cross-intermediate$|^virtual:native:.*|^virtual:nativesdk:.*)"
-BB_HASHBASE_WHITELIST = "TMPDIR FILE PATH PWD BB_TASKHASH TIME DATE BBPATH DL_DIR SSTATE_DIR THISDIR _ FILESEXTRPATHS"
+BB_HASHBASE_WHITELIST = "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER"
+patch_do_patch[vardepsexclude] = "DATE SRCDATE"
-- 
1.7.1

Re: [PATCH 0/3] reworked environment variable whitelisting fixes

[Richard Purdie]

On Tue, 2010-12-14 at 18:30 +0000, Paul Eggleton wrote:
> Adjustments to variable whitelisting to fix sstate package hashing. This supersedes
> the paule/sstate contrib branch.
> 
> The bitbake changes have been discussed briefly with Richard Purdie & Chris Larson.
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: paule/sstate2
>   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=paule/sstate2
> 
> Thanks,
>     Paul Eggleton <paul.eggleton@linux.intel.com>

Merged into master, thanks!

Cheers,

Richard