[PATCH 0/5] Checksum / error reporting improvements

[PATCH 0/5] Checksum / error reporting improvements

[Paul Eggleton]

Some checksum improvements and a few error reporting fixes that I made
in the process. Note that the varflags change won't do anything without
the list of exclusions; an appropriate list has been set in a patch
which will shortly be set to the OE-Core list.


The following changes (against Poky, but which apply cleanly with -p2
against BitBake master) are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib paule/bb-checksum-fixes-2
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-checksum-fixes-2

Paul Eggleton (5):
  lib/bb/data_smart.py: don't report variable in ExpansionError if not
    set
  cooker: fix UnboundLocalError when exception occurs during parsing
  cooker: report recipe being parsed when ExpansionError occurs
  lib/bb/fetch2: ignore remote URIs when doing file checksums
  bitbake: include varflags in checksums

 bitbake/lib/bb/cooker.py          |    8 +++++++-
 bitbake/lib/bb/data.py            |   20 +++++++++++++++++++-
 bitbake/lib/bb/data_smart.py      |    5 ++++-
 bitbake/lib/bb/fetch2/__init__.py |   26 ++++++++++++++++++++------
 bitbake/lib/bb/siggen.py          |    4 ++++
 5 files changed, 54 insertions(+), 9 deletions(-)

-- 
1.7.9.5

[PATCH 1/5] lib/bb/data_smart.py: don't report variable in ExpansionError if not set

[Paul Eggleton]

If the variable name is not specified then don't confuse the error message
by starting off with "Failure expanding variable None...".

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 bitbake/lib/bb/data_smart.py |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/data_smart.py b/bitbake/lib/bb/data_smart.py
index 27fb7d9..2c02cde 100644
--- a/bitbake/lib/bb/data_smart.py
+++ b/bitbake/lib/bb/data_smart.py
@@ -102,7 +102,10 @@ class ExpansionError(Exception):
         self.expression = expression
         self.variablename = varname
         self.exception = exception
-        self.msg = "Failure expanding variable %s, expression was %s which triggered exception %s: %s" % (varname, expression, type(exception).__name__, exception)
+        if varname:
+            self.msg = "Failure expanding variable %s, expression was %s which triggered exception %s: %s" % (varname, expression, type(exception).__name__, exception)
+        else:
+            self.msg = "Failure expanding expression %s which triggered exception %s: %s" % (expression, type(exception).__name__, exception)
         Exception.__init__(self, self.msg)
         self.args = (varname, expression, exception)
     def __str__(self):
-- 
1.7.9.5

[PATCH 2/5] cooker: fix UnboundLocalError when exception occurs during parsing

[Paul Eggleton]

Fix a recent regression where we see the following additional error
after an error occurs during parsing:

ERROR: Command execution failed: Traceback (most recent call last):
  File "/home/paul/poky/poky/bitbake/lib/bb/command.py", line 84, in runAsyncCommand
    self.cooker.updateCache()
  File "/home/paul/poky/poky/bitbake/lib/bb/cooker.py", line 1202, in updateCache
    if not self.parser.parse_next():
  File "/home/paul/poky/poky/bitbake/lib/bb/cooker.py", line 1672, in parse_next
    self.virtuals += len(result)
UnboundLocalError: local variable 'result' referenced before assignment

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 bitbake/lib/bb/cooker.py |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index d1de757..b30945c 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1646,6 +1646,8 @@ class CookerParser(object):
                     yield result
 
     def parse_next(self):
+        result = []
+        parsed = None
         try:
             parsed, result = self.results.next()
         except StopIteration:
-- 
1.7.9.5

[PATCH 3/5] cooker: report recipe being parsed when ExpansionError occurs

[Paul Eggleton]

When an ExpansionError occurs during parsing it is useful to know which
recipe was being parsed when it occurred.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 bitbake/lib/bb/cooker.py |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index b30945c..928b600 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1657,9 +1657,13 @@ class CookerParser(object):
             logger.error('Unable to parse %s: %s' %
                      (exc.recipe, bb.exceptions.to_string(exc.realexception)))
             self.shutdown(clean=False)
-        except (bb.parse.ParseError, bb.data_smart.ExpansionError) as exc:
+        except bb.parse.ParseError as exc:
             logger.error(str(exc))
             self.shutdown(clean=False)
+        except bb.data_smart.ExpansionError as exc:
+            _, value, _ = sys.exc_info()
+            logger.error('ExpansionError during parsing %s: %s', value.recipe, str(exc))
+            self.shutdown(clean=False)
         except SyntaxError as exc:
             logger.error('Unable to parse %s', exc.recipe)
             self.shutdown(clean=False)
-- 
1.7.9.5

[PATCH 4/5] lib/bb/fetch2: ignore remote URIs when doing file checksums

[Paul Eggleton]

Skip evaluating remote URIs when doing local file checksums, because we
don't need to process them and doing so will trigger a parse failure if
SRCREV is not fully specified. Whilst this is just delaying a check
until runtime (when do_fetch runs for the recipe) we're only validating
this here accidentally and if we did wish to check it during parsing it
ought to be done explicitly.

Fixes [YOCTO #2512]

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 bitbake/lib/bb/fetch2/__init__.py |   26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index 6ae69cd..83050e4 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -112,6 +112,9 @@ class NetworkAccess(BBFetchException):
          BBFetchException.__init__(self, msg)
          self.args = (url, cmd)
 
+class NonLocalMethod(Exception):
+    def __init__(self):
+        Exception.__init__(self)
 
 def decodeurl(url):
     """Decodes an URL into the tokens (scheme, network location, path,
@@ -565,17 +568,17 @@ def srcrev_internal_helper(ud, d, name):
 def get_checksum_file_list(d):
     """ Get a list of files checksum in SRC_URI
 
-    Returns the all resolved local path of all local file entries in
+    Returns the resolved local paths of all local file entries in
     SRC_URI as a space-separated string
     """
-    fetch = Fetch([], d)
+    fetch = Fetch([], d, cache = False, localonly = True)
 
     dl_dir = d.getVar('DL_DIR', True)
     filelist = []
     for u in fetch.urls:
         ud = fetch.ud[u]
 
-        if isinstance(ud.method, local.Local):
+        if ud and isinstance(ud.method, local.Local):
             ud.setup_localpath(d)
             f = ud.localpath
             if f.startswith(dl_dir):
@@ -639,7 +642,7 @@ class FetchData(object):
     """
     A class which represents the fetcher state for a given URI.
     """
-    def __init__(self, url, d):
+    def __init__(self, url, d, localonly = False):
         # localpath is the location of a downloaded result. If not set, the file is local.
         self.donestamp = None
         self.localfile = ""
@@ -686,6 +689,9 @@ class FetchData(object):
         if not self.method:
             raise NoMethodError(url)
 
+        if localonly and not isinstance(self.method, local.Local):
+            raise NonLocalMethod()
+
         if hasattr(self.method, "urldata_init"):
             self.method.urldata_init(self, d)
 
@@ -1009,7 +1015,10 @@ class FetchMethod(object):
         return "%s-%s" % (key, d.getVar("PN", True) or "")
 
 class Fetch(object):
-    def __init__(self, urls, d, cache = True):
+    def __init__(self, urls, d, cache = True, localonly = False):
+        if localonly and cache:
+            raise Exception("bb.fetch2.Fetch.__init__: cannot set cache and localonly at same time")
+
         if len(urls) == 0:
             urls = d.getVar("SRC_URI", True).split()
         self.urls = urls
@@ -1022,7 +1031,12 @@ class Fetch(object):
 
         for url in urls:
             if url not in self.ud:
-                self.ud[url] = FetchData(url, d)
+                try:
+                    self.ud[url] = FetchData(url, d, localonly)
+                except NonLocalMethod:
+                    if localonly:
+                        self.ud[url] = None
+                        pass
 
         if cache:
             urldata_cache[fn] = self.ud
-- 
1.7.9.5

[PATCH 5/5] bitbake: include varflags in checksums

[Paul Eggleton]

Add a dependency to each variable on a filtered list of its varflags.
This is intended to catch things such as SRC_URI checksums, varflags
controlling extra functionality from classes (e.g. the recently updated
update-alternatives class in OE-Core), etc. and ensure their values
influence the sstate checksums.

There is an exclusion list which needs to be set via bitbake.conf
(BB_SIGNATURE_EXCLUDE_FLAGS), if this is not set then the functionality
is disabled. The existing vardepsexclude mechanism can also be used to
exclude undesired varflags, but they must be fully specified, e.g.:

do_patch[vardepsexclude] += "do_patch[someflag]"

Implements [YOCTO #2517].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 bitbake/lib/bb/data.py   |   20 +++++++++++++++++++-
 bitbake/lib/bb/siggen.py |    4 ++++
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
index c0636e1..e3ffefe 100644
--- a/bitbake/lib/bb/data.py
+++ b/bitbake/lib/bb/data.py
@@ -279,7 +279,12 @@ def build_dependencies(key, keys, shelldeps, vardepvals, d):
     deps = set()
     vardeps = d.getVarFlag(key, "vardeps", True)
     try:
-        value = d.getVar(key, False)
+        if key[-1] == ']':
+            vf = key[:-1].split('[')
+            value = d.getVarFlag(vf[0], vf[1], False)
+        else:
+            value = d.getVar(key, False)
+
         if key in vardepvals:
            value =  d.getVarFlag(key, "vardepvalue", True)
         elif d.getVarFlag(key, "func"):
@@ -301,6 +306,19 @@ def build_dependencies(key, keys, shelldeps, vardepvals, d):
             parser = d.expandWithRefs(value, key)
             deps |= parser.references
             deps = deps | (keys & parser.execs)
+
+        # Add varflags, assuming an exclusion list is set
+        varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS', True)
+        if varflagsexcl:
+            varfdeps = []
+            varflags = d.getVarFlags(key)
+            if varflags:
+                for f in varflags:
+                    if f not in varflagsexcl:
+                        varfdeps.append('%s[%s]' % (key, f))
+            if varfdeps:
+                deps |= set(varfdeps)
+
         deps |= set((vardeps or "").split())
         deps -= set((d.getVarFlag(key, "vardepsexclude", True) or "").split())
     except:
diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index daf5677..c4b7c39 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -108,6 +108,10 @@ class SignatureGeneratorBasic(SignatureGenerator):
                 data = data + dep
                 if dep in lookupcache:
                     var = lookupcache[dep]
+                elif dep[-1] == ']':
+                    vf = dep[:-1].split('[')
+                    var = d.getVarFlag(vf[0], vf[1], False)
+                    lookupcache[dep] = var
                 else:
                     var = d.getVar(dep, False)
                     lookupcache[dep] = var
-- 
1.7.9.5

Re: [PATCH 0/5] Checksum / error reporting improvements

[Richard Purdie]

On Wed, 2012-05-30 at 17:17 +0100, Paul Eggleton wrote:
> Some checksum improvements and a few error reporting fixes that I made
> in the process. Note that the varflags change won't do anything without
> the list of exclusions; an appropriate list has been set in a patch
> which will shortly be set to the OE-Core list.
> 
> 
> The following changes (against Poky, but which apply cleanly with -p2
> against BitBake master) are available in the git repository at:
> 
>   git://git.yoctoproject.org/poky-contrib paule/bb-checksum-fixes-2
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-checksum-fixes-2
> 
> Paul Eggleton (5):
>   lib/bb/data_smart.py: don't report variable in ExpansionError if not
>     set
>   cooker: fix UnboundLocalError when exception occurs during parsing
>   cooker: report recipe being parsed when ExpansionError occurs
>   lib/bb/fetch2: ignore remote URIs when doing file checksums
>   bitbake: include varflags in checksums

Merged to master, thanks.

Richard