[PATCH 0/6] bitbake: Improve error/exception handling

[PATCH 0/6] bitbake: Improve error/exception handling

[Robert Yang]

The following changes since commit eddff2b361928e88e3628ebc22a1a0ebb119e01b:

  curl: actually apply latest CVE patches (2018-11-09 17:46:18 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib rbt/bb_improve_error
  http://git.pokylinux.org/cgit.cgi//log/?h=rbt/bb_improve_error

Robert Yang (6):
  bitbake: utils: better_compile(): Fix line number when report errors
  bitbake: parse/ast: fix line number for anonymous function
  bitbake: data_smart: Add original traceback to ExpansionError
  bitbake: data_smart: fix filename for compile()
  bitbake: server/process: print a message when no logfile
  bitbake: BBHandler: Fix __python_func_regexp__ for comment lines

 bitbake/lib/bb/data_smart.py               | 9 +++++++--
 bitbake/lib/bb/parse/ast.py                | 2 +-
 bitbake/lib/bb/parse/parse_py/BBHandler.py | 2 +-
 bitbake/lib/bb/server/process.py           | 4 ++++
 bitbake/lib/bb/utils.py                    | 7 +++++--
 5 files changed, 18 insertions(+), 6 deletions(-)

-- 
2.7.4

[PATCH 1/6] bitbake: utils: better_compile(): Fix line number when report errors

[Robert Yang]

Fixed:
- Add an error line in base.bbclass, e.g.:
  15
  16 def oe_import(d):
  17     import sys
  18     Compile error
  19     bbpath = d.getVar("BBPATH").split(":")
  [snip]

  Note the "Compile error" line, I added it for reporting errors.

  $ bitbake -p
  ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 15:

  The code lines resulting in this error were:
       0014:    import oe.data
       0015:    for toimport in oe.data.typed_value("OE_IMPORTS", d):
       0016:        imported = __import__(toimport)
       0017:        inject(toimport.split(".", 1)[0], imported)
   *** 0018:
       0019:    return ""
       0020:
  SyntaxError: invalid syntax (base.bbclass, line 18)

  There are 2 problems:
  - The "line 15" is incorrect, it is a blank line, not the error line.
  - The "*** 0018" points to incorrect position.

  These two problems would mislead people a lot sometimes.

- Now fix it to:
  $ bitbake -p
  ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 18:

  The code lines resulting in this error were:
       0001:def oe_import(d):
       0002:    import sys
   *** 0003:    Compile error
       0004:    bbpath = d.getVar("BBPATH").split(":")
                [snip]
  SyntaxError: invalid syntax (base.bbclass, line 18)

Please see comments in the code for more details on how it is fixed.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/utils.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index 73b6cb4..0322ffb 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -317,10 +317,13 @@ def better_compile(text, file, realfile, mode = "exec", lineno = 0):
         error = []
         # split the text into lines again
         body = text.split('\n')
-        error.append("Error in compiling python function in %s, line %s:\n" % (realfile, lineno))
+        error.append("Error in compiling python function in %s, line %s:\n" % (realfile, e.lineno))
         if hasattr(e, "lineno"):
             error.append("The code lines resulting in this error were:")
-            error.extend(_print_trace(body, e.lineno))
+            # e.lineno: line's position in reaflile
+            # lineno: function name's "position -1" in realfile
+            # e.lineno - lineno: line's relative position in function
+            error.extend(_print_trace(body, e.lineno - lineno))
         else:
             error.append("The function causing this error was:")
             for line in body:
-- 
2.7.4

[PATCH 2/6] bitbake: parse/ast: fix line number for anonymous function

[Robert Yang]

Fixed:
- Define an error anonymous function in base.bbclass:
  15
  16 python() {
  17     Compile error
  18 }

  $ bitbake -p
  ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 18:

  The code lines resulting in this error were:
       0001:def __anon_18__buildarea1_lyang1_poky_meta_classes_base_bbclass(d):
   *** 0002:    Compile error
       0003:
  SyntaxError: invalid syntax (base.bbclass, line 18)

  The lineno should be 17, but it reported 18, this would mislead people a lot
  when there more lines.

- Now fix it to:
  ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 17:

  The code lines resulting in this error were:
       0001:def __anon_18__buildarea1_lyang1_poky_meta_classes_base_bbclass(d):
   *** 0002:    Compile error
       0003:
  SyntaxError: invalid syntax (base.bbclass, line 17)

This is because the anonymous function is constructed by:
text = "def %s(d):\n" % (funcname) + text

The len(self.body) doesn't include the "def " line, the length of the function
should be "len(self.body) + 1", so we need pass "self.lineno - (len(self.body) + 1)"
which is the same as 'self.lineno - len(self.body) - 1' to
bb.methodpool.insert_method() as we already had done to named function. Otherwise, the
lineno is wrong, and would cause other problems such as report which line is
wrong, but the line is not what we want since it reports incorrect line.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/parse/ast.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py
index 9d20c32..6d7c80b 100644
--- a/bitbake/lib/bb/parse/ast.py
+++ b/bitbake/lib/bb/parse/ast.py
@@ -178,7 +178,7 @@ class MethodNode(AstNode):
             funcname = ("__anon_%s_%s" % (self.lineno, self.filename.translate(MethodNode.tr_tbl)))
             self.python = True
             text = "def %s(d):\n" % (funcname) + text
-            bb.methodpool.insert_method(funcname, text, self.filename, self.lineno - len(self.body))
+            bb.methodpool.insert_method(funcname, text, self.filename, self.lineno - len(self.body) - 1)
             anonfuncs = data.getVar('__BBANONFUNCS', False) or []
             anonfuncs.append(funcname)
             data.setVar('__BBANONFUNCS', anonfuncs)
-- 
2.7.4

[PATCH 3/6] bitbake: data_smart: Add original traceback to ExpansionError

[Robert Yang]

This can make it print clearer errors, for exmaple:

Add Runtime_error to 'def oe_import(d)"

 16 def oe_import(d):
 17     import sys
 18     Runtime_error
[snip]

* Before the patch:
  $ bitbake -p
  ERROR: Unable to parse /buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py
  Traceback (most recent call last):
    File "/buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py", line 430, in DataSmart.expandWithRefs(s='${@oe_import(d)}', varname='OE_IMPORTED[:=]'):
                   except Exception as exc:
      >                raise ExpansionError(varname, s, exc) from exc

  bb.data_smart.ExpansionError: Failure expanding variable OE_IMPORTED[:=], expression was ${@oe_import(d)} which triggered exception NameError: name 'Runtime_error' is not defined

  This error message has two problems:
  - "Unable to parse data_smart.py": This isn't the real cause.
  - It pionts to "raise ExpansionError(varname, s, exc) from exc" which isn't clear enough.

* After the patch:
  $ bitbake -p
  ERROR: Unable to parse OE_IMPORTED[:=]
  Traceback (most recent call last):
    File "OE_IMPORTED[:=]", line 1, in <module>
    File "/buildarea1/lyang1/poky/meta/classes/base.bbclass", line 18, in oe_import(d=<bb.data_smart.DataSmart object at 0x7f9257e7a0b8>):
           import sys
      >    Runtime_error

  bb.data_smart.ExpansionError: Failure expanding variable OE_IMPORTED[:=], expression was ${@oe_import(d)} which triggered exception NameError: name 'Runtime_error' is not defined

This one is more clearer than before.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/data_smart.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/data_smart.py b/bitbake/lib/bb/data_smart.py
index 6b94fc4..efa5a79 100644
--- a/bitbake/lib/bb/data_smart.py
+++ b/bitbake/lib/bb/data_smart.py
@@ -427,7 +427,8 @@ class DataSmart(MutableMapping):
             except bb.parse.SkipRecipe:
                 raise
             except Exception as exc:
-                raise ExpansionError(varname, s, exc) from exc
+                tb = sys.exc_info()[2]
+                raise ExpansionError(varname, s, exc).with_traceback(tb) from exc
 
         varparse.value = s
 
-- 
2.7.4

[PATCH 4/6] bitbake: data_smart: fix filename for compile()

[Robert Yang]

Fixed:
Add the following two lines to conf/local.conf:
FOO = "${@foo = 5}"
HOSTTOOLS += "${FOO}"

* Before the patch
  $ bitbake -p
  Check the first lines of bitbake bitbake-cookerdaemon.log
  [snip]
  File "/buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py", line 125, in python_sub
    codeobj = compile(code.strip(), self.varname or "<expansion>", "eval")
  File "FOO", line 1
  [snip]

  There isn't a file named 'FOO', but a variable name.

* After the patch
  $ bitbake -p
  [snip]
  File "/buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py", line 129, in python_sub
    codeobj = compile(code.strip(), varname, "eval")
  File "Var <FOO>", line 1
    foo = 5

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/data_smart.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/data_smart.py b/bitbake/lib/bb/data_smart.py
index efa5a79..67af380 100644
--- a/bitbake/lib/bb/data_smart.py
+++ b/bitbake/lib/bb/data_smart.py
@@ -122,7 +122,11 @@ class VariableParse:
                 connector = self.d["_remote_data"]
                 return connector.expandPythonRef(self.varname, code, self.d)
 
-            codeobj = compile(code.strip(), self.varname or "<expansion>", "eval")
+            if self.varname:
+                varname = 'Var <%s>' % self.varname
+            else:
+                varname = '<expansion>'
+            codeobj = compile(code.strip(), varname, "eval")
 
             parser = bb.codeparser.PythonParser(self.varname, logger)
             parser.parse_python(code)
-- 
2.7.4

[PATCH 5/6] bitbake: server/process: print a message when no logfile

[Robert Yang]

[YOCTO #12898]

There might be no bitbake-cookerdaemon.log, print a message for debugging.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/server/process.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/bitbake/lib/bb/server/process.py b/bitbake/lib/bb/server/process.py
index ed930c6..12f52b0 100644
--- a/bitbake/lib/bb/server/process.py
+++ b/bitbake/lib/bb/server/process.py
@@ -428,7 +428,11 @@ class BitBakeServer(object):
                         bb.error("Last 10 lines of server log for this session (%s):\n%s" % (logfile, "".join(lines[-10:])))
                     else:
                         bb.error("Server log for this session (%s):\n%s" % (logfile, "".join(lines)))
+            else:
+                bb.error("%s doesn't exist" % logfile)
+
             raise SystemExit(1)
+
         ready.close()
 
     def _startServer(self):
-- 
2.7.4

[PATCH 6/6] bitbake: BBHandler: Fix __python_func_regexp__ for comment lines

[Robert Yang]

Fixed:
- Add a comment in base.bbclass:
  def oe_import(d):
      import sys
  # Comment
      bbpath = d.getVar("BBPATH").split(":")
  [snip]

  Note, '# Comment' is started with '#', it is legal in python's syntax
  (though maybe not a good style), but bitbake reported errors:

  $ bitbake -p
  ERROR: ParseError at /path/to/base.bbclass:20: unparsed line: '    bbpath = d.getVar("BBPATH").split(":")'

  This error report would mislead people, the real problem is that '# Comment'
  is not supported, but it reports the next line, this may make it hard to debug
  the code are complicated.

We can make __python_func_regexp__ handle '^#' to fix the problem, since it
already can handle blank line "^$" in a python function, so it would be pretty
safe to handle "^#" as well.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/parse/parse_py/BBHandler.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py b/bitbake/lib/bb/parse/parse_py/BBHandler.py
index e5039e3..01fc47e 100644
--- a/bitbake/lib/bb/parse/parse_py/BBHandler.py
+++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py
@@ -45,7 +45,7 @@ __addtask_regexp__       = re.compile("addtask\s+(?P<func>\w+)\s*((before\s*(?P<
 __deltask_regexp__       = re.compile("deltask\s+(?P<func>\w+)")
 __addhandler_regexp__    = re.compile( r"addhandler\s+(.+)" )
 __def_regexp__           = re.compile( r"def\s+(\w+).*:" )
-__python_func_regexp__   = re.compile( r"(\s+.*)|(^$)" )
+__python_func_regexp__   = re.compile( r"(\s+.*)|(^$)|(^#)" )
 
 __infunc__ = []
 __inpython__ = False
-- 
2.7.4