[PATCH 0/3] Fixes for tinfoil usage issues

[PATCH 0/3] Fixes for tinfoil usage issues

[Paul Eggleton]

Fixes for a recursion issue triggered by RSS in OE, and a logging issue
caused by the tinfoil2 rework.


The following changes since commit 751b06c25d22eea8240f9429cb49874082245e52:

  bitbake-diffsigs: Add debug support (2017-03-22 14:52:39 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib paule/tinfoil-fixes-bb2
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/tinfoil-fixes-bb2

Paul Eggleton (3):
  fetch2/git: prevent recursion on getting latest revision
  lib/bb/msg: introduce logger_create() function
  tinfoil: enable client-side logger handling by default

 bin/bitbake-diffsigs | 14 ++------------
 bin/bitbake-dumpsig  | 14 ++------------
 bin/bitbake-layers   | 25 +++++--------------------
 lib/bb/fetch2/git.py | 28 ++++++++++++++++++++--------
 lib/bb/msg.py        | 15 +++++++++++++++
 lib/bb/tinfoil.py    |  6 +++++-
 6 files changed, 49 insertions(+), 53 deletions(-)

-- 
2.9.3

[PATCH 1/3] fetch2/git: prevent recursion on getting latest revision

[Paul Eggleton]

We call git ls-remote to get the latest revision from a git repository,
however by calling runfetchcmd() we can end up recursively running
git ls-remote a number of times with OE e.g. if ${SRCPV} is in PV, ${PV}
is in WORKDIR, and ${WORKDIR} is in PATH (as a result of recipe-specific
sysroots), our call to runfetchcmd() exports PATH so _lsremote() will
get called again - with the end result that we run git ls-remote 30
times in quick succession (!). Prevent that from happening by using a
guard variable and returning a dummy value if it's called recursively.

Fixes [YOCTO #11185].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/fetch2/git.py | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index a8be859..2550bde 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -379,14 +379,26 @@ class Git(FetchMethod):
         """
         Run git ls-remote with the specified search string
         """
-        repourl = self._get_repo_url(ud)
-        cmd = "%s ls-remote %s %s" % \
-              (ud.basecmd, repourl, search)
-        if ud.proto.lower() != 'file':
-            bb.fetch2.check_network_access(d, cmd, repourl)
-        output = runfetchcmd(cmd, d, True)
-        if not output:
-            raise bb.fetch2.FetchError("The command %s gave empty output unexpectedly" % cmd, ud.url)
+        # Prevent recursion e.g. in OE if SRCPV is in PV, PV is in WORKDIR,
+        # and WORKDIR is in PATH (as a result of RSS), our call to
+        # runfetchcmd() exports PATH so this function will get called again (!)
+        # In this scenario the return call of the function isn't actually
+        # important - WORKDIR isn't needed in PATH to call git ls-remote
+        # anyway.
+        if d.getVar('_BB_GIT_IN_LSREMOTE', False):
+            return ''
+        d.setVar('_BB_GIT_IN_LSREMOTE', '1')
+        try:
+            repourl = self._get_repo_url(ud)
+            cmd = "%s ls-remote %s %s" % \
+                (ud.basecmd, repourl, search)
+            if ud.proto.lower() != 'file':
+                bb.fetch2.check_network_access(d, cmd, repourl)
+            output = runfetchcmd(cmd, d, True)
+            if not output:
+                raise bb.fetch2.FetchError("The command %s gave empty output unexpectedly" % cmd, ud.url)
+        finally:
+            d.delVar('_BB_GIT_IN_LSREMOTE')
         return output
 
     def _latest_revision(self, ud, d, name):
-- 
2.9.3

[PATCH 2/3] lib/bb/msg: introduce logger_create() function

[Paul Eggleton]

We use this code to set up a logger with colour in a number of different
places, so create one function that does this and make some of bitbake's
utility scripts use it.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 bin/bitbake-diffsigs | 14 ++------------
 bin/bitbake-dumpsig  | 14 ++------------
 bin/bitbake-layers   | 25 +++++--------------------
 lib/bb/msg.py        | 15 +++++++++++++++
 4 files changed, 24 insertions(+), 44 deletions(-)

diff --git a/bin/bitbake-diffsigs b/bin/bitbake-diffsigs
index c087f99..1e3de09 100755
--- a/bin/bitbake-diffsigs
+++ b/bin/bitbake-diffsigs
@@ -30,19 +30,9 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), '
 
 import bb.tinfoil
 import bb.siggen
+import bb.msg
 
-def logger_create(name, output=sys.stderr):
-    logger = logging.getLogger(name)
-    console = logging.StreamHandler(output)
-    format = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
-    if output.isatty():
-        format.enable_color()
-    console.setFormatter(format)
-    logger.addHandler(console)
-    logger.setLevel(logging.INFO)
-    return logger
-
-logger = logger_create('bitbake-diffsigs')
+logger = bb.msg.logger_create('bitbake-diffsigs')
 
 def find_compare_task(bbhandler, pn, taskname):
     """ Find the most recent signature files for the specified PN/task and compare them """
diff --git a/bin/bitbake-dumpsig b/bin/bitbake-dumpsig
index 38efd22..95ebd93 100755
--- a/bin/bitbake-dumpsig
+++ b/bin/bitbake-dumpsig
@@ -29,19 +29,9 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), '
 
 import bb.tinfoil
 import bb.siggen
+import bb.msg
 
-def logger_create(name, output=sys.stderr):
-    logger = logging.getLogger(name)
-    console = logging.StreamHandler(output)
-    format = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
-    if output.isatty():
-        format.enable_color()
-    console.setFormatter(format)
-    logger.addHandler(console)
-    logger.setLevel(logging.INFO)
-    return logger
-
-logger = logger_create('bitbake-dumpsig')
+logger = bb.msg.logger_create('bitbake-dumpsig')
 
 def find_siginfo_task(bbhandler, pn, taskname):
     """ Find the most recent signature file for the specified PN/task """
diff --git a/bin/bitbake-layers b/bin/bitbake-layers
index 66fc7ca..390ad05 100755
--- a/bin/bitbake-layers
+++ b/bin/bitbake-layers
@@ -30,26 +30,9 @@ topdir = os.path.dirname(bindir)
 sys.path[0:0] = [os.path.join(topdir, 'lib')]
 
 import bb.tinfoil
+import bb.msg
 
-def logger_create(name, output=sys.stderr):
-    logger = logging.getLogger(name)
-    loggerhandler = logging.StreamHandler(output)
-    loggerhandler.setFormatter(logging.Formatter("%(levelname)s: %(message)s"))
-    logger.addHandler(loggerhandler)
-    logger.setLevel(logging.INFO)
-    return logger
-
-def logger_setup_color(logger, color='auto'):
-    from bb.msg import BBLogFormatter
-    console = logging.StreamHandler(sys.stdout)
-    formatter = BBLogFormatter("%(levelname)s: %(message)s")
-    console.setFormatter(formatter)
-    logger.handlers = [console]
-    if color == 'always' or (color == 'auto' and console.stream.isatty()):
-        formatter.enable_color()
-
-
-logger = logger_create('bitbake-layers', sys.stdout)
+logger = bb.msg.logger_create('bitbake-layers', sys.stdout)
 
 def main():
     parser = argparse.ArgumentParser(
@@ -74,7 +57,9 @@ def main():
     elif global_args.quiet:
         logger.setLevel(logging.ERROR)
 
-    logger_setup_color(logger, global_args.color)
+    # Need to re-run logger_create with color argument
+    # (will be the same logger since it has the same name)
+    bb.msg.logger_create('bitbake-layers', output=sys.stdout, color=global_args.color)
 
     plugins = []
     tinfoil = bb.tinfoil.Tinfoil(tracking=True)
diff --git a/lib/bb/msg.py b/lib/bb/msg.py
index b7c39fa..90b1582 100644
--- a/lib/bb/msg.py
+++ b/lib/bb/msg.py
@@ -201,3 +201,18 @@ def fatal(msgdomain, msg):
         logger = logging.getLogger("BitBake")
     logger.critical(msg)
     sys.exit(1)
+
+def logger_create(name, output=sys.stderr, level=logging.INFO, preserve_handlers=False, color='auto'):
+    """Standalone logger creation function"""
+    logger = logging.getLogger(name)
+    console = logging.StreamHandler(output)
+    format = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
+    if color == 'always' or (color == 'auto' and output.isatty()):
+        format.enable_color()
+    console.setFormatter(format)
+    if preserve_handlers:
+        logger.addHandler(console)
+    else:
+        logger.handlers = [console]
+    logger.setLevel(level)
+    return logger
-- 
2.9.3

[PATCH 3/3] tinfoil: enable client-side logger handling by default

[Paul Eggleton]

If you had a script that uses tinfoil and it failed to connect to the
BitBake server, you did't see any of the expected messages - this was
because client-side logging wasn't being handled at all. Since you'll
almost always want this when using tinfoil, have it use the new
bb.msg.logger_create() function to enable client-side logging by
default.

Relates to [YOCTO #11185].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/tinfoil.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py
index 19b41be..c2ee707 100644
--- a/lib/bb/tinfoil.py
+++ b/lib/bb/tinfoil.py
@@ -210,13 +210,17 @@ class TinfoilCookerAdapter:
 
 class Tinfoil:
 
-    def __init__(self, output=sys.stdout, tracking=False):
+    def __init__(self, output=sys.stdout, tracking=False, setup_logging=True):
         self.logger = logging.getLogger('BitBake')
         self.config_data = None
         self.cooker = None
         self.tracking = tracking
         self.ui_module = None
         self.server_connection = None
+        if setup_logging:
+            # This is the *client-side* logger, nothing to do with
+            # logging messages from the server
+            bb.msg.logger_create('BitBake', output)
 
     def __enter__(self):
         return self
-- 
2.9.3

Re: [PATCH 3/3] tinfoil: enable client-side logger handling by default

[Paul Eggleton]

On Monday, 27 March 2017 1:17:28 PM NZDT Paul Eggleton wrote:
> If you had a script that uses tinfoil and it failed to connect to the
> BitBake server, you did't see any of the expected messages - this was
> because client-side logging wasn't being handled at all. Since you'll
> almost always want this when using tinfoil, have it use the new
> bb.msg.logger_create() function to enable client-side logging by
> default.

Sigh... this causes a bunch of debug messages from the codeparser to get shown 
when using devtool add. I am still tracking that issue down - perhaps it's 
better than not seeing any messages at all though.

- Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre