[PATCH 0/2] Shell message fixes (BitBake side)

[PATCH 0/2] Shell message fixes (BitBake side)

[Paul Eggleton]

BitBake side of some fixes for regressions caused by the recent shell
message changes.


The following changes since commit 5c5f8da509f6bbc1fad263462142519ef3d54a35:

  tests/data: Add new data tests (2015-07-12 22:50:27 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib paule/bb-shell-logging-fixes
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-shell-logging-fixes

Paul Eggleton (2):
  lib/bb: provide mechanism to bypass UI log suppression
  lib/bb/process: check output of select() before reading extrafiles

 lib/bb/__init__.py |  8 ++++----
 lib/bb/build.py    |  7 ++++++-
 lib/bb/process.py  | 21 +++++++++++----------
 3 files changed, 21 insertions(+), 15 deletions(-)

-- 
2.1.0

[PATCH 1/2] lib/bb: provide mechanism to bypass UI log suppression

[Paul Eggleton]

The recent change to connect through the shell logging functions had an
unexpected side-effect - bb.error() and bb.fatal() cause a flag to be
set internally such that BitBake's UI will not print the full task log
on failure; unfortunately we have in places within the OpenEmbedded
metadata called these shell logging functions under error situations
where we still want to see the full log (i.e., the message we're sending
doesn't include the full error). Thus, provide a mechanism to fatally
exit with an error but unset the flag, utilising the built-in python
logging functionality that allows extra values to be passed in the log
record.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/__init__.py | 8 ++++----
 lib/bb/build.py    | 7 ++++++-
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/lib/bb/__init__.py b/lib/bb/__init__.py
index 1a30fa1..1f7946e 100644
--- a/lib/bb/__init__.py
+++ b/lib/bb/__init__.py
@@ -94,11 +94,11 @@ def note(*args):
 def warn(*args):
     logger.warn(''.join(args))
 
-def error(*args):
-    logger.error(''.join(args))
+def error(*args, **kwargs):
+    logger.error(''.join(args), extra=kwargs)
 
-def fatal(*args):
-    logger.critical(''.join(args))
+def fatal(*args, **kwargs):
+    logger.critical(''.join(args), extra=kwargs)
     raise BBHandledException()
 
 def deprecated(func, name=None, advice=""):
diff --git a/lib/bb/build.py b/lib/bb/build.py
index cce01fe..3439964 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -350,6 +350,8 @@ exit $?
                 # The caller will call exit themselves, so bb.error() is
                 # what we want here rather than bb.fatal()
                 bb.error(value)
+            elif cmd == 'bbfatal_log':
+                bb.error(value, forcelog=True)
             elif cmd == 'bbdebug':
                 splitval = value.split(' ', 1)
                 level = int(splitval[0])
@@ -446,7 +448,10 @@ def _exec_task(fn, task, d, quieterr):
             self.triggered = False
             logging.Handler.__init__(self, logging.ERROR)
         def emit(self, record):
-            self.triggered = True
+            if getattr(record, 'forcelog', False):
+                self.triggered = False
+            else:
+                self.triggered = True
 
     # Handle logfiles
     si = open('/dev/null', 'r')
-- 
2.1.0

[PATCH 2/2] lib/bb/process: check output of select() before reading extrafiles

[Paul Eggleton]

We're calling select() to find the fds that have data available, so we
really ought to check the return to see if the extra file is in there
before trying to read from it. This is part of the fix for the
performance regression that this code introduced (5-10 minutes extra in
a reasonable size OE build); the rest is down to an issue in the way
that pseudo currently handles FIFOs and will need to be addressed there,
see: https://bugzilla.yoctoproject.org/show_bug.cgi?id=7993

Solution suggested by Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/process.py | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/lib/bb/process.py b/lib/bb/process.py
index d95a03d..7c79785 100644
--- a/lib/bb/process.py
+++ b/lib/bb/process.py
@@ -83,15 +83,16 @@ def _logged_communicate(pipe, log, input, extrafiles):
         bb.utils.nonblockingfd(fobj.fileno())
         rin.append(fobj)
 
-    def readextras():
+    def readextras(selected):
         for fobj, func in extrafiles:
-            try:
-                data = fobj.read()
-            except IOError as err:
-                if err.errno == errno.EAGAIN or err.errno == errno.EWOULDBLOCK:
-                    data = None
-            if data is not None:
-                func(data)
+            if fobj in selected:
+                try:
+                    data = fobj.read()
+                except IOError as err:
+                    if err.errno == errno.EAGAIN or err.errno == errno.EWOULDBLOCK:
+                        data = None
+                if data is not None:
+                    func(data)
 
     try:
         while pipe.poll() is None:
@@ -114,12 +115,12 @@ def _logged_communicate(pipe, log, input, extrafiles):
                     errdata.append(data)
                     log.write(data)
 
-            readextras()
+            readextras(r)
 
     finally:    
         log.flush()
 
-    readextras()
+    readextras([fobj for fobj, _ in extrafiles])
 
     if pipe.stdout is not None:
         pipe.stdout.close()
-- 
2.1.0