[PATCH v4 0/18] Multiplexed task logs / bitbake fixes for fetch2 / stamp features

[PATCH v4 0/18] Multiplexed task logs / bitbake fixes for fetch2 / stamp features

[Jason Wessel]

This is the v4 patch set for various fixes/features to bitbake. 

The patches 13-15 are probably bit more of an RFC vs ready for merge,
in particular the patch 14.  Bitbake's setscene engine really
shouldn't know the specifics about what stamp (in this case do_patch)
should invalidate the use of setscene for any given task.  At the
moment there is no provision to be able to pass this from oe-core to
bitbake such that this can be generically specified.  That being said
the patch 14 does serve a real purpose to allow for incremental
building while preserving the setscene data without wiping out the
existing compile sources (which cleansstate does).

Thanks,
Jason.


-- New in v4 patch set --
   Patch 1 - Fix bug where the LogTee did not have a flush operation
             and caused a python trace
   Patch 13 - Add --no-setscene to avoid running any setscene task
              and force builds via the standard tasks
   Patch 14 - Invalidate setscene tasks if you have built locally
              to support using your build area as a "true scratch area"
   Patch 15 - Add bitbake -i STAMP to invalidate a specific stamp
              and run the do_build target

--  Previous information from v3 patch set --
   Patch 8 - knotty would not flush in a timely manner through |tee
   Patch 9 - knotty2 - fix reporting of set scene tasks
   Patch 10 - knotty2 - fix line wrap issues
   Patch 11 - merge knotty2 functionality into knotty
   Patch 12 - dynamically toggle log locations
   Patch 18 - OE Core specific change to make screen_inline the default 

--  Previous information from v2 patch set --

New in this series:
   Patch 5 - Change to the api for passing the screen connection
   Patch 6 - Fix logging in fetch2
   Patch 7 - Fix fetch2 failures to a read-only GITDIR
   Patch 16/17 - OE Core specific changes related to screen as a reference

-- Previous information from first patch set --

This patch set is intended to provide a semi real time aspect to the
logging to stdout.  Originally I was just interested in the output of
the do_compile and I will still consider adding that capability, but I
ended up using the logger generically.  It provides the ability to see
all the task logs multiplexed to stdout by PID.

More recently I decided I wanted to be able to toggle the logs on and
off while bitbake was already running, so that was added into the
series as well.  After seeing how that worked, I also decided I wanted
to be able to run the dev shell in a single terminal window.  The last
patch in the series is for the OE-Core, but since I made reference to
this patch I included in this series for completeness.

As a foot note, I have actually been using a more primitive version of
this patch set for the last 4 weeks that was controlled only by the
local.conf setting and used it to diagnose quite a few different
problems as well as to just understand what it is that bitbake is
doing at times.

--

Bitbake specific short log:

Jason Wessel (15):
      process.py,build.py: Fix log truncation problems with flush()
      knotty: Extend knotty to support "real time" log tail for tasks
      knotty: Add the ability to dynamically select loglevel from stdin
      msg.py, knotty.py: Allow dynamic toggle of the debug log level
      knotty.py: Add the ability to spawn screen directly from knotty with OE_TERMINAL=screen_inline
      fetch2: Fix missing output from stderr in fetcher logs
      fetch2/git.py: Use local download dir, when using an external read-only GITDIR
      knotty.py, knotty2,py: Flush and update footer on dynamic log level change
      knotty2.py: Fix knotty2 to report something other than 0 of 0 for setscene tasks
      knotty2: Properly adjust for lines longer than terminal size
      knotty: Merge knotty2 functionality into knotty
      knotty: Add ability to dynamically toggle log location
      bitbake,runqueue: Add --no-setscene to skip all setscene tasks
      runqueue.py,build.py: Invalidate setscene tasks based on do_unpack stamps
      bitbake,runqueue.py: Add -i to invalidate a stamp and rebuild the target

 bin/bitbake                      |    5 ++
 lib/bb/build.py                  |   15 +++++-
 lib/bb/fetch2/__init__.py        |    5 +-
 lib/bb/fetch2/git.py             |    5 +-
 lib/bb/msg.py                    |    9 +++-
 lib/bb/process.py                |   29 ++++++++++-
 lib/bb/runqueue.py               |   38 +++++++++++---
 lib/bb/ui/crumbs/multilogtail.py |  135 ++++++++++++++++++++++++++++++++++++++++++++++++
 lib/bb/ui/knotty.py              |  283 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----
 lib/bb/ui/knotty2.py             |   51 +++++++++++++++++--
 lib/bb/ui/uihelper.py            |    5 ++
 11 files changed, 550 insertions(+), 30 deletions(-)
 create mode 100644 lib/bb/ui/crumbs/multilogtail.py

[PATCH v4 01/18] process.py, build.py: Fix log truncation problems with flush()

[Jason Wessel]

There are two problems with the _logged_communicate that are both
caused as a result of buffering I/O, instead of flushing it out to the
log files as it arrives.

1) log truncation when python dumps
   I have seen the task logs missing data that was not flushed when
   bitbake crashes.

2) While a bitbake task is running it is impossible to see what is
   going on if it is only writing a small incremental log that is
   smaller than the buffer, or you get only a partial log, up until
   the task exists.  It is worse in the case that stderr and stdout
   are separate file handles, because previous code blocks on the read
   of stdout and then stderr, serially.

The right approach is simply to use select() to determine if there is
data available and then to flush the log buffers as they arrive.  This
still makes use of buffering in the cases where there is more than 1
byte of data, but the buffers are much more dynamic.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/build.py   |    3 ++-
 lib/bb/process.py |   29 +++++++++++++++++++++++++++--
 2 files changed, 29 insertions(+), 3 deletions(-)

diff --git a/lib/bb/build.py b/lib/bb/build.py
index 95f1dcf..da4c911 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -135,7 +135,8 @@ class LogTee(object):
 
     def __repr__(self):
         return '<LogTee {0}>'.format(self.name)
-
+    def flush(self):
+        self.outfile.flush()
 
 def exec_func(func, d, dirs = None):
     """Execute an BB 'function'"""
diff --git a/lib/bb/process.py b/lib/bb/process.py
index b74cb18..9848fc3 100644
--- a/lib/bb/process.py
+++ b/lib/bb/process.py
@@ -1,6 +1,9 @@
 import logging
 import signal
 import subprocess
+import fcntl
+import errno
+import select
 
 logger = logging.getLogger('BitBake.Process')
 
@@ -70,18 +73,40 @@ def _logged_communicate(pipe, log, input):
 
     bufsize = 512
     outdata, errdata = [], []
+    rin = []
+
+    if pipe.stdout is not None:
+        fd = pipe.stdout.fileno()
+        fl = fcntl.fcntl(fd, fcntl.F_GETFL)
+        fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
+        rin.append(pipe.stdout)
+    if pipe.stderr is not None:
+        fd = pipe.stderr.fileno()
+        fl = fcntl.fcntl(fd, fcntl.F_GETFL)
+        fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
+        rin.append(pipe.stderr)
+
     while pipe.poll() is None:
-        if pipe.stdout is not None:
+        rlist = rin
+        try:
+            r,w,e = select.select (rlist, [], [])
+        except OSError, e:
+            if e.errno != errno.EINTR:
+                raise
+
+        if pipe.stdout in r:
             data = pipe.stdout.read(bufsize)
             if data is not None:
                 outdata.append(data)
                 log.write(data)
+                log.flush()
 
-        if pipe.stderr is not None:
+        if pipe.stderr in r:
             data = pipe.stderr.read(bufsize)
             if data is not None:
                 errdata.append(data)
                 log.write(data)
+                log.flush()
     return ''.join(outdata), ''.join(errdata)
 
 def run(cmd, input=None, log=None, **options):
-- 
1.7.10

[PATCH v4 02/18] knotty: Extend knotty to support "real time" log tail for tasks

[Jason Wessel]

This patch provides the capability to knotty to have a higher log
level where the individual tasks logs can be multiplexed to stdio.
This is highly desirable for several different use cases.

1) You can visually spot "hot spots" in building parallel tasks

2) The relative order an sequence of events can be determined in
   difficult to diagnose inter-related library problems.

3) It provides a nice way to see what all is actually going on and
   getting logged by various tasks such as a kernel compile where you
   might want to actually look at the output in a build over build
   scenario.

4) There are most certainly other aspects where this is useful.

The primary control is to set the BB_RT_LOGLEVEL = "2" in your
local.conf.  Additional capabilities will follow in successive
patches.  When parallel tasks are enabled, the logs will be
multiplexed and prefixed with the PID of the task.  In the
non-parallel case there is no PID prefix as everything is linear.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/build.py                  |    4 ++
 lib/bb/ui/crumbs/multilogtail.py |  135 ++++++++++++++++++++++++++++++++++++++
 lib/bb/ui/knotty.py              |   46 +++++++++++++
 3 files changed, 185 insertions(+)
 create mode 100644 lib/bb/ui/crumbs/multilogtail.py

diff --git a/lib/bb/build.py b/lib/bb/build.py
index da4c911..fb61b00 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -88,6 +88,10 @@ class TaskBase(event.Event):
 class TaskStarted(TaskBase):
     """Task execution started"""
 
+    def __init__(self, t, d):
+        self.logfile = d.getVar("BB_LOGFILE", True) or "";
+        super(TaskStarted, self).__init__(t, d)
+
 class TaskSucceeded(TaskBase):
     """Task execution completed"""
 
diff --git a/lib/bb/ui/crumbs/multilogtail.py b/lib/bb/ui/crumbs/multilogtail.py
new file mode 100644
index 0000000..da8ae68
--- /dev/null
+++ b/lib/bb/ui/crumbs/multilogtail.py
@@ -0,0 +1,135 @@
+#
+# Multipile Log Tail for real time logging
+#
+# Copyright (C) 2012 Wind River Systems, Inc.
+#
+# Authored by Jason Wessel <jason.wessel@windriver.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import re
+import time
+import sys
+
+class LogTail(object):
+    """Tail a single log file"""
+
+    def __init__(self):
+        self.file = None
+        self.offset = 0
+        self.filterLines = True
+        self.limit = 8192
+        self.cnt = 0
+        self.storedLine = ""
+
+    def openLog(self, filename, filterLines=True, limit=8192):
+        try:
+            self.file = open(filename, 'rb')
+            self.limit = limit
+            self.filterLines = filterLines
+        except:
+            return False
+        return True
+
+    def getLeftLine(self):
+        (left, right) = self.storedLine.split('\r',1)
+        self.storedLine = right
+        self.cnt = len(right)
+        return left
+
+    def getLine(self):
+        if self.cnt > 0 and '\r' in self.storedLine:
+            return self.getLeftLine()
+        while 1:
+            self.offset = self.file.tell()
+            line = self.file.readline()
+            if line:
+                if self.filterLines:
+                    line = line.replace('\r\n','\r');
+                    line = line.replace('\n','\r');
+                    self.storedLine += line
+                    self.cnt = len(self.storedLine)
+                    if (self.cnt > self.limit):
+                        if not '\r' in self.storedLine:
+                            self.storedLine += '\r'
+                        return self.getLeftLine()
+                    if '\r' in self.storedLine:
+                        return self.getLeftLine()
+                    continue
+                else:
+                    return line
+            self.file.seek(self.offset)
+            return None
+
+    def closeLog(self):
+        if self.file:
+            self.file.close()
+            self.file = None
+
+    def __del__(self):
+        self.closeLog()
+
+class MultiLogTail(object):
+    """Tail mulitple log files"""
+
+    def __init__(self, printPid=True):
+        self.filelist = []
+        self.trackPid = []
+        self.printPid = printPid
+
+    def openLog(self, filename, pid=0, limit=8192):
+        logtail = LogTail()
+        if logtail.openLog(filename, True, limit):
+            self.filelist.append((logtail, filename, pid))
+        else:
+            if pid > 0:
+                self.trackPid.append(pid)
+
+    def displayLogs(self):
+        for (logtail, filename, pid) in self.filelist:
+            while 1:
+                output = logtail.getLine()
+                if output == None:
+                    break
+                if self.printPid:
+                    print " " + str(pid) + ":" + output
+                else:
+                    print output
+
+    def closeLogs(self):
+        while 1:
+            try:
+                (logtail, filename, pid) = self.filelist.pop()
+                logtail.closeLog()
+            except:
+                break
+
+    def closeLog(self, rm_filename):
+        for (logtail, filename, pid) in self.filelist:
+            if filename == rm_filename:
+                self.filelist.remove((logtail, filename, pid))
+                logtail.closeLog()
+                return
+        raise Exception("File %s not in the list" % rm_filename)
+
+    def closeLogPid(self, rm_pid):
+        for (logtail, filename, pid) in self.filelist:
+            if pid == rm_pid:
+                self.filelist.remove((logtail, filename, pid))
+                logtail.closeLog()
+                return
+        if rm_pid in self.trackPid:
+            self.trackPid.remove(rm_pid)
+            return
+        print "ERROR: PID %s not in the list" % rm_pid
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 0167416..fd86ed5 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -22,11 +22,15 @@ from __future__ import division
 
 import os
 import sys
+import select
+import termios
+import copy
 import xmlrpclib
 import logging
 import progressbar
 import bb.msg
 from bb.ui import uihelper
+from bb.ui.crumbs.multilogtail import MultiLogTail
 
 logger = logging.getLogger("BitBake")
 interactive = sys.stdout.isatty()
@@ -104,6 +108,25 @@ class TerminalFilter(object):
     def finish(self):
         return
 
+class RtLogLevel:
+    def __init__(self, mlt):
+        self.displaytail = False
+        self.mlt = mlt
+
+    def displayLogs(self):
+        if self.displaytail:
+            self.mlt.displayLogs()
+
+    def setLevel(self, input, verbose):
+        if input == "1":
+            if verbose:
+                print "NOTE: Turning off real time log tail"
+            self.displaytail = False
+        elif input == "2":
+            if verbose:
+                print "NOTE: Turning on real time log tail"
+            self.displaytail = True
+
 def main(server, eventHandler, tf = TerminalFilter):
 
     # Get values of variables which control our output
@@ -111,11 +134,22 @@ def main(server, eventHandler, tf = TerminalFilter):
     loglines = server.runCommand(["getVariable", "BBINCLUDELOGS_LINES"])
     consolelogfile = server.runCommand(["getVariable", "BB_CONSOLELOG"])
 
+    # MultiTail affected varialbles
+    numthreads = server.runCommand(["getVariable", "BB_NUMBER_THREADS"])
+    if numthreads is None or numthreads == "1":
+        mlt = MultiLogTail(False)
+    else:
+        mlt = MultiLogTail(True)
+
     helper = uihelper.BBUIHelper()
 
     console = logging.StreamHandler(sys.stdout)
     format = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
     bb.msg.addDefaultlogFilter(console)
+    rtloglevel = RtLogLevel(mlt)
+    bb_rt_loglevel = server.runCommand(["getVariable", "BB_RT_LOGLEVEL"])
+    if bb_rt_loglevel and bb_rt_loglevel != "":
+        rtloglevel.setLevel(bb_rt_loglevel, False)
     console.setFormatter(format)
     logger.addHandler(console)
     if consolelogfile:
@@ -155,11 +189,23 @@ def main(server, eventHandler, tf = TerminalFilter):
         try:
             termfilter.updateFooter()
             event = eventHandler.waitEvent(0.25)
+            # Always try printing any accumulated log files first
+            rtloglevel.displayLogs()
             if event is None:
                 if main.shutdown > 1:
                     break
                 continue
             helper.eventHandler(event)
+            if isinstance(event, bb.build.TaskStarted):
+                mlt.openLog(event.logfile, event.pid)
+                rtloglevel.displayLogs()
+
+            if isinstance(event, bb.build.TaskSucceeded):
+                mlt.closeLogPid(event.pid)
+
+            if isinstance(event, bb.build.TaskFailed):
+                mlt.closeLogPid(event.pid)
+
             if isinstance(event, bb.runqueue.runQueueExitWait):
                 if not main.shutdown:
                     main.shutdown = 1
-- 
1.7.10

[PATCH v4 03/18] knotty: Add the ability to dynamically select loglevel from stdin

[Jason Wessel]

If stdin is a controlling terminal, make it possible to dynamically
select a log level with the keys 1 and 2.

1 = normal bitbake knotty logs
2 = realtime multiple tail of task logs

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/ui/knotty.py |   32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index fd86ed5..f72f0ad 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -108,6 +108,32 @@ class TerminalFilter(object):
     def finish(self):
         return
 
+class StdinMgr:
+    def __init__(self):
+        self.stdinbackup = None
+        self.fd = None
+        if sys.stdin.isatty():
+            self.fd = sys.stdin.fileno()
+            self.stdinbackup = termios.tcgetattr(self.fd)
+            new = termios.tcgetattr(self.fd)
+            new[3] = new[3] & ~termios.ICANON & ~termios.ECHO
+            termios.tcsetattr(self.fd, termios.TCSANOW, new)
+
+    def poll(self):
+        if not self.stdinbackup:
+            return False
+        return select.select([sys.stdin], [], [], 0) == ([sys.stdin], [], [])
+
+    def restore(self):
+        if self.stdinbackup:
+            termios.tcsetattr(self.fd, termios.TCSADRAIN, self.stdinbackup)
+            self.stdinbackup = None
+        # Force echo back on "just in case" something went haywire with an exception
+        if sys.stdin.isatty():
+            new = termios.tcgetattr(self.fd)
+            new[3] = new[3] | termios.ECHO
+            termios.tcsetattr(self.fd, termios.TCSANOW, new)
+
 class RtLogLevel:
     def __init__(self, mlt):
         self.displaytail = False
@@ -184,11 +210,15 @@ def main(server, eventHandler, tf = TerminalFilter):
     taskfailures = []
 
     termfilter = tf(main, helper, console, format)
+    stdin_mgr = StdinMgr()
 
     while True:
         try:
             termfilter.updateFooter()
             event = eventHandler.waitEvent(0.25)
+            if stdin_mgr.poll():
+                keyinput = sys.stdin.read(1)
+                rtloglevel.setLevel(keyinput, True)
             # Always try printing any accumulated log files first
             rtloglevel.displayLogs()
             if event is None:
@@ -364,6 +394,7 @@ def main(server, eventHandler, tf = TerminalFilter):
             if ioerror.args[0] == 4:
                 pass
         except KeyboardInterrupt:
+            stdin_mgr.restore()
             termfilter.clearFooter()
             if main.shutdown == 1:
                 print("\nSecond Keyboard Interrupt, stopping...\n")
@@ -375,6 +406,7 @@ def main(server, eventHandler, tf = TerminalFilter):
             main.shutdown = main.shutdown + 1
             pass
 
+    stdin_mgr.restore()
     summary = ""
     if taskfailures:
         summary += pluralise("\nSummary: %s task failed:",
-- 
1.7.10

[PATCH v4 04/18] msg.py, knotty.py: Allow dynamic toggle of the debug log level

[Jason Wessel]

If stdin is a controlling tty, make it possible to have
log levels 1-4 where you can press a key to dynamically
change the log level, while bitbake is running to see
what is going on.

1 = default log level
2 = multiple log tail of bitbake tasks
3 = bit bake debug messages
4 = bitbake debug messages + multiple lot tail of bitbake tasks

In order to make this possible a slight modification to
the msg logfilter was needed to make it possible to get
an object so as to change the filter log level.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/msg.py       |    9 ++++++++-
 lib/bb/ui/knotty.py |   21 ++++++++++++++++++---
 2 files changed, 26 insertions(+), 4 deletions(-)

diff --git a/lib/bb/msg.py b/lib/bb/msg.py
index 9b39325..4b2f90e 100644
--- a/lib/bb/msg.py
+++ b/lib/bb/msg.py
@@ -86,6 +86,13 @@ class BBLogFilter(object):
         handler.setLevel(loglevel)
         handler.addFilter(self)
 
+    def setFiltLevel(self, handler, level):
+        self.stdlevel = level
+        handler.setLevel(level)
+
+    def getFiltLevel(self):
+        return self.stdlevel
+
     def filter(self, record):
         if record.levelno >= self.stdlevel:
             return True
@@ -131,7 +138,7 @@ def addDefaultlogFilter(handler):
         dlevel = len(tuple(iterator))
         debug_domains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
 
-    BBLogFilter(handler, level, debug_domains)
+    return BBLogFilter(handler, level, debug_domains)
 
 #
 # Message handling functions
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index f72f0ad..7a7a1b2 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -135,8 +135,11 @@ class StdinMgr:
             termios.tcsetattr(self.fd, termios.TCSANOW, new)
 
 class RtLogLevel:
-    def __init__(self, mlt):
+    def __init__(self, handler, logfilter, mlt):
         self.displaytail = False
+        self.handler = handler
+        self.logfilter = logfilter
+        self.defaultLevel = logfilter.getFiltLevel()
         self.mlt = mlt
 
     def displayLogs(self):
@@ -147,10 +150,22 @@ class RtLogLevel:
         if input == "1":
             if verbose:
                 print "NOTE: Turning off real time log tail"
+            self.logfilter.setFiltLevel(self.handler, self.defaultLevel)
             self.displaytail = False
         elif input == "2":
             if verbose:
                 print "NOTE: Turning on real time log tail"
+            self.logfilter.setFiltLevel(self.handler, self.defaultLevel)
+            self.displaytail = True
+        elif input == "3":
+            if verbose:
+                print "NOTE: Turning on DEBUG logging"
+            self.logfilter.setFiltLevel(self.handler, logging.DEBUG)
+            self.displaytail = False
+        elif input == "4":
+            if verbose:
+                print "NOTE: Turning on DEBUG logging + real time log tail"
+            self.logfilter.setFiltLevel(self.handler, logging.DEBUG)
             self.displaytail = True
 
 def main(server, eventHandler, tf = TerminalFilter):
@@ -171,8 +186,8 @@ def main(server, eventHandler, tf = TerminalFilter):
 
     console = logging.StreamHandler(sys.stdout)
     format = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
-    bb.msg.addDefaultlogFilter(console)
-    rtloglevel = RtLogLevel(mlt)
+    logfilter = bb.msg.addDefaultlogFilter(console)
+    rtloglevel = RtLogLevel(console, logfilter, mlt)
     bb_rt_loglevel = server.runCommand(["getVariable", "BB_RT_LOGLEVEL"])
     if bb_rt_loglevel and bb_rt_loglevel != "":
         rtloglevel.setLevel(bb_rt_loglevel, False)
-- 
1.7.10

[PATCH v4 05/18] knotty.py: Add the ability to spawn screen directly from knotty with OE_TERMINAL=screen_inline

[Jason Wessel]

The oe-core has a change pending to add the screen_inline class.

This change allows the simple use case of directly using a devshell
from the controlling terminal that started bitbake which is a
significant convience for some use cases.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/ui/knotty.py |   10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 7a7a1b2..b4125f6 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -174,6 +174,10 @@ def main(server, eventHandler, tf = TerminalFilter):
     includelogs = server.runCommand(["getVariable", "BBINCLUDELOGS"])
     loglines = server.runCommand(["getVariable", "BBINCLUDELOGS_LINES"])
     consolelogfile = server.runCommand(["getVariable", "BB_CONSOLELOG"])
+    if sys.stdin.isatty() and sys.stdout.isatty():
+        runscreen = True
+    else:
+        runscreen = False
 
     # MultiTail affected varialbles
     numthreads = server.runCommand(["getVariable", "BB_NUMBER_THREADS"])
@@ -260,6 +264,12 @@ def main(server, eventHandler, tf = TerminalFilter):
                     errors = errors + 1
                     return_value = 1
                 elif event.levelno == format.WARNING:
+                    msg = event.getMessage()
+                    if runscreen and event.name == "BitBake.OE.Terminal" and msg.startswith("Screen Connect Command: "):
+                        msg = msg[24:]
+                        print "Starting terminal with: %s" % msg
+                        os.system(msg)
+                        continue
                     warnings = warnings + 1
                 # For "normal" logging conditions, don't show note logs from tasks
                 # but do show them if the user has changed the default log level to 
-- 
1.7.10

[PATCH v4 06/18] fetch2: Fix missing output from stderr in fetcher logs

[Jason Wessel]

There are actually two problems to fix

1) The exception for bb.process.ExecutionError must be processed first
   because it is a derived from the bb.process.CmdError class and
   we never reach the ExecutionError otherwise.

2) The stderr needs to be printed as well as stdout to determine
   the root cause of a fetch failure.

The example I have is that I got a log that looked like:

--
ERROR: Function failed: Network access disabled through
  BB_NO_NETWORK but access requested with command
   /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P
   /localds 'http://downloads.yoctoproject.org/[...CLIPPED...] url None)
--

That really didn't tell me much, but with this patch I get error above
plus the following:

--
STDERR: /net/[...CLIPPED...]kernel-tools.git: Read-only file system
--

Having the ability to see that the build was trying to write to a
read-only file system was a good clue that the fetcher is broken in
some other way to be fixed in a follow on patch, but let us fix the
logging problem first.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/fetch2/__init__.py |    5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 3391e6a..ba562a8 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -421,11 +421,10 @@ def runfetchcmd(cmd, d, quiet = False, cleanup = []):
         success = True
     except bb.process.NotFoundError as e:
         error_message = "Fetch command %s" % (e.command)
+    except bb.process.ExecutionError as e:
+        error_message = "Fetch command %s failed with exit code %s, output:\nSTDOUT: %s\nSTDERR: %s" % (e.command, e.exitcode, e.stdout, e.stderr)
     except bb.process.CmdError as e:
         error_message = "Fetch command %s could not be run:\n%s" % (e.command, e.msg)
-    except bb.process.ExecutionError as e:
-        error_message = "Fetch command %s failed with exit code %s, output:\n%s" % (e.command, e.exitcode, e.stderr)
-
     if not success:
         for f in cleanup:
             try:
-- 
1.7.10

[PATCH v4 07/18] fetch2/git.py: Use local download dir, when using an external read-only GITDIR

[Jason Wessel]

When building the yocto kernel or yocto kernel tools, if the GITDIR is
on a read-only server that is shared and updated by via some cron
jobs, the fetch2 should not try to write to that area.  The end result
will be odd failures if you have download turned off, or extra
needless downloading if you have download turned on.

The fetch2 should prefer to create mirror the mirror clones in the
local "downloads" directory in the work space.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/fetch2/git.py |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index bcc0da5..59d95cb 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -136,8 +136,11 @@ class Git(FetchMethod):
                 gitsrcname = gitsrcname + '_' + ud.revisions[name]
         ud.mirrortarball = 'git2_%s.tar.gz' % (gitsrcname)
         ud.fullmirror = os.path.join(data.getVar("DL_DIR", d, True), ud.mirrortarball)
-        ud.clonedir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
+        ud.gitdir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
+        ud.clonedir = os.path.join(data.getVar("DL_DIR", d, True), gitsrcname)
 
+        if os.path.exists(ud.gitdir):
+            ud.clonedir = ud.gitdir
         ud.localfile = ud.clonedir
 
     def localpath(self, url, ud, d):
-- 
1.7.10

[PATCH v4 08/18] knotty.py, knotty2, py: Flush and update footer on dynamic log level change

[Jason Wessel]

This patch adds a call to update the task list after changing log
modes dynamically such that you can immediately see the remaining
tasks.  After adding this functionality, it became obvious there was a
problem flushing data loaded into stdout because the task list would
not appear immediately.  Doing something like "bitbake IMAGE | tee
/tmp/log" is where the flush problem manifests itself the most.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/ui/knotty.py  |   14 ++++++++++++--
 lib/bb/ui/knotty2.py |    4 ++++
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index b4125f6..d7b1e19 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -82,9 +82,16 @@ class TerminalFilter(object):
     def clearFooter(self):
         return
 
+    def updateFooterForce(self):
+        self.printFooter(True)
+
     def updateFooter(self):
-        if not main.shutdown or not self.helper.needUpdate:
-            return
+        self.printFooter(False)
+
+    def printFooter(self, force_update):
+        if not force_update:
+            if not main.shutdown or not self.helper.needUpdate:
+                return
 
         activetasks = self.helper.running_tasks
         runningpids = self.helper.running_pids
@@ -238,6 +245,9 @@ def main(server, eventHandler, tf = TerminalFilter):
             if stdin_mgr.poll():
                 keyinput = sys.stdin.read(1)
                 rtloglevel.setLevel(keyinput, True)
+                termfilter.updateFooterForce()
+                sys.stdout.flush()
+
             # Always try printing any accumulated log files first
             rtloglevel.displayLogs()
             if event is None:
diff --git a/lib/bb/ui/knotty2.py b/lib/bb/ui/knotty2.py
index aa6a408..01693e9 100644
--- a/lib/bb/ui/knotty2.py
+++ b/lib/bb/ui/knotty2.py
@@ -73,6 +73,10 @@ class TerminalFilter2(object):
             sys.stdout.write(self.curses.tparm(self.ed))
         self.footer_present = False
 
+    def updateFooterForce(self):
+        self.footer_present = False
+        self.updateFooter()
+
     def updateFooter(self):
         if not self.cuu:
             return
-- 
1.7.10

[PATCH v4 09/18] knotty2.py: Fix knotty2 to report something other than 0 of 0 for setscene tasks

[Jason Wessel]

Any time knotty2 was running through the setscene tasks it was always
reporting the top line as follows for a core-image-minimal build.

    Currently 7 running tasks (0 of 0):

With this patch the number of SetScene tasks are tracked and reported
with knotty such that it nows looks something like:

    Currently 7 running SetScene tasks (342 of 479):

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/ui/knotty2.py  |    5 ++++-
 lib/bb/ui/uihelper.py |    5 +++++
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/lib/bb/ui/knotty2.py b/lib/bb/ui/knotty2.py
index 01693e9..9e3619b 100644
--- a/lib/bb/ui/knotty2.py
+++ b/lib/bb/ui/knotty2.py
@@ -97,7 +97,10 @@ class TerminalFilter2(object):
         if self.main.shutdown:
             print("Waiting for %s running tasks to finish:" % len(activetasks))
         else:
-            print("Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total))
+            if self.helper.tasknumber_current == 0:
+                print("Currently %s running SetScene tasks (%s of %s):" % (len(activetasks), self.helper.sc_tasknumber_current, self.helper.sc_tasknumber_total))
+            else:
+                print("Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total))
         for tasknum, task in enumerate(tasks):
             print("%s: %s" % (tasknum, task))
             lines = lines + 1
diff --git a/lib/bb/ui/uihelper.py b/lib/bb/ui/uihelper.py
index 2c78695..84af5a3 100644
--- a/lib/bb/ui/uihelper.py
+++ b/lib/bb/ui/uihelper.py
@@ -28,6 +28,8 @@ class BBUIHelper:
         self.failed_tasks = []
         self.tasknumber_current = 0
         self.tasknumber_total = 0
+        self.sc_tasknumber_current = 0
+        self.sc_tasknumber_current = 0
 
     def eventHandler(self, event):
         if isinstance(event, bb.build.TaskStarted):
@@ -48,6 +50,9 @@ class BBUIHelper:
             self.running_pids.remove(event.pid)
             self.failed_tasks.append( { 'title' : "%s %s" % (event._package, event._task)})
             self.needUpdate = True
+        if isinstance(event, bb.runqueue.sceneQueueTaskStarted):
+            self.sc_tasknumber_current = event.stats.completed + event.stats.active + event.stats.failed + 1
+            self.sc_tasknumber_total = event.stats.total
         if isinstance(event, bb.runqueue.runQueueTaskStarted):
             self.tasknumber_current = event.stats.completed + event.stats.active + event.stats.failed + 1
             self.tasknumber_total = event.stats.total
-- 
1.7.10

[PATCH v4 10/18] knotty2: Properly adjust for lines longer than terminal size

[Jason Wessel]

knotty2 does not handle being resized nor does it work properly
if a task line is longer than the terminal size.

The filter handler relies on the sigwinch to properly
handle a terminal resize.  The cache and parse tasks will
try to take over the sigwinch handler, so it must be restored
when either of these tasks completes.
---
 lib/bb/ui/knotty.py  |    5 +++++
 lib/bb/ui/knotty2.py |   42 +++++++++++++++++++++++++++++++++++++++---
 2 files changed, 44 insertions(+), 3 deletions(-)

diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index d7b1e19..58fb8f1 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -82,6 +82,9 @@ class TerminalFilter(object):
     def clearFooter(self):
         return
 
+    def sigUpdate(self):
+        return
+
     def updateFooterForce(self):
         self.printFooter(True)
 
@@ -332,6 +335,7 @@ def main(server, eventHandler, tf = TerminalFilter):
                 parseprogress.finish()
                 print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors."
                     % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors)))
+                termfilter.sigUpdate()
                 continue
 
             if isinstance(event, bb.event.CacheLoadStarted):
@@ -343,6 +347,7 @@ def main(server, eventHandler, tf = TerminalFilter):
             if isinstance(event, bb.event.CacheLoadCompleted):
                 cacheprogress.finish()
                 print("Loaded %d entries from dependency cache." % event.num_entries)
+                termfilter.sigUpdate()
                 continue
 
             if isinstance(event, bb.command.CommandFailed):
diff --git a/lib/bb/ui/knotty2.py b/lib/bb/ui/knotty2.py
index 9e3619b..cfa4bf7 100644
--- a/lib/bb/ui/knotty2.py
+++ b/lib/bb/ui/knotty2.py
@@ -35,6 +35,39 @@ class InteractConsoleLogFilter(logging.Filter):
         return True
 
 class TerminalFilter2(object):
+    columns = 80
+
+    def sigwinch_handle(self, sig, data):
+        self.columns = self.getTerminalColumns()
+
+    def sigUpdate(self):
+        import signal
+        if self.interactive and self.cuu:
+            signal.signal(signal.SIGWINCH, self.sigwinch_handle)
+
+    def getTerminalColumns(self):
+        def ioctl_GWINSZ(fd):
+            try:
+                import fcntl, termios, struct, os
+                cr = struct.unpack('hh', fcntl.ioctl(fd, termios.TIOCGWINSZ, '1234'))
+            except:
+                return None
+            return cr
+        cr = ioctl_GWINSZ(0) or ioctl_GWINSZ(1) or ioctl_GWINSZ(2)
+        if not cr:
+            try:
+                fd = os.open(os.ctermid(), os.O_RDONLY)
+                cr = ioctl_GWINSZ(fd)
+                os.close(fd)
+            except:
+                pass
+        if not cr:
+            try:
+                cr = (env['LINES'], env['COLUMNS'])
+            except:
+                cr = (25, 80)
+        return cr[1]
+
     def __init__(self, main, helper, console, format):
         self.main = main
         self.helper = helper
@@ -62,7 +95,9 @@ class TerminalFilter2(object):
             self.ed = curses.tigetstr("ed")
             if self.ed:
                 self.cuu = curses.tigetstr("cuu")
-        except:
+            self.columns = self.getTerminalColumns()
+            self.sigUpdate()
+        except Exception:
             self.cuu = None
         console.addFilter(InteractConsoleLogFilter(self, format))
 
@@ -102,8 +137,9 @@ class TerminalFilter2(object):
             else:
                 print("Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total))
         for tasknum, task in enumerate(tasks):
-            print("%s: %s" % (tasknum, task))
-            lines = lines + 1
+            content = "%s: %s" % (tasknum, task)
+            print content
+            lines = lines + 1 + int(len(content) / (self.columns + 1))
         self.footer_present = lines
         self.lastpids = runningpids[:]
 
-- 
1.7.10

[PATCH v4 11/18] knotty: Merge knotty2 functionality into knotty

[Jason Wessel]

The knotty 2 functionality can be activated in realtime
by pressing "t" for top mode or reverted with "0" for
normal mode.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/ui/knotty.py |  172 ++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 150 insertions(+), 22 deletions(-)

diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 58fb8f1..765ad27 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -74,50 +74,156 @@ def pluralise(singular, plural, qty):
     else:
         return plural % qty
 
+class InteractConsoleLogFilter(logging.Filter):
+    def __init__(self, tf, format):
+        self.tf = tf
+        self.format = format
+
+    def filter(self, record):
+        if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("package ")):
+            return False
+        self.tf.clearFooter()
+        return True
+
 class TerminalFilter(object):
     def __init__(self, main, helper, console, format):
+        import curses
         self.main = main
+        self.console = console
         self.helper = helper
+        self.topMode = False
+        self.cuu = None
+        self.interactive = sys.stdout.isatty()
+        self.footer_present = False
+        self.lastpids = []
+        self.curses = curses
+        self.confilter = InteractConsoleLogFilter(self, format)
+        self.curses_init = False
+        self.columns = 80
+
+    def cursesInit(self):
+        if self.curses_init:
+            return
+        self.curses_init = True
+        if self.interactive:
+            try:
+                self.curses.setupterm()
+                self.ed = self.curses.tigetstr("ed")
+                if self.ed:
+                    self.cuu = self.curses.tigetstr("cuu")
+                self.columns = self.getTerminalColumns()
+                self.sigUpdate()
+            except Exception as e:
+                self.cuu = None
+
+    def setTopMode(self):
+        if self.topMode:
+            return
+        self.cursesInit()
+        self.topMode = True
+        self.console.addFilter(self.confilter)
+
+    def setNormalMode(self):
+        if not self.topMode:
+            return
+        self.topMode = False
+        try:
+            self.console.removeFilter(self.confilter)
+        except:
+            pass
 
     def clearFooter(self):
-        return
+        if not self.topMode:
+            return
+        if self.footer_present:
+            lines = self.footer_present
+            sys.stdout.write(self.curses.tparm(self.cuu, lines))
+            sys.stdout.write(self.curses.tparm(self.ed))
+        self.footer_present = False
 
-    def sigUpdate(self):
-        return
 
     def updateFooterForce(self):
+        self.footer_present = False
         self.printFooter(True)
 
     def updateFooter(self):
         self.printFooter(False)
 
     def printFooter(self, force_update):
-        if not force_update:
-            if not main.shutdown or not self.helper.needUpdate:
-                return
-
         activetasks = self.helper.running_tasks
+        failedtasks = self.helper.failed_tasks
         runningpids = self.helper.running_pids
+        if self.topMode:
+            if not self.cuu:
+                return
+            if self.footer_present and (self.lastpids == runningpids):
+                return
+            if self.footer_present:
+                self.clearFooter()
+        else:
+            if not force_update:
+                if not main.shutdown or not self.helper.needUpdate:
+                    return
+            if len(runningpids) == 0:
+                return
+            self.helper.getTasks()
 
-        if len(runningpids) == 0:
+        if not activetasks:
             return
-
-        self.helper.getTasks()
-
+        lines = 1
         tasks = []
         for t in runningpids:
             tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
 
-        if main.shutdown:
+        if self.main.shutdown:
             print("Waiting for %s running tasks to finish:" % len(activetasks))
         else:
-            print("Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total))
+            if self.helper.tasknumber_current == 0:
+                print("Currently %s running SetScene tasks (%s of %s):" % (len(activetasks), self.helper.sc_tasknumber_current, self.helper.sc_tasknumber_total))
+            else:
+                print("Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total))
         for tasknum, task in enumerate(tasks):
-            print("%s: %s" % (tasknum, task))
+            content = "%s: %s" % (tasknum, task)
+            print content
+            lines = lines + 1 + int(len(content) / (self.columns + 1))
+        self.footer_present = lines
+        self.lastpids = runningpids[:]
 
     def finish(self):
         return
 
+    def sigwinch_handle(self, sig, data):
+        self.columns = self.getTerminalColumns()
+
+    def sigUpdate(self):
+        import signal
+        if self.interactive and self.cuu:
+            signal.signal(signal.SIGWINCH, self.sigwinch_handle)
+
+    def getTerminalColumns(self):
+        def ioctl_GWINSZ(fd):
+            try:
+                import fcntl, termios, struct, os
+                cr = struct.unpack('hh', fcntl.ioctl(fd, termios.TIOCGWINSZ, '1234'))
+            except:
+                return None
+            return cr
+        cr = ioctl_GWINSZ(0) or ioctl_GWINSZ(1) or ioctl_GWINSZ(2)
+        if not cr:
+            try:
+                fd = os.open(os.ctermid(), os.O_RDONLY)
+                cr = ioctl_GWINSZ(fd)
+                os.close(fd)
+            except:
+                pass
+        if not cr:
+            try:
+                cr = (env['LINES'], env['COLUMNS'])
+            except:
+                cr = (25, 80)
+        return cr[1]
+
+
 class StdinMgr:
     def __init__(self):
         self.stdinbackup = None
@@ -145,23 +251,26 @@ class StdinMgr:
             termios.tcsetattr(self.fd, termios.TCSANOW, new)
 
 class RtLogLevel:
-    def __init__(self, handler, logfilter, mlt):
+    def __init__(self, handler, logfilter, mlt, tf):
         self.displaytail = False
         self.handler = handler
         self.logfilter = logfilter
         self.defaultLevel = logfilter.getFiltLevel()
         self.mlt = mlt
+        self.tf = tf
 
     def displayLogs(self):
         if self.displaytail:
             self.mlt.displayLogs()
 
     def setLevel(self, input, verbose):
-        if input == "1":
+        if input == "1" or input == "0":
             if verbose:
                 print "NOTE: Turning off real time log tail"
             self.logfilter.setFiltLevel(self.handler, self.defaultLevel)
             self.displaytail = False
+            if input == "0" and isinstance(self.tf, TerminalFilter):
+                self.tf.setNormalMode()
         elif input == "2":
             if verbose:
                 print "NOTE: Turning on real time log tail"
@@ -177,6 +286,23 @@ class RtLogLevel:
                 print "NOTE: Turning on DEBUG logging + real time log tail"
             self.logfilter.setFiltLevel(self.handler, logging.DEBUG)
             self.displaytail = True
+        elif input == "t":
+            if verbose:
+                print "NOTE: Activing task \"top\" mode"
+            if isinstance(self.tf, TerminalFilter):
+                self.tf.setTopMode()
+        elif input == "h":
+            print "============================================="
+            print "Interaction help commands:"
+            print " 0 - default with default logs"
+            print " 1 - turn off real time log tail"
+            print " 2 - turn on real time log tail"
+            print " 3 - turn on debug logging"
+            print " 4 - turn on debug logging and real time log tail"
+            print " t - Display tasks in \"top\" mode (formerly knotty2 mode)"
+            print " h - display commands"
+            return False
+        return True
 
 def main(server, eventHandler, tf = TerminalFilter):
 
@@ -201,10 +327,6 @@ def main(server, eventHandler, tf = TerminalFilter):
     console = logging.StreamHandler(sys.stdout)
     format = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
     logfilter = bb.msg.addDefaultlogFilter(console)
-    rtloglevel = RtLogLevel(console, logfilter, mlt)
-    bb_rt_loglevel = server.runCommand(["getVariable", "BB_RT_LOGLEVEL"])
-    if bb_rt_loglevel and bb_rt_loglevel != "":
-        rtloglevel.setLevel(bb_rt_loglevel, False)
     console.setFormatter(format)
     logger.addHandler(console)
     if consolelogfile:
@@ -241,14 +363,20 @@ def main(server, eventHandler, tf = TerminalFilter):
     termfilter = tf(main, helper, console, format)
     stdin_mgr = StdinMgr()
 
+    rtloglevel = RtLogLevel(console, logfilter, mlt, termfilter)
+    bb_rt_loglevel = server.runCommand(["getVariable", "BB_RT_LOGLEVEL"])
+    if bb_rt_loglevel and bb_rt_loglevel != "":
+        for inputkey in bb_rt_loglevel:
+            rtloglevel.setLevel(inputkey, False)
     while True:
         try:
             termfilter.updateFooter()
             event = eventHandler.waitEvent(0.25)
             if stdin_mgr.poll():
                 keyinput = sys.stdin.read(1)
-                rtloglevel.setLevel(keyinput, True)
-                termfilter.updateFooterForce()
+                termfilter.clearFooter()
+                if (rtloglevel.setLevel(keyinput, True)):
+                    termfilter.updateFooterForce()
                 sys.stdout.flush()
 
             # Always try printing any accumulated log files first
-- 
1.7.10

[PATCH v4 12/18] knotty: Add ability to dynamically toggle log location

[Jason Wessel]

Add the ability to see where all the per task logs
were written to disk.

The resulting output looks like where [...] is the full path:

NOTE: LOG: [...]/python-2.7.2-r2.20/temp/log.do_package_setscene.8719
NOTE: LOG: [...]/perl-5.14.2-r7/temp/log.do_package_setscene.8760

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/ui/knotty.py |   13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 765ad27..e485c7e 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -253,6 +253,7 @@ class StdinMgr:
 class RtLogLevel:
     def __init__(self, handler, logfilter, mlt, tf):
         self.displaytail = False
+        self.displayLogLocations = False
         self.handler = handler
         self.logfilter = logfilter
         self.defaultLevel = logfilter.getFiltLevel()
@@ -291,6 +292,14 @@ class RtLogLevel:
                 print "NOTE: Activing task \"top\" mode"
             if isinstance(self.tf, TerminalFilter):
                 self.tf.setTopMode()
+        elif input == "l":
+            if verbose:
+                print "NOTE: Activating log locations display"
+            self.displayLogLocations = True
+        elif input == "L":
+            if verbose:
+                print "NOTE: Disable log locations display"
+            self.displayLogLocations = False
         elif input == "h":
             print "============================================="
             print "Interaction help commands:"
@@ -300,6 +309,7 @@ class RtLogLevel:
             print " 3 - turn on debug logging"
             print " 4 - turn on debug logging and real time log tail"
             print " t - Display tasks in \"top\" mode (formerly knotty2 mode)"
+            print " l - emit log locations (L to turn off)"
             print " h - display commands"
             return False
         return True
@@ -387,6 +397,9 @@ def main(server, eventHandler, tf = TerminalFilter):
                 continue
             helper.eventHandler(event)
             if isinstance(event, bb.build.TaskStarted):
+                if (rtloglevel.displayLogLocations):
+                    termfilter.clearFooter()
+                    print "NOTE: LOG: %s" % event.logfile
                 mlt.openLog(event.logfile, event.pid)
                 rtloglevel.displayLogs()
 
-- 
1.7.10

[PATCH v4 13/18] bitbake, runqueue: Add --no-setscene to skip all setscene tasks

[Jason Wessel]

Mainly intended for the purpose of debugging or forcing builds
from source, the --no-setscene will prevent any setscene
tasks from running.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 bin/bitbake        |    2 ++
 lib/bb/runqueue.py |   13 +++++++------
 2 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/bin/bitbake b/bin/bitbake
index 420e69d..702d99c 100755
--- a/bin/bitbake
+++ b/bin/bitbake
@@ -170,6 +170,8 @@ Default BBFILES are the .bb files in the current directory.""")
 
     parser.add_option("-B", "--bind", help = "The name/address for the bitbake server to bind to",
                action = "store", dest = "bind", default = False)
+    parser.add_option("", "--no-setscene", help = "Do not run any setscene tasks, forces builds",
+               action = "store_true", dest = "nosetscene", default = False)
     options, args = parser.parse_args(sys.argv)
 
     configuration = BBConfiguration(options)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index b870caf..da3fdf9 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -699,13 +699,14 @@ class RunQueueData:
 
         # Interate over the task list looking for tasks with a 'setscene' function
         self.runq_setscene = []
-        for task in range(len(self.runq_fnid)):
-            setscene = taskData.gettask_id(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task] + "_setscene", False)
-            if not setscene:
-                continue
-            self.runq_setscene.append(task)
+        if not self.cooker.configuration.nosetscene:
+            for task in range(len(self.runq_fnid)):
+                setscene = taskData.gettask_id(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task] + "_setscene", False)
+                if not setscene:
+                    continue
+                self.runq_setscene.append(task)
 
-        # Interate over the task list and call into the siggen code
+        # Iterate over the task list and call into the siggen code
         dealtwith = set()
         todeal = set(range(len(self.runq_fnid)))
         while len(todeal) > 0:
-- 
1.7.10

[PATCH v4 14/18] runqueue.py, build.py: Invalidate setscene tasks based on do_unpack stamps

[Jason Wessel]

If you have a fully populated sstate cache and have used it to
execute a build, it is not possible to invalidate repackage
an intermediate build after you have forced a compiled

Example when you have build from a complete sstate cache build:
  bitbake core-image-sato
  bitbake -c patch acl
     *** Make some changes to the C files for experimentation.
  bitbake -f -c compile acl
  bitbake acl

The bitbake will refuse to build the acl package at this
point and instead keep populating it from the sstate.  Using
the cleanstate is no longer a good option because it will
also erase the scratch area.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/build.py    |    8 ++++++++
 lib/bb/runqueue.py |   14 ++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/lib/bb/build.py b/lib/bb/build.py
index fb61b00..18c28aa 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -463,6 +463,14 @@ def del_stamp(task, d, file_name = None):
     stamp = stamp_internal(task, d, file_name)
     bb.utils.remove(stamp)
 
+def exists_stamp(task, d, file_name = None):
+    """
+    Removes a stamp for a given task
+    (d can be a data dict or dataCache)
+    """
+    stamp = stamp_internal(task, d, file_name)
+    return os.path.exists(stamp)
+
 def stampfile(taskname, d, file_name = None):
     """
     Return the stamp for a given task
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index da3fdf9..6c802af 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -718,6 +718,20 @@ class RunQueueData:
                     for dep in self.runq_depends[task]:
                         procdep.append(self.taskData.fn_index[self.runq_fnid[dep]] + "." + self.runq_task[dep])
                     self.runq_hash[task] = bb.parse.siggen.get_taskhash(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], procdep, self.dataCache)
+        try:
+            new_setscene = []
+            for task in self.runq_setscene:
+                try:
+                    fn = self.taskData.fn_index[self.rq.rqdata.runq_fnid[task]]
+                    if bb.build.exists_stamp("do_unpack", self.dataCache, fn):
+                        logger.debug(2, 'Removing task %s from queue because do_unpack exists', task)
+                    else:
+                        new_setscene.append(task)
+                except:
+                    logger.debug(2, 'Failed do_unpack check for %s', task)
+            self.runq_setscene = new_setscene
+        except:
+            logger.debug(2, 'Failed to update runq_setscene')
 
         self.hashes = {}
         self.hash_deps = {}
-- 
1.7.10

[PATCH v4 15/18] bitbake, runqueue.py: Add -i to invalidate a stamp and rebuild the target

[Jason Wessel]

It is highly desirable to be able to invalidate a stamp used for
do_compile and to be able to continue on to build the entire
package's build rule.

If invalidating a stamp, the setscene rules for the specified targets
should be automatically canceled so as to allow a forced build
of an individual target.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 bin/bitbake        |    3 +++
 lib/bb/runqueue.py |   13 ++++++++++++-
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/bin/bitbake b/bin/bitbake
index 702d99c..bf81a0c 100755
--- a/bin/bitbake
+++ b/bin/bitbake
@@ -150,6 +150,9 @@ Default BBFILES are the .bb files in the current directory.""")
     parser.add_option("-I", "--ignore-deps", help = """Assume these dependencies don't exist and are already provided (equivalent to ASSUME_PROVIDED). Useful to make dependency graphs more appealing""",
                 action = "append", dest = "extra_assume_provided", default = [])
 
+    parser.add_option("-i", "--invalidate-stamp", help = "Invalidate the specified stamp for a command such as 'compile' and build",
+                action = "store", dest = "invalidate_stamp")
+
     parser.add_option("-l", "--log-domains", help = """Show debug logging for the specified logging domains""",
                 action = "append", dest = "debug_domains", default = [])
 
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 6c802af..3dbfaad 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -720,10 +720,14 @@ class RunQueueData:
                     self.runq_hash[task] = bb.parse.siggen.get_taskhash(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], procdep, self.dataCache)
         try:
             new_setscene = []
+            test_fns = []
+            if self.cooker.configuration.invalidate_stamp:
+                for tgt in self.target_pairs:
+                    test_fns.append(tgt[0])
             for task in self.runq_setscene:
                 try:
                     fn = self.taskData.fn_index[self.rq.rqdata.runq_fnid[task]]
-                    if bb.build.exists_stamp("do_unpack", self.dataCache, fn):
+                    if bb.build.exists_stamp("do_unpack", self.dataCache, fn) or (self.cooker.configuration.invalidate_stamp and fn in test_fns):
                         logger.debug(2, 'Removing task %s from queue because do_unpack exists', task)
                     else:
                         new_setscene.append(task)
@@ -746,6 +750,13 @@ class RunQueueData:
                 deps.append(depidentifier)
             self.hash_deps[identifier] = deps
 
+        # Remove stamps for forced invalidation
+        if self.cooker.configuration.invalidate_stamp:
+            for (fn, target) in self.target_pairs:
+                for st in self.cooker.configuration.invalidate_stamp.split(','):
+                    logger.verbose("Remove stamp %s, %s", st, fn)
+                    bb.build.del_stamp("do_%s" % st, self.dataCache, fn)
+
         # Remove stamps for targets if force mode active
         if self.cooker.configuration.force:
             for (fn, target) in self.target_pairs:
-- 
1.7.10

[PATCH v4 16/18] meta/lib/oe/terminal.py: Add an additional reference to screen called screen_inline

[Jason Wessel]

This is a new mode for screen with the explicit purpose of invoking
screen on the same terminal that you started bitbake.  The bitbake knotty
will attempt to launch screen directly when using this setting for
OE_TERMINAL.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 meta/lib/oe/terminal.py |    7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/meta/lib/oe/terminal.py b/meta/lib/oe/terminal.py
index 43639d5..ca94e73 100644
--- a/meta/lib/oe/terminal.py
+++ b/meta/lib/oe/terminal.py
@@ -92,6 +92,13 @@ class Rxvt(XTerminal):
     command = 'rxvt -T "{title}" -e {command}'
     priority = 1
 
+class Screen_inline(Terminal):
+    command = 'screen -D -m -t "{title}" -S devshell {command}'
+
+    def __init__(self, command, title=None, env=None):
+        Terminal.__init__(self, command, title, env)
+        logger.warn('Screen Connect Command: screen -r devshell')
+
 class Screen(Terminal):
     command = 'screen -D -m -t "{title}" -S devshell {command}'
 
-- 
1.7.10

[PATCH v4 17/18] terminal.bbclass: Allow OE_TERMINAL_PREFERRED to override the fallback order for OE_TERMINAL

[Jason Wessel]

With the addition of two different modes for the use of screen, an end
user may prefer to set specific fall back order for an X vs tty
environment.  This patch modifies the terminal.bbclass and terminal.py
to allow the variable OE_TERMINAL_PREFERRED to control the order of
trying the various supported OE_TERMINAL's.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 meta/classes/terminal.bbclass |    3 ++-
 meta/lib/oe/terminal.py       |    8 +++++---
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/meta/classes/terminal.bbclass b/meta/classes/terminal.bbclass
index 3cfc84b..2be6974 100644
--- a/meta/classes/terminal.bbclass
+++ b/meta/classes/terminal.bbclass
@@ -4,6 +4,7 @@ OE_TERMINAL[choices] = 'auto none \
                         ${@" ".join(o.name \
                                     for o in oe.terminal.prioritized())}'
 
+OE_TERMINAL_PREFERRED ?= "auto"
 OE_TERMINAL_EXPORTS = 'XAUTHORITY SHELL DBUS_SESSION_BUS_ADDRESS DISPLAY EXTRA_OEMAKE SCREENDIR'
 OE_TERMINAL_EXPORTS[type] = 'list'
 
@@ -34,7 +35,7 @@ def oe_terminal(command, title, d):
             bb.fatal('Unable to spawn terminal %s: %s' % (terminal, exc))
 
     try:
-        oe.terminal.spawn_preferred(command, title)
+        oe.terminal.spawn_preferred(d.getVar('OE_TERMINAL_PREFERRED', True), command, title)
     except oe.terminal.NoSupportedTerminals:
         bb.fatal('No valid terminal found, unable to open devshell')
     except oe.terminal.ExecutionError as exc:
diff --git a/meta/lib/oe/terminal.py b/meta/lib/oe/terminal.py
index ca94e73..943e5db 100644
--- a/meta/lib/oe/terminal.py
+++ b/meta/lib/oe/terminal.py
@@ -111,11 +111,13 @@ class Screen(Terminal):
 def prioritized():
     return Registry.prioritized()
 
-def spawn_preferred(command, title=None, env=None):
+def spawn_preferred(priorityList, command, title=None, env=None):
     """Spawn the first supported terminal, by priority"""
-    for terminal in prioritized():
+    if priorityList == "auto":
+        priorityList = ' '.join(o.name for o in prioritized())
+    for terminal in priorityList.split():
         try:
-            spawn(terminal.name, command, title, env)
+            spawn(terminal, command, title, env)
             break
         except UnsupportedTerminal:
             continue
-- 
1.7.10

[PATCH v4 18/18] terminal.py: Make screen_inline have a higher priority than screen

[Jason Wessel]

The screen_inline is far easier to use and less confusing for
beginner users of oe-core.  Making this the default improves
the end user experience, and it is possible to change
the OE_TERMINAL variable as well as the default fall back list
with OE_TERMINAL_PREFERRED for anyone who desires the original
behavior.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 meta/lib/oe/terminal.py |   11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/meta/lib/oe/terminal.py b/meta/lib/oe/terminal.py
index 943e5db..3122d56 100644
--- a/meta/lib/oe/terminal.py
+++ b/meta/lib/oe/terminal.py
@@ -55,11 +55,11 @@ class XTerminal(Terminal):
 
 class Gnome(XTerminal):
     command = 'gnome-terminal --disable-factory -t "{title}" -x {command}'
-    priority = 2
+    priority = 3
 
 class Xfce(XTerminal):
     command = 'Terminal -T "{title}" -e "{command}"'
-    priority = 2
+    priority = 3
 
     def __init__(self, command, title=None, env=None):
         # Upstream binary name is Terminal but Debian/Ubuntu use
@@ -73,7 +73,7 @@ class Xfce(XTerminal):
 
 class Konsole(XTerminal):
     command = 'konsole -T "{title}" -e {command}'
-    priority = 2
+    priority = 3
 
     def __init__(self, command, title=None, env=None):
         # Check version
@@ -86,14 +86,15 @@ class Konsole(XTerminal):
 
 class XTerm(XTerminal):
     command = 'xterm -T "{title}" -e {command}'
-    priority = 1
+    priority = 2
 
 class Rxvt(XTerminal):
     command = 'rxvt -T "{title}" -e {command}'
-    priority = 1
+    priority = 2
 
 class Screen_inline(Terminal):
     command = 'screen -D -m -t "{title}" -S devshell {command}'
+    priority = 1
 
     def __init__(self, command, title=None, env=None):
         Terminal.__init__(self, command, title, env)
-- 
1.7.10

Re: [PATCH v4 14/18] runqueue.py, build.py: Invalidate setscene tasks based on do_unpack stamps

[Richard Purdie]

On Fri, 2012-06-08 at 08:41 -0500, Jason Wessel wrote:
> If you have a fully populated sstate cache and have used it to
> execute a build, it is not possible to invalidate repackage
> an intermediate build after you have forced a compiled
> 
> Example when you have build from a complete sstate cache build:
>   bitbake core-image-sato
>   bitbake -c patch acl
>      *** Make some changes to the C files for experimentation.
>   bitbake -f -c compile acl
>   bitbake acl
> 
> The bitbake will refuse to build the acl package at this
> point and instead keep populating it from the sstate.  Using
> the cleanstate is no longer a good option because it will
> also erase the scratch area.
> 
> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
> ---
>  lib/bb/build.py    |    8 ++++++++
>  lib/bb/runqueue.py |   14 ++++++++++++++
>  2 files changed, 22 insertions(+)
> 
> diff --git a/lib/bb/build.py b/lib/bb/build.py
> index fb61b00..18c28aa 100644
> --- a/lib/bb/build.py
> +++ b/lib/bb/build.py
> @@ -463,6 +463,14 @@ def del_stamp(task, d, file_name = None):
>      stamp = stamp_internal(task, d, file_name)
>      bb.utils.remove(stamp)
>  
> +def exists_stamp(task, d, file_name = None):
> +    """
> +    Removes a stamp for a given task
> +    (d can be a data dict or dataCache)
> +    """
> +    stamp = stamp_internal(task, d, file_name)
> +    return os.path.exists(stamp)
> +
>  def stampfile(taskname, d, file_name = None):
>      """
>      Return the stamp for a given task
> diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
> index da3fdf9..6c802af 100644
> --- a/lib/bb/runqueue.py
> +++ b/lib/bb/runqueue.py
> @@ -718,6 +718,20 @@ class RunQueueData:
>                      for dep in self.runq_depends[task]:
>                          procdep.append(self.taskData.fn_index[self.runq_fnid[dep]] + "." + self.runq_task[dep])
>                      self.runq_hash[task] = bb.parse.siggen.get_taskhash(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], procdep, self.dataCache)
> +        try:
> +            new_setscene = []
> +            for task in self.runq_setscene:
> +                try:
> +                    fn = self.taskData.fn_index[self.rq.rqdata.runq_fnid[task]]
> +                    if bb.build.exists_stamp("do_unpack", self.dataCache, fn):
> +                        logger.debug(2, 'Removing task %s from queue because do_unpack exists', task)
> +                    else:
> +                        new_setscene.append(task)
> +                except:
> +                    logger.debug(2, 'Failed do_unpack check for %s', task)
> +            self.runq_setscene = new_setscene
> +        except:
> +            logger.debug(2, 'Failed to update runq_setscene')

We've gone to quite a bit of trouble to keep knowledge of the specific
tasks out of bitbake. I've noticed a few of your patches are "blurring"
the separation between bitbake and the metadata.

Here, bitbake shouldn't have knowledge of "unpack" or why its special.
I'd also argue its not in fact special and there are probably other
"force" scenarios which would break with a similar problem even with
this patch. I agree there is a problem here but we need to fix it in a
more generic way. There is already a bug open on this kind of problem
(#2256).

Cheers,

Richard

Re: [PATCH v4 14/18] runqueue.py, build.py: Invalidate setscene tasks based on do_unpack stamps

[Jason Wessel]

On 06/11/2012 08:52 AM, Richard Purdie wrote:
> On Fri, 2012-06-08 at 08:41 -0500, Jason Wessel wrote:
>> If you have a fully populated sstate cache and have used it to
>> execute a build, it is not possible to invalidate repackage
>> an intermediate build after you have forced a compiled
>>
>> Example when you have build from a complete sstate cache build:
>>   bitbake core-image-sato
>>   bitbake -c patch acl
>>      *** Make some changes to the C files for experimentation.
>>   bitbake -f -c compile acl
>>   bitbake acl
>>
>> The bitbake will refuse to build the acl package at this
>> point and instead keep populating it from the sstate.  Using
>> the cleanstate is no longer a good option because it will
>> also erase the scratch area.
>>
>> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
>> ---
>>  lib/bb/build.py    |    8 ++++++++
>>  lib/bb/runqueue.py |   14 ++++++++++++++
>>  2 files changed, 22 insertions(+)
>>
>> diff --git a/lib/bb/build.py b/lib/bb/build.py
>> index fb61b00..18c28aa 100644
>> --- a/lib/bb/build.py
>> +++ b/lib/bb/build.py
>> @@ -463,6 +463,14 @@ def del_stamp(task, d, file_name = None):
>>      stamp = stamp_internal(task, d, file_name)
>>      bb.utils.remove(stamp)
>>  
>> +def exists_stamp(task, d, file_name = None):
>> +    """
>> +    Removes a stamp for a given task
>> +    (d can be a data dict or dataCache)
>> +    """
>> +    stamp = stamp_internal(task, d, file_name)
>> +    return os.path.exists(stamp)
>> +
>>  def stampfile(taskname, d, file_name = None):
>>      """
>>      Return the stamp for a given task
>> diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
>> index da3fdf9..6c802af 100644
>> --- a/lib/bb/runqueue.py
>> +++ b/lib/bb/runqueue.py
>> @@ -718,6 +718,20 @@ class RunQueueData:
>>                      for dep in self.runq_depends[task]:
>>                          procdep.append(self.taskData.fn_index[self.runq_fnid[dep]] + "." + self.runq_task[dep])
>>                      self.runq_hash[task] = bb.parse.siggen.get_taskhash(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], procdep, self.dataCache)
>> +        try:
>> +            new_setscene = []
>> +            for task in self.runq_setscene:
>> +                try:
>> +                    fn = self.taskData.fn_index[self.rq.rqdata.runq_fnid[task]]
>> +                    if bb.build.exists_stamp("do_unpack", self.dataCache, fn):
>> +                        logger.debug(2, 'Removing task %s from queue because do_unpack exists', task)
>> +                    else:
>> +                        new_setscene.append(task)
>> +                except:
>> +                    logger.debug(2, 'Failed do_unpack check for %s', task)
>> +            self.runq_setscene = new_setscene
>> +        except:
>> +            logger.debug(2, 'Failed to update runq_setscene')
> We've gone to quite a bit of trouble to keep knowledge of the specific
> tasks out of bitbake. I've noticed a few of your patches are "blurring"
> the separation between bitbake and the metadata.
>
> Here, bitbake shouldn't have knowledge of "unpack" or why its special.
> I'd also argue its not in fact special and there are probably other
> "force" scenarios which would break with a similar problem even with
> this patch. I agree there is a problem here but we need to fix it in a
> more generic way. There is already a bug open on this kind of problem
> (#2256).

As I mentioned in the 0/18 summary this is certainly the case that there needs to be an API and this should be treated as more of an RFC.

I will break whole series down into a few smaller chunks since there are some important fixes vs the features.


Jason.