[PATCH 0/4] Improve logger setup in devtool/recipetool/scriptutils

[PATCH 0/4] Improve logger setup in devtool/recipetool/scriptutils

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This improves logger setup in devtool and recipetool to send ERRORs to stderr,
the way bitbake itself does, and also sends WARNINGs there, as those are
essentially out of band as well. This makes it easier to script with devtool
and recipetool, as any *real* subcommand output can be more easily captured
without having to manually filter out the cruft.

The following changes since commit b11725db2d5549dc45d8ae36fbf94a5c8e342d69:

  xserver-xorg: Fix build errors with clang (2019-03-04 14:26:41 +0000)

are available in the Git repository at:

  git@github.com:kergoth/openembedded-core scriptutils-log-improvements

for you to fetch changes up to c4f37568236d2e3d8c40d57399d0b275ad5b7c8b:

  oe.scriptutils: also send WARNING to stderr (2019-03-04 21:31:00 +0000)

----------------------------------------------------------------
Christopher Larson (4):
      oe.scriptutils: enable color in a more flexible way
      oe.scriptutils: add logger_setup_filters
      recipetool, devtool: set up the logging filters
      oe.scriptutils: also send WARNING to stderr

 scripts/devtool            |  2 ++
 scripts/lib/scriptutils.py | 30 ++++++++++++++++++++++++------
 scripts/recipetool         |  2 ++
 3 files changed, 28 insertions(+), 6 deletions(-)

-- 
2.20.1

[PATCH 1/4] oe.scriptutils: enable color in a more flexible way

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

Rather than recreating handlers and forcing them, iterate over the handlers
and enable color on ones we can handle. This makes it easier to handle color
properly when we introduce the bb.msg default log filters.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 scripts/lib/scriptutils.py | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
index 3c60c3a1e6..0633c7066e 100644
--- a/scripts/lib/scriptutils.py
+++ b/scripts/lib/scriptutils.py
@@ -39,12 +39,12 @@ def logger_create(name, stream=None):
 
 def logger_setup_color(logger, color='auto'):
     from bb.msg import BBLogFormatter
-    console = logging.StreamHandler(sys.stdout)
-    formatter = BBLogFormatter("%(levelname)s: %(message)s")
-    console.setFormatter(formatter)
-    logger.handlers = [console]
-    if color == 'always' or (color=='auto' and console.stream.isatty()):
-        formatter.enable_color()
+
+    for handler in logger.handlers:
+        if (isinstance(handler, logging.StreamHandler) and
+            isinstance(handler.formatter, BBLogFormatter)):
+            if color == 'always' or (color == 'auto' and handler.stream.isatty()):
+                handler.formatter.enable_color()
 
 
 def load_plugins(logger, plugins, pluginpath):
@@ -69,6 +69,7 @@ def load_plugins(logger, plugins, pluginpath):
                 plugin.plugin_init(plugins)
             plugins.append(plugin)
 
+
 def git_convert_standalone_clone(repodir):
     """If specified directory is a git repository, ensure it's a standalone clone"""
     import bb.process
-- 
2.20.1

[PATCH 2/4] oe.scriptutils: add logger_setup_filters

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This function sets up the default bb.msg formatter and log filters, which by
default sends ERROR messages to stderr rather than stdout.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 scripts/lib/scriptutils.py | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
index 0633c7066e..c361d0741d 100644
--- a/scripts/lib/scriptutils.py
+++ b/scripts/lib/scriptutils.py
@@ -29,6 +29,7 @@ import tempfile
 import importlib
 from importlib import machinery
 
+
 def logger_create(name, stream=None):
     logger = logging.getLogger(name)
     loggerhandler = logging.StreamHandler(stream=stream)
@@ -37,6 +38,20 @@ def logger_create(name, stream=None):
     logger.setLevel(logging.INFO)
     return logger
 
+
+def logger_setup_filters(logger):
+    import bb.msg
+
+    console = logging.StreamHandler(sys.stdout)
+    errconsole = logging.StreamHandler(sys.stderr)
+    bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut)
+    bb.msg.addDefaultlogFilter(errconsole, bb.msg.BBLogFilterStdErr)
+    format_str = "%(levelname)s: %(message)s"
+    console.setFormatter(bb.msg.BBLogFormatter(format_str))
+    errconsole.setFormatter(bb.msg.BBLogFormatter(format_str))
+    logger.handlers = [console, errconsole]
+
+
 def logger_setup_color(logger, color='auto'):
     from bb.msg import BBLogFormatter
 
-- 
2.20.1

[PATCH 3/4] recipetool, devtool: set up the logging filters

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

We want errors going to stderr, the way they do in bitbake itself.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 scripts/devtool    | 2 ++
 scripts/recipetool | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/scripts/devtool b/scripts/devtool
index 0e578c0de3..de1847eef3 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -282,11 +282,13 @@ def main():
         logger.debug('Using standard bitbake path %s' % bitbakepath)
         scriptpath.add_oe_lib_path()
 
+    scriptutils.logger_setup_filters(logger)
     scriptutils.logger_setup_color(logger, global_args.color)
 
     if global_args.bbpath is None:
         try:
             tinfoil = setup_tinfoil(config_only=True, basepath=basepath)
+            tinfoil.logger.handlers = logger.handlers
             try:
                 global_args.bbpath = tinfoil.config_data.getVar('BBPATH')
             finally:
diff --git a/scripts/recipetool b/scripts/recipetool
index 3a3c9b7445..bac9c72107 100755
--- a/scripts/recipetool
+++ b/scripts/recipetool
@@ -75,9 +75,11 @@ def main():
     logger.debug('Found bitbake path: %s' % bitbakepath)
     scriptpath.add_oe_lib_path()
 
+    scriptutils.logger_setup_filters(logger)
     scriptutils.logger_setup_color(logger, global_args.color)
 
     tinfoil = tinfoil_init(False)
+    tinfoil.logger.handlers = logger.handlers
     try:
         for path in (tinfoil.config_data.getVar('BBPATH').split(':')
                      + [scripts_path]):
-- 
2.20.1

[PATCH 4/4] oe.scriptutils: also send WARNING to stderr

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

stderr isn't just for errors, but anything out of band. It's highly unlikely
that warning messages are anything but out of band, which makes scripting our
tools more troublesome, since the script calling it has to filter out the
warnings from stdout to get the real output.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 scripts/lib/scriptutils.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
index c361d0741d..d1a50207c5 100644
--- a/scripts/lib/scriptutils.py
+++ b/scripts/lib/scriptutils.py
@@ -43,9 +43,11 @@ def logger_setup_filters(logger):
     import bb.msg
 
     console = logging.StreamHandler(sys.stdout)
+    bb.msg.addDefaultlogFilter(console)
     errconsole = logging.StreamHandler(sys.stderr)
-    bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut)
-    bb.msg.addDefaultlogFilter(errconsole, bb.msg.BBLogFilterStdErr)
+    bb.msg.addDefaultlogFilter(errconsole)
+    console.addFilter(lambda r: r.levelno < logging.WARNING)
+    errconsole.addFilter(lambda r: r.levelno >= logging.WARNING)
     format_str = "%(levelname)s: %(message)s"
     console.setFormatter(bb.msg.BBLogFormatter(format_str))
     errconsole.setFormatter(bb.msg.BBLogFormatter(format_str))
-- 
2.20.1

Re: [PATCH 0/4] Improve logger setup in devtool/recipetool/scriptutils

[Paul Eggleton]

On Tuesday, 5 March 2019 10:36:07 AM NZDT Christopher Larson wrote:
> From: Christopher Larson <chris_larson@mentor.com>
> 
> This improves logger setup in devtool and recipetool to send ERRORs to 
stderr,
> the way bitbake itself does, and also sends WARNINGs there, as those are
> essentially out of band as well. This makes it easier to script with devtool
> and recipetool, as any *real* subcommand output can be more easily captured
> without having to manually filter out the cruft.
> 
> The following changes since commit b11725db2d5549dc45d8ae36fbf94a5c8e342d69:
> 
>   xserver-xorg: Fix build errors with clang (2019-03-04 14:26:41 +0000)
> 
> are available in the Git repository at:
> 
>   git@github.com:kergoth/openembedded-core scriptutils-log-improvements
> 
> for you to fetch changes up to c4f37568236d2e3d8c40d57399d0b275ad5b7c8b:
> 
>   oe.scriptutils: also send WARNING to stderr (2019-03-04 21:31:00 +0000)
> 
> ----------------------------------------------------------------
> Christopher Larson (4):
>       oe.scriptutils: enable color in a more flexible way
>       oe.scriptutils: add logger_setup_filters
>       recipetool, devtool: set up the logging filters
>       oe.scriptutils: also send WARNING to stderr
> 

All of these look good to me.

Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com>

Cheers
Paul


-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/4] Improve logger setup in devtool/recipetool/scriptutils

[Richard Purdie]

On Tue, 2019-03-05 at 16:50 +1300, Paul Eggleton wrote:
> On Tuesday, 5 March 2019 10:36:07 AM NZDT Christopher Larson wrote:
> > From: Christopher Larson <chris_larson@mentor.com>
> > 
> > This improves logger setup in devtool and recipetool to send ERRORs
> > to 
> stderr,
> > the way bitbake itself does, and also sends WARNINGs there, as
> > those are
> > essentially out of band as well. This makes it easier to script
> > with devtool
> > and recipetool, as any *real* subcommand output can be more easily
> > captured
> > without having to manually filter out the cruft.
> > 
> > The following changes since commit
> > b11725db2d5549dc45d8ae36fbf94a5c8e342d69:
> > 
> >   xserver-xorg: Fix build errors with clang (2019-03-04 14:26:41
> > +0000)
> > 
> > are available in the Git repository at:
> > 
> >   git@github.com:kergoth/openembedded-core scriptutils-log-
> > improvements
> > 
> > for you to fetch changes up to
> > c4f37568236d2e3d8c40d57399d0b275ad5b7c8b:
> > 
> >   oe.scriptutils: also send WARNING to stderr (2019-03-04 21:31:00
> > +0000)
> > 
> > ----------------------------------------------------------------
> > Christopher Larson (4):
> >       oe.scriptutils: enable color in a more flexible way
> >       oe.scriptutils: add logger_setup_filters
> >       recipetool, devtool: set up the logging filters
> >       oe.scriptutils: also send WARNING to stderr
> > 
> 
> All of these look good to me.
> 
> Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com>

Unfortunately there were a couple of issues on the autobuilder in the
devtool selftests with these applied:

https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/297

Cheers,

Richard