[PATCHv2 0/6] recipetool/devtool/oe-selftest: pull from BBPATH

[PATCHv2 0/6] recipetool/devtool/oe-selftest: pull from BBPATH

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

Ensures that recipetool and devtool pull plugins from layers, and
oe-selftest will pull tests from layers.

Unrelated to that, also prevents a traceback on parsing failure, and adds
a tiny feature to appendsrc to facilitate reuse by other sub-commands.

[v2 Update] `devtool: also load plugins from BBPATH` was fixed, so it no longer
breaks the devtool selftests.


The following changes since commit b283b82ec59a2137cf717811172765848fa813fa:

  core-image-lsb-sdk: use kernel-devsrc to ensure kernel module builds work (2015-07-20 10:39:02 +0100)

are available in the git repository at:

  git@github.com:kergoth/openembedded-core yocto-bug-7625

for you to fetch changes up to 35a9ce57747248192dc5c0dd7916e15e58a5835e:

  oe-selftest: add libdirs from BBPATH to sys.path (2015-07-20 15:11:59 -0700)

----------------------------------------------------------------
Christopher Larson (6):
      recipetool: catch BBHandledException from parsing
      recipetool.append: add extralines arg to appendsrc
      recipetool: also load plugins from BBPATH
      devtool: also load plugins from BBPATH
      oe-selftest: obey oeqa.selftest.__path__
      oe-selftest: add libdirs from BBPATH to sys.path

 scripts/devtool                  | 57 +++++++++++++++++++++++++---------------
 scripts/lib/devtool/__init__.py  |  4 +--
 scripts/lib/recipetool/append.py |  4 +--
 scripts/oe-selftest              | 18 +++++++++----
 scripts/recipetool               | 56 ++++++++++++++++++++++++---------------
 5 files changed, 88 insertions(+), 51 deletions(-)

-- 
2.2.1

[PATCHv2 1/6] recipetool: catch BBHandledException from parsing

[Christopher Larson]

This ensures that we don't see a traceback on parsing failures.

Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 scripts/recipetool | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/scripts/recipetool b/scripts/recipetool
index c68bef4..3063cf7 100755
--- a/scripts/recipetool
+++ b/scripts/recipetool
@@ -82,9 +82,11 @@ def main():
 
     scriptutils.logger_setup_color(logger, args.color)
 
-    tinfoil_init(getattr(args, 'parserecipes', False))
-
-    ret = args.func(args)
+    try:
+        tinfoil_init(getattr(args, 'parserecipes', False))
+        ret = args.func(args)
+    except bb.BBHandledException:
+        ret = 1
 
     return ret
 
-- 
2.2.1

[PATCHv2 2/6] recipetool.append: add extralines arg to appendsrc

[Christopher Larson]

This makes the function more reusable for other sub-commands.

Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 scripts/lib/recipetool/append.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/recipetool/append.py b/scripts/lib/recipetool/append.py
index a2133f7..fb6b090 100644
--- a/scripts/lib/recipetool/append.py
+++ b/scripts/lib/recipetool/append.py
@@ -337,7 +337,7 @@ def appendfile(args):
         return 3
 
 
-def appendsrc(args, files, rd):
+def appendsrc(args, files, rd, extralines=None):
     import oe.recipeutils
 
     srcdir = rd.getVar('S', True)
@@ -352,7 +352,7 @@ def appendsrc(args, files, rd):
         simplified[simple_uri] = uri
 
     copyfiles = {}
-    extralines = []
+    extralines = extralines or []
     for newfile, srcfile in files.iteritems():
         src_destdir = os.path.dirname(srcfile)
         if not args.use_workdir:
-- 
2.2.1

[PATCHv2 3/6] recipetool: also load plugins from BBPATH

[Christopher Larson]

This makes it easier to extend, as a layer can add its own sub-commands.

[YOCTO #7625]

Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 scripts/recipetool | 52 ++++++++++++++++++++++++++++++++--------------------
 1 file changed, 32 insertions(+), 20 deletions(-)

diff --git a/scripts/recipetool b/scripts/recipetool
index 3063cf7..6061d7b 100755
--- a/scripts/recipetool
+++ b/scripts/recipetool
@@ -36,11 +36,8 @@ def tinfoil_init(parserecipes):
     import logging
     tinfoil = bb.tinfoil.Tinfoil()
     tinfoil.prepare(not parserecipes)
-
-    for plugin in plugins:
-        if hasattr(plugin, 'tinfoil_init'):
-            plugin.tinfoil_init(tinfoil)
     tinfoil.logger.setLevel(logger.getEffectiveLevel())
+    return tinfoil
 
 def main():
 
@@ -49,28 +46,22 @@ def main():
         sys.exit(1)
 
     parser = argparse.ArgumentParser(description="OpenEmbedded recipe tool",
+                                     add_help=False,
                                      epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
     parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
     parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
     parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR')
-    subparsers = parser.add_subparsers(title='subcommands', metavar='<subcommand>')
-
-    scriptutils.load_plugins(logger, plugins, os.path.join(scripts_path, 'lib', 'recipetool'))
-    registered = False
-    for plugin in plugins:
-        if hasattr(plugin, 'register_command'):
-            registered = True
-            plugin.register_command(subparsers)
 
-    if not registered:
-        logger.error("No commands registered - missing plugins?")
-        sys.exit(1)
+    initial_args, unparsed_args = parser.parse_known_args(sys.argv[1:])
 
-    args = parser.parse_args()
+    # Help is added here rather than via add_help=True, as we don't want it to
+    # be handled by parse_known_args()
+    parser.add_argument('-h', '--help', action='help', default=argparse.SUPPRESS,
+                        help='show this help message and exit')
 
-    if args.debug:
+    if initial_args.debug:
         logger.setLevel(logging.DEBUG)
-    elif args.quiet:
+    elif initial_args.quiet:
         logger.setLevel(logging.ERROR)
 
     import scriptpath
@@ -80,10 +71,31 @@ def main():
         sys.exit(1)
     logger.debug('Found bitbake path: %s' % bitbakepath)
 
-    scriptutils.logger_setup_color(logger, args.color)
+    scriptutils.logger_setup_color(logger, initial_args.color)
+
+    subparsers = parser.add_subparsers(title='subcommands', metavar='<subcommand>')
+    tinfoil = tinfoil_init(False)
+    for path in ([scripts_path] +
+                 tinfoil.config_data.getVar('BBPATH', True).split(':')):
+        pluginpath = os.path.join(path, 'lib', 'recipetool')
+        scriptutils.load_plugins(logger, plugins, pluginpath)
+    registered = False
+    for plugin in plugins:
+        if hasattr(plugin, 'register_command'):
+            registered = True
+            plugin.register_command(subparsers)
+        if hasattr(plugin, 'tinfoil_init'):
+            plugin.tinfoil_init(tinfoil)
+
+    if not registered:
+        logger.error("No commands registered - missing plugins?")
+        sys.exit(1)
+
+    args = parser.parse_args(unparsed_args, namespace=initial_args)
 
     try:
-        tinfoil_init(getattr(args, 'parserecipes', False))
+        if getattr(args, 'parserecipes', False):
+            tinfoil.parseRecipes()
         ret = args.func(args)
     except bb.BBHandledException:
         ret = 1
-- 
2.2.1

Re: [PATCHv2 3/6] recipetool: also load plugins from BBPATH

[Paul Eggleton]

Hi Chris,

On Monday 20 July 2015 15:15:53 Christopher Larson wrote:
> This makes it easier to extend, as a layer can add its own sub-commands.
> 
> [YOCTO #7625]
> 
> Signed-off-by: Christopher Larson <kergoth@gmail.com>
> Signed-off-by: Ross Burton <ross.burton@intel.com>
> ---
>  scripts/recipetool | 52
> ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 32
> insertions(+), 20 deletions(-)
> 
> diff --git a/scripts/recipetool b/scripts/recipetool
> index 3063cf7..6061d7b 100755
> --- a/scripts/recipetool
> +++ b/scripts/recipetool
> @@ -36,11 +36,8 @@ def tinfoil_init(parserecipes):
>      import logging
>      tinfoil = bb.tinfoil.Tinfoil()
>      tinfoil.prepare(not parserecipes)
> -
> -    for plugin in plugins:
> -        if hasattr(plugin, 'tinfoil_init'):
> -            plugin.tinfoil_init(tinfoil)
>      tinfoil.logger.setLevel(logger.getEffectiveLevel())
> +    return tinfoil
> 
>  def main():
> 
> @@ -49,28 +46,22 @@ def main():
>          sys.exit(1)
> 
>      parser = argparse.ArgumentParser(description="OpenEmbedded recipe
> tool", +                                     add_help=False,
>                                       epilog="Use %(prog)s <subcommand>
> --help to get help on a specific command") parser.add_argument('-d',
> '--debug', help='Enable debug output', action='store_true')
> parser.add_argument('-q', '--quiet', help='Print only errors',
> action='store_true') parser.add_argument('--color', choices=['auto',
> 'always', 'never'], default='auto', help='Colorize output (where
> %(metavar)s is %(choices)s)', metavar='COLOR') -    subparsers =
> parser.add_subparsers(title='subcommands', metavar='<subcommand>') -
> -    scriptutils.load_plugins(logger, plugins, os.path.join(scripts_path,
> 'lib', 'recipetool')) -    registered = False
> -    for plugin in plugins:
> -        if hasattr(plugin, 'register_command'):
> -            registered = True
> -            plugin.register_command(subparsers)
> 
> -    if not registered:
> -        logger.error("No commands registered - missing plugins?")
> -        sys.exit(1)
> +    initial_args, unparsed_args = parser.parse_known_args(sys.argv[1:])
> 
> -    args = parser.parse_args()
> +    # Help is added here rather than via add_help=True, as we don't want it
> to +    # be handled by parse_known_args()
> +    parser.add_argument('-h', '--help', action='help',
> default=argparse.SUPPRESS, +                        help='show this help
> message and exit')
> 
> -    if args.debug:
> +    if initial_args.debug:
>          logger.setLevel(logging.DEBUG)
> -    elif args.quiet:
> +    elif initial_args.quiet:
>          logger.setLevel(logging.ERROR)
> 
>      import scriptpath
> @@ -80,10 +71,31 @@ def main():
>          sys.exit(1)
>      logger.debug('Found bitbake path: %s' % bitbakepath)
> 
> -    scriptutils.logger_setup_color(logger, args.color)
> +    scriptutils.logger_setup_color(logger, initial_args.color)
> +
> +    subparsers = parser.add_subparsers(title='subcommands',
> metavar='<subcommand>') +    tinfoil = tinfoil_init(False)
> +    for path in ([scripts_path] +
> +                 tinfoil.config_data.getVar('BBPATH', True).split(':')):
> +        pluginpath = os.path.join(path, 'lib', 'recipetool')
> +        scriptutils.load_plugins(logger, plugins, pluginpath)
> +    registered = False
> +    for plugin in plugins:
> +        if hasattr(plugin, 'register_command'):
> +            registered = True
> +            plugin.register_command(subparsers)
> +        if hasattr(plugin, 'tinfoil_init'):
> +            plugin.tinfoil_init(tinfoil)
> +
> +    if not registered:
> +        logger.error("No commands registered - missing plugins?")
> +        sys.exit(1)
> +
> +    args = parser.parse_args(unparsed_args, namespace=initial_args)
> 
>      try:
> -        tinfoil_init(getattr(args, 'parserecipes', False))
> +        if getattr(args, 'parserecipes', False):
> +            tinfoil.parseRecipes()
>          ret = args.func(args)
>      except bb.BBHandledException:
>          ret = 1

I must admit it wasn't immediately obvious to me why you'd made some of the 
changes here, although after several reads it does make sense - would you be 
able to add a slightly more detailed commit message explaining the changes? 
(same for the equivalent devtool patch).

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCHv2 3/6] recipetool: also load plugins from BBPATH

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1178 bytes --]

On Wed, Jul 22, 2015 at 3:55 AM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> I must admit it wasn't immediately obvious to me why you'd made some of the
> changes here, although after several reads it does make sense - would you
> be
> able to add a slightly more detailed commit message explaining the changes?
> (same for the equivalent devtool patch).
>

Will do, thanks.

- The tinfoil setup had to be split, as we need access to the tinfoil
object to get BBPATH.
- The argument parsing was split into two, first just the global options,
the second for everything else, to let us enable debugging or quiet mode
before the plugins are loaded, so we can see details about that process,
rather than doing it all at once. —help, while global, shouldn’t be handled
in the first pass, otherwise the help info will lack info about the plugins.

I’ll add that info to the commit message, and also rename initial_args to
global_args to hopefully clarify it a bit better in the code.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1753 bytes --]

[PATCHv2 4/6] devtool: also load plugins from BBPATH

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This makes it easier to extend, as a layer can add its own sub-commands.

tinfoil is now passed into the commands, as we needed to parse the
configuration metadata to get BBPATH, and we don't want to construct tinfoil
more than once, otherwise we have to deal with startup and shutdown of cooker.

[YOCTO #7625]

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 scripts/devtool                 | 57 ++++++++++++++++++++++++++---------------
 scripts/lib/devtool/__init__.py |  4 +--
 2 files changed, 38 insertions(+), 23 deletions(-)

diff --git a/scripts/devtool b/scripts/devtool
index fa799f6..b2ffc31 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -35,7 +35,7 @@ context = None
 scripts_path = os.path.dirname(os.path.realpath(__file__))
 lib_path = scripts_path + '/lib'
 sys.path = sys.path + [lib_path]
-from devtool import DevtoolError
+from devtool import DevtoolError, setup_tinfoil
 import scriptutils
 logger = scriptutils.logger_create('devtool')
 
@@ -186,37 +186,28 @@ def main():
         pth = os.path.dirname(pth)
 
     parser = argparse.ArgumentParser(description="OpenEmbedded development tool",
+                                     add_help=False,
                                      epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
     parser.add_argument('--basepath', help='Base directory of SDK / build directory')
     parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
     parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
     parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR')
 
-    subparsers = parser.add_subparsers(dest="subparser_name", title='subcommands', metavar='<subcommand>')
+    initial_args, unparsed_args = parser.parse_known_args()
 
-    if not context.fixed_setup:
-        parser_create_workspace = subparsers.add_parser('create-workspace',
-                                                        help='Set up a workspace',
-                                                        description='Sets up a new workspace. NOTE: other devtool subcommands will create a workspace automatically as needed, so you only need to use %(prog)s if you want to specify where the workspace should be located.')
-        parser_create_workspace.add_argument('layerpath', nargs='?', help='Path in which the workspace layer should be created')
-        parser_create_workspace.add_argument('--create-only', action="store_true", help='Only create the workspace layer, do not alter configuration')
-        parser_create_workspace.set_defaults(func=create_workspace)
+    # Help is added here rather than via add_help=True, as we don't want it to
+    # be handled by parse_known_args()
+    parser.add_argument('-h', '--help', action='help', default=argparse.SUPPRESS,
+                        help='show this help message and exit')
 
-    scriptutils.load_plugins(logger, plugins, os.path.join(scripts_path, 'lib', 'devtool'))
-    for plugin in plugins:
-        if hasattr(plugin, 'register_commands'):
-            plugin.register_commands(subparsers, context)
-
-    args = parser.parse_args()
-
-    if args.debug:
+    if initial_args.debug:
         logger.setLevel(logging.DEBUG)
-    elif args.quiet:
+    elif initial_args.quiet:
         logger.setLevel(logging.ERROR)
 
-    if args.basepath:
+    if initial_args.basepath:
         # Override
-        basepath = args.basepath
+        basepath = initial_args.basepath
     elif not context.fixed_setup:
         basepath = os.environ.get('BUILDDIR')
         if not basepath:
@@ -246,7 +237,31 @@ def main():
         logger.debug('Using standard bitbake path %s' % bitbakepath)
         scriptpath.add_oe_lib_path()
 
-    scriptutils.logger_setup_color(logger, args.color)
+    scriptutils.logger_setup_color(logger, initial_args.color)
+
+    tinfoil = setup_tinfoil(config_only=True)
+    for path in ([scripts_path] +
+                 tinfoil.config_data.getVar('BBPATH', True).split(':')):
+        pluginpath = os.path.join(path, 'lib', 'devtool')
+        scriptutils.load_plugins(logger, plugins, pluginpath)
+    tinfoil.cooker.shutdown(force=True)
+    tinfoil.cooker.unlockBitbake()
+
+    subparsers = parser.add_subparsers(dest="subparser_name", title='subcommands', metavar='<subcommand>')
+
+    if not context.fixed_setup:
+        parser_create_workspace = subparsers.add_parser('create-workspace',
+                                                        help='Set up a workspace',
+                                                        description='Sets up a new workspace. NOTE: other devtool subcommands will create a workspace automatically as needed, so you only need to use %(prog)s if you want to specify where the workspace should be located.')
+        parser_create_workspace.add_argument('layerpath', nargs='?', help='Path in which the workspace layer should be created')
+        parser_create_workspace.add_argument('--create-only', action="store_true", help='Only create the workspace layer, do not alter configuration')
+        parser_create_workspace.set_defaults(func=create_workspace)
+
+    for plugin in plugins:
+        if hasattr(plugin, 'register_commands'):
+            plugin.register_commands(subparsers, context)
+
+    args = parser.parse_args(unparsed_args, namespace=initial_args)
 
     if args.subparser_name != 'create-workspace':
         read_workspace()
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 61b810c..b54ddf5 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -96,7 +96,7 @@ def exec_fakeroot(d, cmd, **kwargs):
             newenv[splitval[0]] = splitval[1]
     return subprocess.call("%s %s" % (fakerootcmd, cmd), env=newenv, **kwargs)
 
-def setup_tinfoil():
+def setup_tinfoil(config_only=False):
     """Initialize tinfoil api from bitbake"""
     import scriptpath
     bitbakepath = scriptpath.add_bitbake_lib_path()
@@ -106,7 +106,7 @@ def setup_tinfoil():
 
     import bb.tinfoil
     tinfoil = bb.tinfoil.Tinfoil()
-    tinfoil.prepare(False)
+    tinfoil.prepare(config_only)
     tinfoil.logger.setLevel(logger.getEffectiveLevel())
     return tinfoil
 
-- 
2.2.1

Re: [PATCHv2 4/6] devtool: also load plugins from BBPATH

[Paul Eggleton]

On Monday 20 July 2015 15:15:54 Christopher Larson wrote:
> From: Christopher Larson <chris_larson@mentor.com>
> 
> This makes it easier to extend, as a layer can add its own sub-commands.
> 
> tinfoil is now passed into the commands, as we needed to parse the
> configuration metadata to get BBPATH, and we don't want to construct tinfoil
> more than once, otherwise we have to deal with startup and shutdown of
> cooker.

Hmm, but tinfoil is being constructed twice, right? I think what you've 
implemented is correct (and the only way it can really work), but this comment 
doesn't seem to match up with the actual changes.

> +    tinfoil = setup_tinfoil(config_only=True)
> +    for path in ([scripts_path] +
> +                 tinfoil.config_data.getVar('BBPATH', True).split(':')):
> +        pluginpath = os.path.join(path, 'lib', 'devtool')
> +        scriptutils.load_plugins(logger, plugins, pluginpath)
> +    tinfoil.cooker.shutdown(force=True)
> +    tinfoil.cooker.unlockBitbake()

FYI I think Richard is about to add a shutdown method to tinfoil itself that 
will do what's needed here; however we can tidy this up after that's added.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCHv2 4/6] devtool: also load plugins from BBPATH

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2032 bytes --]

On Wed, Jul 22, 2015 at 4:02 AM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> On Monday 20 July 2015 15:15:54 Christopher Larson wrote:
> > From: Christopher Larson <chris_larson@mentor.com>
> >
> > This makes it easier to extend, as a layer can add its own sub-commands.
> >
> > tinfoil is now passed into the commands, as we needed to parse the
> > configuration metadata to get BBPATH, and we don't want to construct
> tinfoil
> > more than once, otherwise we have to deal with startup and shutdown of
> > cooker.
>
> Hmm, but tinfoil is being constructed twice, right? I think what you've
> implemented is correct (and the only way it can really work), but this
> comment
> doesn't seem to match up with the actual changes.
>

Yeah, that’s correct. My initial implementation did what the commit message
says, but of course that’s a problem for devtool commands that call out to
recipetool, since recipetool constructs its own, so needs to acquire the
bitbake lock itself. So I reworked it to construct and then throw away the
initial instance instead, and leave it up to the subcommands to control
whether they need a tinfoil instance the way they did previously. I’ll fix
the commit message.

> +    tinfoil = setup_tinfoil(config_only=True)
> > +    for path in ([scripts_path] +
> > +                 tinfoil.config_data.getVar('BBPATH', True).split(':')):
> > +        pluginpath = os.path.join(path, 'lib', 'devtool')
> > +        scriptutils.load_plugins(logger, plugins, pluginpath)
> > +    tinfoil.cooker.shutdown(force=True)
> > +    tinfoil.cooker.unlockBitbake()
>
> FYI I think Richard is about to add a shutdown method to tinfoil itself
> that

will do what's needed here; however we can tidy this up after that's added.


Cool, that would definitely be cleaner than poking into its internals.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2924 bytes --]

[PATCHv2 5/6] oe-selftest: obey oeqa.selftest.__path__

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This ensures that all paths that hold selftest tests will be checked
(oeqa.selftest is a namespace package).

[YOCTO #7625]

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 scripts/oe-selftest | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index fa6245a..c19c692 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -142,11 +142,12 @@ def get_tests(exclusive_modules=[], include_hidden=False):
     for x in exclusive_modules:
         testslist.append('oeqa.selftest.' + x)
     if not testslist:
-        testpath = os.path.abspath(os.path.dirname(oeqa.selftest.__file__))
-        files = sorted([f for f in os.listdir(testpath) if f.endswith('.py') and not (f.startswith('_') and not include_hidden) and not f.startswith('__') and f != 'base.py'])
-        for f in files:
-            module = 'oeqa.selftest.' + f[:-3]
-            testslist.append(module)
+        for testpath in oeqa.selftest.__path__:
+            files = sorted([f for f in os.listdir(testpath) if f.endswith('.py') and not (f.startswith('_') and not include_hidden) and not f.startswith('__') and f != 'base.py'])
+            for f in files:
+                module = 'oeqa.selftest.' + f[:-3]
+                if module not in testslist:
+                    testslist.append(module)
 
     return testslist
 
-- 
2.2.1

[PATCHv2 6/6] oe-selftest: add libdirs from BBPATH to sys.path

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This ensures that oeqa.selftest.* from layers are found.

[YOCTO #7625]

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 scripts/oe-selftest | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index c19c692..60f9bb8 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -155,6 +155,13 @@ def main():
     parser = get_args_parser()
     args = parser.parse_args()
 
+    # Add <layer>/lib to sys.path, so layers can add selftests
+    log.info("Running bitbake -e to get BBPATH")
+    bbpath = get_bb_var('BBPATH').split(':')
+    layer_libdirs = [p for p in (os.path.join(l, 'lib') for l in bbpath) if os.path.exists(p)]
+    sys.path.extend(layer_libdirs)
+    reload(oeqa.selftest)
+
     if args.list_allclasses:
         args.list_modules = True
 
-- 
2.2.1

Re: [PATCHv2 0/6] recipetool/devtool/oe-selftest: pull from BBPATH

[Paul Eggleton]

On Monday 20 July 2015 15:15:38 Christopher Larson wrote:
> From: Christopher Larson <chris_larson@mentor.com>
> 
> Ensures that recipetool and devtool pull plugins from layers, and
> oe-selftest will pull tests from layers.
> 
> Unrelated to that, also prevents a traceback on parsing failure, and adds
> a tiny feature to appendsrc to facilitate reuse by other sub-commands.
> 
> [v2 Update] `devtool: also load plugins from BBPATH` was fixed, so it no
> longer breaks the devtool selftests.
> 
> 
> The following changes since commit b283b82ec59a2137cf717811172765848fa813fa:
> 
>   core-image-lsb-sdk: use kernel-devsrc to ensure kernel module builds work
> (2015-07-20 10:39:02 +0100)
> 
> are available in the git repository at:
> 
>   git@github.com:kergoth/openembedded-core yocto-bug-7625
> 
> for you to fetch changes up to 35a9ce57747248192dc5c0dd7916e15e58a5835e:
> 
>   oe-selftest: add libdirs from BBPATH to sys.path (2015-07-20 15:11:59
> -0700)
> 
> ----------------------------------------------------------------
> Christopher Larson (6):
>       recipetool: catch BBHandledException from parsing
>       recipetool.append: add extralines arg to appendsrc
>       recipetool: also load plugins from BBPATH
>       devtool: also load plugins from BBPATH
>       oe-selftest: obey oeqa.selftest.__path__
>       oe-selftest: add libdirs from BBPATH to sys.path
> 
>  scripts/devtool                  | 57
> +++++++++++++++++++++++++--------------- scripts/lib/devtool/__init__.py  |
>  4 +--
>  scripts/lib/recipetool/append.py |  4 +--
>  scripts/oe-selftest              | 18 +++++++++----
>  scripts/recipetool               | 56
> ++++++++++++++++++++++++--------------- 5 files changed, 88 insertions(+),
> 51 deletions(-)

Overall I'm happy with this set of changes apart from the commit messages for 
the devtool and recipetool plugin loading commits as separately noted. (The 
other changes could conceivably go in ahead).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCHv2 0/6] recipetool/devtool/oe-selftest: pull from BBPATH

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2397 bytes --]

On Wed, Jul 22, 2015 at 4:02 AM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> On Monday 20 July 2015 15:15:38 Christopher Larson wrote:
> > From: Christopher Larson <chris_larson@mentor.com>
> >
> > Ensures that recipetool and devtool pull plugins from layers, and
> > oe-selftest will pull tests from layers.
> >
> > Unrelated to that, also prevents a traceback on parsing failure, and adds
> > a tiny feature to appendsrc to facilitate reuse by other sub-commands.
> >
> > [v2 Update] `devtool: also load plugins from BBPATH` was fixed, so it no
> > longer breaks the devtool selftests.
> >
> >
> > The following changes since commit
> b283b82ec59a2137cf717811172765848fa813fa:
> >
> >   core-image-lsb-sdk: use kernel-devsrc to ensure kernel module builds
> work
> > (2015-07-20 10:39:02 +0100)
> >
> > are available in the git repository at:
> >
> >   git@github.com:kergoth/openembedded-core yocto-bug-7625
> >
> > for you to fetch changes up to 35a9ce57747248192dc5c0dd7916e15e58a5835e:
> >
> >   oe-selftest: add libdirs from BBPATH to sys.path (2015-07-20 15:11:59
> > -0700)
> >
> > ----------------------------------------------------------------
> > Christopher Larson (6):
> >       recipetool: catch BBHandledException from parsing
> >       recipetool.append: add extralines arg to appendsrc
> >       recipetool: also load plugins from BBPATH
> >       devtool: also load plugins from BBPATH
> >       oe-selftest: obey oeqa.selftest.__path__
> >       oe-selftest: add libdirs from BBPATH to sys.path
> >
> >  scripts/devtool                  | 57
> > +++++++++++++++++++++++++---------------
> scripts/lib/devtool/__init__.py  |
> >  4 +--
> >  scripts/lib/recipetool/append.py |  4 +--
> >  scripts/oe-selftest              | 18 +++++++++----
> >  scripts/recipetool               | 56
> > ++++++++++++++++++++++++--------------- 5 files changed, 88
> insertions(+),
> > 51 deletions(-)
>
> Overall I'm happy with this set of changes apart from the commit messages
> for
> the devtool and recipetool plugin loading commits as separately noted. (The
> other changes could conceivably go in ahead).


Thanks for the feedback, will get a v3 out today.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3136 bytes --]