[PATCH 0/6] recipetool/devtool/oe-selftest: pull from BBPATH

[PATCH 0/6] recipetool/devtool/oe-selftest: pull from BBPATH

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

Ensures that recipetool and devtool pull plugins from layers, and
oe-selftest will pull tests from layers.

Unrelated to that, also prevents a traceback on parsing failure, and adds
a tiny feature to appendsrc to facilitate reuse by other sub-commands.

[YOCTO #7625]

The following changes since commit 6be698b7270f73f40d38713ecf13f12aec0ced61:

  dpkg: Fix for Fedora22 and new versions of tar (2015-07-13 13:46:45 +0100)

are available in the git repository at:

  git@github.com:kergoth/openembedded-core yocto-bug-7625

for you to fetch changes up to ab9d004953408688ed82d1b8de1ae0887f9989b5:

  oe-selftest: add libdirs from BBPATH to sys.path (2015-07-13 12:31:29 -0700)

----------------------------------------------------------------
Christopher Larson (6):
      recipetool: catch BBHandledException from parsing
      recipetool.append: add extralines arg to appendsrc
      recipetool: also load plugins from BBPATH
      devtool: also load plugins from BBPATH
      oe-selftest: obey oeqa.selftest.__path__
      oe-selftest: add libdirs from BBPATH to sys.path

 scripts/devtool                  | 57 ++++++++++++++++++++++++----------------
 scripts/lib/devtool/__init__.py  |  4 +--
 scripts/lib/devtool/deploy.py    |  6 ++---
 scripts/lib/devtool/standard.py  | 15 +++++------
 scripts/lib/recipetool/append.py |  4 +--
 scripts/oe-selftest              | 18 +++++++++----
 scripts/recipetool               | 56 ++++++++++++++++++++++++---------------
 7 files changed, 96 insertions(+), 64 deletions(-)

-- 
2.2.1

[PATCH 1/6] recipetool: catch BBHandledException from parsing

[Christopher Larson]

This ensures that we don't see a traceback on parsing failures.

Signed-off-by: Christopher Larson <kergoth@gmail.com>
---
 scripts/recipetool | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/scripts/recipetool b/scripts/recipetool
index c68bef4..3063cf7 100755
--- a/scripts/recipetool
+++ b/scripts/recipetool
@@ -82,9 +82,11 @@ def main():
 
     scriptutils.logger_setup_color(logger, args.color)
 
-    tinfoil_init(getattr(args, 'parserecipes', False))
-
-    ret = args.func(args)
+    try:
+        tinfoil_init(getattr(args, 'parserecipes', False))
+        ret = args.func(args)
+    except bb.BBHandledException:
+        ret = 1
 
     return ret
 
-- 
2.2.1

[PATCH 2/6] recipetool.append: add extralines arg to appendsrc

[Christopher Larson]

This makes the function more reusable for other sub-commands.

Signed-off-by: Christopher Larson <kergoth@gmail.com>
---
 scripts/lib/recipetool/append.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/recipetool/append.py b/scripts/lib/recipetool/append.py
index a2133f7..fb6b090 100644
--- a/scripts/lib/recipetool/append.py
+++ b/scripts/lib/recipetool/append.py
@@ -337,7 +337,7 @@ def appendfile(args):
         return 3
 
 
-def appendsrc(args, files, rd):
+def appendsrc(args, files, rd, extralines=None):
     import oe.recipeutils
 
     srcdir = rd.getVar('S', True)
@@ -352,7 +352,7 @@ def appendsrc(args, files, rd):
         simplified[simple_uri] = uri
 
     copyfiles = {}
-    extralines = []
+    extralines = extralines or []
     for newfile, srcfile in files.iteritems():
         src_destdir = os.path.dirname(srcfile)
         if not args.use_workdir:
-- 
2.2.1

[PATCH 3/6] recipetool: also load plugins from BBPATH

[Christopher Larson]

This makes it easier to extend, as a layer can add its own sub-commands.

[YOCTO #7625]

Signed-off-by: Christopher Larson <kergoth@gmail.com>
---
 scripts/recipetool | 52 ++++++++++++++++++++++++++++++++--------------------
 1 file changed, 32 insertions(+), 20 deletions(-)

diff --git a/scripts/recipetool b/scripts/recipetool
index 3063cf7..6061d7b 100755
--- a/scripts/recipetool
+++ b/scripts/recipetool
@@ -36,11 +36,8 @@ def tinfoil_init(parserecipes):
     import logging
     tinfoil = bb.tinfoil.Tinfoil()
     tinfoil.prepare(not parserecipes)
-
-    for plugin in plugins:
-        if hasattr(plugin, 'tinfoil_init'):
-            plugin.tinfoil_init(tinfoil)
     tinfoil.logger.setLevel(logger.getEffectiveLevel())
+    return tinfoil
 
 def main():
 
@@ -49,28 +46,22 @@ def main():
         sys.exit(1)
 
     parser = argparse.ArgumentParser(description="OpenEmbedded recipe tool",
+                                     add_help=False,
                                      epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
     parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
     parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
     parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR')
-    subparsers = parser.add_subparsers(title='subcommands', metavar='<subcommand>')
-
-    scriptutils.load_plugins(logger, plugins, os.path.join(scripts_path, 'lib', 'recipetool'))
-    registered = False
-    for plugin in plugins:
-        if hasattr(plugin, 'register_command'):
-            registered = True
-            plugin.register_command(subparsers)
 
-    if not registered:
-        logger.error("No commands registered - missing plugins?")
-        sys.exit(1)
+    initial_args, unparsed_args = parser.parse_known_args(sys.argv[1:])
 
-    args = parser.parse_args()
+    # Help is added here rather than via add_help=True, as we don't want it to
+    # be handled by parse_known_args()
+    parser.add_argument('-h', '--help', action='help', default=argparse.SUPPRESS,
+                        help='show this help message and exit')
 
-    if args.debug:
+    if initial_args.debug:
         logger.setLevel(logging.DEBUG)
-    elif args.quiet:
+    elif initial_args.quiet:
         logger.setLevel(logging.ERROR)
 
     import scriptpath
@@ -80,10 +71,31 @@ def main():
         sys.exit(1)
     logger.debug('Found bitbake path: %s' % bitbakepath)
 
-    scriptutils.logger_setup_color(logger, args.color)
+    scriptutils.logger_setup_color(logger, initial_args.color)
+
+    subparsers = parser.add_subparsers(title='subcommands', metavar='<subcommand>')
+    tinfoil = tinfoil_init(False)
+    for path in ([scripts_path] +
+                 tinfoil.config_data.getVar('BBPATH', True).split(':')):
+        pluginpath = os.path.join(path, 'lib', 'recipetool')
+        scriptutils.load_plugins(logger, plugins, pluginpath)
+    registered = False
+    for plugin in plugins:
+        if hasattr(plugin, 'register_command'):
+            registered = True
+            plugin.register_command(subparsers)
+        if hasattr(plugin, 'tinfoil_init'):
+            plugin.tinfoil_init(tinfoil)
+
+    if not registered:
+        logger.error("No commands registered - missing plugins?")
+        sys.exit(1)
+
+    args = parser.parse_args(unparsed_args, namespace=initial_args)
 
     try:
-        tinfoil_init(getattr(args, 'parserecipes', False))
+        if getattr(args, 'parserecipes', False):
+            tinfoil.parseRecipes()
         ret = args.func(args)
     except bb.BBHandledException:
         ret = 1
-- 
2.2.1

[PATCH 4/6] devtool: also load plugins from BBPATH

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This makes it easier to extend, as a layer can add its own sub-commands.

tinfoil is now passed into the commands, as we needed to parse the
configuration metadata to get BBPATH, and we don't want to construct tinfoil
more than once, otherwise we have to deal with startup and shutdown of cooker.

[YOCTO #7625]

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 scripts/devtool                 | 57 +++++++++++++++++++++++++----------------
 scripts/lib/devtool/__init__.py |  4 +--
 scripts/lib/devtool/deploy.py   |  6 ++---
 scripts/lib/devtool/standard.py | 15 +++++------
 4 files changed, 46 insertions(+), 36 deletions(-)

diff --git a/scripts/devtool b/scripts/devtool
index fa799f6..01bb412 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -35,7 +35,7 @@ context = None
 scripts_path = os.path.dirname(os.path.realpath(__file__))
 lib_path = scripts_path + '/lib'
 sys.path = sys.path + [lib_path]
-from devtool import DevtoolError
+from devtool import DevtoolError, setup_tinfoil
 import scriptutils
 logger = scriptutils.logger_create('devtool')
 
@@ -186,37 +186,28 @@ def main():
         pth = os.path.dirname(pth)
 
     parser = argparse.ArgumentParser(description="OpenEmbedded development tool",
+                                     add_help=False,
                                      epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
     parser.add_argument('--basepath', help='Base directory of SDK / build directory')
     parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
     parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
     parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR')
 
-    subparsers = parser.add_subparsers(dest="subparser_name", title='subcommands', metavar='<subcommand>')
+    initial_args, unparsed_args = parser.parse_known_args()
 
-    if not context.fixed_setup:
-        parser_create_workspace = subparsers.add_parser('create-workspace',
-                                                        help='Set up a workspace',
-                                                        description='Sets up a new workspace. NOTE: other devtool subcommands will create a workspace automatically as needed, so you only need to use %(prog)s if you want to specify where the workspace should be located.')
-        parser_create_workspace.add_argument('layerpath', nargs='?', help='Path in which the workspace layer should be created')
-        parser_create_workspace.add_argument('--create-only', action="store_true", help='Only create the workspace layer, do not alter configuration')
-        parser_create_workspace.set_defaults(func=create_workspace)
+    # Help is added here rather than via add_help=True, as we don't want it to
+    # be handled by parse_known_args()
+    parser.add_argument('-h', '--help', action='help', default=argparse.SUPPRESS,
+                        help='show this help message and exit')
 
-    scriptutils.load_plugins(logger, plugins, os.path.join(scripts_path, 'lib', 'devtool'))
-    for plugin in plugins:
-        if hasattr(plugin, 'register_commands'):
-            plugin.register_commands(subparsers, context)
-
-    args = parser.parse_args()
-
-    if args.debug:
+    if initial_args.debug:
         logger.setLevel(logging.DEBUG)
-    elif args.quiet:
+    elif initial_args.quiet:
         logger.setLevel(logging.ERROR)
 
-    if args.basepath:
+    if initial_args.basepath:
         # Override
-        basepath = args.basepath
+        basepath = initial_args.basepath
     elif not context.fixed_setup:
         basepath = os.environ.get('BUILDDIR')
         if not basepath:
@@ -246,13 +237,35 @@ def main():
         logger.debug('Using standard bitbake path %s' % bitbakepath)
         scriptpath.add_oe_lib_path()
 
-    scriptutils.logger_setup_color(logger, args.color)
+    scriptutils.logger_setup_color(logger, initial_args.color)
+
+    tinfoil = setup_tinfoil(config_only=True)
+    for path in ([scripts_path] +
+                 tinfoil.config_data.getVar('BBPATH', True).split(':')):
+        pluginpath = os.path.join(path, 'lib', 'devtool')
+        scriptutils.load_plugins(logger, plugins, pluginpath)
+
+    subparsers = parser.add_subparsers(dest="subparser_name", title='subcommands', metavar='<subcommand>')
+
+    if not context.fixed_setup:
+        parser_create_workspace = subparsers.add_parser('create-workspace',
+                                                        help='Set up a workspace',
+                                                        description='Sets up a new workspace. NOTE: other devtool subcommands will create a workspace automatically as needed, so you only need to use %(prog)s if you want to specify where the workspace should be located.')
+        parser_create_workspace.add_argument('layerpath', nargs='?', help='Path in which the workspace layer should be created')
+        parser_create_workspace.add_argument('--create-only', action="store_true", help='Only create the workspace layer, do not alter configuration')
+        parser_create_workspace.set_defaults(func=create_workspace)
+
+    for plugin in plugins:
+        if hasattr(plugin, 'register_commands'):
+            plugin.register_commands(subparsers, context)
+
+    args = parser.parse_args(unparsed_args, namespace=initial_args)
 
     if args.subparser_name != 'create-workspace':
         read_workspace()
 
     try:
-        ret = args.func(args, config, basepath, workspace)
+        ret = args.func(args, config, basepath, workspace, tinfoil)
     except DevtoolError as err:
         if str(err):
             logger.error(str(err))
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 61b810c..b54ddf5 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -96,7 +96,7 @@ def exec_fakeroot(d, cmd, **kwargs):
             newenv[splitval[0]] = splitval[1]
     return subprocess.call("%s %s" % (fakerootcmd, cmd), env=newenv, **kwargs)
 
-def setup_tinfoil():
+def setup_tinfoil(config_only=False):
     """Initialize tinfoil api from bitbake"""
     import scriptpath
     bitbakepath = scriptpath.add_bitbake_lib_path()
@@ -106,7 +106,7 @@ def setup_tinfoil():
 
     import bb.tinfoil
     tinfoil = bb.tinfoil.Tinfoil()
-    tinfoil.prepare(False)
+    tinfoil.prepare(config_only)
     tinfoil.logger.setLevel(logger.getEffectiveLevel())
     return tinfoil
 
diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py
index 448db96..3bc59de 100644
--- a/scripts/lib/devtool/deploy.py
+++ b/scripts/lib/devtool/deploy.py
@@ -28,7 +28,7 @@ def plugin_init(pluginlist):
     pass
 
 
-def deploy(args, config, basepath, workspace):
+def deploy(args, config, basepath, workspace, tinfoil):
     """Entry point for the devtool 'deploy' subcommand"""
     import re
     import oe.recipeutils
@@ -46,7 +46,7 @@ def deploy(args, config, basepath, workspace):
     deploy_dir = os.path.join(basepath, 'target_deploy', args.target)
     deploy_file = os.path.join(deploy_dir, args.recipename + '.list')
 
-    tinfoil = setup_tinfoil()
+    tinfoil.parseRecipes()
     try:
         rd = oe.recipeutils.parse_recipe_simple(tinfoil.cooker, args.recipename, tinfoil.config_data)
     except Exception as e:
@@ -100,7 +100,7 @@ def deploy(args, config, basepath, workspace):
 
     return 0
 
-def undeploy(args, config, basepath, workspace):
+def undeploy(args, config, basepath, workspace, tinfoil):
     """Entry point for the devtool 'undeploy' subcommand"""
     deploy_file = os.path.join(basepath, 'target_deploy', args.target, args.recipename + '.list')
     if not os.path.exists(deploy_file):
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index ea21877..8fe32b7 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -193,12 +193,11 @@ def _ls_tree(directory):
     return ret
 
 
-def extract(args, config, basepath, workspace):
+def extract(args, config, basepath, workspace, tinfoil):
     """Entry point for the devtool 'extract' subcommand"""
     import bb
 
-    tinfoil = setup_tinfoil()
-
+    tinfoil.parseRecipes()
     rd = _parse_recipe(config, tinfoil, args.recipename, True)
     if not rd:
         return 1
@@ -409,7 +408,7 @@ def _check_preserve(config, recipename):
                     tf.write(line)
     os.rename(newfile, origfile)
 
-def modify(args, config, basepath, workspace):
+def modify(args, config, basepath, workspace, tinfoil):
     """Entry point for the devtool 'modify' subcommand"""
     import bb
     import oe.recipeutils
@@ -423,8 +422,7 @@ def modify(args, config, basepath, workspace):
                            "(specify -x to extract source from recipe)" %
                            args.srctree)
 
-    tinfoil = setup_tinfoil()
-
+    tinfoil.parseRecipes()
     rd = _parse_recipe(config, tinfoil, args.recipename, True)
     if not rd:
         return 1
@@ -747,7 +745,7 @@ def _update_recipe_patch(args, config, srctree, rd, config_data):
 
     _remove_patch_files(args, removepatches, destpath)
 
-def update_recipe(args, config, basepath, workspace):
+def update_recipe(args, config, basepath, workspace, tinfoil):
     """Entry point for the devtool 'update-recipe' subcommand"""
     if not args.recipename in workspace:
         raise DevtoolError("no recipe named %s in your workspace" %
@@ -761,8 +759,7 @@ def update_recipe(args, config, basepath, workspace):
             raise DevtoolError('conf/layer.conf not found in bbappend '
                                'destination layer "%s"' % args.append)
 
-    tinfoil = setup_tinfoil()
-
+    tinfoil.parseRecipes()
     rd = _parse_recipe(config, tinfoil, args.recipename, True)
     if not rd:
         return 1
-- 
2.2.1

Re: [PATCH 4/6] devtool: also load plugins from BBPATH

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 2104 bytes --]

On 14 July 2015 at 16:57, Christopher Larson <kergoth@gmail.com> wrote:

> +        parser_create_workspace =
> subparsers.add_parser('create-workspace',
> +                                                        help='Set up a
> workspace',
> +                                                        description='Sets
> up a new workspace. NOTE: other devtool subcommands will create a
> workspace automatically as needed, so you only need to use %(prog)s if you
> want to specify where the workspace should be located.')
> +        parser_create_workspace.add_argument('layerpath', nargs='?',
> help='Path in which the workspace layer should be created')
> +        parser_create_workspace.add_argument('--create-only',
> action="store_true", help='Only create the workspace layer, do not alter
> configuration')
> +        parser_create_workspace.set_defaults(func=create_workspace)
>

This is causing the selftests to abort:

FAIL: test_create_workspace (oeqa.selftest.devtool.DevtoolTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ross/Yocto/poky/meta/lib/oeqa/utils/decorators.py", line 102,
in wrapped_f
    return func(*args)
  File "/home/ross/Yocto/poky/meta/lib/oeqa/selftest/devtool.py", line 97,
in test_create_workspace
    result = runCmd('devtool create-workspace %s' % tempdir)
  File "/home/ross/Yocto/poky/meta/lib/oeqa/utils/commands.py", line 104,
in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status
%d:\n%s" % (command, result.status, result.output))
AssertionError: Command 'devtool create-workspace /tmp/devtoolqaCuFJBv'
returned non-zero exit status 1:
Traceback (most recent call last):
  File "/home/ross/Yocto/poky/scripts/devtool", line 279, in <module>
    ret = main()
  File "/home/ross/Yocto/poky/scripts/devtool", line 268, in main
    ret = args.func(args, config, basepath, workspace, tinfoil)
TypeError: create_workspace() takes exactly 4 arguments (5 given)

(MACHINE=qemux86 oe-selftest --run-tests devtool)

Ross

[-- Attachment #2: Type: text/html, Size: 3406 bytes --]

Re: [PATCH 4/6] devtool: also load plugins from BBPATH

[ChenQi]

Hi Christopher,

I suspect this patch is causing failure for 'populate_sdk_ext' task.
Could you please take a look at it?

See error message below.

Best Regards,
Chen Qi

=======================

ERROR: Error executing a python function in 
/buildarea2/chenqi/poky/meta/recipes-core/images/core-image-minimal.bb:

The stack trace of python calls that resulted in this exception/failure was:
File: 'copy_buildsystem', lineno: 127, function: <module>
      0123:    with open(baseoutpath + '/conf/work-config.inc', 'w') as f:
      0124:        pass
      0125:
      0126:
  *** 0127:copy_buildsystem(d)
      0128:
File: 'copy_buildsystem', lineno: 53, function: copy_buildsystem
      0049:    with open(os.path.join(baseoutpath, 'conf', 
'devtool.conf'), 'w') as f:
      0050:        config.write(f)
      0051:
      0052:    # Create a layer for new recipes / appends
  *** 0053:    bb.process.run("devtool --basepath %s create-workspace 
--create-only %s" % (baseoutpath, os.path.join(baseoutpath, 'workspace')))
      0054:
      0055:    # Create bblayers.conf
      0056:    bb.utils.mkdirhier(baseoutpath + '/conf')
      0057:    with open(baseoutpath + '/conf/bblayers.conf', 'w') as f:
File: '/buildarea2/chenqi/poky/bitbake/lib/bb/process.py', lineno: 152, 
function: run
      0148:    else:
      0149:        stdout, stderr = pipe.communicate(input)
      0150:
      0151:    if pipe.returncode != 0:
  *** 0152:        raise ExecutionError(cmd, pipe.returncode, stdout, 
stderr)
      0153:    return stdout, stderr
Exception: ExecutionError: Execution of 'devtool --basepath 
/buildarea2/chenqi/poky/build-systemd/tmp-2/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/sdk-ext/image//opt/poky/1.8+snapshot 
create-workspace --create-only 
/buildarea2/chenqi/poky/build-systemd/tmp-2/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/sdk-ext/image//opt/poky/1.8+snapshot/workspace' 
failed with exit code 1:
ERROR: Only one copy of bitbake should be run against a build directory
Traceback (most recent call last):
   File "/buildarea2/chenqi/poky/scripts/devtool", line 281, in <module>
     ret = main()
   File "/buildarea2/chenqi/poky/scripts/devtool", line 242, in main
     tinfoil = setup_tinfoil(config_only=True)
   File "/buildarea2/chenqi/poky/scripts/lib/devtool/__init__.py", line 
108, in setup_tinfoil
     tinfoil = bb.tinfoil.Tinfoil()
   File 
"/buildarea2/chenqi/poky/build-systemd/tmp-2/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/sdk-ext/image//opt/poky/1.8+snapshot/layers/poky/bitbake/lib/bb/tinfoil.py", 
line 54, in __init__
     self.cooker = BBCooker(self.config, features)
   File 
"/buildarea2/chenqi/poky/build-systemd/tmp-2/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/sdk-ext/image//opt/poky/1.8+snapshot/layers/poky/bitbake/lib/bb/cooker.py", 
line 155, in __init__
     bb.fatal("Only one copy of bitbake should be run against a build 
directory")
BBHandledException


ERROR: Function failed: copy_buildsystem
ERROR: Logfile of failure stored in: 
/buildarea2/chenqi/poky/build-systemd/tmp-2/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/temp/log.do_populate_sdk_ext.3616
ERROR: Task 12 
(/buildarea2/chenqi/poky/meta/recipes-core/images/core-image-minimal.bb, 
do_populate_sdk_ext) failed with exit code '1'




On 07/14/2015 11:57 PM, Christopher Larson wrote:
> From: Christopher Larson <chris_larson@mentor.com>
>
> This makes it easier to extend, as a layer can add its own sub-commands.
>
> tinfoil is now passed into the commands, as we needed to parse the
> configuration metadata to get BBPATH, and we don't want to construct tinfoil
> more than once, otherwise we have to deal with startup and shutdown of cooker.
>
> [YOCTO #7625]
>
> Signed-off-by: Christopher Larson <chris_larson@mentor.com>
> ---
>   scripts/devtool                 | 57 +++++++++++++++++++++++++----------------
>   scripts/lib/devtool/__init__.py |  4 +--
>   scripts/lib/devtool/deploy.py   |  6 ++---
>   scripts/lib/devtool/standard.py | 15 +++++------
>   4 files changed, 46 insertions(+), 36 deletions(-)
>
> diff --git a/scripts/devtool b/scripts/devtool
> index fa799f6..01bb412 100755
> --- a/scripts/devtool
> +++ b/scripts/devtool
> @@ -35,7 +35,7 @@ context = None
>   scripts_path = os.path.dirname(os.path.realpath(__file__))
>   lib_path = scripts_path + '/lib'
>   sys.path = sys.path + [lib_path]
> -from devtool import DevtoolError
> +from devtool import DevtoolError, setup_tinfoil
>   import scriptutils
>   logger = scriptutils.logger_create('devtool')
>   
> @@ -186,37 +186,28 @@ def main():
>           pth = os.path.dirname(pth)
>   
>       parser = argparse.ArgumentParser(description="OpenEmbedded development tool",
> +                                     add_help=False,
>                                        epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
>       parser.add_argument('--basepath', help='Base directory of SDK / build directory')
>       parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
>       parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
>       parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR')
>   
> -    subparsers = parser.add_subparsers(dest="subparser_name", title='subcommands', metavar='<subcommand>')
> +    initial_args, unparsed_args = parser.parse_known_args()
>   
> -    if not context.fixed_setup:
> -        parser_create_workspace = subparsers.add_parser('create-workspace',
> -                                                        help='Set up a workspace',
> -                                                        description='Sets up a new workspace. NOTE: other devtool subcommands will create a workspace automatically as needed, so you only need to use %(prog)s if you want to specify where the workspace should be located.')
> -        parser_create_workspace.add_argument('layerpath', nargs='?', help='Path in which the workspace layer should be created')
> -        parser_create_workspace.add_argument('--create-only', action="store_true", help='Only create the workspace layer, do not alter configuration')
> -        parser_create_workspace.set_defaults(func=create_workspace)
> +    # Help is added here rather than via add_help=True, as we don't want it to
> +    # be handled by parse_known_args()
> +    parser.add_argument('-h', '--help', action='help', default=argparse.SUPPRESS,
> +                        help='show this help message and exit')
>   
> -    scriptutils.load_plugins(logger, plugins, os.path.join(scripts_path, 'lib', 'devtool'))
> -    for plugin in plugins:
> -        if hasattr(plugin, 'register_commands'):
> -            plugin.register_commands(subparsers, context)
> -
> -    args = parser.parse_args()
> -
> -    if args.debug:
> +    if initial_args.debug:
>           logger.setLevel(logging.DEBUG)
> -    elif args.quiet:
> +    elif initial_args.quiet:
>           logger.setLevel(logging.ERROR)
>   
> -    if args.basepath:
> +    if initial_args.basepath:
>           # Override
> -        basepath = args.basepath
> +        basepath = initial_args.basepath
>       elif not context.fixed_setup:
>           basepath = os.environ.get('BUILDDIR')
>           if not basepath:
> @@ -246,13 +237,35 @@ def main():
>           logger.debug('Using standard bitbake path %s' % bitbakepath)
>           scriptpath.add_oe_lib_path()
>   
> -    scriptutils.logger_setup_color(logger, args.color)
> +    scriptutils.logger_setup_color(logger, initial_args.color)
> +
> +    tinfoil = setup_tinfoil(config_only=True)
> +    for path in ([scripts_path] +
> +                 tinfoil.config_data.getVar('BBPATH', True).split(':')):
> +        pluginpath = os.path.join(path, 'lib', 'devtool')
> +        scriptutils.load_plugins(logger, plugins, pluginpath)
> +
> +    subparsers = parser.add_subparsers(dest="subparser_name", title='subcommands', metavar='<subcommand>')
> +
> +    if not context.fixed_setup:
> +        parser_create_workspace = subparsers.add_parser('create-workspace',
> +                                                        help='Set up a workspace',
> +                                                        description='Sets up a new workspace. NOTE: other devtool subcommands will create a workspace automatically as needed, so you only need to use %(prog)s if you want to specify where the workspace should be located.')
> +        parser_create_workspace.add_argument('layerpath', nargs='?', help='Path in which the workspace layer should be created')
> +        parser_create_workspace.add_argument('--create-only', action="store_true", help='Only create the workspace layer, do not alter configuration')
> +        parser_create_workspace.set_defaults(func=create_workspace)
> +
> +    for plugin in plugins:
> +        if hasattr(plugin, 'register_commands'):
> +            plugin.register_commands(subparsers, context)
> +
> +    args = parser.parse_args(unparsed_args, namespace=initial_args)
>   
>       if args.subparser_name != 'create-workspace':
>           read_workspace()
>   
>       try:
> -        ret = args.func(args, config, basepath, workspace)
> +        ret = args.func(args, config, basepath, workspace, tinfoil)
>       except DevtoolError as err:
>           if str(err):
>               logger.error(str(err))
> diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
> index 61b810c..b54ddf5 100644
> --- a/scripts/lib/devtool/__init__.py
> +++ b/scripts/lib/devtool/__init__.py
> @@ -96,7 +96,7 @@ def exec_fakeroot(d, cmd, **kwargs):
>               newenv[splitval[0]] = splitval[1]
>       return subprocess.call("%s %s" % (fakerootcmd, cmd), env=newenv, **kwargs)
>   
> -def setup_tinfoil():
> +def setup_tinfoil(config_only=False):
>       """Initialize tinfoil api from bitbake"""
>       import scriptpath
>       bitbakepath = scriptpath.add_bitbake_lib_path()
> @@ -106,7 +106,7 @@ def setup_tinfoil():
>   
>       import bb.tinfoil
>       tinfoil = bb.tinfoil.Tinfoil()
> -    tinfoil.prepare(False)
> +    tinfoil.prepare(config_only)
>       tinfoil.logger.setLevel(logger.getEffectiveLevel())
>       return tinfoil
>   
> diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py
> index 448db96..3bc59de 100644
> --- a/scripts/lib/devtool/deploy.py
> +++ b/scripts/lib/devtool/deploy.py
> @@ -28,7 +28,7 @@ def plugin_init(pluginlist):
>       pass
>   
>   
> -def deploy(args, config, basepath, workspace):
> +def deploy(args, config, basepath, workspace, tinfoil):
>       """Entry point for the devtool 'deploy' subcommand"""
>       import re
>       import oe.recipeutils
> @@ -46,7 +46,7 @@ def deploy(args, config, basepath, workspace):
>       deploy_dir = os.path.join(basepath, 'target_deploy', args.target)
>       deploy_file = os.path.join(deploy_dir, args.recipename + '.list')
>   
> -    tinfoil = setup_tinfoil()
> +    tinfoil.parseRecipes()
>       try:
>           rd = oe.recipeutils.parse_recipe_simple(tinfoil.cooker, args.recipename, tinfoil.config_data)
>       except Exception as e:
> @@ -100,7 +100,7 @@ def deploy(args, config, basepath, workspace):
>   
>       return 0
>   
> -def undeploy(args, config, basepath, workspace):
> +def undeploy(args, config, basepath, workspace, tinfoil):
>       """Entry point for the devtool 'undeploy' subcommand"""
>       deploy_file = os.path.join(basepath, 'target_deploy', args.target, args.recipename + '.list')
>       if not os.path.exists(deploy_file):
> diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
> index ea21877..8fe32b7 100644
> --- a/scripts/lib/devtool/standard.py
> +++ b/scripts/lib/devtool/standard.py
> @@ -193,12 +193,11 @@ def _ls_tree(directory):
>       return ret
>   
>   
> -def extract(args, config, basepath, workspace):
> +def extract(args, config, basepath, workspace, tinfoil):
>       """Entry point for the devtool 'extract' subcommand"""
>       import bb
>   
> -    tinfoil = setup_tinfoil()
> -
> +    tinfoil.parseRecipes()
>       rd = _parse_recipe(config, tinfoil, args.recipename, True)
>       if not rd:
>           return 1
> @@ -409,7 +408,7 @@ def _check_preserve(config, recipename):
>                       tf.write(line)
>       os.rename(newfile, origfile)
>   
> -def modify(args, config, basepath, workspace):
> +def modify(args, config, basepath, workspace, tinfoil):
>       """Entry point for the devtool 'modify' subcommand"""
>       import bb
>       import oe.recipeutils
> @@ -423,8 +422,7 @@ def modify(args, config, basepath, workspace):
>                              "(specify -x to extract source from recipe)" %
>                              args.srctree)
>   
> -    tinfoil = setup_tinfoil()
> -
> +    tinfoil.parseRecipes()
>       rd = _parse_recipe(config, tinfoil, args.recipename, True)
>       if not rd:
>           return 1
> @@ -747,7 +745,7 @@ def _update_recipe_patch(args, config, srctree, rd, config_data):
>   
>       _remove_patch_files(args, removepatches, destpath)
>   
> -def update_recipe(args, config, basepath, workspace):
> +def update_recipe(args, config, basepath, workspace, tinfoil):
>       """Entry point for the devtool 'update-recipe' subcommand"""
>       if not args.recipename in workspace:
>           raise DevtoolError("no recipe named %s in your workspace" %
> @@ -761,8 +759,7 @@ def update_recipe(args, config, basepath, workspace):
>               raise DevtoolError('conf/layer.conf not found in bbappend '
>                                  'destination layer "%s"' % args.append)
>   
> -    tinfoil = setup_tinfoil()
> -
> +    tinfoil.parseRecipes()
>       rd = _parse_recipe(config, tinfoil, args.recipename, True)
>       if not rd:
>           return 1

Re: [PATCH 4/6] devtool: also load plugins from BBPATH

[Paul Eggleton]

Hmm, I guess we will have to add a command-line parameter to specify BBPATH 
(or the full plugin search path) so that tinfoil doesn't have to be 
instantiated when it's specified.

Cheers,
Paul

On Wednesday 29 July 2015 14:58:37 ChenQi wrote:
> Hi Christopher,
> 
> I suspect this patch is causing failure for 'populate_sdk_ext' task.
> Could you please take a look at it?
> 
> See error message below.
> 
> Best Regards,
> Chen Qi
> 
> =======================
> 
> ERROR: Error executing a python function in
> /buildarea2/chenqi/poky/meta/recipes-core/images/core-image-minimal.bb:
> 
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'copy_buildsystem', lineno: 127, function: <module>
>       0123:    with open(baseoutpath + '/conf/work-config.inc', 'w') as f:
>       0124:        pass
>       0125:
>       0126:
>   *** 0127:copy_buildsystem(d)
>       0128:
> File: 'copy_buildsystem', lineno: 53, function: copy_buildsystem
>       0049:    with open(os.path.join(baseoutpath, 'conf',
> 'devtool.conf'), 'w') as f:
>       0050:        config.write(f)
>       0051:
>       0052:    # Create a layer for new recipes / appends
>   *** 0053:    bb.process.run("devtool --basepath %s create-workspace
> --create-only %s" % (baseoutpath, os.path.join(baseoutpath, 'workspace')))
>       0054:
>       0055:    # Create bblayers.conf
>       0056:    bb.utils.mkdirhier(baseoutpath + '/conf')
>       0057:    with open(baseoutpath + '/conf/bblayers.conf', 'w') as f:
> File: '/buildarea2/chenqi/poky/bitbake/lib/bb/process.py', lineno: 152,
> function: run
>       0148:    else:
>       0149:        stdout, stderr = pipe.communicate(input)
>       0150:
>       0151:    if pipe.returncode != 0:
>   *** 0152:        raise ExecutionError(cmd, pipe.returncode, stdout,
> stderr)
>       0153:    return stdout, stderr
> Exception: ExecutionError: Execution of 'devtool --basepath
> /buildarea2/chenqi/poky/build-systemd/tmp-2/work/qemuarm-poky-linux-gnueabi/
> core-image-minimal/1.0-r0/sdk-ext/image//opt/poky/1.8+snapshot
> create-workspace --create-only
> /buildarea2/chenqi/poky/build-systemd/tmp-2/work/qemuarm-poky-linux-gnueabi/
> core-image-minimal/1.0-r0/sdk-ext/image//opt/poky/1.8+snapshot/workspace'
> failed with exit code 1:
> ERROR: Only one copy of bitbake should be run against a build directory
> Traceback (most recent call last):
>    File "/buildarea2/chenqi/poky/scripts/devtool", line 281, in <module>
>      ret = main()
>    File "/buildarea2/chenqi/poky/scripts/devtool", line 242, in main
>      tinfoil = setup_tinfoil(config_only=True)
>    File "/buildarea2/chenqi/poky/scripts/lib/devtool/__init__.py", line
> 108, in setup_tinfoil
>      tinfoil = bb.tinfoil.Tinfoil()
>    File
> "/buildarea2/chenqi/poky/build-systemd/tmp-2/work/qemuarm-poky-linux-gnueabi
> /core-image-minimal/1.0-r0/sdk-ext/image//opt/poky/1.8+snapshot/layers/poky/
> bitbake/lib/bb/tinfoil.py", line 54, in __init__
>      self.cooker = BBCooker(self.config, features)
>    File
> "/buildarea2/chenqi/poky/build-systemd/tmp-2/work/qemuarm-poky-linux-gnueabi
> /core-image-minimal/1.0-r0/sdk-ext/image//opt/poky/1.8+snapshot/layers/poky/
> bitbake/lib/bb/cooker.py", line 155, in __init__
>      bb.fatal("Only one copy of bitbake should be run against a build
> directory")
> BBHandledException
> 
> 
> ERROR: Function failed: copy_buildsystem
> ERROR: Logfile of failure stored in:
> /buildarea2/chenqi/poky/build-systemd/tmp-2/work/qemuarm-poky-linux-gnueabi/
> core-image-minimal/1.0-r0/temp/log.do_populate_sdk_ext.3616 ERROR: Task 12
> (/buildarea2/chenqi/poky/meta/recipes-core/images/core-image-minimal.bb,
> do_populate_sdk_ext) failed with exit code '1'
> 
> On 07/14/2015 11:57 PM, Christopher Larson wrote:
> > From: Christopher Larson <chris_larson@mentor.com>
> > 
> > This makes it easier to extend, as a layer can add its own sub-commands.
> > 
> > tinfoil is now passed into the commands, as we needed to parse the
> > configuration metadata to get BBPATH, and we don't want to construct
> > tinfoil more than once, otherwise we have to deal with startup and
> > shutdown of cooker.
> > 
> > [YOCTO #7625]
> > 
> > Signed-off-by: Christopher Larson <chris_larson@mentor.com>
> > ---
> > 
> >   scripts/devtool                 | 57
> >   +++++++++++++++++++++++++----------------
> >   scripts/lib/devtool/__init__.py |  4 +--
> >   scripts/lib/devtool/deploy.py   |  6 ++---
> >   scripts/lib/devtool/standard.py | 15 +++++------
> >   4 files changed, 46 insertions(+), 36 deletions(-)
> > 
> > diff --git a/scripts/devtool b/scripts/devtool
> > index fa799f6..01bb412 100755
> > --- a/scripts/devtool
> > +++ b/scripts/devtool
> > @@ -35,7 +35,7 @@ context = None
> > 
> >   scripts_path = os.path.dirname(os.path.realpath(__file__))
> >   lib_path = scripts_path + '/lib'
> >   sys.path = sys.path + [lib_path]
> > 
> > -from devtool import DevtoolError
> > +from devtool import DevtoolError, setup_tinfoil
> > 
> >   import scriptutils
> >   logger = scriptutils.logger_create('devtool')
> > 
> > @@ -186,37 +186,28 @@ def main():
> >           pth = os.path.dirname(pth)
> >       
> >       parser = argparse.ArgumentParser(description="OpenEmbedded
> >       development tool",> 
> > +                                     add_help=False,
> > 
> >                                        epilog="Use %(prog)s <subcommand>
> >                                        --help to get help on a specific
> >                                        command")>       
> >       parser.add_argument('--basepath', help='Base directory of SDK /
> >       build directory') parser.add_argument('-d', '--debug', help='Enable
> >       debug output', action='store_true') parser.add_argument('-q',
> >       '--quiet', help='Print only errors', action='store_true')
> >       parser.add_argument('--color', choices=['auto', 'always', 'never'],
> >       default='auto', help='Colorize output (where %(metavar)s is
> >       %(choices)s)', metavar='COLOR')> 
> > -    subparsers = parser.add_subparsers(dest="subparser_name",
> > title='subcommands', metavar='<subcommand>') +    initial_args,
> > unparsed_args = parser.parse_known_args()
> > 
> > -    if not context.fixed_setup:
> > -        parser_create_workspace =
> > subparsers.add_parser('create-workspace', -                              
> >                          help='Set up a workspace', -                    
> >                                    description='Sets up a new workspace.
> > NOTE: other devtool subcommands will create a workspace automatically as
> > needed, so you only need to use %(prog)s if you want to specify where the
> > workspace should be located.') -       
> > parser_create_workspace.add_argument('layerpath', nargs='?', help='Path
> > in which the workspace layer should be created') -       
> > parser_create_workspace.add_argument('--create-only',
> > action="store_true", help='Only create the workspace layer, do not alter
> > configuration') -       
> > parser_create_workspace.set_defaults(func=create_workspace) +    # Help
> > is added here rather than via add_help=True, as we don't want it to +   
> > # be handled by parse_known_args()
> > +    parser.add_argument('-h', '--help', action='help',
> > default=argparse.SUPPRESS, +                        help='show this help
> > message and exit')
> > 
> > -    scriptutils.load_plugins(logger, plugins, os.path.join(scripts_path,
> > 'lib', 'devtool')) -    for plugin in plugins:
> > -        if hasattr(plugin, 'register_commands'):
> > -            plugin.register_commands(subparsers, context)
> > -
> > -    args = parser.parse_args()
> > -
> > -    if args.debug:
> > 
> > +    if initial_args.debug:
> >           logger.setLevel(logging.DEBUG)
> > 
> > -    elif args.quiet:
> > 
> > +    elif initial_args.quiet:
> >           logger.setLevel(logging.ERROR)
> > 
> > -    if args.basepath:
> > 
> > +    if initial_args.basepath:
> >           # Override
> > 
> > -        basepath = args.basepath
> > +        basepath = initial_args.basepath
> > 
> >       elif not context.fixed_setup:
> >           basepath = os.environ.get('BUILDDIR')
> > 
> >           if not basepath:
> > @@ -246,13 +237,35 @@ def main():
> >           logger.debug('Using standard bitbake path %s' % bitbakepath)
> >           scriptpath.add_oe_lib_path()
> > 
> > -    scriptutils.logger_setup_color(logger, args.color)
> > +    scriptutils.logger_setup_color(logger, initial_args.color)
> > +
> > +    tinfoil = setup_tinfoil(config_only=True)
> > +    for path in ([scripts_path] +
> > +                 tinfoil.config_data.getVar('BBPATH', True).split(':')):
> > +        pluginpath = os.path.join(path, 'lib', 'devtool')
> > +        scriptutils.load_plugins(logger, plugins, pluginpath)
> > +
> > +    subparsers = parser.add_subparsers(dest="subparser_name",
> > title='subcommands', metavar='<subcommand>') +
> > +    if not context.fixed_setup:
> > +        parser_create_workspace =
> > subparsers.add_parser('create-workspace', +                              
> >                          help='Set up a workspace', +                    
> >                                    description='Sets up a new workspace.
> > NOTE: other devtool subcommands will create a workspace automatically as
> > needed, so you only need to use %(prog)s if you want to specify where the
> > workspace should be located.') +       
> > parser_create_workspace.add_argument('layerpath', nargs='?', help='Path
> > in which the workspace layer should be created') +       
> > parser_create_workspace.add_argument('--create-only',
> > action="store_true", help='Only create the workspace layer, do not alter
> > configuration') +       
> > parser_create_workspace.set_defaults(func=create_workspace) +
> > +    for plugin in plugins:
> > +        if hasattr(plugin, 'register_commands'):
> > +            plugin.register_commands(subparsers, context)
> > +
> > +    args = parser.parse_args(unparsed_args, namespace=initial_args)
> > 
> >       if args.subparser_name != 'create-workspace':
> >           read_workspace()
> >       
> >       try:
> > -        ret = args.func(args, config, basepath, workspace)
> > +        ret = args.func(args, config, basepath, workspace, tinfoil)
> > 
> >       except DevtoolError as err:
> >           if str(err):
> >               logger.error(str(err))
> > 
> > diff --git a/scripts/lib/devtool/__init__.py
> > b/scripts/lib/devtool/__init__.py index 61b810c..b54ddf5 100644
> > --- a/scripts/lib/devtool/__init__.py
> > +++ b/scripts/lib/devtool/__init__.py
> > 
> > @@ -96,7 +96,7 @@ def exec_fakeroot(d, cmd, **kwargs):
> >               newenv[splitval[0]] = splitval[1]
> >       
> >       return subprocess.call("%s %s" % (fakerootcmd, cmd), env=newenv,
> >       **kwargs)
> > 
> > -def setup_tinfoil():
> > 
> > +def setup_tinfoil(config_only=False):
> >       """Initialize tinfoil api from bitbake"""
> >       import scriptpath
> >       bitbakepath = scriptpath.add_bitbake_lib_path()
> > 
> > @@ -106,7 +106,7 @@ def setup_tinfoil():
> >       import bb.tinfoil
> >       tinfoil = bb.tinfoil.Tinfoil()
> > 
> > -    tinfoil.prepare(False)
> > +    tinfoil.prepare(config_only)
> > 
> >       tinfoil.logger.setLevel(logger.getEffectiveLevel())
> >       return tinfoil
> > 
> > diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py
> > index 448db96..3bc59de 100644
> > --- a/scripts/lib/devtool/deploy.py
> > +++ b/scripts/lib/devtool/deploy.py
> > 
> > @@ -28,7 +28,7 @@ def plugin_init(pluginlist):
> >       pass
> > 
> > -def deploy(args, config, basepath, workspace):
> > 
> > +def deploy(args, config, basepath, workspace, tinfoil):
> >       """Entry point for the devtool 'deploy' subcommand"""
> >       import re
> >       import oe.recipeutils
> > 
> > @@ -46,7 +46,7 @@ def deploy(args, config, basepath, workspace):
> >       deploy_dir = os.path.join(basepath, 'target_deploy', args.target)
> >       deploy_file = os.path.join(deploy_dir, args.recipename + '.list')
> > 
> > -    tinfoil = setup_tinfoil()
> > +    tinfoil.parseRecipes()
> > 
> >       try:
> >           rd = oe.recipeutils.parse_recipe_simple(tinfoil.cooker,
> >           args.recipename, tinfoil.config_data)>       
> >       except Exception as e:
> > @@ -100,7 +100,7 @@ def deploy(args, config, basepath, workspace):
> >       return 0
> > 
> > -def undeploy(args, config, basepath, workspace):
> > 
> > +def undeploy(args, config, basepath, workspace, tinfoil):
> >       """Entry point for the devtool 'undeploy' subcommand"""
> >       deploy_file = os.path.join(basepath, 'target_deploy', args.target,
> >       args.recipename + '.list')> 
> >       if not os.path.exists(deploy_file):
> > diff --git a/scripts/lib/devtool/standard.py
> > b/scripts/lib/devtool/standard.py index ea21877..8fe32b7 100644
> > --- a/scripts/lib/devtool/standard.py
> > +++ b/scripts/lib/devtool/standard.py
> > 
> > @@ -193,12 +193,11 @@ def _ls_tree(directory):
> >       return ret
> > 
> > -def extract(args, config, basepath, workspace):
> > 
> > +def extract(args, config, basepath, workspace, tinfoil):
> >       """Entry point for the devtool 'extract' subcommand"""
> >       import bb
> > 
> > -    tinfoil = setup_tinfoil()
> > -
> > +    tinfoil.parseRecipes()
> > 
> >       rd = _parse_recipe(config, tinfoil, args.recipename, True)
> >       
> >       if not rd:
> >           return 1
> > 
> > @@ -409,7 +408,7 @@ def _check_preserve(config, recipename):
> >                       tf.write(line)
> >       
> >       os.rename(newfile, origfile)
> > 
> > -def modify(args, config, basepath, workspace):
> > 
> > +def modify(args, config, basepath, workspace, tinfoil):
> >       """Entry point for the devtool 'modify' subcommand"""
> >       import bb
> >       import oe.recipeutils
> > 
> > @@ -423,8 +422,7 @@ def modify(args, config, basepath, workspace):
> >                              "(specify -x to extract source from recipe)"
> >                              %
> >                              args.srctree)
> > 
> > -    tinfoil = setup_tinfoil()
> > -
> > +    tinfoil.parseRecipes()
> > 
> >       rd = _parse_recipe(config, tinfoil, args.recipename, True)
> >       
> >       if not rd:
> >           return 1
> > 
> > @@ -747,7 +745,7 @@ def _update_recipe_patch(args, config, srctree, rd, 
config_data):
> >       _remove_patch_files(args, removepatches, destpath)
> > 
> > -def update_recipe(args, config, basepath, workspace):
> > 
> > +def update_recipe(args, config, basepath, workspace, tinfoil):
> >       """Entry point for the devtool 'update-recipe' subcommand"""
> >       
> >       if not args.recipename in workspace:
> >           raise DevtoolError("no recipe named %s in your workspace" %
> > 
> > @@ -761,8 +759,7 @@ def update_recipe(args, config, basepath, workspace):
> >               raise DevtoolError('conf/layer.conf not found in bbappend '
> >               
> >                                  'destination layer "%s"' % args.append)
> > 
> > -    tinfoil = setup_tinfoil()
> > -
> > +    tinfoil.parseRecipes()
> > 
> >       rd = _parse_recipe(config, tinfoil, args.recipename, True)
> >       
> >       if not rd:
> >           return 1

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 4/6] devtool: also load plugins from BBPATH

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

On Wed, Jul 29, 2015 at 2:56 AM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> Hmm, I guess we will have to add a command-line parameter to specify BBPATH
> (or the full plugin search path) so that tinfoil doesn't have to be
> instantiated when it's specified.
>

Indeed, sorry about that, I didn’t even realize populate_sdk_ext used
devtool from a bbclass :)
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 993 bytes --]

Re: [PATCH 4/6] devtool: also load plugins from BBPATH

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

On Wed, Jul 29, 2015 at 9:40 AM, Christopher Larson <kergoth@gmail.com>
wrote:

> On Wed, Jul 29, 2015 at 2:56 AM, Paul Eggleton <
> paul.eggleton@linux.intel.com> wrote:
>
>> Hmm, I guess we will have to add a command-line parameter to specify
>> BBPATH
>> (or the full plugin search path) so that tinfoil doesn't have to be
>> instantiated when it's specified.
>>
>
> Indeed, sorry about that, I didn’t even realize populate_sdk_ext used
> devtool from a bbclass :)
>

I'll submit patches for this and one other populate_sdk_ext issue shortly.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1327 bytes --]

[PATCH 5/6] oe-selftest: obey oeqa.selftest.__path__

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This ensures that all paths that hold selftest tests will be checked
(oeqa.selftest is a namespace package).

[YOCTO #7625]

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 scripts/oe-selftest | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index fa6245a..c19c692 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -142,11 +142,12 @@ def get_tests(exclusive_modules=[], include_hidden=False):
     for x in exclusive_modules:
         testslist.append('oeqa.selftest.' + x)
     if not testslist:
-        testpath = os.path.abspath(os.path.dirname(oeqa.selftest.__file__))
-        files = sorted([f for f in os.listdir(testpath) if f.endswith('.py') and not (f.startswith('_') and not include_hidden) and not f.startswith('__') and f != 'base.py'])
-        for f in files:
-            module = 'oeqa.selftest.' + f[:-3]
-            testslist.append(module)
+        for testpath in oeqa.selftest.__path__:
+            files = sorted([f for f in os.listdir(testpath) if f.endswith('.py') and not (f.startswith('_') and not include_hidden) and not f.startswith('__') and f != 'base.py'])
+            for f in files:
+                module = 'oeqa.selftest.' + f[:-3]
+                if module not in testslist:
+                    testslist.append(module)
 
     return testslist
 
-- 
2.2.1

[PATCH 6/6] oe-selftest: add libdirs from BBPATH to sys.path

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This ensures that oeqa.selftest.* from layers are found.

[YOCTO #7625]

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 scripts/oe-selftest | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index c19c692..60f9bb8 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -155,6 +155,13 @@ def main():
     parser = get_args_parser()
     args = parser.parse_args()
 
+    # Add <layer>/lib to sys.path, so layers can add selftests
+    log.info("Running bitbake -e to get BBPATH")
+    bbpath = get_bb_var('BBPATH').split(':')
+    layer_libdirs = [p for p in (os.path.join(l, 'lib') for l in bbpath) if os.path.exists(p)]
+    sys.path.extend(layer_libdirs)
+    reload(oeqa.selftest)
+
     if args.list_allclasses:
         args.list_modules = True
 
-- 
2.2.1

Re: [PATCH 0/6] recipetool/devtool/oe-selftest: pull from BBPATH

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 678 bytes --]

On Tue, Jul 14, 2015 at 8:57 AM, Christopher Larson <kergoth@gmail.com>
wrote:

> From: Christopher Larson <chris_larson@mentor.com>
>
> Ensures that recipetool and devtool pull plugins from layers, and
> oe-selftest will pull tests from layers.
>
> Unrelated to that, also prevents a traceback on parsing failure, and adds
> a tiny feature to appendsrc to facilitate reuse by other sub-commands.
>
> [YOCTO #7625]
>

This series supercedes '[PATCH 0/3] One recipetool fix and a couple
enhancements’.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1180 bytes --]