[PATCH 00/11] Improve build-image plugin

[PATCH 00/11] Improve build-image plugin

[Ed Bartosh]

This set of improvements for build-image plugin hopefully addresses
Paul's review comments and suggestions.

Paul, thank you for review!

The following changes since commit 6b17c3831897ee1d46a763d807c5bd863d426bc1:

  linux-yocto-3.14: Update to latest revisions (2015-08-30 12:47:51 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/devtool/8135
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/devtool/8135

Ed Bartosh (11):
  devtool: make 2 functions public
  devtool: build-image: stop using add_md5
  Revert "devtool: make add_md5 a public API"
  devtool: build-image: rename LOG -> logger
  devtool: build-image: improve help and description
  devtool: build-image: filter out recipes
  devtool: build-image: generate notification callback
  devtool: build-image: add extra logging
  devtool: build-image: remove <image>.bbappend
  devtool: build-image: add comments
  oe-selftest: test devtool build-image plugin

 meta/lib/oeqa/selftest/devtool.py  | 26 ++++++++++++++++
 scripts/lib/devtool/__init__.py    | 32 ++++++++++++++++----
 scripts/lib/devtool/build-image.py | 61 +++++++++++++++++++++++++++++++-------
 scripts/lib/devtool/standard.py    | 49 +++++++++---------------------
 4 files changed, 116 insertions(+), 52 deletions(-)

--
Regards,
Ed

[PATCH 01/11] devtool: make 2 functions public

[Ed Bartosh]

Moved standard.py:_parse_recipe -> __init__.py:parse_recipe and
      standard.py:_get_recipe_file ->  __init__.py:get_recipe_file
to be able to call them from other modules.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/devtool/__init__.py | 27 +++++++++++++++++++++++++++
 scripts/lib/devtool/standard.py | 36 ++++--------------------------------
 2 files changed, 31 insertions(+), 32 deletions(-)

diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 1747fff..97ac6ae 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -116,3 +116,30 @@ def add_md5(config, recipename, filename):
     md5 = bb.utils.md5_file(filename)
     with open(os.path.join(config.workspace_path, '.devtool_md5'), 'a') as f:
         f.write('%s|%s|%s\n' % (recipename, os.path.relpath(filename, config.workspace_path), md5))
+
+def get_recipe_file(cooker, pn):
+    """Find recipe file corresponding a package name"""
+    import oe.recipeutils
+    recipefile = oe.recipeutils.pn_to_recipe(cooker, pn)
+    if not recipefile:
+        skipreasons = oe.recipeutils.get_unavailable_reasons(cooker, pn)
+        if skipreasons:
+            logger.error('\n'.join(skipreasons))
+        else:
+            logger.error("Unable to find any recipe file matching %s" % pn)
+    return recipefile
+
+def parse_recipe(config, tinfoil, pn, appends):
+    """Parse recipe of a package"""
+    import oe.recipeutils
+    recipefile = get_recipe_file(tinfoil.cooker, pn)
+    if not recipefile:
+        # Error already logged
+        return None
+    if appends:
+        append_files = tinfoil.cooker.collection.get_file_appends(recipefile)
+        # Filter out appends from the workspace
+        append_files = [path for path in append_files if
+                        not path.startswith(config.workspace_path)]
+    return oe.recipeutils.parse_recipe(recipefile, append_files,
+                                       tinfoil.config_data)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 4aa6ebd..97c45d9 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -26,7 +26,7 @@ import argparse
 import scriptutils
 import errno
 from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
-from devtool import add_md5
+from devtool import add_md5, parse_recipe
 
 logger = logging.getLogger('devtool')
 
@@ -157,34 +157,6 @@ def _check_compatible_recipe(pn, d):
                            "from working. You will need to disable this "
                            "first." % pn)
 
-def _get_recipe_file(cooker, pn):
-    """Find recipe file corresponding a package name"""
-    import oe.recipeutils
-    recipefile = oe.recipeutils.pn_to_recipe(cooker, pn)
-    if not recipefile:
-        skipreasons = oe.recipeutils.get_unavailable_reasons(cooker, pn)
-        if skipreasons:
-            logger.error('\n'.join(skipreasons))
-        else:
-            logger.error("Unable to find any recipe file matching %s" % pn)
-    return recipefile
-
-def _parse_recipe(config, tinfoil, pn, appends):
-    """Parse recipe of a package"""
-    import oe.recipeutils
-    recipefile = _get_recipe_file(tinfoil.cooker, pn)
-    if not recipefile:
-        # Error already logged
-        return None
-    if appends:
-        append_files = tinfoil.cooker.collection.get_file_appends(recipefile)
-        # Filter out appends from the workspace
-        append_files = [path for path in append_files if
-                        not path.startswith(config.workspace_path)]
-    return oe.recipeutils.parse_recipe(recipefile, append_files,
-                                       tinfoil.config_data)
-
-
 def _ls_tree(directory):
     """Recursive listing of files in a directory"""
     ret = []
@@ -200,7 +172,7 @@ def extract(args, config, basepath, workspace):
 
     tinfoil = setup_tinfoil()
 
-    rd = _parse_recipe(config, tinfoil, args.recipename, True)
+    rd = parse_recipe(config, tinfoil, args.recipename, True)
     if not rd:
         return 1
 
@@ -420,7 +392,7 @@ def modify(args, config, basepath, workspace):
 
     tinfoil = setup_tinfoil()
 
-    rd = _parse_recipe(config, tinfoil, args.recipename, True)
+    rd = parse_recipe(config, tinfoil, args.recipename, True)
     if not rd:
         return 1
     recipefile = rd.getVar('FILE', True)
@@ -762,7 +734,7 @@ def update_recipe(args, config, basepath, workspace):
 
     tinfoil = setup_tinfoil()
 
-    rd = _parse_recipe(config, tinfoil, args.recipename, True)
+    rd = parse_recipe(config, tinfoil, args.recipename, True)
     if not rd:
         return 1
 
-- 
2.1.4

[PATCH 02/11] devtool: build-image: stop using add_md5

[Ed Bartosh]

It doesn't make sense to use it as image recipe is not
in workspace. It means that we can't do 'devtool reset'
for the recipe, which is a main point of using add_md5.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/devtool/build-image.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/scripts/lib/devtool/build-image.py b/scripts/lib/devtool/build-image.py
index d8e7b12..ebd9c59 100644
--- a/scripts/lib/devtool/build-image.py
+++ b/scripts/lib/devtool/build-image.py
@@ -21,7 +21,7 @@ import os
 import logging
 
 from bb.process import ExecutionError
-from devtool import exec_build_env_command, add_md5
+from devtool import exec_build_env_command
 
 LOG = logging.getLogger('devtool')
 
@@ -38,8 +38,6 @@ def build_image(args, config, basepath, workspace):
         afile.write('IMAGE_INSTALL_append = " %s"\n' % \
                     ' '.join(workspace.keys()))
 
-    add_md5(config, image, appendfile)
-
     try:
         exec_build_env_command(config.init_path, basepath,
                                'bitbake %s' % image, watch=True)
-- 
2.1.4

[PATCH 03/11] Revert "devtool: make add_md5 a public API"

[Ed Bartosh]

This reverts commit 69c63728dae38d5b1cc9874268f235a07e04d3db.

Moved add_md5 back to standard.py as it's not used in
any plugin anymore.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/devtool/__init__.py |  7 -------
 scripts/lib/devtool/standard.py | 15 +++++++++++----
 2 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 97ac6ae..404d3e6 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -110,13 +110,6 @@ def setup_tinfoil(config_only=False):
     tinfoil.logger.setLevel(logger.getEffectiveLevel())
     return tinfoil
 
-def add_md5(config, recipename, filename):
-    """Record checksum of a recipe to the md5-file of the workspace"""
-    import bb.utils
-    md5 = bb.utils.md5_file(filename)
-    with open(os.path.join(config.workspace_path, '.devtool_md5'), 'a') as f:
-        f.write('%s|%s|%s\n' % (recipename, os.path.relpath(filename, config.workspace_path), md5))
-
 def get_recipe_file(cooker, pn):
     """Find recipe file corresponding a package name"""
     import oe.recipeutils
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 97c45d9..de7afd9 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -26,7 +26,7 @@ import argparse
 import scriptutils
 import errno
 from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
-from devtool import add_md5, parse_recipe
+from devtool import parse_recipe
 
 logger = logging.getLogger('devtool')
 
@@ -105,7 +105,7 @@ def add(args, config, basepath, workspace):
     except bb.process.ExecutionError as e:
         raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
 
-    add_md5(config, args.recipename, recipefile)
+    _add_md5(config, args.recipename, recipefile)
 
     initial_rev = None
     if os.path.exists(os.path.join(srctree, '.git')):
@@ -121,7 +121,7 @@ def add(args, config, basepath, workspace):
         if initial_rev:
             f.write('\n# initial_rev: %s\n' % initial_rev)
 
-    add_md5(config, args.recipename, appendfile)
+    _add_md5(config, args.recipename, appendfile)
 
     return 0
 
@@ -344,6 +344,13 @@ def _extract_source(srctree, keep_temp, devbranch, d):
             shutil.rmtree(tempdir)
     return initial_rev
 
+def _add_md5(config, recipename, filename):
+    """Record checksum of a recipe to the md5-file of the workspace"""
+    import bb.utils
+    md5 = bb.utils.md5_file(filename)
+    with open(os.path.join(config.workspace_path, '.devtool_md5'), 'a') as f:
+        f.write('%s|%s|%s\n' % (recipename, os.path.relpath(filename, config.workspace_path), md5))
+
 def _check_preserve(config, recipename):
     """Check if a recipe was manually changed and needs to be saved in 'attic'
        directory"""
@@ -471,7 +478,7 @@ def modify(args, config, basepath, workspace):
             for commit in commits:
                 f.write('# commit: %s\n' % commit)
 
-    add_md5(config, args.recipename, appendfile)
+    _add_md5(config, args.recipename, appendfile)
 
     logger.info('Recipe %s now set up to build from %s' % (args.recipename, srctree))
 
-- 
2.1.4

[PATCH 04/11] devtool: build-image: rename LOG -> logger

[Ed Bartosh]

Used logger variable name instead of LOG as it is used in the rest
of devtool code.
Although Pylint complains about 'logger' being invalid constant name,
but it's better to be consistent in naming than make Pylint happy.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/devtool/build-image.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/devtool/build-image.py b/scripts/lib/devtool/build-image.py
index ebd9c59..d875715 100644
--- a/scripts/lib/devtool/build-image.py
+++ b/scripts/lib/devtool/build-image.py
@@ -23,7 +23,7 @@ import logging
 from bb.process import ExecutionError
 from devtool import exec_build_env_command
 
-LOG = logging.getLogger('devtool')
+logger = logging.getLogger('devtool')
 
 def plugin_init(pluginlist):
     """Plugin initialization"""
@@ -44,7 +44,7 @@ def build_image(args, config, basepath, workspace):
     except ExecutionError as err:
         return err.exitcode
 
-    LOG.info('Successfully built %s', image)
+    logger.info('Successfully built %s', image)
 
 def register_commands(subparsers, context):
     """Register devtool subcommands from the build-image plugin"""
-- 
2.1.4

[PATCH 05/11] devtool: build-image: improve help and description

[Ed Bartosh]

Made parser help message and description more clear in
build-image plugin.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/devtool/build-image.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/devtool/build-image.py b/scripts/lib/devtool/build-image.py
index d875715..708120a 100644
--- a/scripts/lib/devtool/build-image.py
+++ b/scripts/lib/devtool/build-image.py
@@ -48,7 +48,9 @@ def build_image(args, config, basepath, workspace):
 
 def register_commands(subparsers, context):
     """Register devtool subcommands from the build-image plugin"""
-    parser_package = subparsers.add_parser('build-image', help='Build image')
-    parser_package.add_argument('recipe', help='Image recipe to build')
-    parser_package.set_defaults(func=build_image)
-
+    parser = subparsers.add_parser('build-image',
+                                   help='Build image including workspace recipe packages',
+                                   description='Builds an image, extending it to include '
+                                   'packages from recipes in the workspace')
+    parser.add_argument('recipe', help='Image recipe to build')
+    parser.set_defaults(func=build_image)
-- 
2.1.4

[PATCH 06/11] devtool: build-image: filter out recipes

[Ed Bartosh]

Filtered out non-target recipes and recipes with
recipe name != package name in build-image plugin.

Isolated all logic of getting recipes in _get_recipes
function.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/devtool/build-image.py | 25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/devtool/build-image.py b/scripts/lib/devtool/build-image.py
index 708120a..341ab28 100644
--- a/scripts/lib/devtool/build-image.py
+++ b/scripts/lib/devtool/build-image.py
@@ -21,7 +21,7 @@ import os
 import logging
 
 from bb.process import ExecutionError
-from devtool import exec_build_env_command
+from devtool import exec_build_env_command, setup_tinfoil, parse_recipe
 
 logger = logging.getLogger('devtool')
 
@@ -29,14 +29,31 @@ def plugin_init(pluginlist):
     """Plugin initialization"""
     pass
 
+def _get_recipes(workspace, config):
+    """Get list of target recipes from the workspace."""
+    result = []
+    tinfoil = setup_tinfoil()
+    for recipe in workspace:
+        data = parse_recipe(config, tinfoil, recipe, True)
+        if 'class-target' in data.getVar('OVERRIDES', True).split(':'):
+            if recipe in data.getVar('PACKAGES', True):
+                result.append(recipe)
+            else:
+                logger.warning("Skipping recipe %s as it doesn't produce "
+                               "package with the same name", recipe)
+    tinfoil.shutdown()
+    return result
+
 def build_image(args, config, basepath, workspace):
     """Entry point for the devtool 'build-image' subcommand."""
     image = args.recipe
     appendfile = os.path.join(config.workspace_path, 'appends',
                               '%s.bbappend' % image)
-    with open(appendfile, 'w') as afile:
-        afile.write('IMAGE_INSTALL_append = " %s"\n' % \
-                    ' '.join(workspace.keys()))
+
+    recipes = _get_recipes(workspace, config)
+    if recipes:
+        with open(appendfile, 'w') as afile:
+            afile.write('IMAGE_INSTALL_append = " %s"\n' % ' '.join(recipes))
 
     try:
         exec_build_env_command(config.init_path, basepath,
-- 
2.1.4

[PATCH 07/11] devtool: build-image: generate notification callback

[Ed Bartosh]

Added notification callback to <image>.bbapend to notify
user that image is modified by build-image plugin.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/devtool/build-image.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/scripts/lib/devtool/build-image.py b/scripts/lib/devtool/build-image.py
index 341ab28..d246fad 100644
--- a/scripts/lib/devtool/build-image.py
+++ b/scripts/lib/devtool/build-image.py
@@ -55,6 +55,15 @@ def build_image(args, config, basepath, workspace):
         with open(appendfile, 'w') as afile:
             afile.write('IMAGE_INSTALL_append = " %s"\n' % ' '.join(recipes))
 
+            # Generate notification callback devtool_warn_image_extended
+            afile.write('do_rootfs[prefuncs] += "devtool_warn_image_extended"\n\n')
+            afile.write("python devtool_warn_image_extended() {\n")
+            afile.write("    bb.plain('NOTE: %%s: building with additional '\n"
+                        "             'packages due to \"devtool build-image\"'"
+                        "              %% d.getVar('PN', True))\n"
+                        "    bb.plain('NOTE: delete %%s to clear this' %% \\\n"
+                        "             '%s')\n" % os.path.relpath(appendfile, basepath))
+            afile.write("}\n")
     try:
         exec_build_env_command(config.init_path, basepath,
                                'bitbake %s' % image, watch=True)
-- 
2.1.4

[PATCH 08/11] devtool: build-image: add extra logging

[Ed Bartosh]

Added logger calls to show if image is modified by the
plugin or not.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/devtool/build-image.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/lib/devtool/build-image.py b/scripts/lib/devtool/build-image.py
index d246fad..563563b 100644
--- a/scripts/lib/devtool/build-image.py
+++ b/scripts/lib/devtool/build-image.py
@@ -64,6 +64,12 @@ def build_image(args, config, basepath, workspace):
                         "    bb.plain('NOTE: delete %%s to clear this' %% \\\n"
                         "             '%s')\n" % os.path.relpath(appendfile, basepath))
             afile.write("}\n")
+
+            logger.info('Building image %s with the following '
+                        'additional packages: %s', image, ' '.join(recipes))
+    else:
+        logger.warning('No recipes in workspace, building image %s unmodified', image)
+
     try:
         exec_build_env_command(config.init_path, basepath,
                                'bitbake %s' % image, watch=True)
-- 
2.1.4

[PATCH 09/11] devtool: build-image: remove <image>.bbappend

[Ed Bartosh]

Removed <image>.bbappend before generating it again as
it may cause tinfoil to fail due to its wrong content.

It's safe to do as <image>.bbappend is regenerated anyway.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/devtool/build-image.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/scripts/lib/devtool/build-image.py b/scripts/lib/devtool/build-image.py
index 563563b..5bc8213 100644
--- a/scripts/lib/devtool/build-image.py
+++ b/scripts/lib/devtool/build-image.py
@@ -50,6 +50,11 @@ def build_image(args, config, basepath, workspace):
     appendfile = os.path.join(config.workspace_path, 'appends',
                               '%s.bbappend' % image)
 
+    # remove <image>.bbapend to make sure setup_tinfoil doesn't
+    # break because of it
+    if os.path.isfile(appendfile):
+        os.unlink(appendfile)
+
     recipes = _get_recipes(workspace, config)
     if recipes:
         with open(appendfile, 'w') as afile:
-- 
2.1.4

[PATCH 10/11] devtool: build-image: add comments

[Ed Bartosh]

Added couple of hopefully useful comments to the code.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/devtool/build-image.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/lib/devtool/build-image.py b/scripts/lib/devtool/build-image.py
index 5bc8213..817703a 100644
--- a/scripts/lib/devtool/build-image.py
+++ b/scripts/lib/devtool/build-image.py
@@ -58,6 +58,7 @@ def build_image(args, config, basepath, workspace):
     recipes = _get_recipes(workspace, config)
     if recipes:
         with open(appendfile, 'w') as afile:
+            # include selected recipes into the image
             afile.write('IMAGE_INSTALL_append = " %s"\n' % ' '.join(recipes))
 
             # Generate notification callback devtool_warn_image_extended
@@ -75,6 +76,7 @@ def build_image(args, config, basepath, workspace):
     else:
         logger.warning('No recipes in workspace, building image %s unmodified', image)
 
+    # run bitbake to build image
     try:
         exec_build_env_command(config.init_path, basepath,
                                'bitbake %s' % image, watch=True)
-- 
2.1.4

[PATCH 11/11] oe-selftest: test devtool build-image plugin

[Ed Bartosh]

Added test case to test functionaly of build-image plugin:

Add two packages to workspace and run 'devtool build-image
core-image-minimal'.

Checked if command is successful.

Checked if expected package is added to generated
core-image-minimal.bbappend file.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 70ee634..a6474b7 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -887,3 +887,29 @@ class DevtoolTests(DevtoolBase):
             result = runCmd('devtool undeploy-target -c %s root@%s' % (testrecipe, qemu.ip))
             result = runCmd('ssh %s root@%s %s' % (sshargs, qemu.ip, testcommand), ignore_status=True)
             self.assertNotEqual(result, 0, 'undeploy-target did not remove command as it should have')
+
+    def test_devtool_build_image(self):
+        """Test devtool build-image plugin"""
+        # Check preconditions
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        image = 'core-image-minimal'
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        self.add_command_to_tearDown('bitbake -c clean %s' % image)
+        bitbake('%s -c clean' % image)
+        # Add target and native recipes to workspace
+        for recipe in ('mdadm', 'parted-native'):
+            tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+            self.track_for_cleanup(tempdir)
+            self.add_command_to_tearDown('bitbake -c clean %s' % recipe)
+            runCmd('devtool modify %s -x %s' % (recipe, tempdir))
+        # Try to build image
+        result = runCmd('devtool build-image %s' % image)
+        self.assertNotEqual(result, 0, 'devtool build-image failed')
+        # Check if image.bbappend has required content
+        bbappend = os.path.join(workspacedir, 'appends', image+'.bbappend')
+        self.assertTrue(os.path.isfile(bbappend), 'bbappend not created %s' % result.output)
+        # NOTE: native recipe parted-native should not be in IMAGE_INSTALL_append
+        self.assertTrue('IMAGE_INSTALL_append = " mdadm"\n' in open(bbappend).readlines(),
+                        'IMAGE_INSTALL_append = " mdadm" not found in %s' % bbappend)
-- 
2.1.4

Re: [PATCH 01/11] devtool: make 2 functions public

[Jussi Kukkonen]

[-- Attachment #1: Type: text/plain, Size: 5610 bytes --]

On 30 August 2015 at 19:08, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
>
> Moved standard.py:_parse_recipe -> __init__.py:parse_recipe and
>       standard.py:_get_recipe_file ->  __init__.py:get_recipe_file
> to be able to call them from other modules.

Hi Ed,

Can you take a look if the failures in
https://autobuilder.yoctoproject.org/main/builders/nightly-oe-selftest/builds/173/steps/Running%20oe-selftest/logs/stdio
are related to this patch (there's lots of "NameError: global name
'_parse_recipe' is not defined")

Jussi

>
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  scripts/lib/devtool/__init__.py | 27 +++++++++++++++++++++++++++
>  scripts/lib/devtool/standard.py | 36 ++++--------------------------------
>  2 files changed, 31 insertions(+), 32 deletions(-)
>
> diff --git a/scripts/lib/devtool/__init__.py
b/scripts/lib/devtool/__init__.py
> index 1747fff..97ac6ae 100644
> --- a/scripts/lib/devtool/__init__.py
> +++ b/scripts/lib/devtool/__init__.py
> @@ -116,3 +116,30 @@ def add_md5(config, recipename, filename):
>      md5 = bb.utils.md5_file(filename)
>      with open(os.path.join(config.workspace_path, '.devtool_md5'), 'a')
as f:
>          f.write('%s|%s|%s\n' % (recipename, os.path.relpath(filename,
config.workspace_path), md5))
> +
> +def get_recipe_file(cooker, pn):
> +    """Find recipe file corresponding a package name"""
> +    import oe.recipeutils
> +    recipefile = oe.recipeutils.pn_to_recipe(cooker, pn)
> +    if not recipefile:
> +        skipreasons = oe.recipeutils.get_unavailable_reasons(cooker, pn)
> +        if skipreasons:
> +            logger.error('\n'.join(skipreasons))
> +        else:
> +            logger.error("Unable to find any recipe file matching %s" %
pn)
> +    return recipefile
> +
> +def parse_recipe(config, tinfoil, pn, appends):
> +    """Parse recipe of a package"""
> +    import oe.recipeutils
> +    recipefile = get_recipe_file(tinfoil.cooker, pn)
> +    if not recipefile:
> +        # Error already logged
> +        return None
> +    if appends:
> +        append_files =
tinfoil.cooker.collection.get_file_appends(recipefile)
> +        # Filter out appends from the workspace
> +        append_files = [path for path in append_files if
> +                        not path.startswith(config.workspace_path)]
> +    return oe.recipeutils.parse_recipe(recipefile, append_files,
> +                                       tinfoil.config_data)
> diff --git a/scripts/lib/devtool/standard.py
b/scripts/lib/devtool/standard.py
> index 4aa6ebd..97c45d9 100644
> --- a/scripts/lib/devtool/standard.py
> +++ b/scripts/lib/devtool/standard.py
> @@ -26,7 +26,7 @@ import argparse
>  import scriptutils
>  import errno
>  from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
> -from devtool import add_md5
> +from devtool import add_md5, parse_recipe
>
>  logger = logging.getLogger('devtool')
>
> @@ -157,34 +157,6 @@ def _check_compatible_recipe(pn, d):
>                             "from working. You will need to disable this "
>                             "first." % pn)
>
> -def _get_recipe_file(cooker, pn):
> -    """Find recipe file corresponding a package name"""
> -    import oe.recipeutils
> -    recipefile = oe.recipeutils.pn_to_recipe(cooker, pn)
> -    if not recipefile:
> -        skipreasons = oe.recipeutils.get_unavailable_reasons(cooker, pn)
> -        if skipreasons:
> -            logger.error('\n'.join(skipreasons))
> -        else:
> -            logger.error("Unable to find any recipe file matching %s" %
pn)
> -    return recipefile
> -
> -def _parse_recipe(config, tinfoil, pn, appends):
> -    """Parse recipe of a package"""
> -    import oe.recipeutils
> -    recipefile = _get_recipe_file(tinfoil.cooker, pn)
> -    if not recipefile:
> -        # Error already logged
> -        return None
> -    if appends:
> -        append_files =
tinfoil.cooker.collection.get_file_appends(recipefile)
> -        # Filter out appends from the workspace
> -        append_files = [path for path in append_files if
> -                        not path.startswith(config.workspace_path)]
> -    return oe.recipeutils.parse_recipe(recipefile, append_files,
> -                                       tinfoil.config_data)
> -
> -
>  def _ls_tree(directory):
>      """Recursive listing of files in a directory"""
>      ret = []
> @@ -200,7 +172,7 @@ def extract(args, config, basepath, workspace):
>
>      tinfoil = setup_tinfoil()
>
> -    rd = _parse_recipe(config, tinfoil, args.recipename, True)
> +    rd = parse_recipe(config, tinfoil, args.recipename, True)
>      if not rd:
>          return 1
>
> @@ -420,7 +392,7 @@ def modify(args, config, basepath, workspace):
>
>      tinfoil = setup_tinfoil()
>
> -    rd = _parse_recipe(config, tinfoil, args.recipename, True)
> +    rd = parse_recipe(config, tinfoil, args.recipename, True)
>      if not rd:
>          return 1
>      recipefile = rd.getVar('FILE', True)
> @@ -762,7 +734,7 @@ def update_recipe(args, config, basepath, workspace):
>
>      tinfoil = setup_tinfoil()
>
> -    rd = _parse_recipe(config, tinfoil, args.recipename, True)
> +    rd = parse_recipe(config, tinfoil, args.recipename, True)
>      if not rd:
>          return 1
>
> --
> 2.1.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

[-- Attachment #2: Type: text/html, Size: 7238 bytes --]

Re: [PATCH 01/11] devtool: make 2 functions public

[Ed Bartosh]

On Wed, Sep 02, 2015 at 02:16:58PM +0300, Jussi Kukkonen wrote:
> On 30 August 2015 at 19:08, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> >
> > Moved standard.py:_parse_recipe -> __init__.py:parse_recipe and
> >       standard.py:_get_recipe_file ->  __init__.py:get_recipe_file
> > to be able to call them from other modules.
> 
> Hi Ed,
> 
> Can you take a look if the failures in
> https://autobuilder.yoctoproject.org/main/builders/nightly-oe-selftest/builds/173/steps/Running%20oe-selftest/logs/stdio
> are related to this patch (there's lots of "NameError: global name
> '_parse_recipe' is not defined")
> 

The autobuilder log looks very strange to me. I don't see any mentioning
of _parse_recipe in standard.py on master. Moreover, there is no
_prep_extract_operation function mentioned in traceback.

I've just run failed test and it passed without any issues.


It looks like autobuilder messed the code from some old branch and tests
from master.

Regards,
Ed

> 
> >
> > Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> > ---
> >  scripts/lib/devtool/__init__.py | 27 +++++++++++++++++++++++++++
> >  scripts/lib/devtool/standard.py | 36 ++++--------------------------------
> >  2 files changed, 31 insertions(+), 32 deletions(-)
> >
> > diff --git a/scripts/lib/devtool/__init__.py
> b/scripts/lib/devtool/__init__.py
> > index 1747fff..97ac6ae 100644
> > --- a/scripts/lib/devtool/__init__.py
> > +++ b/scripts/lib/devtool/__init__.py
> > @@ -116,3 +116,30 @@ def add_md5(config, recipename, filename):
> >      md5 = bb.utils.md5_file(filename)
> >      with open(os.path.join(config.workspace_path, '.devtool_md5'), 'a')
> as f:
> >          f.write('%s|%s|%s\n' % (recipename, os.path.relpath(filename,
> config.workspace_path), md5))
> > +
> > +def get_recipe_file(cooker, pn):
> > +    """Find recipe file corresponding a package name"""
> > +    import oe.recipeutils
> > +    recipefile = oe.recipeutils.pn_to_recipe(cooker, pn)
> > +    if not recipefile:
> > +        skipreasons = oe.recipeutils.get_unavailable_reasons(cooker, pn)
> > +        if skipreasons:
> > +            logger.error('\n'.join(skipreasons))
> > +        else:
> > +            logger.error("Unable to find any recipe file matching %s" %
> pn)
> > +    return recipefile
> > +
> > +def parse_recipe(config, tinfoil, pn, appends):
> > +    """Parse recipe of a package"""
> > +    import oe.recipeutils
> > +    recipefile = get_recipe_file(tinfoil.cooker, pn)
> > +    if not recipefile:
> > +        # Error already logged
> > +        return None
> > +    if appends:
> > +        append_files =
> tinfoil.cooker.collection.get_file_appends(recipefile)
> > +        # Filter out appends from the workspace
> > +        append_files = [path for path in append_files if
> > +                        not path.startswith(config.workspace_path)]
> > +    return oe.recipeutils.parse_recipe(recipefile, append_files,
> > +                                       tinfoil.config_data)
> > diff --git a/scripts/lib/devtool/standard.py
> b/scripts/lib/devtool/standard.py
> > index 4aa6ebd..97c45d9 100644
> > --- a/scripts/lib/devtool/standard.py
> > +++ b/scripts/lib/devtool/standard.py
> > @@ -26,7 +26,7 @@ import argparse
> >  import scriptutils
> >  import errno
> >  from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
> > -from devtool import add_md5
> > +from devtool import add_md5, parse_recipe
> >
> >  logger = logging.getLogger('devtool')
> >
> > @@ -157,34 +157,6 @@ def _check_compatible_recipe(pn, d):
> >                             "from working. You will need to disable this "
> >                             "first." % pn)
> >
> > -def _get_recipe_file(cooker, pn):
> > -    """Find recipe file corresponding a package name"""
> > -    import oe.recipeutils
> > -    recipefile = oe.recipeutils.pn_to_recipe(cooker, pn)
> > -    if not recipefile:
> > -        skipreasons = oe.recipeutils.get_unavailable_reasons(cooker, pn)
> > -        if skipreasons:
> > -            logger.error('\n'.join(skipreasons))
> > -        else:
> > -            logger.error("Unable to find any recipe file matching %s" %
> pn)
> > -    return recipefile
> > -
> > -def _parse_recipe(config, tinfoil, pn, appends):
> > -    """Parse recipe of a package"""
> > -    import oe.recipeutils
> > -    recipefile = _get_recipe_file(tinfoil.cooker, pn)
> > -    if not recipefile:
> > -        # Error already logged
> > -        return None
> > -    if appends:
> > -        append_files =
> tinfoil.cooker.collection.get_file_appends(recipefile)
> > -        # Filter out appends from the workspace
> > -        append_files = [path for path in append_files if
> > -                        not path.startswith(config.workspace_path)]
> > -    return oe.recipeutils.parse_recipe(recipefile, append_files,
> > -                                       tinfoil.config_data)
> > -
> > -
> >  def _ls_tree(directory):
> >      """Recursive listing of files in a directory"""
> >      ret = []
> > @@ -200,7 +172,7 @@ def extract(args, config, basepath, workspace):
> >
> >      tinfoil = setup_tinfoil()
> >
> > -    rd = _parse_recipe(config, tinfoil, args.recipename, True)
> > +    rd = parse_recipe(config, tinfoil, args.recipename, True)
> >      if not rd:
> >          return 1
> >
> > @@ -420,7 +392,7 @@ def modify(args, config, basepath, workspace):
> >
> >      tinfoil = setup_tinfoil()
> >
> > -    rd = _parse_recipe(config, tinfoil, args.recipename, True)
> > +    rd = parse_recipe(config, tinfoil, args.recipename, True)
> >      if not rd:
> >          return 1
> >      recipefile = rd.getVar('FILE', True)
> > @@ -762,7 +734,7 @@ def update_recipe(args, config, basepath, workspace):
> >
> >      tinfoil = setup_tinfoil()
> >
> > -    rd = _parse_recipe(config, tinfoil, args.recipename, True)
> > +    rd = parse_recipe(config, tinfoil, args.recipename, True)
> >      if not rd:
> >          return 1
> >
> > --
> > 2.1.4
> >
> > --
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH 01/11] devtool: make 2 functions public

[Richard Purdie]

On Wed, 2015-09-02 at 16:54 +0300, Ed Bartosh wrote:
> On Wed, Sep 02, 2015 at 02:16:58PM +0300, Jussi Kukkonen wrote:
> > On 30 August 2015 at 19:08, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> > >
> > > Moved standard.py:_parse_recipe -> __init__.py:parse_recipe and
> > >       standard.py:_get_recipe_file ->  __init__.py:get_recipe_file
> > > to be able to call them from other modules.
> > 
> > Hi Ed,
> > 
> > Can you take a look if the failures in
> > https://autobuilder.yoctoproject.org/main/builders/nightly-oe-selftest/builds/173/steps/Running%20oe-selftest/logs/stdio
> > are related to this patch (there's lots of "NameError: global name
> > '_parse_recipe' is not defined")
> > 
> 
> The autobuilder log looks very strange to me. I don't see any mentioning
> of _parse_recipe in standard.py on master. Moreover, there is no
> _prep_extract_operation function mentioned in traceback.
> 
> I've just run failed test and it passed without any issues.
> 
> 
> It looks like autobuilder messed the code from some old branch and tests
> from master.

Its building master-next:

http://git.yoctoproject.org/cgit.cgi/poky/log/?h=master-next

and its therefore probably the devtool patches from Markus that are at
fault and need rebasing after your changes?

Cheers,

Richard