[PATCH v2 0/1] devtool: upgrade feature

[PATCH v2 0/1] devtool: upgrade feature

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

Complete RE-IMPLEMENTATION since v1. This patch does not use any of the 
auto-upgrade-helper code [1] and the reason is that this code needs
some (i.e. use the oe.recipeutils module, auh touchs tracked recipes, etc.)
work before it can be reused into the devtool framework. Instead
of AUH, the recipetool script is used to get the new checksums. 
Bugzilla Tracking page is [2]

[1] http://git.yoctoproject.org/cgit/cgit.cgi/auto-upgrade-helper/
[2] https://bugzilla.yoctoproject.org/show_bug.cgi?id=7642

The following changes since commit 645435a645a0817cec94ce1433eb74fbe7388416:

  bitbake: toastergui: Added IDs to elements used in testing (2015-08-17 08:48:28 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib lsandov1/devtool-upgrade
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lsandov1/devtool-upgrade

Leonardo Sandoval (2):
  devtool: upgrade feature
  oeqa/selftest: new tests for devtool upgrage feature

 meta/lib/oeqa/selftest/devtool.py |  36 +++++
 scripts/lib/devtool/standard.py   |   4 +-
 scripts/lib/devtool/upgrade.py    | 314 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 353 insertions(+), 1 deletion(-)
 create mode 100644 scripts/lib/devtool/upgrade.py

-- 
1.8.4.5

[PATCH v2 1/2] devtool: upgrade feature

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

Upgrades a recipe to a particular version and downloads the source code
into srctree. User can avoid patching the source code. These are the
general steps of the upgrade function:
   * Extract current recipe source code into srctree and create branch
   * Extract upgrade recipe source code into srctree and rebase with
     previous branch. This step also creates a temporal recipe (created
     using recipetool), containing the correct checksums.
   * Creates the new recipe under the workspace

[YOCTO #7642]

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
 scripts/lib/devtool/standard.py |   4 +-
 scripts/lib/devtool/upgrade.py  | 314 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 317 insertions(+), 1 deletion(-)
 create mode 100644 scripts/lib/devtool/upgrade.py

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index ea21877..cd5a3ed 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -205,6 +205,8 @@ def extract(args, config, basepath, workspace):
 
     srctree = os.path.abspath(args.srctree)
     initial_rev = _extract_source(srctree, args.keep_temp, args.branch, rd)
+    logger.info('Source tree extracted to %s' % srctree)
+
     if initial_rev:
         return 0
     else:
@@ -360,7 +362,6 @@ def _extract_source(srctree, keep_temp, devbranch, d):
                 bb.process.run('git checkout patches', cwd=srcsubdir)
 
         shutil.move(srcsubdir, srctree)
-        logger.info('Source tree extracted to %s' % srctree)
     finally:
         bb.logger.setLevel(origlevel)
 
@@ -439,6 +440,7 @@ def modify(args, config, basepath, workspace):
         initial_rev = _extract_source(args.srctree, False, args.branch, rd)
         if not initial_rev:
             return 1
+        logger.info('Source tree extracted to %s' % srctree)
         # Get list of commits since this revision
         (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=args.srctree)
         commits = stdout.split()
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
new file mode 100644
index 0000000..9bef984
--- /dev/null
+++ b/scripts/lib/devtool/upgrade.py
@@ -0,0 +1,314 @@
+# Development tool - upgrade command plugin
+#
+# Copyright (C) 2014-2015 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# DESCRIPTION
+# Created a new recipe under workspace/recipes/<recipename> and place the
+# source code into <srctree>.
+# The upgrade feature executes the following steps:
+#     * Extract current recipe source code into srctree and create branch
+#     * Extract upgrade recipe source code into srctree and rebase with 
+#       previous branch. This step also creates a temporal recipe (created
+#       using recipetool), containing the correct checksums.
+#     * Creates the new recipe under the workspace
+"""Devtool upgrade plugin"""
+
+import os
+import sys
+import re
+import shutil
+import tempfile
+import logging
+import argparse
+import scriptutils
+import errno
+from devtool import standard
+from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
+
+logger = logging.getLogger('devtool')
+
+def plugin_init(pluginlist):
+    """Plugin initialization"""
+    pass
+
+def _extract_upgrade_source(args, devbranch, config, basepath, d, recipepostfix='tmp'):
+    """Extract sources of a recipe with PV given on args.version
+
+    On the target source tree folder, a new branch (<devbranch>_<args.version>)
+    and tag (<devbranch>-base_<args.version>) will be created. In case patches
+    are applied, another tag is created (<devbranch>-patched_<args.version>).
+
+    Returns: 1) The (git) initial revision ID
+             2) The full path of a temporal recipe containing the correct checksums
+    """
+    import oe.recipeutils
+
+    initial_rev = None
+    srctree = os.path.abspath(args.srctree)
+
+    pn = d.getVar('PN', True)
+
+    standard._check_compatible_recipe(pn, d)
+
+    recipepath = os.path.join(config.workspace_path, 'recipes', pn)
+    bb.utils.mkdirhier(recipepath)
+    recipefile = os.path.join(recipepath, "%s-%s.bb" % (pn,recipepostfix))
+    tmpsrcbasetree = tempfile.mkdtemp(prefix='devtool')
+
+    # Change PV and get URL
+    crd = d.createCopy()
+    pv = d.getVar('PV', True)
+    crd.setVar('PV', args.version)
+    src_uri = crd.getVar('SRC_URI', True)
+    if src_uri:
+        url = src_uri.split()[0]
+
+    # Generate recipe and fetch new source
+    try:
+        cmdopts = "-o %s -x %s -V %s" % (recipefile, tmpsrcbasetree, args.version)
+        cmd = 'recipetool create "%s" %s' % (url, cmdopts)
+        stdout, _ = exec_build_env_command(config.init_path, basepath, cmd)
+    except bb.process.ExecutionError as e:
+        raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
+
+    tmpsrctree = os.path.join(tmpsrcbasetree, pn + '-' + args.version)
+
+    try:
+        # branch from devtool-base (original source code without patches) before copying new source code
+        bb.process.run('git checkout -b %s devtool-base' % devbranch, cwd=srctree)
+        bb.process.run('git tag -f devtool-base_%s' % args.version, cwd=srctree)
+
+        # Copy tmpsrctree  into srctree
+        src_files = standard._ls_tree(tmpsrctree)
+        for path in src_files:
+            tgt_dir = os.path.join(srctree, os.path.dirname(path))
+            bb.utils.mkdirhier(tgt_dir)
+            tgt_path = os.path.join(srctree, path)
+            os.rename(os.path.join(tmpsrctree, path), tgt_path)
+
+        # Track modified and untracked files
+        (stdout,_) = bb.process.run('git ls-files --modified --others --exclude-standard', cwd=srctree)
+        add_files = stdout.splitlines()
+        for add_file in add_files:
+            bb.process.run('git add "%s"' % add_file, cwd=srctree)
+
+        if len(add_files):
+            bb.process.run('git commit -q -m "Initial commit from upstream at version %s"' % args.version, cwd=srctree)
+        (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
+        initial_rev = stdout.rstrip()
+
+        if args.no_patch:
+            patches = oe.recipeutils.get_recipe_patches(crd)
+            if len(patches):
+                logger.warn('By user choice, the following (%s_%s) patches will NOT be applied into' %(pn,pv))
+                for patch in patches:
+                    logger.warn("\t%s" % os.path.basename(patch))
+        else:
+            try:
+                bb.process.run('git rebase devtool-patched', cwd=srctree)
+            except bb.process.ExecutionError as e:
+                # this is the only case where we do not propagate the error
+                # user will have the change to correct merges
+                logger.error('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
+            bb.process.run('git tag -f devtool-patched_%s' % args.version, cwd=srctree)
+    except bb.process.ExecutionError as e:
+        raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
+    finally:
+        if args.keep_temp:
+            logger.info('Preserving temporary directory %s' % tmpsrcbasetree)
+        else:
+            shutil.rmtree(tmpsrcbasetree)
+    return initial_rev, recipefile
+
+def _create_upgrade_recipe(args, config, recipetmp, config_data, d):
+    """Creates the new recipe under workspace
+
+    Returns the full path on the new created recipe
+    """
+    import oe.recipeutils
+
+    def _get_checksums(recipefile):
+        import re
+        checksums = {}
+        with open(recipefile) as rf:
+            for line in rf:
+                for cs in ['md5sum', 'sha256sum']:
+                    m = re.match("^SRC_URI\[%s\].*=.*\"(.*)\"" % cs, line)
+                    if m:
+                        checksums[cs] = m.group(1)
+        return checksums
+
+    def _replace_checksums(recipefile, checksums):
+        import re
+        with open(recipefile + ".tmp", "w+") as tmprf:
+            with open(recipefile) as rf:
+                for line in rf:
+                    m = None
+                    for cs in ['md5sum', 'sha256sum']:
+                        m = re.match("^SRC_URI\[%s\].*=.*\"(.*)\"" % cs, line)
+                        if m:
+                            if cs in checksums:
+                                oldcheck = m.group(1)
+                                newcheck = checksums[cs]
+                                line = line.replace(oldcheck, newcheck)
+                            break
+                    tmprf.write(line)
+
+        os.rename(recipefile + ".tmp", recipefile)
+
+    def _rename_patch_dirs(recipefolder, oldpv, newpv):
+        for root, dirs, files in os.walk(recipefolder):
+            for olddir in dirs:
+                if olddir.find(oldpv) != -1:
+                    newdir = olddir.replace(oldpv, newpv)
+                    bb.process.run('mv %s %s' % (olddir, newdir))
+
+    def _remove_patch_dirs(recipefolder):
+        for root, dirs, files in os.walk(recipefolder):
+            for d in dirs:
+                shutil.rmtree(os.path.join(root,d))
+
+    def _recipe_contains(recipefile, var):
+        import re
+        found = False
+        with open(recipefile) as rf:
+            for line in rf:
+                if re.match("^%s.*=.*" % var, line):
+                    found = True
+                    break
+        return found
+
+    if not os.path.exists(recipetmp):
+        raise DevtoolError("Temporal recipe %s does not exist" % recipetmp)
+
+    # Copy current recipe into workspace
+    pn = d.getVar('PN', True)
+    recipepath = os.path.join(config.workspace_path, 'recipes', pn)
+    oe.recipeutils.copy_recipe_files(d, recipepath)
+
+    # Rename recipe
+    pv = d.getVar('PV', True)
+    recipename = "%s_%s.bb" % (pn, pv)
+    newrecipename = "%s_%s.bb" % (pn, args.version)
+    if os.path.isfile(os.path.join(recipepath, recipename)):
+        bb.process.run('mv %s %s' % (recipename, newrecipename), cwd=recipepath)
+    else:
+        # Check if it is a git recipe
+        recipename = newrecipename = "%s_git.bb" % pn
+        if not os.path.isfile(os.path.join(recipepath, recipename)):
+            raise DevtoolError("Original recipe not found on workspace")
+
+    # Rename folders
+    _rename_patch_dirs(recipepath, pv, args.version)
+
+    # Update PV, just in case it is present
+    if _recipe_contains(os.path.join(recipepath, newrecipename), 'PV'):
+        oe.recipeutils.patch_recipe(d, os.path.join(recipepath, newrecipename), {'PV':args.version})
+
+    # Update checksums
+    checksums = _get_checksums(os.path.join(recipepath, recipetmp))
+    _replace_checksums(os.path.join(recipepath, newrecipename), checksums)
+
+    # Remove recipe created by the recipe-tool
+    bb.process.run('rm %s' % recipetmp)
+
+    return os.path.join(recipepath,newrecipename)
+
+def upgrade(args, config, basepath, workspace):
+    """Entry point for the devtool 'upgrade' subcommand"""
+    import bb
+    import oe.recipeutils
+
+    if args.recipename in workspace:
+        raise DevtoolError("recipe %s is already in your workspace" %
+                            args.recipename)
+    if not args.version:
+        raise DevtoolError("Provide a version through the parameter --version/-V")
+
+    reason = oe.recipeutils.validate_pn(args.recipename)
+    if reason:
+        raise DevtoolError(reason)
+
+    tinfoil = setup_tinfoil()
+
+    rd = standard._parse_recipe(config, tinfoil, args.recipename, True)
+    if not rd:
+        return 1
+
+    pn = rd.getVar('PV', True)
+    if pn == args.version:
+        raise DevtoolError("Current and upgrade versions are the same %s" % pn)
+
+    srctree = os.path.abspath(args.srctree)
+
+    try:
+        # Extract source from current recipe
+        initial_rev_base = standard._extract_source(srctree, False, args.branch, rd)
+
+        # We need to shutdown tinfoil temporally because recipetool will be used
+        tinfoil.shutdown()
+
+        # Extract new source code
+        branch_upgrade = "%s_%s" % (args.branch, args.version)
+        initial_rev_upgrade, recipe_file_tmp = _extract_upgrade_source(args, branch_upgrade, config, basepath, rd)
+        # Start again tinfoil
+        tinfoil = setup_tinfoil()
+
+        recipe_file = _create_upgrade_recipe(args, config, recipe_file_tmp, tinfoil.config_data, rd)
+
+    except DevtoolError as e:
+        logger.error(e)
+        # remove workspace recipe and srctree
+        pn = rd.getVar('PN', True)
+        recipespath = os.path.join(config.workspace_path, 'recipes')
+        recipepath = os.path.join(recipespath, pn)
+        if os.path.exists(recipepath):
+            shutil.rmtree(recipepath)
+            if not len(os.listdir(recipespath)):
+                os.rmdir(recipespath)
+        # remove srctree
+        if os.path.exists(srctree):
+            shutil.rmtree(srctree)
+        raise DevtoolError(e)
+
+    appendpath = os.path.join(config.workspace_path, 'appends')
+    if not os.path.exists(appendpath):
+        os.makedirs(appendpath)
+
+    recipe_file_base = os.path.basename(os.path.splitext(recipe_file)[0])
+    appendfile = os.path.join(appendpath, '%s.bbappend' % recipe_file_base)
+    with open(appendfile, 'w') as f:
+        f.write('inherit externalsrc\n')
+        f.write('EXTERNALSRC = "%s"\n' % srctree)
+        f.write('EXTERNALSRC_BUILD = "%s"\n' % srctree)
+        f.write('\n# initial_rev: %s\n' % initial_rev_upgrade)
+
+    standard._add_md5(config, args.recipename, appendfile)
+    logger.info('Source tree extracted to %s' % srctree)
+    return 0
+
+def register_commands(subparsers, context):
+    """Register devtool subcommands from this plugin"""
+    parser_upgrade = subparsers.add_parser('upgrade', help='Upgrade an existing recipe',
+                                       description='Upgrades an existing recipe',
+                                       formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser_upgrade.add_argument('recipename', help='Name for recipe to extract the source for')
+    parser_upgrade.add_argument('srctree', help='Path to where to extract the source tree')
+    parser_upgrade.add_argument('--version', '-V', help='Version to upgrade (PV)')
+    parser_upgrade.add_argument('--branch', '-b', default="devtool", help='Name for development branch to checkout')
+    parser_upgrade.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
+    parser_upgrade.add_argument('--no-patch', action="store_true", help='Do not patch the new source code')
+    parser_upgrade.set_defaults(func=upgrade)
-- 
1.8.4.5

[PATCH v2 2/2] oeqa/selftest: new tests for devtool upgrage feature

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

Basic tests for the devtool's upgrade feature, including:
      * Parameter check
      * Upgrading a real recipe (e2fsprogrs) without patching and
      	checing its output
      * Devtool status after upgrade

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index b59db15..f72e010 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -857,3 +857,39 @@ class DevtoolTests(DevtoolBase):
             result = runCmd('devtool undeploy-target -c %s root@%s' % (testrecipe, qemu.ip))
             result = runCmd('ssh %s root@%s %s' % (sshargs, qemu.ip, testcommand), ignore_status=True)
             self.assertNotEqual(result, 0, 'undeploy-target did not remove command as it should have')
+
+    def test_devtool_upgrade(self):
+        # Check preconditions
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        # Check parameters
+        result = runCmd('devtool upgrade -h')
+        for param in 'recipename srctree --version -V --branch -b --keep-temp --no-patch'.split():
+            self.assertIn(param, result.output)
+        # For the moment, we are using a real recipe.
+        recipe='e2fsprogs'
+        version='1.42.13'
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        # Check that recipe is not already under devtool control
+        result = runCmd('devtool status')
+        self.assertNotIn(recipe, result.output)
+        # Check upgrade. Code does not check if new PV is older or newer that current PV, so, it may be that
+        # we are downgrading instead of upgrading.
+        result = runCmd('devtool upgrade %s %s -V %s --no-patch' % (recipe, tempdir, version))
+        # Check if srctree at least is populated
+        self.assertTrue(len(os.listdir(tempdir)) > 0, 'scrtree (%s) should be populated with new (%s) source code' % (tempdir, version))
+        # Check new recipe folder is present
+        self.assertTrue(os.path.exists(os.path.join(workspacedir,'recipes',recipe)), 'Recipe folder should exist')
+        # Check new recipe file is present
+        self.assertTrue(os.path.exists(os.path.join(workspacedir,'recipes',recipe,"%s_%s.bb" % (recipe,version))), 'Recipe folder should exist')
+        # Check devtool status and make sure recipe is present
+        result = runCmd('devtool status')
+        self.assertIn(recipe, result.output)
+        self.assertIn(tempdir, result.output)
+        # Check devtool reset recipe
+        result = runCmd('devtool reset %s -n' % recipe)
+        result = runCmd('devtool status')
+        self.assertNotIn(recipe, result.output)
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
-- 
1.8.4.5

Re: [PATCH v2 1/2] devtool: upgrade feature

[Aníbal Limón]

Comments below,

On 26/08/15 02:43, leonardo.sandoval.gonzalez@linux.intel.com wrote:
> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
>
> Upgrades a recipe to a particular version and downloads the source code
> into srctree. User can avoid patching the source code. These are the
> general steps of the upgrade function:
>     * Extract current recipe source code into srctree and create branch
>     * Extract upgrade recipe source code into srctree and rebase with
>       previous branch. This step also creates a temporal recipe (created
>       using recipetool), containing the correct checksums.
>     * Creates the new recipe under the workspace
>
> [YOCTO #7642]
>
> Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
> ---
>   scripts/lib/devtool/standard.py |   4 +-
>   scripts/lib/devtool/upgrade.py  | 314 ++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 317 insertions(+), 1 deletion(-)
>   create mode 100644 scripts/lib/devtool/upgrade.py
>
> diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
> index ea21877..cd5a3ed 100644
> --- a/scripts/lib/devtool/standard.py
> +++ b/scripts/lib/devtool/standard.py
> @@ -205,6 +205,8 @@ def extract(args, config, basepath, workspace):
>   
>       srctree = os.path.abspath(args.srctree)
>       initial_rev = _extract_source(srctree, args.keep_temp, args.branch, rd)
> +    logger.info('Source tree extracted to %s' % srctree)
> +
>       if initial_rev:
>           return 0
>       else:
> @@ -360,7 +362,6 @@ def _extract_source(srctree, keep_temp, devbranch, d):
>                   bb.process.run('git checkout patches', cwd=srcsubdir)
>   
>           shutil.move(srcsubdir, srctree)
> -        logger.info('Source tree extracted to %s' % srctree)
Why you move this logging outside?
>       finally:
>           bb.logger.setLevel(origlevel)
>   
> @@ -439,6 +440,7 @@ def modify(args, config, basepath, workspace):
>           initial_rev = _extract_source(args.srctree, False, args.branch, rd)
>           if not initial_rev:
>               return 1
> +        logger.info('Source tree extracted to %s' % srctree)
>           # Get list of commits since this revision
>           (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=args.srctree)
>           commits = stdout.split()
> diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
> new file mode 100644
> index 0000000..9bef984
> --- /dev/null
> +++ b/scripts/lib/devtool/upgrade.py
> @@ -0,0 +1,314 @@
> +# Development tool - upgrade command plugin
> +#
> +# Copyright (C) 2014-2015 Intel Corporation
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License version 2 as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +#
> +# DESCRIPTION
> +# Created a new recipe under workspace/recipes/<recipename> and place the
> +# source code into <srctree>.
> +# The upgrade feature executes the following steps:
> +#     * Extract current recipe source code into srctree and create branch
> +#     * Extract upgrade recipe source code into srctree and rebase with
> +#       previous branch. This step also creates a temporal recipe (created
> +#       using recipetool), containing the correct checksums.
> +#     * Creates the new recipe under the workspace
> +"""Devtool upgrade plugin"""
> +
> +import os
> +import sys
> +import re
> +import shutil
> +import tempfile
> +import logging
> +import argparse
> +import scriptutils
> +import errno
> +from devtool import standard
> +from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
> +
> +logger = logging.getLogger('devtool')
> +
> +def plugin_init(pluginlist):
> +    """Plugin initialization"""
> +    pass
> +
> +def _extract_upgrade_source(args, devbranch, config, basepath, d, recipepostfix='tmp'):
> +    """Extract sources of a recipe with PV given on args.version
> +
> +    On the target source tree folder, a new branch (<devbranch>_<args.version>)
> +    and tag (<devbranch>-base_<args.version>) will be created. In case patches
> +    are applied, another tag is created (<devbranch>-patched_<args.version>).
> +
> +    Returns: 1) The (git) initial revision ID
> +             2) The full path of a temporal recipe containing the correct checksums
> +    """
> +    import oe.recipeutils
> +
> +    initial_rev = None
> +    srctree = os.path.abspath(args.srctree)
> +
> +    pn = d.getVar('PN', True)
> +
> +    standard._check_compatible_recipe(pn, d)
> +
> +    recipepath = os.path.join(config.workspace_path, 'recipes', pn)
> +    bb.utils.mkdirhier(recipepath)
> +    recipefile = os.path.join(recipepath, "%s-%s.bb" % (pn,recipepostfix))
> +    tmpsrcbasetree = tempfile.mkdtemp(prefix='devtool')
> +
> +    # Change PV and get URL
> +    crd = d.createCopy()
> +    pv = d.getVar('PV', True)
> +    crd.setVar('PV', args.version)
> +    src_uri = crd.getVar('SRC_URI', True)
> +    if src_uri:
> +        url = src_uri.split()[0]
> +
> +    # Generate recipe and fetch new source
> +    try:
> +        cmdopts = "-o %s -x %s -V %s" % (recipefile, tmpsrcbasetree, args.version)
> +        cmd = 'recipetool create "%s" %s' % (url, cmdopts)
> +        stdout, _ = exec_build_env_command(config.init_path, basepath, cmd)
> +    except bb.process.ExecutionError as e:
> +        raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
> +
> +    tmpsrctree = os.path.join(tmpsrcbasetree, pn + '-' + args.version)
> +
> +    try:
> +        # branch from devtool-base (original source code without patches) before copying new source code
> +        bb.process.run('git checkout -b %s devtool-base' % devbranch, cwd=srctree)
> +        bb.process.run('git tag -f devtool-base_%s' % args.version, cwd=srctree)
> +
> +        # Copy tmpsrctree  into srctree
> +        src_files = standard._ls_tree(tmpsrctree)
> +        for path in src_files:
> +            tgt_dir = os.path.join(srctree, os.path.dirname(path))
> +            bb.utils.mkdirhier(tgt_dir)
> +            tgt_path = os.path.join(srctree, path)
> +            os.rename(os.path.join(tmpsrctree, path), tgt_path)
> +
> +        # Track modified and untracked files
> +        (stdout,_) = bb.process.run('git ls-files --modified --others --exclude-standard', cwd=srctree)
> +        add_files = stdout.splitlines()
> +        for add_file in add_files:
> +            bb.process.run('git add "%s"' % add_file, cwd=srctree)
> +
> +        if len(add_files):
> +            bb.process.run('git commit -q -m "Initial commit from upstream at version %s"' % args.version, cwd=srctree)
I don't know if is good to commit changes without user intervention. 
This tool is an interactive one may be ask to the
user?
> +        (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
> +        initial_rev = stdout.rstrip()
> +
> +        if args.no_patch:
> +            patches = oe.recipeutils.get_recipe_patches(crd)
> +            if len(patches):
> +                logger.warn('By user choice, the following (%s_%s) patches will NOT be applied into' %(pn,pv))
> +                for patch in patches:
> +                    logger.warn("\t%s" % os.path.basename(patch))
> +        else:
> +            try:
> +                bb.process.run('git rebase devtool-patched', cwd=srctree)
> +            except bb.process.ExecutionError as e:
> +                # this is the only case where we do not propagate the error
> +                # user will have the change to correct merges
> +                logger.error('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
> +            bb.process.run('git tag -f devtool-patched_%s' % args.version, cwd=srctree)
> +    except bb.process.ExecutionError as e:
> +        raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
> +    finally:
> +        if args.keep_temp:
> +            logger.info('Preserving temporary directory %s' % tmpsrcbasetree)
> +        else:
> +            shutil.rmtree(tmpsrcbasetree)
> +    return initial_rev, recipefile
> +
> +def _create_upgrade_recipe(args, config, recipetmp, config_data, d):
> +    """Creates the new recipe under workspace
> +
> +    Returns the full path on the new created recipe
> +    """
> +    import oe.recipeutils
> +
> +    def _get_checksums(recipefile):
> +        import re
> +        checksums = {}
> +        with open(recipefile) as rf:
> +            for line in rf:
> +                for cs in ['md5sum', 'sha256sum']:
> +                    m = re.match("^SRC_URI\[%s\].*=.*\"(.*)\"" % cs, line)
> +                    if m:
> +                        checksums[cs] = m.group(1)
> +        return checksums
> +
> +    def _replace_checksums(recipefile, checksums):
> +        import re
> +        with open(recipefile + ".tmp", "w+") as tmprf:
> +            with open(recipefile) as rf:
> +                for line in rf:
> +                    m = None
> +                    for cs in ['md5sum', 'sha256sum']:
> +                        m = re.match("^SRC_URI\[%s\].*=.*\"(.*)\"" % cs, line)
> +                        if m:
> +                            if cs in checksums:
> +                                oldcheck = m.group(1)
> +                                newcheck = checksums[cs]
> +                                line = line.replace(oldcheck, newcheck)
> +                            break
> +                    tmprf.write(line)
> +
> +        os.rename(recipefile + ".tmp", recipefile)
> +
> +    def _rename_patch_dirs(recipefolder, oldpv, newpv):
> +        for root, dirs, files in os.walk(recipefolder):
> +            for olddir in dirs:
> +                if olddir.find(oldpv) != -1:
> +                    newdir = olddir.replace(oldpv, newpv)
> +                    bb.process.run('mv %s %s' % (olddir, newdir))
> +
> +    def _remove_patch_dirs(recipefolder):
> +        for root, dirs, files in os.walk(recipefolder):
> +            for d in dirs:
> +                shutil.rmtree(os.path.join(root,d))
> +
> +    def _recipe_contains(recipefile, var):
> +        import re
> +        found = False
> +        with open(recipefile) as rf:
> +            for line in rf:
> +                if re.match("^%s.*=.*" % var, line):
> +                    found = True
> +                    break
> +        return found
> +
> +    if not os.path.exists(recipetmp):
> +        raise DevtoolError("Temporal recipe %s does not exist" % recipetmp)
> +
> +    # Copy current recipe into workspace
> +    pn = d.getVar('PN', True)
> +    recipepath = os.path.join(config.workspace_path, 'recipes', pn)
> +    oe.recipeutils.copy_recipe_files(d, recipepath)
> +
> +    # Rename recipe
> +    pv = d.getVar('PV', True)
> +    recipename = "%s_%s.bb" % (pn, pv)
> +    newrecipename = "%s_%s.bb" % (pn, args.version)
> +    if os.path.isfile(os.path.join(recipepath, recipename)):
> +        bb.process.run('mv %s %s' % (recipename, newrecipename), cwd=recipepath)
> +    else:
> +        # Check if it is a git recipe
> +        recipename = newrecipename = "%s_git.bb" % pn
> +        if not os.path.isfile(os.path.join(recipepath, recipename)):
> +            raise DevtoolError("Original recipe not found on workspace")
> +
> +    # Rename folders
> +    _rename_patch_dirs(recipepath, pv, args.version)
> +
> +    # Update PV, just in case it is present
> +    if _recipe_contains(os.path.join(recipepath, newrecipename), 'PV'):
> +        oe.recipeutils.patch_recipe(d, os.path.join(recipepath, newrecipename), {'PV':args.version})
> +
> +    # Update checksums
> +    checksums = _get_checksums(os.path.join(recipepath, recipetmp))
> +    _replace_checksums(os.path.join(recipepath, newrecipename), checksums)
> +
> +    # Remove recipe created by the recipe-tool
> +    bb.process.run('rm %s' % recipetmp)
> +
> +    return os.path.join(recipepath,newrecipename)
> +
> +def upgrade(args, config, basepath, workspace):
> +    """Entry point for the devtool 'upgrade' subcommand"""
> +    import bb
> +    import oe.recipeutils
> +
> +    if args.recipename in workspace:
> +        raise DevtoolError("recipe %s is already in your workspace" %
> +                            args.recipename)
> +    if not args.version:
> +        raise DevtoolError("Provide a version through the parameter --version/-V")
> +
> +    reason = oe.recipeutils.validate_pn(args.recipename)
> +    if reason:
> +        raise DevtoolError(reason)
> +
> +    tinfoil = setup_tinfoil()
> +
> +    rd = standard._parse_recipe(config, tinfoil, args.recipename, True)
> +    if not rd:
> +        return 1
> +
> +    pn = rd.getVar('PV', True)
> +    if pn == args.version:
> +        raise DevtoolError("Current and upgrade versions are the same %s" % pn)
> +
> +    srctree = os.path.abspath(args.srctree)
> +
> +    try:
> +        # Extract source from current recipe
> +        initial_rev_base = standard._extract_source(srctree, False, args.branch, rd)
> +
> +        # We need to shutdown tinfoil temporally because recipetool will be used
> +        tinfoil.shutdown()
> +
> +        # Extract new source code
> +        branch_upgrade = "%s_%s" % (args.branch, args.version)
> +        initial_rev_upgrade, recipe_file_tmp = _extract_upgrade_source(args, branch_upgrade, config, basepath, rd)
> +        # Start again tinfoil
> +        tinfoil = setup_tinfoil()
> +
> +        recipe_file = _create_upgrade_recipe(args, config, recipe_file_tmp, tinfoil.config_data, rd)
> +
> +    except DevtoolError as e:
> +        logger.error(e)
> +        # remove workspace recipe and srctree
> +        pn = rd.getVar('PN', True)
> +        recipespath = os.path.join(config.workspace_path, 'recipes')
> +        recipepath = os.path.join(recipespath, pn)
> +        if os.path.exists(recipepath):
> +            shutil.rmtree(recipepath)
> +            if not len(os.listdir(recipespath)):
> +                os.rmdir(recipespath)
> +        # remove srctree
> +        if os.path.exists(srctree):
> +            shutil.rmtree(srctree)

May be will be better to put this inside a clean function?

> +        raise DevtoolError(e)
> +
> +    appendpath = os.path.join(config.workspace_path, 'appends')
> +    if not os.path.exists(appendpath):
> +        os.makedirs(appendpath)
> +
> +    recipe_file_base = os.path.basename(os.path.splitext(recipe_file)[0])
> +    appendfile = os.path.join(appendpath, '%s.bbappend' % recipe_file_base)
> +    with open(appendfile, 'w') as f:
> +        f.write('inherit externalsrc\n')
> +        f.write('EXTERNALSRC = "%s"\n' % srctree)
> +        f.write('EXTERNALSRC_BUILD = "%s"\n' % srctree)
> +        f.write('\n# initial_rev: %s\n' % initial_rev_upgrade)
> +
> +    standard._add_md5(config, args.recipename, appendfile)
> +    logger.info('Source tree extracted to %s' % srctree)
> +    return 0
> +
> +def register_commands(subparsers, context):
> +    """Register devtool subcommands from this plugin"""
> +    parser_upgrade = subparsers.add_parser('upgrade', help='Upgrade an existing recipe',
> +                                       description='Upgrades an existing recipe',
> +                                       formatter_class=argparse.ArgumentDefaultsHelpFormatter)
> +    parser_upgrade.add_argument('recipename', help='Name for recipe to extract the source for')
> +    parser_upgrade.add_argument('srctree', help='Path to where to extract the source tree')
> +    parser_upgrade.add_argument('--version', '-V', help='Version to upgrade (PV)')
> +    parser_upgrade.add_argument('--branch', '-b', default="devtool", help='Name for development branch to checkout')
> +    parser_upgrade.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
> +    parser_upgrade.add_argument('--no-patch', action="store_true", help='Do not patch the new source code')
> +    parser_upgrade.set_defaults(func=upgrade)

Re: [PATCH v2 2/2] oeqa/selftest: new tests for devtool upgrage feature

[Aníbal Limón]

Comments below.

On 26/08/15 02:43, leonardo.sandoval.gonzalez@linux.intel.com wrote:
> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
>
> Basic tests for the devtool's upgrade feature, including:
>        * Parameter check
>        * Upgrading a real recipe (e2fsprogrs) without patching and
>        	checing its output
>        * Devtool status after upgrade
>
> Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
> ---
>   meta/lib/oeqa/selftest/devtool.py | 36 ++++++++++++++++++++++++++++++++++++
>   1 file changed, 36 insertions(+)
>
> diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
> index b59db15..f72e010 100644
> --- a/meta/lib/oeqa/selftest/devtool.py
> +++ b/meta/lib/oeqa/selftest/devtool.py
> @@ -857,3 +857,39 @@ class DevtoolTests(DevtoolBase):
>               result = runCmd('devtool undeploy-target -c %s root@%s' % (testrecipe, qemu.ip))
>               result = runCmd('ssh %s root@%s %s' % (sshargs, qemu.ip, testcommand), ignore_status=True)
>               self.assertNotEqual(result, 0, 'undeploy-target did not remove command as it should have')
> +
> +    def test_devtool_upgrade(self):
> +        # Check preconditions
> +        workspacedir = os.path.join(self.builddir, 'workspace')
> +        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
> +        # Check parameters
> +        result = runCmd('devtool upgrade -h')
> +        for param in 'recipename srctree --version -V --branch -b --keep-temp --no-patch'.split():
> +            self.assertIn(param, result.output)
> +        # For the moment, we are using a real recipe.
> +        recipe='e2fsprogs'
> +        version='1.42.13'

Isn't a guarantee that this version will be newer ever (i.e. if someone 
upgrade e2fsprogs to 1.42.13) what happen in this case?
> +        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
> +        # Check that recipe is not already under devtool control
> +        result = runCmd('devtool status')
> +        self.assertNotIn(recipe, result.output)
> +        # Check upgrade. Code does not check if new PV is older or newer that current PV, so, it may be that
> +        # we are downgrading instead of upgrading.
> +        result = runCmd('devtool upgrade %s %s -V %s --no-patch' % (recipe, tempdir, version))
> +        # Check if srctree at least is populated
> +        self.assertTrue(len(os.listdir(tempdir)) > 0, 'scrtree (%s) should be populated with new (%s) source code' % (tempdir, version))
> +        # Check new recipe folder is present
> +        self.assertTrue(os.path.exists(os.path.join(workspacedir,'recipes',recipe)), 'Recipe folder should exist')
> +        # Check new recipe file is present
> +        self.assertTrue(os.path.exists(os.path.join(workspacedir,'recipes',recipe,"%s_%s.bb" % (recipe,version))), 'Recipe folder should exist')
> +        # Check devtool status and make sure recipe is present
> +        result = runCmd('devtool status')
> +        self.assertIn(recipe, result.output)
> +        self.assertIn(tempdir, result.output)
> +        # Check devtool reset recipe
> +        result = runCmd('devtool reset %s -n' % recipe)
> +        result = runCmd('devtool status')
> +        self.assertNotIn(recipe, result.output)
> +        self.track_for_cleanup(tempdir)
> +        self.track_for_cleanup(workspacedir)
> +        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')

Re: [PATCH v2 1/2] devtool: upgrade feature

[Leonardo Sandoval]

On 08/26/2015 11:09 AM, Aníbal Limón wrote:
> Comments below,
>
> On 26/08/15 02:43, leonardo.sandoval.gonzalez@linux.intel.com wrote:
>> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
>>
>> Upgrades a recipe to a particular version and downloads the source code
>> into srctree. User can avoid patching the source code. These are the
>> general steps of the upgrade function:
>>     * Extract current recipe source code into srctree and create branch
>>     * Extract upgrade recipe source code into srctree and rebase with
>>       previous branch. This step also creates a temporal recipe (created
>>       using recipetool), containing the correct checksums.
>>     * Creates the new recipe under the workspace
>>
>> [YOCTO #7642]
>>
>> Signed-off-by: Leonardo Sandoval
>> <leonardo.sandoval.gonzalez@linux.intel.com>
>> ---
>>   scripts/lib/devtool/standard.py |   4 +-
>>   scripts/lib/devtool/upgrade.py  | 314
>> ++++++++++++++++++++++++++++++++++++++++
>>   2 files changed, 317 insertions(+), 1 deletion(-)
>>   create mode 100644 scripts/lib/devtool/upgrade.py
>>
>> diff --git a/scripts/lib/devtool/standard.py
>> b/scripts/lib/devtool/standard.py
>> index ea21877..cd5a3ed 100644
>> --- a/scripts/lib/devtool/standard.py
>> +++ b/scripts/lib/devtool/standard.py
>> @@ -205,6 +205,8 @@ def extract(args, config, basepath, workspace):
>>       srctree = os.path.abspath(args.srctree)
>>       initial_rev = _extract_source(srctree, args.keep_temp,
>> args.branch, rd)
>> +    logger.info('Source tree extracted to %s' % srctree)
>> +
>>       if initial_rev:
>>           return 0
>>       else:
>> @@ -360,7 +362,6 @@ def _extract_source(srctree, keep_temp, devbranch,
>> d):
>>                   bb.process.run('git checkout patches', cwd=srcsubdir)
>>           shutil.move(srcsubdir, srctree)
>> -        logger.info('Source tree extracted to %s' % srctree)
> Why you move this logging outside?

the upgrade functions does two extractions, so I did not want two prints 
indicating the same message.

>>       finally:
>>           bb.logger.setLevel(origlevel)
>> @@ -439,6 +440,7 @@ def modify(args, config, basepath, workspace):
>>           initial_rev = _extract_source(args.srctree, False,
>> args.branch, rd)
>>           if not initial_rev:
>>               return 1
>> +        logger.info('Source tree extracted to %s' % srctree)
>>           # Get list of commits since this revision
>>           (stdout, _) = bb.process.run('git rev-list --reverse
>> %s..HEAD' % initial_rev, cwd=args.srctree)
>>           commits = stdout.split()
>> diff --git a/scripts/lib/devtool/upgrade.py
>> b/scripts/lib/devtool/upgrade.py
>> new file mode 100644
>> index 0000000..9bef984
>> --- /dev/null
>> +++ b/scripts/lib/devtool/upgrade.py
>> @@ -0,0 +1,314 @@
>> +# Development tool - upgrade command plugin
>> +#
>> +# Copyright (C) 2014-2015 Intel Corporation
>> +#
>> +# This program is free software; you can redistribute it and/or modify
>> +# it under the terms of the GNU General Public License version 2 as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> along
>> +# with this program; if not, write to the Free Software Foundation,
>> Inc.,
>> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
>> +#
>> +# DESCRIPTION
>> +# Created a new recipe under workspace/recipes/<recipename> and place
>> the
>> +# source code into <srctree>.
>> +# The upgrade feature executes the following steps:
>> +#     * Extract current recipe source code into srctree and create
>> branch
>> +#     * Extract upgrade recipe source code into srctree and rebase with
>> +#       previous branch. This step also creates a temporal recipe
>> (created
>> +#       using recipetool), containing the correct checksums.
>> +#     * Creates the new recipe under the workspace
>> +"""Devtool upgrade plugin"""
>> +
>> +import os
>> +import sys
>> +import re
>> +import shutil
>> +import tempfile
>> +import logging
>> +import argparse
>> +import scriptutils
>> +import errno
>> +from devtool import standard
>> +from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
>> +
>> +logger = logging.getLogger('devtool')
>> +
>> +def plugin_init(pluginlist):
>> +    """Plugin initialization"""
>> +    pass
>> +
>> +def _extract_upgrade_source(args, devbranch, config, basepath, d,
>> recipepostfix='tmp'):
>> +    """Extract sources of a recipe with PV given on args.version
>> +
>> +    On the target source tree folder, a new branch
>> (<devbranch>_<args.version>)
>> +    and tag (<devbranch>-base_<args.version>) will be created. In
>> case patches
>> +    are applied, another tag is created
>> (<devbranch>-patched_<args.version>).
>> +
>> +    Returns: 1) The (git) initial revision ID
>> +             2) The full path of a temporal recipe containing the
>> correct checksums
>> +    """
>> +    import oe.recipeutils
>> +
>> +    initial_rev = None
>> +    srctree = os.path.abspath(args.srctree)
>> +
>> +    pn = d.getVar('PN', True)
>> +
>> +    standard._check_compatible_recipe(pn, d)
>> +
>> +    recipepath = os.path.join(config.workspace_path, 'recipes', pn)
>> +    bb.utils.mkdirhier(recipepath)
>> +    recipefile = os.path.join(recipepath, "%s-%s.bb" %
>> (pn,recipepostfix))
>> +    tmpsrcbasetree = tempfile.mkdtemp(prefix='devtool')
>> +
>> +    # Change PV and get URL
>> +    crd = d.createCopy()
>> +    pv = d.getVar('PV', True)
>> +    crd.setVar('PV', args.version)
>> +    src_uri = crd.getVar('SRC_URI', True)
>> +    if src_uri:
>> +        url = src_uri.split()[0]
>> +
>> +    # Generate recipe and fetch new source
>> +    try:
>> +        cmdopts = "-o %s -x %s -V %s" % (recipefile, tmpsrcbasetree,
>> args.version)
>> +        cmd = 'recipetool create "%s" %s' % (url, cmdopts)
>> +        stdout, _ = exec_build_env_command(config.init_path,
>> basepath, cmd)
>> +    except bb.process.ExecutionError as e:
>> +        raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command,
>> e.stdout))
>> +
>> +    tmpsrctree = os.path.join(tmpsrcbasetree, pn + '-' + args.version)
>> +
>> +    try:
>> +        # branch from devtool-base (original source code without
>> patches) before copying new source code
>> +        bb.process.run('git checkout -b %s devtool-base' % devbranch,
>> cwd=srctree)
>> +        bb.process.run('git tag -f devtool-base_%s' % args.version,
>> cwd=srctree)
>> +
>> +        # Copy tmpsrctree  into srctree
>> +        src_files = standard._ls_tree(tmpsrctree)
>> +        for path in src_files:
>> +            tgt_dir = os.path.join(srctree, os.path.dirname(path))
>> +            bb.utils.mkdirhier(tgt_dir)
>> +            tgt_path = os.path.join(srctree, path)
>> +            os.rename(os.path.join(tmpsrctree, path), tgt_path)
>> +
>> +        # Track modified and untracked files
>> +        (stdout,_) = bb.process.run('git ls-files --modified --others
>> --exclude-standard', cwd=srctree)
>> +        add_files = stdout.splitlines()
>> +        for add_file in add_files:
>> +            bb.process.run('git add "%s"' % add_file, cwd=srctree)
>> +
>> +        if len(add_files):
>> +            bb.process.run('git commit -q -m "Initial commit from
>> upstream at version %s"' % args.version, cwd=srctree)
> I don't know if is good to commit changes without user intervention.
> This tool is an interactive one may be ask to the
> user?

I do not get your question, can you please elaborate a more?

Let me explain a bit more: Internally, code creates a git repository and 
branches, so the user can start pushing commits from it. This does not 
require user intervention; at least this is how is done with the rest of 
the commands.

>> +        (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
>> +        initial_rev = stdout.rstrip()
>> +
>> +        if args.no_patch:
>> +            patches = oe.recipeutils.get_recipe_patches(crd)
>> +            if len(patches):
>> +                logger.warn('By user choice, the following (%s_%s)
>> patches will NOT be applied into' %(pn,pv))
>> +                for patch in patches:
>> +                    logger.warn("\t%s" % os.path.basename(patch))
>> +        else:
>> +            try:
>> +                bb.process.run('git rebase devtool-patched',
>> cwd=srctree)
>> +            except bb.process.ExecutionError as e:
>> +                # this is the only case where we do not propagate the
>> error
>> +                # user will have the change to correct merges
>> +                logger.error('Command \'%s\' failed:\n%s' %
>> (e.command, e.stdout))
>> +            bb.process.run('git tag -f devtool-patched_%s' %
>> args.version, cwd=srctree)
>> +    except bb.process.ExecutionError as e:
>> +        raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command,
>> e.stdout))
>> +    finally:
>> +        if args.keep_temp:
>> +            logger.info('Preserving temporary directory %s' %
>> tmpsrcbasetree)
>> +        else:
>> +            shutil.rmtree(tmpsrcbasetree)
>> +    return initial_rev, recipefile
>> +
>> +def _create_upgrade_recipe(args, config, recipetmp, config_data, d):
>> +    """Creates the new recipe under workspace
>> +
>> +    Returns the full path on the new created recipe
>> +    """
>> +    import oe.recipeutils
>> +
>> +    def _get_checksums(recipefile):
>> +        import re
>> +        checksums = {}
>> +        with open(recipefile) as rf:
>> +            for line in rf:
>> +                for cs in ['md5sum', 'sha256sum']:
>> +                    m = re.match("^SRC_URI\[%s\].*=.*\"(.*)\"" % cs,
>> line)
>> +                    if m:
>> +                        checksums[cs] = m.group(1)
>> +        return checksums
>> +
>> +    def _replace_checksums(recipefile, checksums):
>> +        import re
>> +        with open(recipefile + ".tmp", "w+") as tmprf:
>> +            with open(recipefile) as rf:
>> +                for line in rf:
>> +                    m = None
>> +                    for cs in ['md5sum', 'sha256sum']:
>> +                        m = re.match("^SRC_URI\[%s\].*=.*\"(.*)\"" %
>> cs, line)
>> +                        if m:
>> +                            if cs in checksums:
>> +                                oldcheck = m.group(1)
>> +                                newcheck = checksums[cs]
>> +                                line = line.replace(oldcheck, newcheck)
>> +                            break
>> +                    tmprf.write(line)
>> +
>> +        os.rename(recipefile + ".tmp", recipefile)
>> +
>> +    def _rename_patch_dirs(recipefolder, oldpv, newpv):
>> +        for root, dirs, files in os.walk(recipefolder):
>> +            for olddir in dirs:
>> +                if olddir.find(oldpv) != -1:
>> +                    newdir = olddir.replace(oldpv, newpv)
>> +                    bb.process.run('mv %s %s' % (olddir, newdir))
>> +
>> +    def _remove_patch_dirs(recipefolder):
>> +        for root, dirs, files in os.walk(recipefolder):
>> +            for d in dirs:
>> +                shutil.rmtree(os.path.join(root,d))
>> +
>> +    def _recipe_contains(recipefile, var):
>> +        import re
>> +        found = False
>> +        with open(recipefile) as rf:
>> +            for line in rf:
>> +                if re.match("^%s.*=.*" % var, line):
>> +                    found = True
>> +                    break
>> +        return found
>> +
>> +    if not os.path.exists(recipetmp):
>> +        raise DevtoolError("Temporal recipe %s does not exist" %
>> recipetmp)
>> +
>> +    # Copy current recipe into workspace
>> +    pn = d.getVar('PN', True)
>> +    recipepath = os.path.join(config.workspace_path, 'recipes', pn)
>> +    oe.recipeutils.copy_recipe_files(d, recipepath)
>> +
>> +    # Rename recipe
>> +    pv = d.getVar('PV', True)
>> +    recipename = "%s_%s.bb" % (pn, pv)
>> +    newrecipename = "%s_%s.bb" % (pn, args.version)
>> +    if os.path.isfile(os.path.join(recipepath, recipename)):
>> +        bb.process.run('mv %s %s' % (recipename, newrecipename),
>> cwd=recipepath)
>> +    else:
>> +        # Check if it is a git recipe
>> +        recipename = newrecipename = "%s_git.bb" % pn
>> +        if not os.path.isfile(os.path.join(recipepath, recipename)):
>> +            raise DevtoolError("Original recipe not found on workspace")
>> +
>> +    # Rename folders
>> +    _rename_patch_dirs(recipepath, pv, args.version)
>> +
>> +    # Update PV, just in case it is present
>> +    if _recipe_contains(os.path.join(recipepath, newrecipename), 'PV'):
>> +        oe.recipeutils.patch_recipe(d, os.path.join(recipepath,
>> newrecipename), {'PV':args.version})
>> +
>> +    # Update checksums
>> +    checksums = _get_checksums(os.path.join(recipepath, recipetmp))
>> +    _replace_checksums(os.path.join(recipepath, newrecipename),
>> checksums)
>> +
>> +    # Remove recipe created by the recipe-tool
>> +    bb.process.run('rm %s' % recipetmp)
>> +
>> +    return os.path.join(recipepath,newrecipename)
>> +
>> +def upgrade(args, config, basepath, workspace):
>> +    """Entry point for the devtool 'upgrade' subcommand"""
>> +    import bb
>> +    import oe.recipeutils
>> +
>> +    if args.recipename in workspace:
>> +        raise DevtoolError("recipe %s is already in your workspace" %
>> +                            args.recipename)
>> +    if not args.version:
>> +        raise DevtoolError("Provide a version through the parameter
>> --version/-V")
>> +
>> +    reason = oe.recipeutils.validate_pn(args.recipename)
>> +    if reason:
>> +        raise DevtoolError(reason)
>> +
>> +    tinfoil = setup_tinfoil()
>> +
>> +    rd = standard._parse_recipe(config, tinfoil, args.recipename, True)
>> +    if not rd:
>> +        return 1
>> +
>> +    pn = rd.getVar('PV', True)
>> +    if pn == args.version:
>> +        raise DevtoolError("Current and upgrade versions are the same
>> %s" % pn)
>> +
>> +    srctree = os.path.abspath(args.srctree)
>> +
>> +    try:
>> +        # Extract source from current recipe
>> +        initial_rev_base = standard._extract_source(srctree, False,
>> args.branch, rd)
>> +
>> +        # We need to shutdown tinfoil temporally because recipetool
>> will be used
>> +        tinfoil.shutdown()
>> +
>> +        # Extract new source code
>> +        branch_upgrade = "%s_%s" % (args.branch, args.version)
>> +        initial_rev_upgrade, recipe_file_tmp =
>> _extract_upgrade_source(args, branch_upgrade, config, basepath, rd)
>> +        # Start again tinfoil
>> +        tinfoil = setup_tinfoil()
>> +
>> +        recipe_file = _create_upgrade_recipe(args, config,
>> recipe_file_tmp, tinfoil.config_data, rd)
>> +
>> +    except DevtoolError as e:
>> +        logger.error(e)
>> +        # remove workspace recipe and srctree
>> +        pn = rd.getVar('PN', True)
>> +        recipespath = os.path.join(config.workspace_path, 'recipes')
>> +        recipepath = os.path.join(recipespath, pn)
>> +        if os.path.exists(recipepath):
>> +            shutil.rmtree(recipepath)
>> +            if not len(os.listdir(recipespath)):
>> +                os.rmdir(recipespath)
>> +        # remove srctree
>> +        if os.path.exists(srctree):
>> +            shutil.rmtree(srctree)
>
> May be will be better to put this inside a clean function?
>
>> +        raise DevtoolError(e)
>> +
>> +    appendpath = os.path.join(config.workspace_path, 'appends')
>> +    if not os.path.exists(appendpath):
>> +        os.makedirs(appendpath)
>> +
>> +    recipe_file_base =
>> os.path.basename(os.path.splitext(recipe_file)[0])
>> +    appendfile = os.path.join(appendpath, '%s.bbappend' %
>> recipe_file_base)
>> +    with open(appendfile, 'w') as f:
>> +        f.write('inherit externalsrc\n')
>> +        f.write('EXTERNALSRC = "%s"\n' % srctree)
>> +        f.write('EXTERNALSRC_BUILD = "%s"\n' % srctree)
>> +        f.write('\n# initial_rev: %s\n' % initial_rev_upgrade)
>> +
>> +    standard._add_md5(config, args.recipename, appendfile)
>> +    logger.info('Source tree extracted to %s' % srctree)
>> +    return 0
>> +
>> +def register_commands(subparsers, context):
>> +    """Register devtool subcommands from this plugin"""
>> +    parser_upgrade = subparsers.add_parser('upgrade', help='Upgrade
>> an existing recipe',
>> +                                       description='Upgrades an
>> existing recipe',
>> +
>> formatter_class=argparse.ArgumentDefaultsHelpFormatter)
>> +    parser_upgrade.add_argument('recipename', help='Name for recipe
>> to extract the source for')
>> +    parser_upgrade.add_argument('srctree', help='Path to where to
>> extract the source tree')
>> +    parser_upgrade.add_argument('--version', '-V', help='Version to
>> upgrade (PV)')
>> +    parser_upgrade.add_argument('--branch', '-b', default="devtool",
>> help='Name for development branch to checkout')
>> +    parser_upgrade.add_argument('--keep-temp', action="store_true",
>> help='Keep temporary directory (for debugging)')
>> +    parser_upgrade.add_argument('--no-patch', action="store_true",
>> help='Do not patch the new source code')
>> +    parser_upgrade.set_defaults(func=upgrade)
>

Re: [PATCH v2 2/2] oeqa/selftest: new tests for devtool upgrage feature

[Leonardo Sandoval]

On 08/26/2015 11:15 AM, Aníbal Limón wrote:
> Comments below.
>
> On 26/08/15 02:43, leonardo.sandoval.gonzalez@linux.intel.com wrote:
>> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
>>
>> Basic tests for the devtool's upgrade feature, including:
>>        * Parameter check
>>        * Upgrading a real recipe (e2fsprogrs) without patching and
>>            checing its output
>>        * Devtool status after upgrade
>>
>> Signed-off-by: Leonardo Sandoval
>> <leonardo.sandoval.gonzalez@linux.intel.com>
>> ---
>>   meta/lib/oeqa/selftest/devtool.py | 36
>> ++++++++++++++++++++++++++++++++++++
>>   1 file changed, 36 insertions(+)
>>
>> diff --git a/meta/lib/oeqa/selftest/devtool.py
>> b/meta/lib/oeqa/selftest/devtool.py
>> index b59db15..f72e010 100644
>> --- a/meta/lib/oeqa/selftest/devtool.py
>> +++ b/meta/lib/oeqa/selftest/devtool.py
>> @@ -857,3 +857,39 @@ class DevtoolTests(DevtoolBase):
>>               result = runCmd('devtool undeploy-target -c %s root@%s'
>> % (testrecipe, qemu.ip))
>>               result = runCmd('ssh %s root@%s %s' % (sshargs, qemu.ip,
>> testcommand), ignore_status=True)
>>               self.assertNotEqual(result, 0, 'undeploy-target did not
>> remove command as it should have')
>> +
>> +    def test_devtool_upgrade(self):
>> +        # Check preconditions
>> +        workspacedir = os.path.join(self.builddir, 'workspace')
>> +        self.assertTrue(not os.path.exists(workspacedir), 'This test
>> cannot be run with a workspace directory under the build directory')
>> +        # Check parameters
>> +        result = runCmd('devtool upgrade -h')
>> +        for param in 'recipename srctree --version -V --branch -b
>> --keep-temp --no-patch'.split():
>> +            self.assertIn(param, result.output)
>> +        # For the moment, we are using a real recipe.
>> +        recipe='e2fsprogs'
>> +        version='1.42.13'
>
> Isn't a guarantee that this version will be newer ever (i.e. if someone
> upgrade e2fsprogs to 1.42.13) what happen in this case?

Good question. In fact, the code will fail just in case current and 
upgrade version are the same. The tool can also do downgrades, so at the 
end it does not matter what the version is. I will correct this point 
and send a v3.

As mentioned by Paul in a previous message, we need 'upgradable' recipes 
which whenever the test is done, it can be upgrade. The only point here 
is that we need a tarball somewhere, so the URL is always live. Not sure 
if poky repository is a good place for the latter, any suggestion?


>> +        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
>> +        # Check that recipe is not already under devtool control
>> +        result = runCmd('devtool status')
>> +        self.assertNotIn(recipe, result.output)
>> +        # Check upgrade. Code does not check if new PV is older or
>> newer that current PV, so, it may be that
>> +        # we are downgrading instead of upgrading.
>> +        result = runCmd('devtool upgrade %s %s -V %s --no-patch' %
>> (recipe, tempdir, version))
>> +        # Check if srctree at least is populated
>> +        self.assertTrue(len(os.listdir(tempdir)) > 0, 'scrtree (%s)
>> should be populated with new (%s) source code' % (tempdir, version))
>> +        # Check new recipe folder is present
>> +
>> self.assertTrue(os.path.exists(os.path.join(workspacedir,'recipes',recipe)),
>> 'Recipe folder should exist')
>> +        # Check new recipe file is present
>> +
>> self.assertTrue(os.path.exists(os.path.join(workspacedir,'recipes',recipe,"%s_%s.bb"
>> % (recipe,version))), 'Recipe folder should exist')
>> +        # Check devtool status and make sure recipe is present
>> +        result = runCmd('devtool status')
>> +        self.assertIn(recipe, result.output)
>> +        self.assertIn(tempdir, result.output)
>> +        # Check devtool reset recipe
>> +        result = runCmd('devtool reset %s -n' % recipe)
>> +        result = runCmd('devtool status')
>> +        self.assertNotIn(recipe, result.output)
>> +        self.track_for_cleanup(tempdir)
>> +        self.track_for_cleanup(workspacedir)
>> +        self.add_command_to_tearDown('bitbake-layers remove-layer
>> */workspace')
>

Re: [PATCH v2 2/2] oeqa/selftest: new tests for devtool upgrage feature

[Paul Eggleton]

On Wednesday 26 August 2015 11:29:37 Leonardo Sandoval wrote:
> On 08/26/2015 11:15 AM, Aníbal Limón wrote:
> > Comments below.
> > 
> > On 26/08/15 02:43, leonardo.sandoval.gonzalez@linux.intel.com wrote:
> >> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
> >> 
> >> Basic tests for the devtool's upgrade feature, including:
> >>        * Parameter check
> >>        * Upgrading a real recipe (e2fsprogrs) without patching and
> >>        
> >>            checing its output
> >>        
> >>        * Devtool status after upgrade
> >> 
> >> Signed-off-by: Leonardo Sandoval
> >> <leonardo.sandoval.gonzalez@linux.intel.com>
> >> ---
> >> 
> >>   meta/lib/oeqa/selftest/devtool.py | 36
> >> 
> >> ++++++++++++++++++++++++++++++++++++
> >> 
> >>   1 file changed, 36 insertions(+)
> >> 
> >> diff --git a/meta/lib/oeqa/selftest/devtool.py
> >> b/meta/lib/oeqa/selftest/devtool.py
> >> index b59db15..f72e010 100644
> >> --- a/meta/lib/oeqa/selftest/devtool.py
> >> +++ b/meta/lib/oeqa/selftest/devtool.py
> >> 
> >> @@ -857,3 +857,39 @@ class DevtoolTests(DevtoolBase):
> >>               result = runCmd('devtool undeploy-target -c %s root@%s'
> >> 
> >> % (testrecipe, qemu.ip))
> >> 
> >>               result = runCmd('ssh %s root@%s %s' % (sshargs, qemu.ip,
> >> 
> >> testcommand), ignore_status=True)
> >> 
> >>               self.assertNotEqual(result, 0, 'undeploy-target did not
> >> 
> >> remove command as it should have')
> >> +
> >> +    def test_devtool_upgrade(self):
> >> +        # Check preconditions
> >> +        workspacedir = os.path.join(self.builddir, 'workspace')
> >> +        self.assertTrue(not os.path.exists(workspacedir), 'This test
> >> cannot be run with a workspace directory under the build directory')
> >> +        # Check parameters
> >> +        result = runCmd('devtool upgrade -h')
> >> +        for param in 'recipename srctree --version -V --branch -b
> >> --keep-temp --no-patch'.split():
> >> +            self.assertIn(param, result.output)
> >> +        # For the moment, we are using a real recipe.
> >> +        recipe='e2fsprogs'
> >> +        version='1.42.13'
> > 
> > Isn't a guarantee that this version will be newer ever (i.e. if someone
> > upgrade e2fsprogs to 1.42.13) what happen in this case?
> 
> Good question. In fact, the code will fail just in case current and
> upgrade version are the same. The tool can also do downgrades, so at the
> end it does not matter what the version is. I will correct this point
> and send a v3.
> 
> As mentioned by Paul in a previous message, we need 'upgradable' recipes
> which whenever the test is done, it can be upgrade. The only point here
> is that we need a tarball somewhere, so the URL is always live. Not sure
> if poky repository is a good place for the latter, any suggestion?

This is what we have meta-selftest for. We can put a recipe (or multiple 
recipes) in there that's named specially so as not to overlap with one in OE-
Core and is a known old version.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH v2 2/2] oeqa/selftest: new tests for devtool upgrage feature

[Leonardo Sandoval]

On 08/26/2015 11:47 AM, Paul Eggleton wrote:
> On Wednesday 26 August 2015 11:29:37 Leonardo Sandoval wrote:
>> On 08/26/2015 11:15 AM, Aníbal Limón wrote:
>>> Comments below.
>>>
>>> On 26/08/15 02:43, leonardo.sandoval.gonzalez@linux.intel.com wrote:
>>>> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
>>>>
>>>> Basic tests for the devtool's upgrade feature, including:
>>>>         * Parameter check
>>>>         * Upgrading a real recipe (e2fsprogrs) without patching and
>>>>
>>>>             checing its output
>>>>
>>>>         * Devtool status after upgrade
>>>>
>>>> Signed-off-by: Leonardo Sandoval
>>>> <leonardo.sandoval.gonzalez@linux.intel.com>
>>>> ---
>>>>
>>>>    meta/lib/oeqa/selftest/devtool.py | 36
>>>>
>>>> ++++++++++++++++++++++++++++++++++++
>>>>
>>>>    1 file changed, 36 insertions(+)
>>>>
>>>> diff --git a/meta/lib/oeqa/selftest/devtool.py
>>>> b/meta/lib/oeqa/selftest/devtool.py
>>>> index b59db15..f72e010 100644
>>>> --- a/meta/lib/oeqa/selftest/devtool.py
>>>> +++ b/meta/lib/oeqa/selftest/devtool.py
>>>>
>>>> @@ -857,3 +857,39 @@ class DevtoolTests(DevtoolBase):
>>>>                result = runCmd('devtool undeploy-target -c %s root@%s'
>>>>
>>>> % (testrecipe, qemu.ip))
>>>>
>>>>                result = runCmd('ssh %s root@%s %s' % (sshargs, qemu.ip,
>>>>
>>>> testcommand), ignore_status=True)
>>>>
>>>>                self.assertNotEqual(result, 0, 'undeploy-target did not
>>>>
>>>> remove command as it should have')
>>>> +
>>>> +    def test_devtool_upgrade(self):
>>>> +        # Check preconditions
>>>> +        workspacedir = os.path.join(self.builddir, 'workspace')
>>>> +        self.assertTrue(not os.path.exists(workspacedir), 'This test
>>>> cannot be run with a workspace directory under the build directory')
>>>> +        # Check parameters
>>>> +        result = runCmd('devtool upgrade -h')
>>>> +        for param in 'recipename srctree --version -V --branch -b
>>>> --keep-temp --no-patch'.split():
>>>> +            self.assertIn(param, result.output)
>>>> +        # For the moment, we are using a real recipe.
>>>> +        recipe='e2fsprogs'
>>>> +        version='1.42.13'
>>>
>>> Isn't a guarantee that this version will be newer ever (i.e. if someone
>>> upgrade e2fsprogs to 1.42.13) what happen in this case?
>>
>> Good question. In fact, the code will fail just in case current and
>> upgrade version are the same. The tool can also do downgrades, so at the
>> end it does not matter what the version is. I will correct this point
>> and send a v3.
>>
>> As mentioned by Paul in a previous message, we need 'upgradable' recipes
>> which whenever the test is done, it can be upgrade. The only point here
>> is that we need a tarball somewhere, so the URL is always live. Not sure
>> if poky repository is a good place for the latter, any suggestion?
>
> This is what we have meta-selftest for. We can put a recipe (or multiple
> recipes) in there that's named specially so as not to overlap with one in OE-
> Core and is a known old version.

Good. I will add a new recipe there and a tarball. Sending a V3 today.

>
> Cheers,
> Paul
>

Re: [PATCH v2 1/2] devtool: upgrade feature

[Paul Eggleton]

On Wednesday 26 August 2015 07:43:23 
leonardo.sandoval.gonzalez@linux.intel.com wrote:
> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
> 
> Upgrades a recipe to a particular version and downloads the source code
> into srctree. User can avoid patching the source code. These are the
> general steps of the upgrade function:
>    * Extract current recipe source code into srctree and create branch
>    * Extract upgrade recipe source code into srctree and rebase with
>      previous branch. This step also creates a temporal recipe (created
>      using recipetool), containing the correct checksums.
>    * Creates the new recipe under the workspace

OK, I haven't tested this yet but some comments just looking over the code 
below.


> [YOCTO #7642]
> 
> Signed-off-by: Leonardo Sandoval
> <leonardo.sandoval.gonzalez@linux.intel.com> ---
>  scripts/lib/devtool/standard.py |   4 +-
>  scripts/lib/devtool/upgrade.py  | 314
> ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 317
> insertions(+), 1 deletion(-)
>  create mode 100644 scripts/lib/devtool/upgrade.py
> 
> diff --git a/scripts/lib/devtool/standard.py
> b/scripts/lib/devtool/standard.py index ea21877..cd5a3ed 100644
> --- a/scripts/lib/devtool/standard.py
> +++ b/scripts/lib/devtool/standard.py
> @@ -205,6 +205,8 @@ def extract(args, config, basepath, workspace):
> 
>      srctree = os.path.abspath(args.srctree)
>      initial_rev = _extract_source(srctree, args.keep_temp, args.branch, rd)
> +    logger.info('Source tree extracted to %s' % srctree)
> +
>      if initial_rev:
>          return 0
>      else:
> @@ -360,7 +362,6 @@ def _extract_source(srctree, keep_temp, devbranch, d):
>                  bb.process.run('git checkout patches', cwd=srcsubdir)
> 
>          shutil.move(srcsubdir, srctree)
> -        logger.info('Source tree extracted to %s' % srctree)
>      finally:
>          bb.logger.setLevel(origlevel)
> 
> @@ -439,6 +440,7 @@ def modify(args, config, basepath, workspace):
>          initial_rev = _extract_source(args.srctree, False, args.branch, rd)
> if not initial_rev:
>              return 1
> +        logger.info('Source tree extracted to %s' % srctree)
>          # Get list of commits since this revision
>          (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' %
> initial_rev, cwd=args.srctree) commits = stdout.split()
> diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
> new file mode 100644
> index 0000000..9bef984
> --- /dev/null
> +++ b/scripts/lib/devtool/upgrade.py
> @@ -0,0 +1,314 @@
> +# Development tool - upgrade command plugin
> +#
> +# Copyright (C) 2014-2015 Intel Corporation
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License version 2 as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +#
> +# DESCRIPTION
> +# Created a new recipe under workspace/recipes/<recipename> and place the
> +# source code into <srctree>.
> +# The upgrade feature executes the following steps:
> +#     * Extract current recipe source code into srctree and create branch
> +#     * Extract upgrade recipe source code into srctree and rebase with
> +#       previous branch. This step also creates a temporal recipe (created
> +#       using recipetool), containing the correct checksums.
> +#     * Creates the new recipe under the workspace
> +"""Devtool upgrade plugin"""
> +
> +import os
> +import sys
> +import re
> +import shutil
> +import tempfile
> +import logging
> +import argparse
> +import scriptutils
> +import errno
> +from devtool import standard
> +from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
> +
> +logger = logging.getLogger('devtool')
> +
> +def plugin_init(pluginlist):
> +    """Plugin initialization"""
> +    pass
> +
> +def _extract_upgrade_source(args, devbranch, config, basepath, d,
> recipepostfix='tmp'): +    """Extract sources of a recipe with PV given on
> args.version
> +
> +    On the target source tree folder, a new branch
> (<devbranch>_<args.version>) +    and tag (<devbranch>-base_<args.version>)
> will be created. In case patches +    are applied, another tag is created
> (<devbranch>-patched_<args.version>). +
> +    Returns: 1) The (git) initial revision ID
> +             2) The full path of a temporal recipe containing the correct
> checksums +    """
> +    import oe.recipeutils
> +
> +    initial_rev = None
> +    srctree = os.path.abspath(args.srctree)
> +
> +    pn = d.getVar('PN', True)
> +
> +    standard._check_compatible_recipe(pn, d)
> +
> +    recipepath = os.path.join(config.workspace_path, 'recipes', pn)
> +    bb.utils.mkdirhier(recipepath)
> +    recipefile = os.path.join(recipepath, "%s-%s.bb" % (pn,recipepostfix))
> +    tmpsrcbasetree = tempfile.mkdtemp(prefix='devtool')
> +
> +    # Change PV and get URL
> +    crd = d.createCopy()
> +    pv = d.getVar('PV', True)
> +    crd.setVar('PV', args.version)
> +    src_uri = crd.getVar('SRC_URI', True)
> +    if src_uri:
> +        url = src_uri.split()[0]
> +
> +    # Generate recipe and fetch new source
> +    try:
> +        cmdopts = "-o %s -x %s -V %s" % (recipefile, tmpsrcbasetree,
> args.version) +        cmd = 'recipetool create "%s" %s' % (url, cmdopts)
> +        stdout, _ = exec_build_env_command(config.init_path, basepath, cmd)
> +    except bb.process.ExecutionError as e:
> +        raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command,
> e.stdout)) +
> +    tmpsrctree = os.path.join(tmpsrcbasetree, pn + '-' + args.version)
> +
> +    try:
> +        # branch from devtool-base (original source code without patches)
> before copying new source code +        bb.process.run('git checkout -b %s
> devtool-base' % devbranch, cwd=srctree) +        bb.process.run('git tag -f
> devtool-base_%s' % args.version, cwd=srctree) +
> +        # Copy tmpsrctree  into srctree
> +        src_files = standard._ls_tree(tmpsrctree)
> +        for path in src_files:
> +            tgt_dir = os.path.join(srctree, os.path.dirname(path))
> +            bb.utils.mkdirhier(tgt_dir)
> +            tgt_path = os.path.join(srctree, path)
> +            os.rename(os.path.join(tmpsrctree, path), tgt_path)
> +
> +        # Track modified and untracked files
> +        (stdout,_) = bb.process.run('git ls-files --modified --others
> --exclude-standard', cwd=srctree) +        add_files = stdout.splitlines()
> +        for add_file in add_files:
> +            bb.process.run('git add "%s"' % add_file, cwd=srctree)
> +
> +        if len(add_files):
> +            bb.process.run('git commit -q -m "Initial commit from upstream
> at version %s"' % args.version, cwd=srctree) +        (stdout, _) =
> bb.process.run('git rev-parse HEAD', cwd=srctree) +        initial_rev =
> stdout.rstrip()
> +
> +        if args.no_patch:
> +            patches = oe.recipeutils.get_recipe_patches(crd)
> +            if len(patches):
> +                logger.warn('By user choice, the following (%s_%s) patches
> will NOT be applied into' %(pn,pv)) +                for patch in patches:
> +                    logger.warn("\t%s" % os.path.basename(patch))
> +        else:
> +            try:
> +                bb.process.run('git rebase devtool-patched', cwd=srctree)
> +            except bb.process.ExecutionError as e:
> +                # this is the only case where we do not propagate the error
> +                # user will have the change to correct merges
> +                logger.error('Command \'%s\' failed:\n%s' % (e.command,
> e.stdout)) 
> +            bb.process.run('git tag -f devtool-patched_%s' %
> args.version, cwd=srctree) 

Could you use 'devtool-patched-%s' so we avoid mixing - and _ here ?


> +    except bb.process.ExecutionError as e:
> +        raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command,
> e.stdout)) +    finally:
> +        if args.keep_temp:
> +            logger.info('Preserving temporary directory %s' %
> tmpsrcbasetree) +        else:
> +            shutil.rmtree(tmpsrcbasetree)
> +    return initial_rev, recipefile
> +
> +def _create_upgrade_recipe(args, config, recipetmp, config_data, d):
> +    """Creates the new recipe under workspace
> +
> +    Returns the full path on the new created recipe
> +    """
> +    import oe.recipeutils
> +
> +    def _get_checksums(recipefile):
> +        import re
> +        checksums = {}
> +        with open(recipefile) as rf:
> +            for line in rf:
> +                for cs in ['md5sum', 'sha256sum']:
> +                    m = re.match("^SRC_URI\[%s\].*=.*\"(.*)\"" % cs, line)
> +                    if m:
> +                        checksums[cs] = m.group(1)
> +        return checksums
> +
> +    def _replace_checksums(recipefile, checksums):
> +        import re
> +        with open(recipefile + ".tmp", "w+") as tmprf:
> +            with open(recipefile) as rf:
> +                for line in rf:
> +                    m = None
> +                    for cs in ['md5sum', 'sha256sum']:
> +                        m = re.match("^SRC_URI\[%s\].*=.*\"(.*)\"" % cs,
> line) +                        if m:
> +                            if cs in checksums:
> +                                oldcheck = m.group(1)
> +                                newcheck = checksums[cs]
> +                                line = line.replace(oldcheck, newcheck)
> +                            break
> +                    tmprf.write(line)
> +
> +        os.rename(recipefile + ".tmp", recipefile)

We already have code for editing recipes like this in recipeutils (and in 
lib/bb/utils as well) - in fact I see you're using patch_recipe() later on, so 
why do it by hand here?

> +    def _rename_patch_dirs(recipefolder, oldpv, newpv):
> +        for root, dirs, files in os.walk(recipefolder):
> +            for olddir in dirs:
> +                if olddir.find(oldpv) != -1:
> +                    newdir = olddir.replace(oldpv, newpv)
> +                    bb.process.run('mv %s %s' % (olddir, newdir))
> +
> +    def _remove_patch_dirs(recipefolder):
> +        for root, dirs, files in os.walk(recipefolder):
> +            for d in dirs:
> +                shutil.rmtree(os.path.join(root,d))
> +
> +    def _recipe_contains(recipefile, var):
> +        import re
> +        found = False
> +        with open(recipefile) as rf:
> +            for line in rf:
> +                if re.match("^%s.*=.*" % var, line):
> +                    found = True
> +                    break
> +        return found
> +
> +    if not os.path.exists(recipetmp):
> +        raise DevtoolError("Temporal recipe %s does not exist" % recipetmp)
> +
> +    # Copy current recipe into workspace
> +    pn = d.getVar('PN', True)
> +    recipepath = os.path.join(config.workspace_path, 'recipes', pn)
> +    oe.recipeutils.copy_recipe_files(d, recipepath)
> +
> +    # Rename recipe
> +    pv = d.getVar('PV', True)
> +    recipename = "%s_%s.bb" % (pn, pv)
> +    newrecipename = "%s_%s.bb" % (pn, args.version)
> +    if os.path.isfile(os.path.join(recipepath, recipename)):
> +        bb.process.run('mv %s %s' % (recipename, newrecipename),
> cwd=recipepath) 
> +    else:
> +        # Check if it is a git recipe
> +        recipename = newrecipename = "%s_git.bb" % pn
> +        if not os.path.isfile(os.path.join(recipepath, recipename)):
> +            raise DevtoolError("Original recipe not found on workspace")
> +
> +    # Rename folders
> +    _rename_patch_dirs(recipepath, pv, args.version)
> +
> +    # Update PV, just in case it is present
> +    if _recipe_contains(os.path.join(recipepath, newrecipename), 'PV'):
> +        oe.recipeutils.patch_recipe(d, os.path.join(recipepath,
> newrecipename), {'PV':args.version}) +
> +    # Update checksums
> +    checksums = _get_checksums(os.path.join(recipepath, recipetmp))
> +    _replace_checksums(os.path.join(recipepath, newrecipename), checksums)
> +
> +    # Remove recipe created by the recipe-tool
> +    bb.process.run('rm %s' % recipetmp)
> +
> +    return os.path.join(recipepath,newrecipename)
> +
> +def upgrade(args, config, basepath, workspace):
> +    """Entry point for the devtool 'upgrade' subcommand"""
> +    import bb
> +    import oe.recipeutils
> +
> +    if args.recipename in workspace:
> +        raise DevtoolError("recipe %s is already in your workspace" %
> +                            args.recipename)
> +    if not args.version:
> +        raise DevtoolError("Provide a version through the parameter
> --version/-V") +
> +    reason = oe.recipeutils.validate_pn(args.recipename)
> +    if reason:
> +        raise DevtoolError(reason)
> +
> +    tinfoil = setup_tinfoil()
> +
> +    rd = standard._parse_recipe(config, tinfoil, args.recipename, True)
> +    if not rd:
> +        return 1
> +
> +    pn = rd.getVar('PV', True)
> +    if pn == args.version:
> +        raise DevtoolError("Current and upgrade versions are the same %s" %
> pn) +
> +    srctree = os.path.abspath(args.srctree)
> +
> +    try:
> +        # Extract source from current recipe
> +        initial_rev_base = standard._extract_source(srctree, False,
> args.branch, rd) +
> +        # We need to shutdown tinfoil temporally because recipetool will be
> used 
> +        tinfoil.shutdown()
> +
> +        # Extract new source code
> +        branch_upgrade = "%s_%s" % (args.branch, args.version)
> +        initial_rev_upgrade, recipe_file_tmp =
> _extract_upgrade_source(args, branch_upgrade, config, basepath, rd) +      
>  # Start again tinfoil
> +        tinfoil = setup_tinfoil()
> +
> +        recipe_file = _create_upgrade_recipe(args, config, recipe_file_tmp,
> tinfoil.config_data, rd) +
> +    except DevtoolError as e:
> +        logger.error(e)
> +        # remove workspace recipe and srctree
> +        pn = rd.getVar('PN', True)
> +        recipespath = os.path.join(config.workspace_path, 'recipes')
> +        recipepath = os.path.join(recipespath, pn)
> +        if os.path.exists(recipepath):
> +            shutil.rmtree(recipepath)
> +            if not len(os.listdir(recipespath)):
> +                os.rmdir(recipespath)
> +        # remove srctree
> +        if os.path.exists(srctree):
> +            shutil.rmtree(srctree)
> +        raise DevtoolError(e)
> +
> +    appendpath = os.path.join(config.workspace_path, 'appends')
> +    if not os.path.exists(appendpath):
> +        os.makedirs(appendpath)
> +
> +    recipe_file_base = os.path.basename(os.path.splitext(recipe_file)[0])
> +    appendfile = os.path.join(appendpath, '%s.bbappend' % recipe_file_base)
> +    with open(appendfile, 'w') as f:
> +        f.write('inherit externalsrc\n')
> +        f.write('EXTERNALSRC = "%s"\n' % srctree)
> +        f.write('EXTERNALSRC_BUILD = "%s"\n' % srctree)

Why is this unconditionally setting EXTERNALSRC_BUILD? Surely we should be 
using the same logic as used elsewhere for this?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH v2 1/2] devtool: upgrade feature

[Leonardo Sandoval]

On 08/26/2015 07:04 PM, Paul Eggleton wrote:
> On Wednesday 26 August 2015 07:43:23
> leonardo.sandoval.gonzalez@linux.intel.com wrote:
>> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
>>
>> Upgrades a recipe to a particular version and downloads the source code
>> into srctree. User can avoid patching the source code. These are the
>> general steps of the upgrade function:
>>     * Extract current recipe source code into srctree and create branch
>>     * Extract upgrade recipe source code into srctree and rebase with
>>       previous branch. This step also creates a temporal recipe (created
>>       using recipetool), containing the correct checksums.
>>     * Creates the new recipe under the workspace
>
> OK, I haven't tested this yet but some comments just looking over the code
> below.

Ok, let me know if you find something when testing.

>
>
>> [YOCTO #7642]
>>
>> Signed-off-by: Leonardo Sandoval
>> <leonardo.sandoval.gonzalez@linux.intel.com> ---
>>   scripts/lib/devtool/standard.py |   4 +-
>>   scripts/lib/devtool/upgrade.py  | 314
>> ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 317
>> insertions(+), 1 deletion(-)
>>   create mode 100644 scripts/lib/devtool/upgrade.py
>>
>> diff --git a/scripts/lib/devtool/standard.py
>> b/scripts/lib/devtool/standard.py index ea21877..cd5a3ed 100644
>> --- a/scripts/lib/devtool/standard.py
>> +++ b/scripts/lib/devtool/standard.py
>> @@ -205,6 +205,8 @@ def extract(args, config, basepath, workspace):
>>
>>       srctree = os.path.abspath(args.srctree)
>>       initial_rev = _extract_source(srctree, args.keep_temp, args.branch, rd)
>> +    logger.info('Source tree extracted to %s' % srctree)
>> +
>>       if initial_rev:
>>           return 0
>>       else:
>> @@ -360,7 +362,6 @@ def _extract_source(srctree, keep_temp, devbranch, d):
>>                   bb.process.run('git checkout patches', cwd=srcsubdir)
>>
>>           shutil.move(srcsubdir, srctree)
>> -        logger.info('Source tree extracted to %s' % srctree)
>>       finally:
>>           bb.logger.setLevel(origlevel)
>>
>> @@ -439,6 +440,7 @@ def modify(args, config, basepath, workspace):
>>           initial_rev = _extract_source(args.srctree, False, args.branch, rd)
>> if not initial_rev:
>>               return 1
>> +        logger.info('Source tree extracted to %s' % srctree)
>>           # Get list of commits since this revision
>>           (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' %
>> initial_rev, cwd=args.srctree) commits = stdout.split()
>> diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
>> new file mode 100644
>> index 0000000..9bef984
>> --- /dev/null
>> +++ b/scripts/lib/devtool/upgrade.py
>> @@ -0,0 +1,314 @@
>> +# Development tool - upgrade command plugin
>> +#
>> +# Copyright (C) 2014-2015 Intel Corporation
>> +#
>> +# This program is free software; you can redistribute it and/or modify
>> +# it under the terms of the GNU General Public License version 2 as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License along
>> +# with this program; if not, write to the Free Software Foundation, Inc.,
>> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
>> +#
>> +# DESCRIPTION
>> +# Created a new recipe under workspace/recipes/<recipename> and place the
>> +# source code into <srctree>.
>> +# The upgrade feature executes the following steps:
>> +#     * Extract current recipe source code into srctree and create branch
>> +#     * Extract upgrade recipe source code into srctree and rebase with
>> +#       previous branch. This step also creates a temporal recipe (created
>> +#       using recipetool), containing the correct checksums.
>> +#     * Creates the new recipe under the workspace
>> +"""Devtool upgrade plugin"""
>> +
>> +import os
>> +import sys
>> +import re
>> +import shutil
>> +import tempfile
>> +import logging
>> +import argparse
>> +import scriptutils
>> +import errno
>> +from devtool import standard
>> +from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
>> +
>> +logger = logging.getLogger('devtool')
>> +
>> +def plugin_init(pluginlist):
>> +    """Plugin initialization"""
>> +    pass
>> +
>> +def _extract_upgrade_source(args, devbranch, config, basepath, d,
>> recipepostfix='tmp'): +    """Extract sources of a recipe with PV given on
>> args.version
>> +
>> +    On the target source tree folder, a new branch
>> (<devbranch>_<args.version>) +    and tag (<devbranch>-base_<args.version>)
>> will be created. In case patches +    are applied, another tag is created
>> (<devbranch>-patched_<args.version>). +
>> +    Returns: 1) The (git) initial revision ID
>> +             2) The full path of a temporal recipe containing the correct
>> checksums +    """
>> +    import oe.recipeutils
>> +
>> +    initial_rev = None
>> +    srctree = os.path.abspath(args.srctree)
>> +
>> +    pn = d.getVar('PN', True)
>> +
>> +    standard._check_compatible_recipe(pn, d)
>> +
>> +    recipepath = os.path.join(config.workspace_path, 'recipes', pn)
>> +    bb.utils.mkdirhier(recipepath)
>> +    recipefile = os.path.join(recipepath, "%s-%s.bb" % (pn,recipepostfix))
>> +    tmpsrcbasetree = tempfile.mkdtemp(prefix='devtool')
>> +
>> +    # Change PV and get URL
>> +    crd = d.createCopy()
>> +    pv = d.getVar('PV', True)
>> +    crd.setVar('PV', args.version)
>> +    src_uri = crd.getVar('SRC_URI', True)
>> +    if src_uri:
>> +        url = src_uri.split()[0]
>> +
>> +    # Generate recipe and fetch new source
>> +    try:
>> +        cmdopts = "-o %s -x %s -V %s" % (recipefile, tmpsrcbasetree,
>> args.version) +        cmd = 'recipetool create "%s" %s' % (url, cmdopts)
>> +        stdout, _ = exec_build_env_command(config.init_path, basepath, cmd)
>> +    except bb.process.ExecutionError as e:
>> +        raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command,
>> e.stdout)) +
>> +    tmpsrctree = os.path.join(tmpsrcbasetree, pn + '-' + args.version)
>> +
>> +    try:
>> +        # branch from devtool-base (original source code without patches)
>> before copying new source code +        bb.process.run('git checkout -b %s
>> devtool-base' % devbranch, cwd=srctree) +        bb.process.run('git tag -f
>> devtool-base_%s' % args.version, cwd=srctree) +
>> +        # Copy tmpsrctree  into srctree
>> +        src_files = standard._ls_tree(tmpsrctree)
>> +        for path in src_files:
>> +            tgt_dir = os.path.join(srctree, os.path.dirname(path))
>> +            bb.utils.mkdirhier(tgt_dir)
>> +            tgt_path = os.path.join(srctree, path)
>> +            os.rename(os.path.join(tmpsrctree, path), tgt_path)
>> +
>> +        # Track modified and untracked files
>> +        (stdout,_) = bb.process.run('git ls-files --modified --others
>> --exclude-standard', cwd=srctree) +        add_files = stdout.splitlines()
>> +        for add_file in add_files:
>> +            bb.process.run('git add "%s"' % add_file, cwd=srctree)
>> +
>> +        if len(add_files):
>> +            bb.process.run('git commit -q -m "Initial commit from upstream
>> at version %s"' % args.version, cwd=srctree) +        (stdout, _) =
>> bb.process.run('git rev-parse HEAD', cwd=srctree) +        initial_rev =
>> stdout.rstrip()
>> +
>> +        if args.no_patch:
>> +            patches = oe.recipeutils.get_recipe_patches(crd)
>> +            if len(patches):
>> +                logger.warn('By user choice, the following (%s_%s) patches
>> will NOT be applied into' %(pn,pv)) +                for patch in patches:
>> +                    logger.warn("\t%s" % os.path.basename(patch))
>> +        else:
>> +            try:
>> +                bb.process.run('git rebase devtool-patched', cwd=srctree)
>> +            except bb.process.ExecutionError as e:
>> +                # this is the only case where we do not propagate the error
>> +                # user will have the change to correct merges
>> +                logger.error('Command \'%s\' failed:\n%s' % (e.command,
>> e.stdout))
>> +            bb.process.run('git tag -f devtool-patched_%s' %
>> args.version, cwd=srctree)
>
> Could you use 'devtool-patched-%s' so we avoid mixing - and _ here ?
>

Sounds good. The original intention was to have a similar syntax as 
recipes, where PN and PV is divided by '_'.

>
>> +    except bb.process.ExecutionError as e:
>> +        raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command,
>> e.stdout)) +    finally:
>> +        if args.keep_temp:
>> +            logger.info('Preserving temporary directory %s' %
>> tmpsrcbasetree) +        else:
>> +            shutil.rmtree(tmpsrcbasetree)
>> +    return initial_rev, recipefile
>> +
>> +def _create_upgrade_recipe(args, config, recipetmp, config_data, d):
>> +    """Creates the new recipe under workspace
>> +
>> +    Returns the full path on the new created recipe
>> +    """
>> +    import oe.recipeutils
>> +
>> +    def _get_checksums(recipefile):
>> +        import re
>> +        checksums = {}
>> +        with open(recipefile) as rf:
>> +            for line in rf:
>> +                for cs in ['md5sum', 'sha256sum']:
>> +                    m = re.match("^SRC_URI\[%s\].*=.*\"(.*)\"" % cs, line)
>> +                    if m:
>> +                        checksums[cs] = m.group(1)
>> +        return checksums
>> +
>> +    def _replace_checksums(recipefile, checksums):
>> +        import re
>> +        with open(recipefile + ".tmp", "w+") as tmprf:
>> +            with open(recipefile) as rf:
>> +                for line in rf:
>> +                    m = None
>> +                    for cs in ['md5sum', 'sha256sum']:
>> +                        m = re.match("^SRC_URI\[%s\].*=.*\"(.*)\"" % cs,
>> line) +                        if m:
>> +                            if cs in checksums:
>> +                                oldcheck = m.group(1)
>> +                                newcheck = checksums[cs]
>> +                                line = line.replace(oldcheck, newcheck)
>> +                            break
>> +                    tmprf.write(line)
>> +
>> +        os.rename(recipefile + ".tmp", recipefile)
>
> We already have code for editing recipes like this in recipeutils (and in
> lib/bb/utils as well) - in fact I see you're using patch_recipe() later on, so
> why do it by hand here?
>

can 'oe.recipeutils.patch_recipe' patch variable's flags? I did not try 
but looking at the function definition, seems like it does not.

>> +    def _rename_patch_dirs(recipefolder, oldpv, newpv):
>> +        for root, dirs, files in os.walk(recipefolder):
>> +            for olddir in dirs:
>> +                if olddir.find(oldpv) != -1:
>> +                    newdir = olddir.replace(oldpv, newpv)
>> +                    bb.process.run('mv %s %s' % (olddir, newdir))
>> +
>> +    def _remove_patch_dirs(recipefolder):
>> +        for root, dirs, files in os.walk(recipefolder):
>> +            for d in dirs:
>> +                shutil.rmtree(os.path.join(root,d))
>> +
>> +    def _recipe_contains(recipefile, var):
>> +        import re
>> +        found = False
>> +        with open(recipefile) as rf:
>> +            for line in rf:
>> +                if re.match("^%s.*=.*" % var, line):
>> +                    found = True
>> +                    break
>> +        return found
>> +
>> +    if not os.path.exists(recipetmp):
>> +        raise DevtoolError("Temporal recipe %s does not exist" % recipetmp)
>> +
>> +    # Copy current recipe into workspace
>> +    pn = d.getVar('PN', True)
>> +    recipepath = os.path.join(config.workspace_path, 'recipes', pn)
>> +    oe.recipeutils.copy_recipe_files(d, recipepath)
>> +
>> +    # Rename recipe
>> +    pv = d.getVar('PV', True)
>> +    recipename = "%s_%s.bb" % (pn, pv)
>> +    newrecipename = "%s_%s.bb" % (pn, args.version)
>> +    if os.path.isfile(os.path.join(recipepath, recipename)):
>> +        bb.process.run('mv %s %s' % (recipename, newrecipename),
>> cwd=recipepath)
>> +    else:
>> +        # Check if it is a git recipe
>> +        recipename = newrecipename = "%s_git.bb" % pn
>> +        if not os.path.isfile(os.path.join(recipepath, recipename)):
>> +            raise DevtoolError("Original recipe not found on workspace")
>> +
>> +    # Rename folders
>> +    _rename_patch_dirs(recipepath, pv, args.version)
>> +
>> +    # Update PV, just in case it is present
>> +    if _recipe_contains(os.path.join(recipepath, newrecipename), 'PV'):
>> +        oe.recipeutils.patch_recipe(d, os.path.join(recipepath,
>> newrecipename), {'PV':args.version}) +
>> +    # Update checksums
>> +    checksums = _get_checksums(os.path.join(recipepath, recipetmp))
>> +    _replace_checksums(os.path.join(recipepath, newrecipename), checksums)
>> +
>> +    # Remove recipe created by the recipe-tool
>> +    bb.process.run('rm %s' % recipetmp)
>> +
>> +    return os.path.join(recipepath,newrecipename)
>> +
>> +def upgrade(args, config, basepath, workspace):
>> +    """Entry point for the devtool 'upgrade' subcommand"""
>> +    import bb
>> +    import oe.recipeutils
>> +
>> +    if args.recipename in workspace:
>> +        raise DevtoolError("recipe %s is already in your workspace" %
>> +                            args.recipename)
>> +    if not args.version:
>> +        raise DevtoolError("Provide a version through the parameter
>> --version/-V") +
>> +    reason = oe.recipeutils.validate_pn(args.recipename)
>> +    if reason:
>> +        raise DevtoolError(reason)
>> +
>> +    tinfoil = setup_tinfoil()
>> +
>> +    rd = standard._parse_recipe(config, tinfoil, args.recipename, True)
>> +    if not rd:
>> +        return 1
>> +
>> +    pn = rd.getVar('PV', True)
>> +    if pn == args.version:
>> +        raise DevtoolError("Current and upgrade versions are the same %s" %
>> pn) +
>> +    srctree = os.path.abspath(args.srctree)
>> +
>> +    try:
>> +        # Extract source from current recipe
>> +        initial_rev_base = standard._extract_source(srctree, False,
>> args.branch, rd) +
>> +        # We need to shutdown tinfoil temporally because recipetool will be
>> used
>> +        tinfoil.shutdown()
>> +
>> +        # Extract new source code
>> +        branch_upgrade = "%s_%s" % (args.branch, args.version)
>> +        initial_rev_upgrade, recipe_file_tmp =
>> _extract_upgrade_source(args, branch_upgrade, config, basepath, rd) +
>>   # Start again tinfoil
>> +        tinfoil = setup_tinfoil()
>> +
>> +        recipe_file = _create_upgrade_recipe(args, config, recipe_file_tmp,
>> tinfoil.config_data, rd) +
>> +    except DevtoolError as e:
>> +        logger.error(e)
>> +        # remove workspace recipe and srctree
>> +        pn = rd.getVar('PN', True)
>> +        recipespath = os.path.join(config.workspace_path, 'recipes')
>> +        recipepath = os.path.join(recipespath, pn)
>> +        if os.path.exists(recipepath):
>> +            shutil.rmtree(recipepath)
>> +            if not len(os.listdir(recipespath)):
>> +                os.rmdir(recipespath)
>> +        # remove srctree
>> +        if os.path.exists(srctree):
>> +            shutil.rmtree(srctree)
>> +        raise DevtoolError(e)
>> +
>> +    appendpath = os.path.join(config.workspace_path, 'appends')
>> +    if not os.path.exists(appendpath):
>> +        os.makedirs(appendpath)
>> +
>> +    recipe_file_base = os.path.basename(os.path.splitext(recipe_file)[0])
>> +    appendfile = os.path.join(appendpath, '%s.bbappend' % recipe_file_base)
>> +    with open(appendfile, 'w') as f:
>> +        f.write('inherit externalsrc\n')
>> +        f.write('EXTERNALSRC = "%s"\n' % srctree)
>> +        f.write('EXTERNALSRC_BUILD = "%s"\n' % srctree)
>
> Why is this unconditionally setting EXTERNALSRC_BUILD? Surely we should be
> using the same logic as used elsewhere for this?

Good point. I will add the required parameter ('--same-dir') and write 
EXTERNALSRC_BUILD if present.

>
> Cheers,
> Paul
>

Re: [PATCH v2 1/2] devtool: upgrade feature

[Otavio Salvador]

On Thu, Aug 27, 2015 at 10:24 AM, Leonardo Sandoval
<leonardo.sandoval.gonzalez@linux.intel.com> wrote:
> On 08/26/2015 07:04 PM, Paul Eggleton wrote:
>> We already have code for editing recipes like this in recipeutils (and in
>> lib/bb/utils as well) - in fact I see you're using patch_recipe() later
>> on, so
>> why do it by hand here?
>>
>
> can 'oe.recipeutils.patch_recipe' patch variable's flags? I did not try but
> looking at the function definition, seems like it does not.

So add this feature there if you need it.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH v2 1/2] devtool: upgrade feature

[Leonardo Sandoval]

On 08/27/2015 08:29 AM, Otavio Salvador wrote:
> On Thu, Aug 27, 2015 at 10:24 AM, Leonardo Sandoval
> <leonardo.sandoval.gonzalez@linux.intel.com> wrote:
>> On 08/26/2015 07:04 PM, Paul Eggleton wrote:
>>> We already have code for editing recipes like this in recipeutils (and in
>>> lib/bb/utils as well) - in fact I see you're using patch_recipe() later
>>> on, so
>>> why do it by hand here?
>>>
>>
>> can 'oe.recipeutils.patch_recipe' patch variable's flags? I did not try but
>> looking at the function definition, seems like it does not.
>
> So add this feature there if you need it.
Hi Otavio!

you are completely right. I gave it some hours today without much luck. 
What I have right now is that it appends the new checksums at the end of 
the recipe instead of replacing them. This works fine because the last 
assignment is the one remaining after the parsing stage, but not really 
elegant to have duplicate values in a single recipe.

Paul,
For this particular feature, I believe we should focus on bitbake's 
edit_metadata function instead of recipeutils's patch_recipe. In 
general, we should refactor many things on the devtool code, now that 
many stuff is just in a single file (standard.py). This seems to me like 
work for me targeting 2.1 release.

>