[PATCH 00/14] devtool upgrade fixes

[PATCH 00/14] devtool upgrade fixes

[Paul Eggleton]

I had occasion to look at devtool upgrade recently and found quite a
number of issues. Fix those, improve the tests and fix a couple of minor
things in devtool add and devtool modify as well.


The following changes since commit 205b446f3fc4a9885179a66a8dab9d81bcc63dca:

  uclibc: Do not use immediate expansion operator (2016-02-22 20:42:34 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/devtool15-oe
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool15-oe

Paul Eggleton (14):
  devtool: add: tweak auto-determining name failure message
  devtool: modify: fix source tree default name when mapping virtuals
  devtool: upgrade: make source tree path optional
  devtool: upgrade: drop superfluous call to validate_pn
  devtool: upgrade: fix version argument checking
  devtool: upgrade: fix moving version-specific files directory
  devtool: upgrade: fix renaming of recipe if PV is not in name
  devtool: upgrade: fix constructing new branch from tarball releases
  devtool: upgrade: fix several issues with extraction of new source
  devtool: upgrade: eliminate unnecessary datastore copy
  devtool: upgrade: drop PR on upgrade
  devtool: upgrade: print new recipe name
  oe-selftest: devtool: rework devtool upgrade test
  oe-selftest: devtool: add an additional test for devtool upgrade

 ...01-Add-a-note-line-to-the-quick-reference.patch |  25 ++++++
 .../devtool/devtool-upgrade-test1_1.5.3.bb         |  16 ++++
 .../devtool-upgrade-test1_1.5.3.bb.upgraded        |  14 ++++
 .../devtool/devtool-upgrade-test2_git.bb           |  18 ++++
 .../devtool/devtool-upgrade-test2_git.bb.upgraded  |  17 ++++
 .../recipes-test/devtool/devtool-upgrade_0.1.bb    |  25 ------
 ...-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch |  27 ------
 .../devtool/files/devtool-upgrade-0.1.tar.gz       | Bin 411 -> 0 bytes
 .../devtool/files/devtool-upgrade-0.2.tar.gz       | Bin 411 -> 0 bytes
 meta/lib/oeqa/selftest/devtool.py                  |  59 +++++++++++--
 scripts/lib/devtool/standard.py                    |  37 +++++----
 scripts/lib/devtool/upgrade.py                     |  91 +++++++++++++--------
 12 files changed, 216 insertions(+), 113 deletions(-)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test1-1.5.3/0001-Add-a-note-line-to-the-quick-reference.patch
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test1_1.5.3.bb
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test1_1.5.3.bb.upgraded
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test2_git.bb
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test2_git.bb.upgraded
 delete mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade_0.1.bb
 delete mode 100644 meta-selftest/recipes-test/devtool/files/0001-helloword.c-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch
 delete mode 100644 meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.1.tar.gz
 delete mode 100644 meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.2.tar.gz

-- 
2.5.0

[PATCH 01/14] devtool: add: tweak auto-determining name failure message

[Paul Eggleton]

As suggested by Khem Raj.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index daf18c9..bcf6520 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -150,7 +150,7 @@ def add(args, config, basepath, workspace):
             stdout, _ = exec_build_env_command(config.init_path, basepath, 'recipetool --color=%s create -o %s "%s" %s' % (color, tempdir, source, extracmdopts))
         except bb.process.ExecutionError as e:
             if e.exitcode == 15:
-                raise DevtoolError('Unable to auto-determine recipe name from source tree, please specify it on the command line')
+                raise DevtoolError('Could not auto-determine recipe name, please specify it on the command line')
             else:
                 raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
 
-- 
2.5.0

[PATCH 02/14] devtool: modify: fix source tree default name when mapping virtuals

[Paul Eggleton]

If you for example ran devtool modify virtual/libusb0 without specifying
a source tree path, the default was <workspace>/sources/virtual/libusb0
which isn't correct - it should be using the mapped name i.e.
libusb-compat (in the default OE-Core configuration). Reorder some of
the code to ensure that the mapped name is used.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 35 ++++++++++++++++++-----------------
 1 file changed, 18 insertions(+), 17 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index bcf6520..acd21c7 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -425,10 +425,12 @@ class PatchTaskExecutor(BbTaskExecutor):
                 bb.process.run('git add .; git commit -a -m "Committing changes from %s\n\n%s"' % (func, GitApplyTree.ignore_commit_prefix + ' - from %s' % func), cwd=srcsubdir)
 
 
-def _prep_extract_operation(config, basepath, recipename):
+def _prep_extract_operation(config, basepath, recipename, tinfoil=None):
     """HACK: Ugly workaround for making sure that requirements are met when
        trying to extract a package. Returns the tinfoil instance to be used."""
-    tinfoil = setup_tinfoil(basepath=basepath)
+    if not tinfoil:
+        tinfoil = setup_tinfoil(basepath=basepath)
+
     rd = parse_recipe(config, tinfoil, recipename, True)
     if not rd:
         return None
@@ -680,33 +682,32 @@ def modify(args, config, basepath, workspace):
         raise DevtoolError("recipe %s is already in your workspace" %
                            args.recipename)
 
+    tinfoil = setup_tinfoil(basepath=basepath)
+    rd = parse_recipe(config, tinfoil, args.recipename, True)
+    if not rd:
+        return 1
+
+    pn = rd.getVar('PN', True)
+    if pn != args.recipename:
+        logger.info('Mapping %s to %s' % (args.recipename, pn))
+    if pn in workspace:
+        raise DevtoolError("recipe %s is already in your workspace" %
+                           pn)
+
     if args.srctree:
         srctree = os.path.abspath(args.srctree)
     else:
-        srctree = get_default_srctree(config, args.recipename)
+        srctree = get_default_srctree(config, pn)
 
     if args.no_extract and not os.path.isdir(srctree):
         raise DevtoolError("--no-extract specified and source path %s does "
                            "not exist or is not a directory" %
                            srctree)
     if not args.no_extract:
-        tinfoil = _prep_extract_operation(config, basepath, args.recipename)
+        tinfoil = _prep_extract_operation(config, basepath, pn, tinfoil)
         if not tinfoil:
             # Error already shown
             return 1
-    else:
-        tinfoil = setup_tinfoil(basepath=basepath)
-
-    rd = parse_recipe(config, tinfoil, args.recipename, True)
-    if not rd:
-        return 1
-
-    pn = rd.getVar('PN', True)
-    if pn != args.recipename:
-        logger.info('Mapping %s to %s' % (args.recipename, pn))
-    if pn in workspace:
-        raise DevtoolError("recipe %s is already in your workspace" %
-                           pn)
 
     recipefile = rd.getVar('FILE', True)
     appendfile = recipe_to_append(recipefile, config, args.wildcard)
-- 
2.5.0

[PATCH 03/14] devtool: upgrade: make source tree path optional

[Paul Eggleton]

Make devtool upgrade consistent with devtool add/modify in defaulting to
sources/<recipename> under the workspace if no source tree path is
specified.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/upgrade.py | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 0e53c82..a446c55 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -313,36 +313,44 @@ def upgrade(args, config, basepath, workspace):
     if pn in workspace:
         raise DevtoolError("recipe %s is already in your workspace" % pn)
 
+    if args.srctree:
+        srctree = os.path.abspath(args.srctree)
+    else:
+        srctree = standard.get_default_srctree(config, pn)
+
     standard._check_compatible_recipe(pn, rd)
     if rd.getVar('PV', True) == args.version and rd.getVar('SRCREV', True) == args.srcrev:
         raise DevtoolError("Current and upgrade versions are the same version" % version)
 
     rf = None
     try:
-        rev1 = standard._extract_source(args.srctree, False, 'devtool-orig', False, rd)
-        rev2, md5, sha256 = _extract_new_source(args.version, args.srctree, args.no_patch,
+        rev1 = standard._extract_source(srctree, False, 'devtool-orig', False, rd)
+        rev2, md5, sha256 = _extract_new_source(args.version, srctree, args.no_patch,
                                                 args.srcrev, args.branch, args.keep_temp,
                                                 tinfoil, rd)
         rf = _create_new_recipe(args.version, md5, sha256, args.srcrev, args.srcbranch, config.workspace_path, tinfoil, rd)
     except bb.process.CmdError as e:
-        _upgrade_error(e, rf, args.srctree)
+        _upgrade_error(e, rf, srctree)
     except DevtoolError as e:
-        _upgrade_error(e, rf, args.srctree)
+        _upgrade_error(e, rf, srctree)
     standard._add_md5(config, pn, os.path.dirname(rf))
 
-    af = _write_append(rf, args.srctree, args.same_dir, args.no_same_dir, rev2,
+    af = _write_append(rf, srctree, args.same_dir, args.no_same_dir, rev2,
                        config.workspace_path, rd)
     standard._add_md5(config, pn, af)
-    logger.info('Upgraded source extracted to %s' % args.srctree)
+    logger.info('Upgraded source extracted to %s' % srctree)
     return 0
 
 def register_commands(subparsers, context):
     """Register devtool subcommands from this plugin"""
+
+    defsrctree = standard.get_default_srctree(context.config)
+
     parser_upgrade = subparsers.add_parser('upgrade', help='Upgrade an existing recipe',
                                            description='Upgrades an existing recipe to a new upstream version. Puts the upgraded recipe file into the workspace along with any associated files, and extracts the source tree to a specified location (in case patches need rebasing or adding to as a result of the upgrade).',
                                            group='starting')
     parser_upgrade.add_argument('recipename', help='Name of recipe to upgrade (just name - no version, path or extension)')
-    parser_upgrade.add_argument('srctree', help='Path to where to extract the source tree')
+    parser_upgrade.add_argument('srctree',  nargs='?', help='Path to where to extract the source tree. If not specified, a subdirectory of %s will be used.' % defsrctree)
     parser_upgrade.add_argument('--version', '-V', help='Version to upgrade to (PV)')
     parser_upgrade.add_argument('--srcrev', '-S', help='Source revision to upgrade to (if fetching from an SCM such as git)')
     parser_upgrade.add_argument('--srcbranch', '-B', help='Branch in source repository containing the revision to use (if fetching from an SCM such as git)')
-- 
2.5.0

[PATCH 04/14] devtool: upgrade: drop superfluous call to validate_pn

[Paul Eggleton]

The recipename argument to devtool upgrade specifies an existing recipe,
so by definition the name will be valid (or it won't exist) - we don't
need to validate it ourselves, that's only needed for situations like in
devtool add where we're creating a new recipe.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/upgrade.py | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index a446c55..279a65f 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -297,12 +297,7 @@ def upgrade(args, config, basepath, workspace):
     if args.srcbranch and not args.srcrev:
         raise DevtoolError("If you specify --srcbranch/-B then you must use --srcrev/-S to specify the revision" % args.recipename)
 
-    reason = oe.recipeutils.validate_pn(args.recipename)
-    if reason:
-        raise DevtoolError(reason)
-
     tinfoil = setup_tinfoil(basepath=basepath, tracking=True)
-
     rd = parse_recipe(config, tinfoil, args.recipename, True)
     if not rd:
         return 1
-- 
2.5.0

[PATCH 05/14] devtool: upgrade: fix version argument checking

[Paul Eggleton]

For recipes that specify SRCREV, the code here wasn't quite doing the
right thing. If the recipe has a SRCREV then that needs changing on
upgrade, so ensure that the user specifies it. If it doesn't, then it'll
be "INVALID" not None since the former is the actual default, so handle
that properly as well. Additionally an unset variable was being
erroneously passed when raising the error about the version being the
same leading to a traceback, so fix that as well.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/upgrade.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 279a65f..77b48f4 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -314,8 +314,13 @@ def upgrade(args, config, basepath, workspace):
         srctree = standard.get_default_srctree(config, pn)
 
     standard._check_compatible_recipe(pn, rd)
-    if rd.getVar('PV', True) == args.version and rd.getVar('SRCREV', True) == args.srcrev:
-        raise DevtoolError("Current and upgrade versions are the same version" % version)
+    old_srcrev = rd.getVar('SRCREV', True)
+    if old_srcrev == 'INVALID':
+        old_srcrev = None
+    if old_srcrev and not args.srcrev:
+        raise DevtoolError("Recipe specifies a SRCREV value; you must specify a new one when upgrading")
+    if rd.getVar('PV', True) == args.version and old_srcrev == args.srcrev:
+        raise DevtoolError("Current and upgrade versions are the same version")
 
     rf = None
     try:
-- 
2.5.0

[PATCH 06/14] devtool: upgrade: fix moving version-specific files directory

[Paul Eggleton]

We were trying to move this from the current directory instead of the
path. Let's just use shutil.move() instead of shelling out to mv.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/upgrade.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 77b48f4..f2a93b7 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -81,7 +81,7 @@ def _rename_recipe_dirs(oldpv, newpv, path):
             if olddir.find(oldpv) != -1:
                 newdir = olddir.replace(oldpv, newpv)
                 if olddir != newdir:
-                    _run('mv %s %s' % (olddir, newdir))
+                    shutil.move(os.path.join(path, olddir), os.path.join(path, newdir))
 
 def _rename_recipe_file(bpn, oldpv, newpv, path):
     oldrecipe = "%s_%s.bb" % (bpn, oldpv)
-- 
2.5.0

[PATCH 07/14] devtool: upgrade: fix renaming of recipe if PV is not in name

[Paul Eggleton]

If the actual value of PV isn't in the name of the recipe (for example,
a git or svn recipe) there's no point trying to rename it. Additionally,
we already have the original filename, there's no need to guess it -
just pass it in.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/upgrade.py | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index f2a93b7..9a308b1 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -83,21 +83,19 @@ def _rename_recipe_dirs(oldpv, newpv, path):
                 if olddir != newdir:
                     shutil.move(os.path.join(path, olddir), os.path.join(path, newdir))
 
-def _rename_recipe_file(bpn, oldpv, newpv, path):
-    oldrecipe = "%s_%s.bb" % (bpn, oldpv)
-    newrecipe = "%s_%s.bb" % (bpn, newpv)
-    if os.path.isfile(os.path.join(path, oldrecipe)):
+def _rename_recipe_file(oldrecipe, bpn, oldpv, newpv, path):
+    oldrecipe = os.path.basename(oldrecipe)
+    if oldrecipe.endswith('_%s.bb' % oldpv):
+        newrecipe = '%s_%s.bb' % (bpn, newpv)
         if oldrecipe != newrecipe:
-            _run('mv %s %s' % (oldrecipe, newrecipe), cwd=path)
+            shutil.move(os.path.join(path, oldrecipe), os.path.join(path, newrecipe))
     else:
-        recipe = "%s_git.bb" % bpn
-        if os.path.isfile(os.path.join(path, recipe)):
-            newrecipe = recipe
+        newrecipe = oldrecipe
     return os.path.join(path, newrecipe)
 
-def _rename_recipe_files(bpn, oldpv, newpv, path):
+def _rename_recipe_files(oldrecipe, bpn, oldpv, newpv, path):
     _rename_recipe_dirs(oldpv, newpv, path)
-    return _rename_recipe_file(bpn, oldpv, newpv, path)
+    return _rename_recipe_file(oldrecipe, bpn, oldpv, newpv, path)
 
 def _write_append(rc, srctree, same_dir, no_same_dir, rev, workspace, d):
     """Writes an append file"""
@@ -243,7 +241,9 @@ def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil
     oldpv = crd.getVar('PV', True)
     if not newpv:
         newpv = oldpv
-    fullpath = _rename_recipe_files(bpn, oldpv, newpv, path)
+    origpath = rd.getVar('FILE', True)
+    fullpath = _rename_recipe_files(origpath, bpn, oldpv, newpv, path)
+    logger.debug('Upgraded %s => %s' % (origpath, fullpath))
 
     newvalues = {}
     if _recipe_contains(rd, 'PV') and newpv != oldpv:
-- 
2.5.0

[PATCH 08/14] devtool: upgrade: fix constructing new branch from tarball releases

[Paul Eggleton]

When we do an upgrade from one tarball version to another we want to:

1) Check out the old version as a new branch
2) Record the changes between the old and new versions as a commit
3) Check out the old version with patches applied
4) Rebase that onto the new branch

Where we went wrong was step #1 where instead we checked out the old
version with patches applied as the new branch, which meant the rebase
didn't do anything and any changes made by the patches to files still in
the new version were wiped out.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/upgrade.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 9a308b1..fab7b15 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -179,7 +179,7 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, branch, keep_temp, tin
         md5 = None
         sha256 = None
     else:
-        __run('git checkout -b devtool-%s' % newpv)
+        __run('git checkout devtool-base -b devtool-%s' % newpv)
 
         tmpdir = tempfile.mkdtemp(prefix='devtool')
         try:
-- 
2.5.0

[PATCH 09/14] devtool: upgrade: fix several issues with extraction of new source

[Paul Eggleton]

Fix several issues when extracting the new version source over the top
of the old one (when the recipe is not fetching from a git repo):

* Delete the old source first so we ensure files deleted in the new
  version are deleted. This also has the side-effect of fixing any
  issues where files aren't marked writeable in the old source and thus
  overwriting them failed (harfbuzz 1.1.3 contains such files).
* Fix incorrect variable name in abspath statement that made it a no-op

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/upgrade.py | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index fab7b15..fae20ef 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -188,9 +188,19 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, branch, keep_temp, tin
             raise DevtoolError(e)
 
         tmpsrctree = _get_srctree(tmpdir)
+        srctree = os.path.abspath(srctree)
+
+        # Delete all sources so we ensure no stray files are left over
+        for item in os.listdir(srctree):
+            if item in ['.git', 'oe-local-files']:
+                continue
+            itempath = os.path.join(srctree, item)
+            if os.path.isdir(itempath):
+                shutil.rmtree(itempath)
+            else:
+                os.remove(itempath)
 
-        scrtree = os.path.abspath(srctree)
-
+        # Copy in new ones
         _copy_source_code(tmpsrctree, srctree)
 
         (stdout,_) = __run('git ls-files --modified --others --exclude-standard')
-- 
2.5.0

[PATCH 10/14] devtool: upgrade: eliminate unnecessary datastore copy

[Paul Eggleton]

We aren't modifying the datastore copy here, so we don't need a copy at
all.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/upgrade.py | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index fae20ef..50c670e 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -241,14 +241,13 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, branch, keep_temp, tin
 
 def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil, rd):
     """Creates the new recipe under workspace"""
-    crd = rd.createCopy()
 
-    bpn = crd.getVar('BPN', True)
+    bpn = rd.getVar('BPN', True)
     path = os.path.join(workspace, 'recipes', bpn)
     bb.utils.mkdirhier(path)
-    oe.recipeutils.copy_recipe_files(crd, path)
+    oe.recipeutils.copy_recipe_files(rd, path)
 
-    oldpv = crd.getVar('PV', True)
+    oldpv = rd.getVar('PV', True)
     if not newpv:
         newpv = oldpv
     origpath = rd.getVar('FILE', True)
-- 
2.5.0

[PATCH 11/14] devtool: upgrade: drop PR on upgrade

[Paul Eggleton]

The PR value should be reset to the default when upgrading, so we need
to drop it from the newly created file.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/upgrade.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 50c670e..14dc14c 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -286,13 +286,14 @@ def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil
         if changed:
             newvalues['SRC_URI'] = ' '.join(new_src_uri)
 
+    newvalues['PR'] = None
+
     if md5 and sha256:
         newvalues['SRC_URI[md5sum]'] = md5
         newvalues['SRC_URI[sha256sum]'] = sha256
 
-    if newvalues:
-        rd = oe.recipeutils.parse_recipe(fullpath, None, tinfoil.config_data)
-        oe.recipeutils.patch_recipe(rd, fullpath, newvalues)
+    rd = oe.recipeutils.parse_recipe(fullpath, None, tinfoil.config_data)
+    oe.recipeutils.patch_recipe(rd, fullpath, newvalues)
 
     return fullpath
 
-- 
2.5.0

[PATCH 12/14] devtool: upgrade: print new recipe name

[Paul Eggleton]

Make this consistent with "devtool add" so that the user knows where to
find the new recipe.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/upgrade.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 14dc14c..680cbf1 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -349,6 +349,7 @@ def upgrade(args, config, basepath, workspace):
                        config.workspace_path, rd)
     standard._add_md5(config, pn, af)
     logger.info('Upgraded source extracted to %s' % srctree)
+    logger.info('New recipe is %s' % rf)
     return 0
 
 def register_commands(subparsers, context):
-- 
2.5.0

[PATCH 13/14] oe-selftest: devtool: rework devtool upgrade test

[Paul Eggleton]

* Use a more real-world test of a recipe pointing to a remote file
* The cleanup tracking / teardown commands need to be added towards the
  top, or they won't have the desired effect of cleaning up if the test
  fails.
* Check that a versioned subdirectory gets renamed to match the new
  version
* Ensure the recipe contents gets changed as we expect it to
* Check that the recipe directory is deleted by devtool reset at the end

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 ...01-Add-a-note-line-to-the-quick-reference.patch |  25 +++++++++++++++++++
 .../devtool/devtool-upgrade-test1_1.5.3.bb         |  16 ++++++++++++
 .../devtool-upgrade-test1_1.5.3.bb.upgraded        |  14 +++++++++++
 .../recipes-test/devtool/devtool-upgrade_0.1.bb    |  25 -------------------
 ...-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch |  27 ---------------------
 .../devtool/files/devtool-upgrade-0.1.tar.gz       | Bin 411 -> 0 bytes
 .../devtool/files/devtool-upgrade-0.2.tar.gz       | Bin 411 -> 0 bytes
 meta/lib/oeqa/selftest/devtool.py                  |  27 ++++++++++++++-------
 8 files changed, 73 insertions(+), 61 deletions(-)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test1-1.5.3/0001-Add-a-note-line-to-the-quick-reference.patch
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test1_1.5.3.bb
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test1_1.5.3.bb.upgraded
 delete mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade_0.1.bb
 delete mode 100644 meta-selftest/recipes-test/devtool/files/0001-helloword.c-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch
 delete mode 100644 meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.1.tar.gz
 delete mode 100644 meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.2.tar.gz

diff --git a/meta-selftest/recipes-test/devtool/devtool-upgrade-test1-1.5.3/0001-Add-a-note-line-to-the-quick-reference.patch b/meta-selftest/recipes-test/devtool/devtool-upgrade-test1-1.5.3/0001-Add-a-note-line-to-the-quick-reference.patch
new file mode 100644
index 0000000..4ea3d74
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/devtool-upgrade-test1-1.5.3/0001-Add-a-note-line-to-the-quick-reference.patch
@@ -0,0 +1,25 @@
+From 1478846ebfac690684e9c48049d08e0065f97a36 Mon Sep 17 00:00:00 2001
+From: Paul Eggleton <paul.eggleton@linux.intel.com>
+Date: Wed, 24 Feb 2016 17:43:03 +1300
+Subject: [PATCH] Add a note line to the quick reference
+
+A test patch so we have a file to move around.
+---
+ doc/quickref.1.in | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/doc/quickref.1.in b/doc/quickref.1.in
+index 389008b..226615c 100644
+--- a/doc/quickref.1.in
++++ b/doc/quickref.1.in
+@@ -560,6 +560,7 @@ you want, at runtime, to change the parameters of.
+ .P
+ If you find any other problems, please report them.
+ 
++NOTE: this is an important note.
+ 
+ .SH REPORTING BUGS
+ Report bugs in
+-- 
+2.5.0
+
diff --git a/meta-selftest/recipes-test/devtool/devtool-upgrade-test1_1.5.3.bb b/meta-selftest/recipes-test/devtool/devtool-upgrade-test1_1.5.3.bb
new file mode 100644
index 0000000..e93b0d5
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/devtool-upgrade-test1_1.5.3.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Pipe viewer test recipe for devtool upgrade test"
+LICENSE = "Artistic-2.0"
+LIC_FILES_CHKSUM = "file://doc/COPYING;md5=9c50db2589ee3ef10a9b7b2e50ce1d02"
+
+SRC_URI = "http://www.ivarch.com/programs/sources/pv-${PV}.tar.gz \
+           file://0001-Add-a-note-line-to-the-quick-reference.patch"
+
+SRC_URI[md5sum] = "9365d86bd884222b4bf1039b5a9ed1bd"
+SRC_URI[sha256sum] = "681bcca9784bf3cb2207e68236d1f68e2aa7b80f999b5750dc77dcd756e81fbc"
+
+PR = "r5"
+
+S = "${WORKDIR}/pv-${PV}"
+
+inherit autotools
+
diff --git a/meta-selftest/recipes-test/devtool/devtool-upgrade-test1_1.5.3.bb.upgraded b/meta-selftest/recipes-test/devtool/devtool-upgrade-test1_1.5.3.bb.upgraded
new file mode 100644
index 0000000..afcc4aa
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/devtool-upgrade-test1_1.5.3.bb.upgraded
@@ -0,0 +1,14 @@
+SUMMARY = "Pipe viewer test recipe for devtool upgrade test"
+LICENSE = "Artistic-2.0"
+LIC_FILES_CHKSUM = "file://doc/COPYING;md5=9c50db2589ee3ef10a9b7b2e50ce1d02"
+
+SRC_URI = "http://www.ivarch.com/programs/sources/pv-${PV}.tar.gz \
+           file://0001-Add-a-note-line-to-the-quick-reference.patch"
+
+SRC_URI[md5sum] = "062bca5ff33df1dd09472e7fc3bbe332"
+SRC_URI[sha256sum] = "9dd45391806b0ed215abee4c5ac1597d018c386fe9c1f5afd2f6bc3b07fd82c3"
+
+S = "${WORKDIR}/pv-${PV}"
+
+inherit autotools
+
diff --git a/meta-selftest/recipes-test/devtool/devtool-upgrade_0.1.bb b/meta-selftest/recipes-test/devtool/devtool-upgrade_0.1.bb
deleted file mode 100644
index 33ffc88..0000000
--- a/meta-selftest/recipes-test/devtool/devtool-upgrade_0.1.bb
+++ /dev/null
@@ -1,25 +0,0 @@
-#
-# This file was derived from the 'Hello World!' example recipe in the
-# Yocto Project Development Manual.
-#
-
-DESCRIPTION = "Simple helloworld application used to test the devtool upgrade feature"
-SECTION = "devtool"
-LICENSE = "MIT"
-LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
-PR = "r0"
-
-SRC_URI = "file://${THISDIR}/files/${P}.tar.gz \
-           file://0001-helloword.c-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch \
-           "
-
-S = "${WORKDIR}/${P}"
-
-do_compile() {
-	     ${CC} helloworld.c -o helloworld
-}
-
-do_install() {
-	     install -d ${D}${bindir}
-	     install -m 0755 helloworld ${D}${bindir}
-}
diff --git a/meta-selftest/recipes-test/devtool/files/0001-helloword.c-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch b/meta-selftest/recipes-test/devtool/files/0001-helloword.c-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch
deleted file mode 100644
index 2294a09..0000000
--- a/meta-selftest/recipes-test/devtool/files/0001-helloword.c-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch
+++ /dev/null
@@ -1,27 +0,0 @@
-From 0f37affbc6e6c71687301d99d7259f1968e57c48 Mon Sep 17 00:00:00 2001
-From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
-Date: Wed, 26 Aug 2015 12:42:23 +0000
-Subject: [PATCH] helloword.c: exit with EXIT_SUCCESS instead of a magic number
-
----
- helloworld.c | 3 ++-
- 1 file changed, 2 insertions(+), 1 deletion(-)
-
-diff --git a/helloworld.c b/helloworld.c
-index 71f2e46..54bf50b 100644
---- a/helloworld.c
-+++ b/helloworld.c
-@@ -1,8 +1,9 @@
- #include <stdio.h>
-+#include <stdlib.h>
- 
- int main(int argc, char **argv)
- {
-     printf("Hello World!\n");
- 
--    return 0;
-+    return EXIT_SUCCESS;
- }
--- 
-1.8.4.5
-
diff --git a/meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.1.tar.gz b/meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.1.tar.gz
deleted file mode 100644
index 06a1c49cd9b440bba0cc16363b60d11b206895bf..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001

literal 411
zcmV;M0c8FkiwFRs0OVBw1MSs8Z-Ous2XM}Oiq2isOzbNymALKfakpJ|X@DsSf+^rk
zmVNiNCX1Wm=7!Ojz2Bwa6WR*@r#F01t4X5bM3)};AjB|EmQ_(?gL*O5=JNsdz-fgV
z0q5({*d8hP%mF4Om}BOHL(M8cIAqms+QZakrL<W}+xu*9?n3yK%&T;s+yUq2_ZO0Z
z7X*CdQx4KKRrRF`fyV{2RmN;J+JEi2cZLqp5cc7*=dgK@LfDZ3h4ndX`oTIbTlx2;
zbci_iLYjQ!KPy!p7t6h^t}ac7{N_Fvn*XNfzw?hUOR}tZDRdTl(I4jcx&MQG{(<y;
z^IDkifs_*a|4FcL)BmXmSs(*1q#*?1i2b(xf2p2(W9T4F`EMEb9iQ^w*#Cixo&0m^
zkduVypZEXkG>@`6PRMOp#cAQq?p-&{D>7GU-n9oxPoo<W&6FnnzByU;+*g+v3vJ5J
z-Oj|mE#%R@E>}-^r#E)jWt~*D&Iuj6Z`cbELI@#*5JCtcgb+dqA%qY*jd!bMh~WSz
F005Y))ZYLA

diff --git a/meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.2.tar.gz b/meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.2.tar.gz
deleted file mode 100644
index 9b0dcf4b6c1210f2b907ea62b62288d917effc4b..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001

literal 411
zcmV;M0c8FkiwFQe0OVBw1MSs8Z-Ous0ASDjiq2is%-WWgLfm%txZ5tfG{BSu!4z~R
z%l`WwB#YDG=7!Ojz3)=mS1E<QCmlVQ^&~b?YzxYTuf-^eS7n~3!{uUXVSLD_=(NIs
zCuQA&-Ij4zbhuEOR~-%GTqz+tWZiDr!z_!^Sa_7S_u1Xtg>V^}m&rW7<5I%=SDuhm
zOW${y473ALvlZon20nwRVs$j!|J!r#3>~5&*+<7-qfTq3s#b8u>ptJqgF0Py@?XL<
zqK^F_O+NBJEld{WE53goRLF1cbD{a)^!>m5Bh2D7&0lhxMl}4x8b8;6UVVR0yDl6H
zm?vP+#rl5=BG}Y_rUKz<j|ME@Uf_%0mjA8jxi^Lm(v<%#)BTH2`QKRoxm4Tv&$X6N
z|4))c|GfTRCs~*-qnO+lWt8M}cJDYzR+70%vR-vz>@>U~;mlYv7(i#$cU~O=78b(K
zz3!wsE#$E}E>}-kw?B63u#L;5%?KMiZ&(WuLI@#*5JCtcgb+dqA%qY*jdwLBEfxSM
F003pq%-sM0

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 0361464..4d280bb 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -1095,14 +1095,18 @@ class DevtoolTests(DevtoolBase):
     def test_devtool_upgrade(self):
         # Check preconditions
         self.assertTrue(not os.path.exists(self.workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
         # Check parameters
         result = runCmd('devtool upgrade -h')
         for param in 'recipename srctree --version -V --branch -b --keep-temp --no-patch'.split():
             self.assertIn(param, result.output)
         # For the moment, we are using a real recipe.
-        recipe='devtool-upgrade'
-        version='0.2'
+        recipe = 'devtool-upgrade-test1'
+        version = '1.6.0'
+        oldrecipefile = get_bb_var('FILE', recipe)
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
         # Check that recipe is not already under devtool control
         result = runCmd('devtool status')
         self.assertNotIn(recipe, result.output)
@@ -1110,22 +1114,27 @@ class DevtoolTests(DevtoolBase):
         # we are downgrading instead of upgrading.
         result = runCmd('devtool upgrade %s %s -V %s' % (recipe, tempdir, version))
         # Check if srctree at least is populated
-        self.assertTrue(len(os.listdir(tempdir)) > 0, 'scrtree (%s) should be populated with new (%s) source code' % (tempdir, version))
-        # Check new recipe folder is present
-        self.assertTrue(os.path.exists(os.path.join(self.workspacedir,'recipes',recipe)), 'Recipe folder should exist')
+        self.assertTrue(len(os.listdir(tempdir)) > 0, 'srctree (%s) should be populated with new (%s) source code' % (tempdir, version))
+        # Check new recipe subdirectory is present
+        self.assertTrue(os.path.exists(os.path.join(self.workspacedir, 'recipes', recipe, '%s-%s' % (recipe, version))), 'Recipe folder should exist')
         # Check new recipe file is present
-        self.assertTrue(os.path.exists(os.path.join(self.workspacedir,'recipes',recipe,"%s_%s.bb" % (recipe,version))), 'Recipe folder should exist')
+        newrecipefile = os.path.join(self.workspacedir, 'recipes', recipe, '%s_%s.bb' % (recipe, version))
+        self.assertTrue(os.path.exists(newrecipefile), 'Recipe file should exist after upgrade')
         # Check devtool status and make sure recipe is present
         result = runCmd('devtool status')
         self.assertIn(recipe, result.output)
         self.assertIn(tempdir, result.output)
+        # Check recipe got changed as expected
+        with open(oldrecipefile + '.upgraded', 'r') as f:
+            desiredlines = f.readlines()
+        with open(newrecipefile, 'r') as f:
+            newlines = f.readlines()
+        self.assertEqual(desiredlines, newlines)
         # Check devtool reset recipe
         result = runCmd('devtool reset %s -n' % recipe)
         result = runCmd('devtool status')
         self.assertNotIn(recipe, result.output)
-        self.track_for_cleanup(tempdir)
-        self.track_for_cleanup(self.workspacedir)
-        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        self.assertFalse(os.path.exists(os.path.join(self.workspacedir, 'recipes', recipe)), 'Recipe directory should not exist after resetting')
 
     @testcase(1352)
     def test_devtool_layer_plugins(self):
-- 
2.5.0

[PATCH 14/14] oe-selftest: devtool: add an additional test for devtool upgrade

[Paul Eggleton]

Add a test for devtool upgrade with a recipe pointing to a git
repository, since this uses several different code paths.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 .../devtool/devtool-upgrade-test2_git.bb           | 18 +++++++++++
 .../devtool/devtool-upgrade-test2_git.bb.upgraded  | 17 ++++++++++
 meta/lib/oeqa/selftest/devtool.py                  | 36 ++++++++++++++++++++++
 3 files changed, 71 insertions(+)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test2_git.bb
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test2_git.bb.upgraded

diff --git a/meta-selftest/recipes-test/devtool/devtool-upgrade-test2_git.bb b/meta-selftest/recipes-test/devtool/devtool-upgrade-test2_git.bb
new file mode 100644
index 0000000..9bfce0a
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/devtool-upgrade-test2_git.bb
@@ -0,0 +1,18 @@
+SUMMARY = "A simple tool to wait for a specific signal over DBus"
+HOMEPAGE = "http://git.yoctoproject.org/cgit/cgit.cgi/dbus-wait"
+SECTION = "base"
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
+
+DEPENDS = "dbus"
+
+# Note: this is intentionally not the latest version in the original .bb
+SRCREV = "1a3e1343761b30750bed70e0fd688f6d3c7b3717"
+PV = "0.1+git${SRCPV}"
+PR = "r2"
+
+SRC_URI = "git://git.yoctoproject.org/dbus-wait"
+
+S = "${WORKDIR}/git"
+
+inherit autotools pkgconfig
diff --git a/meta-selftest/recipes-test/devtool/devtool-upgrade-test2_git.bb.upgraded b/meta-selftest/recipes-test/devtool/devtool-upgrade-test2_git.bb.upgraded
new file mode 100644
index 0000000..9b947ed
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/devtool-upgrade-test2_git.bb.upgraded
@@ -0,0 +1,17 @@
+SUMMARY = "A simple tool to wait for a specific signal over DBus"
+HOMEPAGE = "http://git.yoctoproject.org/cgit/cgit.cgi/dbus-wait"
+SECTION = "base"
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
+
+DEPENDS = "dbus"
+
+# Note: this is intentionally not the latest version in the original .bb
+SRCREV = "6cc6077a36fe2648a5f993fe7c16c9632f946517"
+PV = "0.1+git${SRCPV}"
+
+SRC_URI = "git://git.yoctoproject.org/dbus-wait"
+
+S = "${WORKDIR}/git"
+
+inherit autotools pkgconfig
diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 4d280bb..d41af2f 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -1136,6 +1136,42 @@ class DevtoolTests(DevtoolBase):
         self.assertNotIn(recipe, result.output)
         self.assertFalse(os.path.exists(os.path.join(self.workspacedir, 'recipes', recipe)), 'Recipe directory should not exist after resetting')
 
+    def test_devtool_upgrade_git(self):
+        # Check preconditions
+        self.assertTrue(not os.path.exists(self.workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        recipe = 'devtool-upgrade-test2'
+        commit = '6cc6077a36fe2648a5f993fe7c16c9632f946517'
+        oldrecipefile = get_bb_var('FILE', recipe)
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        # Check that recipe is not already under devtool control
+        result = runCmd('devtool status')
+        self.assertNotIn(recipe, result.output)
+        # Check upgrade
+        result = runCmd('devtool upgrade %s %s -S %s' % (recipe, tempdir, commit))
+        # Check if srctree at least is populated
+        self.assertTrue(len(os.listdir(tempdir)) > 0, 'srctree (%s) should be populated with new (%s) source code' % (tempdir, commit))
+        # Check new recipe file is present
+        newrecipefile = os.path.join(self.workspacedir, 'recipes', recipe, os.path.basename(oldrecipefile))
+        self.assertTrue(os.path.exists(newrecipefile), 'Recipe file should exist after upgrade')
+        # Check devtool status and make sure recipe is present
+        result = runCmd('devtool status')
+        self.assertIn(recipe, result.output)
+        self.assertIn(tempdir, result.output)
+        # Check recipe got changed as expected
+        with open(oldrecipefile + '.upgraded', 'r') as f:
+            desiredlines = f.readlines()
+        with open(newrecipefile, 'r') as f:
+            newlines = f.readlines()
+        self.assertEqual(desiredlines, newlines)
+        # Check devtool reset recipe
+        result = runCmd('devtool reset %s -n' % recipe)
+        result = runCmd('devtool status')
+        self.assertNotIn(recipe, result.output)
+        self.assertFalse(os.path.exists(os.path.join(self.workspacedir, 'recipes', recipe)), 'Recipe directory should not exist after resetting')
+
     @testcase(1352)
     def test_devtool_layer_plugins(self):
         """Test that devtool can use plugins from other layers.
-- 
2.5.0