[PATCH 0/9] devtool improvements

[PATCH 0/9] devtool improvements

[Paul Eggleton]

Adds "upgrade" and "package" subcommands to devtool, provides a way to
disable parallel make, improves the default behaviour of update-recipe
with git recipes, and fixes running devtool from a different directory.

This rolls up and fixes some previously posted patches; changes from
the previously posted versions:

* devtool: add upgrade feature:
  - Handle upgrading git recipes i.e. where version may not change, we
    need to be able to specify the SRCREV, md5sum/sha256sum aren't used
    and most importantly we don't want to munge the existing repository
  - Fix rebasing logic - we want to be rebasing the old patched branch
    onto the new branch, not the other way around
  - Fix branching/tagging
  - Use branch name for the new version branch not the old one
  - Ensure we delete/preserve copied in recipe files when resetting
  - Use common fetch_uri function instead of duplicating
  - Fix handling recipes that use BBCLASSEXTEND
  - Tweak some messages and command line help
  - Squash in test commits (we probably need additional tests - to be
    added later)
* devtool: Allow disabling make parallelism on build command:
  - Add a short option (-s for "single")
  - Use traditional spacing in created postfile
  - Delete postfile after building, there's not much point keeping it
  - Tweak log output
* devtool: add package plugin that lets you create package via devtool:
  - Fix config file reading
  - Make this available only within the extensible SDK (since it's not
    particularly useful outside of it)
* devtool: update-recipe: better 'auto' mode
  - Fix oe-selftest tests for this behaviour change


The following changes since commit 8402958cd2cb87b8283c8ee4e2d08e1a6717d67a:

  pseudo_1.7.3.bb: New version of pseudo (2015-09-06 15:24:28 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/devtool7
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool7

Brendan Le Foll (1):
  devtool: add package plugin that lets you create package via devtool

Leonardo Sandoval (3):
  devtool: add upgrade feature
  devtool: Create a single file for the build devtool feature
  devtool: Allow disabling make parallelism on build command

Markus Lehtonen (1):
  devtool: update-recipe: better 'auto' mode

Paul Eggleton (4):
  scriptutils: split out simple fetching function from recipetool
  devtool: improve modified file preservation to handle directory
    structures
  devtool: update-recipe: add a warning when recipe is inside the
    workspace
  devtool: fix running from a different directory

 .../recipes-test/devtool/devtool-upgrade_0.1.bb    |  25 ++
 ...-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch |  27 ++
 .../devtool/files/devtool-upgrade-0.1.tar.gz       | Bin 0 -> 411 bytes
 .../devtool/files/devtool-upgrade-0.2.tar.gz       | Bin 0 -> 411 bytes
 meta/lib/oeqa/selftest/devtool.py                  |  68 +++-
 scripts/devtool                                    |   3 +
 scripts/lib/devtool/build.py                       |  77 +++++
 scripts/lib/devtool/package.py                     |  61 ++++
 scripts/lib/devtool/standard.py                    |  94 +++---
 scripts/lib/devtool/upgrade.py                     | 354 +++++++++++++++++++++
 scripts/lib/recipetool/create.py                   |  30 +-
 scripts/lib/scriptutils.py                         |  27 ++
 12 files changed, 692 insertions(+), 74 deletions(-)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade_0.1.bb
 create mode 100644 meta-selftest/recipes-test/devtool/files/0001-helloword.c-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch
 create mode 100644 meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.1.tar.gz
 create mode 100644 meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.2.tar.gz
 create mode 100644 scripts/lib/devtool/build.py
 create mode 100644 scripts/lib/devtool/package.py
 create mode 100644 scripts/lib/devtool/upgrade.py

-- 
2.1.0

[PATCH 1/9] scriptutils: split out simple fetching function from recipetool

[Paul Eggleton]

This will now also be used by "devtool upgrade".

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 30 +-----------------------------
 scripts/lib/scriptutils.py       | 27 +++++++++++++++++++++++++++
 2 files changed, 28 insertions(+), 29 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index ec6e107..c4754db 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -58,34 +58,6 @@ class RecipeHandler():
 
 
 
-def fetch_source(uri, destdir, srcrev):
-    import bb.data
-    bb.utils.mkdirhier(destdir)
-    localdata = bb.data.createCopy(tinfoil.config_data)
-    bb.data.update_data(localdata)
-    localdata.setVar('BB_STRICT_CHECKSUM', '')
-    localdata.setVar('SRCREV', srcrev)
-    ret = (None, None)
-    olddir = os.getcwd()
-    try:
-        fetcher = bb.fetch2.Fetch([uri], localdata)
-        for u in fetcher.ud:
-            ud = fetcher.ud[u]
-            ud.ignore_checksums = True
-        fetcher.download()
-        fetcher.unpack(destdir)
-        for u in fetcher.ud:
-            ud = fetcher.ud[u]
-            if ud.method.recommends_checksum(ud):
-                md5value = bb.utils.md5_file(ud.localpath)
-                sha256value = bb.utils.sha256_file(ud.localpath)
-                ret = (md5value, sha256value)
-    except bb.fetch2.BBFetchException, e:
-        raise bb.build.FuncFailed(e)
-    finally:
-        os.chdir(olddir)
-    return ret
-
 def supports_srcrev(uri):
     localdata = bb.data.createCopy(tinfoil.config_data)
     # This is a bit sad, but if you don't have this set there can be some
@@ -123,7 +95,7 @@ def create_recipe(args):
         tempsrc = tempfile.mkdtemp(prefix='recipetool-')
         srctree = tempsrc
         logger.info('Fetching %s...' % srcuri)
-        checksums = fetch_source(args.source, srctree, srcrev)
+        checksums = scriptutils.fetch_uri(tinfoil.config_data, args.source, srctree, srcrev)
         dirlist = os.listdir(srctree)
         if 'git.indirectionsymlink' in dirlist:
             dirlist.remove('git.indirectionsymlink')
diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
index fdf4b5d..5d103a5 100644
--- a/scripts/lib/scriptutils.py
+++ b/scripts/lib/scriptutils.py
@@ -69,3 +69,30 @@ def git_convert_standalone_clone(repodir):
             # of the contents is shared
             bb.process.run('git repack -a', cwd=repodir)
             os.remove(alternatesfile)
+
+def fetch_uri(d, uri, destdir, srcrev=None):
+    """Fetch a URI to a local directory"""
+    import bb.data
+    bb.utils.mkdirhier(destdir)
+    localdata = bb.data.createCopy(d)
+    localdata.setVar('BB_STRICT_CHECKSUM', '')
+    localdata.setVar('SRCREV', srcrev)
+    ret = (None, None)
+    olddir = os.getcwd()
+    try:
+        fetcher = bb.fetch2.Fetch([uri], localdata)
+        for u in fetcher.ud:
+            ud = fetcher.ud[u]
+            ud.ignore_checksums = True
+        fetcher.download()
+        fetcher.unpack(destdir)
+        for u in fetcher.ud:
+            ud = fetcher.ud[u]
+            if ud.method.recommends_checksum(ud):
+                md5value = bb.utils.md5_file(ud.localpath)
+                sha256value = bb.utils.sha256_file(ud.localpath)
+                ret = (md5value, sha256value)
+    finally:
+        os.chdir(olddir)
+    return ret
+
-- 
2.1.0

[PATCH 2/9] devtool: improve modified file preservation to handle directory structures

[Paul Eggleton]

Allow the _add_md5() function to be called with a directory in order to
recursively add the files under it. Additionally, we need to skip
preserving empty directories (since directories aren't listed in the md5
file).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index e59fb5e..6d65d57 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -362,14 +362,23 @@ def _extract_source(srctree, keep_temp, devbranch, d):
     return initial_rev
 
 def _add_md5(config, recipename, filename):
-    """Record checksum of a recipe to the md5-file of the workspace"""
+    """Record checksum of a file (or recursively for a directory) to the md5-file of the workspace"""
     import bb.utils
-    md5 = bb.utils.md5_file(filename)
-    with open(os.path.join(config.workspace_path, '.devtool_md5'), 'a') as f:
-        f.write('%s|%s|%s\n' % (recipename, os.path.relpath(filename, config.workspace_path), md5))
+
+    def addfile(fn):
+        md5 = bb.utils.md5_file(fn)
+        with open(os.path.join(config.workspace_path, '.devtool_md5'), 'a') as f:
+            f.write('%s|%s|%s\n' % (recipename, os.path.relpath(fn, config.workspace_path), md5))
+
+    if os.path.isdir(filename):
+        for root, _, files in os.walk(os.path.dirname(filename)):
+            for f in files:
+                addfile(os.path.join(root, f))
+    else:
+        addfile(filename)
 
 def _check_preserve(config, recipename):
-    """Check if a recipe was manually changed and needs to be saved in 'attic'
+    """Check if a file was manually changed and needs to be saved in 'attic'
        directory"""
     import bb.utils
     origfile = os.path.join(config.workspace_path, '.devtool_md5')
@@ -830,10 +839,13 @@ def reset(args, config, basepath, workspace):
         preservepath = os.path.join(config.workspace_path, 'attic', pn)
         def preservedir(origdir):
             if os.path.exists(origdir):
-                for fn in os.listdir(origdir):
-                    logger.warn('Preserving %s in %s' % (fn, preservepath))
-                    bb.utils.mkdirhier(preservepath)
-                    shutil.move(os.path.join(origdir, fn), os.path.join(preservepath, fn))
+                for root, dirs, files in os.walk(origdir):
+                    for fn in files:
+                        logger.warn('Preserving %s in %s' % (fn, preservepath))
+                        bb.utils.mkdirhier(preservepath)
+                        shutil.move(os.path.join(origdir, fn), os.path.join(preservepath, fn))
+                    for dn in dirs:
+                        os.rmdir(os.path.join(root, dn))
                 os.rmdir(origdir)
 
         preservedir(os.path.join(config.workspace_path, 'recipes', pn))
-- 
2.1.0

[PATCH 3/9] devtool: add upgrade feature

[Paul Eggleton]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

Upgrades a recipe to a particular version and downloads the source code
into a folder. User can avoid patching the source code.

These are the general steps of the upgrade function:

   - Extract current recipe source code into srctree and create a branch
   - Extract upgrade recipe source code into srctree and rebase with
     previous branch. In case the rebase is not correctly applied, source
     code will not be deleted, so user correct the patches
   - Creates the new recipe under the workspace

[YOCTO #7642]

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 .../recipes-test/devtool/devtool-upgrade_0.1.bb    |  25 ++
 ...-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch |  27 ++
 .../devtool/files/devtool-upgrade-0.1.tar.gz       | Bin 0 -> 411 bytes
 .../devtool/files/devtool-upgrade-0.2.tar.gz       | Bin 0 -> 411 bytes
 meta/lib/oeqa/selftest/devtool.py                  |  36 +++
 scripts/lib/devtool/standard.py                    |   4 +-
 scripts/lib/devtool/upgrade.py                     | 354 +++++++++++++++++++++
 7 files changed, 445 insertions(+), 1 deletion(-)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade_0.1.bb
 create mode 100644 meta-selftest/recipes-test/devtool/files/0001-helloword.c-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch
 create mode 100644 meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.1.tar.gz
 create mode 100644 meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.2.tar.gz
 create mode 100644 scripts/lib/devtool/upgrade.py

diff --git a/meta-selftest/recipes-test/devtool/devtool-upgrade_0.1.bb b/meta-selftest/recipes-test/devtool/devtool-upgrade_0.1.bb
new file mode 100644
index 0000000..33ffc88
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/devtool-upgrade_0.1.bb
@@ -0,0 +1,25 @@
+#
+# This file was derived from the 'Hello World!' example recipe in the
+# Yocto Project Development Manual.
+#
+
+DESCRIPTION = "Simple helloworld application used to test the devtool upgrade feature"
+SECTION = "devtool"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
+PR = "r0"
+
+SRC_URI = "file://${THISDIR}/files/${P}.tar.gz \
+           file://0001-helloword.c-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch \
+           "
+
+S = "${WORKDIR}/${P}"
+
+do_compile() {
+	     ${CC} helloworld.c -o helloworld
+}
+
+do_install() {
+	     install -d ${D}${bindir}
+	     install -m 0755 helloworld ${D}${bindir}
+}
diff --git a/meta-selftest/recipes-test/devtool/files/0001-helloword.c-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch b/meta-selftest/recipes-test/devtool/files/0001-helloword.c-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch
new file mode 100644
index 0000000..2294a09
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/files/0001-helloword.c-exit-with-EXIT_SUCCESS-instead-of-a-magi.patch
@@ -0,0 +1,27 @@
+From 0f37affbc6e6c71687301d99d7259f1968e57c48 Mon Sep 17 00:00:00 2001
+From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
+Date: Wed, 26 Aug 2015 12:42:23 +0000
+Subject: [PATCH] helloword.c: exit with EXIT_SUCCESS instead of a magic number
+
+---
+ helloworld.c | 3 ++-
+ 1 file changed, 2 insertions(+), 1 deletion(-)
+
+diff --git a/helloworld.c b/helloworld.c
+index 71f2e46..54bf50b 100644
+--- a/helloworld.c
++++ b/helloworld.c
+@@ -1,8 +1,9 @@
+ #include <stdio.h>
++#include <stdlib.h>
+ 
+ int main(int argc, char **argv)
+ {
+     printf("Hello World!\n");
+ 
+-    return 0;
++    return EXIT_SUCCESS;
+ }
+-- 
+1.8.4.5
+
diff --git a/meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.1.tar.gz b/meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.1.tar.gz
new file mode 100644
index 0000000000000000000000000000000000000000..06a1c49cd9b440bba0cc16363b60d11b206895bf
GIT binary patch
literal 411
zcmV;M0c8FkiwFRs0OVBw1MSs8Z-Ous2XM}Oiq2isOzbNymALKfakpJ|X@DsSf+^rk
zmVNiNCX1Wm=7!Ojz2Bwa6WR*@r#F01t4X5bM3)};AjB|EmQ_(?gL*O5=JNsdz-fgV
z0q5({*d8hP%mF4Om}BOHL(M8cIAqms+QZakrL<W}+xu*9?n3yK%&T;s+yUq2_ZO0Z
z7X*CdQx4KKRrRF`fyV{2RmN;J+JEi2cZLqp5cc7*=dgK@LfDZ3h4ndX`oTIbTlx2;
zbci_iLYjQ!KPy!p7t6h^t}ac7{N_Fvn*XNfzw?hUOR}tZDRdTl(I4jcx&MQG{(<y;
z^IDkifs_*a|4FcL)BmXmSs(*1q#*?1i2b(xf2p2(W9T4F`EMEb9iQ^w*#Cixo&0m^
zkduVypZEXkG>@`6PRMOp#cAQq?p-&{D>7GU-n9oxPoo<W&6FnnzByU;+*g+v3vJ5J
z-Oj|mE#%R@E>}-^r#E)jWt~*D&Iuj6Z`cbELI@#*5JCtcgb+dqA%qY*jd!bMh~WSz
F005Y))ZYLA

literal 0
HcmV?d00001

diff --git a/meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.2.tar.gz b/meta-selftest/recipes-test/devtool/files/devtool-upgrade-0.2.tar.gz
new file mode 100644
index 0000000000000000000000000000000000000000..9b0dcf4b6c1210f2b907ea62b62288d917effc4b
GIT binary patch
literal 411
zcmV;M0c8FkiwFQe0OVBw1MSs8Z-Ous0ASDjiq2is%-WWgLfm%txZ5tfG{BSu!4z~R
z%l`WwB#YDG=7!Ojz3)=mS1E<QCmlVQ^&~b?YzxYTuf-^eS7n~3!{uUXVSLD_=(NIs
zCuQA&-Ij4zbhuEOR~-%GTqz+tWZiDr!z_!^Sa_7S_u1Xtg>V^}m&rW7<5I%=SDuhm
zOW${y473ALvlZon20nwRVs$j!|J!r#3>~5&*+<7-qfTq3s#b8u>ptJqgF0Py@?XL<
zqK^F_O+NBJEld{WE53goRLF1cbD{a)^!>m5Bh2D7&0lhxMl}4x8b8;6UVVR0yDl6H
zm?vP+#rl5=BG}Y_rUKz<j|ME@Uf_%0mjA8jxi^Lm(v<%#)BTH2`QKRoxm4Tv&$X6N
z|4))c|GfTRCs~*-qnO+lWt8M}cJDYzR+70%vR-vz>@>U~;mlYv7(i#$cU~O=78b(K
zz3!wsE#$E}E>}-kw?B63u#L;5%?KMiZ&(WuLI@#*5JCtcgb+dqA%qY*jdwLBEfxSM
F003pq%-sM0

literal 0
HcmV?d00001

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index a6474b7..255f2c3 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -913,3 +913,39 @@ class DevtoolTests(DevtoolBase):
         # NOTE: native recipe parted-native should not be in IMAGE_INSTALL_append
         self.assertTrue('IMAGE_INSTALL_append = " mdadm"\n' in open(bbappend).readlines(),
                         'IMAGE_INSTALL_append = " mdadm" not found in %s' % bbappend)
+
+    def test_devtool_upgrade(self):
+        # Check preconditions
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        # Check parameters
+        result = runCmd('devtool upgrade -h')
+        for param in 'recipename srctree --version -V --branch -b --keep-temp --no-patch'.split():
+            self.assertIn(param, result.output)
+        # For the moment, we are using a real recipe.
+        recipe='devtool-upgrade'
+        version='0.2'
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        # Check that recipe is not already under devtool control
+        result = runCmd('devtool status')
+        self.assertNotIn(recipe, result.output)
+        # Check upgrade. Code does not check if new PV is older or newer that current PV, so, it may be that
+        # we are downgrading instead of upgrading.
+        result = runCmd('devtool upgrade %s %s -V %s' % (recipe, tempdir, version))
+        # Check if srctree at least is populated
+        self.assertTrue(len(os.listdir(tempdir)) > 0, 'scrtree (%s) should be populated with new (%s) source code' % (tempdir, version))
+        # Check new recipe folder is present
+        self.assertTrue(os.path.exists(os.path.join(workspacedir,'recipes',recipe)), 'Recipe folder should exist')
+        # Check new recipe file is present
+        self.assertTrue(os.path.exists(os.path.join(workspacedir,'recipes',recipe,"%s_%s.bb" % (recipe,version))), 'Recipe folder should exist')
+        # Check devtool status and make sure recipe is present
+        result = runCmd('devtool status')
+        self.assertIn(recipe, result.output)
+        self.assertIn(tempdir, result.output)
+        # Check devtool reset recipe
+        result = runCmd('devtool reset %s -n' % recipe)
+        result = runCmd('devtool status')
+        self.assertNotIn(recipe, result.output)
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 6d65d57..60d81c3 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -178,6 +178,8 @@ def extract(args, config, basepath, workspace):
 
     srctree = os.path.abspath(args.srctree)
     initial_rev = _extract_source(srctree, args.keep_temp, args.branch, rd)
+    logger.info('Source tree extracted to %s' % srctree)
+
     if initial_rev:
         return 0
     else:
@@ -351,7 +353,6 @@ def _extract_source(srctree, keep_temp, devbranch, d):
                 bb.process.run('git checkout patches', cwd=srcsubdir)
 
         shutil.move(srcsubdir, srctree)
-        logger.info('Source tree extracted to %s' % srctree)
     finally:
         bb.logger.setLevel(origlevel)
 
@@ -451,6 +452,7 @@ def modify(args, config, basepath, workspace):
         initial_rev = _extract_source(args.srctree, False, args.branch, rd)
         if not initial_rev:
             return 1
+        logger.info('Source tree extracted to %s' % srctree)
         # Get list of commits since this revision
         (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=args.srctree)
         commits = stdout.split()
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
new file mode 100644
index 0000000..86443b0
--- /dev/null
+++ b/scripts/lib/devtool/upgrade.py
@@ -0,0 +1,354 @@
+# Development tool - upgrade command plugin
+#
+# Copyright (C) 2014-2015 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+"""Devtool upgrade plugin"""
+
+import os
+import sys
+import re
+import shutil
+import tempfile
+import logging
+import argparse
+import scriptutils
+import errno
+import bb
+import oe.recipeutils
+from devtool import standard
+from devtool import exec_build_env_command, setup_tinfoil, DevtoolError, parse_recipe
+
+logger = logging.getLogger('devtool')
+
+def plugin_init(pluginlist):
+    """Plugin initialization"""
+    pass
+
+def _run(cmd, cwd=''):
+    logger.debug("Running command %s> %s" % (cwd,cmd))
+    return bb.process.run('%s' % cmd, cwd=cwd)
+
+def _get_srctree(tmpdir):
+    srctree = tmpdir
+    dirs = os.listdir(tmpdir)
+    if len(dirs) == 1:
+        srctree = os.path.join(tmpdir, dirs[0])
+    return srctree
+
+def _copy_source_code(orig, dest):
+    for path in standard._ls_tree(orig):
+        dest_dir = os.path.join(dest, os.path.dirname(path))
+        bb.utils.mkdirhier(dest_dir)
+        dest_path = os.path.join(dest, path)
+        os.rename(os.path.join(orig, path), dest_path)
+
+def _get_checksums(rf):
+    import re
+    checksums = {}
+    with open(rf) as f:
+        for line in f:
+            for cs in ['md5sum', 'sha256sum']:
+                m = re.match("^SRC_URI\[%s\].*=.*\"(.*)\"" % cs, line)
+                if m:
+                    checksums[cs] = m.group(1)
+    return checksums
+
+def _replace_checksums(rf, md5, sha256):
+    if not md5 and not sha256:
+        return
+    checksums = {'md5sum':md5, 'sha256sum':sha256}
+    with open(rf + ".tmp", "w+") as tmprf:
+        with open(rf) as f:
+            for line in f:
+                m = None
+                for cs in checksums.keys():
+                    m = re.match("^SRC_URI\[%s\].*=.*\"(.*)\"" % cs, line)
+                    if m:
+                        if checksums[cs]:
+                            oldcheck = m.group(1)
+                            newcheck = checksums[cs]
+                            line = line.replace(oldcheck, newcheck)
+                        break
+                tmprf.write(line)
+    os.rename(rf + ".tmp", rf)
+
+
+def _remove_patch_dirs(recipefolder):
+    for root, dirs, files in os.walk(recipefolder):
+        for d in dirs:
+            shutil.rmtree(os.path.join(root,d))
+
+def _recipe_contains(rf, var):
+    import re
+    found = False
+    with open(rf) as f:
+        for line in f:
+            if re.match("^%s.*=.*" % var, line):
+                found = True
+                break
+    return found
+
+def _rename_recipe_dirs(oldpv, newpv, path):
+    for root, dirs, files in os.walk(path):
+        for olddir in dirs:
+            if olddir.find(oldpv) != -1:
+                newdir = olddir.replace(oldpv, newpv)
+                if olddir != newdir:
+                    _run('mv %s %s' % (olddir, newdir))
+
+def _rename_recipe_file(bpn, oldpv, newpv, path):
+    oldrecipe = "%s_%s.bb" % (bpn, oldpv)
+    newrecipe = "%s_%s.bb" % (bpn, newpv)
+    if os.path.isfile(os.path.join(path, oldrecipe)):
+        if oldrecipe != newrecipe:
+            _run('mv %s %s' % (oldrecipe, newrecipe), cwd=path)
+    else:
+        recipe = "%s_git.bb" % bpn
+        if os.path.isfile(os.path.join(path, recipe)):
+            newrecipe = recipe
+            raise DevtoolError("Original recipe not found on workspace")
+    return os.path.join(path, newrecipe)
+
+def _rename_recipe_files(bpn, oldpv, newpv, path):
+    _rename_recipe_dirs(oldpv, newpv, path)
+    return _rename_recipe_file(bpn, oldpv, newpv, path)
+
+def _use_external_build(same_dir, no_same_dir, d):
+    b_is_s = True
+    if no_same_dir:
+        logger.info('using separate build directory since --no-same-dir specified')
+        b_is_s = False
+    elif same_dir:
+        logger.info('using source tree as build directory since --same-dir specified')
+    elif bb.data.inherits_class('autotools-brokensep', d):
+        logger.info('using source tree as build directory since original recipe inherits autotools-brokensep')
+    elif d.getVar('B', True) == os.path.abspath(d.getVar('S', True)):
+        logger.info('using source tree as build directory since that is the default for this recipe')
+    else:
+        b_is_s = False
+    return b_is_s
+
+def _write_append(rc, srctree, same_dir, no_same_dir, rev, workspace, d):
+    """Writes an append file"""
+    if not os.path.exists(rc):
+        raise DevtoolError("bbappend not created because %s does not exist" % rc)
+
+    appendpath = os.path.join(workspace, 'appends')
+    if not os.path.exists(appendpath):
+        bb.utils.mkdirhier(appendpath)
+
+    brf = os.path.basename(os.path.splitext(rc)[0]) # rc basename
+
+    srctree = os.path.abspath(srctree)
+    pn = d.getVar('PN',True)
+    af = os.path.join(appendpath, '%s.bbappend' % brf)
+    with open(af, 'w') as f:
+        f.write('FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"\n\n')
+        f.write('inherit externalsrc\n')
+        f.write(('# NOTE: We use pn- overrides here to avoid affecting'
+                 'multiple variants in the case where the recipe uses BBCLASSEXTEND\n'))
+        f.write('EXTERNALSRC_pn-%s = "%s"\n' % (pn, srctree))
+        if _use_external_build(same_dir, no_same_dir, d):
+            f.write('EXTERNALSRC_BUILD_pn-%s = "%s"\n' % (pn, srctree))
+        if rev:
+            f.write('\n# initial_rev: %s\n' % rev)
+    return af
+
+def _cleanup_on_error(rf, srctree):
+    rfp = os.path.split(rf)[0] # recipe folder
+    rfpp = os.path.split(rfp)[0] # recipes folder
+    if os.path.exists(rfp):
+        shutil.rmtree(b)
+    if not len(os.listdir(rfpp)):
+        os.rmdir(rfpp)
+    srctree = os.path.abspath(srctree)
+    if os.path.exists(srctree):
+        shutil.rmtree(srctree)
+
+def _upgrade_error(e, rf, srctree):
+    if rf:
+        cleanup_on_error(rf, srctree)
+    logger.error(e)
+    raise DevtoolError(e)
+
+def _get_uri(rd):
+    srcuris = rd.getVar('SRC_URI', True).split()
+    if not len(srcuris):
+        raise DevtoolError('SRC_URI not found on recipe')
+    srcuri = srcuris[0] # it is assumed, URI is at first position
+    srcrev = '${AUTOREV}'
+    if '://' in srcuri:
+        # Fetch a URL
+        rev_re = re.compile(';rev=([^;]+)')
+        res = rev_re.search(srcuri)
+        if res:
+            srcrev = res.group(1)
+            srcuri = rev_re.sub('', srcuri)
+    return srcuri, srcrev
+
+def _extract_new_source(newpv, srctree, no_patch, srcrev, branch, keep_temp, tinfoil, rd):
+    """Extract sources of a recipe with a new version"""
+
+    def __run(cmd):
+        """Simple wrapper which calls _run with srctree as cwd"""
+        return _run(cmd, srctree)
+
+    crd = rd.createCopy()
+
+    pv = crd.getVar('PV', True)
+    crd.setVar('PV', newpv)
+
+    tmpsrctree = None
+    uri, rev = _get_uri(crd)
+    if srcrev:
+        rev = srcrev
+    if uri.startswith('git://'):
+        __run('git checkout %s' % rev)
+        __run('git tag -f devtool-base-new')
+        md5 = None
+        sha256 = None
+    else:
+        __run('git checkout -b devtool-%s' % newpv)
+
+        tmpdir = tempfile.mkdtemp(prefix='devtool')
+        try:
+            md5, sha256 = scriptutils.fetch_uri(tinfoil.config_data, uri, tmpdir, rev)
+        except bb.fetch2.FetchError as e:
+            raise DevtoolError(e)
+
+        tmpsrctree = _get_srctree(tmpdir)
+
+        scrtree = os.path.abspath(srctree)
+
+        _copy_source_code(tmpsrctree, srctree)
+
+        (stdout,_) = __run('git ls-files --modified --others --exclude-standard')
+        for f in stdout.splitlines():
+            __run('git add "%s"' % f)
+
+        __run('git commit -q -m "Commit of upstream changes at version %s" --allow-empty' % newpv)
+        __run('git tag -f devtool-base-%s' % newpv)
+
+    (stdout, _) = __run('git rev-parse HEAD')
+    rev = stdout.rstrip()
+
+    if no_patch:
+        patches = oe.recipeutils.get_recipe_patches(crd)
+        if len(patches):
+            logger.warn('By user choice, the following patches will NOT be applied')
+            for patch in patches:
+                logger.warn("%s" % os.path.basename(patch))
+    else:
+        try:
+            __run('git checkout devtool-patched -b %s' % branch)
+            __run('git rebase %s' % rev)
+            if uri.startswith('git://'):
+                suffix = 'new'
+            else:
+                suffix = newpv
+            __run('git tag -f devtool-patched-%s' % suffix)
+        except bb.process.ExecutionError as e:
+            logger.warn('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
+
+    if tmpsrctree:
+        if keep_temp:
+            logger.info('Preserving temporary directory %s' % tmpsrctree)
+        else:
+            shutil.rmtree(tmpsrctree)
+
+    return (rev, md5, sha256)
+
+def _create_new_recipe(newpv, md5, sha256, workspace, rd):
+    """Creates the new recipe under workspace"""
+    crd = rd.createCopy()
+
+    bpn = crd.getVar('BPN', True)
+    path = os.path.join(workspace, 'recipes', bpn)
+    bb.utils.mkdirhier(path)
+    oe.recipeutils.copy_recipe_files(crd, path)
+
+    oldpv = crd.getVar('PV', True)
+    if not newpv:
+        newpv = oldpv
+    fullpath = _rename_recipe_files(bpn, oldpv, newpv, path)
+
+    if _recipe_contains(fullpath, 'PV') and newpv != oldpv:
+        oe.recipeutils.patch_recipe(d, fullpath, {'PV':newpv})
+
+    if md5 and sha256:
+        # Unfortunately, oe.recipeutils.patch_recipe cannot update flags.
+        # once the latter feature is implemented, we should call patch_recipe
+        # instead of the following function
+        _replace_checksums(fullpath, md5, sha256)
+
+    return fullpath
+
+def upgrade(args, config, basepath, workspace):
+    """Entry point for the devtool 'upgrade' subcommand"""
+
+    if args.recipename in workspace:
+        raise DevtoolError("recipe %s is already in your workspace" % args.recipename)
+    if not args.version and not args.srcrev:
+        raise DevtoolError("You must provide a version using the --version/-V option, or for recipes that fetch from an SCM such as git, the --srcrev/-S option")
+
+    reason = oe.recipeutils.validate_pn(args.recipename)
+    if reason:
+        raise DevtoolError(reason)
+
+    tinfoil = setup_tinfoil()
+
+    rd = parse_recipe(config, tinfoil, args.recipename, True)
+    if not rd:
+        return 1
+
+    standard._check_compatible_recipe(args.recipename, rd)
+    if rd.getVar('PV', True) == args.version and rd.getVar('SRCREV', True) == args.srcrev:
+        raise DevtoolError("Current and upgrade versions are the same version" % version)
+
+    rf = None
+    try:
+        rev1 = standard._extract_source(args.srctree, False, 'devtool-orig', rd)
+        rev2, md5, sha256 = _extract_new_source(args.version, args.srctree, args.no_patch,
+                                                args.srcrev, args.branch, args.keep_temp,
+                                                tinfoil, rd)
+        rf = _create_new_recipe(args.version, md5, sha256, config.workspace_path, rd)
+    except bb.process.CmdError as e:
+        _upgrade_error(e, rf, args.srctree)
+    except DevtoolError as e:
+        _upgrade_error(e, rf, args.srctree)
+    standard._add_md5(config, args.recipename, os.path.dirname(rf))
+
+    af = _write_append(rf, args.srctree, args.same_dir, args.no_same_dir, rev2,
+                       config.workspace_path, rd)
+    standard._add_md5(config, args.recipename, af)
+    logger.info('Upgraded source extracted to %s' % args.srctree)
+    return 0
+
+def register_commands(subparsers, context):
+    """Register devtool subcommands from this plugin"""
+    parser_upgrade = subparsers.add_parser('upgrade', help='Upgrade an existing recipe',
+                                           description='Upgrades an existing recipe to a new upstream version')
+    parser_upgrade.add_argument('recipename', help='Name for recipe to extract the source for')
+    parser_upgrade.add_argument('srctree', help='Path to where to extract the source tree')
+    parser_upgrade.add_argument('--version', '-V', help='Version to upgrade to (PV)')
+    parser_upgrade.add_argument('--srcrev', '-S', help='Source revision to upgrade to (if fetching from an SCM such as git)')
+    parser_upgrade.add_argument('--branch', '-b', default="devtool", help='Name for new development branch to checkout (default "%(default)s")')
+    parser_upgrade.add_argument('--no-patch', action="store_true", help='Do not apply patches from the recipe to the new source code')
+    group = parser_upgrade.add_mutually_exclusive_group()
+    group.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
+    group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true")
+    parser_upgrade.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
+    parser_upgrade.set_defaults(func=upgrade)
-- 
2.1.0

[PATCH 4/9] devtool: Create a single file for the build devtool feature

[Paul Eggleton]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

The intention is to have a single file for each devtool feature
so devtool can grow in a modular way. In this direction, this patch creates
build.py, moving all related build features from standard.py to build.py.

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
 scripts/lib/devtool/build.py    | 50 +++++++++++++++++++++++++++++++++++++++++
 scripts/lib/devtool/standard.py | 22 ------------------
 2 files changed, 50 insertions(+), 22 deletions(-)
 create mode 100644 scripts/lib/devtool/build.py

diff --git a/scripts/lib/devtool/build.py b/scripts/lib/devtool/build.py
new file mode 100644
index 0000000..0f848e2
--- /dev/null
+++ b/scripts/lib/devtool/build.py
@@ -0,0 +1,50 @@
+# Development tool - build command plugin
+#
+# Copyright (C) 2014-2015 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+"""Devtool build plugin"""
+
+import logging
+import argparse
+from devtool import exec_build_env_command
+
+logger = logging.getLogger('devtool')
+
+def plugin_init(pluginlist):
+    """Plugin initialization"""
+    pass
+
+def build(args, config, basepath, workspace):
+    """Entry point for the devtool 'build' subcommand"""
+    import bb
+    if not args.recipename in workspace:
+        raise DevtoolError("no recipe named %s in your workspace" %
+                           args.recipename)
+    build_task = config.get('Build', 'build_task', 'populate_sysroot')
+    try:
+        exec_build_env_command(config.init_path, basepath, 'bitbake -c %s %s' % (build_task, args.recipename), watch=True)
+    except bb.process.ExecutionError as e:
+        # We've already seen the output since watch=True, so just ensure we return something to the user
+        return e.exitcode
+
+    return 0
+
+def register_commands(subparsers, context):
+    """Register devtool subcommands from this plugin"""
+    parser_build = subparsers.add_parser('build', help='Build a recipe',
+                                         description='Builds the specified recipe using bitbake',
+                                         formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser_build.add_argument('recipename', help='Recipe to build')
+    parser_build.set_defaults(func=build)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 60d81c3..f2dc745 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -857,22 +857,6 @@ def reset(args, config, basepath, workspace):
     return 0
 
 
-def build(args, config, basepath, workspace):
-    """Entry point for the devtool 'build' subcommand"""
-    import bb
-    if not args.recipename in workspace:
-        raise DevtoolError("no recipe named %s in your workspace" %
-                           args.recipename)
-    build_task = config.get('Build', 'build_task', 'populate_sysroot')
-    try:
-        exec_build_env_command(config.init_path, basepath, 'bitbake -c %s %s' % (build_task, args.recipename), watch=True)
-    except bb.process.ExecutionError as e:
-        # We've already seen the output since watch=True, so just ensure we return something to the user
-        return e.exitcode
-
-    return 0
-
-
 def register_commands(subparsers, context):
     """Register devtool subcommands from this plugin"""
     parser_add = subparsers.add_parser('add', help='Add a new recipe',
@@ -921,12 +905,6 @@ def register_commands(subparsers, context):
                                           formatter_class=argparse.ArgumentDefaultsHelpFormatter)
     parser_status.set_defaults(func=status)
 
-    parser_build = subparsers.add_parser('build', help='Build a recipe',
-                                         description='Builds the specified recipe using bitbake',
-                                         formatter_class=argparse.ArgumentDefaultsHelpFormatter)
-    parser_build.add_argument('recipename', help='Recipe to build')
-    parser_build.set_defaults(func=build)
-
     parser_reset = subparsers.add_parser('reset', help='Remove a recipe from your workspace',
                                          description='Removes the specified recipe from your workspace (resetting its state)',
                                          formatter_class=argparse.ArgumentDefaultsHelpFormatter)
-- 
2.1.0

[PATCH 5/9] devtool: Allow disabling make parallelism on build command

[Paul Eggleton]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

Through -s/--disable-parallel-make, the user can turn off parallelism
on the make tool. This can be useful when debuging race condition issues.
In order to set PARALLEL_MAKE = "" a post-config file created and then
passed into the build.

[YOCTO #7589]

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/build.py | 33 ++++++++++++++++++++++++++++++---
 1 file changed, 30 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/devtool/build.py b/scripts/lib/devtool/build.py
index 0f848e2..335aff5 100644
--- a/scripts/lib/devtool/build.py
+++ b/scripts/lib/devtool/build.py
@@ -16,9 +16,12 @@
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 """Devtool build plugin"""
 
+import os
+import bb
 import logging
 import argparse
-from devtool import exec_build_env_command
+import tempfile
+from devtool import exec_build_env_command, DevtoolError
 
 logger = logging.getLogger('devtool')
 
@@ -26,18 +29,41 @@ def plugin_init(pluginlist):
     """Plugin initialization"""
     pass
 
+def _create_conf_file(values, conf_file=None):
+    if not conf_file:
+        fd, conf_file = tempfile.mkstemp(suffix='.conf')
+    elif not os.path.exists(os.path.dirname(conf_file)):
+        logger.debug("Creating folder %s" % os.path.dirname(conf_file))
+        bb.utils.mkdirhier(os.path.dirname(conf_file))
+    with open(conf_file, 'w') as f:
+        for key, value in values.iteritems():
+            f.write('%s = "%s"\n' % (key, value))
+    return conf_file
+
 def build(args, config, basepath, workspace):
     """Entry point for the devtool 'build' subcommand"""
-    import bb
     if not args.recipename in workspace:
         raise DevtoolError("no recipe named %s in your workspace" %
                            args.recipename)
+
     build_task = config.get('Build', 'build_task', 'populate_sysroot')
+
+    postfile_param = ""
+    postfile = ""
+    if args.disable_parallel_make:
+        logger.info("Disabling 'make' parallelism")
+        postfile = os.path.join(basepath, 'conf', 'disable_parallelism.conf')
+        _create_conf_file({'PARALLEL_MAKE':''}, postfile)
+        postfile_param = "-R %s" % postfile
     try:
-        exec_build_env_command(config.init_path, basepath, 'bitbake -c %s %s' % (build_task, args.recipename), watch=True)
+        exec_build_env_command(config.init_path, basepath, 'bitbake -c %s %s %s' % (build_task, postfile_param, args.recipename), watch=True)
     except bb.process.ExecutionError as e:
         # We've already seen the output since watch=True, so just ensure we return something to the user
         return e.exitcode
+    finally:
+        if postfile:
+            logger.debug('Removing postfile')
+            os.remove(postfile)
 
     return 0
 
@@ -47,4 +73,5 @@ def register_commands(subparsers, context):
                                          description='Builds the specified recipe using bitbake',
                                          formatter_class=argparse.ArgumentDefaultsHelpFormatter)
     parser_build.add_argument('recipename', help='Recipe to build')
+    parser_build.add_argument('-s', '--disable-parallel-make', action="store_true", help='Disable make parallelism')
     parser_build.set_defaults(func=build)
-- 
2.1.0

[PATCH 6/9] devtool: update-recipe: better 'auto' mode

[Paul Eggleton]

From: Markus Lehtonen <markus.lehtonen@linux.intel.com>

Enhance the logic behind the 'auto' mode a bit by only updating the
SRCREV if the changes are already found upstream. The logic is simple:
update SRCREV only if the current local HEAD commit is found in the
remote branch (i.e. 'origin/<branch_name>'). Otherwise resort to
patching.

This affects a couple of the oe-selftest tests so update those as well.

[YOCTO #7907]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py | 32 ++++++++++++++++++++++++++------
 scripts/lib/devtool/standard.py   | 34 +++++++++++++++++++++++++++-------
 2 files changed, 53 insertions(+), 13 deletions(-)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 255f2c3..6e731d6 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -547,8 +547,8 @@ class DevtoolTests(DevtoolBase):
         result = runCmd('echo "A new file" > devtool-new-file', cwd=tempdir)
         result = runCmd('git add devtool-new-file', cwd=tempdir)
         result = runCmd('git commit -m "Add a new file"', cwd=tempdir)
-        self.add_command_to_tearDown('cd %s; git checkout %s %s' % (os.path.dirname(recipefile), testrecipe, os.path.basename(recipefile)))
-        result = runCmd('devtool update-recipe %s' % testrecipe)
+        self.add_command_to_tearDown('cd %s; rm -rf %s; git checkout %s %s' % (os.path.dirname(recipefile), testrecipe, testrecipe, os.path.basename(recipefile)))
+        result = runCmd('devtool update-recipe -m srcrev %s' % testrecipe)
         result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile))
         self.assertNotEqual(result.output.strip(), "", '%s recipe should be modified' % testrecipe)
         status = result.output.splitlines()
@@ -585,6 +585,26 @@ class DevtoolTests(DevtoolBase):
                         matched = True
                         break
                 self.assertTrue(matched, 'Unexpected diff remove line: %s' % line)
+        # Now try with auto mode
+        runCmd('cd %s; git checkout %s %s' % (os.path.dirname(recipefile), testrecipe, os.path.basename(recipefile)))
+        result = runCmd('devtool update-recipe %s' % testrecipe)
+        result = runCmd('git rev-parse --show-toplevel')
+        topleveldir = result.output.strip()
+        result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile))
+        status = result.output.splitlines()
+        relpatchpath = os.path.join(os.path.relpath(os.path.dirname(recipefile), topleveldir), testrecipe)
+        expectedstatus = [('M', os.path.relpath(recipefile, topleveldir)),
+                          ('??', '%s/0001-Change-the-Makefile.patch' % relpatchpath),
+                          ('??', '%s/0002-Add-a-new-file.patch' % relpatchpath)]
+        for line in status:
+            statusline = line.split(None, 1)
+            for fstatus, fn in expectedstatus:
+                if fn == statusline[1]:
+                    if fstatus != statusline[0]:
+                        self.fail('Unexpected status in line: %s' % line)
+                    break
+            else:
+                self.fail('Unexpected modified file in line: %s' % line)
 
     @testcase(1170)
     def test_devtool_update_recipe_append(self):
@@ -708,7 +728,7 @@ class DevtoolTests(DevtoolBase):
         self.add_command_to_tearDown('bitbake-layers remove-layer %s || true' % templayerdir)
         result = runCmd('bitbake-layers add-layer %s' % templayerdir, cwd=self.builddir)
         # Create the bbappend
-        result = runCmd('devtool update-recipe %s -a %s' % (testrecipe, templayerdir))
+        result = runCmd('devtool update-recipe -m srcrev %s -a %s' % (testrecipe, templayerdir))
         self.assertNotIn('WARNING:', result.output)
         # Check recipe is still clean
         result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile))
@@ -729,12 +749,12 @@ class DevtoolTests(DevtoolBase):
             self.assertEqual(expectedlines, f.readlines())
 
         # Check we can run it again and bbappend isn't modified
-        result = runCmd('devtool update-recipe %s -a %s' % (testrecipe, templayerdir))
+        result = runCmd('devtool update-recipe -m srcrev %s -a %s' % (testrecipe, templayerdir))
         with open(bbappendfile, 'r') as f:
             self.assertEqual(expectedlines, f.readlines())
         # Drop new commit and check SRCREV changes
         result = runCmd('git reset HEAD^', cwd=tempsrcdir)
-        result = runCmd('devtool update-recipe %s -a %s' % (testrecipe, templayerdir))
+        result = runCmd('devtool update-recipe -m srcrev %s -a %s' % (testrecipe, templayerdir))
         self.assertFalse(os.path.exists(os.path.join(appenddir, testrecipe)), 'Patch directory should not be created')
         result = runCmd('git rev-parse HEAD', cwd=tempsrcdir)
         expectedlines = ['SRCREV = "%s"\n' % result.output,
@@ -747,7 +767,7 @@ class DevtoolTests(DevtoolBase):
         os.remove(bbappendfile)
         result = runCmd('git commit -a -m "Change the Makefile"', cwd=tempsrcdir)
         result = runCmd('bitbake-layers remove-layer %s' % templayerdir, cwd=self.builddir)
-        result = runCmd('devtool update-recipe %s -a %s' % (testrecipe, templayerdir))
+        result = runCmd('devtool update-recipe -m srcrev %s -a %s' % (testrecipe, templayerdir))
         self.assertIn('WARNING: Specified layer is not currently enabled in bblayers.conf', result.output)
         self.assertFalse(os.path.exists(os.path.join(appenddir, testrecipe)), 'Patch directory should not be created')
         result = runCmd('git rev-parse HEAD', cwd=tempsrcdir)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index f2dc745..c398e4c 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -757,6 +757,31 @@ def _update_recipe_patch(args, config, srctree, rd, config_data):
 
     _remove_patch_files(args, removepatches, destpath)
 
+def _guess_recipe_update_mode(srctree, rdata):
+    """Guess the recipe update mode to use"""
+    src_uri = (rdata.getVar('SRC_URI', False) or '').split()
+    git_uris = [uri for uri in src_uri if uri.startswith('git://')]
+    if not git_uris:
+        return 'patch'
+    # Just use the first URI for now
+    uri = git_uris[0]
+    # Check remote branch
+    upstr_branch = 'master'
+    for paramdef in uri.split(';')[1:]:
+        name, value = paramdef.split('=', 1)
+        if name == 'branch':
+            upstr_branch = value
+    # Check if current branch HEAD is found in upstream branch
+    stdout, _ = bb.process.run('git rev-parse HEAD', cwd=srctree)
+    head_rev = stdout.rstrip()
+    stdout, _ = bb.process.run('git branch -r --contains %s' % head_rev,
+                               cwd=srctree)
+    remote_brs = [branch.strip() for branch in stdout.splitlines()]
+    if 'origin/' + upstr_branch in remote_brs:
+        return 'srcrev'
+
+    return 'patch'
+
 def update_recipe(args, config, basepath, workspace):
     """Entry point for the devtool 'update-recipe' subcommand"""
     if not args.recipename in workspace:
@@ -777,17 +802,12 @@ def update_recipe(args, config, basepath, workspace):
     if not rd:
         return 1
 
-    orig_src_uri = rd.getVar('SRC_URI', False) or ''
+    srctree = workspace[args.recipename]['srctree']
     if args.mode == 'auto':
-        if 'git://' in orig_src_uri:
-            mode = 'srcrev'
-        else:
-            mode = 'patch'
+        mode = _guess_recipe_update_mode(srctree, rd)
     else:
         mode = args.mode
 
-    srctree = workspace[args.recipename]['srctree']
-
     if mode == 'srcrev':
         _update_recipe_srcrev(args, srctree, rd, tinfoil.config_data)
     elif mode == 'patch':
-- 
2.1.0

[PATCH 7/9] devtool: update-recipe: add a warning when recipe is inside the workspace

[Paul Eggleton]

If a recipe has been added to the workspace via "devtool add"
or "devtool upgrade" then the recipe file itself will be in the
workspace layer; if you run "devtool update-recipe" particularly in the
upgrade case you might think it would apply the upgrade to the original
recipe, but it will not - in order to remain consistent it has to update
the recipe that's in use which is the one in the workspace. Warn the
user in this situation so that they know what they need to do when they
are finished.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index c398e4c..4ae41cf 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -815,6 +815,10 @@ def update_recipe(args, config, basepath, workspace):
     else:
         raise DevtoolError('update_recipe: invalid mode %s' % mode)
 
+    rf = rd.getVar('FILE', True)
+    if rf.startswith(config.workspace_path):
+        logger.warn('Recipe file %s has been updated but is inside the workspace - you will need to move it (and any associated files next to it) out to the desired layer before using "devtool reset" in order to keep any changes' % rf)
+
     return 0
 
 
-- 
2.1.0

[PATCH 8/9] devtool: add package plugin that lets you create package via devtool

[Paul Eggleton]

From: Brendan Le Foll <brendan.le.foll@intel.com>

Enables creating packages using devtool within the extensible SDK. (This
is only enabled within the extensible SDK because it provides no
advantage over just running bitbake directly there).

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/package.py | 61 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)
 create mode 100644 scripts/lib/devtool/package.py

diff --git a/scripts/lib/devtool/package.py b/scripts/lib/devtool/package.py
new file mode 100644
index 0000000..3a7a36b
--- /dev/null
+++ b/scripts/lib/devtool/package.py
@@ -0,0 +1,61 @@
+# Development tool - package command plugin
+#
+# Copyright (C) 2014-2015 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+"""Devtool plugin containing the package subcommands"""
+
+import os
+import subprocess
+import logging
+from bb.process import ExecutionError
+from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
+
+logger = logging.getLogger('devtool')
+
+def plugin_init(pluginlist):
+    """Plugin initialization"""
+    pass
+
+def package(args, config, basepath, workspace):
+    """Entry point for the devtool 'package' subcommand"""
+    if not args.recipename in workspace:
+        raise DevtoolError("no recipe named %s in your workspace" %
+                           args.recipename)
+
+    image_pkgtype = config.get('Package', 'image_pkgtype', '')
+    if not image_pkgtype:
+        tinfoil = setup_tinfoil()
+        try:
+            tinfoil.prepare(config_only=True)
+            image_pkgtype = tinfoil.config_data.getVar('IMAGE_PKGTYPE', True)
+        finally:
+            tinfoil.shutdown()
+
+    package_task = config.get('Package', 'package_task', 'package_write_%s' % image_pkgtype)
+    try:
+        exec_build_env_command(config.init_path, basepath, 'bitbake -c %s %s' % (package_task, args.recipename), watch=True)
+    except bb.process.ExecutionError as e:
+        # We've already seen the output since watch=True, so just ensure we return something to the user
+        return e.exitcode
+    logger.info('Your packages are in %s/tmp/deploy/%s' % (basepath, image_pkgtype))
+
+    return 0
+
+def register_commands(subparsers, context):
+    """Register devtool subcommands from the package plugin"""
+    if context.fixed_setup:
+        parser_package = subparsers.add_parser('package', help='Build packages for a recipe', description='Builds packages for a recipe\'s output files')
+        parser_package.add_argument('recipename', help='Recipe to package')
+        parser_package.set_defaults(func=package)
-- 
2.1.0

[PATCH 9/9] devtool: fix running from a different directory

[Paul Eggleton]

Fixes the following error when running devtool from a directory other
than the build directory (or the SDK base path when using within the
extensible SDK):

  The BBPATH variable is not set and bitbake did not find a
  conf/bblayers.conf file in the expected location.
  Maybe you accidentally invoked bitbake from the wrong directory?

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/devtool | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/devtool b/scripts/devtool
index b9d3bb9..87df951 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -221,6 +221,9 @@ def main():
     if not config.read():
         return -1
 
+    # We need to be in this directory or we won't be able to initialise tinfoil
+    os.chdir(basepath)
+
     bitbake_subdir = config.get('General', 'bitbake_subdir', '')
     if bitbake_subdir:
         # Normally set for use within the SDK
-- 
2.1.0