[PATCH 00/15] devtool and externalsrc improvements

[PATCH 00/15] devtool and externalsrc improvements

[Paul Eggleton]

I've been attempting to enable usage of devtool with the kernel (which
effectively means using the externalsrc class with the kernel). This
doesn't really work yet due to lack of handling the kernel configuration,
so you still can't use externalsrc properly with the kernel, but this
at least gets us further - you can now extract the source successfully
and run a build. I also discovered and fixed a number of related issues
along the way, and made a few behavioural improvements in both devtool
and externalsrc.


The following changes since commit ad016fa7492ba23c751cf33dd1befde7782facfa:

  lzo: Upgrade to 2.09 and add ptest (2015-02-19 07:50:44 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/devtool-fixes
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool-fixes

Paul Eggleton (15):
  recipetool: fix regression caused by previous commit
  lib/oe/patch: fix PATCHTOOL = "git" with source in a subdirectory
  classes/externalsrc: fix source being wiped out on clean with kernel
  classes/externalsrc: add workaround for recipes that use SRCPV in PV
  classes/externalsrc: show a warning on compiling
  classes/kernel-yocto: extend SRCTREECOVEREDTASKS
  devtool: add/modify: add option to build in same directory
  devtool: reset: run bitbake -c clean on recipe
  devtool: build: run do_populate_sysroot instead of do_install
  lib/oe/recipeutils: minor improvements to patch_recipe()
  devtool: update-recipe: add handling for git recipes
  devtool: prevent extract/modify use with image recipes
  devtool: modify: handle recipes that use a shared workdir
  devtool: support source extraction for recipes that use a shared
    workdir
  devtool: fix broken clones of git recipes

 meta/classes/externalsrc.bbclass  |  24 ++++
 meta/classes/kernel-yocto.bbclass |   2 +-
 meta/lib/oe/patch.py              |   7 +-
 meta/lib/oe/recipeutils.py        |   4 +-
 meta/lib/oeqa/selftest/devtool.py | 147 ++++++++++++++++++-
 scripts/lib/devtool/standard.py   | 292 +++++++++++++++++++++++++-------------
 scripts/lib/recipetool/create.py  |   8 +-
 7 files changed, 377 insertions(+), 107 deletions(-)

-- 
1.9.3

[PATCH 01/15] recipetool: fix regression caused by previous commit

[Paul Eggleton]

Option was renamed in the setup code but not in the code that used it.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 290fc13..ae599cb 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -110,8 +110,8 @@ def create_recipe(args):
     if '://' in args.source:
         # Fetch a URL
         srcuri = args.source
-        if args.externalsrc:
-            srctree = args.externalsrc
+        if args.extract_to:
+            srctree = args.extract_to
         else:
             tempsrc = tempfile.mkdtemp(prefix='recipetool-')
             srctree = tempsrc
@@ -126,8 +126,8 @@ def create_recipe(args):
             srctree = os.path.join(srctree, srcsubdir)
     else:
         # Assume we're pointing to an existing source tree
-        if args.externalsrc:
-            logger.error('externalsrc cannot be specified if source is a directory')
+        if args.extract_to:
+            logger.error('--extract-to cannot be specified if source is a directory')
             sys.exit(1)
         if not os.path.isdir(args.source):
             logger.error('Invalid source directory %s' % args.source)
-- 
1.9.3

[PATCH 02/15] lib/oe/patch: fix PATCHTOOL = "git" with source in a subdirectory

[Paul Eggleton]

For recipes that have their actual source in a subdirectory of what is
fetched (e.g. mkelfimage), we need to find the root of the repository
within the GitApplyTree code that attempts to set up the required git
hooks and use that, rather than expecting the root to be the same as
${S}.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/patch.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index b838be8..f68d40f 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -313,9 +313,12 @@ class GitApplyTree(PatchTree):
             return runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
 
         # Add hooks which add a pointer to the original patch file name in the commit message
-        commithook = os.path.join(self.dir, '.git', 'hooks', 'commit-msg')
+        reporoot = (runcmd("git rev-parse --show-toplevel".split(), self.dir) or '').strip()
+        if not reporoot:
+            raise Exception("Cannot get repository root for directory %s" % self.dir)
+        commithook = os.path.join(reporoot, '.git', 'hooks', 'commit-msg')
         commithook_backup = commithook + '.devtool-orig'
-        applyhook = os.path.join(self.dir, '.git', 'hooks', 'applypatch-msg')
+        applyhook = os.path.join(reporoot, '.git', 'hooks', 'applypatch-msg')
         applyhook_backup = applyhook + '.devtool-orig'
         if os.path.exists(commithook):
             shutil.move(commithook, commithook_backup)
-- 
1.9.3

[PATCH 03/15] classes/externalsrc: fix source being wiped out on clean with kernel

[Paul Eggleton]

kernel.bbclass adds ${S} do do_clean[cleandirs], but this means if you
run bitbake -c clean <kernelrecipe> then your external source tree will
be trashed, which could be a disaster. For safety, remove ${S} from
cleandirs for every task. We also have to do the same for ${B} in the
case where EXTERNALSRC_BUILD is set to the same value as EXTERNALSRC.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/externalsrc.bbclass | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 4e429d7..e372392 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -47,6 +47,20 @@ python () {
                 # Since configure will likely touch ${S}, ensure only we lock so one task has access at a time
                 d.appendVarFlag(task, "lockfiles", "${S}/singletask.lock")
 
+            # We do not want our source to be wiped out, ever (kernel.bbclass does this for do_clean)
+            cleandirs = d.getVarFlag(task, 'cleandirs', False)
+            if cleandirs:
+                cleandirs = cleandirs.split()
+                setvalue = False
+                if '${S}' in cleandirs:
+                    cleandirs.remove('${S}')
+                    setvalue = True
+                if externalsrcbuild == externalsrc and '${B}' in cleandirs:
+                    cleandirs.remove('${B}')
+                    setvalue = True
+                if setvalue:
+                    d.setVarFlag(task, 'cleandirs', ' '.join(cleandirs))
+
         for task in d.getVar("SRCTREECOVEREDTASKS", True).split():
             bb.build.deltask(task, d)
 
-- 
1.9.3

[PATCH 04/15] classes/externalsrc: add workaround for recipes that use SRCPV in PV

[Paul Eggleton]

Here we set SRC_URI to blank, however doing so means that the function
that is called when you expand the default value of SRCPV
(i.e. bb.fetch2.get_srcrev()) will fail, so any recipe that references
SRCPV in PV couldn't previously be used with externalsrc.

(At some point we may fix the function to work in the externalsrc case,
but then we would also need to ensure that ${B} did not change as a
result of PV changing any time the HEAD revision changes in the external
source tree, or you'll lose any intermediate build artifacts.)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/externalsrc.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index e372392..234bd8d 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -37,6 +37,10 @@ python () {
             d.setVar('B', '${WORKDIR}/${BPN}-${PV}/')
         d.setVar('SRC_URI', '')
 
+        if '{SRCPV}' in d.getVar('PV', False):
+            # Dummy value because the default function can't be called with blank SRC_URI
+            d.setVar('SRCPV', '999')
+
         tasks = filter(lambda k: d.getVarFlag(k, "task"), d.keys())
 
         for task in tasks:
-- 
1.9.3

[PATCH 05/15] classes/externalsrc: show a warning on compiling

[Paul Eggleton]

Make sure there's no chance of anyone forgetting they have a recipe set
up for externalsrc; otherwise you could get confused about what is going
on. (With our default logging setup we can't make it a note because the
UI doesn't forward those; otherwise I would have used bb.note().)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/externalsrc.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 234bd8d..25f0be9 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -68,7 +68,13 @@ python () {
         for task in d.getVar("SRCTREECOVEREDTASKS", True).split():
             bb.build.deltask(task, d)
 
+        d.prependVarFlag('do_compile', 'prefuncs', "externalsrc_compile_prefunc ")
+
         # Ensure compilation happens every time
         d.setVarFlag('do_compile', 'nostamp', '1')
 }
 
+python externalsrc_compile_prefunc() {
+    # Make it obvious that this is happening, since forgetting about it could lead to much confusion
+    bb.warn('Compiling %s from external source %s' % (d.getVar('PN', True), d.getVar('EXTERNALSRC', True)))
+}
-- 
1.9.3

[PATCH 06/15] classes/kernel-yocto: extend SRCTREECOVEREDTASKS

[Paul Eggleton]

* Add do_shared_workdir which was added recently
* Add do_fetch and do_unpack to this list, because at the moment if you
  enable externalsrc through a bbappend the += in this class wipes out
  the original value from externalsrc (which is set with ?=)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/kernel-yocto.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
index 223825e..c3f1ac7 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -1,5 +1,5 @@
 # remove tasks that modify the source tree in case externalsrc is inherited
-SRCTREECOVEREDTASKS += "do_kernel_link_vmlinux do_kernel_configme do_validate_branches do_kernel_configcheck do_kernel_checkout do_patch"
+SRCTREECOVEREDTASKS += "do_kernel_link_vmlinux do_kernel_configme do_validate_branches do_kernel_configcheck do_kernel_checkout do_shared_workdir do_fetch do_unpack do_patch"
 
 # returns local (absolute) path names for all valid patches in the
 # src_uri
-- 
1.9.3

[PATCH 07/15] devtool: add/modify: add option to build in same directory

[Paul Eggleton]

The default behaviour is to build in a separate directory to the source,
however some projects can't be built this way, so add an option to do
that (or override the automatic behaviour in the case of modify).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index ae64840..d503111 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -77,6 +77,8 @@ def add(args, config, basepath, workspace):
     with open(appendfile, 'w') as f:
         f.write('inherit externalsrc\n')
         f.write('EXTERNALSRC = "%s"\n' % srctree)
+        if args.same_dir:
+            f.write('EXTERNALSRC_BUILD = "%s"\n' % srctree)
         if initial_rev:
             f.write('\n# initial_rev: %s\n' % initial_rev)
 
@@ -323,8 +325,11 @@ def modify(args, config, basepath, workspace):
         f.write('inherit externalsrc\n')
         f.write('# NOTE: We use pn- overrides here to avoid affecting multiple variants in the case where the recipe uses BBCLASSEXTEND\n')
         f.write('EXTERNALSRC_pn-%s = "%s"\n' % (args.recipename, srctree))
-        if bb.data.inherits_class('autotools-brokensep', rd):
-            logger.info('using source tree as build directory since original recipe inherits autotools-brokensep')
+        if args.same_dir or bb.data.inherits_class('autotools-brokensep', rd):
+            if args.same_dir:
+                logger.info('using source tree as build directory since --same-dir specified')
+            else:
+                logger.info('using source tree as build directory since original recipe inherits autotools-brokensep')
             f.write('EXTERNALSRC_BUILD_pn-%s = "%s"\n' % (args.recipename, srctree))
         if initial_rev:
             f.write('\n# initial_rev: %s\n' % initial_rev)
@@ -503,6 +508,7 @@ def register_commands(subparsers, context):
                                        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
     parser_add.add_argument('recipename', help='Name for new recipe to add')
     parser_add.add_argument('srctree', help='Path to external source tree')
+    parser_add.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
     parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
     parser_add.set_defaults(func=add)
 
@@ -513,6 +519,7 @@ def register_commands(subparsers, context):
     parser_add.add_argument('srctree', help='Path to external source tree')
     parser_add.add_argument('--wildcard', '-w', action="store_true", help='Use wildcard for unversioned bbappend')
     parser_add.add_argument('--extract', '-x', action="store_true", help='Extract source as well')
+    parser_add.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
     parser_add.add_argument('--branch', '-b', default="devtool", help='Name for development branch to checkout (only when using -x)')
     parser_add.set_defaults(func=modify)
 
-- 
1.9.3

[PATCH 08/15] devtool: reset: run bitbake -c clean on recipe

[Paul Eggleton]

If you've added a new recipe, you want the output cleaned when you do
devtool reset, otherwise cruft from building the recipe may remain which
could interfere with future builds.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py | 4 ++++
 scripts/lib/devtool/standard.py   | 6 ++++++
 2 files changed, 10 insertions(+)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 74fb325..33f2e91 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -179,9 +179,13 @@ class DevtoolTests(oeSelfTest):
                 if line.startswith('.TH'):
                     self.assertEqual(line.rstrip(), '.TH MDADM 8 "" v9.999-custom', 'man file not modified')
         # Test devtool reset
+        stampprefix = get_bb_var('STAMP', 'mdadm')
         result = runCmd('devtool reset mdadm')
         result = runCmd('devtool status')
         self.assertNotIn('mdadm', result.output)
+        self.assertTrue(stampprefix, 'Unable to get STAMP value for recipe mdadm')
+        matches = glob.glob(stampprefix + '*')
+        self.assertFalse(matches, 'Stamp files exist for recipe mdadm that should have been cleaned')
 
     def test_devtool_update_recipe(self):
         # Check preconditions
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index d503111..763177d 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -475,6 +475,11 @@ def reset(args, config, basepath, workspace):
     if not args.recipename in workspace:
         logger.error("no recipe named %s in your workspace" % args.recipename)
         return -1
+
+    if not args.no_clean:
+        logger.info('Cleaning sysroot for recipe %s...' % args.recipename)
+        exec_build_env_command(config.init_path, basepath, 'bitbake -c clean %s' % args.recipename)
+
     _check_preserve(config, args.recipename)
 
     preservepath = os.path.join(config.workspace_path, 'attic', args.recipename)
@@ -555,5 +560,6 @@ def register_commands(subparsers, context):
                                          description='Removes the specified recipe from your workspace (resetting its state)',
                                          formatter_class=argparse.ArgumentDefaultsHelpFormatter)
     parser_reset.add_argument('recipename', help='Recipe to reset')
+    parser_reset.add_argument('--no-clean', '-n', action="store_true", help='Don\'t clean the sysroot to remove recipe output')
     parser_reset.set_defaults(func=reset)
 
-- 
1.9.3

[PATCH 09/15] devtool: build: run do_populate_sysroot instead of do_install

[Paul Eggleton]

If you want to be able to make use of libraries in conjunction with
devtool then we need to install them into the sysroot for other recipes
to use. Make it a configuration option in case it needs to be changed at
runtime.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py | 40 +++++++++++++++++++++++++++++++++++++++
 scripts/lib/devtool/standard.py   |  5 +++--
 2 files changed, 43 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 33f2e91..8caf07a 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -136,6 +136,46 @@ class DevtoolTests(oeSelfTest):
             bindir = bindir[1:]
         self.assertTrue(os.path.isfile(os.path.join(installdir, bindir, 'pv')), 'pv binary not found in D')
 
+    def test_devtool_add_library(self):
+        # Check preconditions
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        # We don't have the ability to pick up this dependency automatically yet...
+        bitbake('libusb1')
+        # Fetch source
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        url = 'http://www.intra2net.com/en/developer/libftdi/download/libftdi1-1.1.tar.bz2'
+        result = runCmd('wget %s' % url, cwd=tempdir)
+        result = runCmd('tar xfv libftdi1-1.1.tar.bz2', cwd=tempdir)
+        srcdir = os.path.join(tempdir, 'libftdi1-1.1')
+        self.assertTrue(os.path.isfile(os.path.join(srcdir, 'CMakeLists.txt')), 'Unable to find CMakeLists.txt in source directory')
+        # Test devtool add
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        result = runCmd('devtool add libftdi %s' % srcdir)
+        self.assertTrue(os.path.exists(os.path.join(workspacedir, 'conf', 'layer.conf')), 'Workspace directory not created')
+        # Test devtool status
+        result = runCmd('devtool status')
+        self.assertIn('libftdi', result.output)
+        self.assertIn(srcdir, result.output)
+        # Clean up anything in the workdir/sysroot/sstate cache (have to do this *after* devtool add since the recipe only exists then)
+        bitbake('libftdi -c cleansstate')
+        # Test devtool build
+        result = runCmd('devtool build libftdi')
+        staging_libdir = get_bb_var('STAGING_LIBDIR', 'libftdi')
+        self.assertTrue(staging_libdir, 'Could not query STAGING_LIBDIR variable')
+        self.assertTrue(os.path.isfile(os.path.join(staging_libdir, 'libftdi1.so.2.1.0')), 'libftdi binary not found in STAGING_LIBDIR')
+        # Test devtool reset
+        stampprefix = get_bb_var('STAMP', 'libftdi')
+        result = runCmd('devtool reset libftdi')
+        result = runCmd('devtool status')
+        self.assertNotIn('libftdi', result.output)
+        self.assertTrue(stampprefix, 'Unable to get STAMP value for recipe libftdi')
+        matches = glob.glob(stampprefix + '*')
+        self.assertFalse(matches, 'Stamp files exist for recipe libftdi that should have been cleaned')
+        self.assertFalse(os.path.isfile(os.path.join(staging_libdir, 'libftdi1.so.2.1.0')), 'libftdi binary still found in STAGING_LIBDIR after cleaning')
+
     def test_devtool_modify(self):
         # Check preconditions
         workspacedir = os.path.join(self.builddir, 'workspace')
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 763177d..9b5a085 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -502,7 +502,8 @@ def build(args, config, basepath, workspace):
     if not args.recipename in workspace:
         logger.error("no recipe named %s in your workspace" % args.recipename)
         return -1
-    exec_build_env_command(config.init_path, basepath, 'bitbake -c install %s' % args.recipename, watch=True)
+    build_task = config.get('Build', 'build_task', 'populate_sysroot')
+    exec_build_env_command(config.init_path, basepath, 'bitbake -c %s %s' % (build_task, args.recipename), watch=True)
 
     return 0
 
@@ -551,7 +552,7 @@ def register_commands(subparsers, context):
     parser_status.set_defaults(func=status)
 
     parser_build = subparsers.add_parser('build', help='Build a recipe',
-                                         description='Builds the specified recipe using bitbake (up to do_install)',
+                                         description='Builds the specified recipe using bitbake',
                                          formatter_class=argparse.ArgumentDefaultsHelpFormatter)
     parser_build.add_argument('recipename', help='Recipe to build')
     parser_build.set_defaults(func=build)
-- 
1.9.3

[PATCH 10/15] lib/oe/recipeutils: minor improvements to patch_recipe()

[Paul Eggleton]

* Ensure it knows where to put SRCREV and S
* Handle prepend/append and functions in general

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/recipeutils.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 1758dce..159a103 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -18,7 +18,7 @@ from collections import OrderedDict, defaultdict
 
 
 # Help us to find places to insert values
-recipe_progression = ['SUMMARY', 'DESCRIPTION', 'HOMEPAGE', 'BUGTRACKER', 'SECTION', 'LICENSE', 'LIC_FILES_CHKSUM', 'PROVIDES', 'DEPENDS', 'PR', 'PV', 'SRC_URI', 'do_fetch', 'do_unpack', 'do_patch', 'EXTRA_OECONF', 'do_configure', 'EXTRA_OEMAKE', 'do_compile', 'do_install', 'do_populate_sysroot', 'INITSCRIPT', 'USERADD', 'GROUPADD', 'PACKAGES', 'FILES', 'RDEPENDS', 'RRECOMMENDS', 'RSUGGESTS', 'RPROVIDES', 'RREPLACES', 'RCONFLICTS', 'ALLOW_EMPTY', 'do_package', 'do_deploy']
+recipe_progression = ['SUMMARY', 'DESCRIPTION', 'HOMEPAGE', 'BUGTRACKER', 'SECTION', 'LICENSE', 'LIC_FILES_CHKSUM', 'PROVIDES', 'DEPENDS', 'PR', 'PV', 'SRCREV', 'SRC_URI', 'S', 'do_fetch', 'do_unpack', 'do_patch', 'EXTRA_OECONF', 'do_configure', 'EXTRA_OEMAKE', 'do_compile', 'do_install', 'do_populate_sysroot', 'INITSCRIPT', 'USERADD', 'GROUPADD', 'PACKAGES', 'FILES', 'RDEPENDS', 'RRECOMMENDS', 'RSUGGESTS', 'RPROVIDES', 'RREPLACES', 'RCONFLICTS', 'ALLOW_EMPTY', 'do_package', 'do_deploy']
 # Variables that sometimes are a bit long but shouldn't be wrapped
 nowrap_vars = ['SUMMARY', 'HOMEPAGE', 'BUGTRACKER']
 list_vars = ['SRC_URI', 'LIC_FILES_CHKSUM']
@@ -113,7 +113,7 @@ def patch_recipe_file(fn, values, patch=False, relpath=''):
                         insert = False
                         for k in remainingnames.keys():
                             for p in recipe_progression:
-                                if re.match('^%s[ ?:=]' % p, line):
+                                if re.match('^%s(_prepend|_append)*[ ?:=(]' % p, line):
                                     if remainingnames[k] > -1 and recipe_progression.index(p) > remainingnames[k] and runthrough > 1 and not k in existingnames:
                                         outputvalue(k)
                                         del remainingnames[k]
-- 
1.9.3

[PATCH 11/15] devtool: update-recipe: add handling for git recipes

[Paul Eggleton]

When updating git-based recipes, in a lot of cases what you want is to
push the changes to the repository and update SRCREV rather than to
apply patches within the recipe. Updating SRCREV is now the default
behaviour for recipes that fetch from git, but this can be overridden
in both directions using a new -m/--mode option.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py |  70 +++++++++++-
 scripts/lib/devtool/standard.py   | 230 ++++++++++++++++++++++++--------------
 2 files changed, 213 insertions(+), 87 deletions(-)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 8caf07a..f147f24 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -233,6 +233,8 @@ class DevtoolTests(oeSelfTest):
         self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
         testrecipe = 'minicom'
         recipefile = get_bb_var('FILE', testrecipe)
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        self.assertNotIn('git://', src_uri, 'This test expects the %s recipe to NOT be a git recipe' % testrecipe)
         result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile))
         self.assertEqual(result.output.strip(), "", '%s recipe is not clean' % testrecipe)
         # First, modify a recipe
@@ -266,11 +268,77 @@ class DevtoolTests(oeSelfTest):
                 self.assertEqual(line[:3], '?? ', 'Unexpected status in line: %s' % line)
             elif line.endswith('0002-Add-a-new-file.patch'):
                 self.assertEqual(line[:3], '?? ', 'Unexpected status in line: %s' % line)
-            elif re.search('minicom_[^_]*.bb$', line):
+            elif re.search('%s_[^_]*.bb$' % testrecipe, line):
                 self.assertEqual(line[:3], ' M ', 'Unexpected status in line: %s' % line)
             else:
                 raise AssertionError('Unexpected modified file in status: %s' % line)
 
+    def test_devtool_update_recipe_git(self):
+        # Check preconditions
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        testrecipe = 'mtd-utils'
+        recipefile = get_bb_var('FILE', testrecipe)
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        self.assertIn('git://', src_uri, 'This test expects the %s recipe to be a git recipe' % testrecipe)
+        result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile))
+        self.assertEqual(result.output.strip(), "", '%s recipe is not clean' % testrecipe)
+        # First, modify a recipe
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        # (don't bother with cleaning the recipe on teardown, we won't be building it)
+        result = runCmd('devtool modify %s -x %s' % (testrecipe, tempdir))
+        # Check git repo
+        self.assertTrue(os.path.isdir(os.path.join(tempdir, '.git')), 'git repository for external source tree not found')
+        result = runCmd('git status --porcelain', cwd=tempdir)
+        self.assertEqual(result.output.strip(), "", 'Created git repo is not clean')
+        result = runCmd('git symbolic-ref HEAD', cwd=tempdir)
+        self.assertEqual(result.output.strip(), "refs/heads/devtool", 'Wrong branch in git repo')
+        # Add a couple of commits
+        # FIXME: this only tests adding, need to also test update and remove
+        result = runCmd('echo "# Additional line" >> Makefile', cwd=tempdir)
+        result = runCmd('git commit -a -m "Change the Makefile"', cwd=tempdir)
+        result = runCmd('echo "A new file" > devtool-new-file', cwd=tempdir)
+        result = runCmd('git add devtool-new-file', cwd=tempdir)
+        result = runCmd('git commit -m "Add a new file"', cwd=tempdir)
+        self.add_command_to_tearDown('cd %s; git checkout %s %s' % (os.path.dirname(recipefile), testrecipe, os.path.basename(recipefile)))
+        result = runCmd('devtool update-recipe %s' % testrecipe)
+        result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile))
+        self.assertNotEqual(result.output.strip(), "", '%s recipe should be modified' % testrecipe)
+        status = result.output.splitlines()
+        self.assertEqual(len(status), 3, 'Less/more files modified than expected. Entire status:\n%s' % result.output)
+        for line in status:
+            if line.endswith('add-exclusion-to-mkfs-jffs2-git-2.patch'):
+                self.assertEqual(line[:3], ' D ', 'Unexpected status in line: %s' % line)
+            elif line.endswith('fix-armv7-neon-alignment.patch'):
+                self.assertEqual(line[:3], ' D ', 'Unexpected status in line: %s' % line)
+            elif re.search('%s_[^_]*.bb$' % testrecipe, line):
+                self.assertEqual(line[:3], ' M ', 'Unexpected status in line: %s' % line)
+            else:
+                raise AssertionError('Unexpected modified file in status: %s' % line)
+        result = runCmd('git diff %s' % os.path.basename(recipefile), cwd=os.path.dirname(recipefile))
+        addlines = ['SRCREV = ".*"', 'SRC_URI = "git://git.infradead.org/mtd-utils.git"']
+        removelines = ['SRCREV = ".*"', 'SRC_URI = "git://git.infradead.org/mtd-utils.git \\\\', 'file://add-exclusion-to-mkfs-jffs2-git-2.patch \\\\', 'file://fix-armv7-neon-alignment.patch \\\\', '"']
+        for line in result.output.splitlines():
+            if line.startswith('+++') or line.startswith('---'):
+                continue
+            elif line.startswith('+'):
+                matched = False
+                for item in addlines:
+                    if re.match(item, line[1:].strip()):
+                        matched = True
+                        break
+                self.assertTrue(matched, 'Unexpected diff add line: %s' % line)
+            elif line.startswith('-'):
+                matched = False
+                for item in removelines:
+                    if re.match(item, line[1:].strip()):
+                        matched = True
+                        break
+                self.assertTrue(matched, 'Unexpected diff remove line: %s' % line)
+
     def test_devtool_extract(self):
         # Check preconditions
         workspacedir = os.path.join(self.builddir, 'workspace')
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 9b5a085..3a8c66c 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -359,104 +359,162 @@ def update_recipe(args, config, basepath, workspace):
     from oe.patch import GitApplyTree
     import oe.recipeutils
 
-    srctree = workspace[args.recipename]
-    commits = []
-    update_rev = None
-    if args.initial_rev:
-        initial_rev = args.initial_rev
-    else:
-        initial_rev = None
-        with open(appends[0], 'r') as f:
-            for line in f:
-                if line.startswith('# initial_rev:'):
-                    initial_rev = line.split(':')[-1].strip()
-                elif line.startswith('# commit:'):
-                    commits.append(line.split(':')[-1].strip())
-
-        if initial_rev:
-            # Find first actually changed revision
-            (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree)
-            newcommits = stdout.split()
-            for i in xrange(min(len(commits), len(newcommits))):
-                if newcommits[i] == commits[i]:
-                    update_rev = commits[i]
-
-    if not initial_rev:
-        logger.error('Unable to find initial revision - please specify it with --initial-rev')
-        return -1
-
-    if not update_rev:
-        update_rev = initial_rev
-
-    # Find list of existing patches in recipe file
     recipefile = _get_recipe_file(tinfoil.cooker, args.recipename)
     if not recipefile:
         # Error already logged
         return -1
     rd = oe.recipeutils.parse_recipe(recipefile, tinfoil.config_data)
-    existing_patches = oe.recipeutils.get_recipe_patches(rd)
 
-    removepatches = []
-    if not args.no_remove:
-        # Get all patches from source tree and check if any should be removed
+    orig_src_uri = rd.getVar('SRC_URI', False) or ''
+    if args.mode == 'auto':
+        if 'git://' in orig_src_uri:
+            mode = 'srcrev'
+        else:
+            mode = 'patch'
+    else:
+        mode = args.mode
+
+    def remove_patches(srcuri, patchlist):
+        # Remove any patches that we don't need
+        updated = False
+        for patch in patchlist:
+            patchfile = os.path.basename(patch)
+            for i in xrange(len(srcuri)):
+                if srcuri[i].startswith('file://') and os.path.basename(srcuri[i]).split(';')[0] == patchfile:
+                    logger.info('Removing patch %s' % patchfile)
+                    srcuri.pop(i)
+                    # FIXME "git rm" here would be nice if the file in question is tracked
+                    # FIXME there's a chance that this file is referred to by another recipe, in which case deleting wouldn't be the right thing to do
+                    if patch.startswith(os.path.dirname(recipefile)):
+                        os.remove(patch)
+                    updated = True
+                    break
+        return updated
+
+    srctree = workspace[args.recipename]
+    if mode == 'srcrev':
+        (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
+        srcrev = stdout.strip()
+        if len(srcrev) != 40:
+            logger.error('Invalid hash returned by git: %s' % stdout)
+            return 1
+
+        logger.info('Updating SRCREV in recipe %s' % os.path.basename(recipefile))
+        patchfields = {}
+        patchfields['SRCREV'] = srcrev
+        if not args.no_remove:
+            # Find list of existing patches in recipe file
+            existing_patches = oe.recipeutils.get_recipe_patches(rd)
+
+            old_srcrev = (rd.getVar('SRCREV', False) or '')
+            tempdir = tempfile.mkdtemp(prefix='devtool')
+            removepatches = []
+            try:
+                GitApplyTree.extractPatches(srctree, old_srcrev, tempdir)
+                newpatches = os.listdir(tempdir)
+                for patch in existing_patches:
+                    patchfile = os.path.basename(patch)
+                    if patchfile in newpatches:
+                        removepatches.append(patch)
+            finally:
+                shutil.rmtree(tempdir)
+            if removepatches:
+                srcuri = (rd.getVar('SRC_URI', False) or '').split()
+                if remove_patches(srcuri, removepatches):
+                    patchfields['SRC_URI'] = ' '.join(srcuri)
+
+        oe.recipeutils.patch_recipe(rd, recipefile, patchfields)
+
+        if not 'git://' in orig_src_uri:
+            logger.info('You will need to update SRC_URI within the recipe to point to a git repository where you have pushed your changes')
+
+    elif mode == 'patch':
+        commits = []
+        update_rev = None
+        if args.initial_rev:
+            initial_rev = args.initial_rev
+        else:
+            initial_rev = None
+            with open(appends[0], 'r') as f:
+                for line in f:
+                    if line.startswith('# initial_rev:'):
+                        initial_rev = line.split(':')[-1].strip()
+                    elif line.startswith('# commit:'):
+                        commits.append(line.split(':')[-1].strip())
+
+            if initial_rev:
+                # Find first actually changed revision
+                (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree)
+                newcommits = stdout.split()
+                for i in xrange(min(len(commits), len(newcommits))):
+                    if newcommits[i] == commits[i]:
+                        update_rev = commits[i]
+
+        if not initial_rev:
+            logger.error('Unable to find initial revision - please specify it with --initial-rev')
+            return -1
+
+        if not update_rev:
+            update_rev = initial_rev
+
+        # Find list of existing patches in recipe file
+        existing_patches = oe.recipeutils.get_recipe_patches(rd)
+
+        removepatches = []
+        if not args.no_remove:
+            # Get all patches from source tree and check if any should be removed
+            tempdir = tempfile.mkdtemp(prefix='devtool')
+            try:
+                GitApplyTree.extractPatches(srctree, initial_rev, tempdir)
+                newpatches = os.listdir(tempdir)
+                for patch in existing_patches:
+                    patchfile = os.path.basename(patch)
+                    if patchfile not in newpatches:
+                        removepatches.append(patch)
+            finally:
+                shutil.rmtree(tempdir)
+
+        # Get updated patches from source tree
         tempdir = tempfile.mkdtemp(prefix='devtool')
         try:
-            GitApplyTree.extractPatches(srctree, initial_rev, tempdir)
+            GitApplyTree.extractPatches(srctree, update_rev, tempdir)
+
+            # Match up and replace existing patches with corresponding new patches
+            updatepatches = False
+            updaterecipe = False
             newpatches = os.listdir(tempdir)
             for patch in existing_patches:
                 patchfile = os.path.basename(patch)
-                if patchfile not in newpatches:
-                    removepatches.append(patch)
+                if patchfile in newpatches:
+                    logger.info('Updating patch %s' % patchfile)
+                    shutil.move(os.path.join(tempdir, patchfile), patch)
+                    newpatches.remove(patchfile)
+                    updatepatches = True
+            srcuri = (rd.getVar('SRC_URI', False) or '').split()
+            if newpatches:
+                # Add any patches left over
+                patchdir = os.path.join(os.path.dirname(recipefile), rd.getVar('BPN', True))
+                bb.utils.mkdirhier(patchdir)
+                for patchfile in newpatches:
+                    logger.info('Adding new patch %s' % patchfile)
+                    shutil.move(os.path.join(tempdir, patchfile), os.path.join(patchdir, patchfile))
+                    srcuri.append('file://%s' % patchfile)
+                    updaterecipe = True
+            if removepatches:
+                if remove_patches(srcuri, removepatches):
+                    updaterecipe = True
+            if updaterecipe:
+                logger.info('Updating recipe %s' % os.path.basename(recipefile))
+                oe.recipeutils.patch_recipe(rd, recipefile, {'SRC_URI': ' '.join(srcuri)})
+            elif not updatepatches:
+                # Neither patches nor recipe were updated
+                logger.info('No patches need updating')
         finally:
             shutil.rmtree(tempdir)
 
-    # Get updated patches from source tree
-    tempdir = tempfile.mkdtemp(prefix='devtool')
-    try:
-        GitApplyTree.extractPatches(srctree, update_rev, tempdir)
-
-        # Match up and replace existing patches with corresponding new patches
-        updatepatches = False
-        updaterecipe = False
-        newpatches = os.listdir(tempdir)
-        for patch in existing_patches:
-            patchfile = os.path.basename(patch)
-            if patchfile in newpatches:
-                logger.info('Updating patch %s' % patchfile)
-                shutil.move(os.path.join(tempdir, patchfile), patch)
-                newpatches.remove(patchfile)
-                updatepatches = True
-        srcuri = (rd.getVar('SRC_URI', False) or '').split()
-        if newpatches:
-            # Add any patches left over
-            patchdir = os.path.join(os.path.dirname(recipefile), rd.getVar('BPN', True))
-            bb.utils.mkdirhier(patchdir)
-            for patchfile in newpatches:
-                logger.info('Adding new patch %s' % patchfile)
-                shutil.move(os.path.join(tempdir, patchfile), os.path.join(patchdir, patchfile))
-                srcuri.append('file://%s' % patchfile)
-                updaterecipe = True
-        if removepatches:
-            # Remove any patches that we don't need
-            for patch in removepatches:
-                patchfile = os.path.basename(patch)
-                for i in xrange(len(srcuri)):
-                    if srcuri[i].startswith('file://') and os.path.basename(srcuri[i]).split(';')[0] == patchfile:
-                        logger.info('Removing patch %s' % patchfile)
-                        srcuri.pop(i)
-                        # FIXME "git rm" here would be nice if the file in question is tracked
-                        # FIXME there's a chance that this file is referred to by another recipe, in which case deleting wouldn't be the right thing to do
-                        os.remove(patch)
-                        updaterecipe = True
-                        break
-        if updaterecipe:
-            logger.info('Updating recipe %s' % os.path.basename(recipefile))
-            oe.recipeutils.patch_recipe(rd, recipefile, {'SRC_URI': ' '.join(srcuri)})
-        elif not updatepatches:
-            # Neither patches nor recipe were updated
-            logger.info('No patches need updating')
-    finally:
-        shutil.rmtree(tempdir)
+    else:
+        logger.error('update_recipe: invalid mode %s' % mode)
+        return 1
 
     return 0
 
@@ -539,9 +597,9 @@ def register_commands(subparsers, context):
     parser_add.set_defaults(func=extract)
 
     parser_add = subparsers.add_parser('update-recipe', help='Apply changes from external source tree to recipe',
-                                       description='Applies changes from external source tree to a recipe (updating/adding/removing patches as necessary)',
-                                       formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+                                       description='Applies changes from external source tree to a recipe (updating/adding/removing patches as necessary, or by updating SRCREV)')
     parser_add.add_argument('recipename', help='Name of recipe to update')
+    parser_add.add_argument('--mode', '-m', choices=['patch', 'srcrev', 'auto'], default='auto', help='Update mode (where %(metavar)s is %(choices)s; default is %(default)s)', metavar='MODE')
     parser_add.add_argument('--initial-rev', help='Starting revision for patches')
     parser_add.add_argument('--no-remove', '-n', action="store_true", help='Don\'t remove patches, only add or update')
     parser_add.set_defaults(func=update_recipe)
-- 
1.9.3

[PATCH 12/15] devtool: prevent extract/modify use with image recipes

[Paul Eggleton]

There wouldn't be any point to using these with an image recipe, so
disallow it.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 3a8c66c..9ab3aa6 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -141,6 +141,10 @@ def _extract_source(srctree, keep_temp, devbranch, d):
         logger.error("The %s recipe uses a shared workdir which this tool does not currently support" % pn)
         return None
 
+    if bb.data.inherits_class('image', d):
+        logger.error("The %s recipe is an image, and therefore is not supported by this tool" % pn)
+        return None
+
     if bb.data.inherits_class('externalsrc', d) and d.getVar('EXTERNALSRC', True):
         logger.error("externalsrc is currently enabled for the %s recipe. This prevents the normal do_patch task from working. You will need to disable this first." % pn)
         return None
@@ -290,6 +294,10 @@ def modify(args, config, basepath, workspace):
         return -1
     rd = oe.recipeutils.parse_recipe(recipefile, tinfoil.config_data)
 
+    if bb.data.inherits_class('image', rd):
+        logger.error("The %s recipe is an image, and therefore is not supported by this tool" % args.recipename)
+        return None
+
     initial_rev = None
     commits = []
     srctree = os.path.abspath(args.srctree)
-- 
1.9.3

[PATCH 13/15] devtool: modify: handle recipes that use a shared workdir

[Paul Eggleton]

If S is outside of WORKDIR then we shouldn't try to get the relative
path in order to work out where the source subdirectory is as we do by
default.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 9ab3aa6..4cb03f3 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -313,12 +313,14 @@ def modify(args, config, basepath, workspace):
             (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=args.srctree)
             initial_rev = stdout.rstrip()
 
-    # Handle if S is set to a subdirectory of the source
+    # Check that recipe isn't using a shared workdir
     s = rd.getVar('S', True)
     workdir = rd.getVar('WORKDIR', True)
-    if s != workdir and os.path.dirname(s) != workdir:
-        srcsubdir = os.sep.join(os.path.relpath(s, workdir).split(os.sep)[1:])
-        srctree = os.path.join(srctree, srcsubdir)
+    if s.startswith(workdir):
+        # Handle if S is set to a subdirectory of the source
+        if s != workdir and os.path.dirname(s) != workdir:
+            srcsubdir = os.sep.join(os.path.relpath(s, workdir).split(os.sep)[1:])
+            srctree = os.path.join(srctree, srcsubdir)
 
     appendpath = os.path.join(config.workspace_path, 'appends')
     if not os.path.exists(appendpath):
-- 
1.9.3

[PATCH 14/15] devtool: support source extraction for recipes that use a shared workdir

[Paul Eggleton]

Enable source extraction used by devtool extract / devtool modify -x for
recipes that use a shared workdir (e.g. the kernel and gcc).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 4cb03f3..bf18aae 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -137,10 +137,6 @@ def _extract_source(srctree, keep_temp, devbranch, d):
         logger.error("The perf recipe does not actually check out source and thus cannot be supported by this tool")
         return None
 
-    if 'work-shared' in d.getVar('S', True):
-        logger.error("The %s recipe uses a shared workdir which this tool does not currently support" % pn)
-        return None
-
     if bb.data.inherits_class('image', d):
         logger.error("The %s recipe is an image, and therefore is not supported by this tool" % pn)
         return None
@@ -169,6 +165,16 @@ def _extract_source(srctree, keep_temp, devbranch, d):
         workdir = os.path.join(tempdir, 'workdir')
         crd.setVar('WORKDIR', workdir)
         crd.setVar('T', os.path.join(tempdir, 'temp'))
+        if not crd.getVar('S', True).startswith(workdir):
+            # Usually a shared workdir recipe (kernel, gcc)
+            # Try to set a reasonable default
+            if bb.data.inherits_class('kernel', d):
+                crd.setVar('S', '${WORKDIR}/source')
+            else:
+                crd.setVar('S', '${WORKDIR}/${BP}')
+        if bb.data.inherits_class('kernel', d):
+            # We don't want to move the source to STAGING_KERNEL_DIR here
+            crd.setVar('STAGING_KERNEL_DIR', '${S}')
 
         # FIXME: This is very awkward. Unfortunately it's not currently easy to properly
         # execute tasks outside of bitbake itself, until then this has to suffice if we
@@ -204,7 +210,10 @@ def _extract_source(srctree, keep_temp, devbranch, d):
             else:
                 os.rmdir(patchdir)
 
-        if not bb.data.inherits_class('kernel-yocto', d):
+        if bb.data.inherits_class('kernel-yocto', d):
+            (stdout, _) = bb.process.run('git --git-dir="%s" rev-parse HEAD' % crd.expand('${WORKDIR}/git'), cwd=srcsubdir)
+            initial_rev = stdout.rstrip()
+        else:
             if not os.listdir(srcsubdir):
                 logger.error("no source unpacked to S, perhaps the %s recipe doesn't use any source?" % pn)
                 return None
-- 
1.9.3

[PATCH 15/15] devtool: fix broken clones of git recipes

[Paul Eggleton]

Because we move the workdir when extracting source, then move the source
and delete the temporary workdir, you lose the indirection symlink
pointed to by the alternates file (which is created when the fetcher
clones it from DL_DIR with -s) and the resulting repository is broken.
In any case, for a source repo that the user may put their own changes
into, we can't really rely on a clone made with -s in case the
original goes away - because of cleanall, DL_DIR disappearing, etc. So
repack the repository so that it is a complete, non-shared clone after
unpacking.

(While I'm at it, add a test for devtool modify with a git recipe which
verifies that this works.)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py | 33 +++++++++++++++++++++++++++++++++
 scripts/lib/devtool/standard.py   |  7 +++++++
 2 files changed, 40 insertions(+)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index f147f24..932d6b9 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -227,6 +227,39 @@ class DevtoolTests(oeSelfTest):
         matches = glob.glob(stampprefix + '*')
         self.assertFalse(matches, 'Stamp files exist for recipe mdadm that should have been cleaned')
 
+    def test_devtool_modify_git(self):
+        # Check preconditions
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        testrecipe = 'mkelfimage'
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        self.assertIn('git://', src_uri, 'This test expects the %s recipe to be a git recipe' % testrecipe)
+        # Clean up anything in the workdir/sysroot/sstate cache
+        bitbake('%s -c cleansstate' % testrecipe)
+        # Try modifying a recipe
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        self.add_command_to_tearDown('bitbake -c clean %s' % testrecipe)
+        result = runCmd('devtool modify %s -x %s' % (testrecipe, tempdir))
+        self.assertTrue(os.path.exists(os.path.join(tempdir, 'Makefile')), 'Extracted source could not be found')
+        self.assertTrue(os.path.isdir(os.path.join(tempdir, '.git')), 'git repository for external source tree not found')
+        self.assertTrue(os.path.exists(os.path.join(workspacedir, 'conf', 'layer.conf')), 'Workspace directory not created')
+        matches = glob.glob(os.path.join(workspacedir, 'appends', 'mkelfimage_*.bbappend'))
+        self.assertTrue(matches, 'bbappend not created')
+        # Test devtool status
+        result = runCmd('devtool status')
+        self.assertIn(testrecipe, result.output)
+        self.assertIn(tempdir, result.output)
+        # Check git repo
+        result = runCmd('git status --porcelain', cwd=tempdir)
+        self.assertEqual(result.output.strip(), "", 'Created git repo is not clean')
+        result = runCmd('git symbolic-ref HEAD', cwd=tempdir)
+        self.assertEqual(result.output.strip(), "refs/heads/devtool", 'Wrong branch in git repo')
+        # Try building
+        bitbake(testrecipe)
+
     def test_devtool_update_recipe(self):
         # Check preconditions
         workspacedir = os.path.join(self.builddir, 'workspace')
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index bf18aae..cabf3fe 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -202,6 +202,13 @@ def _extract_source(srctree, keep_temp, devbranch, d):
             # Handle if S is set to a subdirectory of the source
             srcsubdir = os.path.join(workdir, os.path.relpath(srcsubdir, workdir).split(os.sep)[0])
 
+        if os.path.exists(os.path.join(srcsubdir, '.git')):
+            alternatesfile = os.path.join(srcsubdir, '.git', 'objects', 'info', 'alternates')
+            if os.path.exists(alternatesfile):
+                # This will have been cloned with -s, so we need to convert it to a full clone
+                bb.process.run('git repack -a', cwd=srcsubdir)
+                os.remove(alternatesfile)
+
         patchdir = os.path.join(srcsubdir, 'patches')
         haspatches = False
         if os.path.exists(patchdir):
-- 
1.9.3

Re: [PATCH 00/15] devtool and externalsrc improvements

[Bruce Ashfield]

On Thu, Feb 19, 2015 at 11:39 AM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> I've been attempting to enable usage of devtool with the kernel (which
> effectively means using the externalsrc class with the kernel). This
> doesn't really work yet due to lack of handling the kernel configuration,

I've always insisted/argued and designed in that externalsrc shouldn't touch the
kernel configuration at all.

You are building a tree that is completely managed outside of the build system,
and getting the kernel configuration in place is part of that management.

So I'm not convinced it is a problem that needs to be solved.

Cheers,

Bruce

> so you still can't use externalsrc properly with the kernel, but this
> at least gets us further - you can now extract the source successfully
> and run a build. I also discovered and fixed a number of related issues
> along the way, and made a few behavioural improvements in both devtool
> and externalsrc.
>
>
> The following changes since commit ad016fa7492ba23c751cf33dd1befde7782facfa:
>
>   lzo: Upgrade to 2.09 and add ptest (2015-02-19 07:50:44 +0000)
>
> are available in the git repository at:
>
>   git://git.openembedded.org/openembedded-core-contrib paule/devtool-fixes
>   http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool-fixes
>
> Paul Eggleton (15):
>   recipetool: fix regression caused by previous commit
>   lib/oe/patch: fix PATCHTOOL = "git" with source in a subdirectory
>   classes/externalsrc: fix source being wiped out on clean with kernel
>   classes/externalsrc: add workaround for recipes that use SRCPV in PV
>   classes/externalsrc: show a warning on compiling
>   classes/kernel-yocto: extend SRCTREECOVEREDTASKS
>   devtool: add/modify: add option to build in same directory
>   devtool: reset: run bitbake -c clean on recipe
>   devtool: build: run do_populate_sysroot instead of do_install
>   lib/oe/recipeutils: minor improvements to patch_recipe()
>   devtool: update-recipe: add handling for git recipes
>   devtool: prevent extract/modify use with image recipes
>   devtool: modify: handle recipes that use a shared workdir
>   devtool: support source extraction for recipes that use a shared
>     workdir
>   devtool: fix broken clones of git recipes
>
>  meta/classes/externalsrc.bbclass  |  24 ++++
>  meta/classes/kernel-yocto.bbclass |   2 +-
>  meta/lib/oe/patch.py              |   7 +-
>  meta/lib/oe/recipeutils.py        |   4 +-
>  meta/lib/oeqa/selftest/devtool.py | 147 ++++++++++++++++++-
>  scripts/lib/devtool/standard.py   | 292 +++++++++++++++++++++++++-------------
>  scripts/lib/recipetool/create.py  |   8 +-
>  7 files changed, 377 insertions(+), 107 deletions(-)
>
> --
> 1.9.3
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: [PATCH 00/15] devtool and externalsrc improvements

[Paul Eggleton]

On Thursday 19 February 2015 11:47:01 Bruce Ashfield wrote:
> On Thu, Feb 19, 2015 at 11:39 AM, Paul Eggleton
> <paul.eggleton@linux.intel.com> wrote:
> > I've been attempting to enable usage of devtool with the kernel (which
> > effectively means using the externalsrc class with the kernel). This
> > doesn't really work yet due to lack of handling the kernel configuration,
> 
> I've always insisted/argued and designed in that externalsrc shouldn't touch
> the kernel configuration at all.
> 
> You are building a tree that is completely managed outside of the build
> system, and getting the kernel configuration in place is part of that
> management.
> 
> So I'm not convinced it is a problem that needs to be solved.

So, externalsrc still doesn't touch the configuration, and it is not my plan to 
make it do so. However, users do need some way of setting up that configuration 
when checking out those sources for the first time, and if that's to be of any 
use to someone who just wants to make some minor changes, it should be the 
exact same configuration you get when you build the recipe as normal. That is 
what I believe "devtool modify" (at least with -x) should be doing, given that 
it is one means you can use to set up a development environment if you do not 
have one already.

We can certainly talk about the mechanism and work out the best way to do it, 
such that it fits in with how linux-yocto works and meets user expectations. My 
intent with this series was to get us to the point where we can at least 
experiment and have that discussion.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 00/15] devtool and externalsrc improvements

[Richard Purdie]

On Thu, 2015-02-19 at 16:55 +0000, Paul Eggleton wrote:
> On Thursday 19 February 2015 11:47:01 Bruce Ashfield wrote:
> > On Thu, Feb 19, 2015 at 11:39 AM, Paul Eggleton
> > <paul.eggleton@linux.intel.com> wrote:
> > > I've been attempting to enable usage of devtool with the kernel (which
> > > effectively means using the externalsrc class with the kernel). This
> > > doesn't really work yet due to lack of handling the kernel configuration,
> > 
> > I've always insisted/argued and designed in that externalsrc shouldn't touch
> > the kernel configuration at all.
> > 
> > You are building a tree that is completely managed outside of the build
> > system, and getting the kernel configuration in place is part of that
> > management.
> > 
> > So I'm not convinced it is a problem that needs to be solved.
> 
> So, externalsrc still doesn't touch the configuration, and it is not my plan to 
> make it do so. However, users do need some way of setting up that configuration 
> when checking out those sources for the first time, and if that's to be of any 
> use to someone who just wants to make some minor changes, it should be the 
> exact same configuration you get when you build the recipe as normal. That is 
> what I believe "devtool modify" (at least with -x) should be doing, given that 
> it is one means you can use to set up a development environment if you do not 
> have one already.
> 
> We can certainly talk about the mechanism and work out the best way to do it, 
> such that it fits in with how linux-yocto works and meets user expectations. My 
> intent with this series was to get us to the point where we can at least 
> experiment and have that discussion.

Having the option of triggering the standard configuration process as
per a usual linux-yocto build but on an external direction, standalone
may be the best option?

Cheers,

Richard

Re: [PATCH 00/15] devtool and externalsrc improvements

[Bruce Ashfield]

On Thu, Feb 19, 2015 at 12:00 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Thu, 2015-02-19 at 16:55 +0000, Paul Eggleton wrote:
>> On Thursday 19 February 2015 11:47:01 Bruce Ashfield wrote:
>> > On Thu, Feb 19, 2015 at 11:39 AM, Paul Eggleton
>> > <paul.eggleton@linux.intel.com> wrote:
>> > > I've been attempting to enable usage of devtool with the kernel (which
>> > > effectively means using the externalsrc class with the kernel). This
>> > > doesn't really work yet due to lack of handling the kernel configuration,
>> >
>> > I've always insisted/argued and designed in that externalsrc shouldn't touch
>> > the kernel configuration at all.
>> >
>> > You are building a tree that is completely managed outside of the build
>> > system, and getting the kernel configuration in place is part of that
>> > management.
>> >
>> > So I'm not convinced it is a problem that needs to be solved.
>>
>> So, externalsrc still doesn't touch the configuration, and it is not my plan to
>> make it do so. However, users do need some way of setting up that configuration
>> when checking out those sources for the first time, and if that's to be of any
>> use to someone who just wants to make some minor changes, it should be the
>> exact same configuration you get when you build the recipe as normal. That is
>> what I believe "devtool modify" (at least with -x) should be doing, given that
>> it is one means you can use to set up a development environment if you do not
>> have one already.
>>
>> We can certainly talk about the mechanism and work out the best way to do it,
>> such that it fits in with how linux-yocto works and meets user expectations. My
>> intent with this series was to get us to the point where we can at least
>> experiment and have that discussion.
>
> Having the option of triggering the standard configuration process as
> per a usual linux-yocto build but on an external direction, standalone
> may be the best option?

I did try that before, and the tools to run on random trees (I use it
for building
the linux-yocto kernel source out of tree), but they need the fragments and
meta-data to do that configuration.

and when a externalsrc tree is unpacked and checked out, that data won't
by lying around (that being said, I haven't tried the series to see what
exactly is happening).

I always considered externalsrc to be: "here's my tree, I've set it up and
it is ready to build .. go do it!", so I need to think about the best way to
get that next step done (for any kernel, not just linux yocto).

Bruce

>
> Cheers,
>
> Richard
>



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: [PATCH 00/15] devtool and externalsrc improvements

[Bruce Ashfield]

On Thu, Feb 19, 2015 at 11:55 AM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> On Thursday 19 February 2015 11:47:01 Bruce Ashfield wrote:
>> On Thu, Feb 19, 2015 at 11:39 AM, Paul Eggleton
>> <paul.eggleton@linux.intel.com> wrote:
>> > I've been attempting to enable usage of devtool with the kernel (which
>> > effectively means using the externalsrc class with the kernel). This
>> > doesn't really work yet due to lack of handling the kernel configuration,
>>
>> I've always insisted/argued and designed in that externalsrc shouldn't touch
>> the kernel configuration at all.
>>
>> You are building a tree that is completely managed outside of the build
>> system, and getting the kernel configuration in place is part of that
>> management.
>>
>> So I'm not convinced it is a problem that needs to be solved.
>
> So, externalsrc still doesn't touch the configuration, and it is not my plan to
> make it do so. However, users do need some way of setting up that configuration
> when checking out those sources for the first time, and if that's to be of any

Right. Something to think on.

> use to someone who just wants to make some minor changes, it should be the
> exact same configuration you get when you build the recipe as normal. That is
> what I believe "devtool modify" (at least with -x) should be doing, given that
> it is one means you can use to set up a development environment if you do not
> have one already.
>
> We can certainly talk about the mechanism and work out the best way to do it,
> such that it fits in with how linux-yocto works and meets user expectations. My
> intent with this series was to get us to the point where we can at least
> experiment and have that discussion.

Agreed.

Bruce

>
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"