[PATCH 0/7] A bunch of fixes for devtool update-recipe

[PATCH 0/7] A bunch of fixes for devtool update-recipe

[Paul Eggleton]

A bunch of fixes for devtool update-recipe and associated oe-selftest tests.

The following changes since commit 9303d8055c45a0f6af295d70a6f6a8b9d8d8a7c9:

  devtool: add "rename" subcommand (2016-11-07 11:04:17 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/devtool23-oe
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool23-oe

Paul Eggleton (7):
  devtool: update-recipe: check output before treating it as a string
  oe-selftest: devtool: test update-recipe with only local files
  lib/oe/patch: fix handling of patches with no header
  devtool: update-recipe: fix handling of compressed local patches
  devtool: update-recipe: support replacing remote patches
  lib/oe/recipeutils: ignore archives by default in
    get_recipe_local_files()
  oe-selftest: devtool: test that updating a file with subdir= works

 .../recipes-test/devtool/devtool-test-localonly.bb |   6 ++
 .../devtool/devtool-test-localonly/file1           |   1 +
 .../devtool/devtool-test-localonly/file2           |   1 +
 .../recipes-test/devtool/devtool-test-patch-gz.bb  |  16 ++++
 .../devtool/devtool-test-patch-gz/readme.patch.gz  | Bin 0 -> 449 bytes
 .../recipes-test/devtool/devtool-test-subdir.bb    |   7 ++
 .../devtool-test-subdir/devtool-test-subdir.tar.gz | Bin 0 -> 181 bytes
 .../devtool/devtool-test-subdir/testfile           |   1 +
 meta/classes/patch.bbclass                         |   9 ++-
 meta/lib/oe/patch.py                               |   8 +-
 meta/lib/oe/recipeutils.py                         |  22 +++--
 meta/lib/oeqa/selftest/devtool.py                  |  67 +++++++++++++++
 scripts/lib/devtool/standard.py                    |  90 +++++++++++++++------
 13 files changed, 193 insertions(+), 35 deletions(-)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-localonly.bb
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-localonly/file1
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-localonly/file2
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-patch-gz.bb
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-patch-gz/readme.patch.gz
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-subdir.bb
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-subdir/devtool-test-subdir.tar.gz
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-subdir/testfile

-- 
2.5.5

[PATCH 1/7] devtool: update-recipe: check output before treating it as a string

[Paul Eggleton]

As of the move to Python 3 and the fixes we applied at that time,
bb.process.run() will return a byte array of length 0 rather than an
empty string if the output is empty. That may be a bug that we should
fix, but for now it's easiest to just check the result here before
treating it as a string. This fixes running "devtool update-recipe" or
"devtool finish" on a recipe which has no source tree, for example
initramfs-framework.

Fixes [YOCTO #10563].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 4523048..1511641 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -332,10 +332,11 @@ def _git_ls_tree(repodir, treeish='HEAD', recursive=False):
         cmd.append('-r')
     out, _ = bb.process.run(cmd, cwd=repodir)
     ret = {}
-    for line in out.split('\0'):
-        if line:
-            split = line.split(None, 4)
-            ret[split[3]] = split[0:3]
+    if out:
+        for line in out.split('\0'):
+            if line:
+                split = line.split(None, 4)
+                ret[split[3]] = split[0:3]
     return ret
 
 def _git_exclude_path(srctree, path):
-- 
2.5.5

[PATCH 2/7] oe-selftest: devtool: test update-recipe with only local files

[Paul Eggleton]

Add a test to ensure devtool update-recipe works properly on recipes
that contain only local files (since the other tests we have didn't test
that).

Relates to [YOCTO #10563].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 .../recipes-test/devtool/devtool-test-localonly.bb     |  6 ++++++
 .../recipes-test/devtool/devtool-test-localonly/file1  |  1 +
 .../recipes-test/devtool/devtool-test-localonly/file2  |  1 +
 meta/lib/oeqa/selftest/devtool.py                      | 18 ++++++++++++++++++
 4 files changed, 26 insertions(+)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-localonly.bb
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-localonly/file1
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-localonly/file2

diff --git a/meta-selftest/recipes-test/devtool/devtool-test-localonly.bb b/meta-selftest/recipes-test/devtool/devtool-test-localonly.bb
new file mode 100644
index 0000000..28ff49e
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/devtool-test-localonly.bb
@@ -0,0 +1,6 @@
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI = "file://file1 \
+           file://file2"
+
diff --git a/meta-selftest/recipes-test/devtool/devtool-test-localonly/file1 b/meta-selftest/recipes-test/devtool/devtool-test-localonly/file1
new file mode 100644
index 0000000..f4bdcfc
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/devtool-test-localonly/file1
@@ -0,0 +1 @@
+The first file
diff --git a/meta-selftest/recipes-test/devtool/devtool-test-localonly/file2 b/meta-selftest/recipes-test/devtool/devtool-test-localonly/file2
new file mode 100644
index 0000000..a7e2414
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/devtool-test-localonly/file2
@@ -0,0 +1 @@
+The second file
diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 46f5a0b..f6226c1 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -930,6 +930,24 @@ class DevtoolTests(DevtoolBase):
                            ('??', '.*/0001-Add-new-file.patch$')]
         self._check_repo_status(os.path.dirname(recipefile), expected_status)
 
+    def test_devtool_update_recipe_local_files_3(self):
+        # First, modify the recipe
+        testrecipe = 'devtool-test-localonly'
+        recipefile = get_bb_var('FILE', testrecipe)
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        # (don't bother with cleaning the recipe on teardown, we won't be building it)
+        result = runCmd('devtool modify %s' % testrecipe)
+        # Modify one file
+        runCmd('echo "Another line" >> file2', cwd=os.path.join(self.workspacedir, 'sources', testrecipe, 'oe-local-files'))
+        self.add_command_to_tearDown('cd %s; rm %s/*; git checkout %s %s' % (os.path.dirname(recipefile), testrecipe, testrecipe, os.path.basename(recipefile)))
+        result = runCmd('devtool update-recipe %s' % testrecipe)
+        expected_status = [(' M', '.*/%s/file2$' % testrecipe)]
+        self._check_repo_status(os.path.dirname(recipefile), expected_status)
+
     @testcase(1163)
     def test_devtool_extract(self):
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
-- 
2.5.5

[PATCH 3/7] lib/oe/patch: fix handling of patches with no header

[Paul Eggleton]

If a patch applied by a recipe has no header and we turn the recipe's
source into a git tree (when PATCHTOOL = "git" or when using devtool
extract / modify / upgrade), the commit message ends up consisting only
of the original filename marker ("%% original patch: filename.patch").
When we come to do turn the commits back into a set of patches in
extractPatches(), this first line ends up in the "Subject: " part of
the file, but we were ignoring it because the line didn't start with the
marker text. The end result was we weren't able to get the original
patch name. Strip off any "Subject [PATCH x/y]" part before looking for
the marker text to fix.

This caused "devtool modify openssl" followed by "devtool update-recipe
openssl" (without any changes in-between) to remove version-script.patch
because that patch has no header and we weren't able to determine the
original filename.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/patch.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index 0332f10..dbefd28 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -428,6 +428,7 @@ class GitApplyTree(PatchTree):
     def extractPatches(tree, startcommit, outdir, paths=None):
         import tempfile
         import shutil
+        import re
         tempdir = tempfile.mkdtemp(prefix='oepatch')
         try:
             shellcmd = ["git", "format-patch", startcommit, "-o", tempdir]
@@ -443,10 +444,13 @@ class GitApplyTree(PatchTree):
                         try:
                             with open(srcfile, 'r', encoding=encoding) as f:
                                 for line in f:
-                                    if line.startswith(GitApplyTree.patch_line_prefix):
+                                    checkline = line
+                                    if checkline.startswith('Subject: '):
+                                        checkline = re.sub(r'\[.+?\]\s*', '', checkline[9:])
+                                    if checkline.startswith(GitApplyTree.patch_line_prefix):
                                         outfile = line.split()[-1].strip()
                                         continue
-                                    if line.startswith(GitApplyTree.ignore_commit_prefix):
+                                    if checkline.startswith(GitApplyTree.ignore_commit_prefix):
                                         continue
                                     patchlines.append(line)
                         except UnicodeDecodeError:
-- 
2.5.5

[PATCH 4/7] devtool: update-recipe: fix handling of compressed local patches

[Paul Eggleton]

It is possible to use gzip or bzip2 to compress patches and still refer
to them in compressed form in the SRC_URI value within a recipe. If you
run "devtool modify" on such a recipe, make changes to the commit for
the patch and then run devtool update-recipe, we need to correctly
associate the commit back to the compressed patch file and re-compress
the patch, neither of which we were doing previously.

Additionally, add an oe-selftest test to ensure this doesn't regress in
future.

Fixes [YOCTO #8278].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 .../recipes-test/devtool/devtool-test-patch-gz.bb  |  16 ++++++
 .../devtool/devtool-test-patch-gz/readme.patch.gz  | Bin 0 -> 449 bytes
 meta/classes/patch.bbclass                         |   9 ++--
 meta/lib/oe/recipeutils.py                         |   6 +--
 meta/lib/oeqa/selftest/devtool.py                  |  24 +++++++++
 scripts/lib/devtool/standard.py                    |  55 ++++++++++++++-------
 6 files changed, 85 insertions(+), 25 deletions(-)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-patch-gz.bb
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-patch-gz/readme.patch.gz

diff --git a/meta-selftest/recipes-test/devtool/devtool-test-patch-gz.bb b/meta-selftest/recipes-test/devtool/devtool-test-patch-gz.bb
new file mode 100644
index 0000000..719a5f1
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/devtool-test-patch-gz.bb
@@ -0,0 +1,16 @@
+LICENSE = "GPLv2+"
+LIC_FILES_CHKSUM = "file://COPYING;md5=94d55d512a9ba36caa9b7df079bae19f"
+
+DEPENDS = "libxres libxext virtual/libx11 ncurses"
+
+SRC_URI = "http://downloads.yoctoproject.org/releases/xrestop/xrestop-0.4.tar.gz \
+           file://readme.patch.gz \
+           "
+
+S = "${WORKDIR}/xrestop-0.4"
+
+SRC_URI[md5sum] = "d8a54596cbaf037e62b80c4585a3ca9b"
+SRC_URI[sha256sum] = "67c2fc94a7ecedbaae0d1837e82e93d1d98f4a6d759828860e552119af3ce257"
+
+inherit autotools pkgconfig
+
diff --git a/meta-selftest/recipes-test/devtool/devtool-test-patch-gz/readme.patch.gz b/meta-selftest/recipes-test/devtool/devtool-test-patch-gz/readme.patch.gz
new file mode 100644
index 0000000000000000000000000000000000000000..4752492ccd661e2c360d8fe2ca4dcef61dea89ce
GIT binary patch
literal 449
zcmV;y0Y3g8iwFpRQ6X3W19D|yWNl?GaA9;~XaJQ_O>5jR5WVYHyjxnlc4f!1V^fNk
zWJ7aE3E3?KLJs~Id)3;KYs+B+{qddME-gJ44dTsc=FQC0?PLdJlx8wZ;!344qp+#@
zPUA$yDr(w9R#{f3l}d1L4Yo8QOd(}{GDK2_;+AQ0Jd{h1P1p70IPv4iZa`<Z=(Sl6
zfi{l%pti#&aZ@_Vv0G?`D}1r9%oHX}orfw9FX4rkQi|<TeWTjtcz#&#u0OwE-83kT
zb$gn?lmutN4kjv_frR)v&W`cmZ(Zq@ImewT$8oabq`4oS9nDUg?nQ{L?hG})ZCk&3
z``_Vx-xqk?tZ(i&_=yOfClI35J)z#0rlV%XQEg_LoVI3u=e-xA(QOOg@3cet;apsu
zfzYO*1JWwKl4%(Pm5OSmDuhxdaeVCer$%_5_qVPn;D@WDUMq#75O{i|d1puH8EafP
zJ6^C*-;YAs9<7})SfibtSOq-cy4PeJc+OW=bfVcVQ%zXrvZDZk&z<A&k45v_N$Dd$
r=-|lAyjdVT;qCU^jP0L~(dLx?d(410fQSMWC?S3U{t;{<bpikYONih}

literal 0
HcmV?d00001

diff --git a/meta/classes/patch.bbclass b/meta/classes/patch.bbclass
index 1f6927b..2c1f58c 100644
--- a/meta/classes/patch.bbclass
+++ b/meta/classes/patch.bbclass
@@ -10,13 +10,13 @@ PATCH_GIT_USER_EMAIL ?= "oe.patch@oe"
 
 inherit terminal
 
-def src_patches(d, all = False ):
+def src_patches(d, all=False, expand=True):
     workdir = d.getVar('WORKDIR', True)
     fetch = bb.fetch2.Fetch([], d)
     patches = []
     sources = []
     for url in fetch.urls:
-        local = patch_path(url, fetch, workdir)
+        local = patch_path(url, fetch, workdir, expand)
         if not local:
             if all:
                 local = fetch.localpath(url)
@@ -55,13 +55,14 @@ def src_patches(d, all = False ):
 
     return patches
 
-def patch_path(url, fetch, workdir):
+def patch_path(url, fetch, workdir, expand=True):
     """Return the local path of a patch, or None if this isn't a patch"""
 
     local = fetch.localpath(url)
     base, ext = os.path.splitext(os.path.basename(local))
     if ext in ('.gz', '.bz2', '.Z'):
-        local = os.path.join(workdir, base)
+        if expand:
+            local = os.path.join(workdir, base)
         ext = os.path.splitext(base)[1]
 
     urldata = fetch.ud[url]
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 6caae5f..ab4177a 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -397,7 +397,7 @@ def get_recipe_local_files(d, patches=False):
     for uri in uris:
         if fetch.ud[uri].type == 'file':
             if (not patches and
-                    bb.utils.exec_flat_python_func('patch_path', uri, fetch, '')):
+                    bb.utils.exec_flat_python_func('patch_path', uri, fetch, '', expand=False)):
                 continue
             # Skip files that are referenced by absolute path
             fname = fetch.ud[uri].basepath
@@ -418,7 +418,7 @@ def get_recipe_patches(d):
     patchfiles = []
     # Execute src_patches() defined in patch.bbclass - this works since that class
     # is inherited globally
-    patches = bb.utils.exec_flat_python_func('src_patches', d)
+    patches = bb.utils.exec_flat_python_func('src_patches', d, expand=False)
     for patch in patches:
         _, _, local, _, _, parm = bb.fetch.decodeurl(patch)
         patchfiles.append(local)
@@ -437,7 +437,7 @@ def get_recipe_patched_files(d):
     import oe.patch
     # Execute src_patches() defined in patch.bbclass - this works since that class
     # is inherited globally
-    patches = bb.utils.exec_flat_python_func('src_patches', d)
+    patches = bb.utils.exec_flat_python_func('src_patches', d, expand=False)
     patchedfiles = {}
     for patch in patches:
         _, _, patchfile, _, _, parm = bb.fetch.decodeurl(patch)
diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index f6226c1..92dc5e5 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -948,6 +948,30 @@ class DevtoolTests(DevtoolBase):
         expected_status = [(' M', '.*/%s/file2$' % testrecipe)]
         self._check_repo_status(os.path.dirname(recipefile), expected_status)
 
+    def test_devtool_update_recipe_local_patch_gz(self):
+        # First, modify the recipe
+        testrecipe = 'devtool-test-patch-gz'
+        recipefile = get_bb_var('FILE', testrecipe)
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        # (don't bother with cleaning the recipe on teardown, we won't be building it)
+        result = runCmd('devtool modify %s' % testrecipe)
+        # Modify one file
+        srctree = os.path.join(self.workspacedir, 'sources', testrecipe)
+        runCmd('echo "Another line" >> README', cwd=srctree)
+        runCmd('git commit -a --amend --no-edit', cwd=srctree)
+        self.add_command_to_tearDown('cd %s; rm %s/*; git checkout %s %s' % (os.path.dirname(recipefile), testrecipe, testrecipe, os.path.basename(recipefile)))
+        result = runCmd('devtool update-recipe %s' % testrecipe)
+        expected_status = [(' M', '.*/%s/readme.patch.gz$' % testrecipe)]
+        self._check_repo_status(os.path.dirname(recipefile), expected_status)
+        patch_gz = os.path.join(os.path.dirname(recipefile), testrecipe, 'readme.patch.gz')
+        result = runCmd('file %s' % patch_gz)
+        if 'gzip compressed data' not in result.output:
+            self.fail('New patch file is not gzipped - file reports:\n%s' % result.output)
+
     @testcase(1163)
     def test_devtool_extract(self):
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 1511641..af0d467 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1128,7 +1128,7 @@ def _remove_source_files(append, files, destpath):
                     raise
 
 
-def _export_patches(srctree, rd, start_rev, destdir):
+def _export_patches(srctree, rd, start_rev, destdir, changed_revs=None):
     """Export patches from srctree to given location.
        Returns three-tuple of dicts:
          1. updated - patches that already exist in SRCURI
@@ -1157,18 +1157,44 @@ def _export_patches(srctree, rd, start_rev, destdir):
         # revision This does assume that people are using unique shortlog
         # values, but they ought to be anyway...
         new_basename = seqpatch_re.match(new_patch).group(2)
-        found = False
+        match_name = None
         for old_patch in existing_patches:
             old_basename = seqpatch_re.match(old_patch).group(2)
-            if new_basename == old_basename:
-                updated[new_patch] = existing_patches.pop(old_patch)
-                found = True
-                # Rename patch files
-                if new_patch != old_patch:
-                    os.rename(os.path.join(destdir, new_patch),
-                              os.path.join(destdir, old_patch))
+            old_basename_splitext = os.path.splitext(old_basename)
+            if old_basename.endswith(('.gz', '.bz2', '.Z')) and old_basename_splitext[0] == new_basename:
+                old_patch_noext = os.path.splitext(old_patch)[0]
+                match_name = old_patch_noext
                 break
-        if not found:
+            elif new_basename == old_basename:
+                match_name = old_patch
+                break
+        if match_name:
+            # Rename patch files
+            if new_patch != match_name:
+                os.rename(os.path.join(destdir, new_patch),
+                          os.path.join(destdir, match_name))
+            # Need to pop it off the list now before checking changed_revs
+            oldpath = existing_patches.pop(old_patch)
+            if changed_revs is not None:
+                # Avoid updating patches that have not actually changed
+                with open(os.path.join(destdir, match_name), 'r') as f:
+                    firstlineitems = f.readline().split()
+                    # Looking for "From <hash>" line
+                    if len(firstlineitems) > 1 and len(firstlineitems[1]) == 40:
+                        if not firstlineitems[1] in changed_revs:
+                            continue
+            # Recompress if necessary
+            if oldpath.endswith(('.gz', '.Z')):
+                bb.process.run(['gzip', match_name], cwd=destdir)
+                if oldpath.endswith('.gz'):
+                    match_name += '.gz'
+                else:
+                    match_name += '.Z'
+            elif oldpath.endswith('.bz2'):
+                bb.process.run(['bzip2', match_name], cwd=destdir)
+                match_name += '.bz2'
+            updated[match_name] = oldpath
+        else:
             added[new_patch] = None
     return (updated, added, existing_patches)
 
@@ -1415,7 +1441,7 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
         # Get updated patches from source tree
         patches_dir = tempfile.mkdtemp(dir=tempdir)
         upd_p, new_p, del_p = _export_patches(srctree, rd, update_rev,
-                                              patches_dir)
+                                              patches_dir, changed_revs)
         updatefiles = False
         updaterecipe = False
         destpath = None
@@ -1453,13 +1479,6 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
                 updatefiles = True
             for basepath, path in upd_p.items():
                 patchfn = os.path.join(patches_dir, basepath)
-                if changed_revs is not None:
-                    # Avoid updating patches that have not actually changed
-                    with open(patchfn, 'r') as f:
-                        firstlineitems = f.readline().split()
-                        if len(firstlineitems) > 1 and len(firstlineitems[1]) == 40:
-                            if not firstlineitems[1] in changed_revs:
-                                continue
                 logger.info('Updating patch %s' % basepath)
                 _move_file(patchfn, path)
                 updatefiles = True
-- 
2.5.5

[PATCH 5/7] devtool: update-recipe: support replacing remote patches

[Paul Eggleton]

If you have a patch remotely fetched in a recipe (e.g. from an http
server) that needs updating then add a local version and substitute the
entry in SRC_URI to point to it.

One can argue about how desirable it is to be modifying patches fetched
in this way, but then one can argue about how desirable it is to have
such patches in the recipe in the first place - and in any case if
devtool update-recipe is to correctly transfer changes to such patches
made in the git repository within the source tree to the recipe then
there isn't much choice but to do it this way.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index af0d467..34de7bd 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1104,6 +1104,15 @@ def _remove_file_entries(srcuri, filelist):
                 break
     return entries, remaining
 
+def _replace_srcuri_entry(srcuri, filename, newentry):
+    """Replace entry corresponding to specified file with a new entry"""
+    basename = os.path.basename(filename)
+    for i in range(len(srcuri)):
+        if os.path.basename(srcuri[i].split(';')[0]) == basename:
+            srcuri.pop(i)
+            srcuri.insert(i, newentry)
+            break
+
 def _remove_source_files(append, files, destpath):
     """Unlink existing patch files"""
     for path in files:
@@ -1424,6 +1433,10 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
         raise DevtoolError('Unable to find initial revision - please specify '
                            'it with --initial-rev')
 
+    dl_dir = rd.getVar('DL_DIR', True)
+    if not dl_dir.endswith('/'):
+        dl_dir += '/'
+
     tempdir = tempfile.mkdtemp(prefix='devtool')
     try:
         local_files_dir = tempfile.mkdtemp(dir=tempdir)
@@ -1468,6 +1481,7 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
                 logger.info('No patches or local source files needed updating')
         else:
             # Update existing files
+            files_dir = _determine_files_dir(rd)
             for basepath, path in upd_f.items():
                 logger.info('Updating file %s' % basepath)
                 if os.path.isabs(basepath):
@@ -1479,11 +1493,19 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
                 updatefiles = True
             for basepath, path in upd_p.items():
                 patchfn = os.path.join(patches_dir, basepath)
-                logger.info('Updating patch %s' % basepath)
+                if os.path.dirname(path) + '/' == dl_dir:
+                    # This is a a downloaded patch file - we now need to
+                    # replace the entry in SRC_URI with our local version
+                    logger.info('Replacing remote patch %s with updated local version' % basepath)
+                    path = os.path.join(files_dir, basepath)
+                    _replace_srcuri_entry(srcuri, basepath, 'file://%s' % basepath)
+                    updaterecipe = True
+                else:
+                    logger.info('Updating patch %s' % basepath)
+                logger.debug('Moving new patch %s to %s' % (patchfn, path))
                 _move_file(patchfn, path)
                 updatefiles = True
             # Add any new files
-            files_dir = _determine_files_dir(rd)
             for basepath, path in new_f.items():
                 logger.info('Adding new file %s' % basepath)
                 _move_file(os.path.join(local_files_dir, basepath),
-- 
2.5.5

[PATCH 6/7] lib/oe/recipeutils: ignore archives by default in get_recipe_local_files()

[Paul Eggleton]

By default, have get_recipe_local_files() not return any archive
files. This prevents a local tarball from being erroneously removed
from SRC_URI if you run "devtool modify" on a recipe followed by
"devtool update-recipe". It doesn't actually help you to directly
update the contents of such tarballs, but at least now it won't break
the recipe.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/recipeutils.py | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index ab4177a..1589feb 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -389,10 +389,15 @@ def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True):
     return copied, remotes
 
 
-def get_recipe_local_files(d, patches=False):
+def get_recipe_local_files(d, patches=False, archives=False):
     """Get a list of local files in SRC_URI within a recipe."""
     uris = (d.getVar('SRC_URI', True) or "").split()
     fetch = bb.fetch2.Fetch(uris, d)
+    # FIXME this list should be factored out somewhere else (such as the
+    # fetcher) though note that this only encompasses actual container formats
+    # i.e. that can contain multiple files as opposed to those that only
+    # contain a compressed stream (i.e. .tar.gz as opposed to just .gz)
+    archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z']
     ret = {}
     for uri in uris:
         if fetch.ud[uri].type == 'file':
@@ -409,7 +414,14 @@ def get_recipe_local_files(d, patches=False):
                 if os.path.isabs(subdir):
                     continue
                 fname = os.path.join(subdir, fname)
-            ret[fname] = fetch.localpath(uri)
+            localpath = fetch.localpath(uri)
+            if not archives:
+                # Ignore archives that will be unpacked
+                if localpath.endswith(tuple(archive_exts)):
+                    unpack = fetch.ud[uri].parm.get('unpack', True)
+                    if unpack:
+                        continue
+            ret[fname] = localpath
     return ret
 
 
-- 
2.5.5

[PATCH 7/7] oe-selftest: devtool: test that updating a file with subdir= works

[Paul Eggleton]

If you have a file:// entry in SRC_URI with a subdir= parameter that
makes it extract into the source tree, then when you update that file in
oe-local-files and run devtool update-recipe then you want the original
file to be updated. This was made to work by OE-Core commit
9069fef5dad5a873c8a8f720f7bcbc7625556309 together with
31f1bbad248c36a8c86dde4ff57ce42efc664082, however until now there was no
oe-selftest test to verify it.

Note that in order to succeed this test also requires the fix
"lib/oe/recipeutils: ignore archives by default in
get_recipe_local_files()" since the test recipe uses a local tarball.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 .../recipes-test/devtool/devtool-test-subdir.bb    |   7 ++++++
 .../devtool-test-subdir/devtool-test-subdir.tar.gz | Bin 0 -> 181 bytes
 .../devtool/devtool-test-subdir/testfile           |   1 +
 meta/lib/oeqa/selftest/devtool.py                  |  25 +++++++++++++++++++++
 4 files changed, 33 insertions(+)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-subdir.bb
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-subdir/devtool-test-subdir.tar.gz
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-subdir/testfile

diff --git a/meta-selftest/recipes-test/devtool/devtool-test-subdir.bb b/meta-selftest/recipes-test/devtool/devtool-test-subdir.bb
new file mode 100644
index 0000000..1a163f8
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/devtool-test-subdir.bb
@@ -0,0 +1,7 @@
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI = "file://devtool-test-subdir.tar.gz \
+           file://testfile;subdir=${BPN}"
+
+S = "${WORKDIR}/${BPN}"
diff --git a/meta-selftest/recipes-test/devtool/devtool-test-subdir/devtool-test-subdir.tar.gz b/meta-selftest/recipes-test/devtool/devtool-test-subdir/devtool-test-subdir.tar.gz
new file mode 100644
index 0000000000000000000000000000000000000000..3d44f803cb996f47ddb9566a6a2c278bd2fd38a7
GIT binary patch
literal 181
zcmb2|=3q$6J|51%{Pv<F*C7Le)`x3sUG)NdZmC(erLQpYQrL27neF~E#f~elloTHQ
z&rMTkto#sVzkFZ6>!)O`V=r&CCF&amo1dGRnw5U#@gCt_C1KIoKB3%8wHJFS`o2@h
zUS??8Jmt|PpTgJocYl|^y6(&VH_HWT*GYRHymoE<Z~d>q|Cg=lnV=Qkpj&_Me<H{8
f3#NhR>Li@+zuhOx!GHt~9AEb)s!EYTgMk46=?79i

literal 0
HcmV?d00001

diff --git a/meta-selftest/recipes-test/devtool/devtool-test-subdir/testfile b/meta-selftest/recipes-test/devtool/devtool-test-subdir/testfile
new file mode 100644
index 0000000..12b519c
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/devtool-test-subdir/testfile
@@ -0,0 +1 @@
+Modified version
diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 92dc5e5..2a08721 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -972,6 +972,31 @@ class DevtoolTests(DevtoolBase):
         if 'gzip compressed data' not in result.output:
             self.fail('New patch file is not gzipped - file reports:\n%s' % result.output)
 
+    def test_devtool_update_recipe_local_files_subdir(self):
+        # Try devtool extract on a recipe that has a file with subdir= set in
+        # SRC_URI such that it overwrites a file that was in an archive that
+        # was also in SRC_URI
+        # First, modify the recipe
+        testrecipe = 'devtool-test-subdir'
+        recipefile = get_bb_var('FILE', testrecipe)
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        # (don't bother with cleaning the recipe on teardown, we won't be building it)
+        result = runCmd('devtool modify %s' % testrecipe)
+        testfile = os.path.join(self.workspacedir, 'sources', testrecipe, 'testfile')
+        self.assertTrue(os.path.exists(testfile), 'Extracted source could not be found')
+        with open(testfile, 'r') as f:
+            contents = f.read().rstrip()
+        self.assertEqual(contents, 'Modified version', 'File has apparently not been overwritten as it should have been')
+        # Test devtool update-recipe without modifying any files
+        self.add_command_to_tearDown('cd %s; rm %s/*; git checkout %s %s' % (os.path.dirname(recipefile), testrecipe, testrecipe, os.path.basename(recipefile)))
+        result = runCmd('devtool update-recipe %s' % testrecipe)
+        expected_status = []
+        self._check_repo_status(os.path.dirname(recipefile), expected_status)
+
     @testcase(1163)
     def test_devtool_extract(self):
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
-- 
2.5.5