[PATCH 0/4] Fixes for square brackets in installed filenames

[PATCH 0/4] Fixes for square brackets in installed filenames

[Paul Eggleton]

When attempting to create a recipe for something that installs files
with square brackets in their name I stumbled across a couple of issues,
here are the fixes.


The following changes since commit 5e21afc9395060b489156d3f90505a372b713f37:

  Revert "selftest/wic: extending test coverage for WIC script options" (2016-12-20 17:06:01 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/bracket-fixes
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/bracket-fixes

Paul Eggleton (4):
  classes/package_rpm: handle square brackets in filenames
  lib/oe/path: add warning comment about oe.path.remove() with wildcarded filenames
  classes/sstate: handle filenames containing square brackets
  classes/sstate: fix file conflict message

 meta/classes/package_rpm.bbclass |  6 ++++++
 meta/classes/sstate.bbclass      | 24 +++++++++++++-----------
 meta/lib/oe/path.py              |  9 ++++++++-
 3 files changed, 27 insertions(+), 12 deletions(-)

-- 
2.5.5

[PATCH 1/4] classes/package_rpm: handle square brackets in filenames

[Paul Eggleton]

When constructing a spec file we list files and directory paths in the
%files section. If ] or [ characters are in a file or directory name,
rpm treats them as wildcards which will mean it won't properly match the
filename. Instead, transform these into an ? wildcard so they don't
cause a problem.

(This fixes packaging the npm package "file-set" and anything that
happens to depend upon it, since it includes tests with files that
contain unusual characters including ] and [).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/package_rpm.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 638cc1b..b9f049e 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -197,6 +197,8 @@ python write_specfile () {
             if path.endswith("DEBIAN") or path.endswith("CONTROL"):
                 continue
             path = path.replace("%", "%%%%%%%%")
+            path = path.replace("[", "?")
+            path = path.replace("]", "?")
 
             # Treat all symlinks to directories as normal files.
             # os.walk() lists them as directories.
@@ -216,6 +218,8 @@ python write_specfile () {
                     if dir == "CONTROL" or dir == "DEBIAN":
                         continue
                     dir = dir.replace("%", "%%%%%%%%")
+                    dir = dir.replace("[", "?")
+                    dir = dir.replace("]", "?")
                     # All packages own the directories their files are in...
                     target.append('%dir "' + path + '/' + dir + '"')
             else:
@@ -230,6 +234,8 @@ python write_specfile () {
                 if file == "CONTROL" or file == "DEBIAN":
                     continue
                 file = file.replace("%", "%%%%%%%%")
+                file = file.replace("[", "?")
+                file = file.replace("]", "?")
                 if conffiles.count(path + '/' + file):
                     target.append('%config "' + path + '/' + file + '"')
                 else:
-- 
2.5.5

[PATCH 2/4] lib/oe/path: add warning comment about oe.path.remove() with wildcarded filenames

[Paul Eggleton]

Add a warning in the doc comment for oe.path.remove() about using that
function on paths that may contain wildcards in the actual
file/directory names.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/path.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index 804ecd5..d468540 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -92,7 +92,14 @@ def copyhardlinktree(src, dst):
         copytree(src, dst)
 
 def remove(path, recurse=True):
-    """Equivalent to rm -f or rm -rf"""
+    """
+    Equivalent to rm -f or rm -rf
+    NOTE: be careful about passing paths that may contain filenames with
+    wildcards in them (as opposed to passing an actual wildcarded path) -
+    since we use glob.glob() to expand the path. Filenames containing
+    square brackets are particularly problematic since the they may not
+    actually expand to match the original filename.
+    """
     for name in glob.glob(path):
         try:
             os.unlink(name)
-- 
2.5.5

[PATCH 3/4] classes/sstate: handle filenames containing square brackets

[Paul Eggleton]

If a recipe installs a file or directory whose name contains square
brackets [ ] that form a valid glob expression and that file then they
won't be correctly removed from the sysroot, because we pass each path
in the sstate manifest to our oe.path.remove() function which calls
glob.glob() on the path passed into it and the expression won't
actually match the original filename. Since we don't expect to put any
wildcarded expressions in the sstate manifests, and we already have a
try...except around this, we can actually use os.remove() here instead.

Similarly, when we pass existing file paths to "grep" looking through
the manifests, we don't want those paths to be treated as regexes - so
use grep's -F command line switch.

Fixes [YOCTO #10836].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/sstate.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index a7cd9ec..755bf59 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -213,7 +213,7 @@ def sstate_install(ss, d):
                     break
             if realmatch:
                 match.append(f)
-                sstate_search_cmd = "grep -rl '%s' %s --exclude=master.list | sed -e 's:^.*/::' -e 's:\.populate-sysroot::'" % (f, d.expand("${SSTATE_MANIFESTS}"))
+                sstate_search_cmd = "grep -rlF '%s' %s --exclude=master.list | sed -e 's:^.*/::' -e 's:\.populate-sysroot::'" % (f, d.expand("${SSTATE_MANIFESTS}"))
                 search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0]
                 if search_output != "":
                     match.append("Matched in %s" % search_output.rstrip())
@@ -406,7 +406,7 @@ def sstate_clean_manifest(manifest, d):
                 elif os.path.exists(entry) and len(os.listdir(entry)) == 0:
                     os.rmdir(entry[:-1])
             else:
-                oe.path.remove(entry)
+                os.remove(entry)
         except OSError:
             pass
 
-- 
2.5.5

[PATCH 4/4] classes/sstate: fix file conflict message

[Paul Eggleton]

* Fix not decoding output from grep ("Matched in b'manifest...')
* Fix showing "Matched in b''" if no match (show "not matched to any
  task" instead)
* Drop the filtering out of .populate-sysroot from matched manifest
  names - it should have been .populate_sysroot so it doesn't work, and
  in any case the value of removing the task name is questionable given
  that we aren't removing it for any other task, and that the rest of
  the filename isn't only the task name, we might as well have the whole
  thing. At least then you can do a find on that exact name without
  wildcards and find it.
* Fix indenting of file list entries and indent "matched in" further
  underneath
* Minor punctuation fixes

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/sstate.bbclass | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 755bf59..afdb4a3 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -213,25 +213,27 @@ def sstate_install(ss, d):
                     break
             if realmatch:
                 match.append(f)
-                sstate_search_cmd = "grep -rlF '%s' %s --exclude=master.list | sed -e 's:^.*/::' -e 's:\.populate-sysroot::'" % (f, d.expand("${SSTATE_MANIFESTS}"))
+                sstate_search_cmd = "grep -rlF '%s' %s --exclude=master.list | sed -e 's:^.*/::'" % (f, d.expand("${SSTATE_MANIFESTS}"))
                 search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0]
-                if search_output != "":
-                    match.append("Matched in %s" % search_output.rstrip())
+                if search_output:
+                    match.append("  (matched in %s)" % search_output.decode('utf-8').rstrip())
+                else:
+                    match.append("  (not matched to any task)")
     if match:
         bb.error("The recipe %s is trying to install files into a shared " \
           "area when those files already exist. Those files and their manifest " \
-          "location are:\n   %s\nPlease verify which recipe should provide the " \
-          "above files.\nThe build has stopped as continuing in this scenario WILL " \
-          "break things, if not now, possibly in the future (we've seen builds fail " \
+          "location are:\n  %s\nPlease verify which recipe should provide the " \
+          "above files.\n\nThe build has stopped, as continuing in this scenario WILL " \
+          "break things - if not now, possibly in the future (we've seen builds fail " \
           "several months later). If the system knew how to recover from this " \
-          "automatically it would however there are several different scenarios " \
+          "automatically it would, however there are several different scenarios " \
           "which can result in this and we don't know which one this is. It may be " \
           "you have switched providers of something like virtual/kernel (e.g. from " \
           "linux-yocto to linux-yocto-dev), in that case you need to execute the " \
           "clean task for both recipes and it will resolve this error. It may be " \
           "you changed DISTRO_FEATURES from systemd to udev or vice versa. Cleaning " \
-          "those recipes should again resolve this error however switching " \
-          "DISTRO_FEATURES on an existing build directory is not supported, you " \
+          "those recipes should again resolve this error, however switching " \
+          "DISTRO_FEATURES on an existing build directory is not supported - you " \
           "should really clean out tmp and rebuild (reusing sstate should be safe). " \
           "It could be the overlapping files detected are harmless in which case " \
           "adding them to SSTATE_DUPWHITELIST may be the correct solution. It could " \
@@ -239,7 +241,7 @@ def sstate_install(ss, d):
           "things (e.g. bluez 4 and bluez 5 and the correct solution for that would " \
           "be to resolve the conflict. If in doubt, please ask on the mailing list, " \
           "sharing the error and filelist above." % \
-          (d.getVar('PN'), "\n ".join(match)))
+          (d.getVar('PN'), "\n  ".join(match)))
         bb.fatal("If the above message is too much, the simpler version is you're advised to wipe out tmp and rebuild (reusing sstate is fine). That will likely fix things in most (but not all) cases.")
 
     # Write out the manifest
-- 
2.5.5