* [PATCH 1/4] classes/package_rpm: handle square brackets in filenames
2016-12-22 2:19 [PATCH 0/4] Fixes for square brackets in installed filenames Paul Eggleton
@ 2016-12-22 2:19 ` Paul Eggleton
2016-12-22 2:19 ` [PATCH 2/4] lib/oe/path: add warning comment about oe.path.remove() with wildcarded filenames Paul Eggleton
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Paul Eggleton @ 2016-12-22 2:19 UTC (permalink / raw)
To: openembedded-core
When constructing a spec file we list files and directory paths in the
%files section. If ] or [ characters are in a file or directory name,
rpm treats them as wildcards which will mean it won't properly match the
filename. Instead, transform these into an ? wildcard so they don't
cause a problem.
(This fixes packaging the npm package "file-set" and anything that
happens to depend upon it, since it includes tests with files that
contain unusual characters including ] and [).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/package_rpm.bbclass | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 638cc1b..b9f049e 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -197,6 +197,8 @@ python write_specfile () {
if path.endswith("DEBIAN") or path.endswith("CONTROL"):
continue
path = path.replace("%", "%%%%%%%%")
+ path = path.replace("[", "?")
+ path = path.replace("]", "?")
# Treat all symlinks to directories as normal files.
# os.walk() lists them as directories.
@@ -216,6 +218,8 @@ python write_specfile () {
if dir == "CONTROL" or dir == "DEBIAN":
continue
dir = dir.replace("%", "%%%%%%%%")
+ dir = dir.replace("[", "?")
+ dir = dir.replace("]", "?")
# All packages own the directories their files are in...
target.append('%dir "' + path + '/' + dir + '"')
else:
@@ -230,6 +234,8 @@ python write_specfile () {
if file == "CONTROL" or file == "DEBIAN":
continue
file = file.replace("%", "%%%%%%%%")
+ file = file.replace("[", "?")
+ file = file.replace("]", "?")
if conffiles.count(path + '/' + file):
target.append('%config "' + path + '/' + file + '"')
else:
--
2.5.5
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH 2/4] lib/oe/path: add warning comment about oe.path.remove() with wildcarded filenames
2016-12-22 2:19 [PATCH 0/4] Fixes for square brackets in installed filenames Paul Eggleton
2016-12-22 2:19 ` [PATCH 1/4] classes/package_rpm: handle square brackets in filenames Paul Eggleton
@ 2016-12-22 2:19 ` Paul Eggleton
2016-12-22 2:19 ` [PATCH 3/4] classes/sstate: handle filenames containing square brackets Paul Eggleton
2016-12-22 2:19 ` [PATCH 4/4] classes/sstate: fix file conflict message Paul Eggleton
3 siblings, 0 replies; 5+ messages in thread
From: Paul Eggleton @ 2016-12-22 2:19 UTC (permalink / raw)
To: openembedded-core
Add a warning in the doc comment for oe.path.remove() about using that
function on paths that may contain wildcards in the actual
file/directory names.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/path.py | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index 804ecd5..d468540 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -92,7 +92,14 @@ def copyhardlinktree(src, dst):
copytree(src, dst)
def remove(path, recurse=True):
- """Equivalent to rm -f or rm -rf"""
+ """
+ Equivalent to rm -f or rm -rf
+ NOTE: be careful about passing paths that may contain filenames with
+ wildcards in them (as opposed to passing an actual wildcarded path) -
+ since we use glob.glob() to expand the path. Filenames containing
+ square brackets are particularly problematic since the they may not
+ actually expand to match the original filename.
+ """
for name in glob.glob(path):
try:
os.unlink(name)
--
2.5.5
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH 3/4] classes/sstate: handle filenames containing square brackets
2016-12-22 2:19 [PATCH 0/4] Fixes for square brackets in installed filenames Paul Eggleton
2016-12-22 2:19 ` [PATCH 1/4] classes/package_rpm: handle square brackets in filenames Paul Eggleton
2016-12-22 2:19 ` [PATCH 2/4] lib/oe/path: add warning comment about oe.path.remove() with wildcarded filenames Paul Eggleton
@ 2016-12-22 2:19 ` Paul Eggleton
2016-12-22 2:19 ` [PATCH 4/4] classes/sstate: fix file conflict message Paul Eggleton
3 siblings, 0 replies; 5+ messages in thread
From: Paul Eggleton @ 2016-12-22 2:19 UTC (permalink / raw)
To: openembedded-core
If a recipe installs a file or directory whose name contains square
brackets [ ] that form a valid glob expression and that file then they
won't be correctly removed from the sysroot, because we pass each path
in the sstate manifest to our oe.path.remove() function which calls
glob.glob() on the path passed into it and the expression won't
actually match the original filename. Since we don't expect to put any
wildcarded expressions in the sstate manifests, and we already have a
try...except around this, we can actually use os.remove() here instead.
Similarly, when we pass existing file paths to "grep" looking through
the manifests, we don't want those paths to be treated as regexes - so
use grep's -F command line switch.
Fixes [YOCTO #10836].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/sstate.bbclass | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index a7cd9ec..755bf59 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -213,7 +213,7 @@ def sstate_install(ss, d):
break
if realmatch:
match.append(f)
- sstate_search_cmd = "grep -rl '%s' %s --exclude=master.list | sed -e 's:^.*/::' -e 's:\.populate-sysroot::'" % (f, d.expand("${SSTATE_MANIFESTS}"))
+ sstate_search_cmd = "grep -rlF '%s' %s --exclude=master.list | sed -e 's:^.*/::' -e 's:\.populate-sysroot::'" % (f, d.expand("${SSTATE_MANIFESTS}"))
search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0]
if search_output != "":
match.append("Matched in %s" % search_output.rstrip())
@@ -406,7 +406,7 @@ def sstate_clean_manifest(manifest, d):
elif os.path.exists(entry) and len(os.listdir(entry)) == 0:
os.rmdir(entry[:-1])
else:
- oe.path.remove(entry)
+ os.remove(entry)
except OSError:
pass
--
2.5.5
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH 4/4] classes/sstate: fix file conflict message
2016-12-22 2:19 [PATCH 0/4] Fixes for square brackets in installed filenames Paul Eggleton
` (2 preceding siblings ...)
2016-12-22 2:19 ` [PATCH 3/4] classes/sstate: handle filenames containing square brackets Paul Eggleton
@ 2016-12-22 2:19 ` Paul Eggleton
3 siblings, 0 replies; 5+ messages in thread
From: Paul Eggleton @ 2016-12-22 2:19 UTC (permalink / raw)
To: openembedded-core
* Fix not decoding output from grep ("Matched in b'manifest...')
* Fix showing "Matched in b''" if no match (show "not matched to any
task" instead)
* Drop the filtering out of .populate-sysroot from matched manifest
names - it should have been .populate_sysroot so it doesn't work, and
in any case the value of removing the task name is questionable given
that we aren't removing it for any other task, and that the rest of
the filename isn't only the task name, we might as well have the whole
thing. At least then you can do a find on that exact name without
wildcards and find it.
* Fix indenting of file list entries and indent "matched in" further
underneath
* Minor punctuation fixes
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/sstate.bbclass | 22 ++++++++++++----------
1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 755bf59..afdb4a3 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -213,25 +213,27 @@ def sstate_install(ss, d):
break
if realmatch:
match.append(f)
- sstate_search_cmd = "grep -rlF '%s' %s --exclude=master.list | sed -e 's:^.*/::' -e 's:\.populate-sysroot::'" % (f, d.expand("${SSTATE_MANIFESTS}"))
+ sstate_search_cmd = "grep -rlF '%s' %s --exclude=master.list | sed -e 's:^.*/::'" % (f, d.expand("${SSTATE_MANIFESTS}"))
search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0]
- if search_output != "":
- match.append("Matched in %s" % search_output.rstrip())
+ if search_output:
+ match.append(" (matched in %s)" % search_output.decode('utf-8').rstrip())
+ else:
+ match.append(" (not matched to any task)")
if match:
bb.error("The recipe %s is trying to install files into a shared " \
"area when those files already exist. Those files and their manifest " \
- "location are:\n %s\nPlease verify which recipe should provide the " \
- "above files.\nThe build has stopped as continuing in this scenario WILL " \
- "break things, if not now, possibly in the future (we've seen builds fail " \
+ "location are:\n %s\nPlease verify which recipe should provide the " \
+ "above files.\n\nThe build has stopped, as continuing in this scenario WILL " \
+ "break things - if not now, possibly in the future (we've seen builds fail " \
"several months later). If the system knew how to recover from this " \
- "automatically it would however there are several different scenarios " \
+ "automatically it would, however there are several different scenarios " \
"which can result in this and we don't know which one this is. It may be " \
"you have switched providers of something like virtual/kernel (e.g. from " \
"linux-yocto to linux-yocto-dev), in that case you need to execute the " \
"clean task for both recipes and it will resolve this error. It may be " \
"you changed DISTRO_FEATURES from systemd to udev or vice versa. Cleaning " \
- "those recipes should again resolve this error however switching " \
- "DISTRO_FEATURES on an existing build directory is not supported, you " \
+ "those recipes should again resolve this error, however switching " \
+ "DISTRO_FEATURES on an existing build directory is not supported - you " \
"should really clean out tmp and rebuild (reusing sstate should be safe). " \
"It could be the overlapping files detected are harmless in which case " \
"adding them to SSTATE_DUPWHITELIST may be the correct solution. It could " \
@@ -239,7 +241,7 @@ def sstate_install(ss, d):
"things (e.g. bluez 4 and bluez 5 and the correct solution for that would " \
"be to resolve the conflict. If in doubt, please ask on the mailing list, " \
"sharing the error and filelist above." % \
- (d.getVar('PN'), "\n ".join(match)))
+ (d.getVar('PN'), "\n ".join(match)))
bb.fatal("If the above message is too much, the simpler version is you're advised to wipe out tmp and rebuild (reusing sstate is fine). That will likely fix things in most (but not all) cases.")
# Write out the manifest
--
2.5.5
^ permalink raw reply related [flat|nested] 5+ messages in thread