* [PATCH 1/3] lib/oe/buildhistory_analysis: drop related field feature
2018-07-11 14:56 [PATCH 0/3] Buildhistory fixes Paul Eggleton
@ 2018-07-11 14:56 ` Paul Eggleton
2018-07-11 14:56 ` [PATCH 2/3] classes/buildhistory: handle packaged files with names containing spaces Paul Eggleton
2018-07-11 14:56 ` [PATCH 3/3] classes/buildhistory: properly process escaped chars from pkgdata Paul Eggleton
2 siblings, 0 replies; 5+ messages in thread
From: Paul Eggleton @ 2018-07-11 14:56 UTC (permalink / raw)
To: openembedded-core
The original idea here was that changes to certain fields might be able
to be explained if there was a change to another field, for example if
RDEPENDS changed it might be because DEPENDS changed. Thus we were
printing this kind of thing out with each change. Unfortunately in
practice this turned out to be noisy and not particularly useful, so we
might as well remove it.
Fixes [YOCTO #7336].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/buildhistory_analysis.py | 26 --------------------------
1 file changed, 26 deletions(-)
diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index b0365abcedf..ff7815d7c94 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -31,13 +31,6 @@ ver_monitor_fields = ['PKGE', 'PKGV', 'PKGR']
monitor_numeric_threshold = 10
# Image files to monitor (note that image-info.txt is handled separately)
img_monitor_files = ['installed-package-names.txt', 'files-in-image.txt']
-# Related context fields for reporting (note: PE, PV & PR are always reported for monitored package fields)
-related_fields = {}
-related_fields['RDEPENDS'] = ['DEPENDS']
-related_fields['RRECOMMENDS'] = ['DEPENDS']
-related_fields['FILELIST'] = ['FILES']
-related_fields['files-in-image.txt'] = ['installed-package-names.txt', 'USER_CLASSES', 'IMAGE_CLASSES', 'ROOTFS_POSTPROCESS_COMMAND', 'IMAGE_POSTPROCESS_COMMAND']
-related_fields['installed-package-names.txt'] = ['IMAGE_FEATURES', 'IMAGE_LINGUAS', 'IMAGE_INSTALL', 'BAD_RECOMMENDATIONS', 'NO_RECOMMENDATIONS', 'PACKAGE_EXCLUDE']
colours = {
'colour_default': '',
@@ -67,7 +60,6 @@ class ChangeRecord:
self.oldvalue = oldvalue
self.newvalue = newvalue
self.monitored = monitored
- self.related = []
self.filechanges = None
def __str__(self):
@@ -206,13 +198,6 @@ class ChangeRecord:
else:
out = '{} changed from "{colour_remove}{}{colour_default}" to "{colour_add}{}{colour_default}"'.format(self.fieldname, self.oldvalue, self.newvalue, **colours)
- if self.related:
- for chg in self.related:
- if not outer and chg.fieldname in ['PE', 'PV', 'PR']:
- continue
- for line in chg._str_internal(False).splitlines():
- out += '\n * %s' % line
-
return '%s%s' % (prefix, out) if out else ''
class FileChange:
@@ -635,17 +620,6 @@ def process_changes(repopath, revision1, revision2='HEAD', report_all=False, rep
chg = ChangeRecord(path, filename[7:], d.a_blob.data_stream.read().decode('utf-8'), '', True)
changes.append(chg)
- # Link related changes
- for chg in changes:
- if chg.monitored:
- for chg2 in changes:
- # (Check dirname in the case of fields from recipe info files)
- if chg.path == chg2.path or os.path.dirname(chg.path) == chg2.path:
- if chg2.fieldname in related_fields.get(chg.fieldname, []):
- chg.related.append(chg2)
- elif chg.path == chg2.path and chg.path.startswith('packages/') and chg2.fieldname in ['PE', 'PV', 'PR']:
- chg.related.append(chg2)
-
# filter out unwanted paths
if exclude_path:
for chg in changes:
--
2.17.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH 2/3] classes/buildhistory: handle packaged files with names containing spaces
2018-07-11 14:56 [PATCH 0/3] Buildhistory fixes Paul Eggleton
2018-07-11 14:56 ` [PATCH 1/3] lib/oe/buildhistory_analysis: drop related field feature Paul Eggleton
@ 2018-07-11 14:56 ` Paul Eggleton
2018-07-11 14:56 ` [PATCH 3/3] classes/buildhistory: properly process escaped chars from pkgdata Paul Eggleton
2 siblings, 0 replies; 5+ messages in thread
From: Paul Eggleton @ 2018-07-11 14:56 UTC (permalink / raw)
To: openembedded-core
The FILELIST field of the package info file in the buildhistory
repository is a space-separated list of all of the files in the package.
If a name of a file packaged by a recipe contains a space character then
of course the result was that we didn't handle its name properly. To fix
that, use quotes around any filename containing spaces and at the other
end use these quotes to extract the proper entries.
Fixes [YOCTO #12742].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/buildhistory.bbclass | 3 ++-
meta/lib/oe/buildhistory_analysis.py | 16 ++++++++++++----
2 files changed, 14 insertions(+), 5 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 63980f72a52..2e5213e66ec 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -77,6 +77,7 @@ python buildhistory_emit_pkghistory() {
import re
import json
+ import shlex
import errno
pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE')
@@ -287,7 +288,7 @@ python buildhistory_emit_pkghistory() {
dictval = json.loads(val)
filelist = list(dictval.keys())
filelist.sort()
- pkginfo.filelist = " ".join(filelist)
+ pkginfo.filelist = " ".join([shlex.quote(x) for x in filelist])
pkginfo.size = int(pkgdata['PKGSIZE'])
diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index ff7815d7c94..ad7fceb8bbc 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -13,6 +13,7 @@ import os.path
import difflib
import git
import re
+import shlex
import hashlib
import collections
import bb.utils
@@ -115,10 +116,13 @@ class ChangeRecord:
aitems = pkglist_combine(depvera)
bitems = pkglist_combine(depverb)
else:
- aitems = self.oldvalue.split()
- bitems = self.newvalue.split()
if self.fieldname == 'FILELIST':
+ aitems = shlex.split(self.oldvalue)
+ bitems = shlex.split(self.newvalue)
renamed_dirs, aitems, bitems = detect_renamed_dirs(aitems, bitems)
+ else:
+ aitems = self.oldvalue.split()
+ bitems = self.newvalue.split()
removed = list(set(aitems) - set(bitems))
added = list(set(bitems) - set(aitems))
@@ -409,9 +413,13 @@ def compare_dict_blobs(path, ablob, bblob, report_all, report_ver):
(depvera, depverb) = compare_pkg_lists(astr, bstr)
if depvera == depverb:
continue
- alist = astr.split()
+ if key == 'FILELIST':
+ alist = shlex.split(astr)
+ blist = shlex.split(bstr)
+ else:
+ alist = astr.split()
+ blist = bstr.split()
alist.sort()
- blist = bstr.split()
blist.sort()
# We don't care about the removal of self-dependencies
if pkgname in alist and not pkgname in blist:
--
2.17.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH 3/3] classes/buildhistory: properly process escaped chars from pkgdata
2018-07-11 14:56 [PATCH 0/3] Buildhistory fixes Paul Eggleton
2018-07-11 14:56 ` [PATCH 1/3] lib/oe/buildhistory_analysis: drop related field feature Paul Eggleton
2018-07-11 14:56 ` [PATCH 2/3] classes/buildhistory: handle packaged files with names containing spaces Paul Eggleton
@ 2018-07-11 14:56 ` Paul Eggleton
2 siblings, 0 replies; 5+ messages in thread
From: Paul Eggleton @ 2018-07-11 14:56 UTC (permalink / raw)
To: openembedded-core
All values written out to pkgdata are escaped (see write_if_exists() in
package.bbclass). In practice there tend not to be characters that need
escaping except in the scriptlets (pkg_preinst, pkg_postinst, pkg_prerm
and pkg_postrm) where currently we still see the escape codes in the
corresponding files within buildhistory (e.g. \n and \t) and thus also
in the output of buildhistory-diff, hindering proper diffing of changes.
To fix this, when we read values from pkgdata and write them out to
buildhistory, we need to interpret the escape codes by doing the exact
reverse of what we do in package.bbclass.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/buildhistory.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 2e5213e66ec..40b292b139f 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -245,7 +245,7 @@ python buildhistory_emit_pkghistory() {
key = item[0]
if key.endswith('_' + pkg):
key = key[:-len(pkg)-1]
- pkgdata[key] = item[1]
+ pkgdata[key] = item[1].encode('latin-1').decode('unicode_escape')
pkge = pkgdata.get('PKGE', '0')
pkgv = pkgdata['PKGV']
--
2.17.1
^ permalink raw reply related [flat|nested] 5+ messages in thread