* [PATCH 1/7] classes/buildhistory: squash spaces out of image variables
2012-02-14 13:40 [PATCH 0/7] buildhistory fixes Paul Eggleton
@ 2012-02-14 13:40 ` Paul Eggleton
2012-02-14 13:40 ` [PATCH 2/7] classes/buildhistory: sort and de-dupe dependency graphs Paul Eggleton
` (7 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Paul Eggleton @ 2012-02-14 13:40 UTC (permalink / raw)
To: openembedded-core
Values of image variables that are lists (e.g. IMAGE_INSTALL) are easier
to read if there are no extraneous spaces in them, so ensure that there
is only one space between each item.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/buildhistory.bbclass | 16 ++++++++++++----
1 files changed, 12 insertions(+), 4 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 1b6b249..d1a9670 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -123,9 +123,6 @@ python buildhistory_emit_pkghistory() {
except EnvironmentError:
return None
- def squashspaces(string):
- return re.sub("\s+", " ", string)
-
def sortpkglist(string):
pkgiter = re.finditer(r'[a-zA-Z0-9.-]+( \([><=]+ [^ )]+\))?', string, 0)
pkglist = [p.group(0) for p in pkgiter]
@@ -349,12 +346,23 @@ def buildhistory_get_layers(d):
return layertext
+def squashspaces(string):
+ import re
+ return re.sub("\s+", " ", string).strip()
+
+
def buildhistory_get_imagevars(d):
imagevars = "DISTRO DISTRO_VERSION USER_CLASSES IMAGE_CLASSES IMAGE_FEATURES IMAGE_LINGUAS IMAGE_INSTALL BAD_RECOMMENDATIONS ROOTFS_POSTPROCESS_COMMAND IMAGE_POSTPROCESS_COMMAND"
+ listvars = "USER_CLASSES IMAGE_CLASSES IMAGE_FEATURES IMAGE_LINGUAS IMAGE_INSTALL BAD_RECOMMENDATIONS"
+ imagevars = imagevars.split()
+ listvars = listvars.split()
ret = ""
- for var in imagevars.split():
+ for var in imagevars:
value = d.getVar(var, True) or ""
+ if var in listvars:
+ # Squash out spaces
+ value = squashspaces(value)
ret += "%s = %s\n" % (var, value)
return ret.rstrip('\n')
--
1.7.5.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 2/7] classes/buildhistory: sort and de-dupe dependency graphs
2012-02-14 13:40 [PATCH 0/7] buildhistory fixes Paul Eggleton
2012-02-14 13:40 ` [PATCH 1/7] classes/buildhistory: squash spaces out of image variables Paul Eggleton
@ 2012-02-14 13:40 ` Paul Eggleton
2012-02-14 13:40 ` [PATCH 3/7] classes/buildhistory: sort FILELIST in package info Paul Eggleton
` (6 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Paul Eggleton @ 2012-02-14 13:40 UTC (permalink / raw)
To: openembedded-core
Sort dependencies of each package which sometimes change order and cause
noise in the buildhistory repo, and at the same time remove duplicates
(which seem to be common especially for the RPM package query output).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/buildhistory.bbclass | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index d1a9670..0ee6a33 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -286,12 +286,12 @@ buildhistory_get_image_installed() {
echo $pkgsize $pkg >> ${BUILDHISTORY_DIR_IMAGE}/installed-package-sizes.tmp
fi
- deps=`list_package_depends $pkg`
+ deps=`list_package_depends $pkg | sort | uniq`
for dep in $deps ; do
echo "$pkg OPP $dep;" | sed -e 's:-:_:g' -e 's:\.:_:g' -e 's:+::g' | sed 's:OPP:->:g' >> ${BUILDHISTORY_DIR_IMAGE}/depends.dot
done
- recs=`list_package_recommends $pkg`
+ recs=`list_package_recommends $pkg | sort | uniq`
for rec in $recs ; do
echo "$pkg OPP $rec [style=dotted];" | sed -e 's:-:_:g' -e 's:\.:_:g' -e 's:+::g' | sed 's:OPP:->:g' >> ${BUILDHISTORY_DIR_IMAGE}/depends.dot
done
--
1.7.5.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 3/7] classes/buildhistory: sort FILELIST in package info
2012-02-14 13:40 [PATCH 0/7] buildhistory fixes Paul Eggleton
2012-02-14 13:40 ` [PATCH 1/7] classes/buildhistory: squash spaces out of image variables Paul Eggleton
2012-02-14 13:40 ` [PATCH 2/7] classes/buildhistory: sort and de-dupe dependency graphs Paul Eggleton
@ 2012-02-14 13:40 ` Paul Eggleton
2012-02-14 13:40 ` [PATCH 4/7] classes/buildhistory: use hostname instead of reading /etc/hostname Paul Eggleton
` (5 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Paul Eggleton @ 2012-02-14 13:40 UTC (permalink / raw)
To: openembedded-core
The FILELIST order can vary depending on the order the files were
written which may change between builds with no ill effect, so sort the
list prior to writing it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/buildhistory.bbclass | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 0ee6a33..3fbe3a8 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -186,6 +186,7 @@ python buildhistory_emit_pkghistory() {
fstat = os.lstat(os.path.join(root, f))
pkginfo.size += fstat.st_size
filelist.append(os.sep + os.path.join(relpth, f))
+ filelist.sort()
pkginfo.filelist = " ".join(filelist)
write_pkghistory(pkginfo, d)
--
1.7.5.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 4/7] classes/buildhistory: use hostname instead of reading /etc/hostname
2012-02-14 13:40 [PATCH 0/7] buildhistory fixes Paul Eggleton
` (2 preceding siblings ...)
2012-02-14 13:40 ` [PATCH 3/7] classes/buildhistory: sort FILELIST in package info Paul Eggleton
@ 2012-02-14 13:40 ` Paul Eggleton
2012-02-14 14:28 ` Koen Kooi
2012-02-14 13:40 ` [PATCH 5/7] classes/buildhistory: fix splitting on + in package list fields Paul Eggleton
` (4 subsequent siblings)
8 siblings, 1 reply; 12+ messages in thread
From: Paul Eggleton @ 2012-02-14 13:40 UTC (permalink / raw)
To: openembedded-core
For the purposes of querying the hostname to include it in the commit
message, it seems "cat /etc/hostname" does not work on the Yocto Project
autobuilder machines, and it's likely that the hostname command will be
more generally reliable, so use that instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/buildhistory.bbclass | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 3fbe3a8..dfc9b73 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -383,7 +383,7 @@ buildhistory_commit() {
repostatus=`git status --porcelain`
if [ "$repostatus" != "" ] ; then
git add ${BUILDHISTORY_DIR}/*
- HOSTNAME=`cat /etc/hostname 2>/dev/null || echo unknown`
+ HOSTNAME=`hostname 2>/dev/null || echo unknown`
git commit ${BUILDHISTORY_DIR}/ -m "Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
git push -q ${BUILDHISTORY_PUSH_REPO}
--
1.7.5.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH 4/7] classes/buildhistory: use hostname instead of reading /etc/hostname
2012-02-14 13:40 ` [PATCH 4/7] classes/buildhistory: use hostname instead of reading /etc/hostname Paul Eggleton
@ 2012-02-14 14:28 ` Koen Kooi
0 siblings, 0 replies; 12+ messages in thread
From: Koen Kooi @ 2012-02-14 14:28 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
Op 14 feb. 2012, om 05:40 heeft Paul Eggleton het volgende geschreven:
> For the purposes of querying the hostname to include it in the commit
> message, it seems "cat /etc/hostname" does not work on the Yocto Project
> autobuilder machines,
/etc/hostname is a debianism, but systemd is trying to make it a standard across all distros: http://0pointer.de/blog/projects/the-new-configuration-files.html
regards,
Koen
> and it's likely that the hostname command will be
> more generally reliable, so use that instead.
>
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
> meta/classes/buildhistory.bbclass | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
> index 3fbe3a8..dfc9b73 100644
> --- a/meta/classes/buildhistory.bbclass
> +++ b/meta/classes/buildhistory.bbclass
> @@ -383,7 +383,7 @@ buildhistory_commit() {
> repostatus=`git status --porcelain`
> if [ "$repostatus" != "" ] ; then
> git add ${BUILDHISTORY_DIR}/*
> - HOSTNAME=`cat /etc/hostname 2>/dev/null || echo unknown`
> + HOSTNAME=`hostname 2>/dev/null || echo unknown`
> git commit ${BUILDHISTORY_DIR}/ -m "Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
> if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
> git push -q ${BUILDHISTORY_PUSH_REPO}
> --
> 1.7.5.4
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 5/7] classes/buildhistory: fix splitting on + in package list fields
2012-02-14 13:40 [PATCH 0/7] buildhistory fixes Paul Eggleton
` (3 preceding siblings ...)
2012-02-14 13:40 ` [PATCH 4/7] classes/buildhistory: use hostname instead of reading /etc/hostname Paul Eggleton
@ 2012-02-14 13:40 ` Paul Eggleton
2012-02-14 13:40 ` [PATCH 6/7] buildhistory_analysis: avoid noise due to reordering Paul Eggleton
` (3 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Paul Eggleton @ 2012-02-14 13:40 UTC (permalink / raw)
To: openembedded-core
Ensure we do not erroneously split on + in RDEPENDS/RRECOMMENDS e.g.
libstdc++-dev was being split into libstdc and -dev.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/buildhistory.bbclass | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index dfc9b73..6c2d4e9 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -124,7 +124,7 @@ python buildhistory_emit_pkghistory() {
return None
def sortpkglist(string):
- pkgiter = re.finditer(r'[a-zA-Z0-9.-]+( \([><=]+ [^ )]+\))?', string, 0)
+ pkgiter = re.finditer(r'[a-zA-Z0-9.+-]+( \([><=]+ [^ )]+\))?', string, 0)
pkglist = [p.group(0) for p in pkgiter]
pkglist.sort()
return ' '.join(pkglist)
--
1.7.5.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 6/7] buildhistory_analysis: avoid noise due to reordering
2012-02-14 13:40 [PATCH 0/7] buildhistory fixes Paul Eggleton
` (4 preceding siblings ...)
2012-02-14 13:40 ` [PATCH 5/7] classes/buildhistory: fix splitting on + in package list fields Paul Eggleton
@ 2012-02-14 13:40 ` Paul Eggleton
2012-02-14 13:40 ` [PATCH 7/7] buildhistory_analysis: correctly split RDEPENDS/RRECOMMENDS Paul Eggleton
` (2 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Paul Eggleton @ 2012-02-14 13:40 UTC (permalink / raw)
To: openembedded-core
Avoid noise in the output due to reordering of list variables (except
for PACKAGES where we just report that the order changed). Recent
changes to the buildhistory class itself will avoid this reordering
from occurring but this allows us to examine the results before and
after those changes.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/buildhistory_analysis.py | 17 ++++++++++++++---
1 files changed, 14 insertions(+), 3 deletions(-)
diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index 103cfb4..bef6cd4 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -15,7 +15,8 @@ import git
# How to display fields
-list_fields = ['DEPENDS', 'RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILES', 'FILELIST', 'USER_CLASSES', 'IMAGE_CLASSES', 'IMAGE_FEATURES', 'IMAGE_LINGUAS', 'IMAGE_INSTALL', 'BAD_RECOMMENDATIONS']
+list_fields = ['DEPENDS', 'RDEPENDS', 'RRECOMMENDS', 'FILES', 'FILELIST', 'USER_CLASSES', 'IMAGE_CLASSES', 'IMAGE_FEATURES', 'IMAGE_LINGUAS', 'IMAGE_INSTALL', 'BAD_RECOMMENDATIONS']
+list_order_fields = ['PACKAGES']
numeric_fields = ['PKGSIZE', 'IMAGESIZE']
# Fields to monitor
monitor_fields = ['RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILELIST', 'PKGSIZE', 'IMAGESIZE']
@@ -52,12 +53,15 @@ class ChangeRecord:
else:
prefix = ''
- if self.fieldname in list_fields:
+ if self.fieldname in list_fields or self.fieldname in list_order_fields:
aitems = self.oldvalue.split()
bitems = self.newvalue.split()
removed = list(set(aitems) - set(bitems))
added = list(set(bitems) - set(aitems))
- out = '%s:%s%s' % (self.fieldname, ' removed "%s"' % ' '.join(removed) if removed else '', ' added "%s"' % ' '.join(added) if added else '')
+ if removed or added:
+ out = '%s:%s%s' % (self.fieldname, ' removed "%s"' % ' '.join(removed) if removed else '', ' added "%s"' % ' '.join(added) if added else '')
+ else:
+ out = '%s changed order' % self.fieldname
elif self.fieldname in numeric_fields:
aval = int(self.oldvalue or 0)
bval = int(self.newvalue or 0)
@@ -237,6 +241,13 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
percentchg = 100
if percentchg < monitor_numeric_threshold:
continue
+ elif (not report_all) and key in list_fields:
+ alist = astr.split()
+ alist.sort()
+ blist = bstr.split()
+ blist.sort()
+ if ' '.join(alist) == ' '.join(blist):
+ continue
chg = ChangeRecord(path, key, astr, bstr, key in monitor_fields)
changes.append(chg)
return changes
--
1.7.5.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 7/7] buildhistory_analysis: correctly split RDEPENDS/RRECOMMENDS
2012-02-14 13:40 [PATCH 0/7] buildhistory fixes Paul Eggleton
` (5 preceding siblings ...)
2012-02-14 13:40 ` [PATCH 6/7] buildhistory_analysis: avoid noise due to reordering Paul Eggleton
@ 2012-02-14 13:40 ` Paul Eggleton
2012-02-17 11:36 ` [PATCH 0/7] buildhistory fixes Paul Eggleton
2012-02-21 18:45 ` Saul Wold
8 siblings, 0 replies; 12+ messages in thread
From: Paul Eggleton @ 2012-02-14 13:40 UTC (permalink / raw)
To: openembedded-core
Split RDEPENDS and RRECOMMENDS correctly (which may contain version
number specifications after each item).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/buildhistory_analysis.py | 14 ++++++++++++--
1 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index bef6cd4..4f3e635 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -12,6 +12,7 @@ import sys
import os.path
import difflib
import git
+import re
# How to display fields
@@ -53,9 +54,18 @@ class ChangeRecord:
else:
prefix = ''
+ def pkglist_split(pkgs):
+ pkgit = re.finditer(r'[a-zA-Z0-9.+-]+( \([><=]+ [^ )]+\))?', pkgs, 0)
+ pkglist = [p.group(0) for p in pkgit]
+ return pkglist
+
if self.fieldname in list_fields or self.fieldname in list_order_fields:
- aitems = self.oldvalue.split()
- bitems = self.newvalue.split()
+ if self.fieldname in ['RDEPENDS', 'RRECOMMENDS']:
+ aitems = pkglist_split(self.oldvalue)
+ bitems = pkglist_split(self.newvalue)
+ else:
+ aitems = self.oldvalue.split()
+ bitems = self.newvalue.split()
removed = list(set(aitems) - set(bitems))
added = list(set(bitems) - set(aitems))
if removed or added:
--
1.7.5.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH 0/7] buildhistory fixes
2012-02-14 13:40 [PATCH 0/7] buildhistory fixes Paul Eggleton
` (6 preceding siblings ...)
2012-02-14 13:40 ` [PATCH 7/7] buildhistory_analysis: correctly split RDEPENDS/RRECOMMENDS Paul Eggleton
@ 2012-02-17 11:36 ` Paul Eggleton
2012-02-21 14:57 ` Paul Eggleton
2012-02-21 18:45 ` Saul Wold
8 siblings, 1 reply; 12+ messages in thread
From: Paul Eggleton @ 2012-02-17 11:36 UTC (permalink / raw)
To: openembedded-core
On Tuesday 14 February 2012 13:40:10 Paul Eggleton wrote:
> Analysis of the recent output from buildhistory on the Yocto Project and
> Angstrom autobuilders showed a few anomalies and annoyances; this
> set of patches addresses all of the ones I noticed.
I've got one more patch to add to this, please ignore this set as v2 is on its
way.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH 0/7] buildhistory fixes
2012-02-14 13:40 [PATCH 0/7] buildhistory fixes Paul Eggleton
` (7 preceding siblings ...)
2012-02-17 11:36 ` [PATCH 0/7] buildhistory fixes Paul Eggleton
@ 2012-02-21 18:45 ` Saul Wold
8 siblings, 0 replies; 12+ messages in thread
From: Saul Wold @ 2012-02-21 18:45 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer; +Cc: Paul Eggleton
On 02/14/2012 05:40 AM, Paul Eggleton wrote:
> Analysis of the recent output from buildhistory on the Yocto Project and
> Angstrom autobuilders showed a few anomalies and annoyances; this
> set of patches addresses all of the ones I noticed.
>
> The following changes since commit e5ad03093dfc4364d1407183f458df79f347c7a1:
>
> guile: fix cross configure failure (2012-02-10 13:38:16 +0000)
>
> are available in the git repository at:
> git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes4
> http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes4
>
> Paul Eggleton (7):
> classes/buildhistory: squash spaces out of image variables
> classes/buildhistory: sort and de-dupe dependency graphs
> classes/buildhistory: sort FILELIST in package info
> classes/buildhistory: use hostname instead of reading /etc/hostname
> classes/buildhistory: fix splitting on + in package list fields
> buildhistory_analysis: avoid noise due to reordering
> buildhistory_analysis: correctly split RDEPENDS/RRECOMMENDS
>
> meta/classes/buildhistory.bbclass | 25 +++++++++++++++++--------
> meta/lib/oe/buildhistory_analysis.py | 31 ++++++++++++++++++++++++++-----
> 2 files changed, 43 insertions(+), 13 deletions(-)
>
Merged into OE-core
Thanks
Sau!
^ permalink raw reply [flat|nested] 12+ messages in thread