[PATCH 0/7] buildhistory fixes

[PATCH 0/7] buildhistory fixes

[Paul Eggleton]

Analysis of the recent output from buildhistory on the Yocto Project and
Angstrom autobuilders showed a few anomalies and annoyances; this
set of patches addresses all of the ones I noticed.

The following changes since commit e5ad03093dfc4364d1407183f458df79f347c7a1:

  guile: fix cross configure failure (2012-02-10 13:38:16 +0000)

are available in the git repository at:
  git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes4
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes4

Paul Eggleton (7):
  classes/buildhistory: squash spaces out of image variables
  classes/buildhistory: sort and de-dupe dependency graphs
  classes/buildhistory: sort FILELIST in package info
  classes/buildhistory: use hostname instead of reading /etc/hostname
  classes/buildhistory: fix splitting on + in package list fields
  buildhistory_analysis: avoid noise due to reordering
  buildhistory_analysis: correctly split RDEPENDS/RRECOMMENDS

 meta/classes/buildhistory.bbclass    |   25 +++++++++++++++++--------
 meta/lib/oe/buildhistory_analysis.py |   31 ++++++++++++++++++++++++++-----
 2 files changed, 43 insertions(+), 13 deletions(-)

-- 
1.7.5.4

[PATCH 1/7] classes/buildhistory: squash spaces out of image variables

[Paul Eggleton]

Values of image variables that are lists (e.g. IMAGE_INSTALL) are easier
to read if there are no extraneous spaces in them, so ensure that there
is only one space between each item.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass |   16 ++++++++++++----
 1 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 1b6b249..d1a9670 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -123,9 +123,6 @@ python buildhistory_emit_pkghistory() {
 		except EnvironmentError:
 			return None
 
-	def squashspaces(string):
-		return re.sub("\s+", " ", string)
-
 	def sortpkglist(string):
 		pkgiter = re.finditer(r'[a-zA-Z0-9.-]+( \([><=]+ [^ )]+\))?', string, 0)
 		pkglist = [p.group(0) for p in pkgiter]
@@ -349,12 +346,23 @@ def buildhistory_get_layers(d):
 	return layertext
 
 
+def squashspaces(string):
+	import re
+	return re.sub("\s+", " ", string).strip()
+
+
 def buildhistory_get_imagevars(d):
 	imagevars = "DISTRO DISTRO_VERSION USER_CLASSES IMAGE_CLASSES IMAGE_FEATURES IMAGE_LINGUAS IMAGE_INSTALL BAD_RECOMMENDATIONS ROOTFS_POSTPROCESS_COMMAND IMAGE_POSTPROCESS_COMMAND"
+	listvars = "USER_CLASSES IMAGE_CLASSES IMAGE_FEATURES IMAGE_LINGUAS IMAGE_INSTALL BAD_RECOMMENDATIONS"
 
+	imagevars = imagevars.split()
+	listvars = listvars.split()
 	ret = ""
-	for var in imagevars.split():
+	for var in imagevars:
 		value = d.getVar(var, True) or ""
+		if var in listvars:
+			# Squash out spaces
+			value = squashspaces(value)
 		ret += "%s = %s\n" % (var, value)
 	return ret.rstrip('\n')
 
-- 
1.7.5.4

[PATCH 2/7] classes/buildhistory: sort and de-dupe dependency graphs

[Paul Eggleton]

Sort dependencies of each package which sometimes change order and cause
noise in the buildhistory repo, and at the same time remove duplicates
(which seem to be common especially for the RPM package query output).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index d1a9670..0ee6a33 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -286,12 +286,12 @@ buildhistory_get_image_installed() {
 			echo $pkgsize $pkg >> ${BUILDHISTORY_DIR_IMAGE}/installed-package-sizes.tmp
 		fi
 
-		deps=`list_package_depends $pkg`
+		deps=`list_package_depends $pkg | sort | uniq`
 		for dep in $deps ; do
 			echo "$pkg OPP $dep;" | sed -e 's:-:_:g' -e 's:\.:_:g' -e 's:+::g' | sed 's:OPP:->:g' >> ${BUILDHISTORY_DIR_IMAGE}/depends.dot
 		done
 
-		recs=`list_package_recommends $pkg`
+		recs=`list_package_recommends $pkg | sort | uniq`
 		for rec in $recs ; do
 			echo "$pkg OPP $rec [style=dotted];" | sed -e 's:-:_:g' -e 's:\.:_:g' -e 's:+::g' | sed 's:OPP:->:g' >> ${BUILDHISTORY_DIR_IMAGE}/depends.dot
 		done
-- 
1.7.5.4

[PATCH 3/7] classes/buildhistory: sort FILELIST in package info

[Paul Eggleton]

The FILELIST order can vary depending on the order the files were
written which may change between builds with no ill effect, so sort the
list prior to writing it.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 0ee6a33..3fbe3a8 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -186,6 +186,7 @@ python buildhistory_emit_pkghistory() {
 				fstat = os.lstat(os.path.join(root, f))
 				pkginfo.size += fstat.st_size
 				filelist.append(os.sep + os.path.join(relpth, f))
+		filelist.sort()
 		pkginfo.filelist = " ".join(filelist)
 
 		write_pkghistory(pkginfo, d)
-- 
1.7.5.4

[PATCH 4/7] classes/buildhistory: use hostname instead of reading /etc/hostname

[Paul Eggleton]

For the purposes of querying the hostname to include it in the commit
message, it seems "cat /etc/hostname" does not work on the Yocto Project
autobuilder machines, and it's likely that the hostname command will be
more generally reliable, so use that instead.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 3fbe3a8..dfc9b73 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -383,7 +383,7 @@ buildhistory_commit() {
 		repostatus=`git status --porcelain`
 		if [ "$repostatus" != "" ] ; then
 			git add ${BUILDHISTORY_DIR}/*
-			HOSTNAME=`cat /etc/hostname 2>/dev/null || echo unknown`
+			HOSTNAME=`hostname 2>/dev/null || echo unknown`
 			git commit ${BUILDHISTORY_DIR}/ -m "Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
 			if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
 				git push -q ${BUILDHISTORY_PUSH_REPO}
-- 
1.7.5.4

[PATCH 5/7] classes/buildhistory: fix splitting on + in package list fields

[Paul Eggleton]

Ensure we do not erroneously split on + in RDEPENDS/RRECOMMENDS e.g.
libstdc++-dev was being split into libstdc and -dev.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index dfc9b73..6c2d4e9 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -124,7 +124,7 @@ python buildhistory_emit_pkghistory() {
 			return None
 
 	def sortpkglist(string):
-		pkgiter = re.finditer(r'[a-zA-Z0-9.-]+( \([><=]+ [^ )]+\))?', string, 0)
+		pkgiter = re.finditer(r'[a-zA-Z0-9.+-]+( \([><=]+ [^ )]+\))?', string, 0)
 		pkglist = [p.group(0) for p in pkgiter]
 		pkglist.sort()
 		return ' '.join(pkglist)
-- 
1.7.5.4

[PATCH 6/7] buildhistory_analysis: avoid noise due to reordering

[Paul Eggleton]

Avoid noise in the output due to reordering of list variables (except
for PACKAGES where we just report that the order changed). Recent
changes to the buildhistory class itself will avoid this reordering
from occurring but this allows us to examine the results before and
after those changes.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/buildhistory_analysis.py |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index 103cfb4..bef6cd4 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -15,7 +15,8 @@ import git
 
 
 # How to display fields
-list_fields = ['DEPENDS', 'RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILES', 'FILELIST', 'USER_CLASSES', 'IMAGE_CLASSES', 'IMAGE_FEATURES', 'IMAGE_LINGUAS', 'IMAGE_INSTALL', 'BAD_RECOMMENDATIONS']
+list_fields = ['DEPENDS', 'RDEPENDS', 'RRECOMMENDS', 'FILES', 'FILELIST', 'USER_CLASSES', 'IMAGE_CLASSES', 'IMAGE_FEATURES', 'IMAGE_LINGUAS', 'IMAGE_INSTALL', 'BAD_RECOMMENDATIONS']
+list_order_fields = ['PACKAGES']
 numeric_fields = ['PKGSIZE', 'IMAGESIZE']
 # Fields to monitor
 monitor_fields = ['RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILELIST', 'PKGSIZE', 'IMAGESIZE']
@@ -52,12 +53,15 @@ class ChangeRecord:
         else:
             prefix = ''
 
-        if self.fieldname in list_fields:
+        if self.fieldname in list_fields or self.fieldname in list_order_fields:
             aitems = self.oldvalue.split()
             bitems = self.newvalue.split()
             removed = list(set(aitems) - set(bitems))
             added = list(set(bitems) - set(aitems))
-            out = '%s:%s%s' % (self.fieldname, ' removed "%s"' % ' '.join(removed) if removed else '', ' added "%s"' % ' '.join(added) if added else '')
+            if removed or added:
+                out = '%s:%s%s' % (self.fieldname, ' removed "%s"' % ' '.join(removed) if removed else '', ' added "%s"' % ' '.join(added) if added else '')
+            else:
+                out = '%s changed order' % self.fieldname
         elif self.fieldname in numeric_fields:
             aval = int(self.oldvalue or 0)
             bval = int(self.newvalue or 0)
@@ -237,6 +241,13 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
                     percentchg = 100
                 if percentchg < monitor_numeric_threshold:
                     continue
+            elif (not report_all) and key in list_fields:
+                alist = astr.split()
+                alist.sort()
+                blist = bstr.split()
+                blist.sort()
+                if ' '.join(alist) == ' '.join(blist):
+                    continue
             chg = ChangeRecord(path, key, astr, bstr, key in monitor_fields)
             changes.append(chg)
     return changes
-- 
1.7.5.4

[PATCH 7/7] buildhistory_analysis: correctly split RDEPENDS/RRECOMMENDS

[Paul Eggleton]

Split RDEPENDS and RRECOMMENDS correctly (which may contain version
number specifications after each item).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/buildhistory_analysis.py |   14 ++++++++++++--
 1 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index bef6cd4..4f3e635 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -12,6 +12,7 @@ import sys
 import os.path
 import difflib
 import git
+import re
 
 
 # How to display fields
@@ -53,9 +54,18 @@ class ChangeRecord:
         else:
             prefix = ''
 
+        def pkglist_split(pkgs):
+            pkgit = re.finditer(r'[a-zA-Z0-9.+-]+( \([><=]+ [^ )]+\))?', pkgs, 0)
+            pkglist = [p.group(0) for p in pkgit]
+            return pkglist
+
         if self.fieldname in list_fields or self.fieldname in list_order_fields:
-            aitems = self.oldvalue.split()
-            bitems = self.newvalue.split()
+            if self.fieldname in ['RDEPENDS', 'RRECOMMENDS']:
+                aitems = pkglist_split(self.oldvalue)
+                bitems = pkglist_split(self.newvalue)
+            else:
+                aitems = self.oldvalue.split()
+                bitems = self.newvalue.split()
             removed = list(set(aitems) - set(bitems))
             added = list(set(bitems) - set(aitems))
             if removed or added:
-- 
1.7.5.4

Re: [PATCH 4/7] classes/buildhistory: use hostname instead of reading /etc/hostname

[Koen Kooi]

Op 14 feb. 2012, om 05:40 heeft Paul Eggleton het volgende geschreven:

> For the purposes of querying the hostname to include it in the commit
> message, it seems "cat /etc/hostname" does not work on the Yocto Project
> autobuilder machines,

/etc/hostname is a debianism, but systemd is trying to make it a standard across all distros: http://0pointer.de/blog/projects/the-new-configuration-files.html

regards,

Koen

> and it's likely that the hostname command will be
> more generally reliable, so use that instead.
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
> meta/classes/buildhistory.bbclass |    2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
> index 3fbe3a8..dfc9b73 100644
> --- a/meta/classes/buildhistory.bbclass
> +++ b/meta/classes/buildhistory.bbclass
> @@ -383,7 +383,7 @@ buildhistory_commit() {
> 		repostatus=`git status --porcelain`
> 		if [ "$repostatus" != "" ] ; then
> 			git add ${BUILDHISTORY_DIR}/*
> -			HOSTNAME=`cat /etc/hostname 2>/dev/null || echo unknown`
> +			HOSTNAME=`hostname 2>/dev/null || echo unknown`
> 			git commit ${BUILDHISTORY_DIR}/ -m "Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
> 			if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
> 				git push -q ${BUILDHISTORY_PUSH_REPO}
> -- 
> 1.7.5.4
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [PATCH 0/7] buildhistory fixes

[Paul Eggleton]

On Tuesday 14 February 2012 13:40:10 Paul Eggleton wrote:
> Analysis of the recent output from buildhistory on the Yocto Project and
> Angstrom autobuilders showed a few anomalies and annoyances; this
> set of patches addresses all of the ones I noticed.

I've got one more patch to add to this, please ignore this set as v2 is on its 
way.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/7] buildhistory fixes

[Paul Eggleton]

On Friday 17 February 2012 11:36:37 Paul Eggleton wrote:
> I've got one more patch to add to this, please ignore this set as v2 is on
> its way.

Actually scratch that, as mentioned in another thread I got stuck on a pseudo 
issue; please merge this series.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/7] buildhistory fixes

[Saul Wold]

On 02/14/2012 05:40 AM, Paul Eggleton wrote:
> Analysis of the recent output from buildhistory on the Yocto Project and
> Angstrom autobuilders showed a few anomalies and annoyances; this
> set of patches addresses all of the ones I noticed.
>
> The following changes since commit e5ad03093dfc4364d1407183f458df79f347c7a1:
>
>    guile: fix cross configure failure (2012-02-10 13:38:16 +0000)
>
> are available in the git repository at:
>    git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes4
>    http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes4
>
> Paul Eggleton (7):
>    classes/buildhistory: squash spaces out of image variables
>    classes/buildhistory: sort and de-dupe dependency graphs
>    classes/buildhistory: sort FILELIST in package info
>    classes/buildhistory: use hostname instead of reading /etc/hostname
>    classes/buildhistory: fix splitting on + in package list fields
>    buildhistory_analysis: avoid noise due to reordering
>    buildhistory_analysis: correctly split RDEPENDS/RRECOMMENDS
>
>   meta/classes/buildhistory.bbclass    |   25 +++++++++++++++++--------
>   meta/lib/oe/buildhistory_analysis.py |   31 ++++++++++++++++++++++++++-----
>   2 files changed, 43 insertions(+), 13 deletions(-)
>

Merged into OE-core

Thanks
	Sau!