[PATCH 0/8] buildhistory improvements

[PATCH 0/8] buildhistory improvements

[Paul Eggleton]

A set of improvements for the buildhistory class: 
* A fix for not pushing after making "No changes" commits
* Four fixes for the recently added package file listing functionality
* A patch to avoid errors if additional files are added at the recipe
  level in extension classes
* A couple of patches improving the commit messages


The following changes since commit f07045fcae859c902434062d1725f1348f42d1dd:

  oeqa/oetest.py: add better package search for hasPackage() (2015-08-26 08:26:37 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes10
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes10

Paul Eggleton (8):
  classes/buildhistory: ensure we push when "no changes" commits are made
  classes/buildhistory: exclude . in file listings
  classes/buildhistory: indent recently added function consistently
  classes/buildhistory: tweak buildhistory_list_pkg_files
  classes/buildhistory: fix permissions in package file listing
  classes/buildhistory: handle additional files at recipe level
  classes/buildhistory: include metadata revisions in commit message
  classes/buildhistory: add build result to commit message

 meta/classes/buildhistory.bbclass | 95 ++++++++++++++++++++++++++++-----------
 1 file changed, 70 insertions(+), 25 deletions(-)

-- 
2.1.0

[PATCH 1/8] classes/buildhistory: ensure we push when "no changes" commits are made

[Paul Eggleton]

If there aren't any changes, we still make a commit to the buildhistory
repo, but this wasn't being pushed if BUILDHISTORY_PUSH_REPO is set.
Move the push to the end to make it unconditional.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index cad5116..e60cdd7 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -621,11 +621,11 @@ END
 				git commit $entry metadata-revs -m "$entry: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" -m "cmd: $CMDLINE" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
 			done
 			git gc --auto --quiet
-			if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
-				git push -q ${BUILDHISTORY_PUSH_REPO}
-			fi
 		else
 			git commit ${BUILDHISTORY_DIR}/ --allow-empty -m "No changes: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" -m "cmd: $CMDLINE" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
+		fi
+		if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
+			git push -q ${BUILDHISTORY_PUSH_REPO}
 		fi) || true
 }
 
-- 
2.1.0

[PATCH 2/8] classes/buildhistory: exclude . in file listings

[Paul Eggleton]

We don't care about the permissions on the top-level directory in which
the files are contained, just everything under it; this also avoids
lists with just this entry in it for empty packages. Affects file
listings for both images and packages.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index e60cdd7..77a3c86 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -435,7 +435,7 @@ buildhistory_get_sdk_installed_target() {
 buildhistory_list_files() {
 	# List the files in the specified directory, but exclude date/time etc.
 	# This awk script is somewhat messy, but handles where the size is not printed for device files under pseudo
-	( cd $1 && find . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 )
+	( cd $1 && find . ! -path . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 )
 }
 
 buildhistory_list_pkg_files() {
-- 
2.1.0

[PATCH 3/8] classes/buildhistory: indent recently added function consistently

[Paul Eggleton]

Shell functions use tabs in this file.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 77a3c86..054a81f 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -439,20 +439,20 @@ buildhistory_list_files() {
 }
 
 buildhistory_list_pkg_files() {
-        file_prefix="files-in-"
-
-        # Create individual files-in-package for each recipe's package
-        for pkgdir in $(find ${PKGDEST}/* -maxdepth 0 -type d); do
-                pkgname=$(basename ${pkgdir})
-                outfolder="${BUILDHISTORY_DIR_PACKAGE}/${pkgname}"
-                outfile="${outfolder}/${file_prefix}${pkgname}.txt"
-                # Make sure the output folder, exist so we can create the files-in-$pkgname.txt file
-                if [ ! -d ${outfolder} ] ; then
-                        bbdebug 2 "Folder ${outfolder} does not exist, file ${outfile} not created"
-                        continue
-                fi
-                buildhistory_list_files ${pkgdir} ${outfile}
-        done
+	file_prefix="files-in-"
+
+	# Create individual files-in-package for each recipe's package
+	for pkgdir in $(find ${PKGDEST}/* -maxdepth 0 -type d); do
+		pkgname=$(basename ${pkgdir})
+		outfolder="${BUILDHISTORY_DIR_PACKAGE}/${pkgname}"
+		outfile="${outfolder}/${file_prefix}${pkgname}.txt"
+		# Make sure the output folder, exist so we can create the files-in-$pkgname.txt file
+		if [ ! -d ${outfolder} ] ; then
+			bbdebug 2 "Folder ${outfolder} does not exist, file ${outfile} not created"
+			continue
+		fi
+		buildhistory_list_files ${pkgdir} ${outfile}
+	done
 }
 
 buildhistory_get_imageinfo() {
-- 
2.1.0

[PATCH 4/8] classes/buildhistory: tweak buildhistory_list_pkg_files

[Paul Eggleton]

* Avoid using ${...} for shell variables (since they could be
  expanded as bitbake variables if present)
* Use files-in-package.txt rather than files-in-<packagename>.txt; the
  file is already in a subdirectory named with the package name and this
  naming is consistent with that of files-in-image.txt.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 054a81f..3d459c7 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -439,16 +439,14 @@ buildhistory_list_files() {
 }
 
 buildhistory_list_pkg_files() {
-	file_prefix="files-in-"
-
 	# Create individual files-in-package for each recipe's package
 	for pkgdir in $(find ${PKGDEST}/* -maxdepth 0 -type d); do
-		pkgname=$(basename ${pkgdir})
-		outfolder="${BUILDHISTORY_DIR_PACKAGE}/${pkgname}"
-		outfile="${outfolder}/${file_prefix}${pkgname}.txt"
-		# Make sure the output folder, exist so we can create the files-in-$pkgname.txt file
-		if [ ! -d ${outfolder} ] ; then
-			bbdebug 2 "Folder ${outfolder} does not exist, file ${outfile} not created"
+		pkgname=$(basename $pkgdir)
+		outfolder="${BUILDHISTORY_DIR_PACKAGE}/$pkgname"
+		outfile="$outfolder/files-in-package.txt"
+		# Make sure the output folder exists so we can create the file
+		if [ ! -d $outfolder ] ; then
+			bbdebug 2 "Folder $outfolder does not exist, file $outfile not created"
 			continue
 		fi
 		buildhistory_list_files ${pkgdir} ${outfile}
-- 
2.1.0

[PATCH 5/8] classes/buildhistory: fix permissions in package file listing

[Paul Eggleton]

If we want the correct file permissions to show up here as they would on
the target, we need to run the command under pseudo. Normally we'd set
the fakeroot varflag on the function and that would be enough, but it
turns out that setting fakeroot on a non-task function that you execute
using bb.build.exec_func() isn't working at the moment. Work around this
by simply using FAKEROOTENV and FAKEROOTCMD. Unfortunately that means we
have to duplicate the command for the two cases but I couldn't find a
better means of doing that that actually works.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 3d459c7..190c38e 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -435,7 +435,11 @@ buildhistory_get_sdk_installed_target() {
 buildhistory_list_files() {
 	# List the files in the specified directory, but exclude date/time etc.
 	# This awk script is somewhat messy, but handles where the size is not printed for device files under pseudo
-	( cd $1 && find . ! -path . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 )
+	if [ "$3" = "fakeroot" ] ; then
+		( cd $1 && ${FAKEROOTENV} ${FAKEROOTCMD} find . ! -path . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 )
+	else
+		( cd $1 && find . ! -path . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 )
+	fi
 }
 
 buildhistory_list_pkg_files() {
@@ -449,7 +453,7 @@ buildhistory_list_pkg_files() {
 			bbdebug 2 "Folder $outfolder does not exist, file $outfile not created"
 			continue
 		fi
-		buildhistory_list_files ${pkgdir} ${outfile}
+		buildhistory_list_files $pkgdir $outfile fakeroot
 	done
 }
 
-- 
2.1.0

[PATCH 6/8] classes/buildhistory: handle additional files at recipe level

[Paul Eggleton]

Avoid an error when attempting to remove previous data if it's not a
subdirectory - we were assuming that anything that wasn't named "latest"
or "latest_srcrev" had to be a directory. This makes it possible to have
a buildhistory_emit_pkghistory_append which writes additional files at
the recipe level.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 190c38e..57dc1e9 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -172,10 +172,13 @@ python buildhistory_emit_pkghistory() {
         for item in os.listdir(pkghistdir):
             if item != "latest" and item != "latest_srcrev":
                 if item not in packagelist:
-                    subdir = os.path.join(pkghistdir, item)
-                    for subfile in os.listdir(subdir):
-                        os.unlink(os.path.join(subdir, subfile))
-                    os.rmdir(subdir)
+                    itempath = os.path.join(pkghistdir, item)
+                    if os.path.isdir(itempath):
+                        for subfile in os.listdir(itempath):
+                            os.unlink(os.path.join(itempath, subfile))
+                        os.rmdir(itempath)
+                    else:
+                        os.unlink(itempath)
 
     rcpinfo = RecipeInfo(pn)
     rcpinfo.pe = pe
-- 
2.1.0

[PATCH 7/8] classes/buildhistory: include metadata revisions in commit message

[Paul Eggleton]

We do already commit these into the repository itself, but have them in
the commit message as well as a reference. As part of this, refactor out
running "git commit" into a separate function so we don't have to
duplicate the code in the two places we call it.

Implements [YOCTO #7966].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 57dc1e9..135d17a 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -585,6 +585,27 @@ def buildhistory_get_cmdline(d):
     return '%s %s' % (bincmd, ' '.join(sys.argv[1:]))
 
 
+buildhistory_single_commit() {
+	if [ "$3" = "" ] ; then
+		commitopts="${BUILDHISTORY_DIR}/ --allow-empty"
+		item="No changes"
+	else
+		commitopts="$3 metadata-revs"
+		item="$3"
+	fi
+	commitmsgfile=`mktemp`
+	cat > $commitmsgfile << END
+$item: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $2
+
+cmd: $1
+
+metadata revisions:
+END
+	cat ${BUILDHISTORY_DIR}/metadata-revs >> $commitmsgfile
+	git commit $commitopts -F $commitmsgfile --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
+	rm $commitmsgfile
+}
+
 buildhistory_commit() {
 	if [ ! -d ${BUILDHISTORY_DIR} ] ; then
 		# Code above that creates this dir never executed, so there can't be anything to commit
@@ -623,11 +644,11 @@ END
 			# porcelain output looks like "?? packages/foo/bar"
 			# Ensure we commit metadata-revs with the first commit
 			for entry in `echo "$repostatus" | awk '{print $2}' | awk -F/ '{print $1}' | sort | uniq` ; do
-				git commit $entry metadata-revs -m "$entry: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" -m "cmd: $CMDLINE" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
+				buildhistory_single_commit "$CMDLINE" "$HOSTNAME" "$entry"
 			done
 			git gc --auto --quiet
 		else
-			git commit ${BUILDHISTORY_DIR}/ --allow-empty -m "No changes: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" -m "cmd: $CMDLINE" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
+			buildhistory_single_commit "$CMDLINE" "$HOSTNAME"
 		fi
 		if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
 			git push -q ${BUILDHISTORY_PUSH_REPO}
-- 
2.1.0

[PATCH 8/8] classes/buildhistory: add build result to commit message

[Paul Eggleton]

We have the command in the commit message, we might as well have the
build result as well (succeeded/failed and whether or not it was
interrupted by the user). The interrupted part relies upon a change to
BitBake to extend the BuildCompleted event to include an attribute for
that, but will not fail if the attribute is not present.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 135d17a..4db0441 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -593,12 +593,27 @@ buildhistory_single_commit() {
 		commitopts="$3 metadata-revs"
 		item="$3"
 	fi
+	if [ "${BUILDHISTORY_BUILD_FAILURES}" = "0" ] ; then
+		result="succeeded"
+	else
+		result="failed"
+	fi
+	case ${BUILDHISTORY_BUILD_INTERRUPTED} in
+		1)
+			result="$result (interrupted)"
+			;;
+		2)
+			result="$result (force interrupted)"
+			;;
+	esac
 	commitmsgfile=`mktemp`
 	cat > $commitmsgfile << END
 $item: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $2
 
 cmd: $1
 
+result: $result
+
 metadata revisions:
 END
 	cat ${BUILDHISTORY_DIR}/metadata-revs >> $commitmsgfile
@@ -659,7 +674,11 @@ python buildhistory_eventhandler() {
     if e.data.getVar('BUILDHISTORY_FEATURES', True).strip():
         if e.data.getVar("BUILDHISTORY_COMMIT", True) == "1":
             bb.note("Writing buildhistory")
-            bb.build.exec_func("buildhistory_commit", e.data)
+            localdata = bb.data.createCopy(e.data)
+            localdata.setVar('BUILDHISTORY_BUILD_FAILURES', str(e._failures))
+            interrupted = getattr(e, '_interrupted', 0)
+            localdata.setVar('BUILDHISTORY_BUILD_INTERRUPTED', str(interrupted))
+            bb.build.exec_func("buildhistory_commit", localdata)
 }
 
 addhandler buildhistory_eventhandler
-- 
2.1.0