[RFC v2][PATCH 1/3] buildhistory: record all builds

[RFC v2][PATCH 1/3] buildhistory: record all builds

[Koen Kooi]

Allow empty commits, this also give a nice speedup since 'git status --porcelain' doesn't need to get run.

Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
---
 meta/classes/buildhistory.bbclass |   17 +++++++----------
 1 files changed, 7 insertions(+), 10 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index f99aa7f..1926d12 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -396,16 +396,13 @@ buildhistory_commit() {
 		if [ ! -d .git ] ; then
 			git init -q
 		fi
-		# Ensure there are new/changed files to commit
-		repostatus=`git status --porcelain`
-		if [ "$repostatus" != "" ] ; then
-			git add ${BUILDHISTORY_DIR}/*
-			HOSTNAME=`hostname 2>/dev/null || echo unknown`
-			git commit ${BUILDHISTORY_DIR}/ -m "Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
-			if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
-				git push -q ${BUILDHISTORY_PUSH_REPO}
-			fi
-		fi) || true
+		git add ${BUILDHISTORY_DIR}/*
+		HOSTNAME=`hostname 2>/dev/null || echo unknown`
+		git commit ${BUILDHISTORY_DIR}/ --allow-empty -m "Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
+		if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
+			git push -q ${BUILDHISTORY_PUSH_REPO}
+		fi
+	) || true
 }
 
 python buildhistory_eventhandler() {
-- 
1.7.2.5

[RFC v2][PATCH 2/3] buildhistory: make seperate commits for every entry in the buildhistory dir

[Koen Kooi]

This seperates out image changes from package changes making the image diffs a lot easier to read. The downside of this is 3 commits for each and every build.

Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
---
 meta/classes/buildhistory.bbclass |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 1926d12..3aec325 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -398,7 +398,9 @@ buildhistory_commit() {
 		fi
 		git add ${BUILDHISTORY_DIR}/*
 		HOSTNAME=`hostname 2>/dev/null || echo unknown`
-		git commit ${BUILDHISTORY_DIR}/ --allow-empty -m "Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
+		for entry in ${BUILDHISTORY_DIR}/* ; do
+			git commit $entry --allow-empty -m "$(basename $entry): Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
+		done
 		if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
 			git push -q ${BUILDHISTORY_PUSH_REPO}
 		fi
-- 
1.7.2.5

[RFC v2][PATCH 3/3] buildhistory: remove duplicate entries from dot graph

[Koen Kooi]

There are various conditions that lead to duplicate entries in the dot graph which need to get fixed, but this patch is a catchall. A previous attempt to address this only works on rpm which gives a \n seperated output, opkg doesn't.

Another benefit is that the sort order is now know, leading to less spurious diffs in buildhistory commits.

Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
---

Changes since v1: remove sort | uniq in the for loop, that only works with RPM

 meta/classes/buildhistory.bbclass |   11 +++++++----
 1 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 3aec325..49d1982 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -291,6 +291,7 @@ buildhistory_get_image_installed() {
 	# Produce installed package file and size lists and dependency graph
 	echo -n > ${BUILDHISTORY_DIR_IMAGE}/installed-packages.txt
 	echo -n > ${BUILDHISTORY_DIR_IMAGE}/installed-package-sizes.tmp
+	echo -n > ${BUILDHISTORY_DIR_IMAGE}/depends.tmp	
 	echo -e "digraph depends {\n    node [shape=plaintext]" > ${BUILDHISTORY_DIR_IMAGE}/depends.dot
 	for pkg in $INSTALLED_PKGS; do
 		pkgfile=`get_package_filename $pkg`
@@ -300,16 +301,18 @@ buildhistory_get_image_installed() {
 			echo $pkgsize $pkg >> ${BUILDHISTORY_DIR_IMAGE}/installed-package-sizes.tmp
 		fi
 
-		deps=`list_package_depends $pkg | sort | uniq`
+		deps=`list_package_depends $pkg`
 		for dep in $deps ; do
-			echo "$pkg OPP $dep;" | sed -e 's:-:_:g' -e 's:\.:_:g' -e 's:+::g' | sed 's:OPP:->:g' >> ${BUILDHISTORY_DIR_IMAGE}/depends.dot
+			echo "$pkg OPP $dep;" | sed -e 's:-:_:g' -e 's:\.:_:g' -e 's:+::g' | sed 's:OPP:->:g' >> ${BUILDHISTORY_DIR_IMAGE}/depends.tmp
 		done
 
-		recs=`list_package_recommends $pkg | sort | uniq`
+		recs=`list_package_recommends $pkg`
 		for rec in $recs ; do
-			echo "$pkg OPP $rec [style=dotted];" | sed -e 's:-:_:g' -e 's:\.:_:g' -e 's:+::g' | sed 's:OPP:->:g' >> ${BUILDHISTORY_DIR_IMAGE}/depends.dot
+			echo "$pkg OPP $rec [style=dotted];" | sed -e 's:-:_:g' -e 's:\.:_:g' -e 's:+::g' | sed 's:OPP:->:g' >> ${BUILDHISTORY_DIR_IMAGE}/depends.tmp
 		done
 	done
+	cat ${BUILDHISTORY_DIR_IMAGE}/depends.tmp | sort | uniq >> ${BUILDHISTORY_DIR_IMAGE}/depends.dot
+	rm ${BUILDHISTORY_DIR_IMAGE}/depends.tmp
 	echo "}" >>  ${BUILDHISTORY_DIR_IMAGE}/depends.dot
 
 	cat ${BUILDHISTORY_DIR_IMAGE}/installed-package-sizes.tmp | sort -n -r | awk '{print $1 "\tKiB " $2}' > ${BUILDHISTORY_DIR_IMAGE}/installed-package-sizes.txt
-- 
1.7.2.5

Re: [RFC v2][PATCH 1/3] buildhistory: record all builds

[Paul Eggleton]

On Tuesday 20 March 2012 14:38:52 Koen Kooi wrote:
> Allow empty commits, this also give a nice speedup since 'git status
> --porcelain' doesn't need to get run.

I guess my thinking before (without any verification) was that git commit would 
need to be running the equivalent of 'git status --porcelain' anyway and the 
second time it would be cached so there wouldn't be much impact. If the 
buildhistory repo is quite large or the machine is busy then it's entirely 
possible that wouldn't work out however.

Since all image builds will force a commit anyway (as at least build-id must 
change), and for the majority of people most builds will be image builds, I 
think this is going to be a net improvement.

Therefore I'll say:

Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com>

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [RFC v2][PATCH 1/3] buildhistory: record all builds

[Koen Kooi]

Op 20 mrt. 2012, om 14:51 heeft Paul Eggleton het volgende geschreven:

> On Tuesday 20 March 2012 14:38:52 Koen Kooi wrote:
>> Allow empty commits, this also give a nice speedup since 'git status
>> --porcelain' doesn't need to get run.
> 
> I guess my thinking before (without any verification) was that git commit would 
> need to be running the equivalent of 'git status --porcelain' anyway

It does

> and the 
> second time it would be cached so there wouldn't be much impact.

On my non-SSD machine the cache doesn't seem to be working out :(

> If the 
> buildhistory repo is quite large or the machine is busy then it's entirely 
> possible that wouldn't work out however.

Exactly!

regards,

Koen

Re: [RFC v2][PATCH 1/3] buildhistory: record all builds

[Koen Kooi]

Op 20 mrt. 2012, om 14:51 heeft Paul Eggleton het volgende geschreven:

> On Tuesday 20 March 2012 14:38:52 Koen Kooi wrote:
>> Allow empty commits, this also give a nice speedup since 'git status
>> --porcelain' doesn't need to get run.
> 
> I guess my thinking before (without any verification) was that git commit would 
> need to be running the equivalent of 'git status --porcelain' anyway and the 
> second time it would be cached so there wouldn't be much impact. If the 
> buildhistory repo is quite large or the machine is busy then it's entirely 
> possible that wouldn't work out however.
> 
> Since all image builds will force a commit anyway (as at least build-id must 
> change), and for the majority of people most builds will be image builds, I 
> think this is going to be a net improvement.
> 
> Therefore I'll say:
> 
> Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com>

After a few more days of usage I'm going to create a follow-up patch that will do:

1) reinstate the global git status --porcelain and make a single, empty commit if nothing changed instead of N empty commits.
2) either parse the output of git status or re-run it to see which top level entries need to get committed and only run 'git add ; git commit' on those.

That will keep tracking all builds, but will be less noisy. But it will be slightly slower when the OS doesn't cache the git status, but I decided that I can personally live with that.

So what do you think about that? If you like the idea, would you like it as follow up or as a respin of the series?

regards,

Koen

Re: [RFC v2][PATCH 1/3] buildhistory: record all builds

[Paul Eggleton]

On Thursday 22 March 2012 11:35:30 Koen Kooi wrote:
> After a few more days of usage I'm going to create a follow-up patch that
> will do:
> 
> 1) reinstate the global git status --porcelain and make a single, empty
> commit if nothing changed instead of N empty commits. 2) either parse the
> output of git status or re-run it to see which top level entries need to
> get committed and only run 'git add ; git commit' on those.
>
> That will keep tracking all builds, but will be less noisy. But it will be
> slightly slower when the OS doesn't cache the git status, but I decided
> that I can personally live with that.
> 
> So what do you think about that? 

Well, avoiding >1 empty commits sounds good. I'm happy to go with your 
assessment as you're more likely to be observing actual performance than me 
(I'm not really monitoring the performance of buildhistory on our autobuilder, 
only the output).

FYI I didn't comment on the splitting into separate commits patch earlier 
because it doesn't really bother me either way. I do think people will find it 
easier to use the buildhistory-diff tool rather than looking at the git log 
directly (or the web-based equivalent when I finish that.); but I'm happy to 
continue supporting those that prefer to read the log.

> If you like the idea, would you like it as follow up or as a respin of the
> series?

Since it hasn't been merged I'll go for a respin, if you don't mind.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [RFC v2][PATCH 1/3] buildhistory: record all builds

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1948 bytes --]

On Thu, Mar 22, 2012 at 02:25:38PM +0000, Paul Eggleton wrote:
> On Thursday 22 March 2012 11:35:30 Koen Kooi wrote:
> > After a few more days of usage I'm going to create a follow-up patch that
> > will do:
> > 
> > 1) reinstate the global git status --porcelain and make a single, empty
> > commit if nothing changed instead of N empty commits. 2) either parse the
> > output of git status or re-run it to see which top level entries need to
> > get committed and only run 'git add ; git commit' on those.
> >
> > That will keep tracking all builds, but will be less noisy. But it will be
> > slightly slower when the OS doesn't cache the git status, but I decided
> > that I can personally live with that.
> > 
> > So what do you think about that? 
> 
> Well, avoiding >1 empty commits sounds good. I'm happy to go with your 
> assessment as you're more likely to be observing actual performance than me 
> (I'm not really monitoring the performance of buildhistory on our autobuilder, 
> only the output).

Koen please make it to reuse output..

git status --porcelain takes ages here and even more on my 2nd buildhost
so even with 2nd run just after 1st probably faster I hope it will be
worth reparsing.

OE @ ~/shr-core/tmp-eglibc/buildhistory $ du -sh .
520M    .
and after git gc..
OE @ ~/shr-core/tmp-eglibc/buildhistory $ du -sh .
378M    .

> FYI I didn't comment on the splitting into separate commits patch earlier 
> because it doesn't really bother me either way. I do think people will find it 
> easier to use the buildhistory-diff tool rather than looking at the git log 
> directly (or the web-based equivalent when I finish that.); but I'm happy to 
> continue supporting those that prefer to read the log.

same here

> > If you like the idea, would you like it as follow up or as a respin of the
> > series?

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [RFC v2][PATCH 1/3] buildhistory: record all builds

[Koen Kooi]

Op 22 mrt. 2012, om 15:25 heeft Paul Eggleton het volgende geschreven:

> On Thursday 22 March 2012 11:35:30 Koen Kooi wrote:
>> After a few more days of usage I'm going to create a follow-up patch that
>> will do:
>> 
>> 1) reinstate the global git status --porcelain and make a single, empty
>> commit if nothing changed instead of N empty commits. 2) either parse the
>> output of git status or re-run it to see which top level entries need to
>> get committed and only run 'git add ; git commit' on those.
>> 
>> That will keep tracking all builds, but will be less noisy. But it will be
>> slightly slower when the OS doesn't cache the git status, but I decided
>> that I can personally live with that.
>> 
>> So what do you think about that? 
> 
> Well, avoiding >1 empty commits sounds good. I'm happy to go with your 
> assessment as you're more likely to be observing actual performance than me 
> (I'm not really monitoring the performance of buildhistory on our autobuilder, 
> only the output).
> 
> FYI I didn't comment on the splitting into separate commits patch earlier 
> because it doesn't really bother me either way. I do think people will find it 
> easier to use the buildhistory-diff tool rather than looking at the git log 
> directly (or the web-based equivalent when I finish that.); but I'm happy to 
> continue supporting those that prefer to read the log.
> 
>> If you like the idea, would you like it as follow up or as a respin of the
>> series?
> 
> Since it hasn't been merged I'll go for a respin, if you don't mind.

I'll respin and take Martins concerns in as well. Not sure when the new patchset will be sent, 2 weeks of tradeshows, bspfests and summits ahead :)

regards,

Koen