[PATCH OSSTEST v3] mg-all-branch-statuses: Show how up to date each branch is

[PATCH OSSTEST v3] mg-all-branch-statuses: Show how up to date each branch is

[Ian Campbell]

Using report_find_push_age_info allows us to provide counts of
attempts since the last baseline on current tip as well as the first
attempt of each of those.

Since everything serialises on the repo lock I didn't bother trying to
parallelise anything.

It's a little terse to keep it in 80 chars.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Use new report_find_push_age_info functionality, output condensed.

v3: - Correctly quote parameters to printf, so empty ones don't misalign the
      rest.
    - Drop dates in favour of number of days, and n/a in those cases
    - Print "Error!" if no tip is available and "UpToDate" if tip==basis.

time with an recently updates set of Repos says:

57.34user 28.28system 5:34.16elapsed 25%CPU (0avgtext+0avgdata 47256maxresident)k
100216inputs+600outputs (673major+2332436minor)pagefaults 0swaps

So it's not quick...

Example output:
Branch                       Basis    Tip      #Tip #Tot 1stTip     1stNew
libvirt                      d10a5f58 845184b2    0   10 n/a        8 days
linux-3.0                    e1c63f9f 5dba9ddd    2    9 2 days     868 days
linux-3.10                   b3d78448 UpToDate
linux-3.14                   762167f9 UpToDate
linux-3.16                   162d6432 26749e75    1    1 1 day      1 day
linux-3.18                   d048c068 ea5dd38e    3    3 2 days     2 days
linux-3.4                    bb4a05a0 cf1b3dad   15  161 11 days    212 days
linux-4.1                    b953c0d2 6a010c0a    0    - n/a        n/a
linux-arm-xen                64972ceb UpToDate
linux-linus                  6aaf0da8 4da3064d    0    0 n/a        n/a
linux-mingo-tip-master       d935d0f7 778a1ac5    0   16 n/a        1173 days
linux-next                            c8a9ad22    0  219 n/a        447 days
osstest                      15d2dd50 Error!      0    - n/a        n/a
ovmf                         269e0aeb 288ed590    0    1 n/a        1 day
qemu-mainline                d2966f80 UpToDate
qemu-upstream-4.2-testing    d2382550 UpToDate
qemu-upstream-4.3-testing    efae5e0f UpToDate
qemu-upstream-4.4-testing    32226f42 UpToDate
qemu-upstream-4.5-testing    d9552b0a UpToDate
qemu-upstream-unstable       c4a962ec UpToDate
rumpuserxen                  30d72f3f 3b91e449   61  113 77 days    149 days
seabios                      f24eb2f8 UpToDate
xen-4.0-testing              2692df2a UpToDate
xen-4.1-testing              40feff87 UpToDate
xen-4.2-testing              38fcda22 UpToDate
xen-4.3-testing              e7c02297 UpToDate
xen-4.4-testing              6c1cb3db UpToDate
xen-4.5-testing              e3bd3cef UpToDate
xen-unstable                 c40317f1 3d55a179    0    3 n/a        2 days
---
 mg-all-branch-statuses | 135 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 135 insertions(+)
 create mode 100755 mg-all-branch-statuses

diff --git a/mg-all-branch-statuses b/mg-all-branch-statuses
new file mode 100755
index 0000000..9a78ec2
--- /dev/null
+++ b/mg-all-branch-statuses
@@ -0,0 +1,135 @@
+#!/bin/bash
+# -*- bash -*-
+#
+# Prints the status of each branch
+#
+# Usage:
+#    ./mg-all-branch-statuses [BRANCH....]
+#
+# If no BRANCHes specified, does all that are normally run by
+# cr-daily-branch or out of crontab.
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+set -e
+
+. cri-common
+
+mkdir -p tmp
+
+if [ $# = 0 ]; then
+   set `./mg-list-all-branches`
+fi
+
+gather_info()
+{
+    local branch=$1; shift
+    local basis=$1;  shift
+    local tip=$1;    shift
+
+    select_xenbranch
+
+    local info=`perl -we '
+	use Osstest::Executive;
+	use Osstest;
+	use Data::Dumper;
+	open DEBUG, ">/dev/null" or die $!;
+	#open DEBUG, ">&STDERR" or die $!;
+	csreadconfig();
+	my ($branch,$tree,$basis,$tip) = @ARGV;
+	print DEBUG "branch=$branch tree=$tree basis=$basis tip=$tip\n";
+	my $info = report_find_push_age_info([qw(real adhoc play)],
+					      undef, [($branch)],
+					      $tree, $basis, $tip);
+	print DEBUG Dumper $info;
+	my $onevar = sub {
+		my ($var,$dflt) = @_;
+		$dflt //= "";
+		print "export ".uc($var)."=\"";
+		print $info->{$var}//$dflt;
+		print "\";\n";
+	};
+	my $oneflightvar = sub {
+		my ($flight,$var,$dflt) = @_;
+		$dflt //= "";
+		print "export ".uc($flight)."_".uc($var)."=\"";
+		print $info->{$flight}{$var}//$dflt;
+		print "\";\n";
+	};
+	$onevar->($_, "-") foreach qw(CountTip CountAfterBasis);
+	foreach my $flight (qw(Basis FirstAfterBasis FirstTip)) {
+		$oneflightvar->($flight, $_) foreach qw(flight started);
+	}
+' "$branch" "$tree" "$basis" "$tip"`
+    eval $info
+}
+
+days_since()
+{
+    then=$1 ; shift
+
+    if [ -z "$then" ] ; then
+	echo "n/a"
+	return
+    fi
+
+    local now=$(date +%s)
+    local days=$(( ($now - $then) / 86400 ))
+
+    if [ $days -eq 1 ] ; then
+	echo "$days day"
+    else
+	echo "$days days"
+    fi
+}
+
+printf "%-28s %-8s %-8s %-9s %-10s %-10s\n" \
+	"Branch" "Basis" "Tip" "#Tip #Tot" "1stTip" "1stNew"
+
+for branch in $@; do
+    basis=`./ap-fetch-version-old $branch 2>/dev/null || true`
+    tip=`./ap-fetch-version $branch 2>/dev/null || true`
+
+    gather_info "$branch" "$basis" "$tip"
+
+
+    if [ x$basis != x ] ; then
+	basis=${basis:0:8}
+    else
+	basis=""
+    fi
+
+    if [ x$tip != x ] ; then
+	tip=${tip:0:8}
+    else
+	tip="Error!"
+    fi
+
+    printf "%-28s %-8s " "${branch}" "${basis}"
+
+    if [ x$basis != x$tip ] ; then
+	tipsince=$(days_since "$FIRSTTIP_STARTED")
+	newsince=$(days_since "$FIRSTAFTERBASIS_STARTED")
+
+	printf "%-8s  %3s  %3s %-10s %-10s" "${tip}" \
+	       "$COUNTTIP" "$COUNTAFTERBASIS" "$tipsince" "$newsince"
+    else
+	printf "UpToDate"
+    fi
+
+    printf "\n"
+done
-- 
2.1.4

Re: [PATCH OSSTEST v3] mg-all-branch-statuses: Show how up to date each branch is

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v3] mg-all-branch-statuses: Show how up to date each branch is"):
> Using report_find_push_age_info allows us to provide counts of
> attempts since the last baseline on current tip as well as the first
> attempt of each of those.
> 
> Since everything serialises on the repo lock I didn't bother trying to
> parallelise anything.
...
> Example output:
> Branch                       Basis    Tip      #Tip #Tot 1stTip     1stNew
> libvirt                      d10a5f58 845184b2    0   10 n/a        8 days
> linux-3.0                    e1c63f9f 5dba9ddd    2    9 2 days     868 days

I'm right into nits here, but:

Justification of the "nn days" values is wrong.

> linux-3.10                   b3d78448 UpToDate
> linux-3.14                   762167f9 UpToDate
> linux-3.16                   162d6432 26749e75    1    1 1 day      1 day
> linux-3.18                   d048c068 ea5dd38e    3    3 2 days     2 days
> linux-3.4                    bb4a05a0 cf1b3dad   15  161 11 days    212 days
> linux-4.1                    b953c0d2 6a010c0a    0    - n/a        n/a

This would be easier to read if n/a was RH aligned too.

> linux-arm-xen                64972ceb UpToDate
> linux-linus                  6aaf0da8 4da3064d    0    0 n/a        n/a
> linux-mingo-tip-master       d935d0f7 778a1ac5    0   16 n/a        1173 days
> linux-next                            c8a9ad22    0  219 n/a        447 days
> osstest                      15d2dd50 Error!      0    - n/a        n/a
> ovmf                         269e0aeb 288ed590    0    1 n/a        1 day
> qemu-mainline                d2966f80 UpToDate

That `UpToDate' is the same length and about as visually dense as a
commit id is unfortunate.


I should actually read the code:

> +	my $onevar = sub {
> +		my ($var,$dflt) = @_;
> +		$dflt //= "";
> +		print "export ".uc($var)."=\"";

Why are you exporting these values ?

> +		print $info->{$var}//$dflt;

Why do you abstract away $dflt ?

> +		print "\";\n";
> +	};
> +	my $oneflightvar = sub {

These subs are a lot more sophisticated than they need to be, aren't
they (eg, handling of $dflt) ?  I confess I don't understand why these
are anonymous subrefs rather than simply loop bodies, but fine.

> +		my ($flight,$var,$dflt) = @_;
> +		$dflt //= "";
> +		print "export ".uc($flight)."_".uc($var)."=\"";
> +		print $info->{$flight}{$var}//$dflt;
> +		print "\";\n";
> +	};

A simpler $onevar would let you write

> +	$onevar->($_, "-") foreach qw(CountTip CountAfterBasis);

        $onevar->($info->{$_}, $_)

> +	foreach my $flight (qw(Basis FirstAfterBasis FirstTip)) {
> +		$oneflightvar->($flight, $_) foreach qw(flight started);

                $onevar->($info->{$flight}{$_}, "${flight}_${_}")


> +days_since()
> +{
> +    then=$1 ; shift
> +
> +    if [ -z "$then" ] ; then
> +	echo "n/a"
> +	return
> +    fi
> +
> +    local now=$(date +%s)
> +    local days=$(( ($now - $then) / 86400 ))
> +
> +    if [ $days -eq 1 ] ; then
> +	echo "$days day"
> +    else
> +	echo "$days days"
> +    fi
> +}

Wow, this is quite verbose in shell, isn't it.


> +printf "%-28s %-8s %-8s %-9s %-10s %-10s\n" \
> +	"Branch" "Basis" "Tip" "#Tip #Tot" "1stTip" "1stNew"
> +
> +for branch in $@; do
> +    basis=`./ap-fetch-version-old $branch 2>/dev/null || true`
> +    tip=`./ap-fetch-version $branch 2>/dev/null || true`

This is quite fault-oblivious, isn't it.  Oh well.

> +    if [ x$basis != x ] ; then

I would write "x$basis" in case basis contains spaces.  It _shouldn't_
but the errors if it does will be mysterious.  It's also good
practice.

> +	basis=${basis:0:8}
> +    else
> +	basis=""
> +    fi

This is rather odd.  You truncate it to the first 8 characters but
only if it's not empty.  Why bother checking ?

> +    if [ x$tip != x ] ; then
> +	tip=${tip:0:8}
> +    else
> +	tip="Error!"
> +    fi

Did you know you could write
    ${tip-Error!}
?

Ian.

Re: [PATCH OSSTEST v3] mg-all-branch-statuses: Show how up to date each branch is

[Ian Campbell]

On Thu, 2015-07-02 at 17:30 +0100, Ian Jackson wrote:

> > +printf "%-28s %-8s %-8s %-9s %-10s %-10s\n" \
> > +	"Branch" "Basis" "Tip" "#Tip #Tot" "1stTip" "1stNew"
> > +
> > +for branch in $@; do
> > +    basis=`./ap-fetch-version-old $branch 2>/dev/null || true`
> > +    tip=`./ap-fetch-version $branch 2>/dev/null || true`
> 
> This is quite fault-oblivious, isn't it.  Oh well.

Yes, the problem is that for branches with no baseline:
$ ./ap-fetch-version-old linux-next ; echo $?
HEAD is now at 705bb44... fix
fatal: Couldn't find remote ref tested/linux-next
128
$

And for osstest (where the tip may not be available unless you are
osstest@osstest);
./ap-fetch-version osstest ; echo $?
HEAD is now at 705bb44... fix
fatal: Couldn't find remote ref pretest
fatal: The remote end hung up unexpectedly
128
$

In the latter case I print "Error!" (I could do it here instead of
later, I just noticed).

Not sure what to do about the former. In both cases this obviously
conflates actual failures with expected failures. I'm not sure how to
distinguish. Perhaps I could special case baselineless trees
(linux-next) in ap-fetch-version-old/ap-common and special case osstest
here as the only tree which might plausibly have an unavailable tip?

Ian.

Re: [PATCH OSSTEST v3] mg-all-branch-statuses: Show how up to date each branch is

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST v3] mg-all-branch-statuses: Show how up to date each branch is"):
> On Thu, 2015-07-02 at 17:30 +0100, Ian Jackson wrote:
> > This is quite fault-oblivious, isn't it.  Oh well.
> 
> Yes, the problem is that for branches with no baseline:
> $ ./ap-fetch-version-old linux-next ; echo $?
> HEAD is now at 705bb44... fix
> fatal: Couldn't find remote ref tested/linux-next
> 128
> $
> 
> And for osstest (where the tip may not be available unless you are
> osstest@osstest);
> ./ap-fetch-version osstest ; echo $?
> HEAD is now at 705bb44... fix
> fatal: Couldn't find remote ref pretest
> fatal: The remote end hung up unexpectedly
> 128
> $

Right.  There's not really anything at that layer that knows to say
"no" rather than "aargh".

> In the latter case I print "Error!" (I could do it here instead of
> later, I just noticed).
> 
> Not sure what to do about the former. In both cases this obviously
> conflates actual failures with expected failures. I'm not sure how to
> distinguish. Perhaps I could special case baselineless trees
> (linux-next) in ap-fetch-version-old/ap-common and special case osstest
> here as the only tree which might plausibly have an unavailable tip?

I think that is too big a yak.

Ian.

Re: [PATCH OSSTEST v3] mg-all-branch-statuses: Show how up to date each branch is

[Ian Campbell]

On Fri, 2015-07-03 at 12:26 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST v3] mg-all-branch-statuses: Show how up to date each branch is"):
> > On Thu, 2015-07-02 at 17:30 +0100, Ian Jackson wrote:
> > > This is quite fault-oblivious, isn't it.  Oh well.
> > 
> > Yes, the problem is that for branches with no baseline:
> > $ ./ap-fetch-version-old linux-next ; echo $?
> > HEAD is now at 705bb44... fix
> > fatal: Couldn't find remote ref tested/linux-next
> > 128
> > $
> > 
> > And for osstest (where the tip may not be available unless you are
> > osstest@osstest);
> > ./ap-fetch-version osstest ; echo $?
> > HEAD is now at 705bb44... fix
> > fatal: Couldn't find remote ref pretest
> > fatal: The remote end hung up unexpectedly
> > 128
> > $
> 
> Right.  There's not really anything at that layer that knows to say
> "no" rather than "aargh".
> 
> > In the latter case I print "Error!" (I could do it here instead of
> > later, I just noticed).
> > 
> > Not sure what to do about the former. In both cases this obviously
> > conflates actual failures with expected failures. I'm not sure how to
> > distinguish. Perhaps I could special case baselineless trees
> > (linux-next) in ap-fetch-version-old/ap-common and special case osstest
> > here as the only tree which might plausibly have an unavailable tip?
> 
> I think that is too big a yak.

Phew!