[PATCH OSSTEST v7 1/2] cr-publish-flight-logs: support publishing files from $HOME/public_html

[PATCH OSSTEST v7 1/2] cr-publish-flight-logs: support publishing files from $HOME/public_html

[Ian Campbell]

Any arguments given after the flight are assumed to be files or
directories relative to $HOME/public_html which should be published to
$c{Publish} at the same relative path.

Removing the requirement that a flight be provided, by accepting '',
allows this to be used even in contexts which have no flight.

The results subtree is always published if $c{ResultsPublish} is set,
since it doesn't depend on any particular flight.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v7: New patch
---
 cr-publish-flight-logs | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/cr-publish-flight-logs b/cr-publish-flight-logs
index 94c74c8..0fdd02d 100755
--- a/cr-publish-flight-logs
+++ b/cr-publish-flight-logs
@@ -31,10 +31,10 @@ if (@ARGV && $ARGV[0] eq '--push-harness') {
     $push_harness = 1;
 }
 
-die "usage: ./cr-publish-flight-logs <flight>" unless @ARGV==1;
+die "usage: ./cr-publish-flight-logs <flight|''> [public_html files]" unless @ARGV>=1;
 
 my $flight= shift @ARGV;
-die unless $flight =~ m/^\d+$/;
+die unless $flight =~ m/^\d*$/;
 
 open LOCK, "> $c{GlobalLockDir}/publish-lock" or die $!;
 flock LOCK, LOCK_EX or die $!;
@@ -49,7 +49,7 @@ if ($push_harness) {
     system_checked("ssh $githost 'cd $gitdir && git update-server-info'");
 }
 
-sub copydir ($$) {
+sub copylogs ($$) {
     my ($src,$dst) = @_;
     my @cmd= qw(rsync --compress --compress-level=9 --stats --delete -auH);
     push @cmd, '-e', 'ssh -o batchmode=yes';
@@ -59,5 +59,11 @@ sub copydir ($$) {
     $!=0; $?=0; system @cmd; die "rsync $? $!" if $? or $!;
 }
 
-copydir("$c{Logs}/$flight/", "$c{LogsPublish}/$flight") if $c{LogsPublish};
-copydir("$c{Results}/", "$c{ResultsPublish}") if $c{ResultsPublish};
+copylogs("$c{Logs}/$flight/", "$c{LogsPublish}/$flight")
+    if $flight && $c{LogsPublish};
+copylogs("$c{Results}/", "$c{ResultsPublish}") if $c{ResultsPublish};
+
+if ( $c{Publish} ) {
+    copylogs("$ENV{'HOME'}/public_html/$_", "$c{Publish}/$_")
+	foreach @ARGV;
+}
-- 
2.1.4

[PATCH OSSTEST v7 2/2] mg-all-branch-statuses: Show how up to date each branch is

[Ian Campbell]

Using report_find_push_age_info allows us to provide counts of
attempts since the last baseline on current tip as well as the first
attempt of each of those.

Since everything serialises on the repo lock I didn't bother trying to
parallelise anything.

It's a little terse to keep it in 80 chars.

cron infrastructure is added to run this in a new
$HOME/for-maintjobs.git on a daily basis and push the output to the
location given in the Publish config variable, if it is set (it is set
in production but not for cambridge).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Use new report_find_push_age_info functionality, output condensed.

v3: - Correctly quote parameters to printf, so empty ones don't misalign the
      rest.
    - Drop dates in favour of number of days, and n/a in those cases
    - Print "Error!" if no tip is available and "UpToDate" if tip==basis.

v4: - Simplify $onevar, dropping $dflt, dropping export, subsuming
      $oneflightvar.
    - Right justify some stuff
    - s/UpToDate/same/
    - Simplify some pointless or complex shell conditionals

v5: - Add cron infrastructure

v6: - Propagate cron output to logs (untested apart from "bash -n" due
      to lack of local infra).

v7: - Use cr-publish-flight-logs.

time with an recently updates set of Repos says:

57.34user 28.28system 5:34.16elapsed 25%CPU (0avgtext+0avgdata 47256maxresident)k
100216inputs+600outputs (673major+2332436minor)pagefaults 0swaps

So it's not quick...

Example output:
Branch                       Basis    Tip      #Tip #Tot 1stTip     1stNew
libvirt                      d10a5f58 28540794    0   11   n/a          9 days
linux-3.0                    e1c63f9f 5dba9ddd    2    9     3 days   870 days
linux-3.10                   b3d78448 same
linux-3.14                   762167f9 same
linux-3.16                   26749e75 same
linux-3.18                   d048c068 ea5dd38e    5    5     3 days     3 days
linux-3.4                    bb4a05a0 cf1b3dad   17  163    12 days   213 days
linux-4.1                    b953c0d2 6a010c0a    0    0   n/a        n/a
linux-arm-xen                64972ceb same
linux-linus                  6aaf0da8 9bdc771f    0    1   n/a          1 day
linux-mingo-tip-master       d935d0f7 21316d69    0   16   n/a       1175 days
linux-next                            e58c2435    0  219   n/a        448 days
osstest                      15d2dd50 Error!      0    -   n/a        n/a
ovmf                         3735017c 0bb5d7a5    0    1   n/a          0 days
qemu-mainline                d2966f80 35360642    0    1   n/a          0 days
qemu-upstream-4.2-testing    d2382550 same
qemu-upstream-4.3-testing    efae5e0f same
qemu-upstream-4.4-testing    32226f42 same
qemu-upstream-4.5-testing    d9552b0a same
qemu-upstream-unstable       c4a962ec same
rumpuserxen                  30d72f3f 3b91e449   62  114    78 days   150 days
seabios                      f24eb2f8 6cfebb4e    0    0   n/a        n/a
xen-4.0-testing              2692df2a same
xen-4.1-testing              40feff87 same
xen-4.2-testing              38fcda22 same
xen-4.3-testing              e7c02297 same
xen-4.4-testing              6c1cb3db same
xen-4.5-testing              e3bd3cef same
xen-unstable                 c40317f1 9379af08    0    5   n/a          3 days
---
 cr-all-branch-statuses |  30 +++++++++++++
 crontab                |   1 +
 crontab-cambridge      |   1 +
 mg-all-branch-statuses | 120 +++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 152 insertions(+)
 create mode 100755 cr-all-branch-statuses
 create mode 100755 mg-all-branch-statuses

diff --git a/cr-all-branch-statuses b/cr-all-branch-statuses
new file mode 100755
index 0000000..a31c512
--- /dev/null
+++ b/cr-all-branch-statuses
@@ -0,0 +1,30 @@
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+set -ex
+
+. cri-args-hostlists
+branch=$1; shift
+
+check_stop all-branch-statuses.
+
+abs=$HOME/public_html/all-branch-statuses.txt
+( date ; echo ; bash -x ./mg-all-branch-statuses ) > $abs.new
+mv $abs.new $abs
+
+./cr-publish-flight-logs '' all-branch-statuses.txt
diff --git a/crontab b/crontab
index 5dc8675..436645b 100644
--- a/crontab
+++ b/crontab
@@ -7,4 +7,5 @@ MAILTO=ian.jackson@citrix.com,ian.campbell@eu.citrix.com
 18		4	* * *		cd testing.git && BRANCHES='linux-linus linux-mingo-tip-master linux-3.0 libvirt rumpuserxen' ./cr-for-branches branches -w "./cr-daily-branch --real"
 6-59/15   	*	* * *		cd testing.git && EXTRA_BRANCHES='linux-linus linux-3.0 rumpuserxen libvirt' ./cr-for-branches bisects -w "./cr-try-bisect --real"
 #8-59/5		*	* * *		cd bisects/adhoc.git &&	with-lock-ex -q data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
+22		8	* * *		cd testing.git && BRANCHES=maintjobs		./cr-for-branches . -w ./cr-all-branch-statuses ''
 3		4	* * *		savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
diff --git a/crontab-cambridge b/crontab-cambridge
index 60bb4fd..2146311 100644
--- a/crontab-cambridge
+++ b/crontab-cambridge
@@ -2,4 +2,5 @@ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
 MAILTO=ian.jackson@citrix.com,ian.campbell@eu.citrix.com
 # m            h       dom mon dow     command
 4-59/30        *       * * *           cd testing.git && BRANCHES=osstest              ./cr-for-branches branches -q "./cr-daily-branch --real"
+22		8	* * *		cd testing.git && BRANCHES=maintjobs		./cr-for-branches . -w ./cr-all-branch-statuses ''
 3              4       * * *           savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
diff --git a/mg-all-branch-statuses b/mg-all-branch-statuses
new file mode 100755
index 0000000..8b86e3b
--- /dev/null
+++ b/mg-all-branch-statuses
@@ -0,0 +1,120 @@
+#!/bin/bash
+# -*- bash -*-
+#
+# Prints the status of each branch
+#
+# Usage:
+#    ./mg-all-branch-statuses [BRANCH....]
+#
+# If no BRANCHes specified, does all that are normally run by
+# cr-daily-branch or out of crontab.
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+set -e
+
+. cri-common
+
+mkdir -p tmp
+
+if [ $# = 0 ]; then
+   set `./mg-list-all-branches`
+fi
+
+gather_info()
+{
+    local branch=$1; shift
+    local basis=$1;  shift
+    local tip=$1;    shift
+
+    select_xenbranch
+
+    local info=`perl -we '
+	use Osstest::Executive;
+	use Osstest;
+	use Data::Dumper;
+	open DEBUG, ">/dev/null" or die $!;
+	#open DEBUG, ">&STDERR" or die $!;
+	csreadconfig();
+	my ($branch,$tree,$basis,$tip) = @ARGV;
+	print DEBUG "branch=$branch tree=$tree basis=$basis tip=$tip\n";
+	my $info = report_find_push_age_info([qw(real adhoc play)],
+					      undef, [($branch)],
+					      $tree, $basis, $tip);
+	print DEBUG Dumper $info;
+	my $onevar = sub {
+		my ($var,$val) = @_;
+		$val //= "";
+		print uc($var)."=\"$val\";\n";
+	};
+	$onevar->($_, $info->{$_}//"-")
+		foreach qw(CountTip CountAfterBasis);
+	foreach my $flight (qw(Basis FirstAfterBasis FirstTip)) {
+		$onevar->("${flight}_${_}", $info->{$flight}{$_})
+			foreach qw(flight started);
+	}
+' "$branch" "$tree" "$basis" "$tip"`
+    eval $info
+}
+
+days_since() # Must output 10 characters
+{
+    then=$1 ; shift
+
+    if [ -z "$then" ] ; then
+	echo "  n/a     "
+	return
+    fi
+
+    local now=$(date +%s)
+    local days=$(( ($now - $then) / 86400 ))
+
+    if [ $days -eq 1 ] ; then
+	printf "%5d day " $days
+    else
+	printf "%5d days" $days
+    fi
+}
+
+printf "%-28s %-8s %-8s %-9s %-10s %-10s\n" \
+	"Branch" "Basis" "Tip" "#Tip #Tot" "1stTip" "1stNew"
+
+for branch in $@; do
+    basis=`./ap-fetch-version-old $branch 2>/dev/null || true`
+    tip=`./ap-fetch-version $branch 2>/dev/null || true`
+
+    gather_info "$branch" "$basis" "$tip"
+
+    basis=${basis:0:8}
+
+    tip=${tip:0:8}
+    : ${tip:=Error!}
+
+    printf "%-28s %-8s " "${branch}" "${basis}"
+
+    if [ x$basis != x$tip ] ; then
+	tipsince=$(days_since "$FIRSTTIP_STARTED")
+	newsince=$(days_since "$FIRSTAFTERBASIS_STARTED")
+
+	printf "%-8s  %3s  %3s %10s %10s" "${tip}" \
+	       "$COUNTTIP" "$COUNTAFTERBASIS" "$tipsince" "$newsince"
+    else
+	printf "same"
+    fi
+
+    printf "\n"
+done
-- 
2.1.4

Re: [PATCH OSSTEST v7 1/2] cr-publish-flight-logs: support publishing files from $HOME/public_html

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v7 1/2] cr-publish-flight-logs: support publishing files from $HOME/public_html"):
> Any arguments given after the flight are assumed to be files or
> directories relative to $HOME/public_html which should be published to
> $c{Publish} at the same relative path.
> 
> Removing the requirement that a flight be provided, by accepting '',
> allows this to be used even in contexts which have no flight.
> 
> The results subtree is always published if $c{ResultsPublish} is set,
> since it doesn't depend on any particular flight.

Uh.  Looking at this, I hadn't spotted that in your other patch you
generate the output in ~/public_html.  Shouldn't it be in results/
(which is nowadays a general place for publicly visible output) ?

At the very least, if you do mean ~/public_html, you should be using
WebspaceFile.

Also:

> -sub copydir ($$) {
> +sub copylogs ($$) {

Why did you rename this function ?

Ian.

Re: [PATCH OSSTEST v7 1/2] cr-publish-flight-logs: support publishing files from $HOME/public_html

[Ian Campbell]

On Thu, 2015-07-09 at 13:48 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v7 1/2] cr-publish-flight-logs: support publishing files from $HOME/public_html"):
> > Any arguments given after the flight are assumed to be files or
> > directories relative to $HOME/public_html which should be published to
> > $c{Publish} at the same relative path.
> > 
> > Removing the requirement that a flight be provided, by accepting '',
> > allows this to be used even in contexts which have no flight.
> > 
> > The results subtree is always published if $c{ResultsPublish} is set,
> > since it doesn't depend on any particular flight.
> 
> Uh.  Looking at this, I hadn't spotted that in your other patch you
> generate the output in ~/public_html.  Shouldn't it be in results/
> (which is nowadays a general place for publicly visible output) ?
> 
> At the very least, if you do mean ~/public_html, you should be using
> WebspaceFile.

Results would be ok too, if you prefer (which you seem to so I shall
make that change).

> 
> Also:
> 
> > -sub copydir ($$) {
> > +sub copylogs ($$) {
> 
> Why did you rename this function ?

I meant to mention this in the commit message and forgot, it was just
because the thing I was now copying wasn't a dir...

But if I generate the file into results this isn't need, the existing
publication of Results will do the job for free.

Ian.

Re: [PATCH OSSTEST v7 1/2] cr-publish-flight-logs: support publishing files from $HOME/public_html

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST v7 1/2] cr-publish-flight-logs: support publishing files from $HOME/public_html"):
> On Thu, 2015-07-09 at 13:48 +0100, Ian Jackson wrote:
> > At the very least, if you do mean ~/public_html, you should be using
> > WebspaceFile.
> 
> Results would be ok too, if you prefer (which you seem to so I shall
> make that change).

Thanks.

> > Also:
> > 
> > > -sub copydir ($$) {
> > > +sub copylogs ($$) {
> > 
> > Why did you rename this function ?
> 
> I meant to mention this in the commit message and forgot, it was just
> because the thing I was now copying wasn't a dir...
> 
> But if I generate the file into results this isn't need, the existing
> publication of Results will do the job for free.

Indeed.

Ian.