[OSSTEST PATCH 0/6] Trivial reporting improvements

[OSSTEST PATCH 0/6] Trivial reporting improvements

[Ian Jackson]

Six tiny patches to make the html output from sg-report-flight better,
prompted by looking at the output critically following recent
questions.

 1/6 show_abs_time: break out from sg-report-flight
 2/6 TestSupport: logm: use show_abs_time
 3/6 sg-report-flight: mkdir the per-job directories
 4/6 sg-report-flight: show the job recipe and
 5/6 sg-report-flight: Add some flight information
 6/6 sg-report-flight: report when flight finished

I intend to push these when the push gate has a quiet moment,
unless anyone objects.

[OSSTEST PATCH 1/6] show_abs_time: break out from sg-report-flight

[Ian Jackson]

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest.pm       |    7 ++++++-
 sg-report-flight |    4 +---
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/Osstest.pm b/Osstest.pm
index 1033dd1..fc9698b 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -35,7 +35,7 @@ BEGIN {
                       $dbh_tests db_retry db_retry_retry db_retry_abort
                       db_begin_work
                       ensuredir get_filecontents_core_quiet system_checked
-                      nonempty visible_undef
+                      nonempty visible_undef show_abs_time
                       );
     %EXPORT_TAGS = ( );
 
@@ -286,4 +286,9 @@ sub visible_undef ($) {
     return defined $v ? $v : '<undef>';
 }
 
+sub show_abs_time ($) {
+    my ($timet) = @_;
+    return strftime "%Y-%m-%d %H:%M:%S Z", gmtime $timet;
+}
+
 1;
diff --git a/sg-report-flight b/sg-report-flight
index 69ef252..e6f3339 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -808,9 +808,7 @@ END
             encode_entities($st);
         $issteplog{$logfilename}= 1;
 
-        printf H "<td>%s</td>",
-            encode_entities(strftime "%Y-%m-%d %H:%M:%S Z",
-                            gmtime $step->{started});
+        printf H "<td>%s</td>", show_abs_time $step->{started};
 
 	if (defined $step->{finished} && defined $step->{started}) {
 	    printf H "<td align=right>%d</td>\n",
-- 
1.7.10.4

[OSSTEST PATCH 2/6] TestSupport: logm: use show_abs_time

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm |    4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index a957a69..0db1366 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -175,9 +175,7 @@ sub ts_get_host_guest { # pass this @ARGV
 sub logm ($) {
     my ($m) = @_;
     my @t = gmtime;
-    my $fm = sprintf "%04d-%02d-%02d %02d:%02d:%02d Z %s\n",
-		$t[5]+1900,$t[4]+1,$t[3], $t[2],$t[1],$t[0],
-		$m;
+    my $fm = (show_abs_time time)." $m\n";
     foreach my $h ((ref($logm_handle) eq 'ARRAY')
 		   ? @$logm_handle : $logm_handle) {
 	print $h $fm or die $!;
-- 
1.7.10.4

[OSSTEST PATCH 3/6] sg-report-flight: mkdir the per-job directories

[Ian Jackson]

This makes sure the per-job pages always get created.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-flight |   12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/sg-report-flight b/sg-report-flight
index e6f3339..1a7c316 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -27,6 +27,7 @@ use URI::Escape;
 use POSIX;
 use IPC::Open2;
 use Data::Dumper;
+use File::Path;
 
 use Osstest;
 use Osstest::Executive;
@@ -757,11 +758,12 @@ sub htmloutjob ($$) {
     my ($fi,$job) = @_;
     return unless defined $htmldir;
 
-    my $htmlfile= "$htmldir/$job/$htmlleaf";
-    if (!open H, "> $htmlfile.new") {
-        return if $!==&ENOENT;
-        die "$htmlfile.new $!";
-    }
+    my $jobhtmldir = "$htmldir/$job";
+    mkpath $jobhtmldir;
+
+    my $htmlfile= "$jobhtmldir/$htmlleaf";
+    open H, "> $htmlfile.new" or die "$htmlfile.new $!";
+
     my $title= "Info on flight $fi->{Flight} job $job";
     my $branch= $fi->{FlightInfo}{branch};
 
-- 
1.7.10.4

[OSSTEST PATCH 4/6] sg-report-flight: show the job recipe and overall status

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-flight |   13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/sg-report-flight b/sg-report-flight
index 1a7c316..0bf2b5d 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -767,6 +767,13 @@ sub htmloutjob ($$) {
     my $title= "Info on flight $fi->{Flight} job $job";
     my $branch= $fi->{FlightInfo}{branch};
 
+    our $htmlout_jobq ||= $dbh_tests->prepare(<<END);
+        SELECT * FROM jobs WHERE flight = ? AND job = ?
+END
+    $htmlout_jobq->execute($fi->{Flight}, $job);
+    my $ji = $htmlout_jobq->fetchrow_hashref();
+    die unless $ji;
+
     print H <<END;
 <html><head><title>$title</title><head>
 <body>
@@ -822,6 +829,12 @@ END
 
     print H <<END;
 </table>
+<h2>Job</h2>
+<table>
+<tr><td>Recipe:</td><td>$ji->{recipe}</td></tr>
+<tr><td>Status:</td><td>$ji->{status}</td></tr>
+</table>
+<p>
 <h2>Logfiles etc.</h2>
 For main test script logfiles, see entries in steps table.
 <ul>
-- 
1.7.10.4

[OSSTEST PATCH 5/6] sg-report-flight: Add some flight information

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-flight |   27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/sg-report-flight b/sg-report-flight
index 0bf2b5d..b643816 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -994,7 +994,7 @@ END
     }
     my @vercols= sort keys %vercols;
     if (@vercols) {
-        print H "<h2>targeted versions</h2>\n";
+        print H "<h2>Targeted versions</h2>\n";
         print H "<table rules=all>";
         print H "<tr><td></td>";
         print H "<th>".encode_entities($_)."</th>" foreach @vercols;
@@ -1012,6 +1012,31 @@ END
         }
         print H "</table>";
     }
+    my $started_str = show_abs_time $fi->{FlightInfo}{started};
+
+    print H <<END;
+<h2>Flight information</h2>
+<table>
+<tr>
+<td>"Branch":</td><td>$fi->{FlightInfo}{branch}</td></tr>
+<td>Intended blessing:</td><td>$fi->{FlightInfo}{intended}</td></tr>
+<td>Blessing:</td><td>$fi->{FlightInfo}{blessing}</td></tr>
+<td>Started:</td><td>$started_str</td></tr>
+<td>Test harness revision(s):</td><td>
+END
+
+    our $htmlout_touchedq ||= $dbh_tests->prepare(<<END);
+        SELECT harness FROM flights_harness_touched WHERE flight=?
+END
+    $htmlout_touchedq->execute($fi->{Flight});
+    while (my ($touched) = $htmlout_touchedq->fetchrow_array()) {
+	print H $touched, "\n";
+    }
+
+    print H <<END;
+</td></tr>
+</table>
+END
     print H "</body></html>";
     H->error and die $!;
     close H or die $!;
-- 
1.7.10.4

[OSSTEST PATCH 6/6] sg-report-flight: report when flight finished

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 sg-report-flight |   13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/sg-report-flight b/sg-report-flight
index b643816..4f900c5 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -1014,6 +1014,18 @@ END
     }
     my $started_str = show_abs_time $fi->{FlightInfo}{started};
 
+    our $htmlout_finishedq ||= $dbh_tests->prepare(<<END);
+        SELECT
+ (SELECT max(finished) FROM steps WHERE flight=?)                 max,
+ (SELECT count(*) FROM steps WHERE finished IS NULL AND flight=?) unfinished
+END
+    $htmlout_finishedq->execute($fi->{Flight}, $fi->{Flight});
+    my $frow = $htmlout_finishedq->fetchrow_hashref();
+    my $finished_str =
+	$frow->{unfinished} ? "($frow->{unfinished} unfinished steps)" :
+	!$frow->{max} ? "(no steps)" :
+	show_abs_time($frow->{max});
+
     print H <<END;
 <h2>Flight information</h2>
 <table>
@@ -1022,6 +1034,7 @@ END
 <td>Intended blessing:</td><td>$fi->{FlightInfo}{intended}</td></tr>
 <td>Blessing:</td><td>$fi->{FlightInfo}{blessing}</td></tr>
 <td>Started:</td><td>$started_str</td></tr>
+<td>Finished:</td><td>$finished_str</td></tr>
 <td>Test harness revision(s):</td><td>
 END
 
-- 
1.7.10.4

Re: [OSSTEST PATCH 0/6] Trivial reporting improvements

[Ian Campbell]

On Mon, 2014-04-07 at 17:27 +0100, Ian Jackson wrote:
> Six tiny patches to make the html output from sg-report-flight better,
> prompted by looking at the output critically following recent
> questions.
> 
>  1/6 show_abs_time: break out from sg-report-flight
>  2/6 TestSupport: logm: use show_abs_time
>  3/6 sg-report-flight: mkdir the per-job directories
>  4/6 sg-report-flight: show the job recipe and
>  5/6 sg-report-flight: Add some flight information
>  6/6 sg-report-flight: report when flight finished
> 
> I intend to push these when the push gate has a quiet moment,
> unless anyone objects.

Looks good to me.

One small query: #5/6 reports "Test harness revision(s)" -- can there
really be multiple for a given flight? I thought the per-branch repo was
only updated at the start of the flight?


Ian.

Re: [OSSTEST PATCH 0/6] Trivial reporting improvements

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 0/6] Trivial reporting improvements"):
> One small query: #5/6 reports "Test harness revision(s)" -- can there
> really be multiple for a given flight? I thought the per-branch repo was
> only updated at the start of the flight?

There's not supposed to be for production flights.

But that database table is updated (when using
Osstest::JobDB::Executive) by dbfl_check, which is called numerous
times including from tsreadconfig(), at the start of each ts-* script.

To avoid having to record lies in the database when running tests by
hand etc., the database table (flights_harness_touched) can record
multiple versions of osstest.git which have touched the flight.  ("+"
is appended if the working tree is dirty.)

It's obviously better that sg-report-flight should report such
situations accurately, even if they are abnormal.

Ian.

Re: [OSSTEST PATCH 0/6] Trivial reporting improvements

[Ian Campbell]

On Mon, 2014-04-07 at 17:45 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [OSSTEST PATCH 0/6] Trivial reporting improvements"):
> > One small query: #5/6 reports "Test harness revision(s)" -- can there
> > really be multiple for a given flight? I thought the per-branch repo was
> > only updated at the start of the flight?
> 
> There's not supposed to be for production flights.
> 
> But that database table is updated (when using
> Osstest::JobDB::Executive) by dbfl_check, which is called numerous
> times including from tsreadconfig(), at the start of each ts-* script.
> 
> To avoid having to record lies in the database when running tests by
> hand etc., the database table (flights_harness_touched) can record
> multiple versions of osstest.git which have touched the flight.  ("+"
> is appended if the working tree is dirty.)
> 
> It's obviously better that sg-report-flight should report such
> situations accurately, even if they are abnormal.

Makes sense! Thanks.