[OSSTEST PATCH 00/17] Fix substep (eg XTF) logfile links

[OSSTEST PATCH 00/17] Fix substep (eg XTF) logfile links

[Ian Jackson]

It kept irritating me that the links to the substeps of xtf-run do not
work.  To fix this, it is necessary for the database to contain the
right information - ie, a schema change.

After this series, substep logfile names are recorded and reported
properly, and you can click on them.

At some point in the future, when enough browsers honour `#line=', it
might be worth adding that - that would be another column, probably.
For now, the links for each step just go to the top of the logfile.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 01/17] Developer docs: A rune for flight comparision

[Ian Jackson]

Something like this rune can be useful for comparing testids of
different flights.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 NOTES.flight-comparison | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
 create mode 100644 NOTES.flight-comparison

diff --git a/NOTES.flight-comparison b/NOTES.flight-comparison
new file mode 100644
index 0000000..ea290f8
--- /dev/null
+++ b/NOTES.flight-comparison
@@ -0,0 +1,22 @@
+select
+  l.job, l.stepno, l.testid, l.status,
+  r.job, r.stepno, r.testid, r.status
+from
+  (select * from steps where flight=110909) l
+  full outer join
+  (select * from steps where flight=110951) r
+on
+  regexp_replace(l.testid,'\([0-9]*\)$','') =
+  regexp_replace(r.testid,'\([0-9]*\)$','')
+      and
+  l.job=r.job
+      and
+  (l.stepno=r.stepno or l.stepno+1=r.stepno)
+where
+  coalesce(r.testid,'') != 'syslog-server'
+      and
+  (l.testid is null or r.testid is null or l.status!=r.status)
+order by
+  coalesce(l.job, r.job),
+  l.stepno, r.stepno,
+  l.testid, r.testid;
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 02/17] schema updates: Fix typo in developer doc

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 schema/README.updates | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/schema/README.updates b/schema/README.updates
index 25bc11a..c5abc36 100644
--- a/schema/README.updates
+++ b/schema/README.updates
@@ -109,7 +109,7 @@ on it.  There are typically two schema changes:
 5. Commit: code to populate new column; changing `add' to status
    Needed and `constraint' to status Ready.
 
-6. Optionally commit: code which read new column, but which tolerates
+6. Optionally commit: code which reads new column, but which tolerates
    it containing NULL/DEFAULT.  (`add' is already Needed.)
 
 7. If necessary commit: idempotent utility script to populate missing
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 03/17] schema updates: Document how to remove a used column

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 schema/README.updates | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/schema/README.updates b/schema/README.updates
index c5abc36..ca1d37e 100644
--- a/schema/README.updates
+++ b/schema/README.updates
@@ -147,6 +147,32 @@ finish is needed', this will usually involve passing an appropriate
 `-o' option.
 
 
+Update order for Cleanup
+------------------------
+
+This is for when we want to get rid of some old information in the
+database.  Sometimes the information will have been replaced eg via
+a Populate-then-Rely change, as described above.
+
+There are typically two schema changes:
+
+* To allow the new information to be left blank (eg for the column to
+  be NULL).  Call this `unconstrain'; it is an `Explicit conditional'
+  change.
+
+* To drop the information entirely.  I will call this `drop'.
+  It is `Code first'.
+
+1. Commit: new schema updates `unconstrain', status `Unfinished'
+   and `drop', status `Unfinished'.
+
+2. Commit code which tolerates the information being missing;
+   changing `unconstrain' to Ready.
+
+3. Commit code which no longer writes the information; changing
+   `unconstrain' to Needed and `drop' to Ready.
+
+
 Statuses and rules for push and db update
 -----------------------------------------
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 04/17] schema updates: mg-schema-test-database: Cope with applied updates

[Ian Jackson]

The schema_updates table contents are written by the call to
mg-schema-update.  Trying to copy the contents from the source db
correctly fails with duplicate key errors.  So don't do that.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 mg-schema-test-database | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mg-schema-test-database b/mg-schema-test-database
index 0517ca3..641aadb 100755
--- a/mg-schema-test-database
+++ b/mg-schema-test-database
@@ -467,6 +467,9 @@ END
 END
 
 	for table in $tables; do
+		case $table in
+		schema_updates)	continue ;;
+		esac
 		case " $ftables " in
 		*" $table "*)	condition="flight >= $minflight" ;;
 		*)		condition="1=1" ;;
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 05/17] schema updates: Be willing to apply Ready changes

[Ian Jackson]

If the oldest code declares itself Ready for this change, or
(implausibly) Needed, then it is OK to push.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 mg-schema-update | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mg-schema-update b/mg-schema-update
index 43f313a..44e4283 100755
--- a/mg-schema-update
+++ b/mg-schema-update
@@ -102,7 +102,7 @@ sub getstate () {
 	    if (!$there) {
 		$out->{Todo} = 1;
 		$out->{Msg} = "would break any old code";
-	    } elsif ($tst =~ m/Harmless|Preparatory/) {
+	    } elsif ($tst =~ m/Harmless|Preparatory|Ready|Needed/) {
 		$out->{Todo} = 2;
 		$out->{Msg} = "specified revision can cope";
 	    } else {
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 06/17] db updates: Make all INSERTs (except into steps) name columns explicitly

[Ian Jackson]

This makes the code more robust against schema updates.  Specifically,
if any columns were to be deleted, these runes would break.  (Also if
any column was inserted other than after the existing columns,
although this is not something Postgres can currently do AIUI.)

Affected tables are:
  flights_harness_touched
  jobs
  resources
  runvars
  steps

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/JobDB/Executive.pm |  4 +++-
 Osstest/TestSupport.pm     | 12 +++++++++---
 cs-adjust-flight           |  7 ++++---
 cs-bisection-step          |  3 ++-
 mg-allocate                |  2 +-
 mg-hosts                   |  1 +
 tcl/JobDB-Executive.tcl    |  4 +++-
 7 files changed, 23 insertions(+), 10 deletions(-)

diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index 45aabd4..0d1b3be 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -191,7 +191,9 @@ END
 
     if (!$already) {
         $dbh_tests->do(<<END, {}, $fl,$rev);
-            INSERT INTO flights_harness_touched VALUES (?,?)
+            INSERT INTO flights_harness_touched
+                        (flight, harness)
+                 VALUES (?,?)
 END
     }
 }
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index e6599db..f10d56e 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -309,7 +309,9 @@ sub store_runvar ($$) {
 		  WHERE flight=? AND job=? AND name=? AND synth='t'
 END
         $dbh_tests->do(<<END,{}, $flight,$job, $param,$value);
-            INSERT INTO runvars VALUES (?,?,?,?,'t')
+            INSERT INTO runvars
+                        (flight, job, name, val, synth)
+                 VALUES (?,?,?,?,'t')
 END
     });
     $r{$param}= get_runvar($param, "$flight.$job");
@@ -368,7 +370,9 @@ END
 		  WHERE flight=? AND job=? AND name=? AND synth='t'
 END
 	$dbh_tests->do(<<END, undef, $flight, $job, $param, $value+1);
-            INSERT INTO runvars VALUES (?,?,?,?,'t')
+            INSERT INTO runvars
+                        (flight, job, name, val, synth)
+                 VALUES (?,?,?,?,'t')
 END
     });
     logm("runvar increment: $param=$value");
@@ -1775,7 +1779,9 @@ END
             logm("select_ether $prefix:... $ether (first in flight)");
         }
         $dbh_tests->do(<<END, {}, $flight,$job,$vn,$ether);
-            INSERT INTO runvars VALUES (?,?,?,?,'t')
+            INSERT INTO runvars
+                        (flight, job, name, val, synth)
+                 VALUES (?,?,?,?,'t')
 END
         my $chkrow= $dbh_tests->selectrow_hashref(<<END,{}, $flight);
 	    SELECT val, count(*) FROM runvars WHERE flight=?
diff --git a/cs-adjust-flight b/cs-adjust-flight
index 03257a6..0be104e 100755
--- a/cs-adjust-flight
+++ b/cs-adjust-flight
@@ -196,11 +196,11 @@ sub copy_jobs ($$) {
     my ($srcflight, $jobs) = @_;
     prep_rm_jobs();
     my @job_copy_qs = map { $dbh_tests->prepare($_) }
-        ("INSERT INTO jobs".
+        ("INSERT INTO jobs (flight, job, recipe, status)".
          "     SELECT ?, job, recipe, 'queued'".
          "       FROM jobs".
          "      WHERE flight = ? AND job = ?",
-         "INSERT INTO runvars".
+         "INSERT INTO runvars (flight, job, name, val, synth)".
          "     SELECT ?, job, name, val, 'f'".
          "       FROM runvars".
          "      WHERE flight = ? AND job = ? AND NOT synth");
@@ -246,7 +246,8 @@ sub change__copy_jobs {
 our $runvar_rm_q = $dbh_tests->prepare
     ("DELETE FROM runvars WHERE flight = ? AND job = ? AND name = ?");
 our $runvar_insert_q = $dbh_tests->prepare
-    ("INSERT INTO runvars VALUES (?, ?, ?, ?, 'f')");
+    ("INSERT INTO runvars (flight, job, name, val, synth)".
+     " VALUES (?, ?, ?, ?, 'f')");
 
 sub runvar_set ($$$;$) {
     my ($job, $name, $val, $xwhat) = @_;
diff --git a/cs-bisection-step b/cs-bisection-step
index 76f5010..13ccd7a 100755
--- a/cs-bisection-step
+++ b/cs-bisection-step
@@ -1255,7 +1255,8 @@ END
             " from $copyflight)\n";
 
         $dbh_tests->do(<<END, {}, $popflight,$popjob,$recipe);
-            INSERT INTO jobs VALUES (?, ?, ?, 'queued')
+            INSERT INTO jobs (flight, job, recipe, status)
+                      VALUES (?, ?, ?, 'queued')
 END
         $dbh_tests->do(<<END, {}, $popflight,$popjob);
             INSERT INTO runvars (
diff --git a/mg-allocate b/mg-allocate
index 4f02ce5..dadac3c 100755
--- a/mg-allocate
+++ b/mg-allocate
@@ -180,7 +180,7 @@ sub alloc_1rescand ($$) {
 
     if ($allocate && $restype eq 'share-flight' && $shareix == $tid) {
 	$dbh_tests->do(<<END,{},
-            INSERT INTO resources
+            INSERT INTO resources (restype, resname, shareix, owntaskid)
                  (SELECT ? AS restype,
                          ? AS resname,
                          ? AS shareix,
diff --git a/mg-hosts b/mg-hosts
index bd67a0a..5cdece5 100755
--- a/mg-hosts
+++ b/mg-hosts
@@ -178,6 +178,7 @@ sub cmd_create_like () {
         foreach my $dst (l($dsts)) {
             $dbh_tests->do(<<END,{}, $dst,$src);
                 INSERT INTO resources
+                       (restype, resname, shareix, owntaskid)
                 SELECT restype,
                        ? AS resname,
                        0 AS shareix,
diff --git a/tcl/JobDB-Executive.tcl b/tcl/JobDB-Executive.tcl
index 6b9bcb0..414a77e 100644
--- a/tcl/JobDB-Executive.tcl
+++ b/tcl/JobDB-Executive.tcl
@@ -282,7 +282,9 @@ proc spawn-step-begin {flight job ts stepnovar} {
 		set stepno 1
 	    }
 	    db-execute "
-		INSERT INTO steps
+		INSERT INTO steps (flight, job, stepno,
+                                   step, status,
+                                   testid)
 		    VALUES ([pg_quote $flight], [pg_quote $job], $stepno,
 			    [pg_quote $ts], 'running',
 			    'STARTING')
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 07/17] step logfiles: Break out $logfilename in two places

[Ian Jackson]

No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 determine-failure-reasons | 4 ++--
 sg-report-flight          | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/determine-failure-reasons b/determine-failure-reasons
index ecff332..a3b2100 100755
--- a/determine-failure-reasons
+++ b/determine-failure-reasons
@@ -52,8 +52,8 @@ sub gethosts ($) {
 }
 
 while (my $step= $stepsq->fetchrow_hashref) {
-    $step->{Log}= "$c{Logs}/$step->{flight}/$step->{job}/".
-        "$step->{stepno}.$step->{step}.log";
+    my $logfile = "$step->{stepno}.$step->{step}.log";
+    $step->{Log}= "$c{Logs}/$step->{flight}/$step->{job}/".$logfile;
 
     if (!stat $step->{Log}) {
         die "$step->{Log} $!" unless $!==&ENOENT;
diff --git a/sg-report-flight b/sg-report-flight
index fd8a68b..ec30ed0 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -1167,8 +1167,9 @@ END
 		: ('#ffff00',250);
 	}
 	my $ch = "bgcolor=\"$bgc\"";
+	my $logfilename= "$s->{stepno}.$s->{step}.log";
 	my $h = "<a href=\"".encode_entities($s->{job}).
-	    "/$s->{stepno}.".encode_entities($s->{step}).".log\">";
+	    "/".encode_entities($logfilename)."\">";
 	#print H "[$s->{stepno}] ";
 	$h .= encode_entities($sum_core);
 	$h .= "</a>";
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 08/17] step logfiles: Standalone mode: Do not print uninformative $script value

[Ian Jackson]

Right now, this is sometimes wrong; even if it were right, it's not
going to be interesting in the transcript since the actual script name
is obvious from context.

Now this function does not care if $script is undef.

While we're here, fix the prototype (which is mostly commentary here,
as it is not used for method calls).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/JobDB/Standalone.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Osstest/JobDB/Standalone.pm b/Osstest/JobDB/Standalone.pm
index 4842e69..af93196 100644
--- a/Osstest/JobDB/Standalone.pm
+++ b/Osstest/JobDB/Standalone.pm
@@ -88,9 +88,9 @@ sub current_flight ($) {
 
 sub job_ensure_started ($) { }
 
-sub step_start ($$) {
+sub step_start ($$;$) {
     my ($jd,$testid,$script) = @_;
-    logm("========== $flight.$job step $testid running $script ==========");
+    logm("========== $flight.$job step $testid running ==========");
 }
 
 sub step_finish ($$) { #method
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 09/17] substep logfiles: Move defaulting of $script into Executive

[Ian Jackson]

We are going to want to default this here in a more sophisticated way,
but we can't do that if it's already been defaulted from $0 earlier.

The other consumer of this default was in JobDB/Standalone, but we
have just changed that to ignore it.  So, no functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/JobDB/Executive.pm | 4 ++++
 Osstest/TestSupport.pm     | 4 ----
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index 0d1b3be..fbbb5ac 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -252,6 +252,10 @@ END
         INSERT INTO steps (flight,job,stepno, step,status, testid,started)
             VALUES (?,?,?, ?,'running', ?,?)
 END
+    if (!defined $script) {
+	$script = $0;
+	$script =~ s{^.*/}{};
+    }
     my $stepno;
     db_retry($flight,[qw(running)], $dbh_tests,[qw(flights)],sub {
 	$snq->execute($flight,$job);
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index f10d56e..5a951e5 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -248,10 +248,6 @@ sub complete_testid ($) {
 sub substep_start ($;$) {
     my ($testid,$script) = @_;
     complete_testid(\$testid);
-    if (!defined $script) {
-	$script = $0;
-	$script =~ s{^.*/}{};
-    }
     $mjobdb->step_start($testid,$script);
 }
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 10/17] step logfiles: Provide new column

[Ian Jackson]

We will write things into it, and use it, soon.

This is the start of a "Populate-then-rely" schema change, as
described in schema/README.updates.  Specifically, the steps:
1. Commit: new schema update `add', status Preparatory
2. Commit: new schema update `constraint', status Unfinished

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 schema/steps-logfile-add.sql        |  8 ++++++++
 schema/steps-logfile-constraint.sql | 10 ++++++++++
 2 files changed, 18 insertions(+)
 create mode 100644 schema/steps-logfile-add.sql
 create mode 100644 schema/steps-logfile-constraint.sql

diff --git a/schema/steps-logfile-add.sql b/schema/steps-logfile-add.sql
new file mode 100644
index 0000000..5e37b10
--- /dev/null
+++ b/schema/steps-logfile-add.sql
@@ -0,0 +1,8 @@
+-- ##OSSTEST## 003 Preparatory
+--
+-- Previously, the logfile name was inferred from "stepno" and
+-- "testid".  We want to be able to set it explicitly for substeps.
+--
+-- This change is compatible with old code.
+
+ALTER TABLE steps ADD COLUMN logfile TEXT;
diff --git a/schema/steps-logfile-constraint.sql b/schema/steps-logfile-constraint.sql
new file mode 100644
index 0000000..100bb31
--- /dev/null
+++ b/schema/steps-logfile-constraint.sql
@@ -0,0 +1,10 @@
+-- ##OSSTEST## 004 Unfinished
+--
+-- Previously, the logfile name was inferred from "stepno" and
+-- "testid".  Newer code will need it to be set explicitly.
+
+UPDATE steps
+   SET logfile = stepno || '.' || step || '.log'
+ WHERE logfile is null;
+
+ALTER TABLE steps ALTER COLUMN logfile SET NOT NULL;
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 11/17] substep logfiles: Populate new logfile column, at all, for substeps

[Ian Jackson]

For now we populate it with the sometimes-wrong value based on
$script.  In detail:

* Add a parameter $logfile to $mjobdb->step_start, which currently
  always gets passed undef.

* In JobDB::Executive::step_start, try to use it as the value for the
  new logfile column.  But, if it is not set (which is currently
  always the case), default it to "<stepno>.<script>.log".

  This is the same log filename currently assumed (and therefore used)
  by sg-report-flight.  So with current versions of sg-report-flight,
  this has no visible functional change.

This is part of a "Populate-then-rely" schema change, as described in
schema/README.updates.  Specifically, this is part of the step:

5. Commit: code to populate new column; changing `add' to status
   Needed and `constraint' to status Ready.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/JobDB/Executive.pm   | 14 +++++++++-----
 Osstest/JobDB/Standalone.pm  |  4 ++--
 Osstest/TestSupport.pm       |  2 +-
 schema/steps-logfile-add.sql |  2 +-
 4 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index fbbb5ac..e1e2052 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -242,15 +242,17 @@ END
     logm("starting $flight started=$now") if $count>0;
 }
 
-sub step_start ($$) { #method
-    my ($jd,$testid,$script) = @_;
+sub step_start ($$;$$) { #method
+    my ($jd,$testid,$logfile,$script) = @_;
     my $snq = $dbh_tests->prepare(<<END);
         SELECT max(stepno) AS maxstep FROM steps
             WHERE flight=? AND job=?
 END
     my $createq = $dbh_tests->prepare(<<END);
-        INSERT INTO steps (flight,job,stepno, step,status, testid,started)
-            VALUES (?,?,?, ?,'running', ?,?)
+        INSERT INTO steps (flight,job,stepno,step,
+                           logfile,
+                           status,testid,started)
+            VALUES (?,?,?,?, ?, 'running',?,?)
 END
     if (!defined $script) {
 	$script = $0;
@@ -262,7 +264,9 @@ END
 	($stepno) = $snq->fetchrow_array();
 	$stepno //= 0;
 	$stepno++;
-	$createq->execute($flight,$job,$stepno, $script, $testid,time);
+	$createq->execute($flight,$job,$stepno,$script,
+			  $logfile // "$stepno.$script.log",
+			  $testid,time);
     });
     logm("---------- substep $stepno $testid running ----------");
 }
diff --git a/Osstest/JobDB/Standalone.pm b/Osstest/JobDB/Standalone.pm
index af93196..d9a90fc 100644
--- a/Osstest/JobDB/Standalone.pm
+++ b/Osstest/JobDB/Standalone.pm
@@ -88,8 +88,8 @@ sub current_flight ($) {
 
 sub job_ensure_started ($) { }
 
-sub step_start ($$;$) {
-    my ($jd,$testid,$script) = @_;
+sub step_start ($$;$$) {
+    my ($jd,$testid,$logfile,$script) = @_;
     logm("========== $flight.$job step $testid running ==========");
 }
 
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 5a951e5..a05dd97 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -248,7 +248,7 @@ sub complete_testid ($) {
 sub substep_start ($;$) {
     my ($testid,$script) = @_;
     complete_testid(\$testid);
-    $mjobdb->step_start($testid,$script);
+    $mjobdb->step_start($testid,undef,$script);
 }
 
 sub substep_finish ($$) {
diff --git a/schema/steps-logfile-add.sql b/schema/steps-logfile-add.sql
index 5e37b10..abc89b9 100644
--- a/schema/steps-logfile-add.sql
+++ b/schema/steps-logfile-add.sql
@@ -1,4 +1,4 @@
--- ##OSSTEST## 003 Preparatory
+-- ##OSSTEST## 003 Needed
 --
 -- Previously, the logfile name was inferred from "stepno" and
 -- "testid".  We want to be able to set it explicitly for substeps.
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 12/17] substep logfiles: Set logfile and step columns correctly for substeps

[Ian Jackson]

If we are running under sg-run-job, OSSTEST_TESTID will be set to the
testid of the parent step (ie, the step created by sg-run-job which
relates to the execution of the whole script).

Unless the caller has specified otherwise, the log and script are
the same as for the parent step.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/JobDB/Executive.pm | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index e1e2052..a7a6696 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -254,6 +254,19 @@ END
                            status,testid,started)
             VALUES (?,?,?,?, ?, 'running',?,?)
 END
+    my $parent_testid = $ENV{OSSTEST_TESTID};
+    if ((!defined $logfile || !defined $script) && defined $parent_testid) {
+	my $parentq = $dbh_tests->prepare(<<END);
+            SELECT logfile, step FROM steps
+             WHERE flight=? AND job=? AND testid=?
+END
+	db_retry($flight,[qw(running)], $dbh_tests,[],sub {
+            $parentq->execute($flight,$job,$parent_testid);
+            my $row = $parentq->fetchrow_hashref();
+	    $logfile //= $row->{logfile};
+	    $script //= $row->{script};
+	});
+    }
     if (!defined $script) {
 	$script = $0;
 	$script =~ s{^.*/}{};
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 13/17] step logfiles: Honour new logfile column

[Ian Jackson]

There are the following places in the tree where we construct the
logfile name from stepno and step:
 * tcl/JobDB-Executive            main decision about logfile name  } OK
 * JobDB::Executive               final fallback for new steps      } keep
 * steps-logfile-constraint.sql   default for old data              }
 * sg-report-flight (twice)   } both read stepno and step from the db;
 * determine-failure-reasons  }  here we teach them to try logfile first

Now substep logs will have correct links in the job output, as soon as
the logfile column has the right data.

This is part of a "Populate-then-rely" schema change, as described in
schema/README.updates.  Specifically, this is the step:

6. Optionally commit: code which reads new column, but which tolerates
   it containing NULL/DEFAULT.  (`add' is already Needed.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 determine-failure-reasons | 2 +-
 sg-report-flight          | 5 +++--
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/determine-failure-reasons b/determine-failure-reasons
index a3b2100..89c67e6 100755
--- a/determine-failure-reasons
+++ b/determine-failure-reasons
@@ -52,7 +52,7 @@ sub gethosts ($) {
 }
 
 while (my $step= $stepsq->fetchrow_hashref) {
-    my $logfile = "$step->{stepno}.$step->{step}.log";
+    my $logfile = $step->{logfile} // "$step->{stepno}.$step->{step}.log";
     $step->{Log}= "$c{Logs}/$step->{flight}/$step->{job}/".$logfile;
 
     if (!stat $step->{Log}) {
diff --git a/sg-report-flight b/sg-report-flight
index ec30ed0..4e3560e 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -1001,7 +1001,8 @@ END
         printf H "<td>%s</td>", show_abs_time $step->{started};
 
         my $st= $step->{status};
-        my $logfilename= "$step->{stepno}.$step->{step}.log";
+        my $logfilename= $step->{logfile}
+	    // "$step->{stepno}.$step->{step}.log";
         printf H "<td bgcolor='%s'><a href='%s'>%s</a></td>",
 	    (html_status2_colour_priority($st))[0],
             encode_entities(uri_escape($logfilename)),
@@ -1167,7 +1168,7 @@ END
 		: ('#ffff00',250);
 	}
 	my $ch = "bgcolor=\"$bgc\"";
-	my $logfilename= "$s->{stepno}.$s->{step}.log";
+	my $logfilename= $s->{logfile} // "$s->{stepno}.$s->{step}.log";
 	my $h = "<a href=\"".encode_entities($s->{job}).
 	    "/".encode_entities($logfilename)."\">";
 	#print H "[$s->{stepno}] ";
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 14/17] step logfiles: Make substep_* take $logfile, not $script

[Ian Jackson]

The $script parameter was used by Standalone to print a message, but
this is no longer done.

It is used by Executive as the value for the "step" column in the
steps table, which shows up as "script" in the output from
sg-report-flight.  But, when running under sg-run-job,
JobDB::Executive already has a better source of the script name: the
parent step (as reported in OSSTEST_TESTID).  When not running under
sg-run-job the use of $0 is good enough.

Finally, it is used by Executive as a logfile name.  But when running
under sg-run-job, JobDB::Executive gets the right logfile name anyway
(as above); and when not running under sg-run-job there is no default
logfile.

Conversely, substeps might organise their own logfile.

So abolish $script from the TestSupport substep API.  Replace it in
substep_start and substep_eval with a $logfile parameter.

Update all call sites to no longer pass $script.  None make their own
logfiles, so none pass the new parameter $logfile.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm | 10 +++++-----
 ts-examine-serial-post |  2 +-
 ts-xtf-run             |  2 +-
 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index a05dd97..6e19b28 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -246,9 +246,9 @@ sub complete_testid ($) {
 }
 
 sub substep_start ($;$) {
-    my ($testid,$script) = @_;
+    my ($testid,$logfile) = @_;
     complete_testid(\$testid);
-    $mjobdb->step_start($testid,undef,$script);
+    $mjobdb->step_start($testid,$logfile);
 }
 
 sub substep_finish ($$) {
@@ -258,10 +258,10 @@ sub substep_finish ($$) {
 }
 
 sub substep_eval ($$;$) {
-    # substep_eval(TESTID, [SCRIPTNAME], sub { ... });
+    # substep_eval(TESTID, [LOGFILE], sub { ... });
     my $fn = pop @_;
-    my ($testid,$script) = @_;
-    substep_start($testid,$script);
+    my ($testid,$logfile) = @_;
+    substep_start($testid,$logfile);
     eval { $fn->(); };
     if (length $@) {
 	logm("substep ($testid) failed: $@");
diff --git a/ts-examine-serial-post b/ts-examine-serial-post
index 2d5c103..77d8e5d 100755
--- a/ts-examine-serial-post
+++ b/ts-examine-serial-post
@@ -44,7 +44,7 @@ sub cookies () {
 	my $key = $1;
 	my $substep = "examine-serial/$key";
 	my $ci = { K => $key, C => $r{$rv}, Substep => $substep };
-	substep_start($ci->{Substep}, 'ts-examine-serial-post');
+	substep_start($ci->{Substep});
 	push @cookies, $ci;
     }
 }
diff --git a/ts-xtf-run b/ts-xtf-run
index cef45c8..5b136a6 100755
--- a/ts-xtf-run
+++ b/ts-xtf-run
@@ -62,7 +62,7 @@ sub do_one_test ($) {
     my $ret;
     my $osstest_result;
 
-    substep_start($tid, $cmd);
+    substep_start($tid);
 
     if (!eval {
             $output = target_cmd_output_root($ho, <<END, 600);
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 15/17] step logfiles: sg-run-job: Break out step-log-leafname

[Ian Jackson]

No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 tcl/JobDB-Executive.tcl | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tcl/JobDB-Executive.tcl b/tcl/JobDB-Executive.tcl
index 414a77e..ed636e4 100644
--- a/tcl/JobDB-Executive.tcl
+++ b/tcl/JobDB-Executive.tcl
@@ -437,12 +437,16 @@ proc preserve-task {seconds} {
     exec setsid sleep $seconds > /dev/null < /dev/null 2> /dev/null &
 }
 
+proc step-log-leafname {stepno ts} {
+    return $stepno.$ts.log
+}
+
 proc step-log-filename {flight job stepno ts} {
     global c
     set logdir $c(Logs)/$flight/$job
     file mkdir $c(Logs)/$flight
     file mkdir $logdir
-    return $logdir/$stepno.$ts.log
+    return $logdir/[step-log-leafname $stepno $ts]
 }
 
 }; # namespace eval jobdb
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 16/17] step logfiles: Populate logfile column with logfile name

[Ian Jackson]

This is part of a "Populate-then-rely" schema change, as described in
schema/README.updates.  Specifically, this is the final part of the
step:

5. Commit: code to populate new column; changing `add' to status
   Needed and `constraint' to status Ready.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 schema/steps-logfile-constraint.sql | 2 +-
 tcl/JobDB-Executive.tcl             | 5 +++--
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/schema/steps-logfile-constraint.sql b/schema/steps-logfile-constraint.sql
index 100bb31..462941e 100644
--- a/schema/steps-logfile-constraint.sql
+++ b/schema/steps-logfile-constraint.sql
@@ -1,4 +1,4 @@
--- ##OSSTEST## 004 Unfinished
+-- ##OSSTEST## 004 Ready
 --
 -- Previously, the logfile name was inferred from "stepno" and
 -- "testid".  Newer code will need it to be set explicitly.
diff --git a/tcl/JobDB-Executive.tcl b/tcl/JobDB-Executive.tcl
index ed636e4..0734018 100644
--- a/tcl/JobDB-Executive.tcl
+++ b/tcl/JobDB-Executive.tcl
@@ -281,12 +281,13 @@ proc spawn-step-begin {flight job ts stepnovar} {
 	    } else {
 		set stepno 1
 	    }
+	    set logfile [step-log-leafname $stepno $ts]
 	    db-execute "
 		INSERT INTO steps (flight, job, stepno,
-                                   step, status,
+                                   step, logfile, status,
                                    testid)
 		    VALUES ([pg_quote $flight], [pg_quote $job], $stepno,
-			    [pg_quote $ts], 'running',
+			    [pg_quote $ts], [pg_quote $logfile], 'running',
 			    'STARTING')
 	    "
 	} emsg]} {
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 17/17] step logfiles: Rely on new logfile column

[Ian Jackson]

These are the three places which read logfile from the db, but had a
fallback to a computation involving stepno and step.

But, after steps-logfile-constraint, the logfile column is NOT NULL
(and populated with computed default data if need be) so this is not
needed any more

This is part of a "Populate-then-rely" schema change, as
described in schema/README.updates.  Specifically, the step:
10. Optionally commit: code which relies on new column, and does not
   necessarily tolerate NULL/DEFAULT; changing `constraint' to Needed.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 determine-failure-reasons           | 2 +-
 schema/steps-logfile-constraint.sql | 2 +-
 sg-report-flight                    | 5 ++---
 3 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/determine-failure-reasons b/determine-failure-reasons
index 89c67e6..42791c3 100755
--- a/determine-failure-reasons
+++ b/determine-failure-reasons
@@ -52,7 +52,7 @@ sub gethosts ($) {
 }
 
 while (my $step= $stepsq->fetchrow_hashref) {
-    my $logfile = $step->{logfile} // "$step->{stepno}.$step->{step}.log";
+    my $logfile = $step->{logfile};
     $step->{Log}= "$c{Logs}/$step->{flight}/$step->{job}/".$logfile;
 
     if (!stat $step->{Log}) {
diff --git a/schema/steps-logfile-constraint.sql b/schema/steps-logfile-constraint.sql
index 462941e..b261fd0 100644
--- a/schema/steps-logfile-constraint.sql
+++ b/schema/steps-logfile-constraint.sql
@@ -1,4 +1,4 @@
--- ##OSSTEST## 004 Ready
+-- ##OSSTEST## 004 Needed
 --
 -- Previously, the logfile name was inferred from "stepno" and
 -- "testid".  Newer code will need it to be set explicitly.
diff --git a/sg-report-flight b/sg-report-flight
index 4e3560e..135f0d4 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -1001,8 +1001,7 @@ END
         printf H "<td>%s</td>", show_abs_time $step->{started};
 
         my $st= $step->{status};
-        my $logfilename= $step->{logfile}
-	    // "$step->{stepno}.$step->{step}.log";
+        my $logfilename= $step->{logfile};
         printf H "<td bgcolor='%s'><a href='%s'>%s</a></td>",
 	    (html_status2_colour_priority($st))[0],
             encode_entities(uri_escape($logfilename)),
@@ -1168,7 +1167,7 @@ END
 		: ('#ffff00',250);
 	}
 	my $ch = "bgcolor=\"$bgc\"";
-	my $logfilename= $s->{logfile} // "$s->{stepno}.$s->{step}.log";
+	my $logfilename= $s->{logfile};
 	my $h = "<a href=\"".encode_entities($s->{job}).
 	    "/".encode_entities($logfilename)."\">";
 	#print H "[$s->{stepno}] ";
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel