[OSSTEST PATCH 00/11] Coverity fix and mg-repro-setup improvements

[OSSTEST PATCH 00/11] Coverity fix and mg-repro-setup improvements

[Ian Jackson]

This should hopefully actually fix the coverity upload failures.
Also, I tripped over some infelicities in mg-repro-setup.

I'm going to push this to pretest now.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 01/11] TestSupport: Provide stashfilecontents

[Ian Jackson]

Like target_putfilecontents_* but for files on controller.

No callers yet.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 2b4a1f4..d482e1d 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -50,6 +50,7 @@ BEGIN {
                       store_runvar get_runvar get_runvar_maybe
                       get_runvar_default need_runvars
                       unique_incrementing_runvar next_unique_name
+                      stashfilecontents
 
                       target_cmd_root target_cmd target_cmd_build
                       target_cmd_output_root target_cmd_output
@@ -520,6 +521,16 @@ sub target_putfilecontents_root_stash ($$$$;$) {
     tpfcs_core(\&target_putfile_root, @_);
 }
 
+sub stashfilecontents ($$;$) {
+    my ($filedata,$leafbase, $leafref) = @_;
+    # returns full pathname
+    $$leafref = $leafbase;
+    my $fh = open_unique_stashfile($leafref);
+    print $fh $filedata or die "$$leafref: $!";
+    close $fh or die "$$leafref: $!";
+    return "$stash/$$leafref";
+}
+
 sub target_file_exists ($$) {
     my ($ho,$rfile) = @_;
     my $out= target_cmd_output($ho, "if test -e $rfile; then echo y; fi");
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 02/11] ts-coverity-upload: Pass HttpsProxyMITMCert to curl

[Ian Jackson]

"proxy config: Add ability to install MITM TLS cert" sets this up on
the test hosts, but not the controller.

We don't really want this odd configuration globally on the
controller.  Instead, we have to pass it where it is needed.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-coverity-upload | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/ts-coverity-upload b/ts-coverity-upload
index bcdfa91..c52bb87 100755
--- a/ts-coverity-upload
+++ b/ts-coverity-upload
@@ -47,6 +47,11 @@ sub upload() {
     my $proxy = $c{CoverityHttpProxy} // $c{HttpProxy};
     push @args, qw(-x), $proxy if $proxy;
 
+    if ($c{HttpsProxyMITMCert}) {
+        push @args, qw(--cacert),
+            stashfilecontents($c{HttpsProxyMITMCert}, "cacert.pem");
+    }
+
     push @args, qw(--max-time 7200);
     push @args, qw(--fail); # turn 404 etc into a failure.
     push @args, $r{coverity_submit_url};
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 03/11] truncation: Support globs, and multiple patterns

[Ian Jackson]

Rename the variable to truncate_testids.  It contains glob patterns as
for Tcl `string match', space-separated.

Adjust the two places which set it: cs-bisection-step (which needs to
quote any special characters) and mg-repro-setup (which does not
really process the value, but ought to be able to cope with
space-separated lists.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cs-bisection-step |  4 +++-
 mg-repro-setup    |  2 +-
 sg-run-job        | 15 +++++++++------
 3 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/cs-bisection-step b/cs-bisection-step
index 162e068..76f5010 100755
--- a/cs-bisection-step
+++ b/cs-bisection-step
@@ -1312,7 +1312,9 @@ END
             $hostspec =~ m/=/;
             $addvar->execute($popflight, $job, $`, $'); # '
         }
-	$addvar->execute($popflight, $job, 'truncate_testid', $testid);
+        my $testid_glob = $testid;
+        $testid_glob =~ s/[]\\*?[]/\\$&/g;
+	$addvar->execute($popflight, $job, 'truncate_testids', $testid_glob);
     });
 }
 
diff --git a/mg-repro-setup b/mg-repro-setup
index 5a64b1a..bb429b4 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -186,7 +186,7 @@ fi
 
 progress "setting up flight ..."
 
-adjrunvar truncate_testid $testid
+adjrunvar truncate_testids "$testid"
 
 ./cs-adjust-flight $flight						\
 	copy-jobs $example_flight $job					\
diff --git a/sg-run-job b/sg-run-job
index 104c880..cc24b73 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -41,14 +41,14 @@ proc per-host-finish {} {
 
 proc run-job {job} {
     global jobinfo builds flight ok truncate need_xen_hosts anyfailed
-    global nested_layers_hosts truncate_at
+    global nested_layers_hosts truncate_globs
 
     set ok 1
     set truncate 0
     set anyfailed 0
     jobdb::prepare $job
 
-    set truncate_at [jobdb::read-runvar $flight $job truncate_testid]
+    set truncate_globs [jobdb::read-runvar $flight $job truncate_testids]
 
     set nh [need-hosts/$jobinfo(recipe)]
     if {![string compare $nh BUILD]} {
@@ -244,7 +244,7 @@ proc spawn-ts {iffail testid args} {
 }
 
 proc reap-ts {reap} {
-    global truncate truncate_at
+    global truncate truncate_globs
 
     switch -exact [lindex $reap 0] {
         imm { return [lindex $reap 1] }
@@ -266,9 +266,12 @@ proc reap-ts {reap} {
 
     eval jobdb::step-set-status [lrange $details 0 2] $result
     jobdb::logputs stdout "finished $detstr $result $emsg"
-    if {![string compare $testid $truncate_at]} {
-        jobdb::logputs stdout "truncating job now as instructed"
-        set truncate 1
+    foreach truncate_glob [split $truncate_globs] {
+        if {[string match $truncate_glob $testid]} {
+            jobdb::logputs stdout "truncating job now as instructed"
+            set truncate 1
+            break
+        }
     }
     return [expr {![string compare $result pass]}]
 }
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 04/11] sg-run-job: Break out testid_matches_globs

[Ian Jackson]

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-run-job | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/sg-run-job b/sg-run-job
index cc24b73..8f7c36f 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -133,6 +133,15 @@ proc setstatus {st} {
     jobdb::job-set-status $flight $jobinfo(job) $st
 }
 
+proc testid_matches_globs {testid globs} {
+    foreach glob [split $globs] {
+        if {[string match $glob $testid]} {
+            return 1
+        }
+    }
+    return 0
+}
+
 #---------- test script handling ----------
 
 proc run-ts {args} {
@@ -266,12 +275,9 @@ proc reap-ts {reap} {
 
     eval jobdb::step-set-status [lrange $details 0 2] $result
     jobdb::logputs stdout "finished $detstr $result $emsg"
-    foreach truncate_glob [split $truncate_globs] {
-        if {[string match $truncate_glob $testid]} {
-            jobdb::logputs stdout "truncating job now as instructed"
-            set truncate 1
-            break
-        }
+    if {[testid_matches_globs $testid $truncate_globs]} {
+        jobdb::logputs stdout "truncating job now as instructed"
+        set truncate 1
     }
     return [expr {![string compare $result pass]}]
 }
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 05/11] sg-run-job: Start step a bit later

[Ian Jackson]

We are going to want to process the nearly-finished testid.  So
allocate a stepno at the last possible moment.

No overall functional chagne.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-run-job | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/sg-run-job b/sg-run-job
index 8f7c36f..8cf3b94 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -209,8 +209,6 @@ proc spawn-ts {iffail testid args} {
     set host_testid_suffix {}
     foreach arg $testid_args { append host_testid_suffix "/$arg" }
 
-    jobdb::spawn-step-begin $flight $jobinfo(job) $ts stepno
-
     if {[string match =* $testid]} {
         set testid "$deftestid[string range $testid 1 end]"
     } elseif {![string compare $testid *]} {
@@ -218,6 +216,8 @@ proc spawn-ts {iffail testid args} {
         append testid (*)
     }
     regsub {/\@} $testid $host_testid_suffix testid
+
+    jobdb::spawn-step-begin $flight $jobinfo(job) $ts stepno
     regsub {\(\*\)$} $testid ($stepno) testid
 
     set detstr "$flight.$jobinfo(job) $ts $real_args"
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 06/11] sg-run-job: Honour skip_testids runvar

[Ian Jackson]

This works like truncate_testids, except that the steps which match
are never run (truncate_testids skips steps *after* matching steps,
and also doesn't skip some things like log capture).

If the programmed testid for a step ends in `(*)' to request
substitutions of the stepno, the skip matching takes place against the
unsubstituted value (since it happens before a stepno is allocated).
There is no way to skip only some of a set of steps whose testids are
distinguished only by stepno.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-run-job | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/sg-run-job b/sg-run-job
index 8cf3b94..8e625d9 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -41,14 +41,16 @@ proc per-host-finish {} {
 
 proc run-job {job} {
     global jobinfo builds flight ok truncate need_xen_hosts anyfailed
-    global nested_layers_hosts truncate_globs
+    global nested_layers_hosts truncate_globs skip_globs anyskipped
 
     set ok 1
     set truncate 0
+    set anyskipped 0
     set anyfailed 0
     jobdb::prepare $job
 
     set truncate_globs [jobdb::read-runvar $flight $job truncate_testids]
+    set skip_globs     [jobdb::read-runvar $flight $job skip_testids]
 
     set nh [need-hosts/$jobinfo(recipe)]
     if {![string compare $nh BUILD]} {
@@ -97,7 +99,7 @@ proc run-job {job} {
         }
     }
 
-    if {$truncate} {
+    if {$truncate || $anyskipped} {
        if {$ok} { setstatus truncated                                     }
        set ok 0
     }
@@ -174,7 +176,7 @@ proc iffail-check {iffail okexpr iffail_status_var} {
 }
 
 proc spawn-ts {iffail testid args} {
-    global flight c jobinfo env truncate
+    global flight c jobinfo env truncate skip_globs anyskipped
 
     if {[file exists abort]} {
         jobdb::logputs stdout \
@@ -217,6 +219,13 @@ proc spawn-ts {iffail testid args} {
     }
     regsub {/\@} $testid $host_testid_suffix testid
 
+    if {[testid_matches_globs $testid $skip_globs]} {
+        set anyskipped 1
+        jobdb::logputs stdout \
+            "skipping - not executing $flight.$jobinfo(job) $args"
+        return {imm 1} ;# reap-ts will report success
+    }
+
     jobdb::spawn-step-begin $flight $jobinfo(job) $ts stepno
     regsub {\(\*\)$} $testid ($stepno) testid
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 07/11] mg-repro-setup: skip log capture (by default)

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-repro-setup | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/mg-repro-setup b/mg-repro-setup
index bb429b4..af679d0 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -34,6 +34,7 @@ usage () { cat <<END
  OPTIONs
    -t<duration>         estimated duration (default = 28d)
    --rogue              bypass queuing system and allocate now
+   --capture            do not suppress log capture
    -r<var>=<value>      set runvar
    -r!<var>             delete runvar
    -B<blessing>		default is 'play'
@@ -54,6 +55,7 @@ allocs=()
 logfile=tmp/mg-repro-setup.log
 duration=28d
 blessing=play
+skipcapture=true
 
 while true; do
 	case "$1" in
@@ -67,6 +69,7 @@ while true; do
         -f?*)		refflight=${arg#-f}		;;
 	-t?*)		duration=${arg#-t}		;;
 	--rogue)	duration=''			;;
+	--capture)	skipcapture=false		;;
 	-l*)		logfile=${arg#-l}		;;
 	-r!*)		adjustsets+=("${arg#-r}")	;;
 	-r*=*)		adjustsets+=("${arg#-r}")	;;
@@ -188,6 +191,8 @@ progress "setting up flight ..."
 
 adjrunvar truncate_testids "$testid"
 
+if $skipcapture; then adjrunvar skip_testids "capture-logs*"; fi
+
 ./cs-adjust-flight $flight						\
 	copy-jobs $example_flight $job					\
 	runvar-build-set . '/buildjob$' '^[^\.]+$' $example_flight	\
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 08/11] mg-allocate: Document --info-base in head comment

[Ian Jackson]

This documentation was omitted from
 "mg-repro-setup: Provide useful info for allocation system"

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-allocate | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/mg-allocate b/mg-allocate
index 803abb6..87a4e71 100755
--- a/mg-allocate
+++ b/mg-allocate
@@ -55,6 +55,8 @@
 #
 #                  Not compatible with other options.
 #
+#   --info-base=<info>  Replaces `manual' at start of info put into plan.
+#
 # <task-spec> must exist (and be in a format valid for OSSTEST_TASK).
 
 # This is part of "osstest", an automated testing framework for Xen.
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 09/11] mg-allocate: Support --progress-fd

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-allocate | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/mg-allocate b/mg-allocate
index 87a4e71..2dfbdb1 100755
--- a/mg-allocate
+++ b/mg-allocate
@@ -57,6 +57,8 @@
 #
 #   --info-base=<info>  Replaces `manual' at start of info put into plan.
 #
+#   --progress-fd=<fd>  Print some brief progress messages to <fd>.
+#
 # <task-spec> must exist (and be in a format valid for OSSTEST_TASK).
 
 # This is part of "osstest", an automated testing framework for Xen.
@@ -83,6 +85,7 @@ unshift @INC, qw(.);
 use Osstest;
 use Osstest::TestSupport;
 use Osstest::Executive;
+use IO::Handle;
 
 csreadconfig();
 
@@ -450,6 +453,12 @@ sub plan () {
 	logm("best at $planned->{Start} is ".showposs(\@reqlist));
 	die unless $planned;
 
+        printf MGA_PROGRESS "%s %s @%d %s\n",
+            (show_abs_time time),
+            ($mayalloc && !$planned->{Start} ? "allocating" : "planned"),
+            $planned->{Start},
+            showposs(\@reqlist);
+
         my $worstok=0;
         if ($mayalloc && !$planned->{Start}) {
             $worstok=1;
@@ -501,6 +510,8 @@ sub plan () {
     loggot(@got);
 }
 
+open MGA_PROGRESS, ">/dev/null" or die $!;
+
 while (@ARGV && $ARGV[0] =~ m/^[-0-9]/) {
     $_= shift @ARGV;
     last if m/^\-\-?$/;
@@ -528,6 +539,9 @@ while (@ARGV && $ARGV[0] =~ m/^[-0-9]/) {
 	    $allocinfo_base = $1;
 	} elsif (s/^--stdout-output$/-/) {
 	    $stdout_output = 1;
+	} elsif (s/^--progress-fd=(\d+)$/-/) {
+            open MGA_PROGRESS, ">&$1" or die $!;
+            MGA_PROGRESS->autoflush(1);
         } else {
             die "bad option \`$_'";
         }
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 10/11] mg-execute-flight: Support --progress-fd

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-execute-flight | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/mg-execute-flight b/mg-execute-flight
index bd09df3..5a861b0 100755
--- a/mg-execute-flight
+++ b/mg-execute-flight
@@ -35,6 +35,8 @@ publish=false
 
 badusage () { echo >&2 "bad usage"; usage >&2; exit 126; }
 
+exec 5>/dev/null
+
 while [ $# != 0 ]; do
 	arg=$1; shift
         case "$arg" in
@@ -42,6 +44,8 @@ while [ $# != 0 ]; do
         -E?*)	email=${arg#-E}				;;
 	-f?*)	basisflightopt=--that-flight=${arg#-f}	;;
 	-P)	publish=true				;;
+	--progress-fd=*)
+                exec 5>&${arg#--progress-fd=}           ;;
 	[0-9]*)	if [ "x$flight" != x ]; then badusage; fi
 		flight=$arg				;;
 	*)	badusage				;;
@@ -72,6 +76,8 @@ mkdir -p tmp
  echo "sg-execute-flight $flight $blessing failed: $?") \
  >tmp/$flight.transcript 2>&1
 
+echo >&5 "flight $flight reporting"
+
 ./sg-report-flight $basisflightopt \
 	--html-dir=$OSSTEST_HTMLPUB_DIR/$flight/ \
 	$flight >tmp/$flight.report
@@ -97,6 +103,7 @@ cat <tmp/$flight.transcript
 exec >&2
 
 if $publish; then
+	echo >&5 "flight $flight publishing"
 	./cr-publish-flight-logs $flight >/dev/null
 fi
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 11/11] mg-repro-setup: Use --progress-fd

[Ian Jackson]

Also, print a sensible message if allocation fails.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-repro-setup | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/mg-repro-setup b/mg-repro-setup
index af679d0..0b3137a 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -48,6 +48,8 @@ badusage () { echo >&2 "bad usage"; usage >&2; exit 126; }
 
 set -e -o posix
 
+. ./mgi-common
+
 mgexecflags=()
 adjustsets=()
 adjusts=()
@@ -171,9 +173,11 @@ progress "new flight is $flight"
 if [ "${alloc_idents[*]}" ]; then
 	progress "allocating ${alloc_idents[*]} ..."
 	alloc_output=tmp/$flight.allocations
-	./mg-allocate --stdout-output >$alloc_output $duration 2>&3 \
+	./mg-allocate \
+		--stdout-output >$alloc_output 5>&2 --progress-fd=5 2>&3 \
 		--info-base="[repro] $flight.$job $testid" \
-		"${alloc_specs[@]}"
+		$duration "${alloc_specs[@]}" ||
+                fail "allocation failed."
 	exec <$alloc_output
 	progressf "allocated"
 	for ident in ${alloc_idents[*]}; do
@@ -201,6 +205,7 @@ if $skipcapture; then adjrunvar skip_testids "capture-logs*"; fi
 progress "executing ..."
 
 OSSTEST_NOALLOCATE=1 \
-./mg-execute-flight -B$blessing -f$refflight "${mgexecflags[@]}" $flight
+./mg-execute-flight -B$blessing -f$refflight --progress-fd=2 \
+	"${mgexecflags[@]}" $flight
 
 progress "complete (see tmp/$flight.*)"
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel