All of lore.kernel.org
 help / color / mirror / Atom feed
* [OSSTEST PATCH v2] sg-run-job: Coalesce a couple of repetitions
@ 2015-11-30 16:49 Ian Jackson
  2015-11-30 17:01 ` Ian Campbell
  0 siblings, 1 reply; 2+ messages in thread
From: Ian Jackson @ 2015-11-30 16:49 UTC (permalink / raw)
  To: xen-devel; +Cc: Ian Jackson, Ian Campbell

Fold `guest-localmigrate.2' into `guest-localmigrate/x10' and move
`guest-start.2' to after `guest-start.repeat' (reversing the contents
of the latter so that the start comes before the stop).
(guest-start.2 is still necessary because the start/stop test leaves
the guest stopped, whereas the subsequent destroy test ought happen
with the guest running.)

This change will allow the heisenbug compensator to see more of these
failures as the same failures.

The overall effect includes a reduction of the number of localhost
migrations from 11 to 10, but this is better than leaving a misleading
testid containing the string `x10' (or changing the testid).

It is best to fold this way, keeping the testid of the step which
previously had most of the regressions, because: the alternative,
keeping the testid of the low-repetition step, would allow osstest to
use previous lucky passes of the low-repetition step to justify
current failures of the now-high-repetition step.

To check that the effect of the patch is as intended, I ran a before
and after run with OSSTEST_SIMULATE=1, and (a) collected and sedded
and diffed the sg-run-job transcripts and (b) looked in the db.

I also ran a real test (65261 in the Xen Project test lab) with a very
similar version, which passed, and will re-run that before pushing.

(a):

  c&p transcripts from mg-execute-flight email reports
  perl -i~ -pe 's/\b(38371|38370|65261|38395|38397)\b/FLIGHT/; s/^2015-11-\d\d \S+ /TIME /' [tu]
  diff -u [tu] >v
  grep starting v

 =>

 TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-build-check  build-check(1)
 TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-saverestore host debian guest-saverestore.2
-TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-localmigrate host debian guest-localmigrate.2
 TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-localmigrate x10 host debian guest-localmigrate/x10
 TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-stop host debian guest-stop
+TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-repeat-test 10 ts-guest-start host debian {;} ts-guest-stop host debian guest-start/debian.repeat
 TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-start host debian guest-start.2
-TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-repeat-test 10 ts-guest-stop host debian {;} ts-guest-start host debian guest-start/debian.repeat
 TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-destroy host debian guest-destroy
 TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-leak-check check host leak-check/check
-TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-logs-capture host capture-logs(24)
+TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-logs-capture host capture-logs(23)

(b)

osstestdb=> select * from (select job,stepno,step,status,testid from steps where flight=38370) before full outer join (select job,stepno,step,status,testid from steps where flight=38400) after using (testid) order by coalesce(before.stepno, after.stepno);
          testid           |        job         | stepno |             step             | status |        job         | stepno |             step             | status
---------------------------+--------------------+--------+------------------------------+--------+--------------------+--------+------------------------------+--------
 build-check(1)            | test-amd64-i386-xl |      1 | ts-build-check               | pass   | test-amd64-i386-xl |      1 | ts-build-check               | pass
 hosts-allocate            | test-amd64-i386-xl |      2 | ts-hosts-allocate            | pass   | test-amd64-i386-xl |      2 | ts-hosts-allocate            | pass
 host-install(3)           | test-amd64-i386-xl |      3 | ts-host-install-twice        | pass   | test-amd64-i386-xl |      3 | ts-host-install-twice        | pass
 host-ping-check-native    | test-amd64-i386-xl |      4 | ts-host-ping-check           | pass   | test-amd64-i386-xl |      4 | ts-host-ping-check           | pass
 xen-install               | test-amd64-i386-xl |      5 | ts-xen-install               | pass   | test-amd64-i386-xl |      5 | ts-xen-install               | pass
 xen-boot                  | test-amd64-i386-xl |      6 | ts-host-reboot               | pass   | test-amd64-i386-xl |      6 | ts-host-reboot               | pass
 host-ping-check-xen       | test-amd64-i386-xl |      7 | ts-host-ping-check           | pass   | test-amd64-i386-xl |      7 | ts-host-ping-check           | pass
 leak-check/basis(8)       | test-amd64-i386-xl |      8 | ts-leak-check                | pass   | test-amd64-i386-xl |      8 | ts-leak-check                | pass
 debian-install            | test-amd64-i386-xl |      9 | ts-debian-install            | pass   | test-amd64-i386-xl |      9 | ts-debian-install            | pass
 debian-fixup              | test-amd64-i386-xl |     10 | ts-debian-fixup              | pass   | test-amd64-i386-xl |     10 | ts-debian-fixup              | pass
 guest-start               | test-amd64-i386-xl |     11 | ts-guest-start               | pass   | test-amd64-i386-xl |     11 | ts-guest-start               | pass
 migrate-support-check     | test-amd64-i386-xl |     12 | ts-migrate-support-check     | pass   | test-amd64-i386-xl |     12 | ts-migrate-support-check     | pass
 saverestore-support-check | test-amd64-i386-xl |     13 | ts-saverestore-support-check | pass   | test-amd64-i386-xl |     13 | ts-saverestore-support-check | pass
 guest-saverestore         | test-amd64-i386-xl |     14 | ts-guest-saverestore         | pass   | test-amd64-i386-xl |     14 | ts-guest-saverestore         | pass
 guest-localmigrate        | test-amd64-i386-xl |     15 | ts-guest-localmigrate        | pass   | test-amd64-i386-xl |     15 | ts-guest-localmigrate        | pass
 guest-saverestore.2       | test-amd64-i386-xl |     16 | ts-guest-saverestore         | pass   | test-amd64-i386-xl |     16 | ts-guest-saverestore         | pass
 guest-localmigrate.2      | test-amd64-i386-xl |     17 | ts-guest-localmigrate        | pass   |                    |        |                              |
 guest-localmigrate/x10    | test-amd64-i386-xl |     18 | ts-guest-localmigrate        | pass   | test-amd64-i386-xl |     17 | ts-guest-localmigrate        | pass
 guest-stop                | test-amd64-i386-xl |     19 | ts-guest-stop                | pass   | test-amd64-i386-xl |     18 | ts-guest-stop                | pass
 guest-start.2             | test-amd64-i386-xl |     20 | ts-guest-start               | pass   | test-amd64-i386-xl |     20 | ts-guest-start               | pass
 guest-start/debian.repeat | test-amd64-i386-xl |     21 | ts-repeat-test               | pass   | test-amd64-i386-xl |     19 | ts-repeat-test               | pass
 guest-destroy             | test-amd64-i386-xl |     22 | ts-guest-destroy             | pass   | test-amd64-i386-xl |     21 | ts-guest-destroy             | pass
 leak-check/check          | test-amd64-i386-xl |     23 | ts-leak-check                | pass   | test-amd64-i386-xl |     22 | ts-leak-check                | pass
 capture-logs(23)          |                    |        |                              |        | test-amd64-i386-xl |     23 | ts-logs-capture              | pass
 capture-logs(24)          | test-amd64-i386-xl |     24 | ts-logs-capture              | pass   |                    |        |                              |
(25 rows)

osstestdb=>

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: Do not increment count of migration tests so as to make
     testid misleading.
    Do the change to the start/stop test differently.
---
 sg-run-job |   12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/sg-run-job b/sg-run-job
index 4f6b43a..20ebb64 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -374,10 +374,7 @@ proc test-guest-migr {g} {
 	run-ts . =.2 ts-guest-saverestore + host $g
     }
     if {$can_migrate} {
-        run-ts . =.2 ts-guest-localmigrate + host $g
-    }
-    if {$can_migrate} {
-        run-ts . = ts-guest-localmigrate x10 + host $g
+        run-ts . =/x10 ts-guest-localmigrate + x10 host $g
     }
 }
 
@@ -388,11 +385,12 @@ proc test-guest {g} {
 
 proc test-guest-nomigr {g} {
     run-ts . =   ts-guest-stop    + host $g
-    run-ts . =.2 ts-guest-start   + host $g
 
     repeat-ts 10 =.repeat \
-               + ts-guest-stop    host   $g \; \
-               + ts-guest-start + host + $g +
+                 ts-guest-start + host + $g + \; \
+                 ts-guest-stop    host   $g +
+
+    run-ts . =.2 ts-guest-start   + host $g
 
     run-ts . =   ts-guest-destroy + host $g
 }
-- 
1.7.10.4

^ permalink raw reply related	[flat|nested] 2+ messages in thread
end of thread, other threads:[~2015-11-30 17:02 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-11-30 16:49 [OSSTEST PATCH v2] sg-run-job: Coalesce a couple of repetitions Ian Jackson
2015-11-30 17:01 ` Ian Campbell

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.