* [OSSTEST PATCH] sg-run-job: Coalesce a couple of repetitions
@ 2015-11-27 16:54 Ian Jackson
2015-11-27 17:03 ` Ian Campbell
0 siblings, 1 reply; 4+ messages in thread
From: Ian Jackson @ 2015-11-27 16:54 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson, Ian Campbell
Fold `guest-localmigrate.2' into `guest-localmigrate/x10' and
`guest-start.2' into `guest-start.repeat'. In each case increase the
iteration count of the 2nd test by one so that the total number of
reps remains constant.
This will allow the heisenbug compensator to see more of these
failures as the same failures.
The resulting testid `guest-localmigrate/x10' becomes a bit misleading
in that it now does 11 repetitions. But we don't want to change the
testid.
It is best to fold this way, keeping the testid of the step which
previously had most of the regressions, because: the alternative,
keeping the testid of the low-repetition step, would allow osstest to
use previous lucky passes of the low-repetition step to justify
current failures of the now-high-repetition step.
To check that the effect of the patch is as intended, I ran a before
and after run with OSSTEST_SIMULATE=1, and (a) collected and sedded
and diffed the sg-run-job transcripts and (b) looked in the db.
(a):
c&p transcripts from mg-execute-flight email reports
perl -i~ -pe 's/\b(38371|38370)\b/FLIGHT/; s/^2015-11-27 \S+ /TIME /' [tu]
diff -u [tu] >v
grep starting v
=>
TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-build-check build-check(1)
TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-saverestore host debian guest-saverestore.2
-TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-localmigrate host debian guest-localmigrate.2
-TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-localmigrate x10 host debian guest-localmigrate/x10
+TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-localmigrate x11 host debian guest-localmigrate/x10
TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-stop host debian guest-stop
-TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-start host debian guest-start.2
-TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-repeat-test 10 ts-guest-stop host debian {;} ts-guest-start host debian guest-start/debian.repeat
+TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-repeat-test 11 ts-guest-stop host debian {;} ts-guest-start host debian guest-start/debian.repeat
TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-destroy host debian guest-destroy
TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-leak-check check host leak-check/check
-TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-logs-capture host capture-logs(24)
+TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-logs-capture host capture-logs(22)
(b)
osstestdb=> select * from (select job,stepno,step,status,testid from steps where flight=38370) before full outer join (select job,stepno,step,status,testid from steps where flight=38371) after using (testid) order by coalesce(before.stepno, after.stepno);
testid | job | stepno | step | status | job | stepno | step | status
---------------------------+--------------------+--------+------------------------------+--------+--------------------+--------+------------------------------+--------
build-check(1) | test-amd64-i386-xl | 1 | ts-build-check | pass | test-amd64-i386-xl | 1 | ts-build-check | pass
hosts-allocate | test-amd64-i386-xl | 2 | ts-hosts-allocate | pass | test-amd64-i386-xl | 2 | ts-hosts-allocate | pass
host-install(3) | test-amd64-i386-xl | 3 | ts-host-install-twice | pass | test-amd64-i386-xl | 3 | ts-host-install-twice | pass
host-ping-check-native | test-amd64-i386-xl | 4 | ts-host-ping-check | pass | test-amd64-i386-xl | 4 | ts-host-ping-check | pass
xen-install | test-amd64-i386-xl | 5 | ts-xen-install | pass | test-amd64-i386-xl | 5 | ts-xen-install | pass
xen-boot | test-amd64-i386-xl | 6 | ts-host-reboot | pass | test-amd64-i386-xl | 6 | ts-host-reboot | pass
host-ping-check-xen | test-amd64-i386-xl | 7 | ts-host-ping-check | pass | test-amd64-i386-xl | 7 | ts-host-ping-check | pass
leak-check/basis(8) | test-amd64-i386-xl | 8 | ts-leak-check | pass | test-amd64-i386-xl | 8 | ts-leak-check | pass
debian-install | test-amd64-i386-xl | 9 | ts-debian-install | pass | test-amd64-i386-xl | 9 | ts-debian-install | pass
debian-fixup | test-amd64-i386-xl | 10 | ts-debian-fixup | pass | test-amd64-i386-xl | 10 | ts-debian-fixup | pass
guest-start | test-amd64-i386-xl | 11 | ts-guest-start | pass | test-amd64-i386-xl | 11 | ts-guest-start | pass
migrate-support-check | test-amd64-i386-xl | 12 | ts-migrate-support-check | pass | test-amd64-i386-xl | 12 | ts-migrate-support-check | pass
saverestore-support-check | test-amd64-i386-xl | 13 | ts-saverestore-support-check | pass | test-amd64-i386-xl | 13 | ts-saverestore-support-check | pass
guest-saverestore | test-amd64-i386-xl | 14 | ts-guest-saverestore | pass | test-amd64-i386-xl | 14 | ts-guest-saverestore | pass
guest-localmigrate | test-amd64-i386-xl | 15 | ts-guest-localmigrate | pass | test-amd64-i386-xl | 15 | ts-guest-localmigrate | pass
guest-saverestore.2 | test-amd64-i386-xl | 16 | ts-guest-saverestore | pass | test-amd64-i386-xl | 16 | ts-guest-saverestore | pass
guest-localmigrate.2 | test-amd64-i386-xl | 17 | ts-guest-localmigrate | pass | | | |
guest-localmigrate/x10 | test-amd64-i386-xl | 18 | ts-guest-localmigrate | pass | test-amd64-i386-xl | 17 | ts-guest-localmigrate | pass
guest-stop | test-amd64-i386-xl | 19 | ts-guest-stop | pass | test-amd64-i386-xl | 18 | ts-guest-stop | pass
guest-start.2 | test-amd64-i386-xl | 20 | ts-guest-start | pass | | | |
guest-start/debian.repeat | test-amd64-i386-xl | 21 | ts-repeat-test | pass | test-amd64-i386-xl | 19 | ts-repeat-test | pass
capture-logs(22) | | | | | test-amd64-i386-xl | 22 | ts-logs-capture | pass
guest-destroy | test-amd64-i386-xl | 22 | ts-guest-destroy | pass | test-amd64-i386-xl | 20 | ts-guest-destroy | pass
leak-check/check | test-amd64-i386-xl | 23 | ts-leak-check | pass | test-amd64-i386-xl | 21 | ts-leak-check | pass
capture-logs(24) | test-amd64-i386-xl | 24 | ts-logs-capture | pass | | | |
(25 rows)
osstestdb=>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
sg-run-job | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
diff --git a/sg-run-job b/sg-run-job
index 4f6b43a..a74d638 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -374,10 +374,7 @@ proc test-guest-migr {g} {
run-ts . =.2 ts-guest-saverestore + host $g
}
if {$can_migrate} {
- run-ts . =.2 ts-guest-localmigrate + host $g
- }
- if {$can_migrate} {
- run-ts . = ts-guest-localmigrate x10 + host $g
+ run-ts . =/x10 ts-guest-localmigrate + x11 host $g
}
}
@@ -388,9 +385,8 @@ proc test-guest {g} {
proc test-guest-nomigr {g} {
run-ts . = ts-guest-stop + host $g
- run-ts . =.2 ts-guest-start + host $g
- repeat-ts 10 =.repeat \
+ repeat-ts 11 =.repeat \
+ ts-guest-stop host $g \; \
+ ts-guest-start + host + $g +
--
1.7.10.4
^ permalink raw reply related [flat|nested] 4+ messages in thread* Re: [OSSTEST PATCH] sg-run-job: Coalesce a couple of repetitions
2015-11-27 16:54 [OSSTEST PATCH] sg-run-job: Coalesce a couple of repetitions Ian Jackson
@ 2015-11-27 17:03 ` Ian Campbell
2015-11-27 17:16 ` Ian Jackson
0 siblings, 1 reply; 4+ messages in thread
From: Ian Campbell @ 2015-11-27 17:03 UTC (permalink / raw)
To: Ian Jackson, xen-devel
On Fri, 2015-11-27 at 16:54 +0000, Ian Jackson wrote:
> Fold `guest-localmigrate.2' into `guest-localmigrate/x10' and
> `guest-start.2' into `guest-start.repeat'. In each case increase the
> iteration count of the 2nd test by one so that the total number of
> reps remains constant.
>
> This will allow the heisenbug compensator to see more of these
> failures as the same failures.
>
> The resulting testid `guest-localmigrate/x10' becomes a bit misleading
> in that it now does 11 repetitions. But we don't want to change the
> testid.
I'd be inclined to just accept the possibility of missing a failure on the
11th iteration and just do 10 instead of 10+1.
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> ---
> sg-run-job | 8 ++------
> 1 file changed, 2 insertions(+), 6 deletions(-)
>
> diff --git a/sg-run-job b/sg-run-job
> index 4f6b43a..a74d638 100755
> --- a/sg-run-job
> +++ b/sg-run-job
> @@ -374,10 +374,7 @@ proc test-guest-migr {g} {
> run-ts . =.2 ts-guest-saverestore + host $g
> }
> if {$can_migrate} {
> - run-ts . =.2 ts-guest-localmigrate + host $g
> - }
> - if {$can_migrate} {
> - run-ts . = ts-guest-localmigrate x10 + host $g
> + run-ts . =/x10 ts-guest-localmigrate + x11 host $g
> }
> }
>
> @@ -388,9 +385,8 @@ proc test-guest {g} {
>
> proc test-guest-nomigr {g} {
> run-ts . = ts-guest-stop + host $g
> - run-ts . =.2 ts-guest-start + host $g
>
> - repeat-ts 10 =.repeat \
> + repeat-ts 11 =.repeat \
When this repeat starts the guest is now stopped, whereas before it was
running. Does the body of the loop, which starts with a ts-guest-stop,
handle that?
Maybe you should remove the ts-guest-stop right at the start of this fn
too?
> + ts-guest-stop host $g \; \
> + ts-guest-start + host + $g +
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [OSSTEST PATCH] sg-run-job: Coalesce a couple of repetitions
2015-11-27 17:03 ` Ian Campbell
@ 2015-11-27 17:16 ` Ian Jackson
2015-11-27 17:22 ` Ian Campbell
0 siblings, 1 reply; 4+ messages in thread
From: Ian Jackson @ 2015-11-27 17:16 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("Re: [OSSTEST PATCH] sg-run-job: Coalesce a couple of repetitions"):
> On Fri, 2015-11-27 at 16:54 +0000, Ian Jackson wrote:
> > The resulting testid `guest-localmigrate/x10' becomes a bit misleading
> > in that it now does 11 repetitions. But we don't want to change the
> > testid.
>
> I'd be inclined to just accept the possibility of missing a failure on the
> 11th iteration and just do 10 instead of 10+1.
OK.
> When this repeat starts the guest is now stopped, whereas before it was
> running. Does the body of the loop, which starts with a ts-guest-stop,
> handle that?
No, oops. Clearly I should test this for real...
> Maybe you should remove the ts-guest-stop right at the start of this fn
> too?
It's probably better to swap the order of things in the repeat, and
move the extra start to below the repeat.
That way if guest-stop fails utterly (which it does in several tests)
it won't look like a failure of starting.
Ian.
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [OSSTEST PATCH] sg-run-job: Coalesce a couple of repetitions
2015-11-27 17:16 ` Ian Jackson
@ 2015-11-27 17:22 ` Ian Campbell
0 siblings, 0 replies; 4+ messages in thread
From: Ian Campbell @ 2015-11-27 17:22 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Fri, 2015-11-27 at 17:16 +0000, Ian Jackson wrote:
>
> > Maybe you should remove the ts-guest-stop right at the start of this fn
> > too?
>
> It's probably better to swap the order of things in the repeat, and
> move the extra start to below the repeat.
>
> That way if guest-stop fails utterly (which it does in several tests)
> it won't look like a failure of starting.
I agree.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2015-11-27 17:23 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-11-27 16:54 [OSSTEST PATCH] sg-run-job: Coalesce a couple of repetitions Ian Jackson
2015-11-27 17:03 ` Ian Campbell
2015-11-27 17:16 ` Ian Jackson
2015-11-27 17:22 ` Ian Campbell
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.