From: Ian Campbell <ian.campbell@citrix.com>
To: Ian Jackson <ian.jackson@eu.citrix.com>, xen-devel@lists.xenproject.org
Subject: Re: [OSSTEST PATCH v2] sg-run-job: Coalesce a couple of repetitions
Date: Mon, 30 Nov 2015 17:01:59 +0000 [thread overview]
Message-ID: <1448902919.15768.55.camel@citrix.com> (raw)
In-Reply-To: <1448902146-7558-1-git-send-email-ian.jackson@eu.citrix.com>
On Mon, 2015-11-30 at 16:49 +0000, Ian Jackson wrote:
> Fold `guest-localmigrate.2' into `guest-localmigrate/x10' and move
> `guest-start.2' to after `guest-start.repeat' (reversing the contents
> of the latter so that the start comes before the stop).
> (guest-start.2 is still necessary because the start/stop test leaves
> the guest stopped, whereas the subsequent destroy test ought happen
> with the guest running.)
>
> This change will allow the heisenbug compensator to see more of these
> failures as the same failures.
>
> The overall effect includes a reduction of the number of localhost
> migrations from 11 to 10, but this is better than leaving a misleading
> testid containing the string `x10' (or changing the testid).
>
> It is best to fold this way, keeping the testid of the step which
> previously had most of the regressions, because: the alternative,
> keeping the testid of the low-repetition step, would allow osstest to
> use previous lucky passes of the low-repetition step to justify
> current failures of the now-high-repetition step.
>
> To check that the effect of the patch is as intended, I ran a before
> and after run with OSSTEST_SIMULATE=1, and (a) collected and sedded
> and diffed the sg-run-job transcripts and (b) looked in the db.
>
> I also ran a real test (65261 in the Xen Project test lab) with a very
> similar version, which passed, and will re-run that before pushing.
>
> (a):
>
> c&p transcripts from mg-execute-flight email reports
> perl -i~ -pe 's/\b(38371|38370|65261|38395|38397)\b/FLIGHT/; s/^2015-
> 11-\d\d \S+ /TIME /' [tu]
> diff -u [tu] >v
> grep starting v
>
> =>
>
> TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-build-
> check build-check(1)
> TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-
> saverestore host debian guest-saverestore.2
> -TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-
> localmigrate host debian guest-localmigrate.2
> TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-
> localmigrate x10 host debian guest-localmigrate/x10
> TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-
> stop host debian guest-stop
> +TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-
> repeat-test 10 ts-guest-start host debian {;} ts-guest-stop host debian
> guest-start/debian.repeat
> TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-
> start host debian guest-start.2
> -TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-
> repeat-test 10 ts-guest-stop host debian {;} ts-guest-start host debian
> guest-start/debian.repeat
> TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-
> destroy host debian guest-destroy
> TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-leak-
> check check host leak-check/check
> -TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-logs-
> capture host capture-logs(24)
> +TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-logs-
> capture host capture-logs(23)
>
> (b)
>
> osstestdb=> select * from (select job,stepno,step,status,testid from
> steps where flight=38370) before full outer join (select
> job,stepno,step,status,testid from steps where flight=38400) after using
> (testid) order by coalesce(before.stepno, after.stepno);
> testid | job | stepno
> | step | status | job | stepno
> | step | status
> ---------------------------+--------------------+--------+---------------
> ---------------+--------+--------------------+--------+----------------
> --------------+--------
> build-check(1) | test-amd64-i386-xl | 1 | ts-build-
> check | pass | test-amd64-i386-xl | 1 | ts-build-
> check | pass
> hosts-allocate | test-amd64-i386-xl | 2 | ts-hosts-
> allocate | pass | test-amd64-i386-xl | 2 | ts-hosts-
> allocate | pass
> host-install(3) | test-amd64-i386-xl | 3 | ts-host-
> install-twice | pass | test-amd64-i386-xl | 3 | ts-host-
> install-twice | pass
> host-ping-check-native | test-amd64-i386-xl | 4 | ts-host-ping-
> check | pass | test-amd64-i386-xl | 4 | ts-host-ping-
> check | pass
> xen-install | test-amd64-i386-xl | 5 | ts-xen-
> install | pass | test-amd64-i386-xl | 5 | ts-xen-
> install | pass
> xen-boot | test-amd64-i386-xl | 6 | ts-host-
> reboot | pass | test-amd64-i386-xl | 6 | ts-host-
> reboot | pass
> host-ping-check-xen | test-amd64-i386-xl | 7 | ts-host-ping-
> check | pass | test-amd64-i386-xl | 7 | ts-host-ping-
> check | pass
> leak-check/basis(8) | test-amd64-i386-xl | 8 | ts-leak-
> check | pass | test-amd64-i386-xl | 8 | ts-leak-
> check | pass
> debian-install | test-amd64-i386-xl | 9 | ts-debian-
> install | pass | test-amd64-i386-xl | 9 | ts-debian-
> install | pass
> debian-fixup | test-amd64-i386-xl | 10 | ts-debian-
> fixup | pass | test-amd64-i386-xl | 10 | ts-debian-
> fixup | pass
> guest-start | test-amd64-i386-xl | 11 | ts-guest-
> start | pass | test-amd64-i386-xl | 11 | ts-guest-
> start | pass
> migrate-support-check | test-amd64-i386-xl | 12 | ts-migrate-
> support-check | pass | test-amd64-i386-xl | 12 | ts-migrate-
> support-check | pass
> saverestore-support-check | test-amd64-i386-xl | 13 | ts-
> saverestore-support-check | pass | test-amd64-i386-xl | 13 | ts-
> saverestore-support-check | pass
> guest-saverestore | test-amd64-i386-xl | 14 | ts-guest-
> saverestore | pass | test-amd64-i386-xl | 14 | ts-guest-
> saverestore | pass
> guest-localmigrate | test-amd64-i386-xl | 15 | ts-guest-
> localmigrate | pass | test-amd64-i386-xl | 15 | ts-guest-
> localmigrate | pass
> guest-saverestore.2 | test-amd64-i386-xl | 16 | ts-guest-
> saverestore | pass | test-amd64-i386-xl | 16 | ts-guest-
> saverestore | pass
> guest-localmigrate.2 | test-amd64-i386-xl | 17 | ts-guest-
> localmigrate |
> pass | | | |
> guest-localmigrate/x10 | test-amd64-i386-xl | 18 | ts-guest-
> localmigrate | pass | test-amd64-i386-xl | 17 | ts-guest-
> localmigrate | pass
> guest-stop | test-amd64-i386-xl | 19 | ts-guest-
> stop | pass | test-amd64-i386-xl | 18 | ts-guest-
> stop | pass
> guest-start.2 | test-amd64-i386-xl | 20 | ts-guest-
> start | pass | test-amd64-i386-xl | 20 | ts-guest-
> start | pass
> guest-start/debian.repeat | test-amd64-i386-xl | 21 | ts-repeat-
> test | pass | test-amd64-i386-xl | 19 | ts-repeat-
> test | pass
> guest-destroy | test-amd64-i386-xl | 22 | ts-guest-
> destroy | pass | test-amd64-i386-xl | 21 | ts-guest-
> destroy | pass
> leak-check/check | test-amd64-i386-xl | 23 | ts-leak-
> check | pass | test-amd64-i386-xl | 22 | ts-leak-
> check | pass
> capture-
> logs(23) | | |
> | | test-amd64-i386-xl | 23 | ts-logs-
> capture | pass
> capture-logs(24) | test-amd64-i386-xl | 24 | ts-logs-
> capture |
> pass | | | |
> (25 rows)
>
> osstestdb=>
>
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
prev parent reply other threads:[~2015-11-30 17:02 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-11-30 16:49 [OSSTEST PATCH v2] sg-run-job: Coalesce a couple of repetitions Ian Jackson
2015-11-30 17:01 ` Ian Campbell [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1448902919.15768.55.camel@citrix.com \
--to=ian.campbell@citrix.com \
--cc=ian.jackson@eu.citrix.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.