Re: [OSSTEST PATCH] README.email: Add `Worked example of relevant regression in previous flight'

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-11-27 at 15:38 +0000, Ian Jackson wrote:
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> CC: Jan Beulich <JBeulich@suse.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  README.email |   51 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 51 insertions(+)
> 
> diff --git a/README.email b/README.email
> index 5de63dd..e14a816 100644
> --- a/README.email
> +++ b/README.email
> @@ -89,6 +89,9 @@ history.  Here are some examples:
>        justifiable because they prevent other tests from running and
>        can so conceal bugs.)
>  
> +      See `Worked example of relevant regression in previous flight',
> +      below.
> +
>     fail in 58948 pass in 58965
>     fail in 58948 like 37628
>  
> @@ -159,3 +162,51 @@ X-Osstest-Versions-That:
>       tree     revision
>  
>  `This' is the version being tested, and `That' is the baseline.
> +
> +
> +
> +Worked example of relevant regression in previous flight
> +--------------------------------------------------------
> +
> +Suppose two test steps A and B, which normally run in that order:
> +  job test-foo
> +       A   ./ts-do-some-thing
> +       B   ./ts-do-another-thing
> +
> +Suppose failure of A prevents the execution of B.  (This is the usual
> +case where step A precedes step B; normally later steps in a job
> +depend on the success of earlier steps, because after an earlier
> +failure the testbed state is not necessarily well-defined.)
> +
> +Now suppose A has an intermittent bug, but B is totally broken.
> +
> +With our current policy on intermittent bugs[1], we would allow a push
> +despite the bug in A.  But we should not allow a push despite B: the
> +100% reproducible failure of B should prevent all pushes.
> +
> +But the bug in B only shows up when A happens to pass.  So the
> +heisenbug compensator has to insist on seeing an actual pass of B
> +(which in this hypothetical situation, will not occur).
> +
> +Eg, consider these flights:
> +
> +  100  is now master  A pass, B pass      pushed
> +  200  staging        A pass, B fail      `B REGR. vs 100'
> +  201  staging        A fail, B not run   `B fail in 200 REGR. vs 100'
> +
> +In flight 201, the failure of A is indeed justifiable as a heisenbug
> +because it can be seen to succeed in flight 200.  It is the problem
> +with B which is actually blocking the push - but that failure is only
> +visible in flight 200.
> +
> +If, contrary to the suppositions above, the failure of B is actually a
> +heisenbug, then hopefully eventually both A and then B will happen to
> +pass in the same run.  Even if that particular flight has other
> +problems, a future evaluation of a test of the same version can use
> +that flight's passes of A and B to justify, respectively, whatever
> +failures of A and/or B that it comes across.
> +
> +[1] In principle we could have a different policy: to try to reject
> +intermittent bugs.  But it would require a lot of test resources
> +because all tests would have to be repeated a lot, and naturally
> +intermittent bugs would slip through anyway.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel