[OSSTEST PATCH 1/4] Rename README.bisection to NOTES.bisection

[OSSTEST PATCH 1/4] Rename README.bisection to NOTES.bisection

[Ian Jackson]

This contains osstest-developer-oriented information about the
bisector, and is not really pitched at osstest consumers.  (It's also
rather more of a sketch than a complete algorithm doc, and somewhat
out of date.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 README.bisection => NOTES.bisection |    0
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename README.bisection => NOTES.bisection (100%)

diff --git a/README.bisection b/NOTES.bisection
similarity index 100%
rename from README.bisection
rename to NOTES.bisection
-- 
1.7.10.4

[OSSTEST PATCH 2/4] cri-bisect: Change bisection report intro slightly

[Ian Jackson]

Remove space from `xen branch'; this is `xenbranch' in most other
places.

Use `testid' rather than `test' for the failing step testid.  This is
what it is called elsewhere.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cri-bisect |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cri-bisect b/cri-bisect
index c1d0d92..6a2f616 100644
--- a/cri-bisect
+++ b/cri-bisect
@@ -91,9 +91,9 @@ send_bisection_email () {
 Subject: $subject_prefix $1 $job
 
 branch $xenbranch
-xen branch $xenbranch
+xenbranch $xenbranch
 job $job
-test $testid
+testid $testid
 
 END
         cat >&3 $summaryfile
-- 
1.7.10.4

[OSSTEST PATCH 3/4] cs-bisection-step: Print url for last failure repro flight

[Ian Jackson]

Print the url of the logs for last failure repro flight.  This saves
looking up the graph and manually constructing the url.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cs-bisection-step |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/cs-bisection-step b/cs-bisection-step
index ebecda4..7a97b10 100755
--- a/cs-bisection-step
+++ b/cs-bisection-step
@@ -857,11 +857,14 @@ sub search () {
             next if $nope;
             next unless @passparents;
 
+	    my $fail_last_repro;
             return 1 if need_repro_sequence(3, sub {
                 foreach my $pass (@passparents) {
                     return 1 if need_repro('pass', $pass, "last pass");
                 }
                 return 1 if need_repro('fail', $fail, "first failure");
+		$fail_last_repro = $repro_lastflight;
+		return 0;
             });
 
             $found_exact_failure++;
@@ -885,6 +888,7 @@ END
 Bug is in tree:  $treeinfos[$i]{Name} $treeinfos[$i]{Url}
 Bug introduced:  $failrev[$i]
 Bug not present: $passrev[$i]
+Last fail repro: $c{ReportHtmlPubBaseUrl}/$fail_last_repro/
 END
                 }
             }
-- 
1.7.10.4

[OSSTEST PATCH 4/4] README.bisection: New consumer-oriented document

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
 README.bisection |  214 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 214 insertions(+)
 create mode 100644 README.bisection

diff --git a/README.bisection b/README.bisection
new file mode 100644
index 0000000..cc373cf
--- /dev/null
+++ b/README.bisection
@@ -0,0 +1,214 @@
+BISECTION
+=========
+
+osstest has an automatic bisector which reports primarily by email.
+
+Here we first explain the meaning of a `bisection complete' report,
+and later discuss the bisection algorithm, intermediate reports, etc.
+
+
+Understanding a bisected failure
+================================
+
+The starting point is to:
+
+ - Look at the fingered changeset, as shown in the email.
+
+ - Look at the logs for the most recent failure to understand
+   the problem.   The test grid for final failure repro is in
+   the email.
+
+ - Maybe, look at the revision graph.
+
+
+Email Reports
+=============
+
+Subject line
+------------
+
+  Subject: [xen-unstable bisection] complete test-amd64-i386-xl-qemuu-...
+            ^^^^^^^^^^^^            ^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^...
+       	   "branch"                     |    job which contains
+                                        |      failing test step
+                                        |
+           bisection status, `complete' if it fingers a particular commit
+
+
+Introduction
+------------
+
+  branch xen-unstable
+  xenbranch xen-unstable
+  job test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm
+  testid debian-hvm-install
+
+"branch" is the osstest test series; normally this identifies the push
+gate whose failure triggered the bisection start.  (See README for
+information about terminology like "branch".)
+
+"testid" identifies the individual test step.  Bisection is always of
+a particular step, and distinguishes failures of other steps in a test
+job.
+
+"xenbranch" refers to the version of Xen being used.
+
+
+Trees
+-----
+
+Each test run by osstest draws on source code for a number of projects
+and components.  osstest records for each test, which version of each
+tree is involved.  When doing bisection, osstest considers all of the
+sources used, and tries to narrow the failure down to a single commit
+in a single tree.
+
+The bisection report email lists the relevant trees:
+
+  Tree: linux git://xenbits.xen.org/linux-pvops.git
+  Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
+  Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
+  Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-unstable.git
+  Tree: xen git://xenbits.xen.org/xen.git
+        ^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+         |       url used for fetching the bits
+	 |
+         ` Identifier indicating the purpose for which the tree
+	   is used.
+
+
+Result
+------
+
+  *** Found and reproduced problem changeset ***
+
+    Bug is in tree:  xen git://xenbits.xen.org/xen.git
+    Bug introduced:  c64ef24febcf5bc6c6da733abd256412da8c4895
+    Bug not present: 2fe4c0060c62a012109dab1d981522dc26822110
+
+This should be fairly self-explanatory.  osstest is blaming the
+failure on the commit shown.  osstest has verified that the "not
+present" version works and that the "bug introduced" version breaks
+(with identical versions of everything else).
+
+    Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/63036/
+
+This is the URL of the final failure reproduction test.  If you wanted
+to see what the failure looks like, in the fingered commit, this is a
+good place to start.  You'll probably want to click on the column
+heading for the failing test job, to get to the page for just that
+job.
+
+osstest then reproduces the commit message of the fingered commit.
+
+
+Epilogue
+--------
+
+  For bisection revision-tuple graph see:
+     http://logs.test-lab.xenproject.org/osstest/results/bisect/xen-unstable/test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm.debian-hvm-install.html
+  Revision IDs in each graph node refer, respectively, to the Trees above.
+
+This links to a graph showing all the revisions and test results.
+
+
+
+Debugging output
+----------------
+
+  Running cs-bisection-step --graph-out=/home/logs/results/bisect/xen-unstable/test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm.debian-hvm-install --summary-out=tmp/63036.bisection-summary --basis-template=62711 --blessings=real,real-bisect xen-unstable test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm debian-hvm-install
+  Searching for failure / basis pass:
+
+Most of the remainder of the email is primarily useful for osstest
+developers.
+
+Typically there is also a report of the last test flight run by the
+osstest bisector.  However, this last test flight might be of a pass
+or of a fail.
+
+It is better to look at the results grid of the "Last fail repro"
+flight, mentioned earlier.
+
+
+
+Revision Graph
+==============
+
+The revision graph page shows the osstest bisector's view of the
+situation.
+
+Each node in the graph corresponds to a tuple of revisions: one
+revision for each of the relevant trees.  Each edge changes the
+revision of one of the trees.  (osstest has constructed this graph by
+merging the separate revision graphs of each of the individual trees.)
+
+Each box lists the revisions of the trees, and the osstest flight
+numbers which tested that revision (on the specific host).  The logs
+for a particular flight can be found by adding the flight number onto
+the end of the log base URL.  Eg, to find the logs for flight 63036,
+  http://logs.test-lab.xenproject.org/osstest/logs/63036/
+
+Nodes coloured red or green are fail and pass respectively.  Nodes can
+also be coloured yellow (if the test results are inconsistent at that
+revision) or blue (if the attempt with that particular revision tuple
+was inconclusive because the test step being bisected was blocked by
+an earlier failure).
+
+The revision being fingered (or, when the bisector is running, the
+next revision to test) is highlighed with a double boundary.
+
+The starting points for the bisection are shown hexagonal (failure)
+and oval (pass).
+
+
+Semantics
+=========
+
+osstest does not report a bisection failure unless it has reproduced
+the pass (at the good commit which is the bad commit's parent), and
+the failure (at the bad commit) three times each.
+
+osstest's bisector insists that all the bisection tests are run on the
+same physical host.
+
+So if osstest blames a particular commit, then it is very likely
+indeed that this commit does cause the failure.  (Of course, it may be
+that the blamed commit merely exposes a bug elsewhere.)
+
+
+Merges
+------
+
+Sometimes osstest may blame a merge commit.  The actual broken commit
+might be in an arm of the history which osstest didn't look at,
+because that arm of the history is not descended from osstest's
+baseline pass.
+
+If this happens the problem is likely to be in the changes being
+merged in, rather than the merge itself.
+
+We may improve osstest's handling of merges in the future.
+
+
+Flattened history
+-----------------
+
+In sufficiently pathological situations (for example, if one of the
+trees involved is rewound) osstest may construct a dummy revision
+graph for a particular tree, containing only two commits: the baseline
+pass, and the failure which prompted the bisection.
+
+It will be obvious if osstest fingers the tip of this dummy revision
+graph, since the good commit mentioned in the results mail will not be
+an immediate parent of the alleged bad commit, when viewed in the
+actual revision system.
+
+
+Progress Reports
+================
+
+When the bisector is running, it sends similar-looking emails to the
+osstest-output mailing list.
+
+These contain the roughly same information; instead of "complete" they
+may say "running", "inconclusive", "unreproducible", etc.
-- 
1.7.10.4

Re: [OSSTEST PATCH 4/4] README.bisection: New consumer-oriented document

[Wei Liu]

On Mon, Oct 19, 2015 at 05:15:35PM +0100, Ian Jackson wrote:
[...]
> +
> +The bisection report email lists the relevant trees:
> +
> +  Tree: linux git://xenbits.xen.org/linux-pvops.git
> +  Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
> +  Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
> +  Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-unstable.git
> +  Tree: xen git://xenbits.xen.org/xen.git
> +        ^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +         |       url used for fetching the bits
> +	 |
> +         ` Identifier indicating the purpose for which the tree
> +	   is used.

A tab was inserted here.

> +
> +
> +Result
> +------
[...]
> +Progress Reports
> +================
> +
> +When the bisector is running, it sends similar-looking emails to the
> +osstest-output mailing list.
> +
> +These contain the roughly same information; instead of "complete" they
> +may say "running", "inconclusive", "unreproducible", etc.

On xen-devel there are only "complete" emails. It might be helpful to
send "unreproducible" and / or "inconclusive" emails as well? What I'm
thinking is that it would be helpful to clearly indicate human
intervention is needed.

Wei.

> -- 
> 1.7.10.4

Re: [OSSTEST PATCH 4/4] README.bisection: New consumer-oriented document

[Ian Jackson]

Wei Liu writes ("Re: [OSSTEST PATCH 4/4] README.bisection: New consumer-oriented document"):
> On Mon, Oct 19, 2015 at 05:15:35PM +0100, Ian Jackson wrote:
> > +         |       url used for fetching the bits
> > +	 |
> > +         ` Identifier indicating the purpose for which the tree
> > +	   is used.
> 
> A tab was inserted here.

Thanks, will fix.

> > +Progress Reports
...
> > +These contain the roughly same information; instead of "complete" they
> > +may say "running", "inconclusive", "unreproducible", etc.
> 
> On xen-devel there are only "complete" emails. It might be helpful to
> send "unreproducible" and / or "inconclusive" emails as well? What I'm
> thinking is that it would be helpful to clearly indicate human
> intervention is needed.

There's quite a lot of these.  In principle, around one for each
"REGR" ever mentioned in a main push gate flight report.

I don't think anyone wants to read them routinely.  I certainly don't.
If one is interested in a particular failure, it's easy enough to go
into the list and find the right one.

Ian.

Re: [OSSTEST PATCH 1/4] Rename README.bisection to NOTES.bisection

[Ian Campbell]

On Mon, 2015-10-19 at 17:15 +0100, Ian Jackson wrote:
> This contains osstest-developer-oriented information about the
> bisector, and is not really pitched at osstest consumers.  (It's also
> rather more of a sketch than a complete algorithm doc, and somewhat
> out of date.)
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

All 4 patches here: Acked-by: Ian Campbell <ian.campbell@citrix.com>

Ian.

Re: [OSSTEST PATCH 4/4] README.bisection: New consumer-oriented document

[Wei Liu]

On Mon, Oct 19, 2015 at 05:51:25PM +0100, Ian Jackson wrote:
> Wei Liu writes ("Re: [OSSTEST PATCH 4/4] README.bisection: New consumer-oriented document"):
> > On Mon, Oct 19, 2015 at 05:15:35PM +0100, Ian Jackson wrote:
> > > +         |       url used for fetching the bits
> > > +	 |
> > > +         ` Identifier indicating the purpose for which the tree
> > > +	   is used.
> > 
> > A tab was inserted here.
> 
> Thanks, will fix.
> 
> > > +Progress Reports
> ...
> > > +These contain the roughly same information; instead of "complete" they
> > > +may say "running", "inconclusive", "unreproducible", etc.
> > 
> > On xen-devel there are only "complete" emails. It might be helpful to
> > send "unreproducible" and / or "inconclusive" emails as well? What I'm
> > thinking is that it would be helpful to clearly indicate human
> > intervention is needed.
> 
> There's quite a lot of these.  In principle, around one for each
> "REGR" ever mentioned in a main push gate flight report.
> 
> I don't think anyone wants to read them routinely.  I certainly don't.
> If one is interested in a particular failure, it's easy enough to go
> into the list and find the right one.
> 

OK then.

> Ian.