[xen-4.3-testing bisection] complete build-i386

[xen-4.3-testing bisection] complete build-i386

[xen.org]

branch xen-4.3-testing
xen branch xen-4.3-testing
job build-i386
test xen-build

Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-4.3-testing.git
Tree: xen git://xenbits.xen.org/xen.git

*** Found and reproduced problem changeset ***

  Bug is in tree:  xen git://xenbits.xen.org/xen.git
  Bug introduced:  3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
  Bug not present: c8d233c644cbaba24b823ddd2394e4b4e07f7d9d


  commit 3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
  Author: Ian Jackson <ian.jackson@eu.citrix.com>
  Date:   Mon Jul 1 16:51:43 2013 +0100
  
      Config.mk: Update QEMU_TAG and QEMU_UPSTREAM_REVISION for 4.3
      
      Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>


For bisection revision-tuple graph see:
   http://www.chiark.greenend.org.uk/~xensrcts/results/bisect.xen-4.3-testing.build-i386.xen-build.html
Revision IDs in each graph node refer, respectively, to the Trees above.

----------------------------------------
Searching for failure / basis pass:
 18321 fail [host=lace-bug] / 18261 [host=potato-beetle] 18368 ok.
Failure / basis pass flights: 18321 / 18368
(tree in basispass but not in latest: qemu)
Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-4.3-testing.git
Tree: xen git://xenbits.xen.org/xen.git
Latest 7483e7f15139603380c45ebcd8cc2a57dda5583c f8cc9c2b713b1739b1d3d324716547fa639dce86
Basis pass 7483e7f15139603380c45ebcd8cc2a57dda5583c c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
Generating revisions with ./adhoc-revtuple-generator  git://xenbits.xen.org/staging/qemu-upstream-4.3-testing.git#7483e7f15139603380c45ebcd8cc2a57dda5583c-7483e7f15139603380c45ebcd8cc2a57dda5583c git://xenbits.xen.org/xen.git#c8d233c644cbaba24b823ddd2394e4b4e07f7d9d-f8cc9c2b713b1739b1d3d324716547fa639dce86
using cache /export/home/osstest/repos/git-cache...
using cache /export/home/osstest/repos/git-cache...
locked cache /export/home/osstest/repos/git-cache...
processing ./cacheing-git clone --bare git://xenbits.xen.org/xen.git /export/home/osstest/repos/xen...
Initialized empty Git repository in /export/home/osstest/repos/xen/
Initialized empty Git repository in /export/home/osstest/repos/xen/
updating cache /export/home/osstest/repos/git-cache xen...
using cache /export/home/osstest/repos/git-cache...
using cache /export/home/osstest/repos/git-cache...
locked cache /export/home/osstest/repos/git-cache...
processing ./cacheing-git clone --bare git://xenbits.xen.org/xen.git /export/home/osstest/repos/xen...
Initialized empty Git repository in /export/home/osstest/repos/xen/
updating cache /export/home/osstest/repos/git-cache xen...
Loaded 1001 nodes in revision graph
Searching for test results:
 18261 [host=potato-beetle]
 18321 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c f8cc9c2b713b1739b1d3d324716547fa639dce86
 18346 pass irrelevant
 18348 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c f8cc9c2b713b1739b1d3d324716547fa639dce86
 18349 pass irrelevant
 18350 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c f8cc9c2b713b1739b1d3d324716547fa639dce86
 18351 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c 3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
 18352 pass irrelevant
 18353 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c f8cc9c2b713b1739b1d3d324716547fa639dce86
 18354 pass irrelevant
 18357 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c f8cc9c2b713b1739b1d3d324716547fa639dce86
 18358 pass 7483e7f15139603380c45ebcd8cc2a57dda5583c c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
 18359 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c f8cc9c2b713b1739b1d3d324716547fa639dce86
 18360 pass 7483e7f15139603380c45ebcd8cc2a57dda5583c c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
 18361 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c f8cc9c2b713b1739b1d3d324716547fa639dce86
 18363 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c 3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
 18365 pass 7483e7f15139603380c45ebcd8cc2a57dda5583c c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
 18366 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c 3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
 18368 pass 7483e7f15139603380c45ebcd8cc2a57dda5583c c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
 18370 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c 3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
Searching for interesting versions
 Result found: flight 18358 (pass), for basis pass
 Result found: flight 18359 (fail), for basis failure
 Repro found: flight 18360 (pass), for basis pass
 Repro found: flight 18361 (fail), for basis failure
 0 revisions at 7483e7f15139603380c45ebcd8cc2a57dda5583c c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
No revisions left to test, checking graph state.
 Result found: flight 18358 (pass), for last pass
 Result found: flight 18363 (fail), for first failure
 Repro found: flight 18365 (pass), for last pass
 Repro found: flight 18366 (fail), for first failure
 Repro found: flight 18368 (pass), for last pass
 Repro found: flight 18370 (fail), for first failure

*** Found and reproduced problem changeset ***

  Bug is in tree:  xen git://xenbits.xen.org/xen.git
  Bug introduced:  3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
  Bug not present: c8d233c644cbaba24b823ddd2394e4b4e07f7d9d

using cache /export/home/osstest/repos/git-cache...
using cache /export/home/osstest/repos/git-cache...
locked cache /export/home/osstest/repos/git-cache...
processing ./cacheing-git clone --bare git://xenbits.xen.org/xen.git /export/home/osstest/repos/xen...
Initialized empty Git repository in /export/home/osstest/repos/xen/
updating cache /export/home/osstest/repos/git-cache xen...

  commit 3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
  Author: Ian Jackson <ian.jackson@eu.citrix.com>
  Date:   Mon Jul 1 16:51:43 2013 +0100
  
      Config.mk: Update QEMU_TAG and QEMU_UPSTREAM_REVISION for 4.3
      
      Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Revision graph left in /home/xc_osstest/results/bisect.xen-4.3-testing.build-i386.xen-build.{dot,ps,png,html}.
----------------------------------------
18370: tolerable ALL FAIL

flight 18370 xen-4.3-testing real-bisect [real]
http://www.chiark.greenend.org.uk/~xensrcts/logs/18370/

Failures :-/ but no regressions.

Tests which did not succeed,
including tests which could not be run:
 build-i386                    4 xen-build               fail baseline untested


jobs:
 build-i386                                                   fail    


------------------------------------------------------------
sg-report-flight on woking.cam.xci-test.com
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images

Logs, config files, etc. are available at
    http://www.chiark.greenend.org.uk/~xensrcts/logs

Test harness code can be found at
    http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary

Re: [xen-4.3-testing bisection] complete build-i386

[Ian Jackson]

xen.org writes ("[xen-4.3-testing bisection] complete build-i386"):
> branch xen-4.3-testing
> xen branch xen-4.3-testing
> job build-i386
> test xen-build
> 
> Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-4.3-testing.git
> Tree: xen git://xenbits.xen.org/xen.git
> 
> *** Found and reproduced problem changeset ***
> 
>   Bug is in tree:  xen git://xenbits.xen.org/xen.git
>   Bug introduced:  3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
>   Bug not present: c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
... 
>       Config.mk: Update QEMU_TAG and QEMU_UPSTREAM_REVISION for 4.3

This is because
 (a) qemu-xen-4.3-testing.git (the trad tree) had not had
     git-update-server-info run in it since the xen-4.3.0 tag was
     made.
 (b) Config.mk specifies an http url (rather than a git url) for
     the qemu trees.

I have fixed this and prodded the tester into starting another 4.3
test right away.


Post mortem:

The xen-4.3.0 tag in that tree was made some time last week.  Running
g-u-s-i is in the checklist.  I don't know why the checklist wasn't
followed properly (by me).  That was clearly a mistake.  Sorry.

I think, though, that our release process has become quite complex and
single the ad-hoc checklist file we (mostly I) are using is creaking
under the strain rather.

ACTION: I think we need to write down our actual relase process from a
higher-level point of view, and have a separate file for technical
processes and runes at various stages.

The autotester didn't detect this situation before the release for the
following reasons:

1. There was an intermittent infrastructure problem which broke the
   autotester entirely for much of the last few weeks, for days at a
   time.  The cause of this infrastructure problem (an NFS lockup) has
   not been identified.  We hope to collect more data if it recurs.

   ACTION: We intend to migrate the many services (not just osstest)
   off the server which broke, into their own VMs.  But this is
   dependent on effort and hardware.

2. There was a problem with the way the autotester was trying to test
   4.3 testing which was actually an overlooked hg-specific rune in a
   slightly ad-hoc piece of machinery (which grabs the qemu trad
   version out of a particular Xen's Config.mk).  For reasons which
   still aren't clear to me, this problem only manifested in the 4.3
   testing branch, which was created only recently.  (This bug is now
   fixed.)

3. Because of problem (1) the cron mails reporting (2) didn't arrive
   for several days.  So I didn't become aware of it.

4. I was ill in bed on Monday.

For all of these reasons, there were no autotester test results of 4.3
until today.  But we knew on Tuesday that the tests were missing.

I'm not sure why we didn't detect this problem in any of the ad-hoc
test builds we did after changing Config.mk.

I'm not sure all the test builds we did, but I know about the ones I
did as part of the tarball process.  They wouldn't detect this
situation because the tarball is made to include copies of the checked
out qemu code; thus a test build of the resulting tarball doesn't need
to check out qemu.  (And building the tarball itself is done from
runes in the checklist which do not involve cloning a fresh qemu tree,
so you wouldn't see it then.)

If someone did an ad-hoc test build of changing Config.mk, it is
possible that this is because one of

(a) They didn't clean their tree _entirely thoroughly_.
    "make clean" is not sufficient because it doesn't remove cloned
    subdirectories.  "git-clean -x -d -f" is not sufficient because
    it recurses into the cloned subdirectories, discovers that these
    are separate git repositories, and stops.

    "make distclean" plus "git-clean -x -d -f" should be sufficient.
    In my test just now it is for qemu, however there is a bug in
    the build system which means that it leaves
    tools/firmware/seabios-dir-remote/

    This is not a particularly nice situation but arguably it is
    something that someone doing a final test build would have to
    know.  ACTION: We should perhaps write it down.  Where ?

(b) Their arrangements for building things involve using a local
    .config (or similar) which specify explicitly which qemu to use.

    Doing this for a final release check build after editing Config.mk
    would arguably be simply a mistake.

(c) Their arrangements for building things involve using a local
    .config or environment setting to tell the Xen build system to use
    the git protocol rather than http.

    It is not clear that we could regard this as unequivocally a
    mistake.


Ian.

Re: [xen-4.3-testing bisection] complete build-i386

[George Dunlap]

On 11/07/13 16:07, Ian Jackson wrote:
> If someone did an ad-hoc test build of changing Config.mk, it is
> possible that this is because one of
>
> (a) They didn't clean their tree _entirely thoroughly_.
>      "make clean" is not sufficient because it doesn't remove cloned
>      subdirectories.  "git-clean -x -d -f" is not sufficient because
>      it recurses into the cloned subdirectories, discovers that these
>      are separate git repositories, and stops.

If you specify -f twice it will override this behavior and delete cloned 
subdirectories.   So "git clean -ffdx" should really really really clean it.

  -George

Re: [xen-4.3-testing bisection] complete build-i386

[George Dunlap]

On 11/07/13 16:07, Ian Jackson wrote:
> xen.org writes ("[xen-4.3-testing bisection] complete build-i386"):
>> branch xen-4.3-testing
>> xen branch xen-4.3-testing
>> job build-i386
>> test xen-build
>>
>> Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-4.3-testing.git
>> Tree: xen git://xenbits.xen.org/xen.git
>>
>> *** Found and reproduced problem changeset ***
>>
>>    Bug is in tree:  xen git://xenbits.xen.org/xen.git
>>    Bug introduced:  3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
>>    Bug not present: c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
> ...
>>        Config.mk: Update QEMU_TAG and QEMU_UPSTREAM_REVISION for 4.3
> This is because
>   (a) qemu-xen-4.3-testing.git (the trad tree) had not had
>       git-update-server-info run in it since the xen-4.3.0 tag was
>       made.
>   (b) Config.mk specifies an http url (rather than a git url) for
>       the qemu trees.
>
> I have fixed this and prodded the tester into starting another 4.3
> test right away.
>
>
> Post mortem:
>
> The xen-4.3.0 tag in that tree was made some time last week.  Running
> g-u-s-i is in the checklist.  I don't know why the checklist wasn't
> followed properly (by me).  That was clearly a mistake.  Sorry.
>
> I think, though, that our release process has become quite complex and
> single the ad-hoc checklist file we (mostly I) are using is creaking
> under the strain rather.
>
> ACTION: I think we need to write down our actual relase process from a
> higher-level point of view, and have a separate file for technical
> processes and runes at various stages.

I've put up a skeleton of a release checklist here:

http://wiki.xenproject.org/wiki/ReleaseChecklist

A higher-level overview, plus maybe some "be aware of this" reminders 
before the step-by-step might be useful -- I might add those in tomorrow 
(or you can do it if you have a mind).

I was also thinking it might be nice to have a text-based "to-do" for 
the different stages somewhere, so that whoever was doing it could copy 
the list to their personal to-do file, and delete the items as they did 
them (to make sure nothing got dropped).  Not sure how useful that would be.

  -George