Re: [xen-unstable bisection] complete test-i386-i386-xl

[Ian Campbell <Ian.Campbell@citrix.com>]

On Sat, 2012-01-14 at 12:05 +0000, xen.org wrote:
> branch xen-unstable
> xen branch xen-unstable
> job test-i386-i386-xl
> test guest-localmigrate
> 
> Tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
> Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
> Tree: xen http://xenbits.xen.org/staging/xen-unstable.hg
> 
> *** Found and reproduced problem changeset ***
> 
>   Bug is in tree:  xen http://xenbits.xen.org/staging/xen-unstable.hg
>   Bug introduced:  3c4c5df2cd0e
>   Bug not present: 92630d4b093e
> 
> 
>   changeset:   24471:3c4c5df2cd0e
>   user:        Paul Durrant <paul.durrant@citrix.com>
>   date:        Fri Dec 16 14:54:14 2011 +0000
>       
>       tools: VM generation ID save/restore and migrate.
>       
>       Add code to track the address of the VM generation ID buffer across a
>       save/restore or migrate, and increment it as necessary.
>       The address of the buffer is written into xenstore by hvmloader at
>       boot time. It must be read from xenstore by the caller of
>       xc_domain_save() and then written back again by the caller of
>       xc_domain_restore().
>       
>       Note that the changes to xc_save.c and xc_restore.c are merely
>       sufficient for them to build.
>       
>       Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
>       Committed-by: Ian Jackson <ian.jackson.citrix.com>

The logs show a guest kernel crash but I think this is a second order
effect (which I'll investigate) related to xl not recovering from a
failed migration in a way that guest kernels actually like.

The original failure seems to be a PV migration failure. Paul can you
look into that aspect please. Probably an imbalance for PV save/restore
due to something not correctly gated on HVM on one side or the other.

Thanks,
Ian.

>       
>       
> 
> 
> For bisection revision-tuple graph see:
>    http://www.chiark.greenend.org.uk/~xensrcts/results/bisect.xen-unstable.test-i386-i386-xl.guest-localmigrate.html
> Revision IDs in each graph node refer, respectively, to the Trees above.
> 
> ----------------------------------------
> Searching for failure / basis pass:
>  10700 fail [host=woodlouse] / 10649 ok.
> Failure / basis pass flights: 10700 / 10649
> Tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
> Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
> Tree: xen http://xenbits.xen.org/staging/xen-unstable.hg
> Latest 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 6d8888519e3a
> Basis pass 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 5b2676ac1321
> Generating revisions with ./adhoc-revtuple-generator  git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git#59b1ffe039ce366305d48176b7fccce29ce729d3-59b1ffe039ce366305d48176b7fccce29ce729d3 git://xenbits.xen.org/staging/qemu-xen-unstable.git#bb36d632e4cabf47882adff07a45c6702c4a5b30-bb36d632e4cabf47882adff07a45c6702c4a5b30 http://xenbits.xen.org/staging/xen-unstable.hg#5b2676ac1321-6d8888519e3a
> pulling from ssh://xen@xenbits.xen.org/HG/staging/xen-unstable.hg
> searching for changes
> no changes found
> pulling from ssh://xen@xenbits.xen.org/HG/staging/xen-unstable.hg
> searching for changes
> no changes found
> Loaded 49 nodes in revision graph
> Searching for test results:
>  10697 [host=potato-beetle]
>  10708 fail 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 db22b1aa11d3
>  10689 [host=itch-mite]
>  10698 [host=potato-beetle]
>  10715 fail 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 3c4c5df2cd0e
>  10709 pass 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e
>  10681 [host=potato-beetle]
>  10645 [host=potato-beetle]
>  10699 [host=itch-mite]
>  10646 [host=moss-bug]
>  10649 pass 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 5b2676ac1321
>  10701 [host=itch-mite]
>  10680 [host=lace-bug]
>  10710 fail 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 1e3fa01b0c9e
>  10702 [host=itch-mite]
>  10703 [host=itch-mite]
>  10711 fail 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 3c4c5df2cd0e
>  10700 fail 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 6d8888519e3a
>  10704 [host=itch-mite]
>  10712 pass 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e
>  10705 pass 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 5b2676ac1321
>  10706 fail 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 6d8888519e3a
>  10713 fail 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 3c4c5df2cd0e
>  10707 fail 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 32dd444700bd
>  10714 pass 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e
> Searching for interesting versions
>  Result found: flight 10649 (pass), for basis pass
>  Result found: flight 10700 (fail), for basis failure
>  Repro found: flight 10705 (pass), for basis pass
>  Repro found: flight 10706 (fail), for basis failure
>  0 revisions at 59b1ffe039ce366305d48176b7fccce29ce729d3 bb36d632e4cabf47882adff07a45c6702c4a5b30 92630d4b093e
> No revisions left to test, checking graph state.
>  Result found: flight 10709 (pass), for last pass
>  Result found: flight 10711 (fail), for first failure
>  Repro found: flight 10712 (pass), for last pass
>  Repro found: flight 10713 (fail), for first failure
>  Repro found: flight 10714 (pass), for last pass
>  Repro found: flight 10715 (fail), for first failure
> 
> *** Found and reproduced problem changeset ***
> 
>   Bug is in tree:  xen http://xenbits.xen.org/staging/xen-unstable.hg
>   Bug introduced:  3c4c5df2cd0e
>   Bug not present: 92630d4b093e
> 
> pulling from ssh://xen@xenbits.xen.org/HG/staging/xen-unstable.hg
> searching for changes
> no changes found
> 
>   changeset:   24471:3c4c5df2cd0e
>   user:        Paul Durrant <paul.durrant@citrix.com>
>   date:        Fri Dec 16 14:54:14 2011 +0000
>       
>       tools: VM generation ID save/restore and migrate.
>       
>       Add code to track the address of the VM generation ID buffer across a
>       save/restore or migrate, and increment it as necessary.
>       The address of the buffer is written into xenstore by hvmloader at
>       boot time. It must be read from xenstore by the caller of
>       xc_domain_save() and then written back again by the caller of
>       xc_domain_restore().
>       
>       Note that the changes to xc_save.c and xc_restore.c are merely
>       sufficient for them to build.
>       
>       Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
>       Committed-by: Ian Jackson <ian.jackson.citrix.com>
>       
>       
> 
> Revision graph left in /home/xc_osstest/results/bisect.xen-unstable.test-i386-i386-xl.guest-localmigrate.{dot,ps,png,html}.
> ----------------------------------------
> 10715: ALL FAIL
> 
> flight 10715 xen-unstable real-bisect [real]
> http://www.chiark.greenend.org.uk/~xensrcts/logs/10715/
> 
> 
> jobs:
>  test-i386-i386-xl                                            fail    
> 
> 
> ------------------------------------------------------------
> sg-report-flight on woking.cam.xci-test.com
> logs: /home/xc_osstest/logs
> images: /home/xc_osstest/images
> 
> Logs, config files, etc. are available at
>     http://www.chiark.greenend.org.uk/~xensrcts/logs
> 
> Test harness code can be found at
>     http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel