Re: [xen-unstable bisection] complete test-amd64-amd64-i386-pvgrub

[Wei Liu <wei.liu2@citrix.com>]

On Tue, Nov 07, 2017 at 05:24:32PM +0000, Julien Grall wrote:
> Hi Wei,
> 
> On 07/11/17 15:13, Wei Liu wrote:
> > On Tue, Nov 07, 2017 at 03:09:07PM +0000, Julien Grall wrote:
> >> Hi Wei,
> >>
> >> On 06/11/17 14:55, Wei Liu wrote:
> >>> On Mon, Nov 06, 2017 at 01:47:56PM +0000, osstest service owner wrote:
> >>>> branch xen-unstable
> >>>> xenbranch xen-unstable
> >>>> job test-amd64-amd64-i386-pvgrub
> >>>> testid guest-start
> >>>>
> >>>> Tree: linux git://xenbits.xen.org/linux-pvops.git
> >>>> Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
> >>>> Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git
> >>>> Tree: qemuu git://xenbits.xen.org/qemu-xen.git
> >>>> Tree: xen git://xenbits.xen.org/xen.git
> >>>>
> >>>> *** Found and reproduced problem changeset ***
> >>>>
> >>>>     Bug is in tree:  xen git://xenbits.xen.org/xen.git
> >>>>     Bug introduced:  f48b5449dabc770acdde6d25cfbd265cfb71034d
> >>>>     Bug not present: 86cf189a957129ea1ad6468fe9a0887b9e2819f3
> >>>>     Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/115612/
> >>>>
> >>>>
> >>>>     commit f48b5449dabc770acdde6d25cfbd265cfb71034d
> >>>>     Author: Wei Liu <wei.liu2@citrix.com>
> >>>>     Date:   Thu Oct 12 20:19:07 2017 +0100
> >>>>         tools/dombuilder: Switch to using gfn terminology for console and xenstore rings
> >>>>         The sole use of xc_dom_translated() and xc_dom_p2m() outside of the domain
> >>>>         builder is for libxl_dom() to translate the console and xenstore pfns back
> >>>>         into useful values.  PV guest pfns are only interesting to the domain builder,
> >>>>         and gfns are the address space used by all other hypercalls.
> >>>>         Renaming the fields in xc_dom_image is deliberate, as it will cause
> >>>>         out-of-tree users of the dombuilder to notice the different semantics.
> >>>>         Correct the terminology throughout xc_dom_gnttab{_hvm,}_seed(), which are all
> >>>>         using gfns despite the existing variable names.
> >>>>         Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> >>>>         Reviewed-by: Roger Pau Monn?? <roger.pau@citrix.com>
> >>>>         Acked-by: Wei Liu <wei.liu2@citrix.com>
> >>>>         Tested-by: Julien Grall <julien.grall@arm.com>
> >>>>         Release-acked-by: Julien Grall <julien.grall@linaro.org>
> >>>>         [ wei: fix stubdom build ]
> >>>>         Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> >>>
> >>> This has broken pvgrub. The problem is more than just the name of the
> >>> variables. I have reverted this and its successor patch.
> >>
> >> It looks like osstest is still broken after the patches you reverted (see
> >> [1] and [2]).
> >>
> >> AFAICT, the only series between the two flights is the dombuilder, there are
> >> 2 patches not reverted.
> >>
> >> Do you have an idea of what's going on?
> >>
> >> Cheers,
> >>
> >> [1] http://logs.test-lab.xenproject.org/osstest/logs/115624/
> >> [2]
> >> https://lists.xenproject.org/archives/html/xen-devel/2017-11/msg00391.html
> >>
> > 
> > test-amd64-amd64-xl-qemut-win7-amd64 16 guest-localmigrate/x10 fail REGR. vs.  115526
> > test-armhf-armhf-xl-vhd     15 guest-start/debian.repeat fail REGR. vs.  115526
> 
> The log for the xl-vhd contains ([1])
> 
> libxl: error: libxl_bootloader.c:283:bootloader_local_detached_cb: Domain 11:unable to detach locally attached disk
> libxl: error: libxl_create.c:1246:domcreate_rebuild_done: Domain 11:cannot (re-)build domain: -3
> libxl: debug: libxl_domain.c:1138:devices_destroy_cb: Domain 11:Forked pid 5103 for destroy of domain
> libxl: debug: libxl_create.c:1683:do_domain_create: Domain 0:ao 0x5d6e8: inprogress: poller=0x56ad8, flags=i
> libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x5d6e8: complete, rc=-3
> libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x5d6e8: destroy
> libxl: debug: libxl_domain.c:868:libxl_domain_destroy: Domain 11:ao 0x5a170: create: how=(nil) callback=(nil) poller=0x56ad8
> libxl: error: libxl_domain.c:1000:libxl__destroy_domid: Domain 11:Non-existant domain
> libxl: error: libxl_domain.c:959:domain_destroy_callback: Domain 11:Unable to destroy guest
> libxl: error: libxl_domain.c:886:domain_destroy_cb: Domain 11:Destruction of domain failed
> libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x5a170: complete, rc=-21
> libxl: debug: libxl_domain.c:877:libxl_domain_destroy: Domain 11:ao 0x5a170: inprogress: poller=0x56ad8, flags=ic
> libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x5a170: destroy
> 
> It is in guest repeat and has succeed few times before.
> 
> Looking at the success/failure ([2]), the same configuration passed on the Arndale
> (see 115580) but fails reliably on the cubietruck.
> 

The same test failed on Arndale as well in 115314 and 115526, with the
same error messages.

http://logs.test-lab.xenproject.org/osstest/logs/115526/test-armhf-armhf-xl-vhd/16.ts-guest-start.log

So the failure isn't related to Andrew's series.

> My guess would be the disk is not detached by the previous guest in time.
> Now the question is why? I am not familiar with this area, any ideas? 
> 

I don't have immediate idea either. I've set up a repro flight so that
we can have something to play with.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel