Re: [xen-unstable bisection] complete test-amd64-amd64-i386-pvgrub

[Julien Grall <julien.grall@linaro.org>]

Hi Wei,

On 08/11/17 11:36, Wei Liu wrote:
> On Tue, Nov 07, 2017 at 05:24:32PM +0000, Julien Grall wrote:
>> Hi Wei,
>>
>> On 07/11/17 15:13, Wei Liu wrote:
>>> On Tue, Nov 07, 2017 at 03:09:07PM +0000, Julien Grall wrote:
>>>> Hi Wei,
>>>>
>>>> On 06/11/17 14:55, Wei Liu wrote:
>>>>> On Mon, Nov 06, 2017 at 01:47:56PM +0000, osstest service owner wrote:
>>>>>> branch xen-unstable
>>>>>> xenbranch xen-unstable
>>>>>> job test-amd64-amd64-i386-pvgrub
>>>>>> testid guest-start
>>>>>>
>>>>>> Tree: linux git://xenbits.xen.org/linux-pvops.git
>>>>>> Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
>>>>>> Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git
>>>>>> Tree: qemuu git://xenbits.xen.org/qemu-xen.git
>>>>>> Tree: xen git://xenbits.xen.org/xen.git
>>>>>>
>>>>>> *** Found and reproduced problem changeset ***
>>>>>>
>>>>>>      Bug is in tree:  xen git://xenbits.xen.org/xen.git
>>>>>>      Bug introduced:  f48b5449dabc770acdde6d25cfbd265cfb71034d
>>>>>>      Bug not present: 86cf189a957129ea1ad6468fe9a0887b9e2819f3
>>>>>>      Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/115612/
>>>>>>
>>>>>>
>>>>>>      commit f48b5449dabc770acdde6d25cfbd265cfb71034d
>>>>>>      Author: Wei Liu <wei.liu2@citrix.com>
>>>>>>      Date:   Thu Oct 12 20:19:07 2017 +0100
>>>>>>          tools/dombuilder: Switch to using gfn terminology for console and xenstore rings
>>>>>>          The sole use of xc_dom_translated() and xc_dom_p2m() outside of the domain
>>>>>>          builder is for libxl_dom() to translate the console and xenstore pfns back
>>>>>>          into useful values.  PV guest pfns are only interesting to the domain builder,
>>>>>>          and gfns are the address space used by all other hypercalls.
>>>>>>          Renaming the fields in xc_dom_image is deliberate, as it will cause
>>>>>>          out-of-tree users of the dombuilder to notice the different semantics.
>>>>>>          Correct the terminology throughout xc_dom_gnttab{_hvm,}_seed(), which are all
>>>>>>          using gfns despite the existing variable names.
>>>>>>          Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
>>>>>>          Reviewed-by: Roger Pau Monn?? <roger.pau@citrix.com>
>>>>>>          Acked-by: Wei Liu <wei.liu2@citrix.com>
>>>>>>          Tested-by: Julien Grall <julien.grall@arm.com>
>>>>>>          Release-acked-by: Julien Grall <julien.grall@linaro.org>
>>>>>>          [ wei: fix stubdom build ]
>>>>>>          Signed-off-by: Wei Liu <wei.liu2@citrix.com>
>>>>>
>>>>> This has broken pvgrub. The problem is more than just the name of the
>>>>> variables. I have reverted this and its successor patch.
>>>>
>>>> It looks like osstest is still broken after the patches you reverted (see
>>>> [1] and [2]).
>>>>
>>>> AFAICT, the only series between the two flights is the dombuilder, there are
>>>> 2 patches not reverted.
>>>>
>>>> Do you have an idea of what's going on?
>>>>
>>>> Cheers,
>>>>
>>>> [1] http://logs.test-lab.xenproject.org/osstest/logs/115624/
>>>> [2]
>>>> https://lists.xenproject.org/archives/html/xen-devel/2017-11/msg00391.html
>>>>
>>>
>>> test-amd64-amd64-xl-qemut-win7-amd64 16 guest-localmigrate/x10 fail REGR. vs.  115526

Looking at the osstest result today, this one seem to be intermittent as 
it passed during the night but failed this morning.

>>> test-armhf-armhf-xl-vhd     15 guest-start/debian.repeat fail REGR. vs.  115526
>>
>> The log for the xl-vhd contains ([1])
>>
>> libxl: error: libxl_bootloader.c:283:bootloader_local_detached_cb: Domain 11:unable to detach locally attached disk
>> libxl: error: libxl_create.c:1246:domcreate_rebuild_done: Domain 11:cannot (re-)build domain: -3
>> libxl: debug: libxl_domain.c:1138:devices_destroy_cb: Domain 11:Forked pid 5103 for destroy of domain
>> libxl: debug: libxl_create.c:1683:do_domain_create: Domain 0:ao 0x5d6e8: inprogress: poller=0x56ad8, flags=i
>> libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x5d6e8: complete, rc=-3
>> libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x5d6e8: destroy
>> libxl: debug: libxl_domain.c:868:libxl_domain_destroy: Domain 11:ao 0x5a170: create: how=(nil) callback=(nil) poller=0x56ad8
>> libxl: error: libxl_domain.c:1000:libxl__destroy_domid: Domain 11:Non-existant domain
>> libxl: error: libxl_domain.c:959:domain_destroy_callback: Domain 11:Unable to destroy guest
>> libxl: error: libxl_domain.c:886:domain_destroy_cb: Domain 11:Destruction of domain failed
>> libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x5a170: complete, rc=-21
>> libxl: debug: libxl_domain.c:877:libxl_domain_destroy: Domain 11:ao 0x5a170: inprogress: poller=0x56ad8, flags=ic
>> libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x5a170: destroy
>>
>> It is in guest repeat and has succeed few times before.
>>
>> Looking at the success/failure ([2]), the same configuration passed on the Arndale
>> (see 115580) but fails reliably on the cubietruck.
>>
> 
> The same test failed on Arndale as well in 115314 and 115526, with the
> same error messages.
> 
> http://logs.test-lab.xenproject.org/osstest/logs/115526/test-armhf-armhf-xl-vhd/16.ts-guest-start.log
> 
> So the failure isn't related to Andrew's series.
> 
>> My guess would be the disk is not detached by the previous guest in time.
>> Now the question is why? I am not familiar with this area, any ideas?
>>
> 
> I don't have immediate idea either. I've set up a repro flight so that
> we can have something to play with.

Thanks! Let me know when it is ready.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel