Re: [libvirt test] 55257: regressions - FAIL

[Ian Campbell <ian.campbell@citrix.com>]

On Mon, 2015-05-11 at 12:46 +0000, osstest service user wrote:
> flight 55257 libvirt real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/55257/
> 
> Regressions :-(
> 
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  test-amd64-amd64-libvirt     11 guest-start               fail REGR. vs. 53854

I fear this may be a new heisenbug.

I fear a heisenbug because flight 53854 passed and there is only one
more, completely unrelated change here.

I saw something similar in
http://logs.test-lab.xenproject.org/osstest/logs/53721/ which was an
osstest flight against itself (so not posted to the list). That one had:
>  test-amd64-i386-libvirt      11 guest-start               fail REGR. vs. 53073
>  test-amd64-amd64-libvirt     11 guest-start               fail REGR. vs. 53073

In that case the range of libvirt was more useful than the one commit
here. It was 225aa80246d5..63a368012df, FWIW. Being a heisenbug I'm not
sure if 225aa80246d5 was OK or not

http://logs.test-lab.xenproject.org/osstest/logs/55257/test-amd64-amd64-libvirt/merlot1---var-log-libvirt-libxl-libxl-driver.log ends with:

libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=xvda2 spec.backend=qdisk
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x7f979cd10750: deregister unregistered
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=qdisk
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x7f979ccdd370: deregister unregistered
libxl: debug: libxl_dm.c:1487:libxl__spawn_local_dm: Spawning device-model /usr/local/lib/xen/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   /usr/local/lib/xen/bin/qemu-system-i386
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -xen-domid
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   1
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-1,server,nowait
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -no-shutdown
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-1,server,nowait
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   chardev=libxenstat-cmd,mode=control
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -nodefaults
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -xen-attach
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -name
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   debian.guest.osstest
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -vnc
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   none
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -display
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   none
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -nographic
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -machine
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   xenpv
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -m
libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   512
libxl: debug: libxl_event.c:577:libxl__ev_xswatch_register: watch w=0x7f979cd0f430 wpath=/local/domain/0/device-model/1/state token=3/0: register slotnum=3
libxl: debug: libxl_create.c:1549:do_domain_create: ao 0x7f979cce1b10: inprogress: poller=0x7f979ccdd260, flags=i
libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x7f979cd0f430 wpath=/local/domain/0/device-model/1/state token=3/0: event epath=/local/domain/0/device-model/1/state
libxl: debug: libxl_aoutils.c:87:xswait_timeout_callback: domain 1 device model startup: xswait timeout (path=/local/domain/0/device-model/1/state)
libxl: debug: libxl_event.c:615:libxl__ev_xswatch_deregister: watch w=0x7f979cd0f430 wpath=/local/domain/0/device-model/1/state token=3/0: deregister slotnum=3
libxl: error: libxl_exec.c:393:spawn_watch_event: domain 1 device model: startup timed out
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x7f979cd0f430: deregister unregistered
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x7f979cd0f430: deregister unregistered
libxl: error: libxl_dm.c:1558:device_model_spawn_outcome: domain 1 device model: spawn failed (rc=-3)
libxl: error: libxl_create.c:1351:domcreate_devmodel_started: device model did not start: -3
libxl: debug: libxl_dm.c:1671:kill_device_model: Device Model signaled
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x7f979cd12ce0: deregister unregistered
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x7f979cd13010: deregister unregistered
libxl: info: libxl.c:1701:devices_destroy_cb: forked pid 18657 for destroy of domain 1
libxl: debug: libxl_event.c:1766:libxl__ao_complete: ao 0x7f979cce1b10: complete, rc=-3

The qemu log is sadly empty so I've no clue why this timed out.

Perhaps there is something in 
http://logs.test-lab.xenproject.org/osstest/logs/55257/test-amd64-amd64-libvirt/merlot1---var-log-libvirt-libvirtd.log.gz
I can't make heads nor tail though.


> Tests which did not succeed, but are not blocking:
>  test-amd64-i386-libvirt-xsm  11 guest-start                  fail   never pass
>  test-amd64-amd64-libvirt-xsm 11 guest-start                  fail   never pass
>  test-armhf-armhf-libvirt-xsm  6 xen-boot                     fail   never pass
>  test-amd64-i386-libvirt      12 migrate-support-check        fail   never pass
>  test-armhf-armhf-libvirt     12 migrate-support-check        fail   never pass
> 
> version targeted for testing:
>  libvirt              8910e063dbafc09695b2100c80213be569abb7ef
> baseline version:
>  libvirt              fd74e231751334b64af0934b680c5cc62f652453
>