Re: [libvirt test] 55257: regressions - FAIL

[Jim Fehlig <jfehlig@suse.com>]

Ian Campbell wrote:
> On Mon, 2015-05-11 at 12:46 +0000, osstest service user wrote:
>   
>> flight 55257 libvirt real [real]
>> http://logs.test-lab.xenproject.org/osstest/logs/55257/
>>
>> Regressions :-(
>>
>> Tests which did not succeed and are blocking,
>> including tests which could not be run:
>>  test-amd64-amd64-libvirt     11 guest-start               fail REGR. vs. 53854
>>     
>
> I fear this may be a new heisenbug.
>
> I fear a heisenbug because flight 53854 passed and there is only one
> more, completely unrelated change here.
>
> I saw something similar in
> http://logs.test-lab.xenproject.org/osstest/logs/53721/ which was an
> osstest flight against itself (so not posted to the list). That one had:
>   
>>  test-amd64-i386-libvirt      11 guest-start               fail REGR. vs. 53073
>>  test-amd64-amd64-libvirt     11 guest-start               fail REGR. vs. 53073
>>     
>
> In that case the range of libvirt was more useful than the one commit
> here. It was 225aa80246d5..63a368012df, FWIW. Being a heisenbug I'm not
> sure if 225aa80246d5 was OK or not
>   

225aa80246d5 only touches the qemu driver and should not affect Xen.

> http://logs.test-lab.xenproject.org/osstest/logs/55257/test-amd64-amd64-libvirt/merlot1---var-log-libvirt-libxl-libxl-driver.log ends with:
>
> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=xvda2 spec.backend=qdisk
> libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x7f979cd10750: deregister unregistered
> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=qdisk
> libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x7f979ccdd370: deregister unregistered
> libxl: debug: libxl_dm.c:1487:libxl__spawn_local_dm: Spawning device-model /usr/local/lib/xen/bin/qemu-system-i386 with arguments:
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   /usr/local/lib/xen/bin/qemu-system-i386
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -xen-domid
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   1
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -chardev
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-1,server,nowait
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -no-shutdown
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -mon
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   chardev=libxl-cmd,mode=control
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -chardev
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-1,server,nowait
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -mon
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   chardev=libxenstat-cmd,mode=control
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -nodefaults
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -xen-attach
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -name
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   debian.guest.osstest
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -vnc
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   none
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -display
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   none
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -nographic
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -machine
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   xenpv
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   -m
> libxl: debug: libxl_dm.c:1489:libxl__spawn_local_dm:   512
> libxl: debug: libxl_event.c:577:libxl__ev_xswatch_register: watch w=0x7f979cd0f430 wpath=/local/domain/0/device-model/1/state token=3/0: register slotnum=3
> libxl: debug: libxl_create.c:1549:do_domain_create: ao 0x7f979cce1b10: inprogress: poller=0x7f979ccdd260, flags=i
> libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x7f979cd0f430 wpath=/local/domain/0/device-model/1/state token=3/0: event epath=/local/domain/0/device-model/1/state
> libxl: debug: libxl_aoutils.c:87:xswait_timeout_callback: domain 1 device model startup: xswait timeout (path=/local/domain/0/device-model/1/state)
> libxl: debug: libxl_event.c:615:libxl__ev_xswatch_deregister: watch w=0x7f979cd0f430 wpath=/local/domain/0/device-model/1/state token=3/0: deregister slotnum=3
> libxl: error: libxl_exec.c:393:spawn_watch_event: domain 1 device model: startup timed out
> libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x7f979cd0f430: deregister unregistered
> libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x7f979cd0f430: deregister unregistered
> libxl: error: libxl_dm.c:1558:device_model_spawn_outcome: domain 1 device model: spawn failed (rc=-3)
> libxl: error: libxl_create.c:1351:domcreate_devmodel_started: device model did not start: -3
> libxl: debug: libxl_dm.c:1671:kill_device_model: Device Model signaled
> libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x7f979cd12ce0: deregister unregistered
> libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x7f979cd13010: deregister unregistered
> libxl: info: libxl.c:1701:devices_destroy_cb: forked pid 18657 for destroy of domain 1
> libxl: debug: libxl_event.c:1766:libxl__ao_complete: ao 0x7f979cce1b10: complete, rc=-3
>
> The qemu log is sadly empty so I've no clue why this timed out.
>   

I guess qemu didn't run at all...

> Perhaps there is something in 
> http://logs.test-lab.xenproject.org/osstest/logs/55257/test-amd64-amd64-libvirt/merlot1---var-log-libvirt-libvirtd.log.gz
> I can't make heads nor tail though.
>   

Nothing interesting.  Only the unhelpful

2015-05-11 12:42:17.451+0000: 4280: error : libxlDomainStart:1032 :
internal error: libxenlight failed to create new domain
'debian.guest.osstest'

Off topic, but I'd really like to improve reporting of libxl errors in
libvirt.  Currently, when calls to libxl_foo fail, libvirt simply
reports something like "libxenlight failed foo".  Users must resort to
/var/log/libvirt/libxl/libxl-driver.log and
/var/log/xen/qemu-dm-<domname>.log for details.  Perhaps a topic for the
dev summit.

Regards,
Jim