Re: [libvirt test] 55257: regressions - FAIL

[Jim Fehlig <jfehlig@suse.com>]

Ian Campbell wrote:
> On Wed, 2015-05-13 at 18:46 +0100, Anthony PERARD wrote:
>   
>> On Wed, May 13, 2015 at 09:46:28AM +0100, Ian Campbell wrote:
>>     
>>> On Mon, 2015-05-11 at 10:36 -0600, Jim Fehlig wrote:
>>> [...]
>>>       
>>>>> The qemu log is sadly empty so I've no clue why this timed out.
>>>>>   
>>>>>           
>>>> I guess qemu didn't run at all...
>>>>
>>>>         
>>>>> Perhaps there is something in 
>>>>> http://logs.test-lab.xenproject.org/osstest/logs/55257/test-amd64-amd64-libvirt/merlot1---var-log-libvirt-libvirtd.log.gz
>>>>> I can't make heads nor tail though.
>>>>>   
>>>>>           
>>>> Nothing interesting.  Only the unhelpful
>>>>
>>>> 2015-05-11 12:42:17.451+0000: 4280: error : libxlDomainStart:1032 :
>>>> internal error: libxenlight failed to create new domain
>>>> 'debian.guest.osstest'
>>>>         
>>> This happened again in
>>> http://logs.test-lab.xenproject.org/osstest/logs/55349/test-amd64-amd64-libvirt/info.html
>>>
>>> Is there anything we could tweak in osstest to produce more helpful
>>> logging?
>>>       
>> Well we can find in var-log-libvirt-libvirtd.log.gz this:
>> 2015-05-12 17:39:35.180+0000: 4329: error : libxlDomainStart:1032 : internal error: libxenlight failed to create new domain 'debian.guest.osstest'
>>
>> And for more information we need to look into the driver specific log,
>> libxl logs in var-log-libvirt-libxl-libxl-driver.log:
>> libxl: error: libxl_exec.c:393:spawn_watch_event: domain 1 device model: startup timed out
>>     
>
> Thanks, all of that was mentioned earlier in the thread too, I was
> looking for ways to get more info.
>
>   
>> I'm seeing this error a lot on our OpenStack CI loop, I thought the error
>> was due to the "host" been very busy, but if osstest is having the same
>> issue, then there is probably something wrong with libxl+libvirt :(.
>>     
>
> Are you able to reproduce at will or is it like osstest and just a
> sporadic failure?
>
> I suppose the openstack CI loop doesn't capture anything more
> interesting than osstest does?
>
> FWIW http://logs.test-lab.xenproject.org/osstest/logs/55443/ seems to
> have two more instances of this (amd64 and i386)

More cases of qemu not starting.  I'm not sure how we can get more
details about that.

>  but with no 
> interesting logs still and a different one on ARM:
>
> http://logs.test-lab.xenproject.org/osstest/logs/55443/test-armhf-armhf-libvirt/11.ts-guest-start.log:
> 2015-05-13 09:23:32.193+0000: 16389: info : libvirt version: 1.2.16
> 2015-05-13 09:23:32.193+0000: 16389: warning : virKeepAliveTimerInternal:143 : No response from client 0xb7000c38 after 6 keepalive messages in 35 seconds
> 2015-05-13 09:23:32.193+0000: 16390: warning : virKeepAliveTimerInternal:143 : No response from client 0xb7000c38 after 6 keepalive messages in 35 seconds
> error: Failed to create domain from /etc/xen/debian.guest.osstest.cfg.xml
> error: internal error: received hangup / error event on socket
>   

In this case it seems libvirtd crashed.

> In that case the the libxl-driver log ends with:
> libxl: debug: libxl_dm.c:1495:libxl__spawn_local_dm: Spawning device-model /usr/local/lib/xen/bin/qemu-system-i386 with arguments:
> [...]
> libxl: debug: libxl_event.c:600:libxl__ev_xswatch_register: watch w=0xb2e07bcc wpath=/local/domain/0/device-model/1/state token=3/0: register slotnum=3
> libxl: debug: libxl_create.c:1560:do_domain_create: ao 0xb2e044f0: inprogress: poller=0xb2e07590, flags=i
> libxl: debug: libxl_event.c:537:watchfd_callback: watch w=0xb2e07bcc wpath=/local/domain/0/device-model/1/state token=3/0: event epath=/local/domain/0/device-model/1/state
>
> Which I don't think is complete, i.e. there should be more? Not sure if
> this gives a hint for the x86 case too?
>   

More hint that libvirtd crashed.  Have there been any attempts to
reproduce this outside of the test rig?  Or capture a core dump?

Regards,
Jim