Re: [libvirt test] 55257: regressions - FAIL

[Ian Campbell <ian.campbell@citrix.com>]

On Wed, 2015-05-13 at 18:46 +0100, Anthony PERARD wrote:
> On Wed, May 13, 2015 at 09:46:28AM +0100, Ian Campbell wrote:
> > On Mon, 2015-05-11 at 10:36 -0600, Jim Fehlig wrote:
> > [...]
> > > > The qemu log is sadly empty so I've no clue why this timed out.
> > > >   
> > > 
> > > I guess qemu didn't run at all...
> > > 
> > > > Perhaps there is something in 
> > > > http://logs.test-lab.xenproject.org/osstest/logs/55257/test-amd64-amd64-libvirt/merlot1---var-log-libvirt-libvirtd.log.gz
> > > > I can't make heads nor tail though.
> > > >   
> > > 
> > > Nothing interesting.  Only the unhelpful
> > > 
> > > 2015-05-11 12:42:17.451+0000: 4280: error : libxlDomainStart:1032 :
> > > internal error: libxenlight failed to create new domain
> > > 'debian.guest.osstest'
> > 
> > This happened again in
> > http://logs.test-lab.xenproject.org/osstest/logs/55349/test-amd64-amd64-libvirt/info.html
> > 
> > Is there anything we could tweak in osstest to produce more helpful
> > logging?
> 
> Well we can find in var-log-libvirt-libvirtd.log.gz this:
> 2015-05-12 17:39:35.180+0000: 4329: error : libxlDomainStart:1032 : internal error: libxenlight failed to create new domain 'debian.guest.osstest'
> 
> And for more information we need to look into the driver specific log,
> libxl logs in var-log-libvirt-libxl-libxl-driver.log:
> libxl: error: libxl_exec.c:393:spawn_watch_event: domain 1 device model: startup timed out

Thanks, all of that was mentioned earlier in the thread too, I was
looking for ways to get more info.

> I'm seeing this error a lot on our OpenStack CI loop, I thought the error
> was due to the "host" been very busy, but if osstest is having the same
> issue, then there is probably something wrong with libxl+libvirt :(.

Are you able to reproduce at will or is it like osstest and just a
sporadic failure?

I suppose the openstack CI loop doesn't capture anything more
interesting than osstest does?

FWIW http://logs.test-lab.xenproject.org/osstest/logs/55443/ seems to
have two more instances of this (amd64 and i386), but with no 
interesting logs still and a different one on ARM:

http://logs.test-lab.xenproject.org/osstest/logs/55443/test-armhf-armhf-libvirt/11.ts-guest-start.log:
2015-05-13 09:23:32.193+0000: 16389: info : libvirt version: 1.2.16
2015-05-13 09:23:32.193+0000: 16389: warning : virKeepAliveTimerInternal:143 : No response from client 0xb7000c38 after 6 keepalive messages in 35 seconds
2015-05-13 09:23:32.193+0000: 16390: warning : virKeepAliveTimerInternal:143 : No response from client 0xb7000c38 after 6 keepalive messages in 35 seconds
error: Failed to create domain from /etc/xen/debian.guest.osstest.cfg.xml
error: internal error: received hangup / error event on socket

In that case the the libxl-driver log ends with:
libxl: debug: libxl_dm.c:1495:libxl__spawn_local_dm: Spawning device-model /usr/local/lib/xen/bin/qemu-system-i386 with arguments:
[...]
libxl: debug: libxl_event.c:600:libxl__ev_xswatch_register: watch w=0xb2e07bcc wpath=/local/domain/0/device-model/1/state token=3/0: register slotnum=3
libxl: debug: libxl_create.c:1560:do_domain_create: ao 0xb2e044f0: inprogress: poller=0xb2e07590, flags=i
libxl: debug: libxl_event.c:537:watchfd_callback: watch w=0xb2e07bcc wpath=/local/domain/0/device-model/1/state token=3/0: event epath=/local/domain/0/device-model/1/state

Which I don't think is complete, i.e. there should be more? Not sure if
this gives a hint for the x86 case too?

I don't see anything useful in
http://logs.test-lab.xenproject.org/osstest/logs/55443/test-armhf-armhf-libvirt/arndale-lakeside---var-log-libvirt-libvirtd.log.gz

Ian.