From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jim Fehlig Subject: Re: [libvirt test] 55257: regressions - FAIL Date: Thu, 14 May 2015 15:21:41 -0600 Message-ID: <555511E5.3040304@suse.com> References: <1431350527.8263.64.camel@citrix.com> <5550DA91.4050101@suse.com> <1431506788.8263.202.camel@citrix.com> <20150513174654.GL20952@perard.uk.xensource.com> <1431600438.13579.28.camel@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1431600438.13579.28.camel@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Ian Campbell Cc: Anthony PERARD , xen-devel@lists.xensource.com, ian.jackson@eu.citrix.com List-Id: xen-devel@lists.xenproject.org Ian Campbell wrote: > On Wed, 2015-05-13 at 18:46 +0100, Anthony PERARD wrote: > >> On Wed, May 13, 2015 at 09:46:28AM +0100, Ian Campbell wrote: >> >>> On Mon, 2015-05-11 at 10:36 -0600, Jim Fehlig wrote: >>> [...] >>> >>>>> The qemu log is sadly empty so I've no clue why this timed out. >>>>> >>>>> >>>> I guess qemu didn't run at all... >>>> >>>> >>>>> Perhaps there is something in >>>>> http://logs.test-lab.xenproject.org/osstest/logs/55257/test-amd64-amd64-libvirt/merlot1---var-log-libvirt-libvirtd.log.gz >>>>> I can't make heads nor tail though. >>>>> >>>>> >>>> Nothing interesting. Only the unhelpful >>>> >>>> 2015-05-11 12:42:17.451+0000: 4280: error : libxlDomainStart:1032 : >>>> internal error: libxenlight failed to create new domain >>>> 'debian.guest.osstest' >>>> >>> This happened again in >>> http://logs.test-lab.xenproject.org/osstest/logs/55349/test-amd64-amd64-libvirt/info.html >>> >>> Is there anything we could tweak in osstest to produce more helpful >>> logging? >>> >> Well we can find in var-log-libvirt-libvirtd.log.gz this: >> 2015-05-12 17:39:35.180+0000: 4329: error : libxlDomainStart:1032 : internal error: libxenlight failed to create new domain 'debian.guest.osstest' >> >> And for more information we need to look into the driver specific log, >> libxl logs in var-log-libvirt-libxl-libxl-driver.log: >> libxl: error: libxl_exec.c:393:spawn_watch_event: domain 1 device model: startup timed out >> > > Thanks, all of that was mentioned earlier in the thread too, I was > looking for ways to get more info. > > >> I'm seeing this error a lot on our OpenStack CI loop, I thought the error >> was due to the "host" been very busy, but if osstest is having the same >> issue, then there is probably something wrong with libxl+libvirt :(. >> > > Are you able to reproduce at will or is it like osstest and just a > sporadic failure? > > I suppose the openstack CI loop doesn't capture anything more > interesting than osstest does? > > FWIW http://logs.test-lab.xenproject.org/osstest/logs/55443/ seems to > have two more instances of this (amd64 and i386) More cases of qemu not starting. I'm not sure how we can get more details about that. > but with no > interesting logs still and a different one on ARM: > > http://logs.test-lab.xenproject.org/osstest/logs/55443/test-armhf-armhf-libvirt/11.ts-guest-start.log: > 2015-05-13 09:23:32.193+0000: 16389: info : libvirt version: 1.2.16 > 2015-05-13 09:23:32.193+0000: 16389: warning : virKeepAliveTimerInternal:143 : No response from client 0xb7000c38 after 6 keepalive messages in 35 seconds > 2015-05-13 09:23:32.193+0000: 16390: warning : virKeepAliveTimerInternal:143 : No response from client 0xb7000c38 after 6 keepalive messages in 35 seconds > error: Failed to create domain from /etc/xen/debian.guest.osstest.cfg.xml > error: internal error: received hangup / error event on socket > In this case it seems libvirtd crashed. > In that case the the libxl-driver log ends with: > libxl: debug: libxl_dm.c:1495:libxl__spawn_local_dm: Spawning device-model /usr/local/lib/xen/bin/qemu-system-i386 with arguments: > [...] > libxl: debug: libxl_event.c:600:libxl__ev_xswatch_register: watch w=0xb2e07bcc wpath=/local/domain/0/device-model/1/state token=3/0: register slotnum=3 > libxl: debug: libxl_create.c:1560:do_domain_create: ao 0xb2e044f0: inprogress: poller=0xb2e07590, flags=i > libxl: debug: libxl_event.c:537:watchfd_callback: watch w=0xb2e07bcc wpath=/local/domain/0/device-model/1/state token=3/0: event epath=/local/domain/0/device-model/1/state > > Which I don't think is complete, i.e. there should be more? Not sure if > this gives a hint for the x86 case too? > More hint that libvirtd crashed. Have there been any attempts to reproduce this outside of the test rig? Or capture a core dump? Regards, Jim