Re: [xen-unstable test] 35257: regressions - FAIL

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-02-27 at 10:48 +0000, Wei Liu wrote:
> On Fri, Feb 27, 2015 at 09:42:29AM +0000, Ian Campbell wrote:
> > On Thu, 2015-02-26 at 20:14 +0000, xen.org wrote:
> > > flight 35257 xen-unstable real [real]
> > > http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/
> > > 
> > > Regressions :-(
> > > 
> > > Tests which did not succeed and are blocking,
> > > including tests which could not be run:
> > >  test-armhf-armhf-libvirt     12 guest-start.2             fail REGR. vs. 34629
> > 
> > logs:
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/info.html
> > 
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/12.ts-guest-start.log
> >         2015-02-23 20:21:48 Z executing ssh ... root@10.80.229.106 virsh domxml-from-native xen-xl /etc/xen/debian.guest.osstest.cfg > /etc/xen/debian.guest.osstest.cfg.xml
> >         error: failed to connect to the hypervisor
> >         error: no valid connection
> >         error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused
> > 
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/marilith-n4-output-ps_wwwaxf_-eo_pid%2Ctty%2Cstat%2Ctime%2Cnice%2Cpsr%2Cpcpu%2Cpmem%2Cnwchan%2Cwchan%2325%2Cargs
> > appears to show no libvirtd process.
> > 
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/marilith-n4---var-log-libvirt-libvirtd.log
> > says:
> >         2015-02-23 20:13:15.556+0000: 2133: info : libvirt version: 1.2.13
> >         2015-02-23 20:13:15.556+0000: 2133: error : dnsmasqCapsRefreshInternal:726 : Cannot check dnsmasq binary dnsmasq: No such file or directory
> >         2015-02-23 20:13:15.845+0000: 2133: error : virFirewallValidateBackend:193 : direct firewall backend requested, but /sbin/ebtables is not available: No such file or directory
> >         
> > I think these are just spurious.
> > 
> >         2015-02-23 20:13:15.845+0000: 2133: error : virFirewallApply:936 : out of memory
> >         
> > 
> >         2015-02-23 20:13:16.092+0000: 2133: error : virExec:491 : Cannot find 'pm-is-supported' in path: No such file or directory
> >         2015-02-23 20:13:16.092+0000: 2133: warning : virQEMUCapsInit:999 : Failed to get host power management capabilities
> >         
> > As are these two.
> >         
> >         2015-02-23 20:13:16.400+0000: 2133: error : virFirewallApply:936 : out of memory
> > 
> 
> Last time Ian and I debugged a libvirt crashing bug, "out of memory"
> didn't cause libvirtd to exit. It turned out it's some bug in libxl
> event machinery that caused libvirt to exit, but the assertion message
> was not shown anywhere.
> 
> I think we might need to login to that host and run libvirtd in
> foreground to determine what goes wrong.

That's possible I suppose, but it would be nice to arrange not to have
to in the future.

Perhaps we should be forcing higher log levels on libvirtd when
installing, patching /usr/local/etc/libvirt/libvirtd.conf to set
log_level=2 (or even 1) perhaps? (Default is 3 == warnings+error, 2 is
info, 1 is debug)

Jim, what debug level would you recommend for automated test? Unless it
is super verbose I suppose 1=debug is the way to go?

Adding -v to libvirtd command line would be an easier patch, but only
gives the effect of log_level=2 AFAICT. Perhaps that is considered
sufficient?

Ian.