[xen-unstable test] 35257: regressions - FAIL

[xen-unstable test] 35257: regressions - FAIL

[xen.org]

flight 35257 xen-unstable real [real]
http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 test-armhf-armhf-libvirt     12 guest-start.2             fail REGR. vs. 34629

Regressions which are regarded as allowable (not blocking):
 test-amd64-i386-pair        17 guest-migrate/src_host/dst_host fail like 34629

Tests which did not succeed, but are not blocking:
 test-armhf-armhf-xl-sedf-pin 10 migrate-support-check        fail   never pass
 test-armhf-armhf-xl          10 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-sedf     10 migrate-support-check        fail   never pass
 test-amd64-amd64-rumpuserxen-amd64 13 rumpuserxen-demo-xenstorels/xenstorels.repeat fail never pass
 test-armhf-armhf-xl-credit2  10 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-midway   10 migrate-support-check        fail   never pass
 test-amd64-amd64-xl-pvh-amd   9 guest-start                  fail   never pass
 test-armhf-armhf-libvirt     10 migrate-support-check        fail   never pass
 test-amd64-amd64-xl-pvh-intel  9 guest-start                  fail  never pass
 test-amd64-i386-libvirt      10 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-multivcpu 10 migrate-support-check        fail  never pass
 test-amd64-amd64-xl-pcipt-intel  9 guest-start                 fail never pass
 test-amd64-amd64-libvirt     10 migrate-support-check        fail   never pass
 test-amd64-i386-xl-qemut-winxpsp3 14 guest-stop                fail never pass
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1 14 guest-stop         fail never pass
 test-amd64-i386-xl-qemuu-winxpsp3 14 guest-stop                fail never pass
 test-amd64-amd64-xl-qemut-win7-amd64 14 guest-stop             fail never pass
 test-amd64-i386-xl-qemuu-win7-amd64 14 guest-stop              fail never pass
 test-amd64-amd64-xl-winxpsp3 14 guest-stop                   fail   never pass
 test-amd64-amd64-xl-qemuu-win7-amd64 14 guest-stop             fail never pass
 test-amd64-amd64-xl-win7-amd64 14 guest-stop                   fail never pass
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1 14 guest-stop         fail never pass
 test-amd64-i386-xl-qemut-win7-amd64 14 guest-stop              fail never pass
 test-amd64-i386-xl-win7-amd64 14 guest-stop                   fail  never pass
 test-amd64-i386-xl-winxpsp3-vcpus1 14 guest-stop               fail never pass
 test-amd64-i386-xl-winxpsp3  14 guest-stop                   fail   never pass
 test-amd64-amd64-xl-qemut-winxpsp3 14 guest-stop               fail never pass
 test-amd64-amd64-xl-qemuu-winxpsp3 14 guest-stop               fail never pass

version targeted for testing:
 xen                  24b2b8dea180098a3acc809a91cde6c0cc4c8607
baseline version:
 xen                  cb34a7c8d741aa447d79e1b01d71168a4088a4d7

------------------------------------------------------------
People who touched revisions under test:
  Andrew Cooper <andrew.cooper3@citrix.com>
  Dario Faggioli <dario.faggioli@citrix.com>
  David Scott <dave.scott@citrix.com>
  Don Slutz <dslutz@verizon.com>
  Elena Ufimsteva <ufimtseva@gmail.com>
  George Dunlap <george.dunlap@eu.citrix.com>
  Ian Campbell <ian.campbell@citrix.com>
  Ian Jackson <Ian.Jackson@eu.citrix.com>
  Jan Beulich <jbeulich@suse.com>
  Jintack Lim <jintack@cs.columbia.edu>
  Julien Grall <julien.grall@linaro.org>
  Michael Young <m.a.young@durham.ac.uk>
  Olaf Hering <olaf@aepfle.de>
  Stefano Stabellini <stefano.stabellini@eu.citrix.com>
  Wei Liu <wei.liu2@citrix.com>
------------------------------------------------------------

jobs:
 build-amd64                                                  pass    
 build-armhf                                                  pass    
 build-i386                                                   pass    
 build-amd64-libvirt                                          pass    
 build-armhf-libvirt                                          pass    
 build-i386-libvirt                                           pass    
 build-amd64-oldkern                                          pass    
 build-i386-oldkern                                           pass    
 build-amd64-pvops                                            pass    
 build-armhf-pvops                                            pass    
 build-i386-pvops                                             pass    
 build-amd64-rumpuserxen                                      pass    
 build-i386-rumpuserxen                                       pass    
 test-amd64-amd64-xl                                          pass    
 test-armhf-armhf-xl                                          pass    
 test-amd64-i386-xl                                           pass    
 test-amd64-amd64-xl-pvh-amd                                  fail    
 test-amd64-i386-rhel6hvm-amd                                 pass    
 test-amd64-i386-qemut-rhel6hvm-amd                           pass    
 test-amd64-i386-qemuu-rhel6hvm-amd                           pass    
 test-amd64-amd64-xl-qemut-debianhvm-amd64                    pass    
 test-amd64-i386-xl-qemut-debianhvm-amd64                     pass    
 test-amd64-amd64-xl-qemuu-debianhvm-amd64                    pass    
 test-amd64-i386-xl-qemuu-debianhvm-amd64                     pass    
 test-amd64-i386-freebsd10-amd64                              pass    
 test-amd64-amd64-xl-qemuu-ovmf-amd64                         pass    
 test-amd64-i386-xl-qemuu-ovmf-amd64                          pass    
 test-amd64-amd64-rumpuserxen-amd64                           fail    
 test-amd64-amd64-xl-qemut-win7-amd64                         fail    
 test-amd64-i386-xl-qemut-win7-amd64                          fail    
 test-amd64-amd64-xl-qemuu-win7-amd64                         fail    
 test-amd64-i386-xl-qemuu-win7-amd64                          fail    
 test-amd64-amd64-xl-win7-amd64                               fail    
 test-amd64-i386-xl-win7-amd64                                fail    
 test-amd64-amd64-xl-credit2                                  pass    
 test-armhf-armhf-xl-credit2                                  pass    
 test-amd64-i386-freebsd10-i386                               pass    
 test-amd64-i386-rumpuserxen-i386                             pass    
 test-amd64-amd64-xl-pcipt-intel                              fail    
 test-amd64-amd64-xl-pvh-intel                                fail    
 test-amd64-i386-rhel6hvm-intel                               pass    
 test-amd64-i386-qemut-rhel6hvm-intel                         pass    
 test-amd64-i386-qemuu-rhel6hvm-intel                         pass    
 test-amd64-amd64-libvirt                                     pass    
 test-armhf-armhf-libvirt                                     fail    
 test-amd64-i386-libvirt                                      pass    
 test-armhf-armhf-xl-midway                                   pass    
 test-amd64-amd64-xl-multivcpu                                pass    
 test-armhf-armhf-xl-multivcpu                                pass    
 test-amd64-amd64-pair                                        pass    
 test-amd64-i386-pair                                         fail    
 test-amd64-amd64-xl-sedf-pin                                 pass    
 test-armhf-armhf-xl-sedf-pin                                 pass    
 test-amd64-amd64-xl-sedf                                     pass    
 test-armhf-armhf-xl-sedf                                     pass    
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1                     fail    
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1                     fail    
 test-amd64-i386-xl-winxpsp3-vcpus1                           fail    
 test-amd64-amd64-xl-qemut-winxpsp3                           fail    
 test-amd64-i386-xl-qemut-winxpsp3                            fail    
 test-amd64-amd64-xl-qemuu-winxpsp3                           fail    
 test-amd64-i386-xl-qemuu-winxpsp3                            fail    
 test-amd64-amd64-xl-winxpsp3                                 fail    
 test-amd64-i386-xl-winxpsp3                                  fail    


------------------------------------------------------------
sg-report-flight on osstest.cam.xci-test.com
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images

Logs, config files, etc. are available at
    http://www.chiark.greenend.org.uk/~xensrcts/logs

Test harness code can be found at
    http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary


Not pushing.

(No revision log; it would be 884 lines long.)

Re: [xen-unstable test] 35257: regressions - FAIL

[Ian Campbell]

On Thu, 2015-02-26 at 20:14 +0000, xen.org wrote:
> flight 35257 xen-unstable real [real]
> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/
> 
> Regressions :-(
> 
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  test-armhf-armhf-libvirt     12 guest-start.2             fail REGR. vs. 34629

logs:
http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/info.html

http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/12.ts-guest-start.log
        2015-02-23 20:21:48 Z executing ssh ... root@10.80.229.106 virsh domxml-from-native xen-xl /etc/xen/debian.guest.osstest.cfg > /etc/xen/debian.guest.osstest.cfg.xml
        error: failed to connect to the hypervisor
        error: no valid connection
        error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused

http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/marilith-n4-output-ps_wwwaxf_-eo_pid%2Ctty%2Cstat%2Ctime%2Cnice%2Cpsr%2Cpcpu%2Cpmem%2Cnwchan%2Cwchan%2325%2Cargs
appears to show no libvirtd process.

http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/marilith-n4---var-log-libvirt-libvirtd.log
says:
        2015-02-23 20:13:15.556+0000: 2133: info : libvirt version: 1.2.13
        2015-02-23 20:13:15.556+0000: 2133: error : dnsmasqCapsRefreshInternal:726 : Cannot check dnsmasq binary dnsmasq: No such file or directory
        2015-02-23 20:13:15.845+0000: 2133: error : virFirewallValidateBackend:193 : direct firewall backend requested, but /sbin/ebtables is not available: No such file or directory
        
I think these are just spurious.

        2015-02-23 20:13:15.845+0000: 2133: error : virFirewallApply:936 : out of memory
        

        2015-02-23 20:13:16.092+0000: 2133: error : virExec:491 : Cannot find 'pm-is-supported' in path: No such file or directory
        2015-02-23 20:13:16.092+0000: 2133: warning : virQEMUCapsInit:999 : Failed to get host power management capabilities
        
As are these two.
        
        2015-02-23 20:13:16.400+0000: 2133: error : virFirewallApply:936 : out of memory

Has these OOM messages resulted in libvirtd exiting? I don't see any
evidence of a crash elsewhere in the logs (i.e. no "process segfaulted"
in dmesg, no OOM killing going on etc).

We don't seem to collect dom0 freemem info, but that most likely
wouldn't help given the libvirtd process has exited.

Any ideas where to look next?

Ian.

Re: [xen-unstable test] 35257: regressions - FAIL

[Wei Liu]

On Fri, Feb 27, 2015 at 09:42:29AM +0000, Ian Campbell wrote:
> On Thu, 2015-02-26 at 20:14 +0000, xen.org wrote:
> > flight 35257 xen-unstable real [real]
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/
> > 
> > Regressions :-(
> > 
> > Tests which did not succeed and are blocking,
> > including tests which could not be run:
> >  test-armhf-armhf-libvirt     12 guest-start.2             fail REGR. vs. 34629
> 
> logs:
> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/info.html
> 
> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/12.ts-guest-start.log
>         2015-02-23 20:21:48 Z executing ssh ... root@10.80.229.106 virsh domxml-from-native xen-xl /etc/xen/debian.guest.osstest.cfg > /etc/xen/debian.guest.osstest.cfg.xml
>         error: failed to connect to the hypervisor
>         error: no valid connection
>         error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused
> 
> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/marilith-n4-output-ps_wwwaxf_-eo_pid%2Ctty%2Cstat%2Ctime%2Cnice%2Cpsr%2Cpcpu%2Cpmem%2Cnwchan%2Cwchan%2325%2Cargs
> appears to show no libvirtd process.
> 
> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/marilith-n4---var-log-libvirt-libvirtd.log
> says:
>         2015-02-23 20:13:15.556+0000: 2133: info : libvirt version: 1.2.13
>         2015-02-23 20:13:15.556+0000: 2133: error : dnsmasqCapsRefreshInternal:726 : Cannot check dnsmasq binary dnsmasq: No such file or directory
>         2015-02-23 20:13:15.845+0000: 2133: error : virFirewallValidateBackend:193 : direct firewall backend requested, but /sbin/ebtables is not available: No such file or directory
>         
> I think these are just spurious.
> 
>         2015-02-23 20:13:15.845+0000: 2133: error : virFirewallApply:936 : out of memory
>         
> 
>         2015-02-23 20:13:16.092+0000: 2133: error : virExec:491 : Cannot find 'pm-is-supported' in path: No such file or directory
>         2015-02-23 20:13:16.092+0000: 2133: warning : virQEMUCapsInit:999 : Failed to get host power management capabilities
>         
> As are these two.
>         
>         2015-02-23 20:13:16.400+0000: 2133: error : virFirewallApply:936 : out of memory
> 

Last time Ian and I debugged a libvirt crashing bug, "out of memory"
didn't cause libvirtd to exit. It turned out it's some bug in libxl
event machinery that caused libvirt to exit, but the assertion message
was not shown anywhere.

I think we might need to login to that host and run libvirtd in
foreground to determine what goes wrong.

Wei.


> Has these OOM messages resulted in libvirtd exiting? I don't see any
> evidence of a crash elsewhere in the logs (i.e. no "process segfaulted"
> in dmesg, no OOM killing going on etc).
> 
> We don't seem to collect dom0 freemem info, but that most likely
> wouldn't help given the libvirtd process has exited.
> 
> Any ideas where to look next?
> 
> Ian.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [xen-unstable test] 35257: regressions - FAIL

[Ian Campbell]

On Fri, 2015-02-27 at 10:48 +0000, Wei Liu wrote:
> On Fri, Feb 27, 2015 at 09:42:29AM +0000, Ian Campbell wrote:
> > On Thu, 2015-02-26 at 20:14 +0000, xen.org wrote:
> > > flight 35257 xen-unstable real [real]
> > > http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/
> > > 
> > > Regressions :-(
> > > 
> > > Tests which did not succeed and are blocking,
> > > including tests which could not be run:
> > >  test-armhf-armhf-libvirt     12 guest-start.2             fail REGR. vs. 34629
> > 
> > logs:
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/info.html
> > 
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/12.ts-guest-start.log
> >         2015-02-23 20:21:48 Z executing ssh ... root@10.80.229.106 virsh domxml-from-native xen-xl /etc/xen/debian.guest.osstest.cfg > /etc/xen/debian.guest.osstest.cfg.xml
> >         error: failed to connect to the hypervisor
> >         error: no valid connection
> >         error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused
> > 
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/marilith-n4-output-ps_wwwaxf_-eo_pid%2Ctty%2Cstat%2Ctime%2Cnice%2Cpsr%2Cpcpu%2Cpmem%2Cnwchan%2Cwchan%2325%2Cargs
> > appears to show no libvirtd process.
> > 
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/marilith-n4---var-log-libvirt-libvirtd.log
> > says:
> >         2015-02-23 20:13:15.556+0000: 2133: info : libvirt version: 1.2.13
> >         2015-02-23 20:13:15.556+0000: 2133: error : dnsmasqCapsRefreshInternal:726 : Cannot check dnsmasq binary dnsmasq: No such file or directory
> >         2015-02-23 20:13:15.845+0000: 2133: error : virFirewallValidateBackend:193 : direct firewall backend requested, but /sbin/ebtables is not available: No such file or directory
> >         
> > I think these are just spurious.
> > 
> >         2015-02-23 20:13:15.845+0000: 2133: error : virFirewallApply:936 : out of memory
> >         
> > 
> >         2015-02-23 20:13:16.092+0000: 2133: error : virExec:491 : Cannot find 'pm-is-supported' in path: No such file or directory
> >         2015-02-23 20:13:16.092+0000: 2133: warning : virQEMUCapsInit:999 : Failed to get host power management capabilities
> >         
> > As are these two.
> >         
> >         2015-02-23 20:13:16.400+0000: 2133: error : virFirewallApply:936 : out of memory
> > 
> 
> Last time Ian and I debugged a libvirt crashing bug, "out of memory"
> didn't cause libvirtd to exit. It turned out it's some bug in libxl
> event machinery that caused libvirt to exit, but the assertion message
> was not shown anywhere.
> 
> I think we might need to login to that host and run libvirtd in
> foreground to determine what goes wrong.

That's possible I suppose, but it would be nice to arrange not to have
to in the future.

Perhaps we should be forcing higher log levels on libvirtd when
installing, patching /usr/local/etc/libvirt/libvirtd.conf to set
log_level=2 (or even 1) perhaps? (Default is 3 == warnings+error, 2 is
info, 1 is debug)

Jim, what debug level would you recommend for automated test? Unless it
is super verbose I suppose 1=debug is the way to go?

Adding -v to libvirtd command line would be an easier patch, but only
gives the effect of log_level=2 AFAICT. Perhaps that is considered
sufficient?

Ian.

Re: [xen-unstable test] 35257: regressions - FAIL

[Jim Fehlig]

Ian Campbell wrote:
> On Fri, 2015-02-27 at 10:48 +0000, Wei Liu wrote:
>   
>> On Fri, Feb 27, 2015 at 09:42:29AM +0000, Ian Campbell wrote:
>>     
>>> On Thu, 2015-02-26 at 20:14 +0000, xen.org wrote:
>>>       
>>>> flight 35257 xen-unstable real [real]
>>>> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/
>>>>
>>>> Regressions :-(
>>>>
>>>> Tests which did not succeed and are blocking,
>>>> including tests which could not be run:
>>>>  test-armhf-armhf-libvirt     12 guest-start.2             fail REGR. vs. 34629
>>>>         
>>> logs:
>>> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/info.html
>>>
>>> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/12.ts-guest-start.log
>>>         2015-02-23 20:21:48 Z executing ssh ... root@10.80.229.106 virsh domxml-from-native xen-xl /etc/xen/debian.guest.osstest.cfg > /etc/xen/debian.guest.osstest.cfg.xml
>>>         error: failed to connect to the hypervisor
>>>         error: no valid connection
>>>         error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused
>>>
>>> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/marilith-n4-output-ps_wwwaxf_-eo_pid%2Ctty%2Cstat%2Ctime%2Cnice%2Cpsr%2Cpcpu%2Cpmem%2Cnwchan%2Cwchan%2325%2Cargs
>>> appears to show no libvirtd process.
>>>
>>> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/marilith-n4---var-log-libvirt-libvirtd.log
>>> says:
>>>         2015-02-23 20:13:15.556+0000: 2133: info : libvirt version: 1.2.13
>>>         2015-02-23 20:13:15.556+0000: 2133: error : dnsmasqCapsRefreshInternal:726 : Cannot check dnsmasq binary dnsmasq: No such file or directory
>>>         2015-02-23 20:13:15.845+0000: 2133: error : virFirewallValidateBackend:193 : direct firewall backend requested, but /sbin/ebtables is not available: No such file or directory
>>>         
>>> I think these are just spurious.
>>>
>>>         2015-02-23 20:13:15.845+0000: 2133: error : virFirewallApply:936 : out of memory
>>>         
>>>
>>>         2015-02-23 20:13:16.092+0000: 2133: error : virExec:491 : Cannot find 'pm-is-supported' in path: No such file or directory
>>>         2015-02-23 20:13:16.092+0000: 2133: warning : virQEMUCapsInit:999 : Failed to get host power management capabilities
>>>         
>>> As are these two.
>>>         
>>>         2015-02-23 20:13:16.400+0000: 2133: error : virFirewallApply:936 : out of memory
>>>
>>>       
>> Last time Ian and I debugged a libvirt crashing bug, "out of memory"
>> didn't cause libvirtd to exit. It turned out it's some bug in libxl
>> event machinery that caused libvirt to exit, but the assertion message
>> was not shown anywhere.
>>
>> I think we might need to login to that host and run libvirtd in
>> foreground to determine what goes wrong.
>>     
>
> That's possible I suppose, but it would be nice to arrange not to have
> to in the future.
>
> Perhaps we should be forcing higher log levels on libvirtd when
> installing, patching /usr/local/etc/libvirt/libvirtd.conf to set
> log_level=2 (or even 1) perhaps? (Default is 3 == warnings+error, 2 is
> info, 1 is debug)
>
> Jim, what debug level would you recommend for automated test? Unless it
>   
> is super verbose I suppose 1=debug is the way to go?

I think we need DEBUG log level, although it is rather verbose.  If that
becomes a problem, we could experiment with a minimally useful
log_filters setting, e.g.

log_filters="1:daemon 1:libxl"

> Adding -v to libvirtd command line would be an easier patch, but only
> gives the effect of log_level=2 AFAICT. Perhaps that is considered
> sufficient?
>   

In my experience, if ERROR is insufficient, INFO and WARNING don't
help.  DEBUG is needed.

Regards,
Jim

[PATCH] ts-xen-install: Enable debug level logging in libvirt

[Ian Campbell]

This should make diagnosing libvirt issues a bit easier.

Refactor the existing loop to edit the xencommons kvp style config
file editing code so it can be reused for libvirtd.conf.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Jim Fehlig <jfehlig@suse.com>
---
 ts-xen-install | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/ts-xen-install b/ts-xen-install
index 7cfe344..a102b0d 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -114,19 +114,26 @@ sub adjustconfig () {
     }
     die unless defined $trace_config_file;
 
-    target_editfile_root($ho, $trace_config_file, sub {
+    my $kvp_replace = sub($$) {
+	my ($key,$value) = @_;
         my $prnow;
         $prnow= sub {
-            print EO "XENCONSOLED_TRACE=guest\n" or die $!;
+            print EO "$key=$value\n" or die $!;
             $prnow= sub { };
         };
         while (<EI>) {
-            print EO or die $! unless m/^XENCONSOLED_TRACE/;
-            $prnow->() if m/^#XENCONSOLED_TRACE/;
+            print EO or die $! unless m/^$key/;
+            $prnow->() if m/^#$key/;
         }
         print EO "\n" or die $!;
         $prnow->();
-    });
+    };
+
+    target_editfile_root($ho, $trace_config_file,
+			 sub { $kvp_replace->("XENCONSOLED_TRACE", "guest") });
+
+    target_editfile_root($ho, '/etc/libvirt/libvirtd.conf',
+			 sub { $kvp_replace->("log_level", "1") });
 
     target_cmd_root($ho, 'mkdir -p /var/log/xen/console');
 
-- 
2.1.4

Re: [PATCH] ts-xen-install: Enable debug level logging in libvirt

[Ian Jackson]

Ian Campbell writes ("[PATCH] ts-xen-install: Enable debug level logging in libvirt"):
> This should make diagnosing libvirt issues a bit easier.

If you're generalising this function,

> +            print EO or die $! unless m/^$key/;
                                                ^ \b ?

?

Ian.

[PATCH v2] ts-xen-install: Enable debug level logging in libvirt

[Ian Campbell]

This should make diagnosing libvirt issues a bit easier.

Refactor the existing loop to edit the xencommons kvp style config
file editing code so it can be reused for libvirtd.conf.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Jim Fehlig <jfehlig@suse.com>
---
v2: Match key\b
---
 ts-xen-install | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/ts-xen-install b/ts-xen-install
index 7cfe344..044bcc8 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -114,19 +114,26 @@ sub adjustconfig () {
     }
     die unless defined $trace_config_file;
 
-    target_editfile_root($ho, $trace_config_file, sub {
+    my $kvp_replace = sub($$) {
+	my ($key,$value) = @_;
         my $prnow;
         $prnow= sub {
-            print EO "XENCONSOLED_TRACE=guest\n" or die $!;
+            print EO "$key=$value\n" or die $!;
             $prnow= sub { };
         };
         while (<EI>) {
-            print EO or die $! unless m/^XENCONSOLED_TRACE/;
-            $prnow->() if m/^#XENCONSOLED_TRACE/;
+            print EO or die $! unless m/^$key\b/;
+            $prnow->() if m/^#$key/;
         }
         print EO "\n" or die $!;
         $prnow->();
-    });
+    };
+
+    target_editfile_root($ho, $trace_config_file,
+			 sub { $kvp_replace->("XENCONSOLED_TRACE", "guest") });
+
+    target_editfile_root($ho, '/etc/libvirt/libvirtd.conf',
+			 sub { $kvp_replace->("log_level", "1") });
 
     target_cmd_root($ho, 'mkdir -p /var/log/xen/console');
 
-- 
2.1.4

Re: [PATCH v2] ts-xen-install: Enable debug level logging in libvirt

[Ian Jackson]

Ian Campbell writes ("[PATCH v2] ts-xen-install: Enable debug level logging in libvirt"):
> This should make diagnosing libvirt issues a bit easier.
> 
> Refactor the existing loop to edit the xencommons kvp style config
> file editing code so it can be reused for libvirtd.conf.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH v2] ts-xen-install: Enable debug level logging in libvirt

[Ian Campbell]

On Mon, 2015-03-02 at 11:37 +0000, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH v2] ts-xen-install: Enable debug level logging in libvirt"):
> > This should make diagnosing libvirt issues a bit easier.
> > 
> > Refactor the existing loop to edit the xencommons kvp style config
> > file editing code so it can be reused for libvirtd.conf.
> 
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Thanks. Pushed to pretest.

Ian.

[PATCH v3] ts-xen-install: Enable debug level logging in libvirt

[Ian Campbell]

This should make diagnosing libvirt issues a bit easier.

Refactor the existing loop to edit the xencommons kvp style config
file editing code so it can be reused for libvirtd.conf.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Jim Fehlig <jfehlig@suse.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v3: Only apply if toolstack==libvirt, else we break non-libvirt
    tests. Now tested with test-amd64-amd64-xl as well as -libvirt.
v2: Match key\b
---
 ts-xen-install | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/ts-xen-install b/ts-xen-install
index 7cfe344..5282f0a 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -114,19 +114,27 @@ sub adjustconfig () {
     }
     die unless defined $trace_config_file;
 
-    target_editfile_root($ho, $trace_config_file, sub {
+    my $kvp_replace = sub($$) {
+	my ($key,$value) = @_;
         my $prnow;
         $prnow= sub {
-            print EO "XENCONSOLED_TRACE=guest\n" or die $!;
+            print EO "$key=$value\n" or die $!;
             $prnow= sub { };
         };
         while (<EI>) {
-            print EO or die $! unless m/^XENCONSOLED_TRACE/;
-            $prnow->() if m/^#XENCONSOLED_TRACE/;
+            print EO or die $! unless m/^$key\b/;
+            $prnow->() if m/^#$key/;
         }
         print EO "\n" or die $!;
         $prnow->();
-    });
+    };
+
+    target_editfile_root($ho, $trace_config_file,
+			 sub { $kvp_replace->("XENCONSOLED_TRACE", "guest") });
+
+    target_editfile_root($ho, '/etc/libvirt/libvirtd.conf',
+			 sub { $kvp_replace->("log_level", "1") })
+	if toolstack($ho)->{Name} eq "libvirt";
 
     target_cmd_root($ho, 'mkdir -p /var/log/xen/console');
 
-- 
2.1.4

Re: [PATCH v3] ts-xen-install: Enable debug level logging in libvirt

[Ian Campbell]

On Tue, 2015-03-03 at 10:22 +0000, Ian Campbell wrote:
> This should make diagnosing libvirt issues a bit easier.
> 
> Refactor the existing loop to edit the xencommons kvp style config
> file editing code so it can be reused for libvirtd.conf.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> Cc: Jim Fehlig <jfehlig@suse.com>
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Pushed to pretest as requested on IRC.

Re: [xen-unstable test] 35257: regressions - FAIL

[Ian Campbell]

On Fri, 2015-02-27 at 13:47 -0700, Jim Fehlig wrote:
> I think we need DEBUG log level, although it is rather verbose.[...]
> In my experience, if ERROR is insufficient, INFO and WARNING don't
> help.  DEBUG is needed.

Thanks, I've just sent out an osstest patch to enable log_level=1.

Ian, I think we should apply that to pretest ASAP to help diagnose the
libvirt issue we are seeing at the moment.

Ian.

Re: [xen-unstable test] 35257: regressions - FAIL

[Ian Jackson]

Jim Fehlig writes ("Re: [Xen-devel] [xen-unstable test] 35257: regressions - FAIL"):
...
> In my experience, if ERROR is insufficient, INFO and WARNING don't
> help.  DEBUG is needed.

Where does virtd's stderr go ?  That is where any assertion failure
message would be written.

If we were more clever, osstest would collect corefiles.

Ian.

Re: [xen-unstable test] 35257: regressions - FAIL

[Ian Campbell]

On Mon, 2015-03-02 at 11:14 +0000, Ian Jackson wrote:
> Jim Fehlig writes ("Re: [Xen-devel] [xen-unstable test] 35257: regressions - FAIL"):
> ...
> > In my experience, if ERROR is insufficient, INFO and WARNING don't
> > help.  DEBUG is needed.
> 
> Where does virtd's stderr go ?  That is where any assertion failure
> message would be written.

/dev/null from the looks of the code:
http://libvirt.org/git/?p=libvirt.git;a=blob;f=daemon/libvirtd.c;h=2366d63cd3c70f75df5af82d2c241df163293eab;hb=HEAD#l174

Ian

Re: [xen-unstable test] 35257: regressions - FAIL

[Jim Fehlig]

Ian Campbell wrote:
> On Mon, 2015-03-02 at 11:14 +0000, Ian Jackson wrote:
>   
>> Jim Fehlig writes ("Re: [Xen-devel] [xen-unstable test] 35257: regressions - FAIL"):
>> ...
>>     
>>> In my experience, if ERROR is insufficient, INFO and WARNING don't
>>> help.  DEBUG is needed.
>>>       
>> Where does virtd's stderr go ?  That is where any assertion failure
>> message would be written.
>>     
>
> /dev/null from the looks of the code:
> http://libvirt.org/git/?p=libvirt.git;a=blob;f=daemon/libvirtd.c;h=2366d63cd3c70f75df5af82d2c241df163293eab;hb=HEAD#l174
>   

When daemonized, yes :-/.  You would need to run libvirtd without the
'-d' option to see the assert.  Or collect a core as Ian suggested.

Regards,
Jim

Re: [xen-unstable test] 35257: regressions - FAIL

[Ian Campbell]

On Mon, 2015-03-02 at 10:59 -0700, Jim Fehlig wrote:
> Ian Campbell wrote:
> > On Mon, 2015-03-02 at 11:14 +0000, Ian Jackson wrote:
> >   
> >> Jim Fehlig writes ("Re: [Xen-devel] [xen-unstable test] 35257: regressions - FAIL"):
> >> ...
> >>     
> >>> In my experience, if ERROR is insufficient, INFO and WARNING don't
> >>> help.  DEBUG is needed.
> >>>       
> >> Where does virtd's stderr go ?  That is where any assertion failure
> >> message would be written.
> >>     
> >
> > /dev/null from the looks of the code:
> > http://libvirt.org/git/?p=libvirt.git;a=blob;f=daemon/libvirtd.c;h=2366d63cd3c70f75df5af82d2c241df163293eab;hb=HEAD#l174
> >   
> 
> When daemonized, yes :-/.  You would need to run libvirtd without the
> '-d' option to see the assert.

Which would run in the foreground, I think? So we would need to handle
daemonisation some other way? Probably best to avoid that I think.

>   Or collect a core as Ian suggested.

This would seem to be a generically useful thing to do, as well as
helping here.

I think it could be achieved by ts-host-install editing /etc/sysctl.conf
to set sys.kernel.core_pattern to something (perhaps a new dir
under /var/). http://man7.org/linux/man-pages/man5/core.5.html describes
the substitutions and says it is there since 2.4.21 and 2.6.

Some sort of ulimit frobbing might also be needed I suppose.

Then ts-logs-capture would pick up anything which appeared in the
directory which is in core_pattern. Or perhaps a new test step
ts-core-check could be added which fails the test if anything is found?

Ian, if you think this makes sense I could add to my TODO.

Ian.

Re: [xen-unstable test] 35257: regressions - FAIL

[Jim Fehlig]

Ian Campbell wrote:
> On Thu, 2015-02-26 at 20:14 +0000, xen.org wrote:
>   
>> flight 35257 xen-unstable real [real]
>> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/
>>
>> Regressions :-(
>>
>> Tests which did not succeed and are blocking,
>> including tests which could not be run:
>>  test-armhf-armhf-libvirt     12 guest-start.2             fail REGR. vs. 34629
>>     
>
> logs:
> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/info.html
>
> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/12.ts-guest-start.log
>         2015-02-23 20:21:48 Z executing ssh ... root@10.80.229.106 virsh domxml-from-native xen-xl /etc/xen/debian.guest.osstest.cfg > /etc/xen/debian.guest.osstest.cfg.xml
>         error: failed to connect to the hypervisor
>         error: no valid connection
>         error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused
>
> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/marilith-n4-output-ps_wwwaxf_-eo_pid%2Ctty%2Cstat%2Ctime%2Cnice%2Cpsr%2Cpcpu%2Cpmem%2Cnwchan%2Cwchan%2325%2Cargs
> appears to show no libvirtd process.
>
> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/test-armhf-armhf-libvirt/marilith-n4---var-log-libvirt-libvirtd.log
> says:
>         2015-02-23 20:13:15.556+0000: 2133: info : libvirt version: 1.2.13
>         2015-02-23 20:13:15.556+0000: 2133: error : dnsmasqCapsRefreshInternal:726 : Cannot check dnsmasq binary dnsmasq: No such file or directory
>   
> 2015-02-23 20:13:15.845+0000: 2133: error :
> virFirewallValidateBackend:193 : direct firewall backend requested,
> but /sbin/ebtables is not available: No such file or directory

Odd, since ebtables was found when building

checking for ebtables... /sbin/ebtables

But AFAICT, that wont prevent libvirtd from starting.

>         
> I think these are just spurious.
>
>         2015-02-23 20:13:15.845+0000: 2133: error : virFirewallApply:936 : out of memory
>         
>
>         2015-02-23 20:13:16.092+0000: 2133: error : virExec:491 : Cannot find 'pm-is-supported' in path: No such file or directory
>         2015-02-23 20:13:16.092+0000: 2133: warning : virQEMUCapsInit:999 : Failed to get host power management capabilities
>         
> As are these two.
>         
>         2015-02-23 20:13:16.400+0000: 2133: error : virFirewallApply:936 : out of memory
>
> Has these OOM messages resulted in libvirtd exiting?

No, I don't think so.  The related code is

int
virFirewallApply(virFirewallPtr firewall)
{
    size_t i, j;
    int ret = -1;

    virMutexLock(&ruleLock);

    if (!firewall || firewall->err == ENOMEM) {
        virReportOOMError();
        goto cleanup;
    ...
}

I suspect 'firewall' is null, so OOM error is reported and the function
returns -1.  But I also don't see this preventing libvirtd from
starting.  I've cc'd the libvirt list for verification that these errors
won't prevent libvirtd from starting.

>  I don't see any
> evidence of a crash elsewhere in the logs (i.e. no "process segfaulted"
> in dmesg, no OOM killing going on etc).
>
> We don't seem to collect dom0 freemem info, but that most likely
> wouldn't help given the libvirtd process has exited.
>
>   
> Any ideas where to look next?

Can you access the test environment and try starting libvirtd in the
foreground?  Or enable debug log level in /etc/libvirt/libvirtd.conf?

Regards,
Jim

Re: [libvirt] [Xen-devel] [xen-unstable test] 35257: regressions - FAIL

[Ian Campbell]

On Fri, 2015-02-27 at 11:51 -0700, Jim Fehlig wrote:
> > 2015-02-23 20:13:15.845+0000: 2133: error :
> > virFirewallValidateBackend:193 : direct firewall backend requested,
> > but /sbin/ebtables is not available: No such file or directory
> 
> Odd, since ebtables was found when building
> 
> checking for ebtables... /sbin/ebtables
> 
> But AFAICT, that wont prevent libvirtd from starting.

The build host and the runtime host will likely be different (or at
least reinstalled).

The base set of packages should be the same, but the build one will
install a bunch of libfoo-dev while the runtime host will only get
libfoo. Perhaps some libfoo-dev is pulling in ebtables somehow while
just libfoo is not. I'll have a look next week. I think its probably
non-critical to the error here.

> > I think these are just spurious.
> >
> >         2015-02-23 20:13:15.845+0000: 2133: error : virFirewallApply:936 : out of memory
> >         
> >
> >         2015-02-23 20:13:16.092+0000: 2133: error : virExec:491 : Cannot find 'pm-is-supported' in path: No such file or directory
> >         2015-02-23 20:13:16.092+0000: 2133: warning : virQEMUCapsInit:999 : Failed to get host power management capabilities
> >         
> > As are these two.
> >         
> >         2015-02-23 20:13:16.400+0000: 2133: error : virFirewallApply:936 : out of memory
> >
> > Has these OOM messages resulted in libvirtd exiting?
> 
> No, I don't think so.  The related code is
> 
> int
> virFirewallApply(virFirewallPtr firewall)
> {
>     size_t i, j;
>     int ret = -1;
> 
>     virMutexLock(&ruleLock);
> 
>     if (!firewall || firewall->err == ENOMEM) {
>         virReportOOMError();
>         goto cleanup;
>     ...
> }
> 
> I suspect 'firewall' is null, so OOM error is reported and the function
> returns -1.  But I also don't see this preventing libvirtd from
> starting.  I've cc'd the libvirt list for verification that these errors
> won't prevent libvirtd from starting.

I'm pretty sure libvirtd did successfully start, since we have
successfully done a guest start and stop.

The failing step is a second guest start, so it seems like libvirtd has
either crashed or exited.

I suppose these messages are from start of day and therefore
red-herrings wrt the reason libvirtd went away.

> >  I don't see any
> > evidence of a crash elsewhere in the logs (i.e. no "process segfaulted"
> > in dmesg, no OOM killing going on etc).
> >
> > We don't seem to collect dom0 freemem info, but that most likely
> > wouldn't help given the libvirtd process has exited.
> >
> >   
> > Any ideas where to look next?
> 
> Can you access the test environment and try starting libvirtd in the
> foreground?  Or enable debug log level in /etc/libvirt/libvirtd.conf?

The test env will have been recycled, I could try and replicate it
manually, but I think to start with I should arrange for the test env to
have more logging enabled, in the hopes that if it happens again we get
more information. I had some question around this in my reply Wei in
this thread at <1425042785.14641.188.camel@citrix.com>.

Cheers,
Ian.

Re: [xen-unstable test] 35257: regressions - FAIL

[Ian Campbell]

On Thu, 2015-02-26 at 20:14 +0000, xen.org wrote:
> flight 35257 xen-unstable real [real]
> http://www.chiark.greenend.org.uk/~xensrcts/logs/35257/
> 
> Regressions :-(
> 
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  test-armhf-armhf-libvirt     12 guest-start.2             fail REGR. vs. 34629

libvirt is unreliable right now, Ian already added to allow.all:
        @@-libvirt@@               guest-start

Ian is going to push a change to the allow file to make at least libvirt
guest-state.* and guest-destroy allowable fails but in the meantime:
[...]

> version targeted for testing:
>  xen                  24b2b8dea180098a3acc809a91cde6c0cc4c8607

I have force pushed this revision.

Ian.

osstest@osstest:~/branches/for-xen-unstable.git$ OSSTEST_CONFIG=production-config ./ap-push 24b2b8dea180098a3acc809a91cde6c0cc4c8607
+ branch=24b2b8dea180098a3acc809a91cde6c0cc4c8607
+ revision=
+ . cri-lock-repos
++ . cri-common
+++ . cri-getconfig
+++ umask 002
+++ getconfig Repos
+++ perl -e '
                use Osstest;
                readglobalconfig();
                print $c{"Repos"} or die $!;
        '
++ repos=/export/home/osstest/repos
++ repos_lock=/export/home/osstest/repos/lock
++ '[' x '!=' x/export/home/osstest/repos/lock ']'
++ OSSTEST_REPOS_LOCK_LOCKED=/export/home/osstest/repos/lock
++ exec with-lock-ex -w /export/home/osstest/repos/lock ./ap-push 24b2b8dea180098a3acc809a91cde6c0cc4c8607
+ branch=24b2b8dea180098a3acc809a91cde6c0cc4c8607
+ revision=
+ . cri-lock-repos
++ . cri-common
+++ . cri-getconfig
+++ umask 002
+++ getconfig Repos
+++ perl -e '
                use Osstest;
                readglobalconfig();
                print $c{"Repos"} or die $!;
        '
++ repos=/export/home/osstest/repos
++ repos_lock=/export/home/osstest/repos/lock
++ '[' x/export/home/osstest/repos/lock '!=' x/export/home/osstest/repos/lock ']'
+ . cri-common
++ . cri-getconfig
++ umask 002
+ select_xenbranch
+ case "$branch" in
+ '[' x = xlinux ']'
+ linuxbranch=
+ '[' x = x ']'
+ qemuubranch=qemu-upstream-
+ : tested/2.6.39.x
+ . ap-common
++ : osstest@xenbits.xensource.com
++ : git://xenbits.xen.org/xen.git
++ : osstest@xenbits.xensource.com:/home/xen/git/xen.git
++ : git://xenbits.xen.org/staging/qemu-.git
++ : git://git.kernel.org
++ : git://git.kernel.org/pub/scm/linux/kernel/git
++ : git
++ : git://libvirt.org/libvirt.git
++ : osstest@xenbits.xensource.com:/home/xen/git/libvirt.git
++ : git://xenbits.xen.org/libvirt.git
++ : https://github.com/rumpkernel/rumprun-xen
++ : git
++ : git://xenbits.xen.org/rumpuser-xen.git
++ : osstest@xenbits.xensource.com:/home/xen/git/rumpuser-xen.git
+++ besteffort_repo https://github.com/rumpkernel/rumpkernel-netbsd-src
+++ local repo=https://github.com/rumpkernel/rumpkernel-netbsd-src
+++ cached_repo https://github.com/rumpkernel/rumpkernel-netbsd-src '[fetch=try]'
+++ local repo=https://github.com/rumpkernel/rumpkernel-netbsd-src
+++ local 'options=[fetch=try]'
++++ getconfig GitCacheProxy
++++ perl -e '
                use Osstest;
                readglobalconfig();
                print $c{"GitCacheProxy"} or die $!;
        '
+++ local cache=git://drall.uk.xensource.com:9419/
+++ '[' xgit://drall.uk.xensource.com:9419/ '!=' x ']'
+++ echo 'git://drall.uk.xensource.com:9419/https://github.com/rumpkernel/rumpkernel-netbsd-src%20[fetch=try]'
++ : 'git://drall.uk.xensource.com:9419/https://github.com/rumpkernel/rumpkernel-netbsd-src%20[fetch=try]'
++ : git
++ : git://git.seabios.org/seabios.git
++ : osstest@xenbits.xensource.com:/home/xen/git/osstest/seabios.git
++ : git://xenbits.xen.org/osstest/seabios.git
++ : https://github.com/tianocore/edk2.git
++ : osstest@xenbits.xensource.com:/home/xen/git/osstest/ovmf.git
++ : git://xenbits.xen.org/osstest/ovmf.git
++ : git://xenbits.xen.org/osstest/linux-firmware.git
++ : osstest@xenbits.xensource.com:/home/osstest/ext/linux-firmware.git
++ : git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
++ : osstest@xenbits.xensource.com:/home/xen/git/linux-pvops.git
++ : git://xenbits.xen.org/linux-pvops.git
++ : tested/linux-3.14
++ : tested/linux-arm-xen
++ '[' x = x ']'
++ : git://xenbits.xen.org/linux-pvops.git
++ : tested/linux-3.14
++ '[' x = x ']'
++ : git://xenbits.xen.org/linux-pvops.git
++ : tested/linux-arm-xen
++ : git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
++ : tested/2.6.39.x
++ : daily-cron.24b2b8dea180098a3acc809a91cde6c0cc4c8607
++ : daily-cron.24b2b8dea180098a3acc809a91cde6c0cc4c8607
++ : daily-cron.24b2b8dea180098a3acc809a91cde6c0cc4c8607
++ : daily-cron.24b2b8dea180098a3acc809a91cde6c0cc4c8607
++ : daily-cron.24b2b8dea180098a3acc809a91cde6c0cc4c8607
++ : daily-cron.24b2b8dea180098a3acc809a91cde6c0cc4c8607
++ : http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27
++ : git://xenbits.xen.org/staging/qemu-upstream-.git
++ : daily-cron.24b2b8dea180098a3acc809a91cde6c0cc4c8607
++ : git://git.qemu.org/qemu.git
++ : git://xenbits.xen.org/osstest/qemu.git
++ : osstest@xenbits.xensource.com:/home/xen/git/osstest/qemu.git
++ : daily-cron.24b2b8dea180098a3acc809a91cde6c0cc4c8607
+ TREE_LINUX=osstest@xenbits.xensource.com:/home/xen/git/linux-pvops.git
+ TREE_QEMU_MAINLINE=osstest@xenbits.xensource.com:/home/xen/git/osstest/qemu.git
+ TREE_QEMU_UPSTREAM=osstest@xenbits.xensource.com:/home/xen/git/qemu-upstream-.git
+ TREE_XEN=osstest@xenbits.xensource.com:/home/xen/git/xen.git
+ TREE_LIBVIRT=osstest@xenbits.xensource.com:/home/xen/git/libvirt.git
+ TREE_RUMPUSERXEN=osstest@xenbits.xensource.com:/home/xen/git/rumpuser-xen.git
+ TREE_SEABIOS=osstest@xenbits.xensource.com:/home/xen/git/osstest/seabios.git
+ TREE_OVMF=osstest@xenbits.xensource.com:/home/xen/git/osstest/ovmf.git
+ info_linux_tree 24b2b8dea180098a3acc809a91cde6c0cc4c8607
+ case $1 in
+ return 1
+ case "$branch" in
+ echo 'branch 24b2b8dea180098a3acc809a91cde6c0cc4c8607 ?'
branch 24b2b8dea180098a3acc809a91cde6c0cc4c8607 ?
+ exit 1
osstest@osstest:~/branches/for-xen-unstable.git$ 

Re: [xen-unstable test] 35257: regressions - FAIL

[Ian Campbell]

On Tue, 2015-03-03 at 12:44 +0000, Ian Campbell wrote:
> On Thu, 2015-02-26 at 20:14 +0000, xen.org wrote:
> > version targeted for testing:
> >  xen                  24b2b8dea180098a3acc809a91cde6c0cc4c8607
> 
> I have force pushed this revision.
[...]
> + echo 'branch 24b2b8dea180098a3acc809a91cde6c0cc4c8607 ?'
> branch 24b2b8dea180098a3acc809a91cde6c0cc4c8607 ?
> + exit 1

Ahem. Lets try that again.

osstest@osstest:~/branches/for-xen-unstable.git$ OSSTEST_CONFIG=production-config ./ap-push xen-unstable 24b2b8dea180098a3acc809a91cde6c0cc4c8607
+ branch=xen-unstable
+ revision=24b2b8dea180098a3acc809a91cde6c0cc4c8607
+ . cri-lock-repos
++ . cri-common
+++ . cri-getconfig
+++ umask 002
+++ getconfig Repos
+++ perl -e '
                use Osstest;
                readglobalconfig();
                print $c{"Repos"} or die $!;
        '
++ repos=/export/home/osstest/repos
++ repos_lock=/export/home/osstest/repos/lock
++ '[' x '!=' x/export/home/osstest/repos/lock ']'
++ OSSTEST_REPOS_LOCK_LOCKED=/export/home/osstest/repos/lock
++ exec with-lock-ex -w /export/home/osstest/repos/lock ./ap-push xen-unstable 24b2b8dea180098a3acc809a91cde6c0cc4c8607
+ branch=xen-unstable
+ revision=24b2b8dea180098a3acc809a91cde6c0cc4c8607
+ . cri-lock-repos
++ . cri-common
+++ . cri-getconfig
+++ umask 002
+++ getconfig Repos
+++ perl -e '
                use Osstest;
                readglobalconfig();
                print $c{"Repos"} or die $!;
        '
++ repos=/export/home/osstest/repos
++ repos_lock=/export/home/osstest/repos/lock
++ '[' x/export/home/osstest/repos/lock '!=' x/export/home/osstest/repos/lock ']'
+ . cri-common
++ . cri-getconfig
++ umask 002
+ select_xenbranch
+ case "$branch" in
+ tree=xen
+ xenbranch=xen-unstable
+ '[' xxen = xlinux ']'
+ linuxbranch=
+ '[' x = x ']'
+ qemuubranch=qemu-upstream-unstable
+ : tested/2.6.39.x
+ . ap-common
++ : osstest@xenbits.xensource.com
++ : git://xenbits.xen.org/xen.git
++ : osstest@xenbits.xensource.com:/home/xen/git/xen.git
++ : git://xenbits.xen.org/staging/qemu-xen-unstable.git
++ : git://git.kernel.org
++ : git://git.kernel.org/pub/scm/linux/kernel/git
++ : git
++ : git://libvirt.org/libvirt.git
++ : osstest@xenbits.xensource.com:/home/xen/git/libvirt.git
++ : git://xenbits.xen.org/libvirt.git
++ : https://github.com/rumpkernel/rumprun-xen
++ : git
++ : git://xenbits.xen.org/rumpuser-xen.git
++ : osstest@xenbits.xensource.com:/home/xen/git/rumpuser-xen.git
+++ besteffort_repo https://github.com/rumpkernel/rumpkernel-netbsd-src
+++ local repo=https://github.com/rumpkernel/rumpkernel-netbsd-src
+++ cached_repo https://github.com/rumpkernel/rumpkernel-netbsd-src '[fetch=try]'
+++ local repo=https://github.com/rumpkernel/rumpkernel-netbsd-src
+++ local 'options=[fetch=try]'
++++ getconfig GitCacheProxy
++++ perl -e '
                use Osstest;
                readglobalconfig();
                print $c{"GitCacheProxy"} or die $!;
        '
+++ local cache=git://drall.uk.xensource.com:9419/
+++ '[' xgit://drall.uk.xensource.com:9419/ '!=' x ']'
+++ echo 'git://drall.uk.xensource.com:9419/https://github.com/rumpkernel/rumpkernel-netbsd-src%20[fetch=try]'
++ : 'git://drall.uk.xensource.com:9419/https://github.com/rumpkernel/rumpkernel-netbsd-src%20[fetch=try]'
++ : git
++ : git://git.seabios.org/seabios.git
++ : osstest@xenbits.xensource.com:/home/xen/git/osstest/seabios.git
++ : git://xenbits.xen.org/osstest/seabios.git
++ : https://github.com/tianocore/edk2.git
++ : osstest@xenbits.xensource.com:/home/xen/git/osstest/ovmf.git
++ : git://xenbits.xen.org/osstest/ovmf.git
++ : git://xenbits.xen.org/osstest/linux-firmware.git
++ : osstest@xenbits.xensource.com:/home/osstest/ext/linux-firmware.git
++ : git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
++ : osstest@xenbits.xensource.com:/home/xen/git/linux-pvops.git
++ : git://xenbits.xen.org/linux-pvops.git
++ : tested/linux-3.14
++ : tested/linux-arm-xen
++ '[' x = x ']'
++ : git://xenbits.xen.org/linux-pvops.git
++ : tested/linux-3.14
++ '[' x = x ']'
++ : git://xenbits.xen.org/linux-pvops.git
++ : tested/linux-arm-xen
++ : git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
++ : tested/2.6.39.x
++ : daily-cron.xen-unstable
++ : daily-cron.xen-unstable
++ : daily-cron.xen-unstable
++ : daily-cron.xen-unstable
++ : daily-cron.xen-unstable
++ : daily-cron.xen-unstable
++ : http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27
++ : git://xenbits.xen.org/staging/qemu-upstream-unstable.git
++ : daily-cron.xen-unstable
++ : git://git.qemu.org/qemu.git
++ : git://xenbits.xen.org/osstest/qemu.git
++ : osstest@xenbits.xensource.com:/home/xen/git/osstest/qemu.git
++ : daily-cron.xen-unstable
+ TREE_LINUX=osstest@xenbits.xensource.com:/home/xen/git/linux-pvops.git
+ TREE_QEMU_MAINLINE=osstest@xenbits.xensource.com:/home/xen/git/osstest/qemu.git
+ TREE_QEMU_UPSTREAM=osstest@xenbits.xensource.com:/home/xen/git/qemu-upstream-unstable.git
+ TREE_XEN=osstest@xenbits.xensource.com:/home/xen/git/xen.git
+ TREE_LIBVIRT=osstest@xenbits.xensource.com:/home/xen/git/libvirt.git
+ TREE_RUMPUSERXEN=osstest@xenbits.xensource.com:/home/xen/git/rumpuser-xen.git
+ TREE_SEABIOS=osstest@xenbits.xensource.com:/home/xen/git/osstest/seabios.git
+ TREE_OVMF=osstest@xenbits.xensource.com:/home/xen/git/osstest/ovmf.git
+ info_linux_tree xen-unstable
+ case $1 in
+ return 1
+ case "$branch" in
+ cd /export/home/osstest/repos/xen
+ git push osstest@xenbits.xensource.com:/home/xen/git/xen.git 24b2b8dea180098a3acc809a91cde6c0cc4c8607:master
Total 0 (delta 0), reused 0 (delta 0)
To osstest@xenbits.xensource.com:/home/xen/git/xen.git
   cb34a7c..24b2b8d  24b2b8dea180098a3acc809a91cde6c0cc4c8607 -> master
osstest@osstest:~/branches/for-xen-unstable.git$