Re: [qemu-mainline test] 62028: regressions - FAIL

[Anthony PERARD <anthony.perard@citrix.com>]

On Thu, Sep 17, 2015 at 04:15:37PM +0100, Ian Campbell wrote:
> On Thu, 2015-09-17 at 13:44 +0000, osstest service owner wrote:
> > flight 62028 qemu-mainline real [real]
> > http://logs.test-lab.xenproject.org/osstest/logs/62028/
> > 
> > Regressions :-(
> > 
> > Tests which did not succeed and are blocking,
> > including tests which could not be run:
> >  test-amd64-amd64-xl-pvh-amd   6 xen-boot                  fail REGR. vs. 61666
> >  test-amd64-amd64-xl-pvh-intel  6 xen-boot                 fail REGR. vs. 61666
> >  test-armhf-armhf-xl-vhd       6 xen-boot                  fail REGR. vs. 61666
> [...many more for every test-*...]
> 
> I looked at test-*-*-xl and they all seem to have hung at "Starting QEMU as
> disk backend for dom0".
> 
> In this flight http://logs.test-lab.xenproject.org/osstest/logs/62028/test-
> amd64-amd64-xl/serial-chardonnay1.log ends with:
> 
> Sep 17 11:58:48.589051 Starting QEMU as disk backend for dom0
> Sep 17 11:58:48.589081
> Sep 17 11:59:41.890612 <client 0x93c470 connected - now 1 clients>
> (the new client is going to hit the debug keys after the timeout)
> 
> By comparison 61666 has in http://logs.test-lab.xenproject.org/osstest/logs
> /61666/test-amd64-amd64-xl/serial-elbling0.log
> 
>     Sep 10 13:31:52.273142 Starting QEMU as disk backend for dom0
>     Sep 10 13:31:52.273183 Starting NTP server: ntpd.
>     Sep 10 13:32:18.613286 ^[[r^[[H^[[J
> 
>     Sep 10 13:32:18.652996 Debian GNU/Linux 7 elbling0 hvc0
> 
> i.e. normally dom0's qemu starts almost instantly.
> 
> Given the number of failures here this seems unlikely to be host specific
> and almost certain represents a bug in qemu-mainline.
> 
> It was noted on xen-users this morning that there is no logging from this
> qemu instance -- fixing that would be well worthwhile I think.

I don't think there is anything to do on systems using systemd, we can find
the stdout and stderr in the system logs. What does debian with the stdout
and stderr of a service?

> http://logs.test-lab.xenproject.org/osstest/logs/62028/test-amd64-amd64-xl/chardonnay1-output-ps_wwwaxf_-eo_pid%2Ctty%2Cstat%2Ctime%2Cnice%2Cpsr%2Cpcpu%2Cpmem%2Cnwchan%2Cwchan%2325%2Cargs
> 
> shows the process:
> 
> 3638 ?        SL   00:00:00   0   3  0.3  0.5 1d1054 poll_schedule_timeout      \_ startpar -p 4 -t 20 -T 3 -M start -P N -R 2
>  3815 ?        S    00:00:00   0   0  0.0  0.3  b0482 wait                           \_ /bin/bash /etc/init.d/xencommons start
>  4053 ?        Sl   00:00:00   0   1  0.0  1.6 ffffff pipe_wait                          \_ /usr/local/lib/xen/bin/qemu-system-i386 -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize -monitor /dev/null -serial /dev/null -parallel /dev/null -pidfile /var/run/qemu-dom0.pid
>  4171 ?        Ss   00:00:00   0   2  0.0  0.5 113cd9 futex_wait_queue_me                    \_ /usr/local/lib/xen/bin/qemu-system-i386 -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize -monitor /dev/null -serial /dev/null -parallel /dev/null -pidfile /var/run/qemu-dom0.pid
> 
> The bisector seems to have started working on this in
http://logs.test-lab.xenproject.org/osstest/results/bisect/qemu-mainline/test-amd64-amd64-xl-pvh-amd.xen-boot.html

I found the first commit that have the same behavior:
commit 5243722376873a48e9852a58b91f4d4101ee66e4
rcu: init rcu_registry_lock after fork

-- 
Anthony PERARD