[Qemu-devel] qemu-kvm: sigsegv at exit

[Qemu-devel] qemu-kvm: sigsegv at exit

[Michael S. Tsirkin]

Hi!
I'm sometimes getting segfaults when I kill qemu.
This time I caught it when qemu was under gdb:


Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x411d0940 (LWP 14446)]
0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
    at /home/mst/scm/qemu-kvm/vl.c:1009
1009            if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
(gdb) l
1004        ts->next = *pt;
1005        *pt = ts;
1006
1007        /* Rearm if necessary  */
1008        if (pt == &active_timers[ts->clock->type]) {
1009            if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
1010                qemu_rearm_alarm_timer(alarm_timer);
1011            }
1012            /* Interrupt execution to force deadline recalculation.  */
1013            if (use_icount)
(gdb) p alarm_timer
$1 = (struct qemu_alarm_timer *) 0x0
(gdb) where
#0  0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
    at /home/mst/scm/qemu-kvm/vl.c:1009
#1  0x000000000041aadf in virtio_net_handle_tx (vdev=<value optimized out>, vq=0x19f5af0)
    at /home/mst/scm/qemu-kvm/hw/virtio-net.c:696
#2  0x0000000000421669 in kvm_run (vcpu=0x19d46a0, env=0x19c2250) at /home/mst/scm/qemu-kvm/qemu-kvm.c:797
#3  0x00000000004216d6 in kvm_cpu_exec (env=0x83d0f8) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1714
#4  0x0000000000422981 in ap_main_loop (_env=<value optimized out>) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1969
#5  0x000000377dc06367 in start_thread () from /lib64/libpthread.so.0
#6  0x000000377d0d30ad in clone () from /lib64/libc.so.6
(gdb)

So this probably means that we have already run quit_timers:

static void quit_timers(void)
{
    alarm_timer->stop(alarm_timer);
    alarm_timer = NULL;
}

but kvm vcpu thread is still running.


Not sure what the right fix is here: should we stop
kvm after main loop has exited?

-- 
MST

[Qemu-devel] Re: qemu-kvm: sigsegv at exit

[Marcelo Tosatti]

On Thu, Oct 22, 2009 at 02:00:15PM +0200, Michael S. Tsirkin wrote:
> Hi!
> I'm sometimes getting segfaults when I kill qemu.
> This time I caught it when qemu was under gdb:
> 
> 
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x411d0940 (LWP 14446)]
> 0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
>     at /home/mst/scm/qemu-kvm/vl.c:1009
> 1009            if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
> (gdb) l
> 1004        ts->next = *pt;
> 1005        *pt = ts;
> 1006
> 1007        /* Rearm if necessary  */
> 1008        if (pt == &active_timers[ts->clock->type]) {
> 1009            if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
> 1010                qemu_rearm_alarm_timer(alarm_timer);
> 1011            }
> 1012            /* Interrupt execution to force deadline recalculation.  */
> 1013            if (use_icount)
> (gdb) p alarm_timer
> $1 = (struct qemu_alarm_timer *) 0x0
> (gdb) where
> #0  0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
>     at /home/mst/scm/qemu-kvm/vl.c:1009
> #1  0x000000000041aadf in virtio_net_handle_tx (vdev=<value optimized out>, vq=0x19f5af0)
>     at /home/mst/scm/qemu-kvm/hw/virtio-net.c:696
> #2  0x0000000000421669 in kvm_run (vcpu=0x19d46a0, env=0x19c2250) at /home/mst/scm/qemu-kvm/qemu-kvm.c:797
> #3  0x00000000004216d6 in kvm_cpu_exec (env=0x83d0f8) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1714
> #4  0x0000000000422981 in ap_main_loop (_env=<value optimized out>) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1969
> #5  0x000000377dc06367 in start_thread () from /lib64/libpthread.so.0
> #6  0x000000377d0d30ad in clone () from /lib64/libc.so.6
> (gdb)
> 
> So this probably means that we have already run quit_timers:
> 
> static void quit_timers(void)
> {
>     alarm_timer->stop(alarm_timer);
>     alarm_timer = NULL;
> }
> 
> but kvm vcpu thread is still running.
> 
> 
> Not sure what the right fix is here: should we stop
> kvm after main loop has exited?

kvm_main_loop_wait(env, 0) can process the stop request (signalling
iothread that vcpu is stopped, so its OK to exit) and continue to
kvm_cpu_exec.

Can you please try this:

diff --git a/qemu-kvm.c b/qemu-kvm.c
index 87ece3d..141c8b1 100644
--- a/qemu-kvm.c
+++ b/qemu-kvm.c
@@ -1931,7 +1931,8 @@ static int kvm_main_loop_cpu(CPUState *env)
         }
         if (run_cpu) {
             kvm_main_loop_wait(env, 0);
-            kvm_cpu_exec(env);
+            if (!is_cpu_stopped(env))
+                kvm_cpu_exec(env);
         } else {
             kvm_main_loop_wait(env, 1000);
         }

[Qemu-devel] Re: qemu-kvm: sigsegv at exit

[Michael S. Tsirkin]

On Thu, Oct 22, 2009 at 06:57:27PM -0200, Marcelo Tosatti wrote:
> On Thu, Oct 22, 2009 at 02:00:15PM +0200, Michael S. Tsirkin wrote:
> > Hi!
> > I'm sometimes getting segfaults when I kill qemu.
> > This time I caught it when qemu was under gdb:
> > 
> > 
> > Program received signal SIGSEGV, Segmentation fault.
> > [Switching to Thread 0x411d0940 (LWP 14446)]
> > 0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
> >     at /home/mst/scm/qemu-kvm/vl.c:1009
> > 1009            if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
> > (gdb) l
> > 1004        ts->next = *pt;
> > 1005        *pt = ts;
> > 1006
> > 1007        /* Rearm if necessary  */
> > 1008        if (pt == &active_timers[ts->clock->type]) {
> > 1009            if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
> > 1010                qemu_rearm_alarm_timer(alarm_timer);
> > 1011            }
> > 1012            /* Interrupt execution to force deadline recalculation.  */
> > 1013            if (use_icount)
> > (gdb) p alarm_timer
> > $1 = (struct qemu_alarm_timer *) 0x0
> > (gdb) where
> > #0  0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
> >     at /home/mst/scm/qemu-kvm/vl.c:1009
> > #1  0x000000000041aadf in virtio_net_handle_tx (vdev=<value optimized out>, vq=0x19f5af0)
> >     at /home/mst/scm/qemu-kvm/hw/virtio-net.c:696
> > #2  0x0000000000421669 in kvm_run (vcpu=0x19d46a0, env=0x19c2250) at /home/mst/scm/qemu-kvm/qemu-kvm.c:797
> > #3  0x00000000004216d6 in kvm_cpu_exec (env=0x83d0f8) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1714
> > #4  0x0000000000422981 in ap_main_loop (_env=<value optimized out>) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1969
> > #5  0x000000377dc06367 in start_thread () from /lib64/libpthread.so.0
> > #6  0x000000377d0d30ad in clone () from /lib64/libc.so.6
> > (gdb)
> > 
> > So this probably means that we have already run quit_timers:
> > 
> > static void quit_timers(void)
> > {
> >     alarm_timer->stop(alarm_timer);
> >     alarm_timer = NULL;
> > }
> > 
> > but kvm vcpu thread is still running.
> > 
> > 
> > Not sure what the right fix is here: should we stop
> > kvm after main loop has exited?
> 
> kvm_main_loop_wait(env, 0) can process the stop request (signalling
> iothread that vcpu is stopped, so its OK to exit) and continue to
> kvm_cpu_exec.
> 
> Can you please try this:

I applied this, and have not yet see any segfaults at exit.
Not sure whether this is means anything as the crash is not
100% reproducable. Push it out to Anthony and we'll see, long term?
Based on the knowledge of how to fix this,
how would you go about reproducing it?

> diff --git a/qemu-kvm.c b/qemu-kvm.c
> index 87ece3d..141c8b1 100644
> --- a/qemu-kvm.c
> +++ b/qemu-kvm.c
> @@ -1931,7 +1931,8 @@ static int kvm_main_loop_cpu(CPUState *env)
>          }
>          if (run_cpu) {
>              kvm_main_loop_wait(env, 0);
> -            kvm_cpu_exec(env);
> +            if (!is_cpu_stopped(env))
> +                kvm_cpu_exec(env);
>          } else {
>              kvm_main_loop_wait(env, 1000);
>          }

[Qemu-devel] Re: qemu-kvm: sigsegv at exit

[Marcelo Tosatti]

> On Thu, Oct 22, 2009 at 06:57:27PM -0200, Marcelo Tosatti wrote:
> > On Thu, Oct 22, 2009 at 02:00:15PM +0200, Michael S. Tsirkin wrote:
> > > Hi!
> > > I'm sometimes getting segfaults when I kill qemu.
> > > This time I caught it when qemu was under gdb:
> > > 
> > > 
> > > Program received signal SIGSEGV, Segmentation fault.
> > > [Switching to Thread 0x411d0940 (LWP 14446)]
> > > 0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
> > >     at /home/mst/scm/qemu-kvm/vl.c:1009
> > > 1009            if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
> > > (gdb) l
> > > 1004        ts->next = *pt;
> > > 1005        *pt = ts;
> > > 1006
> > > 1007        /* Rearm if necessary  */
> > > 1008        if (pt == &active_timers[ts->clock->type]) {
> > > 1009            if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
> > > 1010                qemu_rearm_alarm_timer(alarm_timer);
> > > 1011            }
> > > 1012            /* Interrupt execution to force deadline recalculation.  */
> > > 1013            if (use_icount)
> > > (gdb) p alarm_timer
> > > $1 = (struct qemu_alarm_timer *) 0x0
> > > (gdb) where
> > > #0  0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
> > >     at /home/mst/scm/qemu-kvm/vl.c:1009
> > > #1  0x000000000041aadf in virtio_net_handle_tx (vdev=<value optimized out>, vq=0x19f5af0)
> > >     at /home/mst/scm/qemu-kvm/hw/virtio-net.c:696
> > > #2  0x0000000000421669 in kvm_run (vcpu=0x19d46a0, env=0x19c2250) at /home/mst/scm/qemu-kvm/qemu-kvm.c:797
> > > #3  0x00000000004216d6 in kvm_cpu_exec (env=0x83d0f8) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1714
> > > #4  0x0000000000422981 in ap_main_loop (_env=<value optimized out>) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1969
> > > #5  0x000000377dc06367 in start_thread () from /lib64/libpthread.so.0
> > > #6  0x000000377d0d30ad in clone () from /lib64/libc.so.6
> > > (gdb)
> > > 
> > > So this probably means that we have already run quit_timers:
> > > 
> > > static void quit_timers(void)
> > > {
> > >     alarm_timer->stop(alarm_timer);
> > >     alarm_timer = NULL;
> > > }
> > > 
> > > but kvm vcpu thread is still running.
> > > 
> > > 
> > > Not sure what the right fix is here: should we stop
> > > kvm after main loop has exited?
> > 
> > kvm_main_loop_wait(env, 0) can process the stop request (signalling
> > iothread that vcpu is stopped, so its OK to exit) and continue to
> > kvm_cpu_exec.
> > 
> > Can you please try this:
> 
> I applied this, and have not yet see any segfaults at exit.
> Not sure whether this is means anything as the crash is not
> 100% reproducable. Push it out to Anthony and we'll see, long term?
> Based on the knowledge of how to fix this,
> how would you go about reproducing it?

Add code to trigger the race manually, but i'm pretty sure thats it.

Thanks for testing.

[Qemu-devel] Re: qemu-kvm: sigsegv at exit

[Michael S. Tsirkin]

On Mon, Oct 26, 2009 at 04:43:11PM -0200, Marcelo Tosatti wrote:
> > On Thu, Oct 22, 2009 at 06:57:27PM -0200, Marcelo Tosatti wrote:
> > > On Thu, Oct 22, 2009 at 02:00:15PM +0200, Michael S. Tsirkin wrote:
> > > > Hi!
> > > > I'm sometimes getting segfaults when I kill qemu.
> > > > This time I caught it when qemu was under gdb:
> > > > 
> > > > 
> > > > Program received signal SIGSEGV, Segmentation fault.
> > > > [Switching to Thread 0x411d0940 (LWP 14446)]
> > > > 0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
> > > >     at /home/mst/scm/qemu-kvm/vl.c:1009
> > > > 1009            if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
> > > > (gdb) l
> > > > 1004        ts->next = *pt;
> > > > 1005        *pt = ts;
> > > > 1006
> > > > 1007        /* Rearm if necessary  */
> > > > 1008        if (pt == &active_timers[ts->clock->type]) {
> > > > 1009            if ((alarm_timer->flags & ALARM_FLAG_EXPIRED) == 0) {
> > > > 1010                qemu_rearm_alarm_timer(alarm_timer);
> > > > 1011            }
> > > > 1012            /* Interrupt execution to force deadline recalculation.  */
> > > > 1013            if (use_icount)
> > > > (gdb) p alarm_timer
> > > > $1 = (struct qemu_alarm_timer *) 0x0
> > > > (gdb) where
> > > > #0  0x000000000040afb4 in qemu_mod_timer (ts=0x19f0fd0, expire_time=62275467335)
> > > >     at /home/mst/scm/qemu-kvm/vl.c:1009
> > > > #1  0x000000000041aadf in virtio_net_handle_tx (vdev=<value optimized out>, vq=0x19f5af0)
> > > >     at /home/mst/scm/qemu-kvm/hw/virtio-net.c:696
> > > > #2  0x0000000000421669 in kvm_run (vcpu=0x19d46a0, env=0x19c2250) at /home/mst/scm/qemu-kvm/qemu-kvm.c:797
> > > > #3  0x00000000004216d6 in kvm_cpu_exec (env=0x83d0f8) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1714
> > > > #4  0x0000000000422981 in ap_main_loop (_env=<value optimized out>) at /home/mst/scm/qemu-kvm/qemu-kvm.c:1969
> > > > #5  0x000000377dc06367 in start_thread () from /lib64/libpthread.so.0
> > > > #6  0x000000377d0d30ad in clone () from /lib64/libc.so.6
> > > > (gdb)
> > > > 
> > > > So this probably means that we have already run quit_timers:
> > > > 
> > > > static void quit_timers(void)
> > > > {
> > > >     alarm_timer->stop(alarm_timer);
> > > >     alarm_timer = NULL;
> > > > }
> > > > 
> > > > but kvm vcpu thread is still running.
> > > > 
> > > > 
> > > > Not sure what the right fix is here: should we stop
> > > > kvm after main loop has exited?
> > > 
> > > kvm_main_loop_wait(env, 0) can process the stop request (signalling
> > > iothread that vcpu is stopped, so its OK to exit) and continue to
> > > kvm_cpu_exec.
> > > 
> > > Can you please try this:
> > 
> > I applied this, and have not yet see any segfaults at exit.
> > Not sure whether this is means anything as the crash is not
> > 100% reproducable. Push it out to Anthony and we'll see, long term?
> > Based on the knowledge of how to fix this,
> > how would you go about reproducing it?
> 
> Add code to trigger the race manually,

If you like, send a patch adding such code, I will test.

> but i'm pretty sure thats it.
> 
> Thanks for testing.