[Qemu-devel] [old BUG]: -watchdog-action pause hangs QEMU

[Qemu-devel] [old BUG]: -watchdog-action pause hangs QEMU

[Luiz Capitulino]

Hi Richard,

If setup qemu to use a watchdog with action pause, like this:

 # qemu [...] -watchdog i6300esb -watchdog-action pause

qemu will completely hang when the watchdog fires. I started trying to
bisect, but I went as down as qemu v1.7.0 and the bug exists there.

Here goes a gdb backtrace:

#0  0x00007f298ce31759 in syscall () from /lib64/libc.so.6
#1  0x00007f29939b0b56 in futex_wait (ev=0x7f2994548f18, val=4294967295)
    at /home/lcapitulino/work/src/upstream/qmp-unstable/util/qemu-thread-posix.c:304
#2  0x00007f29939b0c68 in qemu_event_wait (ev=0x7f2994548f18)
    at /home/lcapitulino/work/src/upstream/qmp-unstable/util/qemu-thread-posix.c:400
#3  0x00007f299392a066 in qemu_clock_enable (type=QEMU_CLOCK_VIRTUAL, enabled=false)
    at /home/lcapitulino/work/src/upstream/qmp-unstable/qemu-timer.c:171
#4  0x00007f2993653cc1 in pause_all_vcpus () at /home/lcapitulino/work/src/upstream/qmp-unstable/cpus.c:1078
#5  0x00007f2993652d3f in do_vm_stop (state=RUN_STATE_WATCHDOG) at /home/lcapitulino/work/src/upstream/qmp-unstable/cpus.c:532
#6  0x00007f2993654273 in vm_stop (state=RUN_STATE_WATCHDOG) at /home/lcapitulino/work/src/upstream/qmp-unstable/cpus.c:1221
#7  0x00007f29938c638d in watchdog_perform_action () at /home/lcapitulino/work/src/upstream/qmp-unstable/hw/watchdog/watchdog.c:126
#8  0x00007f29938c679d in i6300esb_timer_expired (vp=0x7f2994d17d40)
    at /home/lcapitulino/work/src/upstream/qmp-unstable/hw/watchdog/wdt_i6300esb.c:197
#9  0x00007f299392a8ce in timerlist_run_timers (timer_list=0x7f2994548ec0)
    at /home/lcapitulino/work/src/upstream/qmp-unstable/qemu-timer.c:491
#10 0x00007f299392a916 in qemu_clock_run_timers (type=QEMU_CLOCK_VIRTUAL)
    at /home/lcapitulino/work/src/upstream/qmp-unstable/qemu-timer.c:502
#11 0x00007f299392abfe in qemu_clock_run_all_timers () at /home/lcapitulino/work/src/upstream/qmp-unstable/qemu-timer.c:608
#12 0x00007f299392956a in main_loop_wait (nonblocking=0) at /home/lcapitulino/work/src/upstream/qmp-unstable/main-loop.c:490
#13 0x00007f299374b40c in main_loop () at /home/lcapitulino/work/src/upstream/qmp-unstable/vl.c:2007
#14 0x00007f2993752e5d in main (argc=23, argv=0x7fffc6a5a738, envp=0x7fffc6a5a7f8)
    at /home/lcapitulino/work/src/upstream/qmp-unstable/vl.c:4526

Re: [Qemu-devel] [old BUG]: -watchdog-action pause hangs QEMU

[Paolo Bonzini]

Il 26/06/2014 22:19, Luiz Capitulino ha scritto:
> Here goes a gdb backtrace:
>
> #0  0x00007f298ce31759 in syscall () from /lib64/libc.so.6
> #1  0x00007f29939b0b56 in futex_wait (ev=0x7f2994548f18, val=4294967295)
>     at /home/lcapitulino/work/src/upstream/qmp-unstable/util/qemu-thread-posix.c:304
> #2  0x00007f29939b0c68 in qemu_event_wait (ev=0x7f2994548f18)
>     at /home/lcapitulino/work/src/upstream/qmp-unstable/util/qemu-thread-posix.c:400
> #3  0x00007f299392a066 in qemu_clock_enable (type=QEMU_CLOCK_VIRTUAL, enabled=false)
>     at /home/lcapitulino/work/src/upstream/qmp-unstable/qemu-timer.c:171
> #4  0x00007f2993653cc1 in pause_all_vcpus () at /home/lcapitulino/work/src/upstream/qmp-unstable/cpus.c:1078
> #5  0x00007f2993652d3f in do_vm_stop (state=RUN_STATE_WATCHDOG) at /home/lcapitulino/work/src/upstream/qmp-unstable/cpus.c:532
> #6  0x00007f2993654273 in vm_stop (state=RUN_STATE_WATCHDOG) at /home/lcapitulino/work/src/upstream/qmp-unstable/cpus.c:1221
> #7  0x00007f29938c638d in watchdog_perform_action () at /home/lcapitulino/work/src/upstream/qmp-unstable/hw/watchdog/watchdog.c:126
> #8  0x00007f29938c679d in i6300esb_timer_expired (vp=0x7f2994d17d40)
>     at /home/lcapitulino/work/src/upstream/qmp-unstable/hw/watchdog/wdt_i6300esb.c:197
> #9  0x00007f299392a8ce in timerlist_run_timers (timer_list=0x7f2994548ec0)
>     at /home/lcapitulino/work/src/upstream/qmp-unstable/qemu-timer.c:491
> #10 0x00007f299392a916 in qemu_clock_run_timers (type=QEMU_CLOCK_VIRTUAL)
>     at /home/lcapitulino/work/src/upstream/qmp-unstable/qemu-timer.c:502
> #11 0x00007f299392abfe in qemu_clock_run_all_timers () at /home/lcapitulino/work/src/upstream/qmp-unstable/qemu-timer.c:608
> #12 0x00007f299392956a in main_loop_wait (nonblocking=0) at /home/lcapitulino/work/src/upstream/qmp-unstable/main-loop.c:490
> #13 0x00007f299374b40c in main_loop () at /home/lcapitulino/work/src/upstream/qmp-unstable/vl.c:2007
> #14 0x00007f2993752e5d in main (argc=23, argv=0x7fffc6a5a738, envp=0x7fffc6a5a7f8)
>     at /home/lcapitulino/work/src/upstream/qmp-unstable/vl.c:4526
>

Can you include the backtrace for the VCPU thread(s) too?

Paolo

Re: [Qemu-devel] [old BUG]: -watchdog-action pause hangs QEMU

[Luiz Capitulino]

On Thu, 26 Jun 2014 23:24:09 +0200
Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 26/06/2014 22:19, Luiz Capitulino ha scritto:
> > Here goes a gdb backtrace:
> >
> > #0  0x00007f298ce31759 in syscall () from /lib64/libc.so.6
> > #1  0x00007f29939b0b56 in futex_wait (ev=0x7f2994548f18, val=4294967295)
> >     at /home/lcapitulino/work/src/upstream/qmp-unstable/util/qemu-thread-posix.c:304
> > #2  0x00007f29939b0c68 in qemu_event_wait (ev=0x7f2994548f18)
> >     at /home/lcapitulino/work/src/upstream/qmp-unstable/util/qemu-thread-posix.c:400
> > #3  0x00007f299392a066 in qemu_clock_enable (type=QEMU_CLOCK_VIRTUAL, enabled=false)
> >     at /home/lcapitulino/work/src/upstream/qmp-unstable/qemu-timer.c:171
> > #4  0x00007f2993653cc1 in pause_all_vcpus () at /home/lcapitulino/work/src/upstream/qmp-unstable/cpus.c:1078
> > #5  0x00007f2993652d3f in do_vm_stop (state=RUN_STATE_WATCHDOG) at /home/lcapitulino/work/src/upstream/qmp-unstable/cpus.c:532
> > #6  0x00007f2993654273 in vm_stop (state=RUN_STATE_WATCHDOG) at /home/lcapitulino/work/src/upstream/qmp-unstable/cpus.c:1221
> > #7  0x00007f29938c638d in watchdog_perform_action () at /home/lcapitulino/work/src/upstream/qmp-unstable/hw/watchdog/watchdog.c:126
> > #8  0x00007f29938c679d in i6300esb_timer_expired (vp=0x7f2994d17d40)
> >     at /home/lcapitulino/work/src/upstream/qmp-unstable/hw/watchdog/wdt_i6300esb.c:197
> > #9  0x00007f299392a8ce in timerlist_run_timers (timer_list=0x7f2994548ec0)
> >     at /home/lcapitulino/work/src/upstream/qmp-unstable/qemu-timer.c:491
> > #10 0x00007f299392a916 in qemu_clock_run_timers (type=QEMU_CLOCK_VIRTUAL)
> >     at /home/lcapitulino/work/src/upstream/qmp-unstable/qemu-timer.c:502
> > #11 0x00007f299392abfe in qemu_clock_run_all_timers () at /home/lcapitulino/work/src/upstream/qmp-unstable/qemu-timer.c:608
> > #12 0x00007f299392956a in main_loop_wait (nonblocking=0) at /home/lcapitulino/work/src/upstream/qmp-unstable/main-loop.c:490
> > #13 0x00007f299374b40c in main_loop () at /home/lcapitulino/work/src/upstream/qmp-unstable/vl.c:2007
> > #14 0x00007f2993752e5d in main (argc=23, argv=0x7fffc6a5a738, envp=0x7fffc6a5a7f8)
> >     at /home/lcapitulino/work/src/upstream/qmp-unstable/vl.c:4526
> >
> 
> Can you include the backtrace for the VCPU thread(s) too?

Here you go, I start the guest with only one vCPU.

#0  0x00007f9d89d2a59d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9d89d26179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f9d89d26048 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f9d8b5b4687 in qemu_mutex_lock (mutex=0x7f9d8ba9c340 <qemu_global_mutex>) at /home/lcapitulino/work/src/upstream/qmp-unstable/util/qemu-thread-posix.c:76
#4  0x00007f9d8b257c02 in qemu_mutex_lock_iothread () at /home/lcapitulino/work/src/upstream/qmp-unstable/cpus.c:1044
#5  0x00007f9d8b26e8bb in kvm_cpu_exec (cpu=0x7f9d8e04b830) at /home/lcapitulino/work/src/upstream/qmp-unstable/kvm-all.c:1716
#6  0x00007f9d8b2577c5 in qemu_kvm_cpu_thread_fn (arg=0x7f9d8e04b830) at /home/lcapitulino/work/src/upstream/qmp-unstable/cpus.c:874
#7  0x00007f9d89d23f33 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f9d84a3aded in clone () from /lib64/libc.so.6

Re: [Qemu-devel] [old BUG]: -watchdog-action pause hangs QEMU

[Paolo Bonzini]

Il 27/06/2014 14:45, Luiz Capitulino ha scritto:
> On Thu, 26 Jun 2014 23:24:09 +0200
> Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>> Il 26/06/2014 22:19, Luiz Capitulino ha scritto:
>>> Here goes a gdb backtrace:
>>>
>>> #0  0x00007f298ce31759 in syscall () from /lib64/libc.so.6
>>> #1  0x00007f29939b0b56 in futex_wait (ev=0x7f2994548f18, val=4294967295)
>>>     at /home/lcapitulino/work/src/upstream/qmp-unstable/util/qemu-thread-posix.c:304
>>> #2  0x00007f29939b0c68 in qemu_event_wait (ev=0x7f2994548f18)
>>>     at /home/lcapitulino/work/src/upstream/qmp-unstable/util/qemu-thread-posix.c:400
>>> #3  0x00007f299392a066 in qemu_clock_enable (type=QEMU_CLOCK_VIRTUAL, enabled=false)
>>>     at /home/lcapitulino/work/src/upstream/qmp-unstable/qemu-timer.c:171


/* Disabling the clock will wait for related timerlists to stop
  * executing qemu_run_timers.  Thus, this functions should not
  * be used from the callback of a timer that is based on @clock.
  * Doing so would cause a deadlock.
  */

It does indeed.  I'll post a fix during hard freeze.

Paolo

Re: [Qemu-devel] [old BUG]: -watchdog-action pause hangs QEMU

[Luiz Capitulino]

On Fri, 27 Jun 2014 14:50:27 +0200
Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 27/06/2014 14:45, Luiz Capitulino ha scritto:
> > On Thu, 26 Jun 2014 23:24:09 +0200
> > Paolo Bonzini <pbonzini@redhat.com> wrote:
> >
> >> Il 26/06/2014 22:19, Luiz Capitulino ha scritto:
> >>> Here goes a gdb backtrace:
> >>>
> >>> #0  0x00007f298ce31759 in syscall () from /lib64/libc.so.6
> >>> #1  0x00007f29939b0b56 in futex_wait (ev=0x7f2994548f18, val=4294967295)
> >>>     at /home/lcapitulino/work/src/upstream/qmp-unstable/util/qemu-thread-posix.c:304
> >>> #2  0x00007f29939b0c68 in qemu_event_wait (ev=0x7f2994548f18)
> >>>     at /home/lcapitulino/work/src/upstream/qmp-unstable/util/qemu-thread-posix.c:400
> >>> #3  0x00007f299392a066 in qemu_clock_enable (type=QEMU_CLOCK_VIRTUAL, enabled=false)
> >>>     at /home/lcapitulino/work/src/upstream/qmp-unstable/qemu-timer.c:171
> 
> 
> /* Disabling the clock will wait for related timerlists to stop
>   * executing qemu_run_timers.  Thus, this functions should not
>   * be used from the callback of a timer that is based on @clock.
>   * Doing so would cause a deadlock.
>   */
> 
> It does indeed.  I'll post a fix during hard freeze.

Thanks Paolo!