[Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

Hello,

the issue is not specific to the iothread code because generic
virtio-blk also hangs up:

Given code set like in the
http://www.mail-archive.com/qemu-devel@nongnu.org/msg246164.html,
launch a VM with virtio-blk disk and writeback rbd backend, fire up
fio, migrate once with libvirt:

time virsh migrate vm27842 qemu+tcp://10.6.0.10/system --live
--persistent --undefinesource --timeout 60


real    1m2.969s
user    0m0.016s
sys     0m0.008s

For ones who are unfamiliar with libvirt syntax this means that the
live migration was failed to complete over sixty seconds, VM was
frozen, moved and re-launched at the destination. After this, I/O gets
stuck forever. Any diagnostic information is available upon request if
there will be difficulties repeating the issue.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Sun, Jul 13, 2014 at 4:28 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> Hello,
>
> the issue is not specific to the iothread code because generic
> virtio-blk also hangs up:
>
> Given code set like in the
> http://www.mail-archive.com/qemu-devel@nongnu.org/msg246164.html,
> launch a VM with virtio-blk disk and writeback rbd backend, fire up
> fio, migrate once with libvirt:
>
> time virsh migrate vm27842 qemu+tcp://10.6.0.10/system --live
> --persistent --undefinesource --timeout 60
>
>
> real    1m2.969s
> user    0m0.016s
> sys     0m0.008s
>
> For ones who are unfamiliar with libvirt syntax this means that the
> live migration was failed to complete over sixty seconds, VM was
> frozen, moved and re-launched at the destination. After this, I/O gets
> stuck forever. Any diagnostic information is available upon request if
> there will be difficulties repeating the issue.

Small follow-up: issue have probabilistic nature, as it looks - by
limited number of runs, it is reproducible within three cases:
 1) live migration went well, I/O locked up,
 2) live migration failed by timeout, I/O locked up,
 3) live migration went well and disk was not locked, but on backward
migration we are always hitting 2).

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Amit Shah]

On (Sun) 13 Jul 2014 [16:28:56], Andrey Korolyov wrote:
> Hello,
> 
> the issue is not specific to the iothread code because generic
> virtio-blk also hangs up:

Do you know which version works well?  If you could bisect, that'll
help a lot.

Thanks,
		Amit

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Tue, Jul 15, 2014 at 9:03 AM, Amit Shah <amit.shah@redhat.com> wrote:
> On (Sun) 13 Jul 2014 [16:28:56], Andrey Korolyov wrote:
>> Hello,
>>
>> the issue is not specific to the iothread code because generic
>> virtio-blk also hangs up:
>
> Do you know which version works well?  If you could bisect, that'll
> help a lot.
>
> Thanks,
>                 Amit

Hi,

2.0 works definitely well. I`ll try to finish bisection today, though
every step takes about 10 minutes to complete.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Tue, Jul 15, 2014 at 10:52 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
> On Tue, Jul 15, 2014 at 9:03 AM, Amit Shah <amit.shah@redhat.com> wrote:
>> On (Sun) 13 Jul 2014 [16:28:56], Andrey Korolyov wrote:
>>> Hello,
>>>
>>> the issue is not specific to the iothread code because generic
>>> virtio-blk also hangs up:
>>
>> Do you know which version works well?  If you could bisect, that'll
>> help a lot.
>>
>> Thanks,
>>                 Amit
>
> Hi,
>
> 2.0 works definitely well. I`ll try to finish bisection today, though
> every step takes about 10 minutes to complete.

Yay! It is even outside of virtio-blk.

commit 9b1786829aefb83f37a8f3135e3ea91c56001b56
Author: Marcelo Tosatti <mtosatti@redhat.com>
Date:   Tue Jun 3 13:34:48 2014 -0300

    kvmclock: Ensure proper env->tsc value for kvmclock_current_nsec calculation

    Ensure proper env->tsc value for kvmclock_current_nsec calculation.

    Reported-by: Marcin Gibuła <m.gibula@beyond.pl>
    Cc: qemu-stable@nongnu.org
    Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
    Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Paolo Bonzini]

Il 13/07/2014 17:29, Andrey Korolyov ha scritto:
> Small follow-up: issue have probabilistic nature, as it looks - by
> limited number of runs, it is reproducible within three cases:
>  1) live migration went well, I/O locked up,
>  2) live migration failed by timeout, I/O locked up,
>  3) live migration went well and disk was not locked, but on backward
> migration we are always hitting 2).

Can you provide a gdb backtrace of case (2)?

Paolo

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

[-- Attachment #1: Type: text/plain, Size: 954 bytes --]

On Tue, Jul 15, 2014 at 7:57 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 13/07/2014 17:29, Andrey Korolyov ha scritto:
>
>> Small follow-up: issue have probabilistic nature, as it looks - by
>> limited number of runs, it is reproducible within three cases:
>>  1) live migration went well, I/O locked up,
>>  2) live migration failed by timeout, I/O locked up,
>>  3) live migration went well and disk was not locked, but on backward
>> migration we are always hitting 2).
>
>
> Can you provide a gdb backtrace of case (2)?
>
> Paolo
>

Sorry, I`ve mixed reset into. The timeout on migration happens if
there was already hang before reset. Bt-reset and bt-noreset
corresponds to two backtraces first of which had hang+reset
previously. As it looks by bisection, the problem introduced by fix
targeted to patch exactly the same behavior. Moreover, rolling
problematic commit back (9a833121eb253589143ff8fe30be8a311a2c16b3)
fixes situation for me.

[-- Attachment #2: bt-noreset.txt --]
[-- Type: text/plain, Size: 18103 bytes --]

Thread 33 (Thread 0x7fb632fcc700 (LWP 3023)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fb63ed7e51b in ceph::log::Log::entry (this=0x7fb642f13860) at log/Log.cc:323
#2  0x00007fb63b321e9a in start_thread (arg=0x7fb632fcc700) at pthread_create.c:308
#3  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 32 (Thread 0x7fb6327cb700 (LWP 3024)):
#0  sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:102
#1  0x00007fb63ee57b18 in CephContextServiceThread::entry (this=0x7fb642f15630) at common/ceph_context.cc:57
#2  0x00007fb63b321e9a in start_thread (arg=0x7fb6327cb700) at pthread_create.c:308
#3  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 31 (Thread 0x7fb631fca700 (LWP 3025)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fb63ed9ae6b in Wait (mutex=..., this=0x7fb642f1afa0) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7fb642f1af38) at msg/DispatchQueue.cc:129
#3  0x00007fb63ee035ad in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb631fca700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 30 (Thread 0x7fb6317c9700 (LWP 3026)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fb63edff5f0 in Wait (mutex=..., this=0x7fb642f1b360) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7fb642f1ae50) at msg/SimpleMessenger.cc:206
#3  0x00007fb63ee03e7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:422
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb6317c9700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 29 (Thread 0x7fb630fc8700 (LWP 3027)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00007fb63ed33238 in WaitUntil (when=..., mutex=..., this=0x7fb642f15a40) at common/Cond.h:71
#2  SafeTimer::timer_thread (this=0x7fb642f15a30) at common/Timer.cc:114
#3  0x00007fb63ed3416d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb630fc8700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 28 (Thread 0x7fb6307c7700 (LWP 3028)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fb63ed34bdf in Wait (mutex=..., this=0x7fb642f15b58) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x7fb642f15af8) at common/Finisher.cc:80
#3  0x00007fb63b321e9a in start_thread (arg=0x7fb6307c7700) at pthread_create.c:308
#4  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 27 (Thread 0x7fb62ffc6700 (LWP 3029)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fb63edeee1e in Wait (mutex=..., this=0x7fb642f1d1e8) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7fb642f1d000) at msg/Pipe.cc:1610
#3  0x00007fb63edfa1dd in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb62ffc6700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 26 (Thread 0x7fb62fec5700 (LWP 3030)):
#0  0x00007fb63b043a13 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007fb63ede3f0c in Pipe::tcp_read_wait (this=this@entry=0x7fb642f1d000) at msg/Pipe.cc:2144
#2  0x00007fb63ede41d0 in Pipe::tcp_read (this=this@entry=0x7fb642f1d000, buf=<optimized out>, buf@entry=0x7fb62fec4aef "\377\a", len=len@entry=1) at msg/Pipe.cc:2117
#3  0x00007fb63edf610e in Pipe::reader (this=0x7fb642f1d000) at msg/Pipe.cc:1353
#4  0x00007fb63edfa2fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007fb63b321e9a in start_thread (arg=0x7fb62fec5700) at pthread_create.c:308
#6  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 25 (Thread 0x7fb62fdc4700 (LWP 3031)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00007fb63ed33238 in WaitUntil (when=..., mutex=..., this=0x7fb642f15e48) at common/Cond.h:71
#2  SafeTimer::timer_thread (this=0x7fb642f15e38) at common/Timer.cc:114
#3  0x00007fb63ed3416d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb62fdc4700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 24 (Thread 0x7fb62f5c3700 (LWP 3032)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fb63ed34bdf in Wait (mutex=..., this=0x7fb642f15f68) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x7fb642f15f08) at common/Finisher.cc:80
#3  0x00007fb63b321e9a in start_thread (arg=0x7fb62f5c3700) at pthread_create.c:308
#4  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 23 (Thread 0x7fb62edc2700 (LWP 3033)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fb63ed34bdf in Wait (mutex=..., this=0x7fb642f20858) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x7fb642f207f8) at common/Finisher.cc:80
#3  0x00007fb63b321e9a in start_thread (arg=0x7fb62edc2700) at pthread_create.c:308
#4  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 22 (Thread 0x7fb62e5c1700 (LWP 3034)):

#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00007fb63f9890bc in WaitUntil (when=..., mutex=..., this=0x7fb642f20798) at ./common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7fb642f20798) at ./common/Cond.h:79
#3  ObjectCacher::flusher_entry (this=0x7fb642f205c0) at osdc/ObjectCacher.cc:1458
#4  0x00007fb63f99c20d in ObjectCacher::FlusherThread::entry (this=<optimized out>) at osdc/ObjectCacher.h:354
#5  0x00007fb63b321e9a in start_thread (arg=0x7fb62e5c1700) at pthread_create.c:308
#6  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 21 (Thread 0x7fb62ddc0700 (LWP 3035)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fb63edeee1e in Wait (mutex=..., this=0x7fb642f21548) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7fb642f21360) at msg/Pipe.cc:1610
#3  0x00007fb63edfa1dd in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb62ddc0700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 20 (Thread 0x7fb62dcbf700 (LWP 3036)):
#0  0x00007fb63b043a13 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007fb63ede3f0c in Pipe::tcp_read_wait (this=this@entry=0x7fb642f21360) at msg/Pipe.cc:2144
#2  0x00007fb63ede41d0 in Pipe::tcp_read (this=this@entry=0x7fb642f21360, buf=<optimized out>, buf@entry=0x7fb62dcbeaef "\377\001", len=len@entry=1) at msg/Pipe.cc:2117
#3  0x00007fb63edf610e in Pipe::reader (this=0x7fb642f21360) at msg/Pipe.cc:1353
#4  0x00007fb63edfa2fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007fb63b321e9a in start_thread (arg=0x7fb62dcbf700) at pthread_create.c:308
#6  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 19 (Thread 0x7fb62dbbe700 (LWP 3037)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fb63edeee1e in Wait (mutex=..., this=0x7fb642f21be8) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7fb642f21a00) at msg/Pipe.cc:1610
#3  0x00007fb63edfa1dd in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb62dbbe700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 18 (Thread 0x7fb62dabd700 (LWP 3038)):
#0  0x00007fb63b043a13 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007fb63ede3f0c in Pipe::tcp_read_wait (this=this@entry=0x7fb642f21a00) at msg/Pipe.cc:2144
#2  0x00007fb63ede41d0 in Pipe::tcp_read (this=this@entry=0x7fb642f21a00, buf=<optimized out>, buf@entry=0x7fb62dabcaef "\377\002", len=len@entry=1) at msg/Pipe.cc:2117
#3  0x00007fb63edf610e in Pipe::reader (this=0x7fb642f21a00) at msg/Pipe.cc:1353
#4  0x00007fb63edfa2fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007fb63b321e9a in start_thread (arg=0x7fb62dabd700) at pthread_create.c:308
#6  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7fb62d9bc700 (LWP 3039)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fb63edeee1e in Wait (mutex=..., this=0x7fb642f224f8) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7fb642f22310) at msg/Pipe.cc:1610
#3  0x00007fb63edfa1dd in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb62d9bc700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7fb62d8bb700 (LWP 3040)):
#0  0x00007fb63b043a13 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007fb63ede3f0c in Pipe::tcp_read_wait (this=this@entry=0x7fb642f22310) at msg/Pipe.cc:2144
#2  0x00007fb63ede41d0 in Pipe::tcp_read (this=this@entry=0x7fb642f22310, buf=<optimized out>, buf@entry=0x7fb62d8baaef "\377\033", len=len@entry=1) at msg/Pipe.cc:2117
#3  0x00007fb63edf610e in Pipe::reader (this=0x7fb642f22310) at msg/Pipe.cc:1353
#4  0x00007fb63edfa2fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007fb63b321e9a in start_thread (arg=0x7fb62d8bb700) at pthread_create.c:308
#6  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7fb62d7ba700 (LWP 3041)):
#0  0x00007fb63b047c37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fb6405560a9 in ?? ()
#2  0x00007fb6405561e5 in ?? ()
#3  0x00007fb6405419bc in ?? ()
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb62d7ba700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7fb62cfb9700 (LWP 3042)):
#0  0x00007fb63b047c37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fb6405560a9 in ?? ()
#2  0x00007fb6405561e5 in ?? ()
#3  0x00007fb6405419bc in ?? ()
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb62cfb9700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7fb5fffff700 (LWP 3043)):
#0  0x00007fb63b047c37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fb6405560a9 in ?? ()
#2  0x00007fb6405561e5 in ?? ()
#3  0x00007fb6405419bc in ?? ()
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb5fffff700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7fb5ff7fe700 (LWP 3044)):
#0  0x00007fb63b047c37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fb6405560a9 in ?? ()
#2  0x00007fb6405561e5 in ?? ()
#3  0x00007fb6405419bc in ?? ()
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb5ff7fe700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7fb5feffd700 (LWP 3045)):
#0  0x00007fb63b047c37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fb6405560a9 in ?? ()
#2  0x00007fb6405561e5 in ?? ()
#3  0x00007fb6405419bc in ?? ()
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb5feffd700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7fb5fe7fc700 (LWP 3046)):
#0  0x00007fb63b047c37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fb6405560a9 in ?? ()
#2  0x00007fb6405561e5 in ?? ()
#3  0x00007fb6405419bc in ?? ()
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb5fe7fc700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7fb5fdffb700 (LWP 3047)):
#0  0x00007fb63b047c37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fb6405560a9 in ?? ()
#2  0x00007fb6405561e5 in ?? ()
#3  0x00007fb6405419bc in ?? ()
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb5fdffb700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fb5fd7fa700 (LWP 3048)):
#0  0x00007fb63b047c37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fb6405560a9 in ?? ()
#2  0x00007fb6405561e5 in ?? ()
#3  0x00007fb6405419bc in ?? ()
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb5fd7fa700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fb5fcff9700 (LWP 3049)):
#0  0x00007fb63b047c37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fb6405560a9 in ?? ()
#2  0x00007fb6405561e5 in ?? ()
#3  0x00007fb6405419bc in ?? ()
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb5fcff9700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7fb5dffff700 (LWP 3050)):
#0  0x00007fb63b047c37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fb6405560a9 in ?? ()
#2  0x00007fb6405561e5 in ?? ()
#3  0x00007fb6405419bc in ?? ()
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb5dffff700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fb5df7fe700 (LWP 3051)):
#0  0x00007fb63b047c37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fb6405560a9 in ?? ()
#2  0x00007fb6405561e5 in ?? ()
#3  0x00007fb6405419bc in ?? ()
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb5df7fe700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7fb5deffd700 (LWP 3052)):
#0  0x00007fb63b047c37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fb6405560a9 in ?? ()
#2  0x00007fb6405561e5 in ?? ()
#3  0x00007fb6405419bc in ?? ()
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb5deffd700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7fb62c5ff700 (LWP 3054)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fb63edeee1e in Wait (mutex=..., this=0x7fb643060548) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7fb643060360) at msg/Pipe.cc:1610
#3  0x00007fb63edfa1dd in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007fb63b321e9a in start_thread (arg=0x7fb62c5ff700) at pthread_create.c:308
#5  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7fb62c3ff700 (LWP 3055)):
#0  0x00007fb63b043a13 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007fb63ede3f0c in Pipe::tcp_read_wait (this=this@entry=0x7fb643060360) at msg/Pipe.cc:2144
#2  0x00007fb63ede41d0 in Pipe::tcp_read (this=this@entry=0x7fb643060360, buf=<optimized out>, buf@entry=0x7fb62c3feaef "\377\001", len=len@entry=1) at msg/Pipe.cc:2117
#3  0x00007fb63edf610e in Pipe::reader (this=0x7fb643060360) at msg/Pipe.cc:1353
#4  0x00007fb63edfa2fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007fb63b321e9a in start_thread (arg=0x7fb62c3ff700) at pthread_create.c:308
#6  0x00007fb63b04f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fb640349940 (LWP 3020)):
#0  0x00007fb63b043ae3 in ppoll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, sigmask=<optimized out>) at ../sysdeps/unix/sysv/linux/ppoll.c:58
#1  0x00007fb6407b6679 in ?? ()
#2  0x00007fb6407b5a64 in ?? ()
#3  0x00007fb640513665 in main ()


[-- Attachment #3: bt-reset.txt --]
[-- Type: text/plain, Size: 18099 bytes --]

Thread 33 (Thread 0x7f0098460700 (LWP 2886)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f00a421251b in ceph::log::Log::entry (this=0x7f00a7de5860) at log/Log.cc:323
#2  0x00007f00a07b5e9a in start_thread (arg=0x7f0098460700) at pthread_create.c:308
#3  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 32 (Thread 0x7f0097c5f700 (LWP 2887)):
#0  sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:102
#1  0x00007f00a42ebb18 in CephContextServiceThread::entry (this=0x7f00a7de7630) at common/ceph_context.cc:57
#2  0x00007f00a07b5e9a in start_thread (arg=0x7f0097c5f700) at pthread_create.c:308
#3  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 31 (Thread 0x7f009745e700 (LWP 2888)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f00a422ee6b in Wait (mutex=..., this=0x7f00a7decfa0) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7f00a7decf38) at msg/DispatchQueue.cc:129
#3  0x00007f00a42975ad in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f009745e700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 30 (Thread 0x7f0096c5d700 (LWP 2889)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f00a42935f0 in Wait (mutex=..., this=0x7f00a7ded360) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7f00a7dece50) at msg/SimpleMessenger.cc:206
#3  0x00007f00a4297e7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:422
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f0096c5d700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 29 (Thread 0x7f009645c700 (LWP 2890)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00007f00a41c7238 in WaitUntil (when=..., mutex=..., this=0x7f00a7de7a40) at common/Cond.h:71
#2  SafeTimer::timer_thread (this=0x7f00a7de7a30) at common/Timer.cc:114
#3  0x00007f00a41c816d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f009645c700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 28 (Thread 0x7f0095c5b700 (LWP 2891)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f00a41c8bdf in Wait (mutex=..., this=0x7f00a7de7b58) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x7f00a7de7af8) at common/Finisher.cc:80
#3  0x00007f00a07b5e9a in start_thread (arg=0x7f0095c5b700) at pthread_create.c:308
#4  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 27 (Thread 0x7f009545a700 (LWP 2892)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f00a4282e1e in Wait (mutex=..., this=0x7f00a7def1e8) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7f00a7def000) at msg/Pipe.cc:1610
#3  0x00007f00a428e1dd in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f009545a700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 26 (Thread 0x7f0095359700 (LWP 2893)):
#0  0x00007f00a04d7a13 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007f00a4277f0c in Pipe::tcp_read_wait (this=this@entry=0x7f00a7def000) at msg/Pipe.cc:2144
#2  0x00007f00a42781d0 in Pipe::tcp_read (this=this@entry=0x7f00a7def000, buf=<optimized out>, buf@entry=0x7f0095358aef "\377\b", len=len@entry=1) at msg/Pipe.cc:2117
#3  0x00007f00a428a10e in Pipe::reader (this=0x7f00a7def000) at msg/Pipe.cc:1353
#4  0x00007f00a428e2fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f00a07b5e9a in start_thread (arg=0x7f0095359700) at pthread_create.c:308
#6  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 25 (Thread 0x7f0095258700 (LWP 2894)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00007f00a41c7238 in WaitUntil (when=..., mutex=..., this=0x7f00a7de7e48) at common/Cond.h:71
#2  SafeTimer::timer_thread (this=0x7f00a7de7e38) at common/Timer.cc:114
#3  0x00007f00a41c816d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f0095258700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 24 (Thread 0x7f0094a57700 (LWP 2895)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f00a41c8bdf in Wait (mutex=..., this=0x7f00a7de7f68) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x7f00a7de7f08) at common/Finisher.cc:80
#3  0x00007f00a07b5e9a in start_thread (arg=0x7f0094a57700) at pthread_create.c:308
#4  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 23 (Thread 0x7f0087fff700 (LWP 2896)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f00a41c8bdf in Wait (mutex=..., this=0x7f00a7df2858) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x7f00a7df27f8) at common/Finisher.cc:80
#3  0x00007f00a07b5e9a in start_thread (arg=0x7f0087fff700) at pthread_create.c:308
#4  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 22 (Thread 0x7f00877fe700 (LWP 2897)):

#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00007f00a4e1d0bc in WaitUntil (when=..., mutex=..., this=0x7f00a7df2798) at ./common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f00a7df2798) at ./common/Cond.h:79
#3  ObjectCacher::flusher_entry (this=0x7f00a7df25c0) at osdc/ObjectCacher.cc:1458
#4  0x00007f00a4e3020d in ObjectCacher::FlusherThread::entry (this=<optimized out>) at osdc/ObjectCacher.h:354
#5  0x00007f00a07b5e9a in start_thread (arg=0x7f00877fe700) at pthread_create.c:308
#6  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 21 (Thread 0x7f0094256700 (LWP 2898)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f00a4282e1e in Wait (mutex=..., this=0x7f00a7df3548) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7f00a7df3360) at msg/Pipe.cc:1610
#3  0x00007f00a428e1dd in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f0094256700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 20 (Thread 0x7f0094155700 (LWP 2899)):
#0  0x00007f00a04d7a13 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007f00a4277f0c in Pipe::tcp_read_wait (this=this@entry=0x7f00a7df3360) at msg/Pipe.cc:2144
#2  0x00007f00a42781d0 in Pipe::tcp_read (this=this@entry=0x7f00a7df3360, buf=<optimized out>, buf@entry=0x7f0094154aef "\377\001", len=len@entry=1) at msg/Pipe.cc:2117
#3  0x00007f00a428a10e in Pipe::reader (this=0x7f00a7df3360) at msg/Pipe.cc:1353
#4  0x00007f00a428e2fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f00a07b5e9a in start_thread (arg=0x7f0094155700) at pthread_create.c:308
#6  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 19 (Thread 0x7f0086ffd700 (LWP 2900)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f00a4282e1e in Wait (mutex=..., this=0x7f00a7df3be8) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7f00a7df3a00) at msg/Pipe.cc:1610
#3  0x00007f00a428e1dd in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f0086ffd700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 18 (Thread 0x7f0086efc700 (LWP 2901)):
#0  0x00007f00a04d7a13 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007f00a4277f0c in Pipe::tcp_read_wait (this=this@entry=0x7f00a7df3a00) at msg/Pipe.cc:2144
#2  0x00007f00a42781d0 in Pipe::tcp_read (this=this@entry=0x7f00a7df3a00, buf=<optimized out>, buf@entry=0x7f0086efbaef "\377\002", len=len@entry=1) at msg/Pipe.cc:2117
#3  0x00007f00a428a10e in Pipe::reader (this=0x7f00a7df3a00) at msg/Pipe.cc:1353
#4  0x00007f00a428e2fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f00a07b5e9a in start_thread (arg=0x7f0086efc700) at pthread_create.c:308
#6  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7f0086dfb700 (LWP 2902)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f00a4282e1e in Wait (mutex=..., this=0x7f00a7df44f8) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7f00a7df4310) at msg/Pipe.cc:1610
#3  0x00007f00a428e1dd in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f0086dfb700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7f0086cfa700 (LWP 2903)):
#0  0x00007f00a04d7a13 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007f00a4277f0c in Pipe::tcp_read_wait (this=this@entry=0x7f00a7df4310) at msg/Pipe.cc:2144
#2  0x00007f00a42781d0 in Pipe::tcp_read (this=this@entry=0x7f00a7df4310, buf=<optimized out>, buf@entry=0x7f0086cf9aef "\377.", len=len@entry=1) at msg/Pipe.cc:2117
#3  0x00007f00a428a10e in Pipe::reader (this=0x7f00a7df4310) at msg/Pipe.cc:1353
#4  0x00007f00a428e2fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f00a07b5e9a in start_thread (arg=0x7f0086cfa700) at pthread_create.c:308
#6  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7f0086bf9700 (LWP 2904)):
#0  0x00007f00a04dbc37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f00a59ea0a9 in ?? ()
#2  0x00007f00a59ea1e5 in ?? ()
#3  0x00007f00a59d59bc in ?? ()
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f0086bf9700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7f00863f8700 (LWP 2905)):
#0  0x00007f00a04dbc37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f00a59ea0a9 in ?? ()
#2  0x00007f00a59ea1e5 in ?? ()
#3  0x00007f00a59d59bc in ?? ()
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f00863f8700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7f0085bf7700 (LWP 2906)):
#0  0x00007f00a04dbc37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f00a59ea0a9 in ?? ()
#2  0x00007f00a59ea1e5 in ?? ()
#3  0x00007f00a59d59bc in ?? ()
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f0085bf7700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7f00853f6700 (LWP 2907)):
#0  0x00007f00a04dbc37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f00a59ea0a9 in ?? ()
#2  0x00007f00a59ea1e5 in ?? ()
#3  0x00007f00a59d59bc in ?? ()
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f00853f6700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7f0084bf5700 (LWP 2908)):
#0  0x00007f00a04dbc37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f00a59ea0a9 in ?? ()
#2  0x00007f00a59ea1e5 in ?? ()
#3  0x00007f00a59d59bc in ?? ()
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f0084bf5700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7f0057fff700 (LWP 2909)):
#0  0x00007f00a04dbc37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f00a59ea0a9 in ?? ()
#2  0x00007f00a59ea1e5 in ?? ()
#3  0x00007f00a59d59bc in ?? ()
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f0057fff700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7f00577fe700 (LWP 2910)):
#0  0x00007f00a04dbc37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f00a59ea0a9 in ?? ()
#2  0x00007f00a59ea1e5 in ?? ()
#3  0x00007f00a59d59bc in ?? ()
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f00577fe700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7f0056ffd700 (LWP 2911)):
#0  0x00007f00a04dbc37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f00a59ea0a9 in ?? ()
#2  0x00007f00a59ea1e5 in ?? ()
#3  0x00007f00a59d59bc in ?? ()
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f0056ffd700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f00567fc700 (LWP 2912)):
#0  0x00007f00a04dbc37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f00a59ea0a9 in ?? ()
#2  0x00007f00a59ea1e5 in ?? ()
#3  0x00007f00a59d59bc in ?? ()
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f00567fc700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f0055ffb700 (LWP 2913)):
#0  0x00007f00a04dbc37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f00a59ea0a9 in ?? ()
#2  0x00007f00a59ea1e5 in ?? ()
#3  0x00007f00a59d59bc in ?? ()
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f0055ffb700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f00557fa700 (LWP 2914)):
#0  0x00007f00a04dbc37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f00a59ea0a9 in ?? ()
#2  0x00007f00a59ea1e5 in ?? ()
#3  0x00007f00a59d59bc in ?? ()
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f00557fa700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f0054ff9700 (LWP 2915)):
#0  0x00007f00a04dbc37 in ioctl () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f00a59ea0a9 in ?? ()
#2  0x00007f00a59ea1e5 in ?? ()
#3  0x00007f00a59d59bc in ?? ()
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f0054ff9700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f00841ff700 (LWP 2917)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f00a4282e1e in Wait (mutex=..., this=0x7f00a7f32548) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7f00a7f32360) at msg/Pipe.cc:1610
#3  0x00007f00a428e1dd in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f00a07b5e9a in start_thread (arg=0x7f00841ff700) at pthread_create.c:308
#5  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f00547f8700 (LWP 2918)):
#0  0x00007f00a04d7a13 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007f00a4277f0c in Pipe::tcp_read_wait (this=this@entry=0x7f00a7f32360) at msg/Pipe.cc:2144
#2  0x00007f00a42781d0 in Pipe::tcp_read (this=this@entry=0x7f00a7f32360, buf=<optimized out>, buf@entry=0x7f00547f7aef "\377\001", len=len@entry=1) at msg/Pipe.cc:2117
#3  0x00007f00a428a10e in Pipe::reader (this=0x7f00a7f32360) at msg/Pipe.cc:1353
#4  0x00007f00a428e2fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f00a07b5e9a in start_thread (arg=0x7f00547f8700) at pthread_create.c:308
#6  0x00007f00a04e33dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f00a57dd940 (LWP 2883)):
#0  0x00007f00a04d7ae3 in ppoll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, sigmask=<optimized out>) at ../sysdeps/unix/sysv/linux/ppoll.c:58
#1  0x00007f00a5c4a679 in ?? ()
#2  0x00007f00a5c49a64 in ?? ()
#3  0x00007f00a59a7665 in main ()

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Tue, Jul 15, 2014 at 9:32 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> On Tue, Jul 15, 2014 at 7:57 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> Il 13/07/2014 17:29, Andrey Korolyov ha scritto:
>>
>>> Small follow-up: issue have probabilistic nature, as it looks - by
>>> limited number of runs, it is reproducible within three cases:
>>>  1) live migration went well, I/O locked up,
>>>  2) live migration failed by timeout, I/O locked up,
>>>  3) live migration went well and disk was not locked, but on backward
>>> migration we are always hitting 2).
>>
>>
>> Can you provide a gdb backtrace of case (2)?
>>
>> Paolo
>>
>
> Sorry, I`ve mixed reset into. The timeout on migration happens if
> there was already hang before reset. Bt-reset and bt-noreset
> corresponds to two backtraces first of which had hang+reset
> previously. As it looks by bisection, the problem introduced by fix
> targeted to patch exactly the same behavior. Moreover, rolling
> problematic commit back (9a833121eb253589143ff8fe30be8a311a2c16b3)
> fixes situation for me.

Sorry, wrong hash given, right one is the
9b1786829aefb83f37a8f3135e3ea91c56001b56. I am referring to the
http://marc.info/?l=qemu-devel&m=140174116403095&w=2 above. Also I
forgot to mention that two backtraces are absolutely identical, so the
problem is linked to the emulator state for sure.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcelo Tosatti]

On Tue, Jul 15, 2014 at 06:01:08PM +0400, Andrey Korolyov wrote:
> On Tue, Jul 15, 2014 at 10:52 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
> > On Tue, Jul 15, 2014 at 9:03 AM, Amit Shah <amit.shah@redhat.com> wrote:
> >> On (Sun) 13 Jul 2014 [16:28:56], Andrey Korolyov wrote:
> >>> Hello,
> >>>
> >>> the issue is not specific to the iothread code because generic
> >>> virtio-blk also hangs up:
> >>
> >> Do you know which version works well?  If you could bisect, that'll
> >> help a lot.
> >>
> >> Thanks,
> >>                 Amit
> >
> > Hi,
> >
> > 2.0 works definitely well. I`ll try to finish bisection today, though
> > every step takes about 10 minutes to complete.
> 
> Yay! It is even outside of virtio-blk.
> 
> commit 9b1786829aefb83f37a8f3135e3ea91c56001b56
> Author: Marcelo Tosatti <mtosatti@redhat.com>
> Date:   Tue Jun 3 13:34:48 2014 -0300
> 
>     kvmclock: Ensure proper env->tsc value for kvmclock_current_nsec calculation
> 
>     Ensure proper env->tsc value for kvmclock_current_nsec calculation.
> 
>     Reported-by: Marcin Gibuła <m.gibula@beyond.pl>
>     Cc: qemu-stable@nongnu.org
>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Andrey,

Can you please provide instructions on how to create reproducible 
environment? 

The following patch is equivalent to the original patch, for
the purposes of fixing the kvmclock problem.

Perhaps it becomes easier to spot the reason for the hang you are
experiencing.


diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
index 272a88a..feb5fc5 100644
--- a/hw/i386/kvm/clock.c
+++ b/hw/i386/kvm/clock.c
@@ -17,7 +17,6 @@
 #include "qemu/host-utils.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/kvm.h"
-#include "sysemu/cpus.h"
 #include "hw/sysbus.h"
 #include "hw/kvm/clock.h"
 
@@ -66,7 +65,6 @@ static uint64_t kvmclock_current_nsec(KVMClockState *s)
 
     cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
 
-    assert(time.tsc_timestamp <= migration_tsc);
     delta = migration_tsc - time.tsc_timestamp;
     if (time.tsc_shift < 0) {
         delta >>= -time.tsc_shift;
@@ -125,8 +123,6 @@ static void kvmclock_vm_state_change(void *opaque, int running,
         if (s->clock_valid) {
             return;
         }
-
-        cpu_synchronize_all_states();
         ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
         if (ret < 0) {
             fprintf(stderr, "KVM_GET_CLOCK failed: %s\n", strerror(ret));
diff --git a/migration.c b/migration.c
index 8d675b3..34f2325 100644
--- a/migration.c
+++ b/migration.c
@@ -608,6 +608,7 @@ static void *migration_thread(void *opaque)
                 qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
                 old_vm_running = runstate_is_running();
 
+                cpu_synchronize_all_states();
                 ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
                 if (ret >= 0) {
                     qemu_file_set_rate_limit(s->file, INT64_MAX);

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Wed, Jul 16, 2014 at 1:09 AM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> On Tue, Jul 15, 2014 at 06:01:08PM +0400, Andrey Korolyov wrote:
>> On Tue, Jul 15, 2014 at 10:52 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
>> > On Tue, Jul 15, 2014 at 9:03 AM, Amit Shah <amit.shah@redhat.com> wrote:
>> >> On (Sun) 13 Jul 2014 [16:28:56], Andrey Korolyov wrote:
>> >>> Hello,
>> >>>
>> >>> the issue is not specific to the iothread code because generic
>> >>> virtio-blk also hangs up:
>> >>
>> >> Do you know which version works well?  If you could bisect, that'll
>> >> help a lot.
>> >>
>> >> Thanks,
>> >>                 Amit
>> >
>> > Hi,
>> >
>> > 2.0 works definitely well. I`ll try to finish bisection today, though
>> > every step takes about 10 minutes to complete.
>>
>> Yay! It is even outside of virtio-blk.
>>
>> commit 9b1786829aefb83f37a8f3135e3ea91c56001b56
>> Author: Marcelo Tosatti <mtosatti@redhat.com>
>> Date:   Tue Jun 3 13:34:48 2014 -0300
>>
>>     kvmclock: Ensure proper env->tsc value for kvmclock_current_nsec calculation
>>
>>     Ensure proper env->tsc value for kvmclock_current_nsec calculation.
>>
>>     Reported-by: Marcin Gibuła <m.gibula@beyond.pl>
>>     Cc: qemu-stable@nongnu.org
>>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>
> Andrey,
>
> Can you please provide instructions on how to create reproducible
> environment?
>
> The following patch is equivalent to the original patch, for
> the purposes of fixing the kvmclock problem.
>
> Perhaps it becomes easier to spot the reason for the hang you are
> experiencing.
>
>
> diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
> index 272a88a..feb5fc5 100644
> --- a/hw/i386/kvm/clock.c
> +++ b/hw/i386/kvm/clock.c
> @@ -17,7 +17,6 @@
>  #include "qemu/host-utils.h"
>  #include "sysemu/sysemu.h"
>  #include "sysemu/kvm.h"
> -#include "sysemu/cpus.h"
>  #include "hw/sysbus.h"
>  #include "hw/kvm/clock.h"
>
> @@ -66,7 +65,6 @@ static uint64_t kvmclock_current_nsec(KVMClockState *s)
>
>      cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
>
> -    assert(time.tsc_timestamp <= migration_tsc);
>      delta = migration_tsc - time.tsc_timestamp;
>      if (time.tsc_shift < 0) {
>          delta >>= -time.tsc_shift;
> @@ -125,8 +123,6 @@ static void kvmclock_vm_state_change(void *opaque, int running,
>          if (s->clock_valid) {
>              return;
>          }
> -
> -        cpu_synchronize_all_states();
>          ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
>          if (ret < 0) {
>              fprintf(stderr, "KVM_GET_CLOCK failed: %s\n", strerror(ret));
> diff --git a/migration.c b/migration.c
> index 8d675b3..34f2325 100644
> --- a/migration.c
> +++ b/migration.c
> @@ -608,6 +608,7 @@ static void *migration_thread(void *opaque)
>                  qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
>                  old_vm_running = runstate_is_running();
>
> +                cpu_synchronize_all_states();
>                  ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
>                  if (ret >= 0) {
>                      qemu_file_set_rate_limit(s->file, INT64_MAX);

Marcelo, I do not see way easier than creating PoC deployment
(involving at least two separated physical nodes) which will act for
both as a sender and receiver for migration and for Ceph storage
(http://ceph.com/docs/master/start/). For simplicity you probably want
to disable cephx, therefore not putting the secret in the CLI. Also
you may receive minimal qemu-ready installation using Mirantis` Fuel
with Ceph deployment settings (it`ll deploy some Openstack too as a
side effect, but the main reason to do things this way is a very high
level of provisioning automation, you`ll get necessary environment
with multi-node setting with RBD backend in matter of some clicks and
some hours). In a meantime, I`ll try to reproduce the issue with
iscsi, because I do not want to mess with shared storage and sanlock
plugin.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Paolo Bonzini]

Il 15/07/2014 23:25, Andrey Korolyov ha scritto:
> On Wed, Jul 16, 2014 at 1:09 AM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
>> On Tue, Jul 15, 2014 at 06:01:08PM +0400, Andrey Korolyov wrote:
>>> On Tue, Jul 15, 2014 at 10:52 AM, Andrey Korolyov <andrey@xdel.ru> wrote:
>>>> On Tue, Jul 15, 2014 at 9:03 AM, Amit Shah <amit.shah@redhat.com> wrote:
>>>>> On (Sun) 13 Jul 2014 [16:28:56], Andrey Korolyov wrote:
>>>>>> Hello,
>>>>>>
>>>>>> the issue is not specific to the iothread code because generic
>>>>>> virtio-blk also hangs up:
>>>>>
>>>>> Do you know which version works well?  If you could bisect, that'll
>>>>> help a lot.
>>>>>
>>>>> Thanks,
>>>>>                 Amit
>>>>
>>>> Hi,
>>>>
>>>> 2.0 works definitely well. I`ll try to finish bisection today, though
>>>> every step takes about 10 minutes to complete.
>>>
>>> Yay! It is even outside of virtio-blk.
>>>
>>> commit 9b1786829aefb83f37a8f3135e3ea91c56001b56
>>> Author: Marcelo Tosatti <mtosatti@redhat.com>
>>> Date:   Tue Jun 3 13:34:48 2014 -0300
>>>
>>>     kvmclock: Ensure proper env->tsc value for kvmclock_current_nsec calculation
>>>
>>>     Ensure proper env->tsc value for kvmclock_current_nsec calculation.
>>>
>>>     Reported-by: Marcin Gibuła <m.gibula@beyond.pl>
>>>     Cc: qemu-stable@nongnu.org
>>>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>>>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>>
>> Andrey,
>>
>> Can you please provide instructions on how to create reproducible
>> environment?
>>
>> The following patch is equivalent to the original patch, for
>> the purposes of fixing the kvmclock problem.
>>
>> Perhaps it becomes easier to spot the reason for the hang you are
>> experiencing.
>>
>>
>> diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
>> index 272a88a..feb5fc5 100644
>> --- a/hw/i386/kvm/clock.c
>> +++ b/hw/i386/kvm/clock.c
>> @@ -17,7 +17,6 @@
>>  #include "qemu/host-utils.h"
>>  #include "sysemu/sysemu.h"
>>  #include "sysemu/kvm.h"
>> -#include "sysemu/cpus.h"
>>  #include "hw/sysbus.h"
>>  #include "hw/kvm/clock.h"
>>
>> @@ -66,7 +65,6 @@ static uint64_t kvmclock_current_nsec(KVMClockState *s)
>>
>>      cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
>>
>> -    assert(time.tsc_timestamp <= migration_tsc);
>>      delta = migration_tsc - time.tsc_timestamp;
>>      if (time.tsc_shift < 0) {
>>          delta >>= -time.tsc_shift;
>> @@ -125,8 +123,6 @@ static void kvmclock_vm_state_change(void *opaque, int running,
>>          if (s->clock_valid) {
>>              return;
>>          }
>> -
>> -        cpu_synchronize_all_states();
>>          ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
>>          if (ret < 0) {
>>              fprintf(stderr, "KVM_GET_CLOCK failed: %s\n", strerror(ret));
>> diff --git a/migration.c b/migration.c
>> index 8d675b3..34f2325 100644
>> --- a/migration.c
>> +++ b/migration.c
>> @@ -608,6 +608,7 @@ static void *migration_thread(void *opaque)
>>                  qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
>>                  old_vm_running = runstate_is_running();
>>
>> +                cpu_synchronize_all_states();
>>                  ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
>>                  if (ret >= 0) {
>>                      qemu_file_set_rate_limit(s->file, INT64_MAX);

It could also be useful to apply the above patch _and_ revert 
a096b3a6732f846ec57dc28b47ee9435aa0609bf, then try to reproduce.

Paolo

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Wed, Jul 16, 2014 at 2:01 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 15/07/2014 23:25, Andrey Korolyov ha scritto:
>
>> On Wed, Jul 16, 2014 at 1:09 AM, Marcelo Tosatti <mtosatti@redhat.com>
>> wrote:
>>>
>>> On Tue, Jul 15, 2014 at 06:01:08PM +0400, Andrey Korolyov wrote:
>>>>
>>>> On Tue, Jul 15, 2014 at 10:52 AM, Andrey Korolyov <andrey@xdel.ru>
>>>> wrote:
>>>>>
>>>>> On Tue, Jul 15, 2014 at 9:03 AM, Amit Shah <amit.shah@redhat.com>
>>>>> wrote:
>>>>>>
>>>>>> On (Sun) 13 Jul 2014 [16:28:56], Andrey Korolyov wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> the issue is not specific to the iothread code because generic
>>>>>>> virtio-blk also hangs up:
>>>>>>
>>>>>>
>>>>>> Do you know which version works well?  If you could bisect, that'll
>>>>>> help a lot.
>>>>>>
>>>>>> Thanks,
>>>>>>                 Amit
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> 2.0 works definitely well. I`ll try to finish bisection today, though
>>>>> every step takes about 10 minutes to complete.
>>>>
>>>>
>>>> Yay! It is even outside of virtio-blk.
>>>>
>>>> commit 9b1786829aefb83f37a8f3135e3ea91c56001b56
>>>> Author: Marcelo Tosatti <mtosatti@redhat.com>
>>>> Date:   Tue Jun 3 13:34:48 2014 -0300
>>>>
>>>>     kvmclock: Ensure proper env->tsc value for kvmclock_current_nsec
>>>> calculation
>>>>
>>>>     Ensure proper env->tsc value for kvmclock_current_nsec calculation.
>>>>
>>>>     Reported-by: Marcin Gibuła <m.gibula@beyond.pl>
>>>>     Cc: qemu-stable@nongnu.org
>>>>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>>>>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>>>
>>>
>>> Andrey,
>>>
>>> Can you please provide instructions on how to create reproducible
>>> environment?
>>>
>>> The following patch is equivalent to the original patch, for
>>> the purposes of fixing the kvmclock problem.
>>>
>>> Perhaps it becomes easier to spot the reason for the hang you are
>>> experiencing.
>>>
>>>
>>> diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
>>> index 272a88a..feb5fc5 100644
>>> --- a/hw/i386/kvm/clock.c
>>> +++ b/hw/i386/kvm/clock.c
>>> @@ -17,7 +17,6 @@
>>>  #include "qemu/host-utils.h"
>>>  #include "sysemu/sysemu.h"
>>>  #include "sysemu/kvm.h"
>>> -#include "sysemu/cpus.h"
>>>  #include "hw/sysbus.h"
>>>  #include "hw/kvm/clock.h"
>>>
>>> @@ -66,7 +65,6 @@ static uint64_t kvmclock_current_nsec(KVMClockState *s)
>>>
>>>      cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
>>>
>>> -    assert(time.tsc_timestamp <= migration_tsc);
>>>      delta = migration_tsc - time.tsc_timestamp;
>>>      if (time.tsc_shift < 0) {
>>>          delta >>= -time.tsc_shift;
>>> @@ -125,8 +123,6 @@ static void kvmclock_vm_state_change(void *opaque,
>>> int running,
>>>          if (s->clock_valid) {
>>>              return;
>>>          }
>>> -
>>> -        cpu_synchronize_all_states();
>>>          ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
>>>          if (ret < 0) {
>>>              fprintf(stderr, "KVM_GET_CLOCK failed: %s\n",
>>> strerror(ret));
>>> diff --git a/migration.c b/migration.c
>>> index 8d675b3..34f2325 100644
>>> --- a/migration.c
>>> +++ b/migration.c
>>> @@ -608,6 +608,7 @@ static void *migration_thread(void *opaque)
>>>                  qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
>>>                  old_vm_running = runstate_is_running();
>>>
>>> +                cpu_synchronize_all_states();
>>>                  ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
>>>                  if (ret >= 0) {
>>>                      qemu_file_set_rate_limit(s->file, INT64_MAX);
>
>
> It could also be useful to apply the above patch _and_ revert
> a096b3a6732f846ec57dc28b47ee9435aa0609bf, then try to reproduce.
>
> Paolo

Yes, it solved the issue for me! (it took much time to check because
most of country` debian mirrors went inconsistent by some reason)

Also trivial addition:

diff --git a/migration.c b/migration.c
index 34f2325..65d1c88 100644
--- a/migration.c
+++ b/migration.c
@@ -25,6 +25,7 @@
 #include "qemu/thread.h"
 #include "qmp-commands.h"
 #include "trace.h"
+#include "sysemu/cpus.h"

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcelo Tosatti]

On Wed, Jul 16, 2014 at 03:40:47AM +0400, Andrey Korolyov wrote:
> On Wed, Jul 16, 2014 at 2:01 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > Il 15/07/2014 23:25, Andrey Korolyov ha scritto:
> >
> >> On Wed, Jul 16, 2014 at 1:09 AM, Marcelo Tosatti <mtosatti@redhat.com>
> >> wrote:
> >>>
> >>> On Tue, Jul 15, 2014 at 06:01:08PM +0400, Andrey Korolyov wrote:
> >>>>
> >>>> On Tue, Jul 15, 2014 at 10:52 AM, Andrey Korolyov <andrey@xdel.ru>
> >>>> wrote:
> >>>>>
> >>>>> On Tue, Jul 15, 2014 at 9:03 AM, Amit Shah <amit.shah@redhat.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> On (Sun) 13 Jul 2014 [16:28:56], Andrey Korolyov wrote:
> >>>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> the issue is not specific to the iothread code because generic
> >>>>>>> virtio-blk also hangs up:
> >>>>>>
> >>>>>>
> >>>>>> Do you know which version works well?  If you could bisect, that'll
> >>>>>> help a lot.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>                 Amit
> >>>>>
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> 2.0 works definitely well. I`ll try to finish bisection today, though
> >>>>> every step takes about 10 minutes to complete.
> >>>>
> >>>>
> >>>> Yay! It is even outside of virtio-blk.
> >>>>
> >>>> commit 9b1786829aefb83f37a8f3135e3ea91c56001b56
> >>>> Author: Marcelo Tosatti <mtosatti@redhat.com>
> >>>> Date:   Tue Jun 3 13:34:48 2014 -0300
> >>>>
> >>>>     kvmclock: Ensure proper env->tsc value for kvmclock_current_nsec
> >>>> calculation
> >>>>
> >>>>     Ensure proper env->tsc value for kvmclock_current_nsec calculation.
> >>>>
> >>>>     Reported-by: Marcin Gibuła <m.gibula@beyond.pl>
> >>>>     Cc: qemu-stable@nongnu.org
> >>>>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> >>>>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> >>>
> >>>
> >>> Andrey,
> >>>
> >>> Can you please provide instructions on how to create reproducible
> >>> environment?
> >>>
> >>> The following patch is equivalent to the original patch, for
> >>> the purposes of fixing the kvmclock problem.
> >>>
> >>> Perhaps it becomes easier to spot the reason for the hang you are
> >>> experiencing.
> >>>
> >>>
> >>> diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
> >>> index 272a88a..feb5fc5 100644
> >>> --- a/hw/i386/kvm/clock.c
> >>> +++ b/hw/i386/kvm/clock.c
> >>> @@ -17,7 +17,6 @@
> >>>  #include "qemu/host-utils.h"
> >>>  #include "sysemu/sysemu.h"
> >>>  #include "sysemu/kvm.h"
> >>> -#include "sysemu/cpus.h"
> >>>  #include "hw/sysbus.h"
> >>>  #include "hw/kvm/clock.h"
> >>>
> >>> @@ -66,7 +65,6 @@ static uint64_t kvmclock_current_nsec(KVMClockState *s)
> >>>
> >>>      cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
> >>>
> >>> -    assert(time.tsc_timestamp <= migration_tsc);
> >>>      delta = migration_tsc - time.tsc_timestamp;
> >>>      if (time.tsc_shift < 0) {
> >>>          delta >>= -time.tsc_shift;
> >>> @@ -125,8 +123,6 @@ static void kvmclock_vm_state_change(void *opaque,
> >>> int running,
> >>>          if (s->clock_valid) {
> >>>              return;
> >>>          }
> >>> -
> >>> -        cpu_synchronize_all_states();
> >>>          ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
> >>>          if (ret < 0) {
> >>>              fprintf(stderr, "KVM_GET_CLOCK failed: %s\n",
> >>> strerror(ret));
> >>> diff --git a/migration.c b/migration.c
> >>> index 8d675b3..34f2325 100644
> >>> --- a/migration.c
> >>> +++ b/migration.c
> >>> @@ -608,6 +608,7 @@ static void *migration_thread(void *opaque)
> >>>                  qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
> >>>                  old_vm_running = runstate_is_running();
> >>>
> >>> +                cpu_synchronize_all_states();
> >>>                  ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
> >>>                  if (ret >= 0) {
> >>>                      qemu_file_set_rate_limit(s->file, INT64_MAX);
> >
> >
> > It could also be useful to apply the above patch _and_ revert
> > a096b3a6732f846ec57dc28b47ee9435aa0609bf, then try to reproduce.
> >
> > Paolo
> 
> Yes, it solved the issue for me! (it took much time to check because
> most of country` debian mirrors went inconsistent by some reason)
> 
> Also trivial addition:
> 
> diff --git a/migration.c b/migration.c
> index 34f2325..65d1c88 100644
> --- a/migration.c
> +++ b/migration.c
> @@ -25,6 +25,7 @@
>  #include "qemu/thread.h"
>  #include "qmp-commands.h"
>  #include "trace.h"
> +#include "sysemu/cpus.h"

Can you attach 'git diff' of the qemu tree where the test
was successful please?

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcelo Tosatti]

On Wed, Jul 16, 2014 at 03:40:47AM +0400, Andrey Korolyov wrote:
> On Wed, Jul 16, 2014 at 2:01 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > Il 15/07/2014 23:25, Andrey Korolyov ha scritto:
> >
> >> On Wed, Jul 16, 2014 at 1:09 AM, Marcelo Tosatti <mtosatti@redhat.com>
> >> wrote:
> >>>
> >>> On Tue, Jul 15, 2014 at 06:01:08PM +0400, Andrey Korolyov wrote:
> >>>>
> >>>> On Tue, Jul 15, 2014 at 10:52 AM, Andrey Korolyov <andrey@xdel.ru>
> >>>> wrote:
> >>>>>
> >>>>> On Tue, Jul 15, 2014 at 9:03 AM, Amit Shah <amit.shah@redhat.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> On (Sun) 13 Jul 2014 [16:28:56], Andrey Korolyov wrote:
> >>>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> the issue is not specific to the iothread code because generic
> >>>>>>> virtio-blk also hangs up:
> >>>>>>
> >>>>>>
> >>>>>> Do you know which version works well?  If you could bisect, that'll
> >>>>>> help a lot.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>                 Amit
> >>>>>
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> 2.0 works definitely well. I`ll try to finish bisection today, though
> >>>>> every step takes about 10 minutes to complete.
> >>>>
> >>>>
> >>>> Yay! It is even outside of virtio-blk.
> >>>>
> >>>> commit 9b1786829aefb83f37a8f3135e3ea91c56001b56
> >>>> Author: Marcelo Tosatti <mtosatti@redhat.com>
> >>>> Date:   Tue Jun 3 13:34:48 2014 -0300
> >>>>
> >>>>     kvmclock: Ensure proper env->tsc value for kvmclock_current_nsec
> >>>> calculation
> >>>>
> >>>>     Ensure proper env->tsc value for kvmclock_current_nsec calculation.
> >>>>
> >>>>     Reported-by: Marcin Gibuła <m.gibula@beyond.pl>
> >>>>     Cc: qemu-stable@nongnu.org
> >>>>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> >>>>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> >>>
> >>>
> >>> Andrey,
> >>>
> >>> Can you please provide instructions on how to create reproducible
> >>> environment?
> >>>
> >>> The following patch is equivalent to the original patch, for
> >>> the purposes of fixing the kvmclock problem.
> >>>
> >>> Perhaps it becomes easier to spot the reason for the hang you are
> >>> experiencing.
> >>>
> >>>
> >>> diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
> >>> index 272a88a..feb5fc5 100644
> >>> --- a/hw/i386/kvm/clock.c
> >>> +++ b/hw/i386/kvm/clock.c
> >>> @@ -17,7 +17,6 @@
> >>>  #include "qemu/host-utils.h"
> >>>  #include "sysemu/sysemu.h"
> >>>  #include "sysemu/kvm.h"
> >>> -#include "sysemu/cpus.h"
> >>>  #include "hw/sysbus.h"
> >>>  #include "hw/kvm/clock.h"
> >>>
> >>> @@ -66,7 +65,6 @@ static uint64_t kvmclock_current_nsec(KVMClockState *s)
> >>>
> >>>      cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
> >>>
> >>> -    assert(time.tsc_timestamp <= migration_tsc);
> >>>      delta = migration_tsc - time.tsc_timestamp;
> >>>      if (time.tsc_shift < 0) {
> >>>          delta >>= -time.tsc_shift;
> >>> @@ -125,8 +123,6 @@ static void kvmclock_vm_state_change(void *opaque,
> >>> int running,
> >>>          if (s->clock_valid) {
> >>>              return;
> >>>          }
> >>> -
> >>> -        cpu_synchronize_all_states();
> >>>          ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
> >>>          if (ret < 0) {
> >>>              fprintf(stderr, "KVM_GET_CLOCK failed: %s\n",
> >>> strerror(ret));
> >>> diff --git a/migration.c b/migration.c
> >>> index 8d675b3..34f2325 100644
> >>> --- a/migration.c
> >>> +++ b/migration.c
> >>> @@ -608,6 +608,7 @@ static void *migration_thread(void *opaque)
> >>>                  qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
> >>>                  old_vm_running = runstate_is_running();
> >>>
> >>> +                cpu_synchronize_all_states();
> >>>                  ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
> >>>                  if (ret >= 0) {
> >>>                      qemu_file_set_rate_limit(s->file, INT64_MAX);
> >
> >
> > It could also be useful to apply the above patch _and_ revert
> > a096b3a6732f846ec57dc28b47ee9435aa0609bf, then try to reproduce.
> >
> > Paolo
> 
> Yes, it solved the issue for me! (it took much time to check because
> most of country` debian mirrors went inconsistent by some reason)
> 
> Also trivial addition:
> 
> diff --git a/migration.c b/migration.c
> index 34f2325..65d1c88 100644
> --- a/migration.c
> +++ b/migration.c
> @@ -25,6 +25,7 @@
>  #include "qemu/thread.h"
>  #include "qmp-commands.h"
>  #include "trace.h"
> +#include "sysemu/cpus.h"

And what about not reverting a096b3a6732f846ec57dc28b47ee9435aa0609bf ?

That is, test with a stock qemu.git tree and the patch sent today, 
on this thread, to move cpu_synchronize_all_states ?

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

> Andrey,
>
> Can you please provide instructions on how to create reproducible
> environment?
>
> The following patch is equivalent to the original patch, for
> the purposes of fixing the kvmclock problem.
>
> Perhaps it becomes easier to spot the reason for the hang you are
> experiencing.

Marcelo,

the original reason for patch adding cpu_synchronize_all_states() there 
was because this bug affected non-migration operations as well - 
http://lists.gnu.org/archive/html/qemu-devel/2014-06/msg00472.html.

Won't moving it only to migration code break these things again?

>
> diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
> index 272a88a..feb5fc5 100644
> --- a/hw/i386/kvm/clock.c
> +++ b/hw/i386/kvm/clock.c
> @@ -17,7 +17,6 @@
>   #include "qemu/host-utils.h"
>   #include "sysemu/sysemu.h"
>   #include "sysemu/kvm.h"
> -#include "sysemu/cpus.h"
>   #include "hw/sysbus.h"
>   #include "hw/kvm/clock.h"
>
> @@ -66,7 +65,6 @@ static uint64_t kvmclock_current_nsec(KVMClockState *s)
>
>       cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
>
> -    assert(time.tsc_timestamp <= migration_tsc);
>       delta = migration_tsc - time.tsc_timestamp;
>       if (time.tsc_shift < 0) {
>           delta >>= -time.tsc_shift;
> @@ -125,8 +123,6 @@ static void kvmclock_vm_state_change(void *opaque, int running,
>           if (s->clock_valid) {
>               return;
>           }
> -
> -        cpu_synchronize_all_states();
>           ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
>           if (ret < 0) {
>               fprintf(stderr, "KVM_GET_CLOCK failed: %s\n", strerror(ret));
> diff --git a/migration.c b/migration.c
> index 8d675b3..34f2325 100644
> --- a/migration.c
> +++ b/migration.c
> @@ -608,6 +608,7 @@ static void *migration_thread(void *opaque)
>                   qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
>                   old_vm_running = runstate_is_running();
>
> +                cpu_synchronize_all_states();
>                   ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
>                   if (ret >= 0) {
>                       qemu_file_set_rate_limit(s->file, INT64_MAX);
>


-- 
mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

[-- Attachment #1: Type: text/plain, Size: 5507 bytes --]

On Wed, Jul 16, 2014 at 5:16 AM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> On Wed, Jul 16, 2014 at 03:40:47AM +0400, Andrey Korolyov wrote:
>> On Wed, Jul 16, 2014 at 2:01 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> > Il 15/07/2014 23:25, Andrey Korolyov ha scritto:
>> >
>> >> On Wed, Jul 16, 2014 at 1:09 AM, Marcelo Tosatti <mtosatti@redhat.com>
>> >> wrote:
>> >>>
>> >>> On Tue, Jul 15, 2014 at 06:01:08PM +0400, Andrey Korolyov wrote:
>> >>>>
>> >>>> On Tue, Jul 15, 2014 at 10:52 AM, Andrey Korolyov <andrey@xdel.ru>
>> >>>> wrote:
>> >>>>>
>> >>>>> On Tue, Jul 15, 2014 at 9:03 AM, Amit Shah <amit.shah@redhat.com>
>> >>>>> wrote:
>> >>>>>>
>> >>>>>> On (Sun) 13 Jul 2014 [16:28:56], Andrey Korolyov wrote:
>> >>>>>>>
>> >>>>>>> Hello,
>> >>>>>>>
>> >>>>>>> the issue is not specific to the iothread code because generic
>> >>>>>>> virtio-blk also hangs up:
>> >>>>>>
>> >>>>>>
>> >>>>>> Do you know which version works well?  If you could bisect, that'll
>> >>>>>> help a lot.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>>                 Amit
>> >>>>>
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> 2.0 works definitely well. I`ll try to finish bisection today, though
>> >>>>> every step takes about 10 minutes to complete.
>> >>>>
>> >>>>
>> >>>> Yay! It is even outside of virtio-blk.
>> >>>>
>> >>>> commit 9b1786829aefb83f37a8f3135e3ea91c56001b56
>> >>>> Author: Marcelo Tosatti <mtosatti@redhat.com>
>> >>>> Date:   Tue Jun 3 13:34:48 2014 -0300
>> >>>>
>> >>>>     kvmclock: Ensure proper env->tsc value for kvmclock_current_nsec
>> >>>> calculation
>> >>>>
>> >>>>     Ensure proper env->tsc value for kvmclock_current_nsec calculation.
>> >>>>
>> >>>>     Reported-by: Marcin Gibuła <m.gibula@beyond.pl>
>> >>>>     Cc: qemu-stable@nongnu.org
>> >>>>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>> >>>>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>> >>>
>> >>>
>> >>> Andrey,
>> >>>
>> >>> Can you please provide instructions on how to create reproducible
>> >>> environment?
>> >>>
>> >>> The following patch is equivalent to the original patch, for
>> >>> the purposes of fixing the kvmclock problem.
>> >>>
>> >>> Perhaps it becomes easier to spot the reason for the hang you are
>> >>> experiencing.
>> >>>
>> >>>
>> >>> diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
>> >>> index 272a88a..feb5fc5 100644
>> >>> --- a/hw/i386/kvm/clock.c
>> >>> +++ b/hw/i386/kvm/clock.c
>> >>> @@ -17,7 +17,6 @@
>> >>>  #include "qemu/host-utils.h"
>> >>>  #include "sysemu/sysemu.h"
>> >>>  #include "sysemu/kvm.h"
>> >>> -#include "sysemu/cpus.h"
>> >>>  #include "hw/sysbus.h"
>> >>>  #include "hw/kvm/clock.h"
>> >>>
>> >>> @@ -66,7 +65,6 @@ static uint64_t kvmclock_current_nsec(KVMClockState *s)
>> >>>
>> >>>      cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
>> >>>
>> >>> -    assert(time.tsc_timestamp <= migration_tsc);
>> >>>      delta = migration_tsc - time.tsc_timestamp;
>> >>>      if (time.tsc_shift < 0) {
>> >>>          delta >>= -time.tsc_shift;
>> >>> @@ -125,8 +123,6 @@ static void kvmclock_vm_state_change(void *opaque,
>> >>> int running,
>> >>>          if (s->clock_valid) {
>> >>>              return;
>> >>>          }
>> >>> -
>> >>> -        cpu_synchronize_all_states();
>> >>>          ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
>> >>>          if (ret < 0) {
>> >>>              fprintf(stderr, "KVM_GET_CLOCK failed: %s\n",
>> >>> strerror(ret));
>> >>> diff --git a/migration.c b/migration.c
>> >>> index 8d675b3..34f2325 100644
>> >>> --- a/migration.c
>> >>> +++ b/migration.c
>> >>> @@ -608,6 +608,7 @@ static void *migration_thread(void *opaque)
>> >>>                  qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
>> >>>                  old_vm_running = runstate_is_running();
>> >>>
>> >>> +                cpu_synchronize_all_states();
>> >>>                  ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
>> >>>                  if (ret >= 0) {
>> >>>                      qemu_file_set_rate_limit(s->file, INT64_MAX);
>> >
>> >
>> > It could also be useful to apply the above patch _and_ revert
>> > a096b3a6732f846ec57dc28b47ee9435aa0609bf, then try to reproduce.
>> >
>> > Paolo
>>
>> Yes, it solved the issue for me! (it took much time to check because
>> most of country` debian mirrors went inconsistent by some reason)
>>
>> Also trivial addition:
>>
>> diff --git a/migration.c b/migration.c
>> index 34f2325..65d1c88 100644
>> --- a/migration.c
>> +++ b/migration.c
>> @@ -25,6 +25,7 @@
>>  #include "qemu/thread.h"
>>  #include "qmp-commands.h"
>>  #include "trace.h"
>> +#include "sysemu/cpus.h"
>
> And what about not reverting a096b3a6732f846ec57dc28b47ee9435aa0609bf ?
>
> That is, test with a stock qemu.git tree and the patch sent today,
> on this thread, to move cpu_synchronize_all_states ?
>
>

The main reason for things to work for me is a revert of
9b1786829aefb83f37a8f3135e3ea91c56001b56 on top, not adding any other
patches. I had tested two cases, with Alexander`s patch completely
reverted plus suggestion from Marcelo and only with revert 9b178682
plug same suggestion. The difference is that the until Alexander`
patch is not reverted, live migration is always failing by the timeout
value, and when reverted migration always succeeds in 8-10 seconds.
Appropriate diffs are attached for the reference.

[-- Attachment #2: diff-with-reverted-agraf-patch.txt --]
[-- Type: text/plain, Size: 3271 bytes --]

diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
index 272a88a..93e1829 100644
--- a/hw/i386/kvm/clock.c
+++ b/hw/i386/kvm/clock.c
@@ -17,7 +17,6 @@
 #include "qemu/host-utils.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/kvm.h"
-#include "sysemu/cpus.h"
 #include "hw/sysbus.h"
 #include "hw/kvm/clock.h"
 
@@ -36,49 +35,6 @@ typedef struct KVMClockState {
     bool clock_valid;
 } KVMClockState;
 
-struct pvclock_vcpu_time_info {
-    uint32_t   version;
-    uint32_t   pad0;
-    uint64_t   tsc_timestamp;
-    uint64_t   system_time;
-    uint32_t   tsc_to_system_mul;
-    int8_t     tsc_shift;
-    uint8_t    flags;
-    uint8_t    pad[2];
-} __attribute__((__packed__)); /* 32 bytes */
-
-static uint64_t kvmclock_current_nsec(KVMClockState *s)
-{
-    CPUState *cpu = first_cpu;
-    CPUX86State *env = cpu->env_ptr;
-    hwaddr kvmclock_struct_pa = env->system_time_msr & ~1ULL;
-    uint64_t migration_tsc = env->tsc;
-    struct pvclock_vcpu_time_info time;
-    uint64_t delta;
-    uint64_t nsec_lo;
-    uint64_t nsec_hi;
-    uint64_t nsec;
-
-    if (!(env->system_time_msr & 1ULL)) {
-        /* KVM clock not active */
-        return 0;
-    }
-
-    cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
-
-    assert(time.tsc_timestamp <= migration_tsc);
-    delta = migration_tsc - time.tsc_timestamp;
-    if (time.tsc_shift < 0) {
-        delta >>= -time.tsc_shift;
-    } else {
-        delta <<= time.tsc_shift;
-    }
-
-    mulu64(&nsec_lo, &nsec_hi, delta, time.tsc_to_system_mul);
-    nsec = (nsec_lo >> 32) | (nsec_hi << 32);
-    return nsec + time.system_time;
-}
-
 static void kvmclock_vm_state_change(void *opaque, int running,
                                      RunState state)
 {
@@ -89,15 +45,9 @@ static void kvmclock_vm_state_change(void *opaque, int running,
 
     if (running) {
         struct kvm_clock_data data;
-        uint64_t time_at_migration = kvmclock_current_nsec(s);
 
         s->clock_valid = false;
 
-	/* We can't rely on the migrated clock value, just discard it */
-	if (time_at_migration) {
-	        s->clock = time_at_migration;
-	}
-
         data.clock = s->clock;
         data.flags = 0;
         ret = kvm_vm_ioctl(kvm_state, KVM_SET_CLOCK, &data);
@@ -125,8 +75,6 @@ static void kvmclock_vm_state_change(void *opaque, int running,
         if (s->clock_valid) {
             return;
         }
-
-        cpu_synchronize_all_states();
         ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
         if (ret < 0) {
             fprintf(stderr, "KVM_GET_CLOCK failed: %s\n", strerror(ret));
diff --git a/migration.c b/migration.c
index 8d675b3..65d1c88 100644
--- a/migration.c
+++ b/migration.c
@@ -25,6 +25,7 @@
 #include "qemu/thread.h"
 #include "qmp-commands.h"
 #include "trace.h"
+#include "sysemu/cpus.h"
 
 enum {
     MIG_STATE_ERROR = -1,
@@ -608,6 +609,7 @@ static void *migration_thread(void *opaque)
                 qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
                 old_vm_running = runstate_is_running();
 
+                cpu_synchronize_all_states();
                 ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
                 if (ret >= 0) {
                     qemu_file_set_rate_limit(s->file, INT64_MAX);

[-- Attachment #3: diff-with-only-late-fix-moved.txt --]
[-- Type: text/plain, Size: 1652 bytes --]

diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
index 272a88a..feb5fc5 100644
--- a/hw/i386/kvm/clock.c
+++ b/hw/i386/kvm/clock.c
@@ -17,7 +17,6 @@
 #include "qemu/host-utils.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/kvm.h"
-#include "sysemu/cpus.h"
 #include "hw/sysbus.h"
 #include "hw/kvm/clock.h"
 
@@ -66,7 +65,6 @@ static uint64_t kvmclock_current_nsec(KVMClockState *s)
 
     cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
 
-    assert(time.tsc_timestamp <= migration_tsc);
     delta = migration_tsc - time.tsc_timestamp;
     if (time.tsc_shift < 0) {
         delta >>= -time.tsc_shift;
@@ -125,8 +123,6 @@ static void kvmclock_vm_state_change(void *opaque, int running,
         if (s->clock_valid) {
             return;
         }
-
-        cpu_synchronize_all_states();
         ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
         if (ret < 0) {
             fprintf(stderr, "KVM_GET_CLOCK failed: %s\n", strerror(ret));
diff --git a/migration.c b/migration.c
index 8d675b3..65d1c88 100644
--- a/migration.c
+++ b/migration.c
@@ -25,6 +25,7 @@
 #include "qemu/thread.h"
 #include "qmp-commands.h"
 #include "trace.h"
+#include "sysemu/cpus.h"
 
 enum {
     MIG_STATE_ERROR = -1,
@@ -608,6 +609,7 @@ static void *migration_thread(void *opaque)
                 qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
                 old_vm_running = runstate_is_running();
 
+                cpu_synchronize_all_states();
                 ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
                 if (ret >= 0) {
                     qemu_file_set_rate_limit(s->file, INT64_MAX);

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcelo Tosatti]

[-- Attachment #1: Type: text/plain, Size: 6252 bytes --]

On Wed, Jul 16, 2014 at 12:38:51PM +0400, Andrey Korolyov wrote:
> On Wed, Jul 16, 2014 at 5:16 AM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> > On Wed, Jul 16, 2014 at 03:40:47AM +0400, Andrey Korolyov wrote:
> >> On Wed, Jul 16, 2014 at 2:01 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >> > Il 15/07/2014 23:25, Andrey Korolyov ha scritto:
> >> >
> >> >> On Wed, Jul 16, 2014 at 1:09 AM, Marcelo Tosatti <mtosatti@redhat.com>
> >> >> wrote:
> >> >>>
> >> >>> On Tue, Jul 15, 2014 at 06:01:08PM +0400, Andrey Korolyov wrote:
> >> >>>>
> >> >>>> On Tue, Jul 15, 2014 at 10:52 AM, Andrey Korolyov <andrey@xdel.ru>
> >> >>>> wrote:
> >> >>>>>
> >> >>>>> On Tue, Jul 15, 2014 at 9:03 AM, Amit Shah <amit.shah@redhat.com>
> >> >>>>> wrote:
> >> >>>>>>
> >> >>>>>> On (Sun) 13 Jul 2014 [16:28:56], Andrey Korolyov wrote:
> >> >>>>>>>
> >> >>>>>>> Hello,
> >> >>>>>>>
> >> >>>>>>> the issue is not specific to the iothread code because generic
> >> >>>>>>> virtio-blk also hangs up:
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> Do you know which version works well?  If you could bisect, that'll
> >> >>>>>> help a lot.
> >> >>>>>>
> >> >>>>>> Thanks,
> >> >>>>>>                 Amit
> >> >>>>>
> >> >>>>>
> >> >>>>> Hi,
> >> >>>>>
> >> >>>>> 2.0 works definitely well. I`ll try to finish bisection today, though
> >> >>>>> every step takes about 10 minutes to complete.
> >> >>>>
> >> >>>>
> >> >>>> Yay! It is even outside of virtio-blk.
> >> >>>>
> >> >>>> commit 9b1786829aefb83f37a8f3135e3ea91c56001b56
> >> >>>> Author: Marcelo Tosatti <mtosatti@redhat.com>
> >> >>>> Date:   Tue Jun 3 13:34:48 2014 -0300
> >> >>>>
> >> >>>>     kvmclock: Ensure proper env->tsc value for kvmclock_current_nsec
> >> >>>> calculation
> >> >>>>
> >> >>>>     Ensure proper env->tsc value for kvmclock_current_nsec calculation.
> >> >>>>
> >> >>>>     Reported-by: Marcin Gibuła <m.gibula@beyond.pl>
> >> >>>>     Cc: qemu-stable@nongnu.org
> >> >>>>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> >> >>>>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> >> >>>
> >> >>>
> >> >>> Andrey,
> >> >>>
> >> >>> Can you please provide instructions on how to create reproducible
> >> >>> environment?
> >> >>>
> >> >>> The following patch is equivalent to the original patch, for
> >> >>> the purposes of fixing the kvmclock problem.
> >> >>>
> >> >>> Perhaps it becomes easier to spot the reason for the hang you are
> >> >>> experiencing.
> >> >>>
> >> >>>
> >> >>> diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
> >> >>> index 272a88a..feb5fc5 100644
> >> >>> --- a/hw/i386/kvm/clock.c
> >> >>> +++ b/hw/i386/kvm/clock.c
> >> >>> @@ -17,7 +17,6 @@
> >> >>>  #include "qemu/host-utils.h"
> >> >>>  #include "sysemu/sysemu.h"
> >> >>>  #include "sysemu/kvm.h"
> >> >>> -#include "sysemu/cpus.h"
> >> >>>  #include "hw/sysbus.h"
> >> >>>  #include "hw/kvm/clock.h"
> >> >>>
> >> >>> @@ -66,7 +65,6 @@ static uint64_t kvmclock_current_nsec(KVMClockState *s)
> >> >>>
> >> >>>      cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
> >> >>>
> >> >>> -    assert(time.tsc_timestamp <= migration_tsc);
> >> >>>      delta = migration_tsc - time.tsc_timestamp;
> >> >>>      if (time.tsc_shift < 0) {
> >> >>>          delta >>= -time.tsc_shift;
> >> >>> @@ -125,8 +123,6 @@ static void kvmclock_vm_state_change(void *opaque,
> >> >>> int running,
> >> >>>          if (s->clock_valid) {
> >> >>>              return;
> >> >>>          }
> >> >>> -
> >> >>> -        cpu_synchronize_all_states();
> >> >>>          ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
> >> >>>          if (ret < 0) {
> >> >>>              fprintf(stderr, "KVM_GET_CLOCK failed: %s\n",
> >> >>> strerror(ret));
> >> >>> diff --git a/migration.c b/migration.c
> >> >>> index 8d675b3..34f2325 100644
> >> >>> --- a/migration.c
> >> >>> +++ b/migration.c
> >> >>> @@ -608,6 +608,7 @@ static void *migration_thread(void *opaque)
> >> >>>                  qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
> >> >>>                  old_vm_running = runstate_is_running();
> >> >>>
> >> >>> +                cpu_synchronize_all_states();
> >> >>>                  ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
> >> >>>                  if (ret >= 0) {
> >> >>>                      qemu_file_set_rate_limit(s->file, INT64_MAX);
> >> >
> >> >
> >> > It could also be useful to apply the above patch _and_ revert
> >> > a096b3a6732f846ec57dc28b47ee9435aa0609bf, then try to reproduce.
> >> >
> >> > Paolo
> >>
> >> Yes, it solved the issue for me! (it took much time to check because
> >> most of country` debian mirrors went inconsistent by some reason)
> >>
> >> Also trivial addition:
> >>
> >> diff --git a/migration.c b/migration.c
> >> index 34f2325..65d1c88 100644
> >> --- a/migration.c
> >> +++ b/migration.c
> >> @@ -25,6 +25,7 @@
> >>  #include "qemu/thread.h"
> >>  #include "qmp-commands.h"
> >>  #include "trace.h"
> >> +#include "sysemu/cpus.h"
> >
> > And what about not reverting a096b3a6732f846ec57dc28b47ee9435aa0609bf ?
> >
> > That is, test with a stock qemu.git tree and the patch sent today,
> > on this thread, to move cpu_synchronize_all_states ?
> >
> >
> 
> The main reason for things to work for me is a revert of
> 9b1786829aefb83f37a8f3135e3ea91c56001b56 on top, not adding any other
> patches. I had tested two cases, with Alexander`s patch completely
> reverted plus suggestion from Marcelo and only with revert 9b178682
> plug same suggestion. The difference is that the until Alexander`
> patch is not reverted, live migration is always failing by the timeout
> value, and when reverted migration always succeeds in 8-10 seconds.
> Appropriate diffs are attached for the reference.

Andrey,

Can you please apply only the following attached patch to an upstream
QEMU git tree (move_synchronize_all_states.patch), plus the necessary
header file corrections, and attempt to reproduce?

When you reproduce, please provide a backtrace and version of the QEMU
git tree, and instructions on how to reproduce:

1) full QEMU command line
2) steps to reproduce
 


[-- Attachment #2: move_synchronize_all_states.patch --]
[-- Type: text/plain, Size: 1549 bytes --]


Move cpu_synchronize_all_states to migration.c.

diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
index 272a88a..feb5fc5 100644
--- a/hw/i386/kvm/clock.c
+++ b/hw/i386/kvm/clock.c
@@ -17,7 +17,6 @@
 #include "qemu/host-utils.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/kvm.h"
-#include "sysemu/cpus.h"
 #include "hw/sysbus.h"
 #include "hw/kvm/clock.h"
 
@@ -66,7 +65,6 @@ static uint64_t kvmclock_current_nsec(KVMClockState *s)
 
     cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
 
-    assert(time.tsc_timestamp <= migration_tsc);
     delta = migration_tsc - time.tsc_timestamp;
     if (time.tsc_shift < 0) {
         delta >>= -time.tsc_shift;
@@ -125,8 +123,6 @@ static void kvmclock_vm_state_change(void *opaque, int running,
         if (s->clock_valid) {
             return;
         }
-
-        cpu_synchronize_all_states();
         ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
         if (ret < 0) {
             fprintf(stderr, "KVM_GET_CLOCK failed: %s\n", strerror(ret));
diff --git a/migration.c b/migration.c
index 8d675b3..34f2325 100644
--- a/migration.c
+++ b/migration.c
@@ -608,6 +608,7 @@ static void *migration_thread(void *opaque)
                 qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
                 old_vm_running = runstate_is_running();
 
+                cpu_synchronize_all_states();
                 ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
                 if (ret >= 0) {
                     qemu_file_set_rate_limit(s->file, INT64_MAX);

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcelo Tosatti]

On Wed, Jul 16, 2014 at 09:35:16AM +0200, Marcin Gibuła wrote:
> >Andrey,
> >
> >Can you please provide instructions on how to create reproducible
> >environment?
> >
> >The following patch is equivalent to the original patch, for
> >the purposes of fixing the kvmclock problem.
> >
> >Perhaps it becomes easier to spot the reason for the hang you are
> >experiencing.
> 
> Marcelo,
> 
> the original reason for patch adding cpu_synchronize_all_states()
> there was because this bug affected non-migration operations as well
> -
> http://lists.gnu.org/archive/html/qemu-devel/2014-06/msg00472.html.
> 
> Won't moving it only to migration code break these things again?

Yes - its just for debug purposes.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Wed, Jul 16, 2014 at 3:52 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> On Wed, Jul 16, 2014 at 12:38:51PM +0400, Andrey Korolyov wrote:
>> On Wed, Jul 16, 2014 at 5:16 AM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
>> > On Wed, Jul 16, 2014 at 03:40:47AM +0400, Andrey Korolyov wrote:
>> >> On Wed, Jul 16, 2014 at 2:01 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> >> > Il 15/07/2014 23:25, Andrey Korolyov ha scritto:
>> >> >
>> >> >> On Wed, Jul 16, 2014 at 1:09 AM, Marcelo Tosatti <mtosatti@redhat.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> On Tue, Jul 15, 2014 at 06:01:08PM +0400, Andrey Korolyov wrote:
>> >> >>>>
>> >> >>>> On Tue, Jul 15, 2014 at 10:52 AM, Andrey Korolyov <andrey@xdel.ru>
>> >> >>>> wrote:
>> >> >>>>>
>> >> >>>>> On Tue, Jul 15, 2014 at 9:03 AM, Amit Shah <amit.shah@redhat.com>
>> >> >>>>> wrote:
>> >> >>>>>>
>> >> >>>>>> On (Sun) 13 Jul 2014 [16:28:56], Andrey Korolyov wrote:
>> >> >>>>>>>
>> >> >>>>>>> Hello,
>> >> >>>>>>>
>> >> >>>>>>> the issue is not specific to the iothread code because generic
>> >> >>>>>>> virtio-blk also hangs up:
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>> Do you know which version works well?  If you could bisect, that'll
>> >> >>>>>> help a lot.
>> >> >>>>>>
>> >> >>>>>> Thanks,
>> >> >>>>>>                 Amit
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Hi,
>> >> >>>>>
>> >> >>>>> 2.0 works definitely well. I`ll try to finish bisection today, though
>> >> >>>>> every step takes about 10 minutes to complete.
>> >> >>>>
>> >> >>>>
>> >> >>>> Yay! It is even outside of virtio-blk.
>> >> >>>>
>> >> >>>> commit 9b1786829aefb83f37a8f3135e3ea91c56001b56
>> >> >>>> Author: Marcelo Tosatti <mtosatti@redhat.com>
>> >> >>>> Date:   Tue Jun 3 13:34:48 2014 -0300
>> >> >>>>
>> >> >>>>     kvmclock: Ensure proper env->tsc value for kvmclock_current_nsec
>> >> >>>> calculation
>> >> >>>>
>> >> >>>>     Ensure proper env->tsc value for kvmclock_current_nsec calculation.
>> >> >>>>
>> >> >>>>     Reported-by: Marcin Gibuła <m.gibula@beyond.pl>
>> >> >>>>     Cc: qemu-stable@nongnu.org
>> >> >>>>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>> >> >>>>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>> >> >>>
>> >> >>>
>> >> >>> Andrey,
>> >> >>>
>> >> >>> Can you please provide instructions on how to create reproducible
>> >> >>> environment?
>> >> >>>
>> >> >>> The following patch is equivalent to the original patch, for
>> >> >>> the purposes of fixing the kvmclock problem.
>> >> >>>
>> >> >>> Perhaps it becomes easier to spot the reason for the hang you are
>> >> >>> experiencing.
>> >> >>>
>> >> >>>
>> >> >>> diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
>> >> >>> index 272a88a..feb5fc5 100644
>> >> >>> --- a/hw/i386/kvm/clock.c
>> >> >>> +++ b/hw/i386/kvm/clock.c
>> >> >>> @@ -17,7 +17,6 @@
>> >> >>>  #include "qemu/host-utils.h"
>> >> >>>  #include "sysemu/sysemu.h"
>> >> >>>  #include "sysemu/kvm.h"
>> >> >>> -#include "sysemu/cpus.h"
>> >> >>>  #include "hw/sysbus.h"
>> >> >>>  #include "hw/kvm/clock.h"
>> >> >>>
>> >> >>> @@ -66,7 +65,6 @@ static uint64_t kvmclock_current_nsec(KVMClockState *s)
>> >> >>>
>> >> >>>      cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
>> >> >>>
>> >> >>> -    assert(time.tsc_timestamp <= migration_tsc);
>> >> >>>      delta = migration_tsc - time.tsc_timestamp;
>> >> >>>      if (time.tsc_shift < 0) {
>> >> >>>          delta >>= -time.tsc_shift;
>> >> >>> @@ -125,8 +123,6 @@ static void kvmclock_vm_state_change(void *opaque,
>> >> >>> int running,
>> >> >>>          if (s->clock_valid) {
>> >> >>>              return;
>> >> >>>          }
>> >> >>> -
>> >> >>> -        cpu_synchronize_all_states();
>> >> >>>          ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
>> >> >>>          if (ret < 0) {
>> >> >>>              fprintf(stderr, "KVM_GET_CLOCK failed: %s\n",
>> >> >>> strerror(ret));
>> >> >>> diff --git a/migration.c b/migration.c
>> >> >>> index 8d675b3..34f2325 100644
>> >> >>> --- a/migration.c
>> >> >>> +++ b/migration.c
>> >> >>> @@ -608,6 +608,7 @@ static void *migration_thread(void *opaque)
>> >> >>>                  qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
>> >> >>>                  old_vm_running = runstate_is_running();
>> >> >>>
>> >> >>> +                cpu_synchronize_all_states();
>> >> >>>                  ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
>> >> >>>                  if (ret >= 0) {
>> >> >>>                      qemu_file_set_rate_limit(s->file, INT64_MAX);
>> >> >
>> >> >
>> >> > It could also be useful to apply the above patch _and_ revert
>> >> > a096b3a6732f846ec57dc28b47ee9435aa0609bf, then try to reproduce.
>> >> >
>> >> > Paolo
>> >>
>> >> Yes, it solved the issue for me! (it took much time to check because
>> >> most of country` debian mirrors went inconsistent by some reason)
>> >>
>> >> Also trivial addition:
>> >>
>> >> diff --git a/migration.c b/migration.c
>> >> index 34f2325..65d1c88 100644
>> >> --- a/migration.c
>> >> +++ b/migration.c
>> >> @@ -25,6 +25,7 @@
>> >>  #include "qemu/thread.h"
>> >>  #include "qmp-commands.h"
>> >>  #include "trace.h"
>> >> +#include "sysemu/cpus.h"
>> >
>> > And what about not reverting a096b3a6732f846ec57dc28b47ee9435aa0609bf ?
>> >
>> > That is, test with a stock qemu.git tree and the patch sent today,
>> > on this thread, to move cpu_synchronize_all_states ?
>> >
>> >
>>
>> The main reason for things to work for me is a revert of
>> 9b1786829aefb83f37a8f3135e3ea91c56001b56 on top, not adding any other
>> patches. I had tested two cases, with Alexander`s patch completely
>> reverted plus suggestion from Marcelo and only with revert 9b178682
>> plug same suggestion. The difference is that the until Alexander`
>> patch is not reverted, live migration is always failing by the timeout
>> value, and when reverted migration always succeeds in 8-10 seconds.
>> Appropriate diffs are attached for the reference.
>
> Andrey,
>
> Can you please apply only the following attached patch to an upstream
> QEMU git tree (move_synchronize_all_states.patch), plus the necessary
> header file corrections, and attempt to reproduce?
>
> When you reproduce, please provide a backtrace and version of the QEMU
> git tree, and instructions on how to reproduce:
>
> 1) full QEMU command line
> 2) steps to reproduce
>
>

Marcelo, as I can see, this patch resembles second case from my
previous message exactly (there are diffs from the generic rc). I/O is
not locking up there but live migration failing and libvirt moves a
freezed state. I can try to run the same on top of rc2, but it`ll be
probably the same.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

[-- Attachment #1: Type: text/plain, Size: 7899 bytes --]

On Wed, Jul 16, 2014 at 5:24 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> On Wed, Jul 16, 2014 at 3:52 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
>> On Wed, Jul 16, 2014 at 12:38:51PM +0400, Andrey Korolyov wrote:
>>> On Wed, Jul 16, 2014 at 5:16 AM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
>>> > On Wed, Jul 16, 2014 at 03:40:47AM +0400, Andrey Korolyov wrote:
>>> >> On Wed, Jul 16, 2014 at 2:01 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>> >> > Il 15/07/2014 23:25, Andrey Korolyov ha scritto:
>>> >> >
>>> >> >> On Wed, Jul 16, 2014 at 1:09 AM, Marcelo Tosatti <mtosatti@redhat.com>
>>> >> >> wrote:
>>> >> >>>
>>> >> >>> On Tue, Jul 15, 2014 at 06:01:08PM +0400, Andrey Korolyov wrote:
>>> >> >>>>
>>> >> >>>> On Tue, Jul 15, 2014 at 10:52 AM, Andrey Korolyov <andrey@xdel.ru>
>>> >> >>>> wrote:
>>> >> >>>>>
>>> >> >>>>> On Tue, Jul 15, 2014 at 9:03 AM, Amit Shah <amit.shah@redhat.com>
>>> >> >>>>> wrote:
>>> >> >>>>>>
>>> >> >>>>>> On (Sun) 13 Jul 2014 [16:28:56], Andrey Korolyov wrote:
>>> >> >>>>>>>
>>> >> >>>>>>> Hello,
>>> >> >>>>>>>
>>> >> >>>>>>> the issue is not specific to the iothread code because generic
>>> >> >>>>>>> virtio-blk also hangs up:
>>> >> >>>>>>
>>> >> >>>>>>
>>> >> >>>>>> Do you know which version works well?  If you could bisect, that'll
>>> >> >>>>>> help a lot.
>>> >> >>>>>>
>>> >> >>>>>> Thanks,
>>> >> >>>>>>                 Amit
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> Hi,
>>> >> >>>>>
>>> >> >>>>> 2.0 works definitely well. I`ll try to finish bisection today, though
>>> >> >>>>> every step takes about 10 minutes to complete.
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> Yay! It is even outside of virtio-blk.
>>> >> >>>>
>>> >> >>>> commit 9b1786829aefb83f37a8f3135e3ea91c56001b56
>>> >> >>>> Author: Marcelo Tosatti <mtosatti@redhat.com>
>>> >> >>>> Date:   Tue Jun 3 13:34:48 2014 -0300
>>> >> >>>>
>>> >> >>>>     kvmclock: Ensure proper env->tsc value for kvmclock_current_nsec
>>> >> >>>> calculation
>>> >> >>>>
>>> >> >>>>     Ensure proper env->tsc value for kvmclock_current_nsec calculation.
>>> >> >>>>
>>> >> >>>>     Reported-by: Marcin Gibuła <m.gibula@beyond.pl>
>>> >> >>>>     Cc: qemu-stable@nongnu.org
>>> >> >>>>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>>> >> >>>>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>>> >> >>>
>>> >> >>>
>>> >> >>> Andrey,
>>> >> >>>
>>> >> >>> Can you please provide instructions on how to create reproducible
>>> >> >>> environment?
>>> >> >>>
>>> >> >>> The following patch is equivalent to the original patch, for
>>> >> >>> the purposes of fixing the kvmclock problem.
>>> >> >>>
>>> >> >>> Perhaps it becomes easier to spot the reason for the hang you are
>>> >> >>> experiencing.
>>> >> >>>
>>> >> >>>
>>> >> >>> diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
>>> >> >>> index 272a88a..feb5fc5 100644
>>> >> >>> --- a/hw/i386/kvm/clock.c
>>> >> >>> +++ b/hw/i386/kvm/clock.c
>>> >> >>> @@ -17,7 +17,6 @@
>>> >> >>>  #include "qemu/host-utils.h"
>>> >> >>>  #include "sysemu/sysemu.h"
>>> >> >>>  #include "sysemu/kvm.h"
>>> >> >>> -#include "sysemu/cpus.h"
>>> >> >>>  #include "hw/sysbus.h"
>>> >> >>>  #include "hw/kvm/clock.h"
>>> >> >>>
>>> >> >>> @@ -66,7 +65,6 @@ static uint64_t kvmclock_current_nsec(KVMClockState *s)
>>> >> >>>
>>> >> >>>      cpu_physical_memory_read(kvmclock_struct_pa, &time, sizeof(time));
>>> >> >>>
>>> >> >>> -    assert(time.tsc_timestamp <= migration_tsc);
>>> >> >>>      delta = migration_tsc - time.tsc_timestamp;
>>> >> >>>      if (time.tsc_shift < 0) {
>>> >> >>>          delta >>= -time.tsc_shift;
>>> >> >>> @@ -125,8 +123,6 @@ static void kvmclock_vm_state_change(void *opaque,
>>> >> >>> int running,
>>> >> >>>          if (s->clock_valid) {
>>> >> >>>              return;
>>> >> >>>          }
>>> >> >>> -
>>> >> >>> -        cpu_synchronize_all_states();
>>> >> >>>          ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
>>> >> >>>          if (ret < 0) {
>>> >> >>>              fprintf(stderr, "KVM_GET_CLOCK failed: %s\n",
>>> >> >>> strerror(ret));
>>> >> >>> diff --git a/migration.c b/migration.c
>>> >> >>> index 8d675b3..34f2325 100644
>>> >> >>> --- a/migration.c
>>> >> >>> +++ b/migration.c
>>> >> >>> @@ -608,6 +608,7 @@ static void *migration_thread(void *opaque)
>>> >> >>>                  qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
>>> >> >>>                  old_vm_running = runstate_is_running();
>>> >> >>>
>>> >> >>> +                cpu_synchronize_all_states();
>>> >> >>>                  ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
>>> >> >>>                  if (ret >= 0) {
>>> >> >>>                      qemu_file_set_rate_limit(s->file, INT64_MAX);
>>> >> >
>>> >> >
>>> >> > It could also be useful to apply the above patch _and_ revert
>>> >> > a096b3a6732f846ec57dc28b47ee9435aa0609bf, then try to reproduce.
>>> >> >
>>> >> > Paolo
>>> >>
>>> >> Yes, it solved the issue for me! (it took much time to check because
>>> >> most of country` debian mirrors went inconsistent by some reason)
>>> >>
>>> >> Also trivial addition:
>>> >>
>>> >> diff --git a/migration.c b/migration.c
>>> >> index 34f2325..65d1c88 100644
>>> >> --- a/migration.c
>>> >> +++ b/migration.c
>>> >> @@ -25,6 +25,7 @@
>>> >>  #include "qemu/thread.h"
>>> >>  #include "qmp-commands.h"
>>> >>  #include "trace.h"
>>> >> +#include "sysemu/cpus.h"
>>> >
>>> > And what about not reverting a096b3a6732f846ec57dc28b47ee9435aa0609bf ?
>>> >
>>> > That is, test with a stock qemu.git tree and the patch sent today,
>>> > on this thread, to move cpu_synchronize_all_states ?
>>> >
>>> >
>>>
>>> The main reason for things to work for me is a revert of
>>> 9b1786829aefb83f37a8f3135e3ea91c56001b56 on top, not adding any other
>>> patches. I had tested two cases, with Alexander`s patch completely
>>> reverted plus suggestion from Marcelo and only with revert 9b178682
>>> plug same suggestion. The difference is that the until Alexander`
>>> patch is not reverted, live migration is always failing by the timeout
>>> value, and when reverted migration always succeeds in 8-10 seconds.
>>> Appropriate diffs are attached for the reference.
>>
>> Andrey,
>>
>> Can you please apply only the following attached patch to an upstream
>> QEMU git tree (move_synchronize_all_states.patch), plus the necessary
>> header file corrections, and attempt to reproduce?
>>
>> When you reproduce, please provide a backtrace and version of the QEMU
>> git tree, and instructions on how to reproduce:
>>
>> 1) full QEMU command line
>> 2) steps to reproduce
>>
>>
>
> Marcelo, as I can see, this patch resembles second case from my
> previous message exactly (there are diffs from the generic rc). I/O is
> not locking up there but live migration failing and libvirt moves a
> freezed state. I can try to run the same on top of rc2, but it`ll be
> probably the same.


Tested on iscsi pool, though there are no-cache requirement and rbd
with disabled cache may survive one migration but iscsi backend hangs
always. As it was before, just rolling back problematic commit fixes
the problem and adding cpu_synchronize_all_states to migration.c has
no difference at a glance in a VM` behavior. The problem consist at
least two separate ones: the current hang and behavior with the
unreverted patch from agraf - last one causes live migration with
writeback cache to fail, cache=none works well in any variant which
survives first condition. Marcin, would you mind to check the current
state of the problem on your environments in a spare time? It is
probably easier to reproduce on iscsi because of way smaller time
needed to set it up, command line and libvirt config attached
(v2.1.0-rc2 plus iscsi-1.11.0).

[-- Attachment #2: cli.txt --]
[-- Type: text/plain, Size: 1426 bytes --]

qemu-system-x86_64 -enable-kvm -name vm27842 -S -machine pc-i440fx-2.1,accel=kvm,usb=off -m 256 -realtime mlock=off -smp 12,sockets=1,cores=12,threads=12 -numa node,nodeid=0,cpus=0,mem=256 -uuid 9a44bdae-5702-463b-aa1e-d8d85055f6af -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm27842.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/dev/disk/by-path/ip-10.6.0.1:3260-iscsi-iqn.2014-05.ru.flops:kvmguest-lun-1,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x2,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:10:03:40,bus=pci.0,addr=0x3 -netdev tap,fd=25,id=hostnet1,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:10:03:41,bus=pci.0,addr=0x4 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/vm27842.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1

[-- Attachment #3: libvirt-config.xml --]
[-- Type: text/xml, Size: 2881 bytes --]

<domain type='kvm'>
  <name>vm27842</name>
  <uuid>9a44bdae-5702-463b-aa1e-d8d85055f6af</uuid>
  <description>3145728</description>
  <memory unit='KiB'>262144</memory>
  <currentMemory unit='KiB'>262144</currentMemory>
  <vcpu placement='static' cpuset='0-11'>12</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu>
    <topology sockets='1' cores='12' threads='12'/>
    <numa>
      <cell cpus='0' memory='262144'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup' track='guest'>
      <catchup threshold='123' slew='120' limit='10000'/>
    </timer>
    <timer name='pit' tickpolicy='delay'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.6.0.1:3260-iscsi-iqn.2014-05.ru.flops:kvmguest-lun-1'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </disk>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='bridge'>
      <mac address='52:54:00:10:03:40'/>
      <source bridge='oswbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='0bf5d1b1-a972-40e4-839d-067730f4f20d'/>
      </virtualport>
      <target dev='vp27842l'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:10:03:41'/>
      <source bridge='oswbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='f7ebddab-6f4d-4e01-8a5a-59ee3e434c92'/>
      </virtualport>
      <target dev='vp27842p'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/vm27842.sock'/>
      <target type='virtio' name='org.qemu.guest_agent.1'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <memballoon model='none'/>
  </devices>
  <seclabel type='none'/>
</domain>


[-- Attachment #4: fio.txt --]
[-- Type: text/plain, Size: 148 bytes --]

[random]
rw=randrw
size=768M
directory=/data
iodepth=32
fsync=1
direct=1
blocksize=2M
numjobs=1
nrfiles=4
group_reporting
ioengine=libaio
loops=512

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

> Tested on iscsi pool, though there are no-cache requirement and rbd
> with disabled cache may survive one migration but iscsi backend hangs
> always. As it was before, just rolling back problematic commit fixes
> the problem and adding cpu_synchronize_all_states to migration.c has
> no difference at a glance in a VM` behavior. The problem consist at
> least two separate ones: the current hang and behavior with the
> unreverted patch from agraf - last one causes live migration with
> writeback cache to fail, cache=none works well in any variant which
> survives first condition. Marcin, would you mind to check the current
> state of the problem on your environments in a spare time? It is
> probably easier to reproduce on iscsi because of way smaller time
> needed to set it up, command line and libvirt config attached
> (v2.1.0-rc2 plus iscsi-1.11.0).

Ok, but what exacly do you want me to test?

Just to avoid any confusion, originally there were two problems with 
kvmclock:

1. Commit a096b3a6732f846ec57dc28b47ee9435aa0609bf fixes problem when 
clock drift (?) caused kvmclock in guest to report time in past which 
caused guest kernel to hang. This is hard to reproduce reliably 
(probably as it requires long time for drift to accumulate).

2. Commit 9b1786829aefb83f37a8f3135e3ea91c56001b56 fixes regression 
caused by a096b3a6732f846ec57dc28b47ee9435aa0609bf which occured during 
non-migration operations (drive-mirror + pivot), which also caused guest 
kernel to hang. This is trival to reproduce.

I'm using both of them applied on top of 2.0 in production and have no 
problems with them. I'm using NFS exclusively with cache=none.

So, I shall test vm-migration and drive-migration with 2.1.0-rc2 with no 
extra patches applied or reverted, on VM that is running fio, am I correct?

-- 
mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Thu, Jul 17, 2014 at 1:28 AM, Marcin Gibuła <m.gibula@beyond.pl> wrote:
>> Tested on iscsi pool, though there are no-cache requirement and rbd
>> with disabled cache may survive one migration but iscsi backend hangs
>> always. As it was before, just rolling back problematic commit fixes
>> the problem and adding cpu_synchronize_all_states to migration.c has
>> no difference at a glance in a VM` behavior. The problem consist at
>> least two separate ones: the current hang and behavior with the
>> unreverted patch from agraf - last one causes live migration with
>> writeback cache to fail, cache=none works well in any variant which
>> survives first condition. Marcin, would you mind to check the current
>> state of the problem on your environments in a spare time? It is
>> probably easier to reproduce on iscsi because of way smaller time
>> needed to set it up, command line and libvirt config attached
>> (v2.1.0-rc2 plus iscsi-1.11.0).
>
>
> Ok, but what exacly do you want me to test?
>
> Just to avoid any confusion, originally there were two problems with
> kvmclock:
>
> 1. Commit a096b3a6732f846ec57dc28b47ee9435aa0609bf fixes problem when clock
> drift (?) caused kvmclock in guest to report time in past which caused guest
> kernel to hang. This is hard to reproduce reliably (probably as it requires
> long time for drift to accumulate).
>
> 2. Commit 9b1786829aefb83f37a8f3135e3ea91c56001b56 fixes regression caused
> by a096b3a6732f846ec57dc28b47ee9435aa0609bf which occured during
> non-migration operations (drive-mirror + pivot), which also caused guest
> kernel to hang. This is trival to reproduce.
>
> I'm using both of them applied on top of 2.0 in production and have no
> problems with them. I'm using NFS exclusively with cache=none.
>
> So, I shall test vm-migration and drive-migration with 2.1.0-rc2 with no
> extra patches applied or reverted, on VM that is running fio, am I correct?
>

Yes, exactly. ISCSI-based setup can take some minutes to deploy, given
prepared image, and I have one hundred percent hit rate for the
original issue with it.

> --
> mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

[-- Attachment #1: Type: text/plain, Size: 1029 bytes --]

>> I'm using both of them applied on top of 2.0 in production and have no
>> problems with them. I'm using NFS exclusively with cache=none.
>>
>> So, I shall test vm-migration and drive-migration with 2.1.0-rc2 with no
>> extra patches applied or reverted, on VM that is running fio, am I correct?
>>
>
> Yes, exactly. ISCSI-based setup can take some minutes to deploy, given
> prepared image, and I have one hundred percent hit rate for the
> original issue with it.

I've reproduced your IO hang with 2.0 and both 
9b1786829aefb83f37a8f3135e3ea91c56001b56 and 
a096b3a6732f846ec57dc28b47ee9435aa0609bf applied.

Reverting 9b1786829aefb83f37a8f3135e3ea91c56001b56 indeed fixes the 
problem (but reintroduces block-migration hang). It's seems like qemu 
bug rather than guest problem, as no-kvmclock parameters makes no 
difference. IO just stops, all qemu IO threads die off. Almost like it 
forgets to migrate them:-)

I'm attaching backtrace from guest kernel and qemu and qemu command line.

Going to compile 2.1-rc.

-- 
mg

[-- Attachment #2: guest-backtrace.txt --]
[-- Type: text/plain, Size: 3866 bytes --]

[  254.634525] SysRq : Show Blocked State
[  254.635041]   task                        PC stack   pid father
[  254.635304] kworker/0:2     D ffff88013fc145c0     0    83      2 0x00000000
[  254.635304] Workqueue: xfs-log/vdb xfs_log_worker [xfs]
[  254.635304]  ffff880136bdfa58 0000000000000046 ffff880136bdffd8 00000000000145c0
[  254.635304]  ffff880136bdffd8 00000000000145c0 ffff880136ad8000 ffff88013fc14e88
[  254.635304]  ffff880037bd4380 ffff880037bc5068 ffff880037bd43b0 ffff880037bd4380
[  254.635304] Call Trace:
[  254.635304]  [<ffffffff815e797d>] io_schedule+0x9d/0x140
[  254.635304]  [<ffffffff812921d5>] get_request+0x1b5/0x790
[  254.635304]  [<ffffffff81086ab0>] ? wake_up_bit+0x30/0x30
[  254.635304]  [<ffffffff81294236>] blk_queue_bio+0x96/0x390
[  254.635304]  [<ffffffff812904e2>] generic_make_request+0xe2/0x130
[  254.635304]  [<ffffffff812905a1>] submit_bio+0x71/0x150
[  254.635304]  [<ffffffff811e72c8>] ? bio_alloc_bioset+0x1e8/0x2e0
[  254.635304]  [<ffffffffa03310bb>] _xfs_buf_ioapply+0x2bb/0x3d0 [xfs]
[  254.635304]  [<ffffffffa038d3ef>] ? xlog_bdstrat+0x1f/0x50 [xfs]
[  254.635304]  [<ffffffffa03328e6>] xfs_buf_iorequest+0x46/0xa0 [xfs]
[  254.635304]  [<ffffffffa038d3ef>] xlog_bdstrat+0x1f/0x50 [xfs]
[  254.635304]  [<ffffffffa038f135>] xlog_sync+0x265/0x450 [xfs]
[  254.635304]  [<ffffffffa038f3b2>] xlog_state_release_iclog+0x92/0xb0 [xfs]
[  254.635304]  [<ffffffffa039016a>] _xfs_log_force+0x15a/0x290 [xfs]
[  254.635304]  [<ffffffff810115d6>] ? __switch_to+0x136/0x490
[  254.635304]  [<ffffffffa03902c6>] xfs_log_force+0x26/0x80 [xfs]
[  254.635304]  [<ffffffffa0390344>] xfs_log_worker+0x24/0x50 [xfs]
[  254.635304]  [<ffffffff8107e02b>] process_one_work+0x17b/0x460
[  254.635304]  [<ffffffff8107edfb>] worker_thread+0x11b/0x400
[  254.635304]  [<ffffffff8107ece0>] ? rescuer_thread+0x400/0x400
[  254.635304]  [<ffffffff81085aef>] kthread+0xcf/0xe0
[  254.635304]  [<ffffffff81085a20>] ? kthread_create_on_node+0x140/0x140
[  254.635304]  [<ffffffff815f24ec>] ret_from_fork+0x7c/0xb0
[  254.635304]  [<ffffffff81085a20>] ? kthread_create_on_node+0x140/0x140
[  254.635304] fio             D ffff88013fc145c0     0   772    770 0x00000000
[  254.635304]  ffff8800bba4b8c8 0000000000000082 ffff8800bba4bfd8 00000000000145c0
[  254.635304]  ffff8800bba4bfd8 00000000000145c0 ffff8801376ff1c0 ffff88013fc14e88
[  254.635304]  ffff880037bd4380 ffff880037baba90 ffff880037bd43b0 ffff880037bd4380
[  254.635304] Call Trace:
[  254.635304]  [<ffffffff815e797d>] io_schedule+0x9d/0x140
[  254.635304]  [<ffffffff812921d5>] get_request+0x1b5/0x790
[  254.635304]  [<ffffffff81086ab0>] ? wake_up_bit+0x30/0x30
[  254.635304]  [<ffffffff81294236>] blk_queue_bio+0x96/0x390
[  254.635304]  [<ffffffff812904e2>] generic_make_request+0xe2/0x130
[  254.635304]  [<ffffffff812905a1>] submit_bio+0x71/0x150
[  254.635304]  [<ffffffff811ed26c>] do_blockdev_direct_IO+0x14bc/0x2620
[  254.635304]  [<ffffffffa032bc30>] ? xfs_get_blocks+0x20/0x20 [xfs]
[  254.635304]  [<ffffffff811ee425>] __blockdev_direct_IO+0x55/0x60
[  254.635304]  [<ffffffffa032bc30>] ? xfs_get_blocks+0x20/0x20 [xfs]
[  254.635304]  [<ffffffffa032aaec>] xfs_vm_direct_IO+0x15c/0x180 [xfs]
[  254.635304]  [<ffffffffa032bc30>] ? xfs_get_blocks+0x20/0x20 [xfs]
[  254.635304]  [<ffffffff81143563>] generic_file_aio_read+0x6d3/0x750
[  254.635304]  [<ffffffff810b69c8>] ? ktime_get_ts+0x48/0xe0
[  254.635304]  [<ffffffff811030cf>] ? delayacct_end+0x8f/0xb0
[  254.635304]  [<ffffffff815e6a32>] ? down_read+0x12/0x30
[  254.635304]  [<ffffffffa0337224>] xfs_file_aio_read+0x154/0x2e0 [xfs]
[  254.635304]  [<ffffffffa03370d0>] ? xfs_file_splice_read+0x140/0x140 [xfs]
[  254.635304]  [<ffffffff811fd6a8>] do_io_submit+0x3b8/0x840
[  254.635304]  [<ffffffff811fdb40>] SyS_io_submit+0x10/0x20
[  254.635304]  [<ffffffff815f2599>] system_call_fastpath+0x16/0x1b


[-- Attachment #3: qemu-backtrace.txt --]
[-- Type: text/plain, Size: 2514 bytes --]

Thread 3 (Thread 0x7f4250f50700 (LWP 11955)):
#0  0x00007f4253d1a897 in ioctl () from /lib64/libc.so.6
#1  0x00007f4257f8adf9 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f4258e2aa90, type=type@entry=44672)
    at /var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/kvm-all.c:1796
#2  0x00007f4257f8af35 in kvm_cpu_exec (cpu=cpu@entry=0x7f4258e2aa90) at /var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/kvm-all.c:1681
#3  0x00007f4257f3071c in qemu_kvm_cpu_thread_fn (arg=0x7f4258e2aa90) at /var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/cpus.c:873
#4  0x00007f4253fe8f3a in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4253d22dad in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f424b5ff700 (LWP 11957)):
#0  0x00007f4253fecd0c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f425802c019 in qemu_cond_wait (cond=cond@entry=0x7f4258f0cfc0, mutex=mutex@entry=0x7f4258f0cff0)
    at /var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/util/qemu-thread-posix.c:135
#2  0x00007f4257f2070b in vnc_worker_thread_loop (queue=queue@entry=0x7f4258f0cfc0)
    at /var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/ui/vnc-jobs.c:222
#3  0x00007f4257f20ae0 in vnc_worker_thread (arg=0x7f4258f0cfc0) at /var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/ui/vnc-jobs.c:323
#4  0x00007f4253fe8f3a in start_thread () from /lib64/libpthread.so.0
#5  0x00007f4253d22dad in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f4257cc6900 (LWP 11952)):
#0  0x00007f4253d19286 in ppoll () from /lib64/libc.so.6
#1  0x00007f4257eecd79 in ppoll (__ss=0x0, __timeout=0x7ffffc03af40, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77
#2  qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=883000000)
    at /var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/qemu-timer.c:316
#3  0x00007f4257eb02d4 in os_host_main_loop_wait (timeout=883000000) at /var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/main-loop.c:229
#4  main_loop_wait (nonblocking=<optimized out>) at /var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/main-loop.c:484
#5  0x00007f4257d7c05e in main_loop () at /var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/vl.c:2051
#6  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/vl.c:4507


[-- Attachment #4: qemu-cmdline.txt --]
[-- Type: text/plain, Size: 2522 bytes --]

/usr/bin/qemu-system-x86_64 -machine accel=kvm -name 21eae881-5e6f-4d13-9b7d-0b8279aed737 -S -machine pc-i440fx-2.0,accel=kvm,usb=off -cpu SandyBridge,+kvmclock -m 4096 -realtime mlock=on -smp 4,sockets=2,cores=10,threads=1 -uuid 21eae881-5e6f-4d13-9b7d-0b8279aed737 -smbios type=0,vendor=HAL 9000 -smbios type=1,manufacturer=testcloud -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/21eae881-5e6f-4d13-9b7d-0b8279aed737.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -no-hpet -global kvm-pit.lost_tick_policy=discard -no-shutdown -boot order=dc,menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/mnt/nfs/volumes/e919ceff-8344-4de5-82da-db49a20c4c87/active.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,aio=threads,bps_rd=68157440,bps_wr=68157440,iops_rd=325,iops_wr=325 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/mnt/nfs/volumes/f2fb6c59-2960-4976-aaa1-6154f55f6a66/active.qcow2,if=none,id=drive-virtio-disk1,format=qcow2,cache=none,aio=threads,bps_rd=68157440,bps_wr=68157440,iops_rd=325,iops_wr=325 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1 -drive if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:07:6f:fb,bus=pci.0,addr=0x3 -netdev tap,fd=25,id=hostnet1,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:39:21:d3,bus=pci.0,addr=0x4 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/21eae881-5e6f-4d13-9b7d-0b8279aed737.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/21eae881-5e6f-4d13-9b7d-0b8279aed737.testcloud.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.testcloud.guest_agent.1 -device usb-tablet,id=input0 -vnc 0.0.0.0:1,password -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -incoming tcp:0.0.0.0:49152 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -sandbox on -device pvpanic

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

> I've reproduced your IO hang with 2.0 and both
> 9b1786829aefb83f37a8f3135e3ea91c56001b56 and
> a096b3a6732f846ec57dc28b47ee9435aa0609bf applied.
>
> Reverting 9b1786829aefb83f37a8f3135e3ea91c56001b56 indeed fixes the
> problem (but reintroduces block-migration hang). It's seems like qemu
> bug rather than guest problem, as no-kvmclock parameters makes no
> difference. IO just stops, all qemu IO threads die off. Almost like it
> forgets to migrate them:-)

Some more info:

a) 2.0 + 9b1786829aefb83f37a8f3135e3ea91c56001b56 + 
a096b3a6732f846ec57dc28b47ee9435aa0609bf = hangs

b) 2.0 + 9b1786829aefb83f37a8f3135e3ea91c56001b56 = works

c) 2.0 + 9b1786829aefb83f37a8f3135e3ea91c56001b56 + move 
cpu_synchronize_state to migration.c = works

Tested with NFS (qcow2) + cache=none.

IO is dead only for disk that was being written to during migration.
I.e. if my test VM has two disks: vda and vdb, and I'm running fio on 
vdb and it hangs after migration, I can still issue writes to vda.

Recreation steps:
1. Create VM
2. Run fio (Andrey's config)
3. Live migrate VM couple of times.

-- 
mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

>> Yes, exactly. ISCSI-based setup can take some minutes to deploy, given
>> prepared image, and I have one hundred percent hit rate for the
>> original issue with it.
>
> I've reproduced your IO hang with 2.0 and both
> 9b1786829aefb83f37a8f3135e3ea91c56001b56 and
> a096b3a6732f846ec57dc28b47ee9435aa0609bf applied.
>
> Reverting 9b1786829aefb83f37a8f3135e3ea91c56001b56 indeed fixes the
> problem (but reintroduces block-migration hang). It's seems like qemu
> bug rather than guest problem, as no-kvmclock parameters makes no
> difference. IO just stops, all qemu IO threads die off. Almost like it
> forgets to migrate them:-)
>
> I'm attaching backtrace from guest kernel and qemu and qemu command line.
>
> Going to compile 2.1-rc.

2.1-rc2 behaves exactly the same.

Interestingly enough, reseting guest system causes I/O to work again. So 
it's not qemu that hangs on IO, rather it fails to notify guest about 
completed operations that were issued during migration.

And its somehow caused by calling cpu_synchronize_all_states() inside 
kvmclock_vm_state_change().



As for testing with cache=writeback, I'll try to setup some iscsi to 
test it.

-- 
mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Thu, Jul 17, 2014 at 3:54 PM, Marcin Gibuła <m.gibula@beyond.pl> wrote:
>>> Yes, exactly. ISCSI-based setup can take some minutes to deploy, given
>>> prepared image, and I have one hundred percent hit rate for the
>>> original issue with it.
>>
>>
>> I've reproduced your IO hang with 2.0 and both
>> 9b1786829aefb83f37a8f3135e3ea91c56001b56 and
>> a096b3a6732f846ec57dc28b47ee9435aa0609bf applied.
>>
>> Reverting 9b1786829aefb83f37a8f3135e3ea91c56001b56 indeed fixes the
>> problem (but reintroduces block-migration hang). It's seems like qemu
>> bug rather than guest problem, as no-kvmclock parameters makes no
>> difference. IO just stops, all qemu IO threads die off. Almost like it
>> forgets to migrate them:-)
>>
>> I'm attaching backtrace from guest kernel and qemu and qemu command line.
>>
>> Going to compile 2.1-rc.
>
>
> 2.1-rc2 behaves exactly the same.
>
> Interestingly enough, reseting guest system causes I/O to work again. So
> it's not qemu that hangs on IO, rather it fails to notify guest about
> completed operations that were issued during migration.
>
> And its somehow caused by calling cpu_synchronize_all_states() inside
> kvmclock_vm_state_change().
>
>
>
> As for testing with cache=writeback, I'll try to setup some iscsi to test
> it.

Awesome, thanks! AFAIK you`ll not be able to use write cache with
iscsi for migration. VM which had a reset before hangs always when
freshly launched have a chance to be migrated successfully. And yes,
it looks like lower layer forgetting to notify driver about some
operations at a glance.

>
> --
> mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

[-- Attachment #1: Type: text/plain, Size: 1044 bytes --]

>> 2.1-rc2 behaves exactly the same.
>>
>> Interestingly enough, reseting guest system causes I/O to work again. So
>> it's not qemu that hangs on IO, rather it fails to notify guest about
>> completed operations that were issued during migration.
>>
>> And its somehow caused by calling cpu_synchronize_all_states() inside
>> kvmclock_vm_state_change().
>>
>>
>>
>> As for testing with cache=writeback, I'll try to setup some iscsi to test
>> it.
>
> Awesome, thanks! AFAIK you`ll not be able to use write cache with
> iscsi for migration. VM which had a reset before hangs always when
> freshly launched have a chance to be migrated successfully. And yes,
> it looks like lower layer forgetting to notify driver about some
> operations at a glance.

Andrey,

could you try attached patch? It's an incredibly ugly workaround that 
calls cpu_synchronize_all_states() in a way that bypasses lazy execution 
logic.

But it works for me. If that works for you as well, its somehow related 
to lazy execution of cpu_synchronize_all_states.

-- 
mg

[-- Attachment #2: io-hang.patch --]
[-- Type: text/x-patch, Size: 3151 bytes --]

diff -ru qemu-2.1.0-rc2/cpus.c qemu-2.1.0-rc2-fixed/cpus.c
--- qemu-2.1.0-rc2/cpus.c	2014-07-15 23:49:14.000000000 +0200
+++ qemu-2.1.0-rc2-fixed/cpus.c	2014-07-17 15:09:09.306696284 +0200
@@ -505,6 +505,15 @@
     }
 }
 
+void cpu_synchronize_all_states_always(void)
+{
+    CPUState *cpu;
+
+    CPU_FOREACH(cpu) {
+        cpu_synchronize_state_always(cpu);
+    }
+}
+
 void cpu_synchronize_all_post_reset(void)
 {
     CPUState *cpu;
diff -ru qemu-2.1.0-rc2/hw/i386/kvm/clock.c qemu-2.1.0-rc2-fixed/hw/i386/kvm/clock.c
--- qemu-2.1.0-rc2/hw/i386/kvm/clock.c	2014-07-15 23:49:14.000000000 +0200
+++ qemu-2.1.0-rc2-fixed/hw/i386/kvm/clock.c	2014-07-17 15:08:25.627063756 +0200
@@ -126,7 +126,7 @@
             return;
         }
 
-        cpu_synchronize_all_states();
+        cpu_synchronize_all_states_always();
         ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
         if (ret < 0) {
             fprintf(stderr, "KVM_GET_CLOCK failed: %s\n", strerror(ret));
diff -ru qemu-2.1.0-rc2/include/sysemu/cpus.h qemu-2.1.0-rc2-fixed/include/sysemu/cpus.h
--- qemu-2.1.0-rc2/include/sysemu/cpus.h	2014-07-15 23:49:14.000000000 +0200
+++ qemu-2.1.0-rc2-fixed/include/sysemu/cpus.h	2014-07-17 15:09:23.256578916 +0200
@@ -7,6 +7,7 @@
 void pause_all_vcpus(void);
 void cpu_stop_current(void);
 
+void cpu_synchronize_all_states_always(void);
 void cpu_synchronize_all_states(void);
 void cpu_synchronize_all_post_reset(void);
 void cpu_synchronize_all_post_init(void);
diff -ru qemu-2.1.0-rc2/include/sysemu/kvm.h qemu-2.1.0-rc2-fixed/include/sysemu/kvm.h
--- qemu-2.1.0-rc2/include/sysemu/kvm.h	2014-07-15 23:49:14.000000000 +0200
+++ qemu-2.1.0-rc2-fixed/include/sysemu/kvm.h	2014-07-17 15:11:54.855303171 +0200
@@ -346,9 +346,11 @@
 #endif /* NEED_CPU_H */
 
 void kvm_cpu_synchronize_state(CPUState *cpu);
+void kvm_cpu_synchronize_state_always(CPUState *cpu);
 void kvm_cpu_synchronize_post_reset(CPUState *cpu);
 void kvm_cpu_synchronize_post_init(CPUState *cpu);
 
+
 /* generic hooks - to be moved/refactored once there are more users */
 
 static inline void cpu_synchronize_state(CPUState *cpu)
@@ -358,6 +360,13 @@
     }
 }
 
+static inline void cpu_synchronize_state_always(CPUState *cpu)
+{
+    if (kvm_enabled()) {
+        kvm_cpu_synchronize_state_always(cpu);
+    }
+}
+
 static inline void cpu_synchronize_post_reset(CPUState *cpu)
 {
     if (kvm_enabled()) {
diff -ru qemu-2.1.0-rc2/kvm-all.c qemu-2.1.0-rc2-fixed/kvm-all.c
--- qemu-2.1.0-rc2/kvm-all.c	2014-07-15 23:49:14.000000000 +0200
+++ qemu-2.1.0-rc2-fixed/kvm-all.c	2014-07-17 15:14:04.884208826 +0200
@@ -1652,6 +1652,13 @@
     s->coalesced_flush_in_progress = false;
 }
 
+static void do_kvm_cpu_synchronize_state_always(void *arg)
+{
+    CPUState *cpu = arg;
+
+    kvm_arch_get_registers(cpu);
+}
+
 static void do_kvm_cpu_synchronize_state(void *arg)
 {
     CPUState *cpu = arg;
@@ -1669,6 +1676,11 @@
     }
 }
 
+void kvm_cpu_synchronize_state_always(CPUState *cpu)
+{
+    run_on_cpu(cpu, do_kvm_cpu_synchronize_state_always, cpu);
+}
+
 void kvm_cpu_synchronize_post_reset(CPUState *cpu)
 {
     kvm_arch_put_registers(cpu, KVM_PUT_RESET_STATE);

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Dr. David Alan Gilbert]

I don't know if this is the same case, but Gerd showed me a migration failure
that might be related.  2.0 seems OK, 2.1-rc0 is broken (and I've not found
another working point in between yet).

The test case involves booting a fedora livecd (using an IDE CDROM device)
and after the migration we're seeing squashfs errors and stuff gently
falling apart.

Dave

--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

W dniu 2014-07-17 21:18, Dr. David Alan Gilbert pisze:
> I don't know if this is the same case, but Gerd showed me a migration failure
> that might be related.  2.0 seems OK, 2.1-rc0 is broken (and I've not found
> another working point in between yet).
>
> The test case involves booting a fedora livecd (using an IDE CDROM device)
> and after the migration we're seeing squashfs errors and stuff gently
> falling apart.

Perhaps you could try testing workaround patch I sent earlier? It's not 
proposal for inclusion, just a test patch that seems to fix IO hang for me.

-- 
mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Thu, Jul 17, 2014 at 5:25 PM, Marcin Gibuła <m.gibula@beyond.pl> wrote:
>>> 2.1-rc2 behaves exactly the same.
>>>
>>> Interestingly enough, reseting guest system causes I/O to work again. So
>>> it's not qemu that hangs on IO, rather it fails to notify guest about
>>> completed operations that were issued during migration.
>>>
>>> And its somehow caused by calling cpu_synchronize_all_states() inside
>>> kvmclock_vm_state_change().
>>>
>>>
>>>
>>> As for testing with cache=writeback, I'll try to setup some iscsi to test
>>> it.
>>
>>
>> Awesome, thanks! AFAIK you`ll not be able to use write cache with
>> iscsi for migration. VM which had a reset before hangs always when
>> freshly launched have a chance to be migrated successfully. And yes,
>> it looks like lower layer forgetting to notify driver about some
>> operations at a glance.
>
>
> Andrey,
>
> could you try attached patch? It's an incredibly ugly workaround that calls
> cpu_synchronize_all_states() in a way that bypasses lazy execution logic.
>
> But it works for me. If that works for you as well, its somehow related to
> lazy execution of cpu_synchronize_all_states.
>
> --
> mg

Yes, it working well with writeback cache too.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

>> could you try attached patch? It's an incredibly ugly workaround that calls
>> cpu_synchronize_all_states() in a way that bypasses lazy execution logic.
>>
>> But it works for me. If that works for you as well, its somehow related to
>> lazy execution of cpu_synchronize_all_states.
>>
>> --
>> mg
>
> Yes, it working well with writeback cache too.

Does it fix problem with libvirt migration timing out for you as well?

-- 
mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Fri, Jul 18, 2014 at 12:21 PM, Marcin Gibuła <m.gibula@beyond.pl> wrote:
>>> could you try attached patch? It's an incredibly ugly workaround that
>>> calls
>>> cpu_synchronize_all_states() in a way that bypasses lazy execution logic.
>>>
>>> But it works for me. If that works for you as well, its somehow related
>>> to
>>> lazy execution of cpu_synchronize_all_states.
>>>
>>> --
>>> mg
>>
>>
>> Yes, it working well with writeback cache too.
>
>
> Does it fix problem with libvirt migration timing out for you as well?
>

Oh, forgot to mention - yes, all migration-related problems are fixed.
Though release right now in a freeze phase, I`d like to ask
maintainers to consider possibility of fixing the problem on top of
the current tree instead of just rolling back problematic snippet.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

>> Does it fix problem with libvirt migration timing out for you as well?
>>
>
> Oh, forgot to mention - yes, all migration-related problems are fixed.
> Though release right now in a freeze phase, I`d like to ask
> maintainers to consider possibility of fixing the problem on top of
> the current tree instead of just rolling back problematic snippet.

Paolo,

if patch in its current form is not acceptable for you for inclusion, 
I'll try rewrite it according to your comments.

-- 
mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Paolo Bonzini]

Il 17/07/2014 15:25, Marcin Gibuła ha scritto:
> +static void do_kvm_cpu_synchronize_state_always(void *arg)
> +{
> +    CPUState *cpu = arg;
> +
> +    kvm_arch_get_registers(cpu);
> +}
> +

The name of the hack^Wfunction is tricky, because compared to 
do_kvm_cpu_synchronize_state there are three things you change:

1) you always synchronize the state

2) the next call to do_kvm_cpu_synchronize_state will do 
kvm_arch_get_registers

3) the next CPU entry will call kvm_arch_put_registers:

         if (cpu->kvm_vcpu_dirty) {
             kvm_arch_put_registers(cpu, KVM_PUT_RUNTIME_STATE);
             cpu->kvm_vcpu_dirty = false;
         }

It is easy to find out if the "fix" is related to 1 or 2/3: just write

      if (cpu->kvm_vcpu_dirty) {
          printf ("do_kvm_cpu_synchronize_state_always: look at 2/3\n");
          kvm_arch_get_registers(cpu);
      } else {
          printf ("do_kvm_cpu_synchronize_state_always: look at 1\n");
      }

To further refine between 2 and 3, I suppose you can set a breakpoint on 
cpu_synchronize_all_states and kvm_cpu_exec, and see which is called 
first after cpu_synchronize_all_states_always.

I still lean very much towards reverting the patches now.  We can 
reapply them, fixed, in 2.1.1.

Paolo

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Paolo Bonzini]

Il 18/07/2014 10:44, Marcin Gibuła ha scritto:
>>
>
> Paolo,
>
> if patch in its current form is not acceptable for you for inclusion,
> I'll try rewrite it according to your comments.

The problem is that we don't know _why_ the patch is fixing things.

Considering that your kvmclock bug has been there literally for years, I 
think it's better to delay that one to 2.2.

Paolo

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Amit Shah]

On (Fri) 18 Jul 2014 [10:48:40], Paolo Bonzini wrote:
> Il 17/07/2014 15:25, Marcin Gibuła ha scritto:
> >+static void do_kvm_cpu_synchronize_state_always(void *arg)
> >+{
> >+    CPUState *cpu = arg;
> >+
> >+    kvm_arch_get_registers(cpu);
> >+}
> >+
> 
> The name of the hack^Wfunction is tricky, because compared to
> do_kvm_cpu_synchronize_state there are three things you change:
> 
> 1) you always synchronize the state
> 
> 2) the next call to do_kvm_cpu_synchronize_state will do
> kvm_arch_get_registers
> 
> 3) the next CPU entry will call kvm_arch_put_registers:
> 
>         if (cpu->kvm_vcpu_dirty) {
>             kvm_arch_put_registers(cpu, KVM_PUT_RUNTIME_STATE);
>             cpu->kvm_vcpu_dirty = false;
>         }
> 
> It is easy to find out if the "fix" is related to 1 or 2/3: just write
> 
>      if (cpu->kvm_vcpu_dirty) {
>          printf ("do_kvm_cpu_synchronize_state_always: look at 2/3\n");
>          kvm_arch_get_registers(cpu);
>      } else {
>          printf ("do_kvm_cpu_synchronize_state_always: look at 1\n");
>      }
> 
> To further refine between 2 and 3, I suppose you can set a breakpoint on
> cpu_synchronize_all_states and kvm_cpu_exec, and see which is called first
> after cpu_synchronize_all_states_always.
> 
> I still lean very much towards reverting the patches now.  We can reapply
> them, fixed, in 2.1.1.

FWIW I agree with this plan.

		Amit

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

> The name of the hack^Wfunction is tricky, because compared to
> do_kvm_cpu_synchronize_state there are three things you change:
>
> 1) you always synchronize the state
>
> 2) the next call to do_kvm_cpu_synchronize_state will do
> kvm_arch_get_registers

Yes.

> 3) the next CPU entry will call kvm_arch_put_registers:
>
>          if (cpu->kvm_vcpu_dirty) {
>              kvm_arch_put_registers(cpu, KVM_PUT_RUNTIME_STATE);
>              cpu->kvm_vcpu_dirty = false;
>          }

But, I don't set cpu->kvm_vcpu_dirty anywhere (?).

> I still lean very much towards reverting the patches now.  We can
> reapply them, fixed, in 2.1.1.

That's probably good idea.

-- 
mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Paolo Bonzini]

Il 18/07/2014 11:32, Marcin Gibuła ha scritto:
>
>> 3) the next CPU entry will call kvm_arch_put_registers:
>>
>>          if (cpu->kvm_vcpu_dirty) {
>>              kvm_arch_put_registers(cpu, KVM_PUT_RUNTIME_STATE);
>>              cpu->kvm_vcpu_dirty = false;
>>          }
>
> But, I don't set cpu->kvm_vcpu_dirty anywhere (?).

Yeah, the next CPU entry will *not* call kvm_arch_put_registers with 
your change.  It will call it with vanilla cpu_synchronize_all_states().

Paolo

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

W dniu 2014-07-18 11:37, Paolo Bonzini pisze:
> Il 18/07/2014 11:32, Marcin Gibuła ha scritto:
>>
>>> 3) the next CPU entry will call kvm_arch_put_registers:
>>>
>>>          if (cpu->kvm_vcpu_dirty) {
>>>              kvm_arch_put_registers(cpu, KVM_PUT_RUNTIME_STATE);
>>>              cpu->kvm_vcpu_dirty = false;
>>>          }
>>
>> But, I don't set cpu->kvm_vcpu_dirty anywhere (?).
>
> Yeah, the next CPU entry will *not* call kvm_arch_put_registers with
> your change.  It will call it with vanilla cpu_synchronize_all_states().

That's because in kvmclock, it's used only to read cpu registers, not 
edit them.

Now, because making this call "invisible" makes it work, I'm speculating 
that following happens:

[migration starts]
kvmclock: calls cpu_synchronize_all_states()
somewhere in qemu: completes IO
somewhere in qemu: calls cpu_synchronize_all_states() <- old state


Is it (or something similar) possible? I didn't dig deep enough into 
internals yet, but perhaps you could point if thats the right direction?

-- 
mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Paolo Bonzini]

Il 18/07/2014 10:48, Paolo Bonzini ha scritto:
> 
> It is easy to find out if the "fix" is related to 1 or 2/3: just write
> 
>      if (cpu->kvm_vcpu_dirty) {
>          printf ("do_kvm_cpu_synchronize_state_always: look at 2/3\n");
>          kvm_arch_get_registers(cpu);
>      } else {
>          printf ("do_kvm_cpu_synchronize_state_always: look at 1\n");
>      }
> 
> To further refine between 2 and 3, I suppose you can set a breakpoint on
> cpu_synchronize_all_states and kvm_cpu_exec, and see which is called
> first after cpu_synchronize_all_states_always.

Marcin, have you ever gotten round to doing this?

Paolo

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

On 29.07.2014 18:58, Paolo Bonzini wrote:
> Il 18/07/2014 10:48, Paolo Bonzini ha scritto:
>>
>> It is easy to find out if the "fix" is related to 1 or 2/3: just write
>>
>>       if (cpu->kvm_vcpu_dirty) {
>>           printf ("do_kvm_cpu_synchronize_state_always: look at 2/3\n");
>>           kvm_arch_get_registers(cpu);
>>       } else {
>>           printf ("do_kvm_cpu_synchronize_state_always: look at 1\n");
>>       }
>>
>> To further refine between 2 and 3, I suppose you can set a breakpoint on
>> cpu_synchronize_all_states and kvm_cpu_exec, and see which is called
>> first after cpu_synchronize_all_states_always.
>
> Marcin, have you ever gotten round to doing this?

Source side of migration, without my ugly hack:

called do_kvm_cpu_synchronize_state: vcpu not dirty, getting registers
called do_kvm_cpu_synchronize_state: vcpu not dirty, getting registers
called kvm_cpu_synchronize_state: vcpu dirty
called kvm_cpu_synchronize_state: vcpu dirty
shutting down

without it:

called do_kvm_cpu_synchronize_state_always
called do_kvm_cpu_synchronize_state_always
called do_kvm_cpu_synchronize_state: vcpu not dirty, getting registers
called do_kvm_cpu_synchronize_state: vcpu not dirty, getting registers
shutting down

So it's probably about 2 from your list ("the next call to 
do_kvm_cpu_synchronize_state will do kvm_arch_get_registers").

I've tapped into kvm_cpu_exec() to find out if it's 
kvm_arch_put_registers(), but nothing was logged during migration so 
it's probably not 3.

-- 
mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Paolo Bonzini]

Il 30/07/2014 14:02, Marcin Gibuła ha scritto:
> without it:
> 
> called do_kvm_cpu_synchronize_state_always
> called do_kvm_cpu_synchronize_state_always
> called do_kvm_cpu_synchronize_state: vcpu not dirty, getting registers
> called do_kvm_cpu_synchronize_state: vcpu not dirty, getting registers
> shutting down
> 
> So it's probably about 2 from your list ("the next call to
> do_kvm_cpu_synchronize_state will do kvm_arch_get_registers").

Can you dump *env before and after the call to kvm_arch_get_registers?

Paolo

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

W dniu 2014-07-30 15:38, Paolo Bonzini pisze:
> Il 30/07/2014 14:02, Marcin Gibuła ha scritto:
>> without it:

s/without/with/ of course...

>> called do_kvm_cpu_synchronize_state_always
>> called do_kvm_cpu_synchronize_state_always
>> called do_kvm_cpu_synchronize_state: vcpu not dirty, getting registers
>> called do_kvm_cpu_synchronize_state: vcpu not dirty, getting registers
>> shutting down
>>
>> So it's probably about 2 from your list ("the next call to
>> do_kvm_cpu_synchronize_state will do kvm_arch_get_registers").
>
> Can you dump *env before and after the call to kvm_arch_get_registers?

Yes, but it seems they are equal - I used memcmp() to compare them. Is 
there any other side effect that cpu_synchronize_all_states() may have?

The second caller of this function is qemu_savevm_state_complete().

-- 
mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

>> Can you dump *env before and after the call to kvm_arch_get_registers?
>
> Yes, but it seems they are equal - I used memcmp() to compare them. Is
> there any other side effect that cpu_synchronize_all_states() may have?

I think I found it.

The reason for hang is, because when second call to 
kvm_arch_get_registers() is skipped, it also skips kvm_get_apic() which 
updates cpu->apic_state.

-- 
mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

W dniu 2014-07-31 13:27, Marcin Gibuła pisze:
>>> Can you dump *env before and after the call to kvm_arch_get_registers?
>>
>> Yes, but it seems they are equal - I used memcmp() to compare them. Is
>> there any other side effect that cpu_synchronize_all_states() may have?
>
> I think I found it.
>
> The reason for hang is, because when second call to
> kvm_arch_get_registers() is skipped, it also skips kvm_get_apic() which
> updates cpu->apic_state.

Paolo,

is this analysis deep enough for you? I don't know if that can be fixed 
with existing api as cpu_synchronize_all_states() is all or nothing kind 
of stuff.

Kvmclock needs it only to read current cpu registers, so syncing 
everything is not really necessary. Perhaps exporting one of 
kvm_arch_get_* would be enough. And it wouldn't mess with lazy get/put.

On the other hand, if in future any other driver adds 
cpu_synchronize_all_states() in its change state callback it could 
result in same error so perhaps more generic approach is needed.

-- 
mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Paolo Bonzini]

Il 04/08/2014 18:30, Marcin Gibuła ha scritto:
> 
> 
> is this analysis deep enough for you? I don't know if that can be fixed
> with existing api as cpu_synchronize_all_states() is all or nothing kind
> of stuff.
> 
> Kvmclock needs it only to read current cpu registers, so syncing
> everything is not really necessary. Perhaps exporting one of
> kvm_arch_get_* would be enough. And it wouldn't mess with lazy get/put.
> 
> On the other hand, if in future any other driver adds
> cpu_synchronize_all_states() in its change state callback it could
> result in same error so perhaps more generic approach is needed.

Yeah, I need to sit down and look at the code more closely...  Perhaps a
cpu_mark_all_dirty() is enough.

Paolo

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcelo Tosatti]

On Mon, Aug 04, 2014 at 08:30:48PM +0200, Paolo Bonzini wrote:
> Il 04/08/2014 18:30, Marcin Gibuła ha scritto:
> > 
> > 
> > is this analysis deep enough for you? I don't know if that can be fixed
> > with existing api as cpu_synchronize_all_states() is all or nothing kind
> > of stuff.
> > 
> > Kvmclock needs it only to read current cpu registers, so syncing
> > everything is not really necessary. Perhaps exporting one of
> > kvm_arch_get_* would be enough. And it wouldn't mess with lazy get/put.
> > 
> > On the other hand, if in future any other driver adds
> > cpu_synchronize_all_states() in its change state callback it could
> > result in same error so perhaps more generic approach is needed.
> 
> Yeah, I need to sit down and look at the code more closely...  Perhaps a
> cpu_mark_all_dirty() is enough.

Hi Paolo,

cpu_clean_all_dirty, you mean? Has the same effect.

Marcin's patch to add cpu_synchronize_state_always() has the same
effect.

What do you prefer ?

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Paolo Bonzini]

> > Yeah, I need to sit down and look at the code more closely...  Perhaps a
> > cpu_mark_all_dirty() is enough.
> 
> Hi Paolo,
> 
> cpu_clean_all_dirty, you mean? Has the same effect.
> 
> Marcin's patch to add cpu_synchronize_state_always() has the same
> effect.
> 
> What do you prefer ?

I'd prefer cpu_clean_all_dirty because you can call it from the APIC
load functions.  The bug with your patch is due to the APIC and to
migration, it's not in the way your patch touches the kvmclock
vmstate_change_handler.

Paolo

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Sat, Aug 9, 2014 at 10:35 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>> > Yeah, I need to sit down and look at the code more closely...  Perhaps a
>> > cpu_mark_all_dirty() is enough.
>>
>> Hi Paolo,
>>
>> cpu_clean_all_dirty, you mean? Has the same effect.
>>
>> Marcin's patch to add cpu_synchronize_state_always() has the same
>> effect.
>>
>> What do you prefer ?
>
> I'd prefer cpu_clean_all_dirty because you can call it from the APIC
> load functions.  The bug with your patch is due to the APIC and to
> migration, it's not in the way your patch touches the kvmclock
> vmstate_change_handler.
>
> Paolo

Hello,


JFYI - Windows showing the same behavior after migration using bare
2.1 (frozen disk with latest virtio block drivers). Reverting agraf`s
patches back plus Marcin`s fix saves the situation, so AFAICS problem
is critical to M$ products on KVM.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Thu, Aug 21, 2014 at 7:48 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> On Sat, Aug 9, 2014 at 10:35 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>>> > Yeah, I need to sit down and look at the code more closely...  Perhaps a
>>> > cpu_mark_all_dirty() is enough.
>>>
>>> Hi Paolo,
>>>
>>> cpu_clean_all_dirty, you mean? Has the same effect.
>>>
>>> Marcin's patch to add cpu_synchronize_state_always() has the same
>>> effect.
>>>
>>> What do you prefer ?
>>
>> I'd prefer cpu_clean_all_dirty because you can call it from the APIC
>> load functions.  The bug with your patch is due to the APIC and to
>> migration, it's not in the way your patch touches the kvmclock
>> vmstate_change_handler.
>>
>> Paolo
>
> Hello,
>
>
> JFYI - Windows showing the same behavior after migration using bare
> 2.1 (frozen disk with latest virtio block drivers). Reverting agraf`s
> patches back plus Marcin`s fix saves the situation, so AFAICS problem
> is critical to M$ products on KVM.

Sorry, the test series revealed that the problem is still here, but
with lower hit ratio with modified 2.1-HEAD using selected argument
set. The actual root of the issue is in '-cpu
qemu64,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000'
Windows-specific addition which can be traded for some idle CPU
consumption. Reproduction of bug is quite simple - fire up Windows VM,
migrate it two-three times and then try to log in using rdesktop/VNC -
if disk is frozen, login progress will freeze too. It is not so easy
to detect blocked I/O on Windows in the interactive session due to
lack of soft lockup warning equivalent, so one can try such a sequence
to check.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Paolo Bonzini]

Il 21/08/2014 18:41, Andrey Korolyov ha scritto:
> Sorry, the test series revealed that the problem is still here, but
> with lower hit ratio with modified 2.1-HEAD using selected argument
> set. The actual root of the issue is in '-cpu
> qemu64,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000'
> Windows-specific addition which can be traded for some idle CPU
> consumption. Reproduction of bug is quite simple - fire up Windows VM,
> migrate it two-three times and then try to log in using rdesktop/VNC -
> if disk is frozen, login progress will freeze too. It is not so easy
> to detect blocked I/O on Windows in the interactive session due to
> lack of soft lockup warning equivalent, so one can try such a sequence
> to check.

What kernel version?

The ioapic patches at http://thread.gmane.org/gmane.linux.kernel/1671045
fixed exactly a Windows-specific hang in migration.  But it didn't require
Hyper-V enlightenments.

Paolo

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Thu, Aug 21, 2014 at 8:44 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 21/08/2014 18:41, Andrey Korolyov ha scritto:
>> Sorry, the test series revealed that the problem is still here, but
>> with lower hit ratio with modified 2.1-HEAD using selected argument
>> set. The actual root of the issue is in '-cpu
>> qemu64,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000'
>> Windows-specific addition which can be traded for some idle CPU
>> consumption. Reproduction of bug is quite simple - fire up Windows VM,
>> migrate it two-three times and then try to log in using rdesktop/VNC -
>> if disk is frozen, login progress will freeze too. It is not so easy
>> to detect blocked I/O on Windows in the interactive session due to
>> lack of soft lockup warning equivalent, so one can try such a sequence
>> to check.
>
> What kernel version?
>
> The ioapic patches at http://thread.gmane.org/gmane.linux.kernel/1671045
> fixed exactly a Windows-specific hang in migration.  But it didn't require
> Hyper-V enlightenments.
>
> Paolo

I`m running 3.10, so patches are not here, will try 3.16 soon. Even if
problem will be fixed, it still will be specific for 2.1, earlier
releases working well and I`ll bisect at a time.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

>
> I`m running 3.10, so patches are not here, will try 3.16 soon. Even if
> problem will be fixed, it still will be specific for 2.1, earlier
> releases working well and I`ll bisect at a time.

Thanks, using 3.16 helped indeed. Though the bug remains as is at 2.1
on LTS 3.10, should I find the breaking commit, or it is better to
backport kvm subsystem changes related to apic to 3.10?

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcelo Tosatti]

[-- Attachment #1: Type: text/plain, Size: 599 bytes --]

On Fri, Aug 22, 2014 at 08:44:53PM +0400, Andrey Korolyov wrote:
> >
> > I`m running 3.10, so patches are not here, will try 3.16 soon. Even if
> > problem will be fixed, it still will be specific for 2.1, earlier
> > releases working well and I`ll bisect at a time.
> 
> Thanks, using 3.16 helped indeed. Though the bug remains as is at 2.1
> on LTS 3.10, should I find the breaking commit, or it is better to
> backport kvm subsystem changes related to apic to 3.10?

Andrey,

Can you try the patchset (which includes cpu_clean_all_dirty) please?
It should be equivalent to your proposed patch.



[-- Attachment #2: patches-kvmclock.tar.gz --]
[-- Type: application/gzip, Size: 2605 bytes --]

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Paolo Bonzini]

Il 22/08/2014 18:44, Andrey Korolyov ha scritto:
>>
>> I`m running 3.10, so patches are not here, will try 3.16 soon. Even if
>> problem will be fixed, it still will be specific for 2.1, earlier
>> releases working well and I`ll bisect at a time.
> 
> Thanks, using 3.16 helped indeed. Though the bug remains as is at 2.1
> on LTS 3.10, should I find the breaking commit, or it is better to
> backport kvm subsystem changes related to apic to 3.10?

I think I'd suggest the latter.  If the fix was brought by those ioapic
patches, the bug has always been latent as far as I could see.

I'm afraid that bisection would be pretty time consuming, also because
the bug didn't reproduce 100% for me (and at some point stopped
reproducing even on 3.10, causing some serious wtf...).

Paolo

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Fri, Aug 22, 2014 at 9:45 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> On Fri, Aug 22, 2014 at 08:44:53PM +0400, Andrey Korolyov wrote:
>> >
>> > I`m running 3.10, so patches are not here, will try 3.16 soon. Even if
>> > problem will be fixed, it still will be specific for 2.1, earlier
>> > releases working well and I`ll bisect at a time.
>>
>> Thanks, using 3.16 helped indeed. Though the bug remains as is at 2.1
>> on LTS 3.10, should I find the breaking commit, or it is better to
>> backport kvm subsystem changes related to apic to 3.10?
>
> Andrey,
>
> Can you try the patchset (which includes cpu_clean_all_dirty) please?
> It should be equivalent to your proposed patch.
>
>


Just checked case for Windows, bug is still here (twelve continuous
ping-pong migrations between two hosts). Also there is very weird
observation that the if VM stalled, password window renders in a
cripple way just after connection when no VNC region refreshes are
made yet by mouse, vm-is-alive case rendering after-c-a-d prompt
properly. Probably it`ll be easier to check behaviour against Paolo`
patchset for RH kernel, though I`m still in progress with cutting out
the rest to adapt them on top of bare 3.10, for ones with access to
git log it should be easier :)

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcelo Tosatti]

On Fri, Aug 22, 2014 at 10:39:38PM +0400, Andrey Korolyov wrote:
> On Fri, Aug 22, 2014 at 9:45 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> > On Fri, Aug 22, 2014 at 08:44:53PM +0400, Andrey Korolyov wrote:
> >> >
> >> > I`m running 3.10, so patches are not here, will try 3.16 soon. Even if
> >> > problem will be fixed, it still will be specific for 2.1, earlier
> >> > releases working well and I`ll bisect at a time.
> >>
> >> Thanks, using 3.16 helped indeed. Though the bug remains as is at 2.1
> >> on LTS 3.10, should I find the breaking commit, or it is better to
> >> backport kvm subsystem changes related to apic to 3.10?
> >
> > Andrey,
> >
> > Can you try the patchset (which includes cpu_clean_all_dirty) please?
> > It should be equivalent to your proposed patch.
> >
> >
> 
> 
> Just checked case for Windows, bug is still here (twelve continuous
> ping-pong migrations between two hosts). Also there is very weird
> observation that the if VM stalled, password window renders in a
> cripple way just after connection when no VNC region refreshes are
> made yet by mouse, vm-is-alive case rendering after-c-a-d prompt
> properly. Probably it`ll be easier to check behaviour against Paolo`
> patchset for RH kernel, though I`m still in progress with cutting out
> the rest to adapt them on top of bare 3.10, for ones with access to
> git log it should be easier :)

Argh i forgot cpu_sync_all_states. Will resend shortly.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcelo Tosatti]

On Fri, Aug 22, 2014 at 04:05:07PM -0300, Marcelo Tosatti wrote:
> On Fri, Aug 22, 2014 at 10:39:38PM +0400, Andrey Korolyov wrote:
> > On Fri, Aug 22, 2014 at 9:45 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> > > On Fri, Aug 22, 2014 at 08:44:53PM +0400, Andrey Korolyov wrote:
> > >> >
> > >> > I`m running 3.10, so patches are not here, will try 3.16 soon. Even if
> > >> > problem will be fixed, it still will be specific for 2.1, earlier
> > >> > releases working well and I`ll bisect at a time.
> > >>
> > >> Thanks, using 3.16 helped indeed. Though the bug remains as is at 2.1
> > >> on LTS 3.10, should I find the breaking commit, or it is better to
> > >> backport kvm subsystem changes related to apic to 3.10?
> > >
> > > Andrey,
> > >
> > > Can you try the patchset (which includes cpu_clean_all_dirty) please?
> > > It should be equivalent to your proposed patch.
> > >
> > >
> > 
> > 
> > Just checked case for Windows, bug is still here (twelve continuous
> > ping-pong migrations between two hosts). Also there is very weird
> > observation that the if VM stalled, password window renders in a
> > cripple way just after connection when no VNC region refreshes are
> > made yet by mouse, vm-is-alive case rendering after-c-a-d prompt
> > properly. Probably it`ll be easier to check behaviour against Paolo`
> > patchset for RH kernel, though I`m still in progress with cutting out
> > the rest to adapt them on top of bare 3.10, for ones with access to
> > git log it should be easier :)
> 
> Argh i forgot cpu_sync_all_states. Will resend shortly.

Well, can you add cpu_synchronize_all_states() before 
the cpu_clean_all_dirty() call and retry?

Sorry.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Fri, Aug 22, 2014 at 11:05 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> On Fri, Aug 22, 2014 at 04:05:07PM -0300, Marcelo Tosatti wrote:
>> On Fri, Aug 22, 2014 at 10:39:38PM +0400, Andrey Korolyov wrote:
>> > On Fri, Aug 22, 2014 at 9:45 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
>> > > On Fri, Aug 22, 2014 at 08:44:53PM +0400, Andrey Korolyov wrote:
>> > >> >
>> > >> > I`m running 3.10, so patches are not here, will try 3.16 soon. Even if
>> > >> > problem will be fixed, it still will be specific for 2.1, earlier
>> > >> > releases working well and I`ll bisect at a time.
>> > >>
>> > >> Thanks, using 3.16 helped indeed. Though the bug remains as is at 2.1
>> > >> on LTS 3.10, should I find the breaking commit, or it is better to
>> > >> backport kvm subsystem changes related to apic to 3.10?
>> > >
>> > > Andrey,
>> > >
>> > > Can you try the patchset (which includes cpu_clean_all_dirty) please?
>> > > It should be equivalent to your proposed patch.
>> > >
>> > >
>> >
>> >
>> > Just checked case for Windows, bug is still here (twelve continuous
>> > ping-pong migrations between two hosts). Also there is very weird
>> > observation that the if VM stalled, password window renders in a
>> > cripple way just after connection when no VNC region refreshes are
>> > made yet by mouse, vm-is-alive case rendering after-c-a-d prompt
>> > properly. Probably it`ll be easier to check behaviour against Paolo`
>> > patchset for RH kernel, though I`m still in progress with cutting out
>> > the rest to adapt them on top of bare 3.10, for ones with access to
>> > git log it should be easier :)
>>
>> Argh i forgot cpu_sync_all_states. Will resend shortly.
>
> Well, can you add cpu_synchronize_all_states() before
> the cpu_clean_all_dirty() call and retry?
>
> Sorry.
>

Still nope, and libvirtd-receiver crashed eventually during test, lol.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcelo Tosatti]

On Fri, Aug 22, 2014 at 04:05:46PM -0300, Marcelo Tosatti wrote:
> On Fri, Aug 22, 2014 at 04:05:07PM -0300, Marcelo Tosatti wrote:
> > On Fri, Aug 22, 2014 at 10:39:38PM +0400, Andrey Korolyov wrote:
> > > On Fri, Aug 22, 2014 at 9:45 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> > > > On Fri, Aug 22, 2014 at 08:44:53PM +0400, Andrey Korolyov wrote:
> > > >> >
> > > >> > I`m running 3.10, so patches are not here, will try 3.16 soon. Even if
> > > >> > problem will be fixed, it still will be specific for 2.1, earlier
> > > >> > releases working well and I`ll bisect at a time.
> > > >>
> > > >> Thanks, using 3.16 helped indeed. Though the bug remains as is at 2.1
> > > >> on LTS 3.10, should I find the breaking commit, or it is better to
> > > >> backport kvm subsystem changes related to apic to 3.10?
> > > >
> > > > Andrey,
> > > >
> > > > Can you try the patchset (which includes cpu_clean_all_dirty) please?
> > > > It should be equivalent to your proposed patch.
> > > >
> > > >
> > > 
> > > 
> > > Just checked case for Windows, bug is still here (twelve continuous
> > > ping-pong migrations between two hosts). Also there is very weird
> > > observation that the if VM stalled, password window renders in a
> > > cripple way just after connection when no VNC region refreshes are
> > > made yet by mouse, vm-is-alive case rendering after-c-a-d prompt
> > > properly. Probably it`ll be easier to check behaviour against Paolo`
> > > patchset for RH kernel, though I`m still in progress with cutting out
> > > the rest to adapt them on top of bare 3.10, for ones with access to
> > > git log it should be easier :)
> > 
> > Argh i forgot cpu_sync_all_states. Will resend shortly.
> 
> Well, can you add cpu_synchronize_all_states() before 
> the cpu_clean_all_dirty() call and retry?
> 
> Sorry.

Andrey,

Can you give instructions on how to reproduce please?

- qemu.git codebase (if you have any patches relative to a
given commit id, please provide the patches).
- qemu command line.
- how to recreate guest disk contents.
- how to recreate workload at which point migration 
fails.
- migration command relative to last item.


Thanks

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

>
> Andrey,
>
> Can you give instructions on how to reproduce please?
>

Please find answers inline:

> - qemu.git codebase (if you have any patches relative to a
> given commit id, please provide the patches).


rolled to bare 2.1-release to reproduce, for 3.10 I am hitting issue
with and without patches from previous message. With 3.16.0, I am not
able to reproduce issue anymore on bare 2.1. Both virtio-dp and
regular virtio-blk are affected, though first option hit the issue
always at a single migration check with HV timer, so I`d suggest to
test against it (as in mine args string below). With hvapic option set
emulator tends to hit issue more frequently than without it.

> - qemu command line.

qemu-system-x86_64 -enable-kvm -name vm29107 -S -machine
pc-i440fx-2.1,accel=kvm,usb=off -cpu
qemu64,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -bios
/usr/share/seabios/bios.bin-1.7.4 -m 512 -realtime mlock=off -smp
12,sockets=1,cores=12,threads=12 -numa node,nodeid=0,cpus=0-11,mem=512
-uuid 53646494-fe6c-4b5d-b6d0-c333b4f20582 -no-user-config -nodefaults
-device sga -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm29107.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
usb-ehci,id=usb,bus=pci.0,addr=0x5 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive
file=rbd:dev-rack2/vm29107-WfV:id=qemukvm:key=secret:auth_supported=cephx\;none:mon_host=10.6.0.1\:6789\;10.6.0.3\:6789\;10.6.0.4\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,aio=native
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fds=26:27:28:29,id=hostnet0,vhost=on,vhostfds=30:31:32:33
-device virtio-net-pci,mq=on,vectors=10,netdev=hostnet0,id=net0,mac=52:54:00:10:06:9a,bus=pci.0,addr=0x3
-netdev tap,fds=34:35:36:37,id=hostnet1,vhost=on,vhostfds=38:39:40:41
-device virtio-net-pci,mq=on,vectors=10,netdev=hostnet1,id=net1,mac=52:54:00:10:06:9b,bus=pci.0,addr=0x4
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/vm29107.sock,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1
-vnc 0.0.0.0:0 -k en-us -device VGA,id=video0,bus=pci.0,addr=0x2
-object iothread,id=vm29107blk0 -set
device.virtio-disk0.config-wce=off -set device.virtio-disk0.scsi=off
-set device.virtio-disk0.iothread=vm29107blk0 -m
512,slots=62,maxmem=16384M -object
memory-backend-ram,id=mem0,size=256M -device
pc-dimm,id=dimm0,node=0,memdev=mem0 -object
memory-backend-ram,id=mem1,size=256M -device
pc-dimm,id=dimm1,node=0,memdev=mem1 -object
memory-backend-ram,id=mem2,size=256M -device
pc-dimm,id=dimm2,node=0,memdev=mem2 -object
memory-backend-ram,id=mem3,size=256M -device
pc-dimm,id=dimm3,node=0,memdev=mem3 -object
memory-backend-ram,id=mem4,size=256M -device
pc-dimm,id=dimm4,node=0,memdev=mem4 -object
memory-backend-ram,id=mem5,size=256M -device
pc-dimm,id=dimm5,node=0,memdev=mem5 -object
memory-backend-ram,id=mem6,size=256M -device
pc-dimm,id=dimm6,node=0,memdev=mem6 -object
memory-backend-ram,id=mem7,size=256M -device
pc-dimm,id=dimm7,node=0,memdev=mem7 -object
memory-backend-ram,id=mem8,size=256M -device
pc-dimm,id=dimm8,node=0,memdev=mem8 -object
memory-backend-ram,id=mem9,size=256M -device
pc-dimm,id=dimm9,node=0,memdev=mem9 -object
memory-backend-ram,id=mem10,size=256M -device
pc-dimm,id=dimm10,node=0,memdev=mem10 -object
memory-backend-ram,id=mem11,size=256M -device
pc-dimm,id=dimm11,node=0,memdev=mem11 -object
memory-backend-ram,id=mem12,size=256M -device
pc-dimm,id=dimm12,node=0,memdev=mem12 -object
memory-backend-ram,id=mem13,size=256M -device
pc-dimm,id=dimm13,node=0,memdev=mem13

> - how to recreate guest disk contents.

In my case, it is just bare installation of W2008R2 x64, I can share
it off-list if necessary.

> - how to recreate workload at which point migration
> fails.

Migration does not fail itself, the VM` disk seemingly is.
Reproduction is quite simple - just boot up a VM, migrate it N times
and try to log in. Failed case will hang on the progress screen (or
you may log in before and check the disk availability by any other
convenient method).

> - migration command relative to last item.

It is p2p libvirt live migration:

for i in $(seq 1 6) ; do virsh migrate --live --persistent
--undefinesource vm29107 qemu+tcp://twin2/system ; sleep 5; ssh twin2
"virsh migrate --live --persistent --undefinesource vm29107
qemu+tcp://twin0/system" ; done

>
>
> Thanks

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

[-- Attachment #1: Type: text/plain, Size: 367 bytes --]

Sorry, I was a bit inaccurate in my thoughts at Fri about necessary
amount of work, patch lays perfectly on 3.10 with bit of monkey
rewrites. The attached one fixed problem for me - it represents
0b10a1c87a2b0fb459baaefba9cb163dbb8d3344,
0bc830b05c667218d703f2026ec866c49df974fc,
44847dea79751e95665a439f8c63a65e51da8e1f and
673f7b4257a1fe7b181e1a1182ecc2b6b2b795f1.

[-- Attachment #2: pbonzini-ioapic-3.10.patch --]
[-- Type: text/x-diff, Size: 6818 bytes --]

diff -ru linux-3.10.11/arch/ia64/kvm/kvm-ia64.c linux-3.10.11.patched-ioapic/arch/ia64/kvm/kvm-ia64.c
--- linux-3.10.11/arch/ia64/kvm/kvm-ia64.c	2013-09-08 09:10:14.000000000 +0400
+++ linux-3.10.11.patched-ioapic/arch/ia64/kvm/kvm-ia64.c	2014-08-24 19:49:25.723072383 +0400
@@ -199,6 +199,7 @@
 	case KVM_CAP_IRQCHIP:
 	case KVM_CAP_MP_STATE:
 	case KVM_CAP_IRQ_INJECT_STATUS:
+	case KVM_CAP_IOAPIC_POLARITY_IGNORED:
 		r = 1;
 		break;
 	case KVM_CAP_COALESCED_MMIO:
diff -ru linux-3.10.11/arch/x86/kvm/x86.c linux-3.10.11.patched-ioapic/arch/x86/kvm/x86.c
--- linux-3.10.11/arch/x86/kvm/x86.c	2013-09-08 09:10:14.000000000 +0400
+++ linux-3.10.11.patched-ioapic/arch/x86/kvm/x86.c	2014-08-24 19:50:06.553716276 +0400
@@ -2537,6 +2537,7 @@
 	case KVM_CAP_GET_TSC_KHZ:
 	case KVM_CAP_KVMCLOCK_CTRL:
 	case KVM_CAP_READONLY_MEM:
+	case KVM_CAP_IOAPIC_POLARITY_IGNORED:
 #ifdef CONFIG_KVM_DEVICE_ASSIGNMENT
 	case KVM_CAP_ASSIGN_DEV_IRQ:
 	case KVM_CAP_PCI_2_3:
diff -ru linux-3.10.11/include/uapi/linux/kvm.h linux-3.10.11.patched-ioapic/include/uapi/linux/kvm.h
--- linux-3.10.11/include/uapi/linux/kvm.h	2013-09-08 09:10:14.000000000 +0400
+++ linux-3.10.11.patched-ioapic/include/uapi/linux/kvm.h	2014-08-24 19:51:10.975577204 +0400
@@ -666,6 +666,7 @@
 #define KVM_CAP_IRQ_MPIC 90
 #define KVM_CAP_PPC_RTAS 91
 #define KVM_CAP_IRQ_XICS 92
+#define KVM_CAP_IOAPIC_POLARITY_IGNORED 93
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
diff -ru linux-3.10.11/virt/kvm/ioapic.c linux-3.10.11.patched-ioapic/virt/kvm/ioapic.c
--- linux-3.10.11/virt/kvm/ioapic.c	2013-09-08 09:10:14.000000000 +0400
+++ linux-3.10.11.patched-ioapic/virt/kvm/ioapic.c	2014-08-24 19:59:26.755137527 +0400
@@ -50,7 +50,7 @@
 #else
 #define ioapic_debug(fmt, arg...)
 #endif
-static int ioapic_deliver(struct kvm_ioapic *vioapic, int irq,
+static int ioapic_service(struct kvm_ioapic *vioapic, int irq
 		bool line_status);
 
 static unsigned long ioapic_read_indirect(struct kvm_ioapic *ioapic,
@@ -163,23 +163,67 @@
 	return false;
 }
 
-static int ioapic_service(struct kvm_ioapic *ioapic, unsigned int idx,
-		bool line_status)
+static int ioapic_set_irq(struct kvm_ioapic *ioapic, unsigned int irq,
+		int irq_level, bool line_status)
 {
-	union kvm_ioapic_redirect_entry *pent;
-	int injected = -1;
+	union kvm_ioapic_redirect_entry entry;
+	u32 mask = 1 << irq;
+	u32 old_irr;
+	int edge, ret;
 
-	pent = &ioapic->redirtbl[idx];
+	entry = ioapic->redirtbl[irq];
+	edge = (entry.fields.trig_mode == IOAPIC_EDGE_TRIG);
 
-	if (!pent->fields.mask) {
-		injected = ioapic_deliver(ioapic, idx, line_status);
-		if (injected && pent->fields.trig_mode == IOAPIC_LEVEL_TRIG)
-			pent->fields.remote_irr = 1;
+	if (!irq_level) {
+		ioapic->irr &= ~mask;
+		ret = 1;
+		goto out;
+	}
+
+	/*
+	 * Return 0 for coalesced interrupts; for edge-triggered interrupts,
+	 * this only happens if a previous edge has not been delivered due
+	 * do masking.  For level interrupts, the remote_irr field tells
+	 * us if the interrupt is waiting for an EOI.
+	 *
+	 * RTC is special: it is edge-triggered, but userspace likes to know
+	 * if it has been already ack-ed via EOI because coalesced RTC
+	 * interrupts lead to time drift in Windows guests.  So we track
+	 * EOI manually for the RTC interrupt.
+	 */
+	if (irq == RTC_GSI && line_status &&
+		rtc_irq_check_coalesced(ioapic)) {
+		ret = 0;
+		goto out;
 	}
 
-	return injected;
+	old_irr = ioapic->irr;
+	ioapic->irr |= mask;
+	if ((edge && old_irr == ioapic->irr) ||
+	    (!edge && entry.fields.remote_irr)) {
+		ret = 0;
+		goto out;
+	}
+
+	ret = ioapic_service(ioapic, irq, line_status);
+
+out:
+	trace_kvm_ioapic_set_irq(entry.bits, irq, ret == 0);
+	return ret;
+}
+
+static void kvm_ioapic_inject_all(struct kvm_ioapic *ioapic, unsigned long irr)
+{
+	u32 idx;
+
+	rtc_irq_eoi_tracking_reset(ioapic);
+	for_each_set_bit(idx, &irr, IOAPIC_NUM_PINS)
+		ioapic_set_irq(ioapic, idx, 1, true);
+
+	kvm_rtc_eoi_tracking_restore_all(ioapic);
 }
 
+
 static void update_handled_vectors(struct kvm_ioapic *ioapic)
 {
 	DECLARE_BITMAP(handled_vectors, 256);
@@ -282,12 +326,15 @@
 	}
 }
 
-static int ioapic_deliver(struct kvm_ioapic *ioapic, int irq, bool line_status)
+static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status)
 {
 	union kvm_ioapic_redirect_entry *entry = &ioapic->redirtbl[irq];
 	struct kvm_lapic_irq irqe;
 	int ret;
 
+	if (entry->fields.mask)
+	    return -1;
+
 	ioapic_debug("dest=%x dest_mode=%x delivery_mode=%x "
 		     "vector=%x trig_mode=%x\n",
 		     entry->fields.dest_id, entry->fields.dest_mode,
@@ -302,6 +349,10 @@
 	irqe.level = 1;
 	irqe.shorthand = 0;
 
+	if (irqe.trig_mode == IOAPIC_EDGE_TRIG)
+	    ioapic->irr &= ~(1 << irq);
+
+
 	if (irq == RTC_GSI && line_status) {
 		BUG_ON(ioapic->rtc_status.pending_eoi != 0);
 		ret = kvm_irq_delivery_to_apic(ioapic->kvm, NULL, &irqe,
@@ -309,6 +360,8 @@
 		ioapic->rtc_status.pending_eoi = ret;
 	} else
 		ret = kvm_irq_delivery_to_apic(ioapic->kvm, NULL, &irqe, NULL);
+	if (ret && irqe.trig_mode == IOAPIC_LEVEL_TRIG)
+	    entry->fields.remote_irr = 1;
 
 	return ret;
 }
@@ -316,39 +369,15 @@
 int kvm_ioapic_set_irq(struct kvm_ioapic *ioapic, int irq, int irq_source_id,
 		       int level, bool line_status)
 {
-	u32 old_irr;
-	u32 mask = 1 << irq;
-	union kvm_ioapic_redirect_entry entry;
 	int ret, irq_level;
 
 	BUG_ON(irq < 0 || irq >= IOAPIC_NUM_PINS);
 
 	spin_lock(&ioapic->lock);
-	old_irr = ioapic->irr;
 	irq_level = __kvm_irq_line_state(&ioapic->irq_states[irq],
 					 irq_source_id, level);
-	entry = ioapic->redirtbl[irq];
-	irq_level ^= entry.fields.polarity;
-	if (!irq_level) {
-		ioapic->irr &= ~mask;
-		ret = 1;
-	} else {
-		int edge = (entry.fields.trig_mode == IOAPIC_EDGE_TRIG);
+	ret = ioapic_set_irq(ioapic, irq, irq_level, line_status);
 
-		if (irq == RTC_GSI && line_status &&
-			rtc_irq_check_coalesced(ioapic)) {
-			ret = 0; /* coalesced */
-			goto out;
-		}
-		ioapic->irr |= mask;
-		if ((edge && old_irr != ioapic->irr) ||
-		    (!edge && !entry.fields.remote_irr))
-			ret = ioapic_service(ioapic, irq, line_status);
-		else
-			ret = 0; /* report coalesced interrupt */
-	}
-out:
-	trace_kvm_ioapic_set_irq(entry.bits, irq, ret == 0);
 	spin_unlock(&ioapic->lock);
 
 	return ret;
@@ -394,7 +423,7 @@
 
 		ASSERT(ent->fields.trig_mode == IOAPIC_LEVEL_TRIG);
 		ent->fields.remote_irr = 0;
-		if (!ent->fields.mask && (ioapic->irr & (1 << i)))
+		if (ioapic->irr & (1 << i))
 			ioapic_service(ioapic, i, false);
 	}
 }
@@ -595,9 +624,10 @@
 
 	spin_lock(&ioapic->lock);
 	memcpy(ioapic, state, sizeof(struct kvm_ioapic_state));
+	ioapic->irr = 0;
 	update_handled_vectors(ioapic);
 	kvm_vcpu_request_scan_ioapic(kvm);
-	kvm_rtc_eoi_tracking_restore_all(ioapic);
+	kvm_ioapic_inject_all(ioapic, state->irr);
 	spin_unlock(&ioapic->lock);
 	return 0;
 }

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Paolo Bonzini]

Il 24/08/2014 18:19, Andrey Korolyov ha scritto:
> Sorry, I was a bit inaccurate in my thoughts at Fri about necessary
> amount of work, patch lays perfectly on 3.10 with bit of monkey
> rewrites. The attached one fixed problem for me - it represents
> 0b10a1c87a2b0fb459baaefba9cb163dbb8d3344,
> 0bc830b05c667218d703f2026ec866c49df974fc,
> 44847dea79751e95665a439f8c63a65e51da8e1f and
> 673f7b4257a1fe7b181e1a1182ecc2b6b2b795f1.

So, with these changes, Marcelo's patch does not hang up your guest anymore?

Paolo

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Sun, Aug 24, 2014 at 8:35 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 24/08/2014 18:19, Andrey Korolyov ha scritto:
>> Sorry, I was a bit inaccurate in my thoughts at Fri about necessary
>> amount of work, patch lays perfectly on 3.10 with bit of monkey
>> rewrites. The attached one fixed problem for me - it represents
>> 0b10a1c87a2b0fb459baaefba9cb163dbb8d3344,
>> 0bc830b05c667218d703f2026ec866c49df974fc,
>> 44847dea79751e95665a439f8c63a65e51da8e1f and
>> 673f7b4257a1fe7b181e1a1182ecc2b6b2b795f1.
>
> So, with these changes, Marcelo's patch does not hang up your guest anymore?
>
> Paolo
>

If I may reword, Marcelo`s proposed states sync with revert-revert of
agraf`s patch, does not break anything for Windows (migration works
well for any variant of emulator with modified kernel modules). Let me
check if initially reported issue (lost I/O interrupts) is gone for
the current situation (patched kernel plus Marcelo` patch).

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Sun, Aug 24, 2014 at 8:57 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> On Sun, Aug 24, 2014 at 8:35 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> Il 24/08/2014 18:19, Andrey Korolyov ha scritto:
>>> Sorry, I was a bit inaccurate in my thoughts at Fri about necessary
>>> amount of work, patch lays perfectly on 3.10 with bit of monkey
>>> rewrites. The attached one fixed problem for me - it represents
>>> 0b10a1c87a2b0fb459baaefba9cb163dbb8d3344,
>>> 0bc830b05c667218d703f2026ec866c49df974fc,
>>> 44847dea79751e95665a439f8c63a65e51da8e1f and
>>> 673f7b4257a1fe7b181e1a1182ecc2b6b2b795f1.
>>
>> So, with these changes, Marcelo's patch does not hang up your guest anymore?
>>
>> Paolo
>>
>
> If I may reword, Marcelo`s proposed states sync with revert-revert of
> agraf`s patch, does not break anything for Windows (migration works
> well for any variant of emulator with modified kernel modules). Let me
> check if initially reported issue (lost I/O interrupts) is gone for
> the current situation (patched kernel plus Marcelo` patch).


patched kernel + any 2.1 variant + Windows = works
patched kernel + patched 2.1 + Linux + disk workload = works fine
bare kernel + any 2.1 variant + Windows = disk stale
bare kernel + proposed patch from Marcelo + Linux + disk workload =
works fine (bare kernel + 2.1 release works at this point as tested
earlier)


Also guest 3.10.52 caused literally a rain of WTFs crashing emulator
with an emulation error in every case tested with live migration...
3.10.11 works fine. There will be another report soon I suppose though
it is nearly impossible to do a proper bisection in a sane time.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

Forgot to mention, _actual_ patch from above. Adding
cpu_synchronize_all_states() bringing old bug with lost interrupts
back.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Paolo Bonzini]

Il 24/08/2014 22:14, Andrey Korolyov ha scritto:
> Forgot to mention, _actual_ patch from above. Adding
> cpu_synchronize_all_states() bringing old bug with lost interrupts
> back.

Are you adding it before or after cpu_clean_all_dirty?

Paolo

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Mon, Aug 25, 2014 at 2:45 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 24/08/2014 22:14, Andrey Korolyov ha scritto:
>> Forgot to mention, _actual_ patch from above. Adding
>> cpu_synchronize_all_states() bringing old bug with lost interrupts
>> back.
>
> Are you adding it before or after cpu_clean_all_dirty?
>
> Paolo

I`ve added right before, as suggested.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcelo Tosatti]

On Sun, Aug 24, 2014 at 10:51:38PM +0400, Andrey Korolyov wrote:
> On Sun, Aug 24, 2014 at 8:57 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> > On Sun, Aug 24, 2014 at 8:35 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >> Il 24/08/2014 18:19, Andrey Korolyov ha scritto:
> >>> Sorry, I was a bit inaccurate in my thoughts at Fri about necessary
> >>> amount of work, patch lays perfectly on 3.10 with bit of monkey
> >>> rewrites. The attached one fixed problem for me - it represents
> >>> 0b10a1c87a2b0fb459baaefba9cb163dbb8d3344,
> >>> 0bc830b05c667218d703f2026ec866c49df974fc,
> >>> 44847dea79751e95665a439f8c63a65e51da8e1f and
> >>> 673f7b4257a1fe7b181e1a1182ecc2b6b2b795f1.
> >>
> >> So, with these changes, Marcelo's patch does not hang up your guest anymore?
> >>
> >> Paolo
> >>
> >
> > If I may reword, Marcelo`s proposed states sync with revert-revert of
> > agraf`s patch, does not break anything for Windows (migration works
> > well for any variant of emulator with modified kernel modules). Let me
> > check if initially reported issue (lost I/O interrupts) is gone for
> > the current situation (patched kernel plus Marcelo` patch).
> 
> 
> patched kernel + any 2.1 variant + Windows = works
> patched kernel + patched 2.1 + Linux + disk workload = works fine

Andrey,

What patch is this again ?

> bare kernel + any 2.1 variant + Windows = disk stale
> bare kernel + proposed patch from Marcelo + Linux + disk workload =
> works fine (bare kernel + 2.1 release works at this point as tested
> earlier)
> 
> Also guest 3.10.52 caused literally a rain of WTFs crashing emulator
> with an emulation error in every case tested with live migration...
> 3.10.11 works fine. There will be another report soon I suppose though
> it is nearly impossible to do a proper bisection in a sane time.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Thu, Sep 4, 2014 at 8:38 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> On Sun, Aug 24, 2014 at 10:51:38PM +0400, Andrey Korolyov wrote:
>> On Sun, Aug 24, 2014 at 8:57 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
>> > On Sun, Aug 24, 2014 at 8:35 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> >> Il 24/08/2014 18:19, Andrey Korolyov ha scritto:
>> >>> Sorry, I was a bit inaccurate in my thoughts at Fri about necessary
>> >>> amount of work, patch lays perfectly on 3.10 with bit of monkey
>> >>> rewrites. The attached one fixed problem for me - it represents
>> >>> 0b10a1c87a2b0fb459baaefba9cb163dbb8d3344,
>> >>> 0bc830b05c667218d703f2026ec866c49df974fc,
>> >>> 44847dea79751e95665a439f8c63a65e51da8e1f and
>> >>> 673f7b4257a1fe7b181e1a1182ecc2b6b2b795f1.
>> >>
>> >> So, with these changes, Marcelo's patch does not hang up your guest anymore?
>> >>
>> >> Paolo
>> >>
>> >
>> > If I may reword, Marcelo`s proposed states sync with revert-revert of
>> > agraf`s patch, does not break anything for Windows (migration works
>> > well for any variant of emulator with modified kernel modules). Let me
>> > check if initially reported issue (lost I/O interrupts) is gone for
>> > the current situation (patched kernel plus Marcelo` patch).
>>
>>
>> patched kernel + any 2.1 variant + Windows = works
>> patched kernel + patched 2.1 + Linux + disk workload = works fine
>
> Andrey,
>
> What patch is this again ?
>
>

It is attached to the message which has a highest quoting order above
1), basically it is a monkey adaptation of newer ioapic patches for
3.10. RH kernel includes them, so Centos7/RH7 users should not observe
reported problem.

[1] https://www.mail-archive.com/qemu-devel@nongnu.org/msg252526.html

Using the opportunity - what are plans, if there are any, to revive
agraf`s patch which was the reason of an initial issue in this thread?
The Windows issue is most probably unrelated to it and I`d be happy to
check patch proposals because there are a lot of suffering with VMs
with large uptime and live migration when negative time shift appears.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcelo Tosatti]

On Thu, Sep 04, 2014 at 08:52:00PM +0400, Andrey Korolyov wrote:
> On Thu, Sep 4, 2014 at 8:38 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> > On Sun, Aug 24, 2014 at 10:51:38PM +0400, Andrey Korolyov wrote:
> >> On Sun, Aug 24, 2014 at 8:57 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> >> > On Sun, Aug 24, 2014 at 8:35 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >> >> Il 24/08/2014 18:19, Andrey Korolyov ha scritto:
> >> >>> Sorry, I was a bit inaccurate in my thoughts at Fri about necessary
> >> >>> amount of work, patch lays perfectly on 3.10 with bit of monkey
> >> >>> rewrites. The attached one fixed problem for me - it represents
> >> >>> 0b10a1c87a2b0fb459baaefba9cb163dbb8d3344,
> >> >>> 0bc830b05c667218d703f2026ec866c49df974fc,
> >> >>> 44847dea79751e95665a439f8c63a65e51da8e1f and
> >> >>> 673f7b4257a1fe7b181e1a1182ecc2b6b2b795f1.
> >> >>
> >> >> So, with these changes, Marcelo's patch does not hang up your guest anymore?
> >> >>
> >> >> Paolo
> >> >>
> >> >
> >> > If I may reword, Marcelo`s proposed states sync with revert-revert of
> >> > agraf`s patch, does not break anything for Windows (migration works
> >> > well for any variant of emulator with modified kernel modules). Let me
> >> > check if initially reported issue (lost I/O interrupts) is gone for
> >> > the current situation (patched kernel plus Marcelo` patch).
> >>
> >>
> >> patched kernel + any 2.1 variant + Windows = works
> >> patched kernel + patched 2.1 + Linux + disk workload = works fine
> >
> > Andrey,
> >
> > What patch is this again ?
> >
> >
> 
> It is attached to the message which has a highest quoting order above
> 1), basically it is a monkey adaptation of newer ioapic patches for
> 3.10. RH kernel includes them, so Centos7/RH7 users should not observe
> reported problem.
> 
> [1] https://www.mail-archive.com/qemu-devel@nongnu.org/msg252526.html
> 
> Using the opportunity - what are plans, if there are any, to revive
> agraf`s patch which was the reason of an initial issue in this thread?
> The Windows issue is most probably unrelated to it and I`d be happy to
> check patch proposals because there are a lot of suffering with VMs
> with large uptime and live migration when negative time shift appears.

I'll try to confirm via Marcin's testcase (vfio+VM migration on Linux guests), 
that the proposed patch is fixing the issue.

Getting a BSOD on Win2008-R2-x64 while trying to install virtio-win
driver (viostor related BSOD).

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcelo Tosatti]

On Thu, Sep 04, 2014 at 03:54:01PM -0300, Marcelo Tosatti wrote:
> On Thu, Sep 04, 2014 at 08:52:00PM +0400, Andrey Korolyov wrote:
> > On Thu, Sep 4, 2014 at 8:38 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> > > On Sun, Aug 24, 2014 at 10:51:38PM +0400, Andrey Korolyov wrote:
> > >> On Sun, Aug 24, 2014 at 8:57 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> > >> > On Sun, Aug 24, 2014 at 8:35 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > >> >> Il 24/08/2014 18:19, Andrey Korolyov ha scritto:
> > >> >>> Sorry, I was a bit inaccurate in my thoughts at Fri about necessary
> > >> >>> amount of work, patch lays perfectly on 3.10 with bit of monkey
> > >> >>> rewrites. The attached one fixed problem for me - it represents
> > >> >>> 0b10a1c87a2b0fb459baaefba9cb163dbb8d3344,
> > >> >>> 0bc830b05c667218d703f2026ec866c49df974fc,
> > >> >>> 44847dea79751e95665a439f8c63a65e51da8e1f and
> > >> >>> 673f7b4257a1fe7b181e1a1182ecc2b6b2b795f1.
> > >> >>
> > >> >> So, with these changes, Marcelo's patch does not hang up your guest anymore?
> > >> >>
> > >> >> Paolo
> > >> >>
> > >> >
> > >> > If I may reword, Marcelo`s proposed states sync with revert-revert of
> > >> > agraf`s patch, does not break anything for Windows (migration works
> > >> > well for any variant of emulator with modified kernel modules). Let me
> > >> > check if initially reported issue (lost I/O interrupts) is gone for
> > >> > the current situation (patched kernel plus Marcelo` patch).
> > >>
> > >>
> > >> patched kernel + any 2.1 variant + Windows = works
> > >> patched kernel + patched 2.1 + Linux + disk workload = works fine
> > >
> > > Andrey,
> > >
> > > What patch is this again ?
> > >
> > >
> > 
> > It is attached to the message which has a highest quoting order above
> > 1), basically it is a monkey adaptation of newer ioapic patches for
> > 3.10. RH kernel includes them, so Centos7/RH7 users should not observe
> > reported problem.
> > 
> > [1] https://www.mail-archive.com/qemu-devel@nongnu.org/msg252526.html
> > 
> > Using the opportunity - what are plans, if there are any, to revive
> > agraf`s patch which was the reason of an initial issue in this thread?
> > The Windows issue is most probably unrelated to it and I`d be happy to
> > check patch proposals because there are a lot of suffering with VMs
> > with large uptime and live migration when negative time shift appears.
> 
> I'll try to confirm via Marcin's testcase (vfio+VM migration on Linux guests), 
> that the proposed patch is fixing the issue.
> 
> Getting a BSOD on Win2008-R2-x64 while trying to install virtio-win
> driver (viostor related BSOD).

What version of virtio-win are you using?

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Andrey Korolyov]

On Thu, Sep 4, 2014 at 10:54 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> On Thu, Sep 04, 2014 at 03:54:01PM -0300, Marcelo Tosatti wrote:
>> On Thu, Sep 04, 2014 at 08:52:00PM +0400, Andrey Korolyov wrote:
>> > On Thu, Sep 4, 2014 at 8:38 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
>> > > On Sun, Aug 24, 2014 at 10:51:38PM +0400, Andrey Korolyov wrote:
>> > >> On Sun, Aug 24, 2014 at 8:57 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
>> > >> > On Sun, Aug 24, 2014 at 8:35 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> > >> >> Il 24/08/2014 18:19, Andrey Korolyov ha scritto:
>> > >> >>> Sorry, I was a bit inaccurate in my thoughts at Fri about necessary
>> > >> >>> amount of work, patch lays perfectly on 3.10 with bit of monkey
>> > >> >>> rewrites. The attached one fixed problem for me - it represents
>> > >> >>> 0b10a1c87a2b0fb459baaefba9cb163dbb8d3344,
>> > >> >>> 0bc830b05c667218d703f2026ec866c49df974fc,
>> > >> >>> 44847dea79751e95665a439f8c63a65e51da8e1f and
>> > >> >>> 673f7b4257a1fe7b181e1a1182ecc2b6b2b795f1.
>> > >> >>
>> > >> >> So, with these changes, Marcelo's patch does not hang up your guest anymore?
>> > >> >>
>> > >> >> Paolo
>> > >> >>
>> > >> >
>> > >> > If I may reword, Marcelo`s proposed states sync with revert-revert of
>> > >> > agraf`s patch, does not break anything for Windows (migration works
>> > >> > well for any variant of emulator with modified kernel modules). Let me
>> > >> > check if initially reported issue (lost I/O interrupts) is gone for
>> > >> > the current situation (patched kernel plus Marcelo` patch).
>> > >>
>> > >>
>> > >> patched kernel + any 2.1 variant + Windows = works
>> > >> patched kernel + patched 2.1 + Linux + disk workload = works fine
>> > >
>> > > Andrey,
>> > >
>> > > What patch is this again ?
>> > >
>> > >
>> >
>> > It is attached to the message which has a highest quoting order above
>> > 1), basically it is a monkey adaptation of newer ioapic patches for
>> > 3.10. RH kernel includes them, so Centos7/RH7 users should not observe
>> > reported problem.
>> >
>> > [1] https://www.mail-archive.com/qemu-devel@nongnu.org/msg252526.html
>> >
>> > Using the opportunity - what are plans, if there are any, to revive
>> > agraf`s patch which was the reason of an initial issue in this thread?
>> > The Windows issue is most probably unrelated to it and I`d be happy to
>> > check patch proposals because there are a lot of suffering with VMs
>> > with large uptime and live migration when negative time shift appears.
>>
>> I'll try to confirm via Marcin's testcase (vfio+VM migration on Linux guests),
>> that the proposed patch is fixing the issue.
>>
>> Getting a BSOD on Win2008-R2-x64 while trying to install virtio-win
>> driver (viostor related BSOD).
>
> What version of virtio-win are you using?
>

Bug appears on both 0.1.76 and 0.1.81 (and probably it is at least
specific to virtio, if not to more generic nature). BSOD sounds
strange for me, are you using virtio drivers at the install stage or
you trying to change the disk model after installation? I`ve never
went second way, so can`t tell if BSOD is here or not.

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Eduardo Habkost]

On Mon, Aug 04, 2014 at 06:30:09PM +0200, Marcin Gibuła wrote:
> W dniu 2014-07-31 13:27, Marcin Gibuła pisze:
> >>>Can you dump *env before and after the call to kvm_arch_get_registers?
> >>
> >>Yes, but it seems they are equal - I used memcmp() to compare them. Is
> >>there any other side effect that cpu_synchronize_all_states() may have?
> >
> >I think I found it.
> >
> >The reason for hang is, because when second call to
> >kvm_arch_get_registers() is skipped, it also skips kvm_get_apic() which
> >updates cpu->apic_state.
> 
> Paolo,
> 
> is this analysis deep enough for you? I don't know if that can be fixed with
> existing api as cpu_synchronize_all_states() is all or nothing kind of
> stuff.
> 
> Kvmclock needs it only to read current cpu registers, so syncing everything
> is not really necessary. Perhaps exporting one of kvm_arch_get_* would be
> enough. And it wouldn't mess with lazy get/put.
> 
> On the other hand, if in future any other driver adds
> cpu_synchronize_all_states() in its change state callback it could result in
> same error so perhaps more generic approach is needed.

Does anybody know why the APIC state loaded by the first call to
kvm_arch_get_registers() is wrong, in the first place? What exactly is
different in the APIC state in the second kvm_arch_get_registers() call,
and when/why does it change?

If cpu_synchronize_state() does the wrong thing if it is called at the
wrong moment, then we may have other hidden bugs, because the user can
trigger cpu_synchronize_all_states() calls arbitrarily using monitor
commands.

-- 
Eduardo

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Marcin Gibuła]

> Does anybody know why the APIC state loaded by the first call to
> kvm_arch_get_registers() is wrong, in the first place? What exactly is
> different in the APIC state in the second kvm_arch_get_registers() call,
> and when/why does it change?
>
> If cpu_synchronize_state() does the wrong thing if it is called at the
> wrong moment, then we may have other hidden bugs, because the user can
> trigger cpu_synchronize_all_states() calls arbitrarily using monitor
> commands.

My guess is, it's not wrong, it's just outdated when second call 
occures. Maybe it's an ordering issue - could kvmclock state change 
handler be called before other activity is suspended (?)

I didn't pursue it further, cause I don't know too much (anything 
really) about QEMU/APIC internals and how to track its changes.

-- 
mg

Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration

[Eduardo Habkost]

On Fri, Oct 10, 2014 at 09:33:04AM +0200, Marcin Gibuła wrote:
> >Does anybody know why the APIC state loaded by the first call to
> >kvm_arch_get_registers() is wrong, in the first place? What exactly is
> >different in the APIC state in the second kvm_arch_get_registers() call,
> >and when/why does it change?
> >
> >If cpu_synchronize_state() does the wrong thing if it is called at the
> >wrong moment, then we may have other hidden bugs, because the user can
> >trigger cpu_synchronize_all_states() calls arbitrarily using monitor
> >commands.
> 
> My guess is, it's not wrong, it's just outdated when second call occures.
> Maybe it's an ordering issue - could kvmclock state change handler be called
> before other activity is suspended (?)

Changing the ordering could just hide the problem. If the APIC state in
the kernel can change outside ioctl(KVM_RUN) calls, then it can't be
cached the same way the other registers loaded by
kvm_arch_get_registers() are. The current code invalidates the QEMU
register data (i.e. sets kvm_vcpu_dirty=false) only on 3 cases: 1) VCPU
init; 2) VCPU reset; 3) on kvm_cpu_exec(), immediately before calling
ioctl(KVM_RUN).

We may need a separate API to load APIC state? I don't know what are the
actual users X86CPU.apic_state, today.

-- 
Eduardo