From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55977) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1X7lhK-0005Yw-Mf for qemu-devel@nongnu.org; Thu, 17 Jul 2014 09:26:19 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1X7lhC-0000ZY-8m for qemu-devel@nongnu.org; Thu, 17 Jul 2014 09:26:10 -0400 Received: from mx.beyond.pl ([92.43.117.49]:33465) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1X7lhB-0000ZM-VC for qemu-devel@nongnu.org; Thu, 17 Jul 2014 09:26:02 -0400 Message-ID: <53C7CEE5.4080006@beyond.pl> Date: Thu, 17 Jul 2014 15:25:57 +0200 From: =?UTF-8?B?TWFyY2luIEdpYnXFgmE=?= MIME-Version: 1.0 References: <20140715050318.GD26186@grmbl.mre> <20140715210948.GA20036@amt.cnet> <53C5A4C9.80609@redhat.com> <20140716011634.GA30717@amt.cnet> <20140716115229.GA7741@amt.cnet> <53C6EE7C.60702@beyond.pl> <53C79C41.4000800@beyond.pl> <53C7B989.9000203@beyond.pl> In-Reply-To: Content-Type: multipart/mixed; boundary="------------000704080509070305080901" Subject: Re: [Qemu-devel] latest rc: virtio-blk hangs forever after migration List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Andrey Korolyov Cc: Amit Shah , Paolo Bonzini , Marcelo Tosatti , Fam Zheng , "qemu-devel@nongnu.org" This is a multi-part message in MIME format. --------------000704080509070305080901 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit >> 2.1-rc2 behaves exactly the same. >> >> Interestingly enough, reseting guest system causes I/O to work again. So >> it's not qemu that hangs on IO, rather it fails to notify guest about >> completed operations that were issued during migration. >> >> And its somehow caused by calling cpu_synchronize_all_states() inside >> kvmclock_vm_state_change(). >> >> >> >> As for testing with cache=writeback, I'll try to setup some iscsi to test >> it. > > Awesome, thanks! AFAIK you`ll not be able to use write cache with > iscsi for migration. VM which had a reset before hangs always when > freshly launched have a chance to be migrated successfully. And yes, > it looks like lower layer forgetting to notify driver about some > operations at a glance. Andrey, could you try attached patch? It's an incredibly ugly workaround that calls cpu_synchronize_all_states() in a way that bypasses lazy execution logic. But it works for me. If that works for you as well, its somehow related to lazy execution of cpu_synchronize_all_states. -- mg --------------000704080509070305080901 Content-Type: text/x-patch; name="io-hang.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="io-hang.patch" diff -ru qemu-2.1.0-rc2/cpus.c qemu-2.1.0-rc2-fixed/cpus.c --- qemu-2.1.0-rc2/cpus.c 2014-07-15 23:49:14.000000000 +0200 +++ qemu-2.1.0-rc2-fixed/cpus.c 2014-07-17 15:09:09.306696284 +0200 @@ -505,6 +505,15 @@ } } +void cpu_synchronize_all_states_always(void) +{ + CPUState *cpu; + + CPU_FOREACH(cpu) { + cpu_synchronize_state_always(cpu); + } +} + void cpu_synchronize_all_post_reset(void) { CPUState *cpu; diff -ru qemu-2.1.0-rc2/hw/i386/kvm/clock.c qemu-2.1.0-rc2-fixed/hw/i386/kvm/clock.c --- qemu-2.1.0-rc2/hw/i386/kvm/clock.c 2014-07-15 23:49:14.000000000 +0200 +++ qemu-2.1.0-rc2-fixed/hw/i386/kvm/clock.c 2014-07-17 15:08:25.627063756 +0200 @@ -126,7 +126,7 @@ return; } - cpu_synchronize_all_states(); + cpu_synchronize_all_states_always(); ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data); if (ret < 0) { fprintf(stderr, "KVM_GET_CLOCK failed: %s\n", strerror(ret)); diff -ru qemu-2.1.0-rc2/include/sysemu/cpus.h qemu-2.1.0-rc2-fixed/include/sysemu/cpus.h --- qemu-2.1.0-rc2/include/sysemu/cpus.h 2014-07-15 23:49:14.000000000 +0200 +++ qemu-2.1.0-rc2-fixed/include/sysemu/cpus.h 2014-07-17 15:09:23.256578916 +0200 @@ -7,6 +7,7 @@ void pause_all_vcpus(void); void cpu_stop_current(void); +void cpu_synchronize_all_states_always(void); void cpu_synchronize_all_states(void); void cpu_synchronize_all_post_reset(void); void cpu_synchronize_all_post_init(void); diff -ru qemu-2.1.0-rc2/include/sysemu/kvm.h qemu-2.1.0-rc2-fixed/include/sysemu/kvm.h --- qemu-2.1.0-rc2/include/sysemu/kvm.h 2014-07-15 23:49:14.000000000 +0200 +++ qemu-2.1.0-rc2-fixed/include/sysemu/kvm.h 2014-07-17 15:11:54.855303171 +0200 @@ -346,9 +346,11 @@ #endif /* NEED_CPU_H */ void kvm_cpu_synchronize_state(CPUState *cpu); +void kvm_cpu_synchronize_state_always(CPUState *cpu); void kvm_cpu_synchronize_post_reset(CPUState *cpu); void kvm_cpu_synchronize_post_init(CPUState *cpu); + /* generic hooks - to be moved/refactored once there are more users */ static inline void cpu_synchronize_state(CPUState *cpu) @@ -358,6 +360,13 @@ } } +static inline void cpu_synchronize_state_always(CPUState *cpu) +{ + if (kvm_enabled()) { + kvm_cpu_synchronize_state_always(cpu); + } +} + static inline void cpu_synchronize_post_reset(CPUState *cpu) { if (kvm_enabled()) { diff -ru qemu-2.1.0-rc2/kvm-all.c qemu-2.1.0-rc2-fixed/kvm-all.c --- qemu-2.1.0-rc2/kvm-all.c 2014-07-15 23:49:14.000000000 +0200 +++ qemu-2.1.0-rc2-fixed/kvm-all.c 2014-07-17 15:14:04.884208826 +0200 @@ -1652,6 +1652,13 @@ s->coalesced_flush_in_progress = false; } +static void do_kvm_cpu_synchronize_state_always(void *arg) +{ + CPUState *cpu = arg; + + kvm_arch_get_registers(cpu); +} + static void do_kvm_cpu_synchronize_state(void *arg) { CPUState *cpu = arg; @@ -1669,6 +1676,11 @@ } } +void kvm_cpu_synchronize_state_always(CPUState *cpu) +{ + run_on_cpu(cpu, do_kvm_cpu_synchronize_state_always, cpu); +} + void kvm_cpu_synchronize_post_reset(CPUState *cpu) { kvm_arch_put_registers(cpu, KVM_PUT_RESET_STATE); --------------000704080509070305080901--