From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:35365) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YajYg-0000Kp-Qv for qemu-devel@nongnu.org; Wed, 25 Mar 2015 07:33:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YajYd-00007j-GN for qemu-devel@nongnu.org; Wed, 25 Mar 2015 07:33:14 -0400 Received: from szxga01-in.huawei.com ([58.251.152.64]:5829) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YajYb-000070-FQ for qemu-devel@nongnu.org; Wed, 25 Mar 2015 07:33:11 -0400 Message-ID: <55129CE6.5070302@huawei.com> Date: Wed, 25 Mar 2015 19:32:54 +0800 From: zhanghailiang MIME-Version: 1.0 References: <55128084.2040304@huawei.com> <87a8z12yot.fsf@neno.neno> In-Reply-To: <87a8z12yot.fsf@neno.neno> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [Migration Bug? ] Occasionally, the content of VM's memory is inconsistent between Source and Destination of migration List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: quintela@redhat.com Cc: hangaohuai@huawei.com, Li Zhijian , qemu-devel@nongnu.org, peter.huangpeng@huawei.com, "Dr. David Alan Gilbert (git)" , "Gonglei (Arei)" , Amit Shah , david@gibson.dropbear.id.au On 2015/3/25 17:50, Juan Quintela wrote: > zhanghailiang wrote: >> Hi all, >> >> We found that, sometimes, the content of VM's memory is inconsistent between Source side and Destination side >> when we check it just after finishing migration but before VM continue to Run. >> >> We use a patch like bellow to find this issue, you can find it from affix, >> and Steps to reprduce: >> >> (1) Compile QEMU: >> ./configure --target-list=x86_64-softmmu --extra-ldflags="-lssl" && make >> >> (2) Command and output: >> SRC: # x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu qemu64,-kvmclock -netdev tap,id=hn0-device virtio-net-pci,id=net-pci0,netdev=hn0 -boot c -drive file=/mnt/sdb/pure_IMG/sles/sles11_sp3.img,if=none,id=drive-virtio-disk0,cache=unsafe -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor stdio > > Could you try to reproduce: > - without vhost > - without virtio-net Yes, with e1000, the problem is still exist, i will test it with dropping all the unnecessary device configure, just as Dave's suggestion. > - cache=unsafe is going to give you trouble, but trouble should only > happen after migration of pages have finished. > > What kind of load were you having when reproducing this issue? > Just to confirm, you have been able to reproduce this without COLO > patches, right? > Yes, we reproduce this only in normal migration. >> (qemu) migrate tcp:192.168.3.8:3004 >> before saving ram complete >> ff703f6889ab8701e4e040872d079a28 >> md_host : after saving ram complete >> ff703f6889ab8701e4e040872d079a28 >> >> DST: # x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu qemu64,-kvmclock -netdev tap,id=hn0,vhost=on -device virtio-net-pci,id=net-pci0,netdev=hn0 -boot c -drive file=/mnt/sdb/pure_IMG/sles/sles11_sp3.img,if=none,id=drive-virtio-disk0,cache=unsafe -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor stdio -incoming tcp:0:3004 >> (qemu) QEMU_VM_SECTION_END, after loading ram >> 230e1e68ece9cd4e769630e1bcb5ddfb >> md_host : after loading all vmstate >> 230e1e68ece9cd4e769630e1bcb5ddfb >> md_host : after cpu_synchronize_all_post_init >> 230e1e68ece9cd4e769630e1bcb5ddfb >> >> This happens occasionally, and it is more easy to reproduce when issue migration command during VM's startup time. > > OK, a couple of things. Memory don't have to be exactly identical. > Virtio devices in particular do funny things on "post-load". There > aren't warantees for that as far as I know, we should end with an > equivalent device state in memory. > >> We have done further test and found that some pages has been dirtied but its corresponding migration_bitmap is not set. >> We can't figure out which modules of QEMU has missed setting bitmap when dirty page of VM, >> it is very difficult for us to trace all the actions of dirtying VM's pages. > > This seems to point to a bug in one of the devices. > >> Actually, the first time we found this problem was in the COLO FT development, and it triggered some strange issues in >> VM which all pointed to the issue of inconsistent of VM's memory. (We have try to save all memory of VM to slave side every time >> when do checkpoint in COLO FT, and everything will be OK.) >> >> Is it OK for some pages that not transferred to destination when do migration ? Or is it a bug? > > Pages transferred should be the same, after device state transmission is > when things could change. > >> This issue has blocked our COLO development... :( >> >> Any help will be greatly appreciated! > > Later, Juan. > >> >> Thanks, >> zhanghailiang >> >> --- a/savevm.c >> +++ b/savevm.c >> @@ -51,6 +51,26 @@ >> #define ARP_PTYPE_IP 0x0800 >> #define ARP_OP_REQUEST_REV 0x3 >> >> +#include "qemu/rcu_queue.h" >> +#include >> + >> +static void check_host_md5(void) >> +{ >> + int i; >> + unsigned char md[MD5_DIGEST_LENGTH]; >> + MD5_CTX ctx; >> + RAMBlock *block = QLIST_FIRST_RCU(&ram_list.blocks);/* Only check 'pc.ram' block */ >> + >> + MD5_Init(&ctx); >> + MD5_Update(&ctx, (void *)block->host, block->used_length); >> + MD5_Final(md, &ctx); >> + printf("md_host : "); >> + for(i = 0; i < MD5_DIGEST_LENGTH; i++) { >> + fprintf(stderr, "%02x", md[i]); >> + } >> + fprintf(stderr, "\n"); >> +} >> + >> static int announce_self_create(uint8_t *buf, >> uint8_t *mac_addr) >> { >> @@ -741,7 +761,13 @@ void qemu_savevm_state_complete(QEMUFile *f) >> qemu_put_byte(f, QEMU_VM_SECTION_END); >> qemu_put_be32(f, se->section_id); >> >> + printf("before saving %s complete\n", se->idstr); >> + check_host_md5(); >> + >> ret = se->ops->save_live_complete(f, se->opaque); >> + printf("after saving %s complete\n", se->idstr); >> + check_host_md5(); >> + >> trace_savevm_section_end(se->idstr, se->section_id, ret); >> if (ret < 0) { >> qemu_file_set_error(f, ret); >> @@ -1030,6 +1063,11 @@ int qemu_loadvm_state(QEMUFile *f) >> } >> >> ret = vmstate_load(f, le->se, le->version_id); >> + if (section_type == QEMU_VM_SECTION_END) { >> + printf("QEMU_VM_SECTION_END, after loading %s\n", le->se->idstr); >> + check_host_md5(); >> + } >> + >> if (ret < 0) { >> error_report("error while loading state section id %d(%s)", >> section_id, le->se->idstr); >> @@ -1061,7 +1099,11 @@ int qemu_loadvm_state(QEMUFile *f) >> g_free(buf); >> } >> >> + printf("after loading all vmstate\n"); >> + check_host_md5(); >> cpu_synchronize_all_post_init(); >> + printf("after cpu_synchronize_all_post_init\n"); >> + check_host_md5(); >> >> ret = 0; > > . >