From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51246) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aqJAY-0007BA-68 for qemu-devel@nongnu.org; Wed, 13 Apr 2016 07:41:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aqJAT-0006eC-4N for qemu-devel@nongnu.org; Wed, 13 Apr 2016 07:41:14 -0400 Received: from mx1.redhat.com ([209.132.183.28]:54068) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aqJAS-0006e1-Rg for qemu-devel@nongnu.org; Wed, 13 Apr 2016 07:41:09 -0400 Date: Wed, 13 Apr 2016 12:41:04 +0100 From: "Dr. David Alan Gilbert" Message-ID: <20160413114103.GB2270@work-vm> References: <20160412175501.GB6415@work-vm> <20160413080545.GA2270@work-vm> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <20160413080545.GA2270@work-vm> Subject: Re: [Qemu-devel] post-copy is broken? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Li, Liang Z" , aarcange@redhat.com Cc: Amit Shah , "qemu-devel@nongnu.org" , "quintela@redhat.com" * Dr. David Alan Gilbert (dgilbert@redhat.com) wrote: > * Li, Liang Z (liang.z.li@intel.com) wrote: > > > > > I used the latest qemu code (commit id: 4e71220387e88a22) and ker= nel > > > > (v4.5) to test the post-copy, and find the guest get crashed after > > > > live migration, no matter I did a local live migration or live > > > > migration between two hosts. I just ran the stress as the workload = in > > > > guest. It seems the post-copy is broken? > > > > > > > > > > Stress parameters: stress --vm 2 --vm-hang 1 --vm-bytes 2048M > > > > > --vm-keep QEMU parameters: ./qemu-system-x86_64 --enable-kvm - > > > smp > > > > 4 -m > > > > > 8192 -monitor stdio -drive file=3D/share/centos6u6.qcow > > > > > > > > My test seems to be working here (4.4.6-301.fc23 kernel) same qemu > > > > version. > > > > This is with an f20 guest running google stressapptest. > > > > > > > > What's your last working version? > > > > > > >=20 > > > This is my first try of post-copy after the related patches been merg= ed. > > > I will double check and get back to you. > > >=20 > > > Thanks! > > > Liang > > >=20 > > > > Dave > >=20 > > I tried the v4.4 upstream kernel, the issue was disappeared. It must be= some changes between kernel v4.4 and v4.5 > > breaks post-copy. =20 >=20 > Oh, fun. cc'ing in Andrea. OK, I can confirm this bug on Fedora24 (4.5.0-302); see below for the postcopy test I've written that I intend to add to qemu; it works on my f23 host but not in f24. Dave =46rom 304829b6414dbd070b08ff03c1f155d229b5c492 Mon Sep 17 00:00:00 2001 =46rom: "Dr. David Alan Gilbert" Date: Wed, 13 Apr 2016 12:35:41 +0100 Subject: [PATCH] test: Postcopy This is a postcopy test (x86 only) that actually runs the guest and checks the memory contents. The test runs from an x86 boot block with the hex embedded in the test; the source for this is: =2E.......... =2Ecode16 =2Eorg 0x7c00 .file "fill.s" .text .globl start .type start, @function start: # at 0x7c00 ? cli lgdt gdtdesc mov $1,%eax mov %eax,%cr0 # Protected mode enable data32 ljmp $8,$0x7c20 =2Eorg 0x7c20 =2Ecode32 # A20 enable - not sure I actually need this inb $0x92,%al or $2,%al outb %al, $0x92 # set up DS for the whole of RAM (needed on KVM) mov $16,%eax mov %eax,%ds mov $65,%ax mov $0x3f8,%dx outb %al,%dx # bl keeps a counter so we limit the output speed mov $0, %bl mainloop: # Start from 1MB mov $(1024*1024),%eax innerloop: incb (%eax) add $4096,%eax cmp $(100*1024*1024),%eax jl innerloop inc %bl jnz mainloop mov $66,%ax mov $0x3f8,%dx outb %al,%dx jmp mainloop # GDT magic from old (GPLv2) Grub startup.S .p2align 2 /* force 4-byte alignment */ gdt: .word 0, 0 .byte 0, 0, 0, 0 /* -- code segment -- * base =3D 0x00000000, limit =3D 0xFFFFF (4 KiB Granularity), pres= ent * type =3D 32bit code execute/read, DPL =3D 0 */ .word 0xFFFF, 0 .byte 0, 0x9A, 0xCF, 0 /* -- data segment -- * base =3D 0x00000000, limit 0xFFFFF (4 KiB Granularity), present * type =3D 32 bit data read/write, DPL =3D 0 */ .word 0xFFFF, 0 .byte 0, 0x92, 0xCF, 0 gdtdesc: .word 0x27 /* limit */ .long gdt /* addr */ /* I'm a bootable disk */ =2Eorg 0x7dfe .byte 0x55 .byte 0xAA =2E.......... Signed-off-by: Dr. David Alan Gilbert --- tests/Makefile | 1 + tests/postcopy-test.c | 419 ++++++++++++++++++++++++++++++++++++++++++++++= ++++ 2 files changed, 420 insertions(+) create mode 100644 tests/postcopy-test.c diff --git a/tests/Makefile b/tests/Makefile index 9de9598..6aebddd 100644 --- a/tests/Makefile +++ b/tests/Makefile @@ -222,6 +222,7 @@ endif check-qtest-i386-y +=3D tests/test-netfilter$(EXESUF) check-qtest-i386-y +=3D tests/test-filter-mirror$(EXESUF) check-qtest-i386-y +=3D tests/test-filter-redirector$(EXESUF) +check-qtest-i386-y +=3D tests/postcopy-test$(EXESUF) check-qtest-x86_64-y =3D $(check-qtest-i386-y) gcov-files-i386-y +=3D i386-softmmu/hw/timer/mc146818rtc.c gcov-files-x86_64-y =3D $(subst i386-softmmu/,x86_64-softmmu/,$(gcov-files= -i386-y)) diff --git a/tests/postcopy-test.c b/tests/postcopy-test.c new file mode 100644 index 0000000..5e5940b --- /dev/null +++ b/tests/postcopy-test.c @@ -0,0 +1,419 @@ +/* + * QTest testcase for postcopy + * + * Copyright (c) 2016 Red Hat, Inc. and/or its affiliates + * based on the vhost-user-test.c that is: + * Copyright (c) 2014 Virtual Open Systems Sarl. + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include + +#include "libqtest.h" +#include "qemu/option.h" +#include "qemu/range.h" +#include "sysemu/char.h" +#include "sysemu/sysemu.h" + +#include +#include +#include + +#if defined(__linux__) +#include +#endif + +#if defined(__linux__) && defined(__NR_userfaultfd) && defined(CONFIG_EVEN= TFD) +#include +#include +#include + +static bool ufd_version_check(void) +{ + struct uffdio_api api_struct; + uint64_t ioctl_mask; + + int ufd =3D ufd =3D syscall(__NR_userfaultfd, O_CLOEXEC); + + if (ufd =3D=3D -1) { + g_test_message("Skipping test: userfaultfd not available"); + return false; + } + + api_struct.api =3D UFFD_API; + api_struct.features =3D 0; + if (ioctl(ufd, UFFDIO_API, &api_struct)) { + g_test_message("Skipping test: UFFDIO_API failed"); + return false; + } + + ioctl_mask =3D (__u64)1 << _UFFDIO_REGISTER | + (__u64)1 << _UFFDIO_UNREGISTER; + if ((api_struct.ioctls & ioctl_mask) !=3D ioctl_mask) { + g_test_message("Skipping test: Missing userfault feature"); + return false; + } + + return true; +} + +#else +static bool ufd_version_check(void) +{ + g_test_message("Skipping test: Userfault not available (builtdtime)"); + return false; +} + +#endif + +/* GLIB version compatibility flags */ +#if !GLIB_CHECK_VERSION(2, 26, 0) +#define G_TIME_SPAN_SECOND (G_GINT64_CONSTANT(1000000)) +#endif + +#if GLIB_CHECK_VERSION(2, 28, 0) +#define HAVE_MONOTONIC_TIME +#endif + + +#if !GLIB_CHECK_VERSION(2, 32, 0) +static gboolean g_cond_wait_until(CompatGCond cond, CompatGMutex mutex, + gint64 end_time) +{ + gboolean ret =3D FALSE; + end_time -=3D g_get_monotonic_time(); + GTimeVal time =3D { end_time / G_TIME_SPAN_SECOND, + end_time % G_TIME_SPAN_SECOND }; + ret =3D g_cond_timed_wait(cond, mutex, &time); + return ret; +} +#endif + +static const char *tmpfs; + +/* A simple PC boot sector that modifies memory (1-100MB) quickly + * outputing a 'B' every so often if it's still running. + */ +unsigned char bootsect[] =3D { + 0xfa, 0x0f, 0x01, 0x16, 0x74, 0x7c, 0x66, 0xb8, 0x01, 0x00, 0x00, 0x00, + 0x0f, 0x22, 0xc0, 0x66, 0xea, 0x20, 0x7c, 0x00, 0x00, 0x08, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xe4, 0x92, 0x0c, 0x02, + 0xe6, 0x92, 0xb8, 0x10, 0x00, 0x00, 0x00, 0x8e, 0xd8, 0x66, 0xb8, 0x41, + 0x00, 0x66, 0xba, 0xf8, 0x03, 0xee, 0xb3, 0x00, 0xb8, 0x00, 0x00, 0x10, + 0x00, 0xfe, 0x00, 0x05, 0x00, 0x10, 0x00, 0x00, 0x3d, 0x00, 0x00, 0x40, + 0x06, 0x7c, 0xf2, 0xfe, 0xc3, 0x75, 0xe9, 0x66, 0xb8, 0x42, 0x00, 0x66, + 0xba, 0xf8, 0x03, 0xee, 0xeb, 0xde, 0x66, 0x90, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0xff, 0xff, 0x00, 0x00, 0x00, 0x9a, 0xcf, 0x00, + 0xff, 0xff, 0x00, 0x00, 0x00, 0x92, 0xcf, 0x00, 0x27, 0x00, 0x5c, 0x7c, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x55, 0xaa +}; + +/* + * Wait for some output in the serial output file, + * we get an 'A' followed by an endless string of 'B's + * but on the destination we won't have the A. + */ +static void wait_for_serial(const char *side) +{ + char *serialpath =3D g_strdup_printf("%s/%s", tmpfs, side); + FILE *serialfile =3D fopen(serialpath, "r"); + + do { + int readvalue =3D fgetc(serialfile); + + switch (readvalue) { + case 'A': + /* Fine */ + break; + + case 'B': + /* It's alive! */ + fclose(serialfile); + g_free(serialpath); + return; + + case EOF: + fseek(serialfile, 0, SEEK_SET); + usleep(1000); + break; + + default: + fprintf(stderr, "Unexpected %d on %s serial\n", readvalue, sid= e); + assert(0); + } + } while (true); +} + +/* + * It's tricky to use qemu's migration event capability with qtest, + * events suddenly appearing confuse the qmp()/hmp() responses. + * so wait for a couple of passes to have happened before + * going postcopy. + */ + +static uint64_t get_migration_pass(void) +{ + QDict *rsp, *rsp_return, *rsp_ram; + uint64_t result; + + rsp =3D qmp("{ 'execute': 'query-migrate' }"); + g_assert(qdict_haskey(rsp, "return")); + rsp_return =3D qdict_get_qdict(rsp, "return"); + if (!qdict_haskey(rsp_return, "ram")) { + /* Still in setup */ + result =3D 0; + } else { + rsp_ram =3D qdict_get_qdict(rsp_return, "ram"); + result =3D qdict_get_try_int(rsp_ram, "dirty-sync-count", 0); + QDECREF(rsp); + } + return result; +} + +static void wait_for_migration_complete(void) +{ + QDict *rsp, *rsp_return; + bool completed; + + do { + const char *status; + + rsp =3D qmp("{ 'execute': 'query-migrate' }"); + rsp_return =3D qdict_get_qdict(rsp, "return"); + status =3D qdict_get_str(rsp_return, "status"); + =20 + completed =3D strcmp(status, "completed") =3D=3D 0; + assert(strcmp(status, "failed")); + QDECREF(rsp); + usleep(1000*100); + } while (!completed); +} + +static void wait_for_migration_pass(void) +{ + uint64_t initial_pass =3D get_migration_pass(); + uint64_t pass; + + do { + usleep(1000*100); + pass =3D get_migration_pass(); + } while (pass =3D=3D initial_pass); +} + +static void check_guests_ram(void) +{ + const unsigned start_address =3D 1024 * 1024; + const unsigned end_address =3D 100 * 1024 * 1024; + /* Our ASM test will have been incrementing one byte from each page fr= om + * 1MB to <100MB in order. + * This gives us a constraint that any page's byte should be equal or = less + * than the previous pages byte (mod 256); and they should all be equal + * except for one transition at the point where we meet the incremente= r. + * (We're running this with the guest stopped). + */ + unsigned address; + uint8_t first_byte; + uint8_t last_byte; + bool hit_edge =3D false; + + qtest_memread(global_qtest, start_address, &first_byte, 1); + last_byte =3D first_byte; + + for (address =3D start_address + 4096; address < end_address; address = +=3D 4096) + { + uint8_t b; + qtest_memread(global_qtest, address, &b, 1); + if (b !=3D last_byte) { + if ( ((b + 1) % 255) =3D=3D last_byte && !hit_edge) { + /* This is OK, the guest stopped at the point of + * incrementing the previous page but didn't get + * to us yet. + */ + hit_edge =3D true; + } else { + fprintf(stderr, "Memory content inconsistency at %x" + " first_byte =3D %x last_byte =3D %x curre= nt =3D %x" + " hit_edge =3D %x\n", + address, first_byte, last_byte, b, hit_edg= e); + assert(0); + } + } + last_byte =3D b; + } + fprintf(stderr, "first_byte =3D %x last_byte =3D %x hit_edge =3D %x OK= \n", + first_byte, last_byte, hit_edge); +} + +static void cleanup(const char *filename) +{ + char *path =3D g_strdup_printf("%s/%s", tmpfs, filename); + + unlink(path); +} + +static void test_migrate(void) +{ + char *uri =3D g_strdup_printf("unix:%s/migsocket", tmpfs ); + QTestState *global =3D global_qtest, *from, *to; + gchar *cmd; + QDict *rsp; + + char *bootpath =3D g_strdup_printf("%s/bootsect", tmpfs); + FILE *bootfile =3D fopen(bootpath, "wb"); + + assert(fwrite(bootsect, 512, 1, bootfile) =3D=3D 1); + fclose(bootfile); + + cmd =3D g_strdup_printf("-machine accel=3Dkvm:tcg -m 150M" + " -name pcsource,debug-threads=3Don" + " -serial file:%s/src_serial" + " -drive file=3D%s,format=3Draw", + tmpfs, bootpath); + from =3D qtest_start(cmd); + g_free(cmd); + + cmd =3D g_strdup_printf("-machine accel=3Dkvm:tcg -m 150M" + " -name pcdest,debug-threads=3Don" + " -serial file:%s/dest_serial" + " -drive file=3D%s,format=3Draw" + " -incoming %s", + tmpfs, bootpath, uri); + to =3D qtest_init(cmd); + g_free(cmd); + + global_qtest =3D from; + rsp =3D qmp("{ 'execute': 'migrate-set-capabilities'," + "'arguments': { " + "'capabilities': [ {" + "'capability': 'postcopy-ram'," + "'state': true } ] } }"); + g_assert(qdict_haskey(rsp, "return")); + QDECREF(rsp); + + global_qtest =3D to; + rsp =3D qmp("{ 'execute': 'migrate-set-capabilities'," + "'arguments': { " + "'capabilities': [ {" + "'capability': 'postcopy-ram'," + "'state': true } ] } }"); + g_assert(qdict_haskey(rsp, "return")); + QDECREF(rsp); + + global_qtest =3D from; + rsp =3D qmp("{ 'execute': 'migrate_set_speed'," + "'arguments': { 'value': 100000000 } }"); + g_assert(qdict_haskey(rsp, "return")); + QDECREF(rsp); + + /* Wait for the first serial output from the source */ + wait_for_serial("src_serial"); + + cmd =3D g_strdup_printf("{ 'execute': 'migrate'," + "'arguments': { 'uri': '%s' } }", + uri); + rsp =3D qmp(cmd); + g_free(cmd); + g_assert(qdict_haskey(rsp, "return")); + QDECREF(rsp); + + wait_for_migration_pass(); + + rsp =3D qmp("{ 'execute': 'migrate-start-postcopy' }"); + g_assert(qdict_haskey(rsp, "return")); + QDECREF(rsp); + + qmp_eventwait("STOP"); + + global_qtest =3D to; + qmp_eventwait("RESUME"); + + wait_for_serial("dest_serial"); + global_qtest =3D from; + wait_for_migration_complete(); + + qtest_quit(from); + + global_qtest =3D to; + qmp("{ 'execute' : 'stop'}"); + check_guests_ram(); + + qtest_quit(to); + g_free(uri); + + global_qtest =3D global; + + cleanup("bootsect"); + cleanup("migsocket"); + cleanup("src_serial"); + cleanup("dest_serial"); +} + +int main(int argc, char **argv) +{ + char template[] =3D "/tmp/postcopy-test-XXXXXX"; + int ret; + + g_test_init(&argc, &argv, NULL); + + if (!ufd_version_check()) { + return 0; + } + + tmpfs =3D mkdtemp(template); + if (!tmpfs) { + g_test_message("mkdtemp on path (%s): %s\n", template, strerror(er= rno)); + } + g_assert(tmpfs); + + module_call_init(MODULE_INIT_QOM); + + qtest_add_func("/postcopy", test_migrate); + + ret =3D g_test_run(); + + g_assert_cmpint(ret, =3D=3D, 0); + + ret =3D rmdir(tmpfs); + if (ret !=3D 0) { + g_test_message("unable to rmdir: path (%s): %s\n", + tmpfs, strerror(errno)); + } + + return ret; +} --=20 2.5.5