From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34298) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YQfCF-0007Tm-CY for qemu-devel@nongnu.org; Wed, 25 Feb 2015 11:52:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YQfC9-0000xW-JR for qemu-devel@nongnu.org; Wed, 25 Feb 2015 11:52:27 -0500 Received: from mx1.redhat.com ([209.132.183.28]:54530) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YQfC9-0000wa-CY for qemu-devel@nongnu.org; Wed, 25 Feb 2015 11:52:21 -0500 From: "Dr. David Alan Gilbert (git)" Date: Wed, 25 Feb 2015 16:51:23 +0000 Message-Id: <1424883128-9841-1-git-send-email-dgilbert@redhat.com> Subject: [Qemu-devel] [PATCH v5 00/45] Postcopy implementation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: aarcange@redhat.com, yamahata@private.email.ne.jp, quintela@redhat.com, amit.shah@redhat.com, pbonzini@redhat.com, yanghy@cn.fujitsu.com, david@gibson.dropbear.id.au From: "Dr. David Alan Gilbert" Hi, This is the 5th cut of my version of postcopy; it is designed for use with the Linux kernel additions posted by Andrea Arcangeli here: git clone --reference linux -b userfault16 git://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git (Note this is a different API from the last version) This qemu series can be found at: https://github.com/orbitfp7/qemu.git on the wp3-postcopy-v5 tag. I believe I've fixed most of the issues from the reviews, but there are probably a few more to cover before the final version: * I've got a rare (1/1000 migration) bug I'm fighting * The kernel API is still being discussed v5 New kernel API Uses an atomic copy rather than remap to avoid IPIs Fix builds on older machines Fix seg if block migration enabled (thanks to Gary Hook for reporting) Fix for mlock (thanks to zhanghailiang for reporting) Send postcopy-notify before sending the 'begin' part of device state Disable ballooning during postcopy Don't allocate zero pages as they migrate Stop using sysconf to get pagesizes - using getpagesize() and qemu_host_page_size in target dependent parts Fix for worst case page size ratio (untested) - ARM is 64k host, 1k target; with 32bit longs Drop the assertion check on migration_dirty_pages I don't like killing src vms anyway, and I've not seen it trigger in 9months+ Free up migration_incoming-state at end of postcopy Calculate downtime on source differently for postcopy Use trace rather than dprintf Paolo's comments: Rename postcopy_ram_sensitise_area Tidy up of loadvm loop flags Remove one layer of socket_shutdown abstraction Remove _RAM_ from many of the postcopy symbols as appropriate - most messages/states could now be used for other postcopy state. Used a qemu_event to wait for the end of the main thread finishing rather than spinning on postcopy state Rename inward/outward cmd messages for consistency Fixup old-vm-running check for postcopy Dave Gibson's comments: Lots of cleanup Eric's comments 80col in qapi-schema Dr. David Alan Gilbert (45): Start documenting how postcopy works. Split header writing out of qemu_save_state_begin qemu_ram_foreach_block: pass up error value, and down the ramblock name Add qemu_get_counted_string to read a string prefixed by a count byte Create MigrationIncomingState Provide runtime Target page information Return path: Open a return path on QEMUFile for sockets Return path: socket_writev_buffer: Block even on non-blocking fd's Migration commands Return path: Control commands Return path: Send responses from destination to source Return path: Source handling of return path ram_debug_dump_bitmap: Dump a migration bitmap as text Move loadvm_handlers into MigrationIncomingState Rework loadvm path for subloops Add migration-capability boolean for postcopy-ram. Add wrappers and handlers for sending/receiving the postcopy-ram migration messages. MIG_CMD_PACKAGED: Send a packaged chunk of migration stream migrate_init: Call from savevm Modify savevm handlers for postcopy Add Linux userfaultfd header postcopy: OS support test migrate_start_postcopy: Command to trigger transition to postcopy MIG_STATE_POSTCOPY_ACTIVE: Add new migration state qemu_savevm_state_complete: Postcopy changes Postcopy page-map-incoming (PMI) structure Postcopy: Maintain sentmap and calculate discard postcopy: Incoming initialisation postcopy: ram_enable_notify to switch on userfault Postcopy: Postcopy startup in migration thread Postcopy end in migration_thread Page request: Add MIG_RP_CMD_REQ_PAGES reverse command Page request: Process incoming page request Page request: Consume pages off the post-copy queue postcopy_ram.c: place_page and helpers Postcopy: Use helpers to map pages during migration qemu_ram_block_from_host Don't sync dirty bitmaps in postcopy Host page!=target page: Cleanup bitmaps Postcopy; Handle userfault requests Start up a postcopy/listener thread ready for incoming page data postcopy: Wire up loadvm_postcopy_handle_{run,end} commands End of migration for postcopy Disable mlock around incoming postcopy Inhibit ballooning during postcopy arch_init.c | 908 +++++++++++++++++++++++++++++++-- balloon.c | 11 + docs/migration.txt | 189 +++++++ exec.c | 76 ++- hmp-commands.hx | 15 + hmp.c | 7 + hmp.h | 1 + hw/virtio/virtio-balloon.c | 4 +- include/exec/cpu-all.h | 2 - include/exec/cpu-common.h | 8 +- include/migration/migration.h | 145 +++++- include/migration/postcopy-ram.h | 99 ++++ include/migration/qemu-file.h | 10 + include/migration/vmstate.h | 12 +- include/qemu/typedefs.h | 6 + include/sysemu/balloon.h | 2 + include/sysemu/sysemu.h | 46 +- linux-headers/linux/userfaultfd.h | 150 ++++++ migration/Makefile.objs | 2 +- migration/block.c | 7 +- migration/migration.c | 739 +++++++++++++++++++++++++-- migration/postcopy-ram.c | 1018 +++++++++++++++++++++++++++++++++++++ migration/qemu-file-internal.h | 2 + migration/qemu-file-unix.c | 99 +++- migration/qemu-file.c | 28 + migration/rdma.c | 4 +- qapi-schema.json | 15 +- qmp-commands.hx | 19 + savevm.c | 866 ++++++++++++++++++++++++++++--- trace-events | 80 ++- 30 files changed, 4381 insertions(+), 189 deletions(-) create mode 100644 include/migration/postcopy-ram.h create mode 100644 linux-headers/linux/userfaultfd.h create mode 100644 migration/postcopy-ram.c -- 2.1.0