From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53959) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aORNI-0007Xe-SF for qemu-devel@nongnu.org; Wed, 27 Jan 2016 09:47:19 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aORNF-0007AH-DW for qemu-devel@nongnu.org; Wed, 27 Jan 2016 09:47:12 -0500 Received: from mx2.parallels.com ([199.115.105.18]:57934) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aORNF-0007A9-4e for qemu-devel@nongnu.org; Wed, 27 Jan 2016 09:47:09 -0500 Message-ID: <56A8D865.8020503@virtuozzo.com> Date: Wed, 27 Jan 2016 17:47:01 +0300 From: Vladimir Sementsov-Ogievskiy MIME-Version: 1.0 References: <1446747083-18205-1-git-send-email-dgilbert@redhat.com> <56A8AFEE.4070201@virtuozzo.com> <20160127124254.GC2394@work-vm> In-Reply-To: <20160127124254.GC2394@work-vm> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH v9 00/56] Postcopy implementation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Dr. David Alan Gilbert" Cc: qemu-devel@nongnu.org Ok, thanks, I'll think about it. On 27.01.2016 15:42, Dr. David Alan Gilbert wrote: > * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote: >> Hello, Dr. Gilbert! > Hi Vladimir, > >> As I understand this is not a complete realization of post-copy stage of >> migration, but only realization of ram post-copy. > Yes, that's correct; RAM was the only one we'd had much interest in; but > I can understand wanting to help block as well. > >> I need to implement >> post-copy migration of block-dirty-bitmaps. And it should work with/without >> ram post-copy. Did you plan a possibility of post-copying something except >> ram? If yes, can you please help me with an interface? Should I implement >> separate thread like postcopy_ram_listen_thread and other things, or >> something may be reused? > I think with the structure that's in, it should be possible to add > another device doing postcopy as well. Although maybe it's worth checking > whether these dirty bitmaps behave similarly enough to a non-guest visible > RAM block, so then the existing migrate code would handle it. > > 1) I think you're probably best reusing the postcopy_ram_listen_thread > for all postcopy data on the load side; I think that's probably already > OK to do that; just rename it. The migration stream once in postcopy > mode is just still a normal migration stream, so it can still have a mix > of RAM and disk or whatever. > > 2) The tricky bit I can think of is on the sending side making sure > you interleave the requests for RAM and disk requests sensibly; at the moment > when in postcopy we drop the bandwidth limit so it'll keep doing RAM transfers > and respond to ram requests as soon as possible; you'll need to persuade > it to somehow realise there's a disk request and get the RAM code to > drop out of it's loop to give the disk code a chance. There's already > the opportunity to improve the behaviour with just RAM to stop it flooding > the network buffers with background page requests which delay the responses > to the postcopy requests. > > My knowledge of the block code is pretty minimal, so I've not had a chance > to follow much of your dirty-bitmap series. > > Dave > >> On 05.11.2015 21:10, Dr. David Alan Gilbert (git) wrote: >>> From: "Dr. David Alan Gilbert" >>> >>> This is the 9th cut of my version of postcopy. >>> >>> The userfaultfd linux kernel code is now in the upstream kernel >>> tree, and so 4.3 can be used without modification. >>> >>> This qemu series can be found at: >>> https://github.com/orbitfp7/qemu.git >>> on the wp3-postcopy-v9 tag >>> >>> Testing status: >>> * Tested heavily on x86 >>> * Smoke tested on aarch64 (so it does work on different page sizes) >>> >>> This work has been partially funded by the EU Orbit project: >>> see http://www.orbitproject.eu/about/ >>> >>> v9 >>> Almost all of the changes are changes from review comments, and most are >>> minor. >>> >>> The following are new patches, mostly split out from earlier patches >>> (The exception being adding userfaultfd.h header back in - which was in v5 >>> but we took out, but is needed again due to 1842bdfd) >>> >>> 04/56 Move page_size_init earlier >>> 08/56 qemu_ram_block_by_name >>> 12/56 Factor out host_from_stream_offset call and check >>> 15/56 Add Linux userfaultfd.h header back >>> 21/56 migration_is_setup_or_active >>> 30/56 migration_completion: Take current state >>> 34/56 Maintain unsentmap >>> >>> The previous patches 03,10,13,36/54 went in upstream already. >>> >>> Fix for assert using hotplug (Thanks Bharata for spotting that) >>> Fix for migrate_cancel a second time >>> Rework for migration_bitmap_rcu after Denis's deadlock fix >>> Rework ram_load into a separate postcopy loop >>> The 'sentmap' is now an 'unsentmap' - this saves a complement step at the end >>> The unsentmap creation is now split into a separate patch >>> The unsentmap is now stored in the RCU structure (although it can't >>> really resize during the migrate) >>> Fix for block migration >>> still not a suggested combination. >>> Minor: >>> trace_savevm_send_open_return_path added >>> split migration_is_active function into separate patch and made >>> migration_is_setup_or_active >>> move file reads into migrate_handle_advise and migrate_handle_packaged >>> use of MIN in send-packaged-chunk >>> migration_thread_started -> migration_thread_running >>> updated qemu_get_buffer_in_place with Juan's version (and size_t'ified it) >>> split the host_from_stream_offset change into a separate patch >>> 'ram_load: Factor out host_from_stream_offset call and check' >>> split out ram_find_block_by_id into a separate patch and now >>> called qemu_ram_block_by_name >>> postcopy_discard_send_range etc now take start/length rather than start/end >>> (also added another trace) >>> split ram_save_host_page into ram_save_host_page/ram_save_target_page >>> split host page cleanup function into a core that handles both passes >>> >>> Dave >>> >>> Dr. David Alan Gilbert (56): >>> Add postcopy documentation >>> Provide runtime Target page information >>> Move configuration section writing >>> Move page_size_init earlier >>> qemu_ram_block_from_host >>> qemu_ram_block_by_name >>> Rename mis->file to from_src_file >>> Add qemu_get_buffer_in_place to avoid copies some of the time >>> Add wrapper for setting blocking status on a QEMUFile >>> Add QEMU_MADV_NOHUGEPAGE >>> ram_debug_dump_bitmap: Dump a migration bitmap as text >>> ram_load: Factor out host_from_stream_offset call and check >>> migrate_init: Call from savevm >>> Rename save_live_complete to save_live_complete_precopy >>> Add Linux userfaultfd.h header >>> Return path: Open a return path on QEMUFile for sockets >>> Return path: socket_writev_buffer: Block even on non-blocking fd's >>> Migration commands >>> Return path: Control commands >>> Return path: Send responses from destination to source >>> migration_is_setup_or_active >>> Return path: Source handling of return path >>> Rework loadvm path for subloops >>> Add migration-capability boolean for postcopy-ram. >>> Add wrappers and handlers for sending/receiving the postcopy-ram >>> migration messages. >>> MIG_CMD_PACKAGED: Send a packaged chunk of migration stream >>> Modify save_live_pending for postcopy >>> postcopy: OS support test >>> migrate_start_postcopy: Command to trigger transition to postcopy >>> migration_completion: Take current state >>> MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state >>> Avoid sending vmdescription during postcopy >>> Add qemu_savevm_state_complete_postcopy >>> Postcopy: Maintain unsentmap >>> Postcopy: Calculate discard >>> postcopy: Incoming initialisation >>> postcopy: ram_enable_notify to switch on userfault >>> Postcopy: Postcopy startup in migration thread >>> Postcopy: End of iteration >>> Page request: Add MIG_RP_MSG_REQ_PAGES reverse command >>> Page request: Process incoming page request >>> Page request: Consume pages off the post-copy queue >>> postcopy_ram.c: place_page and helpers >>> Postcopy: Use helpers to map pages during migration >>> postcopy: Check order of received target pages >>> Don't sync dirty bitmaps in postcopy >>> Don't iterate on precopy-only devices during postcopy >>> Host page!=target page: Cleanup bitmaps >>> Round up RAMBlock sizes to host page sizes >>> Postcopy; Handle userfault requests >>> Start up a postcopy/listener thread ready for incoming page data >>> postcopy: Wire up loadvm_postcopy_handle_ commands >>> Postcopy: Mark nohugepage before discard >>> End of migration for postcopy >>> Disable mlock around incoming postcopy >>> Inhibit ballooning during postcopy >>> >>> balloon.c | 11 + >>> docs/migration.txt | 191 ++++++++ >>> exec.c | 92 +++- >>> hmp-commands.hx | 15 + >>> hmp.c | 7 + >>> hmp.h | 1 + >>> hw/ppc/spapr.c | 2 +- >>> hw/virtio/virtio-balloon.c | 4 +- >>> include/exec/cpu-common.h | 4 + >>> include/exec/exec-all.h | 1 - >>> include/exec/ram_addr.h | 2 - >>> include/migration/migration.h | 121 ++++- >>> include/migration/postcopy-ram.h | 99 ++++ >>> include/migration/qemu-file.h | 10 + >>> include/migration/vmstate.h | 8 +- >>> include/qemu-common.h | 1 + >>> include/qemu/osdep.h | 9 + >>> include/qemu/typedefs.h | 3 + >>> include/sysemu/balloon.h | 2 + >>> include/sysemu/sysemu.h | 46 +- >>> kvm-all.c | 1 - >>> linux-headers/linux/userfaultfd.h | 167 +++++++ >>> migration/Makefile.objs | 2 +- >>> migration/block.c | 9 +- >>> migration/migration.c | 723 +++++++++++++++++++++++++-- >>> migration/postcopy-ram.c | 760 +++++++++++++++++++++++++++++ >>> migration/qemu-file-unix.c | 111 ++++- >>> migration/qemu-file.c | 64 +++ >>> migration/ram.c | 997 +++++++++++++++++++++++++++++++++++--- >>> migration/savevm.c | 823 +++++++++++++++++++++++++++---- >>> qapi-schema.json | 18 +- >>> qmp-commands.hx | 19 + >>> qtest.c | 1 - >>> trace-events | 84 +++- >>> vl.c | 1 + >>> 35 files changed, 4159 insertions(+), 250 deletions(-) >>> create mode 100644 include/migration/postcopy-ram.h >>> create mode 100644 linux-headers/linux/userfaultfd.h >>> create mode 100644 migration/postcopy-ram.c >>> >> >> -- >> Best regards, >> Vladimir >> * now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience. >> > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- Best regards, Vladimir * now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.