From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55818) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZwXkl-0004tF-SF for qemu-devel@nongnu.org; Wed, 11 Nov 2015 10:57:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZwXib-0003DH-Pt for qemu-devel@nongnu.org; Wed, 11 Nov 2015 10:56:07 -0500 Received: from mailhub.sw.ru ([195.214.232.25]:36687 helo=relay.sw.ru) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZwXib-00037z-Ct for qemu-devel@nongnu.org; Wed, 11 Nov 2015 10:53:53 -0500 From: Vladimir Sementsov-Ogievskiy Date: Wed, 11 Nov 2015 18:16:30 +0300 Message-Id: <1447255003-2043-1-git-send-email-vsementsov@virtuozzo.com> Subject: [Qemu-devel] [PATCH v7 00/13] Dirty bitmaps migration List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: kwolf@redhat.com, peter.maydell@linaro.org, vsementsov@virtuozzo.com, quintela@redhat.com, dgilbert@redhat.com, stefanha@redhat.com, pbonzini@redhat.com, amit.shah@redhat.com, den@openvz.org, jsnow@redhat.com These patches provide dirty bitmap migration feature. Only named dirty bitmaps are to be migrated. Migration may be enabled using migration capabilities. v7: 0001: main functions are mostly rewritten to be safer. Granularity of serialization is now explicit thing. Only the last chunk of bitmap data may be not aligned. Reviewed-by deleted. 0002: changed accordingly to 0001 chnages Reviewed-by deleted. 0004: use ctz64() instead of ffsll(), accordingly to checkpatch Reviewed-by left. Hope, you don't mind. 0006: since 2.4 -> since 2.6 Reviewed-by's left (Eric Blake's one added). 0007: s/int/size_t for strlen result, add comment Reviewed-by left. Hope, you don't mind. 0008: Rename some static functions for better style: qemu_get_flags -> qemu_get_bitmap_flags qemu_put_flags -> qemu_put_bitmap_flags blk_mig_reset_dirty_cursor -> dirty_bitmap_mig_reset_dirty_cursor Logic, related to 0001 changes: - bdrv_dirty_bitmap_data_size calls with one count parameter are changed by bdrv_dirty_bitmap_serialization_size calls with two parameters, specifying the range - change dirty_bitmap_save_pending for not using bdrv_dirty_bitmap_data_size Rebase on master: - rename _cancel to _cleanup and .cancel to .cleanup - change dirty_bitmap_save_pending (postcopy parameters) - rename .save_live_complete to .save_live_complete_precopy Reviewed-by deleted. also: new function hbitmap_serialization_granularity made public, and may be used for selecting chunk size (it gives the minimum) for migration, storing, loding. But it is not used, because, I think, checking that CHUNK_SIZE from migration/block-dirty-bitmap.c is greater than sizeof(long). 0011: # @md5: md5 checksum of the last bitmap level (since 2.4) to # @md5: md5 checksum (as a hexadecimal string) of the last bitmap level # (since 2.6) Reviewed-by left. Hope, my new comment is ok and you don't mind. 0012: new patch v6: - rebase on master - function qemu_get_string() deleted, used existed function qemu_get_counted_string() - function qemu_put_string() renamed appropriately to qemu_put_counted_string() (and now it is new separate patch) ===================================================== 0004: Reviewed-by: John Snow 0006: (since 2.3) -> (since 2.4) 0007: new patch, mentioned above 0008: printf format fixed, DEBUG definition set back to 0, in dirty_bitmap_live_iterate_is_active conditions swapped. qemu_{get,put}_string removed, used ..counted.. verisions 0011: (since 2.3) -> (since 2.4) 0012: assertIsNotNone -> assertNotEqual for compitibility with old Python Reviewed-by: John Snow v5: - rebase on master - drop [PATCH RFC v4 10/13] iotests: add event_wait to VM class - remove rfc, as incremental backup series by John Snow are in upstream v4 significant changes: 0001: tiny bugfix: out[i] -> out[i-start], same for 'in' 0007: set chunk size to 1kb, disable live iteration for migrating data < 1mb size. tests: only one with md5 sum is here. used function event_wait by John Snow. (I hope, you don't mind me just adding this function with your 'Signed-off-by') rfc: This patch set is based on v13 of "block: incremental backup series" by John Snow, which are not pushed yet. v3: based on v13 of "block: incremental backup series" by John Snow. changes from v2: removed patch for adding dirty parameter (migration capablities used instead). 0001: printf's dropped, qapi used 0002: part0 -> zeroes 0003: part0 -> zeroes 0005: dirty_dirty -> meta add comments about meta bitmap 0006: the format is changed, nodes used instead of devices. other patches are new. rfc: there are two tests. They are the same but using different interfaces: md5 checksum of the bitmap last layer in query-block or separate query-block-dirty-bitmap with dirty bitmap regions. The second form is more appropriate for debugging, the first is more appropriate for simple regression control. Which should go to upstream? v2: 1. bug-fixes, that are already in upstream, and renaming of function bdrv_reset_dirty_bitmap (which is already in Snow's series) are dropped 2. bitmap store/restore: the concept renamed to serialization, added function hbitmap_deserialize_part0, to not transfer zero blocks 3. migration dirty parameter: added description comment 4. Other patches are new. v2.rfc: Actually, in this version of the series I'm trying not use migration/block.c at all. Instead a separate migration unit is added in the new file migration/dirty-bitmap.c. Now bitmaps are migrated like blocks in block migration, they have their "dirty-dirty" bitmaps, for tracking set/unset changes during migration. The advantages are: - no complications of migration/block.c - separate dirty-dirty bitmaps provide handling of "unset's" - more effective meta-data/data ratio - no tiny bitmap-blocks. v1: These patches provide dirty bitmap migration feature. Only named dirty bitmaps are to be migrated. Migration is made as a part of block migration in block-migration.c. Dirty bitmap migration may be enabled by "dirty" parameter for qmp migrate command. If "blk" and "inc" parameters are false when "dirty" is true block migration is actually skipped: no allocatoions, no bdrv_read's, no bdrv_write's, only bitmaps are migrated. Vladimir Sementsov-Ogievskiy (13): hbitmap: serialization block: BdrvDirtyBitmap serialization interface block: tiny refactoring: minimize hbitmap_(set/reset) usage block: add meta bitmaps block: add bdrv_next_dirty_bitmap() qapi: add dirty-bitmaps migration capability migration/qemu-file: add qemu_put_counted_string() migration: add migration/block-dirty-bitmap.c iotests: maintain several vms in test iotests: add add_incoming_migration to VM class qapi: add md5 checksum of last dirty bitmap level to query-block iotests: add default node-name iotests: add dirty bitmap migration test block.c | 93 +++++- include/block/block.h | 22 ++ include/migration/block.h | 1 + include/migration/migration.h | 1 + include/migration/qemu-file.h | 2 + include/qemu/hbitmap.h | 81 +++++ migration/Makefile.objs | 2 +- migration/block-dirty-bitmap.c | 708 +++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 9 + migration/qemu-file.c | 13 + qapi-schema.json | 4 +- qapi/block-core.json | 5 +- tests/qemu-iotests/117 | 84 +++++ tests/qemu-iotests/117.out | 5 + tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 17 +- util/hbitmap.c | 122 +++++++ vl.c | 1 + 18 files changed, 1163 insertions(+), 8 deletions(-) create mode 100644 migration/block-dirty-bitmap.c create mode 100755 tests/qemu-iotests/117 create mode 100644 tests/qemu-iotests/117.out -- 2.1.4