From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55928) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YzuBA-0000RT-WA for qemu-devel@nongnu.org; Tue, 02 Jun 2015 17:57:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YzuB7-0002TV-6N for qemu-devel@nongnu.org; Tue, 02 Jun 2015 17:57:00 -0400 Received: from mx1.redhat.com ([209.132.183.28]:53984) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YzuB6-0002TQ-QU for qemu-devel@nongnu.org; Tue, 02 Jun 2015 17:56:57 -0400 Message-ID: <556E26A1.3080402@redhat.com> Date: Tue, 02 Jun 2015 17:56:49 -0400 From: John Snow MIME-Version: 1.0 References: <1431531007-10269-1-git-send-email-vsementsov@parallels.com> <1431531007-10269-8-git-send-email-vsementsov@parallels.com> In-Reply-To: <1431531007-10269-8-git-send-email-vsementsov@parallels.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [PATCH 07/12] migration: add migration/block-dirty-bitmap.c List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Vladimir Sementsov-Ogievskiy , qemu-devel@nongnu.org Cc: kwolf@redhat.com, peter.maydell@linaro.org, quintela@redhat.com, dgilbert@redhat.com, stefanha@redhat.com, pbonzini@redhat.com, amit.shah@redhat.com, den@openvz.org On 05/13/2015 11:30 AM, Vladimir Sementsov-Ogievskiy wrote: > Live migration of dirty bitmaps. Only named dirty bitmaps, associated w= ith > root nodes and non-root named nodes are migrated. >=20 > If destination qemu is already containing a dirty bitmap with the same = name > as a migrated bitmap (for the same node), than, if their granularities = are > the same the migration will be done, otherwise the error will be genera= ted. >=20 > If destination qemu doesn't contain such bitmap it will be created. >=20 > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > include/migration/block.h | 1 + > migration/Makefile.objs | 2 +- > migration/block-dirty-bitmap.c | 728 +++++++++++++++++++++++++++++++++= ++++++++ > vl.c | 1 + > 4 files changed, 731 insertions(+), 1 deletion(-) > create mode 100644 migration/block-dirty-bitmap.c >=20 > diff --git a/include/migration/block.h b/include/migration/block.h > index ffa8ac0..566bb9f 100644 > --- a/include/migration/block.h > +++ b/include/migration/block.h > @@ -14,6 +14,7 @@ > #ifndef BLOCK_MIGRATION_H > #define BLOCK_MIGRATION_H > =20 > +void dirty_bitmap_mig_init(void); > void blk_mig_init(void); > int blk_mig_active(void); > uint64_t blk_mig_bytes_transferred(void); > diff --git a/migration/Makefile.objs b/migration/Makefile.objs > index d929e96..128612d 100644 > --- a/migration/Makefile.objs > +++ b/migration/Makefile.objs > @@ -6,5 +6,5 @@ common-obj-y +=3D xbzrle.o > common-obj-$(CONFIG_RDMA) +=3D rdma.o > common-obj-$(CONFIG_POSIX) +=3D exec.o unix.o fd.o > =20 > -common-obj-y +=3D block.o > +common-obj-y +=3D block.o block-dirty-bitmap.o > =20 > diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bit= map.c > new file mode 100644 > index 0000000..2d3ba23 > --- /dev/null > +++ b/migration/block-dirty-bitmap.c > @@ -0,0 +1,728 @@ > +/* > + * QEMU dirty bitmap migration > + * > + * Live migration of dirty bitmaps. Only named dirty bitmaps, associat= ed with > + * root nodes and non-root named nodes are migrated. Live iteration is= disabled > + * for small data amount (see MIN_LIVE_SIZE). > + * > + * If destination qemu is already containing a dirty bitmap with the s= ame name > + * as a migrated bitmap (for the same node), than, if their granularit= ies are > + * the same the migration will be done, otherwise the error will be ge= nerated. > + * > + * If destination qemu doesn't contain such bitmap it will be created. > + * > + * format of migration: > + * > + * # Header (shared for different chunk types) > + * 1, 2 or 4 bytes: flags (see qemu_{put,put}_flags) > + * [ 1 byte: node name size ] \ flags & DEVICE_NAME > + * [ n bytes: node name ] / > + * [ 1 byte: bitmap name size ] \ flags & BITMAP_NAME > + * [ n bytes: bitmap name ] / > + * > + * # Start of bitmap migration (flags & START) > + * header > + * be64: granularity > + * > + * # Complete of bitmap migration (flags & COMPLETE) > + * header > + * 1 byte: bitmap enabled flag > + * > + * # Data chunk of bitmap migration > + * header > + * be64: start sector > + * be32: number of sectors > + * [ be64: buffer size ] \ ! (flags & ZEROES) > + * [ n bytes: buffer ] / > + * > + * The last chunk in stream should contain flags & EOS. The chunk may = skip > + * device and/or bitmap names, assuming them to be the same with the p= revious > + * chunk. > + * > + * > + * This file is derived from migration/block.c > + * > + * Author: > + * Vladimir Sementsov-Ogievskiy > + * > + * original copyright message: > + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > + * Copyright IBM, Corp. 2009 > + * > + * Authors: > + * Liran Schour > + * > + * This work is licensed under the terms of the GNU GPL, version 2. S= ee > + * the COPYING file in the top-level directory. > + * > + * Contributions after 2012-01-13 are licensed under the terms of the > + * GNU GPL, version 2 or (at your option) any later version. > + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > + */ > + > +#include "block/block.h" > +#include "block/block_int.h" > +#include "sysemu/block-backend.h" > +#include "qemu/main-loop.h" > +#include "qemu/error-report.h" > +#include "migration/block.h" > +#include "migration/migration.h" > +#include "qemu/hbitmap.h" > +#include > + > +#define CHUNK_SIZE (1 << 10) > +#define MIN_LIVE_SIZE (1 << 20) > + > +/* Flags occupy from one to four bytes. In all but one the 7-th (EXTRA= _FLAGS) > + * bit should be set. */ > +#define DIRTY_BITMAP_MIG_FLAG_EOS 0x01 > +#define DIRTY_BITMAP_MIG_FLAG_ZEROES 0x02 > +#define DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME 0x04 > +#define DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME 0x08 > +#define DIRTY_BITMAP_MIG_FLAG_START 0x10 > +#define DIRTY_BITMAP_MIG_FLAG_COMPLETE 0x20 > +#define DIRTY_BITMAP_MIG_FLAG_BITS 0x40 > + > +#define DIRTY_BITMAP_MIG_EXTRA_FLAGS 0x80 > +#define DIRTY_BITMAP_MIG_FLAGS_SIZE_16 0x8000 > +#define DIRTY_BITMAP_MIG_FLAGS_SIZE_32 0x8080 > + > +#define DEBUG_DIRTY_BITMAP_MIGRATION 1 > + Should be 0 for the final check-in. > +#define DPRINTF(fmt, args...) \ > + do { \ > + if (DEBUG_DIRTY_BITMAP_MIGRATION) { \ > + printf("DMIG %s:%d", __func__, __LINE__); \ > + printf(fmt, ##args); \ > + } \ > + } while (0) > + > +typedef struct DirtyBitmapMigBitmapState { > + /* Written during setup phase. */ > + BlockDriverState *bs; > + const char *node_name; > + BdrvDirtyBitmap *bitmap; > + HBitmap *meta_bitmap; > + uint64_t total_sectors; > + uint64_t sectors_per_chunk; > + QSIMPLEQ_ENTRY(DirtyBitmapMigBitmapState) entry; > + > + /* For bulk phase. */ > + bool bulk_completed; > + uint64_t cur_sector; > + > + /* For dirty phase. */ > + HBitmapIter iter_dirty; > +} DirtyBitmapMigBitmapState; > + > +typedef struct DirtyBitmapMigState { > + QSIMPLEQ_HEAD(dbms_list, DirtyBitmapMigBitmapState) dbms_list; > + > + bool bulk_completed; > + bool is_live_iterative; > + > + /* for send_bitmap_bits() */ > + BlockDriverState *prev_bs; > + BdrvDirtyBitmap *prev_bitmap; > +} DirtyBitmapMigState; > + > +typedef struct DirtyBitmapLoadState { > + uint32_t flags; > + char node_name[256]; > + char bitmap_name[256]; > + BlockDriverState *bs; > + BdrvDirtyBitmap *bitmap; > +} DirtyBitmapLoadState; > + > +static DirtyBitmapMigState dirty_bitmap_mig_state; > + > +static uint32_t qemu_get_flags(QEMUFile *f) > +{ > + uint8_t flags =3D qemu_get_byte(f); > + if (flags & DIRTY_BITMAP_MIG_EXTRA_FLAGS) { > + flags =3D flags << 8 | qemu_get_byte(f); > + if (flags & DIRTY_BITMAP_MIG_EXTRA_FLAGS) { > + flags =3D flags << 16 | qemu_get_be16(f); > + } > + } > + > + return flags; > +} > + > +static void qemu_put_flags(QEMUFile *f, uint32_t flags) > +{ > + if (!(flags & 0xffffff00)) { > + qemu_put_byte(f, flags); > + return; > + } > + > + if (!(flags & 0xffff0000)) { > + qemu_put_be16(f, flags | DIRTY_BITMAP_MIG_FLAGS_SIZE_16); > + return; > + } > + > + qemu_put_be32(f, flags | DIRTY_BITMAP_MIG_FLAGS_SIZE_32); > +} > + > +/* read name from qemu file: > + * format: > + * 1 byte : len =3D name length (<256) > + * len bytes : name without last zero byte > + * > + * name should point to the buffer >=3D 256 bytes length > + */ > +static char *qemu_get_string(QEMUFile *f, char *name) > +{ > + int len =3D qemu_get_byte(f); > + qemu_get_buffer(f, (uint8_t *)name, len); > + name[len] =3D '\0'; > + > + DPRINTF("get name: %d %s\n", len, name); > + > + return name; > +} > + > +/* write name to qemu file: > + * format: > + * same as for qemu_get_string > + * > + * maximum name length is 255 > + */ > +static void qemu_put_string(QEMUFile *f, const char *name) > +{ > + int len =3D strlen(name); > + > + DPRINTF("put name: %d %s\n", len, name); > + > + assert(len < 256); > + qemu_put_byte(f, len); > + qemu_put_buffer(f, (const uint8_t *)name, len); > +} > + > +static void send_bitmap_header(QEMUFile *f, DirtyBitmapMigBitmapState = *dbms, > + uint32_t additional_flags) > +{ > + BlockDriverState *bs =3D dbms->bs; > + BdrvDirtyBitmap *bitmap =3D dbms->bitmap; > + uint32_t flags =3D additional_flags; > + > + if (bs !=3D dirty_bitmap_mig_state.prev_bs) { > + dirty_bitmap_mig_state.prev_bs =3D bs; > + flags |=3D DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME; > + } > + > + if (bitmap !=3D dirty_bitmap_mig_state.prev_bitmap) { > + dirty_bitmap_mig_state.prev_bitmap =3D bitmap; > + flags |=3D DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME; > + } > + > + qemu_put_flags(f, flags); > + > + if (flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) { > + qemu_put_string(f, dbms->node_name); > + } > + > + if (flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) { > + qemu_put_string(f, bdrv_dirty_bitmap_name(bitmap)); > + } > +} > + > +static void send_bitmap_start(QEMUFile *f, DirtyBitmapMigBitmapState *= dbms) > +{ > + send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_START); > + qemu_put_be32(f, bdrv_dirty_bitmap_granularity(dbms->bitmap)); > +} > + > +static void send_bitmap_complete(QEMUFile *f, DirtyBitmapMigBitmapStat= e *dbms) > +{ > + send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_COMPLETE); > + qemu_put_byte(f, bdrv_dirty_bitmap_enabled(dbms->bitmap)); > +} > + > +static void send_bitmap_bits(QEMUFile *f, DirtyBitmapMigBitmapState *d= bms, > + uint64_t start_sector, uint32_t nr_sector= s) > +{ > + /* align for buffer_is_zero() */ > + uint64_t align =3D 4 * sizeof(long); > + uint64_t buf_size =3D > + (bdrv_dirty_bitmap_data_size(dbms->bitmap, nr_sectors) + align= - 1) & > + ~(align - 1); > + uint8_t *buf =3D g_malloc0(buf_size); > + uint32_t flags =3D DIRTY_BITMAP_MIG_FLAG_BITS; > + > + bdrv_dirty_bitmap_serialize_part(dbms->bitmap, buf, > + start_sector, nr_sectors); > + > + if (buffer_is_zero(buf, buf_size)) { > + g_free(buf); > + buf =3D NULL; > + flags |=3D DIRTY_BITMAP_MIG_FLAG_ZEROES; > + } > + > + DPRINTF("parameters:" > + "\n flags: %x" > + "\n start_sector: %" PRIu64 > + "\n nr_sectors: %" PRIu32 > + "\n data_size: %" PRIu64 "\n", > + flags, start_sector, nr_sectors, buf_size); > + > + send_bitmap_header(f, dbms, flags); > + > + qemu_put_be64(f, start_sector); > + qemu_put_be32(f, nr_sectors); > + > + /* if a block is zero we need to flush here since the network > + * bandwidth is now a lot higher than the storage device bandwidth= . > + * thus if we queue zero blocks we slow down the migration. > + * also, skip writing block when migrate only dirty bitmaps. */ > + if (flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) { > + qemu_fflush(f); > + return; > + } > + > + qemu_put_be64(f, buf_size); > + qemu_put_buffer(f, buf, buf_size); > + g_free(buf); > +} > + > + > +/* Called with iothread lock taken. */ > + > +static void set_dirty_tracking(void) > +{ > + DirtyBitmapMigBitmapState *dbms; > + > + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) { > + dbms->meta_bitmap =3D > + bdrv_create_meta_bitmap(dbms->bitmap, CHUNK_SIZE); > + dbms->sectors_per_chunk =3D > + UINT64_C(1) << hbitmap_granularity(dbms->meta_bitmap); > + } > +} > + > +static void unset_dirty_tracking(void) > +{ > + DirtyBitmapMigBitmapState *dbms; > + > + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) { > + bdrv_release_meta_bitmap(dbms->bitmap); > + } > +} > + > +static void init_dirty_bitmap_migration(void) > +{ > + BlockDriverState *bs; > + BdrvDirtyBitmap *bitmap; > + DirtyBitmapMigBitmapState *dbms; > + uint64_t total_bytes =3D 0; > + > + dirty_bitmap_mig_state.bulk_completed =3D false; > + dirty_bitmap_mig_state.prev_bs =3D NULL; > + dirty_bitmap_mig_state.prev_bitmap =3D NULL; > + > + for (bs =3D bdrv_next(NULL); bs; bs =3D bdrv_next(bs)) { > + for (bitmap =3D bdrv_next_dirty_bitmap(bs, NULL); bitmap; > + bitmap =3D bdrv_next_dirty_bitmap(bs, bitmap)) { > + if (!bdrv_dirty_bitmap_name(bitmap)) { > + continue; > + } > + > + if (!bdrv_get_node_name(bs) && blk_bs(bs->blk) !=3D bs) { > + /* not named non-root node */ > + continue; > + } > + > + dbms =3D g_new0(DirtyBitmapMigBitmapState, 1); > + dbms->bs =3D bs; > + dbms->node_name =3D bdrv_get_node_name(bs); > + if (!dbms->node_name || dbms->node_name[0] =3D=3D '\0') { > + dbms->node_name =3D bdrv_get_device_name(bs); > + } > + dbms->bitmap =3D bitmap; > + dbms->total_sectors =3D bdrv_nb_sectors(bs); > + total_bytes +=3D > + bdrv_dirty_bitmap_data_size(bitmap, dbms->total_sector= s); > + > + QSIMPLEQ_INSERT_TAIL(&dirty_bitmap_mig_state.dbms_list, > + dbms, entry); > + } > + } > + > + dirty_bitmap_mig_state.is_live_iterative =3D total_bytes > MIN_LIV= E_SIZE; > +} > + > +/* Called with no lock taken. */ > +static void bulk_phase_send_chunk(QEMUFile *f, DirtyBitmapMigBitmapSta= te *dbms) > +{ > + uint32_t nr_sectors =3D MIN(dbms->total_sectors - dbms->cur_sector= , > + dbms->sectors_per_chunk); > + > + send_bitmap_bits(f, dbms, dbms->cur_sector, nr_sectors); > + > + dbms->cur_sector +=3D nr_sectors; > + if (dbms->cur_sector >=3D dbms->total_sectors) { > + dbms->bulk_completed =3D true; > + } > +} > + > +/* Called with no lock taken. */ > +static void bulk_phase(QEMUFile *f, bool limit) > +{ > + DirtyBitmapMigBitmapState *dbms; > + > + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) { > + while (!dbms->bulk_completed) { > + bulk_phase_send_chunk(f, dbms); > + if (limit && qemu_file_rate_limit(f)) { > + return; > + } > + } > + } > + > + dirty_bitmap_mig_state.bulk_completed =3D true; > +} > + > +static void blk_mig_reset_dirty_cursor(void) > +{ > + DirtyBitmapMigBitmapState *dbms; > + > + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) { > + hbitmap_iter_init(&dbms->iter_dirty, dbms->meta_bitmap, 0); > + } > +} > + > +/* Called with iothread lock taken. */ > +static bool dirty_phase_send_chunk(QEMUFile *f, DirtyBitmapMigBitmapSt= ate *dbms) > +{ > + uint32_t nr_sectors; > + size_t old_pos =3D dbms->iter_dirty.pos; > + int64_t cur =3D hbitmap_iter_next(&dbms->iter_dirty); > + > + /* restart search from the beginning */ > + if (old_pos && cur =3D=3D -1) { > + hbitmap_iter_init(&dbms->iter_dirty, dbms->meta_bitmap, 0); > + cur =3D hbitmap_iter_next(&dbms->iter_dirty); > + } > + > + if (cur =3D=3D -1) { > + hbitmap_iter_init(&dbms->iter_dirty, dbms->meta_bitmap, 0); > + return false; > + } > + > + nr_sectors =3D MIN(dbms->total_sectors - cur, dbms->sectors_per_ch= unk); > + send_bitmap_bits(f, dbms, cur, nr_sectors); > + hbitmap_reset(dbms->meta_bitmap, cur, dbms->sectors_per_chunk); > + > + return true; > +} > + > +/* Called with iothread lock taken. */ > +static void dirty_phase(QEMUFile *f, bool limit) > +{ > + DirtyBitmapMigBitmapState *dbms; > + > + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) { > + while (dirty_phase_send_chunk(f, dbms)) { > + if (limit && qemu_file_rate_limit(f)) { > + return; > + } > + } > + } > +} > + > + > +/* Called with iothread lock taken. */ > +static void dirty_bitmap_mig_cleanup(void) > +{ > + DirtyBitmapMigBitmapState *dbms; > + > + unset_dirty_tracking(); > + > + while ((dbms =3D QSIMPLEQ_FIRST(&dirty_bitmap_mig_state.dbms_list)= ) !=3D NULL) { > + QSIMPLEQ_REMOVE_HEAD(&dirty_bitmap_mig_state.dbms_list, entry)= ; > + g_free(dbms); > + } > +} > + > +static void dirty_bitmap_migration_cancel(void *opaque) > +{ > + dirty_bitmap_mig_cleanup(); > +} > + > +static int dirty_bitmap_save_iterate(QEMUFile *f, void *opaque) > +{ > + DPRINTF("enter\n"); > + > + if (dirty_bitmap_mig_state.bulk_completed) { > + qemu_mutex_lock_iothread(); > + dirty_phase(f, true); > + qemu_mutex_unlock_iothread(); > + } else { > + bulk_phase(f, true); > + } > + > + qemu_put_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS); > + > + return dirty_bitmap_mig_state.bulk_completed; > +} > + > +/* Called with iothread lock taken. */ > + > +static int dirty_bitmap_save_complete(QEMUFile *f, void *opaque) > +{ > + DirtyBitmapMigBitmapState *dbms; > + DPRINTF("enter\n"); > + > + if (!dirty_bitmap_mig_state.bulk_completed) { > + bulk_phase(f, false); > + } > + > + blk_mig_reset_dirty_cursor(); > + dirty_phase(f, false); > + > + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) { > + send_bitmap_complete(f, dbms); > + } > + > + qemu_put_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS); > + > + DPRINTF("Dirty bitmaps migration completed\n"); > + > + dirty_bitmap_mig_cleanup(); > + return 0; > +} > + > +static uint64_t dirty_bitmap_save_pending(QEMUFile *f, void *opaque, > + uint64_t max_size) > +{ > + DirtyBitmapMigBitmapState *dbms; > + uint64_t pending =3D 0; > + > + qemu_mutex_lock_iothread(); > + > + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) { > + uint64_t sectors =3D hbitmap_count(dbms->meta_bitmap); > + if (!dbms->bulk_completed) { > + sectors +=3D dbms->total_sectors - dbms->cur_sector; > + } > + pending +=3D bdrv_dirty_bitmap_data_size(dbms->bitmap, sectors= ); > + } > + > + qemu_mutex_unlock_iothread(); > + > + DPRINTF("pending %" PRIu64 ", max: %" PRIu64 "\n", > + pending, max_size); > + return pending; > +} > + > +/* First occurrence of this bitmap. It should be created if doesn't ex= ist */ > +static int dirty_bitmap_load_start(QEMUFile *f, DirtyBitmapLoadState *= s) > +{ > + uint32_t granularity =3D qemu_get_be32(f); > + if (!s->bitmap) { > + Error *local_err =3D NULL; > + s->bitmap =3D bdrv_create_dirty_bitmap(s->bs, granularity, > + s->bitmap_name, &local_er= r); > + if (!s->bitmap) { > + error_report("%s", error_get_pretty(local_err)); > + error_free(local_err); > + return -EINVAL; > + } > + } else { > + uint32_t dest_granularity =3D > + bdrv_dirty_bitmap_granularity(s->bitmap); > + if (dest_granularity !=3D granularity) { > + fprintf(stderr, > + "Error: " > + "Migrated bitmap granularity (%" PRIu32 ") " > + "doesn't match the destination bitmap '%s' " > + "granularity (%" PRIu32 ")\n", > + granularity, > + bdrv_dirty_bitmap_name(s->bitmap), > + dest_granularity); > + return -EINVAL; > + } > + } > + > + bdrv_disable_dirty_bitmap(s->bitmap); > + > + return 0; > +} > + > +static void dirty_bitmap_load_complete(QEMUFile *f, DirtyBitmapLoadSta= te *s) > +{ > + bool enabled; > + > + bdrv_dirty_bitmap_deserialize_finish(s->bitmap); > + > + enabled =3D qemu_get_byte(f); > + if (enabled) { > + bdrv_enable_dirty_bitmap(s->bitmap); > + } > +} > + > +static int dirty_bitmap_load_bits(QEMUFile *f, DirtyBitmapLoadState *s= ) > +{ > + uint64_t first_sector =3D qemu_get_be64(f); > + uint32_t nr_sectors =3D qemu_get_be32(f); > + DPRINTF("chunk: %lu %u\n", first_sector, nr_sectors); This will break 32 bit compilation, try "%"PRIu64 for first_sector instea= d. > + > + > + if (s->flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) { > + DPRINTF(" - zeroes\n"); > + bdrv_dirty_bitmap_deserialize_zeroes(s->bitmap, first_sector, > + nr_sectors); > + } else { > + uint8_t *buf; > + uint64_t buf_size =3D qemu_get_be64(f); > + uint64_t needed_size =3D > + bdrv_dirty_bitmap_data_size(s->bitmap, nr_sectors); > + > + if (needed_size > buf_size) { > + fprintf(stderr, > + "Error: Migrated bitmap granularity doesn't " > + "match the destination bitmap '%s' granularity\n", > + bdrv_dirty_bitmap_name(s->bitmap)); > + return -EINVAL; > + } > + > + buf =3D g_malloc(buf_size); > + qemu_get_buffer(f, buf, buf_size); > + bdrv_dirty_bitmap_deserialize_part(s->bitmap, buf, > + first_sector, > + nr_sectors); > + g_free(buf); > + } > + > + return 0; > +} > + > +static int dirty_bitmap_load_header(QEMUFile *f, DirtyBitmapLoadState = *s) > +{ > + Error *local_err =3D NULL; > + s->flags =3D qemu_get_flags(f); > + DPRINTF("flags: %x\n", s->flags); > + > + if (s->flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) { > + qemu_get_string(f, s->node_name); > + s->bs =3D bdrv_lookup_bs(s->node_name, s->node_name, &local_er= r); > + if (!s->bs) { > + error_report("%s", error_get_pretty(local_err)); > + error_free(local_err); > + return -EINVAL; > + } > + } else if (!s->bs) { > + fprintf(stderr, "Error: block device name is not set\n"); > + return -EINVAL; > + } > + > + if (s->flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) { > + qemu_get_string(f, s->bitmap_name); > + s->bitmap =3D bdrv_find_dirty_bitmap(s->bs, s->bitmap_name); > + > + /* bitmap may be NULL here, it wouldn't be an error if it is t= he > + * first occurrence of the bitmap */ > + if (!s->bitmap && !(s->flags & DIRTY_BITMAP_MIG_FLAG_START)) { > + fprintf(stderr, "Error: unknown dirty bitmap " > + "'%s' for block device '%s'\n", > + s->bitmap_name, s->node_name); > + return -EINVAL; > + } > + } else if (!s->bitmap) { > + fprintf(stderr, "Error: block device name is not set\n"); > + return -EINVAL; > + } > + > + return 0; > +} > + > +static int dirty_bitmap_load(QEMUFile *f, void *opaque, int version_id= ) > +{ > + static DirtyBitmapLoadState s; > + > + int ret =3D 0; > + > + DPRINTF("load start\n"); > + > + do { > + dirty_bitmap_load_header(f, &s); > + > + if (s.flags & DIRTY_BITMAP_MIG_FLAG_START) { > + ret =3D dirty_bitmap_load_start(f, &s); > + } else if (s.flags & DIRTY_BITMAP_MIG_FLAG_COMPLETE) { > + dirty_bitmap_load_complete(f, &s); > + } else if (s.flags & DIRTY_BITMAP_MIG_FLAG_BITS) { > + ret =3D dirty_bitmap_load_bits(f, &s); > + } > + > + DPRINTF("ret: %d\n", ret); > + if (!ret) { > + ret =3D qemu_file_get_error(f); > + } > + > + DPRINTF("ret: %d\n", ret); > + if (ret) { > + return ret; > + } > + } while (!(s.flags & DIRTY_BITMAP_MIG_FLAG_EOS)); > + > + DPRINTF("load finish\n"); > + return 0; > +} > + > +static int dirty_bitmap_save_setup(QEMUFile *f, void *opaque) > +{ > + DirtyBitmapMigBitmapState *dbms =3D NULL; > + init_dirty_bitmap_migration(); > + > + qemu_mutex_lock_iothread(); > + /* start track dirtiness of dirty bitmaps */ > + set_dirty_tracking(); > + qemu_mutex_unlock_iothread(); > + > + blk_mig_reset_dirty_cursor(); > + > + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) { > + send_bitmap_start(f, dbms); > + } > + qemu_put_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS); > + > + return 0; > +} > + > +static bool dirty_bitmap_is_active(void *opaque) > +{ > + return migrate_dirty_bitmaps(); > +} > + > +static bool dirty_bitmap_live_iterate_is_active(void *opaque) > +{ > + return migrate_dirty_bitmaps() && dirty_bitmap_mig_state.is_live_i= terative; > +} Why not swap the two conditions to avoid the function call? > + > +static SaveVMHandlers savevm_dirty_bitmap_handlers =3D { > + .save_live_setup =3D dirty_bitmap_save_setup, > + .save_live_complete =3D dirty_bitmap_save_complete, > + .save_live_pending =3D dirty_bitmap_save_pending, > + .load_state =3D dirty_bitmap_load, > + .cancel =3D dirty_bitmap_migration_cancel, > + .is_active =3D dirty_bitmap_is_active, > +}; > + > +static SaveVMHandlers savevm_dirty_bitmap_live_iterate_handlers =3D { > + .save_live_iterate =3D dirty_bitmap_save_iterate, > + .load_state =3D dirty_bitmap_load, > + .is_active =3D dirty_bitmap_live_iterate_is_active, > +}; > + > +void dirty_bitmap_mig_init(void) > +{ > + QSIMPLEQ_INIT(&dirty_bitmap_mig_state.dbms_list); > + > + register_savevm_live(NULL, "dirty-bitmap", 0, 1, > + &savevm_dirty_bitmap_handlers, > + &dirty_bitmap_mig_state); > + register_savevm_live(NULL, "dirty-bitmap-live-iterate", 0, 1, > + &savevm_dirty_bitmap_live_iterate_handlers, > + &dirty_bitmap_mig_state); > +} > diff --git a/vl.c b/vl.c > index 15bccc4..83871f5 100644 > --- a/vl.c > +++ b/vl.c > @@ -4174,6 +4174,7 @@ int main(int argc, char **argv, char **envp) > =20 > blk_mig_init(); > ram_mig_init(); > + dirty_bitmap_mig_init(); > =20 > /* If the currently selected machine wishes to override the units-= per-bus > * property of its default HBA interface type, do so now. */ >=20 With the printf format fixed and the DEBUG definition set back to 0: Reviewed-by: John Snow