From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59084) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1W87W5-0003Vj-7j for qemu-devel@nongnu.org; Tue, 28 Jan 2014 07:11:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1W87Vw-0008CL-Oq for qemu-devel@nongnu.org; Tue, 28 Jan 2014 07:11:45 -0500 Received: from mail-ea0-x232.google.com ([2a00:1450:4013:c01::232]:57941) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1W87Vw-0008BR-EG for qemu-devel@nongnu.org; Tue, 28 Jan 2014 07:11:36 -0500 Received: by mail-ea0-f178.google.com with SMTP id a15so187171eae.37 for ; Tue, 28 Jan 2014 04:11:35 -0800 (PST) Date: Tue, 28 Jan 2014 13:11:32 +0100 From: Stefan Hajnoczi Message-ID: <20140128121132.GF16709@stefanha-thinkpad.redhat.com> References: <1389609588-32104-1-git-send-email-famz@redhat.com> <20140117092523.GF31039@stefanha-thinkpad.redhat.com> <20140120025558.GA21516@T430.nay.redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140120025558.GA21516@T430.nay.redhat.com> Subject: Re: [Qemu-devel] [PATCH 0/9] QMP: Introduce incremental drive-backup with in-memory dirty bitmap List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Fam Zheng Cc: kwolf@redhat.com, qemu-devel@nongnu.org, stefanha@redhat.com On Mon, Jan 20, 2014 at 10:55:58AM +0800, Fam Zheng wrote: > On Fri, 01/17 17:25, Stefan Hajnoczi wrote: > > On Mon, Jan 13, 2014 at 06:39:39PM +0800, Fam Zheng wrote: > > > This implements incremental backup. > > > > > > A few new QMP commands related to dirty bitmap are added: > > > > > > dirty-bitmap-add * > > > dirty-bitmap-disable * > > > dirty-bitmap-remove > > > > > > (*: also supported as transactions) > > > > > > As their name implies, they manipulate a block device's dirty bitmap. This > > > doesn't interfere with dirty bitmap used for migration, backup, mirror, etc, > > > which don't have a name and are invisible to user. Only named bitmaps (created > > > by dirty-bitmap-add) can be disabled/removed by user. > > > > > > They are added to support "user controlled write tracking", so as to determine > > > the range of date for incremental backup. > > > > > > A new sync mode for drive-backup is introduced: > > > > > > drive-backup device=.. mode=.. sync=dirty-bitmap bitmap=bitmap0 > > > > > > Which will scan dirty bitmap "bitmap0" and only copy all dirty sectors to > > > target. > > > > > > Now, let's see the usage with a simple example: > > > > > > # Start the guest > > > vm = VM() > > > vm.start() > > > > > > # Fake some guest writes with "qemu-io", this is before creating dirty > > > # bitmap, so it won't be copied > > > vm.hmp('qemu-io ide0-hd0 "write -P 0xa 512k 1M"') > > > > > > # Create a dirty bitmap to track writes > > > vm.qmp("dirty-bitmap-add", device="ide0-hd0", name="dirty-0") > > > > > > # Fake some more guest writes with "qemu-io", this will be copied > > > vm.hmp('qemu-io ide0-hd0 "write -P 0xa 512M 1M"') > > > > > > # Now "disable" the first dirty bitmap, do the backup according to it, > > > # at meantime continue to track dirty with a new dirty bitmap > > > vm.qmp("transaction", actions=[ > > > { > > > 'type': 'dirty-bitmap-disable', 'data': { > > > 'device': 'ide0-hd0', > > > 'name': 'dirty-0' > > > } > > > }, { > > > 'type': 'dirty-bitmap-add', 'data': { > > > 'device': 'ide0-hd0', > > > 'name': 'dirty-1' > > > } > > > }, { > > > 'type': 'drive-backup', 'data': { > > > 'device': 'ide0-hd0', > > > 'target': '/tmp/incre.qcow2', > > > 'bitmap': 'dirty-0', > > > 'sync': 'dirty-bitmap' > > > } > > > } > > > ]) > > > > > > # Once backup job started, the dirty bitmap can be removed (actually only > > > # hidden from user since it is still referenced by block job > > > vm.qmp("dirty-bitmap-remove", device="ide0-hd0", name="dirty-0") > > > > I'm interested in the lifecycle of a dirty bitmap (but haven't reviewed > > the patches yet). In particular, what happens if a bitmap is added to > > more than one drive? Is there a more elegant way to handle the disable, > > drive-backup, remove step (we only need to explicitly disable because we > > still need the bitmap name for the drive-backup command)? Also what > > happens if we add the bitmap again after disabling? > > A same name on that device can't be used again unless it's removed. A bitmap is > associated to (and only) one device, it can't be shared. > > > > > No need to answer all these questions, but it suggests the interface > > exposes a bit of complexity. Maybe it's possible to make it simpler and > > easier to use? > > > > At least the user has to explicitly start tracking, that's the dirty-bitmap-add > step. Alternatively, we can have "disable, drive-backup, remove" step simplified as: > > drive-backup sync=dirty-bitmap bitmap=dirty0 reset-bitmap=true > > where backup job copy out the dirty bitmap (and clears it, as reset-bitmap is > true), and backup with it atomically. > > Of course it doesn't have to actually copy the whole bitmap: it just makes the > old one anonymous, create a new empty one and give it the same name. When > backup is done, the old bitmap is removed. > > What do you think? That simplifies usage. I can think of two options: 1. We want to continue incremental backup so drive-backup should atomically swap the bitmap with a new one of the same name (as you described). 2. We want to end incremental backup so the drive-backup command should consume the bitmap and not create a new one. Then users just need to add a new bitmap and later use drive-backup. They don't need to manually manage the bitmap's lifecycle. Stefan