From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56103) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VnVjw-0007OM-JU for qemu-devel@nongnu.org; Mon, 02 Dec 2013 10:49:01 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1VnVjo-0001mM-6R for qemu-devel@nongnu.org; Mon, 02 Dec 2013 10:48:52 -0500 Received: from mail-wg0-x236.google.com ([2a00:1450:400c:c00::236]:44205) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VnVjn-0001mE-Rh for qemu-devel@nongnu.org; Mon, 02 Dec 2013 10:48:44 -0500 Received: by mail-wg0-f54.google.com with SMTP id n12so10867129wgh.21 for ; Mon, 02 Dec 2013 07:48:42 -0800 (PST) Date: Mon, 2 Dec 2013 16:48:27 +0100 From: Stefan Hajnoczi Message-ID: <20131202154827.GA30510@stefanha-thinkpad.redhat.com> References: <52931F70.5020904@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <52931F70.5020904@redhat.com> Subject: Re: [Qemu-devel] [RFC] Incremental live backup with in memory dirty bitmap List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Fam Zheng Cc: Kevin Wolf , Paolo Bonzini , "qemu-devel@nongnu.org" , Stefan Hajnoczi On Mon, Nov 25, 2013 at 05:59:12PM +0800, Fam Zheng wrote: > This is an idea about allowing online incremental backup of block > device, with drive-backup and (proposed here) in-memory block dirty > bitmap: > > 1. We enable a dirty bitmap on a block device, at the start point of > write tracking: > > (QMP) dirty-bitmap-add device=foo name=bitmap0 granularity=4k > > Since now, all the writes to "foo" will mark dirty bits on it. > > 2.1 Later, when the user wants to export all the dirty data, one > possible approach is using image fleecing: > > # step a. prepare the image fleecing target > (QMP) blockdev-add backing=foo file.filename=backup.qcow2 > id=backup0 if=none driver=qcow2 > > # step b. stop the dirty bitmap and start backup job. These two > steps need to be in a transaction, so the exported data matches the > exported dirty bitmap > (QMP) transaction: > dirty-bitmap-stop device=foo name=bitmap0 > blockdev-backup device=foo sync=none target=backup0 > > # step c. export the snapshot: > (QMP) nbd-server-add device=backup0 > > # step d. export the dirty bitmap as well, so the management > application can access it and decide what sectors to backup > (QMP) dirty-bitmap-nbd-add device=foo name=bitmap0 > > Now the management application can connect to both dirty bitmap > target and image fleecing target, and read whatever data it is > interested in. > > The tricky part is to export the dirty bitmap through NBD, but it > may also be useful for persistent the in-memory dirty bitmap too. > > 2.2 Alternatively, we can avoid exporting the dirty bitmap through > NBD, by adding another sync mode "dirty" to drive-backup, and change > step b to: > > # step b. stop the dirty bitmap and start backup job with the > dirty bitmap > (QMP) transaction: > dirty-bitmap-stop device=foo name=bitmap0 > blockdev-backup device=foo sync=dirty bitmap=bitmap0 > target=backup0 > > With the "dirty" sync mode, drive-backup/blockdev-backup takes a > dirty bitmap name of the device and writes only dirty data to > target. > > Note that NBD is not necessary for backup, we can use "drive-backup" > and omit step c as well. Also, the target could be an NBD server, in > which case we will directly send new data to a remote NBD target. > This part is fully flexible. > > Summary: what we are missing are the dirty bitmap interfaces to > add/stop/remove/query/export it and it's support of transaction, > depends on how we want to use it. E.g. for continuity of the > incremental tracking, the transaction could be: > > (QMP) transaction: > dirty-bitmap-stop device=foo name=bitmap0 > dirty-bitmap-add device=foo name=bitmap1 # Start a new > bitmap atomically > blockdev-backup device=foo sync=dirty bitmap=bitmap0 > target=backup0 > > And the last command is to free the no longer used dirty bitmap: > > (QMP) dirty-bitmap-remove device=foo name=bitmap0 > > So, do these sound good/useful at all? Any comment is welcome! This sounds like a good idea. Incremental backup has been discussed periodically and we're in a good position to actually implement it now. The dirty bitmap object's lifecycle needs to be fully thought through, including how to make the bitmap persistent. I think a reasonable approach is flushing the bitmap to disk when QEMU shuts down. The bitmap file should be marked "in use" by dirty-bitmap-start so the contents are not trusted after a power failure/crash. I like the blockdev-backup sync=dirty mode which can be used to stream dirty blocks to an external NBD server. The minimal feature set that covers persistent dirty bitmaps is: 1. dirty-bitmap-add/dirty-bitmap-remove 2. blockdev-backup sync=dirty The user would not be able to query the bitmap itself but they could receive a copy of changed blocks. Eric: any thoughts on persistent dirty bitmap APIs? Stefan