[Qemu-devel] [RFC] Incremental live backup with in memory dirty bitmap

[Qemu-devel] [RFC] Incremental live backup with in memory dirty bitmap

[Fam Zheng]

Hi all,

This is an idea about allowing online incremental backup of block 
device, with drive-backup and (proposed here) in-memory block dirty bitmap:

1. We enable a dirty bitmap on a block device, at the start point of 
write tracking:

   (QMP) dirty-bitmap-add device=foo name=bitmap0 granularity=4k

Since now, all the writes to "foo" will mark dirty bits on it.

2.1 Later, when the user wants to export all the dirty data, one 
possible approach is using image fleecing:

   # step a. prepare the image fleecing target
   (QMP) blockdev-add backing=foo file.filename=backup.qcow2 id=backup0 
if=none driver=qcow2

   # step b. stop the dirty bitmap and start backup job. These two steps 
need to be in a transaction, so the exported data matches the exported 
dirty bitmap
   (QMP) transaction:
             dirty-bitmap-stop device=foo name=bitmap0
             blockdev-backup device=foo sync=none target=backup0

   # step c. export the snapshot:
   (QMP) nbd-server-add device=backup0

   # step d. export the dirty bitmap as well, so the management 
application can access it and decide what sectors to backup
   (QMP) dirty-bitmap-nbd-add device=foo name=bitmap0

Now the management application can connect to both dirty bitmap target 
and image fleecing target, and read whatever data it is interested in.

The tricky part is to export the dirty bitmap through NBD, but it may 
also be useful for persistent the in-memory dirty bitmap too.

2.2 Alternatively, we can avoid exporting the dirty bitmap through NBD, 
by adding another sync mode "dirty" to drive-backup, and change step b to:

   # step b. stop the dirty bitmap and start backup job with the dirty 
bitmap
   (QMP) transaction:
             dirty-bitmap-stop device=foo name=bitmap0
             blockdev-backup device=foo sync=dirty bitmap=bitmap0 
target=backup0

With the "dirty" sync mode, drive-backup/blockdev-backup takes a dirty 
bitmap name of the device and writes only dirty data to target.

Note that NBD is not necessary for backup, we can use "drive-backup" and 
omit step c as well. Also, the target could be an NBD server, in which 
case we will directly send new data to a remote NBD target. This part is 
fully flexible.

Summary: what we are missing are the dirty bitmap interfaces to 
add/stop/remove/query/export it and it's support of transaction, depends 
on how we want to use it. E.g. for continuity of the incremental 
tracking, the transaction could be:

   (QMP) transaction:
             dirty-bitmap-stop device=foo name=bitmap0
             dirty-bitmap-add device=foo name=bitmap1   # Start a new 
bitmap atomically
             blockdev-backup device=foo sync=dirty bitmap=bitmap0 
target=backup0

And the last command is to free the no longer used dirty bitmap:

   (QMP) dirty-bitmap-remove device=foo name=bitmap0

So, do these sound good/useful at all? Any comment is welcome!

Thanks,
Fam

Re: [Qemu-devel] [RFC] Incremental live backup with in memory dirty bitmap

[Stefan Hajnoczi]

On Mon, Nov 25, 2013 at 05:59:12PM +0800, Fam Zheng wrote:
> This is an idea about allowing online incremental backup of block
> device, with drive-backup and (proposed here) in-memory block dirty
> bitmap:
> 
> 1. We enable a dirty bitmap on a block device, at the start point of
> write tracking:
> 
>   (QMP) dirty-bitmap-add device=foo name=bitmap0 granularity=4k
> 
> Since now, all the writes to "foo" will mark dirty bits on it.
> 
> 2.1 Later, when the user wants to export all the dirty data, one
> possible approach is using image fleecing:
> 
>   # step a. prepare the image fleecing target
>   (QMP) blockdev-add backing=foo file.filename=backup.qcow2
> id=backup0 if=none driver=qcow2
> 
>   # step b. stop the dirty bitmap and start backup job. These two
> steps need to be in a transaction, so the exported data matches the
> exported dirty bitmap
>   (QMP) transaction:
>             dirty-bitmap-stop device=foo name=bitmap0
>             blockdev-backup device=foo sync=none target=backup0
> 
>   # step c. export the snapshot:
>   (QMP) nbd-server-add device=backup0
> 
>   # step d. export the dirty bitmap as well, so the management
> application can access it and decide what sectors to backup
>   (QMP) dirty-bitmap-nbd-add device=foo name=bitmap0
> 
> Now the management application can connect to both dirty bitmap
> target and image fleecing target, and read whatever data it is
> interested in.
> 
> The tricky part is to export the dirty bitmap through NBD, but it
> may also be useful for persistent the in-memory dirty bitmap too.
> 
> 2.2 Alternatively, we can avoid exporting the dirty bitmap through
> NBD, by adding another sync mode "dirty" to drive-backup, and change
> step b to:
> 
>   # step b. stop the dirty bitmap and start backup job with the
> dirty bitmap
>   (QMP) transaction:
>             dirty-bitmap-stop device=foo name=bitmap0
>             blockdev-backup device=foo sync=dirty bitmap=bitmap0
> target=backup0
> 
> With the "dirty" sync mode, drive-backup/blockdev-backup takes a
> dirty bitmap name of the device and writes only dirty data to
> target.
> 
> Note that NBD is not necessary for backup, we can use "drive-backup"
> and omit step c as well. Also, the target could be an NBD server, in
> which case we will directly send new data to a remote NBD target.
> This part is fully flexible.
> 
> Summary: what we are missing are the dirty bitmap interfaces to
> add/stop/remove/query/export it and it's support of transaction,
> depends on how we want to use it. E.g. for continuity of the
> incremental tracking, the transaction could be:
> 
>   (QMP) transaction:
>             dirty-bitmap-stop device=foo name=bitmap0
>             dirty-bitmap-add device=foo name=bitmap1   # Start a new
> bitmap atomically
>             blockdev-backup device=foo sync=dirty bitmap=bitmap0
> target=backup0
> 
> And the last command is to free the no longer used dirty bitmap:
> 
>   (QMP) dirty-bitmap-remove device=foo name=bitmap0
> 
> So, do these sound good/useful at all? Any comment is welcome!

This sounds like a good idea.  Incremental backup has been discussed
periodically and we're in a good position to actually implement it now.

The dirty bitmap object's lifecycle needs to be fully thought through,
including how to make the bitmap persistent.  I think a reasonable
approach is flushing the bitmap to disk when QEMU shuts down.  The
bitmap file should be marked "in use" by dirty-bitmap-start so the
contents are not trusted after a power failure/crash.

I like the blockdev-backup sync=dirty mode which can be used to stream
dirty blocks to an external NBD server.

The minimal feature set that covers persistent dirty bitmaps is:
1. dirty-bitmap-add/dirty-bitmap-remove
2. blockdev-backup sync=dirty

The user would not be able to query the bitmap itself but they could
receive a copy of changed blocks.

Eric: any thoughts on persistent dirty bitmap APIs?

Stefan