[Qemu-devel] [RFC] Differential Backups

[Qemu-devel] [RFC] Differential Backups

[John Snow]

This is a feature that should be very easy to add on top of the existing 
incremental feature, since it's just a difference in how the bitmap is 
treated:

Incremental
- Links to the last incremental (managed by libvirt)
- Clears the bitmap after creation

Differential:
- Links to the last full backup always (managed by libvirt)
- Does not clear the bitmap after creation

No biggie.

How it works currently: Incremental backups are created via the 
MIRROR_SYNC_MODE_DIRTY_BITMAP backup mode. An early version of the 
patchset actually had an additional parameter called the BitmapUseMode 
that controlled how the bitmap was cleared, a concern that was later 
made obsolete for other reasons.

I can add Differential backups in two ways:

(1) rename MIRROR_SYNC_MODE_DIRTY_BITMAP to 
MIRROR_SYNC_MODE_INCREMENTAL, and then add 
MIRROR_SYNC_MODE_DIFFERENTIAL. It's not too late to do this, since 2.4 
has just started.

(2) Re-add the BitmapUseMode parameter and add some enums:
BITMAP_USE_MODE_INCREMENTAL
BITMAP_USE_MODE_DIFFERENTIAL

I think I am partial to #1, if only to cut down on additional 
parameters, especially ones that are only useful to a small subset of 
backup types.



I am also considering adding a QMP primitive to allow people to /copy/ 
bitmaps. This would allow users to differentiate backup chains after 
they've already been started -- e.g.:

- User creates drive0, a full backup, and bitmap monthly0.
- Six days in, user decides it would be very nice to add a "weekly" 
incremental backup series to the same drive, but doesn't want to lose 
out on the monthly chain that was already started.
- User copies the monthly0 bitmap to a new weekly0 target.
- User can continue two independent incremental series intended for 
different periodicity.

Actually, the copy primitive would allow people to do both differentials 
and incrementals with the existing backup mode by copying the bitmap 
before each usage, but that's slightly yucky, so I'd rather just add 
both features to increase the flexibility of the delta backup system in 
general.

Re: [Qemu-devel] [Qemu-block] [RFC] Differential Backups

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 696 bytes --]

On Wed, Apr 29, 2015 at 06:51:08PM -0400, John Snow wrote:
> This is a feature that should be very easy to add on top of the existing
> incremental feature, since it's just a difference in how the bitmap is
> treated:
> 
> Incremental
> - Links to the last incremental (managed by libvirt)
> - Clears the bitmap after creation
> 
> Differential:
> - Links to the last full backup always (managed by libvirt)
> - Does not clear the bitmap after creation
> 
> No biggie.

Differential backups can be done using incremental backup functionality
in QEMU:

The client application points QEMU to the same target repeatedly instead
of keeping separate incremental backups.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [Qemu-block] [RFC] Differential Backups

[John Snow]

On 05/05/2015 06:25 AM, Stefan Hajnoczi wrote:
> On Wed, Apr 29, 2015 at 06:51:08PM -0400, John Snow wrote:
>> This is a feature that should be very easy to add on top of the existing
>> incremental feature, since it's just a difference in how the bitmap is
>> treated:
>>
>> Incremental
>> - Links to the last incremental (managed by libvirt)
>> - Clears the bitmap after creation
>>
>> Differential:
>> - Links to the last full backup always (managed by libvirt)
>> - Does not clear the bitmap after creation
>>
>> No biggie.
>
> Differential backups can be done using incremental backup functionality
> in QEMU:
>
> The client application points QEMU to the same target repeatedly instead
> of keeping separate incremental backups.
>
> Stefan
>

Oh, so you're saying:

[anchor]<--[diff1]

And then when making a new incremental, we re-use diff1 as a target and 
overwrite it so that it becomes:

[anchor]<--[diff2]

In effect giving us a differential.

OK, so it's possible, but we still lose out on some flexibility that a 
slightly different mode would provide us, like the ability to keep 
multiple differentials if desired. (Well, I suppose we *can* create 
those by manually copying differentials after we create them, but that 
seems hackier than necessary.)

Still, it would be such a paltry few lines of code and introduce no real 
complexity to the subsystem, and it might make libvirt's time a little 
easier for managing such things.

--js

Re: [Qemu-devel] [Qemu-block] [RFC] Differential Backups

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2786 bytes --]

On Tue, May 05, 2015 at 11:55:49AM -0400, John Snow wrote:
> On 05/05/2015 06:25 AM, Stefan Hajnoczi wrote:
> >On Wed, Apr 29, 2015 at 06:51:08PM -0400, John Snow wrote:
> >>This is a feature that should be very easy to add on top of the existing
> >>incremental feature, since it's just a difference in how the bitmap is
> >>treated:
> >>
> >>Incremental
> >>- Links to the last incremental (managed by libvirt)
> >>- Clears the bitmap after creation
> >>
> >>Differential:
> >>- Links to the last full backup always (managed by libvirt)
> >>- Does not clear the bitmap after creation
> >>
> >>No biggie.
> >
> >Differential backups can be done using incremental backup functionality
> >in QEMU:
> >
> >The client application points QEMU to the same target repeatedly instead
> >of keeping separate incremental backups.
> >
> >Stefan
> >
> 
> Oh, so you're saying:
> 
> [anchor]<--[diff1]
> 
> And then when making a new incremental, we re-use diff1 as a target and
> overwrite it so that it becomes:
> 
> [anchor]<--[diff2]
> 
> In effect giving us a differential.
> 
> OK, so it's possible, but we still lose out on some flexibility that a
> slightly different mode would provide us, like the ability to keep multiple
> differentials if desired. (Well, I suppose we *can* create those by manually
> copying differentials after we create them, but that seems hackier than
> necessary.)
> 
> Still, it would be such a paltry few lines of code and introduce no real
> complexity to the subsystem, and it might make libvirt's time a little
> easier for managing such things.

I have CCed Eric Blake and the libvirt mailing list.

This is a good time to discuss libvirt requirements for backup APIs.
Current work for QEMU 2.4:
 * QMP command to take incremental backup of guest disks in a single
   atomic operation
 * Dirty bitmap persistence across live migration and QEMU restart

Eric: Do you have any opinion on a differential backup feature in
addition to incremental backup.

My thoughts are that QEMU should only copy out changed blocks and let
the backup application worry about concepts like incremental vs
differential.

From a host performance perspective, copying out the same blocks that
have already been sent in a previous backup is a waste of I/O bandwidth.
Even backup applications that want differential backup may not actually
use the QEMU feature for this reason.

Regarding the size of the patch, there's a cost to adding the tests,
documentation, and commiting to QMP APIs.  These costs dwarf the small
code change that preserves the dirty bitmap.

If there's a concrete requirement for this feature then I'm happy with
it, but let's not add bells and whistles just because we can.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]