Re: [Qemu-devel] [PATCH RESEND 01/17] docs: incremental backup documentation

[John Snow <jsnow@redhat.com>]

On 03/02/2015 12:49 PM, Max Reitz wrote:
> On 2015-02-27 at 19:47, John Snow wrote:
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   docs/bitmaps.md | 303
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 303 insertions(+)
>>   create mode 100644 docs/bitmaps.md
>>
>> diff --git a/docs/bitmaps.md b/docs/bitmaps.md
>> new file mode 100644
>> index 0000000..ebb6ae8
>> --- /dev/null
>> +++ b/docs/bitmaps.md
>> @@ -0,0 +1,303 @@
>> +# Dirty Bitmaps
>> +
>> +* Dirty bitmaps can be created at any time and attached to any node
>> +(not just complete drives.)
>> +
>> +## Dirty Bitmap Names
>> +
>> +* A dirty bitmap's name is unique to the node, but bitmaps attached
>> to different
>> +nodes can share the same name.
>> +
>> +## Bitmap Modes
>> +
>> +* A Bitmap can be "enabled" (tracking writes, the default) or "disabled"
>> +(read-only, I/O is ignored.) This state is currently only changed
>> internally
>> +for the purposes of migration, and otherwise remains enabled.
>> +
>> +* A Bitmap can be "frozen," which means that it is currently in-use
>> by a backup
>> +operation and cannot be deleted, enabled, disabled, renamed, written
>> to, reset,
>> +etc.
>> +
>> +## Basic QMP Usage
>> +
>> +### Supported Commands ###
>> +
>> +* block-dirty-bitmap-add
>> +* block-dirty-bitmap-remove
>> +* block-dirty-bitmap-clear
>> +
>> +### Creation
>> +
>> +* To create a new bitmap, enabled, on the drive with id=drive0:
>> +
>> +```json
>> +{ "execute": "block-dirty-bitmap-add",
>> +  "arguments": {
>> +    "node": "drive0",
>> +    "name": "bitmap0"
>> +  }
>> +}
>> +```
>> +
>> +* This bitmap will have a default granularity that matches the
>> cluster size of
>> +its associated drive, if available, clamped to between [4KiB, 64KiB].
>> +The current default for qcow2 is 64KiB.
>> +
>> +* To create a new bitmap that tracks changes in 32KiB segments:
>> +
>> +```json
>> +{ "execute": "block-dirty-bitmap-add",
>> +  "arguments": {
>> +    "node": "drive0",
>> +    "name": "bitmap0",
>> +    "granularity": 32768
>> +  }
>> +}
>> +```
>> +
>> +### Deletion
>> +
>> +* Can be performed on a disabled bitmap, but not a frozen one.
>> +
>> +* Because bitmaps are only unique to the node to which they are
>> attached,
>> +you must specify the node/drive name here, too.
>> +
>> +```json
>> +{ "execute": "block-dirty-bitmap-remove",
>> +  "arguments": {
>> +    "node": "drive0",
>> +    "name": "bitmap0"
>> +  }
>> +}
>> +```
>> +
>> +### Resetting
>> +
>> +* Resetting a bitmap will clear all information it holds.
>> +* An incremental backup created from an empty bitmap will copy no data,
>> +as if nothing has changed.
>> +
>> +```json
>> +{ "execute": "block-dirty-bitmap-clear",
>> +  "arguments": {
>> +    "node": "drive0",
>> +    "name": "bitmap0"
>> +  }
>> +}
>> +```
>> +
>> +## Transactions (Not yet implemented)
>> +
>> +* Transactional commands are forthcoming in a future version,
>> +  and are not yet available for use. This section serves as
>> +  documentation of intent for their design and usage.
>> +
>> +### Justification
>> +Bitmaps can be safely modified when the VM is paused or halted by using
>> +the basic QMP commands. For instance, you might perform the following
>> actions:
>> +
>> +1. Boot the VM in a paused state.
>> +2. Create a full drive backup of drive0.
>> +3. Create a new bitmap attached to drive0.
>> +4. Resume execution of the VM.
>> +5. Incremental backups are ready to be created.
>> +
>> +At this point, the bitmap and drive backup would be correctly in sync,
>> +and incremental backups made from this point forward would be
>> correctly aligned
>> +to the full drive backup.
>> +
>> +This is not particularly useful if we decide we want to start
>> incremental
>> +backups after the VM has been running for a while, for which we will
>> need to
>> +perform actions such as the following:
>> +
>> +1. Boot the VM and begin execution.
>> +2. Using a single transaction, perform the following operations:
>> +    * Create bitmap0.
>> +    * Create a full drive backup of drive0.
>> +3. Incremental backups are now ready to be created.
>> +
>> +### Supported Bitmap Transactions
>> +
>> +* block-dirty-bitmap-add
>> +* block-dirty-bitmap-clear
>> +
>> +The usages are identical to their respective QMP commands, but see below
>> +for examples.
>> +
>> +### Example: New Incremental Backup
>> +
>> +As outlined in the justification, perhaps we want to create a new
>> incremental
>> +backup chain attached to a drive.
>> +
>> +```json
>> +{ "execute": "transaction",
>> +  "arguments": {
>> +    "actions": [
>> +      {"type": "block-dirty-bitmap-add",
>> +       "data": {"node": "drive0", "name": "bitmap0"} },
>> +      {"type": "drive-backup",
>
> Note that this does not do a full drive-backup during the transaction
> but only starts the block job. Above you said you'd do a single
> transaction in which you'd "Create a full drive backup of drive0" (which
> is not possible right now, however).
>
> So the problem is that any write to the node while the block job is
> running will dirty the bitmap although they will be handled by the block
> job (which is not bad, your bitmap will simply be dirtier than it needs
> to be). I don't know exactly what Stefan proposed in regards to the
> callbacks, but I can imagine two solutions to this:
>

It is my understanding that backup_run will register the 
backup_before_write_notify handler to back up any bits before they are 
written to, which means that the full backup makes a coherent backup 
_before_ any writes, and not a coherent backup _after_ all writes.

The dirty bitmap that results from this should describe new writes that 
occurred during the backup, but aren't reflected in the backup data.

> 1. You add transaction support for completing a block job and clearing a
> dirty bitmap. Then you'd simply start the drive-backup job, add the
> bitmap (both in any order), and then do a transaction of
> block-job-complete and block-dirty-bitmap-clear. But making
> block-job-complete safe for transactions can be difficult.
> 2. You add a command to completely dirty a dirty bitmap (the opposite of
> block-dirty-bitmap-clear). This way, you'd simply create a dirty bitmap,
> mark everything dirty, and then you can start doing incremental backups
> from there (so you'd force a full backup in bitmap mode).
>
> Maybe that helps. Maybe it doesn't. I don't know.
>
>> +       "data": {"device": "drive0", "target":
>> "/path/to/full_backup.img",
>> +                "sync": "full", "format": "qcow2"} }
>> +    ]
>> +  }
>> +}
>> +```
>> +
>> +### Example: New Incremental Backup Anchor Point
>> +
>> +Maybe we just want to create a new full backup with an existing
>> bitmap and
>> +want to reset the bitmap to track the new chain.
>> +
>> +```json
>> +{ "execute": "transaction",
>> +  "arguments": {
>> +    "actions": [
>> +      {"type": "block-dirty-bitmap-clear",
>> +       "data": {"node": "drive0", "name": "bitmap0"} },
>> +      {"type": "drive-backup",
>> +       "data": {"device": "drive0", "target":
>> "/path/to/new_full_backup.img",
>> +                "sync": "full", "format": "qcow2"} }
>> +    ]
>> +  }
>> +}
>> +```
>
> Same here, the dirty bitmap will be dirtier than it needs to be (not
> really bad, but with the above suggestion we can do something about this).
>
>> +
>> +## Incremental Backups
>> +
>> +The star of the show.
>
> I'm fine with a bit of fun in the documentation, but I'll see whether
> others are, too. :-)
>
> (I fear that maybe just being able to imagine someone not allowing fun
> is enough to reinforce the image of the no-fun-allowed German)
>



>> +
>> +**Nota Bene!** Only incremental backups of entire drives are
>> supported for now.
>> +So despite the fact that you can attach a bitmap to any arbitrary
>> node, they are
>> +only currently useful when attached to the root node. This is because
>> +drive-backup only supports drives/devices instead of arbitrary nodes.
>
> Well, not a real reason, since there's blockdev-backup, too. I'd just
> omit this here. If people see that you're using drive-backup, they
> should know that this can only be used for BlockBackends ("full drives").
>
> You can leave it, what I don't like is just that it sounds like it'll be
> really difficult to implement it for single BDS nodes as well; but it's
> just a technical question of drive-backup vs. blockdev-backup. The core
> of everything, block/backup.c, should be agnostic of whether you used
> drive-backup or blockdev-backup, so support is actually there, it's only
> the interface that's missing.
>

I didn't mean to imply it wouldn't ever be useful, it's just factually 
not useful yet. I can try to rephrase this to make it clearer why 
attaching bitmaps to arbitrary nodes is not yet a useful thing to do.

>> +
>> +### Example: First Incremental Backup
>> +
>> +1. Create a full backup and sync it to the dirty bitmap, as in the
>> transactional
>> +examples above; or with the VM offline, manually create a full copy
>> and then
>> +create a new bitmap before the VM begins execution.
>> +
>> +    * Let's assume the full backup is named 'full_backup.img'.
>> +    * Let's assume the bitmap you created is 'bitmap0' attached to
>> 'drive0'.
>> +
>> +2. Create a destination image for the incremental backup that
>> utilizes the
>> +full backup as a backing image.
>> +
>> +    * Let's assume it is named 'incremental.0.img'.
>> +
>> +    ```sh
>> +    # qemu-img create -f qcow2 incremental.0.img -b full_backup.img
>> -F qcow2
>> +    ```
>
> Aha, using .img for qcow2. *g*
>
> Also, we really do need some sort of image-create for QMP at some point
> in time, I guess...
>

It would be convenient, but not necessary. I am not sure what permission 
issues that would create for QEMU. Some installations may have QEMU 
quite locked down on the file creation front.

>> +
>> +3. Issue the incremental backup command:
>> +
>> +   ```json
>> +   { "execute": "drive-backup",
>> +     "arguments": {
>> +       "device": "drive0",
>> +       "bitmap": "bitmap0",
>> +       "target": "incremental.0.img",
>> +       "format": "qcow2",
>> +       "sync": "dirty-bitmap",
>> +       "mode": "existing"
>> +     }
>> +   }
>> +   ```
>> +
>> +### Example: Second Incremental Backup
>> +
>> +1. Create a new destination image for the incremental backup that
>> points to the
>> +   previous one, e.g.: 'incremental.1.img'
>> +
>> +   ```bash
>> +   # qemu-img create -f qcow2 incremental.1.img -b incremental.0.img
>> -F qcow2
>> +   ```
>> +
>> +2. Issue a new incremental backup command. The only difference here
>> is that we
>> +   have changed the target image below.
>> +
>> +   ```json
>> +   { "execute": "drive-backup",
>> +     "arguments": {
>> +       "device": "drive0",
>> +       "bitmap": "bitmap0",
>> +       "target": "incremental.1.img",
>> +       "format": "qcow2",
>> +       "sync": "dirty-bitmap",
>> +       "mode": "existing"
>> +     }
>> +   }
>> +   ```
>> +
>> +## Errors
>> +
>> +* In the event of an unsuccessful incremental backup, either via QMP
>> or a
>> +  QMP transaction, the user will receive a BLOCK_JOB_COMPLETE event with
>> +  a failure message.
>
> Well, except for when the block job did not even start, but you could
> argue that this is not an unsuccessful backup, because it isn't a backup
> at all.
>

Worth elaborating, though. If synchronous part completes, you'll get a 
message. The synchronous portion might fail and you will not get an event.

>> +
>> +* In this case, the incremental backup data contained within the
>> bitmap is
>> +  safely rolled back, and the data within the bitmap is not lost.
>> +
>> +* Once the underlying problem is fixed (e.g. more storage space is
>> freed up),
>> +  you can simply retry the incremental backup command with the same
>> bitmap.
>> +
>> +### Example
>> +
>> +1. Attempt to create an incremental backup.
>> +
>> +   ```sh
>> +   # qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F
>> qcow2
>> +   ```
>> +
>> +   ```json
>> +   { "execute": "drive-backup",
>> +     "arguments": {
>> +       "device": "drive0",
>> +       "bitmap": "bitmap0",
>> +       "target": "incremental.0.img",
>> +       "format": "qcow2",
>> +       "sync": "dirty-bitmap",
>> +       "mode": "existing"
>> +     }
>> +   }
>> +   ```
>
> This looks funny outside of markdown (like if converted to html). It
> looks like you're feeding JSON data to qemu-img. Maybe you should
> explicitly note that this is to be used over QMP while the qemu-img
> command is supposed to be executed outside of qemu (I know it's not
> ambiguous, but it would be more clear nonetheless).
>

OK.

> Oh, and by the way: At least pandoc behaves differently depending on
> whether the ``` block is indented or not. If it's indented, line break
> will not be present in the output, but the language name ("json", "sh",
> etc.) will be present there.
>

I might just convert to plaintext, then. I'm not familiar enough with 
markdown flavors to make something that works and looks nice everywhere. 
I just like it for providing some kind of visual structure.

>> +
>> +2. Receive an event notifying us of failure:
>> +
>> +   ```json
>> +   { "timestamp": { "seconds": 1424709442, "microseconds": 844524 },
>> +     "data": { "speed": 0, "offset": 0, "len": 67108864,
>> +               "error": 'No space left on device',
>
> Single quotes? Did you type this yourself? ;-)
>

Printed via a different debugging utility :)

>> +               "device": "drive1", "type": "backup" },
>> +     "event": "BLOCK_JOB_COMPLETED" }
>> +   ```
>> +
>> +3. Delete the failed incremental, and re-create the image.
>> +
>> +   ```sh
>> +   # rm incremental.0.img
>> +   # qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F
>> qcow2
>> +   ```
>> +
>> +4. Retry the command after fixing the underlaying problem,
>> +   such as freeing up space on the backup volume:
>> +
>> +   ```json
>> +   { "execute": "drive-backup",
>> +     "arguments": {
>> +       "device": "drive0",
>> +       "bitmap": "bitmap0",
>> +       "target": "incremental.0.img",
>> +       "format": "qcow2",
>> +       "sync": "dirty-bitmap",
>> +       "mode": "existing"
>> +     }
>> +   }
>> +   ```
>> +
>> +   ```json
>> +   { "timestamp": { "seconds": 1424709668, "microseconds": 526525 },
>> +     "data": { "device": "drive1", "type": "backup",
>> +               "speed": 0, "len": 67108864, "offset": 67108864},
>> +     "event": "BLOCK_JOB_COMPLETED" }
>> +   ```
>
> The second time stamp is greater than the first one, good. (I had to
> make sure you don't have a time machine or something (but if you did,
> that'd be great (do you?)))
>
> Max

You reviewed my hypothetical timestamps? :)