Re: [Qemu-devel] [PATCH v6 00/21] block: transactionless incremental backup series

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 10230 bytes --]

On Fri, Apr 17, 2015 at 07:49:48PM -0400, John Snow wrote:
> It's spring! The winter snow has thawed and so here is a new
> patch series to enter your life and warm your heart.
> 
> This patchset enables the in-memory part of the incremental backup
> feature, without transactional support.
> 
> Support for transactions was separated into a separate series which
> is also now available on-list. Getting this portion of the series
> committed will help stabilize work on bitmap persistence and bitmap
> migration.
> 
> Thanks to Fam Zheng for the original versions of this patchset,
> And thanks to Max and Eric for reviewing 2,396 versions of it since.
> 
> ===
> v6:
> ===
> 
> 01: s/underlaying/underlying/
>     Removed a reference to 'disabled' bitmaps.
>     Touching up inconsistent list indentation.
>     Added FreeBSD Documentation License, primarily to be difficult
> 07: More in-line documentation for hbitmap_merge, for return value.
>     Fix size cache index to be uint64_t.
> 09: Grammar fixes from Eric Blake, kept R-Bs.
> 10: Moved yield into the do{}while(). Now we check to see if we should
>        yield/cancel after each unsuccessful sector we transfer.
>     Some documentation additions for Eric Blake.
> 15: corrected 'num_elements' to 'start'
> 18: Refactored qmp.py event functions,
>     Added in more explicit exception classes.
>     No changes to iotests.py, just qmp.py.
> 
> Key:
> [----] : patches are identical
> [####] : number of functional differences between upstream/downstream patch
> [down] : patch is downstream-only
> The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
> 
> 001/21:[0057] [FC] 'docs: incremental backup documentation'
> 002/21:[----] [--] 'qapi: Add optional field "name" to block dirty bitmap'
> 003/21:[----] [--] 'qmp: Ensure consistent granularity type'
> 004/21:[----] [--] 'qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove'
> 005/21:[----] [--] 'block: Introduce bdrv_dirty_bitmap_granularity()'
> 006/21:[----] [--] 'hbitmap: cache array lengths'
> 007/21:[0008] [FC] 'hbitmap: add hbitmap_merge'
> 008/21:[----] [--] 'block: Add bitmap disabled status'
> 009/21:[0008] [FC] 'block: Add bitmap successors'
> 010/21:[0013] [FC] 'qmp: Add support of "dirty-bitmap" sync mode for drive-backup'
> 011/21:[----] [--] 'qmp: add block-dirty-bitmap-clear'
> 012/21:[----] [--] 'qmp: Add dirty bitmap status field in query-block'
> 013/21:[----] [--] 'block: add BdrvDirtyBitmap documentation'
> 014/21:[----] [--] 'block: Ensure consistent bitmap function prototypes'
> 015/21:[0002] [FC] 'block: Resize bitmaps on bdrv_truncate'
> 016/21:[----] [--] 'hbitmap: truncate tests'
> 017/21:[----] [-C] 'iotests: add invalid input incremental backup tests'
> 018/21:[0095] [FC] 'iotests: add QMP event waiting queue'
> 019/21:[----] [--] 'iotests: add simple incremental backup case'
> 020/21:[----] [--] 'iotests: add incremental backup failure recovery test'
> 021/21:[----] [--] 'iotests: add incremental backup granularity tests'
> 
> ===
> v5:
> ===
> 
> 10: Code has been moved into backup_run_incremental()
>     'polyrhythm' check is removed,
>         clusters_per_iter variable is introduced instead.
>     If the bitmap granularity is larger than the backup granularity,
>         loop over the backup_do_cow call multiple times.
>     If the bitmap granularity is smaller, skip the iterator ahead as
>         we had been doing previously.
> 14: Only whitespace changes caused by patch 10.
> 15: Changed my approach for clearing out data for the hbitmap
>         truncate shrink case, as suggested by Stefan
> 18: Added a proper timeout mechanism to qmp.pull_event():
>       wait=False or wait=0.0 implies non-blocking.
>       wait=True implies blocking.
>       wait=60.0 implies a 60 second timeout.
>       VM.event_wait() now uses a 60 second timeout by default.
> 19: Many things:
>     The big picture is to add a set of full backups alongside the
>     incremental backups created during the test to be able to test
>     the validity of each incremental at the conclusion of the test
>     when we can shut the VM down.
> 
>     To do this, there are two basic changes:
>     (1) Keep a list of pairs of backup filenames (incremental, reference);
>         create a full reference backup for every incremental created.
>     (2) Refactor the backup helper functions a bit.
> 20: Naming fallout from 19
>     Added calls to vm.shutdown() and check_backups().
> 21: NEW, adds granularity tests that cover the changes in patch 10.
> 
> ===
> v4:
> ===
> 
> 04: Some in-line documentation for block_dirty_bitmap_lookup
>     Changed behavior with respect to aio_context
>       (always acquire, release if pbs == null, give to user otherwise)
> 10: Removed vestigial (currently nop) bdrv_enable_dirty_bitmap call
>     Kept R-B.
> 16: Added some comments to test_check_boundary_bits.
>     Kept R-B.
> 17: Folded in refactor from "incremental transactions v1" (Poor Kitty)
> 18: Pulled forward from "incremental transactions v1"
>     Kept R-B from that series.
> 19: Folded in refactor from "incremental transactions v1"
>     Added offset assertions into wait_incremental
> 20: Removed error tolerance from wait_until_completed, as
>     these patches no longer make use of that feature.
> 
> ===
> v3:
> ===
> 
> 01: Removed enabled/disabled modes information.
>     Elaborated on events that can occur during error cases.
> 04: Added an AioContext out parameter to block_dirty_bitmap_lookup.
> 06: NEW:
>     Cache the array lengths for hbitmap.
> 07: hbitmap_merge now uses the cached array lengths.
> 11: block-dirty-bitmap-clear is edited for the new block_dirty_bitmap_lookup.
> 12: Removed the "disabled" status, leaving just "Frozen."
> 15: Moved bdrv_truncate_dirty_bitmap to be static local
>     Inlined dirty_bitmap_truncate function.
>     Removed size[] caching into new patch (06, above)
>     hbitmap_truncate now keeps correct bit population count
>     hbitmap_truncate now uses hbitmap_reset BEFORE the truncate,
>         to avoid tricky out-of-bounds usages.
>     Remove g_realloc_n call that is not available in glib 2.12 (or 2.22)
>     Renamed "truncate" to "shrink" to make that more clear
>         to people who aren't me (at last count: 7+ billion)
> 16 NEW:
>    hbitmap_truncate tests.
> 
> ===
> v2:
> ===
> 
> 01: Added a new opening blurb.
>     Adjusted codeblock indentations to be 4 spaces instead of 3,
>     so it works as MD or GFMD.
>     Adjusted errors explanation.
>     Make visual separations between json data and shell commands
>     Eliminate any ligering single quotes
> 
> 07: Remember that memset takes bytes, not n_items ...
> 
> ===
> v1:
> ===
> 
> Deletions:
>  - Removed Transactions, to be added later.
>  - Removed Transaction tests, as above.
> 
> Changes:
> 01: Indentation fixes.
>     Removed enable/disable documentation.
>     Added a note that transactions aren't implemented yet.
>     Removed my needless commas
>     Added error case documentation.
> 
> 07: QMP enable/disable commands are deleted.
> 
> 14: Some comments concerning assertions.
>     Scrub re-alloc memory if we expand the array.
>     Do not attempt to scrub memory if fix_count is 0
> 
> Changes made with Reviews kept:
> 
> 02: Since 2.4
> 04: Since 2.4
>     Demingled the QMP command documentation.
> 08: Additions to what was qmp_block_dirty_enable/disable
>     are no longer present as those function no longer exist.
> 09: Since 2.4
> 10: Since 2.4
>     Demingled QMP command documentation.
> 11: Since 2.4
> 15: Test 112 --> 124
> 17: Number of tests altered. (Only 4, now.)
> 
> Fam Zheng (1):
>   qapi: Add optional field "name" to block dirty bitmap
> 
> John Snow (20):
>   docs: incremental backup documentation
>   qmp: Ensure consistent granularity type
>   qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove
>   block: Introduce bdrv_dirty_bitmap_granularity()
>   hbitmap: cache array lengths
>   hbitmap: add hbitmap_merge
>   block: Add bitmap disabled status
>   block: Add bitmap successors
>   qmp: Add support of "dirty-bitmap" sync mode for drive-backup
>   qmp: add block-dirty-bitmap-clear
>   qmp: Add dirty bitmap status field in query-block
>   block: add BdrvDirtyBitmap documentation
>   block: Ensure consistent bitmap function prototypes
>   block: Resize bitmaps on bdrv_truncate
>   hbitmap: truncate tests
>   iotests: add invalid input incremental backup tests
>   iotests: add QMP event waiting queue
>   iotests: add simple incremental backup case
>   iotests: add incremental backup failure recovery test
>   iotests: add incremental backup granularity tests
> 
>  block.c                       | 243 ++++++++++++++++++++++++++--
>  block/backup.c                | 155 +++++++++++++++---
>  block/mirror.c                |  46 +++---
>  blockdev.c                    | 176 +++++++++++++++++++-
>  docs/bitmaps.md               | 352 ++++++++++++++++++++++++++++++++++++++++
>  hmp.c                         |   3 +-
>  include/block/block.h         |  33 +++-
>  include/block/block_int.h     |   4 +-
>  include/qemu/hbitmap.h        |  23 +++
>  migration/block.c             |   9 +-
>  qapi/block-core.json          |  91 ++++++++++-
>  qmp-commands.hx               |  93 ++++++++++-
>  scripts/qmp/qmp.py            |  95 +++++++----
>  tests/qemu-iotests/124        | 363 ++++++++++++++++++++++++++++++++++++++++++
>  tests/qemu-iotests/124.out    |   5 +
>  tests/qemu-iotests/group      |   1 +
>  tests/qemu-iotests/iotests.py |  38 +++++
>  tests/test-hbitmap.c          | 255 +++++++++++++++++++++++++++++
>  util/hbitmap.c                |  85 ++++++++++
>  19 files changed, 1953 insertions(+), 117 deletions(-)
>  create mode 100644 docs/bitmaps.md
>  create mode 100644 tests/qemu-iotests/124
>  create mode 100644 tests/qemu-iotests/124.out
> 
> -- 
> 2.1.0
> 

Thanks, applied to my block-next tree:
https://github.com/stefanha/qemu/commits/block-next

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]