[Qemu-devel] [2.4 PATCH v3 00/19] block: transactionless incremental backup series

[John Snow <jsnow@redhat.com>]

What will life be like after this series gets committed upstream?
Where do we go from here?

This patchset enables the in-memory part of the incremental backup
feature, without transactional support.

Support for transactions was separated into a separate series which
is also now available on-list. Getting this portion of the series
committed will help stabilize work on bitmap persistence and bitmap
migration.

Thanks to Fam Zheng for the original versions of this patchset.

===
v3:
===

01: Removed enabled/disabled modes information.
    Elaborated on events that can occur during error cases.
04: Added an AioContext out parameter to block_dirty_bitmap_lookup.
06: NEW:
    Cache the array lengths for hbitmap.
07: hbitmap_merge now uses the cached array lengths.
11: block-dirty-bitmap-clear is edited for the new block_dirty_bitmap_lookup.
12: Removed the "disabled" status, leaving just "Frozen."
15: Moved bdrv_truncate_dirty_bitmap to be static local
    Inlined dirty_bitmap_truncate function.
    Removed size[] caching into new patch (06, above)
    hbitmap_truncate now keeps correct bit population count
    hbitmap_truncate now uses hbitmap_reset BEFORE the truncate,
        to avoid tricky out-of-bounds usages.
    Remove g_realloc_n call that is not available in glib 2.12 (or 2.22)
    Renamed "truncate" to "shrink" to make that more clear
        to people who aren't me (at last count: 7+ billion)
16 NEW:
   hbitmap_truncate tests.

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/19:[0014] [FC] 'docs: incremental backup documentation'
002/19:[----] [--] 'qapi: Add optional field "name" to block dirty bitmap'
003/19:[----] [--] 'qmp: Ensure consistent granularity type'
004/19:[0012] [FC] 'qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove'
005/19:[----] [--] 'block: Introduce bdrv_dirty_bitmap_granularity()'
006/19:[down] 'hbitmap: cache array lengths'
007/19:[0005] [FC] 'hbitmap: add hbitmap_merge'
008/19:[----] [--] 'block: Add bitmap disabled status'
009/19:[----] [-C] 'block: Add bitmap successors'
010/19:[----] [--] 'qmp: Add support of "dirty-bitmap" sync mode for drive-backup'
011/19:[0007] [FC] 'qmp: add block-dirty-bitmap-clear'
012/19:[down] 'qmp: Add dirty bitmap status field in query-block'
013/19:[----] [--] 'block: add BdrvDirtyBitmap documentation'
014/19:[----] [--] 'block: Ensure consistent bitmap function prototypes'
015/19:[0070] [FC] 'block: Resize bitmaps on bdrv_truncate'
016/19:[down] 'hbitmap: truncate tests'
017/19:[----] [-C] 'iotests: add invalid input incremental backup tests'
018/19:[----] [--] 'iotests: add simple incremental backup case'
019/19:[----] [--] 'iotests: add incremental backup failure recovery test'

===
v2:
===

01: Added a new opening blurb.
    Adjusted codeblock indentations to be 4 spaces instead of 3,
    so it works as MD or GFMD.
    Adjusted errors explanation.
    Make visual separations between json data and shell commands
    Eliminate any ligering single quotes

07: Remember that memset takes bytes, not n_items ...

===
v1:
===

Deletions:
 - Removed Transactions, to be added later.
 - Removed Transaction tests, as above.

Changes:
01: Indentation fixes.
    Removed enable/disable documentation.
    Added a note that transactions aren't implemented yet.
    Removed my needless commas
    Added error case documentation.

07: QMP enable/disable commands are deleted.

14: Some comments concerning assertions.
    Scrub re-alloc memory if we expand the array.
    Do not attempt to scrub memory if fix_count is 0

Changes made with Reviews kept:

02: Since 2.4
04: Since 2.4
    Demingled the QMP command documentation.
08: Additions to what was qmp_block_dirty_enable/disable
    are no longer present as those function no longer exist.
09: Since 2.4
10: Since 2.4
    Demingled QMP command documentation.
11: Since 2.4
15: Test 112 --> 124
17: Number of tests altered. (Only 4, now.)

Fam Zheng (1):
  qapi: Add optional field "name" to block dirty bitmap

John Snow (18):
  docs: incremental backup documentation
  qmp: Ensure consistent granularity type
  qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove
  block: Introduce bdrv_dirty_bitmap_granularity()
  hbitmap: cache array lengths
  hbitmap: add hbitmap_merge
  block: Add bitmap disabled status
  block: Add bitmap successors
  qmp: Add support of "dirty-bitmap" sync mode for drive-backup
  qmp: add block-dirty-bitmap-clear
  qmp: Add dirty bitmap status field in query-block
  block: add BdrvDirtyBitmap documentation
  block: Ensure consistent bitmap function prototypes
  block: Resize bitmaps on bdrv_truncate
  hbitmap: truncate tests
  iotests: add invalid input incremental backup tests
  iotests: add simple incremental backup case
  iotests: add incremental backup failure recovery test

 block.c                       | 243 +++++++++++++++++++++++++++++++--
 block/backup.c                | 149 ++++++++++++++++----
 block/mirror.c                |  46 +++----
 blockdev.c                    | 161 +++++++++++++++++++++-
 docs/bitmaps.md               | 311 ++++++++++++++++++++++++++++++++++++++++++
 hmp.c                         |   3 +-
 include/block/block.h         |  33 ++++-
 include/block/block_int.h     |   4 +-
 include/qemu/hbitmap.h        |  21 +++
 migration/block.c             |   9 +-
 qapi/block-core.json          |  90 +++++++++++-
 qmp-commands.hx               |  92 ++++++++++++-
 tests/qemu-iotests/124        | 266 ++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/124.out    |   5 +
 tests/qemu-iotests/group      |   1 +
 tests/qemu-iotests/iotests.py |   6 +-
 tests/test-hbitmap.c          | 247 +++++++++++++++++++++++++++++++++
 util/hbitmap.c                |  85 ++++++++++++
 18 files changed, 1682 insertions(+), 90 deletions(-)
 create mode 100644 docs/bitmaps.md
 create mode 100644 tests/qemu-iotests/124
 create mode 100644 tests/qemu-iotests/124.out

-- 
1.9.3