[Qemu-devel] [PATCH v2 00/21] qcow2: Support refcount orders != 4

[Max Reitz <mreitz@redhat.com>]

As of version 3, the qcow2 file format supports different widths for
refcount entries, ranging from 1 to 64 bit (only powers of two).
Currently, qemu only supports 16 bit, which is the only width supported
by version 2 (compat=0.10) images.

This series adds support to qemu for all other valid refcount orders.
This is mainly done by adding two function pointers into the
BDRVQcowState structure for reading and writing refcount values
independently of the current refcount entry width; all in-memory
refcount arrays (mostly cached refcount blocks) now are void pointers
and are accessed through these functions alone.

Thanks to previous work of making the qemu code agnostic of e.g. the
number of refcount entries per refcount block, the rest is fairly
trivial. The most complex patch in this series is patch 18 which
implements changing the refcount order through qemu-img amend.

To test different refcount widths, simply invoke the qemu-iotests check
program with -o refcount_width=${your_desired_width}. The final test in
this series adds some tests for operations which do not work with
certain refcount orders and for refcount order amendment.


As of this version, this series depends on version 3 of my
"chardev: Add -qmp-pretty" series (due to different test output of test
067, which makes changing it here much nicer).


v2:
- Patch 2:
  - Added justification for always emitting refcount-width to the commit
    message [Eric]
  - Due to making this series dependent on the -qmp-pretty series, the
    reference output of iotest 067 is changed differently
- Patch 3: Added justification for using int64_t instead of uint64_t to
  the commit message [Eric]
- Patch 6: This replaces patch 7 from v1.
  - Added an assertion and an explanation why refcount_array_byte_size()
    cannot overflow [Eric]
  - Added a helper function for reallocating a refcount array (this is
    what we really want); make use of that function
- Patch 7: Was patch 6 in v1, swapped the order to reduce the diffstat
  and not having to explain why a truncating division does not truncate
  here [Eric]
  - Added overflow check in update_refcount()
  - Consequent use of int64_t for refcounts [Eric]
  - Use g_try_malloc0_n() instead of g_try_malloc0() once to prevent a
    multiplication overflow [Eric]
  - Dropped the temporary on_disk_refblock buffer in
    rebuild_refcount_structure(); just directly write the in-memory
    refcount array to disk
- Patch 8: Added an assertion in set_refcount_ro6() that the MSb of the
  given value is not set (which makes the value fit into an int64_t)
  [Eric]
- Patch 9: Dropped the refcount width specified by the image header from
  the output to prevent overflows [Eric]
- Patch 10: Kept the same, because this calculation was never meant to
  be exact
- Patch 11:
  - Added comments why certain refcount_widths to do not work with some
    tests [Eric]
  - Changed the _unsupported_imgopts argument for allowing only
    refcount_width=16 to disallow any other refcount width instead of
    listing them all (your proposal did not work, though, Eric)
  - Fixed comment in test 007 [Eric]
  - 079 does support any refcount width, we only need to use
    _make_test_img instead of calling qemu-img create manually, so do
    that
  - Disallowing 090 for refcount_width=1 was an artifact from before v1
    (where qcow2_alloc_bytes() simply returned an error when a refcount
    overflow was about to occur, instead of skipping to the next
    cluster), so remove that limitation
- Patch 12: Dropped 079 reference output change because patch 11 took
  care of that (by using _make_test_img)
- Patch 14: Removed trailing full stops in error_report() calls [Eric]
- Patch 17:
  - Fixed indentation for the function header of qcow2_amend_helper_cb()
    [Eric]
  - Renamed arguments of qcow2_amend_helper_cb() to make it more obvious
    what they mean in the context of this function [Eric, in a way]
  - Added to assertions regarding total_operations [Eric]
- Patch 18:
  - Add a typedef for the function pointer which is given to
    walk_over_reftable(); the benefit is that this makes documentation
    of the parameters easier [Eric]
  - Add an "allocated" parameter to that RefblockFinishOp and
    consequently to walk_over_reftable() so that changes in the original
    refcount structures can be tracked; if anything has been changed,
    walk_over_reftable() is rerun with alloc_refblock() until no new
    allocations were necessary anymore
  - Some changes to alloc_refblock() and flush_refblock() to accommodate
    to this change
  - Call the status_cb additionally once at the end of
    walk_over_reftable() [Eric]
  - Fix leak of the new reftable clusters on error [Eric]
- Patch 21:
  - Added comment that snapshotting an image with a small refcount width
    fails for now, but may work in the future [Eric]
  - %s/shoud/should/ [Eric]
  - 64s/qocw2/qcow2/
  - Added comment that failing to amend an image because of refcount
    overflows may work in the future [Eric]
  - 232s/entriy/entry/
  - Added a test for multiple walks during refcount width amendment (for
    the changes to patch 18)


git-backport-diff against v1:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/21:[----] [--] 'qcow2: Add two new fields to BDRVQcowState'
002/21:[0015] [FC] 'qcow2: Add refcount_width to format-specific info'
003/21:[----] [--] 'qcow2: Use 64 bits for refcount values'
004/21:[----] [--] 'qcow2: Respect error in qcow2_alloc_bytes()'
005/21:[----] [--] 'qcow2: Refcount overflow and qcow2_alloc_bytes()'
006/21:[down] 'qcow2: Helper for refcount array reallocation'
007/21:[0062] [FC] 'qcow2: Helper function for refcount modification'
008/21:[0003] [FC] 'qcow2: More helpers for refcount modification'
009/21:[0004] [FC] 'qcow2: Open images with refcount order != 4'
010/21:[----] [--] 'qcow2: refcount_order parameter for qcow2_create2'
011/21:[0096] [FC] 'iotests: Prepare for refcount_width option'
012/21:[0018] [FC] 'qcow2: Allow creation with refcount order != 4'
013/21:[----] [--] 'block: Add opaque value to the amend CB'
014/21:[0018] [FC] 'qcow2: Use error_report() in qcow2_amend_options()'
015/21:[----] [--] 'qcow2: Use abort() instead of assert(false)'
016/21:[----] [--] 'qcow2: Split upgrade/downgrade paths for amend'
017/21:[0014] [FC] 'qcow2: Use intermediate helper CB for amend'
018/21:[0159] [FC] 'qcow2: Add function for refcount order amendment'
019/21:[----] [--] 'qcow2: Invoke refcount order amendment function'
020/21:[----] [--] 'qcow2: Point to amend function in check'
021/21:[0051] [FC] 'iotests: Add test for different refcount widths'


Max Reitz (21):
  qcow2: Add two new fields to BDRVQcowState
  qcow2: Add refcount_width to format-specific info
  qcow2: Use 64 bits for refcount values
  qcow2: Respect error in qcow2_alloc_bytes()
  qcow2: Refcount overflow and qcow2_alloc_bytes()
  qcow2: Helper for refcount array reallocation
  qcow2: Helper function for refcount modification
  qcow2: More helpers for refcount modification
  qcow2: Open images with refcount order != 4
  qcow2: refcount_order parameter for qcow2_create2
  iotests: Prepare for refcount_width option
  qcow2: Allow creation with refcount order != 4
  block: Add opaque value to the amend CB
  qcow2: Use error_report() in qcow2_amend_options()
  qcow2: Use abort() instead of assert(false)
  qcow2: Split upgrade/downgrade paths for amend
  qcow2: Use intermediate helper CB for amend
  qcow2: Add function for refcount order amendment
  qcow2: Invoke refcount order amendment function
  qcow2: Point to amend function in check
  iotests: Add test for different refcount widths

 block.c                          |   4 +-
 block/qcow2-cluster.c            |  23 +-
 block/qcow2-refcount.c           | 946 +++++++++++++++++++++++++++++++++------
 block/qcow2.c                    | 256 ++++++++---
 block/qcow2.h                    |  24 +-
 include/block/block.h            |   4 +-
 include/block/block_int.h        |   4 +-
 qapi/block-core.json             |   5 +-
 qemu-img.c                       |   5 +-
 tests/qemu-iotests/007           |   3 +
 tests/qemu-iotests/015           |   2 +
 tests/qemu-iotests/026           |   7 +
 tests/qemu-iotests/029           |   1 +
 tests/qemu-iotests/049.out       | 112 ++---
 tests/qemu-iotests/051           |   3 +
 tests/qemu-iotests/058           |   2 +
 tests/qemu-iotests/060.out       |   1 +
 tests/qemu-iotests/061.out       |  14 +-
 tests/qemu-iotests/065           |  23 +-
 tests/qemu-iotests/067           |   2 +
 tests/qemu-iotests/067.out       |   5 +
 tests/qemu-iotests/079           |  10 +-
 tests/qemu-iotests/079.out       |  38 +-
 tests/qemu-iotests/080           |   2 +
 tests/qemu-iotests/082.out       |  48 +-
 tests/qemu-iotests/085.out       |  38 +-
 tests/qemu-iotests/089           |   2 +
 tests/qemu-iotests/089.out       |   2 +
 tests/qemu-iotests/108           |   2 +
 tests/qemu-iotests/112           | 252 +++++++++++
 tests/qemu-iotests/112.out       | 131 ++++++
 tests/qemu-iotests/common.filter |   3 +-
 tests/qemu-iotests/group         |   1 +
 33 files changed, 1644 insertions(+), 331 deletions(-)
 create mode 100755 tests/qemu-iotests/112
 create mode 100644 tests/qemu-iotests/112.out

-- 
1.9.3