[PATCH v8 00/43] block: Deal with filters

[Max Reitz <mreitz@redhat.com>]

v6: https://lists.nongnu.org/archive/html/qemu-devel/2019-08/msg01715.html
v7: https://lists.nongnu.org/archive/html/qemu-block/2020-06/msg01357.html

Branch: https://github.com/XanClic/qemu.git child-access-functions-v8
Branch: https://git.xanclic.moe/XanClic/qemu.git child-access-functions-v8


Hi,

In v8, there is not too much that has changed in respect to v7.  I tried
to address all of your comments and hope I got it right.  I also hope I
got the R-bs right.


Changes from v7:
- Patch 1: Let bdrv_primary_child() assert that there’s only a single
           primary child

- Patch 2: Assert that bdrv_do_skip_filters() will not return a filter
           node

- Patch 3: Drop bdrv_unallocated_blocks_are_zero() hunk (that function
           no longer exists)

- Old patch 10: Dropped, it isn’t really important to let mirror-top
                support compressed writes, and
                bdrv_supports_compressed_writes() doesn’t really work
                with this patch

- Old patch 11: Dropped, it isn’t really important to let backup-top
                support compressed writes

- Patch 10: Rebase conflict in init_dirty_bitmap_migration()
            (The modified block is now nested one level deeper)

- Patch 11: Rebase conflict in bdrv_co_block_status()
            (bdrv_unallocated_blocks_are_zero() is no longer used)

- Patch 12:
  - Fix documentation on what the backing node after streaming is going
    to be
  - bdrv_change_backing_file() rebase conflict

- Patch 13: Try to clarify what bdrv_find_overlay() returns in its doc
            comment

- Patch 15: Optimize the overlay_bs finding loop

- Old patch 25: Rolled into the next patch (now patch 23)

- Patch 23:
  - Inline two of the three functions introduced in the old patch 25
  - The other one (bdrv_sum_allocated_file_size()) stays, but is static
    now
  (Turns out we don’t need to let block drivers use any of these
  functions for their BlockDriver.bdrv_get_allocated_File_size().)

- Old patch 27: Dropped, just let blkverify be handled like any other
                filter, because it doesn’t matter, really
                (and this allows us squashing the previous to patches)

- Patch 24: Remove _filter_actual_image_size from iotest 184

- Patch 27: Reference output change due to 184 no longer invoking
            _filter_actual_image_size

- Patch 29:
  - Let the commit message explain why base_overlay is introduced
  - Drop @replaces_node_name from qmp_drive_mirror(), because it isn’t
    needed

- Patch 31:
  - Let blk_commit_all() only commit those nodes that have a backing
    file; and ignore all filters on top of them (instead of just
    implicit filters)
  - In commit_start: %s/\<perms\>/base_perms/
  - Under commit_start.ro_cleanup: Restore the original backing file
    only if necessary, so we do not run into an abort because the
    backing chain is frozen

- Patch 33: Let img_rebase() refer to the unfiltered_bs in three more
            places

- Patch 35:
  - Documentation modifications:
    - Move a chunk of it up where it belongs (from below @backing-file
      to the general area of @block-commit)
    - Explain that you now need to issue block-job-complete whenever
      there’s a writer on @top
    - Also, @backing-file is no longer allowed whenever there’s a writer
      on @top
      - In the code, this should cause a different error message than
        when @top is in the active layer, though
  - Comment note: If @top is a root node without any writers on it, we
    still need to do an active commit, because everything else would be
    an incompatible change

- Patch 36: Trivial rebase conflict in block_int.h (necessary because
            the old patch 25 was dropped)

- Patch 37: Don’t try to fetch ImageInfo.backing-filename-format from
            the backing file open at runtime.  If the image header does
            not specify a format, just leave it empty.

- Patch 40: Use format string instead of ''.format()

- Patch 42: Add -F $IMGFMT to _make_test_img -b


git-backport-diff against v7:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/43:[0007] [FC] 'block: Add child access functions'
002/43:[0007] [FC] 'block: Add chain helper functions'
003/43:[0002] [FC] 'block: bdrv_cow_child() for bdrv_has_zero_init()'
004/43:[----] [--] 'block: bdrv_set_backing_hd() is about bs->backing'
005/43:[----] [--] 'block: Include filters when freezing backing chain'
006/43:[----] [--] 'block: Drop bdrv_is_encrypted()'
007/43:[----] [--] 'block: Add bdrv_supports_compressed_writes()'
008/43:[----] [--] 'throttle: Support compressed writes'
009/43:[----] [--] 'copy-on-read: Support compressed writes'
010/43:[0016] [FC] 'block: Use bdrv_filter_(bs|child) where obvious'
011/43:[0008] [FC] 'block: Use CAFs in block status functions'
012/43:[0017] [FC] 'stream: Deal with filters'
013/43:[0007] [FC] 'block: Use CAFs when working with backing chains'
014/43:[----] [--] 'block: Use bdrv_cow_child() in bdrv_co_truncate()'
015/43:[0009] [FC] 'block: Re-evaluate backing file handling in reopen'
016/43:[----] [--] 'block: Flush all children in generic code'
017/43:[----] [--] 'vmdk: Drop vmdk_co_flush()'
018/43:[----] [--] 'block: Iterate over children in refresh_limits'
019/43:[----] [--] 'block: Use CAFs in bdrv_refresh_filename()'
020/43:[----] [--] 'block: Use CAF in bdrv_co_rw_vmstate()'
021/43:[----] [--] 'block/snapshot: Fix fallback'
022/43:[----] [--] 'block: Use CAFs for debug breakpoints'
023/43:[0031] [FC] 'block: Improve get_allocated_file_size's default'
024/43:[0007] [FC] 'block/null: Implement bdrv_get_allocated_file_size'
025/43:[----] [--] 'blockdev: Use CAF in external_snapshot_prepare()'
026/43:[----] [--] 'block: Report data child for query-blockstats'
027/43:[0002] [FC] 'block: Use child access functions for QAPI queries'
028/43:[----] [--] 'block-copy: Use CAF to find sync=top base'
029/43:[0004] [FC] 'mirror: Deal with filters'
030/43:[----] [--] 'backup: Deal with filters'
031/43:[0020] [FC] 'commit: Deal with filters'
032/43:[----] [--] 'nbd: Use CAF when looking for dirty bitmap'
033/43:[0007] [FC] 'qemu-img: Use child access functions'
034/43:[----] [--] 'block: Drop backing_bs()'
035/43:[0052] [FC] 'blockdev: Fix active commit choice'
036/43:[0001] [FC] 'block: Inline bdrv_co_block_status_from_*()'
037/43:[0013] [FC] 'block: Leave BDS.backing_file constant'
038/43:[----] [--] 'iotests: Test that qcow2's data-file is flushed'
039/43:[----] [--] 'iotests: Let complete_and_wait() work with commit'
040/43:[0004] [FC] 'iotests: Add filter commit test cases'
041/43:[----] [--] 'iotests: Add filter mirror test cases'
042/43:[0008] [FC] 'iotests: Add test for commit in sub directory'
043/43:[----] [--] 'iotests: Test committing to overridden backing'


Max Reitz (43):
  block: Add child access functions
  block: Add chain helper functions
  block: bdrv_cow_child() for bdrv_has_zero_init()
  block: bdrv_set_backing_hd() is about bs->backing
  block: Include filters when freezing backing chain
  block: Drop bdrv_is_encrypted()
  block: Add bdrv_supports_compressed_writes()
  throttle: Support compressed writes
  copy-on-read: Support compressed writes
  block: Use bdrv_filter_(bs|child) where obvious
  block: Use CAFs in block status functions
  stream: Deal with filters
  block: Use CAFs when working with backing chains
  block: Use bdrv_cow_child() in bdrv_co_truncate()
  block: Re-evaluate backing file handling in reopen
  block: Flush all children in generic code
  vmdk: Drop vmdk_co_flush()
  block: Iterate over children in refresh_limits
  block: Use CAFs in bdrv_refresh_filename()
  block: Use CAF in bdrv_co_rw_vmstate()
  block/snapshot: Fix fallback
  block: Use CAFs for debug breakpoints
  block: Improve get_allocated_file_size's default
  block/null: Implement bdrv_get_allocated_file_size
  blockdev: Use CAF in external_snapshot_prepare()
  block: Report data child for query-blockstats
  block: Use child access functions for QAPI queries
  block-copy: Use CAF to find sync=top base
  mirror: Deal with filters
  backup: Deal with filters
  commit: Deal with filters
  nbd: Use CAF when looking for dirty bitmap
  qemu-img: Use child access functions
  block: Drop backing_bs()
  blockdev: Fix active commit choice
  block: Inline bdrv_co_block_status_from_*()
  block: Leave BDS.backing_{file,format} constant
  iotests: Test that qcow2's data-file is flushed
  iotests: Let complete_and_wait() work with commit
  iotests: Add filter commit test cases
  iotests: Add filter mirror test cases
  iotests: Add test for commit in sub directory
  iotests: Test committing to overridden backing

 qapi/block-core.json           |  58 ++--
 include/block/block.h          |   2 +-
 include/block/block_int.h      |  95 ++++---
 block.c                        | 488 ++++++++++++++++++++++++++-------
 block/backup-top.c             |   4 +-
 block/backup.c                 |   9 +-
 block/blkdebug.c               |   7 +-
 block/blklogwrites.c           |   1 -
 block/block-backend.c          |   7 +-
 block/block-copy.c             |   4 +-
 block/commit.c                 |  95 +++++--
 block/copy-on-read.c           |  13 +-
 block/filter-compress.c        |   2 -
 block/io.c                     | 142 +++++-----
 block/mirror.c                 | 119 ++++++--
 block/monitor/block-hmp-cmds.c |   2 +-
 block/null.c                   |   7 +
 block/qapi.c                   |  74 +++--
 block/snapshot.c               | 104 +++++--
 block/stream.c                 |  63 +++--
 block/throttle.c               |  11 +-
 block/vmdk.c                   |  16 --
 blockdev.c                     | 101 +++++--
 migration/block-dirty-bitmap.c |   8 +-
 nbd/server.c                   |   6 +-
 qemu-img.c                     |  43 +--
 tests/qemu-iotests/020         |  44 +++
 tests/qemu-iotests/020.out     |  10 +
 tests/qemu-iotests/040         | 238 ++++++++++++++++
 tests/qemu-iotests/040.out     |   4 +-
 tests/qemu-iotests/041         | 146 +++++++++-
 tests/qemu-iotests/041.out     |   4 +-
 tests/qemu-iotests/153.out     |   2 +-
 tests/qemu-iotests/184         |   3 +-
 tests/qemu-iotests/184.out     |  14 +-
 tests/qemu-iotests/204.out     |   1 +
 tests/qemu-iotests/228         |   6 +-
 tests/qemu-iotests/228.out     |   6 +-
 tests/qemu-iotests/244         |  49 ++++
 tests/qemu-iotests/244.out     |   7 +
 tests/qemu-iotests/245         |   4 +-
 tests/qemu-iotests/273.out     |   4 +-
 tests/qemu-iotests/iotests.py  |  10 +-
 43 files changed, 1591 insertions(+), 442 deletions(-)

-- 
2.26.2