[PATCH v3 0/9] block: cheaper zero handling in backup and commit

["Denis V. Lunev" <den@openvz.org>]

Backup and commit re-query the source block status for every task and
size a task by the copy buffer, so a long run of zeroes turns into a
crowd of small write-zeroes requests. This series reuses what the
up-front scan already learned and lets one write-zeroes task cover a
whole run.

Backup of a 16G qcow2 image holding 1G of data, the rest never
allocated, sync=full to a raw target on ext4:

               tasks    time
    before     16384    1.6s
    after       1088    1.0s

Such an image is the normal case rather than a corner one: a guest with
discard enabled on a disk which is mostly free leaves exactly this
shape behind.

v3, all from Andrey's review of v2:
- 3/9: the commit message says which permission stops whom. A device or
  an export asks for BLK_PERM_CONSISTENT_READ along with BLK_PERM_WRITE
  and the job withholds CONSISTENT_READ above base_overlay, having to
  share WRITE or it would block its own writes to base; a job target
  asks for WRITE alone and is stopped by the op blocker instead.
- 4/9: assertNotIn() rather than a bare assert, and test_bitmap_straddle
  checks the target map, not only the content.
- 9/9: zero_widen is a bool again. A task widened before another task's
  request was refused reached the write loop with the chunk still at its
  full widened size, so it wrote the whole range as one request with no
  BDRV_REQ_NO_FALLBACK, against a target which had just said it cannot
  zero by metadata. The chunk is now bounded whichever way the flag was
  read, and a request carries NO_FALLBACK whenever the target is still
  believed to oblige.
  block_copy_chunk_size() is called under s->lock, as its own comment
  asks for.
  The commit message no longer says a refused request wrote nothing:
  bdrv_co_do_pwrite_zeroes() fragments by bl.max_pwrite_zeroes and a
  driver may refuse a later fragment after an earlier one landed. The
  range is rewritten whole, which is safe because zeroes over zeroes
  change nothing.
- Reviewed-by tags collected on 1-8.
- rebased on master

v2, all from Andrey's review:
- 2/9: holes spelled out in both layouts, and zero runs added, so the
  write-zeroes path of 3/9 is covered too
- 3/9: COMMIT_ZERO_CHUNK has a comment of its own saying what bounds it;
  the cache check drops its dead half and asserts instead
- 4/9: a Case namedtuple pairs each size with its layout, and
  create_image(), write_layout(), dirty_layout() and backup_and_check()
  take out the duplication
- 8/9: g_assert_not_reached() for the sync mode which cannot reach there
- 9/9: a widened write-zeroes request only pays off where the target
  zeroes by metadata. supported_zero_flags rules out the targets which
  cannot, and the first widened request asks for BDRV_REQ_NO_FALLBACK to
  settle the rest, since a driver may advertise it and only learn better
  from a failing call. A target which would write the zeroes out fails
  that request without writing, and the run keeps its requests at the
  buffer chunk size from there on. With the size question settled that
  way the cap became BDRV_REQUEST_MAX_BYTES rather than 256M.
- rebased on master

Signed-off-by: Denis V. Lunev <den@openvz.org>
CC: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
CC: John Snow <jsnow@redhat.com>
CC: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>

Denis V. Lunev (9):
  block/commit: pass BDRV_WANT_PRECISE to block-status
  iotests/040: cover large and fragmented commit runs
  block/commit: batch block-status queries
  iotests/124: cover backup of zero clusters and holes
  block/block-copy: don't reserve memory for zero tasks
  block/block-copy: extract block_copy_set_task_method()
  block/block-copy: track known-zero source clusters
  block/backup: pre-fill zero_bitmap for full/bitmap
  block/block-copy: coalesce write-zeroes tasks

 block/backup.c             |  93 ++++++----
 block/block-copy.c         | 300 ++++++++++++++++++++++++++++----
 block/commit.c             |  58 +++++--
 include/block/block-copy.h |   5 +
 tests/qemu-iotests/040     | 102 ++++++++++-
 tests/qemu-iotests/040.out |   4 +-
 tests/qemu-iotests/124     | 347 ++++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/124.out |   4 +-
 8 files changed, 825 insertions(+), 88 deletions(-)


base-commit: f8296b816fabd370307cd22b0270b610fc0fa279
-- 
2.53.0