[Qemu-devel] [PATCH v2 00/45] Block job improvements for 1.3

[Paolo Bonzini <pbonzini@redhat.com>]

Hi all, this is the resubmission of my block job patches, originally
meant for 1.2.  This still does not include a persistent dirty bitmap,
which I hope to post in October.

The patches are organized as follows:

01-13   preparatory work for block job errors, including support for
        pausing and resuming jobs

14-18   introduce block job errors, and add support in block-stream

19-25   preparatory work for block mirroring: new commands/concepts
        and creating new functions out of existing code.

26-33   introduce a simple version of mirroring.  The initial patch
        add the mirroring logic, followed by the ability to switch to
        the destination of migration and to handle errors during the job.
        All these changes come with testcases.  Removing the ability to
        query the target file is the main change from v1.

34-41   These patches introduce the first optimizations, namely supporting
        an arbitrary granularity for the dirty bitmap.  The current default,
        1M, is too coarse to let the job converge quickly and in almost
        real-time.  These patches reimplement the block device dirty bitmap
        to allow efficient iteration, and add cluster copy-on-write logic.
        Cluster copy-on-write is needed because management will want to
        start the copy before the backing file is in place in the destination;
        if mirroring takes care of copy-on-write, BDRV_O_NO_BACKING can be
        used even if the granularity is smaller than the cluster size.

42-45   A second round optimizations, replacing serialized read-write
        operations with multiple asynchronous I/O operations.  The various
        in-flight operations can be of arbitrary size.  The initial copy
        will end up reading large chunks sequentially (10M by default),
        while subsequent passes can mimic more closely the guest's I/O
        patterns.

All comments from Kevin's partial review are addressed, so I believe
the first 29 patches should be ready to go.  Laszlo already reviewed some
of the subsequent parts.

Please review!

Jeff Cody (1):
  blockdev: rename block_stream_cb to a generic block_job_cb

Paolo Bonzini (44):
  qerror/block: introduce QERR_BLOCK_JOB_NOT_ACTIVE
  block: fix documentation of block_job_cancel_sync
  block: move job APIs to separate files
  block: add block_job_query
  block: add support for job pause/resume
  qmp: add block-job-pause and block-job-resume
  qemu-iotests: add test for pausing a streaming operation
  block: rename block_job_complete to block_job_completed
  iostatus: rename BlockErrorAction, BlockQMPEventAction
  iostatus: move BlockdevOnError declaration to QAPI
  iostatus: change is_read to a bool
  iostatus: reorganize io error code
  block: introduce block job error
  stream: add on-error argument
  blkdebug: process all set_state rules in the old state
  qemu-iotests: map underscore to dash in QMP argument names
  qemu-iotests: add tests for streaming error handling
  block: add bdrv_query_info
  block: add bdrv_query_stats
  block: add bdrv_open_backing_file
  block: introduce new dirty bitmap functionality
  block: export dirty bitmap information in query-block
  block: add block-job-complete
  block: introduce BLOCK_JOB_READY event
  mirror: introduce mirror job
  qmp: add drive-mirror command
  mirror: implement completion
  qemu-iotests: add mirroring test case
  iostatus: forward bdrv_iostatus_reset to block job
  mirror: add support for on-source-error/on-target-error
  qmp: add pull_event function
  qemu-iotests: add testcases for mirroring
    on-source-error/on-target-error
  host-utils: add ffsl
  add hierarchical bitmap data type and test cases
  block: implement dirty bitmap using HBitmap
  block: make round_to_clusters public
  mirror: perform COW if the cluster size is bigger than the
    granularity
  block: return count of dirty sectors, not chunks
  block: allow customizing the granularity of the dirty bitmap
  mirror: allow customizing the granularity
  mirror: switch mirror_iteration to AIO
  mirror: add buf-size argument to drive-mirror
  mirror: support more than one in-flight AIO operation
  mirror: support arbitrarily-sized iterations

 Makefile.objs                 |   5 +-
 QMP/qmp-events.txt            |  42 +++
 QMP/qmp.py                    |  20 ++
 block-migration.c             |   7 +-
 block.c                       | 480 +++++++++++++-----------------
 block.h                       |  37 ++-
 block/Makefile.objs           |   3 +-
 block/blkdebug.c              |  12 +-
 block/mirror.c                | 576 ++++++++++++++++++++++++++++++++++++
 block/stream.c                |  33 ++-
 block_int.h                   | 192 +++---------
 blockdev.c                    | 259 ++++++++++++++---
 blockjob.c                    | 282 ++++++++++++++++++
 blockjob.h                    | 280 ++++++++++++++++++
 hbitmap.c                     | 400 +++++++++++++++++++++++++
 hbitmap.h                     | 207 +++++++++++++
 hmp-commands.hx               |  73 ++++-
 hmp.c                         |  65 ++++-
 hmp.h                         |   4 +
 host-utils.h                  |  26 ++
 hw/fdc.c                      |   4 +-
 hw/ide/core.c                 |  22 +-
 hw/ide/pci.c                  |   4 +-
 hw/scsi-disk.c                |  25 +-
 hw/scsi-generic.c             |   4 +-
 hw/virtio-blk.c               |  23 +-
 monitor.c                     |   2 +
 monitor.h                     |   2 +
 qapi-schema.json              | 206 ++++++++++++-
 qemu-tool.c                   |   6 +
 qerror.h                      |   9 +
 qmp-commands.hx               |  75 ++++-
 tests/Makefile                |   2 +
 tests/qemu-iotests/030        | 260 ++++++++++++++++-
 tests/qemu-iotests/030.out    |   4 +-
 tests/qemu-iotests/040        | 658 ++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/040.out    |   5 +
 tests/qemu-iotests/group      |   5 +-
 tests/qemu-iotests/iotests.py |  19 +-
 tests/test-hbitmap.c          | 408 ++++++++++++++++++++++++++
 trace-events                  |  24 +-
 41 file modificati, 4176 inserzioni(+), 594 rimozioni(-)
 create mode 100644 block/mirror.c
 create mode 100644 blockjob.c
 create mode 100644 blockjob.h
 create mode 100644 hbitmap.c
 create mode 100644 hbitmap.h
 create mode 100755 tests/qemu-iotests/040
 create mode 100644 tests/qemu-iotests/040.out
 create mode 100644 tests/test-hbitmap.c

-- 
1.7.12