[Qemu-devel] [PATCH 00/47] Block job improvements for 1.2

[Paolo Bonzini <pbonzini@redhat.com>]

Hi all, this is the first non-RFC submission of my block job patches
for 1.2.  Everything is there, including multiple in-flight operations
in the mirroring job and new testcases (for all of streaming, mirroring,
hierarchical bitmap).  The tests use blkdebug to test error reporting
for both streaming and mirroring.

This still does not include a persistent dirty bitmap, which will be work
for 1.3.

If you want to tinker with this, everything is available at
git://github.com/bonzini/qemu.git in branch blkmirror-job-1.2.

I know it's a lot of code, I'm sorry for dropping this quite close to
the feature freeze.  Unfortunately, preparing for the Linux merge window
and other non-QEMU tasks have dragged this 1-2 weeks more than I would
have liked.

The patches are organized as follows:

01-12   preparatory work for block job errors, including support for
        pausing and resuming jobs

13-17   introduce block job errors, and add support in block-stream

18-26   preparatory work for block mirroring, including creating new
        new functions out of existing code.

27-34   introduce a simple version of mirroring.  The initial patch
        add the mirroring logic, followed by the ability to switch to
        the destination of migration, to query the target file (for
        example, polling the high-water mark), and to handle errors
        during the job.  All these changes come with testcases.

35-43   These patches introduce the first optimizations, namely supporting
        an arbitrary granularity for the dirty bitmap.  The current default,
        1M, is too coarse to let the job converge quickly and in almost
        real-time.  These patches reimplement the block device dirty bitmap
        to allow efficient iteration, and add cluster copy-on-write logic.
        Cluster copy-on-write is needed because management will want to
        start the copy before the backing file is in place in the destination;
        if mirroring takes care of copy-on-write, BDRV_O_NO_BACKING can be
        used even if the granularity is smaller than the cluster size.

44-47   A second round optimizations, replacing serialized read-write
        operations with multiple asynchronous I/O operations.  The various
        in-flight operations can be of arbitrary size.  The initial copy
        will end up reading large chunks sequentially (10M by default),
        while subsequent passes can mimic more closely the guest's I/O
        patterns.

Compared to v1, the last four patches are entirely new, and so are many
of the testcase changes.  All comments from Eric's review are addressed.
In some cases the patches were modified (reversing if conditions or things
like that) in order to keep later patches simpler.  I also added several
new tracepoints.

Latency is vital to any migration scheme using a dirty bitmap, especially
because completion is entirely asynchronous, so I expect this to be used
either with pretty good storage, or on guests doing relatively little I/O.
I tested this both on my laptop and with moderately high-end SAS disks.

On the SAS disks, time between checkpoints (trace_mirror_before_flush)
on kernel compilation (-j3 to -j12, 4 or 8 vCPUs) is almost always within
1 second, usually much less targeting a local disk.  On hibernation,
which is a worst-case test (sequential I/O happening with no flushes
in between) and failed completely to converge on my lowly laptop hard
disk, a checkpoint was reached every 0.5 to 3 seconds.  When targeting
a local qemu-nbd server performance was similar.  Kernel compilation
showed occasional bumps, but they were fixed in 1.5-7 seconds.

Please review!

Paolo Bonzini (47):
  qapi: generalize documentation of streaming commands
  qerror/block: introduce QERR_BLOCK_JOB_NOT_ACTIVE
  block: move job APIs to separate files
  block: add block_job_query
  block: add support for job pause/resume
  qmp: add block-job-pause and block-job-resume
  qemu-iotests: add test for pausing a streaming operation
  block: rename block_job_complete to block_job_completed
  block: rename BlockErrorAction, BlockQMPEventAction
  block: move BlockdevOnError declaration to QAPI
  block: reorganize io error code
  block: sort BlockDeviceIoStatus errors by severity
  block: introduce block job error
  stream: add on-error argument
  blkdebug: process all set_state rules in the old state
  qemu-iotests: map underscore to dash in QMP argument names
  qemu-iotests: add tests for streaming error handling
  block: live snapshot documentation tweaks
  block: add bdrv_query_info
  block: add bdrv_query_stats
  block: add bdrv_ensure_backing_file
  block: make device optional in BlockInfo
  block: add target info to QMP query-blockjobs command
  block: introduce new dirty bitmap functionality
  block: add block-job-complete
  block: introduce BLOCK_JOB_READY event
  block: introduce mirror job
  qmp: add drive-mirror command
  mirror: support querying target file
  mirror: implement completion
  qemu-iotests: add mirroring test case
  block: forward bdrv_iostatus_reset to block job
  mirror: add support for on-source-error/on-target-error
  qmp: add pull_event function
  qemu-iotests: add testcases for mirroring
    on-source-error/on-target-error
  host-utils: add ffsl and flsl
  add hierarchical bitmap data type and test cases
  block: implement dirty bitmap using HBitmap
  block: make round_to_clusters public
  mirror: perform COW if the cluster size is bigger than the
    granularity
  block: return count of dirty sectors, not chunks
  block: allow customizing the granularity of the dirty bitmap
  mirror: allow customizing the granularity
  mirror: switch mirror_iteration to AIO
  mirror: add buf-size argument to drive-mirror
  mirror: support more than one in-flight AIO operation
  mirror: support arbitrarily-sized iterations

 Makefile.objs                 |    5 +-
 QMP/qmp-events.txt            |   43 +++
 QMP/qmp.py                    |   20 ++
 block-migration.c             |    8 +-
 block.c                       |  486 ++++++++++++------------------
 block.h                       |   37 ++-
 block/Makefile.objs           |    3 +-
 block/blkdebug.c              |   14 +-
 block/mirror.c                |  562 +++++++++++++++++++++++++++++++++++
 block/stream.c                |   33 +-
 block_int.h                   |  192 +++---------
 blockdev.c                    |  257 +++++++++++++---
 blockjob.c                    |  290 ++++++++++++++++++
 blockjob.h                    |  285 ++++++++++++++++++
 hbitmap.c                     |  394 ++++++++++++++++++++++++
 hbitmap.h                     |   51 ++++
 hmp-commands.hx               |   73 ++++-
 hmp.c                         |   65 +++-
 hmp.h                         |    4 +
 host-utils.h                  |   45 +++
 hw/fdc.c                      |    4 +-
 hw/ide/core.c                 |   20 +-
 hw/scsi-disk.c                |   23 +-
 hw/scsi-generic.c             |    4 +-
 hw/virtio-blk.c               |   19 +-
 monitor.c                     |    2 +
 monitor.h                     |    2 +
 qapi-schema.json              |  238 +++++++++++++--
 qemu-tool.c                   |    6 +
 qerror.c                      |   12 +
 qerror.h                      |    9 +
 qmp-commands.hx               |   72 ++++-
 tests/Makefile                |    2 +
 tests/qemu-iotests/030        |  178 ++++++++++-
 tests/qemu-iotests/039        |  661 +++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/group      |    3 +-
 tests/qemu-iotests/iotests.py |   19 +-
 tests/test-hbitmap.c          |  384 ++++++++++++++++++++++++
 trace-events                  |   24 +-
 39 files changed, 3946 insertions(+), 603 deletions(-)
 create mode 100644 block/mirror.c
 create mode 100644 blockjob.c
 create mode 100644 blockjob.h
 create mode 100644 hbitmap.c
 create mode 100644 hbitmap.h
 create mode 100755 tests/qemu-iotests/039
 create mode 100644 tests/test-hbitmap.c

-- 
1.7.10.4