[Qemu-devel] [PATCH v11 00/19] Support streaming to an intermediate layer

[Alberto Garcia <berto@igalia.com>]

Hi all,

here's version 11 of the series.

See below for the list of changes. The most important one is, I think,
the addition of the 'base-node' parameter to 'block-stream'. I decided
to go for this one rather than writing a new 'blockdev-stream' command
because (a) it's not obvious to me that introducing a new command for
such a small API change is justified in this case, and (b) it didn't
seem to me that there was a very clear agreement on the mailing list,
so I decided to go for the easier alternative.

Anyway I don't have a problem with adding a new command if that's what
most people prefer. If everything else in the series is fine I can
change that in v12.

Berto

Changes:

v11:
- Patches 1-2: Add bdrv_drain_all_{begin,end}() and use these
  functions in bdrv_reopen_multiple() instead of pausing all block
  jobs.
- Patch 3: Don't unblock BLOCK_OP_TYPE_DATAPLANE in
  block_job_add_bdrv()
- Patch 7: In commit_start(), don't call block_job_add_bdrv() twice on
  the active node.
- Patch 10: In qmp_block_stream(), don't check twice for op blockers
  in 'bs'.
- Patch 11: Clarify that live-block-ops.txt is incomplete and it only
  covers the block-stream operation.
- Patches 18-19: Add a 'base-node' parameter to 'block-stream', and
  iotests for it.

v10: https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg01047.html
- Rebase the code.
- Pause block jobs during bdrv_reopen().
- Allow jobs in parallel again, and modify jobs to block all involved
  nodes, not just the root one.
- Add more tests.

v9: https://lists.gnu.org/archive/html/qemu-devel/2016-04/msg00411.html
- Rebase the code
- Block the active layer in order to forbid other block jobs in the
  same chain.
- Split the patch that adds block_job_next() into 4 (new patches 1-4).
- Replace the test that performs several block-stream operations in
  parallel with one that check that they're forbidden.
- Remove patches that have already been merged.

v8: https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg05754.html
- Rebased on top of Stefan's block branch (0a35bce416)
- The loop that pauses the block jobs in bdrv_drain_all() is now split
  in two: one that iterates the list of block jobs to stop them, and
  one that iterates the root bds in order to get the aio contexts.

v7: https://lists.gnu.org/archive/html/qemu-devel/2015-05/msg02580.html
- Rebased against the current master
- Updated bdrv_drain_all() to use the new block_job_next() API.

v6: https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg03046.html
- fix the no-op test following Max's suggestions

v5: https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg03006.html
- Fix a few typos
- Minor documentation updates
- Update test_stream_partial() to test no-ops
- New test case: test_stream_parallel()
- New test case: test_stream_overlapping()

v4: https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg01878.html
- Refactor find_block_job to use the error from bdrv_lookup_bs()
- Don't use QERR_DEVICE_IN_USE in block_job_create() since we can be
  dealing with nodes now.
- Fix @device comment in the BlockJobInfo documentation
- stream_start(): simplify the bdrv_reopen() call and use
  bdrv_get_device_or_node_name() for error messages.
- Use a different variable name for BlockDriverState *i
- Documentation fixes in docs/live-block-ops.txt
- Update iotest 30 since now test_device_not_found() returns
  GenericError
- Fix test case test_stream_partial()
- Add new test case test_stream_intermediate()
- Fix typos

v3: https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg00806.html
- Keep a list of block jobs and make qmp_query_block_jobs() iterate
  over it.

v2: https://lists.gnu.org/archive/html/qemu-devel/2015-03/msg04798.html
- The 'block-stream' command does not have a 'node-name' parameter
  anymore and reuses 'device' for that purpose.
- Block jobs can now be owned by any intermediate node, and not just
  by the ones at the root. query-block-jobs is updated to reflect that
  change.
- The 'device' parameter of all 'block-job-*' commands can now take a
  node name.
- The BlockJobInfo type and all BLOCK_JOB_* events report the node
  name in the 'device' field if the node does not have a device name.
- All intermediate nodes are blocked (and checked for blockers) during
  the streaming operation.

v1: https://lists.gnu.org/archive/html/qemu-devel/2015-02/msg04116.html

backport-diff against v10:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/19:[down] 'block: Add bdrv_drain_all_{begin,end}()'
002/19:[0022] [FC] 'block: Pause all jobs during bdrv_reopen_multiple()'
003/19:[0006] [FC] 'block: Add block_job_add_bdrv()'
004/19:[----] [--] 'block: Use block_job_add_bdrv() in mirror_start_job()'
005/19:[----] [--] 'block: Use block_job_add_bdrv() in backup_start()'
006/19:[----] [--] 'block: Check blockers in all nodes involved in a block-commit job'
007/19:[0014] [FC] 'block: Block all nodes involved in the block-commit operation'
008/19:[----] [--] 'block: Block all intermediate nodes in commit_active_start()'
009/19:[----] [--] 'block: Support streaming to an intermediate layer'
010/19:[0004] [FC] 'block: Add QMP support for streaming to an intermediate layer'
011/19:[0005] [FC] 'docs: Document how to stream to an intermediate layer'
012/19:[----] [--] 'qemu-iotests: Test streaming to an intermediate layer'
013/19:[----] [--] 'qemu-iotests: Test block-stream operations in parallel'
014/19:[----] [--] 'qemu-iotests: Test overlapping stream and commit operations'
015/19:[----] [--] 'qemu-iotests: Test block-stream and block-commit in parallel'
016/19:[----] [--] 'qemu-iotests: Add iotests.supports_quorum()'
017/19:[----] [--] 'qemu-iotests: Test streaming to a Quorum child'
018/19:[down] 'block: Add 'base-node' parameter to the 'block-stream' command'
019/19:[down] 'qemu-iotests: Test the 'base-node' parameter of 'block-stream''

Alberto Garcia (19):
  block: Add bdrv_drain_all_{begin,end}()
  block: Pause all jobs during bdrv_reopen_multiple()
  block: Add block_job_add_bdrv()
  block: Use block_job_add_bdrv() in mirror_start_job()
  block: Use block_job_add_bdrv() in backup_start()
  block: Check blockers in all nodes involved in a block-commit job
  block: Block all nodes involved in the block-commit operation
  block: Block all intermediate nodes in commit_active_start()
  block: Support streaming to an intermediate layer
  block: Add QMP support for streaming to an intermediate layer
  docs: Document how to stream to an intermediate layer
  qemu-iotests: Test streaming to an intermediate layer
  qemu-iotests: Test block-stream operations in parallel
  qemu-iotests: Test overlapping stream and commit operations
  qemu-iotests: Test block-stream and block-commit in parallel
  qemu-iotests: Add iotests.supports_quorum()
  qemu-iotests: Test streaming to a Quorum child
  block: Add 'base-node' parameter to the 'block-stream' command
  qemu-iotests: Test the 'base-node' parameter of 'block-stream'

 block.c                       |   9 +-
 block/backup.c                |   5 +-
 block/commit.c                |  14 ++
 block/io.c                    |  24 +++-
 block/mirror.c                |  11 +-
 block/stream.c                |  24 ++++
 blockdev.c                    |  41 +++++-
 blockjob.c                    |  17 ++-
 docs/live-block-ops.txt       |  36 +++--
 docs/qmp-commands.txt         |   7 +-
 hmp.c                         |   2 +-
 include/block/block.h         |   2 +
 include/block/blockjob.h      |  14 ++
 qapi/block-core.json          |  18 ++-
 tests/qemu-iotests/030        | 309 +++++++++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/030.out    |   4 +-
 tests/qemu-iotests/041        |  27 ++--
 tests/qemu-iotests/139        |   3 +-
 tests/qemu-iotests/iotests.py |   5 +-
 19 files changed, 515 insertions(+), 57 deletions(-)

-- 
2.9.3