Re: [Qemu-devel] [PATCH 00/22] dataplane: use QEMU block layer

[Christian Borntraeger <borntraeger@de.ibm.com>]

On 01/05/14 16:54, Stefan Hajnoczi wrote:
> This patch series switches virtio-blk data-plane from a custom Linux AIO
> request queue to the QEMU block layer.  The previous "raw files only"
> limitation is lifted.  All image formats and protocols can now be used with
> virtio-blk data-plane.

Nice. Is there a git branch somewhere, so that we can test this on s390?

Christian
> 
> How to review this series
> -------------------------
> I CCed the maintainer of each block driver that I modified.  You probably don't
> need to review the entire series, just your patch.
> 
> From now on fd handlers, timers, BHs, and event loop wait must explicitly use
> BlockDriverState's AioContext instead of the main loop.  Use
> bdrv_get_aio_context(bs) to get the AioContext.  The following function calls
> need to be converted:
> 
>  * qemu_aio_set_fd_handler() -> aio_set_fd_handler()
>  * timer_new*() -> aio_timer_new()
>  * qemu_bh_new() -> aio_bh_new()
>  * qemu_aio_wait() -> aio_poll(aio_context, true)
> 
> For simple block drivers this modification suffices and it is now safe to use
> outside the QEMU global mutex.
> 
> Block drivers that keep fd handlers, timers, or BHs registered when requests
> have been drained need a little bit more work.  Examples of this are network
> block drivers with keepalive timers, like iSCSI.
> 
> This series adds a new bdrv_set_aio_context(bs, aio_context) function that
> moves a BlockDriverState into a new AioContext.  This function calls the block
> driver's optional .bdrv_detach_aio_context() and .bdrv_attach_aio_context()
> functions.  Implement detach/attach to move the fd handlers, timers, or BHs to
> the new AioContext.
> 
> Finally, block drivers that manage their own child nodes also need to
> implement detach/attach because the generic block layer doesn't know about
> their children.  Both ->file and ->backing_hd are automatically taken care of
> but blkverify, quorum, and VMDK need to manually propagate detach/attach to
> their children.
> 
> I have audited and modified all block drivers.  Block driver maintainers,
> please check I did it correctly and didn't break your code.
> 
> Background
> ----------
> The block layer is currently tied to the QEMU main loop for fd handlers, timer
> callbacks, and BHs.  This means that even on hosts with many cores, parts of
> block I/O processing happen in one thread and depend on the QEMU global mutex.
> 
> virtio-blk data-plane has shown that 1,000,000 IOPS is achievable if we use
> additional threads that are not under the QEMU global mutex.
> 
> It is necessary to make the QEMU block layer aware that there may be more than
> one event loop.  This way BlockDriverState can be used from a thread without
> contention on the QEMU global mutex.
> 
> This series builds on the aio_context_acquire/release() interface that allows a
> thread to temporarily grab an AioContext.  We add bdrv_set_aio_context(bs,
> aio_context) for changing which AioContext a BlockDriverState uses.
> 
> The final patches convert virtio-blk data-plane to use the QEMU block layer and
> let the BlockDriverState run in the IOThread AioContext.
> 
> What's next?
> ------------
> I have already made block I/O throttling work in another AioContext and will
> send the series out next week.
> 
> In order to keep this series reviewable, I'm holding back those patches for
> now.  One could say, "throttling" them.
> 
> Thank you, thank you, I'll be here all night!
> 
> Stefan Hajnoczi (22):
>   block: use BlockDriverState AioContext
>   block: acquire AioContext in bdrv_close_all()
>   block: add bdrv_set_aio_context()
>   blkdebug: use BlockDriverState's AioContext
>   blkverify: implement .bdrv_detach/attach_aio_context()
>   curl: implement .bdrv_detach/attach_aio_context()
>   gluster: use BlockDriverState's AioContext
>   iscsi: implement .bdrv_detach/attach_aio_context()
>   nbd: implement .bdrv_detach/attach_aio_context()
>   nfs: implement .bdrv_detach/attach_aio_context()
>   qed: use BlockDriverState's AioContext
>   quorum: implement .bdrv_detach/attach_aio_context()
>   block/raw-posix: implement .bdrv_detach/attach_aio_context()
>   block/linux-aio: fix memory and fd leak
>   rbd: use BlockDriverState's AioContext
>   sheepdog: implement .bdrv_detach/attach_aio_context()
>   ssh: use BlockDriverState's AioContext
>   vmdk: implement .bdrv_detach/attach_aio_context()
>   dataplane: use the QEMU block layer for I/O
>   dataplane: delete IOQueue since it is no longer used
>   dataplane: implement async flush
>   raw-posix: drop raw_get_aio_fd() since it is no longer used
> 
>  block.c                          |  88 +++++++++++++--
>  block/blkdebug.c                 |   2 +-
>  block/blkverify.c                |  47 +++++---
>  block/curl.c                     | 194 +++++++++++++++++++-------------
>  block/gluster.c                  |   7 +-
>  block/iscsi.c                    |  79 +++++++++----
>  block/linux-aio.c                |  24 +++-
>  block/nbd-client.c               |  24 +++-
>  block/nbd-client.h               |   4 +
>  block/nbd.c                      |  87 +++++++++------
>  block/nfs.c                      |  80 ++++++++++----
>  block/qed-table.c                |   8 +-
>  block/qed.c                      |  35 +++++-
>  block/quorum.c                   |  48 ++++++--
>  block/raw-aio.h                  |   3 +
>  block/raw-posix.c                |  82 ++++++++------
>  block/rbd.c                      |   5 +-
>  block/sheepdog.c                 | 118 +++++++++++++-------
>  block/ssh.c                      |  36 +++---
>  block/vmdk.c                     |  23 ++++
>  hw/block/dataplane/Makefile.objs |   2 +-
>  hw/block/dataplane/ioq.c         | 117 --------------------
>  hw/block/dataplane/ioq.h         |  57 ----------
>  hw/block/dataplane/virtio-blk.c  | 233 +++++++++++++++------------------------
>  include/block/block.h            |  20 ++--
>  include/block/block_int.h        |  36 ++++++
>  26 files changed, 829 insertions(+), 630 deletions(-)
>  delete mode 100644 hw/block/dataplane/ioq.c
>  delete mode 100644 hw/block/dataplane/ioq.h
>