[Qemu-devel] [RFC PATCH 00/18] Fine-grained AioContext critical sections

[Paolo Bonzini <pbonzini@redhat.com>]

This series represents a major enabler for two AioContext projects:
true multiqueue (i.e. multiple AioContexts for the same BDS) and
thread-safe virtio-scsi dataplane.

The patches decouple AioContext's internal locking for the BDS's locking
in two steps.  Patches 1 to 9 make aio_poll and the iothread run
without taking any external locks---only user callbacks are protected
by aio_context_acquire/aio_context_release.  Patches 12 to 16 then push
the lock down to the callbacks themselves, so that the AioContext lock
becomes nothing more than a convenience.

For multiqueue the next step would be to have fine-grained locks so
that e.g. each dataplane thread can proceed independently.  For
thread-safe virtio-scsi dataplane, most of dma-helpers.c is already
running without taking the AioContext lock, and all that's left is to
ensure that dma-helpers.c code is _always_ run without that lock
taken.  Of course the latter is a much smaller endeavor.

I'm fairly confident that patches 1 to 11 are correct and would like to
get them in before patches 12 to 18.  (This second half is much less
tested, especially I have not tested Win32 nor quorum).  I'm
nevertheless posting the whole thing as RFC and not just for review,
since I want to discuss this and the next steps forward at KVM Forum.

Applies on top of the three SCSI patches I have just sent.

Paolo

Paolo Bonzini (18):
  iothread: release iothread around aio_poll
  aio: rename bh_lock to list_lock
  qemu-thread: introduce QemuLockCnt
  aio: make ctx->list_lock a QemuLockCnt, subsuming ctx->walking_bh
  aio: tweak walking in dispatch phase
  aio-posix: remove walking_handlers, protecting AioHandler list with list_lock
  aio-win32: remove walking_handlers, protecting AioHandler list with list_lock
  aio: document locking
  aio: push aio_context_acquire/release down to dispatching
  async: optimize aio_bh_poll
  qemu-timer: optimize timerlist_run_timers
  block: explicitly acquire aiocontext in callbacks that need it
  block: explicitly acquire aiocontext in bottom halves that need it
  block: explicitly acquire aiocontext in timers that need it
  quorum: use atomics for rewrite_count
  quorum: split quorum_fifo_aio_cb from quorum_aio_cb
  block: explicitly acquire aiocontext in aio callbacks that need it
  aio: update locking documentation

 aio-posix.c                     |  78 +++++-----
 aio-win32.c                     |  97 +++++++------
 async.c                         |  63 ++++----
 block/blkverify.c               |   6 +-
 block/curl.c                    |  43 ++++--
 block/gluster.c                 |   2 +
 block/io.c                      |   7 +
 block/iscsi.c                   |  10 ++
 block/linux-aio.c               |  14 +-
 block/mirror.c                  |  12 +-
 block/nbd-client.c              |  14 +-
 block/nfs.c                     |  10 ++
 block/qed.c                     |   2 +
 block/quorum.c                  |  60 +++++---
 block/sheepdog.c                |  29 ++--
 block/ssh.c                     |  45 +++---
 block/throttle-groups.c         |   2 +
 block/win32-aio.c               |   8 +-
 dma-helpers.c                   |   7 +-
 docs/lockcnt.txt                | 308 ++++++++++++++++++++++++++++++++++++++++
 docs/multiple-iothreads.txt     |  95 ++++++++++---
 hw/block/dataplane/virtio-blk.c |   2 +
 hw/block/virtio-blk.c           |   8 ++
 hw/scsi/scsi-bus.c              |   2 +
 hw/scsi/scsi-disk.c             |  18 +++
 hw/scsi/scsi-generic.c          |  20 ++-
 hw/scsi/virtio-scsi-dataplane.c |   6 +
 include/block/aio.h             |  33 ++---
 include/qemu/thread.h           |  17 +++
 iothread.c                      |  11 +-
 nbd.c                           |   4 +
 qemu-coroutine-sleep.c          |   5 +
 qemu-timer.c                    |   4 +
 tests/test-aio.c                |  19 +--
 thread-pool.c                   |  14 +-
 util/Makefile.objs              |   2 +-
 util/lockcnt.c                  | 123 ++++++++++++++++
 37 files changed, 933 insertions(+), 267 deletions(-)
 create mode 100644 docs/lockcnt.txt
 create mode 100644 util/lockcnt.c

-- 
2.4.3