[Qemu-devel] [PATCH 36/40] aio: update locking documentation

[Paolo Bonzini <pbonzini@redhat.com>]

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 docs/multiple-iothreads.txt | 65 +++++++++++++++++++++++++++++++++++++++------
 1 file changed, 57 insertions(+), 8 deletions(-)

diff --git a/docs/multiple-iothreads.txt b/docs/multiple-iothreads.txt
index 4197f62..729c64b 100644
--- a/docs/multiple-iothreads.txt
+++ b/docs/multiple-iothreads.txt
@@ -47,8 +47,11 @@ How to program for IOThreads
 The main difference between legacy code and new code that can run in an
 IOThread is dealing explicitly with the event loop object, AioContext
 (see include/block/aio.h).  Code that only works in the main loop
-implicitly uses the main loop's AioContext.  Code that supports running
-in IOThreads must be aware of its AioContext.
+implicitly uses the main loop's AioContext, and does not need to handle
+locking because it implicitly uses the QEMU global mutex.
+
+Code that supports running in IOThreads, instead, must be aware of its
+AioContext and of synchronization.
 
 AioContext supports the following services:
  * File descriptor monitoring (read/write/error on POSIX hosts)
@@ -79,6 +82,11 @@ iothread_get_aio_context() or for the main loop using qemu_get_aio_context().
 Code that takes an AioContext argument works both in IOThreads or the main
 loop, depending on which AioContext instance the caller passes in.
 
+When running in an IOThread, handlers for bottom halves, file descriptors,
+and timers have to handle their own synchronization.  Block layer AIO
+callbacks also have to handle synchronization.  How to do this is the
+topic of the next sections.
+
 How to synchronize with an IOThread
 -----------------------------------
 AioContext is not thread-safe so some rules must be followed when using file
@@ -87,13 +95,15 @@ descriptors, event notifiers, timers, or BHs across threads:
 1. AioContext functions can always be called safely.  They handle their
 own locking internally.
 
-2. Other threads wishing to access the AioContext must use
-aio_context_acquire()/aio_context_release() for mutual exclusion.  Once the
-context is acquired no other thread can access it or run event loop iterations
-in this AioContext.
+2. Other threads wishing to access block devices on an IOThread's AioContext
+must use a mutex; the block layer does not attempt to provide mutual
+exclusion of any kind.  aio_context_acquire() and aio_context_release()
+are the typical choice; aio_context_acquire()/aio_context_release() calls
+may be nested.
 
-aio_context_acquire()/aio_context_release() calls may be nested.  This
-means you can call them if you're not sure whether #1 applies.
+#2 typically includes bottom half handlers, file descriptor handlers,
+timer handlers and AIO callbacks.  All of these typically acquire and
+release the AioContext.
 
 There is currently no lock ordering rule if a thread needs to acquire multiple
 AioContexts simultaneously.  Therefore, it is only safe for code holding the
@@ -130,9 +140,48 @@ aio_disable_clients() before calling bdrv_drain().  You can then reenable
 guest requests with aio_enable_clients() before finally releasing the
 AioContext and completing the monitor command.
 
+AioContext and coroutines
+-------------------------
 Long-running jobs (usually in the form of coroutines) are best scheduled in the
 BlockDriverState's AioContext to avoid the need to acquire/release around each
 bdrv_*() call.  Be aware that there is currently no mechanism to get notified
 when bdrv_set_aio_context() moves this BlockDriverState to a different
 AioContext (see bdrv_detach_aio_context()/bdrv_attach_aio_context()), so you
 may need to add this if you want to support long-running jobs.
+
+Usually, the coroutine is created and first entered in the main loop.  After
+some time it will yield and, when entered again, it will run in the
+IOThread.  In general, however, the coroutine never has to worry about
+releasing and acquiring the AioContext.  The release will happen in the
+function that entered the coroutine; the acquire will happen (e.g. in
+a bottom half or AIO callback) before the coroutine is entered.  For
+example:
+
+1) the coroutine yields because it waits on I/O (e.g. during a call to
+bdrv_co_readv) or sleeps for a determinate period of time (e.g. with
+co_aio_sleep_ns).  In this case, functions such as bdrv_co_io_em_complete
+or co_sleep_cb take the AioContext lock before re-entering.
+
+2) the coroutine explicitly yields.  In this case, the code implementing
+the long-running job will have a bottom half, AIO callback or similar that
+should acquire the AioContext before re-entering the coroutine.
+
+3) the coroutine yields through a CoQueue (directly or indirectly, e.g.
+through a CoMutex or CoRwLock).  This also "just works", but the mechanism
+is more subtle.  Suppose coroutine C1 has just released a mutex and C2
+was waiting on it.  C2 is woken up at C1's next yield point, or just
+before C1 terminates; C2 then runs *before qemu_coroutine_enter(C1)
+returns to its caller*.  Assuming that C1 was entered with a sequence like
+
+    aio_context_acquire(ctx);
+    qemu_coroutine_enter(co);     // enters C1
+    aio_context_release(ctx);
+
+... then C2 will also be protected by the AioContext lock.
+
+
+Bullet 3 provides the following rule for using coroutine mutual exclusion
+in multiple threads:
+
+    *** If two coroutines can exclude each other through a CoMutex or
+    *** CoRwLock or CoQueue, they should all run under the same AioContext.
-- 
1.8.3.1