[Qemu-devel] [PATCH v2] vl: pause vcpus before stopping iothreads

[Qemu-devel] [PATCH v2] vl: pause vcpus before stopping iothreads

[Stefan Hajnoczi]

Commit dce8921b2baaf95974af8176406881872067adfa ("iothread: Stop threads
before main() quits") introduced iothread_stop_all() to avoid the
following virtio-scsi assertion failure:

  assert(blk_get_aio_context(d->conf.blk) == s->ctx);

Back then the assertion failed because when bdrv_close_all() made
d->conf.blk NULL, blk_get_aio_context() returned the global AioContext
instead of s->ctx.

The same assertion can still fail today when vcpus submit new I/O
requests after iothread_stop_all() has moved the BDS to the global
AioContext.

This patch hardens the iothread_stop_all() approach by pausing vcpus
before calling iothread_stop_all().

Note that the assertion failure is a race condition.  It is not possible
to reproduce it reliably.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
v2:
 * Add comment explaining rationale for this ordering and why it's safe.
   [Kevin]

 vl.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/vl.c b/vl.c
index e517a8d995..b72a9fc6df 100644
--- a/vl.c
+++ b/vl.c
@@ -4766,10 +4766,18 @@ int main(int argc, char **argv, char **envp)
 
     main_loop();
     replay_disable_events();
+
+    /* The ordering of the following is delicate.  Stop vcpus to prevent new
+     * I/O requests being queued by the guest.  Then stop IOThreads (this
+     * includes a drain operation and completes all request processing).  At
+     * this point emulated devices are still associated with their IOThreads
+     * (if any) but no longer have any work to do.  Only then can we close
+     * block devices safely because we know there is no more I/O coming.
+     */
+    pause_all_vcpus();
     iothread_stop_all();
-
-    pause_all_vcpus();
     bdrv_close_all();
+
     res_free();
 
     /* vhost-user must be cleaned up before chardevs.  */
-- 
2.14.3

Re: [Qemu-devel] [PATCH v2] vl: pause vcpus before stopping iothreads

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1210 bytes --]

On Thu, Feb 01, 2018 at 11:07:08AM +0000, Stefan Hajnoczi wrote:
> Commit dce8921b2baaf95974af8176406881872067adfa ("iothread: Stop threads
> before main() quits") introduced iothread_stop_all() to avoid the
> following virtio-scsi assertion failure:
> 
>   assert(blk_get_aio_context(d->conf.blk) == s->ctx);
> 
> Back then the assertion failed because when bdrv_close_all() made
> d->conf.blk NULL, blk_get_aio_context() returned the global AioContext
> instead of s->ctx.
> 
> The same assertion can still fail today when vcpus submit new I/O
> requests after iothread_stop_all() has moved the BDS to the global
> AioContext.
> 
> This patch hardens the iothread_stop_all() approach by pausing vcpus
> before calling iothread_stop_all().
> 
> Note that the assertion failure is a race condition.  It is not possible
> to reproduce it reliably.
> 
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> v2:
>  * Add comment explaining rationale for this ordering and why it's safe.
>    [Kevin]
> 
>  vl.c | 12 ++++++++++--
>  1 file changed, 10 insertions(+), 2 deletions(-)

Thanks, applied to my block tree:
https://github.com/stefanha/qemu/commits/block

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]