Re: [Qemu-devel] [PATCH] util/async: use atomic_mb_set in qemu_bh_cancel

[Paolo Bonzini <pbonzini@redhat.com>]

On 07/11/2017 16:09, Sergio Lopez wrote:
> Commit b7a745d added a qemu_bh_cancel call to the completion function
> as an optimization to prevent it from unnecessarily rescheduling itself.
> 
> This completion function is scheduled from worker_thread, after setting
> the state of a ThreadPoolElement to THREAD_DONE.
> 
> This was considered to be safe, as the completion function restarts the
> loop just after the call to qemu_bh_cancel. But, under certain access
> patterns and scheduling conditions, the loop may wrongly use a
> pre-fetched elem->state value, reading it as THREAD_QUEUED, and ending
> the completion function without having processed a pending TPE linked at
> pool->head.

The commit message probably should include an example of the scenario:


    I/O thread                             worker thread
 --------------------------------------------------------------------------------
                                           speculatively read req->state
    req->state = THREAD_DONE;
    qemu_bh_schedule(pool->completion_bh)
       bh->scheduled = 1;
                                           qemu_bh_cancel(pool->completion_bh)
                                             bh->scheduled = 0;
                                           if (req->state == THREAD_DONE)
                                             // sees THREAD_QUEUED

The source of the misunderstanding was that qemu_bh_cancel is now being
used by the _consumer_ rather than the producer, and therefore now needs
to have acquire semantics just like e.g. aio_bh_poll.

Paolo

> In some situations, if there are no other independent requests in the
> same aio context that could eventually trigger the scheduling of the
> completion function, the omitted TPE and all operations pending on it
> will get stuck forever.
> 
> Signed-off-by: Sergio Lopez <slp@redhat.com>
> ---
>  util/async.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/util/async.c b/util/async.c
> index 355af73ee7..0e1bd8780a 100644
> --- a/util/async.c
> +++ b/util/async.c
> @@ -174,7 +174,7 @@ void qemu_bh_schedule(QEMUBH *bh)
>   */
>  void qemu_bh_cancel(QEMUBH *bh)
>  {
> -    bh->scheduled = 0;
> +    atomic_mb_set(&bh->scheduled, 0);
>  }
>  
>  /* This func is async.The bottom half will do the delete action at the finial
>