From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:49114)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <paolo.bonzini@gmail.com>) id 1WrAFq-0001LG-7Q
	for qemu-devel@nongnu.org; Sun, 01 Jun 2014 14:13:19 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <paolo.bonzini@gmail.com>) id 1WrAFh-0000aH-6w
	for qemu-devel@nongnu.org; Sun, 01 Jun 2014 14:13:10 -0400
Received: from mail-we0-x230.google.com ([2a00:1450:400c:c03::230]:61290)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <paolo.bonzini@gmail.com>) id 1WrAFg-0000Zw-Vo
	for qemu-devel@nongnu.org; Sun, 01 Jun 2014 14:13:01 -0400
Received: by mail-we0-f176.google.com with SMTP id q59so4196198wes.35
	for <qemu-devel@nongnu.org>; Sun, 01 Jun 2014 11:12:59 -0700 (PDT)
Sender: Paolo Bonzini <paolo.bonzini@gmail.com>
Message-ID: <538B6D25.8030106@redhat.com>
Date: Sun, 01 Jun 2014 20:12:53 +0200
From: Paolo Bonzini <pbonzini@redhat.com>
MIME-Version: 1.0
References: <538A1F71.4080203@beyond.pl>
In-Reply-To: <538A1F71.4080203@beyond.pl>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Subject: Re: [Qemu-devel] [PATCH] thread-pool: fix deadlock when callbacks
 depends on each other
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: =?UTF-8?B?TWFyY2luIEdpYnXFgmE=?= <m.gibula@beyond.pl>, "qemu-devel@nongnu.org" <qemu-devel@nongnu.org>
Cc: Stefan Hajnoczi <stefanha@gmail.com>

Il 31/05/2014 20:29, Marcin Gibuła ha scritto:
>
> @@ -185,6 +191,14 @@ restart:
>          }
>          if (elem->state == THREAD_DONE && elem->common.cb) {
>              QLIST_REMOVE(elem, all);
> +            /* If more completed requests are waiting, notifier needs
> +             * to be rearmed so callback can progress with aio_pool().
> +             */
> +            pool->pending_completions--;
> +            if (pool->pending_completions) {
> +                event_notifier_set(notifier);
> +            }
> +
>              /* Read state before ret.  */
>              smp_rmb();
>              elem->common.cb(elem->common.opaque, elem->ret);

Good catch!  The main problem with the patch is that you need to use 
atomic_inc/atomic_dec to increment and decrement pool->pending_completions.

Secondarily, event_notifier_set is pretty heavy-weight, does it work if 
you wrap the loop like this?

restart:
     QLIST_FOREACH_SAFE(elem, &pool->head, all, next) {
         ...
     }
     if (pool->pending_completions) {
         goto restart;
     }
     event_notifier_test_and_clear(notifier);
     if (pool->pending_completions) {
         event_notifier_set(notifier);
         goto restart;
     }

Finally, the same bug is also in block/linux-aio.c and block/win32-aio.c.

Paolo