From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:56427)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <pbonzini@redhat.com>) id 1YdiDg-0003L6-Sf
	for qemu-devel@nongnu.org; Thu, 02 Apr 2015 12:43:53 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <pbonzini@redhat.com>) id 1YdiDd-0006J1-Id
	for qemu-devel@nongnu.org; Thu, 02 Apr 2015 12:43:52 -0400
Received: from mx1.redhat.com ([209.132.183.28]:54513)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <pbonzini@redhat.com>) id 1YdiDd-0006Iv-Cv
	for qemu-devel@nongnu.org; Thu, 02 Apr 2015 12:43:49 -0400
Message-ID: <551D71BF.6050601@redhat.com>
Date: Thu, 02 Apr 2015 18:43:43 +0200
From: Paolo Bonzini <pbonzini@redhat.com>
MIME-Version: 1.0
References: <CAJSP0QU3XK82fpKjkO0T=6eM+_Ow0vAjX2nJ1hRF6ora4VzTpw@mail.gmail.com>
In-Reply-To: <CAJSP0QU3XK82fpKjkO0T=6eM+_Ow0vAjX2nJ1hRF6ora4VzTpw@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Subject: Re: [Qemu-devel] thread-pool.c race condition?
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Stefan Hajnoczi <stefanha@gmail.com>
Cc: Laszlo Ersek <lersek@redhat.com>, John Snow <jsnow@redhat.com>, qemu-devel <qemu-devel@nongnu.org>



On 02/04/2015 18:26, Stefan Hajnoczi wrote:
> John Snow has reported that qemu-io can hang when the host is under
> heavy load.  He made the following observations in gdb:
> 
> 1. The program is sitting in aio_poll() (called by bdrv_prwv_co())
> waiting for request completion.
> 
> 2. The thread pool has a ThreadPoolElement with ->state == THREAD_DONE.
> 
> The ThreadPoolElement should have been reaped by
> thread_pool_completion_bh() and its callback invoked.  For some reason
> this didn't happen and the program is blocked in poll(2) waiting.
> 
> This suggests a race condition in thread-pool.c or qemu_bh_schedule()
> (used to complete ThreadPoolElement from a QEMU event loop).
> 
> I don't have a good theory why this happens yet.  Just wanted to share
> in case someone else hits this problem.

Laszlo hit something very similar fairly easily with virtio-scsi (but
not virtio-blk!) on aarch64 hosts.  Any attempt to debug it (ranging
from compilation with -O0 to tracing) made it disappear.  A reliable
reproducer with qemu-io would be a dream...

Paolo