From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:48133)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <stefanha@gmail.com>) id 1VbSPz-0000ZL-UF
	for qemu-devel@nongnu.org; Wed, 30 Oct 2013 05:50:33 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <stefanha@gmail.com>) id 1VbSPu-0003X5-OL
	for qemu-devel@nongnu.org; Wed, 30 Oct 2013 05:50:27 -0400
Received: from mail-ea0-x22d.google.com ([2a00:1450:4013:c01::22d]:46474)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <stefanha@gmail.com>) id 1VbSPu-0003Wp-AH
	for qemu-devel@nongnu.org; Wed, 30 Oct 2013 05:50:22 -0400
Received: by mail-ea0-f173.google.com with SMTP id g10so515098eak.18
	for <qemu-devel@nongnu.org>; Wed, 30 Oct 2013 02:50:21 -0700 (PDT)
Date: Wed, 30 Oct 2013 10:50:18 +0100
From: Stefan Hajnoczi <stefanha@gmail.com>
Message-ID: <20131030095018.GC11994@stefanha-thinkpad.redhat.com>
References: <526A87E2.9020705@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <526A87E2.9020705@gmail.com>
Subject: Re: [Qemu-devel] [RFC] block io lost in the guest ,
 possible related to qemu?
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Jack Wang <xjtuwjp@gmail.com>
Cc: Kevin Wolf <kwolf@redhat.com>, qemu-devel <qemu-devel@nongnu.org>, Stefan Hajnoczi <stefanha@redhat.com>, Alexey Zaytsev <alexey.zaytsev@gmail.com>

On Fri, Oct 25, 2013 at 05:01:54PM +0200, Jack Wang wrote:
> We've seen guest block io lost in a VM.any response will be helpful
> 
> environment is:
> guest os: Ubuntu 1304
> running busy database workload with xfs on a disk export with virtio-blk
> 
> the exported vdb has very high infight io over 300. Some times later a
> lot io process in D state, looks a lot requests is lost in below storage
> stack.

Is the image file on a local file system or are you using a network
storage system (e.g. NFS, Gluster, Ceph, Sheepdog)?

If you run "vmstat 5" inside the guest, do you see "bi"/"bo" block I/O
activity?  If that number is very low or zero then there may be a
starvation problem.  If that number is reasonable then the workload is
simply bottlenecked on disk I/O.

virtio-blk only has 128 descriptors available so it's not possible to
have 300 requests pending at the virtio-blk layer.

If you suspect QEMU, try building qemu.git/master from source in case
the bug has already been fixed.

If you want to trace I/O requests, you might find this blog post on
writing trace analysis scripts useful:
http://blog.vmsplice.net/2011/03/how-to-write-trace-analysis-scripts-for.html

Stefan