From mboxrd@z Thu Jan 1 00:00:00 1970 From: Antoine Martin Subject: Re: repeatable hang with loop mount and heavy IO in guest [NOT SOLVED] Date: Sun, 24 Jan 2010 18:23:33 +0700 Message-ID: <4B5C2DB5.6090607@nagafix.co.uk> References: <4B588E63.4060700@nagafix.co.uk> <4B595A7B.4010404@msgid.tls.msk.ru> <4B59EE4E.6070005@nagafix.co.uk> <4B59F958.8030709@nagafix.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org To: Michael Tokarev Return-path: Received: from mamba.nagafix.co.uk ([194.145.196.68]:60099 "EHLO mail.nagafix.co.uk" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753509Ab0AXLXk (ORCPT ); Sun, 24 Jan 2010 06:23:40 -0500 In-Reply-To: <4B59F958.8030709@nagafix.co.uk> Sender: kvm-owner@vger.kernel.org List-ID: On 01/23/2010 02:15 AM, Antoine Martin wrote: > On 01/23/2010 01:28 AM, Antoine Martin wrote: >> On 01/22/2010 02:57 PM, Michael Tokarev wrote: >>> Antoine Martin wrote: >>>> I've tried various guests, including most recent Fedora12 kernels, >>>> custom 2.6.32.x >>>> All of them hang around the same point (~1GB written) when I do >>>> heavy IO >>>> write inside the guest. >>> [] >>>> Host is running: 2.6.31.4 >>>> QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88) >>> Please update to last version and repeat. kvm-88 is ancient and >>> _lots_ of stuff fixed and changed since that time, I doubt anyone >>> here will try to dig into kvm-88 problems. >>> >>> Current kvm is qemu-kvm-0.12.2, released yesterday. >> Sorry about that, I didn't realize 88 was so far behind. >> Upgrading to qemu-kvm-0.12.2 did solve my IO problems. > Only for a while. Same problem just re-occurred, only this time it > went a little further. > It is now just sitting there, with a load average of exactly 3.0 (+- 5%) > > Here is a good trace of the symptom during writeback, you can see it > write the data at around 50MB/s, it goes from being idle to sys, but > after a while it just stops writing and goes into mostly wait state: [snip] > From that point onwards, nothing will happen. > The host has disk IO to spare... So what is it waiting for?? Note: if I fill the disk in the guest with zeroes but without going via a loop mounted filesystem, then everything works just fine. Something in using the loopback makes it fall over. Here is the simplest way to make this happen: time dd if=/dev/zero of=./test bs=1048576 count=2048 2147483648 bytes (2.1 GB) copied, 65.1344 s, 33.0 MB/s mkfs.ext3 ./test; mkdir tmp mount -o loop ./test ./tmp time dd if=/dev/zero of=./tmp/test-loop bs=1048576 count=2048 ^this one will never return and you can't just kill "dd", it's stuck. The whole guest has to be killed at this point. > > QEMU PC emulator version 0.12.2 (qemu-kvm-0.12.2), Copyright (c) > 2003-2008 Fabrice Bellard > Guests: various, all recent kernels. > Host: 2.6.31.4 Before anyone suggests this, I have tried with/without elevator=noop, with/without virtio disks. No effect, still hangs. Antoine > Please advise. > > Thanks > Antoine >