Re: repeatable hang with loop mount and heavy IO in guest [NOT SOLVED]

[Antoine Martin <antoine@nagafix.co.uk>]

On 01/23/2010 02:15 AM, Antoine Martin wrote:
> On 01/23/2010 01:28 AM, Antoine Martin wrote:
>> On 01/22/2010 02:57 PM, Michael Tokarev wrote:
>>> Antoine Martin wrote:
>>>> I've tried various guests, including most recent Fedora12 kernels,
>>>> custom 2.6.32.x
>>>> All of them hang around the same point (~1GB written) when I do 
>>>> heavy IO
>>>> write inside the guest.
>>> []
>>>> Host is running: 2.6.31.4
>>>> QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88)
>>> Please update to last version and repeat.  kvm-88 is ancient and
>>> _lots_ of stuff fixed and changed since that time, I doubt anyone
>>> here will try to dig into kvm-88 problems.
>>>
>>> Current kvm is qemu-kvm-0.12.2, released yesterday.
>> Sorry about that, I didn't realize 88 was so far behind.
>> Upgrading to qemu-kvm-0.12.2 did solve my IO problems.
> Only for a while. Same problem just re-occurred, only this time it 
> went a little further.
> It is now just sitting there, with a load average of exactly 3.0 (+- 5%)
>
> Here is a good trace of the symptom during writeback, you can see it 
> write the data at around 50MB/s, it goes from being idle to sys, but 
> after a while it just stops writing and goes into mostly wait state:
[snip]
> From that point onwards, nothing will happen.
> The host has disk IO to spare... So what is it waiting for??
Note: if I fill the disk in the guest with zeroes but without going via 
a loop mounted filesystem, then everything works just fine. Something in 
using the loopback makes it fall over.

Here is the simplest way to make this happen:
time dd if=/dev/zero of=./test bs=1048576 count=2048
2147483648 bytes (2.1 GB) copied, 65.1344 s, 33.0 MB/s

mkfs.ext3 ./test; mkdir tmp
mount -o loop ./test ./tmp
time dd if=/dev/zero of=./tmp/test-loop bs=1048576 count=2048
^this one will never return and you can't just kill "dd", it's stuck.
The whole guest has to be killed at this point.
>
> QEMU PC emulator version 0.12.2 (qemu-kvm-0.12.2), Copyright (c) 
> 2003-2008 Fabrice Bellard
> Guests: various, all recent kernels.
> Host: 2.6.31.4
Before anyone suggests this, I have tried with/without elevator=noop, 
with/without virtio disks.
No effect, still hangs.

Antoine

> Please advise.
>
> Thanks
> Antoine
>