From: Antoine Martin <antoine@nagafix.co.uk>
To: Michael Tokarev <mjt@tls.msk.ru>
Cc: kvm@vger.kernel.org
Subject: Re: repeatable hang with loop mount and heavy IO in guest [NOT SOLVED]
Date: Sat, 23 Jan 2010 02:15:36 +0700 [thread overview]
Message-ID: <4B59F958.8030709@nagafix.co.uk> (raw)
In-Reply-To: <4B59EE4E.6070005@nagafix.co.uk>
On 01/23/2010 01:28 AM, Antoine Martin wrote:
> On 01/22/2010 02:57 PM, Michael Tokarev wrote:
>> Antoine Martin wrote:
>>> I've tried various guests, including most recent Fedora12 kernels,
>>> custom 2.6.32.x
>>> All of them hang around the same point (~1GB written) when I do
>>> heavy IO
>>> write inside the guest.
>> []
>>> Host is running: 2.6.31.4
>>> QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88)
>> Please update to last version and repeat. kvm-88 is ancient and
>> _lots_ of stuff fixed and changed since that time, I doubt anyone
>> here will try to dig into kvm-88 problems.
>>
>> Current kvm is qemu-kvm-0.12.2, released yesterday.
> Sorry about that, I didn't realize 88 was so far behind.
> Upgrading to qemu-kvm-0.12.2 did solve my IO problems.
Only for a while. Same problem just re-occurred, only this time it went
a little further.
It is now just sitting there, with a load average of exactly 3.0 (+- 5%)
Here is a good trace of the symptom during writeback, you can see it
write the data at around 50MB/s, it goes from being idle to sys, but
after a while it just stops writing and goes into mostly wait state:
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
1 0 99 0 0 0| 0 0 | 198B 614B| 0 0 | 36 17
1 0 99 0 0 0| 0 0 | 198B 710B| 0 0 | 31 17
1 1 98 0 0 0| 0 128k| 240B 720B| 0 0 | 39 26
1 1 98 0 0 0| 0 0 | 132B 564B| 0 0 | 31 14
1 0 99 0 0 0| 0 0 | 132B 468B| 0 0 | 31 14
1 1 98 0 0 0| 0 0 | 66B 354B| 0 0 | 30 13
0 4 11 85 0 0| 852k 0 | 444B 1194B| 0 0 | 215 477
2 2 0 96 0 0| 500k 0 | 132B 756B| 0 0 | 169 458
3 57 0 39 1 0| 228k 10M| 132B 692B| 0 0 | 476 5387
6 94 0 0 0 0| 28k 23M| 132B 884B| 0 0 | 373 2142
6 89 0 2 2 0| 40k 38M| 66B 692B| 0 8192B| 502 5651
4 47 0 48 0 0| 140k 34M| 132B 836B| 0 0 | 605 1664
3 64 0 30 2 0| 60k 50M| 132B 370B| 0 60k| 750 631
4 59 0 35 2 0| 48k 45M| 132B 836B| 0 28k| 708 1293
7 81 0 10 2 0| 68k 67M| 132B 788B| 0 124k| 928 1634
5 74 0 20 1 0| 48k 48M| 132B 756B| 0 316k| 830 5715
5 70 0 24 1 0| 168k 48M| 132B 676B| 0 100k| 734 5325
4 70 0 24 1 0| 72k 49M| 132B 948B| 0 88k| 776 3784
5 57 0 37 1 0| 36k 37M| 132B 996B| 0 480k| 602 369
2 21 0 77 0 0| 36k 23M| 132B 724B| 0 72k| 318 1033
4 51 0 43 2 0| 112k 43M| 132B 756B| 0 112k| 681 909
5 55 0 40 0 0| 88k 48M| 140B 926B| 16k 12k| 698 557
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
3 45 0 51 1 0|2248k 29M| 198B 1028B| 28k 44k| 681 5468
1 21 0 78 0 0| 92k 17M|1275B 2049B| 92k 52k| 328 1883
3 30 0 66 1 0| 288k 28M| 498B 2116B| 0 40k| 455 679
1 1 0 98 0 0|4096B 0 | 394B 1340B|4096B 0 | 41 19
1 1 0 98 0 0| 148k 52k| 881B 1592B|4096B 44k| 75 61
1 2 0 97 0 0|1408k 0 | 351B 1727B| 0 0 | 110 109
2 1 0 97 0 0|8192B 0 |1422B 1940B| 0 0 | 53 34
1 0 0 99 0 0|4096B 12k| 328B 1018B| 0 0 | 41 24
1 4 0 95 0 0| 340k 0 |3075B 2152B|4096B 0 | 153 191
4 7 0 89 0 0|1004k 44k|1526B 1906B| 0 0 | 254 244
0 1 0 99 0 0| 76k 0 | 708B 1708B| 0 0 | 67 57
1 1 0 98 0 0| 0 0 | 174B 702B| 0 0 | 32 14
1 1 0 98 0 0| 0 0 | 132B 354B| 0 0 | 32 11
1 0 0 99 0 0| 0 0 | 132B 468B| 0 0 | 32 16
1 0 0 99 0 0| 0 0 | 132B 468B| 0 0 | 32 14
1 1 0 98 0 0| 0 52k| 132B 678B| 0 0 | 41 27
1 0 0 99 0 0| 0 0 | 198B 678B| 0 0 | 35 17
1 1 0 98 0 0| 0 0 | 198B 468B| 0 0 | 34 14
1 0 0 99 0 0| 0 0 | 66B 354B| 0 0 | 28 11
1 0 0 99 0 0| 0 0 | 66B 354B| 0 0 | 28 9
1 1 0 98 0 0| 0 0 | 132B 468B| 0 0 | 34 16
1 0 0 98 0 1| 0 0 | 66B 354B| 0 0 | 30 11
1 1 0 98 0 0| 0 0 | 66B 354B| 0 0 | 29 11
From that point onwards, nothing will happen.
The host has disk IO to spare... So what is it waiting for??
QEMU PC emulator version 0.12.2 (qemu-kvm-0.12.2), Copyright (c)
2003-2008 Fabrice Bellard
Guests: various, all recent kernels.
Host: 2.6.31.4
Please advise.
Thanks
Antoine
next prev parent reply other threads:[~2010-01-22 19:15 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-01-21 17:26 repeatable hang with loop mount and heavy IO in guest Antoine Martin
2010-01-21 20:08 ` RW
2010-01-21 21:08 ` Thomas Beinicke
2010-01-21 21:36 ` RW
2010-01-22 7:57 ` Michael Tokarev
2010-01-22 18:28 ` repeatable hang with loop mount and heavy IO in guest [SOLVED] Antoine Martin
2010-01-22 19:15 ` Antoine Martin [this message]
2010-01-24 11:23 ` repeatable hang with loop mount and heavy IO in guest [NOT SOLVED] Antoine Martin
2010-02-03 19:28 ` Antoine Martin
2010-02-26 17:38 ` repeatable hang with loop mount and heavy IO in guest Antoine Martin
2010-05-21 9:38 ` repeatable hang with loop mount and heavy IO in guest (now in host - not KVM then..) Antoine Martin
2010-05-22 18:10 ` Jim Paris
2010-05-22 19:33 ` Antoine Martin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4B59F958.8030709@nagafix.co.uk \
--to=antoine@nagafix.co.uk \
--cc=kvm@vger.kernel.org \
--cc=mjt@tls.msk.ru \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox