* writes to a virtio block device hungs
@ 2008-09-22 9:16 Michael Tokarev
2008-09-23 7:06 ` Michael Tokarev
0 siblings, 1 reply; 3+ messages in thread
From: Michael Tokarev @ 2008-09-22 9:16 UTC (permalink / raw)
To: kvm-devel
Hello! It's my first email to this list.. ;)
After experimenting for some time with KVM on linux
(both host and guests are linux machines), I placed
one virtual machine into production use, and almost
immediately come... issues. Here's how it looks like
from the guest:
Sep 21 10:35:52 hobbit kernel: INFO: task cleanup:20535 blocked for more than 120 seconds.
Sep 21 10:35:52 hobbit kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 21 10:35:52 hobbit kernel: cleanup D 00000000 0 20535 1570
Sep 21 10:35:52 hobbit kernel: f73b39c0 00200086 00000000 00000000 c3a2ba48 00000000 f7022e00 00000000
Sep 21 10:35:52 hobbit kernel: dbc48ed4 f789c000 c0399080 c0157e48 0000000e 00000000 d05e1b80 d05e1ce4
Sep 21 10:35:52 hobbit kernel: 00000002 00200286 c01322f7 d05e1ce4 c0131ef0 dbc48ec8 00200286 c0132486
Sep 21 10:35:52 hobbit kernel: Call Trace:
Sep 21 10:35:52 hobbit kernel: [<c0157e48>] find_get_pages_tag+0x38/0x80
Sep 21 10:35:52 hobbit kernel: [<c01322f7>] lock_timer_base+0x27/0x60
Sep 21 10:35:52 hobbit kernel: [<c0131ef0>] process_timeout+0x0/0x10
Sep 21 10:35:52 hobbit kernel: [<c0132486>] __mod_timer+0x86/0xa0
Sep 21 10:35:52 hobbit kernel: [<c02c6408>] schedule_timeout+0x58/0xb0
Sep 21 10:35:52 hobbit kernel: [<c0131ef0>] process_timeout+0x0/0x10
Sep 21 10:35:52 hobbit kernel: [<f882db04>] journal_stop+0xa4/0x1b0 [jbd]
Sep 21 10:35:52 hobbit kernel: [<f882ece8>] journal_start+0x88/0xc0 [jbd]
Sep 21 10:35:52 hobbit kernel: [<f8860f20>] ext3_write_inode+0x0/0x40 [ext3]
Sep 21 10:35:52 hobbit kernel: [<f8860f20>] ext3_write_inode+0x0/0x40 [ext3]
Sep 21 10:35:52 hobbit kernel: [<c019d002>] __writeback_single_inode+0x282/0x390
Sep 21 10:35:52 hobbit kernel: [<c015f3c0>] generic_writepages+0x20/0x30
Sep 21 10:35:52 hobbit kernel: [<c015f419>] do_writepages+0x49/0x50
Sep 21 10:35:52 hobbit kernel: [<c0159151>] __filemap_fdatawrite_range+0x71/0x90
Sep 21 10:35:52 hobbit kernel: [<c019d131>] sync_inode+0x21/0x40
Sep 21 10:35:52 hobbit kernel: [<f885f88e>] ext3_sync_file+0x9e/0xc0 [ext3]
Sep 21 10:35:52 hobbit kernel: [<c01a065e>] do_fsync+0x6e/0xb0
Sep 21 10:35:52 hobbit kernel: [<c01a06c7>] __do_fsync+0x27/0x50
Sep 21 10:35:52 hobbit kernel: [<c01032f3>] sysenter_past_esp+0x78/0xb1
Sep 21 10:35:52 hobbit kernel: =======================
It's almost always after fsync, but I guess it's due to the fact that
cleanup (from Postfix) process is the one who does that most often.
After first such message (after which corresponding process will sleep
forever), no write to the corresponding device will succeed - all will
stall the same way. It looks like kvm just "forgets" about each and
every write, effectively turning the device into a black hole -- but
only writes, reads are all ok.
Obviously the system will not reboot in that state, only force-reboot
(echo b > /proc/sysrq-trigger), or a "power-off" from the guest will
help.
The device in question is a virtio block device (vda), which is on top
op a raid1 device on the host (/dev/md_d5, partitioned). The problem
happens after some up-time, from several hours to 2 days, usually under
heavy load.
The system is Asus M3A-H/HDMI motherboard (AMD 780G/SB700 chipset),
with AMD Phenom 9750 CPU and 8Gb ECC memory. Stock 2.6.26.5 kernel,
with KVM optimizations (KVM_TIME etc) turned on in guest. kvm-72.
I'm running it with IDE emulation right now, to see if it will change
something or not.
The question is -- should I try with later kvm (kernel and userspace)
first? The thing is that it's production machine so any downtime is
not good...
Thank you!
/mjt
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: writes to a virtio block device hungs
2008-09-22 9:16 writes to a virtio block device hungs Michael Tokarev
@ 2008-09-23 7:06 ` Michael Tokarev
[not found] ` <20080925230251.GB22929@dmt.cnet>
0 siblings, 1 reply; 3+ messages in thread
From: Michael Tokarev @ 2008-09-23 7:06 UTC (permalink / raw)
To: kvm
[Replying to my own email...]
Michael Tokarev wrote:
> Hello! It's my first email to this list.. ;)
>
> After experimenting for some time with KVM on linux
> (both host and guests are linux machines), I placed
> one virtual machine into production use, and almost
> immediately come... issues. Here's how it looks like
> from the guest:
>
> Sep 21 10:35:52 hobbit kernel: INFO: task cleanup:20535 blocked for more than 120 seconds.
> Sep 21 10:35:52 hobbit kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 21 10:35:52 hobbit kernel: cleanup D 00000000 0 20535 1570
> Sep 21 10:35:52 hobbit kernel: f73b39c0 00200086 00000000 00000000 c3a2ba48 00000000 f7022e00 00000000
> Sep 21 10:35:52 hobbit kernel: dbc48ed4 f789c000 c0399080 c0157e48 0000000e 00000000 d05e1b80 d05e1ce4
> Sep 21 10:35:52 hobbit kernel: 00000002 00200286 c01322f7 d05e1ce4 c0131ef0 dbc48ec8 00200286 c0132486
> Sep 21 10:35:52 hobbit kernel: Call Trace:
> Sep 21 10:35:52 hobbit kernel: [<c0157e48>] find_get_pages_tag+0x38/0x80
> Sep 21 10:35:52 hobbit kernel: [<c01322f7>] lock_timer_base+0x27/0x60
> Sep 21 10:35:52 hobbit kernel: [<c0131ef0>] process_timeout+0x0/0x10
> Sep 21 10:35:52 hobbit kernel: [<c0132486>] __mod_timer+0x86/0xa0
> Sep 21 10:35:52 hobbit kernel: [<c02c6408>] schedule_timeout+0x58/0xb0
> Sep 21 10:35:52 hobbit kernel: [<c0131ef0>] process_timeout+0x0/0x10
> Sep 21 10:35:52 hobbit kernel: [<f882db04>] journal_stop+0xa4/0x1b0 [jbd]
> Sep 21 10:35:52 hobbit kernel: [<f882ece8>] journal_start+0x88/0xc0 [jbd]
> Sep 21 10:35:52 hobbit kernel: [<f8860f20>] ext3_write_inode+0x0/0x40 [ext3]
> Sep 21 10:35:52 hobbit kernel: [<f8860f20>] ext3_write_inode+0x0/0x40 [ext3]
> Sep 21 10:35:52 hobbit kernel: [<c019d002>] __writeback_single_inode+0x282/0x390
> Sep 21 10:35:52 hobbit kernel: [<c015f3c0>] generic_writepages+0x20/0x30
> Sep 21 10:35:52 hobbit kernel: [<c015f419>] do_writepages+0x49/0x50
> Sep 21 10:35:52 hobbit kernel: [<c0159151>] __filemap_fdatawrite_range+0x71/0x90
> Sep 21 10:35:52 hobbit kernel: [<c019d131>] sync_inode+0x21/0x40
> Sep 21 10:35:52 hobbit kernel: [<f885f88e>] ext3_sync_file+0x9e/0xc0 [ext3]
> Sep 21 10:35:52 hobbit kernel: [<c01a065e>] do_fsync+0x6e/0xb0
> Sep 21 10:35:52 hobbit kernel: [<c01a06c7>] __do_fsync+0x27/0x50
> Sep 21 10:35:52 hobbit kernel: [<c01032f3>] sysenter_past_esp+0x78/0xb1
> Sep 21 10:35:52 hobbit kernel: =======================
>
> It's almost always after fsync, but I guess it's due to the fact that
> cleanup (from Postfix) process is the one who does that most often.
>
> After first such message (after which corresponding process will sleep
> forever), no write to the corresponding device will succeed - all will
> stall the same way. It looks like kvm just "forgets" about each and
> every write, effectively turning the device into a black hole -- but
> only writes, reads are all ok.
>
> Obviously the system will not reboot in that state, only force-reboot
> (echo b > /proc/sysrq-trigger), or a "power-off" from the guest will
> help.
>
> The device in question is a virtio block device (vda), which is on top
> op a raid1 device on the host (/dev/md_d5, partitioned). The problem
> happens after some up-time, from several hours to 2 days, usually under
> heavy load.
>
> The system is Asus M3A-H/HDMI motherboard (AMD 780G/SB700 chipset),
> with AMD Phenom 9750 CPU and 8Gb ECC memory. Stock 2.6.26.5 kernel,
> with KVM optimizations (KVM_TIME etc) turned on in guest. kvm-72.
>
> I'm running it with IDE emulation right now, to see if it will change
> something or not.
With IDE (as opposed to virtio) the situation is exactly the same,
switching if=virtio to if=ide didn't change anything at all.
> The question is -- should I try with later kvm (kernel and userspace)
> first? The thing is that it's production machine so any downtime is
> not good...
I'm waiting for opportunity to install a new kernel with new kvm...
in a hope still.
Thank you!
/mjt
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: writes to a virtio block device hungs
[not found] ` <20080925230251.GB22929@dmt.cnet>
@ 2008-09-26 8:28 ` Michael Tokarev
0 siblings, 0 replies; 3+ messages in thread
From: Michael Tokarev @ 2008-09-26 8:28 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: kvm
Marcelo Tosatti wrote:
> On Tue, Sep 23, 2008 at 11:06:11AM +0400, Michael Tokarev wrote:
>>> (both host and guests are linux machines), I placed
>>> one virtual machine into production use, and almost
>>> immediately come... issues. Here's how it looks like
>>> from the guest:
>>>
>>> Sep 21 10:35:52 hobbit kernel: INFO: task cleanup:20535 blocked for more than 120 seconds.
>>> Sep 21 10:35:52 hobbit kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Sep 21 10:35:52 hobbit kernel: cleanup D 00000000 0 20535 1570
>>> Sep 21 10:35:52 hobbit kernel: f73b39c0 00200086 00000000 00000000 c3a2ba48 00000000 f7022e00 00000000
>>> Sep 21 10:35:52 hobbit kernel: dbc48ed4 f789c000 c0399080 c0157e48 0000000e 00000000 d05e1b80 d05e1ce4
>>> Sep 21 10:35:52 hobbit kernel: 00000002 00200286 c01322f7 d05e1ce4 c0131ef0 dbc48ec8 00200286 c0132486
>>> Sep 21 10:35:52 hobbit kernel: Call Trace:
>>> Sep 21 10:35:52 hobbit kernel: [<c0157e48>] find_get_pages_tag+0x38/0x80
>>> Sep 21 10:35:52 hobbit kernel: [<c01322f7>] lock_timer_base+0x27/0x60
>>> Sep 21 10:35:52 hobbit kernel: [<c0131ef0>] process_timeout+0x0/0x10
>>> Sep 21 10:35:52 hobbit kernel: [<c0132486>] __mod_timer+0x86/0xa0
>>> Sep 21 10:35:52 hobbit kernel: [<c02c6408>] schedule_timeout+0x58/0xb0
>>> Sep 21 10:35:52 hobbit kernel: [<c0131ef0>] process_timeout+0x0/0x10
>>> Sep 21 10:35:52 hobbit kernel: [<f882db04>] journal_stop+0xa4/0x1b0 [jbd]
>>> Sep 21 10:35:52 hobbit kernel: [<f882ece8>] journal_start+0x88/0xc0 [jbd]
>>> Sep 21 10:35:52 hobbit kernel: [<f8860f20>] ext3_write_inode+0x0/0x40 [ext3]
>>> Sep 21 10:35:52 hobbit kernel: [<f8860f20>] ext3_write_inode+0x0/0x40 [ext3]
>>> Sep 21 10:35:52 hobbit kernel: [<c019d002>] __writeback_single_inode+0x282/0x390
>>> Sep 21 10:35:52 hobbit kernel: [<c015f3c0>] generic_writepages+0x20/0x30
>>> Sep 21 10:35:52 hobbit kernel: [<c015f419>] do_writepages+0x49/0x50
>>> Sep 21 10:35:52 hobbit kernel: [<c0159151>] __filemap_fdatawrite_range+0x71/0x90
>>> Sep 21 10:35:52 hobbit kernel: [<c019d131>] sync_inode+0x21/0x40
>>> Sep 21 10:35:52 hobbit kernel: [<f885f88e>] ext3_sync_file+0x9e/0xc0 [ext3]
>>> Sep 21 10:35:52 hobbit kernel: [<c01a065e>] do_fsync+0x6e/0xb0
>>> Sep 21 10:35:52 hobbit kernel: [<c01a06c7>] __do_fsync+0x27/0x50
>>> Sep 21 10:35:52 hobbit kernel: [<c01032f3>] sysenter_past_esp+0x78/0xb1
>>> Sep 21 10:35:52 hobbit kernel: =======================
>>>
>>> It's almost always after fsync, but I guess it's due to the fact that
>>> cleanup (from Postfix) process is the one who does that most often.
>>>
>> I'm waiting for opportunity to install a new kernel with new kvm...
>> in a hope still.
Meanwhile I installed kvm-75, which did NOT change anything, -- the system
still hangs. What really changed things is switching guest to single
processor (was 2 before, from 4-core Phenom).
> Are you using ext3 in the host as the filesystem to back the guest
> image? If so, try writeback instead of ordered mode:
On the host there's an MD device (raid1) that hold complete "raw" disk
image for the guest. It was in my email:
>> The device in question is a virtio block device (vda), which is on top
>> op a raid1 device on the host (/dev/md_d5, partitioned). [...]
I'm trying to set up a test system to debug the case further,
because it's impossible to do that on production machine.
/mjt
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2008-09-26 8:28 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-09-22 9:16 writes to a virtio block device hungs Michael Tokarev
2008-09-23 7:06 ` Michael Tokarev
[not found] ` <20080925230251.GB22929@dmt.cnet>
2008-09-26 8:28 ` Michael Tokarev
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).