From: RW <kvm@tauceti.net>
To: Antoine Martin <antoine@nagafix.co.uk>
Cc: kvm@vger.kernel.org
Subject: Re: repeatable hang with loop mount and heavy IO in guest
Date: Thu, 21 Jan 2010 21:08:38 +0100 [thread overview]
Message-ID: <4B58B446.4080801@tauceti.net> (raw)
In-Reply-To: <4B588E63.4060700@nagafix.co.uk>
Some months ago I also thought elevator=noop should be a good idea.
But it isn't. It works good as long as you only do short IO requests.
Try using deadline in host and guest.
Robert
On 01/21/10 18:26, Antoine Martin wrote:
> I've tried various guests, including most recent Fedora12 kernels,
> custom 2.6.32.x
> All of them hang around the same point (~1GB written) when I do heavy IO
> write inside the guest.
> I have waited 30 minutes to see if the guest would recover, but it just
> sits there, not writing back any data, not doing anything - but
> certainly not allowing any new IO writes. The host has some load on it,
> but nothing heavy enough to completely hand a guest for that long.
>
> mount -o loop some_image.fs ./somewhere bs=512
> dd if=/dev/zero of=/somewhere/zero
> then after ~1GB: sync
>
> Host is running: 2.6.31.4
> QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88)
>
> Guests are booted with "elevator=noop" as the filesystems are stored as
> files, accessed as virtio disks.
>
>
> The "hung" backtraces always look similar to these:
> [ 361.460136] INFO: task loop0:2097 blocked for more than 120 seconds.
> [ 361.460139] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 361.460142] loop0 D ffff88000b92c848 0 2097 2
> 0x00000080
> [ 361.460148] ffff88000b92c5d0 0000000000000046 ffff880008c1f810
> ffff880009829fd8
> [ 361.460153] ffff880009829fd8 ffff880009829fd8 ffff88000a21ee80
> ffff88000b92c5d0
> [ 361.460157] ffff880009829610 ffffffff8181b768 ffff880001af33b0
> 0000000000000002
> [ 361.460161] Call Trace:
> [ 361.460216] [<ffffffff8105bf12>] ? sync_page+0x0/0x43
> [ 361.460253] [<ffffffff8151383e>] ? io_schedule+0x2c/0x43
> [ 361.460257] [<ffffffff8105bf50>] ? sync_page+0x3e/0x43
> [ 361.460261] [<ffffffff81513a2a>] ? __wait_on_bit+0x41/0x71
> [ 361.460264] [<ffffffff8105c092>] ? wait_on_page_bit+0x6a/0x70
> [ 361.460283] [<ffffffff810385a7>] ? wake_bit_function+0x0/0x23
> [ 361.460287] [<ffffffff81064975>] ? shrink_page_list+0x3e5/0x61e
> [ 361.460291] [<ffffffff81513992>] ? schedule_timeout+0xa3/0xbe
> [ 361.460305] [<ffffffff81038579>] ? autoremove_wake_function+0x0/0x2e
> [ 361.460308] [<ffffffff8106538f>] ? shrink_zone+0x7e1/0xaf6
> [ 361.460310] [<ffffffff81061725>] ? determine_dirtyable_memory+0xd/0x17
> [ 361.460314] [<ffffffff810637da>] ? isolate_pages_global+0xa3/0x216
> [ 361.460316] [<ffffffff81062712>] ? mark_page_accessed+0x2a/0x39
> [ 361.460335] [<ffffffff810a61db>] ? __find_get_block+0x13b/0x15c
> [ 361.460337] [<ffffffff81065ed4>] ? try_to_free_pages+0x1ab/0x2c9
> [ 361.460340] [<ffffffff81063737>] ? isolate_pages_global+0x0/0x216
> [ 361.460343] [<ffffffff81060baf>] ? __alloc_pages_nodemask+0x394/0x564
> [ 361.460350] [<ffffffff8108250c>] ? __slab_alloc+0x137/0x44f
> [ 361.460371] [<ffffffff812cc4c1>] ? radix_tree_preload+0x1f/0x6a
> [ 361.460374] [<ffffffff81082a08>] ? kmem_cache_alloc+0x5d/0x88
> [ 361.460376] [<ffffffff812cc4c1>] ? radix_tree_preload+0x1f/0x6a
> [ 361.460379] [<ffffffff8105c0b5>] ? add_to_page_cache_locked+0x1d/0xf1
> [ 361.460381] [<ffffffff8105c1b0>] ? add_to_page_cache_lru+0x27/0x57
> [ 361.460384] [<ffffffff8105c25a>] ?
> grab_cache_page_write_begin+0x7a/0xa0
> [ 361.460399] [<ffffffff81104620>] ? ext3_write_begin+0x7e/0x201
> [ 361.460417] [<ffffffff8134648f>] ? do_lo_send_aops+0xa1/0x174
> [ 361.460420] [<ffffffff81081948>] ? virt_to_head_page+0x9/0x2a
> [ 361.460422] [<ffffffff8134686b>] ? loop_thread+0x309/0x48a
> [ 361.460425] [<ffffffff813463ee>] ? do_lo_send_aops+0x0/0x174
> [ 361.460427] [<ffffffff81038579>] ? autoremove_wake_function+0x0/0x2e
> [ 361.460430] [<ffffffff81346562>] ? loop_thread+0x0/0x48a
> [ 361.460432] [<ffffffff8103819b>] ? kthread+0x78/0x80
> [ 361.460441] [<ffffffff810238df>] ? finish_task_switch+0x2b/0x78
> [ 361.460454] [<ffffffff81002f6a>] ? child_rip+0xa/0x20
> [ 361.460460] [<ffffffff81012ac3>] ? native_pax_close_kernel+0x0/0x32
> [ 361.460463] [<ffffffff81038123>] ? kthread+0x0/0x80
> [ 361.460469] [<ffffffff81002f60>] ? child_rip+0x0/0x20
> [ 361.460471] INFO: task kjournald:2098 blocked for more than 120 seconds.
> [ 361.460473] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 361.460474] kjournald D ffff88000b92e558 0 2098 2
> 0x00000080
> [ 361.460477] ffff88000b92e2e0 0000000000000046 ffff88000aad9840
> ffff88000983ffd8
> [ 361.460480] ffff88000983ffd8 ffff88000983ffd8 ffffffff81808e00
> ffff88000b92e2e0
> [ 361.460483] ffff88000983fcf0 ffffffff8181b768 ffff880001af3c40
> 0000000000000002
> [ 361.460486] Call Trace:
> [ 361.460488] [<ffffffff810a6b16>] ? sync_buffer+0x0/0x3c
> [ 361.460491] [<ffffffff8151383e>] ? io_schedule+0x2c/0x43
> [ 361.460494] [<ffffffff810a6b4e>] ? sync_buffer+0x38/0x3c
> [ 361.460496] [<ffffffff81513a2a>] ? __wait_on_bit+0x41/0x71
> [ 361.460499] [<ffffffff810a6b16>] ? sync_buffer+0x0/0x3c
> [ 361.460501] [<ffffffff81513ac4>] ? out_of_line_wait_on_bit+0x6a/0x76
> [ 361.460504] [<ffffffff810385a7>] ? wake_bit_function+0x0/0x23
> [ 361.460514] [<ffffffff8113edad>] ?
> journal_commit_transaction+0x769/0xbb8
> [ 361.460517] [<ffffffff810238df>] ? finish_task_switch+0x2b/0x78
> [ 361.460519] [<ffffffff815137d9>] ? thread_return+0x40/0x79
> [ 361.460522] [<ffffffff8114162d>] ? kjournald+0xc7/0x1cb
> [ 361.460525] [<ffffffff81038579>] ? autoremove_wake_function+0x0/0x2e
> [ 361.460527] [<ffffffff81141566>] ? kjournald+0x0/0x1cb
> [ 361.460530] [<ffffffff8103819b>] ? kthread+0x78/0x80
> [ 361.460532] [<ffffffff810238df>] ? finish_task_switch+0x2b/0x78
> [ 361.460534] [<ffffffff81002f6a>] ? child_rip+0xa/0x20
> [ 361.460537] [<ffffffff81012ac3>] ? native_pax_close_kernel+0x0/0x32
> [ 361.460540] [<ffffffff81038123>] ? kthread+0x0/0x80
> [ 361.460542] [<ffffffff81002f60>] ? child_rip+0x0/0x20
> [ 361.460544] INFO: task dd:2132 blocked for more than 120 seconds.
> [ 361.460546] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 361.460547] dd D ffff88000a21f0f8 0 2132 2090
> 0x00000080
> [ 361.460550] ffff88000a21ee80 0000000000000082 ffff88000a21ee80
> ffff88000b3affd8
> [ 361.460553] ffff88000b3affd8 ffff88000b3affd8 ffffffff81808e00
> ffff880001af3510
> [ 361.460556] ffff88000b78eaf0 ffff88000b3daa00 ffff880008de6c40
> ffff88000ab44a80
> [ 361.460558] Call Trace:
> [ 361.460561] [<ffffffff8113dda5>] ? do_get_write_access+0x1f5/0x3b6
> [ 361.460564] [<ffffffff81061956>] ? get_dirty_limits+0x1dc/0x210
> [ 361.460566] [<ffffffff810385a7>] ? wake_bit_function+0x0/0x23
> [ 361.460569] [<ffffffff810a6218>] ? __getblk+0x1c/0x26c
> [ 361.460576] [<ffffffff8155d1d0>] ? __func__.28446+0x0/0x20
> [ 361.460578] [<ffffffff8113df88>] ? journal_get_write_access+0x22/0x34
> [ 361.460582] [<ffffffff8110dd9b>] ?
> __ext3_journal_get_write_access+0x1e/0x47
> [ 361.460584] [<ffffffff81101c4d>] ? ext3_reserve_inode_write+0x3e/0x75
> [ 361.460587] [<ffffffff81101c9a>] ? ext3_mark_inode_dirty+0x16/0x31
> [ 361.460589] [<ffffffff81101deb>] ? ext3_dirty_inode+0x62/0x7a
> [ 361.460592] [<ffffffff810a10d9>] ? __mark_inode_dirty+0x25/0x134
> [ 361.460595] [<ffffffff81098b80>] ? file_update_time+0xd4/0xfb
> [ 361.460598] [<ffffffff8105ced8>] ? __generic_file_aio_write+0x16c/0x290
> [ 361.460600] [<ffffffff8105d055>] ? generic_file_aio_write+0x59/0x9f
> [ 361.460603] [<ffffffff81087ab5>] ? do_sync_write+0xcd/0x112
> [ 361.460606] [<ffffffff810132d4>] ? pvclock_clocksource_read+0x3a/0x70
> [ 361.460609] [<ffffffff81038579>] ? autoremove_wake_function+0x0/0x2e
> [ 361.460612] [<ffffffff81000d1a>] ? __switch_to+0x177/0x255
> [ 361.460621] [<ffffffff8127891e>] ? selinux_file_permission+0x4d/0xa3
> [ 361.460624] [<ffffffff810883d8>] ? vfs_write+0xfc/0x138
> [ 361.460627] [<ffffffff810884d0>] ? sys_write+0x45/0x6e
> [ 361.460629] [<ffffffff810020ff>] ? system_call_fastpath+0x16/0x1b
> [ 361.460632] INFO: task sync:2164 blocked for more than 120 seconds.
> [ 361.460633] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 361.460639] sync D ffff88000ba11f88 0 2164 2136
> 0x00000080
> [ 361.460642] ffff88000ba11d10 0000000000000086 0000000100000246
> ffff88000b1e9fd8
> [ 361.460645] ffff88000b1e9fd8 ffff88000b1e9fd8 ffffffff81808e00
> ffff88000b3daa00
> [ 361.460648] 00000000000001cc ffff88000b1e9e68 ffff88000b1e9e80
> ffff88000b3daa78
> [ 361.460651] Call Trace:
> [ 361.460653] [<ffffffff8114122b>] ? log_wait_commit+0x9e/0xe0
> [ 361.460656] [<ffffffff81038579>] ? autoremove_wake_function+0x0/0x2e
> [ 361.460659] [<ffffffff81108fe7>] ? ext3_sync_fs+0x42/0x4b
> [ 361.460669] [<ffffffff810c9711>] ? sync_quota_sb+0x45/0xf6
> [ 361.460672] [<ffffffff810a4cd2>] ? __sync_filesystem+0x43/0x70
> [ 361.460675] [<ffffffff810a4d86>] ? sync_filesystems+0x87/0xbd
> [ 361.460677] [<ffffffff810a4e01>] ? sys_sync+0x1c/0x2e
> [ 361.460679] [<ffffffff810020ff>] ? system_call_fastpath+0x16/0x1b
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2010-01-21 20:04 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-01-21 17:26 repeatable hang with loop mount and heavy IO in guest Antoine Martin
2010-01-21 20:08 ` RW [this message]
2010-01-21 21:08 ` Thomas Beinicke
2010-01-21 21:36 ` RW
2010-01-22 7:57 ` Michael Tokarev
2010-01-22 18:28 ` repeatable hang with loop mount and heavy IO in guest [SOLVED] Antoine Martin
2010-01-22 19:15 ` repeatable hang with loop mount and heavy IO in guest [NOT SOLVED] Antoine Martin
2010-01-24 11:23 ` Antoine Martin
2010-02-03 19:28 ` Antoine Martin
2010-02-26 17:38 ` repeatable hang with loop mount and heavy IO in guest Antoine Martin
2010-05-21 9:38 ` repeatable hang with loop mount and heavy IO in guest (now in host - not KVM then..) Antoine Martin
2010-05-22 18:10 ` Jim Paris
2010-05-22 19:33 ` Antoine Martin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4B58B446.4080801@tauceti.net \
--to=kvm@tauceti.net \
--cc=antoine@nagafix.co.uk \
--cc=kvm@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox