* xfs: blocked for more than 120 seconds
@ 2014-01-16 19:51 Ivan Novick
2014-01-16 21:58 ` Dave Chinner
0 siblings, 1 reply; 2+ messages in thread
From: Ivan Novick @ 2014-01-16 19:51 UTC (permalink / raw)
To: xfs
[-- Attachment #1.1: Type: text/plain, Size: 19750 bytes --]
Hello,
I am running a server with heavy workload on a XFS mount:
/dev/mapper/v2-d1 on /d/d1 type xfs (rw,nodev,noatime,inode64,allocsize=16m)
2.6.32-424.el6.x86_64 #1 SMP Mon Oct 14 20:11:50 EDT 2013 x86_64 x86_64
x86_64 GNU/Linux
We get errors in log indicating processes are blocked for more than 120
seconds.
1) Is this expected during heavy workload?
2) What would be the impact on the processes? Are they basically hung in
userspace waiting for IO?
3) Is there anything we tune here?
Below is the output.
Cheers,
Ivan Novick
INFO: task flush-253:1:6882 blocked for more than 120 seconds.
Tainted: P --------------- 2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-253:1 D 0000000000000000 0 6882 2 0x00000000
ffff881011fa5690 0000000000000046 0000000000000000 ffff881013cba040
0000000000000001 ffff88012a447d40 ffff881013cba040 ffff881016eb4830
ffff881013cba5f8 ffff881011fa5fd8 000000000000fbc8 ffff881013cba5f8
Call Trace:
[<ffffffff815289d3>] io_schedule+0x73/0xc0
[<ffffffff81267d18>] get_request_wait+0x108/0x1d0
[<ffffffff8109b4a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff812618ce>] ? elv_merge+0x17e/0x1c0
[<ffffffff81267e79>] blk_queue_bio+0x99/0x620
[<ffffffff81266f00>] generic_make_request+0x240/0x5a0
[<ffffffffa01c3f97>] ? dm_merge_bvec+0xc7/0x100 [dm_mod]
[<ffffffff812672d0>] submit_bio+0x70/0x120
[<ffffffffa03b9c53>] xfs_submit_ioend_bio+0x33/0x40 [xfs]
[<ffffffffa03b9d2c>] xfs_submit_ioend+0xcc/0x140 [xfs]
[<ffffffffa03ba678>] xfs_vm_writepage+0x308/0x590 [xfs]
[<ffffffff81133967>] __writepage+0x17/0x40
[<ffffffff81134c2d>] write_cache_pages+0x1fd/0x4c0
[<ffffffff8100bdae>] ? call_function_single_interrupt+0xe/0x20
[<ffffffff81133950>] ? __writepage+0x0/0x40
[<ffffffff81134f14>] generic_writepages+0x24/0x30
[<ffffffffa03b98ad>] xfs_vm_writepages+0x5d/0x80 [xfs]
[<ffffffff81134f41>] do_writepages+0x21/0x40
[<ffffffff811b52ad>] writeback_single_inode+0xdd/0x290
[<ffffffff811b56ad>] writeback_sb_inodes+0xbd/0x170
[<ffffffff811b580b>] writeback_inodes_wb+0xab/0x1b0
[<ffffffff811b5c03>] wb_writeback+0x2f3/0x410
[<ffffffff81528240>] ? thread_return+0x4e/0x76e
[<ffffffff81084fb2>] ? del_timer_sync+0x22/0x30
[<ffffffff811b5ec5>] wb_do_writeback+0x1a5/0x240
[<ffffffff811b5fc3>] bdi_writeback_task+0x63/0x1b0
[<ffffffff8109b327>] ? bit_waitqueue+0x17/0xd0
[<ffffffff81143c40>] ? bdi_start_fn+0x0/0x100
[<ffffffff81143cc6>] bdi_start_fn+0x86/0x100
[<ffffffff81143c40>] ? bdi_start_fn+0x0/0x100
[<ffffffff8109b0f6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8109b060>] ? kthread+0x0/0xa0
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task ivanserver:6969 blocked for more than 120 seconds.
Tainted: P --------------- 2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver D 0000000000000000 0 6969 6947 0x00000000
ffff881f4ef179d8 0000000000000082 0000000000000000 000000000000000f
0000000000000001 ffff8800622143c0 0000000000000001 ffff880062216840
ffff881f4eeae638 ffff881f4ef17fd8 000000000000fbc8 ffff881f4eeae638
Call Trace:
[<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
[<ffffffff8152a0e2>] __down+0x72/0xb0
[<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffff810a1591>] down+0x41/0x50
[<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
[<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
[<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
[<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
[<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
[<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
[<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
[<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
[<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffff8100bdae>] ? call_function_single_interrupt+0xe/0x20
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
[<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
[<ffffffff8118a5b0>] ? fget+0x30/0x80
[<ffffffff8119f429>] sys_getdents+0x89/0xf0
[<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ivanserver:6972 blocked for more than 120 seconds.
Tainted: P --------------- 2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver D 0000000000000008 0 6972 6948 0x00000000
ffff88100f1219d8 0000000000000086 0000000000000000 000000000000000f
0000000000000001 ffff8810788143c0 0000000000000001 ffff881078816840
ffff88100fd62638 ffff88100f121fd8 000000000000fbc8 ffff88100fd62638
Call Trace:
[<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
[<ffffffff8152a0e2>] __down+0x72/0xb0
[<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffff810a1591>] down+0x41/0x50
[<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
[<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
[<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
[<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
[<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
[<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
[<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
[<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
[<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
[<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
[<ffffffff8119f429>] sys_getdents+0x89/0xf0
[<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ivanserver:6985 blocked for more than 120 seconds.
Tainted: P --------------- 2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver D 0000000000000000 0 6985 6963 0x00000000
ffff8810103ad9d8 0000000000000082 0000000000000000 000000000000000f
0000000000000001 ffff8800622143c0 0000000000000001 ffff880062216840
ffff88100e01e638 ffff8810103adfd8 000000000000fbc8 ffff88100e01e638
Call Trace:
[<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
[<ffffffff8152a0e2>] __down+0x72/0xb0
[<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffff810a1591>] down+0x41/0x50
[<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
[<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
[<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
[<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
[<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
[<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
[<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
[<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
[<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
[<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
[<ffffffff8119f429>] sys_getdents+0x89/0xf0
[<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ivanserver:6989 blocked for more than 120 seconds.
Tainted: P --------------- 2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver D 0000000000000008 0 6989 6981 0x00000000
ffff8810101279d8 0000000000000086 0000000000000000 000000000000000f
0000000000000001 ffff8810788143c0 0000000000000001 ffff881078816840
ffff8810252a4638 ffff881010127fd8 000000000000fbc8 ffff8810252a4638
Call Trace:
[<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
[<ffffffff8152a0e2>] __down+0x72/0xb0
[<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffff810a1591>] down+0x41/0x50
[<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
[<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
[<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
[<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
[<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
[<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
[<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
[<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
[<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
[<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
[<ffffffff8119f429>] sys_getdents+0x89/0xf0
[<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
usxxnovicim1:Downloads novici$ cat dmesg.txt
INFO: task flush-253:1:6882 blocked for more than 120 seconds.
Tainted: P --------------- 2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-253:1 D 0000000000000000 0 6882 2 0x00000000
ffff881011fa5690 0000000000000046 0000000000000000 ffff881013cba040
0000000000000001 ffff88012a447d40 ffff881013cba040 ffff881016eb4830
ffff881013cba5f8 ffff881011fa5fd8 000000000000fbc8 ffff881013cba5f8
Call Trace:
[<ffffffff815289d3>] io_schedule+0x73/0xc0
[<ffffffff81267d18>] get_request_wait+0x108/0x1d0
[<ffffffff8109b4a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff812618ce>] ? elv_merge+0x17e/0x1c0
[<ffffffff81267e79>] blk_queue_bio+0x99/0x620
[<ffffffff81266f00>] generic_make_request+0x240/0x5a0
[<ffffffffa01c3f97>] ? dm_merge_bvec+0xc7/0x100 [dm_mod]
[<ffffffff812672d0>] submit_bio+0x70/0x120
[<ffffffffa03b9c53>] xfs_submit_ioend_bio+0x33/0x40 [xfs]
[<ffffffffa03b9d2c>] xfs_submit_ioend+0xcc/0x140 [xfs]
[<ffffffffa03ba678>] xfs_vm_writepage+0x308/0x590 [xfs]
[<ffffffff81133967>] __writepage+0x17/0x40
[<ffffffff81134c2d>] write_cache_pages+0x1fd/0x4c0
[<ffffffff8100bdae>] ? call_function_single_interrupt+0xe/0x20
[<ffffffff81133950>] ? __writepage+0x0/0x40
[<ffffffff81134f14>] generic_writepages+0x24/0x30
[<ffffffffa03b98ad>] xfs_vm_writepages+0x5d/0x80 [xfs]
[<ffffffff81134f41>] do_writepages+0x21/0x40
[<ffffffff811b52ad>] writeback_single_inode+0xdd/0x290
[<ffffffff811b56ad>] writeback_sb_inodes+0xbd/0x170
[<ffffffff811b580b>] writeback_inodes_wb+0xab/0x1b0
[<ffffffff811b5c03>] wb_writeback+0x2f3/0x410
[<ffffffff81528240>] ? thread_return+0x4e/0x76e
[<ffffffff81084fb2>] ? del_timer_sync+0x22/0x30
[<ffffffff811b5ec5>] wb_do_writeback+0x1a5/0x240
[<ffffffff811b5fc3>] bdi_writeback_task+0x63/0x1b0
[<ffffffff8109b327>] ? bit_waitqueue+0x17/0xd0
[<ffffffff81143c40>] ? bdi_start_fn+0x0/0x100
[<ffffffff81143cc6>] bdi_start_fn+0x86/0x100
[<ffffffff81143c40>] ? bdi_start_fn+0x0/0x100
[<ffffffff8109b0f6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8109b060>] ? kthread+0x0/0xa0
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task ivanserver:6969 blocked for more than 120 seconds.
Tainted: P --------------- 2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver D 0000000000000000 0 6969 6947 0x00000000
ffff881f4ef179d8 0000000000000082 0000000000000000 000000000000000f
0000000000000001 ffff8800622143c0 0000000000000001 ffff880062216840
ffff881f4eeae638 ffff881f4ef17fd8 000000000000fbc8 ffff881f4eeae638
Call Trace:
[<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
[<ffffffff8152a0e2>] __down+0x72/0xb0
[<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffff810a1591>] down+0x41/0x50
[<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
[<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
[<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
[<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
[<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
[<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
[<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
[<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
[<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffff8100bdae>] ? call_function_single_interrupt+0xe/0x20
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
[<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
[<ffffffff8118a5b0>] ? fget+0x30/0x80
[<ffffffff8119f429>] sys_getdents+0x89/0xf0
[<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ivanserver:6972 blocked for more than 120 seconds.
Tainted: P --------------- 2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver D 0000000000000008 0 6972 6948 0x00000000
ffff88100f1219d8 0000000000000086 0000000000000000 000000000000000f
0000000000000001 ffff8810788143c0 0000000000000001 ffff881078816840
ffff88100fd62638 ffff88100f121fd8 000000000000fbc8 ffff88100fd62638
Call Trace:
[<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
[<ffffffff8152a0e2>] __down+0x72/0xb0
[<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffff810a1591>] down+0x41/0x50
[<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
[<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
[<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
[<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
[<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
[<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
[<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
[<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
[<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
[<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
[<ffffffff8119f429>] sys_getdents+0x89/0xf0
[<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ivanserver:6985 blocked for more than 120 seconds.
Tainted: P --------------- 2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver D 0000000000000000 0 6985 6963 0x00000000
ffff8810103ad9d8 0000000000000082 0000000000000000 000000000000000f
0000000000000001 ffff8800622143c0 0000000000000001 ffff880062216840
ffff88100e01e638 ffff8810103adfd8 000000000000fbc8 ffff88100e01e638
Call Trace:
[<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
[<ffffffff8152a0e2>] __down+0x72/0xb0
[<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffff810a1591>] down+0x41/0x50
[<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
[<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
[<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
[<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
[<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
[<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
[<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
[<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
[<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
[<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
[<ffffffff8119f429>] sys_getdents+0x89/0xf0
[<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ivanserver:6989 blocked for more than 120 seconds.
Tainted: P --------------- 2.6.32-424.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ivanserver D 0000000000000008 0 6989 6981 0x00000000
ffff8810101279d8 0000000000000086 0000000000000000 000000000000000f
0000000000000001 ffff8810788143c0 0000000000000001 ffff881078816840
ffff8810252a4638 ffff881010127fd8 000000000000fbc8 ffff8810252a4638
Call Trace:
[<ffffffff815290e5>] schedule_timeout+0x215/0x2e0
[<ffffffff8152a0e2>] __down+0x72/0xb0
[<ffffffffa03bcaa5>] ? _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffff810a1591>] down+0x41/0x50
[<ffffffffa03bc911>] xfs_buf_lock+0x51/0x100 [xfs]
[<ffffffffa03bcaa5>] _xfs_buf_find+0xe5/0x230 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bd266>] ? _xfs_buf_read+0x46/0x60 [xfs]
[<ffffffffa03bcc24>] xfs_buf_get+0x34/0x1b0 [xfs]
[<ffffffffa03bd2ac>] xfs_buf_read+0x2c/0x100 [xfs]
[<ffffffffa03b2c37>] xfs_trans_read_buf+0x197/0x410 [xfs]
[<ffffffffa0385af9>] xfs_da_do_buf+0x279/0x770 [xfs]
[<ffffffffa03babcf>] ? xfs_buf_lru_add+0x6f/0x80 [xfs]
[<ffffffffa03b3288>] ? xfs_trans_brelse+0xe8/0x130 [xfs]
[<ffffffffa038606a>] xfs_da_read_buf+0x2a/0x30 [xfs]
[<ffffffffa038c9e6>] ? xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffffa038c9e6>] xfs_dir2_leaf_getdents+0x266/0x780 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03891e9>] xfs_readdir+0xf9/0x130 [xfs]
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffff8119f020>] ? filldir+0x0/0xe0
[<ffffffffa03be619>] xfs_file_readdir+0x39/0x50 [xfs]
[<ffffffff8119f2a0>] vfs_readdir+0xc0/0xe0
[<ffffffff8119f429>] sys_getdents+0x89/0xf0
[<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
[-- Attachment #1.2: Type: text/html, Size: 26131 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 2+ messages in thread* Re: xfs: blocked for more than 120 seconds
2014-01-16 19:51 xfs: blocked for more than 120 seconds Ivan Novick
@ 2014-01-16 21:58 ` Dave Chinner
0 siblings, 0 replies; 2+ messages in thread
From: Dave Chinner @ 2014-01-16 21:58 UTC (permalink / raw)
To: Ivan Novick; +Cc: xfs
On Thu, Jan 16, 2014 at 11:51:52AM -0800, Ivan Novick wrote:
> Hello,
>
> I am running a server with heavy workload on a XFS mount:
> /dev/mapper/v2-d1 on /d/d1 type xfs (rw,nodev,noatime,inode64,allocsize=16m)
> 2.6.32-424.el6.x86_64 #1 SMP Mon Oct 14 20:11:50 EDT 2013 x86_64 x86_64
> x86_64 GNU/Linux
>
> We get errors in log indicating processes are blocked for more than 120
> seconds.
>
> 1) Is this expected during heavy workload?
> 2) What would be the impact on the processes? Are they basically hung in
> userspace waiting for IO?
> 3) Is there anything we tune here?
>
> Below is the output.
>
> Cheers,
> Ivan Novick
>
> INFO: task flush-253:1:6882 blocked for more than 120 seconds.
> Tainted: P --------------- 2.6.32-424.el6.x86_64 #1
Proprietary kernel module taint on a RHEL/centos kernel, so there's
no guarantee anyone will be able to debug this here. If it's RHEL
you are using, please report it through your support channels...
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> flush-253:1 D 0000000000000000 0 6882 2 0x00000000
> ffff881011fa5690 0000000000000046 0000000000000000 ffff881013cba040
> 0000000000000001 ffff88012a447d40 ffff881013cba040 ffff881016eb4830
> ffff881013cba5f8 ffff881011fa5fd8 000000000000fbc8 ffff881013cba5f8
> Call Trace:
> [<ffffffff815289d3>] io_schedule+0x73/0xc0
> [<ffffffff81267d18>] get_request_wait+0x108/0x1d0
> [<ffffffff8109b4a0>] ? autoremove_wake_function+0x0/0x40
> [<ffffffff812618ce>] ? elv_merge+0x17e/0x1c0
> [<ffffffff81267e79>] blk_queue_bio+0x99/0x620
> [<ffffffff81266f00>] generic_make_request+0x240/0x5a0
Waiting for IO completion. Looks like you've either severely
overloaded your storage subsystem, or it's died and isn't responding
anymore.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2014-01-16 21:58 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-01-16 19:51 xfs: blocked for more than 120 seconds Ivan Novick
2014-01-16 21:58 ` Dave Chinner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).