From: Brian Candler <B.Candler@pobox.com>
To: xfs@oss.sgi.com
Subject: Re: xfs task blocked for more than 120 seconds
Date: Tue, 31 Jan 2012 10:54:08 +0000 [thread overview]
Message-ID: <20120131105408.GA46226@nsrc.org> (raw)
Aside: I saw what seems to be the same error twice yesterday in a setup
without dcrypt, but running bonnie++ on a software RAID6 array (12 drives)
The tasks affected were "kworker" and "xfsaild" - full logs below. OS is
stock ubuntu 11.10 x86_64 server.
Write performance to the RAID6 is admittedly pretty terrible, so maybe it
was just some backlog building up.
Regards,
Brian.
...
Jan 30 10:36:40 storage2 kernel: [ 149.840035] XFS (md127): Mounting Filesystem
Jan 30 10:36:40 storage2 kernel: [ 150.083318] XFS (md127): Ending clean mount
Jan 30 12:00:11 storage2 kernel: [ 5159.737134] INFO: task kworker/0:3:1644 blocked for more than 120 seconds.
Jan 30 12:00:11 storage2 kernel: [ 5159.737507] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 30 12:00:11 storage2 kernel: [ 5159.746598] kworker/0:3 D ffffffff81805120 0 1644 2 0x00000000
Jan 30 12:00:11 storage2 kernel: [ 5159.746603] ffff8802078abb00 0000000000000046 0000000000000004 0000000000000000
Jan 30 12:00:11 storage2 kernel: [ 5159.746608] ffff8802078abfd8 ffff8802078abfd8 ffff8802078abfd8 0000000000012a40
Jan 30 12:00:11 storage2 kernel: [ 5159.746613] ffffffff81c0b020 ffff880212bc4560 ffff8802078abaf0 ffff8802129c2998
Jan 30 12:00:11 storage2 kernel: [ 5159.746618] Call Trace:
Jan 30 12:00:11 storage2 kernel: [ 5159.746625] [<ffffffff8160492f>] schedule+0x3f/0x60
Jan 30 12:00:11 storage2 kernel: [ 5159.746642] [<ffffffffa029f60c>] xlog_wait+0x6c/0x90 [xfs]
Jan 30 12:00:11 storage2 kernel: [ 5159.746646] [<ffffffff810573a0>] ? try_to_wake_up+0x200/0x200
Jan 30 12:00:11 storage2 kernel: [ 5159.746658] [<ffffffffa029f93f>] xlog_cil_push+0x30f/0x380 [xfs]
Jan 30 12:00:11 storage2 kernel: [ 5159.746661] [<ffffffff81086b92>] ? up+0x32/0x50
Jan 30 12:00:11 storage2 kernel: [ 5159.746672] [<ffffffffa02a015e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
Jan 30 12:00:11 storage2 kernel: [ 5159.746676] [<ffffffff811532df>] ? kmem_cache_free+0x2f/0x120
Jan 30 12:00:11 storage2 kernel: [ 5159.746687] [<ffffffffa029e797>] _xfs_log_force_lsn+0x57/0x2e0 [xfs]
Jan 30 12:00:11 storage2 kernel: [ 5159.746690] [<ffffffff811532df>] ? kmem_cache_free+0x2f/0x120
Jan 30 12:00:11 storage2 kernel: [ 5159.746702] [<ffffffffa02abf8d>] _xfs_trans_commit+0x29d/0x2b0 [xfs]
Jan 30 12:00:11 storage2 kernel: [ 5159.746715] [<ffffffffa02bfec0>] ? xfs_sync_inode_attr+0xf0/0xf0 [xfs]
Jan 30 12:00:11 storage2 kernel: [ 5159.746726] [<ffffffffa028f0c4>] xfs_fs_log_dummy+0x64/0x90 [xfs]
Jan 30 12:00:11 storage2 kernel: [ 5159.746738] [<ffffffffa02bff44>] xfs_sync_worker+0x84/0x90 [xfs]
Jan 30 12:00:11 storage2 kernel: [ 5159.746742] [<ffffffff8107b9ca>] process_one_work+0x11a/0x480
Jan 30 12:00:11 storage2 kernel: [ 5159.746745] [<ffffffff8107c775>] worker_thread+0x165/0x370
Jan 30 12:00:11 storage2 kernel: [ 5159.746748] [<ffffffff8107c610>] ? manage_workers.isra.30+0x130/0x130
Jan 30 12:00:11 storage2 kernel: [ 5159.746752] [<ffffffff81080bbc>] kthread+0x8c/0xa0
Jan 30 12:00:11 storage2 kernel: [ 5159.746755] [<ffffffff8160fca4>] kernel_thread_helper+0x4/0x10
Jan 30 12:00:11 storage2 kernel: [ 5159.746759] [<ffffffff81080b30>] ? flush_kthread_worker+0xa0/0xa0
Jan 30 12:00:11 storage2 kernel: [ 5159.746761] [<ffffffff8160fca0>] ? gs_change+0x13/0x13
Jan 30 12:56:11 storage2 kernel: [ 8519.003667] INFO: task xfsaild/md127:1559 blocked for more than 120 seconds.
Jan 30 12:56:11 storage2 kernel: [ 8519.008584] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 30 12:56:11 storage2 kernel: [ 8519.018149] xfsaild/md127 D ffffffff81805120 0 1559 2 0x00000000
Jan 30 12:56:11 storage2 kernel: [ 8519.018153] ffff88020789dcb0 0000000000000046 0000000000000000 0000000000000000
Jan 30 12:56:11 storage2 kernel: [ 8519.018155] ffff88020789dfd8 ffff88020789dfd8 ffff88020789dfd8 0000000000012a40
Jan 30 12:56:11 storage2 kernel: [ 8519.018158] ffff880212d28000 ffff8802117dc560 ffff88020789dca0 000000000000007e
Jan 30 12:56:11 storage2 kernel: [ 8519.018160] Call Trace:
Jan 30 12:56:11 storage2 kernel: [ 8519.018167] [<ffffffff8160492f>] schedule+0x3f/0x60
Jan 30 12:56:11 storage2 kernel: [ 8519.018183] [<ffffffffa02a0147>] xlog_cil_force_lsn+0xf7/0x120 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.018185] [<ffffffff810573a0>] ? try_to_wake_up+0x200/0x200
Jan 30 12:56:11 storage2 kernel: [ 8519.018196] [<ffffffffa029e4c8>] _xfs_log_force+0x68/0x2a0 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.018208] [<ffffffffa02b5fdb>] ? xfs_buf_unlock+0x3b/0xc0 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.018218] [<ffffffffa029e718>] xfs_log_force+0x18/0x40 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.018229] [<ffffffffa02ac804>] xfsaild_push+0x394/0x3a0 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.018239] [<ffffffffa02ac872>] xfsaild+0x62/0xc0 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.018249] [<ffffffffa02ac810>] ? xfsaild_push+0x3a0/0x3a0 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.018252] [<ffffffff81080bbc>] kthread+0x8c/0xa0
Jan 30 12:56:11 storage2 kernel: [ 8519.018254] [<ffffffff8160fca4>] kernel_thread_helper+0x4/0x10
Jan 30 12:56:11 storage2 kernel: [ 8519.018256] [<ffffffff81080b30>] ? flush_kthread_worker+0xa0/0xa0
Jan 30 12:56:11 storage2 kernel: [ 8519.018258] [<ffffffff8160fca0>] ? gs_change+0x13/0x13
Jan 30 12:56:11 storage2 kernel: [ 8519.018260] INFO: task kworker/0:0:1796 blocked for more than 120 seconds.
Jan 30 12:56:11 storage2 kernel: [ 8519.023397] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 30 12:56:11 storage2 kernel: [ 8519.033551] kworker/0:0 D ffffffff81805120 0 1796 2 0x00000000
Jan 30 12:56:11 storage2 kernel: [ 8519.033553] ffff88010c9ebb00 0000000000000046 0000000000000004 0000000000000000
Jan 30 12:56:11 storage2 kernel: [ 8519.033555] ffff88010c9ebfd8 ffff88010c9ebfd8 ffff88010c9ebfd8 0000000000012a40
Jan 30 12:56:11 storage2 kernel: [ 8519.033557] ffffffff81c0b020 ffff880212cedc80 ffff88010c9ebaf0 ffff8802129c2998
Jan 30 12:56:11 storage2 kernel: [ 8519.033560] Call Trace:
Jan 30 12:56:11 storage2 kernel: [ 8519.033562] [<ffffffff8160492f>] schedule+0x3f/0x60
Jan 30 12:56:11 storage2 kernel: [ 8519.033572] [<ffffffffa029f60c>] xlog_wait+0x6c/0x90 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.033574] [<ffffffff810573a0>] ? try_to_wake_up+0x200/0x200
Jan 30 12:56:11 storage2 kernel: [ 8519.033583] [<ffffffffa029f93f>] xlog_cil_push+0x30f/0x380 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.033586] [<ffffffff81086b92>] ? up+0x32/0x50
Jan 30 12:56:11 storage2 kernel: [ 8519.033595] [<ffffffffa02a015e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.033598] [<ffffffff811532df>] ? kmem_cache_free+0x2f/0x120
Jan 30 12:56:11 storage2 kernel: [ 8519.033607] [<ffffffffa029e797>] _xfs_log_force_lsn+0x57/0x2e0 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.033609] [<ffffffff811532df>] ? kmem_cache_free+0x2f/0x120
Jan 30 12:56:11 storage2 kernel: [ 8519.033619] [<ffffffffa02abf8d>] _xfs_trans_commit+0x29d/0x2b0 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.033630] [<ffffffffa02bfec0>] ? xfs_sync_inode_attr+0xf0/0xf0 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.033640] [<ffffffffa028f0c4>] xfs_fs_log_dummy+0x64/0x90 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.033651] [<ffffffffa02bff44>] xfs_sync_worker+0x84/0x90 [xfs]
Jan 30 12:56:11 storage2 kernel: [ 8519.033654] [<ffffffff8107b9ca>] process_one_work+0x11a/0x480
Jan 30 12:56:11 storage2 kernel: [ 8519.033656] [<ffffffff8107c775>] worker_thread+0x165/0x370
Jan 30 12:56:11 storage2 kernel: [ 8519.033657] [<ffffffff8107c610>] ? manage_workers.isra.30+0x130/0x130
Jan 30 12:56:11 storage2 kernel: [ 8519.033660] [<ffffffff81080bbc>] kthread+0x8c/0xa0
Jan 30 12:56:11 storage2 kernel: [ 8519.033661] [<ffffffff8160fca4>] kernel_thread_helper+0x4/0x10
Jan 30 12:56:11 storage2 kernel: [ 8519.033664] [<ffffffff81080b30>] ? flush_kthread_worker+0xa0/0xa0
Jan 30 12:56:11 storage2 kernel: [ 8519.033665] [<ffffffff8160fca0>] ? gs_change+0x13/0x13
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
next reply other threads:[~2012-01-31 10:54 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-01-31 10:54 Brian Candler [this message]
-- strict thread matches above, loose matches on Subject: below --
2012-01-30 0:20 xfs task blocked for more than 120 seconds Sami Liedes
2012-01-30 1:05 ` Dave Chinner
2012-01-30 22:35 ` Sami Liedes
[not found] ` <2504_1327964557_4F27218D_2504_92_2_20120130223527.GH10174@sli.dy.fi>
2012-01-31 23:30 ` Sami Liedes
2012-02-02 0:26 ` Dave Chinner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120131105408.GA46226@nsrc.org \
--to=b.candler@pobox.com \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox