public inbox for linux-xfs@vger.kernel.org
 help / color / mirror / Atom feed
From: Brian Foster <bfoster@redhat.com>
To: "zhao.mingyue@h3c.com" <zhao.mingyue@h3c.com>
Cc: Xudonghai <xu.donghai@h3c.com>, "xfs@oss.sgi.com" <xfs@oss.sgi.com>
Subject: Re: umount hanging problem
Date: Tue, 15 Sep 2015 07:14:20 -0400	[thread overview]
Message-ID: <20150915111420.GB21323@bfoster.bfoster> (raw)
In-Reply-To: <1CB94550540AE44E9CAAE8CD5AF40BF846CEDB68@H3CMLB12-EX.srv.huawei-3com.com>

On Tue, Sep 15, 2015 at 02:59:53AM +0000, zhao.mingyue@h3c.com wrote:
> Hi,everyone
> 
>  I have some question about XFS FILEsystem，can someone help me sovle my problem ？
> 

What kernel?

>  I built a Oracle database on a rbd block which had been created xfs filesystem ，then I removed the rbd block ，then the file directory of the oracle was broken and couldn’t be used，next，I recover the rbd block
> 
> and umount the file directory of the oracle ，what happed next is the question which I was confused ，the process of the umount is hunging there and couldn’t be killed，how to solve the problem ？？
> 

It's waiting to flush everything out that was previously written to the
log and sitting in the AIL list. It sounds like the filesystem shutdown
because the underlying block device was torn down before the filesystem
was unmounted, which is not the way to do things. ;) Recent kernels have
had issues with extent freeing logging such that EFI/EFD objects are not
reference counted correctly and can sit in the AIL and pin it
indefinitely, particularly on races with fs shutdown.

Fixes for (hopefully) most of these problems are pending in 4.3. For the
time being you'll probably have to reboot to recover. Also note that
this should never happen except due to some other crash, caused in this
case by breaking down the block device before unmounting the fs.

Brian

> thanks！
> 
> 
> 
> 
> part of the log is as follows：
> 
> Sep 11 10:22:29 localhost systemd: Started Fingerprint Authentication Daemon.
> Sep 11 10:22:29 localhost fprintd: Launching FprintObject
> Sep 11 10:22:29 localhost fprintd: ** Message: D-Bus service launched with name: net.reactivated.Fprint
> Sep 11 10:22:29 localhost fprintd: ** Message: entering main loop
> Sep 11 10:22:39 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:22:47 localhost systemd-logind: Removed session 1.
> Sep 11 10:22:59 localhost fprintd: ** Message: No devices in use, exit
> Sep 11 10:23:09 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:23:19 localhost systemd-logind: Removed session 136.
> Sep 11 10:23:39 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:23:44 localhost kernel: INFO: task umount:27950 blocked for more than 120 seconds.
> Sep 11 10:23:44 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 11 10:23:44 localhost kernel: umount          D ffff880e1faf3680     0 27950  27775 0x00000084
> Sep 11 10:23:44 localhost kernel: ffff880dd3247d90 0000000000000086 ffff8806f730db00 ffff880dd3247fd8
> Sep 11 10:23:44 localhost kernel: ffff880dd3247fd8 ffff880dd3247fd8 ffff8806f730db00 ffff8806e7ceae80
> Sep 11 10:23:44 localhost kernel: ffff8806cb888b48 ffff8806e7ceaec0 ffff8806e7ceaee8 ffff8806e7ceae90
> Sep 11 10:23:44 localhost kernel: Call Trace:
> Sep 11 10:23:44 localhost kernel: [<ffffffff81609259>] schedule+0x29/0x70
> Sep 11 10:23:44 localhost kernel: [<ffffffffa02f1ac1>] xfs_ail_push_all_sync+0xc1/0x110 [xfs]
> Sep 11 10:23:44 localhost kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
> Sep 11 10:23:44 localhost kernel: [<ffffffffa02a1c48>] xfs_unmountfs+0x68/0x160 [xfs]
> Sep 11 10:23:44 localhost kernel: [<ffffffffa02a24eb>] ? xfs_mru_cache_destroy+0x6b/0x90 [xfs]
> Sep 11 10:23:44 localhost kernel: [<ffffffffa02a3701>] xfs_fs_put_super+0x21/0x60 [xfs]
> Sep 11 10:23:44 localhost kernel: [<ffffffff811c8936>] generic_shutdown_super+0x56/0xe0
> Sep 11 10:23:44 localhost kernel: [<ffffffff811c8c17>] kill_block_super+0x27/0x70
> Sep 11 10:23:44 localhost kernel: [<ffffffff811c8f4d>] deactivate_locked_super+0x3d/0x60
> Sep 11 10:23:44 localhost kernel: [<ffffffff811c9556>] deactivate_super+0x46/0x60
> Sep 11 10:23:44 localhost kernel: [<ffffffff811e6265>] mntput_no_expire+0xc5/0x120
> Sep 11 10:23:44 localhost kernel: [<ffffffff811e739f>] SyS_umount+0x9f/0x3c0
> Sep 11 10:23:44 localhost kernel: [<ffffffff81613da9>] system_call_fastpath+0x16/0x1b
> Sep 11 10:23:57 localhost systemd-logind: Removed session 133.
> Sep 11 10:24:09 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:24:39 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:25:09 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:25:39 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:25:44 localhost kernel: INFO: task umount:27950 blocked for more than 120 seconds.
> Sep 11 10:25:44 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 11 10:25:44 localhost kernel: umount          D ffff880e1faf3680     0 27950  27775 0x00000084
> Sep 11 10:25:44 localhost kernel: ffff880dd3247d90 0000000000000086 ffff8806f730db00 ffff880dd3247fd8
> Sep 11 10:25:44 localhost kernel: ffff880dd3247fd8 ffff880dd3247fd8 ffff8806f730db00 ffff8806e7ceae80
> Sep 11 10:25:44 localhost kernel: ffff8806cb888b48 ffff8806e7ceaec0 ffff8806e7ceaee8 ffff8806e7ceae90
> Sep 11 10:25:44 localhost kernel: Call Trace:
> Sep 11 10:25:44 localhost kernel: [<ffffffff81609259>] schedule+0x29/0x70
> Sep 11 10:25:44 localhost kernel: [<ffffffffa02f1ac1>] xfs_ail_push_all_sync+0xc1/0x110 [xfs]
> Sep 11 10:25:44 localhost kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
> Sep 11 10:25:44 localhost kernel: [<ffffffffa02a1c48>] xfs_unmountfs+0x68/0x160 [xfs]
> Sep 11 10:25:44 localhost kernel: [<ffffffffa02a24eb>] ? xfs_mru_cache_destroy+0x6b/0x90 [xfs]
> Sep 11 10:25:44 localhost kernel: [<ffffffffa02a3701>] xfs_fs_put_super+0x21/0x60 [xfs]
> Sep 11 10:25:44 localhost kernel: [<ffffffff811c8936>] generic_shutdown_super+0x56/0xe0
> Sep 11 10:25:44 localhost kernel: [<ffffffff811c8c17>] kill_block_super+0x27/0x70
> Sep 11 10:25:44 localhost kernel: [<ffffffff811c8f4d>] deactivate_locked_super+0x3d/0x60
> Sep 11 10:25:44 localhost kernel: [<ffffffff811c9556>] deactivate_super+0x46/0x60
> Sep 11 10:25:44 localhost kernel: [<ffffffff811e6265>] mntput_no_expire+0xc5/0x120
> Sep 11 10:25:44 localhost kernel: [<ffffffff811e739f>] SyS_umount+0x9f/0x3c0
> Sep 11 10:25:44 localhost kernel: [<ffffffff81613da9>] system_call_fastpath+0x16/0x1b
> Sep 11 10:26:05 localhost systemd-logind: New session 138 of user root.
> Sep 11 10:26:05 localhost systemd: Starting Session 138 of user root.
> Sep 11 10:26:05 localhost systemd: Started Session 138 of user root.
> Sep 11 10:26:09 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:26:29 localhost systemd-logind: New session 139 of user root.
> Sep 11 10:26:29 localhost systemd: Starting Session 139 of user root.
> Sep 11 10:26:29 localhost systemd: Started Session 139 of user root.
> Sep 11 10:26:32 localhost systemd-logind: Removed session 139.
> Sep 11 10:26:39 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:27:09 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:27:38 localhost goa[28775]: goa-daemon version 3.8.5 starting [main.c:113, main()]
> Sep 11 10:27:38 localhost goa[28782]: GoaKerberosIdentityManager: Using polling for change notification for credential cache type 'KEYRING' [goakerberosidentitymanager.c:1394, monitor_credentials_cache()]
> Sep 11 10:27:40 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:27:44 localhost kernel: INFO: task umount:27950 blocked for more than 120 seconds.
> Sep 11 10:27:44 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 11 10:27:44 localhost kernel: umount          D ffff880e1faf3680     0 27950  27775 0x00000084
> Sep 11 10:27:44 localhost kernel: ffff880dd3247d90 0000000000000086 ffff8806f730db00 ffff880dd3247fd8
> Sep 11 10:27:44 localhost kernel: ffff880dd3247fd8 ffff880dd3247fd8 ffff8806f730db00 ffff8806e7ceae80
> Sep 11 10:27:44 localhost kernel: ffff8806cb888b48 ffff8806e7ceaec0 ffff8806e7ceaee8 ffff8806e7ceae90
> Sep 11 10:27:44 localhost kernel: Call Trace:
> Sep 11 10:27:44 localhost kernel: [<ffffffff81609259>] schedule+0x29/0x70
> Sep 11 10:27:44 localhost kernel: [<ffffffffa02f1ac1>] xfs_ail_push_all_sync+0xc1/0x110 [xfs]
> Sep 11 10:27:44 localhost kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
> Sep 11 10:27:44 localhost kernel: [<ffffffffa02a1c48>] xfs_unmountfs+0x68/0x160 [xfs]
> Sep 11 10:27:44 localhost kernel: [<ffffffffa02a24eb>] ? xfs_mru_cache_destroy+0x6b/0x90 [xfs]
> Sep 11 10:27:44 localhost kernel: [<ffffffffa02a3701>] xfs_fs_put_super+0x21/0x60 [xfs]
> Sep 11 10:27:44 localhost kernel: [<ffffffff811c8936>] generic_shutdown_super+0x56/0xe0
> Sep 11 10:27:44 localhost kernel: [<ffffffff811c8c17>] kill_block_super+0x27/0x70
> Sep 11 10:27:44 localhost kernel: [<ffffffff811c8f4d>] deactivate_locked_super+0x3d/0x60
> Sep 11 10:27:44 localhost kernel: [<ffffffff811c9556>] deactivate_super+0x46/0x60
> Sep 11 10:27:44 localhost kernel: [<ffffffff811e6265>] mntput_no_expire+0xc5/0x120
> Sep 11 10:27:44 localhost kernel: [<ffffffff811e739f>] SyS_umount+0x9f/0x3c0
> Sep 11 10:27:44 localhost kernel: [<ffffffff81613da9>] system_call_fastpath+0x16/0x1b
> Sep 11 10:28:10 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:28:40 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:29:10 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:29:40 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:29:44 localhost kernel: INFO: task umount:27950 blocked for more than 120 seconds.
> Sep 11 10:29:44 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 11 10:29:44 localhost kernel: umount          D ffff880e1faf3680     0 27950  27775 0x00000084
> Sep 11 10:29:44 localhost kernel: ffff880dd3247d90 0000000000000086 ffff8806f730db00 ffff880dd3247fd8
> Sep 11 10:29:44 localhost kernel: ffff880dd3247fd8 ffff880dd3247fd8 ffff8806f730db00 ffff8806e7ceae80
> Sep 11 10:29:44 localhost kernel: ffff8806cb888b48 ffff8806e7ceaec0 ffff8806e7ceaee8 ffff8806e7ceae90
> Sep 11 10:29:44 localhost kernel: Call Trace:
> Sep 11 10:29:44 localhost kernel: [<ffffffff81609259>] schedule+0x29/0x70
> Sep 11 10:29:44 localhost kernel: [<ffffffffa02f1ac1>] xfs_ail_push_all_sync+0xc1/0x110 [xfs]
> Sep 11 10:29:44 localhost kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
> Sep 11 10:29:44 localhost kernel: [<ffffffffa02a1c48>] xfs_unmountfs+0x68/0x160 [xfs]
> Sep 11 10:29:44 localhost kernel: [<ffffffffa02a24eb>] ? xfs_mru_cache_destroy+0x6b/0x90 [xfs]
> Sep 11 10:29:44 localhost kernel: [<ffffffffa02a3701>] xfs_fs_put_super+0x21/0x60 [xfs]
> Sep 11 10:29:44 localhost kernel: [<ffffffff811c8936>] generic_shutdown_super+0x56/0xe0
> Sep 11 10:29:44 localhost kernel: [<ffffffff811c8c17>] kill_block_super+0x27/0x70
> Sep 11 10:29:44 localhost kernel: [<ffffffff811c8f4d>] deactivate_locked_super+0x3d/0x60
> Sep 11 10:29:44 localhost kernel: [<ffffffff811c9556>] deactivate_super+0x46/0x60
> Sep 11 10:29:44 localhost kernel: [<ffffffff811e6265>] mntput_no_expire+0xc5/0x120
> Sep 11 10:29:44 localhost kernel: [<ffffffff811e739f>] SyS_umount+0x9f/0x3c0
> Sep 11 10:29:44 localhost kernel: [<ffffffff81613da9>] system_call_fastpath+0x16/0x1b
> Sep 11 10:29:44 localhost kernel: INFO: task mount:28669 blocked for more than 120 seconds.
> Sep 11 10:29:44 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 11 10:29:44 localhost kernel: mount           D ffff880703cf3680     0 28669  27990 0x00000084
> Sep 11 10:29:44 localhost kernel: ffff8806ca59fc48 0000000000000082 ffff880dea8771c0 ffff8806ca59ffd8
> Sep 11 10:29:44 localhost kernel: ffff8806ca59ffd8 ffff8806ca59ffd8 ffff880dea8771c0 ffff880dea8771c0
> Sep 11 10:29:44 localhost kernel: ffff880dec81f068 ffff880dec81f070 ffffffff00000000 ffff880dec81f078
> Sep 11 10:29:44 localhost kernel: Call Trace:
> Sep 11 10:29:44 localhost kernel: [<ffffffff81609259>] schedule+0x29/0x70
> Sep 11 10:29:44 localhost kernel: [<ffffffff8160ab35>] rwsem_down_write_failed+0x115/0x220
> Sep 11 10:29:44 localhost kernel: [<ffffffff81201481>] ? __blkdev_get+0x221/0x4d0
> Sep 11 10:29:44 localhost kernel: [<ffffffff811c8830>] ? set_bdev_super+0x40/0x40
> Sep 11 10:29:44 localhost kernel: [<ffffffff812e29a3>] call_rwsem_down_write_failed+0x13/0x20
> Sep 11 10:29:44 localhost kernel: [<ffffffff8160864d>] ? down_write+0x2d/0x30
> Sep 11 10:29:44 localhost kernel: [<ffffffff811c917e>] grab_super+0x2e/0xa0
> Sep 11 10:29:44 localhost kernel: [<ffffffff811c9810>] sget+0x2a0/0x3d0
> Sep 11 10:29:44 localhost kernel: [<ffffffff811c87f0>] ? ns_test_super+0x20/0x20
> Sep 11 10:29:44 localhost kernel: [<ffffffff811c9b92>] mount_bdev+0xe2/0x1f0
> Sep 11 10:29:44 localhost kernel: [<ffffffffa02a4580>] ? xfs_parseargs+0xbf0/0xbf0 [xfs]
> Sep 11 10:29:44 localhost kernel: [<ffffffff812d6230>] ? ida_get_new_above+0x230/0x2a0
> Sep 11 10:29:44 localhost kernel: [<ffffffffa02a2775>] xfs_fs_mount+0x15/0x20 [xfs]
> Sep 11 10:29:44 localhost kernel: [<ffffffff811ca599>] mount_fs+0x39/0x1b0
> Sep 11 10:29:44 localhost kernel: [<ffffffff81178420>] ? __alloc_percpu+0x10/0x20
> Sep 11 10:29:44 localhost kernel: [<ffffffff811e5a6f>] vfs_kern_mount+0x5f/0xf0
> Sep 11 10:29:44 localhost kernel: [<ffffffff811e7fbe>] do_mount+0x24e/0xa40
> Sep 11 10:29:44 localhost kernel: [<ffffffff8117317b>] ? strndup_user+0x4b/0xf0
> Sep 11 10:29:44 localhost kernel: [<ffffffff811e8846>] SyS_mount+0x96/0xf0
> Sep 11 10:29:44 localhost kernel: [<ffffffff81613da9>] system_call_fastpath+0x16/0x1b
> Sep 11 10:30:01 localhost systemd: Starting Session 140 of user root.
> Sep 11 10:30:01 localhost systemd: Started Session 140 of user root.
> Sep 11 10:30:10 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:30:40 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:31:10 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:31:40 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> Sep 11 10:31:44 localhost kernel: INFO: task umount:27950 blocked for more than 120 seconds.
> Sep 11 10:31:44 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 11 10:31:44 localhost kernel: umount          D ffff880e1faf3680     0 27950  27775 0x00000084
> Sep 11 10:31:44 localhost kernel: ffff880dd3247d90 0000000000000086 ffff8806f730db00 ffff880dd3247fd8
> Sep 11 10:31:44 localhost kernel: ffff880dd3247fd8 ffff880dd3247fd8 ffff8806f730db00 ffff8806e7ceae80
> Sep 11 10:31:44 localhost kernel: ffff8806cb888b48 ffff8806e7ceaec0 ffff8806e7ceaee8 ffff8806e7ceae90
> Sep 11 10:31:44 localhost kernel: Call Trace:
> Sep 11 10:31:44 localhost kernel: [<ffffffff81609259>] schedule+0x29/0x70
> Sep 11 10:31:44 localhost kernel: [<ffffffffa02f1ac1>] xfs_ail_push_all_sync+0xc1/0x110 [xfs]
> Sep 11 10:31:44 localhost kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
> Sep 11 10:31:44 localhost kernel: [<ffffffffa02a1c48>] xfs_unmountfs+0x68/0x160 [xfs]
> Sep 11 10:31:44 localhost kernel: [<ffffffffa02a24eb>] ? xfs_mru_cache_destroy+0x6b/0x90 [xfs]
> Sep 11 10:31:44 localhost kernel: [<ffffffffa02a3701>] xfs_fs_put_super+0x21/0x60 [xfs]
> Sep 11 10:31:44 localhost kernel: [<ffffffff811c8936>] generic_shutdown_super+0x56/0xe0
> Sep 11 10:31:44 localhost kernel: [<ffffffff811c8c17>] kill_block_super+0x27/0x70
> Sep 11 10:31:44 localhost kernel: [<ffffffff811c8f4d>] deactivate_locked_super+0x3d/0x60
> Sep 11 10:31:44 localhost kernel: [<ffffffff811c9556>] deactivate_super+0x46/0x60
> Sep 11 10:31:44 localhost kernel: [<ffffffff811e6265>] mntput_no_expire+0xc5/0x120
> Sep 11 10:31:44 localhost kernel: [<ffffffff811e739f>] SyS_umount+0x9f/0x3c0
> Sep 11 10:31:44 localhost kernel: [<ffffffff81613da9>] system_call_fastpath+0x16/0x1b
> Sep 11 10:32:10 localhost kernel: XFS (dm-2): xfs_log_force: error 5 returned.
> -------------------------------------------------------------------------------------------------------------------------------------
> 本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出
> 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
> 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
> 邮件！
> This e-mail and its attachments contain confidential information from H3C, which is
> intended only for the person or entity whose address is listed above. Any use of the
> information contained herein in any way (including, but not limited to, total or partial
> disclosure, reproduction, or dissemination) by persons other than the intended
> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!

> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

  reply	other threads:[~2015-09-15 11:14 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-09-15  2:59 umount hanging problem zhao.mingyue
2015-09-15 11:14 ` Brian Foster [this message]
2015-09-15 12:59   ` 答复: " zhao.mingyue
2015-09-15 15:15     ` Brian Foster
2015-09-16 13:34       ` 答复: " zhao.mingyue

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20150915111420.GB21323@bfoster.bfoster \
    --to=bfoster@redhat.com \
    --cc=xfs@oss.sgi.com \
    --cc=xu.donghai@h3c.com \
    --cc=zhao.mingyue@h3c.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox