* XFS deadlock while running generic/019 test
@ 2016-11-04 18:23 Jan Kara
2016-11-04 18:28 ` Darrick J. Wong
0 siblings, 1 reply; 4+ messages in thread
From: Jan Kara @ 2016-11-04 18:23 UTC (permalink / raw)
To: linux-xfs
[-- Attachment #1: Type: text/plain, Size: 2237 bytes --]
Hi,
XFS has deadlocked for me while running generic/019 test on umount. The
command is hanging in:
[ 5429.070390] INFO: task umount:224228 blocked for more than 480 seconds.
[ 5429.088153] Tainted: G W 4.9.0-rc3-64-default+ #29
[ 5429.105523] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5429.126043] umount D ffffffff81618c9d 0 224228 222841 0x00000000
[ 5429.126055] ffff881197089000 0000000000000000 ffff881168dcd2c0 ffff8817da5e0480
[ 5429.126062] ffff8817dd918480 ffffc90033c83d58 ffffffff81618c9d ffffffff810ad30a
[ 5429.126073] ffff8817dcdc0000 ffff881700000047 ffff881168dcd2c0 ffff8811a72ff840
[ 5429.126096] Call Trace:
[ 5429.126125] [<ffffffff81618c9d>] ? __schedule+0x22d/0x6c0
[ 5429.126144] [<ffffffff810ad30a>] ? try_to_wake_up+0x4a/0x380
[ 5429.126155] [<ffffffff81619166>] schedule+0x36/0x80
[ 5429.126271] [<ffffffffa090e0dc>] xfs_ail_push_all_sync+0xac/0xf0 [xfs]
[ 5429.126285] [<ffffffff810c7540>] ? prepare_to_wait_event+0x110/0x110
[ 5429.126386] [<ffffffffa08f45c3>] xfs_unmountfs+0x73/0x1c0 [xfs]
[ 5429.126485] [<ffffffffa08f8ea2>] xfs_fs_put_super+0x32/0x90 [xfs]
[ 5429.126503] [<ffffffff81214a8f>] generic_shutdown_super+0x6f/0x100
[ 5429.126513] [<ffffffff81214ba7>] kill_block_super+0x27/0x70
[ 5429.126523] [<ffffffff812144a3>] deactivate_locked_super+0x43/0x70
[ 5429.126533] [<ffffffff81214516>] deactivate_super+0x46/0x60
[ 5429.126544] [<ffffffff81231c3f>] cleanup_mnt+0x3f/0x80
[ 5429.126554] [<ffffffff81231cd2>] __cleanup_mnt+0x12/0x20
[ 5429.126565] [<ffffffff8109fe23>] task_work_run+0x83/0xb0
[ 5429.126575] [<ffffffff8107c407>] exit_to_usermode_loop+0x66/0x92
[ 5429.126586] [<ffffffff8100391a>] syscall_return_slowpath+0x9a/0xb0
[ 5429.126602] [<ffffffff8161d904>] entry_SYSCALL_64_fastpath+0xa7/0xa9
In between these stuck messages, there are:
[ 5443.405374] XFS (ram1): xfs_log_force: error -5 returned.
errors. Full dmesg from the xfstest run is attached. The problem has been
triggered on XFS mounted with 'dax' mount option over 20G ramdisks. The
kernel is 4.9-rc3 + Ross' DAX PMD patches (latest series) + my ext4 iomap
conversion patches.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
[-- Attachment #2: dmesg.gz --]
[-- Type: application/x-gzip, Size: 17750 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: XFS deadlock while running generic/019 test
2016-11-04 18:23 XFS deadlock while running generic/019 test Jan Kara
@ 2016-11-04 18:28 ` Darrick J. Wong
2016-11-05 21:46 ` Dave Chinner
2016-11-05 22:34 ` Jan Kara
0 siblings, 2 replies; 4+ messages in thread
From: Darrick J. Wong @ 2016-11-04 18:28 UTC (permalink / raw)
To: Jan Kara; +Cc: linux-xfs
On Fri, Nov 04, 2016 at 07:23:08PM +0100, Jan Kara wrote:
> Hi,
>
> XFS has deadlocked for me while running generic/019 test on umount. The
> command is hanging in:
>
> [ 5429.070390] INFO: task umount:224228 blocked for more than 480 seconds.
> [ 5429.088153] Tainted: G W 4.9.0-rc3-64-default+ #29
> [ 5429.105523] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 5429.126043] umount D ffffffff81618c9d 0 224228 222841 0x00000000
> [ 5429.126055] ffff881197089000 0000000000000000 ffff881168dcd2c0 ffff8817da5e0480
> [ 5429.126062] ffff8817dd918480 ffffc90033c83d58 ffffffff81618c9d ffffffff810ad30a
> [ 5429.126073] ffff8817dcdc0000 ffff881700000047 ffff881168dcd2c0 ffff8811a72ff840
> [ 5429.126096] Call Trace:
> [ 5429.126125] [<ffffffff81618c9d>] ? __schedule+0x22d/0x6c0
> [ 5429.126144] [<ffffffff810ad30a>] ? try_to_wake_up+0x4a/0x380
> [ 5429.126155] [<ffffffff81619166>] schedule+0x36/0x80
> [ 5429.126271] [<ffffffffa090e0dc>] xfs_ail_push_all_sync+0xac/0xf0 [xfs]
> [ 5429.126285] [<ffffffff810c7540>] ? prepare_to_wait_event+0x110/0x110
> [ 5429.126386] [<ffffffffa08f45c3>] xfs_unmountfs+0x73/0x1c0 [xfs]
> [ 5429.126485] [<ffffffffa08f8ea2>] xfs_fs_put_super+0x32/0x90 [xfs]
> [ 5429.126503] [<ffffffff81214a8f>] generic_shutdown_super+0x6f/0x100
> [ 5429.126513] [<ffffffff81214ba7>] kill_block_super+0x27/0x70
> [ 5429.126523] [<ffffffff812144a3>] deactivate_locked_super+0x43/0x70
> [ 5429.126533] [<ffffffff81214516>] deactivate_super+0x46/0x60
> [ 5429.126544] [<ffffffff81231c3f>] cleanup_mnt+0x3f/0x80
> [ 5429.126554] [<ffffffff81231cd2>] __cleanup_mnt+0x12/0x20
> [ 5429.126565] [<ffffffff8109fe23>] task_work_run+0x83/0xb0
> [ 5429.126575] [<ffffffff8107c407>] exit_to_usermode_loop+0x66/0x92
> [ 5429.126586] [<ffffffff8100391a>] syscall_return_slowpath+0x9a/0xb0
> [ 5429.126602] [<ffffffff8161d904>] entry_SYSCALL_64_fastpath+0xa7/0xa9
>
> In between these stuck messages, there are:
>
> [ 5443.405374] XFS (ram1): xfs_log_force: error -5 returned.
>
> errors. Full dmesg from the xfstest run is attached. The problem has been
> triggered on XFS mounted with 'dax' mount option over 20G ramdisks. The
> kernel is 4.9-rc3 + Ross' DAX PMD patches (latest series) + my ext4 iomap
> conversion patches.
Hmmm, I posted a patch[1] for a similar unmount hang that Dave found
via generic/388. No idea if that'll fix this problem, but I suppose
it's worth a shot if you can get it to reproduce quickly/consistently.
(AFAIK it's still soaking in linux-next...)
--D
[1] https://www.spinics.net/lists/linux-xfs/msg01737.html
>
> Honza
> --
> Jan Kara <jack@suse.com>
> SUSE Labs, CR
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: XFS deadlock while running generic/019 test
2016-11-04 18:28 ` Darrick J. Wong
@ 2016-11-05 21:46 ` Dave Chinner
2016-11-05 22:34 ` Jan Kara
1 sibling, 0 replies; 4+ messages in thread
From: Dave Chinner @ 2016-11-05 21:46 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: Jan Kara, linux-xfs
On Fri, Nov 04, 2016 at 11:28:17AM -0700, Darrick J. Wong wrote:
> On Fri, Nov 04, 2016 at 07:23:08PM +0100, Jan Kara wrote:
> > Hi,
> >
> > XFS has deadlocked for me while running generic/019 test on umount. The
> > command is hanging in:
> >
> > [ 5429.070390] INFO: task umount:224228 blocked for more than 480 seconds.
> > [ 5429.088153] Tainted: G W 4.9.0-rc3-64-default+ #29
> > [ 5429.105523] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 5429.126043] umount D ffffffff81618c9d 0 224228 222841 0x00000000
> > [ 5429.126055] ffff881197089000 0000000000000000 ffff881168dcd2c0 ffff8817da5e0480
> > [ 5429.126062] ffff8817dd918480 ffffc90033c83d58 ffffffff81618c9d ffffffff810ad30a
> > [ 5429.126073] ffff8817dcdc0000 ffff881700000047 ffff881168dcd2c0 ffff8811a72ff840
> > [ 5429.126096] Call Trace:
> > [ 5429.126125] [<ffffffff81618c9d>] ? __schedule+0x22d/0x6c0
> > [ 5429.126144] [<ffffffff810ad30a>] ? try_to_wake_up+0x4a/0x380
> > [ 5429.126155] [<ffffffff81619166>] schedule+0x36/0x80
> > [ 5429.126271] [<ffffffffa090e0dc>] xfs_ail_push_all_sync+0xac/0xf0 [xfs]
> > [ 5429.126285] [<ffffffff810c7540>] ? prepare_to_wait_event+0x110/0x110
> > [ 5429.126386] [<ffffffffa08f45c3>] xfs_unmountfs+0x73/0x1c0 [xfs]
> > [ 5429.126485] [<ffffffffa08f8ea2>] xfs_fs_put_super+0x32/0x90 [xfs]
> > [ 5429.126503] [<ffffffff81214a8f>] generic_shutdown_super+0x6f/0x100
> > [ 5429.126513] [<ffffffff81214ba7>] kill_block_super+0x27/0x70
> > [ 5429.126523] [<ffffffff812144a3>] deactivate_locked_super+0x43/0x70
> > [ 5429.126533] [<ffffffff81214516>] deactivate_super+0x46/0x60
> > [ 5429.126544] [<ffffffff81231c3f>] cleanup_mnt+0x3f/0x80
> > [ 5429.126554] [<ffffffff81231cd2>] __cleanup_mnt+0x12/0x20
> > [ 5429.126565] [<ffffffff8109fe23>] task_work_run+0x83/0xb0
> > [ 5429.126575] [<ffffffff8107c407>] exit_to_usermode_loop+0x66/0x92
> > [ 5429.126586] [<ffffffff8100391a>] syscall_return_slowpath+0x9a/0xb0
> > [ 5429.126602] [<ffffffff8161d904>] entry_SYSCALL_64_fastpath+0xa7/0xa9
> >
> > In between these stuck messages, there are:
> >
> > [ 5443.405374] XFS (ram1): xfs_log_force: error -5 returned.
> >
> > errors. Full dmesg from the xfstest run is attached. The problem has been
> > triggered on XFS mounted with 'dax' mount option over 20G ramdisks. The
> > kernel is 4.9-rc3 + Ross' DAX PMD patches (latest series) + my ext4 iomap
> > conversion patches.
>
> Hmmm, I posted a patch[1] for a similar unmount hang that Dave found
> via generic/388. No idea if that'll fix this problem, but I suppose
> it's worth a shot if you can get it to reproduce quickly/consistently.
>
> (AFAIK it's still soaking in linux-next...)
Argh. I committed it (it's in for-next), but then tagged the prior
commit when sending the -rc3 fixes to linus. I'll sort it.
-Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: XFS deadlock while running generic/019 test
2016-11-04 18:28 ` Darrick J. Wong
2016-11-05 21:46 ` Dave Chinner
@ 2016-11-05 22:34 ` Jan Kara
1 sibling, 0 replies; 4+ messages in thread
From: Jan Kara @ 2016-11-05 22:34 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: Jan Kara, linux-xfs
On Fri 04-11-16 11:28:17, Darrick J. Wong wrote:
> On Fri, Nov 04, 2016 at 07:23:08PM +0100, Jan Kara wrote:
> > Hi,
> >
> > XFS has deadlocked for me while running generic/019 test on umount. The
> > command is hanging in:
> >
> > [ 5429.070390] INFO: task umount:224228 blocked for more than 480 seconds.
> > [ 5429.088153] Tainted: G W 4.9.0-rc3-64-default+ #29
> > [ 5429.105523] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 5429.126043] umount D ffffffff81618c9d 0 224228 222841 0x00000000
> > [ 5429.126055] ffff881197089000 0000000000000000 ffff881168dcd2c0 ffff8817da5e0480
> > [ 5429.126062] ffff8817dd918480 ffffc90033c83d58 ffffffff81618c9d ffffffff810ad30a
> > [ 5429.126073] ffff8817dcdc0000 ffff881700000047 ffff881168dcd2c0 ffff8811a72ff840
> > [ 5429.126096] Call Trace:
> > [ 5429.126125] [<ffffffff81618c9d>] ? __schedule+0x22d/0x6c0
> > [ 5429.126144] [<ffffffff810ad30a>] ? try_to_wake_up+0x4a/0x380
> > [ 5429.126155] [<ffffffff81619166>] schedule+0x36/0x80
> > [ 5429.126271] [<ffffffffa090e0dc>] xfs_ail_push_all_sync+0xac/0xf0 [xfs]
> > [ 5429.126285] [<ffffffff810c7540>] ? prepare_to_wait_event+0x110/0x110
> > [ 5429.126386] [<ffffffffa08f45c3>] xfs_unmountfs+0x73/0x1c0 [xfs]
> > [ 5429.126485] [<ffffffffa08f8ea2>] xfs_fs_put_super+0x32/0x90 [xfs]
> > [ 5429.126503] [<ffffffff81214a8f>] generic_shutdown_super+0x6f/0x100
> > [ 5429.126513] [<ffffffff81214ba7>] kill_block_super+0x27/0x70
> > [ 5429.126523] [<ffffffff812144a3>] deactivate_locked_super+0x43/0x70
> > [ 5429.126533] [<ffffffff81214516>] deactivate_super+0x46/0x60
> > [ 5429.126544] [<ffffffff81231c3f>] cleanup_mnt+0x3f/0x80
> > [ 5429.126554] [<ffffffff81231cd2>] __cleanup_mnt+0x12/0x20
> > [ 5429.126565] [<ffffffff8109fe23>] task_work_run+0x83/0xb0
> > [ 5429.126575] [<ffffffff8107c407>] exit_to_usermode_loop+0x66/0x92
> > [ 5429.126586] [<ffffffff8100391a>] syscall_return_slowpath+0x9a/0xb0
> > [ 5429.126602] [<ffffffff8161d904>] entry_SYSCALL_64_fastpath+0xa7/0xa9
> >
> > In between these stuck messages, there are:
> >
> > [ 5443.405374] XFS (ram1): xfs_log_force: error -5 returned.
> >
> > errors. Full dmesg from the xfstest run is attached. The problem has been
> > triggered on XFS mounted with 'dax' mount option over 20G ramdisks. The
> > kernel is 4.9-rc3 + Ross' DAX PMD patches (latest series) + my ext4 iomap
> > conversion patches.
>
> Hmmm, I posted a patch[1] for a similar unmount hang that Dave found
> via generic/388. No idea if that'll fix this problem, but I suppose
> it's worth a shot if you can get it to reproduce quickly/consistently.
>
> (AFAIK it's still soaking in linux-next...)
> [1] https://www.spinics.net/lists/linux-xfs/msg01737.html
Thanks. This seems to have fixed the hang for me (only the xfstest still
seems to fail because of some unexpected errors in dmesg).
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-11-05 22:34 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-11-04 18:23 XFS deadlock while running generic/019 test Jan Kara
2016-11-04 18:28 ` Darrick J. Wong
2016-11-05 21:46 ` Dave Chinner
2016-11-05 22:34 ` Jan Kara
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).