* lockdep splat on 4.18.0
@ 2018-09-06 15:44 Jeff Mahoney
2018-09-06 23:40 ` Dave Chinner
0 siblings, 1 reply; 2+ messages in thread
From: Jeff Mahoney @ 2018-09-06 15:44 UTC (permalink / raw)
To: xfs
Hi folks -
I hit this lockdep splat on 4.18.0 this morning (the + in the version is
due to btrfs patch; xfs is unmodified). In my experience lockdep splats
involving mount are false positive, but Eric suggested I drop it here
just the same.
-Jeff
======================================================
WARNING: possible circular locking dependency detected
4.18.0-vanilla+ #8 Not tainted
------------------------------------------------------
kswapd0/56 is trying to acquire lock:
000000002f3c47dc (sb_internal){.+.+}, at: xfs_trans_alloc+0x19d/0x250 [xfs]
but task is already holding lock:
00000000e0553233 (fs_reclaim){+.+.}, at: __fs_reclaim_acquire+0x5/0x40
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (fs_reclaim){+.+.}:
lock_acquire+0xbd/0x220
__fs_reclaim_acquire+0x2c/0x40
kmem_cache_alloc+0x2b/0x320
kmem_zone_alloc+0x95/0x100 [xfs]
xfs_trans_alloc+0x6f/0x250 [xfs]
xlog_recover_process_intents+0x1f6/0x300 [xfs]
xlog_recover_finish+0x18/0xa0 [xfs]
xfs_log_mount_finish+0x6d/0x110 [xfs]
xfs_mountfs+0x6f0/0xa40 [xfs]
xfs_fs_fill_super+0x520/0x6e0 [xfs]
mount_bdev+0x187/0x1c0
mount_fs+0x3a/0x160
vfs_kern_mount+0x66/0x150
do_mount+0x1d9/0xcf0
ksys_mount+0x7e/0xd0
__x64_sys_mount+0x21/0x30
do_syscall_64+0x5d/0x1a0
entry_SYSCALL_64_after_hwframe+0x49/0xbe
-> #0 (sb_internal){.+.+}:
__lock_acquire+0x436/0x770
lock_acquire+0xbd/0x220
__sb_start_write+0x166/0x1d0
xfs_trans_alloc+0x19d/0x250 [xfs]
xfs_iomap_write_allocate+0x1d7/0x330 [xfs]
xfs_map_blocks+0x2d7/0x550 [xfs]
xfs_do_writepage+0x26b/0x7a0 [xfs]
xfs_vm_writepage+0x28/0x50 [xfs]
pageout.isra.51+0x1ca/0x450
shrink_page_list+0x811/0xe30
shrink_inactive_list+0x2e2/0x770
shrink_node_memcg+0x32d/0x750
shrink_node+0xc9/0x470
balance_pgdat+0x175/0x360
kswapd+0x181/0x5d0
kthread+0xf8/0x130
ret_from_fork+0x3a/0x50
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(fs_reclaim);
lock(sb_internal);
lock(fs_reclaim);
lock(sb_internal);
*** DEADLOCK ***
1 lock held by kswapd0/56:
#0: 00000000e0553233 (fs_reclaim){+.+.}, at: __fs_reclaim_acquire+0x5/0x40
stack backtrace:
CPU: 0 PID: 56 Comm: kswapd0 Kdump: loaded Not tainted 4.18.0-vanilla+ #8
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.0.0-prebuilt.qemu-project.org 04/01/2014
Call Trace:
dump_stack+0x78/0xb3
print_circular_bug.isra.37+0x1f3/0x200
check_prev_add.constprop.46+0x717/0x730
validate_chain.isra.41+0x652/0x980
__lock_acquire+0x436/0x770
lock_acquire+0xbd/0x220
? xfs_trans_alloc+0x19d/0x250 [xfs]
__sb_start_write+0x166/0x1d0
? xfs_trans_alloc+0x19d/0x250 [xfs]
xfs_trans_alloc+0x19d/0x250 [xfs]
xfs_iomap_write_allocate+0x1d7/0x330 [xfs]
? xfs_iext_lookup_extent+0x11f/0x140 [xfs]
? xfs_iunlock+0x132/0x2f0 [xfs]
? xfs_map_blocks+0x100/0x550 [xfs]
xfs_map_blocks+0x2d7/0x550 [xfs]
xfs_do_writepage+0x26b/0x7a0 [xfs]
xfs_vm_writepage+0x28/0x50 [xfs]
pageout.isra.51+0x1ca/0x450
shrink_page_list+0x811/0xe30
shrink_inactive_list+0x2e2/0x770
shrink_node_memcg+0x32d/0x750
shrink_node+0xc9/0x470
balance_pgdat+0x175/0x360
kswapd+0x181/0x5d0
? wait_woken+0x80/0x80
kthread+0xf8/0x130
? balance_pgdat+0x360/0x360
? kthread_delayed_work_timer_fn+0x80/0x80
ret_from_fork+0x3a/0x50
--
Jeff Mahoney
SUSE Labs
^ permalink raw reply [flat|nested] 2+ messages in thread* Re: lockdep splat on 4.18.0
2018-09-06 15:44 lockdep splat on 4.18.0 Jeff Mahoney
@ 2018-09-06 23:40 ` Dave Chinner
0 siblings, 0 replies; 2+ messages in thread
From: Dave Chinner @ 2018-09-06 23:40 UTC (permalink / raw)
To: Jeff Mahoney; +Cc: xfs
On Thu, Sep 06, 2018 at 11:44:07AM -0400, Jeff Mahoney wrote:
> Hi folks -
>
> I hit this lockdep splat on 4.18.0 this morning (the + in the version is
> due to btrfs patch; xfs is unmodified). In my experience lockdep splats
> involving mount are false positive, but Eric suggested I drop it here
> just the same.
Thanks Jeff!
tl;dr looks like a false positive we might be able to shut up by
changing the order of code in xfs_trans_alloc().
>
> -Jeff
>
> ======================================================
> WARNING: possible circular locking dependency detected
> 4.18.0-vanilla+ #8 Not tainted
> ------------------------------------------------------
> kswapd0/56 is trying to acquire lock:
> 000000002f3c47dc (sb_internal){.+.+}, at: xfs_trans_alloc+0x19d/0x250 [xfs]
>
> but task is already holding lock:
> 00000000e0553233 (fs_reclaim){+.+.}, at: __fs_reclaim_acquire+0x5/0x40
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #1 (fs_reclaim){+.+.}:
> lock_acquire+0xbd/0x220
> __fs_reclaim_acquire+0x2c/0x40
> kmem_cache_alloc+0x2b/0x320
> kmem_zone_alloc+0x95/0x100 [xfs]
> xfs_trans_alloc+0x6f/0x250 [xfs]
> xlog_recover_process_intents+0x1f6/0x300 [xfs]
> xlog_recover_finish+0x18/0xa0 [xfs]
> xfs_log_mount_finish+0x6d/0x110 [xfs]
> xfs_mountfs+0x6f0/0xa40 [xfs]
> xfs_fs_fill_super+0x520/0x6e0 [xfs]
> mount_bdev+0x187/0x1c0
> mount_fs+0x3a/0x160
> vfs_kern_mount+0x66/0x150
> do_mount+0x1d9/0xcf0
> ksys_mount+0x7e/0xd0
> __x64_sys_mount+0x21/0x30
> do_syscall_64+0x5d/0x1a0
> entry_SYSCALL_64_after_hwframe+0x49/0xbe
>
> -> #0 (sb_internal){.+.+}:
> __lock_acquire+0x436/0x770
> lock_acquire+0xbd/0x220
> __sb_start_write+0x166/0x1d0
> xfs_trans_alloc+0x19d/0x250 [xfs]
> xfs_iomap_write_allocate+0x1d7/0x330 [xfs]
> xfs_map_blocks+0x2d7/0x550 [xfs]
> xfs_do_writepage+0x26b/0x7a0 [xfs]
> xfs_vm_writepage+0x28/0x50 [xfs]
> pageout.isra.51+0x1ca/0x450
> shrink_page_list+0x811/0xe30
> shrink_inactive_list+0x2e2/0x770
> shrink_node_memcg+0x32d/0x750
> shrink_node+0xc9/0x470
> balance_pgdat+0x175/0x360
> kswapd+0x181/0x5d0
> kthread+0xf8/0x130
> ret_from_fork+0x3a/0x50
>
> other info that might help us debug this:
>
> Possible unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(fs_reclaim);
> lock(sb_internal);
> lock(fs_reclaim);
> lock(sb_internal);
Ok, looks like kswapd is doing direct writeback from reclaim (why
hasn't that been killed already?), which takes a freeze reference
before we start the transaction. Then, elsewhere, we do the normal
thing of taking a freeze reference, then allocating the transaction
structure via GFP_KERNEL, triggering then "memory reclaim lock
inversion".
It's not a deadlock - for anything to deadlock in this path, we have
to be in the middle of a freeze and have frozen the transaction
subsystem. Which we cannot do until we've cleaned all the dirty
cached pages in the filesystem and frozen all new writes. Which means
kswapd cannot enter this direct writeback path because we can't have
dirty pages on the filesystem.
So, yeah, yet another false positive.
I suspect we can shut it up by changing the order of operations in
xfs_trans_alloc(). I'll have a look at that.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2018-09-07 4:19 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-09-06 15:44 lockdep splat on 4.18.0 Jeff Mahoney
2018-09-06 23:40 ` Dave Chinner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).