* 5.6-rc1 lockdep warning during dm-thin shutdown
@ 2020-02-13 20:50 Theodore Y. Ts'o
2020-02-20 23:45 ` Theodore Y. Ts'o
0 siblings, 1 reply; 3+ messages in thread
From: Theodore Y. Ts'o @ 2020-02-13 20:50 UTC (permalink / raw)
To: dm-devel
I noticed this while running xfstests generic/347. The lockdep report is below:
[ 3742.062744] run fstests generic/347 at 2020-02-13 13:18:03
[ 3742.431761] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,block_validity
[ 3743.696630] EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: acl,user_xattr,block_validity
[ 3749.384483] device-mapper: thin: 253:4: reached low water mark for data device: sending event.
[ 3750.853156] device-mapper: thin: 253:4: switching pool to out-of-data-space (queue IO) mode
[ 3812.123242] device-mapper: thin: 253:4: switching pool to out-of-data-space (error IO) mode
[ 3812.138217] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 378 starting block 128192)
[ 3812.150804] buffer_io_error: 708 callbacks suppressed
[ 3812.150809] Buffer I/O error on device dm-5, logical block 128000
[ 3812.162553] Buffer I/O error on device dm-5, logical block 128001
[ 3812.169205] Buffer I/O error on device dm-5, logical block 128002
[ 3812.175793] Buffer I/O error on device dm-5, logical block 128003
[ 3812.183686] Buffer I/O error on device dm-5, logical block 128004
[ 3812.191467] Buffer I/O error on device dm-5, logical block 128005
[ 3812.199311] Buffer I/O error on device dm-5, logical block 128006
[ 3812.207204] Buffer I/O error on device dm-5, logical block 128007
[ 3812.214915] Buffer I/O error on device dm-5, logical block 128008
[ 3812.222699] Buffer I/O error on device dm-5, logical block 128009
[ 3812.240796] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 379 starting block 128448)
[ 3812.264563] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 380 starting block 128704)
[ 3812.287052] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 381 starting block 128960)
[ 3812.308866] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 382 starting block 129216)
[ 3812.330810] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 383 starting block 129472)
[ 3812.352898] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 384 starting block 129728)
[ 3812.373989] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 385 starting block 129984)
[ 3812.395096] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 386 starting block 130240)
[ 3812.416946] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 387 starting block 130496)
[ 3813.736289] device-mapper: thin: 253:4: switching pool to write mode
[ 3813.736304] device-mapper: thin: 253:4: growing the data device from 1000 to 1200 blocks
[ 3813.848386] device-mapper: thin: 253:4: reached low water mark for data device: sending event.
[ 3814.349795] ======================================================
[ 3814.356234] WARNING: possible circular locking dependency detected
[ 3814.362533] 5.5.0-xfstests-07635-gc96dceeabf76 #1477 Not tainted
[ 3814.368654] ------------------------------------------------------
[ 3814.374947] dmsetup/10207 is trying to acquire lock:
[ 3814.380079] ffffffff85489a78 (shrinker_rwsem){++++}, at: unregister_shrinker+0x22/0x80
[ 3814.388115]
but task is already holding lock:
[ 3814.394061] ffff9c184c8c95d0 (&pmd->root_lock){++++}, at: dm_pool_metadata_close+0xba/0x120
[ 3814.402530]
which lock already depends on the new lock.
[ 3814.411867]
the existing dependency chain (in reverse order) is:
[ 3814.419464]
-> #3 (&pmd->root_lock){++++}:
[ 3814.425321] lock_acquire+0x9a/0x160
[ 3814.429545] down_read+0x41/0x200
[ 3814.433542] dm_thin_find_block+0x2e/0x80
[ 3814.438590] thin_map+0x172/0x270
[ 3814.442548] __map_bio+0x42/0x210
[ 3814.446501] __split_and_process_non_flush+0xd6/0x150
[ 3814.452292] __split_and_process_bio+0xe0/0x380
[ 3814.457588] dm_process_bio+0x90/0x2b0
[ 3814.462838] dm_make_request+0x54/0x120
[ 3814.467596] generic_make_request+0xd8/0x400
[ 3814.472508] submit_bio+0x3c/0x140
[ 3814.476587] submit_bh_wbc+0x16d/0x190
[ 3814.480975] ext4_read_block_bitmap_nowait+0x41d/0x670
[ 3814.486896] ext4_mb_init_cache+0x163/0x760
[ 3814.491799] ext4_mb_init_group+0x19a/0x260
[ 3814.496759] ext4_mb_good_group+0x16a/0x180
[ 3814.501587] ext4_mb_regular_allocator+0x28f/0x400
[ 3814.507036] ext4_mb_new_blocks+0x57e/0xbe0
[ 3814.512016] ext4_ext_map_blocks+0xb32/0xf30
[ 3814.516992] ext4_map_blocks+0xef/0x580
[ 3814.521743] mpage_map_and_submit_extent+0xe7/0x510
[ 3814.527259] ext4_writepages+0x895/0xb10
[ 3814.531971] do_writepages+0x41/0xe0
[ 3814.536199] __filemap_fdatawrite_range+0xd2/0x120
[ 3814.541752] file_write_and_wait_range+0x71/0xc0
[ 3814.547100] ext4_sync_file+0x130/0x4b0
[ 3814.551580] do_fsync+0x38/0x70
[ 3814.555367] __x64_sys_fsync+0x10/0x20
[ 3814.559760] do_syscall_64+0x50/0x1f0
[ 3814.564195] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 3814.569882]
-> #2 (&ei->i_data_sem){++++}:
[ 3814.575580] lock_acquire+0x9a/0x160
[ 3814.580109] down_write+0x40/0x110
[ 3814.584145] ext4_truncate+0x3ce/0x510
[ 3814.588529] ext4_evict_inode+0x29a/0x5c0
[ 3814.593184] evict+0xd0/0x1a0
[ 3814.596952] vfs_rmdir+0x167/0x190
[ 3814.600990] do_rmdir+0x145/0x1b0
[ 3814.605303] do_syscall_64+0x50/0x1f0
[ 3814.609613] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 3814.615301]
-> #1 (jbd2_handle){++++}:
[ 3814.620744] lock_acquire+0x9a/0x160
[ 3814.625088] start_this_handle+0x226/0x680
[ 3814.629824] jbd2__journal_start+0xf7/0x270
[ 3814.634856] __ext4_journal_start_sb+0x85/0x1b0
[ 3814.640051] ext4_release_dquot+0x68/0xc0
[ 3814.644716] dqput.part.0+0x7f/0x210
[ 3814.648931] __dquot_drop+0x69/0x90
[ 3814.653495] ext4_clear_inode+0x35/0x90
[ 3814.658107] ext4_evict_inode+0x6f/0x5c0
[ 3814.662767] evict+0xd0/0x1a0
[ 3814.666365] dispose_list+0x48/0x70
[ 3814.673422] prune_icache_sb+0x54/0x80
[ 3814.677808] super_cache_scan+0x124/0x1a0
[ 3814.682454] do_shrink_slab+0x171/0x3a0
[ 3814.687036] shrink_slab+0x22c/0x2b0
[ 3814.691322] drop_slab_node+0x2e/0x50
[ 3814.695628] drop_slab+0x34/0x60
[ 3814.699511] drop_caches_sysctl_handler+0x70/0x80
[ 3814.704859] proc_sys_call_handler+0x19d/0x1c0
[ 3814.709944] vfs_write+0xde/0x1b0
[ 3814.713911] ksys_write+0x68/0xe0
[ 3814.718207] do_syscall_64+0x50/0x1f0
[ 3814.722512] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 3814.728895]
-> #0 (shrinker_rwsem){++++}:
[ 3814.734504] check_prev_add+0x8e/0x9f0
[ 3814.738897] __lock_acquire+0xcdf/0xf80
[ 3814.743431] lock_acquire+0x9a/0x160
[ 3814.747651] down_write+0x40/0x110
[ 3814.751806] unregister_shrinker+0x22/0x80
[ 3814.756646] dm_bufio_client_destroy+0xbf/0x220
[ 3814.761922] dm_block_manager_destroy+0x11/0x20
[ 3814.767098] dm_pool_metadata_close+0x113/0x120
[ 3814.772503] __pool_dec+0x96/0x120
[ 3814.776547] pool_dtr+0x2d/0x70
[ 3814.780574] dm_table_destroy+0x58/0x110
[ 3814.785137] __dm_destroy+0x159/0x1b0
[ 3814.789444] dev_remove+0xd3/0x110
[ 3814.793482] ctl_ioctl+0x1ab/0x3b0
[ 3814.797543] dm_ctl_ioctl+0xa/0x10
[ 3814.801608] ksys_ioctl+0x82/0xc0
[ 3814.805593] __x64_sys_ioctl+0x16/0x20
[ 3814.809991] do_syscall_64+0x50/0x1f0
[ 3814.814401] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 3814.820226]
other info that might help us debug this:
[ 3814.828343] Chain exists of:
shrinker_rwsem --> &ei->i_data_sem --> &pmd->root_lock
[ 3814.839823] Possible unsafe locking scenario:
[ 3814.845864] CPU0 CPU1
[ 3814.850625] ---- ----
[ 3814.855274] lock(&pmd->root_lock);
[ 3814.858962] lock(&ei->i_data_sem);
[ 3814.865215] lock(&pmd->root_lock);
[ 3814.871686] lock(shrinker_rwsem);
[ 3814.875377]
*** DEADLOCK ***
[ 3814.881412] 2 locks held by dmsetup/10207:
[ 3814.885637] #0: ffffffff86fc05f0 (&dm_thin_pool_table.mutex){+.+.}, at: pool_dtr+0x1d/0x70
[ 3814.894114] #1: ffff9c184c8c95d0 (&pmd->root_lock){++++}, at: dm_pool_metadata_close+0xba/0x120
[ 3814.903068]
stack backtrace:
[ 3814.907544] CPU: 1 PID: 10207 Comm: dmsetup Not tainted 5.5.0-xfstests-07635-gc96dceeabf76 #1477
[ 3814.916451] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
[ 3814.925915] Call Trace:
[ 3814.928485] dump_stack+0x71/0xa0
[ 3814.931929] check_noncircular+0x162/0x180
[ 3814.936144] check_prev_add+0x8e/0x9f0
[ 3814.940010] __lock_acquire+0xcdf/0xf80
[ 3814.943964] lock_acquire+0x9a/0x160
[ 3814.947733] ? unregister_shrinker+0x22/0x80
[ 3814.952123] down_write+0x40/0x110
[ 3814.955650] ? unregister_shrinker+0x22/0x80
[ 3814.960035] unregister_shrinker+0x22/0x80
[ 3814.964248] dm_bufio_client_destroy+0xbf/0x220
[ 3814.968898] dm_block_manager_destroy+0x11/0x20
[ 3814.973551] dm_pool_metadata_close+0x113/0x120
[ 3814.978307] __pool_dec+0x96/0x120
[ 3814.981829] pool_dtr+0x2d/0x70
[ 3814.985086] dm_table_destroy+0x58/0x110
[ 3814.989156] __dm_destroy+0x159/0x1b0
[ 3814.992939] dev_remove+0xd3/0x110
[ 3814.996494] ? remove_all+0x30/0x30
[ 3815.000136] ctl_ioctl+0x1ab/0x3b0
[ 3815.003672] dm_ctl_ioctl+0xa/0x10
[ 3815.007187] ksys_ioctl+0x82/0xc0
[ 3815.010614] __x64_sys_ioctl+0x16/0x20
[ 3815.014474] do_syscall_64+0x50/0x1f0
[ 3815.018257] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 3815.023440] RIP: 0033:0x7f2c9d82e427
[ 3815.027131] Code: 00 00 90 48 8b 05 69 aa 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 39 aa 0c 00 f7 d8 64 89 01 48
[ 3815.046014] RSP: 002b:00007ffee7570578 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
[ 3815.053715] RAX: ffffffffffffffda RBX: 00007f2c9d90ca7f RCX: 00007f2c9d82e427
[ 3815.061052] RDX: 00005625880b2b40 RSI: 00000000c138fd04 RDI: 0000000000000003
[ 3815.068302] RBP: 00007ffee7570630 R08: 00007f2c9d95b310 R09: 00007ffee75703e0
[ 3815.076018] R10: 00007f2c9d95a65a R11: 0000000000000202 R12: 00007f2c9d95a65a
[ 3815.083398] R13: 00007f2c9d95a65a R14: 00007f2c9d95a65a R15: 00007f2c9d95a65a
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: 5.6-rc1 lockdep warning during dm-thin shutdown
2020-02-13 20:50 5.6-rc1 lockdep warning during dm-thin shutdown Theodore Y. Ts'o
@ 2020-02-20 23:45 ` Theodore Y. Ts'o
2020-02-23 19:54 ` [PATCH] dm thin metadata: fix lockdep complaint Theodore Ts'o
0 siblings, 1 reply; 3+ messages in thread
From: Theodore Y. Ts'o @ 2020-02-20 23:45 UTC (permalink / raw)
To: dm-devel
Ping? It looks like I'm still seeing across the board failures for
generic/347 in 5.6-rc2 and it looks like it's pretty clearly a dm-thin
issue.
If this isn't going to be fixed soon, that's fine. I'm just going to
blacklist generic/347 in kvm-xfstests and gce-xfstests.
Thanks,
- Ted
On Thu, Feb 13, 2020 at 03:50:14PM -0500, Theodore Y. Ts'o wrote:
> I noticed this while running xfstests generic/347. The lockdep report is below:
>
> [ 3742.062744] run fstests generic/347 at 2020-02-13 13:18:03
> [ 3742.431761] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,block_validity
> [ 3743.696630] EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: acl,user_xattr,block_validity
> [ 3749.384483] device-mapper: thin: 253:4: reached low water mark for data device: sending event.
> [ 3750.853156] device-mapper: thin: 253:4: switching pool to out-of-data-space (queue IO) mode
> [ 3812.123242] device-mapper: thin: 253:4: switching pool to out-of-data-space (error IO) mode
> [ 3812.138217] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 378 starting block 128192)
> [ 3812.150804] buffer_io_error: 708 callbacks suppressed
> [ 3812.150809] Buffer I/O error on device dm-5, logical block 128000
> [ 3812.162553] Buffer I/O error on device dm-5, logical block 128001
> [ 3812.169205] Buffer I/O error on device dm-5, logical block 128002
> [ 3812.175793] Buffer I/O error on device dm-5, logical block 128003
> [ 3812.183686] Buffer I/O error on device dm-5, logical block 128004
> [ 3812.191467] Buffer I/O error on device dm-5, logical block 128005
> [ 3812.199311] Buffer I/O error on device dm-5, logical block 128006
> [ 3812.207204] Buffer I/O error on device dm-5, logical block 128007
> [ 3812.214915] Buffer I/O error on device dm-5, logical block 128008
> [ 3812.222699] Buffer I/O error on device dm-5, logical block 128009
> [ 3812.240796] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 379 starting block 128448)
> [ 3812.264563] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 380 starting block 128704)
> [ 3812.287052] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 381 starting block 128960)
> [ 3812.308866] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 382 starting block 129216)
> [ 3812.330810] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 383 starting block 129472)
> [ 3812.352898] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 384 starting block 129728)
> [ 3812.373989] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 385 starting block 129984)
> [ 3812.395096] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 386 starting block 130240)
> [ 3812.416946] EXT4-fs warning (device dm-5): ext4_end_bio:347: I/O error 3 writing to inode 387 starting block 130496)
> [ 3813.736289] device-mapper: thin: 253:4: switching pool to write mode
> [ 3813.736304] device-mapper: thin: 253:4: growing the data device from 1000 to 1200 blocks
> [ 3813.848386] device-mapper: thin: 253:4: reached low water mark for data device: sending event.
>
> [ 3814.349795] ======================================================
> [ 3814.356234] WARNING: possible circular locking dependency detected
> [ 3814.362533] 5.5.0-xfstests-07635-gc96dceeabf76 #1477 Not tainted
> [ 3814.368654] ------------------------------------------------------
> [ 3814.374947] dmsetup/10207 is trying to acquire lock:
> [ 3814.380079] ffffffff85489a78 (shrinker_rwsem){++++}, at: unregister_shrinker+0x22/0x80
> [ 3814.388115]
> but task is already holding lock:
> [ 3814.394061] ffff9c184c8c95d0 (&pmd->root_lock){++++}, at: dm_pool_metadata_close+0xba/0x120
> [ 3814.402530]
> which lock already depends on the new lock.
>
> [ 3814.411867]
> the existing dependency chain (in reverse order) is:
> [ 3814.419464]
> -> #3 (&pmd->root_lock){++++}:
> [ 3814.425321] lock_acquire+0x9a/0x160
> [ 3814.429545] down_read+0x41/0x200
> [ 3814.433542] dm_thin_find_block+0x2e/0x80
> [ 3814.438590] thin_map+0x172/0x270
> [ 3814.442548] __map_bio+0x42/0x210
> [ 3814.446501] __split_and_process_non_flush+0xd6/0x150
> [ 3814.452292] __split_and_process_bio+0xe0/0x380
> [ 3814.457588] dm_process_bio+0x90/0x2b0
> [ 3814.462838] dm_make_request+0x54/0x120
> [ 3814.467596] generic_make_request+0xd8/0x400
> [ 3814.472508] submit_bio+0x3c/0x140
> [ 3814.476587] submit_bh_wbc+0x16d/0x190
> [ 3814.480975] ext4_read_block_bitmap_nowait+0x41d/0x670
> [ 3814.486896] ext4_mb_init_cache+0x163/0x760
> [ 3814.491799] ext4_mb_init_group+0x19a/0x260
> [ 3814.496759] ext4_mb_good_group+0x16a/0x180
> [ 3814.501587] ext4_mb_regular_allocator+0x28f/0x400
> [ 3814.507036] ext4_mb_new_blocks+0x57e/0xbe0
> [ 3814.512016] ext4_ext_map_blocks+0xb32/0xf30
> [ 3814.516992] ext4_map_blocks+0xef/0x580
> [ 3814.521743] mpage_map_and_submit_extent+0xe7/0x510
> [ 3814.527259] ext4_writepages+0x895/0xb10
> [ 3814.531971] do_writepages+0x41/0xe0
> [ 3814.536199] __filemap_fdatawrite_range+0xd2/0x120
> [ 3814.541752] file_write_and_wait_range+0x71/0xc0
> [ 3814.547100] ext4_sync_file+0x130/0x4b0
> [ 3814.551580] do_fsync+0x38/0x70
> [ 3814.555367] __x64_sys_fsync+0x10/0x20
> [ 3814.559760] do_syscall_64+0x50/0x1f0
> [ 3814.564195] entry_SYSCALL_64_after_hwframe+0x49/0xbe
> [ 3814.569882]
> -> #2 (&ei->i_data_sem){++++}:
> [ 3814.575580] lock_acquire+0x9a/0x160
> [ 3814.580109] down_write+0x40/0x110
> [ 3814.584145] ext4_truncate+0x3ce/0x510
> [ 3814.588529] ext4_evict_inode+0x29a/0x5c0
> [ 3814.593184] evict+0xd0/0x1a0
> [ 3814.596952] vfs_rmdir+0x167/0x190
> [ 3814.600990] do_rmdir+0x145/0x1b0
> [ 3814.605303] do_syscall_64+0x50/0x1f0
> [ 3814.609613] entry_SYSCALL_64_after_hwframe+0x49/0xbe
> [ 3814.615301]
> -> #1 (jbd2_handle){++++}:
> [ 3814.620744] lock_acquire+0x9a/0x160
> [ 3814.625088] start_this_handle+0x226/0x680
> [ 3814.629824] jbd2__journal_start+0xf7/0x270
> [ 3814.634856] __ext4_journal_start_sb+0x85/0x1b0
> [ 3814.640051] ext4_release_dquot+0x68/0xc0
> [ 3814.644716] dqput.part.0+0x7f/0x210
> [ 3814.648931] __dquot_drop+0x69/0x90
> [ 3814.653495] ext4_clear_inode+0x35/0x90
> [ 3814.658107] ext4_evict_inode+0x6f/0x5c0
> [ 3814.662767] evict+0xd0/0x1a0
> [ 3814.666365] dispose_list+0x48/0x70
> [ 3814.673422] prune_icache_sb+0x54/0x80
> [ 3814.677808] super_cache_scan+0x124/0x1a0
> [ 3814.682454] do_shrink_slab+0x171/0x3a0
> [ 3814.687036] shrink_slab+0x22c/0x2b0
> [ 3814.691322] drop_slab_node+0x2e/0x50
> [ 3814.695628] drop_slab+0x34/0x60
> [ 3814.699511] drop_caches_sysctl_handler+0x70/0x80
> [ 3814.704859] proc_sys_call_handler+0x19d/0x1c0
> [ 3814.709944] vfs_write+0xde/0x1b0
> [ 3814.713911] ksys_write+0x68/0xe0
> [ 3814.718207] do_syscall_64+0x50/0x1f0
> [ 3814.722512] entry_SYSCALL_64_after_hwframe+0x49/0xbe
> [ 3814.728895]
> -> #0 (shrinker_rwsem){++++}:
> [ 3814.734504] check_prev_add+0x8e/0x9f0
> [ 3814.738897] __lock_acquire+0xcdf/0xf80
> [ 3814.743431] lock_acquire+0x9a/0x160
> [ 3814.747651] down_write+0x40/0x110
> [ 3814.751806] unregister_shrinker+0x22/0x80
> [ 3814.756646] dm_bufio_client_destroy+0xbf/0x220
> [ 3814.761922] dm_block_manager_destroy+0x11/0x20
> [ 3814.767098] dm_pool_metadata_close+0x113/0x120
> [ 3814.772503] __pool_dec+0x96/0x120
> [ 3814.776547] pool_dtr+0x2d/0x70
> [ 3814.780574] dm_table_destroy+0x58/0x110
> [ 3814.785137] __dm_destroy+0x159/0x1b0
> [ 3814.789444] dev_remove+0xd3/0x110
> [ 3814.793482] ctl_ioctl+0x1ab/0x3b0
> [ 3814.797543] dm_ctl_ioctl+0xa/0x10
> [ 3814.801608] ksys_ioctl+0x82/0xc0
> [ 3814.805593] __x64_sys_ioctl+0x16/0x20
> [ 3814.809991] do_syscall_64+0x50/0x1f0
> [ 3814.814401] entry_SYSCALL_64_after_hwframe+0x49/0xbe
> [ 3814.820226]
> other info that might help us debug this:
>
> [ 3814.828343] Chain exists of:
> shrinker_rwsem --> &ei->i_data_sem --> &pmd->root_lock
>
> [ 3814.839823] Possible unsafe locking scenario:
>
> [ 3814.845864] CPU0 CPU1
> [ 3814.850625] ---- ----
> [ 3814.855274] lock(&pmd->root_lock);
> [ 3814.858962] lock(&ei->i_data_sem);
> [ 3814.865215] lock(&pmd->root_lock);
> [ 3814.871686] lock(shrinker_rwsem);
> [ 3814.875377]
> *** DEADLOCK ***
>
> [ 3814.881412] 2 locks held by dmsetup/10207:
> [ 3814.885637] #0: ffffffff86fc05f0 (&dm_thin_pool_table.mutex){+.+.}, at: pool_dtr+0x1d/0x70
> [ 3814.894114] #1: ffff9c184c8c95d0 (&pmd->root_lock){++++}, at: dm_pool_metadata_close+0xba/0x120
> [ 3814.903068]
> stack backtrace:
> [ 3814.907544] CPU: 1 PID: 10207 Comm: dmsetup Not tainted 5.5.0-xfstests-07635-gc96dceeabf76 #1477
> [ 3814.916451] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> [ 3814.925915] Call Trace:
> [ 3814.928485] dump_stack+0x71/0xa0
> [ 3814.931929] check_noncircular+0x162/0x180
> [ 3814.936144] check_prev_add+0x8e/0x9f0
> [ 3814.940010] __lock_acquire+0xcdf/0xf80
> [ 3814.943964] lock_acquire+0x9a/0x160
> [ 3814.947733] ? unregister_shrinker+0x22/0x80
> [ 3814.952123] down_write+0x40/0x110
> [ 3814.955650] ? unregister_shrinker+0x22/0x80
> [ 3814.960035] unregister_shrinker+0x22/0x80
> [ 3814.964248] dm_bufio_client_destroy+0xbf/0x220
> [ 3814.968898] dm_block_manager_destroy+0x11/0x20
> [ 3814.973551] dm_pool_metadata_close+0x113/0x120
> [ 3814.978307] __pool_dec+0x96/0x120
> [ 3814.981829] pool_dtr+0x2d/0x70
> [ 3814.985086] dm_table_destroy+0x58/0x110
> [ 3814.989156] __dm_destroy+0x159/0x1b0
> [ 3814.992939] dev_remove+0xd3/0x110
> [ 3814.996494] ? remove_all+0x30/0x30
> [ 3815.000136] ctl_ioctl+0x1ab/0x3b0
> [ 3815.003672] dm_ctl_ioctl+0xa/0x10
> [ 3815.007187] ksys_ioctl+0x82/0xc0
> [ 3815.010614] __x64_sys_ioctl+0x16/0x20
> [ 3815.014474] do_syscall_64+0x50/0x1f0
> [ 3815.018257] entry_SYSCALL_64_after_hwframe+0x49/0xbe
> [ 3815.023440] RIP: 0033:0x7f2c9d82e427
> [ 3815.027131] Code: 00 00 90 48 8b 05 69 aa 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 39 aa 0c 00 f7 d8 64 89 01 48
> [ 3815.046014] RSP: 002b:00007ffee7570578 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
> [ 3815.053715] RAX: ffffffffffffffda RBX: 00007f2c9d90ca7f RCX: 00007f2c9d82e427
> [ 3815.061052] RDX: 00005625880b2b40 RSI: 00000000c138fd04 RDI: 0000000000000003
> [ 3815.068302] RBP: 00007ffee7570630 R08: 00007f2c9d95b310 R09: 00007ffee75703e0
> [ 3815.076018] R10: 00007f2c9d95a65a R11: 0000000000000202 R12: 00007f2c9d95a65a
> [ 3815.083398] R13: 00007f2c9d95a65a R14: 00007f2c9d95a65a R15: 00007f2c9d95a65a
^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH] dm thin metadata: fix lockdep complaint
2020-02-20 23:45 ` Theodore Y. Ts'o
@ 2020-02-23 19:54 ` Theodore Ts'o
0 siblings, 0 replies; 3+ messages in thread
From: Theodore Ts'o @ 2020-02-23 19:54 UTC (permalink / raw)
To: dm-devel; +Cc: Theodore Ts'o, snitzer, agk
[ 3934.173244] ======================================================
[ 3934.179572] WARNING: possible circular locking dependency detected
[ 3934.185884] 5.4.21-xfstests #1 Not tainted
[ 3934.190151] ------------------------------------------------------
[ 3934.196673] dmsetup/8897 is trying to acquire lock:
[ 3934.201688] ffffffffbce82b18 (shrinker_rwsem){++++}, at: unregister_shrinker+0x22/0x80
[ 3934.210268]
but task is already holding lock:
[ 3934.216489] ffff92a10cc5e1d0 (&pmd->root_lock){++++}, at: dm_pool_metadata_close+0xba/0x120
[ 3934.225083]
which lock already depends on the new lock.
[ 3934.564165] Chain exists of:
shrinker_rwsem --> &journal->j_checkpoint_mutex --> &pmd->root_lock
For a more detailed lockdep report, please see:
https://lore.kernel.org/r/20200220234519.GA620489@mit.edu
We shouldn't need to hold the lock while are just tearing down and
freeing the whole metadata pool structure.
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
---
drivers/md/dm-thin-metadata.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c
index fc9947d6210c..76b6b323bf4b 100644
--- a/drivers/md/dm-thin-metadata.c
+++ b/drivers/md/dm-thin-metadata.c
@@ -960,9 +960,9 @@ int dm_pool_metadata_close(struct dm_pool_metadata *pmd)
DMWARN("%s: __commit_transaction() failed, error = %d",
__func__, r);
}
+ pmd_write_unlock(pmd);
if (!pmd->fail_io)
__destroy_persistent_data_objects(pmd);
- pmd_write_unlock(pmd);
kfree(pmd);
return 0;
--
2.24.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
end of thread, other threads:[~2020-02-23 19:54 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-02-13 20:50 5.6-rc1 lockdep warning during dm-thin shutdown Theodore Y. Ts'o
2020-02-20 23:45 ` Theodore Y. Ts'o
2020-02-23 19:54 ` [PATCH] dm thin metadata: fix lockdep complaint Theodore Ts'o
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.