From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cantor2.suse.de ([195.135.220.15]:49461 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755110AbaCZRBh (ORCPT ); Wed, 26 Mar 2014 13:01:37 -0400 Message-ID: <533307ED.8030700@suse.com> Date: Wed, 26 Mar 2014 13:01:33 -0400 From: Jeff Mahoney MIME-Version: 1.0 To: dsterba@suse.cz, Sasha Levin , clm@fb.com, jbacik@fb.com, linux-btrfs@vger.kernel.org, LKML Subject: Re: btrfs: lock inversion between delayed_node->mutex and found->groups_sem References: <53239AE0.8030306@oracle.com> <20140317130546.GE29256@twin.jikos.cz> In-Reply-To: <20140317130546.GE29256@twin.jikos.cz> Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-btrfs-owner@vger.kernel.org List-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 3/17/14, 9:05 AM, David Sterba wrote: > On Fri, Mar 14, 2014 at 08:12:16PM -0400, Sasha Levin wrote: >> While fuzzing with trinity inside a KVM tools guest running the >> latest -next kernel I've stumbled on the following: >> >> [ 788.458756] CPU0 CPU1 [ 788.459188] >> ---- ---- [ 788.459625] >> lock(&found->groups_sem); [ 788.460041] >> local_irq_disable(); [ 788.460041] >> lock(&delayed_node->mutex); [ 788.460041] >> lock(&found->groups_sem); [ 788.460041] [ >> 788.460041] lock(&delayed_node->mutex); [ 788.460041] [ >> 788.460041] *** DEADLOCK *** [ 788.460041] [ 788.460041] 2 >> locks held by kswapd3/4199: > > I've once (3.14-rc5) seen the same warning also caused by > xfstests/generic/224 I think this is from my sysfs patches. We call kobject_add while holding the group_sem. kobject_add ultimately allocates with GFP_KERNEL, so it can enter reclaim. This particular case isn't dangerous, but it could hit while hot-adding a device. The fix should be pretty simple. - -Jeff > 2 locks held by 224/31203: #0: (shrinker_rwsem){++++..}, at: > [] shrink_slab+0x3d/0x110 #1: > (&type->s_umount_key#32){++++++}, at: [] > grab_super_passive+0x44/0x90 > > the shortest dependencies between 2nd lock and 1st lock: -> > (&found->groups_sem){+++++.} ops: 405561 { HARDIRQ-ON-W at: > [] __lock_acquire+0x7f6/0x1fb0 > [] lock_acquire+0x92/0x120 [] > down_write+0x5c/0xc0 [] > __link_block_group+0x46/0x130 [btrfs] [] > btrfs_read_block_groups+0x341/0x690 [btrfs] [] > open_ctree+0x1880/0x2310 [btrfs] [] > btrfs_mount+0x55b/0x860 [btrfs] [] > mount_fs+0x20/0xe0 [] vfs_kern_mount+0x76/0x160 > [] do_mount+0x31d/0x970 [] > SyS_mount+0x90/0xe0 [] > system_call_fastpath+0x16/0x1b HARDIRQ-ON-R at: > [] __lock_acquire+0x5e5/0x1fb0 > [] lock_acquire+0x92/0x120 [] > down_read+0x4c/0xa0 [] > btrfs_calc_num_tolerated_disk_barrier_failures+0x142/0x240 [btrfs] > [] open_ctree+0x189e/0x2310 [btrfs] > [] btrfs_mount+0x55b/0x860 [btrfs] > [] mount_fs+0x20/0xe0 [] > vfs_kern_mount+0x76/0x160 [] > do_mount+0x31d/0x970 [] SyS_mount+0x90/0xe0 > [] system_call_fastpath+0x16/0x1b SOFTIRQ-ON-W > at: [] __lock_acquire+0x82a/0x1fb0 > [] lock_acquire+0x92/0x120 [] > down_write+0x5c/0xc0 [] > __link_block_group+0x46/0x130 [btrfs] [] > btrfs_read_block_groups+0x341/0x690 [btrfs] [] > open_ctree+0x1880/0x2310 [btrfs] [] > btrfs_mount+0x55b/0x860 [btrfs] [] > mount_fs+0x20/0xe0 [] vfs_kern_mount+0x76/0x160 > [] do_mount+0x31d/0x970 [] > SyS_mount+0x90/0xe0 [] > system_call_fastpath+0x16/0x1b SOFTIRQ-ON-R at: > [] __lock_acquire+0x82a/0x1fb0 > [] lock_acquire+0x92/0x120 [] > down_read+0x4c/0xa0 [] > btrfs_calc_num_tolerated_disk_barrier_failures+0x142/0x240 [btrfs] > [] open_ctree+0x189e/0x2310 [btrfs] > [] btrfs_mount+0x55b/0x860 [btrfs] > [] mount_fs+0x20/0xe0 [] > vfs_kern_mount+0x76/0x160 [] > do_mount+0x31d/0x970 [] SyS_mount+0x90/0xe0 > [] system_call_fastpath+0x16/0x1b RECLAIM_FS-ON-W > at: [] mark_held_locks+0x8c/0x170 > [] lockdep_trace_alloc+0x8a/0xd0 > [] __kmalloc_track_caller+0x47/0x210 > [] kvasprintf+0x5b/0x90 [] > kobject_set_name_vargs+0x2a/0x70 [] > kobject_add+0x5a/0xb0 [] > __link_block_group+0xfd/0x130 [btrfs] [] > btrfs_read_block_groups+0x341/0x690 [btrfs] [] > open_ctree+0x1880/0x2310 [btrfs] [] > btrfs_mount+0x55b/0x860 [btrfs] [] > mount_fs+0x20/0xe0 [] vfs_kern_mount+0x76/0x160 > [] do_mount+0x31d/0x970 [] > SyS_mount+0x90/0xe0 [] > system_call_fastpath+0x16/0x1b INITIAL USE at: [] > __lock_acquire+0x354/0x1fb0 [] > lock_acquire+0x92/0x120 [] down_write+0x5c/0xc0 > [] __link_block_group+0x46/0x130 [btrfs] > [] btrfs_read_block_groups+0x341/0x690 [btrfs] > [] open_ctree+0x1880/0x2310 [btrfs] > [] btrfs_mount+0x55b/0x860 [btrfs] > [] mount_fs+0x20/0xe0 [] > vfs_kern_mount+0x76/0x160 [] > do_mount+0x31d/0x970 [] SyS_mount+0x90/0xe0 > [] system_call_fastpath+0x16/0x1b } ... key > at: [] __key.40139+0x0/0xfffffffffffe2920 > [btrfs] ... acquired at: [] > lock_acquire+0x92/0x120 [] down_read+0x4c/0xa0 > [] find_free_extent+0x7e8/0xbe0 [btrfs] > [] btrfs_reserve_extent+0xa4/0x130 [btrfs] > [] btrfs_alloc_free_block+0x103/0x4c0 [btrfs] > [] __btrfs_cow_block+0x124/0x550 [btrfs] > [] btrfs_cow_block+0x12b/0x1e0 [btrfs] > [] btrfs_search_slot+0x1d9/0xa00 [btrfs] > [] btrfs_insert_empty_items+0x7e/0xe0 [btrfs] > [] btrfs_insert_delayed_items+0x84/0x460 [btrfs] > [] __btrfs_run_delayed_items+0xd9/0x1f0 [btrfs] > [] btrfs_run_delayed_items+0x13/0x20 [btrfs] > [] btrfs_commit_transaction+0x3ae/0xa30 [btrfs] > [] btrfs_mksubvol+0x3a2/0x3b0 [btrfs] > [] btrfs_ioctl_snap_create_transid+0x1b6/0x1c0 > [btrfs] [] btrfs_ioctl_snap_create_v2+0xfe/0x140 > [btrfs] [] btrfs_ioctl+0x652/0x1940 [btrfs] > [] do_vfs_ioctl+0x91/0x560 [] > SyS_ioctl+0x57/0x90 [] > system_call_fastpath+0x16/0x1b > > -> (&delayed_node->mutex){+.+.-.} ops: 8191147 { HARDIRQ-ON-W at: > [] __lock_acquire+0x7f6/0x1fb0 > [] lock_acquire+0x92/0x120 [] > mutex_lock_nested+0x6e/0x3a0 [] > btrfs_delayed_update_inode+0x49/0x670 [btrfs] [] > btrfs_update_inode+0x6a/0x100 [btrfs] [] > btrfs_create+0x16e/0x220 [btrfs] [] > vfs_create+0x89/0xc0 [] do_last+0x794/0xd50 > [] path_openat+0xc7/0x620 [] > do_filp_open+0x4a/0xa0 [] > do_sys_open+0x11e/0x230 [] SyS_open+0x1e/0x20 > [] system_call_fastpath+0x16/0x1b SOFTIRQ-ON-W > at: [] __lock_acquire+0x82a/0x1fb0 > [] lock_acquire+0x92/0x120 [] > mutex_lock_nested+0x6e/0x3a0 [] > btrfs_delayed_update_inode+0x49/0x670 [btrfs] [] > btrfs_update_inode+0x6a/0x100 [btrfs] [] > btrfs_create+0x16e/0x220 [btrfs] [] > vfs_create+0x89/0xc0 [] do_last+0x794/0xd50 > [] path_openat+0xc7/0x620 [] > do_filp_open+0x4a/0xa0 [] > do_sys_open+0x11e/0x230 [] SyS_open+0x1e/0x20 > [] system_call_fastpath+0x16/0x1b IN-RECLAIM_FS-W > at: [] __lock_acquire+0x66c/0x1fb0 > [] lock_acquire+0x92/0x120 [] > mutex_lock_nested+0x6e/0x3a0 [] > __btrfs_release_delayed_node+0x4f/0x220 [btrfs] > [] btrfs_remove_delayed_node+0x24/0x30 [btrfs] > [] btrfs_evict_inode+0x2a7/0x550 [btrfs] > [] evict+0xb8/0x1c0 [] > dispose_list+0x4f/0x60 [] > prune_icache_sb+0x4c/0x60 [] > super_cache_scan+0x126/0x190 [] > shrink_slab_node+0x14e/0x2c0 [] > shrink_slab+0x78/0x110 [] > do_try_to_free_pages+0x24d/0x410 [] > try_to_free_pages+0xe4/0x190 [] > __alloc_pages_nodemask+0x68b/0xa60 [] > cache_alloc_refill+0x40c/0x7b0 [] > kmem_cache_alloc+0x1ef/0x200 [] > copy_process+0x13e/0x1910 [] do_fork+0x65/0x300 > [] SyS_clone+0x16/0x20 [] > stub_clone+0x69/0x90 INITIAL USE at: [] > __lock_acquire+0x354/0x1fb0 [] > lock_acquire+0x92/0x120 [] > mutex_lock_nested+0x6e/0x3a0 [] > btrfs_delayed_update_inode+0x49/0x670 [btrfs] [] > btrfs_update_inode+0x6a/0x100 [btrfs] [] > btrfs_create+0x16e/0x220 [btrfs] [] > vfs_create+0x89/0xc0 [] do_last+0x794/0xd50 > [] path_openat+0xc7/0x620 [] > do_filp_open+0x4a/0xa0 [] > do_sys_open+0x11e/0x230 [] SyS_open+0x1e/0x20 > [] system_call_fastpath+0x16/0x1b } ... key > at: [] __key.35416+0x0/0xfffffffffffde9b8 > [btrfs] ... acquired at: [] > check_usage_forwards+0xaa/0x120 [] > mark_lock+0x1a9/0x430 [] > __lock_acquire+0x66c/0x1fb0 [] > lock_acquire+0x92/0x120 [] > mutex_lock_nested+0x6e/0x3a0 [] > __btrfs_release_delayed_node+0x4f/0x220 [btrfs] > [] btrfs_remove_delayed_node+0x24/0x30 [btrfs] > [] btrfs_evict_inode+0x2a7/0x550 [btrfs] > [] evict+0xb8/0x1c0 [] > dispose_list+0x4f/0x60 [] > prune_icache_sb+0x4c/0x60 [] > super_cache_scan+0x126/0x190 [] > shrink_slab_node+0x14e/0x2c0 [] > shrink_slab+0x78/0x110 [] > do_try_to_free_pages+0x24d/0x410 [] > try_to_free_pages+0xe4/0x190 [] > __alloc_pages_nodemask+0x68b/0xa60 [] > cache_alloc_refill+0x40c/0x7b0 [] > kmem_cache_alloc+0x1ef/0x200 [] > copy_process+0x13e/0x1910 [] do_fork+0x65/0x300 > [] SyS_clone+0x16/0x20 [] > stub_clone+0x69/0x90 > > > stack backtrace: CPU: 1 PID: 31203 Comm: 224 Tainted: G W > 3.14.0-rc5-default #121 Hardware name: Intel Corporation Santa Rosa > platform/Matanzas, BIOS TSRSCRB1.86C.0047.B00.0610170821 10/17/06 > ffffffff82c37220 ffff88003faa9460 ffffffff819fd463 > 0000000000000002 ffffffff82c37220 ffff88003faa94b0 ffffffff810adc97 > ffff88003faa9500 ffffffff81f16ba4 ffff88003faa94cc ffff88003faa5730 > ffff88003faa94c0 Call Trace: [] > dump_stack+0x51/0x6e [] > print_irq_inversion_bug+0x1c7/0x210 [] > check_usage_forwards+0xaa/0x120 [] ? > print_irq_inversion_bug+0x210/0x210 [] > mark_lock+0x1a9/0x430 [] > __lock_acquire+0x66c/0x1fb0 [] ? > print_irq_inversion_bug+0x210/0x210 [] ? > check_irq_usage+0x9f/0xf0 [] ? > __lock_acquire+0x13ab/0x1fb0 [] ? > trace_hardirqs_off+0xd/0x10 [] ? > __btrfs_release_delayed_node+0x4f/0x220 [btrfs] > [] lock_acquire+0x92/0x120 [] ? > __btrfs_release_delayed_node+0x4f/0x220 [btrfs] > [] mutex_lock_nested+0x6e/0x3a0 > [] ? __btrfs_release_delayed_node+0x4f/0x220 > [btrfs] [] ? sched_clock_cpu+0xa8/0xd0 > [] ? __btrfs_release_delayed_node+0x4f/0x220 > [btrfs] [] ? lock_release_holdtime+0x3d/0x1c0 > [] __btrfs_release_delayed_node+0x4f/0x220 > [btrfs] [] btrfs_remove_delayed_node+0x24/0x30 > [btrfs] [] btrfs_evict_inode+0x2a7/0x550 [btrfs] > [] ? _raw_spin_unlock+0x2b/0x40 > [] evict+0xb8/0x1c0 [] ? > insert_inode_locked+0x1a0/0x1a0 [] > dispose_list+0x4f/0x60 [] > prune_icache_sb+0x4c/0x60 [] > super_cache_scan+0x126/0x190 [] > shrink_slab_node+0x14e/0x2c0 [] ? > local_clock+0x16/0x30 [] shrink_slab+0x78/0x110 > [] do_try_to_free_pages+0x24d/0x410 > [] try_to_free_pages+0xe4/0x190 > [] __alloc_pages_nodemask+0x68b/0xa60 > [] ? cache_alloc_refill+0x3ce/0x7b0 > [] cache_alloc_refill+0x40c/0x7b0 > [] ? copy_process+0x13e/0x1910 > [] kmem_cache_alloc+0x1ef/0x200 > [] copy_process+0x13e/0x1910 [] > ? sched_clock_cpu+0xa8/0xd0 [] ? > local_clock+0x16/0x30 [] ? > lock_release_holdtime+0x3d/0x1c0 [] ? > lock_release_non_nested+0x395/0x3e0 [] ? > might_fault+0x66/0xc0 [] do_fork+0x65/0x300 > [] ? might_fault+0x66/0xc0 [] ? > sysret_check+0x1b/0x56 [] SyS_clone+0x16/0x20 > [] stub_clone+0x69/0x90 [] ? > system_call_fastpath+0x16/0x1b -- To unsubscribe from this list: > send the line "unsubscribe linux-btrfs" in the body of a message to > majordomo@vger.kernel.org More majordomo info at > http://vger.kernel.org/majordomo-info.html > - -- Jeff Mahoney SUSE Labs -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) iQIcBAEBAgAGBQJTMwftAAoJEB57S2MheeWy5awP/jYQgrhE6vw/Vv/A02btb4Ey X9EhpYohUaT6MU8WZYamijOA4Kx8OR1wEZ3CFQ3auhD/YceCEeq47Y02PpCCEjzq O78IcKPPRt4vepGrUYkV3FKDpc0N190LG1/Wu1fOQGo44naFxX6z1eH/Vd/o8D+8 NUUC2nDhAGjKM937BuYfGZrqIZiOIuBfWynMNKNXW9x5HY2EbhHbjkyn6iUhssKm 7HSgQ/5upjK9Af+L5hEZWS0qCKjx4QXrj0D9ir5Vll/EOVn2dsL58z09bXk+veor RtWhk6cwrLA2Twwo3LCLSAIIVJwuaCGKIQdv3rjt6Qp541P7q24WK+MUqcfn1oTn YvrOWikWoh4mFxuMn0y6XvCg7LjlY49u+Z69OTWd50PrePjx/UPPrrtwNIf8sKUO vPvk7r5sI1sxIsNjqEn4UUeoR73SbyHzN060T/3GGdySk8Nz+G4LwOYuqP3q6H/x TnoELE0voAbwr4JvufRwQrId4jS4j80p67qhRcUHJkxoEn8NhiXvg4NhqXzDAyVg 6xyZ6mKvzxGfs6DhXqCN7F/gfuOlyxZpxjvj3ZLhPiHhNTdQXRl3CVPsuNg0CSEj DNI4mYpIInlZTEDjiWblW3+bN+hGtmKRshe56UEhEl5yGzZtMvNvcGcSRuyui/6z O1vbSR/A2u+5d2/fLu7C =FMQp -----END PGP SIGNATURE-----