From mboxrd@z Thu Jan 1 00:00:00 1970 From: Daniel J Blueman Subject: 2.6.39-rc2 filesystem balance lock ordering... Date: Thu, 7 Apr 2011 12:37:05 +0800 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Cc: Chris Mason , Josef Bacik To: Linux BTRFS Return-path: List-ID: While trying to reproduce an earlier problem on 2.6.39-rc2, we see a possible deadlock from locks being taken in an inconsistent order: ======================================================= [ INFO: possible circular locking dependency detected ] 2.6.39-rc2-350cd #2 ------------------------------------------------------- btrfs/27867 is trying to acquire lock: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] prealloc_file_extent_cluster+0x59/0x180 but task is already holding lock: (&fs_info->cleaner_mutex){+.+...}, at: [] btrfs_relocate_block_group+0x197/0x2d0 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #2 (&fs_info->cleaner_mutex){+.+...}: [] validate_chain+0x5a7/0x6c0 [] __lock_acquire+0x495/0x920 [] lock_acquire+0x5a/0x70 [] mutex_lock_nested+0x5e/0x390 [] btrfs_commit_super+0x21/0xe0 [] close_ctree+0x258/0x2d0 [] btrfs_put_super+0x18/0x30 [] generic_shutdown_super+0x6d/0xf0 [] kill_anon_super+0x11/0x60 [] deactivate_locked_super+0x45/0x60 [] deactivate_super+0x45/0x60 [] mntput_no_expire+0x99/0xf0 [] sys_umount+0x67/0xd0 [] system_call_fastpath+0x16/0x1b -> #1 (&type->s_umount_key#32){+++++.}: [] validate_chain+0x5a7/0x6c0 [] __lock_acquire+0x495/0x920 [] lock_acquire+0x5a/0x70 [] down_read+0x42/0x60 [] writeback_inodes_sb_nr_if_idle+0x35/0x60 [] shrink_delalloc+0xee/0x180 [] reserve_metadata_bytes+0xc3/0x200 [] btrfs_delalloc_reserve_metadata+0xc4/0x150 [] btrfs_delalloc_reserve_space+0x3b/0x60 [] __btrfs_buffered_write+0x153/0x320 [] btrfs_file_aio_write+0x230/0x310 [] aio_rw_vect_retry+0x74/0x1d0 [] aio_run_iocb+0x61/0x180 [] io_submit_one+0x15f/0x210 [] do_io_submit+0x118/0x1d0 [] sys_io_submit+0xb/0x10 [] system_call_fastpath+0x16/0x1b -> #0 (&sb->s_type->i_mutex_key#13){+.+.+.}: [] check_prev_add+0x705/0x720 [] validate_chain+0x5a7/0x6c0 [] __lock_acquire+0x495/0x920 [] lock_acquire+0x5a/0x70 [] mutex_lock_nested+0x5e/0x390 [] prealloc_file_extent_cluster+0x59/0x180 [] relocate_file_extent_cluster+0x91/0x380 [] relocate_data_extent+0x3f/0xd0 [] relocate_block_group+0x323/0x600 [] btrfs_relocate_block_group+0x1a8/0x2d0 [] btrfs_relocate_chunk+0x6d/0x3b0 [] btrfs_shrink_device+0x21a/0x3d0 [] btrfs_balance+0x10b/0x280 [] btrfs_ioctl+0x450/0x590 [] do_vfs_ioctl+0x8d/0x330 [] sys_ioctl+0x4a/0x80 [] system_call_fastpath+0x16/0x1b other info that might help us debug this: 2 locks held by btrfs/27867: #0: (&fs_info->volume_mutex){+.+...}, at: [] btrfs_balance+0x8b/0x280 #1: (&fs_info->cleaner_mutex){+.+...}, at: [] btrfs_relocate_block_group+0x197/0x2d0 stack backtrace: Pid: 27867, comm: btrfs Tainted: G W 2.6.39-rc2-350cd #2 Call Trace: [] print_circular_bug+0xeb/0xf0 [] check_prev_add+0x705/0x720 [] validate_chain+0x5a7/0x6c0 [] __lock_acquire+0x495/0x920 [] ? unmap_extent_buffer+0xe/0x40 [] ? generic_bin_search+0x19c/0x210 [] ? prealloc_file_extent_cluster+0x59/0x180 [] lock_acquire+0x5a/0x70 [] ? prealloc_file_extent_cluster+0x59/0x180 [] ? add_preempt_count+0x7d/0xd0 [] mutex_lock_nested+0x5e/0x390 [] ? prealloc_file_extent_cluster+0x59/0x180 [] ? mark_held_locks+0x6c/0xa0 [] ? __slab_alloc+0x18d/0x480 [] ? trace_hardirqs_on_caller+0x14d/0x190 [] prealloc_file_extent_cluster+0x59/0x180 [] ? relocate_file_extent_cluster+0x76/0x380 [] relocate_file_extent_cluster+0x91/0x380 [] ? __btrfs_end_transaction+0x15f/0x240 [] relocate_data_extent+0x3f/0xd0 [] relocate_block_group+0x323/0x600 [] btrfs_relocate_block_group+0x1a8/0x2d0 [] btrfs_relocate_chunk+0x6d/0x3b0 [] ? sub_preempt_count+0x9d/0xd0 [] ? unmap_extent_buffer+0xe/0x40 [] ? btrfs_dev_extent_chunk_offset+0xe5/0xf0 [] btrfs_shrink_device+0x21a/0x3d0 [] btrfs_balance+0x10b/0x280 [] ? up_read+0x1e/0x40 [] ? do_page_fault+0x1cc/0x440 [] btrfs_ioctl+0x450/0x590 [] do_vfs_ioctl+0x8d/0x330 [] ? fget_light+0x2bf/0x3c0 [] ? trace_hardirqs_on_caller+0x14d/0x190 [] sys_ioctl+0x4a/0x80 [] system_call_fastpath+0x16/0x1b -- Daniel J Blueman