* [2.6.22] circular lock detected
@ 2007-08-24 21:00 Folkert van Heusden
2007-08-26 23:03 ` Michal Piotrowski
2007-09-02 11:55 ` Andrew Morton
0 siblings, 2 replies; 11+ messages in thread
From: Folkert van Heusden @ 2007-08-24 21:00 UTC (permalink / raw)
To: linux-kernel
Hi,
2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).
[ 346.314640] =======================================================
[ 346.314758] [ INFO: possible circular locking dependency detected ]
[ 346.314815] 2.6.22 #5
[ 346.314862] -------------------------------------------------------
[ 346.314920] tor/2421 is trying to acquire lock:
[ 346.314973] (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
[ 346.315193]
[ 346.315195] but task is already holding lock:
[ 346.316203] (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
[ 346.316457]
[ 346.316458] which lock already depends on the new lock.
[ 346.316461]
[ 346.316585]
[ 346.316586] the existing dependency chain (in reverse order) is:
[ 346.316675]
[ 346.316676] -> #4 (&s->s_dquot.dqptr_sem){----}:
[ 346.316871] [<c103b2ee>] check_prev_add+0x15b/0x281
[ 346.317204] [<c103b49f>] check_prevs_add+0x8b/0xe8
[ 346.317535] [<c103ce2a>] __lock_acquire+0x65d/0xb51
[ 346.317873] [<c103d786>] lock_acquire+0x62/0x81
[ 346.318195] [<c10341af>] down_read+0x2b/0x3d
[ 346.318527] [<c10a485f>] dquot_alloc_space+0x50/0x189
[ 346.318854] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
[ 346.319187] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
[ 346.319513] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
[ 346.319850] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
[ 346.320184] [<c10b8ada>] ext3_getblk+0x97/0x228
[ 346.320530] [<c10b8c85>] ext3_bread+0x1a/0x78
[ 346.320870] [<c10bde64>] ext3_mkdir+0xf4/0x270
[ 346.321188] [<c107cdf6>] vfs_mkdir+0xb3/0x161
[ 346.321506] [<c107cf30>] sys_mkdirat+0x8c/0xc4
[ 346.321820] [<c107cf88>] sys_mkdir+0x20/0x22
[ 346.322131] [<c1003eea>] syscall_call+0x7/0xb
[ 346.322446] [<ffffffff>] 0xffffffff
[ 346.322779]
[ 346.322780] -> #3 (&ei->truncate_mutex){--..}:
[ 346.322965] [<c103b2ee>] check_prev_add+0x15b/0x281
[ 346.323278] [<c103b49f>] check_prevs_add+0x8b/0xe8
[ 346.323602] [<c103ce2a>] __lock_acquire+0x65d/0xb51
[ 346.323919] [<c103d786>] lock_acquire+0x62/0x81
[ 346.324234] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
[ 346.324857] [<c121aa83>] mutex_lock+0x8/0xa
[ 346.325177] [<c10ba380>] ext3_truncate+0x170/0x468
[ 346.325501] [<c1060e9a>] vmtruncate+0xa6/0x116
[ 346.325828] [<c1087937>] inode_setattr+0x145/0x16c
[ 346.326150] [<c10bb301>] ext3_setattr+0x150/0x22f
[ 346.326479] [<c1087cb0>] notify_change+0x352/0x386
[ 346.326802] [<c1071af4>] do_truncate+0x5c/0x7f
[ 346.327116] [<c107c2e3>] may_open+0x1ec/0x231
[ 346.327434] [<c107c4a4>] open_namei+0xdd/0x5a2
[ 346.327770] [<c107275b>] do_filp_open+0x2c/0x53
[ 346.328090] [<c1072a90>] do_sys_open+0x52/0xd8
[ 346.328408] [<c1072b32>] sys_open+0x1c/0x1e
[ 346.328727] [<c1003eea>] syscall_call+0x7/0xb
[ 346.329042] [<ffffffff>] 0xffffffff
[ 346.329368]
[ 346.329370] -> #2 (&inode->i_alloc_sem){--..}:
[ 346.329556] [<c103b2ee>] check_prev_add+0x15b/0x281
[ 346.329879] [<c103b49f>] check_prevs_add+0x8b/0xe8
[ 346.330192] [<c103ce2a>] __lock_acquire+0x65d/0xb51
[ 346.330508] [<c103d786>] lock_acquire+0x62/0x81
[ 346.330826] [<c1034245>] down_write+0x2b/0x45
[ 346.331137] [<c1087c37>] notify_change+0x2d9/0x386
[ 346.331453] [<c1071af4>] do_truncate+0x5c/0x7f
[ 346.331769] [<c107c2e3>] may_open+0x1ec/0x231
[ 346.332087] [<c107c4a4>] open_namei+0xdd/0x5a2
[ 346.332401] [<c107275b>] do_filp_open+0x2c/0x53
[ 346.332720] [<c1072a90>] do_sys_open+0x52/0xd8
[ 346.333037] [<c1072b32>] sys_open+0x1c/0x1e
[ 346.333600] [<c1003eea>] syscall_call+0x7/0xb
[ 346.333912] [<ffffffff>] 0xffffffff
[ 346.334232]
[ 346.334233] -> #1 (&sysfs_inode_imutex_key){--..}:
[ 346.334422] [<c103b2ee>] check_prev_add+0x15b/0x281
[ 346.334734] [<c103b49f>] check_prevs_add+0x8b/0xe8
[ 346.335046] [<c103ce2a>] __lock_acquire+0x65d/0xb51
[ 346.335361] [<c103d786>] lock_acquire+0x62/0x81
[ 346.335685] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
[ 346.336001] [<c121aa83>] mutex_lock+0x8/0xa
[ 346.336313] [<c10b2828>] remove_dir+0x2c/0xf1
[ 346.336629] [<c10b28f5>] sysfs_remove_subdir+0x8/0xa
[ 346.336941] [<c10b3ccf>] sysfs_remove_group+0x67/0x78
[ 346.337256] [<c115cc58>] dpm_sysfs_remove+0x12/0x16
[ 346.337574] [<c115c36f>] device_pm_remove+0x21/0x62
[ 346.337888] [<c1156dc5>] device_del+0x82/0x291
[ 346.338199] [<c1156fdf>] device_unregister+0xb/0x15
[ 346.338516] [<c11571da>] device_destroy+0x8d/0x9a
[ 346.338827] [<c1142712>] vcs_remove_sysfs+0x1c/0x3a
[ 346.339142] [<c1149075>] con_close+0x5e/0x6b
[ 346.339458] [<c113aa5e>] release_dev+0x4c4/0x6ce
[ 346.339767] [<c113b0d6>] tty_release+0x12/0x1c
[ 346.340078] [<c1074479>] __fput+0x145/0x16e
[ 346.340392] [<c1074332>] fput+0x19/0x1b
[ 346.340701] [<c1072bb3>] filp_close+0x3c/0x75
[ 346.341012] [<c1072c55>] sys_close+0x69/0xb4
[ 346.341325] [<c1003eea>] syscall_call+0x7/0xb
[ 346.341892] [<ffffffff>] 0xffffffff
[ 346.342213]
[ 346.342214] -> #0 (tty_mutex){--..}:
[ 346.342397] [<c103b1c7>] check_prev_add+0x34/0x281
[ 346.342708] [<c103b49f>] check_prevs_add+0x8b/0xe8
[ 346.343017] [<c103ce2a>] __lock_acquire+0x65d/0xb51
[ 346.343333] [<c103d786>] lock_acquire+0x62/0x81
[ 346.343646] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
[ 346.343956] [<c121aa83>] mutex_lock+0x8/0xa
[ 346.344263] [<c10a41d5>] print_warning+0x8c/0x15d
[ 346.344575] [<c10a4993>] dquot_alloc_space+0x184/0x189
[ 346.344883] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
[ 346.345193] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
[ 346.345507] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
[ 346.345816] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
[ 346.346127] [<c10b89d8>] ext3_get_block+0x78/0xe3
[ 346.346441] [<c1094802>] __block_prepare_write+0x168/0x415
[ 346.346750] [<c109530e>] block_prepare_write+0x28/0x3b
[ 346.347057] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
[ 346.347369] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
[ 346.347689] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
[ 346.348002] [<c1055d8d>] generic_file_aio_write+0x58/0xc4
[ 346.348314] [<c10b6a93>] ext3_file_write+0x2d/0xba
[ 346.348627] [<c107353b>] do_sync_write+0xc7/0x116
[ 346.348935] [<c10736e2>] vfs_write+0x158/0x15d
[ 346.349246] [<c1073788>] sys_write+0x3d/0x64
[ 346.349560] [<c1003eea>] syscall_call+0x7/0xb
[ 346.349869] [<ffffffff>] 0xffffffff
[ 346.350438]
[ 346.350440] other info that might help us debug this:
[ 346.350442]
[ 346.350562] 3 locks held by tor/2421:
[ 346.350604] #0: (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
[ 346.350827] #1: (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
[ 346.351049] #2: (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
[ 346.351274]
[ 346.351276] stack backtrace:
[ 346.351363] [<c1004cac>] show_trace_log_lvl+0x1a/0x30
[ 346.351446] [<c1004cd4>] show_trace+0x12/0x14
[ 346.351526] [<c1004dce>] dump_stack+0x16/0x18
[ 346.351605] [<c103ab29>] print_circular_bug_tail+0x6f/0x71
[ 346.351686] [<c103b1c7>] check_prev_add+0x34/0x281
[ 346.351764] [<c103b49f>] check_prevs_add+0x8b/0xe8
[ 346.351843] [<c103ce2a>] __lock_acquire+0x65d/0xb51
[ 346.351921] [<c103d786>] lock_acquire+0x62/0x81
[ 346.352002] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
[ 346.352083] [<c121aa83>] mutex_lock+0x8/0xa
[ 346.352160] [<c10a41d5>] print_warning+0x8c/0x15d
[ 346.352240] [<c10a4993>] dquot_alloc_space+0x184/0x189
[ 346.352323] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
[ 346.352402] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
[ 346.352479] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
[ 346.352558] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
[ 346.352638] [<c10b89d8>] ext3_get_block+0x78/0xe3
[ 346.352720] [<c1094802>] __block_prepare_write+0x168/0x415
[ 346.352801] [<c109530e>] block_prepare_write+0x28/0x3b
[ 346.352881] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
[ 346.352961] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
[ 346.353041] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
[ 346.353121] [<c1055d8d>] generic_file_aio_write+0x58/0xc4
[ 346.353200] [<c10b6a93>] ext3_file_write+0x2d/0xba
[ 346.353281] [<c107353b>] do_sync_write+0xc7/0x116
[ 346.353364] [<c10736e2>] vfs_write+0x158/0x15d
[ 346.353444] [<c1073788>] sys_write+0x3d/0x64
[ 346.353524] [<c1003eea>] syscall_call+0x7/0xb
[ 346.353603] =======================
Folkert van Heusden
--
www.vanheusden.com/multitail - win een vlaai van multivlaai! zorg
ervoor dat multitail opgenomen wordt in Fedora Core, AIX, Solaris of
HP/UX en win een vlaai naar keuze
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [2.6.22] circular lock detected 2007-08-24 21:00 [2.6.22] circular lock detected Folkert van Heusden @ 2007-08-26 23:03 ` Michal Piotrowski 2007-09-02 11:55 ` Andrew Morton 1 sibling, 0 replies; 11+ messages in thread From: Michal Piotrowski @ 2007-08-26 23:03 UTC (permalink / raw) To: Folkert van Heusden; +Cc: linux-kernel, linux-fsdevel, linux-ext4 Hi, [Adding fsdevel and linux-ext4 to CC] On 24/08/07, Folkert van Heusden <folkert@vanheusden.com> wrote: > Hi, > > 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2). > > [ 346.314640] ======================================================= > [ 346.314758] [ INFO: possible circular locking dependency detected ] > [ 346.314815] 2.6.22 #5 > [ 346.314862] ------------------------------------------------------- > [ 346.314920] tor/2421 is trying to acquire lock: > [ 346.314973] (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa > [ 346.315193] > [ 346.315195] but task is already holding lock: > [ 346.316203] (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189 > [ 346.316457] > [ 346.316458] which lock already depends on the new lock. > [ 346.316461] > [ 346.316585] > [ 346.316586] the existing dependency chain (in reverse order) is: > [ 346.316675] > [ 346.316676] -> #4 (&s->s_dquot.dqptr_sem){----}: > [ 346.316871] [<c103b2ee>] check_prev_add+0x15b/0x281 > [ 346.317204] [<c103b49f>] check_prevs_add+0x8b/0xe8 > [ 346.317535] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > [ 346.317873] [<c103d786>] lock_acquire+0x62/0x81 > [ 346.318195] [<c10341af>] down_read+0x2b/0x3d > [ 346.318527] [<c10a485f>] dquot_alloc_space+0x50/0x189 > [ 346.318854] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2 > [ 346.319187] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf > [ 346.319513] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b > [ 346.319850] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367 > [ 346.320184] [<c10b8ada>] ext3_getblk+0x97/0x228 > [ 346.320530] [<c10b8c85>] ext3_bread+0x1a/0x78 > [ 346.320870] [<c10bde64>] ext3_mkdir+0xf4/0x270 > [ 346.321188] [<c107cdf6>] vfs_mkdir+0xb3/0x161 > [ 346.321506] [<c107cf30>] sys_mkdirat+0x8c/0xc4 > [ 346.321820] [<c107cf88>] sys_mkdir+0x20/0x22 > [ 346.322131] [<c1003eea>] syscall_call+0x7/0xb > [ 346.322446] [<ffffffff>] 0xffffffff > [ 346.322779] > [ 346.322780] -> #3 (&ei->truncate_mutex){--..}: > [ 346.322965] [<c103b2ee>] check_prev_add+0x15b/0x281 > [ 346.323278] [<c103b49f>] check_prevs_add+0x8b/0xe8 > [ 346.323602] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > [ 346.323919] [<c103d786>] lock_acquire+0x62/0x81 > [ 346.324234] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c > [ 346.324857] [<c121aa83>] mutex_lock+0x8/0xa > [ 346.325177] [<c10ba380>] ext3_truncate+0x170/0x468 > [ 346.325501] [<c1060e9a>] vmtruncate+0xa6/0x116 > [ 346.325828] [<c1087937>] inode_setattr+0x145/0x16c > [ 346.326150] [<c10bb301>] ext3_setattr+0x150/0x22f > [ 346.326479] [<c1087cb0>] notify_change+0x352/0x386 > [ 346.326802] [<c1071af4>] do_truncate+0x5c/0x7f > [ 346.327116] [<c107c2e3>] may_open+0x1ec/0x231 > [ 346.327434] [<c107c4a4>] open_namei+0xdd/0x5a2 > [ 346.327770] [<c107275b>] do_filp_open+0x2c/0x53 > [ 346.328090] [<c1072a90>] do_sys_open+0x52/0xd8 > [ 346.328408] [<c1072b32>] sys_open+0x1c/0x1e > [ 346.328727] [<c1003eea>] syscall_call+0x7/0xb > [ 346.329042] [<ffffffff>] 0xffffffff > [ 346.329368] > [ 346.329370] -> #2 (&inode->i_alloc_sem){--..}: > [ 346.329556] [<c103b2ee>] check_prev_add+0x15b/0x281 > [ 346.329879] [<c103b49f>] check_prevs_add+0x8b/0xe8 > [ 346.330192] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > [ 346.330508] [<c103d786>] lock_acquire+0x62/0x81 > [ 346.330826] [<c1034245>] down_write+0x2b/0x45 > [ 346.331137] [<c1087c37>] notify_change+0x2d9/0x386 > [ 346.331453] [<c1071af4>] do_truncate+0x5c/0x7f > [ 346.331769] [<c107c2e3>] may_open+0x1ec/0x231 > [ 346.332087] [<c107c4a4>] open_namei+0xdd/0x5a2 > [ 346.332401] [<c107275b>] do_filp_open+0x2c/0x53 > [ 346.332720] [<c1072a90>] do_sys_open+0x52/0xd8 > [ 346.333037] [<c1072b32>] sys_open+0x1c/0x1e > [ 346.333600] [<c1003eea>] syscall_call+0x7/0xb > [ 346.333912] [<ffffffff>] 0xffffffff > [ 346.334232] > [ 346.334233] -> #1 (&sysfs_inode_imutex_key){--..}: > [ 346.334422] [<c103b2ee>] check_prev_add+0x15b/0x281 > [ 346.334734] [<c103b49f>] check_prevs_add+0x8b/0xe8 > [ 346.335046] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > [ 346.335361] [<c103d786>] lock_acquire+0x62/0x81 > [ 346.335685] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c > [ 346.336001] [<c121aa83>] mutex_lock+0x8/0xa > [ 346.336313] [<c10b2828>] remove_dir+0x2c/0xf1 > [ 346.336629] [<c10b28f5>] sysfs_remove_subdir+0x8/0xa > [ 346.336941] [<c10b3ccf>] sysfs_remove_group+0x67/0x78 > [ 346.337256] [<c115cc58>] dpm_sysfs_remove+0x12/0x16 > [ 346.337574] [<c115c36f>] device_pm_remove+0x21/0x62 > [ 346.337888] [<c1156dc5>] device_del+0x82/0x291 > [ 346.338199] [<c1156fdf>] device_unregister+0xb/0x15 > [ 346.338516] [<c11571da>] device_destroy+0x8d/0x9a > [ 346.338827] [<c1142712>] vcs_remove_sysfs+0x1c/0x3a > [ 346.339142] [<c1149075>] con_close+0x5e/0x6b > [ 346.339458] [<c113aa5e>] release_dev+0x4c4/0x6ce > [ 346.339767] [<c113b0d6>] tty_release+0x12/0x1c > [ 346.340078] [<c1074479>] __fput+0x145/0x16e > [ 346.340392] [<c1074332>] fput+0x19/0x1b > [ 346.340701] [<c1072bb3>] filp_close+0x3c/0x75 > [ 346.341012] [<c1072c55>] sys_close+0x69/0xb4 > [ 346.341325] [<c1003eea>] syscall_call+0x7/0xb > [ 346.341892] [<ffffffff>] 0xffffffff > [ 346.342213] > [ 346.342214] -> #0 (tty_mutex){--..}: > [ 346.342397] [<c103b1c7>] check_prev_add+0x34/0x281 > [ 346.342708] [<c103b49f>] check_prevs_add+0x8b/0xe8 > [ 346.343017] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > [ 346.343333] [<c103d786>] lock_acquire+0x62/0x81 > [ 346.343646] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c > [ 346.343956] [<c121aa83>] mutex_lock+0x8/0xa > [ 346.344263] [<c10a41d5>] print_warning+0x8c/0x15d > [ 346.344575] [<c10a4993>] dquot_alloc_space+0x184/0x189 > [ 346.344883] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2 > [ 346.345193] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf > [ 346.345507] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b > [ 346.345816] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367 > [ 346.346127] [<c10b89d8>] ext3_get_block+0x78/0xe3 > [ 346.346441] [<c1094802>] __block_prepare_write+0x168/0x415 > [ 346.346750] [<c109530e>] block_prepare_write+0x28/0x3b > [ 346.347057] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e > [ 346.347369] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614 > [ 346.347689] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533 > [ 346.348002] [<c1055d8d>] generic_file_aio_write+0x58/0xc4 > [ 346.348314] [<c10b6a93>] ext3_file_write+0x2d/0xba > [ 346.348627] [<c107353b>] do_sync_write+0xc7/0x116 > [ 346.348935] [<c10736e2>] vfs_write+0x158/0x15d > [ 346.349246] [<c1073788>] sys_write+0x3d/0x64 > [ 346.349560] [<c1003eea>] syscall_call+0x7/0xb > [ 346.349869] [<ffffffff>] 0xffffffff > [ 346.350438] > [ 346.350440] other info that might help us debug this: > [ 346.350442] > [ 346.350562] 3 locks held by tor/2421: > [ 346.350604] #0: (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa > [ 346.350827] #1: (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa > [ 346.351049] #2: (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189 > [ 346.351274] > [ 346.351276] stack backtrace: > [ 346.351363] [<c1004cac>] show_trace_log_lvl+0x1a/0x30 > [ 346.351446] [<c1004cd4>] show_trace+0x12/0x14 > [ 346.351526] [<c1004dce>] dump_stack+0x16/0x18 > [ 346.351605] [<c103ab29>] print_circular_bug_tail+0x6f/0x71 > [ 346.351686] [<c103b1c7>] check_prev_add+0x34/0x281 > [ 346.351764] [<c103b49f>] check_prevs_add+0x8b/0xe8 > [ 346.351843] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > [ 346.351921] [<c103d786>] lock_acquire+0x62/0x81 > [ 346.352002] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c > [ 346.352083] [<c121aa83>] mutex_lock+0x8/0xa > [ 346.352160] [<c10a41d5>] print_warning+0x8c/0x15d > [ 346.352240] [<c10a4993>] dquot_alloc_space+0x184/0x189 > [ 346.352323] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2 > [ 346.352402] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf > [ 346.352479] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b > [ 346.352558] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367 > [ 346.352638] [<c10b89d8>] ext3_get_block+0x78/0xe3 > [ 346.352720] [<c1094802>] __block_prepare_write+0x168/0x415 > [ 346.352801] [<c109530e>] block_prepare_write+0x28/0x3b > [ 346.352881] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e > [ 346.352961] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614 > [ 346.353041] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533 > [ 346.353121] [<c1055d8d>] generic_file_aio_write+0x58/0xc4 > [ 346.353200] [<c10b6a93>] ext3_file_write+0x2d/0xba > [ 346.353281] [<c107353b>] do_sync_write+0xc7/0x116 > [ 346.353364] [<c10736e2>] vfs_write+0x158/0x15d > [ 346.353444] [<c1073788>] sys_write+0x3d/0x64 > [ 346.353524] [<c1003eea>] syscall_call+0x7/0xb > [ 346.353603] ======================= > > > Folkert van Heusden > > -- > www.vanheusden.com/multitail - win een vlaai van multivlaai! zorg > ervoor dat multitail opgenomen wordt in Fedora Core, AIX, Solaris of > HP/UX en win een vlaai naar keuze > ---------------------------------------------------------------------- > Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > Regards, Michal -- LOG http://www.stardust.webpages.pl/log/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [2.6.22] circular lock detected 2007-08-24 21:00 [2.6.22] circular lock detected Folkert van Heusden 2007-08-26 23:03 ` Michal Piotrowski @ 2007-09-02 11:55 ` Andrew Morton 2007-09-02 17:11 ` Folkert van Heusden 2007-09-03 12:27 ` Jan Kara 1 sibling, 2 replies; 11+ messages in thread From: Andrew Morton @ 2007-09-02 11:55 UTC (permalink / raw) To: Folkert van Heusden; +Cc: linux-kernel, Jan Kara > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote: > Hi, > > 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2). > > [ 346.314640] ======================================================= > [ 346.314758] [ INFO: possible circular locking dependency detected ] > [ 346.314815] 2.6.22 #5 > [ 346.314862] ------------------------------------------------------- > [ 346.314920] tor/2421 is trying to acquire lock: > [ 346.314973] (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa > [ 346.315193] > [ 346.315195] but task is already holding lock: > [ 346.316203] (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189 > [ 346.316457] > [ 346.316458] which lock already depends on the new lock. > [ 346.316461] > [ 346.316585] > [ 346.316586] the existing dependency chain (in reverse order) is: > [ 346.316675] > [ 346.316676] -> #4 (&s->s_dquot.dqptr_sem){----}: > [ 346.316871] [<c103b2ee>] check_prev_add+0x15b/0x281 > [ 346.317204] [<c103b49f>] check_prevs_add+0x8b/0xe8 > [ 346.317535] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > [ 346.317873] [<c103d786>] lock_acquire+0x62/0x81 > [ 346.318195] [<c10341af>] down_read+0x2b/0x3d > [ 346.318527] [<c10a485f>] dquot_alloc_space+0x50/0x189 > [ 346.318854] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2 > [ 346.319187] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf > [ 346.319513] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b > [ 346.319850] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367 > [ 346.320184] [<c10b8ada>] ext3_getblk+0x97/0x228 > [ 346.320530] [<c10b8c85>] ext3_bread+0x1a/0x78 > [ 346.320870] [<c10bde64>] ext3_mkdir+0xf4/0x270 > [ 346.321188] [<c107cdf6>] vfs_mkdir+0xb3/0x161 > [ 346.321506] [<c107cf30>] sys_mkdirat+0x8c/0xc4 > [ 346.321820] [<c107cf88>] sys_mkdir+0x20/0x22 > [ 346.322131] [<c1003eea>] syscall_call+0x7/0xb > [ 346.322446] [<ffffffff>] 0xffffffff > [ 346.322779] > [ 346.322780] -> #3 (&ei->truncate_mutex){--..}: > [ 346.322965] [<c103b2ee>] check_prev_add+0x15b/0x281 > [ 346.323278] [<c103b49f>] check_prevs_add+0x8b/0xe8 > [ 346.323602] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > [ 346.323919] [<c103d786>] lock_acquire+0x62/0x81 > [ 346.324234] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c > [ 346.324857] [<c121aa83>] mutex_lock+0x8/0xa > [ 346.325177] [<c10ba380>] ext3_truncate+0x170/0x468 > [ 346.325501] [<c1060e9a>] vmtruncate+0xa6/0x116 > [ 346.325828] [<c1087937>] inode_setattr+0x145/0x16c > [ 346.326150] [<c10bb301>] ext3_setattr+0x150/0x22f > [ 346.326479] [<c1087cb0>] notify_change+0x352/0x386 > [ 346.326802] [<c1071af4>] do_truncate+0x5c/0x7f > [ 346.327116] [<c107c2e3>] may_open+0x1ec/0x231 > [ 346.327434] [<c107c4a4>] open_namei+0xdd/0x5a2 > [ 346.327770] [<c107275b>] do_filp_open+0x2c/0x53 > [ 346.328090] [<c1072a90>] do_sys_open+0x52/0xd8 > [ 346.328408] [<c1072b32>] sys_open+0x1c/0x1e > [ 346.328727] [<c1003eea>] syscall_call+0x7/0xb > [ 346.329042] [<ffffffff>] 0xffffffff > [ 346.329368] > [ 346.329370] -> #2 (&inode->i_alloc_sem){--..}: > [ 346.329556] [<c103b2ee>] check_prev_add+0x15b/0x281 > [ 346.329879] [<c103b49f>] check_prevs_add+0x8b/0xe8 > [ 346.330192] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > [ 346.330508] [<c103d786>] lock_acquire+0x62/0x81 > [ 346.330826] [<c1034245>] down_write+0x2b/0x45 > [ 346.331137] [<c1087c37>] notify_change+0x2d9/0x386 > [ 346.331453] [<c1071af4>] do_truncate+0x5c/0x7f > [ 346.331769] [<c107c2e3>] may_open+0x1ec/0x231 > [ 346.332087] [<c107c4a4>] open_namei+0xdd/0x5a2 > [ 346.332401] [<c107275b>] do_filp_open+0x2c/0x53 > [ 346.332720] [<c1072a90>] do_sys_open+0x52/0xd8 > [ 346.333037] [<c1072b32>] sys_open+0x1c/0x1e > [ 346.333600] [<c1003eea>] syscall_call+0x7/0xb > [ 346.333912] [<ffffffff>] 0xffffffff > [ 346.334232] > [ 346.334233] -> #1 (&sysfs_inode_imutex_key){--..}: > [ 346.334422] [<c103b2ee>] check_prev_add+0x15b/0x281 > [ 346.334734] [<c103b49f>] check_prevs_add+0x8b/0xe8 > [ 346.335046] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > [ 346.335361] [<c103d786>] lock_acquire+0x62/0x81 > [ 346.335685] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c > [ 346.336001] [<c121aa83>] mutex_lock+0x8/0xa > [ 346.336313] [<c10b2828>] remove_dir+0x2c/0xf1 > [ 346.336629] [<c10b28f5>] sysfs_remove_subdir+0x8/0xa > [ 346.336941] [<c10b3ccf>] sysfs_remove_group+0x67/0x78 > [ 346.337256] [<c115cc58>] dpm_sysfs_remove+0x12/0x16 > [ 346.337574] [<c115c36f>] device_pm_remove+0x21/0x62 > [ 346.337888] [<c1156dc5>] device_del+0x82/0x291 > [ 346.338199] [<c1156fdf>] device_unregister+0xb/0x15 > [ 346.338516] [<c11571da>] device_destroy+0x8d/0x9a > [ 346.338827] [<c1142712>] vcs_remove_sysfs+0x1c/0x3a > [ 346.339142] [<c1149075>] con_close+0x5e/0x6b > [ 346.339458] [<c113aa5e>] release_dev+0x4c4/0x6ce > [ 346.339767] [<c113b0d6>] tty_release+0x12/0x1c > [ 346.340078] [<c1074479>] __fput+0x145/0x16e > [ 346.340392] [<c1074332>] fput+0x19/0x1b > [ 346.340701] [<c1072bb3>] filp_close+0x3c/0x75 > [ 346.341012] [<c1072c55>] sys_close+0x69/0xb4 > [ 346.341325] [<c1003eea>] syscall_call+0x7/0xb > [ 346.341892] [<ffffffff>] 0xffffffff > [ 346.342213] > [ 346.342214] -> #0 (tty_mutex){--..}: > [ 346.342397] [<c103b1c7>] check_prev_add+0x34/0x281 > [ 346.342708] [<c103b49f>] check_prevs_add+0x8b/0xe8 > [ 346.343017] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > [ 346.343333] [<c103d786>] lock_acquire+0x62/0x81 > [ 346.343646] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c > [ 346.343956] [<c121aa83>] mutex_lock+0x8/0xa > [ 346.344263] [<c10a41d5>] print_warning+0x8c/0x15d > [ 346.344575] [<c10a4993>] dquot_alloc_space+0x184/0x189 > [ 346.344883] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2 > [ 346.345193] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf > [ 346.345507] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b > [ 346.345816] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367 > [ 346.346127] [<c10b89d8>] ext3_get_block+0x78/0xe3 > [ 346.346441] [<c1094802>] __block_prepare_write+0x168/0x415 > [ 346.346750] [<c109530e>] block_prepare_write+0x28/0x3b > [ 346.347057] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e > [ 346.347369] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614 > [ 346.347689] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533 > [ 346.348002] [<c1055d8d>] generic_file_aio_write+0x58/0xc4 > [ 346.348314] [<c10b6a93>] ext3_file_write+0x2d/0xba > [ 346.348627] [<c107353b>] do_sync_write+0xc7/0x116 > [ 346.348935] [<c10736e2>] vfs_write+0x158/0x15d > [ 346.349246] [<c1073788>] sys_write+0x3d/0x64 > [ 346.349560] [<c1003eea>] syscall_call+0x7/0xb > [ 346.349869] [<ffffffff>] 0xffffffff > [ 346.350438] > [ 346.350440] other info that might help us debug this: > [ 346.350442] > [ 346.350562] 3 locks held by tor/2421: > [ 346.350604] #0: (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa > [ 346.350827] #1: (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa > [ 346.351049] #2: (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189 > [ 346.351274] > [ 346.351276] stack backtrace: > [ 346.351363] [<c1004cac>] show_trace_log_lvl+0x1a/0x30 > [ 346.351446] [<c1004cd4>] show_trace+0x12/0x14 > [ 346.351526] [<c1004dce>] dump_stack+0x16/0x18 > [ 346.351605] [<c103ab29>] print_circular_bug_tail+0x6f/0x71 > [ 346.351686] [<c103b1c7>] check_prev_add+0x34/0x281 > [ 346.351764] [<c103b49f>] check_prevs_add+0x8b/0xe8 > [ 346.351843] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > [ 346.351921] [<c103d786>] lock_acquire+0x62/0x81 > [ 346.352002] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c > [ 346.352083] [<c121aa83>] mutex_lock+0x8/0xa > [ 346.352160] [<c10a41d5>] print_warning+0x8c/0x15d > [ 346.352240] [<c10a4993>] dquot_alloc_space+0x184/0x189 > [ 346.352323] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2 > [ 346.352402] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf > [ 346.352479] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b > [ 346.352558] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367 > [ 346.352638] [<c10b89d8>] ext3_get_block+0x78/0xe3 > [ 346.352720] [<c1094802>] __block_prepare_write+0x168/0x415 > [ 346.352801] [<c109530e>] block_prepare_write+0x28/0x3b > [ 346.352881] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e > [ 346.352961] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614 > [ 346.353041] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533 > [ 346.353121] [<c1055d8d>] generic_file_aio_write+0x58/0xc4 > [ 346.353200] [<c10b6a93>] ext3_file_write+0x2d/0xba > [ 346.353281] [<c107353b>] do_sync_write+0xc7/0x116 > [ 346.353364] [<c10736e2>] vfs_write+0x158/0x15d > [ 346.353444] [<c1073788>] sys_write+0x3d/0x64 > [ 346.353524] [<c1003eea>] syscall_call+0x7/0xb > [ 346.353603] ======================= > Has been reported before, but I don't recall whether we fixed it. Jan, do you know>? ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [2.6.22] circular lock detected 2007-09-02 11:55 ` Andrew Morton @ 2007-09-02 17:11 ` Folkert van Heusden 2007-09-03 12:27 ` Jan Kara 1 sibling, 0 replies; 11+ messages in thread From: Folkert van Heusden @ 2007-09-02 17:11 UTC (permalink / raw) To: Andrew Morton; +Cc: linux-kernel, Jan Kara > > 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2). > > > > [ 346.314640] ======================================================= ... > > [ 346.349869] [<ffffffff>] 0xffffffff > > [ 346.350438] > > [ 346.350440] other info that might help us debug this: > > [ 346.350442] > > [ 346.350562] 3 locks held by tor/2421: > > [ 346.350604] #0: (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa > > [ 346.350827] #1: (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa > > [ 346.351049] #2: (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189 > > [ 346.351274] > > [ 346.351276] stack backtrace: ... > > [ 346.353603] ======================= > > Has been reported before, but I don't recall whether we fixed it. Jan, > do you know>? It is repeatable. Yesterday I changed from ata to sata and it still occurs altough in a different proces: [ 7382.917927] Oh and it also seems it takes longer. [ 7382.917929] ======================================================= [ 7382.918024] [ INFO: possible circular locking dependency detected ] [ 7382.918072] 2.6.22 #2 [ 7382.918114] ------------------------------------------------------- [ 7382.918159] moo/17202 is trying to acquire lock: [ 7382.918204] (tty_mutex){--..}, at: [<c1228353>] mutex_lock+0x8/0xa [ 7382.918392] [ 7382.918393] but task is already holding lock: [ 7382.918474] (&s->s_dquot.dqptr_sem){----}, at: [<c10a47a3>] dquot_alloc_space+0x50/0x189 [ 7382.918665] [ 7382.918666] which lock already depends on the new lock. [ 7382.918668] [ 7382.918788] [ 7382.918789] the existing dependency chain (in reverse order) is: [ 7382.918873] [ 7382.918874] -> #4 (&s->s_dquot.dqptr_sem){----}: [ 7382.919061] [<c103b232>] check_prev_add+0x15b/0x281 [ 7382.919378] [<c103b3e3>] check_prevs_add+0x8b/0xe8 [ 7382.919690] [<c103cd6e>] __lock_acquire+0x65d/0xb51 [ 7382.920004] [<c103d6ca>] lock_acquire+0x62/0x81 [ 7382.920322] [<c10340f3>] down_read+0x2b/0x3d [ 7382.920635] [<c10a47a3>] dquot_alloc_space+0x50/0x189 [ 7382.920957] [<c10b5b99>] ext3_new_blocks+0x44b/0x5a2 [ 7382.921280] [<c10b8128>] ext3_alloc_blocks+0x40/0xdf [ 7382.921592] [<c10b8217>] ext3_alloc_branch+0x50/0x21b [ 7382.921906] [<c10b86f5>] ext3_get_blocks_handle+0x1b8/0x367 [ 7382.922222] [<c10b8a1e>] ext3_getblk+0x97/0x228 [ 7382.922534] [<c10b8bc9>] ext3_bread+0x1a/0x78 [ 7382.922844] [<c10bdda8>] ext3_mkdir+0xf4/0x270 [ 7382.923160] [<c107cd3a>] vfs_mkdir+0xb3/0x161 [ 7382.923474] [<c107ce74>] sys_mkdirat+0x8c/0xc4 [ 7382.923789] [<c107cecc>] sys_mkdir+0x20/0x22 [ 7382.924106] [<c1003eea>] syscall_call+0x7/0xb [ 7382.924425] [<ffffffff>] 0xffffffff [ 7382.924752] [ 7382.924753] -> #3 (&ei->truncate_mutex){--..}: [ 7382.924939] [<c103b232>] check_prev_add+0x15b/0x281 [ 7382.925257] [<c103b3e3>] check_prevs_add+0x8b/0xe8 [ 7382.925571] [<c103cd6e>] __lock_acquire+0x65d/0xb51 [ 7382.925886] [<c103d6ca>] lock_acquire+0x62/0x81 [ 7382.926202] [<c12283d4>] __mutex_lock_slowpath+0x75/0x28c [ 7382.926782] [<c1228353>] mutex_lock+0x8/0xa [ 7382.927101] [<c10ba2c4>] ext3_truncate+0x170/0x468 [ 7382.927414] [<c1060dde>] vmtruncate+0xa6/0x116 [ 7382.927726] [<c108787b>] inode_setattr+0x145/0x16c [ 7382.928041] [<c10bb245>] ext3_setattr+0x150/0x22f [ 7382.928358] [<c1087bf4>] notify_change+0x352/0x386 [ 7382.928672] [<c1071a38>] do_truncate+0x5c/0x7f [ 7382.928988] [<c107c227>] may_open+0x1ec/0x231 [ 7382.929289] [<c107c3e8>] open_namei+0xdd/0x5a2 [ 7382.929298] [<c107269f>] do_filp_open+0x2c/0x53 [ 7382.929308] [<c10729d4>] do_sys_open+0x52/0xd8 [ 7382.929316] [<c1072a76>] sys_open+0x1c/0x1e [ 7382.929323] [<c1003eea>] syscall_call+0x7/0xb [ 7382.929332] [<ffffffff>] 0xffffffff [ 7382.929351] [ 7382.929352] -> #2 (&inode->i_alloc_sem){--..}: [ 7382.929358] [<c103b232>] check_prev_add+0x15b/0x281 [ 7382.929367] [<c103b3e3>] check_prevs_add+0x8b/0xe8 [ 7382.929374] [<c103cd6e>] __lock_acquire+0x65d/0xb51 [ 7382.929382] [<c103d6ca>] lock_acquire+0x62/0x81 [ 7382.929390] [<c1034189>] down_write+0x2b/0x45 [ 7382.929401] [<c1087b7b>] notify_change+0x2d9/0x386 [ 7382.929409] [<c1071a38>] do_truncate+0x5c/0x7f [ 7382.929416] [<c107c227>] may_open+0x1ec/0x231 [ 7382.929423] [<c107c3e8>] open_namei+0xdd/0x5a2 [ 7382.929430] [<c107269f>] do_filp_open+0x2c/0x53 [ 7382.929441] [<c10729d4>] do_sys_open+0x52/0xd8 [ 7382.929449] [<c1072a76>] sys_open+0x1c/0x1e [ 7382.929455] [<c1003eea>] syscall_call+0x7/0xb [ 7382.929463] [<ffffffff>] 0xffffffff [ 7382.929471] [ 7382.929472] -> #1 (&sysfs_inode_imutex_key){--..}: [ 7382.929481] [<c103b232>] check_prev_add+0x15b/0x281 [ 7382.929489] [<c103b3e3>] check_prevs_add+0x8b/0xe8 [ 7382.929496] [<c103cd6e>] __lock_acquire+0x65d/0xb51 [ 7382.929505] [<c103d6ca>] lock_acquire+0x62/0x81 [ 7382.929512] [<c12283d4>] __mutex_lock_slowpath+0x75/0x28c [ 7382.929523] [<c1228353>] mutex_lock+0x8/0xa [ 7382.929531] [<c10b276c>] remove_dir+0x2c/0xf1 [ 7382.929538] [<c10b2839>] sysfs_remove_subdir+0x8/0xa [ 7382.929547] [<c10b3c13>] sysfs_remove_group+0x67/0x78 [ 7382.929554] [<c115cd9c>] dpm_sysfs_remove+0x12/0x16 [ 7382.929563] [<c115c4b3>] device_pm_remove+0x21/0x62 [ 7382.929570] [<c1156f09>] device_del+0x82/0x291 [ 7382.929578] [<c1157123>] device_unregister+0xb/0x15 [ 7382.929585] [<c115731e>] device_destroy+0x8d/0x9a [ 7382.929592] [<c1142856>] vcs_remove_sysfs+0x1c/0x3a [ 7382.929600] [<c11491b9>] con_close+0x5e/0x6b [ 7382.929607] [<c113aba2>] release_dev+0x4c4/0x6ce [ 7382.929614] [<c113b21a>] tty_release+0x12/0x1c [ 7382.929620] [<c10743bd>] __fput+0x145/0x16e [ 7382.929628] [<c1074276>] fput+0x19/0x1b [ 7382.929634] [<c1072af7>] filp_close+0x3c/0x75 [ 7382.929641] [<c1072b99>] sys_close+0x69/0xb4 [ 7382.929648] [<c1003eea>] syscall_call+0x7/0xb [ 7382.929654] [<ffffffff>] 0xffffffff [ 7382.929665] [ 7382.929666] -> #0 (tty_mutex){--..}: [ 7382.929670] [<c103b10b>] check_prev_add+0x34/0x281 [ 7382.929677] [<c103b3e3>] check_prevs_add+0x8b/0xe8 [ 7382.929684] [<c103cd6e>] __lock_acquire+0x65d/0xb51 [ 7382.929691] [<c103d6ca>] lock_acquire+0x62/0x81 [ 7382.929698] [<c12283d4>] __mutex_lock_slowpath+0x75/0x28c [ 7382.929706] [<c1228353>] mutex_lock+0x8/0xa [ 7382.929712] [<c10a4119>] print_warning+0x8c/0x15d [ 7382.929720] [<c10a48d7>] dquot_alloc_space+0x184/0x189 [ 7382.929727] [<c10b5b99>] ext3_new_blocks+0x44b/0x5a2 [ 7382.929735] [<c10b8128>] ext3_alloc_blocks+0x40/0xdf [ 7382.929741] [<c10b8217>] ext3_alloc_branch+0x50/0x21b [ 7382.929748] [<c10b86f5>] ext3_get_blocks_handle+0x1b8/0x367 [ 7382.929755] [<c10b891c>] ext3_get_block+0x78/0xe3 [ 7382.929761] [<c1094746>] __block_prepare_write+0x168/0x415 [ 7382.929770] [<c1095252>] block_prepare_write+0x28/0x3b [ 7382.929777] [<c10b8dc8>] ext3_prepare_write+0xe3/0x17e [ 7382.929784] [<c1055235>] generic_file_buffered_write+0x1b4/0x614 [ 7382.929792] [<c10558cf>] __generic_file_aio_write_nolock+0x23a/0x533 [ 7382.929800] [<c1055cd1>] generic_file_aio_write+0x58/0xc4 [ 7382.929807] [<c10b69d7>] ext3_file_write+0x2d/0xba [ 7382.929815] [<c107347f>] do_sync_write+0xc7/0x116 [ 7382.929822] [<c1073626>] vfs_write+0x158/0x15d [ 7382.929829] [<c10736cc>] sys_write+0x3d/0x64 [ 7382.929835] [<c1003eea>] syscall_call+0x7/0xb [ 7382.929842] [<ffffffff>] 0xffffffff [ 7382.929851] [ 7382.929852] other info that might help us debug this: [ 7382.929853] [ 7382.929855] 3 locks held by moo/17202: [ 7382.929857] #0: (&inode->i_mutex){--..}, at: [<c1228353>] mutex_lock+0x8/0xa [ 7382.929864] #1: (&ei->truncate_mutex){--..}, at: [<c1228353>] mutex_lock+0x8/0xa [ 7382.929871] #2: (&s->s_dquot.dqptr_sem){----}, at: [<c10a47a3>] dquot_alloc_space+0x50/0x189 [ 7382.929877] [ 7382.929878] stack backtrace: [ 7382.929880] [<c1004cac>] show_trace_log_lvl+0x1a/0x30 [ 7382.929885] [<c1004cd4>] show_trace+0x12/0x14 [ 7382.929888] [<c1004dce>] dump_stack+0x16/0x18 [ 7382.929892] [<c103aa6d>] print_circular_bug_tail+0x6f/0x71 [ 7382.929896] [<c103b10b>] check_prev_add+0x34/0x281 [ 7382.929900] [<c103b3e3>] check_prevs_add+0x8b/0xe8 [ 7382.929903] [<c103cd6e>] __lock_acquire+0x65d/0xb51 [ 7382.929907] [<c103d6ca>] lock_acquire+0x62/0x81 [ 7382.929911] [<c12283d4>] __mutex_lock_slowpath+0x75/0x28c [ 7382.929915] [<c1228353>] mutex_lock+0x8/0xa [ 7382.929919] [<c10a4119>] print_warning+0x8c/0x15d [ 7382.929923] [<c10a48d7>] dquot_alloc_space+0x184/0x189 [ 7382.929926] [<c10b5b99>] ext3_new_blocks+0x44b/0x5a2 [ 7382.929931] [<c10b8128>] ext3_alloc_blocks+0x40/0xdf [ 7382.929934] [<c10b8217>] ext3_alloc_branch+0x50/0x21b [ 7382.929937] [<c10b86f5>] ext3_get_blocks_handle+0x1b8/0x367 [ 7382.929941] [<c10b891c>] ext3_get_block+0x78/0xe3 [ 7382.929944] [<c1094746>] __block_prepare_write+0x168/0x415 [ 7382.929949] [<c1095252>] block_prepare_write+0x28/0x3b [ 7382.929953] [<c10b8dc8>] ext3_prepare_write+0xe3/0x17e [ 7382.929957] [<c1055235>] generic_file_buffered_write+0x1b4/0x614 [ 7382.929961] [<c10558cf>] __generic_file_aio_write_nolock+0x23a/0x533 [ 7382.929966] [<c1055cd1>] generic_file_aio_write+0x58/0xc4 [ 7382.929970] [<c10b69d7>] ext3_file_write+0x2d/0xba [ 7382.929974] [<c107347f>] do_sync_write+0xc7/0x116 [ 7382.929978] [<c1073626>] vfs_write+0x158/0x15d [ 7382.929981] [<c10736cc>] sys_write+0x3d/0x64 [ 7382.929985] [<c1003eea>] syscall_call+0x7/0xb [ 7382.929988] ======================= Folkert van Heusden -- Multitail - gibkaja utilita po sledovaniju log-fajlov i vyvoda kommand. Fil'trovanie, raskrašivanie, slijanie, vizual'noe sravnenie, i t.d. http://www.vanheusden.com/multitail/ ---------------------------------------------------------------------- Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [2.6.22] circular lock detected 2007-09-02 11:55 ` Andrew Morton 2007-09-02 17:11 ` Folkert van Heusden @ 2007-09-03 12:27 ` Jan Kara 2007-09-03 12:49 ` Andrew Morton 1 sibling, 1 reply; 11+ messages in thread From: Jan Kara @ 2007-09-03 12:27 UTC (permalink / raw) To: Andrew Morton; +Cc: Folkert van Heusden, linux-kernel, Jan Kara, Peter Zijlstra > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote: > > Hi, > > > > 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2). > > > > [ 346.314640] ======================================================= > > [ 346.314758] [ INFO: possible circular locking dependency detected ] > > [ 346.314815] 2.6.22 #5 > > [ 346.314862] ------------------------------------------------------- > > [ 346.314920] tor/2421 is trying to acquire lock: > > [ 346.314973] (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa > > [ 346.315193] > > [ 346.315195] but task is already holding lock: > > [ 346.316203] (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189 > > [ 346.316457] > > [ 346.316458] which lock already depends on the new lock. > > [ 346.316461] > > [ 346.316585] > > [ 346.316586] the existing dependency chain (in reverse order) is: > > [ 346.316675] > > [ 346.316676] -> #4 (&s->s_dquot.dqptr_sem){----}: > > [ 346.316871] [<c103b2ee>] check_prev_add+0x15b/0x281 > > [ 346.317204] [<c103b49f>] check_prevs_add+0x8b/0xe8 > > [ 346.317535] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > > [ 346.317873] [<c103d786>] lock_acquire+0x62/0x81 > > [ 346.318195] [<c10341af>] down_read+0x2b/0x3d > > [ 346.318527] [<c10a485f>] dquot_alloc_space+0x50/0x189 > > [ 346.318854] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2 > > [ 346.319187] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf > > [ 346.319513] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b > > [ 346.319850] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367 > > [ 346.320184] [<c10b8ada>] ext3_getblk+0x97/0x228 > > [ 346.320530] [<c10b8c85>] ext3_bread+0x1a/0x78 > > [ 346.320870] [<c10bde64>] ext3_mkdir+0xf4/0x270 > > [ 346.321188] [<c107cdf6>] vfs_mkdir+0xb3/0x161 > > [ 346.321506] [<c107cf30>] sys_mkdirat+0x8c/0xc4 > > [ 346.321820] [<c107cf88>] sys_mkdir+0x20/0x22 > > [ 346.322131] [<c1003eea>] syscall_call+0x7/0xb > > [ 346.322446] [<ffffffff>] 0xffffffff > > [ 346.322779] > > [ 346.322780] -> #3 (&ei->truncate_mutex){--..}: > > [ 346.322965] [<c103b2ee>] check_prev_add+0x15b/0x281 > > [ 346.323278] [<c103b49f>] check_prevs_add+0x8b/0xe8 > > [ 346.323602] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > > [ 346.323919] [<c103d786>] lock_acquire+0x62/0x81 > > [ 346.324234] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c > > [ 346.324857] [<c121aa83>] mutex_lock+0x8/0xa > > [ 346.325177] [<c10ba380>] ext3_truncate+0x170/0x468 > > [ 346.325501] [<c1060e9a>] vmtruncate+0xa6/0x116 > > [ 346.325828] [<c1087937>] inode_setattr+0x145/0x16c > > [ 346.326150] [<c10bb301>] ext3_setattr+0x150/0x22f > > [ 346.326479] [<c1087cb0>] notify_change+0x352/0x386 > > [ 346.326802] [<c1071af4>] do_truncate+0x5c/0x7f > > [ 346.327116] [<c107c2e3>] may_open+0x1ec/0x231 > > [ 346.327434] [<c107c4a4>] open_namei+0xdd/0x5a2 > > [ 346.327770] [<c107275b>] do_filp_open+0x2c/0x53 > > [ 346.328090] [<c1072a90>] do_sys_open+0x52/0xd8 > > [ 346.328408] [<c1072b32>] sys_open+0x1c/0x1e > > [ 346.328727] [<c1003eea>] syscall_call+0x7/0xb > > [ 346.329042] [<ffffffff>] 0xffffffff > > [ 346.329368] > > [ 346.329370] -> #2 (&inode->i_alloc_sem){--..}: > > [ 346.329556] [<c103b2ee>] check_prev_add+0x15b/0x281 > > [ 346.329879] [<c103b49f>] check_prevs_add+0x8b/0xe8 > > [ 346.330192] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > > [ 346.330508] [<c103d786>] lock_acquire+0x62/0x81 > > [ 346.330826] [<c1034245>] down_write+0x2b/0x45 > > [ 346.331137] [<c1087c37>] notify_change+0x2d9/0x386 > > [ 346.331453] [<c1071af4>] do_truncate+0x5c/0x7f > > [ 346.331769] [<c107c2e3>] may_open+0x1ec/0x231 > > [ 346.332087] [<c107c4a4>] open_namei+0xdd/0x5a2 > > [ 346.332401] [<c107275b>] do_filp_open+0x2c/0x53 > > [ 346.332720] [<c1072a90>] do_sys_open+0x52/0xd8 > > [ 346.333037] [<c1072b32>] sys_open+0x1c/0x1e > > [ 346.333600] [<c1003eea>] syscall_call+0x7/0xb > > [ 346.333912] [<ffffffff>] 0xffffffff > > [ 346.334232] > > [ 346.334233] -> #1 (&sysfs_inode_imutex_key){--..}: > > [ 346.334422] [<c103b2ee>] check_prev_add+0x15b/0x281 > > [ 346.334734] [<c103b49f>] check_prevs_add+0x8b/0xe8 > > [ 346.335046] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > > [ 346.335361] [<c103d786>] lock_acquire+0x62/0x81 > > [ 346.335685] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c > > [ 346.336001] [<c121aa83>] mutex_lock+0x8/0xa > > [ 346.336313] [<c10b2828>] remove_dir+0x2c/0xf1 > > [ 346.336629] [<c10b28f5>] sysfs_remove_subdir+0x8/0xa > > [ 346.336941] [<c10b3ccf>] sysfs_remove_group+0x67/0x78 > > [ 346.337256] [<c115cc58>] dpm_sysfs_remove+0x12/0x16 > > [ 346.337574] [<c115c36f>] device_pm_remove+0x21/0x62 > > [ 346.337888] [<c1156dc5>] device_del+0x82/0x291 > > [ 346.338199] [<c1156fdf>] device_unregister+0xb/0x15 > > [ 346.338516] [<c11571da>] device_destroy+0x8d/0x9a > > [ 346.338827] [<c1142712>] vcs_remove_sysfs+0x1c/0x3a > > [ 346.339142] [<c1149075>] con_close+0x5e/0x6b > > [ 346.339458] [<c113aa5e>] release_dev+0x4c4/0x6ce > > [ 346.339767] [<c113b0d6>] tty_release+0x12/0x1c > > [ 346.340078] [<c1074479>] __fput+0x145/0x16e > > [ 346.340392] [<c1074332>] fput+0x19/0x1b > > [ 346.340701] [<c1072bb3>] filp_close+0x3c/0x75 > > [ 346.341012] [<c1072c55>] sys_close+0x69/0xb4 > > [ 346.341325] [<c1003eea>] syscall_call+0x7/0xb > > [ 346.341892] [<ffffffff>] 0xffffffff > > [ 346.342213] > > [ 346.342214] -> #0 (tty_mutex){--..}: > > [ 346.342397] [<c103b1c7>] check_prev_add+0x34/0x281 > > [ 346.342708] [<c103b49f>] check_prevs_add+0x8b/0xe8 > > [ 346.343017] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > > [ 346.343333] [<c103d786>] lock_acquire+0x62/0x81 > > [ 346.343646] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c > > [ 346.343956] [<c121aa83>] mutex_lock+0x8/0xa > > [ 346.344263] [<c10a41d5>] print_warning+0x8c/0x15d > > [ 346.344575] [<c10a4993>] dquot_alloc_space+0x184/0x189 > > [ 346.344883] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2 > > [ 346.345193] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf > > [ 346.345507] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b > > [ 346.345816] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367 > > [ 346.346127] [<c10b89d8>] ext3_get_block+0x78/0xe3 > > [ 346.346441] [<c1094802>] __block_prepare_write+0x168/0x415 > > [ 346.346750] [<c109530e>] block_prepare_write+0x28/0x3b > > [ 346.347057] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e > > [ 346.347369] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614 > > [ 346.347689] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533 > > [ 346.348002] [<c1055d8d>] generic_file_aio_write+0x58/0xc4 > > [ 346.348314] [<c10b6a93>] ext3_file_write+0x2d/0xba > > [ 346.348627] [<c107353b>] do_sync_write+0xc7/0x116 > > [ 346.348935] [<c10736e2>] vfs_write+0x158/0x15d > > [ 346.349246] [<c1073788>] sys_write+0x3d/0x64 > > [ 346.349560] [<c1003eea>] syscall_call+0x7/0xb > > [ 346.349869] [<ffffffff>] 0xffffffff > > [ 346.350438] > > [ 346.350440] other info that might help us debug this: > > [ 346.350442] > > [ 346.350562] 3 locks held by tor/2421: > > [ 346.350604] #0: (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa > > [ 346.350827] #1: (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa > > [ 346.351049] #2: (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189 > > [ 346.351274] > > [ 346.351276] stack backtrace: > > [ 346.351363] [<c1004cac>] show_trace_log_lvl+0x1a/0x30 > > [ 346.351446] [<c1004cd4>] show_trace+0x12/0x14 > > [ 346.351526] [<c1004dce>] dump_stack+0x16/0x18 > > [ 346.351605] [<c103ab29>] print_circular_bug_tail+0x6f/0x71 > > [ 346.351686] [<c103b1c7>] check_prev_add+0x34/0x281 > > [ 346.351764] [<c103b49f>] check_prevs_add+0x8b/0xe8 > > [ 346.351843] [<c103ce2a>] __lock_acquire+0x65d/0xb51 > > [ 346.351921] [<c103d786>] lock_acquire+0x62/0x81 > > [ 346.352002] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c > > [ 346.352083] [<c121aa83>] mutex_lock+0x8/0xa > > [ 346.352160] [<c10a41d5>] print_warning+0x8c/0x15d > > [ 346.352240] [<c10a4993>] dquot_alloc_space+0x184/0x189 > > [ 346.352323] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2 > > [ 346.352402] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf > > [ 346.352479] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b > > [ 346.352558] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367 > > [ 346.352638] [<c10b89d8>] ext3_get_block+0x78/0xe3 > > [ 346.352720] [<c1094802>] __block_prepare_write+0x168/0x415 > > [ 346.352801] [<c109530e>] block_prepare_write+0x28/0x3b > > [ 346.352881] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e > > [ 346.352961] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614 > > [ 346.353041] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533 > > [ 346.353121] [<c1055d8d>] generic_file_aio_write+0x58/0xc4 > > [ 346.353200] [<c10b6a93>] ext3_file_write+0x2d/0xba > > [ 346.353281] [<c107353b>] do_sync_write+0xc7/0x116 > > [ 346.353364] [<c10736e2>] vfs_write+0x158/0x15d > > [ 346.353444] [<c1073788>] sys_write+0x3d/0x64 > > [ 346.353524] [<c1003eea>] syscall_call+0x7/0xb > > [ 346.353603] ======================= > > > Has been reported before, but I don't recall whether we fixed it. Jan, > do you know>? I think we at least found a solution: Teach lockdep that I_MUTEX for different filesystems is different. Peter Zilstra wrote a patch for that and Folkert even confirmed that it fixes the problem for him. I'm not sure what happened with the patch afterwards though. Adding Peter to CC :). Honza -- Jan Kara <jack@suse.cz> SuSE CR Labs ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [2.6.22] circular lock detected 2007-09-03 12:27 ` Jan Kara @ 2007-09-03 12:49 ` Andrew Morton 2007-09-03 14:01 ` Jan Kara 0 siblings, 1 reply; 11+ messages in thread From: Andrew Morton @ 2007-09-03 12:49 UTC (permalink / raw) To: Jan Kara; +Cc: folkert, linux-kernel, jack, a.p.zijlstra > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <jack@suse.cz> wrote: > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote: > > > Hi, > > > > > > 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2). > > > > > > [ 346.314640] ======================================================= > > > [ 346.314758] [ INFO: possible circular locking dependency detected ] > > > [ 346.314815] 2.6.22 #5 > > > [ 346.314862] ------------------------------------------------------- > > > [ 346.314920] tor/2421 is trying to acquire lock: > > > [ 346.314973] (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa > > > [ 346.315193] > > > [ 346.315195] but task is already holding lock: > > > [ 346.316203] (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189 > > > [ 346.316457] > > > [ 346.316458] which lock already depends on the new lock. > > ... > > > > [ 346.353603] ======================= > > > > > Has been reported before, but I don't recall whether we fixed it. Jan, > > do you know>? > I think we at least found a solution: Teach lockdep that I_MUTEX for > different filesystems is different. Peter Zilstra wrote a patch for that > and Folkert even confirmed that it fixes the problem for him. I'm not > sure what happened with the patch afterwards though. Adding Peter to CC > :). But this is a tty_lock-versus-dqptr_sem ranking error. Unrelated to i_mutex? ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [2.6.22] circular lock detected 2007-09-03 12:49 ` Andrew Morton @ 2007-09-03 14:01 ` Jan Kara 2007-09-25 8:02 ` Peter Zijlstra 0 siblings, 1 reply; 11+ messages in thread From: Jan Kara @ 2007-09-03 14:01 UTC (permalink / raw) To: Andrew Morton; +Cc: folkert, linux-kernel, a.p.zijlstra On Mon 03-09-07 05:49:59, Andrew Morton wrote: > > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <jack@suse.cz> wrote: > > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote: > > > > Hi, > > > > > > > > 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2). > > > > > > > > [ 346.314640] ======================================================= > > > > [ 346.314758] [ INFO: possible circular locking dependency detected ] > > > > [ 346.314815] 2.6.22 #5 > > > > [ 346.314862] ------------------------------------------------------- > > > > [ 346.314920] tor/2421 is trying to acquire lock: > > > > [ 346.314973] (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa > > > > [ 346.315193] > > > > [ 346.315195] but task is already holding lock: > > > > [ 346.316203] (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189 > > > > [ 346.316457] > > > > [ 346.316458] which lock already depends on the new lock. > > > > ... > > > > > > [ 346.353603] ======================= > > > > > > > Has been reported before, but I don't recall whether we fixed it. Jan, > > > do you know>? > > I think we at least found a solution: Teach lockdep that I_MUTEX for > > different filesystems is different. Peter Zilstra wrote a patch for that > > and Folkert even confirmed that it fixes the problem for him. I'm not > > sure what happened with the patch afterwards though. Adding Peter to CC > > :). > > But this is a tty_lock-versus-dqptr_sem ranking error. Unrelated to i_mutex? The final report is for this ranking but the locking chain (if I understand it right) is: tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir) i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate) truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space) So it complains about tty_mutex vs dqptr_sem (I don't know why it does not complain about tty_mutex vs i_mutex) but the wrong link in the chain is that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate() [ext3] are different and should never depend on each other... Honza -- Jan Kara <jack@suse.cz> SuSE CR Labs ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [2.6.22] circular lock detected 2007-09-03 14:01 ` Jan Kara @ 2007-09-25 8:02 ` Peter Zijlstra 2007-09-25 8:46 ` Jan Kara 0 siblings, 1 reply; 11+ messages in thread From: Peter Zijlstra @ 2007-09-25 8:02 UTC (permalink / raw) To: Jan Kara; +Cc: Andrew Morton, folkert, linux-kernel On Mon, 3 Sep 2007 16:01:35 +0200 Jan Kara <jack@suse.cz> wrote: > On Mon 03-09-07 05:49:59, Andrew Morton wrote: > > > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <jack@suse.cz> wrote: > > > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote: > > > > Has been reported before, but I don't recall whether we fixed it. Jan, > > > > do you know>? > > > I think we at least found a solution: Teach lockdep that I_MUTEX for > > > different filesystems is different. Peter Zilstra wrote a patch for that > > > and Folkert even confirmed that it fixes the problem for him. I'm not > > > sure what happened with the patch afterwards though. Adding Peter to CC > > > :). > > > > But this is a tty_lock-versus-dqptr_sem ranking error. Unrelated to i_mutex? > The final report is for this ranking but the locking chain (if I understand it > right) is: > tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir) > i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate) > truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space) > > So it complains about tty_mutex vs dqptr_sem (I don't know why it does not > complain about tty_mutex vs i_mutex) but the wrong link in the chain is > that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate() > [ext3] are different and should never depend on each other... > Found it again. --- Give each filesystem its own inode lock class. The various filesystems have different locking order wrt the inode locks; esp. the pseudo filesystems differ from the rest. Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> --- fs/inode.c | 12 +++++++++--- include/linux/fs.h | 5 +++++ 2 files changed, 14 insertions(+), 3 deletions(-) Index: linux-2.6/fs/inode.c =================================================================== --- linux-2.6.orig/fs/inode.c +++ linux-2.6/fs/inode.c @@ -142,6 +142,15 @@ static struct inode *alloc_inode(struct return NULL; } + spin_lock_init(&inode->i_lock); + lockdep_set_class(&inode->i_lock, &sb->s_type->i_lock_key); + + mutex_init(&inode->i_mutex); + lockdep_set_class(&inode->i_mutex, &sb->s_type->i_mutex_key); + + init_rwsem(&inode->i_alloc_sem); + lockdep_set_class(&inode->i_alloc_sem, &sb->s_type->i_alloc_sem_key); + mapping->a_ops = &empty_aops; mapping->host = inode; mapping->flags = 0; @@ -190,8 +199,6 @@ void inode_init_once(struct inode *inode INIT_HLIST_NODE(&inode->i_hash); INIT_LIST_HEAD(&inode->i_dentry); INIT_LIST_HEAD(&inode->i_devices); - mutex_init(&inode->i_mutex); - init_rwsem(&inode->i_alloc_sem); INIT_RADIX_TREE(&inode->i_data.page_tree, GFP_ATOMIC); rwlock_init(&inode->i_data.tree_lock); spin_lock_init(&inode->i_data.i_mmap_lock); @@ -199,7 +206,6 @@ void inode_init_once(struct inode *inode spin_lock_init(&inode->i_data.private_lock); INIT_RAW_PRIO_TREE_ROOT(&inode->i_data.i_mmap); INIT_LIST_HEAD(&inode->i_data.i_mmap_nonlinear); - spin_lock_init(&inode->i_lock); i_size_ordered_init(inode); #ifdef CONFIG_INOTIFY INIT_LIST_HEAD(&inode->inotify_watches); Index: linux-2.6/include/linux/fs.h =================================================================== --- linux-2.6.orig/include/linux/fs.h +++ linux-2.6/include/linux/fs.h @@ -1302,8 +1302,13 @@ struct file_system_type { struct module *owner; struct file_system_type * next; struct list_head fs_supers; + struct lock_class_key s_lock_key; struct lock_class_key s_umount_key; + + struct lock_class_key i_lock_key; + struct lock_class_key i_mutex_key; + struct lock_class_key i_alloc_sem_key; }; extern int get_sb_bdev(struct file_system_type *fs_type, ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [2.6.22] circular lock detected 2007-09-25 8:02 ` Peter Zijlstra @ 2007-09-25 8:46 ` Jan Kara 2008-01-04 7:40 ` Simon Arlott 0 siblings, 1 reply; 11+ messages in thread From: Jan Kara @ 2007-09-25 8:46 UTC (permalink / raw) To: Peter Zijlstra; +Cc: Andrew Morton, folkert, linux-kernel On Tue 25-09-07 10:02:43, Peter Zijlstra wrote: > On Mon, 3 Sep 2007 16:01:35 +0200 Jan Kara <jack@suse.cz> wrote: > > > On Mon 03-09-07 05:49:59, Andrew Morton wrote: > > > > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <jack@suse.cz> wrote: > > > > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote: > > > > > > Has been reported before, but I don't recall whether we fixed it. Jan, > > > > > do you know>? > > > > I think we at least found a solution: Teach lockdep that I_MUTEX for > > > > different filesystems is different. Peter Zilstra wrote a patch for that > > > > and Folkert even confirmed that it fixes the problem for him. I'm not > > > > sure what happened with the patch afterwards though. Adding Peter to CC > > > > :). > > > > > > But this is a tty_lock-versus-dqptr_sem ranking error. Unrelated to i_mutex? > > The final report is for this ranking but the locking chain (if I understand it > > right) is: > > tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir) > > i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate) > > truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space) > > > > So it complains about tty_mutex vs dqptr_sem (I don't know why it does not > > complain about tty_mutex vs i_mutex) but the wrong link in the chain is > > that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate() > > [ext3] are different and should never depend on each other... > > > > Found it again. Cool, thanks Peter. Andrew, would you put it into -mm? This should take care of the false lockdep warnings from the quota code. If I recall correctly, one of the reporters even confirmed it fixes the problem for him. The patch looks fine. You can add: Signed-off-by: Jan Kara <jack@suse.cz> Honza > Give each filesystem its own inode lock class. The various filesystems have > different locking order wrt the inode locks; esp. the pseudo filesystems > differ from the rest. > > Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> > --- > fs/inode.c | 12 +++++++++--- > include/linux/fs.h | 5 +++++ > 2 files changed, 14 insertions(+), 3 deletions(-) > > Index: linux-2.6/fs/inode.c > =================================================================== > --- linux-2.6.orig/fs/inode.c > +++ linux-2.6/fs/inode.c > @@ -142,6 +142,15 @@ static struct inode *alloc_inode(struct > return NULL; > } > > + spin_lock_init(&inode->i_lock); > + lockdep_set_class(&inode->i_lock, &sb->s_type->i_lock_key); > + > + mutex_init(&inode->i_mutex); > + lockdep_set_class(&inode->i_mutex, &sb->s_type->i_mutex_key); > + > + init_rwsem(&inode->i_alloc_sem); > + lockdep_set_class(&inode->i_alloc_sem, &sb->s_type->i_alloc_sem_key); > + > mapping->a_ops = &empty_aops; > mapping->host = inode; > mapping->flags = 0; > @@ -190,8 +199,6 @@ void inode_init_once(struct inode *inode > INIT_HLIST_NODE(&inode->i_hash); > INIT_LIST_HEAD(&inode->i_dentry); > INIT_LIST_HEAD(&inode->i_devices); > - mutex_init(&inode->i_mutex); > - init_rwsem(&inode->i_alloc_sem); > INIT_RADIX_TREE(&inode->i_data.page_tree, GFP_ATOMIC); > rwlock_init(&inode->i_data.tree_lock); > spin_lock_init(&inode->i_data.i_mmap_lock); > @@ -199,7 +206,6 @@ void inode_init_once(struct inode *inode > spin_lock_init(&inode->i_data.private_lock); > INIT_RAW_PRIO_TREE_ROOT(&inode->i_data.i_mmap); > INIT_LIST_HEAD(&inode->i_data.i_mmap_nonlinear); > - spin_lock_init(&inode->i_lock); > i_size_ordered_init(inode); > #ifdef CONFIG_INOTIFY > INIT_LIST_HEAD(&inode->inotify_watches); > Index: linux-2.6/include/linux/fs.h > =================================================================== > --- linux-2.6.orig/include/linux/fs.h > +++ linux-2.6/include/linux/fs.h > @@ -1302,8 +1302,13 @@ struct file_system_type { > struct module *owner; > struct file_system_type * next; > struct list_head fs_supers; > + > struct lock_class_key s_lock_key; > struct lock_class_key s_umount_key; > + > + struct lock_class_key i_lock_key; > + struct lock_class_key i_mutex_key; > + struct lock_class_key i_alloc_sem_key; > }; > > extern int get_sb_bdev(struct file_system_type *fs_type, > -- Jan Kara <jack@suse.cz> SUSE Labs, CR ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [2.6.22] circular lock detected 2007-09-25 8:46 ` Jan Kara @ 2008-01-04 7:40 ` Simon Arlott 2008-01-04 9:21 ` Peter Zijlstra 0 siblings, 1 reply; 11+ messages in thread From: Simon Arlott @ 2008-01-04 7:40 UTC (permalink / raw) To: Jan Kara; +Cc: Peter Zijlstra, Andrew Morton, folkert, linux-kernel On 25/09/07 09:46, Jan Kara wrote: > On Tue 25-09-07 10:02:43, Peter Zijlstra wrote: >> On Mon, 3 Sep 2007 16:01:35 +0200 Jan Kara <jack@suse.cz> wrote: >> >> > On Mon 03-09-07 05:49:59, Andrew Morton wrote: >> > > > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <jack@suse.cz> wrote: >> > > > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote: >> >> > > > > Has been reported before, but I don't recall whether we fixed it. Jan, >> > > > > do you know>? >> > > > I think we at least found a solution: Teach lockdep that I_MUTEX for >> > > > different filesystems is different. Peter Zilstra wrote a patch for that >> > > > and Folkert even confirmed that it fixes the problem for him. I'm not >> > > > sure what happened with the patch afterwards though. Adding Peter to CC >> > > > :). >> > > >> > > But this is a tty_lock-versus-dqptr_sem ranking error. Unrelated to i_mutex? >> > The final report is for this ranking but the locking chain (if I understand it >> > right) is: >> > tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir) >> > i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate) >> > truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space) >> > >> > So it complains about tty_mutex vs dqptr_sem (I don't know why it does not >> > complain about tty_mutex vs i_mutex) but the wrong link in the chain is >> > that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate() >> > [ext3] are different and should never depend on each other... >> > >> >> Found it again. > Cool, thanks Peter. Andrew, would you put it into -mm? This should take care of > the false lockdep warnings from the quota code. If I recall correctly, one > of the reporters even confirmed it fixes the problem for him. > The patch looks fine. You can add: > Signed-off-by: Jan Kara <jack@suse.cz> > > Honza > >> Give each filesystem its own inode lock class. The various filesystems have >> different locking order wrt the inode locks; esp. the pseudo filesystems >> differ from the rest. >> >> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> This patch still doesn't exist in 2.6.23.9 and the warning isn't fixed... 03 00:31:34 [1905219.899008] ======================================================= 03 00:31:34 [1905219.907136] [ INFO: possible circular locking dependency detected ] 03 00:31:34 [1905219.913569] 2.6.23.9-git #43 03 00:31:34 [1905219.916624] ------------------------------------------------------- 03 00:31:34 [1905219.923059] bacula-sd/32639 is trying to acquire lock: 03 00:31:34 [1905219.928364] (tty_mutex){--..}, at: [<80389fd7>] mutex_lock+0x1c/0x1f 03 00:31:34 [1905219.935047] 03 00:31:34 [1905219.935049] but task is already holding lock: 03 00:31:34 [1905219.941238] (&s->s_dquot.dqptr_sem){----}, at: [<801839bf>] dquot_alloc_space+0x42/0x15b 03 00:31:34 [1905219.949646] 03 00:31:34 [1905219.949649] which lock already depends on the new lock. 03 00:31:34 [1905219.949652] 03 00:31:34 [1905219.958367] 03 00:31:34 [1905219.958369] the existing dependency chain (in reverse order) is: 03 00:31:34 [1905219.966204] 03 00:31:34 [1905219.966206] -> #4 (&s->s_dquot.dqptr_sem){----}: 03 00:31:34 [1905219.972714] [<80131c5b>] __lock_acquire+0x9ba/0xb96 03 00:31:34 [1905219.978511] [<80132200>] lock_acquire+0x5d/0x75 03 00:31:34 [1905219.983957] [<8012ac3c>] down_read+0x3a/0x4b 03 00:31:34 [1905219.989252] [<801839bf>] dquot_alloc_space+0x42/0x15b 03 00:31:34 [1905219.995217] [<801920e1>] ext3_new_blocks+0x83/0x5ba 03 00:31:34 [1905220.001009] [<80195111>] ext3_get_blocks_handle+0x386/0x822 03 00:31:34 [1905220.007495] [<80195889>] ext3_get_block+0xac/0xc5 03 00:31:34 [1905220.013113] [<80174e54>] __block_prepare_write+0x151/0x3c6 03 00:31:34 [1905220.019603] [<801750ed>] block_prepare_write+0x24/0x32 03 00:31:34 [1905220.025654] [<80196a43>] ext3_prepare_write+0x98/0x153 03 00:31:34 [1905220.031705] [<8013cbbe>] generic_file_buffered_write+0x219/0x57c 03 00:31:34 [1905220.038630] [<8013d384>] __generic_file_aio_write_nolock+0x463/0x4b3 03 00:31:34 [1905220.045894] [<8013d42a>] generic_file_aio_write+0x56/0xb4 03 00:31:34 [1905220.052292] [<8019311b>] ext3_file_write+0x27/0x99 03 00:31:34 [1905220.057996] [<80157742>] do_sync_write+0xc4/0x101 03 00:31:34 [1905220.063615] [<80157f03>] vfs_write+0xaf/0x138 03 00:31:34 [1905220.068885] [<8015840f>] sys_write+0x3d/0x61 03 00:31:34 [1905220.074071] [<80102702>] sysenter_past_esp+0x5f/0x99 03 00:31:34 [1905220.080046] [<ffffffff>] 0xffffffff 03 00:31:34 [1905220.084456] 03 00:31:34 [1905220.084458] -> #3 (&ei->truncate_mutex){--..}: 03 00:31:34 [1905220.090781] [<80131c5b>] __lock_acquire+0x9ba/0xb96 03 00:31:34 [1905220.096573] [<80132200>] lock_acquire+0x5d/0x75 03 00:31:34 [1905220.102020] [<80389e25>] __mutex_lock_slowpath+0xdb/0x271 03 00:31:34 [1905220.108418] [<80389fd7>] mutex_lock+0x1c/0x1f 03 00:31:34 [1905220.113690] [<80196110>] ext3_truncate+0x49e/0x7ec 03 00:31:34 [1905220.119397] [<80148496>] vmtruncate+0x161/0x187 03 00:31:34 [1905220.124847] [<80169794>] inode_setattr+0x73/0x149 03 00:31:34 [1905220.130470] [<8019720c>] ext3_setattr+0x185/0x1e1 03 00:31:34 [1905220.136088] [<8016999a>] notify_change+0x130/0x2d2 03 00:31:34 [1905220.141883] [<80156c7b>] do_truncate+0x60/0x79 03 00:31:34 [1905220.147245] [<8015f377>] may_open+0x1b9/0x201 03 00:31:34 [1905220.152522] [<801612e3>] open_namei+0x240/0x539 03 00:31:34 [1905220.157965] [<8015651b>] do_filp_open+0x26/0x3b 03 00:31:34 [1905220.163411] [<80156575>] do_sys_open+0x45/0xc9 03 00:31:34 [1905220.168857] [<80156631>] sys_open+0x1c/0x1e 03 00:31:34 [1905220.173955] [<80102702>] sysenter_past_esp+0x5f/0x99 03 00:31:34 [1905220.179830] [<ffffffff>] 0xffffffff 03 00:31:34 [1905220.184233] 03 00:31:34 [1905220.184235] -> #2 (&inode->i_alloc_sem){--..}: 03 00:31:34 [1905220.190559] [<80131c5b>] __lock_acquire+0x9ba/0xb96 03 00:31:34 [1905220.196436] [<80132200>] lock_acquire+0x5d/0x75 03 00:31:34 [1905220.201884] [<8012acd6>] down_write+0x3a/0x53 03 00:31:34 [1905220.207152] [<80169952>] notify_change+0xe8/0x2d2 03 00:31:34 [1905220.212772] [<80156c7b>] do_truncate+0x60/0x79 03 00:31:34 [1905220.218130] [<8015f377>] may_open+0x1b9/0x201 03 00:31:34 [1905220.223485] [<801612e3>] open_namei+0x240/0x539 03 00:31:34 [1905220.228926] [<8015651b>] do_filp_open+0x26/0x3b 03 00:31:34 [1905220.234372] [<80156575>] do_sys_open+0x45/0xc9 03 00:31:34 [1905220.239727] [<80156631>] sys_open+0x1c/0x1e 03 00:31:34 [1905220.244819] [<80102702>] sysenter_past_esp+0x5f/0x99 03 00:31:34 [1905220.250760] [<ffffffff>] 0xffffffff 03 00:31:34 [1905220.255156] 03 00:31:34 [1905220.255158] -> #1 (&inode->i_mutex){--..}: 03 00:31:34 [1905220.261135] [<80131c5b>] __lock_acquire+0x9ba/0xb96 03 00:31:34 [1905220.268005] [<80132200>] lock_acquire+0x5d/0x75 03 00:31:34 [1905220.273448] [<80389e25>] __mutex_lock_slowpath+0xdb/0x271 03 00:31:34 [1905220.279752] [<80389fd7>] mutex_lock+0x1c/0x1f 03 00:31:34 [1905220.285024] [<80190cf6>] get_node+0x1d/0x46 03 00:31:34 [1905220.290119] [<80190e84>] devpts_get_tty+0xb/0x3b 03 00:31:34 [1905220.295646] [<80214fb5>] init_dev+0x21/0x4a2 03 00:31:34 [1905220.300837] [<80217c95>] ptmx_open+0xfd/0x1bc 03 00:31:34 [1905220.306102] [<80159f91>] chrdev_open+0x14a/0x180 03 00:31:34 [1905220.311618] [<801563b2>] __dentry_open+0xfd/0x1af 03 00:31:34 [1905220.317232] [<801564e5>] nameidata_to_filp+0x27/0x37 03 00:31:34 [1905220.323097] [<80156528>] do_filp_open+0x33/0x3b 03 00:31:34 [1905220.328535] [<80156575>] do_sys_open+0x45/0xc9 03 00:31:34 [1905220.333874] [<80156631>] sys_open+0x1c/0x1e 03 00:31:34 [1905220.338981] [<80102702>] sysenter_past_esp+0x5f/0x99 03 00:31:34 [1905220.344855] [<ffffffff>] 0xffffffff 03 00:31:34 [1905220.349256] 03 00:31:34 [1905220.349258] -> #0 (tty_mutex){--..}: 03 00:31:34 [1905220.354717] [<80131b4b>] __lock_acquire+0x8aa/0xb96 03 00:31:34 [1905220.360505] [<80132200>] lock_acquire+0x5d/0x75 03 00:31:34 [1905220.365956] [<80389e25>] __mutex_lock_slowpath+0xdb/0x271 03 00:31:34 [1905220.372258] [<80389fd7>] mutex_lock+0x1c/0x1f 03 00:31:34 [1905220.377517] [<80182ea9>] print_warning+0x92/0x137 03 00:31:34 [1905220.383139] [<80183ab8>] dquot_alloc_space+0x13b/0x15b 03 00:31:34 [1905220.389176] [<801920e1>] ext3_new_blocks+0x83/0x5ba 03 00:31:34 [1905220.394961] [<80195111>] ext3_get_blocks_handle+0x386/0x822 03 00:31:34 [1905220.401429] [<80195889>] ext3_get_block+0xac/0xc5 03 00:31:34 [1905220.407047] [<80174e54>] __block_prepare_write+0x151/0x3c6 03 00:31:34 [1905220.413455] [<801750ed>] block_prepare_write+0x24/0x32 03 00:31:34 [1905220.419506] [<80196a43>] ext3_prepare_write+0x98/0x153 03 00:31:34 [1905220.425553] [<8013cbbe>] generic_file_buffered_write+0x219/0x57c 03 00:31:34 [1905220.432465] [<8013d384>] __generic_file_aio_write_nolock+0x463/0x4b3 03 00:31:34 [1905220.439724] [<8013d42a>] generic_file_aio_write+0x56/0xb4 03 00:31:34 [1905220.446027] [<8019311b>] ext3_file_write+0x27/0x99 03 00:31:34 [1905220.451728] [<80157742>] do_sync_write+0xc4/0x101 03 00:31:34 [1905220.457338] [<80157f03>] vfs_write+0xaf/0x138 03 00:31:34 [1905220.462600] [<8015840f>] sys_write+0x3d/0x61 03 00:31:34 [1905220.467771] [<80102702>] sysenter_past_esp+0x5f/0x99 03 00:31:34 [1905220.473639] [<ffffffff>] 0xffffffff 03 00:31:34 [1905220.478040] 03 00:31:34 [1905220.478043] other info that might help us debug this: 03 00:31:34 [1905220.478046] 03 00:31:34 [1905220.486588] 3 locks held by bacula-sd/32639: 03 00:31:34 [1905220.491025] #0: (&inode->i_mutex){--..}, at: [<80389fd7>] mutex_lock+0x1c/0x1f 03 00:31:34 [1905220.498668] #1: (&ei->truncate_mutex){--..}, at: [<80389fd7>] mutex_lock+0x1c/0x1f 03 00:31:34 [1905220.506659] #2: (&s->s_dquot.dqptr_sem){----}, at: [<801839bf>] dquot_alloc_space+0x42/0x15b 03 00:31:34 [1905220.515511] 03 00:31:34 [1905220.515513] stack backtrace: 03 00:31:34 [1905220.520230] [<80103694>] show_trace_log_lvl+0x1a/0x2f 03 00:31:34 [1905220.525555] [<8010421b>] show_trace+0x12/0x14 03 00:31:34 [1905220.530185] [<80104232>] dump_stack+0x15/0x17 03 00:31:34 [1905220.534817] [<801303c4>] print_circular_bug_tail+0x5e/0x67 03 00:31:34 [1905220.540580] [<80131b4b>] __lock_acquire+0x8aa/0xb96 03 00:31:34 [1905220.545731] [<80132200>] lock_acquire+0x5d/0x75 03 00:31:34 [1905220.550536] [<80389e25>] __mutex_lock_slowpath+0xdb/0x271 03 00:31:34 [1905220.556203] [<80389fd7>] mutex_lock+0x1c/0x1f 03 00:31:34 [1905220.560838] [<80182ea9>] print_warning+0x92/0x137 03 00:31:34 [1905220.565819] [<80183ab8>] dquot_alloc_space+0x13b/0x15b 03 00:31:34 [1905220.571230] [<801920e1>] ext3_new_blocks+0x83/0x5ba 03 00:31:34 [1905220.576381] [<80195111>] ext3_get_blocks_handle+0x386/0x822 03 00:31:34 [1905220.582226] [<80195889>] ext3_get_block+0xac/0xc5 03 00:31:34 [1905220.587205] [<80174e54>] __block_prepare_write+0x151/0x3c6 03 00:31:34 [1905220.592963] [<801750ed>] block_prepare_write+0x24/0x32 03 00:31:34 [1905220.598372] [<80196a43>] ext3_prepare_write+0x98/0x153 03 00:31:34 [1905220.603785] [<8013cbbe>] generic_file_buffered_write+0x219/0x57c 03 00:31:34 [1905220.610063] [<8013d384>] __generic_file_aio_write_nolock+0x463/0x4b3 03 00:31:34 [1905220.616687] [<8013d42a>] generic_file_aio_write+0x56/0xb4 03 00:31:34 [1905220.622362] [<8019311b>] ext3_file_write+0x27/0x99 03 00:31:34 [1905220.627426] [<80157742>] do_sync_write+0xc4/0x101 03 00:31:34 [1905220.632401] [<80157f03>] vfs_write+0xaf/0x138 03 00:31:34 [1905220.637033] [<8015840f>] sys_write+0x3d/0x61 03 00:31:34 [1905220.641574] [<80102702>] sysenter_past_esp+0x5f/0x99 03 00:31:34 [1905220.646811] ======================= >> --- >> fs/inode.c | 12 +++++++++--- >> include/linux/fs.h | 5 +++++ >> 2 files changed, 14 insertions(+), 3 deletions(-) >> >> Index: linux-2.6/fs/inode.c >> =================================================================== >> --- linux-2.6.orig/fs/inode.c >> +++ linux-2.6/fs/inode.c >> @@ -142,6 +142,15 @@ static struct inode *alloc_inode(struct >> return NULL; >> } >> >> + spin_lock_init(&inode->i_lock); >> + lockdep_set_class(&inode->i_lock, &sb->s_type->i_lock_key); >> + >> + mutex_init(&inode->i_mutex); >> + lockdep_set_class(&inode->i_mutex, &sb->s_type->i_mutex_key); >> + >> + init_rwsem(&inode->i_alloc_sem); >> + lockdep_set_class(&inode->i_alloc_sem, &sb->s_type->i_alloc_sem_key); >> + >> mapping->a_ops = &empty_aops; >> mapping->host = inode; >> mapping->flags = 0; >> @@ -190,8 +199,6 @@ void inode_init_once(struct inode *inode >> INIT_HLIST_NODE(&inode->i_hash); >> INIT_LIST_HEAD(&inode->i_dentry); >> INIT_LIST_HEAD(&inode->i_devices); >> - mutex_init(&inode->i_mutex); >> - init_rwsem(&inode->i_alloc_sem); >> INIT_RADIX_TREE(&inode->i_data.page_tree, GFP_ATOMIC); >> rwlock_init(&inode->i_data.tree_lock); >> spin_lock_init(&inode->i_data.i_mmap_lock); >> @@ -199,7 +206,6 @@ void inode_init_once(struct inode *inode >> spin_lock_init(&inode->i_data.private_lock); >> INIT_RAW_PRIO_TREE_ROOT(&inode->i_data.i_mmap); >> INIT_LIST_HEAD(&inode->i_data.i_mmap_nonlinear); >> - spin_lock_init(&inode->i_lock); >> i_size_ordered_init(inode); >> #ifdef CONFIG_INOTIFY >> INIT_LIST_HEAD(&inode->inotify_watches); >> Index: linux-2.6/include/linux/fs.h >> =================================================================== >> --- linux-2.6.orig/include/linux/fs.h >> +++ linux-2.6/include/linux/fs.h >> @@ -1302,8 +1302,13 @@ struct file_system_type { >> struct module *owner; >> struct file_system_type * next; >> struct list_head fs_supers; >> + >> struct lock_class_key s_lock_key; >> struct lock_class_key s_umount_key; >> + >> + struct lock_class_key i_lock_key; >> + struct lock_class_key i_mutex_key; >> + struct lock_class_key i_alloc_sem_key; >> }; >> >> extern int get_sb_bdev(struct file_system_type *fs_type, >> -- Simon Arlott ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [2.6.22] circular lock detected 2008-01-04 7:40 ` Simon Arlott @ 2008-01-04 9:21 ` Peter Zijlstra 0 siblings, 0 replies; 11+ messages in thread From: Peter Zijlstra @ 2008-01-04 9:21 UTC (permalink / raw) To: Simon Arlott; +Cc: Jan Kara, Andrew Morton, folkert, linux-kernel [-- Attachment #1: Type: text/plain, Size: 2478 bytes --] On Fri, 2008-01-04 at 07:40 +0000, Simon Arlott wrote: > On 25/09/07 09:46, Jan Kara wrote: > > On Tue 25-09-07 10:02:43, Peter Zijlstra wrote: > >> On Mon, 3 Sep 2007 16:01:35 +0200 Jan Kara <jack@suse.cz> wrote: > >> > >> > On Mon 03-09-07 05:49:59, Andrew Morton wrote: > >> > > > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <jack@suse.cz> wrote: > >> > > > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote: > >> > >> > > > > Has been reported before, but I don't recall whether we fixed it. Jan, > >> > > > > do you know>? > >> > > > I think we at least found a solution: Teach lockdep that I_MUTEX for > >> > > > different filesystems is different. Peter Zilstra wrote a patch for that > >> > > > and Folkert even confirmed that it fixes the problem for him. I'm not > >> > > > sure what happened with the patch afterwards though. Adding Peter to CC > >> > > > :). > >> > > > >> > > But this is a tty_lock-versus-dqptr_sem ranking error. Unrelated to i_mutex? > >> > The final report is for this ranking but the locking chain (if I understand it > >> > right) is: > >> > tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir) > >> > i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate) > >> > truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space) > >> > > >> > So it complains about tty_mutex vs dqptr_sem (I don't know why it does not > >> > complain about tty_mutex vs i_mutex) but the wrong link in the chain is > >> > that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate() > >> > [ext3] are different and should never depend on each other... > >> > > >> > >> Found it again. > > Cool, thanks Peter. Andrew, would you put it into -mm? This should take care of > > the false lockdep warnings from the quota code. If I recall correctly, one > > of the reporters even confirmed it fixes the problem for him. > > The patch looks fine. You can add: > > Signed-off-by: Jan Kara <jack@suse.cz> > > > > Honza > > > >> Give each filesystem its own inode lock class. The various filesystems have > >> different locking order wrt the inode locks; esp. the pseudo filesystems > >> differ from the rest. > >> > >> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> > > This patch still doesn't exist in 2.6.23.9 and the warning isn't fixed... It would have been if you'd used .24-rc :-) [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2008-01-04 9:22 UTC | newest] Thread overview: 11+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2007-08-24 21:00 [2.6.22] circular lock detected Folkert van Heusden 2007-08-26 23:03 ` Michal Piotrowski 2007-09-02 11:55 ` Andrew Morton 2007-09-02 17:11 ` Folkert van Heusden 2007-09-03 12:27 ` Jan Kara 2007-09-03 12:49 ` Andrew Morton 2007-09-03 14:01 ` Jan Kara 2007-09-25 8:02 ` Peter Zijlstra 2007-09-25 8:46 ` Jan Kara 2008-01-04 7:40 ` Simon Arlott 2008-01-04 9:21 ` Peter Zijlstra
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox