From: Simon Arlott <simon@fire.lp0.eu>
To: Jan Kara <jack@suse.cz>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>,
Andrew Morton <akpm@linux-foundation.org>,
folkert@vanheusden.com, linux-kernel@vger.kernel.org
Subject: Re: [2.6.22] circular lock detected
Date: Fri, 04 Jan 2008 07:40:41 +0000 [thread overview]
Message-ID: <477DE2F9.2050208@simon.arlott.org.uk> (raw)
In-Reply-To: <20070925084630.GA412@duck.suse.cz>
On 25/09/07 09:46, Jan Kara wrote:
> On Tue 25-09-07 10:02:43, Peter Zijlstra wrote:
>> On Mon, 3 Sep 2007 16:01:35 +0200 Jan Kara <jack@suse.cz> wrote:
>>
>> > On Mon 03-09-07 05:49:59, Andrew Morton wrote:
>> > > > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <jack@suse.cz> wrote:
>> > > > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote:
>>
>> > > > > Has been reported before, but I don't recall whether we fixed it. Jan,
>> > > > > do you know>?
>> > > > I think we at least found a solution: Teach lockdep that I_MUTEX for
>> > > > different filesystems is different. Peter Zilstra wrote a patch for that
>> > > > and Folkert even confirmed that it fixes the problem for him. I'm not
>> > > > sure what happened with the patch afterwards though. Adding Peter to CC
>> > > > :).
>> > >
>> > > But this is a tty_lock-versus-dqptr_sem ranking error. Unrelated to i_mutex?
>> > The final report is for this ranking but the locking chain (if I understand it
>> > right) is:
>> > tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir)
>> > i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate)
>> > truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space)
>> >
>> > So it complains about tty_mutex vs dqptr_sem (I don't know why it does not
>> > complain about tty_mutex vs i_mutex) but the wrong link in the chain is
>> > that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate()
>> > [ext3] are different and should never depend on each other...
>> >
>>
>> Found it again.
> Cool, thanks Peter. Andrew, would you put it into -mm? This should take care of
> the false lockdep warnings from the quota code. If I recall correctly, one
> of the reporters even confirmed it fixes the problem for him.
> The patch looks fine. You can add:
> Signed-off-by: Jan Kara <jack@suse.cz>
>
> Honza
>
>> Give each filesystem its own inode lock class. The various filesystems have
>> different locking order wrt the inode locks; esp. the pseudo filesystems
>> differ from the rest.
>>
>> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
This patch still doesn't exist in 2.6.23.9 and the warning isn't fixed...
03 00:31:34 [1905219.899008] =======================================================
03 00:31:34 [1905219.907136] [ INFO: possible circular locking dependency detected ]
03 00:31:34 [1905219.913569] 2.6.23.9-git #43
03 00:31:34 [1905219.916624] -------------------------------------------------------
03 00:31:34 [1905219.923059] bacula-sd/32639 is trying to acquire lock:
03 00:31:34 [1905219.928364] (tty_mutex){--..}, at: [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905219.935047]
03 00:31:34 [1905219.935049] but task is already holding lock:
03 00:31:34 [1905219.941238] (&s->s_dquot.dqptr_sem){----}, at: [<801839bf>] dquot_alloc_space+0x42/0x15b
03 00:31:34 [1905219.949646]
03 00:31:34 [1905219.949649] which lock already depends on the new lock.
03 00:31:34 [1905219.949652]
03 00:31:34 [1905219.958367]
03 00:31:34 [1905219.958369] the existing dependency chain (in reverse order) is:
03 00:31:34 [1905219.966204]
03 00:31:34 [1905219.966206] -> #4 (&s->s_dquot.dqptr_sem){----}:
03 00:31:34 [1905219.972714] [<80131c5b>] __lock_acquire+0x9ba/0xb96
03 00:31:34 [1905219.978511] [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905219.983957] [<8012ac3c>] down_read+0x3a/0x4b
03 00:31:34 [1905219.989252] [<801839bf>] dquot_alloc_space+0x42/0x15b
03 00:31:34 [1905219.995217] [<801920e1>] ext3_new_blocks+0x83/0x5ba
03 00:31:34 [1905220.001009] [<80195111>] ext3_get_blocks_handle+0x386/0x822
03 00:31:34 [1905220.007495] [<80195889>] ext3_get_block+0xac/0xc5
03 00:31:34 [1905220.013113] [<80174e54>] __block_prepare_write+0x151/0x3c6
03 00:31:34 [1905220.019603] [<801750ed>] block_prepare_write+0x24/0x32
03 00:31:34 [1905220.025654] [<80196a43>] ext3_prepare_write+0x98/0x153
03 00:31:34 [1905220.031705] [<8013cbbe>] generic_file_buffered_write+0x219/0x57c
03 00:31:34 [1905220.038630] [<8013d384>] __generic_file_aio_write_nolock+0x463/0x4b3
03 00:31:34 [1905220.045894] [<8013d42a>] generic_file_aio_write+0x56/0xb4
03 00:31:34 [1905220.052292] [<8019311b>] ext3_file_write+0x27/0x99
03 00:31:34 [1905220.057996] [<80157742>] do_sync_write+0xc4/0x101
03 00:31:34 [1905220.063615] [<80157f03>] vfs_write+0xaf/0x138
03 00:31:34 [1905220.068885] [<8015840f>] sys_write+0x3d/0x61
03 00:31:34 [1905220.074071] [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.080046] [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.084456]
03 00:31:34 [1905220.084458] -> #3 (&ei->truncate_mutex){--..}:
03 00:31:34 [1905220.090781] [<80131c5b>] __lock_acquire+0x9ba/0xb96
03 00:31:34 [1905220.096573] [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.102020] [<80389e25>] __mutex_lock_slowpath+0xdb/0x271
03 00:31:34 [1905220.108418] [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.113690] [<80196110>] ext3_truncate+0x49e/0x7ec
03 00:31:34 [1905220.119397] [<80148496>] vmtruncate+0x161/0x187
03 00:31:34 [1905220.124847] [<80169794>] inode_setattr+0x73/0x149
03 00:31:34 [1905220.130470] [<8019720c>] ext3_setattr+0x185/0x1e1
03 00:31:34 [1905220.136088] [<8016999a>] notify_change+0x130/0x2d2
03 00:31:34 [1905220.141883] [<80156c7b>] do_truncate+0x60/0x79
03 00:31:34 [1905220.147245] [<8015f377>] may_open+0x1b9/0x201
03 00:31:34 [1905220.152522] [<801612e3>] open_namei+0x240/0x539
03 00:31:34 [1905220.157965] [<8015651b>] do_filp_open+0x26/0x3b
03 00:31:34 [1905220.163411] [<80156575>] do_sys_open+0x45/0xc9
03 00:31:34 [1905220.168857] [<80156631>] sys_open+0x1c/0x1e
03 00:31:34 [1905220.173955] [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.179830] [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.184233]
03 00:31:34 [1905220.184235] -> #2 (&inode->i_alloc_sem){--..}:
03 00:31:34 [1905220.190559] [<80131c5b>] __lock_acquire+0x9ba/0xb96
03 00:31:34 [1905220.196436] [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.201884] [<8012acd6>] down_write+0x3a/0x53
03 00:31:34 [1905220.207152] [<80169952>] notify_change+0xe8/0x2d2
03 00:31:34 [1905220.212772] [<80156c7b>] do_truncate+0x60/0x79
03 00:31:34 [1905220.218130] [<8015f377>] may_open+0x1b9/0x201
03 00:31:34 [1905220.223485] [<801612e3>] open_namei+0x240/0x539
03 00:31:34 [1905220.228926] [<8015651b>] do_filp_open+0x26/0x3b
03 00:31:34 [1905220.234372] [<80156575>] do_sys_open+0x45/0xc9
03 00:31:34 [1905220.239727] [<80156631>] sys_open+0x1c/0x1e
03 00:31:34 [1905220.244819] [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.250760] [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.255156]
03 00:31:34 [1905220.255158] -> #1 (&inode->i_mutex){--..}:
03 00:31:34 [1905220.261135] [<80131c5b>] __lock_acquire+0x9ba/0xb96
03 00:31:34 [1905220.268005] [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.273448] [<80389e25>] __mutex_lock_slowpath+0xdb/0x271
03 00:31:34 [1905220.279752] [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.285024] [<80190cf6>] get_node+0x1d/0x46
03 00:31:34 [1905220.290119] [<80190e84>] devpts_get_tty+0xb/0x3b
03 00:31:34 [1905220.295646] [<80214fb5>] init_dev+0x21/0x4a2
03 00:31:34 [1905220.300837] [<80217c95>] ptmx_open+0xfd/0x1bc
03 00:31:34 [1905220.306102] [<80159f91>] chrdev_open+0x14a/0x180
03 00:31:34 [1905220.311618] [<801563b2>] __dentry_open+0xfd/0x1af
03 00:31:34 [1905220.317232] [<801564e5>] nameidata_to_filp+0x27/0x37
03 00:31:34 [1905220.323097] [<80156528>] do_filp_open+0x33/0x3b
03 00:31:34 [1905220.328535] [<80156575>] do_sys_open+0x45/0xc9
03 00:31:34 [1905220.333874] [<80156631>] sys_open+0x1c/0x1e
03 00:31:34 [1905220.338981] [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.344855] [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.349256]
03 00:31:34 [1905220.349258] -> #0 (tty_mutex){--..}:
03 00:31:34 [1905220.354717] [<80131b4b>] __lock_acquire+0x8aa/0xb96
03 00:31:34 [1905220.360505] [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.365956] [<80389e25>] __mutex_lock_slowpath+0xdb/0x271
03 00:31:34 [1905220.372258] [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.377517] [<80182ea9>] print_warning+0x92/0x137
03 00:31:34 [1905220.383139] [<80183ab8>] dquot_alloc_space+0x13b/0x15b
03 00:31:34 [1905220.389176] [<801920e1>] ext3_new_blocks+0x83/0x5ba
03 00:31:34 [1905220.394961] [<80195111>] ext3_get_blocks_handle+0x386/0x822
03 00:31:34 [1905220.401429] [<80195889>] ext3_get_block+0xac/0xc5
03 00:31:34 [1905220.407047] [<80174e54>] __block_prepare_write+0x151/0x3c6
03 00:31:34 [1905220.413455] [<801750ed>] block_prepare_write+0x24/0x32
03 00:31:34 [1905220.419506] [<80196a43>] ext3_prepare_write+0x98/0x153
03 00:31:34 [1905220.425553] [<8013cbbe>] generic_file_buffered_write+0x219/0x57c
03 00:31:34 [1905220.432465] [<8013d384>] __generic_file_aio_write_nolock+0x463/0x4b3
03 00:31:34 [1905220.439724] [<8013d42a>] generic_file_aio_write+0x56/0xb4
03 00:31:34 [1905220.446027] [<8019311b>] ext3_file_write+0x27/0x99
03 00:31:34 [1905220.451728] [<80157742>] do_sync_write+0xc4/0x101
03 00:31:34 [1905220.457338] [<80157f03>] vfs_write+0xaf/0x138
03 00:31:34 [1905220.462600] [<8015840f>] sys_write+0x3d/0x61
03 00:31:34 [1905220.467771] [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.473639] [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.478040]
03 00:31:34 [1905220.478043] other info that might help us debug this:
03 00:31:34 [1905220.478046]
03 00:31:34 [1905220.486588] 3 locks held by bacula-sd/32639:
03 00:31:34 [1905220.491025] #0: (&inode->i_mutex){--..}, at: [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.498668] #1: (&ei->truncate_mutex){--..}, at: [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.506659] #2: (&s->s_dquot.dqptr_sem){----}, at: [<801839bf>] dquot_alloc_space+0x42/0x15b
03 00:31:34 [1905220.515511]
03 00:31:34 [1905220.515513] stack backtrace:
03 00:31:34 [1905220.520230] [<80103694>] show_trace_log_lvl+0x1a/0x2f
03 00:31:34 [1905220.525555] [<8010421b>] show_trace+0x12/0x14
03 00:31:34 [1905220.530185] [<80104232>] dump_stack+0x15/0x17
03 00:31:34 [1905220.534817] [<801303c4>] print_circular_bug_tail+0x5e/0x67
03 00:31:34 [1905220.540580] [<80131b4b>] __lock_acquire+0x8aa/0xb96
03 00:31:34 [1905220.545731] [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.550536] [<80389e25>] __mutex_lock_slowpath+0xdb/0x271
03 00:31:34 [1905220.556203] [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.560838] [<80182ea9>] print_warning+0x92/0x137
03 00:31:34 [1905220.565819] [<80183ab8>] dquot_alloc_space+0x13b/0x15b
03 00:31:34 [1905220.571230] [<801920e1>] ext3_new_blocks+0x83/0x5ba
03 00:31:34 [1905220.576381] [<80195111>] ext3_get_blocks_handle+0x386/0x822
03 00:31:34 [1905220.582226] [<80195889>] ext3_get_block+0xac/0xc5
03 00:31:34 [1905220.587205] [<80174e54>] __block_prepare_write+0x151/0x3c6
03 00:31:34 [1905220.592963] [<801750ed>] block_prepare_write+0x24/0x32
03 00:31:34 [1905220.598372] [<80196a43>] ext3_prepare_write+0x98/0x153
03 00:31:34 [1905220.603785] [<8013cbbe>] generic_file_buffered_write+0x219/0x57c
03 00:31:34 [1905220.610063] [<8013d384>] __generic_file_aio_write_nolock+0x463/0x4b3
03 00:31:34 [1905220.616687] [<8013d42a>] generic_file_aio_write+0x56/0xb4
03 00:31:34 [1905220.622362] [<8019311b>] ext3_file_write+0x27/0x99
03 00:31:34 [1905220.627426] [<80157742>] do_sync_write+0xc4/0x101
03 00:31:34 [1905220.632401] [<80157f03>] vfs_write+0xaf/0x138
03 00:31:34 [1905220.637033] [<8015840f>] sys_write+0x3d/0x61
03 00:31:34 [1905220.641574] [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.646811] =======================
>> ---
>> fs/inode.c | 12 +++++++++---
>> include/linux/fs.h | 5 +++++
>> 2 files changed, 14 insertions(+), 3 deletions(-)
>>
>> Index: linux-2.6/fs/inode.c
>> ===================================================================
>> --- linux-2.6.orig/fs/inode.c
>> +++ linux-2.6/fs/inode.c
>> @@ -142,6 +142,15 @@ static struct inode *alloc_inode(struct
>> return NULL;
>> }
>>
>> + spin_lock_init(&inode->i_lock);
>> + lockdep_set_class(&inode->i_lock, &sb->s_type->i_lock_key);
>> +
>> + mutex_init(&inode->i_mutex);
>> + lockdep_set_class(&inode->i_mutex, &sb->s_type->i_mutex_key);
>> +
>> + init_rwsem(&inode->i_alloc_sem);
>> + lockdep_set_class(&inode->i_alloc_sem, &sb->s_type->i_alloc_sem_key);
>> +
>> mapping->a_ops = &empty_aops;
>> mapping->host = inode;
>> mapping->flags = 0;
>> @@ -190,8 +199,6 @@ void inode_init_once(struct inode *inode
>> INIT_HLIST_NODE(&inode->i_hash);
>> INIT_LIST_HEAD(&inode->i_dentry);
>> INIT_LIST_HEAD(&inode->i_devices);
>> - mutex_init(&inode->i_mutex);
>> - init_rwsem(&inode->i_alloc_sem);
>> INIT_RADIX_TREE(&inode->i_data.page_tree, GFP_ATOMIC);
>> rwlock_init(&inode->i_data.tree_lock);
>> spin_lock_init(&inode->i_data.i_mmap_lock);
>> @@ -199,7 +206,6 @@ void inode_init_once(struct inode *inode
>> spin_lock_init(&inode->i_data.private_lock);
>> INIT_RAW_PRIO_TREE_ROOT(&inode->i_data.i_mmap);
>> INIT_LIST_HEAD(&inode->i_data.i_mmap_nonlinear);
>> - spin_lock_init(&inode->i_lock);
>> i_size_ordered_init(inode);
>> #ifdef CONFIG_INOTIFY
>> INIT_LIST_HEAD(&inode->inotify_watches);
>> Index: linux-2.6/include/linux/fs.h
>> ===================================================================
>> --- linux-2.6.orig/include/linux/fs.h
>> +++ linux-2.6/include/linux/fs.h
>> @@ -1302,8 +1302,13 @@ struct file_system_type {
>> struct module *owner;
>> struct file_system_type * next;
>> struct list_head fs_supers;
>> +
>> struct lock_class_key s_lock_key;
>> struct lock_class_key s_umount_key;
>> +
>> + struct lock_class_key i_lock_key;
>> + struct lock_class_key i_mutex_key;
>> + struct lock_class_key i_alloc_sem_key;
>> };
>>
>> extern int get_sb_bdev(struct file_system_type *fs_type,
>>
--
Simon Arlott
next prev parent reply other threads:[~2008-01-04 7:41 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-08-24 21:00 [2.6.22] circular lock detected Folkert van Heusden
2007-08-26 23:03 ` Michal Piotrowski
2007-09-02 11:55 ` Andrew Morton
2007-09-02 17:11 ` Folkert van Heusden
2007-09-03 12:27 ` Jan Kara
2007-09-03 12:49 ` Andrew Morton
2007-09-03 14:01 ` Jan Kara
2007-09-25 8:02 ` Peter Zijlstra
2007-09-25 8:46 ` Jan Kara
2008-01-04 7:40 ` Simon Arlott [this message]
2008-01-04 9:21 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=477DE2F9.2050208@simon.arlott.org.uk \
--to=simon@fire.lp0.eu \
--cc=a.p.zijlstra@chello.nl \
--cc=akpm@linux-foundation.org \
--cc=folkert@vanheusden.com \
--cc=jack@suse.cz \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.