From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754360AbZCTWZj (ORCPT ); Fri, 20 Mar 2009 18:25:39 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752685AbZCTWZa (ORCPT ); Fri, 20 Mar 2009 18:25:30 -0400 Received: from thunk.org ([69.25.196.29]:59513 "EHLO thunker.thunk.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751060AbZCTWZ2 (ORCPT ); Fri, 20 Mar 2009 18:25:28 -0400 Date: Fri, 20 Mar 2009 18:25:22 -0400 From: Theodore Tso To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Greg Kroah-Hartman , Kay Sievers Subject: Re: Lockdep problem involving sysfs_mutex in ext4 in linux-next Message-ID: <20090320222522.GB4559@mit.edu> Mail-Followup-To: Theodore Tso , Peter Zijlstra , linux-kernel@vger.kernel.org, Greg Kroah-Hartman , Kay Sievers References: <1237071908.8939.788.camel@laptop> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1237071908.8939.788.camel@laptop> User-Agent: Mutt/1.5.18 (2008-05-17) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: tytso@mit.edu X-SA-Exim-Scanned: No (on thunker.thunk.org); SAEximRunCond expanded to false Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sun, Mar 15, 2009 at 12:05:08AM +0100, Peter Zijlstra wrote: > On Sat, 2009-03-14 at 10:02 -0400, Theodore Ts'o wrote: > > I'm occasionally seeing a circular locking dependency in linux-next when > > I unmount an ext4 filesystem, apparently involving sysfs_mutex in > > sysfs_addrm_start(), and I have no idea what's going on. Some help from > > kobject experts would be greatly appreciated. I assume I must be doing > > something wrong. I took the basic pattern from btrfs, but I suspect I > > may have broken some of the kobject lifetime rules when I adapted what I > > needed for ext4. > > > > It doesn't happen all the time, and it seems to be caused by how I'm > > releasing the ext4's sysfs kobject. In ext4_put_super(): > > > > kobject_put(&sbi->s_kobj); > > wait_for_completion(&sbi->s_kobj_unregister); > > Right that would require sysfs_mutex() while you're holding s_lock. Really? I can't see anywhere else in the kernel where code outside of fs/sysfs which grabs sysfs_mutex? > And here you snipped the interesting bit here it tells us how you > normally have the reverse lock order, namely: > > sysfs_mutex. > s_lock Ok, here's another such report: [42993.140759] the existing dependency chain (in reverse order) is: [42993.140762] [42993.140762] -> #4 (&type->s_lock_key#6){--..}: [42993.140769] [] __lock_acquire+0x9a6/0xb19 [42993.140776] [] lock_acquire+0x5b/0x81 [42993.140780] [] __mutex_lock_common+0xdc/0x33a [42993.140787] [] mutex_lock_nested+0x33/0x3b [42993.140791] [] lock_super+0x26/0x28 [42993.140795] [] ext4_orphan_add+0x1ab/0x1ca [42993.140801] [] ext4_setattr+0x19b/0x2de [42993.140805] [] notify_change+0x164/0x2af [42993.140816] [] do_truncate+0x6b/0x84 [42993.140822] [] may_open+0x196/0x19c [42993.140826] [] do_filp_open+0x341/0x680 [42993.140830] [] do_sys_open+0x47/0xbc [42993.140834] [] sys_open+0x23/0x2b [42993.140839] [] syscall_call+0x7/0xb [42993.140844] [] 0xffffffff [42993.140852] [42993.140853] -> #3 (jbd2_handle){--..}: [42993.140858] [] __lock_acquire+0x9a6/0xb19 [42993.140862] [] lock_acquire+0x5b/0x81 [42993.140866] [] jbd2_journal_start+0xea/0xf7 [42993.140871] [] ext4_journal_start_sb+0x49/0x69 [42993.140876] [] ext4_setattr+0x182/0x2de [42993.140880] [] notify_change+0x164/0x2af [42993.140884] [] do_truncate+0x6b/0x84 [42993.140888] [] may_open+0x196/0x19c [42993.140892] [] do_filp_open+0x341/0x680 [42993.140896] [] do_sys_open+0x47/0xbc [42993.140900] [] sys_open+0x23/0x2b [42993.140905] [] syscall_call+0x7/0xb [42993.140909] [] 0xffffffff [42993.140915] [42993.140915] -> #2 (&sb->s_type->i_alloc_sem_key#4){----}: [42993.140921] [] __lock_acquire+0x9a6/0xb19 [42993.140925] [] lock_acquire+0x5b/0x81 [42993.140929] [] down_read+0x37/0x74 [42993.140933] [] ext4_page_mkwrite+0x36/0x16a [42993.140937] [] do_wp_page+0x1ad/0x60a [42993.140942] [] handle_mm_fault+0x676/0x728 [42993.140946] [] do_page_fault+0x333/0x7ce [42993.140950] [] error_code+0x77/0x7c [42993.140954] [] 0xffffffff [42993.140958] [42993.140959] -> #1 (&mm->mmap_sem){----}: [42993.140964] [] __lock_acquire+0x9a6/0xb19 [42993.140968] [] lock_acquire+0x5b/0x81 [42993.140972] [] might_fault+0x65/0x85 [42993.140975] [] copy_to_user+0x31/0x100 [42993.140980] [] filldir64+0x9c/0xd2 [42993.140984] [] sysfs_readdir+0x11c/0x150 [42993.140988] [] vfs_readdir+0x6d/0x99 [42993.140992] [] sys_getdents64+0x68/0xa5 [42993.140996] [] syscall_call+0x7/0xb [42993.141000] [] 0xffffffff [42993.141009] [42993.141009] -> #0 (sysfs_mutex){--..}: [42993.141014] [] __lock_acquire+0x87b/0xb19 [42993.141018] [] lock_acquire+0x5b/0x81 [42993.141022] [] __mutex_lock_common+0xdc/0x33a [42993.141026] [] mutex_lock_nested+0x33/0x3b [42993.141030] [] sysfs_addrm_start+0x28/0x9a [42993.141034] [] sysfs_remove_dir+0x77/0xab [42993.141039] [] kobject_del+0xf/0x2c [42993.141042] [] kobject_release+0x134/0x1cb [42993.141046] [] kref_put+0x3c/0x4a [42993.141050] [] kobject_put+0x37/0x3c [42993.141054] [] ext4_put_super+0xab/0x215 [42993.141058] [] generic_shutdown_super+0x62/0xe3 [42993.141062] [] kill_block_super+0x22/0x36 [42993.141066] [] deactivate_super+0x5c/0x6f [42993.141070] [] mntput_no_expire+0xd4/0x106 [42993.141074] [] sys_umount+0x2a1/0x2c6 [42993.141078] [] sys_oldumount+0x12/0x14 [42993.141082] [] syscall_call+0x7/0xb [42993.141086] [] 0xffffffff [42993.141092] [42993.141093] other info that might help us debug this: [42993.141094] [42993.141097] 2 locks held by umount/30941: [42993.141099] #0: (&type->s_umount_key#30){----}, at: [] deactivate_super+0x57/0x6f [42993.141107] #1: (&type->s_lock_key#6){--..}, at: [] lock_super+0x26/0x28 [42993.141115] So if I'm reading this right, the problem is happening because someplace in the kernel, fs/sysfs code is grabbing mmap_sem while it is holding sysfs_mutex, right? But I can't see where that might be happening... - Ted