From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2993178AbXDSIgi (ORCPT ); Thu, 19 Apr 2007 04:36:38 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1031221AbXDSIgh (ORCPT ); Thu, 19 Apr 2007 04:36:37 -0400 Received: from brick.kernel.dk ([80.160.20.94]:5686 "EHLO kernel.dk" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1031220AbXDSIgg (ORCPT ); Thu, 19 Apr 2007 04:36:36 -0400 Date: Thu, 19 Apr 2007 10:34:07 +0200 From: Jens Axboe To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-aio@kvack.org, reiserfs-dev@namesys.com, "Vladimir V. Saveliev" , linux-mm@kvack.org Subject: Re: dio_get_page() lockdep complaints Message-ID: <20070419083407.GD20928@kernel.dk> References: <20070419073828.GB20928@kernel.dk> <20070419010142.5b7b00cd.akpm@linux-foundation.org> <20070419080157.GC20928@kernel.dk> <20070419012540.bed394e2.akpm@linux-foundation.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070419012540.bed394e2.akpm@linux-foundation.org> Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Apr 19 2007, Andrew Morton wrote: > On Thu, 19 Apr 2007 10:01:57 +0200 Jens Axboe wrote: > > > On Thu, Apr 19 2007, Andrew Morton wrote: > > > On Thu, 19 Apr 2007 09:38:30 +0200 Jens Axboe wrote: > > > > > > > Hi, > > > > > > > > Doing some testing on CFQ, I ran into this 100% reproducible report: > > > > > > > > ======================================================= > > > > [ INFO: possible circular locking dependency detected ] > > > > 2.6.21-rc7 #5 > > > > ------------------------------------------------------- > > > > fio/9741 is trying to acquire lock: > > > > (&mm->mmap_sem){----}, at: [] dio_get_page+0x54/0x161 > > > > > > > > but task is already holding lock: > > > > (&inode->i_mutex){--..}, at: [] mutex_lock+0x1c/0x1f > > > > > > > > which lock already depends on the new lock. > > > > > > > > > > This is the correct ranking: i_mutex outside mmap_sem. > > > > > > > > > > > the existing dependency chain (in reverse order) is: > > > > > > > > -> #1 (&inode->i_mutex){--..}: > > > > [] __lock_acquire+0xdee/0xf9c > > > > [] lock_acquire+0x57/0x70 > > > > [] __mutex_lock_slowpath+0x73/0x297 > > > > [] mutex_lock+0x1c/0x1f > > > > [] reiserfs_file_release+0x54/0x447 > > > > [] __fput+0x53/0x101 > > > > [] fput+0x19/0x1c > > > > [] remove_vma+0x3b/0x4d > > > > [] do_munmap+0x17f/0x1cf > > > > [] sys_munmap+0x32/0x42 > > > > [] sysenter_past_esp+0x5d/0x99 > > > > [] 0xffffffff > > > > > > > > -> #0 (&mm->mmap_sem){----}: > > > > [] __lock_acquire+0xc4c/0xf9c > > > > [] lock_acquire+0x57/0x70 > > > > [] down_read+0x3a/0x4c > > > > [] dio_get_page+0x54/0x161 > > > > [] __blockdev_direct_IO+0x514/0xe2a > > > > [] ext3_direct_IO+0x98/0x1e5 > > > > [] generic_file_direct_IO+0x63/0x133 > > > > [] generic_file_aio_read+0x16b/0x222 > > > > [] aio_rw_vect_retry+0x5a/0x116 > > > > [] aio_run_iocb+0x69/0x129 > > > > [] io_submit_one+0x194/0x2eb > > > > [] sys_io_submit+0x92/0xe7 > > > > [] syscall_call+0x7/0xb > > > > [] 0xffffffff > > > > > > But here reiserfs is taking i_mutex in its file_operations.release(), > > > which can be called under mmap_sem. > > > > > > Vladimir's recent de14569f94513279e3d44d9571a421e9da1759ae. > > > "resierfs: avoid tail packing if an inode was ever mmapped" comes real > > > close to this code, but afaict it did not cause this bug. > > > > > > I can't think of anything which we've done in the 2.6.21 cycle which > > > would have caused this to start happening. Odd. > > > > The bug may be holder, let me know if you want me to check 2.6.20 or > > earlier. > > Would be great if you could test 2.6.20. I have a feeling that I missed > something, but what? We didn't change the refcounting of lifetime of > vma.vm_file... 2.6.20.7 tested, same lockdep triggers. Attached for reference. > > > > The test run was fio, the job file used is: > > > > > > > > # fio job file snip below > > > > [global] > > > > bs=4k > > > > buffered=0 > > > > ioengine=libaio > > > > iodepth=4 > > > > thread > > > > > > > > [readers] > > > > numjobs=8 > > > > size=128m > > > > rw=read > > > > # fio job file snip above > > > > > > > > Filesystem was ext3, default mkfs and mount options. Kernel was > > > > 2.6.21-rc7 as of this morning, with some CFQ patches applied. > > > > > > > > > > It's interesting that lockdep learned the (wrong) ranking from a reiserfs > > > operation then later detected it being violated by ext3. > > > > It's a scratch test box, which for some reason has reiserfs as the > > rootfs. So reiser gets to run first :-) > > direct-io reads against reiserfs also will take i_mutex outside mmap_sem. > As will pagefaults inside generic_file_write() (which is where this ranking > is primarily defined). > > So an all-reiserfs system should be getting the same reports. Obviously, > that isn't happening. > > It's a bit odd that reiserfs is playing with file contents within > file_operations.release(): there could be other files open against that > inode. One would expect this sort of thing to be happening in an > inode_operation. But it's been like that for a long time. > > Is it possible that fio was changed? That it was changed to close() the fd > before doing the munmapping whereas it used to hold the file open? It's been a while since I tested on this box, so I don't really recall. But fio does close() the fd before doing munmap(). This particular test case doesn't use mmap(), though. ======================================================= [ INFO: possible circular locking dependency detected ] 2.6.20.7 #1 ------------------------------------------------------- fio/6651 is trying to acquire lock: (&mm->mmap_sem){----}, at: [] dio_get_page+0x54/0x161 but task is already holding lock: (&inode->i_mutex){--..}, at: [] mutex_lock+0x1c/0x1f which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #1 (&inode->i_mutex){--..}: [] __lock_acquire+0xc86/0xd64 [] lock_acquire+0x57/0x70 [] __mutex_lock_slowpath+0x73/0x297 [] mutex_lock+0x1c/0x1f [] reiserfs_file_release+0x54/0x44b [] __fput+0x53/0x101 [] fput+0x19/0x1c [] remove_vma+0x37/0x49 [] do_munmap+0x17f/0x1d0 [] sys_munmap+0x32/0x42 [] sysenter_past_esp+0x5d/0x99 [] 0xffffffff -> #0 (&mm->mmap_sem){----}: [] __lock_acquire+0xb08/0xd64 [] lock_acquire+0x57/0x70 [] down_read+0x3a/0x4c [] dio_get_page+0x54/0x161 [] __blockdev_direct_IO+0x514/0xe2a [] ext3_direct_IO+0x98/0x1e5 [] generic_file_direct_IO+0x63/0x133 [] generic_file_aio_read+0x16b/0x222 [] aio_rw_vect_retry+0x5a/0x116 [] aio_run_iocb+0x69/0x129 [] io_submit_one+0x194/0x2ec [] sys_io_submit+0x92/0xe6 [] syscall_call+0x7/0xb [] 0xffffffff other info that might help us debug this: 1 lock held by fio/6651: #0: (&inode->i_mutex){--..}, at: [] mutex_lock+0x1c/0x1f stack backtrace: [] show_trace_log_lvl+0x1a/0x30 [] show_trace+0x12/0x14 [] dump_stack+0x16/0x18 [] print_circular_bug_tail+0x68/0x71 [] __lock_acquire+0xb08/0xd64 [] lock_acquire+0x57/0x70 [] down_read+0x3a/0x4c [] dio_get_page+0x54/0x161 [] __blockdev_direct_IO+0x514/0xe2a [] ext3_direct_IO+0x98/0x1e5 [] generic_file_direct_IO+0x63/0x133 [] generic_file_aio_read+0x16b/0x222 [] aio_rw_vect_retry+0x5a/0x116 [] aio_run_iocb+0x69/0x129 [] io_submit_one+0x194/0x2ec [] sys_io_submit+0x92/0xe6 [] syscall_call+0x7/0xb ======================= -- Jens Axboe