* XFS lock warning, 2.6.29-rc8
@ 2009-03-15 12:59 Denys Fedoryschenko
2009-03-15 22:19 ` Dave Chinner
0 siblings, 1 reply; 7+ messages in thread
From: Denys Fedoryschenko @ 2009-03-15 12:59 UTC (permalink / raw)
To: linux-xfs, linux-kernel
hi
Here is what i got suddently. But seems everything is working.
If any other information required, please let me know.
/dev/md2 on / type xfs (rw,noatime)
[39993.560200]
[39993.560202] =======================================================
[39993.560206] [ INFO: possible circular locking dependency detected ]
[39993.560208] 2.6.29-rc8-home1 #2
[39993.560210] -------------------------------------------------------
[39993.560212] kmail/5077 is trying to acquire lock:
[39993.560214] (&(&ip->i_iolock)->mr_lock){----}, at: [<ffffffff811426aa>]
xfs_ilock+0x27/0x79
[39993.560222]
[39993.560223] but task is already holding lock:
[39993.560225] (&mm->mmap_sem){----}, at: [<ffffffff81095090>]
sys_munmap+0x32/0x59
[39993.560230]
[39993.560231] which lock already depends on the new lock.
[39993.560232]
[39993.560233]
[39993.560234] the existing dependency chain (in reverse order) is:
[39993.560236]
[39993.560236] -> #1 (&mm->mmap_sem){----}:
[39993.560240] [<ffffffff8105b9fc>] __lock_acquire+0x12b0/0x161a
[39993.560244] [<ffffffff8105bdbb>] lock_acquire+0x55/0x71
[39993.560247] [<ffffffff813fc2a5>] down_read+0x34/0x41
[39993.560252] [<ffffffff813ffbfe>] do_page_fault+0x3fa/0x848
[39993.560256] [<ffffffff813fda2f>] page_fault+0x1f/0x30
[39993.560258] [<ffffffff8107e13d>] generic_file_aio_read+0x35f/0x5da
[39993.560268] [<ffffffff811650d8>] xfs_read+0x17d/0x1f5
[39993.560272] [<ffffffff81161646>] xfs_file_aio_read+0x5f/0x61
[39993.560275] [<ffffffff810a88ff>] do_sync_read+0xe7/0x12d
[39993.560284] [<ffffffff810a928c>] vfs_read+0xab/0x134
[39993.560287] [<ffffffff810a93d9>] sys_read+0x47/0x6f
[39993.560289] [<ffffffff8100c0db>] system_call_fastpath+0x16/0x1b
[39993.560293] [<ffffffffffffffff>] 0xffffffffffffffff
[39993.560302]
[39993.560302] -> #0 (&(&ip->i_iolock)->mr_lock){----}:
[39993.560306] [<ffffffff8105b6cb>] __lock_acquire+0xf7f/0x161a
[39993.560309] [<ffffffff8105bdbb>] lock_acquire+0x55/0x71
[39993.560312] [<ffffffff8104ebde>] down_write_nested+0x37/0x46
[39993.560316] [<ffffffff811426aa>] xfs_ilock+0x27/0x79
[39993.560318] [<ffffffff8115b40e>] xfs_inactive+0x14d/0x411
[39993.560322] [<ffffffff8116552f>] xfs_fs_clear_inode+0x83/0x85
[39993.560325] [<ffffffff810ba8f7>] clear_inode+0x8a/0xe3
[39993.560329] [<ffffffff810bb044>] generic_delete_inode+0xf8/0x173
[39993.560332] [<ffffffff810bb0d6>] generic_drop_inode+0x17/0x1d5
[39993.560335] [<ffffffff810ba1c7>] iput+0x61/0x65
[39993.560338] [<ffffffff810b7554>] dentry_iput+0xa9/0xc1
[39993.560341] [<ffffffff810b7651>] d_kill+0x40/0x60
[39993.560344] [<ffffffff810b942d>] dput+0x141/0x14e
[39993.560347] [<ffffffff810a9a85>] __fput+0x173/0x198
[39993.560350] [<ffffffff810a9ac2>] fput+0x18/0x1a
[39993.560353] [<ffffffff8109411e>] remove_vma+0x3e/0x63
[39993.560355] [<ffffffff8109503c>] do_munmap+0x2e9/0x30b
[39993.560358] [<ffffffff8109509e>] sys_munmap+0x40/0x59
[39993.560361] [<ffffffff8100c0db>] system_call_fastpath+0x16/0x1b
[39993.560363] [<ffffffffffffffff>] 0xffffffffffffffff
[39993.560366]
[39993.560367] other info that might help us debug this:
[39993.560368]
[39993.560370] 1 lock held by kmail/5077:
[39993.560371] #0: (&mm->mmap_sem){----}, at: [<ffffffff81095090>]
sys_munmap+0x32/0x59
[39993.560376]
[39993.560377] stack backtrace:
[39993.560379] Pid: 5077, comm: kmail Not tainted 2.6.29-rc8-home1 #2
[39993.560381] Call Trace:
[39993.560384] [<ffffffff8105a366>] print_circular_bug_tail+0xc5/0xd0
[39993.560388] [<ffffffff8105b6cb>] __lock_acquire+0xf7f/0x161a
[39993.560391] [<ffffffff8105bdbb>] lock_acquire+0x55/0x71
[39993.560393] [<ffffffff811426aa>] ? xfs_ilock+0x27/0x79
[39993.560396] [<ffffffff8104ebde>] down_write_nested+0x37/0x46
[39993.560399] [<ffffffff811426aa>] ? xfs_ilock+0x27/0x79
[39993.560401] [<ffffffff811426aa>] xfs_ilock+0x27/0x79
[39993.560404] [<ffffffff8115b40e>] xfs_inactive+0x14d/0x411
[39993.560407] [<ffffffff8116552f>] xfs_fs_clear_inode+0x83/0x85
[39993.560410] [<ffffffff810ba8f7>] clear_inode+0x8a/0xe3
[39993.560413] [<ffffffff810bb044>] generic_delete_inode+0xf8/0x173
[39993.560416] [<ffffffff810bb0d6>] generic_drop_inode+0x17/0x1d5
[39993.560419] [<ffffffff810ba1c7>] iput+0x61/0x65
[39993.560421] [<ffffffff810b7554>] dentry_iput+0xa9/0xc1
[39993.560424] [<ffffffff810b7651>] d_kill+0x40/0x60
[39993.560427] [<ffffffff810b942d>] dput+0x141/0x14e
[39993.560429] [<ffffffff810a9a85>] __fput+0x173/0x198
[39993.560432] [<ffffffff810a9ac2>] fput+0x18/0x1a
[39993.560434] [<ffffffff8109411e>] remove_vma+0x3e/0x63
[39993.560437] [<ffffffff8109503c>] do_munmap+0x2e9/0x30b
[39993.560440] [<ffffffff8109509e>] sys_munmap+0x40/0x59
[39993.560442] [<ffffffff8100c0db>] system_call_fastpath+0x16/0x1b
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: XFS lock warning, 2.6.29-rc8
2009-03-15 12:59 XFS lock warning, 2.6.29-rc8 Denys Fedoryschenko
@ 2009-03-15 22:19 ` Dave Chinner
0 siblings, 0 replies; 7+ messages in thread
From: Dave Chinner @ 2009-03-15 22:19 UTC (permalink / raw)
To: Denys Fedoryschenko; +Cc: linux-kernel, xfs
[ changed cc to xfs@oss.sgi.com. linux-xfs@vger.kernel.org does not
exist. ]
On Sun, Mar 15, 2009 at 02:59:01PM +0200, Denys Fedoryschenko wrote:
> hi
>
> Here is what i got suddently. But seems everything is working.
> If any other information required, please let me know.
Known issue.
> [39993.560234] the existing dependency chain (in reverse order) is:
> [39993.560236]
> [39993.560236] -> #1 (&mm->mmap_sem){----}:
> [39993.560240] [<ffffffff8105b9fc>] __lock_acquire+0x12b0/0x161a
> [39993.560244] [<ffffffff8105bdbb>] lock_acquire+0x55/0x71
> [39993.560247] [<ffffffff813fc2a5>] down_read+0x34/0x41
> [39993.560252] [<ffffffff813ffbfe>] do_page_fault+0x3fa/0x848
> [39993.560256] [<ffffffff813fda2f>] page_fault+0x1f/0x30
> [39993.560258] [<ffffffff8107e13d>] generic_file_aio_read+0x35f/0x5da
> [39993.560268] [<ffffffff811650d8>] xfs_read+0x17d/0x1f5
> [39993.560272] [<ffffffff81161646>] xfs_file_aio_read+0x5f/0x61
> [39993.560275] [<ffffffff810a88ff>] do_sync_read+0xe7/0x12d
> [39993.560284] [<ffffffff810a928c>] vfs_read+0xab/0x134
> [39993.560287] [<ffffffff810a93d9>] sys_read+0x47/0x6f
> [39993.560289] [<ffffffff8100c0db>] system_call_fastpath+0x16/0x1b
> [39993.560293] [<ffffffffffffffff>] 0xffffffffffffffff
> [39993.560302]
> [39993.560302] -> #0 (&(&ip->i_iolock)->mr_lock){----}:
> [39993.560306] [<ffffffff8105b6cb>] __lock_acquire+0xf7f/0x161a
> [39993.560309] [<ffffffff8105bdbb>] lock_acquire+0x55/0x71
> [39993.560312] [<ffffffff8104ebde>] down_write_nested+0x37/0x46
> [39993.560316] [<ffffffff811426aa>] xfs_ilock+0x27/0x79
> [39993.560318] [<ffffffff8115b40e>] xfs_inactive+0x14d/0x411
> [39993.560322] [<ffffffff8116552f>] xfs_fs_clear_inode+0x83/0x85
> [39993.560325] [<ffffffff810ba8f7>] clear_inode+0x8a/0xe3
> [39993.560329] [<ffffffff810bb044>] generic_delete_inode+0xf8/0x173
> [39993.560332] [<ffffffff810bb0d6>] generic_drop_inode+0x17/0x1d5
> [39993.560335] [<ffffffff810ba1c7>] iput+0x61/0x65
> [39993.560338] [<ffffffff810b7554>] dentry_iput+0xa9/0xc1
> [39993.560341] [<ffffffff810b7651>] d_kill+0x40/0x60
> [39993.560344] [<ffffffff810b942d>] dput+0x141/0x14e
> [39993.560347] [<ffffffff810a9a85>] __fput+0x173/0x198
> [39993.560350] [<ffffffff810a9ac2>] fput+0x18/0x1a
> [39993.560353] [<ffffffff8109411e>] remove_vma+0x3e/0x63
> [39993.560355] [<ffffffff8109503c>] do_munmap+0x2e9/0x30b
> [39993.560358] [<ffffffff8109509e>] sys_munmap+0x40/0x59
> [39993.560361] [<ffffffff8100c0db>] system_call_fastpath+0x16/0x1b
> [39993.560363] [<ffffffffffffffff>] 0xffffffffffffffff
This is a VM problem where it calls fput() with the mmap_sem() held
in remove_vma(). It makes the incorrect assumption that filesystems
will never use the same lock in the IO path and the inode release path.
This can deadlock if you are really unlucky.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: XFS lock warning, 2.6.29-rc8
@ 2009-03-15 22:19 ` Dave Chinner
0 siblings, 0 replies; 7+ messages in thread
From: Dave Chinner @ 2009-03-15 22:19 UTC (permalink / raw)
To: Denys Fedoryschenko; +Cc: xfs, linux-kernel
[ changed cc to xfs@oss.sgi.com. linux-xfs@vger.kernel.org does not
exist. ]
On Sun, Mar 15, 2009 at 02:59:01PM +0200, Denys Fedoryschenko wrote:
> hi
>
> Here is what i got suddently. But seems everything is working.
> If any other information required, please let me know.
Known issue.
> [39993.560234] the existing dependency chain (in reverse order) is:
> [39993.560236]
> [39993.560236] -> #1 (&mm->mmap_sem){----}:
> [39993.560240] [<ffffffff8105b9fc>] __lock_acquire+0x12b0/0x161a
> [39993.560244] [<ffffffff8105bdbb>] lock_acquire+0x55/0x71
> [39993.560247] [<ffffffff813fc2a5>] down_read+0x34/0x41
> [39993.560252] [<ffffffff813ffbfe>] do_page_fault+0x3fa/0x848
> [39993.560256] [<ffffffff813fda2f>] page_fault+0x1f/0x30
> [39993.560258] [<ffffffff8107e13d>] generic_file_aio_read+0x35f/0x5da
> [39993.560268] [<ffffffff811650d8>] xfs_read+0x17d/0x1f5
> [39993.560272] [<ffffffff81161646>] xfs_file_aio_read+0x5f/0x61
> [39993.560275] [<ffffffff810a88ff>] do_sync_read+0xe7/0x12d
> [39993.560284] [<ffffffff810a928c>] vfs_read+0xab/0x134
> [39993.560287] [<ffffffff810a93d9>] sys_read+0x47/0x6f
> [39993.560289] [<ffffffff8100c0db>] system_call_fastpath+0x16/0x1b
> [39993.560293] [<ffffffffffffffff>] 0xffffffffffffffff
> [39993.560302]
> [39993.560302] -> #0 (&(&ip->i_iolock)->mr_lock){----}:
> [39993.560306] [<ffffffff8105b6cb>] __lock_acquire+0xf7f/0x161a
> [39993.560309] [<ffffffff8105bdbb>] lock_acquire+0x55/0x71
> [39993.560312] [<ffffffff8104ebde>] down_write_nested+0x37/0x46
> [39993.560316] [<ffffffff811426aa>] xfs_ilock+0x27/0x79
> [39993.560318] [<ffffffff8115b40e>] xfs_inactive+0x14d/0x411
> [39993.560322] [<ffffffff8116552f>] xfs_fs_clear_inode+0x83/0x85
> [39993.560325] [<ffffffff810ba8f7>] clear_inode+0x8a/0xe3
> [39993.560329] [<ffffffff810bb044>] generic_delete_inode+0xf8/0x173
> [39993.560332] [<ffffffff810bb0d6>] generic_drop_inode+0x17/0x1d5
> [39993.560335] [<ffffffff810ba1c7>] iput+0x61/0x65
> [39993.560338] [<ffffffff810b7554>] dentry_iput+0xa9/0xc1
> [39993.560341] [<ffffffff810b7651>] d_kill+0x40/0x60
> [39993.560344] [<ffffffff810b942d>] dput+0x141/0x14e
> [39993.560347] [<ffffffff810a9a85>] __fput+0x173/0x198
> [39993.560350] [<ffffffff810a9ac2>] fput+0x18/0x1a
> [39993.560353] [<ffffffff8109411e>] remove_vma+0x3e/0x63
> [39993.560355] [<ffffffff8109503c>] do_munmap+0x2e9/0x30b
> [39993.560358] [<ffffffff8109509e>] sys_munmap+0x40/0x59
> [39993.560361] [<ffffffff8100c0db>] system_call_fastpath+0x16/0x1b
> [39993.560363] [<ffffffffffffffff>] 0xffffffffffffffff
This is a VM problem where it calls fput() with the mmap_sem() held
in remove_vma(). It makes the incorrect assumption that filesystems
will never use the same lock in the IO path and the inode release path.
This can deadlock if you are really unlucky.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 7+ messages in thread
* fput under mmap_sem
2009-03-15 22:19 ` Dave Chinner
@ 2009-03-18 7:13 ` Christoph Hellwig
-1 siblings, 0 replies; 7+ messages in thread
From: Christoph Hellwig @ 2009-03-18 7:13 UTC (permalink / raw)
To: xfs, linux-mm
On Mon, Mar 16, 2009 at 09:19:21AM +1100, Dave Chinner wrote:
> This is a VM problem where it calls fput() with the mmap_sem() held
> in remove_vma(). It makes the incorrect assumption that filesystems
> will never use the same lock in the IO path and the inode release path.
>
> This can deadlock if you are really unlucky.
I really wonder why other filesystems haven't hit this yet. Any chance
we can get the fput moved out of mmap_sem to get rid of this class of
problems?
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 7+ messages in thread
* fput under mmap_sem
@ 2009-03-18 7:13 ` Christoph Hellwig
0 siblings, 0 replies; 7+ messages in thread
From: Christoph Hellwig @ 2009-03-18 7:13 UTC (permalink / raw)
To: xfs, linux-mm
On Mon, Mar 16, 2009 at 09:19:21AM +1100, Dave Chinner wrote:
> This is a VM problem where it calls fput() with the mmap_sem() held
> in remove_vma(). It makes the incorrect assumption that filesystems
> will never use the same lock in the IO path and the inode release path.
>
> This can deadlock if you are really unlucky.
I really wonder why other filesystems haven't hit this yet. Any chance
we can get the fput moved out of mmap_sem to get rid of this class of
problems?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: fput under mmap_sem
2009-03-18 7:13 ` Christoph Hellwig
@ 2009-03-18 12:37 ` Nick Piggin
-1 siblings, 0 replies; 7+ messages in thread
From: Nick Piggin @ 2009-03-18 12:37 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: linux-mm, xfs
On Wednesday 18 March 2009 18:13:13 Christoph Hellwig wrote:
> On Mon, Mar 16, 2009 at 09:19:21AM +1100, Dave Chinner wrote:
> > This is a VM problem where it calls fput() with the mmap_sem() held
> > in remove_vma(). It makes the incorrect assumption that filesystems
> > will never use the same lock in the IO path and the inode release path.
> >
> > This can deadlock if you are really unlucky.
>
> I really wonder why other filesystems haven't hit this yet. Any chance
> we can get the fput moved out of mmap_sem to get rid of this class of
> problems?
Yes I don't think there is any reason against holding the file refcount
a little longer, but it can be pretty nasty to do in practice because
of deep call chains and sometimes multiple fputs within a given lock
section.
Possibly the easiest and quickest way will be to move aio's deferred
__fput into a usable API and try that. If there are any performance
problems, then we can try to move those fputs out from the mmap_sem
one at a time (eg. I don't expect fput for vma replacement/merging to
often cause the refcount to reach 0, and this is one of the harder
places; wheras fput for a simple munmap might be more often causing
refcount to reach 0 but it should be simpler to move out of mmap_sem
if it is a problem).
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: fput under mmap_sem
@ 2009-03-18 12:37 ` Nick Piggin
0 siblings, 0 replies; 7+ messages in thread
From: Nick Piggin @ 2009-03-18 12:37 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: xfs, linux-mm
On Wednesday 18 March 2009 18:13:13 Christoph Hellwig wrote:
> On Mon, Mar 16, 2009 at 09:19:21AM +1100, Dave Chinner wrote:
> > This is a VM problem where it calls fput() with the mmap_sem() held
> > in remove_vma(). It makes the incorrect assumption that filesystems
> > will never use the same lock in the IO path and the inode release path.
> >
> > This can deadlock if you are really unlucky.
>
> I really wonder why other filesystems haven't hit this yet. Any chance
> we can get the fput moved out of mmap_sem to get rid of this class of
> problems?
Yes I don't think there is any reason against holding the file refcount
a little longer, but it can be pretty nasty to do in practice because
of deep call chains and sometimes multiple fputs within a given lock
section.
Possibly the easiest and quickest way will be to move aio's deferred
__fput into a usable API and try that. If there are any performance
problems, then we can try to move those fputs out from the mmap_sem
one at a time (eg. I don't expect fput for vma replacement/merging to
often cause the refcount to reach 0, and this is one of the harder
places; wheras fput for a simple munmap might be more often causing
refcount to reach 0 but it should be simpler to move out of mmap_sem
if it is a problem).
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2009-03-18 12:38 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-03-15 12:59 XFS lock warning, 2.6.29-rc8 Denys Fedoryschenko
2009-03-15 22:19 ` Dave Chinner
2009-03-15 22:19 ` Dave Chinner
2009-03-18 7:13 ` fput under mmap_sem Christoph Hellwig
2009-03-18 7:13 ` Christoph Hellwig
2009-03-18 12:37 ` Nick Piggin
2009-03-18 12:37 ` Nick Piggin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.