* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
@ 2009-05-05 23:16 ` bugzilla-daemon
2009-05-12 16:56 ` bugzilla-daemon
` (14 subsequent siblings)
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-05-05 23:16 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
Rafael J. Wysocki <rjw@sisk.pl> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |rjw@sisk.pl
Blocks| |12398
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
2009-05-05 23:16 ` [Bug 13232] " bugzilla-daemon
@ 2009-05-12 16:56 ` bugzilla-daemon
2009-05-13 13:48 ` Jan Kara
2009-05-13 13:48 ` bugzilla-daemon
` (13 subsequent siblings)
15 siblings, 1 reply; 25+ messages in thread
From: bugzilla-daemon @ 2009-05-12 16:56 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
--- Comment #2 from Jan Kara <jack@suse.cz> 2009-05-12 16:56:04 ---
Hi,
> (switched to email. Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> > SysRq : Show Blocked State
> > task PC stack pid father
> > kjournald D c01384df 0 2525 2
> > cfcb5f0c 00000082 de27d500 c01384df cfcb5ef4 c02cb5c0 00000001 de32ca00
> > de324814 dd037ebc de324814 de324934 dd037ebc cfcb5f5c cfcb5f90 c01bd4bb
> > 00000046 c0424110 de324a0c de324814 de324800 00000000 00000002 dd037e80
> > Call Trace:
> > [<c01384df>] ? trace_hardirqs_on+0xb/0xd
> > [<c01bd4bb>] journal_commit_transaction+0xea/0xeaf
> > [<c02c534a>] ? _spin_unlock_irqrestore+0x38/0x3f
> > [<c0138489>] ? trace_hardirqs_on_caller+0x145/0x190
> > [<c012b6ee>] ? autoremove_wake_function+0x0/0x38
> > [<c0122a25>] ? del_timer+0x50/0x59
> > [<c01c0c75>] kjournald+0xad/0x1bb
> > [<c012b6ee>] ? autoremove_wake_function+0x0/0x38
> > [<c01c0bc8>] ? kjournald+0x0/0x1bb
> > [<c012b442>] kthread+0x37/0x59
> > [<c012b40b>] ? kthread+0x0/0x59
> > [<c0103667>] kernel_thread_helper+0x7/0x10
>
> I assume this is
>
> while (commit_transaction->t_updates) {
> DEFINE_WAIT(wait);
>
> prepare_to_wait(&journal->j_wait_updates, &wait,
> TASK_UNINTERRUPTIBLE);
> if (commit_transaction->t_updates) {
> spin_unlock(&commit_transaction->t_handle_lock);
> spin_unlock(&journal->j_state_lock);
> schedule();
Yes.
> I'm wondering about
>
> : commit e219cca082f52e7dfea41f3be264b7b5eb204227
> : Author: Theodore Ts'o <tytso@mit.edu>
> : AuthorDate: Thu Nov 6 22:37:59 2008 -0500
> : Commit: Theodore Ts'o <tytso@mit.edu>
> : CommitDate: Thu Nov 6 22:37:59 2008 -0500
> :
> : jbd: don't give up looking for space so easily in __log_wait_for_space
>
> but that patch was present in 2.6.28.
Hmm, I don't see what made this deadlock happening - as far as I can
see it's there for quite some time. See below...
> > pickup D c01384df 0 2597 2594
> > cfaa9e5c 00000086 df9faa80 c01384df cfaa9e44 00000282 cfaa9e74 de32ca00
> > cfaa9e5c c012b8b7 00000002 de324800 0000014f cfaa9e74 cfaa9e94 c01c0539
> > 00000000 de3248c8 de324910 de324814 00000000 df9faa80 c012b6ee de3248e4
> > Call Trace:
> > [<c01384df>] ? trace_hardirqs_on+0xb/0xd
> > [<c012b8b7>] ? prepare_to_wait+0x42/0x4a
> > [<c01c0539>] log_wait_commit+0x90/0xf7
> > [<c012b6ee>] ? autoremove_wake_function+0x0/0x38
> > [<c01bba9d>] journal_stop+0x1c8/0x288
> > [<c01b4236>] __ext3_journal_stop+0x1c/0x38
> > [<c01aeb96>] ext3_delete_inode+0x90/0xc2
> > [<c01aeb06>] ? ext3_delete_inode+0x0/0xc2
> > [<c017ab82>] generic_delete_inode+0x72/0x100
> > [<c02c4ee1>] ? _spin_lock+0x3a/0x40
> > [<c017ad4c>] generic_drop_inode+0x13c/0x1da
> > [<c01d4068>] ? _atomic_dec_and_lock+0x10/0x38
> > [<c017a4e7>] iput+0x47/0x4e
> > [<c0173db0>] do_unlinkat+0xc1/0x137
> > [<c0102f87>] ? sysenter_exit+0xf/0x18
> > [<c0138489>] ? trace_hardirqs_on_caller+0x145/0x190
> > [<c0173e36>] sys_unlink+0x10/0x12
> > [<c0102f55>] sysenter_do_call+0x12/0x35
In generic_delete_inode() we mark inode as I_FREEING. Then
ext3_delete_inode() is called and because of O_SYNC it starts a
transaction commit and waits for it.
> > postdrop D c01384df 0 2664 2663
> > cfcbfd6c 00000086 dd13f700 c01384df cfcbfd54 c02cb5c0 00000001 deedc780
> > c03a1690 c1402348 c03a1690 cfcbfd7c c1402348 cfcbfd90 cfcbfd9c c017a55b
> > dd758c48 00000007 00000000 dd13f700 c012b726 c1402364 c1402364 00152b13
> > Call Trace:
> > [<c01384df>] ? trace_hardirqs_on+0xb/0xd
> > [<c017a55b>] __wait_on_freeing_inode+0x6d/0x88
> > [<c012b726>] ? wake_bit_function+0x0/0x47
> > [<c017a5b5>] find_inode_fast+0x3f/0x4a
> > [<c017ba05>] insert_inode_locked+0x50/0xeb
> > [<c01ab90b>] ext3_new_inode+0x738/0x88f
> > [<c01bc550>] ? journal_start+0xab/0x100
> > [<c01b259a>] ext3_create+0x59/0xbf
> > [<c01722c4>] vfs_create+0x75/0xb0
> > [<c02c4dda>] ? _spin_unlock+0x1d/0x20
> > [<c01b2541>] ? ext3_create+0x0/0xbf
> > [<c0174bc3>] do_filp_open+0x644/0x713
> > [<c02c4dda>] ? _spin_unlock+0x1d/0x20
> > [<c01692ce>] do_sys_open+0x45/0xce
> > [<c0102f87>] ? sysenter_exit+0xf/0x18
> > [<c0138489>] ? trace_hardirqs_on_caller+0x145/0x190
> > [<c01693a3>] sys_open+0x23/0x2b
> > [<c0102f55>] sysenter_do_call+0x12/0x35
Here, we have started a transaction in ext3_create() and then wait in
find_inode_fast() for I_FREEING to be cleared (obviously we have
reallocated the inode and squeezed the allocation before journal_stop()
from the delete was called).
Nasty deadlock and I don't see how to fix it now - have to go home for
today... Tomorrow I'll have a look what we can do about it.
Honza
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-12 16:56 ` bugzilla-daemon
@ 2009-05-13 13:48 ` Jan Kara
2009-05-13 16:07 ` Theodore Tso
2009-05-13 16:52 ` Al Viro
0 siblings, 2 replies; 25+ messages in thread
From: Jan Kara @ 2009-05-13 13:48 UTC (permalink / raw)
To: bugzilla-daemon; +Cc: linux-ext4, Al Viro
> http://bugzilla.kernel.org/show_bug.cgi?id=13232
>
> --- Comment #2 from Jan Kara <jack@suse.cz> 2009-05-12 16:56:04 ---
> Hi,
>
> > (switched to email. Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
>
> > > SysRq : Show Blocked State
> > > task PC stack pid father
> > > kjournald D c01384df 0 2525 2
> > > cfcb5f0c 00000082 de27d500 c01384df cfcb5ef4 c02cb5c0 00000001 de32ca00
> > > de324814 dd037ebc de324814 de324934 dd037ebc cfcb5f5c cfcb5f90 c01bd4bb
> > > 00000046 c0424110 de324a0c de324814 de324800 00000000 00000002 dd037e80
> > > Call Trace:
> > > [<c01384df>] ? trace_hardirqs_on+0xb/0xd
> > > [<c01bd4bb>] journal_commit_transaction+0xea/0xeaf
> > > [<c02c534a>] ? _spin_unlock_irqrestore+0x38/0x3f
> > > [<c0138489>] ? trace_hardirqs_on_caller+0x145/0x190
> > > [<c012b6ee>] ? autoremove_wake_function+0x0/0x38
> > > [<c0122a25>] ? del_timer+0x50/0x59
> > > [<c01c0c75>] kjournald+0xad/0x1bb
> > > [<c012b6ee>] ? autoremove_wake_function+0x0/0x38
> > > [<c01c0bc8>] ? kjournald+0x0/0x1bb
> > > [<c012b442>] kthread+0x37/0x59
> > > [<c012b40b>] ? kthread+0x0/0x59
> > > [<c0103667>] kernel_thread_helper+0x7/0x10
> >
> > I assume this is
> >
> > while (commit_transaction->t_updates) {
> > DEFINE_WAIT(wait);
> >
> > prepare_to_wait(&journal->j_wait_updates, &wait,
> > TASK_UNINTERRUPTIBLE);
> > if (commit_transaction->t_updates) {
> > spin_unlock(&commit_transaction->t_handle_lock);
> > spin_unlock(&journal->j_state_lock);
> > schedule();
> Yes.
>
> > I'm wondering about
> >
> > : commit e219cca082f52e7dfea41f3be264b7b5eb204227
> > : Author: Theodore Ts'o <tytso@mit.edu>
> > : AuthorDate: Thu Nov 6 22:37:59 2008 -0500
> > : Commit: Theodore Ts'o <tytso@mit.edu>
> > : CommitDate: Thu Nov 6 22:37:59 2008 -0500
> > :
> > : jbd: don't give up looking for space so easily in __log_wait_for_space
> >
> > but that patch was present in 2.6.28.
> Hmm, I don't see what made this deadlock happening - as far as I can
> see it's there for quite some time. See below...
>
> > > pickup D c01384df 0 2597 2594
> > > cfaa9e5c 00000086 df9faa80 c01384df cfaa9e44 00000282 cfaa9e74 de32ca00
> > > cfaa9e5c c012b8b7 00000002 de324800 0000014f cfaa9e74 cfaa9e94 c01c0539
> > > 00000000 de3248c8 de324910 de324814 00000000 df9faa80 c012b6ee de3248e4
> > > Call Trace:
> > > [<c01384df>] ? trace_hardirqs_on+0xb/0xd
> > > [<c012b8b7>] ? prepare_to_wait+0x42/0x4a
> > > [<c01c0539>] log_wait_commit+0x90/0xf7
> > > [<c012b6ee>] ? autoremove_wake_function+0x0/0x38
> > > [<c01bba9d>] journal_stop+0x1c8/0x288
> > > [<c01b4236>] __ext3_journal_stop+0x1c/0x38
> > > [<c01aeb96>] ext3_delete_inode+0x90/0xc2
> > > [<c01aeb06>] ? ext3_delete_inode+0x0/0xc2
> > > [<c017ab82>] generic_delete_inode+0x72/0x100
> > > [<c02c4ee1>] ? _spin_lock+0x3a/0x40
> > > [<c017ad4c>] generic_drop_inode+0x13c/0x1da
> > > [<c01d4068>] ? _atomic_dec_and_lock+0x10/0x38
> > > [<c017a4e7>] iput+0x47/0x4e
> > > [<c0173db0>] do_unlinkat+0xc1/0x137
> > > [<c0102f87>] ? sysenter_exit+0xf/0x18
> > > [<c0138489>] ? trace_hardirqs_on_caller+0x145/0x190
> > > [<c0173e36>] sys_unlink+0x10/0x12
> > > [<c0102f55>] sysenter_do_call+0x12/0x35
> In generic_delete_inode() we mark inode as I_FREEING. Then
> ext3_delete_inode() is called and because of O_SYNC it starts a
> transaction commit and waits for it.
>
> > > postdrop D c01384df 0 2664 2663
> > > cfcbfd6c 00000086 dd13f700 c01384df cfcbfd54 c02cb5c0 00000001 deedc780
> > > c03a1690 c1402348 c03a1690 cfcbfd7c c1402348 cfcbfd90 cfcbfd9c c017a55b
> > > dd758c48 00000007 00000000 dd13f700 c012b726 c1402364 c1402364 00152b13
> > > Call Trace:
> > > [<c01384df>] ? trace_hardirqs_on+0xb/0xd
> > > [<c017a55b>] __wait_on_freeing_inode+0x6d/0x88
> > > [<c012b726>] ? wake_bit_function+0x0/0x47
> > > [<c017a5b5>] find_inode_fast+0x3f/0x4a
> > > [<c017ba05>] insert_inode_locked+0x50/0xeb
> > > [<c01ab90b>] ext3_new_inode+0x738/0x88f
> > > [<c01bc550>] ? journal_start+0xab/0x100
> > > [<c01b259a>] ext3_create+0x59/0xbf
> > > [<c01722c4>] vfs_create+0x75/0xb0
> > > [<c02c4dda>] ? _spin_unlock+0x1d/0x20
> > > [<c01b2541>] ? ext3_create+0x0/0xbf
> > > [<c0174bc3>] do_filp_open+0x644/0x713
> > > [<c02c4dda>] ? _spin_unlock+0x1d/0x20
> > > [<c01692ce>] do_sys_open+0x45/0xce
> > > [<c0102f87>] ? sysenter_exit+0xf/0x18
> > > [<c0138489>] ? trace_hardirqs_on_caller+0x145/0x190
> > > [<c01693a3>] sys_open+0x23/0x2b
> > > [<c0102f55>] sysenter_do_call+0x12/0x35
> Here, we have started a transaction in ext3_create() and then wait in
> find_inode_fast() for I_FREEING to be cleared (obviously we have
> reallocated the inode and squeezed the allocation before journal_stop()
> from the delete was called).
> Nasty deadlock and I don't see how to fix it now - have to go home for
> today... Tomorrow I'll have a look what we can do about it.
OK, the deadlock has been introduced by ext3 variant of
261bca86ed4f7f391d1938167624e78da61dcc6b (adding Al to CC). The deadlock
is really tough to avoid - we have to first allocate inode on disk so
that we know the inode number. For this we need transaction open but we
cannot afford waiting for old inode with same INO to be freed when we have
transaction open because of the above deadlock. So we'd have to wait for
inode release only after everything is done and we closed the transaction. But
that would mean reordering a lot of code in ext3/namei.c so that all the
dcache handling is done after all the IO is done.
Hmm, maybe we could change the delete side of the deadlock but that's
going to be tricky as well :(.
Al, any idea if we could somehow get away without waiting on
I_FREEING?
Honza
--
Jan Kara <jack@suse.cz>
SuSE CR Labs
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-13 13:48 ` Jan Kara
@ 2009-05-13 16:07 ` Theodore Tso
2009-05-18 12:45 ` Jan Kara
2009-05-13 16:52 ` Al Viro
1 sibling, 1 reply; 25+ messages in thread
From: Theodore Tso @ 2009-05-13 16:07 UTC (permalink / raw)
To: Jan Kara; +Cc: bugzilla-daemon, linux-ext4, Al Viro
On Wed, May 13, 2009 at 03:48:02PM +0200, Jan Kara wrote:
> OK, the deadlock has been introduced by ext3 variant of
> 261bca86ed4f7f391d1938167624e78da61dcc6b (adding Al to CC).
What do you mean by this?
I'm puzzled why we haven't hit this before. This looks like
long-standing issue; what unmasked it now?
> The deadlock
> is really tough to avoid - we have to first allocate inode on disk so
> that we know the inode number.
Well, the simple thing to do is to have a way of quickly determining
that a particular inode number is in the I_FREEING state, and simply
try to avoid using that inode number. If there are no inodes
available, it can simply close the handle (since nothing else has
changed at that point), wait for the current transaction to close, and
then try again. That should fix the problem, I think.
- Ted
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-13 16:07 ` Theodore Tso
@ 2009-05-18 12:45 ` Jan Kara
0 siblings, 0 replies; 25+ messages in thread
From: Jan Kara @ 2009-05-18 12:45 UTC (permalink / raw)
To: Theodore Tso; +Cc: bugzilla-daemon, linux-ext4, Al Viro
On Wed 13-05-09 12:07:24, Theodore Tso wrote:
> On Wed, May 13, 2009 at 03:48:02PM +0200, Jan Kara wrote:
> > OK, the deadlock has been introduced by ext3 variant of
> > 261bca86ed4f7f391d1938167624e78da61dcc6b (adding Al to CC).
>
> What do you mean by this?
>
> I'm puzzled why we haven't hit this before. This looks like
> long-standing issue; what unmasked it now?
Unless you mount the fs with 'sync' option, hitting this is much harder
(the window is quite small in nosync case). I think that is the main reason
why we didn't see this earlier.
> > The deadlock
> > is really tough to avoid - we have to first allocate inode on disk so
> > that we know the inode number.
>
> Well, the simple thing to do is to have a way of quickly determining
> that a particular inode number is in the I_FREEING state, and simply
> try to avoid using that inode number. If there are no inodes
> available, it can simply close the handle (since nothing else has
> changed at that point), wait for the current transaction to close, and
> then try again. That should fix the problem, I think.
Yes, we could work-around it like that but other filesystems might need
similar things and generally it would be nicer if we could avoid using this
vfs-internal information in the filesystems. Al seems to have found some
other solution without changing filesystems so that would be easier for us...
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-13 13:48 ` Jan Kara
2009-05-13 16:07 ` Theodore Tso
@ 2009-05-13 16:52 ` Al Viro
2009-05-13 18:13 ` Al Viro
2009-05-18 12:53 ` Jan Kara
1 sibling, 2 replies; 25+ messages in thread
From: Al Viro @ 2009-05-13 16:52 UTC (permalink / raw)
To: Jan Kara; +Cc: bugzilla-daemon, linux-ext4
On Wed, May 13, 2009 at 03:48:02PM +0200, Jan Kara wrote:
> > Here, we have started a transaction in ext3_create() and then wait in
> > find_inode_fast() for I_FREEING to be cleared (obviously we have
> > reallocated the inode and squeezed the allocation before journal_stop()
> > from the delete was called).
> > Nasty deadlock and I don't see how to fix it now - have to go home for
> > today... Tomorrow I'll have a look what we can do about it.
> OK, the deadlock has been introduced by ext3 variant of
> 261bca86ed4f7f391d1938167624e78da61dcc6b (adding Al to CC). The deadlock
> is really tough to avoid - we have to first allocate inode on disk so
> that we know the inode number. For this we need transaction open but we
> cannot afford waiting for old inode with same INO to be freed when we have
> transaction open because of the above deadlock. So we'd have to wait for
> inode release only after everything is done and we closed the transaction. But
> that would mean reordering a lot of code in ext3/namei.c so that all the
> dcache handling is done after all the IO is done.
> Hmm, maybe we could change the delete side of the deadlock but that's
> going to be tricky as well :(.
> Al, any idea if we could somehow get away without waiting on
> I_FREEING?
At which point do we actually run into deadlock on delete side? We could,
in principle, skip everything like that in insert_inode_locked(), but
I would rather avoid the "two inodes in icache at the same time, with the
same inumber" situations completely. We might get away with that, since
everything else *will* wait, so we can afford a bunch of inodes past the
point in foo_delete_inode() that has cleared it in bitmap + new locked
one, but if it's at all possible to avoid, I'd rather avoid it.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-13 16:52 ` Al Viro
@ 2009-05-13 18:13 ` Al Viro
2009-05-18 13:15 ` Theodore Tso
2009-05-18 14:10 ` Jan Kara
2009-05-18 12:53 ` Jan Kara
1 sibling, 2 replies; 25+ messages in thread
From: Al Viro @ 2009-05-13 18:13 UTC (permalink / raw)
To: Jan Kara; +Cc: bugzilla-daemon, linux-ext4, linux-fsdevel
On Wed, May 13, 2009 at 05:52:54PM +0100, Al Viro wrote:
> On Wed, May 13, 2009 at 03:48:02PM +0200, Jan Kara wrote:
> > > Here, we have started a transaction in ext3_create() and then wait in
> > > find_inode_fast() for I_FREEING to be cleared (obviously we have
> > > reallocated the inode and squeezed the allocation before journal_stop()
> > > from the delete was called).
> > > Nasty deadlock and I don't see how to fix it now - have to go home for
> > > today... Tomorrow I'll have a look what we can do about it.
> > OK, the deadlock has been introduced by ext3 variant of
> > 261bca86ed4f7f391d1938167624e78da61dcc6b (adding Al to CC). The deadlock
> > is really tough to avoid - we have to first allocate inode on disk so
> > that we know the inode number. For this we need transaction open but we
> > cannot afford waiting for old inode with same INO to be freed when we have
> > transaction open because of the above deadlock. So we'd have to wait for
> > inode release only after everything is done and we closed the transaction. But
> > that would mean reordering a lot of code in ext3/namei.c so that all the
> > dcache handling is done after all the IO is done.
> > Hmm, maybe we could change the delete side of the deadlock but that's
> > going to be tricky as well :(.
> > Al, any idea if we could somehow get away without waiting on
> > I_FREEING?
>
> At which point do we actually run into deadlock on delete side? We could,
> in principle, skip everything like that in insert_inode_locked(), but
> I would rather avoid the "two inodes in icache at the same time, with the
> same inumber" situations completely. We might get away with that, since
> everything else *will* wait, so we can afford a bunch of inodes past the
> point in foo_delete_inode() that has cleared it in bitmap + new locked
> one, but if it's at all possible to avoid, I'd rather avoid it.
OK, that's probably the easiest way to do that, as much as I don't like it...
Since iget() et.al. will not accept I_FREEING (will wait to go away
and restart), and since we'd better have serialization between new/free
on fs data structures anyway, we can afford simply skipping I_FREEING
et.al. in insert_inode_locked().
We do that from new_inode, so it won't race with free_inode in any interesting
ways and it won't race with iget (of any origin; nfsd or in case of fs
corruption a lookup) since both still will wait for I_LOCK.
Tentative patch follow; folks, I would very much like review on that one,
since I'm far too low on caffeine and the area is nasty.
diff --git a/fs/inode.c b/fs/inode.c
index 9d26490..4406952 100644
--- a/fs/inode.c
+++ b/fs/inode.c
@@ -1053,13 +1053,22 @@ int insert_inode_locked(struct inode *inode)
struct super_block *sb = inode->i_sb;
ino_t ino = inode->i_ino;
struct hlist_head *head = inode_hashtable + hash(sb, ino);
- struct inode *old;
inode->i_state |= I_LOCK|I_NEW;
while (1) {
+ struct hlist_node *node;
+ struct inode *old = NULL;
spin_lock(&inode_lock);
- old = find_inode_fast(sb, head, ino);
- if (likely(!old)) {
+ hlist_for_each_entry(old, node, head, i_hash) {
+ if (old->i_ino != ino)
+ continue;
+ if (old->i_sb != sb)
+ continue;
+ if (old->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE))
+ continue;
+ break;
+ }
+ if (likely(!node)) {
hlist_add_head(&inode->i_hash, head);
spin_unlock(&inode_lock);
return 0;
@@ -1081,14 +1090,24 @@ int insert_inode_locked4(struct inode *inode, unsigned long hashval,
{
struct super_block *sb = inode->i_sb;
struct hlist_head *head = inode_hashtable + hash(sb, hashval);
- struct inode *old;
inode->i_state |= I_LOCK|I_NEW;
while (1) {
+ struct hlist_node *node;
+ struct inode *old = NULL;
+
spin_lock(&inode_lock);
- old = find_inode(sb, head, test, data);
- if (likely(!old)) {
+ hlist_for_each_entry(old, node, head, i_hash) {
+ if (old->i_sb != sb)
+ continue;
+ if (!test(old, data))
+ continue;
+ if (old->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE))
+ continue;
+ break;
+ }
+ if (likely(!node)) {
hlist_add_head(&inode->i_hash, head);
spin_unlock(&inode_lock);
return 0;
^ permalink raw reply related [flat|nested] 25+ messages in thread* Re: [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-13 18:13 ` Al Viro
@ 2009-05-18 13:15 ` Theodore Tso
2009-05-18 14:10 ` Jan Kara
1 sibling, 0 replies; 25+ messages in thread
From: Theodore Tso @ 2009-05-18 13:15 UTC (permalink / raw)
To: Al Viro; +Cc: Jan Kara, bugzilla-daemon, linux-ext4, linux-fsdevel
On Wed, May 13, 2009 at 07:13:40PM +0100, Al Viro wrote:
>
> OK, that's probably the easiest way to do that, as much as I don't like it...
> Since iget() et.al. will not accept I_FREEING (will wait to go away
> and restart), and since we'd better have serialization between new/free
> on fs data structures anyway, we can afford simply skipping I_FREEING
> et.al. in insert_inode_locked().
>
> We do that from new_inode, so it won't race with free_inode in any interesting
> ways and it won't race with iget (of any origin; nfsd or in case of fs
> corruption a lookup) since both still will wait for I_LOCK.
>
> Tentative patch follow; folks, I would very much like review on that one,
> since I'm far too low on caffeine and the area is nasty.
Sorry for not having time to review this until now. This looks good
to me.
Reviewed-by: "Theodore Ts'o" <tytso@mit.edu>
So Bug #13232 is currently marked as a 2.6.28 regression; do we feel
confident enough to push this to Linus for 2.6.30?
- Ted
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-13 18:13 ` Al Viro
2009-05-18 13:15 ` Theodore Tso
@ 2009-05-18 14:10 ` Jan Kara
1 sibling, 0 replies; 25+ messages in thread
From: Jan Kara @ 2009-05-18 14:10 UTC (permalink / raw)
To: Al Viro; +Cc: bugzilla-daemon, linux-ext4, linux-fsdevel
On Wed 13-05-09 19:13:40, Al Viro wrote:
> On Wed, May 13, 2009 at 05:52:54PM +0100, Al Viro wrote:
> > On Wed, May 13, 2009 at 03:48:02PM +0200, Jan Kara wrote:
> > > > Here, we have started a transaction in ext3_create() and then wait in
> > > > find_inode_fast() for I_FREEING to be cleared (obviously we have
> > > > reallocated the inode and squeezed the allocation before journal_stop()
> > > > from the delete was called).
> > > > Nasty deadlock and I don't see how to fix it now - have to go home for
> > > > today... Tomorrow I'll have a look what we can do about it.
> > > OK, the deadlock has been introduced by ext3 variant of
> > > 261bca86ed4f7f391d1938167624e78da61dcc6b (adding Al to CC). The deadlock
> > > is really tough to avoid - we have to first allocate inode on disk so
> > > that we know the inode number. For this we need transaction open but we
> > > cannot afford waiting for old inode with same INO to be freed when we have
> > > transaction open because of the above deadlock. So we'd have to wait for
> > > inode release only after everything is done and we closed the transaction. But
> > > that would mean reordering a lot of code in ext3/namei.c so that all the
> > > dcache handling is done after all the IO is done.
> > > Hmm, maybe we could change the delete side of the deadlock but that's
> > > going to be tricky as well :(.
> > > Al, any idea if we could somehow get away without waiting on
> > > I_FREEING?
> >
> > At which point do we actually run into deadlock on delete side? We could,
> > in principle, skip everything like that in insert_inode_locked(), but
> > I would rather avoid the "two inodes in icache at the same time, with the
> > same inumber" situations completely. We might get away with that, since
> > everything else *will* wait, so we can afford a bunch of inodes past the
> > point in foo_delete_inode() that has cleared it in bitmap + new locked
> > one, but if it's at all possible to avoid, I'd rather avoid it.
>
> OK, that's probably the easiest way to do that, as much as I don't like it...
> Since iget() et.al. will not accept I_FREEING (will wait to go away
> and restart), and since we'd better have serialization between new/free
> on fs data structures anyway, we can afford simply skipping I_FREEING
> et.al. in insert_inode_locked().
>
> We do that from new_inode, so it won't race with free_inode in any interesting
> ways and it won't race with iget (of any origin; nfsd or in case of fs
> corruption a lookup) since both still will wait for I_LOCK.
>
> Tentative patch follow; folks, I would very much like review on that one,
> since I'm far too low on caffeine and the area is nasty.
The patch looks fine. Everyone else will either get new inode and wait
for I_LOCK or get old inode and wait for I_FREEING so everything should be
fine... You can add.
Acked-by: Jan Kara <jack@suse.cz>
Honza
>
> diff --git a/fs/inode.c b/fs/inode.c
> index 9d26490..4406952 100644
> --- a/fs/inode.c
> +++ b/fs/inode.c
> @@ -1053,13 +1053,22 @@ int insert_inode_locked(struct inode *inode)
> struct super_block *sb = inode->i_sb;
> ino_t ino = inode->i_ino;
> struct hlist_head *head = inode_hashtable + hash(sb, ino);
> - struct inode *old;
>
> inode->i_state |= I_LOCK|I_NEW;
> while (1) {
> + struct hlist_node *node;
> + struct inode *old = NULL;
> spin_lock(&inode_lock);
> - old = find_inode_fast(sb, head, ino);
> - if (likely(!old)) {
> + hlist_for_each_entry(old, node, head, i_hash) {
> + if (old->i_ino != ino)
> + continue;
> + if (old->i_sb != sb)
> + continue;
> + if (old->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE))
> + continue;
> + break;
> + }
> + if (likely(!node)) {
> hlist_add_head(&inode->i_hash, head);
> spin_unlock(&inode_lock);
> return 0;
> @@ -1081,14 +1090,24 @@ int insert_inode_locked4(struct inode *inode, unsigned long hashval,
> {
> struct super_block *sb = inode->i_sb;
> struct hlist_head *head = inode_hashtable + hash(sb, hashval);
> - struct inode *old;
>
> inode->i_state |= I_LOCK|I_NEW;
>
> while (1) {
> + struct hlist_node *node;
> + struct inode *old = NULL;
> +
> spin_lock(&inode_lock);
> - old = find_inode(sb, head, test, data);
> - if (likely(!old)) {
> + hlist_for_each_entry(old, node, head, i_hash) {
> + if (old->i_sb != sb)
> + continue;
> + if (!test(old, data))
> + continue;
> + if (old->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE))
> + continue;
> + break;
> + }
> + if (likely(!node)) {
> hlist_add_head(&inode->i_hash, head);
> spin_unlock(&inode_lock);
> return 0;
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-13 16:52 ` Al Viro
2009-05-13 18:13 ` Al Viro
@ 2009-05-18 12:53 ` Jan Kara
1 sibling, 0 replies; 25+ messages in thread
From: Jan Kara @ 2009-05-18 12:53 UTC (permalink / raw)
To: Al Viro; +Cc: bugzilla-daemon, linux-ext4
On Wed 13-05-09 17:52:54, Al Viro wrote:
> On Wed, May 13, 2009 at 03:48:02PM +0200, Jan Kara wrote:
> > > Here, we have started a transaction in ext3_create() and then wait in
> > > find_inode_fast() for I_FREEING to be cleared (obviously we have
> > > reallocated the inode and squeezed the allocation before journal_stop()
> > > from the delete was called).
> > > Nasty deadlock and I don't see how to fix it now - have to go home for
> > > today... Tomorrow I'll have a look what we can do about it.
> > OK, the deadlock has been introduced by ext3 variant of
> > 261bca86ed4f7f391d1938167624e78da61dcc6b (adding Al to CC). The deadlock
> > is really tough to avoid - we have to first allocate inode on disk so
> > that we know the inode number. For this we need transaction open but we
> > cannot afford waiting for old inode with same INO to be freed when we have
> > transaction open because of the above deadlock. So we'd have to wait for
> > inode release only after everything is done and we closed the transaction. But
> > that would mean reordering a lot of code in ext3/namei.c so that all the
> > dcache handling is done after all the IO is done.
> > Hmm, maybe we could change the delete side of the deadlock but that's
> > going to be tricky as well :(.
> > Al, any idea if we could somehow get away without waiting on
> > I_FREEING?
>
> At which point do we actually run into deadlock on delete side? We could,
> in principle, skip everything like that in insert_inode_locked(), but
> I would rather avoid the "two inodes in icache at the same time, with the
> same inumber" situations completely. We might get away with that, since
> everything else *will* wait, so we can afford a bunch of inodes past the
> point in foo_delete_inode() that has cleared it in bitmap + new locked
> one, but if it's at all possible to avoid, I'd rather avoid it.
The ordering we see on delete when the filesystem is mounted with 'sync'
option is:
DELETE CREATE
generic_delete_inode()
set I_FREEING
ext3_delete_inode
get transaction handle
do work
get transaction handle
ext3_new_inode()
reallocate inode
insert_inode_locked()
stop transaction, wait for it to commit
(waiting for CREATE process to drop its
transaction reference)
Now similar race can happen even without 'sync' mount option but it's
much harder to hit:
DELETE CREATE
generic_delete_inode()
set I_FREEING
ext3_delete_inode
get transaction handle
ext3_new_inode()
reallocate inode
insert_inode_locked()
try to get transaction handle -
- transaction is too big so we send
current transaction to commit which
again waits for CREATE to drop its
reference.
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 25+ messages in thread
* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
2009-05-05 23:16 ` [Bug 13232] " bugzilla-daemon
2009-05-12 16:56 ` bugzilla-daemon
@ 2009-05-13 13:48 ` bugzilla-daemon
2009-05-13 16:07 ` bugzilla-daemon
` (12 subsequent siblings)
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-05-13 13:48 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
--- Comment #3 from Jan Kara <jack@suse.cz> 2009-05-13 13:48:04 ---
> http://bugzilla.kernel.org/show_bug.cgi?id=13232
>
> --- Comment #2 from Jan Kara <jack@suse.cz> 2009-05-12 16:56:04 ---
> Hi,
>
> > (switched to email. Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
>
> > > SysRq : Show Blocked State
> > > task PC stack pid father
> > > kjournald D c01384df 0 2525 2
> > > cfcb5f0c 00000082 de27d500 c01384df cfcb5ef4 c02cb5c0 00000001 de32ca00
> > > de324814 dd037ebc de324814 de324934 dd037ebc cfcb5f5c cfcb5f90 c01bd4bb
> > > 00000046 c0424110 de324a0c de324814 de324800 00000000 00000002 dd037e80
> > > Call Trace:
> > > [<c01384df>] ? trace_hardirqs_on+0xb/0xd
> > > [<c01bd4bb>] journal_commit_transaction+0xea/0xeaf
> > > [<c02c534a>] ? _spin_unlock_irqrestore+0x38/0x3f
> > > [<c0138489>] ? trace_hardirqs_on_caller+0x145/0x190
> > > [<c012b6ee>] ? autoremove_wake_function+0x0/0x38
> > > [<c0122a25>] ? del_timer+0x50/0x59
> > > [<c01c0c75>] kjournald+0xad/0x1bb
> > > [<c012b6ee>] ? autoremove_wake_function+0x0/0x38
> > > [<c01c0bc8>] ? kjournald+0x0/0x1bb
> > > [<c012b442>] kthread+0x37/0x59
> > > [<c012b40b>] ? kthread+0x0/0x59
> > > [<c0103667>] kernel_thread_helper+0x7/0x10
> >
> > I assume this is
> >
> > while (commit_transaction->t_updates) {
> > DEFINE_WAIT(wait);
> >
> > prepare_to_wait(&journal->j_wait_updates, &wait,
> > TASK_UNINTERRUPTIBLE);
> > if (commit_transaction->t_updates) {
> > spin_unlock(&commit_transaction->t_handle_lock);
> > spin_unlock(&journal->j_state_lock);
> > schedule();
> Yes.
>
> > I'm wondering about
> >
> > : commit e219cca082f52e7dfea41f3be264b7b5eb204227
> > : Author: Theodore Ts'o <tytso@mit.edu>
> > : AuthorDate: Thu Nov 6 22:37:59 2008 -0500
> > : Commit: Theodore Ts'o <tytso@mit.edu>
> > : CommitDate: Thu Nov 6 22:37:59 2008 -0500
> > :
> > : jbd: don't give up looking for space so easily in __log_wait_for_space
> >
> > but that patch was present in 2.6.28.
> Hmm, I don't see what made this deadlock happening - as far as I can
> see it's there for quite some time. See below...
>
> > > pickup D c01384df 0 2597 2594
> > > cfaa9e5c 00000086 df9faa80 c01384df cfaa9e44 00000282 cfaa9e74 de32ca00
> > > cfaa9e5c c012b8b7 00000002 de324800 0000014f cfaa9e74 cfaa9e94 c01c0539
> > > 00000000 de3248c8 de324910 de324814 00000000 df9faa80 c012b6ee de3248e4
> > > Call Trace:
> > > [<c01384df>] ? trace_hardirqs_on+0xb/0xd
> > > [<c012b8b7>] ? prepare_to_wait+0x42/0x4a
> > > [<c01c0539>] log_wait_commit+0x90/0xf7
> > > [<c012b6ee>] ? autoremove_wake_function+0x0/0x38
> > > [<c01bba9d>] journal_stop+0x1c8/0x288
> > > [<c01b4236>] __ext3_journal_stop+0x1c/0x38
> > > [<c01aeb96>] ext3_delete_inode+0x90/0xc2
> > > [<c01aeb06>] ? ext3_delete_inode+0x0/0xc2
> > > [<c017ab82>] generic_delete_inode+0x72/0x100
> > > [<c02c4ee1>] ? _spin_lock+0x3a/0x40
> > > [<c017ad4c>] generic_drop_inode+0x13c/0x1da
> > > [<c01d4068>] ? _atomic_dec_and_lock+0x10/0x38
> > > [<c017a4e7>] iput+0x47/0x4e
> > > [<c0173db0>] do_unlinkat+0xc1/0x137
> > > [<c0102f87>] ? sysenter_exit+0xf/0x18
> > > [<c0138489>] ? trace_hardirqs_on_caller+0x145/0x190
> > > [<c0173e36>] sys_unlink+0x10/0x12
> > > [<c0102f55>] sysenter_do_call+0x12/0x35
> In generic_delete_inode() we mark inode as I_FREEING. Then
> ext3_delete_inode() is called and because of O_SYNC it starts a
> transaction commit and waits for it.
>
> > > postdrop D c01384df 0 2664 2663
> > > cfcbfd6c 00000086 dd13f700 c01384df cfcbfd54 c02cb5c0 00000001 deedc780
> > > c03a1690 c1402348 c03a1690 cfcbfd7c c1402348 cfcbfd90 cfcbfd9c c017a55b
> > > dd758c48 00000007 00000000 dd13f700 c012b726 c1402364 c1402364 00152b13
> > > Call Trace:
> > > [<c01384df>] ? trace_hardirqs_on+0xb/0xd
> > > [<c017a55b>] __wait_on_freeing_inode+0x6d/0x88
> > > [<c012b726>] ? wake_bit_function+0x0/0x47
> > > [<c017a5b5>] find_inode_fast+0x3f/0x4a
> > > [<c017ba05>] insert_inode_locked+0x50/0xeb
> > > [<c01ab90b>] ext3_new_inode+0x738/0x88f
> > > [<c01bc550>] ? journal_start+0xab/0x100
> > > [<c01b259a>] ext3_create+0x59/0xbf
> > > [<c01722c4>] vfs_create+0x75/0xb0
> > > [<c02c4dda>] ? _spin_unlock+0x1d/0x20
> > > [<c01b2541>] ? ext3_create+0x0/0xbf
> > > [<c0174bc3>] do_filp_open+0x644/0x713
> > > [<c02c4dda>] ? _spin_unlock+0x1d/0x20
> > > [<c01692ce>] do_sys_open+0x45/0xce
> > > [<c0102f87>] ? sysenter_exit+0xf/0x18
> > > [<c0138489>] ? trace_hardirqs_on_caller+0x145/0x190
> > > [<c01693a3>] sys_open+0x23/0x2b
> > > [<c0102f55>] sysenter_do_call+0x12/0x35
> Here, we have started a transaction in ext3_create() and then wait in
> find_inode_fast() for I_FREEING to be cleared (obviously we have
> reallocated the inode and squeezed the allocation before journal_stop()
> from the delete was called).
> Nasty deadlock and I don't see how to fix it now - have to go home for
> today... Tomorrow I'll have a look what we can do about it.
OK, the deadlock has been introduced by ext3 variant of
261bca86ed4f7f391d1938167624e78da61dcc6b (adding Al to CC). The deadlock
is really tough to avoid - we have to first allocate inode on disk so
that we know the inode number. For this we need transaction open but we
cannot afford waiting for old inode with same INO to be freed when we have
transaction open because of the above deadlock. So we'd have to wait for
inode release only after everything is done and we closed the transaction. But
that would mean reordering a lot of code in ext3/namei.c so that all the
dcache handling is done after all the IO is done.
Hmm, maybe we could change the delete side of the deadlock but that's
going to be tricky as well :(.
Al, any idea if we could somehow get away without waiting on
I_FREEING?
Honza
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
` (2 preceding siblings ...)
2009-05-13 13:48 ` bugzilla-daemon
@ 2009-05-13 16:07 ` bugzilla-daemon
2009-05-13 16:18 ` bugzilla-daemon
` (11 subsequent siblings)
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-05-13 16:07 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
--- Comment #4 from Theodore Tso <tytso@mit.edu> 2009-05-13 16:07:33 ---
On Wed, May 13, 2009 at 03:48:02PM +0200, Jan Kara wrote:
> OK, the deadlock has been introduced by ext3 variant of
> 261bca86ed4f7f391d1938167624e78da61dcc6b (adding Al to CC).
What do you mean by this?
I'm puzzled why we haven't hit this before. This looks like
long-standing issue; what unmasked it now?
> The deadlock
> is really tough to avoid - we have to first allocate inode on disk so
> that we know the inode number.
Well, the simple thing to do is to have a way of quickly determining
that a particular inode number is in the I_FREEING state, and simply
try to avoid using that inode number. If there are no inodes
available, it can simply close the handle (since nothing else has
changed at that point), wait for the current transaction to close, and
then try again. That should fix the problem, I think.
- Ted
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
` (3 preceding siblings ...)
2009-05-13 16:07 ` bugzilla-daemon
@ 2009-05-13 16:18 ` bugzilla-daemon
2009-05-13 16:52 ` bugzilla-daemon
` (10 subsequent siblings)
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-05-13 16:18 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
Vincent Li <mchun.li@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |mchun.li@gmail.com
--- Comment #5 from Vincent Li <mchun.li@gmail.com> 2009-05-13 16:18:57 ---
I tried to run Postfix on ubuntu 9.04 server version and followed the reporduce
command, had the same problem.
kernel version: 2.6.30-rc4
Distribution: Ubuntu 9.04
Software Environment: Postfix 2.5.1-2ubuntu1.2
also have following message poping up on control terminal:
[844,700070] INFO: task kjournald: 628 blocked for more than 120 seconds,
[844,700148] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
meassage.
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
` (4 preceding siblings ...)
2009-05-13 16:18 ` bugzilla-daemon
@ 2009-05-13 16:52 ` bugzilla-daemon
2009-05-13 18:13 ` bugzilla-daemon
` (9 subsequent siblings)
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-05-13 16:52 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
--- Comment #6 from Anonymous Emailer <anonymous@kernel-bugs.osdl.org> 2009-05-13 16:52:56 ---
Reply-To: viro@ZenIV.linux.org.uk
On Wed, May 13, 2009 at 03:48:02PM +0200, Jan Kara wrote:
> > Here, we have started a transaction in ext3_create() and then wait in
> > find_inode_fast() for I_FREEING to be cleared (obviously we have
> > reallocated the inode and squeezed the allocation before journal_stop()
> > from the delete was called).
> > Nasty deadlock and I don't see how to fix it now - have to go home for
> > today... Tomorrow I'll have a look what we can do about it.
> OK, the deadlock has been introduced by ext3 variant of
> 261bca86ed4f7f391d1938167624e78da61dcc6b (adding Al to CC). The deadlock
> is really tough to avoid - we have to first allocate inode on disk so
> that we know the inode number. For this we need transaction open but we
> cannot afford waiting for old inode with same INO to be freed when we have
> transaction open because of the above deadlock. So we'd have to wait for
> inode release only after everything is done and we closed the transaction. But
> that would mean reordering a lot of code in ext3/namei.c so that all the
> dcache handling is done after all the IO is done.
> Hmm, maybe we could change the delete side of the deadlock but that's
> going to be tricky as well :(.
> Al, any idea if we could somehow get away without waiting on
> I_FREEING?
At which point do we actually run into deadlock on delete side? We could,
in principle, skip everything like that in insert_inode_locked(), but
I would rather avoid the "two inodes in icache at the same time, with the
same inumber" situations completely. We might get away with that, since
everything else *will* wait, so we can afford a bunch of inodes past the
point in foo_delete_inode() that has cleared it in bitmap + new locked
one, but if it's at all possible to avoid, I'd rather avoid it.
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
` (5 preceding siblings ...)
2009-05-13 16:52 ` bugzilla-daemon
@ 2009-05-13 18:13 ` bugzilla-daemon
2009-05-18 12:45 ` bugzilla-daemon
` (8 subsequent siblings)
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-05-13 18:13 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
--- Comment #7 from Anonymous Emailer <anonymous@kernel-bugs.osdl.org> 2009-05-13 18:13:42 ---
Reply-To: viro@ZenIV.linux.org.uk
On Wed, May 13, 2009 at 05:52:54PM +0100, Al Viro wrote:
> On Wed, May 13, 2009 at 03:48:02PM +0200, Jan Kara wrote:
> > > Here, we have started a transaction in ext3_create() and then wait in
> > > find_inode_fast() for I_FREEING to be cleared (obviously we have
> > > reallocated the inode and squeezed the allocation before journal_stop()
> > > from the delete was called).
> > > Nasty deadlock and I don't see how to fix it now - have to go home for
> > > today... Tomorrow I'll have a look what we can do about it.
> > OK, the deadlock has been introduced by ext3 variant of
> > 261bca86ed4f7f391d1938167624e78da61dcc6b (adding Al to CC). The deadlock
> > is really tough to avoid - we have to first allocate inode on disk so
> > that we know the inode number. For this we need transaction open but we
> > cannot afford waiting for old inode with same INO to be freed when we have
> > transaction open because of the above deadlock. So we'd have to wait for
> > inode release only after everything is done and we closed the transaction. But
> > that would mean reordering a lot of code in ext3/namei.c so that all the
> > dcache handling is done after all the IO is done.
> > Hmm, maybe we could change the delete side of the deadlock but that's
> > going to be tricky as well :(.
> > Al, any idea if we could somehow get away without waiting on
> > I_FREEING?
>
> At which point do we actually run into deadlock on delete side? We could,
> in principle, skip everything like that in insert_inode_locked(), but
> I would rather avoid the "two inodes in icache at the same time, with the
> same inumber" situations completely. We might get away with that, since
> everything else *will* wait, so we can afford a bunch of inodes past the
> point in foo_delete_inode() that has cleared it in bitmap + new locked
> one, but if it's at all possible to avoid, I'd rather avoid it.
OK, that's probably the easiest way to do that, as much as I don't like it...
Since iget() et.al. will not accept I_FREEING (will wait to go away
and restart), and since we'd better have serialization between new/free
on fs data structures anyway, we can afford simply skipping I_FREEING
et.al. in insert_inode_locked().
We do that from new_inode, so it won't race with free_inode in any interesting
ways and it won't race with iget (of any origin; nfsd or in case of fs
corruption a lookup) since both still will wait for I_LOCK.
Tentative patch follow; folks, I would very much like review on that one,
since I'm far too low on caffeine and the area is nasty.
diff --git a/fs/inode.c b/fs/inode.c
index 9d26490..4406952 100644
--- a/fs/inode.c
+++ b/fs/inode.c
@@ -1053,13 +1053,22 @@ int insert_inode_locked(struct inode *inode)
struct super_block *sb = inode->i_sb;
ino_t ino = inode->i_ino;
struct hlist_head *head = inode_hashtable + hash(sb, ino);
- struct inode *old;
inode->i_state |= I_LOCK|I_NEW;
while (1) {
+ struct hlist_node *node;
+ struct inode *old = NULL;
spin_lock(&inode_lock);
- old = find_inode_fast(sb, head, ino);
- if (likely(!old)) {
+ hlist_for_each_entry(old, node, head, i_hash) {
+ if (old->i_ino != ino)
+ continue;
+ if (old->i_sb != sb)
+ continue;
+ if (old->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE))
+ continue;
+ break;
+ }
+ if (likely(!node)) {
hlist_add_head(&inode->i_hash, head);
spin_unlock(&inode_lock);
return 0;
@@ -1081,14 +1090,24 @@ int insert_inode_locked4(struct inode *inode, unsigned
long hashval,
{
struct super_block *sb = inode->i_sb;
struct hlist_head *head = inode_hashtable + hash(sb, hashval);
- struct inode *old;
inode->i_state |= I_LOCK|I_NEW;
while (1) {
+ struct hlist_node *node;
+ struct inode *old = NULL;
+
spin_lock(&inode_lock);
- old = find_inode(sb, head, test, data);
- if (likely(!old)) {
+ hlist_for_each_entry(old, node, head, i_hash) {
+ if (old->i_sb != sb)
+ continue;
+ if (!test(old, data))
+ continue;
+ if (old->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE))
+ continue;
+ break;
+ }
+ if (likely(!node)) {
hlist_add_head(&inode->i_hash, head);
spin_unlock(&inode_lock);
return 0;
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply related [flat|nested] 25+ messages in thread* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
` (6 preceding siblings ...)
2009-05-13 18:13 ` bugzilla-daemon
@ 2009-05-18 12:45 ` bugzilla-daemon
2009-05-18 12:54 ` bugzilla-daemon
` (7 subsequent siblings)
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-05-18 12:45 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
--- Comment #8 from Jan Kara <jack@suse.cz> 2009-05-18 12:45:23 ---
On Wed 13-05-09 12:07:24, Theodore Tso wrote:
> On Wed, May 13, 2009 at 03:48:02PM +0200, Jan Kara wrote:
> > OK, the deadlock has been introduced by ext3 variant of
> > 261bca86ed4f7f391d1938167624e78da61dcc6b (adding Al to CC).
>
> What do you mean by this?
>
> I'm puzzled why we haven't hit this before. This looks like
> long-standing issue; what unmasked it now?
Unless you mount the fs with 'sync' option, hitting this is much harder
(the window is quite small in nosync case). I think that is the main reason
why we didn't see this earlier.
> > The deadlock
> > is really tough to avoid - we have to first allocate inode on disk so
> > that we know the inode number.
>
> Well, the simple thing to do is to have a way of quickly determining
> that a particular inode number is in the I_FREEING state, and simply
> try to avoid using that inode number. If there are no inodes
> available, it can simply close the handle (since nothing else has
> changed at that point), wait for the current transaction to close, and
> then try again. That should fix the problem, I think.
Yes, we could work-around it like that but other filesystems might need
similar things and generally it would be nicer if we could avoid using this
vfs-internal information in the filesystems. Al seems to have found some
other solution without changing filesystems so that would be easier for us...
Honza
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
` (7 preceding siblings ...)
2009-05-18 12:45 ` bugzilla-daemon
@ 2009-05-18 12:54 ` bugzilla-daemon
2009-05-18 13:16 ` bugzilla-daemon
` (6 subsequent siblings)
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-05-18 12:54 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
--- Comment #9 from Jan Kara <jack@suse.cz> 2009-05-18 12:54:00 ---
On Wed 13-05-09 17:52:54, Al Viro wrote:
> On Wed, May 13, 2009 at 03:48:02PM +0200, Jan Kara wrote:
> > > Here, we have started a transaction in ext3_create() and then wait in
> > > find_inode_fast() for I_FREEING to be cleared (obviously we have
> > > reallocated the inode and squeezed the allocation before journal_stop()
> > > from the delete was called).
> > > Nasty deadlock and I don't see how to fix it now - have to go home for
> > > today... Tomorrow I'll have a look what we can do about it.
> > OK, the deadlock has been introduced by ext3 variant of
> > 261bca86ed4f7f391d1938167624e78da61dcc6b (adding Al to CC). The deadlock
> > is really tough to avoid - we have to first allocate inode on disk so
> > that we know the inode number. For this we need transaction open but we
> > cannot afford waiting for old inode with same INO to be freed when we have
> > transaction open because of the above deadlock. So we'd have to wait for
> > inode release only after everything is done and we closed the transaction. But
> > that would mean reordering a lot of code in ext3/namei.c so that all the
> > dcache handling is done after all the IO is done.
> > Hmm, maybe we could change the delete side of the deadlock but that's
> > going to be tricky as well :(.
> > Al, any idea if we could somehow get away without waiting on
> > I_FREEING?
>
> At which point do we actually run into deadlock on delete side? We could,
> in principle, skip everything like that in insert_inode_locked(), but
> I would rather avoid the "two inodes in icache at the same time, with the
> same inumber" situations completely. We might get away with that, since
> everything else *will* wait, so we can afford a bunch of inodes past the
> point in foo_delete_inode() that has cleared it in bitmap + new locked
> one, but if it's at all possible to avoid, I'd rather avoid it.
The ordering we see on delete when the filesystem is mounted with 'sync'
option is:
DELETE CREATE
generic_delete_inode()
set I_FREEING
ext3_delete_inode
get transaction handle
do work
get transaction handle
ext3_new_inode()
reallocate inode
insert_inode_locked()
stop transaction, wait for it to commit
(waiting for CREATE process to drop its
transaction reference)
Now similar race can happen even without 'sync' mount option but it's
much harder to hit:
DELETE CREATE
generic_delete_inode()
set I_FREEING
ext3_delete_inode
get transaction handle
ext3_new_inode()
reallocate inode
insert_inode_locked()
try to get transaction handle -
- transaction is too big so we send
current transaction to commit which
again waits for CREATE to drop its
reference.
Honza
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
` (8 preceding siblings ...)
2009-05-18 12:54 ` bugzilla-daemon
@ 2009-05-18 13:16 ` bugzilla-daemon
2009-05-18 14:10 ` bugzilla-daemon
` (5 subsequent siblings)
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-05-18 13:16 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
--- Comment #10 from Theodore Tso <tytso@mit.edu> 2009-05-18 13:16:04 ---
On Wed, May 13, 2009 at 07:13:40PM +0100, Al Viro wrote:
>
> OK, that's probably the easiest way to do that, as much as I don't like it...
> Since iget() et.al. will not accept I_FREEING (will wait to go away
> and restart), and since we'd better have serialization between new/free
> on fs data structures anyway, we can afford simply skipping I_FREEING
> et.al. in insert_inode_locked().
>
> We do that from new_inode, so it won't race with free_inode in any interesting
> ways and it won't race with iget (of any origin; nfsd or in case of fs
> corruption a lookup) since both still will wait for I_LOCK.
>
> Tentative patch follow; folks, I would very much like review on that one,
> since I'm far too low on caffeine and the area is nasty.
Sorry for not having time to review this until now. This looks good
to me.
Reviewed-by: "Theodore Ts'o" <tytso@mit.edu>
So Bug #13232 is currently marked as a 2.6.28 regression; do we feel
confident enough to push this to Linus for 2.6.30?
- Ted
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
` (9 preceding siblings ...)
2009-05-18 13:16 ` bugzilla-daemon
@ 2009-05-18 14:10 ` bugzilla-daemon
2009-05-19 20:38 ` bugzilla-daemon
` (4 subsequent siblings)
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-05-18 14:10 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
--- Comment #11 from Jan Kara <jack@suse.cz> 2009-05-18 14:10:15 ---
On Wed 13-05-09 19:13:40, Al Viro wrote:
> On Wed, May 13, 2009 at 05:52:54PM +0100, Al Viro wrote:
> > On Wed, May 13, 2009 at 03:48:02PM +0200, Jan Kara wrote:
> > > > Here, we have started a transaction in ext3_create() and then wait in
> > > > find_inode_fast() for I_FREEING to be cleared (obviously we have
> > > > reallocated the inode and squeezed the allocation before journal_stop()
> > > > from the delete was called).
> > > > Nasty deadlock and I don't see how to fix it now - have to go home for
> > > > today... Tomorrow I'll have a look what we can do about it.
> > > OK, the deadlock has been introduced by ext3 variant of
> > > 261bca86ed4f7f391d1938167624e78da61dcc6b (adding Al to CC). The deadlock
> > > is really tough to avoid - we have to first allocate inode on disk so
> > > that we know the inode number. For this we need transaction open but we
> > > cannot afford waiting for old inode with same INO to be freed when we have
> > > transaction open because of the above deadlock. So we'd have to wait for
> > > inode release only after everything is done and we closed the transaction. But
> > > that would mean reordering a lot of code in ext3/namei.c so that all the
> > > dcache handling is done after all the IO is done.
> > > Hmm, maybe we could change the delete side of the deadlock but that's
> > > going to be tricky as well :(.
> > > Al, any idea if we could somehow get away without waiting on
> > > I_FREEING?
> >
> > At which point do we actually run into deadlock on delete side? We could,
> > in principle, skip everything like that in insert_inode_locked(), but
> > I would rather avoid the "two inodes in icache at the same time, with the
> > same inumber" situations completely. We might get away with that, since
> > everything else *will* wait, so we can afford a bunch of inodes past the
> > point in foo_delete_inode() that has cleared it in bitmap + new locked
> > one, but if it's at all possible to avoid, I'd rather avoid it.
>
> OK, that's probably the easiest way to do that, as much as I don't like it...
> Since iget() et.al. will not accept I_FREEING (will wait to go away
> and restart), and since we'd better have serialization between new/free
> on fs data structures anyway, we can afford simply skipping I_FREEING
> et.al. in insert_inode_locked().
>
> We do that from new_inode, so it won't race with free_inode in any interesting
> ways and it won't race with iget (of any origin; nfsd or in case of fs
> corruption a lookup) since both still will wait for I_LOCK.
>
> Tentative patch follow; folks, I would very much like review on that one,
> since I'm far too low on caffeine and the area is nasty.
The patch looks fine. Everyone else will either get new inode and wait
for I_LOCK or get old inode and wait for I_FREEING so everything should be
fine... You can add.
Acked-by: Jan Kara <jack@suse.cz>
Honza
>
> diff --git a/fs/inode.c b/fs/inode.c
> index 9d26490..4406952 100644
> --- a/fs/inode.c
> +++ b/fs/inode.c
> @@ -1053,13 +1053,22 @@ int insert_inode_locked(struct inode *inode)
> struct super_block *sb = inode->i_sb;
> ino_t ino = inode->i_ino;
> struct hlist_head *head = inode_hashtable + hash(sb, ino);
> - struct inode *old;
>
> inode->i_state |= I_LOCK|I_NEW;
> while (1) {
> + struct hlist_node *node;
> + struct inode *old = NULL;
> spin_lock(&inode_lock);
> - old = find_inode_fast(sb, head, ino);
> - if (likely(!old)) {
> + hlist_for_each_entry(old, node, head, i_hash) {
> + if (old->i_ino != ino)
> + continue;
> + if (old->i_sb != sb)
> + continue;
> + if (old->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE))
> + continue;
> + break;
> + }
> + if (likely(!node)) {
> hlist_add_head(&inode->i_hash, head);
> spin_unlock(&inode_lock);
> return 0;
> @@ -1081,14 +1090,24 @@ int insert_inode_locked4(struct inode *inode, unsigned long hashval,
> {
> struct super_block *sb = inode->i_sb;
> struct hlist_head *head = inode_hashtable + hash(sb, hashval);
> - struct inode *old;
>
> inode->i_state |= I_LOCK|I_NEW;
>
> while (1) {
> + struct hlist_node *node;
> + struct inode *old = NULL;
> +
> spin_lock(&inode_lock);
> - old = find_inode(sb, head, test, data);
> - if (likely(!old)) {
> + hlist_for_each_entry(old, node, head, i_hash) {
> + if (old->i_sb != sb)
> + continue;
> + if (!test(old, data))
> + continue;
> + if (old->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE))
> + continue;
> + break;
> + }
> + if (likely(!node)) {
> hlist_add_head(&inode->i_hash, head);
> spin_unlock(&inode_lock);
> return 0;
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
` (10 preceding siblings ...)
2009-05-18 14:10 ` bugzilla-daemon
@ 2009-05-19 20:38 ` bugzilla-daemon
2009-05-19 20:39 ` bugzilla-daemon
` (3 subsequent siblings)
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-05-19 20:38 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
--- Comment #12 from Theodore Tso <tytso@mit.edu> 2009-05-19 20:38:32 ---
Created an attachment (id=21436)
--> (http://bugzilla.kernel.org/attachment.cgi?id=21436)
Proposed patch from Al Viro
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
` (11 preceding siblings ...)
2009-05-19 20:38 ` bugzilla-daemon
@ 2009-05-19 20:39 ` bugzilla-daemon
2009-06-07 19:56 ` bugzilla-daemon
` (2 subsequent siblings)
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-05-19 20:39 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
Theodore Tso <tytso@mit.edu> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #21436|application/octet-stream |text/plain
mime type| |
Attachment #21436|0 |1
is patch| |
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
` (12 preceding siblings ...)
2009-05-19 20:39 ` bugzilla-daemon
@ 2009-06-07 19:56 ` bugzilla-daemon
2009-06-07 19:56 ` bugzilla-daemon
2009-06-07 20:44 ` bugzilla-daemon
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-06-07 19:56 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
Theodore Tso <tytso@mit.edu> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |tytso@mit.edu
--- Comment #13 from Theodore Tso <tytso@mit.edu> 2009-06-07 19:56:04 ---
This patch is now in the upstream kernel, so it will be in 2.6.30.
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
` (13 preceding siblings ...)
2009-06-07 19:56 ` bugzilla-daemon
@ 2009-06-07 19:56 ` bugzilla-daemon
2009-06-07 20:44 ` bugzilla-daemon
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-06-07 19:56 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
Theodore Tso <tytso@mit.edu> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |CODE_FIX
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread* [Bug 13232] ext3/4 with synchronous writes gets wedged by Postfix
2009-05-05 21:58 [Bug 13232] New: ext3/4 with synchronous writes gets wedged by Postfix bugzilla-daemon
` (14 preceding siblings ...)
2009-06-07 19:56 ` bugzilla-daemon
@ 2009-06-07 20:44 ` bugzilla-daemon
15 siblings, 0 replies; 25+ messages in thread
From: bugzilla-daemon @ 2009-06-07 20:44 UTC (permalink / raw)
To: linux-ext4
http://bugzilla.kernel.org/show_bug.cgi?id=13232
Rafael J. Wysocki <rjw@sisk.pl> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |CLOSED
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 25+ messages in thread