* [Bug 42763] directory access hangs without error
2012-02-13 3:41 [Bug 42763] New: directory access hangs without error bugzilla-daemon
@ 2012-02-13 15:55 ` bugzilla-daemon
2012-02-13 16:02 ` bugzilla-daemon
` (10 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: bugzilla-daemon @ 2012-02-13 15:55 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=42763
--- Comment #1 from Eric Buddington <ebuddington@wesleyan.edu> 2012-02-13 15:55:11 ---
More info:
After a reboot (and forced but uneventful fsck), the problem is gone until I
run my program again. The program consists of multiple threads traversing and
stat()ing different directory trees simultaneously.
The program will actually work fine for several iterations before getting
stuck,
but once it gets stuck, that directory is a trap for all processes until
reboot.
--
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 15+ messages in thread* [Bug 42763] directory access hangs without error
2012-02-13 3:41 [Bug 42763] New: directory access hangs without error bugzilla-daemon
2012-02-13 15:55 ` [Bug 42763] " bugzilla-daemon
@ 2012-02-13 16:02 ` bugzilla-daemon
2012-02-13 16:56 ` bugzilla-daemon
` (9 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: bugzilla-daemon @ 2012-02-13 16:02 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=42763
--- Comment #2 from Eric Buddington <ebuddington@wesleyan.edu> 2012-02-13 16:02:51 ---
Created an attachment (id=72366)
--> (https://bugzilla.kernel.org/attachment.cgi?id=72366)
C program to trigger the bug
compile with --std=gnu99 -lpthread -lrt
invocation is ./edu <directory> <number of threads>
triggers the bug after about a dozen iteratons. Program will stall showing
"1a", indicating one still-active thread, and fail to exit.
--
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 15+ messages in thread* [Bug 42763] directory access hangs without error
2012-02-13 3:41 [Bug 42763] New: directory access hangs without error bugzilla-daemon
2012-02-13 15:55 ` [Bug 42763] " bugzilla-daemon
2012-02-13 16:02 ` bugzilla-daemon
@ 2012-02-13 16:56 ` bugzilla-daemon
2012-02-13 17:57 ` bugzilla-daemon
` (8 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: bugzilla-daemon @ 2012-02-13 16:56 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=42763
Eric Sandeen <sandeen@redhat.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |sandeen@redhat.com
--- Comment #3 from Eric Sandeen <sandeen@redhat.com> 2012-02-13 16:56:54 ---
Did fsck ever find any problems with the filesystem?
I'm not clear on whether a simple reboot will clear it up, or if it requires a
fsck to get things happy again.
--
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 15+ messages in thread* [Bug 42763] directory access hangs without error
2012-02-13 3:41 [Bug 42763] New: directory access hangs without error bugzilla-daemon
` (2 preceding siblings ...)
2012-02-13 16:56 ` bugzilla-daemon
@ 2012-02-13 17:57 ` bugzilla-daemon
2012-02-13 18:21 ` bugzilla-daemon
` (7 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: bugzilla-daemon @ 2012-02-13 17:57 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=42763
--- Comment #4 from Eric Buddington <ebuddington@wesleyan.edu> 2012-02-13 17:57:16 ---
fsck did not find any problems with the filesystem.
I rebooted without any fsck, and was found the same symptoms; a couple dozen
successful runs, then a hang (this time, it seems to be on two directories
simultaneously).
I have been dropping caches between runs. I haven't tried to trigger the bug
without dropping caches.
--
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 15+ messages in thread* [Bug 42763] directory access hangs without error
2012-02-13 3:41 [Bug 42763] New: directory access hangs without error bugzilla-daemon
` (3 preceding siblings ...)
2012-02-13 17:57 ` bugzilla-daemon
@ 2012-02-13 18:21 ` bugzilla-daemon
2012-02-13 18:30 ` bugzilla-daemon
` (6 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: bugzilla-daemon @ 2012-02-13 18:21 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=42763
--- Comment #5 from Eric Sandeen <sandeen@redhat.com> 2012-02-13 18:21:23 ---
Ok, I'm traveling w/ limited time to try to reproduce but thanks for all the
info.
A sysrq-W or sysrq-T when it's hung might show us where the threads are at.
-Eric
--
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 15+ messages in thread* [Bug 42763] directory access hangs without error
2012-02-13 3:41 [Bug 42763] New: directory access hangs without error bugzilla-daemon
` (4 preceding siblings ...)
2012-02-13 18:21 ` bugzilla-daemon
@ 2012-02-13 18:30 ` bugzilla-daemon
2012-02-14 14:22 ` Jan Kara
2012-02-13 20:29 ` bugzilla-daemon
` (5 subsequent siblings)
11 siblings, 1 reply; 15+ messages in thread
From: bugzilla-daemon @ 2012-02-13 18:30 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=42763
--- Comment #6 from Eric Buddington <ebuddington@wesleyan.edu> 2012-02-13 18:30:27 ---
The stuck threads look like this:
edu D c023a2f4 0 9912 1 0x00000004
f50b2b80 00000086 00000000 c023a2f4 f7b2b400 d5350000 c09f6d80 00000000
c09f6d80 c1c5f500 0000000a c33dbee0 c023f172 00000000 d53515cc c33dbee0
000015cc d5352000 c8c4b4a4 c33dbee0 c1c5f500 f0e05dac c01558a1 00000246
Call Trace:
[<c023a2f4>] ? ext4_getblk+0x8b/0x13d
[<c023f172>] ? search_dirblock+0x76/0xaf
[<c01558a1>] ? arch_local_irq_save+0xf/0x14
[<c0651740>] ? _raw_spin_lock_irqsave+0x8/0x2c
[<c01c2cc3>] ? inode_wait+0x5/0x8
[<c0650c36>] ? __wait_on_bit+0x2f/0x54
[<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
[<c0650cba>] ? out_of_line_wait_on_bit+0x5f/0x67
[<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
[<c014532b>] ? autoremove_wake_function+0x2f/0x2f
[<c01c3610>] ? wait_on_bit.constprop.13+0x22/0x25
[<c01c3c8b>] ? iget_locked+0x42/0xc5
[<c023aad8>] ? ext4_iget+0x24/0x5be
[<c01bad90>] ? do_lookup+0x1e4/0x224
[<c01243a6>] ? should_resched+0x5/0x1e
[<c06506de>] ? _cond_resched+0x5/0x18
[<c01a61c9>] ? slab_pre_alloc_hook.isra.62+0x20/0x23
[<c01a6d76>] ? kmem_cache_alloc+0x1c/0xb1
[<c0240b61>] ? ext4_lookup.part.29+0x50/0xc6
[<c01c1cfc>] ? __d_alloc+0xec/0xfb
[<c01ba3b0>] ? d_alloc_and_lookup+0x2c/0x49
[<c01bacf9>] ? do_lookup+0x14d/0x224
[<c01bae26>] ? walk_component+0x56/0x10c
[<c01baf15>] ? lookup_last+0x39/0x3f
[<c01bb9c3>] ? path_lookupat+0x74/0x238
[<c01243a6>] ? should_resched+0x5/0x1e
[<c06506de>] ? _cond_resched+0x5/0x18
[<c01bbba1>] ? do_path_lookup+0x1a/0x4f
[<c01bcd77>] ? user_path_at_empty+0x3d/0x69
[<c01b6065>] ? cp_new_stat64+0xec/0xfe
[<c01bcdbb>] ? user_path_at+0x18/0x1d
[<c01b6410>] ? vfs_fstatat+0x3f/0x67
[<c01b644e>] ? vfs_lstat+0x16/0x18
[<c01b6645>] ? sys_lstat64+0xe/0x21
[<c01b81a4>] ? flush_old_exec+0x29/0x81
[<c0655f63>] ? sysenter_do_call+0x12/0x2c
--
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [Bug 42763] directory access hangs without error
2012-02-13 18:30 ` bugzilla-daemon
@ 2012-02-14 14:22 ` Jan Kara
2012-02-17 5:33 ` Dave Chinner
0 siblings, 1 reply; 15+ messages in thread
From: Jan Kara @ 2012-02-14 14:22 UTC (permalink / raw)
To: bugzilla-daemon; +Cc: linux-ext4, Al Viro, Dave Chinner
On Mon 13-02-12 18:30:28, bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=42763
> --- Comment #6 from Eric Buddington <ebuddington@wesleyan.edu> 2012-02-13 18:30:27 ---
> The stuck threads look like this:
>
> edu D c023a2f4 0 9912 1 0x00000004
> f50b2b80 00000086 00000000 c023a2f4 f7b2b400 d5350000 c09f6d80 00000000
> c09f6d80 c1c5f500 0000000a c33dbee0 c023f172 00000000 d53515cc c33dbee0
> 000015cc d5352000 c8c4b4a4 c33dbee0 c1c5f500 f0e05dac c01558a1 00000246
> Call Trace:
> [<c023a2f4>] ? ext4_getblk+0x8b/0x13d
> [<c023f172>] ? search_dirblock+0x76/0xaf
> [<c01558a1>] ? arch_local_irq_save+0xf/0x14
> [<c0651740>] ? _raw_spin_lock_irqsave+0x8/0x2c
> [<c01c2cc3>] ? inode_wait+0x5/0x8
> [<c0650c36>] ? __wait_on_bit+0x2f/0x54
> [<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
> [<c0650cba>] ? out_of_line_wait_on_bit+0x5f/0x67
> [<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
> [<c014532b>] ? autoremove_wake_function+0x2f/0x2f
> [<c01c3610>] ? wait_on_bit.constprop.13+0x22/0x25
> [<c01c3c8b>] ? iget_locked+0x42/0xc5
> [<c023aad8>] ? ext4_iget+0x24/0x5be
...
Interesting. So this isn't ext4 related at all. Rather it's a generic bug
in VFS's I_NEW handling introduced by 250df6ed (adding Dave and Al to CC).
That commit removed wake_up_inode() (in particular a memory barrier before
wake_up_bit()) on the basis that i_state transitions are protected by
i_lock. That would be fine if all the readers of i_state were using i_lock
as well. But they don't - in particular wait_on_inode() from
include/linux/writeback.h does not. So that commit opened a reordering
possibility where __I_NEW can be cleared *after* wake_up_bit() in
unlock_new_inode() happens and so wait_on_bit() in wait_on_inode() goes
to sleep indefinitely.
It seems to me the intent was that wait_on_inode() should use i_lock as
well so it would opencode bit waiting similarly to
__wait_on_freeing_inode(). Am I right? Alternatively, we'd have to back out
changes of unlock_new_inode() and wake_up_inode()...
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Bug 42763] directory access hangs without error
2012-02-14 14:22 ` Jan Kara
@ 2012-02-17 5:33 ` Dave Chinner
0 siblings, 0 replies; 15+ messages in thread
From: Dave Chinner @ 2012-02-17 5:33 UTC (permalink / raw)
To: Jan Kara; +Cc: bugzilla-daemon, linux-ext4, Al Viro, Dave Chinner
On Tue, Feb 14, 2012 at 03:22:31PM +0100, Jan Kara wrote:
> On Mon 13-02-12 18:30:28, bugzilla-daemon@bugzilla.kernel.org wrote:
> > https://bugzilla.kernel.org/show_bug.cgi?id=42763
> > --- Comment #6 from Eric Buddington <ebuddington@wesleyan.edu> 2012-02-13 18:30:27 ---
> > The stuck threads look like this:
> >
> > edu D c023a2f4 0 9912 1 0x00000004
> > f50b2b80 00000086 00000000 c023a2f4 f7b2b400 d5350000 c09f6d80 00000000
> > c09f6d80 c1c5f500 0000000a c33dbee0 c023f172 00000000 d53515cc c33dbee0
> > 000015cc d5352000 c8c4b4a4 c33dbee0 c1c5f500 f0e05dac c01558a1 00000246
> > Call Trace:
> > [<c023a2f4>] ? ext4_getblk+0x8b/0x13d
> > [<c023f172>] ? search_dirblock+0x76/0xaf
> > [<c01558a1>] ? arch_local_irq_save+0xf/0x14
> > [<c0651740>] ? _raw_spin_lock_irqsave+0x8/0x2c
> > [<c01c2cc3>] ? inode_wait+0x5/0x8
> > [<c0650c36>] ? __wait_on_bit+0x2f/0x54
> > [<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
> > [<c0650cba>] ? out_of_line_wait_on_bit+0x5f/0x67
> > [<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
> > [<c014532b>] ? autoremove_wake_function+0x2f/0x2f
> > [<c01c3610>] ? wait_on_bit.constprop.13+0x22/0x25
> > [<c01c3c8b>] ? iget_locked+0x42/0xc5
> > [<c023aad8>] ? ext4_iget+0x24/0x5be
> ...
> Interesting. So this isn't ext4 related at all. Rather it's a generic bug
> in VFS's I_NEW handling introduced by 250df6ed (adding Dave and Al to CC).
> That commit removed wake_up_inode() (in particular a memory barrier before
> wake_up_bit()) on the basis that i_state transitions are protected by
> i_lock. That would be fine if all the readers of i_state were using i_lock
> as well.
Hmmmm. I guess I missed that one.
> But they don't - in particular wait_on_inode() from
> include/linux/writeback.h does not. So that commit opened a reordering
> possibility where __I_NEW can be cleared *after* wake_up_bit() in
> unlock_new_inode() happens and so wait_on_bit() in wait_on_inode() goes
> to sleep indefinitely.
>
> It seems to me the intent was that wait_on_inode() should use i_lock as
> well so it would opencode bit waiting similarly to
> __wait_on_freeing_inode().
Yeah, more like inode_wait_for_writeback() rather than
__wait_on_freeing_inode(), though, as we should loop until the bit
is cleared.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Bug 42763] directory access hangs without error
2012-02-13 3:41 [Bug 42763] New: directory access hangs without error bugzilla-daemon
` (5 preceding siblings ...)
2012-02-13 18:30 ` bugzilla-daemon
@ 2012-02-13 20:29 ` bugzilla-daemon
2012-02-14 5:07 ` bugzilla-daemon
` (4 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: bugzilla-daemon @ 2012-02-13 20:29 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=42763
--- Comment #7 from Eric Buddington <ebuddington@wesleyan.edu> 2012-02-13 20:29:13 ---
If I run the multi-threaded 'du' *without* flushing caches between, I don't
seem to trigger the problem (after ~250 iterations, whereas ~20 trigger it
otherwise)
The command I use that triggers the bug (/packages is about 17G of ordinary
software installs):
n=1; while [ $n -lt 128 ]; do sync; echo 3 > /proc/sys/vm/drop_caches; ./edu
/packages $n; n=$[$n+1]; done
--
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 15+ messages in thread* [Bug 42763] directory access hangs without error
2012-02-13 3:41 [Bug 42763] New: directory access hangs without error bugzilla-daemon
` (6 preceding siblings ...)
2012-02-13 20:29 ` bugzilla-daemon
@ 2012-02-14 5:07 ` bugzilla-daemon
2012-02-20 17:31 ` bugzilla-daemon
` (3 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: bugzilla-daemon @ 2012-02-14 5:07 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=42763
--- Comment #8 from Eric Buddington <ebuddington@wesleyan.edu> 2012-02-14 05:07:39 ---
I just discovered that one RAID drive has large and increasing SMART values for
Raw_Read_Error_Rate and Hardware_ECC_Recovered.
Apparently, any disk read errors are being recovered, since I'm not seeing any
errors in dmesg or /proc/mdstat - so I still think there's an ext4 problem
(though perhaps it takes a flooded queue and a very-slow-response disk to
trigger...)
--
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 15+ messages in thread* [Bug 42763] directory access hangs without error
2012-02-13 3:41 [Bug 42763] New: directory access hangs without error bugzilla-daemon
` (7 preceding siblings ...)
2012-02-14 5:07 ` bugzilla-daemon
@ 2012-02-20 17:31 ` bugzilla-daemon
2012-02-20 17:32 ` bugzilla-daemon
` (2 subsequent siblings)
11 siblings, 0 replies; 15+ messages in thread
From: bugzilla-daemon @ 2012-02-20 17:31 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=42763
Jan Kara <jack@suse.cz> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |jack@suse.cz
--- Comment #9 from Jan Kara <jack@suse.cz> 2012-02-20 17:31:29 ---
Hmm, it seems mirroring of emails and bugzilla is broken. Anyway, thanks for
the traces. It's a generic VFS bug. I'll attach here a fix in a moment.
--
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 15+ messages in thread* [Bug 42763] directory access hangs without error
2012-02-13 3:41 [Bug 42763] New: directory access hangs without error bugzilla-daemon
` (8 preceding siblings ...)
2012-02-20 17:31 ` bugzilla-daemon
@ 2012-02-20 17:32 ` bugzilla-daemon
2012-02-24 2:22 ` bugzilla-daemon
2012-08-30 14:35 ` bugzilla-daemon
11 siblings, 0 replies; 15+ messages in thread
From: bugzilla-daemon @ 2012-02-20 17:32 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=42763
--- Comment #10 from Jan Kara <jack@suse.cz> 2012-02-20 17:32:36 ---
Created an attachment (id=72448)
--> (https://bugzilla.kernel.org/attachment.cgi?id=72448)
Fix missed wakeup on I_NEW
Can you please try this patch? Thanks.
--
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 15+ messages in thread* [Bug 42763] directory access hangs without error
2012-02-13 3:41 [Bug 42763] New: directory access hangs without error bugzilla-daemon
` (9 preceding siblings ...)
2012-02-20 17:32 ` bugzilla-daemon
@ 2012-02-24 2:22 ` bugzilla-daemon
2012-08-30 14:35 ` bugzilla-daemon
11 siblings, 0 replies; 15+ messages in thread
From: bugzilla-daemon @ 2012-02-24 2:22 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=42763
--- Comment #11 from Eric Buddington <ebuddington@wesleyan.edu> 2012-02-24 02:22:15 ---
Applied to 3.2.7, and it seems to fix it; at least I ran the same test for
about 4x as long as before without the problem exhibiting itself.
Thanks!
--
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 15+ messages in thread* [Bug 42763] directory access hangs without error
2012-02-13 3:41 [Bug 42763] New: directory access hangs without error bugzilla-daemon
` (10 preceding siblings ...)
2012-02-24 2:22 ` bugzilla-daemon
@ 2012-08-30 14:35 ` bugzilla-daemon
11 siblings, 0 replies; 15+ messages in thread
From: bugzilla-daemon @ 2012-08-30 14:35 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=42763
Alan <alan@lxorguk.ukuu.org.uk> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
CC| |alan@lxorguk.ukuu.org.uk
Resolution| |CODE_FIX
--
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 15+ messages in thread