[Bug 42763] New: directory access hangs without error

[Bug 42763] New: directory access hangs without error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42763

           Summary: directory access hangs without error
           Product: File System
           Version: 2.5
    Kernel Version: 3.2.5
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: ext4
        AssignedTo: fs_ext4@kernel-bugs.osdl.org
        ReportedBy: ebuddington@wesleyan.edu
        Regression: No


Kernel 3.2.5
ext4 over RAID-6

I have a specific directory that freezes all processes that try to getdents()
or open() a new file. In some cases, the kernel gives  me "blocked for more
than 120 seconds" messages, often reporting that the process is stuck in
ext4_getblk.

I have found no errors, however. The stuck processes stay stuck forever (at
least hours), and dmesg shows no complaints about RAID, filesystem, or anything
else. Full reads of the RAID devices work without hanging, and drive self-tests
pass.

Other processes are able to access other parts of the filesystem normally; this
is not a system-wide or fs-wide hang.

Rebooting and fscking made the directory accessible again, but now there is a
different directory exhibiting the problem (unknown whether it had the problem
before the fsck/reboot)

This particular problem exhibited itself for the first time after tests of my
multi-threaded 'du' that beat the filesystem with a few dozen threads
simultaneously.

Given that this seems to very reproducible, I have many opportunities to poke
at a hung process, query the filesystem, or recompile the kernel in any way
that would be helpful; I just don't know to approach it from here.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 42763] directory access hangs without error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42763





--- Comment #1 from Eric Buddington <ebuddington@wesleyan.edu>  2012-02-13 15:55:11 ---
More info:

After a reboot (and forced but uneventful fsck), the problem is gone until I
run my program again. The program consists of multiple threads traversing and
stat()ing different directory trees simultaneously.

The program will actually work fine for several iterations before getting
stuck,
but once it gets stuck, that directory is a trap for all processes until
reboot.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 42763] directory access hangs without error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42763





--- Comment #2 from Eric Buddington <ebuddington@wesleyan.edu>  2012-02-13 16:02:51 ---
Created an attachment (id=72366)
 --> (https://bugzilla.kernel.org/attachment.cgi?id=72366)
C program to trigger the bug

compile with --std=gnu99 -lpthread -lrt

invocation is ./edu <directory> <number of threads>

triggers the bug after about a dozen iteratons. Program will stall showing
"1a", indicating one still-active thread, and fail to exit.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 42763] directory access hangs without error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42763


Eric Sandeen <sandeen@redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sandeen@redhat.com




--- Comment #3 from Eric Sandeen <sandeen@redhat.com>  2012-02-13 16:56:54 ---
Did fsck ever find any problems with the filesystem?

I'm not clear on whether a simple reboot will clear it up, or if it requires a
fsck to get things happy again.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 42763] directory access hangs without error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42763





--- Comment #4 from Eric Buddington <ebuddington@wesleyan.edu>  2012-02-13 17:57:16 ---
fsck did not find any problems with the filesystem.

I rebooted without any fsck, and was found the same symptoms; a couple dozen
successful runs, then a hang (this time, it seems to be on two directories
simultaneously).

I have been dropping caches between runs. I haven't tried to trigger the bug
without dropping caches.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 42763] directory access hangs without error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42763





--- Comment #5 from Eric Sandeen <sandeen@redhat.com>  2012-02-13 18:21:23 ---
Ok, I'm traveling w/ limited time to try to reproduce but thanks for all the
info.

A sysrq-W or sysrq-T when it's hung might show us where the threads are at.

-Eric

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 42763] directory access hangs without error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42763





--- Comment #6 from Eric Buddington <ebuddington@wesleyan.edu>  2012-02-13 18:30:27 ---
The stuck threads look like this:

edu             D c023a2f4     0  9912      1 0x00000004
f50b2b80 00000086 00000000 c023a2f4 f7b2b400 d5350000 c09f6d80 00000000
c09f6d80 c1c5f500 0000000a c33dbee0 c023f172 00000000 d53515cc c33dbee0
000015cc d5352000 c8c4b4a4 c33dbee0 c1c5f500 f0e05dac c01558a1 00000246
Call Trace:
[<c023a2f4>] ? ext4_getblk+0x8b/0x13d
[<c023f172>] ? search_dirblock+0x76/0xaf
[<c01558a1>] ? arch_local_irq_save+0xf/0x14
[<c0651740>] ? _raw_spin_lock_irqsave+0x8/0x2c
[<c01c2cc3>] ? inode_wait+0x5/0x8
[<c0650c36>] ? __wait_on_bit+0x2f/0x54
[<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
[<c0650cba>] ? out_of_line_wait_on_bit+0x5f/0x67
[<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
[<c014532b>] ? autoremove_wake_function+0x2f/0x2f
[<c01c3610>] ? wait_on_bit.constprop.13+0x22/0x25
[<c01c3c8b>] ? iget_locked+0x42/0xc5
[<c023aad8>] ? ext4_iget+0x24/0x5be
[<c01bad90>] ? do_lookup+0x1e4/0x224
[<c01243a6>] ? should_resched+0x5/0x1e
[<c06506de>] ? _cond_resched+0x5/0x18
[<c01a61c9>] ? slab_pre_alloc_hook.isra.62+0x20/0x23
[<c01a6d76>] ? kmem_cache_alloc+0x1c/0xb1
[<c0240b61>] ? ext4_lookup.part.29+0x50/0xc6
[<c01c1cfc>] ? __d_alloc+0xec/0xfb
[<c01ba3b0>] ? d_alloc_and_lookup+0x2c/0x49
[<c01bacf9>] ? do_lookup+0x14d/0x224
[<c01bae26>] ? walk_component+0x56/0x10c
[<c01baf15>] ? lookup_last+0x39/0x3f
[<c01bb9c3>] ? path_lookupat+0x74/0x238
[<c01243a6>] ? should_resched+0x5/0x1e
[<c06506de>] ? _cond_resched+0x5/0x18
[<c01bbba1>] ? do_path_lookup+0x1a/0x4f
[<c01bcd77>] ? user_path_at_empty+0x3d/0x69
[<c01b6065>] ? cp_new_stat64+0xec/0xfe
[<c01bcdbb>] ? user_path_at+0x18/0x1d
[<c01b6410>] ? vfs_fstatat+0x3f/0x67
[<c01b644e>] ? vfs_lstat+0x16/0x18
[<c01b6645>] ? sys_lstat64+0xe/0x21
[<c01b81a4>] ? flush_old_exec+0x29/0x81
[<c0655f63>] ? sysenter_do_call+0x12/0x2c

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 42763] directory access hangs without error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42763





--- Comment #7 from Eric Buddington <ebuddington@wesleyan.edu>  2012-02-13 20:29:13 ---
If I run the multi-threaded 'du' *without* flushing caches between, I don't
seem to trigger the problem (after ~250 iterations, whereas ~20 trigger it
otherwise)

The command I use that triggers the bug (/packages is about 17G of ordinary
software installs):

n=1; while [ $n -lt 128 ]; do sync; echo 3 > /proc/sys/vm/drop_caches; ./edu
/packages $n; n=$[$n+1]; done

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 42763] directory access hangs without error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42763





--- Comment #8 from Eric Buddington <ebuddington@wesleyan.edu>  2012-02-14 05:07:39 ---
I just discovered that one RAID drive has large and increasing SMART values for
Raw_Read_Error_Rate and Hardware_ECC_Recovered.

Apparently, any disk read errors are being recovered, since I'm not seeing any
errors in dmesg or /proc/mdstat - so I still think there's an ext4 problem
(though perhaps it takes a flooded queue and a very-slow-response disk to
trigger...)

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

Re: [Bug 42763] directory access hangs without error

[Jan Kara]

On Mon 13-02-12 18:30:28, bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=42763
> --- Comment #6 from Eric Buddington <ebuddington@wesleyan.edu>  2012-02-13 18:30:27 ---
> The stuck threads look like this:
> 
> edu             D c023a2f4     0  9912      1 0x00000004
> f50b2b80 00000086 00000000 c023a2f4 f7b2b400 d5350000 c09f6d80 00000000
> c09f6d80 c1c5f500 0000000a c33dbee0 c023f172 00000000 d53515cc c33dbee0
> 000015cc d5352000 c8c4b4a4 c33dbee0 c1c5f500 f0e05dac c01558a1 00000246
> Call Trace:
> [<c023a2f4>] ? ext4_getblk+0x8b/0x13d
> [<c023f172>] ? search_dirblock+0x76/0xaf
> [<c01558a1>] ? arch_local_irq_save+0xf/0x14
> [<c0651740>] ? _raw_spin_lock_irqsave+0x8/0x2c
> [<c01c2cc3>] ? inode_wait+0x5/0x8
> [<c0650c36>] ? __wait_on_bit+0x2f/0x54
> [<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
> [<c0650cba>] ? out_of_line_wait_on_bit+0x5f/0x67
> [<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
> [<c014532b>] ? autoremove_wake_function+0x2f/0x2f
> [<c01c3610>] ? wait_on_bit.constprop.13+0x22/0x25
> [<c01c3c8b>] ? iget_locked+0x42/0xc5
> [<c023aad8>] ? ext4_iget+0x24/0x5be
  ...
  Interesting. So this isn't ext4 related at all. Rather it's a generic bug
in VFS's I_NEW handling introduced by 250df6ed (adding Dave and Al to CC).
That commit removed wake_up_inode() (in particular a memory barrier before
wake_up_bit()) on the basis that i_state transitions are protected by
i_lock. That would be fine if all the readers of i_state were using i_lock
as well. But they don't - in particular wait_on_inode() from
include/linux/writeback.h does not. So that commit opened a reordering
possibility where __I_NEW can be cleared *after* wake_up_bit() in
unlock_new_inode() happens and so wait_on_bit() in wait_on_inode() goes
to sleep indefinitely.

It seems to me the intent was that wait_on_inode() should use i_lock as
well so it would opencode bit waiting similarly to
__wait_on_freeing_inode(). Am I right? Alternatively, we'd have to back out
changes of unlock_new_inode() and wake_up_inode()... 

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [Bug 42763] directory access hangs without error

[Dave Chinner]

On Tue, Feb 14, 2012 at 03:22:31PM +0100, Jan Kara wrote:
> On Mon 13-02-12 18:30:28, bugzilla-daemon@bugzilla.kernel.org wrote:
> > https://bugzilla.kernel.org/show_bug.cgi?id=42763
> > --- Comment #6 from Eric Buddington <ebuddington@wesleyan.edu>  2012-02-13 18:30:27 ---
> > The stuck threads look like this:
> > 
> > edu             D c023a2f4     0  9912      1 0x00000004
> > f50b2b80 00000086 00000000 c023a2f4 f7b2b400 d5350000 c09f6d80 00000000
> > c09f6d80 c1c5f500 0000000a c33dbee0 c023f172 00000000 d53515cc c33dbee0
> > 000015cc d5352000 c8c4b4a4 c33dbee0 c1c5f500 f0e05dac c01558a1 00000246
> > Call Trace:
> > [<c023a2f4>] ? ext4_getblk+0x8b/0x13d
> > [<c023f172>] ? search_dirblock+0x76/0xaf
> > [<c01558a1>] ? arch_local_irq_save+0xf/0x14
> > [<c0651740>] ? _raw_spin_lock_irqsave+0x8/0x2c
> > [<c01c2cc3>] ? inode_wait+0x5/0x8
> > [<c0650c36>] ? __wait_on_bit+0x2f/0x54
> > [<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
> > [<c0650cba>] ? out_of_line_wait_on_bit+0x5f/0x67
> > [<c01c2cbe>] ? inode_owner_or_capable+0x30/0x30
> > [<c014532b>] ? autoremove_wake_function+0x2f/0x2f
> > [<c01c3610>] ? wait_on_bit.constprop.13+0x22/0x25
> > [<c01c3c8b>] ? iget_locked+0x42/0xc5
> > [<c023aad8>] ? ext4_iget+0x24/0x5be
>   ...
>   Interesting. So this isn't ext4 related at all. Rather it's a generic bug
> in VFS's I_NEW handling introduced by 250df6ed (adding Dave and Al to CC).
> That commit removed wake_up_inode() (in particular a memory barrier before
> wake_up_bit()) on the basis that i_state transitions are protected by
> i_lock. That would be fine if all the readers of i_state were using i_lock
> as well.

Hmmmm. I guess I missed that one.

> But they don't - in particular wait_on_inode() from
> include/linux/writeback.h does not. So that commit opened a reordering
> possibility where __I_NEW can be cleared *after* wake_up_bit() in
> unlock_new_inode() happens and so wait_on_bit() in wait_on_inode() goes
> to sleep indefinitely.
> 
> It seems to me the intent was that wait_on_inode() should use i_lock as
> well so it would opencode bit waiting similarly to
> __wait_on_freeing_inode().

Yeah, more like inode_wait_for_writeback() rather than
__wait_on_freeing_inode(), though, as we should loop until the bit
is cleared.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[Bug 42763] directory access hangs without error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42763


Jan Kara <jack@suse.cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jack@suse.cz




--- Comment #9 from Jan Kara <jack@suse.cz>  2012-02-20 17:31:29 ---
Hmm, it seems mirroring of emails and bugzilla is broken. Anyway, thanks for
the traces. It's a generic VFS bug. I'll attach here a fix in a moment.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 42763] directory access hangs without error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42763





--- Comment #10 from Jan Kara <jack@suse.cz>  2012-02-20 17:32:36 ---
Created an attachment (id=72448)
 --> (https://bugzilla.kernel.org/attachment.cgi?id=72448)
Fix missed wakeup on I_NEW

Can you please try this patch? Thanks.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 42763] directory access hangs without error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42763





--- Comment #11 from Eric Buddington <ebuddington@wesleyan.edu>  2012-02-24 02:22:15 ---
Applied to 3.2.7, and it seems to fix it; at least I ran the same test for
about 4x as long as before without the problem exhibiting itself.

Thanks!

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 42763] directory access hangs without error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=42763


Alan <alan@lxorguk.ukuu.org.uk> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |alan@lxorguk.ukuu.org.uk
         Resolution|                            |CODE_FIX




-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.