Re: processes hung after sys_renameat, and 'missing' processes

[Dave Jones <davej@redhat.com>]

On Mon, Jun 04, 2012 at 12:28:20AM +0100, Al Viro wrote:
 > On Mon, Jun 04, 2012 at 12:17:09AM +0100, Al Viro wrote:
 > > 
 > > > Also, sysrq-w is usually way more interesting than 't' when there are
 > > > processes stuck on a mutex.
 > > > 
 > > > Because yes, it looks like you have a boattload of trinity processes
 > > > stuck on an inode mutex. Looks like every single one of them is in
 > > > 'lock_rename()'. It *shouldn't* be an ABBA deadlock, since lockdep
 > > > should have noticed that, but who knows.
 > > 
 > > lock_rename() is a bit of a red herring here - they appear to be all
 > > within-directory renames, so it's just a "trying to rename something
 > > in a directory that has ->i_mutex held by something else".
 > > 
 > > IOW, something else in there is holding ->i_mutex - something that
 > > either hadn't been through lock_rename() at all or has already
 > > passed through it and still hadn't got around to unlock_rename().
 > > In either case, suspects won't have lock_rename() in the trace...
 > 
 > Everything in lock_rename() appears to be at lock_rename+0x3e.  Unless
 > there's a really huge amount of filesystems on that box, this has to
 > be
 >                 mutex_lock_nested(&p1->d_inode->i_mutex, I_MUTEX_PARENT);
 > and everything on that sucker is not holding any locks yet.  IOW, that's
 > the tail hanging off whatever deadlock is there.
 > 
 > One possibility is that something has left the kernel without releasing
 > i_mutex on some directory, which would make atomic_open patches the most
 > obvious suspects.

Just hit this again on a different box, though this time the stack traces
of the stuck processes seems to vary between fchmod/fchown/getdents calls.

partial dmesg at http://fpaste.org/jBVM/  
sysrq-w: http://fpaste.org/uYtj/
sysrq-d: http://fpaste.org/Xxur/

does this give any new clues that the previous traces didn't ?

	Dave