Re: processes hung after sys_renameat, and 'missing' processes

[Dave Jones <davej@redhat.com>]

On Sun, Jun 03, 2012 at 06:36:17PM -0400, Dave Jones wrote:
 > I noticed I had a ton of core dumps (like 70G worth) in a directory
 > I hadn't cleaned up in a while, and set about deleting them.
 > After a while I noticed the rm wasn't making any progress.
 > Even more strange, the rm process doesn't show up in the process list.
 > The shell that spawned it is still there, with no child processes,
 > but it hasn't returned to accept new input. (no message of oom kills or
 > anything, just totally missing pids).
 > 
 > 
 > I did sysrq-t to see if it showed up there. It didn't, but.. I noticed
 > a ton of processes from my syscall fuzzer were still around, and all
 > of them were stuck in this trace..
 > 
 > 
 > trinity-child2  D 0000000000000000  5528 13066      1 0x00000004
 >  ffff880100a37ce8 0000000000000046 0000000000000006 ffff880129070000
 >  ffff880129070000 ffff880100a37fd8 ffff880100a37fd8 ffff880100a37fd8
 >  ffff880145ec4d60 ffff880129070000 ffff880100a37cd8 ffff88014784e2a0
 > Call Trace:
 >  [<ffffffff8164b919>] schedule+0x29/0x70
 >  [<ffffffff8164bca8>] schedule_preempt_disabled+0x18/0x30
 >  [<ffffffff8164a186>] mutex_lock_nested+0x196/0x3b0
 >  [<ffffffff811b6d6e>] ? lock_rename+0x3e/0xf0
 >  [<ffffffff811b6d6e>] ? lock_rename+0x3e/0xf0
 >  [<ffffffff811b6d6e>] lock_rename+0x3e/0xf0
 >  [<ffffffff811bcaca>] sys_renameat+0x11a/0x230
 >  [<ffffffff8164d738>] ? _raw_spin_unlock_irqrestore+0x38/0x80
 >  [<ffffffff81050e1c>] ? do_setitimer+0x1cc/0x310
 >  [<ffffffff810b1d7e>] ? put_lock_stats.isra.23+0xe/0x40
 >  [<ffffffff8164d6d0>] ? _raw_spin_unlock_irq+0x30/0x60
 >  [<ffffffff81086f81>] ? get_parent_ip+0x11/0x50
 >  [<ffffffff81655177>] ? sysret_check+0x1b/0x56
 >  [<ffffffff810b7cd5>] ? trace_hardirqs_on_caller+0x115/0x1a0
 >  [<ffffffff813264be>] ? trace_hardirqs_on_thunk+0x3a/0x3f
 >  [<ffffffff811bcbfb>] sys_rename+0x1b/0x20
 >  [<ffffffff81655152>] system_call_fastpath+0x16/0x1b
 > 
 > The whole sysrq-t is attached.
 > 
 > I ran mc to try and kill off all those core files, as I was running low on disk space,
 > and it deleted them without problem.
 > 
 > The two bash processes are chewing up 100% CPU, though strace shows no output.

trying to run perf causes hung perf processes too. hrmph, messed up.

	Dave

perf            D 0000000000000000  3944  1525   1613 0x00000004
 ffff880103e49d58 0000000000000046 0000000000000006 ffff88012c0d4d60
 ffff88012c0d4d60 ffff880103e49fd8 ffff880103e49fd8 ffff880103e49fd8
 ffff880145edcd60 ffff88012c0d4d60 ffff880103e49d48 ffff88013ea31310
Call Trace:
 [<ffffffff8164b919>] schedule+0x29/0x70
 [<ffffffff8164bca8>] schedule_preempt_disabled+0x18/0x30
 [<ffffffff816498a6>] mutex_lock_killable_nested+0x1a6/0x470
 [<ffffffff81046134>] ? mm_access+0x34/0xc0
 [<ffffffff81046134>] ? mm_access+0x34/0xc0
 [<ffffffff81046134>] mm_access+0x34/0xc0
 [<ffffffff8106eef0>] ? pid_task+0xd0/0xd0
 [<ffffffff81213b2c>] m_start+0x7c/0x190
 [<ffffffff811cfa50>] seq_read+0xa0/0x3e0
 [<ffffffff811aae1c>] vfs_read+0xac/0x180
 [<ffffffff811aaf3d>] sys_read+0x4d/0x90
 [<ffffffff81655152>] system_call_fastpath+0x16/0x1b
perf            x 0000000000000000  5496  1526   1525 0x00000004
 ffff88013fbf7cb8 0000000000000046 ffff88013fbf7c68 ffffffff810b248c
 ffff8801423f8000 ffff88013fbf7fd8 ffff88013fbf7fd8 ffff88013fbf7fd8
 ffff880145ee8000 ffff8801423f8000 ffff88013fbf7ca8 ffff8801423f87e0
Call Trace:
 [<ffffffff810b248c>] ? lock_release_holdtime.part.24+0xcc/0x140
 [<ffffffff8164b919>] schedule+0x29/0x70
 [<ffffffff8104f960>] do_exit+0x670/0xb90
 [<ffffffff810627e1>] ? get_signal_to_deliver+0x291/0x930
 [<ffffffff810501cc>] do_group_exit+0x4c/0xc0
 [<ffffffff8106281e>] get_signal_to_deliver+0x2ce/0x930
 [<ffffffff8100225f>] do_signal+0x3f/0x610
 [<ffffffff812b5cc5>] ? security_file_permission+0x95/0xb0
 [<ffffffff811aa961>] ? rw_verify_area+0x61/0xf0
 [<ffffffff816551cc>] ? sysret_signal+0x5/0x47
 [<ffffffff810028d8>] do_notify_resume+0x88/0xc0
 [<ffffffff8165545a>] int_signal+0x12/0x17