Re: odd userspace lockups.

[Dave Jones <davej@redhat.com>]

On Tue, Nov 26, 2013 at 10:44:16AM -0800, Linus Torvalds wrote:
 
 > > At first I thought it was just being really slow, (especially after seeing
 > > all those congestion related symbols in the stack traces), but even after
 > > being left for hours, it doesn't seem to progress.
 > 
 > It *does* seem to be hung in some congestion thing. Possibly brought
 > on by being low on memory due to a memory leak.
 > 
 > I'm not seeing any suspicious locks: the sshd that holds the mmap_sem
 > locks for a fork() seems to be in congestion-wait for the disk due to
 > trying to allocate memory, exactly like all the trinity children are.
 > 
 > You don't have memory information in your dump, but to me it looks
 > like you're basically out of memory (there's a *lot* of
 > trinity-children trying to allocate pages), and the oom killer isn't
 > triggering for whatever reason.
 > 
 > Of course, it could easily be a kernel memory leak too (rather than
 > trinity), triggered by your odd workload.

So, completely different machine: Yesterday I got back from my vacation to
find my router running 3.12+ from a week earlier had hung completely.
On the screen was some oom killer output.  After bringing it back up
I looked at the mrtg graphs of it for the last week:
http://codemonkey.org.uk/junk/mem.free-week.png
Those stair-steps are around the time of morning that cron runs.

Normally I reboot that machine every day to update to the latest kernel,
but because I was on vacation, it had enough uptime to accumulate a leak.

That machine is just routing packets (and making pretty mrtg graphs) though,
not doing any fuzz testing or anything weird.

Who knows, maybe it's something totally unrelated. I'll keep poking at things,
and try and find an easier way to reproduce it.

	Dave