Re: Out of memory, but no OOM Killer? (2.6.9-ac11)

[Nick Piggin <nickpiggin@yahoo.com.au>]

Ralf Hildebrandt wrote:
> * Nick Piggin <nickpiggin@yahoo.com.au>:
> 
> 
>>This could be the problem where fragmented memory causes atomic higher
>>order allocations to fail, for which there is a fix in -mm, which should
>>make its way into 2.6.11.
> 
> 
> I see. rsync requested a big chunk of memory, but failed due to the
> fragmentation of free memory? my "sar" output shows lots of free memory and
> lots of unused swap:
> 

Basically, yes. Well not *exactly* rsync - your network drivers. I guess
rsync is showing up in process context most often because that is the
process causing most of the network activity.

Yep, it looks like fragmentation is indeed the problem here. See you have
a lot of memory that is able to be reclaimed, but the failing allocations
themselves can't reclaim any of it because they are happening from
interrupts. What they should be doing is telling `kswapd` to start freeing
memory for them - however this currently doesn't happen properly for
allocations which are order greater than 0.

Fortunately that is usually not a big problem, but as you have seen, it
can be. Anyway, expect 2.6.10 to be better (ie. good enough), and 2.6.11
should have even more complete fixes.

> 
>>As a temporary workaround, you can increase /proc/sys/vm/min_free_kbytes
> 
> 
> # cat /proc/sys/vm/min_free_kbytes
> 724
> 
> I increased that to 7240 now.
> 

OK that should be fine. If you should upgrade to a 2.6.10 or later kernel,
put this value back to the default, and report further problems if they
occur.

> 
>>BTW. what does `free` say when the allocation failures are happening?
> 
> 
> see sar output above.
> 

Thanks Ralf.