Re: [BUG?] OOM with large cache....(x86_64, 2.6.24-rc3-git1, nohz)

[iank@bredband.net]

On Tue, Nov 20, 2007 at 09:47:32PM +1100, Nick Piggin wrote:
> On Tuesday 20 November 2007 20:09, Ian Kumlien wrote:
> > On tis, 2007-11-20 at 15:13 +1100, Nick Piggin wrote:
> 
> > > It's also used up all your 2.5GB of swap. The output of your `free` shows
> > > a fair bit of disk cache there, but it also shows a lot of swap free,
> > > which isn't the case at oom-time.
> >
> > Yes, as i said those was from several hours later...
> 
> OK, missed that.

It all happened at home while i was at work and the machine there went
down due to hardware failure, so i have no logs at all.
(It has a maxtor disk that sometimes just hangs... )

> > > Unfortunately, we don't show NR_ANON_PAGES in these stats, but at a
> > > guess, I'd say that the file cache is mostly shrunk and you still don't
> > > have enough memory. trackerd probably has a memory leak in it, or else is
> > > just trying to allocate more memory than you have. Is this a regression?
> >
> > I have had it happen twice before, without tracker running...
> 
> Does the machine recover afterward, or is the memory freed up after
> the OOM kill? How about if you kill X and do a sysrq+E then sysrq+I
> (to kill all tasks)?

Yes it recovers, it only happens during high I/O with rtorrent.
In this case, downloading a 8gb image at ~6mb/s.

> If the memory still isn't freed after that, then we could have a
> kernel memory leak.

I don't think it's a kernel memoryleak, my suspicion is that the kernel,
for some reason, doesn't want to return the cache when it needs more
memory. Perhaps due to the read and write patterns that rtorrent
creates.

> > It didn't quite get to the oom stage, it just failed alot of allocations
> > while having 1.5 or more memory locked in cache.
> >
> > http://marc.info/?l=linux-kernel&m=118895576924867&w=2
> 
> OK, yes this is a different case, and it is very far off the OOM
> killing stage. Atomic allocations can fail quite easily, but
> kswapd will have started up and will start freeing memory.

Ok, this ONLY happens with rtorrent and large images...
I'll try to reproduce it locally as soon as my schedule permits.

> Actually there are some patches gone into 2.6.24 that have fixed
> some corner cases with the network stack making order-1 allocations
> when it should be order-0. That might help your atomic allocation
> failures, but they aren't really a bug anyway (unless networking
> fails to recover).

Yeah, i remember reading about that, thats why i upgraded =)

> Thanks for reporting all this...

I just hope that something worthwhile comes out of it =)

PS. Perhaps i should have mentioned that all writes goes to a RAID5
array. but i doubt it matters..
DS.

Wow, i really need caffine =)

I/O, I/O, I/O off to work I go