Re: Limit hash table size

[Steve Lord <lord@xfs.org>]

Andrew Morton wrote:
> Steve Lord <lord@xfs.org> wrote:
> 
>> I have seen some dire cases with the dcache, SGI had some boxes with
>> millions of files out there, and every night a cron job would come
>> along and suck them all into memory. Resources got tight at some point,
>> and as more inodes and dentries were being read in, the try to free
>> pages path was continually getting called. There was always something
>> in filesystem cache which could get freed, and the inodes and dentries
>> kept getting more and more of the memory.
> 
> 
> There are a number of variables here.  Certainly, the old
> inodes-pinned-by-highmem pagecache will cause this to happen - badly.  2.6
> is pretty aggressive at killing off those inodes.
> 
> What kernel was it?
> 
> Was it a highmem box?  If so, was the filesystem in question placing
> directory pagecache in highmem?  If so, that was really bad on older 2.4:
> the directory pagecache in highmem pins down all directory inodes.
> 

This is where my memory gets a little hazy, its been a few months.
This would have been a 2.4 kernel (probably around 2.4.21)
on an Altix, the filesystem was XFS. So no highmem, but definitely
not your standard kernel.

I never had time to dig into it too much, but I always thought that
on machines with large amounts of memory it was too easy for the
inode and dcache pools to get very large at the expense of the
regular memory zones. While there were pages available, the dcache
and inode zones could just keep on growing. If you run a big
enough find you get lots of memory into the dcache zone and
have a hard time getting it out again.

And it was try_to_free_pages I was referring to.

It does look like 2.6 does better, but I don't have quite the
amount of memory on my laptop....

Steve