Re: Cache memory never gets released

Re: Cache memory never gets released

[David Ashley]

I tried upgrading to 2.4.26 and this has the CONFIG_OOM_KILLER which will
probably improve the situation, but the kernel still has the cache problem.

Is there some way I can get a report of all the cached memory, wether it
is inodes or blocks or whatever? I can deal with modifying the kernel to
put in printk's if needbe.

The approach I'm thinking of is
1) Get report on all kernel cached memory
2) do what it takes to increase the cached memory so that it can't get reduced
3) Get another report, and see what's changed.

Thanks for any suggestions.

-Dave
PS Basically what causes the problem the worst is repeatedly doing this:
1) Launch mozilla browser with latest flash plugin (x86)
2) Load a flash site that uses a large japanese unicode font
3) goto step 2 (that is, reload)

Mozilla + flash have a memory leak so every time the japanese font is reloaded
mozilla uses up 3 more megs of ram. A watchdog mechanism kills mozilla when
it uses up too much memory. But after a while doing this, the cached memory
as reported by "free" grows and can't be reduced. So finally rather than the
watchdog killing mozilla and all being ok, the linux kernel kills the XFree86
server in order to free up memory and the system is dead. This last is the
original reason for us tracking down the problem, however the kernel killing
processes is not required for the cache problem to occur.

Re: Cache memory never gets released

[Marcelo Tosatti]

On Thu, Jun 24, 2004 at 05:31:41PM -0700, David Ashley wrote:
> I tried upgrading to 2.4.26 and this has the CONFIG_OOM_KILLER which will
> probably improve the situation, but the kernel still has the cache problem.
> 
> Is there some way I can get a report of all the cached memory, wether it
> is inodes or blocks or whatever? I can deal with modifying the kernel to
> put in printk's if needbe.
> 
> The approach I'm thinking of is
> 1) Get report on all kernel cached memory
> 2) do what it takes to increase the cached memory so that it can't get reduced
> 3) Get another report, and see what's changed.
> 
> Thanks for any suggestions.
> 
> -Dave
> PS Basically what causes the problem the worst is repeatedly doing this:
> 1) Launch mozilla browser with latest flash plugin (x86)
> 2) Load a flash site that uses a large japanese unicode font
> 3) goto step 2 (that is, reload)
> 
> Mozilla + flash have a memory leak so every time the japanese font is reloaded
> mozilla uses up 3 more megs of ram. A watchdog mechanism kills mozilla when
> it uses up too much memory. But after a while doing this, the cached memory
> as reported by "free" grows and can't be reduced. 

Cached memory can be easily reclaimed, take a look at /proc/meminfo "Inactive" list.

> So finally rather than the
> watchdog killing mozilla and all being ok, the linux kernel kills the XFree86
> server in order to free up memory and the system is dead.

Add more swap.

>  This last is the
> original reason for us tracking down the problem, however the kernel killing
> processes is not required for the cache problem to occur.

Re: Cache memory never gets released

[David Ashley]

Marcelo Tosatti wrote:
>Cached memory can be easily reclaimed, take a look at /proc/meminfo "Inactive"
>list.

Here is /proc/meminfo from a box that has all but exhausted its free memory:
        total:    used:    free:  shared: buffers:  cached:
Mem:  122064896 84348928 37715968        0   634880 74829824
Swap:        0        0        0
MemTotal:       119204 kB
MemFree:         36832 kB
MemShared:           0 kB
Buffers:           620 kB
Cached:          73076 kB
SwapCached:          0 kB
Active:          70380 kB
Inactive:         7324 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       119204 kB
LowFree:         36832 kB
SwapTotal:           0 kB
SwapFree:            0 kB

>Add more swap.

Might as well suggest walking on water. The hardware is set in stone, this is
a software issue :^).

Something is preventing the cached memory to get reused, it's like it's gone
for good.

Is there any count of how often a cached block is accessed? If there is such
a count, and the count has an effect on whether to allow the release of the
cached block, that could explain this behaviour. Because it could turn out
that the cached blocks are accessed thousands of times.

Thanks--
Dave

Re: Cache memory never gets released

[Richard B. Johnson]

On Fri, 25 Jun 2004, David Ashley wrote:

> Marcelo Tosatti wrote:
> >Cached memory can be easily reclaimed, take a look at /proc/meminfo "Inactive"
> >list.
>
> Here is /proc/meminfo from a box that has all but exhausted its free memory:
>         total:    used:    free:  shared: buffers:  cached:
> Mem:  122064896 84348928 37715968        0   634880 74829824
> Swap:        0        0        0
> MemTotal:       119204 kB
> MemFree:         36832 kB
> MemShared:           0 kB
> Buffers:           620 kB
> Cached:          73076 kB
> SwapCached:          0 kB
> Active:          70380 kB
> Inactive:         7324 kB
> HighTotal:           0 kB
> HighFree:            0 kB
> LowTotal:       119204 kB
> LowFree:         36832 kB
> SwapTotal:           0 kB
> SwapFree:            0 kB
>
> >Add more swap.
>
> Might as well suggest walking on water. The hardware is set in stone, this is
> a software issue :^).
>
> Something is preventing the cached memory to get reused, it's like it's gone
> for good.
>
> Is there any count of how often a cached block is accessed? If there is such
> a count, and the count has an effect on whether to allow the release of the
> cached block, that could explain this behaviour. Because it could turn out
> that the cached blocks are accessed thousands of times.
>
> Thanks--
> Dave

Are you sure you have a problem? If you do `ls -R /` on a file-system
and then look at the cached RAM, you see a lot. It's the dircache.
It is some of the first to be used when the system needs RAM. If you
compile the kernel with `make -j 20 bzImage` so you have a lot of
tasks, needing lots of RAM, the cached value goes way down. It
will eventually be "free" after the compile completes.

Basically, the only way to return cached RAM to the free-list
is to run programs requiring RAM. What is listed as "cached" really
means "RAM that was used, but wasn't freed because we don't need
it yet.... Freeing RAM is as expensive as acquiring it so if it's
used by a kernel buffer, it isn't freed until it's needed.

This is linux-2.4.26. If you are using an exprimental kernel,
memory-allocation might be broken but is probably not.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: Cache memory never gets released

[David Ashley]

Richard B. Johnson wrote:
>Are you sure you have a problem?
>...
>This is linux-2.4.26. If you are using an exprimental kernel,
>memory-allocation might be broken but is probably not.

Yes, I am absolutely sure I have a problem. The cached memory gets
lost so I can't ever use it again over time. I have a simple hog.c program
that just allocs blocks of 1M and fills them with data:
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>

/* This program just hogs a lot of memory */

#define SIZE 0x100000
int main(int argc,char **argv)
{
char *p;
int i=0;
        for(;;)
        {
                p=malloc(SIZE);
                if(!p) break;
                memset(p,0,SIZE);
                i+=SIZE;printf("Allocated %dM\n",i/1000000);
//              sleep(1);
        }
        exit(0);
}


On a pristine system I can work the cached line output by "free"
down to around 6M. After usage, over time, the cached amount works its
way up to be basically all of system memory, and I can't ever work it
down. The hog program above runs but the kernel kills it after only
a small amount of memory can be allocated.

I'm not doing anything mysterious. The kernel is stock 2.4.26 except for
some changes to these files:
Documentation/Configure.help
Makefile
arch/i386/config.in
drivers/sound/via82cxxx_audio.c
fs/proc/array.c
fs/proc/proc_misc.c
include/asm-i386/param.h
kernel/signal.c
kernel/sys.c
net/ipv4/icmp.c

Nothing related to caching/memory management has been touched. I've applied
the variable HZ patch so ticks happen 500/second up from 100/second.

Is there any valid reason why cached memory would stay locked at 54M for
example and never go lower, even when there are almost no processes running
on the system?

Thanks--
Dave
PS Is there some way that cached blocks can somehow form interdependencies?
For example block A can't be released until block B is, and vice-versa?
And that these can build up over time? This is the kind of behaviour I'm
seeing.