Kernel doesn't free Cached Memory

Kernel doesn't free Cached Memory

[Vinicius]

[-- Attachment #1: Mail message body --]
[-- Type: text/plain, Size: 654 bytes --]

Hi all! 

   I have a server with 2 Pentium 4 HT processors and 32 GB of RAM, this 
server runs lots of applications that consume lots of memory to. When I stop 
this applications, the kernel doesn't free memory (the  memory still in use) 
and the server cache lots of memory (~27GB). When I start this applications, 
the kernel sends  "Out of Memory" messages and kill some random 
applications. 

   Anyone know how can I reduce the kernel cached memory on RHEL 3 (kernel 
2.4.21-32.ELsmp - Trial version)? There is a way to reduce the kernel cached 
memory utilization? 

Thanks in advance (sorry my bad english). 

Vinicius. 
Protolink Consultoria. 

Re: Kernel doesn't free Cached Memory

[Bernd Petrovitsch]

On Fri, 2005-07-22 at 08:27 -0300, Vinicius wrote:
[...]
>    I have a server with 2 Pentium 4 HT processors and 32 GB of RAM, this 
> server runs lots of applications that consume lots of memory to. When I stop 
> this applications, the kernel doesn't free memory (the  memory still in use) 
> and the server cache lots of memory (~27GB). When I start this applications, 
> the kernel sends  "Out of Memory" messages and kill some random 
> applications. 
> 
>    Anyone know how can I reduce the kernel cached memory on RHEL 3 (kernel 
> 2.4.21-32.ELsmp - Trial version)? There is a way to reduce the kernel cached 
> memory utilization? 

Probably RedHat's support can answer this for RHEL 3.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: Kernel doesn't free Cached Memory

[linux-os (Dick Johnson)]

On Fri, 22 Jul 2005, Bernd Petrovitsch wrote:

> On Fri, 2005-07-22 at 08:27 -0300, Vinicius wrote:
> [...]
>>    I have a server with 2 Pentium 4 HT processors and 32 GB of RAM, this
>> server runs lots of applications that consume lots of memory to. When I stop
>> this applications, the kernel doesn't free memory (the  memory still in use)
>> and the server cache lots of memory (~27GB). When I start this applications,
>> the kernel sends  "Out of Memory" messages and kill some random
>> applications.
>>
>>    Anyone know how can I reduce the kernel cached memory on RHEL 3 (kernel
>> 2.4.21-32.ELsmp - Trial version)? There is a way to reduce the kernel cached
>> memory utilization?
>
> Probably RedHat's support can answer this for RHEL 3.
>
> 	Bernd
> --
> Firmix Software GmbH                   http://www.firmix.at/
> mobil: +43 664 4416156                 fax: +43 1 7890849-55
>          Embedded Linux Development and Services
>




How do you know the memory is "still in use?"
I think that if you run a program that uses so
much memory that some swap is used, but not
allowing all free memory to be exhausted, you
can run the same program over and over again
forever. Therefore, the memory must have been
freed and available for the next time you
run the program.

The problem is that you are exhausting all
free memory causing the kernel to kill processes
just to stay alive. Then you look at some
memory "tool" and think that it shows something
about what is, or has been freed. You need
to set process limits on the amount of resources
each process can use and, depending upon the
implementation details of your "server", you
might even need to turn memory over-commit off:
  echo "0" > /proc/sys/vm/overcommit_memory

The fact is that unused memory is stored in
caches and buffers. That's how it doesn't
get wasted. In fact, with a busy system
an attempt is made to use all available
real RAM and minimize swap.

Some server programs are written much like
your defective test program. You can't
attempt to allocate all virtual memory
using malloc(), expecting it to return
NULL when there is no more. Malloc() will
return NULL when it has gotten trashed
by somebody overwriting buffer boundaries
or when address-space is exhausted.

It has no clue how much RAM is available
and, in fact if it did, it would represent
an information leak. Programs should
allocate what they need, not what they
can get.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.12 on an i686 machine (5537.79 BogoMips).
Warning : 98.36% of all statistics are fiction.
.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

RE: Kernel doesn't free Cached Memory

[Al Boldi]

Dick Johnson wrote: { 
> On Fri, 2005-07-22 at 08:27 -0300, Vinicius wrote:
> [...]
>>    I have a server with 2 Pentium 4 HT processors and 32 GB of RAM, 
>> this server runs lots of applications that consume lots of memory to. 
>> When I stop this applications, the kernel doesn't free memory (the  
>> memory still in use) and the server cache lots of memory (~27GB). 
>> When I start this applications, the kernel sends  "Out of Memory" 
>> messages and kill some random applications.

...you might even need to turn memory over-commit off:
  echo "0" > /proc/sys/vm/overcommit_memory
}

That's in 2.4. In 2.6 it's:
  echo "2" > /proc/sys/vm/overcommit_memory

But the kernel doesn't honor no-overcommit in either version, i.e. it still
overcommits/pages-out loaded/running procs, thus invoking OOM!

Is there a way to make the kernel strictly honor the no-overcommit request?

Thanks!

Re: Kernel doesn't free Cached Memory

[Bill Davidsen]

Al Boldi wrote:
> Dick Johnson wrote: { 
> 
>>On Fri, 2005-07-22 at 08:27 -0300, Vinicius wrote:
>>[...]
>>
>>>   I have a server with 2 Pentium 4 HT processors and 32 GB of RAM, 
>>>this server runs lots of applications that consume lots of memory to. 
>>>When I stop this applications, the kernel doesn't free memory (the  
>>>memory still in use) and the server cache lots of memory (~27GB). 
>>>When I start this applications, the kernel sends  "Out of Memory" 
>>>messages and kill some random applications.
> 
> 
> ...you might even need to turn memory over-commit off:
>   echo "0" > /proc/sys/vm/overcommit_memory
> }
> 
> That's in 2.4. In 2.6 it's:
>   echo "2" > /proc/sys/vm/overcommit_memory

RHEL3 *is* a 2.4 kernel.
> 
> But the kernel doesn't honor no-overcommit in either version, i.e. it still
> overcommits/pages-out loaded/running procs, thus invoking OOM!
> 
> Is there a way to make the kernel strictly honor the no-overcommit request?
> 

Don't have swap?

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

RE: Kernel doesn't free Cached Memory

[Al Boldi]

Bill Davidsen wrote: {
Al Boldi wrote:
> Dick Johnson wrote: {
> 
>>On Fri, 2005-07-22 at 08:27 -0300, Vinicius wrote:
>>[...]
>>
>>>   I have a server with 2 Pentium 4 HT processors and 32 GB of RAM, 
>>>this server runs lots of applications that consume lots of memory to.
>>>When I stop this applications, the kernel doesn't free memory (the 
>>>memory still in use) and the server cache lots of memory (~27GB).
>>>When I start this applications, the kernel sends  "Out of Memory" 
>>>messages and kill some random applications.
> 
> 
> ...you might even need to turn memory over-commit off:
>   echo "0" > /proc/sys/vm/overcommit_memory
> }
> 
> That's in 2.4. In 2.6 it's:
>   echo "2" > /proc/sys/vm/overcommit_memory

RHEL3 *is* a 2.4 kernel.
> 
> But the kernel doesn't honor no-overcommit in either version, i.e. it
still
> overcommits/pages-out loaded/running procs, thus invoking OOM!
> 
> Is there a way to make the kernel strictly honor the no-overcommit
request?
> 

Don't have swap?
}

Turn off swap and things get worse!

Paolo Ornati wrote:{
Bill Davidsen <davidsen@tmr.com> wrote:
> And IMHO Linux is *way* too willing to evicy clean pages of my 
> programs to use as disk buffer, so that when system memory is full I 
> pay  the overhead of TWO disk i/o's, one to finally write the data to 
> the  disk and one to read my program back in. If free software is 
> about  choice, I wish there was more in the area of how memory is 
> used.

isn't this tuned enough by "/proc/sys/vm/swappiness" ?
}

Swappiness tunes but does not inhibit overcommit!

So the question remains:
	Why Is there no way to make the kernel _strictly_ honor the
no-overcommit request?

--
Al

Re: Kernel doesn't free Cached Memory

[Alan Cox]

On Gwe, 2005-07-22 at 08:27 -0300, Vinicius wrote:
> Hi all! 
> 
>    I have a server with 2 Pentium 4 HT processors and 32 GB of RAM, this 
> server runs lots of applications that consume lots of memory to. When I stop 
> this applications, the kernel doesn't free memory (the  memory still in use) 

See any FAQ on the Linux memory management - memory is reclaimed when
needed not when nobody is using it. That makes things more efficient.

> and the server cache lots of memory (~27GB). When I start this applications, 
> the kernel sends  "Out of Memory" messages and kill some random 
> applications. 

Some RHEL3 kernels had a problem with very large memory sizes and 2.4.
That should not be the case in the current RHEL3 kernels. 2.6 handles
very large systems a lot lot better, and of course the fact real
computers now have 64bit processors has also rather improved life.

Alan

 

Re: Kernel doesn't free Cached Memory

[Vinicius]

[-- Attachment #1: Mail message body --]
[-- Type: text/plain, Size: 1378 bytes --]




Em (15:49:49), Alan Cox escreveu: 


>On Gwe, 2005-07-22 at 08:27 -0300, Vinicius wrote: 
>> Hi all! 
>> 
>> I have a server with 2 Pentium 4 HT processors and 32 GB of RAM, this 
>> server runs lots of applications that consume lots of memory to. When I 
>stop 
>> this applications, the kernel doesn't free memory (the memory still in 
>use) 
> 
>See any FAQ on the Linux memory management - memory is reclaimed when 
>needed not when nobody is using it. That makes things more efficient. 
> 
>> and the server cache lots of memory (~27GB). When I start this 
>applications, 
>> the kernel sends "Out of Memory" messages and kill some random 
>> applications. 
> 
>Some RHEL3 kernels had a problem with very large memory sizes and 2.4. 
>That should not be the case in the current RHEL3 kernels. 2.6 handles 
>very large systems a lot lot better, and of course the fact real 
>computers now have 64bit processors has also rather improved life. 
> 
>Alan 
> 
>---------- 

Thanks Alan, 

   I also read on the Linux-Kernel that the problem may be related to an 
exhaustion of your kernels address space, I read that the hugemem-kernel 
might be the solution to this case since it has 4GB for the kernel memory 
plus 4GB for user process. 
How can I define if my kernel memory is beeing exhausted? Does this 
exhaustion of kernel memory can cause Out Of memory errors ? 

Re: Kernel doesn't free Cached Memory

[Alan Cox]

On Gwe, 2005-07-22 at 13:00 -0300, Vinicius wrote:
>    I also read on the Linux-Kernel that the problem may be related to an 
> exhaustion of your kernels address space, I read that the hugemem-kernel 
> might be the solution to this case since it has 4GB for the kernel memory 
> plus 4GB for user process. 

If its x86-32 then only the hugemem kernel will even see the memory.
There are big problems with 32Gb+ on a 32bit processor because there is
so little memory usable at a time that even the page tables become
problematic. Thankfully all sane machines with that much ram are 64bit.