Re: Memory Management

Re: Memory Management

[Márcio Oliveira]

Arjan van de Ven wrote:

>On Wed, 2005-07-20 at 11:23 -0300, Márcio Oliveira wrote:
>  
>
>>Arjan van de Ven wrote:
>>
>>    
>>
>>>I'm sure RH support will be able to help you with that; I doubt many
>>>other people care about an ancient kernel like that, and a vendor one to
>>>boot.
>>>
>>>(Also I assume you are using the -hugemem kernel as the documentation
>>>recommends you to do)
>>>
>>> 
>>>
>>>      
>>>
>>Arjan,
>>
>>   I'd like to know/understand more about memory management  on  Linux 
>>Kernel and I belive this concept is applyable to the Red Hat Linux Kernel.
>>    
>>
>
>Only on the highest of levels. The RHEL3 kernel has a VM that resembles
>almost no other linux kernel in many many ways. 
>
>
>  
>
>>  I have some doubts about the ZONE divison (DMA, NORMAL, HIGHMEM), 
>>Shared Memory utilization, HugeTLB feature and OOM with large memory and 
>>the kernel management of memory on SMP machines. I believe these 
>>features are common to the Linux kernel in general(Red Hat, Debian, 
>>SuSe, kernel.org), right?
>>    
>>
>
>nope. These things are very much different between the kernels you
>mention.
>
>What do you want to use the knowledge for? Fixing the VM? Tuning your
>server? The goal of your question determines what kind of answer you
>want to your questions....
>  
>
It's about tunning the VM parameters...

That's my first question on list:

"Is HugeTBL proc memory parameters only to hugetlbfs "filesystem" or are 
these parameters affect ramfs, shm and tmpfs too?

What is the basic difference between ramfs, hugetlbfs, shm and tmpfs to 
the memory management / process VLM utilization?

thanks,

Márcio."

Re: Memory Management

[Neil Horman]

On Thu, Jul 21, 2005 at 09:34:14AM -0300, Márcio Oliveira wrote:
> Arjan van de Ven wrote:
> 
> >On Wed, 2005-07-20 at 11:23 -0300, Márcio Oliveira wrote:
> > 
> >
> >>Arjan van de Ven wrote:
> >>
> >>   
> >>
> >>>I'm sure RH support will be able to help you with that; I doubt many
> >>>other people care about an ancient kernel like that, and a vendor one to
> >>>boot.
> >>>
> >>>(Also I assume you are using the -hugemem kernel as the documentation
> >>>recommends you to do)
> >>>
> >>>
> >>>
> >>>     
> >>>
> >>Arjan,
> >>
> >>  I'd like to know/understand more about memory management  on  Linux 
> >>Kernel and I belive this concept is applyable to the Red Hat Linux Kernel.
> >>   
> >>
> >
> >Only on the highest of levels. The RHEL3 kernel has a VM that resembles
> >almost no other linux kernel in many many ways. 
> >
> >
> > 
> >
> >> I have some doubts about the ZONE divison (DMA, NORMAL, HIGHMEM), 
> >>Shared Memory utilization, HugeTLB feature and OOM with large memory and 
> >>the kernel management of memory on SMP machines. I believe these 
> >>features are common to the Linux kernel in general(Red Hat, Debian, 
> >>SuSe, kernel.org), right?
> >>   
> >>
> >
> >nope. These things are very much different between the kernels you
> >mention.
> >
> >What do you want to use the knowledge for? Fixing the VM? Tuning your
> >server? The goal of your question determines what kind of answer you
> >want to your questions....
> > 
> >
> It's about tunning the VM parameters...
> 
> That's my first question on list:
> 
> "Is HugeTBL proc memory parameters only to hugetlbfs "filesystem" or are 
> these parameters affect ramfs, shm and tmpfs too?
> 
> What is the basic difference between ramfs, hugetlbfs, shm and tmpfs to 
> the memory management / process VLM utilization?
> 
> thanks,
> 
http://people.redhat.com/nhorman/papers/rhel3_vm.pdf
I wrote this with norm awhile back.  It may help you out.
Regards
Neil

> Márcio."
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
/***************************************************
 *Neil Horman
 *Software Engineer
 *Red Hat, Inc.
 *nhorman@redhat.com
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/

Re: Memory Management

[Márcio Oliveira]

>http://people.redhat.com/nhorman/papers/rhel3_vm.pdf
>I wrote this with norm awhile back.  It may help you out.
>Regards
>Neil
>  
>
Neil,

   Thanks.~10-12GB of total RAM (16GB) are

   How can Proc virtual memory parameters like inactive_clean_percent, 
overcommit_memory, overcommit_ratio and page_cache help me to solve / 
reduce Out Of Memory conditions on servers with 16GB RAM and lots of GB 
swap?

   Kernel does not free cached memory (~10-12GB of total RAM - 16GB). Is 
there some way to force the kernel to free cached memory?

/proc/meminfo:

              total:    used:    free:  shared: buffers:  cached:
Mem:    16603488256 16523333632 80154624        0 70651904 13194563584
Swap:   17174257664 11771904 17162485760
MemTotal:     16214344 kB
MemFree:         78276 kB
Buffers:         68996 kB
Cached:       12874808 kB

Thanks to all.

Marcio.

Re: Memory Management

[Neil Horman]

On Thu, Jul 21, 2005 at 10:40:54AM -0300, Márcio Oliveira wrote:
> 
> >http://people.redhat.com/nhorman/papers/rhel3_vm.pdf
> >I wrote this with norm awhile back.  It may help you out.
> >Regards
> >Neil
> > 
> >
> Neil,
> 
>   Thanks.~10-12GB of total RAM (16GB) are
> 
>   How can Proc virtual memory parameters like inactive_clean_percent, 
> overcommit_memory, overcommit_ratio and page_cache help me to solve / 
> reduce Out Of Memory conditions on servers with 16GB RAM and lots of GB 
> swap?
> 
I wouldn't touch memory overcommit if you are already seeing out of memory
issues.  If you are using lots of pagecache, I would suggest increasing
inactive_clean percent, reducing the pagecahce.max value, and modifying the
bdflush parameters in the above document such that bdflush runs sooner, more
often, and does more work per iteration.  This will help you move data in
pagecache back to disk more aggressively so that memory will be available for
other purposes, like heap allocations. Also if you're using a Red Hat kernel and
you have 16GB of ram in your system, you're a good candidate for the hugemem
kernel.  Rather than a straightforward out of memory condition, you may be
seeing a exhaustion of your kernels address space (check LowFree in
/proc/meminfo).  In this even the hugemem kernel will help you in that it
increases your Low Memory address space from 1GB to 4GB, preventing some OOM
conditions.


>   Kernel does not free cached memory (~10-12GB of total RAM - 16GB). Is 
> there some way to force the kernel to free cached memory?
> 
Cached memory is freed on demand.  Just because its listed under the cached line
below doesn't mean it can't be freed and used for another purpose.  Implement
the tunings above, and your situation should improve.

Regards
Neil

> /proc/meminfo:
> 
>              total:    used:    free:  shared: buffers:  cached:
> Mem:    16603488256 16523333632 80154624        0 70651904 13194563584
> Swap:   17174257664 11771904 17162485760
> MemTotal:     16214344 kB
> MemFree:         78276 kB
> Buffers:         68996 kB
> Cached:       12874808 kB
> 
> Thanks to all.
> 
> Marcio.
> 
> 

-- 
/***************************************************
 *Neil Horman
 *Software Engineer
 *Red Hat, Inc.
 *nhorman@redhat.com
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/

Re: Memory Management

[Márcio Oliveira]

Neil Horman wrote:

>On Thu, Jul 21, 2005 at 10:40:54AM -0300, Márcio Oliveira wrote:
>  
>
>>>http://people.redhat.com/nhorman/papers/rhel3_vm.pdf
>>>I wrote this with norm awhile back.  It may help you out.
>>>Regards
>>>Neil
>>>
>>>
>>>      
>>>
>>Neil,
>>
>>  Thanks.~10-12GB of total RAM (16GB) are
>>
>>  How can Proc virtual memory parameters like inactive_clean_percent, 
>>overcommit_memory, overcommit_ratio and page_cache help me to solve / 
>>reduce Out Of Memory conditions on servers with 16GB RAM and lots of GB 
>>swap?
>>
>>    
>>
>I wouldn't touch memory overcommit if you are already seeing out of memory
>issues.  If you are using lots of pagecache, I would suggest increasing
>inactive_clean percent, reducing the pagecahce.max value, and modifying the
>bdflush parameters in the above document such that bdflush runs sooner, more
>often, and does more work per iteration.  This will help you move data in
>pagecache back to disk more aggressively so that memory will be available for
>other purposes, like heap allocations. Also if you're using a Red Hat kernel and
>you have 16GB of ram in your system, you're a good candidate for the hugemem
>kernel.  Rather than a straightforward out of memory condition, you may be
>seeing a exhaustion of your kernels address space (check LowFree in
>/proc/meminfo).  In this even the hugemem kernel will help you in that it
>increases your Low Memory address space from 1GB to 4GB, preventing some OOM
>conditions.
>
>
>  
>
>>  Kernel does not free cached memory (~10-12GB of total RAM - 16GB). Is 
>>there some way to force the kernel to free cached memory?
>>
>>    
>>
>Cached memory is freed on demand.  Just because its listed under the cached line
>below doesn't mean it can't be freed and used for another purpose.  Implement
>the tunings above, and your situation should improve.
>
>Regards
>Neil
>
>  
>
>>/proc/meminfo:
>>
>>             total:    used:    free:  shared: buffers:  cached:
>>Mem:    16603488256 16523333632 80154624        0 70651904 13194563584
>>Swap:   17174257664 11771904 17162485760
>>MemTotal:     16214344 kB
>>MemFree:         78276 kB
>>Buffers:         68996 kB
>>Cached:       12874808 kB
>>
>>Thanks to all.
>>
>>Marcio.
>>    
>>

Neil,

   Thanks for the answers!

The following lines are the Out Of Memory log:

Jul 20 13:45:44 server kernel: Out of Memory: Killed process 23716 (oracle).
Jul 20 13:45:44 server kernel: Fixed up OOM kill of mm-less task
Jul 20 13:45:45 server su(pam_unix)[3848]: session closed for user root
Jul 20 13:45:48 server kernel: Mem-info:
Jul 20 13:45:48 server kernel: Zone:DMA freepages:  1884 min:     0 
low:     0 high:     0
Jul 20 13:45:48 server kernel: Zone:Normal freepages:  1084 min:  1279 
low:  4544 high:  6304
Jul 20 13:45:48 server kernel: Zone:HighMem freepages:386679 min:   255 
low: 61952 high: 92928
Jul 20 13:45:48 server kernel: Free pages:      389647 (386679 HighMem)
Jul 20 13:45:48 server kernel: ( Active: 2259787/488777, 
inactive_laundry: 244282, inactive_clean: 244366, free: 389647 )
Jul 20 13:45:48 server kernel:   aa:0 ac:0 id:0 il:0 ic:0 fr:1884
Jul 20 13:45:48 server kernel:   aa:1620 ac:1801 id:231 il:15 ic:0 fr:1085
Jul 20 13:45:48 server kernel:   aa:1099230 ac:1157136 id:488536 
il:244277 ic:244366 fr:386679
Jul 20 13:45:48 server kernel: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 
1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7536kB)Jul 20 13:45:48 
server kernel: 55*4kB 9*8kB 19*16kB 9*32kB 0*64kB 1*128kB 1*256kB 
0*512kB 1*1024kB 1*2048kB 0*4096kB = 4340kB)
Jul 20 13:45:48 server kernel: 291229*4kB 46179*8kB 711*16kB 1*32kB 
1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1546716kB)
Jul 20 13:45:48 server kernel: Swap cache: add 192990, delete 189665, 
find 21145/90719, race 0+0
Jul 20 13:45:48 server kernel: 139345 pages of slabcache
Jul 20 13:45:48 server kernel: 1890 pages of kernel stacks
Jul 20 13:45:48 server kernel: 0 lowmem pagetables, 274854 highmem 
pagetables
Jul 20 13:45:48 server kernel: Free swap:       16749720kB
Jul 20 13:45:49 server kernel: 4194304 pages of RAM
Jul 20 13:45:49 server kernel: 3899360 pages of HIGHMEM
Jul 20 13:45:49 server kernel: 140718 reserved pages
Jul 20 13:45:49 server kernel: 35350398 pages shared
Jul 20 13:45:49 server kernel: 3325 pages swap cached

/proc/meminfo LowFree info:
LowFree:         17068 kB    ------> Do you think this value is too low?

Zone:Normal freepages:  1084 min:  1279 low:  4544 high:  6304   ---->  
(freepages < min) It's normal?
Zone:HighMem freepages:386679 min:   255 low: 61952 high: 92928  ----> 
(freepages < min) It's normal?

Thanks a lot Neil!

Márcio Oliveira.

Re: Memory Management

[Neil Horman]

On Fri, Jul 22, 2005 at 11:32:52AM -0300, Márcio Oliveira wrote:
> Neil Horman wrote:
> 
> >On Thu, Jul 21, 2005 at 10:40:54AM -0300, Márcio Oliveira wrote:
> > 
> >
> >>>http://people.redhat.com/nhorman/papers/rhel3_vm.pdf
> >>>I wrote this with norm awhile back.  It may help you out.
> >>>Regards
> >>>Neil
> >>>
> >>>
> >>>     
> >>>
> >>Neil,
> >>
> >> Thanks.~10-12GB of total RAM (16GB) are
> >>
> >> How can Proc virtual memory parameters like inactive_clean_percent, 
> >>overcommit_memory, overcommit_ratio and page_cache help me to solve / 
> >>reduce Out Of Memory conditions on servers with 16GB RAM and lots of GB 
> >>swap?
> >>
> >>   
> >>
> >I wouldn't touch memory overcommit if you are already seeing out of memory
> >issues.  If you are using lots of pagecache, I would suggest increasing
> >inactive_clean percent, reducing the pagecahce.max value, and modifying the
> >bdflush parameters in the above document such that bdflush runs sooner, 
> >more
> >often, and does more work per iteration.  This will help you move data in
> >pagecache back to disk more aggressively so that memory will be available 
> >for
> >other purposes, like heap allocations. Also if you're using a Red Hat 
> >kernel and
> >you have 16GB of ram in your system, you're a good candidate for the 
> >hugemem
> >kernel.  Rather than a straightforward out of memory condition, you may be
> >seeing a exhaustion of your kernels address space (check LowFree in
> >/proc/meminfo).  In this even the hugemem kernel will help you in that it
> >increases your Low Memory address space from 1GB to 4GB, preventing some 
> >OOM
> >conditions.
> >
> >
> > 
> >
> >> Kernel does not free cached memory (~10-12GB of total RAM - 16GB). Is 
> >>there some way to force the kernel to free cached memory?
> >>
> >>   
> >>
> >Cached memory is freed on demand.  Just because its listed under the 
> >cached line
> >below doesn't mean it can't be freed and used for another purpose.  
> >Implement
> >the tunings above, and your situation should improve.
> >
> >Regards
> >Neil
> >
> > 
> >
> >>/proc/meminfo:
> >>
> >>            total:    used:    free:  shared: buffers:  cached:
> >>Mem:    16603488256 16523333632 80154624        0 70651904 13194563584
> >>Swap:   17174257664 11771904 17162485760
> >>MemTotal:     16214344 kB
> >>MemFree:         78276 kB
> >>Buffers:         68996 kB
> >>Cached:       12874808 kB
> >>
> >>Thanks to all.
> >>
> >>Marcio.
> >>   
> >>
> 
> Neil,
> 
>   Thanks for the answers!
> 
> The following lines are the Out Of Memory log:
> 
> Jul 20 13:45:44 server kernel: Out of Memory: Killed process 23716 (oracle).
> Jul 20 13:45:44 server kernel: Fixed up OOM kill of mm-less task
> Jul 20 13:45:45 server su(pam_unix)[3848]: session closed for user root
> Jul 20 13:45:48 server kernel: Mem-info:
> Jul 20 13:45:48 server kernel: Zone:DMA freepages:  1884 min:     0 
> low:     0 high:     0
> Jul 20 13:45:48 server kernel: Zone:Normal freepages:  1084 min:  1279 
> low:  4544 high:  6304
> Jul 20 13:45:48 server kernel: Zone:HighMem freepages:386679 min:   255 
> low: 61952 high: 92928
> Jul 20 13:45:48 server kernel: Free pages:      389647 (386679 HighMem)
> Jul 20 13:45:48 server kernel: ( Active: 2259787/488777, 
> inactive_laundry: 244282, inactive_clean: 244366, free: 389647 )
> Jul 20 13:45:48 server kernel:   aa:0 ac:0 id:0 il:0 ic:0 fr:1884
> Jul 20 13:45:48 server kernel:   aa:1620 ac:1801 id:231 il:15 ic:0 fr:1085
> Jul 20 13:45:48 server kernel:   aa:1099230 ac:1157136 id:488536 
> il:244277 ic:244366 fr:386679
> Jul 20 13:45:48 server kernel: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 
> 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7536kB)Jul 20 13:45:48 
> server kernel: 55*4kB 9*8kB 19*16kB 9*32kB 0*64kB 1*128kB 1*256kB 
> 0*512kB 1*1024kB 1*2048kB 0*4096kB = 4340kB)
> Jul 20 13:45:48 server kernel: 291229*4kB 46179*8kB 711*16kB 1*32kB 
> 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1546716kB)
> Jul 20 13:45:48 server kernel: Swap cache: add 192990, delete 189665, 
> find 21145/90719, race 0+0
> Jul 20 13:45:48 server kernel: 139345 pages of slabcache
> Jul 20 13:45:48 server kernel: 1890 pages of kernel stacks
> Jul 20 13:45:48 server kernel: 0 lowmem pagetables, 274854 highmem 
> pagetables
> Jul 20 13:45:48 server kernel: Free swap:       16749720kB
> Jul 20 13:45:49 server kernel: 4194304 pages of RAM
> Jul 20 13:45:49 server kernel: 3899360 pages of HIGHMEM
> Jul 20 13:45:49 server kernel: 140718 reserved pages
> Jul 20 13:45:49 server kernel: 35350398 pages shared
> Jul 20 13:45:49 server kernel: 3325 pages swap cached
> 
> /proc/meminfo LowFree info:
> LowFree:         17068 kB    ------> Do you think this value is too low?
> 
No that should be plenty of lowFree, but that number can change quickly
depending on workload.

> Zone:Normal freepages:  1084 min:  1279 low:  4544 high:  6304   ---->  
> (freepages < min) It's normal?
> Zone:HighMem freepages:386679 min:   255 low: 61952 high: 92928  ----> 
> (freepages < min) It's normal?
> 
You're beneath your low water mark in the normal (lowmem) zone for free pages,
so your kernel is likely trying to get lots of data moved to disk.  Although
given that you're largest buddy list has a 2048K chunk free, I'm hard pressed to
see how you aren't able to get memory when you need it.  Do you have a module
loaded in your kernel that might require such large memory allocations.
Neil

> Thanks a lot Neil!
> 
> Márcio Oliveira.
> 
> 

-- 
/***************************************************
 *Neil Horman
 *Software Engineer
 *Red Hat, Inc.
 *nhorman@redhat.com
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/

Re: Memory Management

[Márcio Oliveira]

Neil Horman wrote:

>On Fri, Jul 22, 2005 at 11:32:52AM -0300, Márcio Oliveira wrote:
>  
>
>>Neil Horman wrote:
>>
>>    
>>
>>>On Thu, Jul 21, 2005 at 10:40:54AM -0300, Márcio Oliveira wrote:
>>>
>>>
>>>      
>>>
>>>>>http://people.redhat.com/nhorman/papers/rhel3_vm.pdf
>>>>>I wrote this with norm awhile back.  It may help you out.
>>>>>Regards
>>>>>Neil
>>>>>
>>>>>
>>>>>    
>>>>>
>>>>>          
>>>>>
>>>>Neil,
>>>>
>>>>Thanks.~10-12GB of total RAM (16GB) are
>>>>
>>>>How can Proc virtual memory parameters like inactive_clean_percent, 
>>>>overcommit_memory, overcommit_ratio and page_cache help me to solve / 
>>>>reduce Out Of Memory conditions on servers with 16GB RAM and lots of GB 
>>>>swap?
>>>>
>>>>  
>>>>
>>>>        
>>>>
>>>I wouldn't touch memory overcommit if you are already seeing out of memory
>>>issues.  If you are using lots of pagecache, I would suggest increasing
>>>inactive_clean percent, reducing the pagecahce.max value, and modifying the
>>>bdflush parameters in the above document such that bdflush runs sooner, 
>>>more
>>>often, and does more work per iteration.  This will help you move data in
>>>pagecache back to disk more aggressively so that memory will be available 
>>>for
>>>other purposes, like heap allocations. Also if you're using a Red Hat 
>>>kernel and
>>>you have 16GB of ram in your system, you're a good candidate for the 
>>>hugemem
>>>kernel.  Rather than a straightforward out of memory condition, you may be
>>>seeing a exhaustion of your kernels address space (check LowFree in
>>>/proc/meminfo).  In this even the hugemem kernel will help you in that it
>>>increases your Low Memory address space from 1GB to 4GB, preventing some 
>>>OOM
>>>conditions.
>>>
>>>
>>>
>>>
>>>      
>>>
>>>>Kernel does not free cached memory (~10-12GB of total RAM - 16GB). Is 
>>>>there some way to force the kernel to free cached memory?
>>>>
>>>>  
>>>>
>>>>        
>>>>
>>>Cached memory is freed on demand.  Just because its listed under the 
>>>cached line
>>>below doesn't mean it can't be freed and used for another purpose.  
>>>Implement
>>>the tunings above, and your situation should improve.
>>>
>>>Regards
>>>Neil
>>>
>>>
>>>
>>>      
>>>
>>>>/proc/meminfo:
>>>>
>>>>           total:    used:    free:  shared: buffers:  cached:
>>>>Mem:    16603488256 16523333632 80154624        0 70651904 13194563584
>>>>Swap:   17174257664 11771904 17162485760
>>>>MemTotal:     16214344 kB
>>>>MemFree:         78276 kB
>>>>Buffers:         68996 kB
>>>>Cached:       12874808 kB
>>>>
>>>>Thanks to all.
>>>>
>>>>Marcio.
>>>>  
>>>>
>>>>        
>>>>
>>Neil,
>>
>>  Thanks for the answers!
>>
>>The following lines are the Out Of Memory log:
>>
>>Jul 20 13:45:44 server kernel: Out of Memory: Killed process 23716 (oracle).
>>Jul 20 13:45:44 server kernel: Fixed up OOM kill of mm-less task
>>Jul 20 13:45:45 server su(pam_unix)[3848]: session closed for user root
>>Jul 20 13:45:48 server kernel: Mem-info:
>>Jul 20 13:45:48 server kernel: Zone:DMA freepages:  1884 min:     0 
>>low:     0 high:     0
>>Jul 20 13:45:48 server kernel: Zone:Normal freepages:  1084 min:  1279 
>>low:  4544 high:  6304
>>Jul 20 13:45:48 server kernel: Zone:HighMem freepages:386679 min:   255 
>>low: 61952 high: 92928
>>Jul 20 13:45:48 server kernel: Free pages:      389647 (386679 HighMem)
>>Jul 20 13:45:48 server kernel: ( Active: 2259787/488777, 
>>inactive_laundry: 244282, inactive_clean: 244366, free: 389647 )
>>Jul 20 13:45:48 server kernel:   aa:0 ac:0 id:0 il:0 ic:0 fr:1884
>>Jul 20 13:45:48 server kernel:   aa:1620 ac:1801 id:231 il:15 ic:0 fr:1085
>>Jul 20 13:45:48 server kernel:   aa:1099230 ac:1157136 id:488536 
>>il:244277 ic:244366 fr:386679
>>Jul 20 13:45:48 server kernel: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 
>>1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7536kB)Jul 20 13:45:48 
>>server kernel: 55*4kB 9*8kB 19*16kB 9*32kB 0*64kB 1*128kB 1*256kB 
>>0*512kB 1*1024kB 1*2048kB 0*4096kB = 4340kB)
>>Jul 20 13:45:48 server kernel: 291229*4kB 46179*8kB 711*16kB 1*32kB 
>>1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1546716kB)
>>Jul 20 13:45:48 server kernel: Swap cache: add 192990, delete 189665, 
>>find 21145/90719, race 0+0
>>Jul 20 13:45:48 server kernel: 139345 pages of slabcache
>>Jul 20 13:45:48 server kernel: 1890 pages of kernel stacks
>>Jul 20 13:45:48 server kernel: 0 lowmem pagetables, 274854 highmem 
>>pagetables
>>Jul 20 13:45:48 server kernel: Free swap:       16749720kB
>>Jul 20 13:45:49 server kernel: 4194304 pages of RAM
>>Jul 20 13:45:49 server kernel: 3899360 pages of HIGHMEM
>>Jul 20 13:45:49 server kernel: 140718 reserved pages
>>Jul 20 13:45:49 server kernel: 35350398 pages shared
>>Jul 20 13:45:49 server kernel: 3325 pages swap cached
>>
>>/proc/meminfo LowFree info:
>>LowFree:         17068 kB    ------> Do you think this value is too low?
>>
>>    
>>
>No that should be plenty of lowFree, but that number can change quickly
>depending on workload.
>
>  
>
>>Zone:Normal freepages:  1084 min:  1279 low:  4544 high:  6304   ---->  
>>(freepages < min) It's normal?
>>Zone:HighMem freepages:386679 min:   255 low: 61952 high: 92928  ----> 
>>(freepages < min) It's normal?
>>
>>    
>>
>You're beneath your low water mark in the normal (lowmem) zone for free pages,
>so your kernel is likely trying to get lots of data moved to disk.  Although
>given that you're largest buddy list has a 2048K chunk free, I'm hard pressed to
>see how you aren't able to get memory when you need it.  Do you have a module
>loaded in your kernel that might require such large memory allocations.
>Neil
>
>  
>
>>Thanks a lot Neil!
>>
>>Márcio Oliveira.
>>
>>
>>    
>>
>
>  
>
Neil,

   Thanks for the help.

   I have a storage attached to the server. Maybe the storage module 
require lots of memory.
   Maybe the "LowFree" be wrong (out of OOM time), so there is possible 
that "LowFree" value be too small on the OOM condition.

  Is there a way to identify if the Low Memory is too small?  (some 
program, command, daemon...)

  The server has 16GB RAM and 16GB swap. When the OOM kill conditions 
happens, the system has ~6GB RAM used, ~10GB RAM cached and 16GB free 
swap. Is that indicate that the server can't allocate Low Memory and 
starts OOM conditions? Because the High Memory is OK, right?

Thanks again!

Márcio.

RE: Memory Management

[Roger Heflin]

I have seen RH3.0 crash on 32GB systems because it has too
much memory tied up in write cache.  It required update 2 
(this was a while ago) and a change of a parameter in /proc
to prevent the crash, it was because of a overagressive
write caching change RH implemented in the kernel resulted
in the crash.  This crash was an OOM related crash.  To
duplicate the bug, you booted the machine and ran a dd
to create a very large file filling the disk.

We did test and did determine that it did not appear to have
the issue if you had less than 28GB of ram, this was on an
itanium machine, so I don't know if it occurs on other arches,
and if it occurs at the same memory limits on the other arches
either.

                        Roger

 

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org 
> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of 
> Márcio Oliveira
> Sent: Friday, July 22, 2005 2:42 PM
> To: Neil Horman
> Cc: arjanv@redhat.com; linux-kernel@vger.kernel.org
> Subject: Re: Memory Management
> 
> Neil Horman wrote:
> 
> >On Fri, Jul 22, 2005 at 11:32:52AM -0300, Márcio Oliveira wrote:
> >  
> >
> >>Neil Horman wrote:
> >>
> >>    
> >>
> >>>On Thu, Jul 21, 2005 at 10:40:54AM -0300, Márcio Oliveira wrote:
> >>>
> >>>
> >>>      
> >>>
> >>>>>http://people.redhat.com/nhorman/papers/rhel3_vm.pdf
> >>>>>I wrote this with norm awhile back.  It may help you out.
> >>>>>Regards
> >>>>>Neil
> >>>>>
> >>>>>
> >>>>>    
> >>>>>
> >>>>>          
> >>>>>
> >>>>Neil,
> >>>>
> >>>>Thanks.~10-12GB of total RAM (16GB) are
> >>>>
> >>>>How can Proc virtual memory parameters like 
> inactive_clean_percent, 
> >>>>overcommit_memory, overcommit_ratio and page_cache help 
> me to solve 
> >>>>/ reduce Out Of Memory conditions on servers with 16GB 
> RAM and lots 
> >>>>of GB swap?
> >>>>
> >>>>  
> >>>>
> >>>>        
> >>>>
> >>>I wouldn't touch memory overcommit if you are already 
> seeing out of 
> >>>memory issues.  If you are using lots of pagecache, I 
> would suggest 
> >>>increasing inactive_clean percent, reducing the 
> pagecahce.max value, 
> >>>and modifying the bdflush parameters in the above document 
> such that 
> >>>bdflush runs sooner, more often, and does more work per 
> iteration.  
> >>>This will help you move data in pagecache back to disk more 
> >>>aggressively so that memory will be available for other purposes, 
> >>>like heap allocations. Also if you're using a Red Hat 
> kernel and you 
> >>>have 16GB of ram in your system, you're a good candidate for the 
> >>>hugemem kernel.  Rather than a straightforward out of memory 
> >>>condition, you may be seeing a exhaustion of your kernels address 
> >>>space (check LowFree in /proc/meminfo).  In this even the hugemem 
> >>>kernel will help you in that it increases your Low Memory address 
> >>>space from 1GB to 4GB, preventing some OOM conditions.
> >>>
> >>>
> >>>
> >>>
> >>>      
> >>>
> >>>>Kernel does not free cached memory (~10-12GB of total RAM 
> - 16GB). Is 
> >>>>there some way to force the kernel to free cached memory?
> >>>>
> >>>>  
> >>>>
> >>>>        
> >>>>
> >>>Cached memory is freed on demand.  Just because its listed 
> under the 
> >>>cached line
> >>>below doesn't mean it can't be freed and used for another 
> purpose.  
> >>>Implement
> >>>the tunings above, and your situation should improve.
> >>>
> >>>Regards
> >>>Neil
> >>>
> >>>
> >>>
> >>>      
> >>>
> >>>>/proc/meminfo:
> >>>>
> >>>>           total:    used:    free:  shared: buffers:  cached:
> >>>>Mem:    16603488256 16523333632 80154624        0 
> 70651904 13194563584
> >>>>Swap:   17174257664 11771904 17162485760
> >>>>MemTotal:     16214344 kB
> >>>>MemFree:         78276 kB
> >>>>Buffers:         68996 kB
> >>>>Cached:       12874808 kB
> >>>>
> >>>>Thanks to all.
> >>>>
> >>>>Marcio.
> >>>>  
> >>>>
> >>>>        
> >>>>
> >>Neil,
> >>
> >>  Thanks for the answers!
> >>
> >>The following lines are the Out Of Memory log:
> >>
> >>Jul 20 13:45:44 server kernel: Out of Memory: Killed 
> process 23716 (oracle).
> >>Jul 20 13:45:44 server kernel: Fixed up OOM kill of mm-less task
> >>Jul 20 13:45:45 server su(pam_unix)[3848]: session closed 
> for user root
> >>Jul 20 13:45:48 server kernel: Mem-info:
> >>Jul 20 13:45:48 server kernel: Zone:DMA freepages:  1884 min:     0 
> >>low:     0 high:     0
> >>Jul 20 13:45:48 server kernel: Zone:Normal freepages:  1084 
> min:  1279 
> >>low:  4544 high:  6304
> >>Jul 20 13:45:48 server kernel: Zone:HighMem 
> freepages:386679 min:   255 
> >>low: 61952 high: 92928
> >>Jul 20 13:45:48 server kernel: Free pages:      389647 
> (386679 HighMem)
> >>Jul 20 13:45:48 server kernel: ( Active: 2259787/488777, 
> >>inactive_laundry: 244282, inactive_clean: 244366, free: 389647 )
> >>Jul 20 13:45:48 server kernel:   aa:0 ac:0 id:0 il:0 ic:0 fr:1884
> >>Jul 20 13:45:48 server kernel:   aa:1620 ac:1801 id:231 
> il:15 ic:0 fr:1085
> >>Jul 20 13:45:48 server kernel:   aa:1099230 ac:1157136 id:488536 
> >>il:244277 ic:244366 fr:386679
> >>Jul 20 13:45:48 server kernel: 0*4kB 0*8kB 1*16kB 1*32kB 
> 1*64kB 0*128kB 
> >>1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7536kB)Jul 20 13:45:48 
> >>server kernel: 55*4kB 9*8kB 19*16kB 9*32kB 0*64kB 1*128kB 1*256kB 
> >>0*512kB 1*1024kB 1*2048kB 0*4096kB = 4340kB)
> >>Jul 20 13:45:48 server kernel: 291229*4kB 46179*8kB 711*16kB 1*32kB 
> >>1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 
> 1546716kB)
> >>Jul 20 13:45:48 server kernel: Swap cache: add 192990, 
> delete 189665, 
> >>find 21145/90719, race 0+0
> >>Jul 20 13:45:48 server kernel: 139345 pages of slabcache
> >>Jul 20 13:45:48 server kernel: 1890 pages of kernel stacks
> >>Jul 20 13:45:48 server kernel: 0 lowmem pagetables, 274854 highmem 
> >>pagetables
> >>Jul 20 13:45:48 server kernel: Free swap:       16749720kB
> >>Jul 20 13:45:49 server kernel: 4194304 pages of RAM
> >>Jul 20 13:45:49 server kernel: 3899360 pages of HIGHMEM
> >>Jul 20 13:45:49 server kernel: 140718 reserved pages
> >>Jul 20 13:45:49 server kernel: 35350398 pages shared
> >>Jul 20 13:45:49 server kernel: 3325 pages swap cached
> >>
> >>/proc/meminfo LowFree info:
> >>LowFree:         17068 kB    ------> Do you think this 
> value is too low?
> >>
> >>    
> >>
> >No that should be plenty of lowFree, but that number can 
> change quickly
> >depending on workload.
> >
> >  
> >
> >>Zone:Normal freepages:  1084 min:  1279 low:  4544 high:  
> 6304   ---->  
> >>(freepages < min) It's normal?
> >>Zone:HighMem freepages:386679 min:   255 low: 61952 high: 
> 92928  ----> 
> >>(freepages < min) It's normal?
> >>
> >>    
> >>
> >You're beneath your low water mark in the normal (lowmem) 
> zone for free pages,
> >so your kernel is likely trying to get lots of data moved to 
> disk.  Although
> >given that you're largest buddy list has a 2048K chunk free, 
> I'm hard pressed to
> >see how you aren't able to get memory when you need it.  Do 
> you have a module
> >loaded in your kernel that might require such large memory 
> allocations.
> >Neil
> >
> >  
> >
> >>Thanks a lot Neil!
> >>
> >>Márcio Oliveira.
> >>
> >>
> >>    
> >>
> >
> >  
> >
> Neil,
> 
>    Thanks for the help.
> 
>    I have a storage attached to the server. Maybe the storage module 
> require lots of memory.
>    Maybe the "LowFree" be wrong (out of OOM time), so there 
> is possible 
> that "LowFree" value be too small on the OOM condition.
> 
>   Is there a way to identify if the Low Memory is too small?  (some 
> program, command, daemon...)
> 
>   The server has 16GB RAM and 16GB swap. When the OOM kill conditions 
> happens, the system has ~6GB RAM used, ~10GB RAM cached and 16GB free 
> swap. Is that indicate that the server can't allocate Low Memory and 
> starts OOM conditions? Because the High Memory is OK, right?
> 
> Thanks again!
> 
> Márcio.
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Memory Management

[Márcio Oliveira]

Roger Heflin wrote:

>I have seen RH3.0 crash on 32GB systems because it has too
>much memory tied up in write cache.  It required update 2 
>(this was a while ago) and a change of a parameter in /proc
>to prevent the crash, it was because of a overagressive
>write caching change RH implemented in the kernel resulted
>in the crash.  This crash was an OOM related crash.  To
>duplicate the bug, you booted the machine and ran a dd
>to create a very large file filling the disk.
>
>We did test and did determine that it did not appear to have
>the issue if you had less than 28GB of ram, this was on an
>itanium machine, so I don't know if it occurs on other arches,
>and if it occurs at the same memory limits on the other arches
>either.
>
>                        Roger
>
> 
>
>  
>
>>-----Original Message-----
>>From: linux-kernel-owner@vger.kernel.org 
>>[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of 
>>Márcio Oliveira
>>Sent: Friday, July 22, 2005 2:42 PM
>>To: Neil Horman
>>Cc: arjanv@redhat.com; linux-kernel@vger.kernel.org
>>Subject: Re: Memory Management
>>
>>Neil Horman wrote:
>>
>>    
>>
>>>On Fri, Jul 22, 2005 at 11:32:52AM -0300, Márcio Oliveira wrote:
>>> 
>>>
>>>      
>>>
>>>>Neil Horman wrote:
>>>>
>>>>   
>>>>
>>>>        
>>>>
>>>>>On Thu, Jul 21, 2005 at 10:40:54AM -0300, Márcio Oliveira wrote:
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>>>>http://people.redhat.com/nhorman/papers/rhel3_vm.pdf
>>>>>>>I wrote this with norm awhile back.  It may help you out.
>>>>>>>Regards
>>>>>>>Neil
>>>>>>>
>>>>>>>
>>>>>>>   
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>Neil,
>>>>>>
>>>>>>Thanks.~10-12GB of total RAM (16GB) are
>>>>>>
>>>>>>How can Proc virtual memory parameters like 
>>>>>>            
>>>>>>
>>inactive_clean_percent, 
>>    
>>
>>>>>>overcommit_memory, overcommit_ratio and page_cache help 
>>>>>>            
>>>>>>
>>me to solve 
>>    
>>
>>>>>>/ reduce Out Of Memory conditions on servers with 16GB 
>>>>>>            
>>>>>>
>>RAM and lots 
>>    
>>
>>>>>>of GB swap?
>>>>>>
>>>>>> 
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>I wouldn't touch memory overcommit if you are already 
>>>>>          
>>>>>
>>seeing out of 
>>    
>>
>>>>>memory issues.  If you are using lots of pagecache, I 
>>>>>          
>>>>>
>>would suggest 
>>    
>>
>>>>>increasing inactive_clean percent, reducing the 
>>>>>          
>>>>>
>>pagecahce.max value, 
>>    
>>
>>>>>and modifying the bdflush parameters in the above document 
>>>>>          
>>>>>
>>such that 
>>    
>>
>>>>>bdflush runs sooner, more often, and does more work per 
>>>>>          
>>>>>
>>iteration.  
>>    
>>
>>>>>This will help you move data in pagecache back to disk more 
>>>>>aggressively so that memory will be available for other purposes, 
>>>>>like heap allocations. Also if you're using a Red Hat 
>>>>>          
>>>>>
>>kernel and you 
>>    
>>
>>>>>have 16GB of ram in your system, you're a good candidate for the 
>>>>>hugemem kernel.  Rather than a straightforward out of memory 
>>>>>condition, you may be seeing a exhaustion of your kernels address 
>>>>>space (check LowFree in /proc/meminfo).  In this even the hugemem 
>>>>>kernel will help you in that it increases your Low Memory address 
>>>>>space from 1GB to 4GB, preventing some OOM conditions.
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>Kernel does not free cached memory (~10-12GB of total RAM 
>>>>>>            
>>>>>>
>>- 16GB). Is 
>>    
>>
>>>>>>there some way to force the kernel to free cached memory?
>>>>>>            
>>>>>>
>>>>>Cached memory is freed on demand.  Just because its listed 
>>>>>          
>>>>>
>>under the 
>>    
>>
>>>>>cached line
>>>>>below doesn't mean it can't be freed and used for another 
>>>>>          
>>>>>
>>purpose.  
>>    
>>
>>>>>Implement
>>>>>the tunings above, and your situation should improve.
>>>>>
>>>>>Regards
>>>>>Neil
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>>>/proc/meminfo:
>>>>>>
>>>>>>          total:    used:    free:  shared: buffers:  cached:
>>>>>>Mem:    16603488256 16523333632 80154624        0 
>>>>>>            
>>>>>>
>>70651904 13194563584
>>    
>>
>>>>>>Swap:   17174257664 11771904 17162485760
>>>>>>MemTotal:     16214344 kB
>>>>>>MemFree:         78276 kB
>>>>>>Buffers:         68996 kB
>>>>>>Cached:       12874808 kB
>>>>>>
>>>>>>Thanks to all.
>>>>>>
>>>>>>Marcio.
>>>>>> 
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>Neil,
>>>>
>>>> Thanks for the answers!
>>>>
>>>>The following lines are the Out Of Memory log:
>>>>
>>>>Jul 20 13:45:44 server kernel: Out of Memory: Killed 
>>>>        
>>>>
>>process 23716 (oracle).
>>    
>>
>>>>Jul 20 13:45:44 server kernel: Fixed up OOM kill of mm-less task
>>>>Jul 20 13:45:45 server su(pam_unix)[3848]: session closed 
>>>>        
>>>>
>>for user root
>>    
>>
>>>>Jul 20 13:45:48 server kernel: Mem-info:
>>>>Jul 20 13:45:48 server kernel: Zone:DMA freepages:  1884 min:     0 
>>>>low:     0 high:     0
>>>>Jul 20 13:45:48 server kernel: Zone:Normal freepages:  1084 
>>>>        
>>>>
>>min:  1279 
>>    
>>
>>>>low:  4544 high:  6304
>>>>Jul 20 13:45:48 server kernel: Zone:HighMem 
>>>>        
>>>>
>>freepages:386679 min:   255 
>>    
>>
>>>>low: 61952 high: 92928
>>>>Jul 20 13:45:48 server kernel: Free pages:      389647 
>>>>        
>>>>
>>(386679 HighMem)
>>    
>>
>>>>Jul 20 13:45:48 server kernel: ( Active: 2259787/488777, 
>>>>inactive_laundry: 244282, inactive_clean: 244366, free: 389647 )
>>>>Jul 20 13:45:48 server kernel:   aa:0 ac:0 id:0 il:0 ic:0 fr:1884
>>>>Jul 20 13:45:48 server kernel:   aa:1620 ac:1801 id:231 
>>>>        
>>>>
>>il:15 ic:0 fr:1085
>>    
>>
>>>>Jul 20 13:45:48 server kernel:   aa:1099230 ac:1157136 id:488536 
>>>>il:244277 ic:244366 fr:386679
>>>>Jul 20 13:45:48 server kernel: 0*4kB 0*8kB 1*16kB 1*32kB 
>>>>        
>>>>
>>1*64kB 0*128kB 
>>    
>>
>>>>1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7536kB)Jul 20 13:45:48 
>>>>server kernel: 55*4kB 9*8kB 19*16kB 9*32kB 0*64kB 1*128kB 1*256kB 
>>>>0*512kB 1*1024kB 1*2048kB 0*4096kB = 4340kB)
>>>>Jul 20 13:45:48 server kernel: 291229*4kB 46179*8kB 711*16kB 1*32kB 
>>>>1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 
>>>>        
>>>>
>>1546716kB)
>>    
>>
>>>>Jul 20 13:45:48 server kernel: Swap cache: add 192990, 
>>>>        
>>>>
>>delete 189665, 
>>    
>>
>>>>find 21145/90719, race 0+0
>>>>Jul 20 13:45:48 server kernel: 139345 pages of slabcache
>>>>Jul 20 13:45:48 server kernel: 1890 pages of kernel stacks
>>>>Jul 20 13:45:48 server kernel: 0 lowmem pagetables, 274854 highmem 
>>>>pagetables
>>>>Jul 20 13:45:48 server kernel: Free swap:       16749720kB
>>>>Jul 20 13:45:49 server kernel: 4194304 pages of RAM
>>>>Jul 20 13:45:49 server kernel: 3899360 pages of HIGHMEM
>>>>Jul 20 13:45:49 server kernel: 140718 reserved pages
>>>>Jul 20 13:45:49 server kernel: 35350398 pages shared
>>>>Jul 20 13:45:49 server kernel: 3325 pages swap cached
>>>>
>>>>/proc/meminfo LowFree info:
>>>>LowFree:         17068 kB    ------> Do you think this 
>>>>        
>>>>
>>value is too low?
>>    
>>
>>>>   
>>>>
>>>>        
>>>>
>>>No that should be plenty of lowFree, but that number can 
>>>      
>>>
>>change quickly
>>    
>>
>>>depending on workload.
>>>
>>> 
>>>
>>>      
>>>
>>>>Zone:Normal freepages:  1084 min:  1279 low:  4544 high:  
>>>>        
>>>>
>>6304   ---->  
>>    
>>
>>>>(freepages < min) It's normal?
>>>>Zone:HighMem freepages:386679 min:   255 low: 61952 high: 
>>>>        
>>>>
>>92928  ----> 
>>    
>>
>>>>(freepages < min) It's normal?
>>>>
>>>>   
>>>>
>>>>        
>>>>
>>>You're beneath your low water mark in the normal (lowmem) 
>>>      
>>>
>>zone for free pages,
>>    
>>
>>>so your kernel is likely trying to get lots of data moved to 
>>>      
>>>
>>disk.  Although
>>    
>>
>>>given that you're largest buddy list has a 2048K chunk free, 
>>>      
>>>
>>I'm hard pressed to
>>    
>>
>>>see how you aren't able to get memory when you need it.  Do 
>>>      
>>>
>>you have a module
>>    
>>
>>>loaded in your kernel that might require such large memory 
>>>      
>>>
>>allocations.
>>    
>>
>>>Neil
>>>
>>> 
>>>
>>>      
>>>
>>>>Thanks a lot Neil!
>>>>
>>>>Márcio Oliveira.
>>>>
>>>>
>>>>   
>>>>
>>>>        
>>>>
>>> 
>>>
>>>      
>>>
>>Neil,
>>
>>   Thanks for the help.
>>
>>   I have a storage attached to the server. Maybe the storage module 
>>require lots of memory.
>>   Maybe the "LowFree" be wrong (out of OOM time), so there 
>>is possible 
>>that "LowFree" value be too small on the OOM condition.
>>
>>  Is there a way to identify if the Low Memory is too small?  (some 
>>program, command, daemon...)
>>
>>  The server has 16GB RAM and 16GB swap. When the OOM kill conditions 
>>happens, the system has ~6GB RAM used, ~10GB RAM cached and 16GB free 
>>swap. Is that indicate that the server can't allocate Low Memory and 
>>starts OOM conditions? Because the High Memory is OK, right?
>>
>>Thanks again!
>>
>>Márcio.
>>-
>>To unsubscribe from this list: send the line "unsubscribe 
>>linux-kernel" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>Please read the FAQ at  http://www.tux.org/lkml/
>>
>>    
>>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>  
>
Roger, thanks for the information.

   I'm using Update 4 kernels (2.4.21-27.ELsmp - This kernel have some 
mm / oom fixes) and don't have big problems when create large files, 
plus the server is a 32-bit machine. 

   Neil said that the problem can be Low Memory and I think it too.

   I read the following message on the list:

http://marc.theaimsgroup.com/?l=linux-kernel&m=112044530919567&w=4

   The problem seems like a I/O issue. Can I/O (like storage devices) 
consume a large amount of low memory? Neil also said that the kernel is 
trying to move lots of data to the disk and it's a module might require 
such large memory. Somebody know how can I identify what is using Low 
Memory on my system?

   The older questions in the message are:

 The server has 16GB RAM and 16GB swap. When the OOM kill conditions happens, the system has ~6GB RAM used, ~10GB RAM cached and 16GB free swap. Is that indicate that the server can't allocate Low Memory and starts OOM conditions? Because the High Memory is OK, right?

Thanks to all!

Regards,
Márcio

Re: Memory Management

[Neil Horman]

On Fri, Jul 22, 2005 at 08:23:19PM -0300, Márcio Oliveira wrote:
<snip>
> >
> Roger, thanks for the information.
> 
>   I'm using Update 4 kernels (2.4.21-27.ELsmp - This kernel have some 
> mm / oom fixes) and don't have big problems when create large files, 
> plus the server is a 32-bit machine. 
> 
>   Neil said that the problem can be Low Memory and I think it too.
> 
>   I read the following message on the list:
> 
> http://marc.theaimsgroup.com/?l=linux-kernel&m=112044530919567&w=4
> 
>   The problem seems like a I/O issue. Can I/O (like storage devices) 
> consume a large amount of low memory? Neil also said that the kernel is 
> trying to move lots of data to the disk and it's a module might require 
> such large memory. Somebody know how can I identify what is using Low 
> Memory on my system?
> 
The best way I can think to do that is take a look at /proc/slabinfo.  That will
likely give you a pointer to which area of code is eating up your memory.

>   The older questions in the message are:
> 
> The server has 16GB RAM and 16GB swap. When the OOM kill conditions 
> happens, the system has ~6GB RAM used, ~10GB RAM cached and 16GB free swap. 
> Is that indicate that the server can't allocate Low Memory and starts OOM 
> conditions? Because the High Memory is OK, right?
> 
Based on the sysrq-m info you posted it looks like due to fragmentation the
largest chunk of memory you can allocate is 2MB (perhaps less depending on
address space availability).  If you can build a test kernel to do a show_state
rather than a show_mem at the beginning of oom_kil, then you should be able to
tell who is trying to do an allocation that leads to kswapd calling
out_of_memory.

> Thanks to all!
> 
> Regards,
> Márcio
> 
> 

-- 
/***************************************************
 *Neil Horman
 *Software Engineer
 *Red Hat, Inc.
 *nhorman@redhat.com
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/

Re: Memory Management

[Márcio Oliveira]

Neil,

>The best way I can think to do that is take a look at /proc/slabinfo.  That will
>likely give you a pointer to which area of code is eating up your memory.
>  
>
OK. I will monitor the /proc/slabinfo file.

>Based on the sysrq-m info you posted it looks like due to fragmentation the
>largest chunk of memory you can allocate is 2MB (perhaps less depending on
>address space availability).  If you can build a test kernel to do a show_state
>rather than a show_mem at the beginning of oom_kil, then you should be able to
>tell who is trying to do an allocation that leads to kswapd calling
>out_of_memory.
>  
>
Neil, I'm trying to recompile the kernel source 2.4.21-32.0.1 and get 
some error messages:

In file included from 
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/prefetch.h:13,
                 from 
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/list.h:6,
                 from 
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:12,
                 from 3w-xxxx.c:172:
/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:61: warning: 
parameter names (without types) in function declaration
/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:61: field 
`loops_per_jiffy_R_ver_str' declared as a function
/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: invalid 
suffix on integer constant
/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: syntax error 
before numeric constant
/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: warning: 
function declaration isn't a prototype
/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: invalid 
suffix on integer constant
/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: syntax 
error before numeric constant
/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: warning: 
function declaration isn't a prototype
/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:273: warning: 
parameter names (without types) in function declaration
In file included from 3w-xxxx.c:172:
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: invalid 
suffix on integer constant
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: syntax error 
before numeric constant
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: 
`inter_module_register_R_ver_str' declared as function returning a function
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: warning: 
function declaration isn't a prototype
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: invalid 
suffix on integer constant
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: syntax error 
before numeric constant
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: 
`inter_module_unregister_R_ver_str' declared as function returning a 
function
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: warning: 
function declaration isn't a prototype
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:192: 
`inter_module_get_R_ver_str' declared as function returning a function
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:192: warning: 
parameter names (without types) in function declaration
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:193: 
`inter_module_get_request_R_ver_str' declared as function returning a 
function
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:193: warning: 
parameter names (without types) in function declaration
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: invalid 
suffix on integer constant
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: syntax error 
before numeric constant
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: 
`inter_module_put_R_ver_str' declared as function returning a function
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: warning: 
function declaration isn't a prototype
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:203: 
`try_inc_mod_count_R_ver_str' declared as function returning a function
/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:203: warning: 
parameter names (without types) in function declaration
make[3]: *** [3w-xxxx_10200033.o] Error 1
make[3]: Leaving directory 
`/usr/src/linux-2.4.21-32.0.1.EL/drivers/addon/3w-xxxx_10200033'
make[2]: *** [_modsubdir_3w-xxxx_10200033] Error 2
make[2]: Leaving directory `/usr/src/linux-2.4.21-32.0.1.EL/drivers/addon'
make[1]: *** [_modsubdir_addon] Error 2
make[1]: Leaving directory `/usr/src/linux-2.4.21-32.0.1.EL/drivers'
make: *** [_mod_drivers] Error 2

Is there any relationship between the sysrq-m changes to do show_state() 
rather than a show_mem() and the compiling erros?

/usr/src/linux-2.4.21-32.0.1.EL/include/linux/prefetch.h, line 13:
    #include <asm/processor.h>

/usr/src/linux-2.4.21-32.0.1.EL/include/linux/list.h ,line 6:
    #include <linux/prefetch.h>

/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h, line 12:
    #include <linux/list.h>

3w-xxxx.c, line 172:
    #include <linux/module.h>

Any ideia about the kernel compiling erros?

(If I try to recompile a kernel.org kernel, it is compiled fine).

Thanks again.

Márcio.

Re: Memory Management

[Neil Horman]

On Sat, Jul 23, 2005 at 08:16:20PM -0300, Márcio Oliveira wrote:
> Neil,
> 
> >The best way I can think to do that is take a look at /proc/slabinfo.  
> >That will
> >likely give you a pointer to which area of code is eating up your memory.
> > 
> >
> OK. I will monitor the /proc/slabinfo file.
> 
> >Based on the sysrq-m info you posted it looks like due to fragmentation the
> >largest chunk of memory you can allocate is 2MB (perhaps less depending on
> >address space availability).  If you can build a test kernel to do a 
> >show_state
> >rather than a show_mem at the beginning of oom_kil, then you should be 
> >able to
> >tell who is trying to do an allocation that leads to kswapd calling
> >out_of_memory.
> > 
> >
> Neil, I'm trying to recompile the kernel source 2.4.21-32.0.1 and get 
> some error messages:
> 
> In file included from 
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/prefetch.h:13,
>                 from 
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/list.h:6,
>                 from 
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:12,
>                 from 3w-xxxx.c:172:
> /usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:61: warning: 
> parameter names (without types) in function declaration
> /usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:61: field 
> `loops_per_jiffy_R_ver_str' declared as a function
> /usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: invalid 
> suffix on integer constant
> /usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: syntax error 
> before numeric constant
> /usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: warning: 
> function declaration isn't a prototype
> /usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: invalid 
> suffix on integer constant
> /usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: syntax 
> error before numeric constant
> /usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: warning: 
> function declaration isn't a prototype
> /usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:273: warning: 
> parameter names (without types) in function declaration
> In file included from 3w-xxxx.c:172:
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: invalid 
> suffix on integer constant
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: syntax error 
> before numeric constant
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: 
> `inter_module_register_R_ver_str' declared as function returning a function
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: warning: 
> function declaration isn't a prototype
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: invalid 
> suffix on integer constant
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: syntax error 
> before numeric constant
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: 
> `inter_module_unregister_R_ver_str' declared as function returning a 
> function
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: warning: 
> function declaration isn't a prototype
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:192: 
> `inter_module_get_R_ver_str' declared as function returning a function
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:192: warning: 
> parameter names (without types) in function declaration
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:193: 
> `inter_module_get_request_R_ver_str' declared as function returning a 
> function
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:193: warning: 
> parameter names (without types) in function declaration
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: invalid 
> suffix on integer constant
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: syntax error 
> before numeric constant
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: 
> `inter_module_put_R_ver_str' declared as function returning a function
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: warning: 
> function declaration isn't a prototype
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:203: 
> `try_inc_mod_count_R_ver_str' declared as function returning a function
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:203: warning: 
> parameter names (without types) in function declaration
> make[3]: *** [3w-xxxx_10200033.o] Error 1
> make[3]: Leaving directory 
> `/usr/src/linux-2.4.21-32.0.1.EL/drivers/addon/3w-xxxx_10200033'
> make[2]: *** [_modsubdir_3w-xxxx_10200033] Error 2
> make[2]: Leaving directory `/usr/src/linux-2.4.21-32.0.1.EL/drivers/addon'
> make[1]: *** [_modsubdir_addon] Error 2
> make[1]: Leaving directory `/usr/src/linux-2.4.21-32.0.1.EL/drivers'
> make: *** [_mod_drivers] Error 2
> 
> Is there any relationship between the sysrq-m changes to do show_state() 
> rather than a show_mem() and the compiling erros?
> 
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/prefetch.h, line 13:
>    #include <asm/processor.h>
> 
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/list.h ,line 6:
>    #include <linux/prefetch.h>
> 
> /usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h, line 12:
>    #include <linux/list.h>
> 
> 3w-xxxx.c, line 172:
>    #include <linux/module.h>
> 
> Any ideia about the kernel compiling erros?
> 
> (If I try to recompile a kernel.org kernel, it is compiled fine).
> 
> Thanks again.
> 
> Márcio.
> 
I honestly don't know.  I expect you haven't patched something correctly, have
you built the source tree with rpmbuild, or are you just extracting the tar file
from the rpm?
Neil

-- 
/***************************************************
 *Neil Horman
 *Software Engineer
 *Red Hat, Inc.
 *nhorman@redhat.com
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/

Re: Memory Management

[Márcio Oliveira]

Neil Horman wrote:

>On Sat, Jul 23, 2005 at 08:16:20PM -0300, Márcio Oliveira wrote:
>  
>
>>Neil,
>>
>>    
>>
>>>The best way I can think to do that is take a look at /proc/slabinfo.  
>>>That will
>>>likely give you a pointer to which area of code is eating up your memory.
>>>
>>>
>>>      
>>>
>>OK. I will monitor the /proc/slabinfo file.
>>
>>    
>>
>>>Based on the sysrq-m info you posted it looks like due to fragmentation the
>>>largest chunk of memory you can allocate is 2MB (perhaps less depending on
>>>address space availability).  If you can build a test kernel to do a 
>>>show_state
>>>rather than a show_mem at the beginning of oom_kil, then you should be 
>>>able to
>>>tell who is trying to do an allocation that leads to kswapd calling
>>>out_of_memory.
>>>
>>>
>>>      
>>>
>>Neil, I'm trying to recompile the kernel source 2.4.21-32.0.1 and get 
>>some error messages:
>>
>>In file included from 
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/prefetch.h:13,
>>                from 
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/list.h:6,
>>                from 
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:12,
>>                from 3w-xxxx.c:172:
>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:61: warning: 
>>parameter names (without types) in function declaration
>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:61: field 
>>`loops_per_jiffy_R_ver_str' declared as a function
>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: invalid 
>>suffix on integer constant
>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: syntax error 
>>before numeric constant
>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: warning: 
>>function declaration isn't a prototype
>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: invalid 
>>suffix on integer constant
>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: syntax 
>>error before numeric constant
>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: warning: 
>>function declaration isn't a prototype
>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:273: warning: 
>>parameter names (without types) in function declaration
>>In file included from 3w-xxxx.c:172:
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: invalid 
>>suffix on integer constant
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: syntax error 
>>before numeric constant
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: 
>>`inter_module_register_R_ver_str' declared as function returning a function
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: warning: 
>>function declaration isn't a prototype
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: invalid 
>>suffix on integer constant
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: syntax error 
>>before numeric constant
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: 
>>`inter_module_unregister_R_ver_str' declared as function returning a 
>>function
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: warning: 
>>function declaration isn't a prototype
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:192: 
>>`inter_module_get_R_ver_str' declared as function returning a function
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:192: warning: 
>>parameter names (without types) in function declaration
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:193: 
>>`inter_module_get_request_R_ver_str' declared as function returning a 
>>function
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:193: warning: 
>>parameter names (without types) in function declaration
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: invalid 
>>suffix on integer constant
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: syntax error 
>>before numeric constant
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: 
>>`inter_module_put_R_ver_str' declared as function returning a function
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: warning: 
>>function declaration isn't a prototype
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:203: 
>>`try_inc_mod_count_R_ver_str' declared as function returning a function
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:203: warning: 
>>parameter names (without types) in function declaration
>>make[3]: *** [3w-xxxx_10200033.o] Error 1
>>make[3]: Leaving directory 
>>`/usr/src/linux-2.4.21-32.0.1.EL/drivers/addon/3w-xxxx_10200033'
>>make[2]: *** [_modsubdir_3w-xxxx_10200033] Error 2
>>make[2]: Leaving directory `/usr/src/linux-2.4.21-32.0.1.EL/drivers/addon'
>>make[1]: *** [_modsubdir_addon] Error 2
>>make[1]: Leaving directory `/usr/src/linux-2.4.21-32.0.1.EL/drivers'
>>make: *** [_mod_drivers] Error 2
>>
>>Is there any relationship between the sysrq-m changes to do show_state() 
>>rather than a show_mem() and the compiling erros?
>>
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/prefetch.h, line 13:
>>   #include <asm/processor.h>
>>
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/list.h ,line 6:
>>   #include <linux/prefetch.h>
>>
>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h, line 12:
>>   #include <linux/list.h>
>>
>>3w-xxxx.c, line 172:
>>   #include <linux/module.h>
>>
>>Any ideia about the kernel compiling erros?
>>
>>(If I try to recompile a kernel.org kernel, it is compiled fine).
>>
>>Thanks again.
>>
>>Márcio.
>>
>>    
>>
>I honestly don't know.  I expect you haven't patched something correctly, have
>you built the source tree with rpmbuild, or are you just extracting the tar file
>from the rpm?
>Neil
>
>  
>
     I'm using the kernel-source package and trying to compiling the 
source (in /usr/src/linux-2.4 directory) with "make config", "make dep", 
"make clean", "make bzImage", "make modules" and "make 
modules_install".  I also try to compile a RHEL3 kernel-source without 
the sysrq-m changes on other RHEL3 systems and get the same errors...

    If I try to compile a kernel source provide by kernel.org, I don't 
get the errors above, and the kernel compile works. When I try to 
compile the RHEL3 kernel-source, the "make config", "make dep", "make 
clean" and "make bzImage" commands works fine but when I run "make 
modules" command I get the errors above. I think that is some issue with 
the RHEL3 kernel-source package because other kernel sources (not Red 
Hat kernel) compile without problems.

   I will try to compile the kernel using rpmbuild and check the 
results, plus the slabinfo informations.

Thanks again,

Márcio.

Re: Memory Management

[Seiji Kihara]

Hello,

At Sun, 24 Jul 2005 22:40:19 -0300,
Márcio Oliveira wrote:
>      I'm using the kernel-source package and trying to compiling the 
> source (in /usr/src/linux-2.4 directory) with "make config", "make dep", 
> "make clean", "make bzImage", "make modules" and "make 
> modules_install".  I also try to compile a RHEL3 kernel-source without 
> the sysrq-m changes on other RHEL3 systems and get the same errors...

I wonder if the topic is suitable for the list...  Anyway, I think
"make mrproper" is required before "make config" for the Red Hat
kernel-source package.  The reason may be described in

http://people.redhat.com/dledford/README.mod_devel_kit_theory.html

Regards,

Seiji Kihara
NTT Laboratories, Japan

Re: Memory Management

[Neil Horman]

On Sun, Jul 24, 2005 at 10:40:19PM -0300, Márcio Oliveira wrote:
> Neil Horman wrote:
> 
> >On Sat, Jul 23, 2005 at 08:16:20PM -0300, Márcio Oliveira wrote:
> > 
> >
> >>Neil,
> >>
> >>   
> >>
> >>>The best way I can think to do that is take a look at /proc/slabinfo.  
> >>>That will
> >>>likely give you a pointer to which area of code is eating up your memory.
> >>>
> >>>
> >>>     
> >>>
> >>OK. I will monitor the /proc/slabinfo file.
> >>
> >>   
> >>
> >>>Based on the sysrq-m info you posted it looks like due to fragmentation 
> >>>the
> >>>largest chunk of memory you can allocate is 2MB (perhaps less depending 
> >>>on
> >>>address space availability).  If you can build a test kernel to do a 
> >>>show_state
> >>>rather than a show_mem at the beginning of oom_kil, then you should be 
> >>>able to
> >>>tell who is trying to do an allocation that leads to kswapd calling
> >>>out_of_memory.
> >>>
> >>>
> >>>     
> >>>
> >>Neil, I'm trying to recompile the kernel source 2.4.21-32.0.1 and get 
> >>some error messages:
> >>
> >>In file included from 
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/prefetch.h:13,
> >>               from 
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/list.h:6,
> >>               from 
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:12,
> >>               from 3w-xxxx.c:172:
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:61: warning: 
> >>parameter names (without types) in function declaration
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:61: field 
> >>`loops_per_jiffy_R_ver_str' declared as a function
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: invalid 
> >>suffix on integer constant
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: syntax error 
> >>before numeric constant
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: warning: 
> >>function declaration isn't a prototype
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: invalid 
> >>suffix on integer constant
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: syntax 
> >>error before numeric constant
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: warning: 
> >>function declaration isn't a prototype
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:273: warning: 
> >>parameter names (without types) in function declaration
> >>In file included from 3w-xxxx.c:172:
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: invalid 
> >>suffix on integer constant
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: syntax error 
> >>before numeric constant
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: 
> >>`inter_module_register_R_ver_str' declared as function returning a 
> >>function
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: warning: 
> >>function declaration isn't a prototype
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: invalid 
> >>suffix on integer constant
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: syntax error 
> >>before numeric constant
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: 
> >>`inter_module_unregister_R_ver_str' declared as function returning a 
> >>function
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: warning: 
> >>function declaration isn't a prototype
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:192: 
> >>`inter_module_get_R_ver_str' declared as function returning a function
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:192: warning: 
> >>parameter names (without types) in function declaration
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:193: 
> >>`inter_module_get_request_R_ver_str' declared as function returning a 
> >>function
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:193: warning: 
> >>parameter names (without types) in function declaration
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: invalid 
> >>suffix on integer constant
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: syntax error 
> >>before numeric constant
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: 
> >>`inter_module_put_R_ver_str' declared as function returning a function
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: warning: 
> >>function declaration isn't a prototype
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:203: 
> >>`try_inc_mod_count_R_ver_str' declared as function returning a function
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:203: warning: 
> >>parameter names (without types) in function declaration
> >>make[3]: *** [3w-xxxx_10200033.o] Error 1
> >>make[3]: Leaving directory 
> >>`/usr/src/linux-2.4.21-32.0.1.EL/drivers/addon/3w-xxxx_10200033'
> >>make[2]: *** [_modsubdir_3w-xxxx_10200033] Error 2
> >>make[2]: Leaving directory `/usr/src/linux-2.4.21-32.0.1.EL/drivers/addon'
> >>make[1]: *** [_modsubdir_addon] Error 2
> >>make[1]: Leaving directory `/usr/src/linux-2.4.21-32.0.1.EL/drivers'
> >>make: *** [_mod_drivers] Error 2
> >>
> >>Is there any relationship between the sysrq-m changes to do show_state() 
> >>rather than a show_mem() and the compiling erros?
> >>
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/prefetch.h, line 13:
> >>  #include <asm/processor.h>
> >>
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/list.h ,line 6:
> >>  #include <linux/prefetch.h>
> >>
> >>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h, line 12:
> >>  #include <linux/list.h>
> >>
> >>3w-xxxx.c, line 172:
> >>  #include <linux/module.h>
> >>
> >>Any ideia about the kernel compiling erros?
> >>
> >>(If I try to recompile a kernel.org kernel, it is compiled fine).
> >>
> >>Thanks again.
> >>
> >>Márcio.
> >>
> >>   
> >>
> >I honestly don't know.  I expect you haven't patched something correctly, 
> >have
> >you built the source tree with rpmbuild, or are you just extracting the 
> >tar file
> >from the rpm?
> >Neil
> >
> > 
> >
I honestly have no idea.  Suffice it to say, I never build out of the
kernel-source package.  Better to use the kernel src rpm, do an rpmbuild -bp on
the kernel src rpm, setting your target arch accordingly.  I have no problems
with that on any RHEL kernels.
Neil

>     I'm using the kernel-source package and trying to compiling the 
> source (in /usr/src/linux-2.4 directory) with "make config", "make dep", 
> "make clean", "make bzImage", "make modules" and "make 
> modules_install".  I also try to compile a RHEL3 kernel-source without 
> the sysrq-m changes on other RHEL3 systems and get the same errors...
> 
>    If I try to compile a kernel source provide by kernel.org, I don't 
> get the errors above, and the kernel compile works. When I try to 
> compile the RHEL3 kernel-source, the "make config", "make dep", "make 
> clean" and "make bzImage" commands works fine but when I run "make 
> modules" command I get the errors above. I think that is some issue with 
> the RHEL3 kernel-source package because other kernel sources (not Red 
> Hat kernel) compile without problems.
> 
>   I will try to compile the kernel using rpmbuild and check the 
> results, plus the slabinfo informations.
> 
> Thanks again,
> 
> Márcio.

-- 
/***************************************************
 *Neil Horman
 *Software Engineer
 *Red Hat, Inc.
 *nhorman@redhat.com
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/

Re: Memory Management

[Márcio Oliveira]

Neil Horman wrote:

>On Sun, Jul 24, 2005 at 10:40:19PM -0300, Márcio Oliveira wrote:
>  
>
>>Neil Horman wrote:
>>
>>    
>>
>>>On Sat, Jul 23, 2005 at 08:16:20PM -0300, Márcio Oliveira wrote:
>>>
>>>
>>>      
>>>
>>>>Neil,
>>>>
>>>>  
>>>>
>>>>        
>>>>
>>>>>The best way I can think to do that is take a look at /proc/slabinfo.  
>>>>>That will
>>>>>likely give you a pointer to which area of code is eating up your memory.
>>>>>
>>>>>
>>>>>    
>>>>>
>>>>>          
>>>>>
>>>>OK. I will monitor the /proc/slabinfo file.
>>>>
>>>>  
>>>>
>>>>        
>>>>
>>>>>Based on the sysrq-m info you posted it looks like due to fragmentation 
>>>>>the
>>>>>largest chunk of memory you can allocate is 2MB (perhaps less depending 
>>>>>on
>>>>>address space availability).  If you can build a test kernel to do a 
>>>>>show_state
>>>>>rather than a show_mem at the beginning of oom_kil, then you should be 
>>>>>able to
>>>>>tell who is trying to do an allocation that leads to kswapd calling
>>>>>out_of_memory.
>>>>>
>>>>>
>>>>>    
>>>>>
>>>>>          
>>>>>
>>>>Neil, I'm trying to recompile the kernel source 2.4.21-32.0.1 and get 
>>>>some error messages:
>>>>
>>>>In file included from 
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/prefetch.h:13,
>>>>              from 
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/list.h:6,
>>>>              from 
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:12,
>>>>              from 3w-xxxx.c:172:
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:61: warning: 
>>>>parameter names (without types) in function declaration
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:61: field 
>>>>`loops_per_jiffy_R_ver_str' declared as a function
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: invalid 
>>>>suffix on integer constant
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: syntax error 
>>>>before numeric constant
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:84: warning: 
>>>>function declaration isn't a prototype
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: invalid 
>>>>suffix on integer constant
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: syntax 
>>>>error before numeric constant
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:269: warning: 
>>>>function declaration isn't a prototype
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/asm/processor.h:273: warning: 
>>>>parameter names (without types) in function declaration
>>>>In file included from 3w-xxxx.c:172:
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: invalid 
>>>>suffix on integer constant
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: syntax error 
>>>>before numeric constant
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: 
>>>>`inter_module_register_R_ver_str' declared as function returning a 
>>>>function
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:190: warning: 
>>>>function declaration isn't a prototype
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: invalid 
>>>>suffix on integer constant
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: syntax error 
>>>>before numeric constant
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: 
>>>>`inter_module_unregister_R_ver_str' declared as function returning a 
>>>>function
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:191: warning: 
>>>>function declaration isn't a prototype
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:192: 
>>>>`inter_module_get_R_ver_str' declared as function returning a function
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:192: warning: 
>>>>parameter names (without types) in function declaration
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:193: 
>>>>`inter_module_get_request_R_ver_str' declared as function returning a 
>>>>function
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:193: warning: 
>>>>parameter names (without types) in function declaration
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: invalid 
>>>>suffix on integer constant
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: syntax error 
>>>>before numeric constant
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: 
>>>>`inter_module_put_R_ver_str' declared as function returning a function
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:194: warning: 
>>>>function declaration isn't a prototype
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:203: 
>>>>`try_inc_mod_count_R_ver_str' declared as function returning a function
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h:203: warning: 
>>>>parameter names (without types) in function declaration
>>>>make[3]: *** [3w-xxxx_10200033.o] Error 1
>>>>make[3]: Leaving directory 
>>>>`/usr/src/linux-2.4.21-32.0.1.EL/drivers/addon/3w-xxxx_10200033'
>>>>make[2]: *** [_modsubdir_3w-xxxx_10200033] Error 2
>>>>make[2]: Leaving directory `/usr/src/linux-2.4.21-32.0.1.EL/drivers/addon'
>>>>make[1]: *** [_modsubdir_addon] Error 2
>>>>make[1]: Leaving directory `/usr/src/linux-2.4.21-32.0.1.EL/drivers'
>>>>make: *** [_mod_drivers] Error 2
>>>>
>>>>Is there any relationship between the sysrq-m changes to do show_state() 
>>>>rather than a show_mem() and the compiling erros?
>>>>
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/prefetch.h, line 13:
>>>> #include <asm/processor.h>
>>>>
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/list.h ,line 6:
>>>> #include <linux/prefetch.h>
>>>>
>>>>/usr/src/linux-2.4.21-32.0.1.EL/include/linux/module.h, line 12:
>>>> #include <linux/list.h>
>>>>
>>>>3w-xxxx.c, line 172:
>>>> #include <linux/module.h>
>>>>
>>>>Any ideia about the kernel compiling erros?
>>>>
>>>>(If I try to recompile a kernel.org kernel, it is compiled fine).
>>>>
>>>>Thanks again.
>>>>
>>>>Márcio.
>>>>
>>>>  
>>>>
>>>>        
>>>>
>>>I honestly don't know.  I expect you haven't patched something correctly, 
>>>have
>>>you built the source tree with rpmbuild, or are you just extracting the 
>>>tar file
>>>      
>>>
>>>from the rpm?
>>    
>>
>>>Neil
>>>
>>>
>>>
>>>      
>>>
>I honestly have no idea.  Suffice it to say, I never build out of the
>kernel-source package.  Better to use the kernel src rpm, do an rpmbuild -bp on
>the kernel src rpm, setting your target arch accordingly.  I have no problems
>with that on any RHEL kernels.
>Neil
>
>  
>
>>    I'm using the kernel-source package and trying to compiling the 
>>source (in /usr/src/linux-2.4 directory) with "make config", "make dep", 
>>"make clean", "make bzImage", "make modules" and "make 
>>modules_install".  I also try to compile a RHEL3 kernel-source without 
>>the sysrq-m changes on other RHEL3 systems and get the same errors...
>>
>>   If I try to compile a kernel source provide by kernel.org, I don't 
>>get the errors above, and the kernel compile works. When I try to 
>>compile the RHEL3 kernel-source, the "make config", "make dep", "make 
>>clean" and "make bzImage" commands works fine but when I run "make 
>>modules" command I get the errors above. I think that is some issue with 
>>the RHEL3 kernel-source package because other kernel sources (not Red 
>>Hat kernel) compile without problems.
>>
>>  I will try to compile the kernel using rpmbuild and check the 
>>results, plus the slabinfo informations.
>>
>>Thanks again,
>>
>>Márcio.
>>    
>>
>
>  
>
Neil,

  Can I increase the number of pages in Zone_NORMAL using RHEL 3 
hugemem kernel?

Thanks.

Márcio.

Memoy Management

[Márcio Oliveira]

Hi all,

  Somebody can help me with some memory management issues (like Out Of 
Memory) in Linux kernel 2.4 (with some backports from 2.6 kernel. eg. 
Red Hat Enterprise Kernel) and SMP machines (4 processors) with a lot of 
memory (16GB)?

Thanks a lot.

Márcio.

Re: Memoy Management

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 570 bytes --]

On Wed, 2005-07-20 at 10:10 -0300, Márcio Oliveira wrote:
> Hi all,
> 
>   Somebody can help me with some memory management issues (like Out Of 
> Memory) in Linux kernel 2.4 (with some backports from 2.6 kernel. eg. 
> Red Hat Enterprise Kernel) and SMP machines (4 processors) with a lot of 
> memory (16GB)?

I'm sure RH support will be able to help you with that; I doubt many
other people care about an ancient kernel like that, and a vendor one to
boot.

(Also I assume you are using the -hugemem kernel as the documentation
recommends you to do)


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Memory Management

[Márcio Oliveira]

Arjan van de Ven wrote:

>I'm sure RH support will be able to help you with that; I doubt many
>other people care about an ancient kernel like that, and a vendor one to
>boot.
>
>(Also I assume you are using the -hugemem kernel as the documentation
>recommends you to do)
>
>  
>
Arjan,

   I'd like to know/understand more about memory management  on  Linux 
Kernel and I belive this concept is applyable to the Red Hat Linux Kernel.

  I have some doubts about the ZONE divison (DMA, NORMAL, HIGHMEM), 
Shared Memory utilization, HugeTLB feature and OOM with large memory and 
the kernel management of memory on SMP machines. I believe these 
features are common to the Linux kernel in general(Red Hat, Debian, 
SuSe, kernel.org), right?
   I read a tons of docs regarding symposiums, The Linux Memory 
Management Book and lots of docs about Oracle memory management but 
memory management still not clear to me.

  If somebody can help me...

Thanks.

Re: Memory Management

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 1239 bytes --]

On Wed, 2005-07-20 at 11:23 -0300, Márcio Oliveira wrote:
> Arjan van de Ven wrote:
> 
> >I'm sure RH support will be able to help you with that; I doubt many
> >other people care about an ancient kernel like that, and a vendor one to
> >boot.
> >
> >(Also I assume you are using the -hugemem kernel as the documentation
> >recommends you to do)
> >
> >  
> >
> Arjan,
> 
>    I'd like to know/understand more about memory management  on  Linux 
> Kernel and I belive this concept is applyable to the Red Hat Linux Kernel.

Only on the highest of levels. The RHEL3 kernel has a VM that resembles
almost no other linux kernel in many many ways. 


>   I have some doubts about the ZONE divison (DMA, NORMAL, HIGHMEM), 
> Shared Memory utilization, HugeTLB feature and OOM with large memory and 
> the kernel management of memory on SMP machines. I believe these 
> features are common to the Linux kernel in general(Red Hat, Debian, 
> SuSe, kernel.org), right?

nope. These things are very much different between the kernels you
mention.

What do you want to use the knowledge for? Fixing the VM? Tuning your
server? The goal of your question determines what kind of answer you
want to your questions....

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]