Re: Memory usage per memory zone

Re: Memory usage per memory zone

[Wu Fengguang]

Hi jack,

On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> Hello,
> 
> I have a box where the oom-killer is killing processes due to running
> out of memory in zone_normal. I can see using slabtop that the inode

How do you know that the memory pressure on zone normal stand out alone?

> caches are using up lots of memory and guess this is the problem, so
> have cleared them using an echo to drop_caches.

It would better be backed by concrete numbers...

> 
> I would quite like to not guess though - is it possible to use slabtop
> (or any other way) to view ram usage per zone so I can pick out the
> culprit?

/proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
Some of them deal with slabs.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> Hi jack,
>
> On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
>> Hello,
>>
>> I have a box where the oom-killer is killing processes due to running
>> out of memory in zone_normal. I can see using slabtop that the inode
>
> How do you know that the memory pressure on zone normal stand out alone?

For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:

kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
active:1304kB inactive:128kB present:901120kB pages_scanned:168951
all_unreclaimable? yes

>> caches are using up lots of memory and guess this is the problem, so
>> have cleared them using an echo to drop_caches.
>
> It would better be backed by concrete numbers...
>
>>
>> I would quite like to not guess though - is it possible to use slabtop
>> (or any other way) to view ram usage per zone so I can pick out the
>> culprit?
>
> /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> Some of them deal with slabs.

Thanks, I'll read up on how to interpret these.

Do you recommend these two files for tracking down memory usage per
process per zone?

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Wu Fengguang]

On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> > Hi jack,
> >
> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> >> Hello,
> >>
> >> I have a box where the oom-killer is killing processes due to running
> >> out of memory in zone_normal. I can see using slabtop that the inode
> >
> > How do you know that the memory pressure on zone normal stand out alone?
> 
> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
> 
> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
> all_unreclaimable? yes

It's normal behavior.  Linux kernel tries hard to utilize most of
the free memory for caching files :)

> >> caches are using up lots of memory and guess this is the problem, so
> >> have cleared them using an echo to drop_caches.
> >
> > It would better be backed by concrete numbers...
> >
> >>
> >> I would quite like to not guess though - is it possible to use slabtop
> >> (or any other way) to view ram usage per zone so I can pick out the
> >> culprit?
> >
> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> > Some of them deal with slabs.
> 
> Thanks, I'll read up on how to interpret these.
> 
> Do you recommend these two files for tracking down memory usage per
> process per zone?

No, the two interfaces provide system wide counters.  We have the well
known tools "ps" and "top" for per-process numbers, hehe.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
>> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
>> > Hi jack,
>> >
>> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
>> >> Hello,
>> >>
>> >> I have a box where the oom-killer is killing processes due to running
>> >> out of memory in zone_normal. I can see using slabtop that the inode
>> >
>> > How do you know that the memory pressure on zone normal stand out alone?
>>
>> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
>>
>> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
>> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
>> all_unreclaimable? yes
>
> It's normal behavior.  Linux kernel tries hard to utilize most of
> the free memory for caching files :)

With all_unreclaimable = yes is normal?

>
>> >> caches are using up lots of memory and guess this is the problem, so
>> >> have cleared them using an echo to drop_caches.
>> >
>> > It would better be backed by concrete numbers...
>> >
>> >>
>> >> I would quite like to not guess though - is it possible to use slabtop
>> >> (or any other way) to view ram usage per zone so I can pick out the
>> >> culprit?
>> >
>> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
>> > Some of them deal with slabs.
>>
>> Thanks, I'll read up on how to interpret these.
>>
>> Do you recommend these two files for tracking down memory usage per
>> process per zone?
>
> No, the two interfaces provide system wide counters.  We have the well
> known tools "ps" and "top" for per-process numbers, hehe.

ps and top do not have per-zone numbers. How do I get those?

>
> Thanks,
> Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Wu Fengguang]

On Wed, Mar 11, 2009 at 02:16:38PM +0200, jack marrow wrote:
> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> > On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
> >> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> >> > Hi jack,
> >> >
> >> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> >> >> Hello,
> >> >>
> >> >> I have a box where the oom-killer is killing processes due to running
> >> >> out of memory in zone_normal. I can see using slabtop that the inode
> >> >
> >> > How do you know that the memory pressure on zone normal stand out alone?
> >>
> >> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
> >>
> >> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
> >> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
> >> all_unreclaimable? yes
> >
> > It's normal behavior. A Linux kernel tries hard to utilize most of
> > the free memory for caching files :)
> 
> With all_unreclaimable = yes is normal?

Ah, perhaps not.

Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
/proc/slabinfo? Thank you.

> >
> >> >> caches are using up lots of memory and guess this is the problem, so
> >> >> have cleared them using an echo to drop_caches.
> >> >
> >> > It would better be backed by concrete numbers...
> >> >
> >> >>
> >> >> I would quite like to not guess though - is it possible to use slabtop
> >> >> (or any other way) to view ram usage per zone so I can pick out the
> >> >> culprit?
> >> >
> >> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> >> > Some of them deal with slabs.
> >>
> >> Thanks, I'll read up on how to interpret these.
> >>
> >> Do you recommend these two files for tracking down memory usage per
> >> process per zone?
> >
> > No, the two interfaces provide system wide counters. A We have the well
> > known tools "ps" and "top" for per-process numbers, hehe.
> 
> ps and top do not have per-zone numbers. How do I get those?

Maybe through the pagemap interface: Documentation/vm/pagemap.txt

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

> Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> /proc/slabinfo? Thank you.

Sure, but I don't know if it will help.

The oom info was from in the night, the rest is from now. I have no zoneinfo.

http://pastebin.com/m67409bc0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> > /proc/slabinfo? Thank you.
> 
> Sure, but I don't know if it will help.
> 
> The oom info was from in the night, the rest is from now. I have no zoneinfo.
> 
> http://pastebin.com/m67409bc0

Thank you! So you are running a pretty old kernel?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

(Didn't reply all:)

2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
>> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
>> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
>> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
>> >> > /proc/slabinfo? Thank you.
>> >>
>> >> Sure, but I don't know if it will help.
>> >>
>> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
>> >>
>> >> http://pastebin.com/m67409bc0
>> >
>> > Thank you! So you are running a pretty old kernel?
>> >
>>
>> Yes. Kernel.2.6.9-78.
>>
>> Added more output from the other oom kills here:
>>  http://pastebin.com/m76fc473d
>>
>> If I could just find a way to find out what is using up all the memory
>> in a zone I could go away happy :)
>
> But the highmem zone wont help you much, since you have a large 900M
> normal zone and a tiny 100M highmem zone ;)
>
> The cached files seem to be the memory killer:

I ran an echo 3 > drop_caches yesterday, I was hoping to come in and
find no oom kill. Oh well :)

I thought the kernel dropped caches if a program needs the ram?

>
> MemTotal:      1034496 kB
> MemFree:         95600 kB
> Buffers:         49916 kB
> Cached:         761544 kB
> SwapCached:          0 kB
> Active:          80484 kB
> Inactive:       749960 kB
> HighTotal:      131008 kB
> HighFree:        68480 kB
> LowTotal:       903488 kB
> LowFree:         27120 kB
> SwapTotal:     2040212 kB
> SwapFree:      2039780 kB
> Dirty:               4 kB
> Writeback:           0 kB
> Mapped:          32636 kB
> Slab:            93856 kB
> CommitLimit:   2557460 kB
> Committed_AS:   129980 kB
> PageTables:       1800 kB
> VmallocTotal:   106488 kB
> VmallocUsed:      3372 kB
> VmallocChunk:   102616 kB
> HugePages_Total:     0
> HugePages_Free:      0
> Hugepagesize:     2048 kB
>
> Is upgrading the kernel an option for you?

No :(

I think shoving some more ram in the box is the best doable option.
Would this help here?

To do that I need to say "look at how much cache we are using for
files, that cache is in the high mem zone (look here) so let's put
some more ram in". Does the cache always live in the high mem zone?

>
> Thanks,
> Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 10:17:44AM +0200, jack marrow wrote:
> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> > On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
> >> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> >> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> >> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> >> >> > /proc/slabinfo? Thank you.
> >> >>
> >> >> Sure, but I don't know if it will help.
> >> >>
> >> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
> >> >>
> >> >> http://pastebin.com/m67409bc0
> >> >
> >> > Thank you! So you are running a pretty old kernel?
> >> >
> >>
> >> Yes. Kernel.2.6.9-78.
> >>
> >> Added more output from the other oom kills here:
> >> A http://pastebin.com/m76fc473d
> >>
> >> If I could just find a way to find out what is using up all the memory
> >> in a zone I could go away happy :)
> >
> > But the highmem zone wont help you much, since you have a large 900M
> > normal zone and a tiny 100M highmem zone ;)
> >
> > The cached files seem to be the memory killer:
> 
> I ran an echo 3 > drop_caches yesterday, I was hoping to come in and
> find no oom kill. Oh well :)

Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?

> I thought the kernel dropped caches if a program needs the ram?

Sure, but something is unreclaimable... Maybe some process is taking a
lot of shared memory(shm)? What's the output of `lsof`?

> >
> > MemTotal: A  A  A 1034496 kB
> > MemFree: A  A  A  A  95600 kB
> > Buffers: A  A  A  A  49916 kB
> > Cached: A  A  A  A  761544 kB
> > SwapCached: A  A  A  A  A 0 kB
> > Active: A  A  A  A  A 80484 kB
> > Inactive: A  A  A  749960 kB
> > HighTotal: A  A  A 131008 kB
> > HighFree: A  A  A  A 68480 kB
> > LowTotal: A  A  A  903488 kB
> > LowFree: A  A  A  A  27120 kB
> > SwapTotal: A  A  2040212 kB
> > SwapFree: A  A  A 2039780 kB
> > Dirty: A  A  A  A  A  A  A  4 kB
> > Writeback: A  A  A  A  A  0 kB
> > Mapped: A  A  A  A  A 32636 kB
> > Slab: A  A  A  A  A  A 93856 kB
> > CommitLimit: A  2557460 kB
> > Committed_AS: A  129980 kB
> > PageTables: A  A  A  1800 kB
> > VmallocTotal: A  106488 kB
> > VmallocUsed: A  A  A 3372 kB
> > VmallocChunk: A  102616 kB
> > HugePages_Total: A  A  0
> > HugePages_Free: A  A  A 0
> > Hugepagesize: A  A  2048 kB
> >
> > Is upgrading the kernel an option for you?
> 
> No :(
> 
> I think shoving some more ram in the box is the best doable option.
> Would this help here?

There have been huge amounts of change sets in mm area since 2.6.9...

> To do that I need to say "look at how much cache we are using for
> files, that cache is in the high mem zone (look here) so let's put
> some more ram in". Does the cache always live in the high mem zone?

Both highmem and normal zones will be used for caches.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 10:48:42AM +0200, jack marrow wrote:
> (Didn't reply all:)
>
> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> > On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
> >> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> >> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> >> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> >> >> > /proc/slabinfo? Thank you.
> >> >>
> >> >> Sure, but I don't know if it will help.
> >> >>
> >> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
> >> >>
> >> >> http://pastebin.com/m67409bc0
> >> >
> >> > Thank you! So you are running a pretty old kernel?
> >> >
> >>
> >> Yes. Kernel.2.6.9-78.
> >>
> >> Added more output from the other oom kills here:
> >> A http://pastebin.com/m76fc473d
> >>
> >> If I could just find a way to find out what is using up all the memory
> >> in a zone I could go away happy :)
> >
> > But the highmem zone wont help you much, since you have a large 900M
> > normal zone and a tiny 100M highmem zone ;)

btw, how much physical memory do you have?
It's weird that meminfo says 1G but Mem-Info says 4G...

> MemTotal: A  A  A 1034496 kB
> MemFree: A  A  A  A  95600 kB
> Buffers: A  A  A  A  49916 kB
> Cached: A  A  A  A  761544 kB
> SwapCached: A  A  A  A  A 0 kB
> Active: A  A  A  A  A 80484 kB
> Inactive: A  A  A  749960 kB
> HighTotal: A  A  A 131008 kB
> HighFree: A  A  A  A 68480 kB
> LowTotal: A  A  A  903488 kB
> LowFree: A  A  A  A  27120 kB
> SwapTotal: A  A  2040212 kB
> SwapFree: A  A  A 2039780 kB

Free pages:       16808kB (1664kB HighMem)
Active:457312 inactive:273805 dirty:0 writeback:0 unstable:0 free:4202 slab:5897 mapped:390707 pagetables:5068
DMA free:12408kB min:64kB low:128kB high:192kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:2736kB min:3728kB low:7456kB high:11184kB active:256kB inactive:16kB present:901120kB pages_scanned:2782 all_unre
claimable? yes
protections[]: 0 0 0
HighMem free:1664kB min:512kB low:1024kB high:1536kB active:1828992kB inactive:1095204kB present:3014656kB pages_scanned:0 al
l_unreclaimable? no
protections[]: 0 0 0
DMA: 4*4kB 5*8kB 2*16kB 1*32kB 4*64kB 2*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12408kB
Normal: 0*4kB 0*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2736kB
HighMem: 320*4kB 0*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1664kB
448110 pagecache pages
Swap cache: add 787243, delete 731019, find 235217/281007, race 0+3
0 bounce buffer pages
Free swap:       1558736kB
983040 pages of RAM
753648 pages of HIGHMEM
9253 reserved pages
253474 pages shared
56224 pages swap cached
Out of Memory: Killed process 28258 (oracle).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

> Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?

http://pastebin.com/mce24730

>> I thought the kernel dropped caches if a program needs the ram?
>
> Sure, but something is unreclaimable... Maybe some process is taking a
> lot of shared memory(shm)? What's the output of `lsof`?

I can't paste that, but I expect oracle is using it.

>
>> >
>> > MemTotal:      1034496 kB
>> > MemFree:         95600 kB
>> > Buffers:         49916 kB
>> > Cached:         761544 kB
>> > SwapCached:          0 kB
>> > Active:          80484 kB
>> > Inactive:       749960 kB
>> > HighTotal:      131008 kB
>> > HighFree:        68480 kB
>> > LowTotal:       903488 kB
>> > LowFree:         27120 kB
>> > SwapTotal:     2040212 kB
>> > SwapFree:      2039780 kB
>> > Dirty:               4 kB
>> > Writeback:           0 kB
>> > Mapped:          32636 kB
>> > Slab:            93856 kB
>> > CommitLimit:   2557460 kB
>> > Committed_AS:   129980 kB
>> > PageTables:       1800 kB
>> > VmallocTotal:   106488 kB
>> > VmallocUsed:      3372 kB
>> > VmallocChunk:   102616 kB
>> > HugePages_Total:     0
>> > HugePages_Free:      0
>> > Hugepagesize:     2048 kB
>> >
>> > Is upgrading the kernel an option for you?
>>
>> No :(
>>
>> I think shoving some more ram in the box is the best doable option.
>> Would this help here?
>
> There have been huge amounts of change sets in mm area since 2.6.9...
>
>> To do that I need to say "look at how much cache we are using for
>> files, that cache is in the high mem zone (look here) so let's put
>> some more ram in". Does the cache always live in the high mem zone?
>
> Both highmem and normal zones will be used for caches.
>
> Thanks,
> Fengguang
>
>

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

> btw, how much physical memory do you have?
> It's weird that meminfo says 1G but Mem-Info says 4G...

4 gigs.

# free -m
             total       used       free     shared    buffers     cached
Mem:          3804       1398       2405          0          1        240
-/+ buffers/cache:       1156       2648
Swap:         1992        403       1588

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

>> Sure, but something is unreclaimable... Maybe some process is taking a
>> lot of shared memory(shm)? What's the output of `lsof`?
>
> I can't paste that, but I expect oracle is using it.

Maybe this is helpful:

#  ipcs |grep oracle
0x00000000 2293770    oracle    640        4194304    22
0x00000000 2326539    oracle    640        536870912  22
0x880f3334 2359308    oracle    640        266338304  22
0x0f9b5efc 1933312    oracle    640        44

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 01:38:46PM +0200, jack marrow wrote:
> > Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?
> 
> http://pastebin.com/mce24730
> 
> >> I thought the kernel dropped caches if a program needs the ram?
> >
> > Sure, but something is unreclaimable... Maybe some process is taking a
> > lot of shared memory(shm)? What's the output of `lsof`?
> 
> I can't paste that, but I expect oracle is using it.

Oh well...

But from the meminfo, there are 1.2G mapped pages. That could be a big
trouble for reclaiming.  Recent kernels can better handle this situation.

Thanks,
Fengguang
---

# cat /proc/meminfo
MemTotal:      3895404 kB
MemFree:       2472656 kB
Buffers:           412 kB
Cached:         239716 kB
SwapCached:     202652 kB
Active:        1275212 kB
Inactive:        34584 kB
HighTotal:     3014592 kB
HighFree:      1684032 kB
LowTotal:       880812 kB
LowFree:        788624 kB
SwapTotal:     2040212 kB
SwapFree:      1626756 kB
Dirty:             104 kB
Writeback:           0 kB
Mapped:        1247000 kB
Slab:            80040 kB
CommitLimit:   3987912 kB
Committed_AS:  8189040 kB
PageTables:      18792 kB
VmallocTotal:   106488 kB
VmallocUsed:      3072 kB
VmallocChunk:   102980 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> On Thu, Mar 12, 2009 at 01:38:46PM +0200, jack marrow wrote:
>> > Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?
>>
>> http://pastebin.com/mce24730
>>
>> >> I thought the kernel dropped caches if a program needs the ram?
>> >
>> > Sure, but something is unreclaimable... Maybe some process is taking a
>> > lot of shared memory(shm)? What's the output of `lsof`?
>>
>> I can't paste that, but I expect oracle is using it.
>
> Oh well...
>
> But from the meminfo, there are 1.2G mapped pages. That could be a big
> trouble for reclaiming.  Recent kernels can better handle this situation.
>
> Thanks,
> Fengguang
> ---
>
> # cat /proc/meminfo
> MemTotal:      3895404 kB
> MemFree:       2472656 kB
> Buffers:           412 kB
> Cached:         239716 kB
> SwapCached:     202652 kB
> Active:        1275212 kB
> Inactive:        34584 kB
> HighTotal:     3014592 kB
> HighFree:      1684032 kB
> LowTotal:       880812 kB
> LowFree:        788624 kB
> SwapTotal:     2040212 kB
> SwapFree:      1626756 kB
> Dirty:             104 kB
> Writeback:           0 kB
> Mapped:        1247000 kB
> Slab:            80040 kB
> CommitLimit:   3987912 kB
> Committed_AS:  8189040 kB
> PageTables:      18792 kB
> VmallocTotal:   106488 kB
> VmallocUsed:      3072 kB
> VmallocChunk:   102980 kB
> HugePages_Total:     0
> HugePages_Free:      0
> Hugepagesize:     2048 kB
>

Thanks for all your help.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 01:48:50PM +0200, jack marrow wrote:
> >> Sure, but something is unreclaimable... Maybe some process is taking a
> >> lot of shared memory(shm)? What's the output of `lsof`?
> >
> > I can't paste that, but I expect oracle is using it.
> 
> Maybe this is helpful:
> 
> #  ipcs |grep oracle
> 0x00000000 2293770    oracle    640        4194304    22
> 0x00000000 2326539    oracle    640        536870912  22
> 0x880f3334 2359308    oracle    640        266338304  22
> 0x0f9b5efc 1933312    oracle    640        44

Up to 800M shm...

http://lwn.net/Articles/286485/
http://feedblog.org/2009/01/25/splitlru-patch-in-kernel-2628-must-have-for-mysql-and-innodb/

which reads: 

        If youa??re running MySQL with InnoDB and an in-memory buffer pool, and
        having paging issues, you probably should upgrade to 2.6.28 ASAP.


Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>