Memory usage per memory zone

Memory usage per memory zone

[jack marrow]

Hello,

I have a box where the oom-killer is killing processes due to running
out of memory in zone_normal. I can see using slabtop that the inode
caches are using up lots of memory and guess this is the problem, so
have cleared them using an echo to drop_caches.

I would quite like to not guess though - is it possible to use slabtop
(or any other way) to view ram usage per zone so I can pick out the
culprit?

(I am not subscribed to the list, please cc any replies to me.)

Thanks a lot.

p.s. the kernel traffic link on http://www.tux.org/lkml/ is now a spam site

Re: Memory usage per memory zone

[Wu Fengguang]

Hi jack,

On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> Hello,
> 
> I have a box where the oom-killer is killing processes due to running
> out of memory in zone_normal. I can see using slabtop that the inode

How do you know that the memory pressure on zone normal stand out alone?

> caches are using up lots of memory and guess this is the problem, so
> have cleared them using an echo to drop_caches.

It would better be backed by concrete numbers...

> 
> I would quite like to not guess though - is it possible to use slabtop
> (or any other way) to view ram usage per zone so I can pick out the
> culprit?

/proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
Some of them deal with slabs.

Thanks,
Fengguang

Re: Memory usage per memory zone

[Wu Fengguang]

Hi jack,

On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> Hello,
> 
> I have a box where the oom-killer is killing processes due to running
> out of memory in zone_normal. I can see using slabtop that the inode

How do you know that the memory pressure on zone normal stand out alone?

> caches are using up lots of memory and guess this is the problem, so
> have cleared them using an echo to drop_caches.

It would better be backed by concrete numbers...

> 
> I would quite like to not guess though - is it possible to use slabtop
> (or any other way) to view ram usage per zone so I can pick out the
> culprit?

/proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
Some of them deal with slabs.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> Hi jack,
>
> On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
>> Hello,
>>
>> I have a box where the oom-killer is killing processes due to running
>> out of memory in zone_normal. I can see using slabtop that the inode
>
> How do you know that the memory pressure on zone normal stand out alone?

For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:

kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
active:1304kB inactive:128kB present:901120kB pages_scanned:168951
all_unreclaimable? yes

>> caches are using up lots of memory and guess this is the problem, so
>> have cleared them using an echo to drop_caches.
>
> It would better be backed by concrete numbers...
>
>>
>> I would quite like to not guess though - is it possible to use slabtop
>> (or any other way) to view ram usage per zone so I can pick out the
>> culprit?
>
> /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> Some of them deal with slabs.

Thanks, I'll read up on how to interpret these.

Do you recommend these two files for tracking down memory usage per
process per zone?

Thanks.

Re: Memory usage per memory zone

[jack marrow]

2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> Hi jack,
>
> On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
>> Hello,
>>
>> I have a box where the oom-killer is killing processes due to running
>> out of memory in zone_normal. I can see using slabtop that the inode
>
> How do you know that the memory pressure on zone normal stand out alone?

For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:

kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
active:1304kB inactive:128kB present:901120kB pages_scanned:168951
all_unreclaimable? yes

>> caches are using up lots of memory and guess this is the problem, so
>> have cleared them using an echo to drop_caches.
>
> It would better be backed by concrete numbers...
>
>>
>> I would quite like to not guess though - is it possible to use slabtop
>> (or any other way) to view ram usage per zone so I can pick out the
>> culprit?
>
> /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> Some of them deal with slabs.

Thanks, I'll read up on how to interpret these.

Do you recommend these two files for tracking down memory usage per
process per zone?

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Wu Fengguang]

On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> > Hi jack,
> >
> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> >> Hello,
> >>
> >> I have a box where the oom-killer is killing processes due to running
> >> out of memory in zone_normal. I can see using slabtop that the inode
> >
> > How do you know that the memory pressure on zone normal stand out alone?
> 
> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
> 
> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
> all_unreclaimable? yes

It's normal behavior.  Linux kernel tries hard to utilize most of
the free memory for caching files :)

> >> caches are using up lots of memory and guess this is the problem, so
> >> have cleared them using an echo to drop_caches.
> >
> > It would better be backed by concrete numbers...
> >
> >>
> >> I would quite like to not guess though - is it possible to use slabtop
> >> (or any other way) to view ram usage per zone so I can pick out the
> >> culprit?
> >
> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> > Some of them deal with slabs.
> 
> Thanks, I'll read up on how to interpret these.
> 
> Do you recommend these two files for tracking down memory usage per
> process per zone?

No, the two interfaces provide system wide counters.  We have the well
known tools "ps" and "top" for per-process numbers, hehe.

Thanks,
Fengguang

Re: Memory usage per memory zone

[Wu Fengguang]

On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> > Hi jack,
> >
> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> >> Hello,
> >>
> >> I have a box where the oom-killer is killing processes due to running
> >> out of memory in zone_normal. I can see using slabtop that the inode
> >
> > How do you know that the memory pressure on zone normal stand out alone?
> 
> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
> 
> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
> all_unreclaimable? yes

It's normal behavior.  Linux kernel tries hard to utilize most of
the free memory for caching files :)

> >> caches are using up lots of memory and guess this is the problem, so
> >> have cleared them using an echo to drop_caches.
> >
> > It would better be backed by concrete numbers...
> >
> >>
> >> I would quite like to not guess though - is it possible to use slabtop
> >> (or any other way) to view ram usage per zone so I can pick out the
> >> culprit?
> >
> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> > Some of them deal with slabs.
> 
> Thanks, I'll read up on how to interpret these.
> 
> Do you recommend these two files for tracking down memory usage per
> process per zone?

No, the two interfaces provide system wide counters.  We have the well
known tools "ps" and "top" for per-process numbers, hehe.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
>> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
>> > Hi jack,
>> >
>> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
>> >> Hello,
>> >>
>> >> I have a box where the oom-killer is killing processes due to running
>> >> out of memory in zone_normal. I can see using slabtop that the inode
>> >
>> > How do you know that the memory pressure on zone normal stand out alone?
>>
>> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
>>
>> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
>> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
>> all_unreclaimable? yes
>
> It's normal behavior.  Linux kernel tries hard to utilize most of
> the free memory for caching files :)

With all_unreclaimable = yes is normal?

>
>> >> caches are using up lots of memory and guess this is the problem, so
>> >> have cleared them using an echo to drop_caches.
>> >
>> > It would better be backed by concrete numbers...
>> >
>> >>
>> >> I would quite like to not guess though - is it possible to use slabtop
>> >> (or any other way) to view ram usage per zone so I can pick out the
>> >> culprit?
>> >
>> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
>> > Some of them deal with slabs.
>>
>> Thanks, I'll read up on how to interpret these.
>>
>> Do you recommend these two files for tracking down memory usage per
>> process per zone?
>
> No, the two interfaces provide system wide counters.  We have the well
> known tools "ps" and "top" for per-process numbers, hehe.

ps and top do not have per-zone numbers. How do I get those?

>
> Thanks,
> Fengguang

Re: Memory usage per memory zone

[jack marrow]

2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
>> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
>> > Hi jack,
>> >
>> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
>> >> Hello,
>> >>
>> >> I have a box where the oom-killer is killing processes due to running
>> >> out of memory in zone_normal. I can see using slabtop that the inode
>> >
>> > How do you know that the memory pressure on zone normal stand out alone?
>>
>> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
>>
>> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
>> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
>> all_unreclaimable? yes
>
> It's normal behavior.  Linux kernel tries hard to utilize most of
> the free memory for caching files :)

With all_unreclaimable = yes is normal?

>
>> >> caches are using up lots of memory and guess this is the problem, so
>> >> have cleared them using an echo to drop_caches.
>> >
>> > It would better be backed by concrete numbers...
>> >
>> >>
>> >> I would quite like to not guess though - is it possible to use slabtop
>> >> (or any other way) to view ram usage per zone so I can pick out the
>> >> culprit?
>> >
>> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
>> > Some of them deal with slabs.
>>
>> Thanks, I'll read up on how to interpret these.
>>
>> Do you recommend these two files for tracking down memory usage per
>> process per zone?
>
> No, the two interfaces provide system wide counters.  We have the well
> known tools "ps" and "top" for per-process numbers, hehe.

ps and top do not have per-zone numbers. How do I get those?

>
> Thanks,
> Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Wu Fengguang]

On Wed, Mar 11, 2009 at 02:16:38PM +0200, jack marrow wrote:
> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> > On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
> >> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> >> > Hi jack,
> >> >
> >> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> >> >> Hello,
> >> >>
> >> >> I have a box where the oom-killer is killing processes due to running
> >> >> out of memory in zone_normal. I can see using slabtop that the inode
> >> >
> >> > How do you know that the memory pressure on zone normal stand out alone?
> >>
> >> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
> >>
> >> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
> >> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
> >> all_unreclaimable? yes
> >
> > It's normal behavior.  Linux kernel tries hard to utilize most of
> > the free memory for caching files :)
> 
> With all_unreclaimable = yes is normal?

Ah, perhaps not.

Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
/proc/slabinfo? Thank you.

> >
> >> >> caches are using up lots of memory and guess this is the problem, so
> >> >> have cleared them using an echo to drop_caches.
> >> >
> >> > It would better be backed by concrete numbers...
> >> >
> >> >>
> >> >> I would quite like to not guess though - is it possible to use slabtop
> >> >> (or any other way) to view ram usage per zone so I can pick out the
> >> >> culprit?
> >> >
> >> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> >> > Some of them deal with slabs.
> >>
> >> Thanks, I'll read up on how to interpret these.
> >>
> >> Do you recommend these two files for tracking down memory usage per
> >> process per zone?
> >
> > No, the two interfaces provide system wide counters.  We have the well
> > known tools "ps" and "top" for per-process numbers, hehe.
> 
> ps and top do not have per-zone numbers. How do I get those?

Maybe through the pagemap interface: Documentation/vm/pagemap.txt

Thanks,
Fengguang

Re: Memory usage per memory zone

[Wu Fengguang]

On Wed, Mar 11, 2009 at 02:16:38PM +0200, jack marrow wrote:
> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> > On Wed, Mar 11, 2009 at 01:51:32PM +0200, jack marrow wrote:
> >> 2009/3/11 Wu Fengguang <fengguang.wu@intel.com>:
> >> > Hi jack,
> >> >
> >> > On Wed, Mar 11, 2009 at 11:41:43AM +0100, jack marrow wrote:
> >> >> Hello,
> >> >>
> >> >> I have a box where the oom-killer is killing processes due to running
> >> >> out of memory in zone_normal. I can see using slabtop that the inode
> >> >
> >> > How do you know that the memory pressure on zone normal stand out alone?
> >>
> >> For the normal zone only, I see "all_unreclaimable: yes" and 3 megs of free ram:
> >>
> >> kernel: Normal free:2576kB min:3728kB low:7456kB high:11184kB
> >> active:1304kB inactive:128kB present:901120kB pages_scanned:168951
> >> all_unreclaimable? yes
> >
> > It's normal behavior. A Linux kernel tries hard to utilize most of
> > the free memory for caching files :)
> 
> With all_unreclaimable = yes is normal?

Ah, perhaps not.

Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
/proc/slabinfo? Thank you.

> >
> >> >> caches are using up lots of memory and guess this is the problem, so
> >> >> have cleared them using an echo to drop_caches.
> >> >
> >> > It would better be backed by concrete numbers...
> >> >
> >> >>
> >> >> I would quite like to not guess though - is it possible to use slabtop
> >> >> (or any other way) to view ram usage per zone so I can pick out the
> >> >> culprit?
> >> >
> >> > /proc/zoneinfo and /proc/vmstat do have some per-zone numbers.
> >> > Some of them deal with slabs.
> >>
> >> Thanks, I'll read up on how to interpret these.
> >>
> >> Do you recommend these two files for tracking down memory usage per
> >> process per zone?
> >
> > No, the two interfaces provide system wide counters. A We have the well
> > known tools "ps" and "top" for per-process numbers, hehe.
> 
> ps and top do not have per-zone numbers. How do I get those?

Maybe through the pagemap interface: Documentation/vm/pagemap.txt

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

> Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> /proc/slabinfo? Thank you.

Sure, but I don't know if it will help.

The oom info was from in the night, the rest is from now. I have no zoneinfo.

http://pastebin.com/m67409bc0

Re: Memory usage per memory zone

[jack marrow]

> Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> /proc/slabinfo? Thank you.

Sure, but I don't know if it will help.

The oom info was from in the night, the rest is from now. I have no zoneinfo.

http://pastebin.com/m67409bc0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> > /proc/slabinfo? Thank you.
> 
> Sure, but I don't know if it will help.
> 
> The oom info was from in the night, the rest is from now. I have no zoneinfo.
> 
> http://pastebin.com/m67409bc0

Thank you! So you are running a pretty old kernel?

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> > /proc/slabinfo? Thank you.
> 
> Sure, but I don't know if it will help.
> 
> The oom info was from in the night, the rest is from now. I have no zoneinfo.
> 
> http://pastebin.com/m67409bc0

Thank you! So you are running a pretty old kernel?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

(Didn't reply all:)

2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
>> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
>> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
>> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
>> >> > /proc/slabinfo? Thank you.
>> >>
>> >> Sure, but I don't know if it will help.
>> >>
>> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
>> >>
>> >> http://pastebin.com/m67409bc0
>> >
>> > Thank you! So you are running a pretty old kernel?
>> >
>>
>> Yes. Kernel.2.6.9-78.
>>
>> Added more output from the other oom kills here:
>>  http://pastebin.com/m76fc473d
>>
>> If I could just find a way to find out what is using up all the memory
>> in a zone I could go away happy :)
>
> But the highmem zone wont help you much, since you have a large 900M
> normal zone and a tiny 100M highmem zone ;)
>
> The cached files seem to be the memory killer:

I ran an echo 3 > drop_caches yesterday, I was hoping to come in and
find no oom kill. Oh well :)

I thought the kernel dropped caches if a program needs the ram?

>
> MemTotal:      1034496 kB
> MemFree:         95600 kB
> Buffers:         49916 kB
> Cached:         761544 kB
> SwapCached:          0 kB
> Active:          80484 kB
> Inactive:       749960 kB
> HighTotal:      131008 kB
> HighFree:        68480 kB
> LowTotal:       903488 kB
> LowFree:         27120 kB
> SwapTotal:     2040212 kB
> SwapFree:      2039780 kB
> Dirty:               4 kB
> Writeback:           0 kB
> Mapped:          32636 kB
> Slab:            93856 kB
> CommitLimit:   2557460 kB
> Committed_AS:   129980 kB
> PageTables:       1800 kB
> VmallocTotal:   106488 kB
> VmallocUsed:      3372 kB
> VmallocChunk:   102616 kB
> HugePages_Total:     0
> HugePages_Free:      0
> Hugepagesize:     2048 kB
>
> Is upgrading the kernel an option for you?

No :(

I think shoving some more ram in the box is the best doable option.
Would this help here?

To do that I need to say "look at how much cache we are using for
files, that cache is in the high mem zone (look here) so let's put
some more ram in". Does the cache always live in the high mem zone?

>
> Thanks,
> Fengguang

Re: Memory usage per memory zone

[jack marrow]

(Didn't reply all:)

2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
>> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
>> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
>> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
>> >> > /proc/slabinfo? Thank you.
>> >>
>> >> Sure, but I don't know if it will help.
>> >>
>> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
>> >>
>> >> http://pastebin.com/m67409bc0
>> >
>> > Thank you! So you are running a pretty old kernel?
>> >
>>
>> Yes. Kernel.2.6.9-78.
>>
>> Added more output from the other oom kills here:
>>  http://pastebin.com/m76fc473d
>>
>> If I could just find a way to find out what is using up all the memory
>> in a zone I could go away happy :)
>
> But the highmem zone wont help you much, since you have a large 900M
> normal zone and a tiny 100M highmem zone ;)
>
> The cached files seem to be the memory killer:

I ran an echo 3 > drop_caches yesterday, I was hoping to come in and
find no oom kill. Oh well :)

I thought the kernel dropped caches if a program needs the ram?

>
> MemTotal:      1034496 kB
> MemFree:         95600 kB
> Buffers:         49916 kB
> Cached:         761544 kB
> SwapCached:          0 kB
> Active:          80484 kB
> Inactive:       749960 kB
> HighTotal:      131008 kB
> HighFree:        68480 kB
> LowTotal:       903488 kB
> LowFree:         27120 kB
> SwapTotal:     2040212 kB
> SwapFree:      2039780 kB
> Dirty:               4 kB
> Writeback:           0 kB
> Mapped:          32636 kB
> Slab:            93856 kB
> CommitLimit:   2557460 kB
> Committed_AS:   129980 kB
> PageTables:       1800 kB
> VmallocTotal:   106488 kB
> VmallocUsed:      3372 kB
> VmallocChunk:   102616 kB
> HugePages_Total:     0
> HugePages_Free:      0
> Hugepagesize:     2048 kB
>
> Is upgrading the kernel an option for you?

No :(

I think shoving some more ram in the box is the best doable option.
Would this help here?

To do that I need to say "look at how much cache we are using for
files, that cache is in the high mem zone (look here) so let's put
some more ram in". Does the cache always live in the high mem zone?

>
> Thanks,
> Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 10:48:42AM +0200, jack marrow wrote:
> (Didn't reply all:)
>
> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> > On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
> >> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> >> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> >> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> >> >> > /proc/slabinfo? Thank you.
> >> >>
> >> >> Sure, but I don't know if it will help.
> >> >>
> >> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
> >> >>
> >> >> http://pastebin.com/m67409bc0
> >> >
> >> > Thank you! So you are running a pretty old kernel?
> >> >
> >>
> >> Yes. Kernel.2.6.9-78.
> >>
> >> Added more output from the other oom kills here:
> >>  http://pastebin.com/m76fc473d
> >>
> >> If I could just find a way to find out what is using up all the memory
> >> in a zone I could go away happy :)
> >
> > But the highmem zone wont help you much, since you have a large 900M
> > normal zone and a tiny 100M highmem zone ;)

btw, how much physical memory do you have?
It's weird that meminfo says 1G but Mem-Info says 4G...

> MemTotal:      1034496 kB
> MemFree:         95600 kB
> Buffers:         49916 kB
> Cached:         761544 kB
> SwapCached:          0 kB
> Active:          80484 kB
> Inactive:       749960 kB
> HighTotal:      131008 kB
> HighFree:        68480 kB
> LowTotal:       903488 kB
> LowFree:         27120 kB
> SwapTotal:     2040212 kB
> SwapFree:      2039780 kB

Free pages:       16808kB (1664kB HighMem)
Active:457312 inactive:273805 dirty:0 writeback:0 unstable:0 free:4202 slab:5897 mapped:390707 pagetables:5068
DMA free:12408kB min:64kB low:128kB high:192kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:2736kB min:3728kB low:7456kB high:11184kB active:256kB inactive:16kB present:901120kB pages_scanned:2782 all_unre
claimable? yes
protections[]: 0 0 0
HighMem free:1664kB min:512kB low:1024kB high:1536kB active:1828992kB inactive:1095204kB present:3014656kB pages_scanned:0 al
l_unreclaimable? no
protections[]: 0 0 0
DMA: 4*4kB 5*8kB 2*16kB 1*32kB 4*64kB 2*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12408kB
Normal: 0*4kB 0*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2736kB
HighMem: 320*4kB 0*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1664kB
448110 pagecache pages
Swap cache: add 787243, delete 731019, find 235217/281007, race 0+3
0 bounce buffer pages
Free swap:       1558736kB
983040 pages of RAM
753648 pages of HIGHMEM
9253 reserved pages
253474 pages shared
56224 pages swap cached
Out of Memory: Killed process 28258 (oracle).

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 10:48:42AM +0200, jack marrow wrote:
> (Didn't reply all:)
>
> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> > On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
> >> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> >> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> >> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> >> >> > /proc/slabinfo? Thank you.
> >> >>
> >> >> Sure, but I don't know if it will help.
> >> >>
> >> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
> >> >>
> >> >> http://pastebin.com/m67409bc0
> >> >
> >> > Thank you! So you are running a pretty old kernel?
> >> >
> >>
> >> Yes. Kernel.2.6.9-78.
> >>
> >> Added more output from the other oom kills here:
> >> A http://pastebin.com/m76fc473d
> >>
> >> If I could just find a way to find out what is using up all the memory
> >> in a zone I could go away happy :)
> >
> > But the highmem zone wont help you much, since you have a large 900M
> > normal zone and a tiny 100M highmem zone ;)

btw, how much physical memory do you have?
It's weird that meminfo says 1G but Mem-Info says 4G...

> MemTotal: A  A  A 1034496 kB
> MemFree: A  A  A  A  95600 kB
> Buffers: A  A  A  A  49916 kB
> Cached: A  A  A  A  761544 kB
> SwapCached: A  A  A  A  A 0 kB
> Active: A  A  A  A  A 80484 kB
> Inactive: A  A  A  749960 kB
> HighTotal: A  A  A 131008 kB
> HighFree: A  A  A  A 68480 kB
> LowTotal: A  A  A  903488 kB
> LowFree: A  A  A  A  27120 kB
> SwapTotal: A  A  2040212 kB
> SwapFree: A  A  A 2039780 kB

Free pages:       16808kB (1664kB HighMem)
Active:457312 inactive:273805 dirty:0 writeback:0 unstable:0 free:4202 slab:5897 mapped:390707 pagetables:5068
DMA free:12408kB min:64kB low:128kB high:192kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:2736kB min:3728kB low:7456kB high:11184kB active:256kB inactive:16kB present:901120kB pages_scanned:2782 all_unre
claimable? yes
protections[]: 0 0 0
HighMem free:1664kB min:512kB low:1024kB high:1536kB active:1828992kB inactive:1095204kB present:3014656kB pages_scanned:0 al
l_unreclaimable? no
protections[]: 0 0 0
DMA: 4*4kB 5*8kB 2*16kB 1*32kB 4*64kB 2*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12408kB
Normal: 0*4kB 0*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2736kB
HighMem: 320*4kB 0*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1664kB
448110 pagecache pages
Swap cache: add 787243, delete 731019, find 235217/281007, race 0+3
0 bounce buffer pages
Free swap:       1558736kB
983040 pages of RAM
753648 pages of HIGHMEM
9253 reserved pages
253474 pages shared
56224 pages swap cached
Out of Memory: Killed process 28258 (oracle).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

> btw, how much physical memory do you have?
> It's weird that meminfo says 1G but Mem-Info says 4G...

4 gigs.

# free -m
             total       used       free     shared    buffers     cached
Mem:          3804       1398       2405          0          1        240
-/+ buffers/cache:       1156       2648
Swap:         1992        403       1588

Re: Memory usage per memory zone

[jack marrow]

> btw, how much physical memory do you have?
> It's weird that meminfo says 1G but Mem-Info says 4G...

4 gigs.

# free -m
             total       used       free     shared    buffers     cached
Mem:          3804       1398       2405          0          1        240
-/+ buffers/cache:       1156       2648
Swap:         1992        403       1588

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 10:17:44AM +0200, jack marrow wrote:
> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> > On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
> >> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> >> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> >> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> >> >> > /proc/slabinfo? Thank you.
> >> >>
> >> >> Sure, but I don't know if it will help.
> >> >>
> >> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
> >> >>
> >> >> http://pastebin.com/m67409bc0
> >> >
> >> > Thank you! So you are running a pretty old kernel?
> >> >
> >>
> >> Yes. Kernel.2.6.9-78.
> >>
> >> Added more output from the other oom kills here:
> >>  http://pastebin.com/m76fc473d
> >>
> >> If I could just find a way to find out what is using up all the memory
> >> in a zone I could go away happy :)
> >
> > But the highmem zone wont help you much, since you have a large 900M
> > normal zone and a tiny 100M highmem zone ;)
> >
> > The cached files seem to be the memory killer:
> 
> I ran an echo 3 > drop_caches yesterday, I was hoping to come in and
> find no oom kill. Oh well :)

Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?

> I thought the kernel dropped caches if a program needs the ram?

Sure, but something is unreclaimable... Maybe some process is taking a
lot of shared memory(shm)? What's the output of `lsof`?

> >
> > MemTotal:      1034496 kB
> > MemFree:         95600 kB
> > Buffers:         49916 kB
> > Cached:         761544 kB
> > SwapCached:          0 kB
> > Active:          80484 kB
> > Inactive:       749960 kB
> > HighTotal:      131008 kB
> > HighFree:        68480 kB
> > LowTotal:       903488 kB
> > LowFree:         27120 kB
> > SwapTotal:     2040212 kB
> > SwapFree:      2039780 kB
> > Dirty:               4 kB
> > Writeback:           0 kB
> > Mapped:          32636 kB
> > Slab:            93856 kB
> > CommitLimit:   2557460 kB
> > Committed_AS:   129980 kB
> > PageTables:       1800 kB
> > VmallocTotal:   106488 kB
> > VmallocUsed:      3372 kB
> > VmallocChunk:   102616 kB
> > HugePages_Total:     0
> > HugePages_Free:      0
> > Hugepagesize:     2048 kB
> >
> > Is upgrading the kernel an option for you?
> 
> No :(
> 
> I think shoving some more ram in the box is the best doable option.
> Would this help here?

There have been huge amounts of change sets in mm area since 2.6.9...

> To do that I need to say "look at how much cache we are using for
> files, that cache is in the high mem zone (look here) so let's put
> some more ram in". Does the cache always live in the high mem zone?

Both highmem and normal zones will be used for caches.

Thanks,
Fengguang

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 10:17:44AM +0200, jack marrow wrote:
> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> > On Thu, Mar 12, 2009 at 10:04:17AM +0200, jack marrow wrote:
> >> 2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> >> > On Thu, Mar 12, 2009 at 09:53:27AM +0200, jack marrow wrote:
> >> >> > Can you paste /proc/vmstat, /proc/meminfo, /proc/zoneinfo and
> >> >> > /proc/slabinfo? Thank you.
> >> >>
> >> >> Sure, but I don't know if it will help.
> >> >>
> >> >> The oom info was from in the night, the rest is from now. I have no zoneinfo.
> >> >>
> >> >> http://pastebin.com/m67409bc0
> >> >
> >> > Thank you! So you are running a pretty old kernel?
> >> >
> >>
> >> Yes. Kernel.2.6.9-78.
> >>
> >> Added more output from the other oom kills here:
> >> A http://pastebin.com/m76fc473d
> >>
> >> If I could just find a way to find out what is using up all the memory
> >> in a zone I could go away happy :)
> >
> > But the highmem zone wont help you much, since you have a large 900M
> > normal zone and a tiny 100M highmem zone ;)
> >
> > The cached files seem to be the memory killer:
> 
> I ran an echo 3 > drop_caches yesterday, I was hoping to come in and
> find no oom kill. Oh well :)

Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?

> I thought the kernel dropped caches if a program needs the ram?

Sure, but something is unreclaimable... Maybe some process is taking a
lot of shared memory(shm)? What's the output of `lsof`?

> >
> > MemTotal: A  A  A 1034496 kB
> > MemFree: A  A  A  A  95600 kB
> > Buffers: A  A  A  A  49916 kB
> > Cached: A  A  A  A  761544 kB
> > SwapCached: A  A  A  A  A 0 kB
> > Active: A  A  A  A  A 80484 kB
> > Inactive: A  A  A  749960 kB
> > HighTotal: A  A  A 131008 kB
> > HighFree: A  A  A  A 68480 kB
> > LowTotal: A  A  A  903488 kB
> > LowFree: A  A  A  A  27120 kB
> > SwapTotal: A  A  2040212 kB
> > SwapFree: A  A  A 2039780 kB
> > Dirty: A  A  A  A  A  A  A  4 kB
> > Writeback: A  A  A  A  A  0 kB
> > Mapped: A  A  A  A  A 32636 kB
> > Slab: A  A  A  A  A  A 93856 kB
> > CommitLimit: A  2557460 kB
> > Committed_AS: A  129980 kB
> > PageTables: A  A  A  1800 kB
> > VmallocTotal: A  106488 kB
> > VmallocUsed: A  A  A 3372 kB
> > VmallocChunk: A  102616 kB
> > HugePages_Total: A  A  0
> > HugePages_Free: A  A  A 0
> > Hugepagesize: A  A  2048 kB
> >
> > Is upgrading the kernel an option for you?
> 
> No :(
> 
> I think shoving some more ram in the box is the best doable option.
> Would this help here?

There have been huge amounts of change sets in mm area since 2.6.9...

> To do that I need to say "look at how much cache we are using for
> files, that cache is in the high mem zone (look here) so let's put
> some more ram in". Does the cache always live in the high mem zone?

Both highmem and normal zones will be used for caches.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

> Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?

http://pastebin.com/mce24730

>> I thought the kernel dropped caches if a program needs the ram?
>
> Sure, but something is unreclaimable... Maybe some process is taking a
> lot of shared memory(shm)? What's the output of `lsof`?

I can't paste that, but I expect oracle is using it.

>
>> >
>> > MemTotal:      1034496 kB
>> > MemFree:         95600 kB
>> > Buffers:         49916 kB
>> > Cached:         761544 kB
>> > SwapCached:          0 kB
>> > Active:          80484 kB
>> > Inactive:       749960 kB
>> > HighTotal:      131008 kB
>> > HighFree:        68480 kB
>> > LowTotal:       903488 kB
>> > LowFree:         27120 kB
>> > SwapTotal:     2040212 kB
>> > SwapFree:      2039780 kB
>> > Dirty:               4 kB
>> > Writeback:           0 kB
>> > Mapped:          32636 kB
>> > Slab:            93856 kB
>> > CommitLimit:   2557460 kB
>> > Committed_AS:   129980 kB
>> > PageTables:       1800 kB
>> > VmallocTotal:   106488 kB
>> > VmallocUsed:      3372 kB
>> > VmallocChunk:   102616 kB
>> > HugePages_Total:     0
>> > HugePages_Free:      0
>> > Hugepagesize:     2048 kB
>> >
>> > Is upgrading the kernel an option for you?
>>
>> No :(
>>
>> I think shoving some more ram in the box is the best doable option.
>> Would this help here?
>
> There have been huge amounts of change sets in mm area since 2.6.9...
>
>> To do that I need to say "look at how much cache we are using for
>> files, that cache is in the high mem zone (look here) so let's put
>> some more ram in". Does the cache always live in the high mem zone?
>
> Both highmem and normal zones will be used for caches.
>
> Thanks,
> Fengguang
>
>

Thanks.

Re: Memory usage per memory zone

[jack marrow]

> Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?

http://pastebin.com/mce24730

>> I thought the kernel dropped caches if a program needs the ram?
>
> Sure, but something is unreclaimable... Maybe some process is taking a
> lot of shared memory(shm)? What's the output of `lsof`?

I can't paste that, but I expect oracle is using it.

>
>> >
>> > MemTotal:      1034496 kB
>> > MemFree:         95600 kB
>> > Buffers:         49916 kB
>> > Cached:         761544 kB
>> > SwapCached:          0 kB
>> > Active:          80484 kB
>> > Inactive:       749960 kB
>> > HighTotal:      131008 kB
>> > HighFree:        68480 kB
>> > LowTotal:       903488 kB
>> > LowFree:         27120 kB
>> > SwapTotal:     2040212 kB
>> > SwapFree:      2039780 kB
>> > Dirty:               4 kB
>> > Writeback:           0 kB
>> > Mapped:          32636 kB
>> > Slab:            93856 kB
>> > CommitLimit:   2557460 kB
>> > Committed_AS:   129980 kB
>> > PageTables:       1800 kB
>> > VmallocTotal:   106488 kB
>> > VmallocUsed:      3372 kB
>> > VmallocChunk:   102616 kB
>> > HugePages_Total:     0
>> > HugePages_Free:      0
>> > Hugepagesize:     2048 kB
>> >
>> > Is upgrading the kernel an option for you?
>>
>> No :(
>>
>> I think shoving some more ram in the box is the best doable option.
>> Would this help here?
>
> There have been huge amounts of change sets in mm area since 2.6.9...
>
>> To do that I need to say "look at how much cache we are using for
>> files, that cache is in the high mem zone (look here) so let's put
>> some more ram in". Does the cache always live in the high mem zone?
>
> Both highmem and normal zones will be used for caches.
>
> Thanks,
> Fengguang
>
>

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

>> Sure, but something is unreclaimable... Maybe some process is taking a
>> lot of shared memory(shm)? What's the output of `lsof`?
>
> I can't paste that, but I expect oracle is using it.

Maybe this is helpful:

#  ipcs |grep oracle
0x00000000 2293770    oracle    640        4194304    22
0x00000000 2326539    oracle    640        536870912  22
0x880f3334 2359308    oracle    640        266338304  22
0x0f9b5efc 1933312    oracle    640        44

Re: Memory usage per memory zone

[jack marrow]

>> Sure, but something is unreclaimable... Maybe some process is taking a
>> lot of shared memory(shm)? What's the output of `lsof`?
>
> I can't paste that, but I expect oracle is using it.

Maybe this is helpful:

#  ipcs |grep oracle
0x00000000 2293770    oracle    640        4194304    22
0x00000000 2326539    oracle    640        536870912  22
0x880f3334 2359308    oracle    640        266338304  22
0x0f9b5efc 1933312    oracle    640        44

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 01:48:50PM +0200, jack marrow wrote:
> >> Sure, but something is unreclaimable... Maybe some process is taking a
> >> lot of shared memory(shm)? What's the output of `lsof`?
> >
> > I can't paste that, but I expect oracle is using it.
> 
> Maybe this is helpful:
> 
> #  ipcs |grep oracle
> 0x00000000 2293770    oracle    640        4194304    22
> 0x00000000 2326539    oracle    640        536870912  22
> 0x880f3334 2359308    oracle    640        266338304  22
> 0x0f9b5efc 1933312    oracle    640        44

Up to 800M shm...

http://lwn.net/Articles/286485/
http://feedblog.org/2009/01/25/splitlru-patch-in-kernel-2628-must-have-for-mysql-and-innodb/

which reads: 

        If you’re running MySQL with InnoDB and an in-memory buffer pool, and
        having paging issues, you probably should upgrade to 2.6.28 ASAP.


Thanks,
Fengguang

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 01:48:50PM +0200, jack marrow wrote:
> >> Sure, but something is unreclaimable... Maybe some process is taking a
> >> lot of shared memory(shm)? What's the output of `lsof`?
> >
> > I can't paste that, but I expect oracle is using it.
> 
> Maybe this is helpful:
> 
> #  ipcs |grep oracle
> 0x00000000 2293770    oracle    640        4194304    22
> 0x00000000 2326539    oracle    640        536870912  22
> 0x880f3334 2359308    oracle    640        266338304  22
> 0x0f9b5efc 1933312    oracle    640        44

Up to 800M shm...

http://lwn.net/Articles/286485/
http://feedblog.org/2009/01/25/splitlru-patch-in-kernel-2628-must-have-for-mysql-and-innodb/

which reads: 

        If youa??re running MySQL with InnoDB and an in-memory buffer pool, and
        having paging issues, you probably should upgrade to 2.6.28 ASAP.


Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 01:38:46PM +0200, jack marrow wrote:
> > Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?
> 
> http://pastebin.com/mce24730
> 
> >> I thought the kernel dropped caches if a program needs the ram?
> >
> > Sure, but something is unreclaimable... Maybe some process is taking a
> > lot of shared memory(shm)? What's the output of `lsof`?
> 
> I can't paste that, but I expect oracle is using it.

Oh well...

But from the meminfo, there are 1.2G mapped pages. That could be a big
trouble for reclaiming.  Recent kernels can better handle this situation.

Thanks,
Fengguang
---

# cat /proc/meminfo
MemTotal:      3895404 kB
MemFree:       2472656 kB
Buffers:           412 kB
Cached:         239716 kB
SwapCached:     202652 kB
Active:        1275212 kB
Inactive:        34584 kB
HighTotal:     3014592 kB
HighFree:      1684032 kB
LowTotal:       880812 kB
LowFree:        788624 kB
SwapTotal:     2040212 kB
SwapFree:      1626756 kB
Dirty:             104 kB
Writeback:           0 kB
Mapped:        1247000 kB
Slab:            80040 kB
CommitLimit:   3987912 kB
Committed_AS:  8189040 kB
PageTables:      18792 kB
VmallocTotal:   106488 kB
VmallocUsed:      3072 kB
VmallocChunk:   102980 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

Re: Memory usage per memory zone

[Wu Fengguang]

On Thu, Mar 12, 2009 at 01:38:46PM +0200, jack marrow wrote:
> > Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?
> 
> http://pastebin.com/mce24730
> 
> >> I thought the kernel dropped caches if a program needs the ram?
> >
> > Sure, but something is unreclaimable... Maybe some process is taking a
> > lot of shared memory(shm)? What's the output of `lsof`?
> 
> I can't paste that, but I expect oracle is using it.

Oh well...

But from the meminfo, there are 1.2G mapped pages. That could be a big
trouble for reclaiming.  Recent kernels can better handle this situation.

Thanks,
Fengguang
---

# cat /proc/meminfo
MemTotal:      3895404 kB
MemFree:       2472656 kB
Buffers:           412 kB
Cached:         239716 kB
SwapCached:     202652 kB
Active:        1275212 kB
Inactive:        34584 kB
HighTotal:     3014592 kB
HighFree:      1684032 kB
LowTotal:       880812 kB
LowFree:        788624 kB
SwapTotal:     2040212 kB
SwapFree:      1626756 kB
Dirty:             104 kB
Writeback:           0 kB
Mapped:        1247000 kB
Slab:            80040 kB
CommitLimit:   3987912 kB
Committed_AS:  8189040 kB
PageTables:      18792 kB
VmallocTotal:   106488 kB
VmallocUsed:      3072 kB
VmallocChunk:   102980 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[jack marrow]

2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> On Thu, Mar 12, 2009 at 01:38:46PM +0200, jack marrow wrote:
>> > Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?
>>
>> http://pastebin.com/mce24730
>>
>> >> I thought the kernel dropped caches if a program needs the ram?
>> >
>> > Sure, but something is unreclaimable... Maybe some process is taking a
>> > lot of shared memory(shm)? What's the output of `lsof`?
>>
>> I can't paste that, but I expect oracle is using it.
>
> Oh well...
>
> But from the meminfo, there are 1.2G mapped pages. That could be a big
> trouble for reclaiming.  Recent kernels can better handle this situation.
>
> Thanks,
> Fengguang
> ---
>
> # cat /proc/meminfo
> MemTotal:      3895404 kB
> MemFree:       2472656 kB
> Buffers:           412 kB
> Cached:         239716 kB
> SwapCached:     202652 kB
> Active:        1275212 kB
> Inactive:        34584 kB
> HighTotal:     3014592 kB
> HighFree:      1684032 kB
> LowTotal:       880812 kB
> LowFree:        788624 kB
> SwapTotal:     2040212 kB
> SwapFree:      1626756 kB
> Dirty:             104 kB
> Writeback:           0 kB
> Mapped:        1247000 kB
> Slab:            80040 kB
> CommitLimit:   3987912 kB
> Committed_AS:  8189040 kB
> PageTables:      18792 kB
> VmallocTotal:   106488 kB
> VmallocUsed:      3072 kB
> VmallocChunk:   102980 kB
> HugePages_Total:     0
> HugePages_Free:      0
> Hugepagesize:     2048 kB
>

Thanks for all your help.

Re: Memory usage per memory zone

[jack marrow]

2009/3/12 Wu Fengguang <fengguang.wu@intel.com>:
> On Thu, Mar 12, 2009 at 01:38:46PM +0200, jack marrow wrote:
>> > Can you paste the /proc/meminfo after doing 'echo 3 > drop_caches'?
>>
>> http://pastebin.com/mce24730
>>
>> >> I thought the kernel dropped caches if a program needs the ram?
>> >
>> > Sure, but something is unreclaimable... Maybe some process is taking a
>> > lot of shared memory(shm)? What's the output of `lsof`?
>>
>> I can't paste that, but I expect oracle is using it.
>
> Oh well...
>
> But from the meminfo, there are 1.2G mapped pages. That could be a big
> trouble for reclaiming.  Recent kernels can better handle this situation.
>
> Thanks,
> Fengguang
> ---
>
> # cat /proc/meminfo
> MemTotal:      3895404 kB
> MemFree:       2472656 kB
> Buffers:           412 kB
> Cached:         239716 kB
> SwapCached:     202652 kB
> Active:        1275212 kB
> Inactive:        34584 kB
> HighTotal:     3014592 kB
> HighFree:      1684032 kB
> LowTotal:       880812 kB
> LowFree:        788624 kB
> SwapTotal:     2040212 kB
> SwapFree:      1626756 kB
> Dirty:             104 kB
> Writeback:           0 kB
> Mapped:        1247000 kB
> Slab:            80040 kB
> CommitLimit:   3987912 kB
> Committed_AS:  8189040 kB
> PageTables:      18792 kB
> VmallocTotal:   106488 kB
> VmallocUsed:      3072 kB
> VmallocChunk:   102980 kB
> HugePages_Total:     0
> HugePages_Free:      0
> Hugepagesize:     2048 kB
>

Thanks for all your help.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory usage per memory zone

[Thomas Schoebel-Theuer]

[-- Attachment #1: Type: text/plain, Size: 2098 bytes --]

Am Mittwoch, 11. März 2009 11:41:43 schrieb jack marrow:
> I have a box where the oom-killer is killing processes due to running
> out of memory in zone_normal. I can see using slabtop that the inode
> caches are using up lots of memory and guess this is the problem, so
> have cleared them using an echo to drop_caches.

Hi Jack,

my experience with plain old 2.6.24 on 32bit _production_ boxes was that under 
heavy load and after >30days uptime we saw an sudden inflation of oom-killers 
on some of them until those boxes died. The standard kernel statistics about 
memory looked much the same as yours (and I suspect they could have been 
wrong or at least misleading, but I have neither checked nor tried to fix).

> is it possible to use slabtop
> (or any other way) to view ram usage per zone so I can pick out the
> culprit?

Try the attached experimental hack which can provide you with some insight 
about whats really going on in the _physical_ memory. Since it does not 
allocate any memory for the purpose of displaying those memory patterns it 
wants to examine, you have to allocate a large enough buffer in userspace. 
Don't use cat, but something like dd with parameters such as bs=4M (as 
mentioned in the comment). Probably you have to adjust the patch to some 
newer kernel versions, and/or to fix some sysctl table checks if you want to 
get it upstreams (I will not). And, of course, you can visualize more/other 
flags as well.

After gaining some insight with /proc/sys/vm/mempattern and some development 
of further experimental patches which successfully reduced fragmentation (but 
finally only _delayed_ the oom problems without being able to _fundamentally_ 
resolve them), the ultimate solution was just to use CONFIG_VMSPLIT_2G or 
even CONFIG_VMSPLIT_1G in order to overcome the artificial shortening of 
zone_normal.

This supports old wisdom that an OS cannot give you resources it just does not 
possess... Just make sure you have enough resources for your working set. 
Thats all.

In the hope of being helpful,

Thomas

[-- Attachment #2: fragmentation-visualisation-v2.patch --]
[-- Type: text/x-diff, Size: 5440 bytes --]

diff -urw linux-2.6.24.fragmentation-base/include/linux/mmzone.h linux-2.6.24.fragmentation-info/include/linux/mmzone.h
--- linux-2.6.24.fragmentation-base/include/linux/mmzone.h	2008-09-30 12:50:05.000000000 +0200
+++ linux-2.6.24.fragmentation-info/include/linux/mmzone.h	2009-01-27 14:27:42.000000000 +0100
@@ -687,6 +687,11 @@
 			struct file *, void __user *, size_t *, loff_t *);
 extern char numa_zonelist_order[];
 #define NUMA_ZONELIST_ORDER_LEN 16	/* string buffer size */
+int sysctl_mempattern(struct ctl_table *table, int __user *name, int nlen,
+		      void __user *oldval, size_t __user *oldlenp,
+		      void __user *newval, size_t newlen);
+int mempattern_sysctl_handler(struct ctl_table *, int, struct file *, 
+			      void __user *, size_t *, loff_t *);
 
 #include <linux/topology.h>
 /* Returns the number of the current Node. */
diff -urw linux-2.6.24.fragmentation-base/include/linux/sysctl.h linux-2.6.24.fragmentation-info/include/linux/sysctl.h
--- linux-2.6.24.fragmentation-base/include/linux/sysctl.h	2008-11-12 12:41:47.000000000 +0100
+++ linux-2.6.24.fragmentation-info/include/linux/sysctl.h	2009-01-27 14:05:17.000000000 +0100
@@ -225,6 +225,7 @@
 	VM_PANIC_ON_OOM=33,	/* panic at out-of-memory */
 	VM_VDSO_ENABLED=34,	/* map VDSO into new processes? */
 	VM_MIN_SLAB=35,		 /* Percent pages ignored by zone reclaim */
+	VM_MEMPATTERN=36,	 /* output physical memory patterns */
 };
 
 
diff -urw linux-2.6.24.fragmentation-base/kernel/sysctl.c linux-2.6.24.fragmentation-info/kernel/sysctl.c
--- linux-2.6.24.fragmentation-base/kernel/sysctl.c	2008-11-12 12:41:47.000000000 +0100
+++ linux-2.6.24.fragmentation-info/kernel/sysctl.c	2009-01-27 14:43:08.000000000 +0100
@@ -987,6 +987,13 @@
 		.extra1		= &zero,
 	},
 	{
+		.ctl_name	= VM_MEMPATTERN,
+		.procname	= "mempattern",
+		.mode		= 0444,
+		.proc_handler	= &mempattern_sysctl_handler,
+		.strategy	= &sysctl_mempattern,
+	},
+	{
 		.ctl_name	= VM_PERCPU_PAGELIST_FRACTION,
 		.procname	= "percpu_pagelist_fraction",
 		.data		= &percpu_pagelist_fraction,
diff -urw linux-2.6.24.fragmentation-base/mm/page_alloc.c linux-2.6.24.fragmentation-info/mm/page_alloc.c
--- linux-2.6.24.fragmentation-base/mm/page_alloc.c	2008-09-30 12:50:05.000000000 +0200
+++ linux-2.6.24.fragmentation-info/mm/page_alloc.c	2009-01-29 15:54:31.000000000 +0100
@@ -1947,6 +1947,126 @@
 	show_swap_cache_info();
 }
 
+/* Strategy routine for dumping memory patterns.
+ * It does not allocate kernel memory, but rather copies everything
+ * directly to userspace.
+ * The userspace buffer should be large enough, otherwise output is
+ * truncated.
+ * Use something like "dd if=/proc/sys/vm/mempattern bs=1M"
+ * to reserve such a large buffer in userspace.
+ */
+size_t do_mempattern(void __user *buf, size_t bufsize)
+{
+	struct zone *zone;
+        size_t pos = 0;
+
+#define WRITE_CHAR(out,order,pfn)				\
+        if(bufsize -  pos <= (1 << (order)))			\
+	          goto done;					\
+        if(!((pfn) % 64)) {					\
+	        put_user('\n', (char __user *)(buf + pos));	\
+		pos++;						\
+	}							\
+        put_user((out), (char __user *)(buf + pos));		\
+        pos++;
+
+        //printk("called do_mempattern bufsize=%ld\n", bufsize);
+	for_each_zone(zone) {
+	        size_t len = strlen(zone->name);
+		int pfn;
+
+		if(bufsize -  pos <= len+7)
+		        goto done;
+		if(copy_to_user(buf + pos, "Zone: ", 6))
+		        return -EFAULT;
+		pos += 6;
+		if(copy_to_user(buf + pos, zone->name, len))
+		        return -EFAULT;
+		pos += len;
+		WRITE_CHAR('\n', 0, -1);
+
+		for(pfn = zone->zone_start_pfn; pfn < zone->zone_start_pfn+zone->spanned_pages; pfn++) {
+			struct page *page;
+			int order = 0;
+			int i;
+			char out;
+		        if(!pfn_valid(pfn))
+			        continue;
+			page = pfn_to_page(pfn);
+			if(PageBuddy(page)) {
+			        order = page_order(page);
+				out = 'X';
+			} else if(PageSlab(page)) {
+				out = 's';
+			} else if(PageDirty(page)) {
+				out = '#';
+				if(PageLRU(page)) {
+				        out = '@';
+				}
+			} else if(PageLRU(page)) {
+				out = ':';
+			} else {
+				out = '.';
+			}
+			WRITE_CHAR(out, order, pfn);
+			for(i = 1; i < (1 << order); i++) {
+				pfn++;
+			        WRITE_CHAR('x', 0, pfn);
+			}
+		}
+		WRITE_CHAR('\n', 0, -1);
+	}
+ done:
+	return pos;
+}
+
+int sysctl_mempattern(struct ctl_table *table, int __user *name, int nlen,
+		  void __user *oldval, size_t __user *oldlenp,
+		  void __user *newval, size_t newlen)
+{
+        size_t res;
+        size_t bufsize;
+        printk("called sysctl_mempattern\n");
+
+	if (!table->data || !table->maxlen) 
+		return -ENOTDIR;
+
+	if(!oldval || !oldlenp) {
+	        return 0;
+	}
+
+	if (get_user(bufsize, oldlenp))
+		return -EFAULT;
+
+	res = do_mempattern(oldval, bufsize);
+	if(res >= 0) {
+		if (put_user(res, oldlenp))
+		        return -EFAULT;
+	        return 1;
+	}
+	return res;
+}
+
+int mempattern_sysctl_handler(ctl_table *table, int write, 
+	struct file *file, void __user *buffer, size_t *length, loff_t *ppos)
+{
+        size_t res;
+        //printk("called mempattern_sysctl_handler write=%d length=%ld\n", write, *length);
+	if (!*length || *ppos) {
+	        *length = 0;
+		return 0;
+	}
+	res = do_mempattern(buffer, *length);
+        //printk("res=%ld\n", res);
+	if(res > 0) {
+	        *ppos += res;
+		*length = res;
+		return 0;
+	}
+	return res;
+}
+
+
 /*
  * Builds allocation fallback zone lists.
  *

Re: Memory usage per memory zone

[jack marrow]

2009/3/11 Thomas Schoebel-Theuer <thomas.schoebel-theuer@1und1.de>:
> Am Mittwoch, 11. März 2009 11:41:43 schrieb jack marrow:
>> I have a box where the oom-killer is killing processes due to running
>> out of memory in zone_normal. I can see using slabtop that the inode
>> caches are using up lots of memory and guess this is the problem, so
>> have cleared them using an echo to drop_caches.
>
> Hi Jack,
>
> my experience with plain old 2.6.24 on 32bit _production_ boxes was that under
> heavy load and after >30days uptime we saw an sudden inflation of oom-killers
> on some of them until those boxes died. The standard kernel statistics about
> memory looked much the same as yours (and I suspect they could have been
> wrong or at least misleading, but I have neither checked nor tried to fix).
>
>> is it possible to use slabtop
>> (or any other way) to view ram usage per zone so I can pick out the
>> culprit?
>
> Try the attached experimental hack which can provide you with some insight
> about whats really going on in the _physical_ memory. Since it does not
> allocate any memory for the purpose of displaying those memory patterns it
> wants to examine, you have to allocate a large enough buffer in userspace.
> Don't use cat, but something like dd with parameters such as bs=4M (as
> mentioned in the comment). Probably you have to adjust the patch to some
> newer kernel versions, and/or to fix some sysctl table checks if you want to
> get it upstreams (I will not). And, of course, you can visualize more/other
> flags as well.
>
> After gaining some insight with /proc/sys/vm/mempattern and some development
> of further experimental patches which successfully reduced fragmentation (but
> finally only _delayed_ the oom problems without being able to _fundamentally_
> resolve them), the ultimate solution was just to use CONFIG_VMSPLIT_2G or
> even CONFIG_VMSPLIT_1G in order to overcome the artificial shortening of
> zone_normal.
>
> This supports old wisdom that an OS cannot give you resources it just does not
> possess... Just make sure you have enough resources for your working set.
> Thats all.
>
> In the hope of being helpful,
>
> Thomas
>

Thanks a lot for this Thomas. I'm not able to run a patched kernel on
this box, but I'll keep this in mind for future.