VM - a plenty of inactive memory

VM - a plenty of inactive memory

[Andreas Grimm]

Hello everybody,

i got a weird problem with one of my servers. It's a Intel SR2500AL
with 32GB of RAM.
Looking at the memory usage of the system, something is going totally
wrong. The crucial numbers from /proc/meminfo are:

MemTotal: 33265916 kB
MemFree: 416168 kB
Inactive: 24630428 kB (24GB? whooaaa)

Another system with only 16GB, same amount of users and load, shows a
more normal behaviour:

MemTotal: 16619808 kB
MemFree: 6912676 kB
Inactive: 1774364 kB

Why does the 32GB-System have this plenty of inactive memory. Is there
a way to find out, what the kernel is holding in readiness (that's the
definition of inactive memory afaik)?

OS: SLES 10 SP1
Kernel : 2.6.16.27-0.9-bigsmp

Thanks in advance.

Andreas Grimm

Re: VM - a plenty of inactive memory

[Johannes Weiner]

Hi,

"Andreas Grimm" <agrimm61@gmail.com> writes:

> Hello everybody,
>
> i got a weird problem with one of my servers. It's a Intel SR2500AL
> with 32GB of RAM.
> Looking at the memory usage of the system, something is going totally
> wrong. The crucial numbers from /proc/meminfo are:
>
> MemTotal: 33265916 kB
> MemFree: 416168 kB
> Inactive: 24630428 kB (24GB? whooaaa)
>
> Another system with only 16GB, same amount of users and load, shows a
> more normal behaviour:
>
> MemTotal: 16619808 kB
> MemFree: 6912676 kB
> Inactive: 1774364 kB
>
> Why does the 32GB-System have this plenty of inactive memory. Is there
> a way to find out, what the kernel is holding in readiness (that's the
> definition of inactive memory afaik)?

Inactive pages are marked in use but haven't been touched for some time.
These are candidates for memory reclaiming.

If nothing memory consuming happens, the kswapd should reclaim them back
eventually.

	Hannes

Re: VM - a plenty of inactive memory

[Andreas Grimm]

Hi Johannes,

i know this. But why the kernel locks that memory for a so long time
(2 days now)? Is there a way to enforce the reclaiming? And how can i
find out, which process owns that memory. The problem is, that i can't
accept, that the free memory fell down to 50MB, when i have 24GB in
the nirvana. The system was recently very close to the awkward
situation to swap to disk, and i bet it will do so in the next few
days, because it happened before. Unintelligible, if one got that much
ram.

Bye, Andreas


2008/4/9, Johannes Weiner <hannes@saeurebad.de>:
> Hi,
>
>
>  "Andreas Grimm" <agrimm61@gmail.com> writes:
>
>  > Hello everybody,
>  >
>  > i got a weird problem with one of my servers. It's a Intel SR2500AL
>  > with 32GB of RAM.
>  > Looking at the memory usage of the system, something is going totally
>  > wrong. The crucial numbers from /proc/meminfo are:
>  >
>  > MemTotal: 33265916 kB
>  > MemFree: 416168 kB
>  > Inactive: 24630428 kB (24GB? whooaaa)
>  >
>  > Another system with only 16GB, same amount of users and load, shows a
>  > more normal behaviour:
>  >
>  > MemTotal: 16619808 kB
>  > MemFree: 6912676 kB
>  > Inactive: 1774364 kB
>  >
>  > Why does the 32GB-System have this plenty of inactive memory. Is there
>  > a way to find out, what the kernel is holding in readiness (that's the
>  > definition of inactive memory afaik)?
>
>
> Inactive pages are marked in use but haven't been touched for some time.
>  These are candidates for memory reclaiming.
>
>  If nothing memory consuming happens, the kswapd should reclaim them back
>  eventually.
>
>         Hannes
>

Re: VM - a plenty of inactive memory

[Dan Noe]

Andreas Grimm wrote:
> i know this. But why the kernel locks that memory for a so long time
> (2 days now)? Is there a way to enforce the reclaiming? And how can i
> find out, which process owns that memory. The problem is, that i can't
> accept, that the free memory fell down to 50MB, when i have 24GB in
> the nirvana. The system was recently very close to the awkward
> situation to swap to disk, and i bet it will do so in the next few
> days, because it happened before. Unintelligible, if one got that much
> ram.

Have you investigated the sys.vm.swappiness tunable?  It allows you to 
control how readily the kernel swaps out unused mapped memory.  This 
seems to be relevant to your issue.

See http://lwn.net/Articles/83588/

Cheers,
Dan

-- 
                     /--------------- - -  -  -   -   -
                     |  Daniel Noe
                     |  http://isomerica.net/~dpn/

Re: VM - a plenty of inactive memory

[KOSAKI Motohiro]

Hi

>  i know this. But why the kernel locks that memory for a so long time
>  (2 days now)?

maybe, page cache consume it.
it isn't mean that waste memory.

> Is there a way to enforce the reclaiming?

following command drop all cache. (but I don't reccomend it)

# echo 3 > /proc/sys/vm/drop_caches


>  And how can i
>  find out, which process owns that memory.

please use free command and see second line(+ buffer/cache'ed free mem).

>  The problem is, that i can't
>  accept, that the free memory fell down to 50MB, when i have 24GB in
>  the nirvana. The system was recently very close to the awkward
>  situation to swap to disk, and i bet it will do so in the next few
>  days, because it happened before. Unintelligible, if one got that much
>  ram.

may be, it is not happend forever.

Re: VM - a plenty of inactive memory

[Andreas Grimm]

Hi,

the output of free:
-/+ buffers/cache:   30718212    2547704
Information in frees man page is very sparse, what's the
interpretation? I will try the drop_cache thing in the night.
Thanks!

2008/4/9, KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>:
> Hi
>
>
>  >  i know this. But why the kernel locks that memory for a so long time
>  >  (2 days now)?
>
>
> maybe, page cache consume it.
>  it isn't mean that waste memory.
>
>  > Is there a way to enforce the reclaiming?
>
>  following command drop all cache. (but I don't reccomend it)
>
>  # echo 3 > /proc/sys/vm/drop_caches
>
>
>
>  >  And how can i
>  >  find out, which process owns that memory.
>
>
> please use free command and see second line(+ buffer/cache'ed free mem).
>
>
>  >  The problem is, that i can't
>  >  accept, that the free memory fell down to 50MB, when i have 24GB in
>  >  the nirvana. The system was recently very close to the awkward
>  >  situation to swap to disk, and i bet it will do so in the next few
>  >  days, because it happened before. Unintelligible, if one got that much
>  >  ram.
>
>
> may be, it is not happend forever.
>

Re: VM - a plenty of inactive memory

[Andreas Grimm]

Hi Kosaki,

>  # echo 3 > /proc/sys/vm/drop_caches
>
that frees exactly the 1,5Gigs in the buffers and the cache. Inactive
memory is still at 25GB.
Any other hint?

Andreas

Re: VM - a plenty of inactive memory

[Johannes Weiner]

Hi,

"Andreas Grimm" <agrimm61@gmail.com> writes:

> Hi Kosaki,
>
>>  # echo 3 > /proc/sys/vm/drop_caches
>>
> that frees exactly the 1,5Gigs in the buffers and the cache. Inactive
> memory is still at 25GB.
> Any other hint?

Could you supply the rest of /proc/meminfo?

	Hannes

Re: VM - a plenty of inactive memory

[Andreas Grimm]

Hi Johnannes,

of course:
MemTotal:     33265916 kB
MemFree:        438764 kB
Buffers:         98544 kB
Cached:        1837180 kB
SwapCached:          0 kB
Active:        7086300 kB
Inactive:     25168628 kB
HighTotal:    32631216 kB
HighFree:       429252 kB
LowTotal:       634700 kB
LowFree:          9512 kB
SwapTotal:     2104504 kB
SwapFree:      2104504 kB
Dirty:           14480 kB
Writeback:           0 kB
Mapped:         515524 kB
Slab:           506636 kB
CommitLimit:  18737460 kB
Committed_AS: 16388640 kB
PageTables:      37036 kB
VmallocTotal:   112632 kB
VmallocUsed:     15024 kB
VmallocChunk:    95268 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Here the meminfo from a "normal" system, nearly same amount of users,
and stress, but only 16GB of ram:

MemTotal:     16619808 kB
MemFree:       5497672 kB
Buffers:        422592 kB
Cached:        7018072 kB
SwapCached:          0 kB
Active:        8769608 kB
Inactive:      1979508 kB
HighTotal:    15854036 kB
HighFree:      5484136 kB
LowTotal:       765772 kB
LowFree:         13536 kB
SwapTotal:     4200988 kB
SwapFree:      4200828 kB
Dirty:           16956 kB
Writeback:           0 kB
Mapped:         864284 kB
Slab:           309456 kB
CommitLimit:  12510892 kB
Committed_AS: 14933584 kB
PageTables:      33248 kB
VmallocTotal:   112632 kB
VmallocUsed:      9980 kB
VmallocChunk:   102480 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Thanks,
Andreas

2008/4/9, Johannes Weiner <hannes@saeurebad.de>:
> Hi,
>
>  "Andreas Grimm" <agrimm61@gmail.com> writes:
>
>
> > Hi Kosaki,
>  >
>  >>  # echo 3 > /proc/sys/vm/drop_caches
>  >>
>  > that frees exactly the 1,5Gigs in the buffers and the cache. Inactive
>  > memory is still at 25GB.
>  > Any other hint?
>
>
> Could you supply the rest of /proc/meminfo?
>
>         Hannes
>

Re: VM - a plenty of inactive memory

[Peter Zijlstra]

On Wed, 2008-04-09 at 15:57 +0200, Andreas Grimm wrote:
> Hi Johannes,
> 
> i know this. But why the kernel locks that memory for a so long time
> (2 days now)?

It should not be locked (being on the inactive list does not imply that
in any way). Being on the inactive list just means its first in line to
be looked at when memory needs to be reclaimed.

>  Is there a way to enforce the reclaiming? 

Why would you want to do that for? Would you not rather have a copy of
some page in memory than having to go back to disk to fetch it? Free
memory is a waste, better have something in it that might potentially be
used again.

> And how can i
> find out, which process owns that memory. The problem is, that i can't
> accept, that the free memory fell down to 50MB, when i have 24GB in
> the nirvana. 

See above, free memory is a waste. Really, you don't want 24GB of free
memory.

> The system was recently very close to the awkward
> situation to swap to disk, and i bet it will do so in the next few
> days, because it happened before. Unintelligible, if one got that much
> ram.

If it really would have hit swap then it just means you have a _lot_ of
anonymous memory and little page cache pages on that inactive list.
Check Cached and AnonPages.

Re: VM - a plenty of inactive memory

[Andreas Grimm]

Hi everybody,

i investigated this further. The tunables swappiness, drop_caches etc.
are no options to solve this. The problem is becoming very unpleasent,
because the system isn't able to cache that much:

The 16GB system (expected behaviour):
MemTotal:     16619808 kB
MemFree:       4490032 kB
Cached:        6929448 kB
Inactive:      1670812 kB

The 32GB system :
MemTotal:     33265916 kB
MemFree:        600000 kB
Cached:        1561124 kB
Inactive:     25873128 kB

Don't you think this is insane? Wasting 25GB at the expense of caching
(compare the cached value)? I'm clueless about this issue. Who (which
process) owns that memory? Is there a way to flush this inactive
memory? I assume that this memory hasn't been reclaimed for days,
because the amount of inactive memory is at this level for days now.

All values in /proc/sys/vm are at the defaults again, on both systems.

Any help would be appreciated.

Andreas

2008/4/9, Johannes Weiner <hannes@saeurebad.de>:
> Hi,
>
>
>  "Andreas Grimm" <agrimm61@gmail.com> writes:
>
>  > Hello everybody,
>  >
>  > i got a weird problem with one of my servers. It's a Intel SR2500AL
>  > with 32GB of RAM.
>  > Looking at the memory usage of the system, something is going totally
>  > wrong. The crucial numbers from /proc/meminfo are:
>  >
>  > MemTotal: 33265916 kB
>  > MemFree: 416168 kB
>  > Inactive: 24630428 kB (24GB? whooaaa)
>  >
>  > Another system with only 16GB, same amount of users and load, shows a
>  > more normal behaviour:
>  >
>  > MemTotal: 16619808 kB
>  > MemFree: 6912676 kB
>  > Inactive: 1774364 kB
>  >
>  > Why does the 32GB-System have this plenty of inactive memory. Is there
>  > a way to find out, what the kernel is holding in readiness (that's the
>  > definition of inactive memory afaik)?
>
>
> Inactive pages are marked in use but haven't been touched for some time.
>  These are candidates for memory reclaiming.
>
>  If nothing memory consuming happens, the kswapd should reclaim them back
>  eventually.
>
>         Hannes
>

Re: VM - a plenty of inactive memory

[linux-os (Dick Johnson)]

On Fri, 11 Apr 2008, Andreas Grimm wrote:

> Hi everybody,
>
> i investigated this further. The tunables swappiness, drop_caches etc.
> are no options to solve this. The problem is becoming very unpleasent,
> because the system isn't able to cache that much:
>
> The 16GB system (expected behaviour):
> MemTotal:     16619808 kB
> MemFree:       4490032 kB
> Cached:        6929448 kB
> Inactive:      1670812 kB
>
> The 32GB system :
> MemTotal:     33265916 kB
> MemFree:        600000 kB
> Cached:        1561124 kB
> Inactive:     25873128 kB
>
> Don't you think this is insane? Wasting 25GB at the expense of caching
> (compare the cached value)? I'm clueless about this issue. Who (which
> process) owns that memory? Is there a way to flush this inactive
> memory? I assume that this memory hasn't been reclaimed for days,
> because the amount of inactive memory is at this level for days now.
>
> All values in /proc/sys/vm are at the defaults again, on both systems.
>
> Any help would be appreciated.
>
> Andreas

Wasting 25GB? It looks to me as though the kernel cached
(didn't write pages to disk yet) just about all it needed,
and there is lots of memory that it just doesn't need
  --yet! This means that if you have tasks that need memory
in a hurry, they'll get it without any disk accesses.

The kernel doesn't expand to use all the memory just to
spread itself all over the place. It uses what it needs,
caches some buffered data, and keepts track of all the
memory it has. If you want to use more cache, do '-lsR /'.
That'll read every directory into memory. It probably
won't go away until someody needs more memory.


Cheers,
Dick Johnson
Penguin : Linux version 2.6.22.1 on an i686 machine (5588.29 BogoMips).
My book : http://www.AbominableFirebug.com/
_


****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: VM - a plenty of inactive memory

[Robert Hancock]

Andreas Grimm wrote:
> Hi Johannes,
> 
> i know this. But why the kernel locks that memory for a so long time
> (2 days now)? Is there a way to enforce the reclaiming? And how can i
> find out, which process owns that memory. The problem is, that i can't
> accept, that the free memory fell down to 50MB, when i have 24GB in
> the nirvana. The system was recently very close to the awkward
> situation to swap to disk, and i bet it will do so in the next few
> days, because it happened before. Unintelligible, if one got that much
> ram.
> 
> Bye, Andreas

The memory should not be locked. If the kernel has no reason to reclaim 
the memory, it won't, though.