Help Understanding Linux memory management

Help Understanding Linux memory management

[Mouawad, Tony]

Hello,

I've been a MS Windows based programmer for a very long time and was
recently tossed in an environment where I am developing embedded apps on
the m68k / Linux platform.  That makes me a Linux newbie.  I started
asking a few questions on various IRC channels and was directed to this
group.  So, I will start asking:

These are questions derived from observations on the MCF 5475 CPU
running Linux 2.6.10 built for the m68k platform.  Disk swapping is
disabled.

1) When physical memory runs low, the memory manager will try to use
memory currently allocated to the pagecache.  Is this true?

2) When vm.overcommit_memory = 2 (overcommit disabled), and memory runs
low, it appears that the memory manager does not try to use memory
currently allocated to pagecache.  Is this true?

3) Is it possible to disable the pagecache?

Cheers,
Tony

Re: Help Understanding Linux memory management

[Robert Hancock]

Mouawad, Tony wrote:
> Hello,
> 
> I've been a MS Windows based programmer for a very long time and was
> recently tossed in an environment where I am developing embedded apps on
> the m68k / Linux platform.  That makes me a Linux newbie.  I started
> asking a few questions on various IRC channels and was directed to this
> group.  So, I will start asking:
> 
> These are questions derived from observations on the MCF 5475 CPU
> running Linux 2.6.10 built for the m68k platform.  Disk swapping is
> disabled.
> 
> 1) When physical memory runs low, the memory manager will try to use
> memory currently allocated to the pagecache.  Is this true?

Yes.

> 2) When vm.overcommit_memory = 2 (overcommit disabled), and memory runs
> low, it appears that the memory manager does not try to use memory
> currently allocated to pagecache.  Is this true?

Shouldn't be, what makes you think this is happening?

> 3) Is it possible to disable the pagecache?

Not as far as I know, it's unlikely this would be of any benefit.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: Help Understanding Linux memory management

[Laurent Pinchart]

Hi,

> I've been a MS Windows based programmer for a very long time and was
> recently tossed in an environment where I am developing embedded apps on
> the m68k / Linux platform.  That makes me a Linux newbie.  I started
> asking a few questions on various IRC channels and was directed to this
> group.  So, I will start asking:
>
> These are questions derived from observations on the MCF 5475 CPU
> running Linux 2.6.10 built for the m68k platform.  Disk swapping is
> disabled.
>
> 1) When physical memory runs low, the memory manager will try to use
> memory currently allocated to the pagecache.  Is this true?
>
> 2) When vm.overcommit_memory = 2 (overcommit disabled), and memory runs
> low, it appears that the memory manager does not try to use memory
> currently allocated to pagecache.  Is this true?
>
> 3) Is it possible to disable the pagecache?

Have you read the "Understanding the Linux Virtual Memory Manager" book by Mel 
Gorman ? It describes the virtual memory manager in details. I was impressed 
by the amount of technical information Mel Gorman gathered and explained 
clearly in a single book. Given the ever-changing nature of the Linux kernel, 
such detailed documentation about core subsystem is often hard to find.

You can get the book at 
http://www.phptr.com/bookstore/product.asp?isbn=0131453483&rl=1#

Laurent Pinchart

RE: Help Understanding Linux memory management

[Mouawad, Tony]

When vm.overcommit_memory = 2 and there appears to be about 2M of memory
readily available and about 12M of memory allocated to pagecache (this
is info gathered from /proc/meminfo) , a call to malloc(5000000) returns
NULL.  I would have expected that somehow, the call to malloc(5000000)
would request the memory in pagecache to be released for use by the app.


-----Original Message-----
From: Robert Hancock [mailto:hancockr@shaw.ca] 
Sent: Tuesday, April 10, 2007 10:15 AM
To: Mouawad, Tony
Cc: linux-kernel@vger.kernel.org
Subject: Re: Help Understanding Linux memory management

Mouawad, Tony wrote:
> Hello,
> 
> I've been a MS Windows based programmer for a very long time and was
> recently tossed in an environment where I am developing embedded apps
on
> the m68k / Linux platform.  That makes me a Linux newbie.  I started
> asking a few questions on various IRC channels and was directed to
this
> group.  So, I will start asking:
> 
> These are questions derived from observations on the MCF 5475 CPU
> running Linux 2.6.10 built for the m68k platform.  Disk swapping is
> disabled.
> 
> 1) When physical memory runs low, the memory manager will try to use
> memory currently allocated to the pagecache.  Is this true?

Yes.

> 2) When vm.overcommit_memory = 2 (overcommit disabled), and memory
runs
> low, it appears that the memory manager does not try to use memory
> currently allocated to pagecache.  Is this true?

Shouldn't be, what makes you think this is happening?

> 3) Is it possible to disable the pagecache?

Not as far as I know, it's unlikely this would be of any benefit.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: Help Understanding Linux memory management

[Alan Cox]

On Tue, 10 Apr 2007 10:31:34 -0400
"Mouawad, Tony" <Tony.Mouawad@christiedigital.com> wrote:

> When vm.overcommit_memory = 2 and there appears to be about 2M of memory
> readily available and about 12M of memory allocated to pagecache (this
> is info gathered from /proc/meminfo) , a call to malloc(5000000) returns
> NULL.  I would have expected that somehow, the call to malloc(5000000)
> would request the memory in pagecache to be released for use by the app.

That can only be done if the memory in the page cache is not being used.
In a system with swap it is possible for anonymous pages to be pushed to
disk, on a system without swap only unchanged disk mapped pages such as
executable progam pages (some of them anyway) can be thrown out if they
are still in use.

More importantly overcommit is address *space* not pages. It deals with
the theoretical worst cases and ensure that a page write won't cause an
out of memory situation. That means it is more pessimistic than the usual
workload. It also by default allows lots of room for worst case kernel
behaviour that is appropriate to a real system with swap but not embedded
so you may well want to tune that via the overcommit ratio proc interface

Alan

RE: Help Understanding Linux memory management

[Mouawad, Tony]

If I understand you correctly, are you suggesting that in an embedded
system where there is no disk swapping, it makes sense to disable the
memory overcommit?

I'm testing a system with overcommit disabled and the overcommit ratio
set to 100%.  I am pleased with the fact that malloc() will actually
return NULL when we are about to exceed the commit limit.  If you look
at COMMITED_AS in /proc/meminfo, it looks to me like it reflects that
amount of memory currently allocated in the system.  I looked into
COMMIT_LIMIT and found that I can raise it to about 120% of physical or
so before the system is really out of memory.  When I load the system, I
see COMMITED_AS actually exceed the amount of physical RAM that I have.
Things appear to work.  My question now is how exactly is COMMITED_AS
calculated?


-----Original Message-----
From: Alan Cox [mailto:alan@lxorguk.ukuu.org.uk] 
Sent: Tuesday, April 10, 2007 12:17 PM
To: Mouawad, Tony
Cc: Robert Hancock; linux-kernel@vger.kernel.org
Subject: Re: Help Understanding Linux memory management

On Tue, 10 Apr 2007 10:31:34 -0400
"Mouawad, Tony" <Tony.Mouawad@christiedigital.com> wrote:

> When vm.overcommit_memory = 2 and there appears to be about 2M of
memory
> readily available and about 12M of memory allocated to pagecache (this
> is info gathered from /proc/meminfo) , a call to malloc(5000000)
returns
> NULL.  I would have expected that somehow, the call to malloc(5000000)
> would request the memory in pagecache to be released for use by the
app.

That can only be done if the memory in the page cache is not being used.
In a system with swap it is possible for anonymous pages to be pushed to
disk, on a system without swap only unchanged disk mapped pages such as
executable progam pages (some of them anyway) can be thrown out if they
are still in use.

More importantly overcommit is address *space* not pages. It deals with
the theoretical worst cases and ensure that a page write won't cause an
out of memory situation. That means it is more pessimistic than the
usual
workload. It also by default allows lots of room for worst case kernel
behaviour that is appropriate to a real system with swap but not
embedded
so you may well want to tune that via the overcommit ratio proc
interface

Alan

RE: Help Understanding Linux memory management

[David Schwartz]

> 1) When physical memory runs low, the memory manager will try to use
> memory currently allocated to the pagecache.  Is this true?

Yes.

> 2) When vm.overcommit_memory = 2 (overcommit disabled), and memory runs
> low, it appears that the memory manager does not try to use memory
> currently allocated to pagecache.  Is this true?

It does try, that doesn't mean it will succeed. If overcommit is disabled,
the OS must have enoug (RAM+swap) to handle the maximum memory consumption
is has allowed to take place.

Perhaps you are laboring under the incorrect assumption that the pagecache
can always be shrunk to zero? Not all the data in the pagecache is
discardable. For example, any page that has been modified from its disk copy
cannot be discarded.

> 3) Is it possible to disable the pagecache?

No, because huge amounts of capability would become impossible. It is not
even clear to me how you could execute self-modifying code without a
pagecache.

DS