Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Ihar 'Philips' Filipau]

Rik van Riel wrote:
> On Tue, 25 Nov 2003, Ihar 'Philips' Filipau wrote:
> 
> 
>>># echo 2 > /proc/sys/vm/overcommit_memory
>>>
>>>Then try again.
>>
>>   What do you know what is not said in docs?
>>   What '2' means?
> 
> Strict non-overcommit mode.  You can allocate as much
> non-file-backed virtual memory as will fit in swap,
> plus /proc/sys/vm/overcommit_percentage worth of memory.
> 

   [ s/overcommit_percentage/overcommit_ratio/ ]

   Thanks! On 2.6 it works as expected. Test with two concurrent memory 
allocations took some time, but both apps stops exactly when memory was 
depleted. Great.

   Did rmap has something todo with this?
   As I see from implementation of do_mmap_pgoff() - it changed from 2.4 
to 2.6 - but there are a lot of common things.
   If I will do dumb back port of this check to 2.4 - do you think it 
will work? 2.4->2.6 memory accounting changed?
   I didn't found this check in your rmap patches for 2.4.22. (btw 
thanks for keeping them up-to-date).

-- 
Ihar 'Philips' Filipau  / with best regards from Saarbruecken.
--                                                           _ _ _
  Because the kernel depends on it existing. "init"          |_|*|_|
  literally _is_ special from a kernel standpoint,           |_|_|*|
  because its' the "reaper of zombies" (and, may I add,      |*|*|*|
  that would be a great name for a rock band).
                                 -- Linus Torvalds

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[William Lee Irwin III]

Rik van Riel wrote:
>> Strict non-overcommit mode.  You can allocate as much
>> non-file-backed virtual memory as will fit in swap,
>> plus /proc/sys/vm/overcommit_percentage worth of memory.

On Wed, Nov 26, 2003 at 11:30:23AM +0100, Ihar 'Philips' Filipau wrote:
>   [ s/overcommit_percentage/overcommit_ratio/ ]
>   Thanks! On 2.6 it works as expected. Test with two concurrent memory 
> allocations took some time, but both apps stops exactly when memory was 
> depleted. Great.
>   Did rmap has something todo with this?
>   As I see from implementation of do_mmap_pgoff() - it changed from 2.4 
> to 2.6 - but there are a lot of common things.
>   If I will do dumb back port of this check to 2.4 - do you think it 
> will work? 2.4->2.6 memory accounting changed?
>   I didn't found this check in your rmap patches for 2.4.22. (btw 
> thanks for keeping them up-to-date).

In principle, non-overcommit shouldn't be dependent on rmap, as it
largely consists of keeping track of the sum of MAP_PRIVATE virtual
mappings' sizes and refusing them when they exceed RAM + swap.

-- wli

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Ihar 'Philips' Filipau]

William Lee Irwin III wrote:
> Rik van Riel wrote:
> 
>>>Strict non-overcommit mode.  You can allocate as much
>>>non-file-backed virtual memory as will fit in swap,
>>>plus /proc/sys/vm/overcommit_percentage worth of memory.
> 
> 
> On Wed, Nov 26, 2003 at 11:30:23AM +0100, Ihar 'Philips' Filipau wrote:
> 
>>  [ s/overcommit_percentage/overcommit_ratio/ ]
>>  Thanks! On 2.6 it works as expected. Test with two concurrent memory 
>>allocations took some time, but both apps stops exactly when memory was 
>>depleted. Great.
>>  Did rmap has something todo with this?
>>  As I see from implementation of do_mmap_pgoff() - it changed from 2.4 
>>to 2.6 - but there are a lot of common things.
>>  If I will do dumb back port of this check to 2.4 - do you think it 
>>will work? 2.4->2.6 memory accounting changed?
>>  I didn't found this check in your rmap patches for 2.4.22. (btw 
>>thanks for keeping them up-to-date).
> 
> 
> In principle, non-overcommit shouldn't be dependent on rmap, as it
> largely consists of keeping track of the sum of MAP_PRIVATE virtual
> mappings' sizes and refusing them when they exceed RAM + swap.
> 

   That's the point of my question. I know a few about MM in Linux. As I 
understand memory accounting is most complicated: 1st how to account 
kernel allocatable memory, 2nd how to reliably calculate already 
allocated memory. (1st looks like not present even in 2.6, 2nd not 
present in 2.4.)

   As I understood, default overcommit_ratio=90% is made especially to 
protect kernel from running out of memory. And 2.6 does offset available 
memory by 3% for all non-root allocation checks.

   But I cannot find any similar accounting stuff in 2.4...
   Hard to draw parallels.

   Will appreciate any advice.

-- 
Ihar 'Philips' Filipau  / with best regards from Saarbruecken.
--                                                           _ _ _
  Because the kernel depends on it existing. "init"          |_|*|_|
  literally _is_ special from a kernel standpoint,           |_|_|*|
  because its' the "reaper of zombies" (and, may I add,      |*|*|*|
  that would be a great name for a rock band).
                                 -- Linus Torvalds

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Ihar 'Philips' Filipau]

Arjan van de Ven wrote:
> 
> that is due to the overcommit policy that your admin has set. 
> You can set it to disabled and then malloc will return NULL in userspace
> 

    Target (patched by mvista) system works as expected in case of 
memory being touch.
    But in case of "for(;;) malloc(N)" it still gets 1.8GB memory 
allocated. (this is ppc32 - looks like 2/2 memory split) So it doesn't 
look like working at all. So basicly pool allocation used in carrier 
grade systems goes south: even with overcommit_memory=-1 && malloc()!=0 
you can not be sure that memory is really allocated. Not good.

    Vanilla 2.4.22 (this is x86) (with HZ=1024, if it does matter).

    after '# echo -1 >/proc/sys/vm/overcommit_memory'
    1. test app with memory touch still gets killed by oom_killer. (so 
no malloc() == NULL)
    2. test app w/o memory touch still can happily allocate 2.8GB of 
memory (x86 - looks like 3/1 memory split) and only then gets NULL 
pointer - oom_killer is silent.

    But thanks for pointers in any way...

-- 
Ihar 'Philips' Filipau  / with best regards from Saarbruecken.
--                                                           _ _ _
  Because the kernel depends on it existing. "init"          |_|*|_|
  literally _is_ special from a kernel standpoint,           |_|_|*|
  because its' the "reaper of zombies" (and, may I add,      |*|*|*|
  that would be a great name for a rock band).
                                 -- Linus Torvalds

2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Ihar 'Philips' Filipau]

[-- Attachment #1: Type: text/plain, Size: 2213 bytes --]

Hello!

   I just wondering: do last three stable branches of LK able to return 
malloc()==NULL and/or ENOMEM?

   2.2: I cannot test this stuff right now - but it was hanging hard on 
"for (;;) memset(malloc(N), 0, N);" So we do not have NULL from malloc().
   2.4: same behaviour if OOM disabled. But by default (OOM even has no 
configuration entry - so always on) it just kills offending process. No 
NULL pointer either.
   2.6: the same as 2.4 with oom killer (default conf). I have no test 
system to check 2.6. w/o oom killer.

   Resume: we malloc() never returns NULL. so man-pages are incorrect ;-)
   [ AFAIK only kmalloc(GFP_ATOMIC) can potentially return NULL - but I 
didn't yet tested my modules under memory pressure. TODO. ]

   Most interesting thing is that on 2.4 "for(;;) malloc(N)" able to 
allocate about 1.8GB of memory on system with only 256M of physical RAM. 
Laghtingly fast. Then malloc() returns NULL. Good signs.
   But as soon as I will put "memset()" -  app will be killed or box 
will go bananas. Neither of this two things are appropriate. Especially 
when applications are tuned to handle memmory allocation errors.

   Can anyone comment on this?

   Does any one has patches with replacement VM? I'm very interesting in 
knowing all aspect of VM<->block layer<->rest of the kernel integration.

   Pointers to on-line memory management algorithms will be valuable 
too, if some one knows any.

P.S. I know about linux-mm.org - out-dated, but I do read it right now. 
Any other pointers will be appreciated.
But looks like general Linux crowd is satisfied with current (and past) 
status of MM...

P.P.S. Small memory eating app is attached. lines 16 & 17 (for() loop) 
used to touch every kbyte of memory to make it really allocated.

-- 
Ihar 'Philips' Filipau  / with best regards from Saarbruecken.
--                                                           _ _ _
  Because the kernel depends on it existing. "init"          |_|*|_|
  literally _is_ special from a kernel standpoint,           |_|_|*|
  because its' the "reaper of zombies" (and, may I add,      |*|*|*|
  that would be a great name for a rock band).
                                 -- Linus Torvalds

[-- Attachment #2: malloc.c --]
[-- Type: text/plain, Size: 368 bytes --]


#include <stdlib.h>
#include <stdio.h>

#define CHUNK_SIZE	(1<<16)

int main()
{
	char *b;
	unsigned long sz = 0;
	for (;;) {
		fprintf(stderr, "alloc ");
		if ((b = malloc( CHUNK_SIZE ))) {
			unsigned int i;
			sz += CHUNK_SIZE;
			for (i=0; i<(CHUNK_SIZE>>10); ++i)
				b[i<<10] = '0';
		} else
			exit(1);
		fprintf(stderr, "done. [%lu%s]\n", sz>>10, "k");
	}
}

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 856 bytes --]

On Tue, 2003-11-25 at 14:27, Ihar 'Philips' Filipau wrote:
> Hello!
> 
>    I just wondering: do last three stable branches of LK able to return 
> malloc()==NULL and/or ENOMEM?
> 
>    2.2: I cannot test this stuff right now - but it was hanging hard on 
> "for (;;) memset(malloc(N), 0, N);" So we do not have NULL from malloc().
>    2.4: same behaviour if OOM disabled. But by default (OOM even has no 
> configuration entry - so always on) it just kills offending process. No 
> NULL pointer either.
>    2.6: the same as 2.4 with oom killer (default conf). I have no test 
> system to check 2.6. w/o oom killer.
> 
>    Resume: we malloc() never returns NULL. so man-pages are incorrect ;-)

that is due to the overcommit policy that your admin has set. 
You can set it to disabled and then malloc will return NULL in userspace



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Rik van Riel]

On Tue, 25 Nov 2003, Ihar 'Philips' Filipau wrote:

>    2.6: the same as 2.4 with oom killer (default conf). I have no test
> system to check 2.6. w/o oom killer.

# echo 2 > /proc/sys/vm/overcommit_memory

Then try again.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Ihar 'Philips' Filipau]

Rik van Riel wrote:
> On Tue, 25 Nov 2003, Ihar 'Philips' Filipau wrote:
> 
>> 2.6: the same as 2.4 with oom killer (default conf). I have no test
>>  system to check 2.6. w/o oom killer.
> 
> 
> # echo 2 > /proc/sys/vm/overcommit_memory
> 
> Then try again.
> 

   What do you know what is not said in docs?
   What '2' means?

   I'll try as soon as I will have again access to 2.6 box.

   this is what 2.6-test10 says:

> overcommit_memory
> 
> This file  contains  one  value.  The following algorithm is used to
> decide if there's enough  memory:  if  the  value of
> overcommit_memory is positive, then there's always  enough  memory.
> This is a useful feature, since programs often malloc() huge  amounts
> of  memory 'just in case', while they only use a small part of  it.
> Leaving  this value at 0 will lead to the failure of such a huge 
> malloc(), when in fact the system has enough memory for the program
> to run.
> 
> On the  other  hand,  enabling this feature can cause you to run out
> of memory and thrash the system to death, so large and/or important
> servers will want to set this value to 0.

   Could this special case 'sysctl_overcommit_memory > 1' be added and 
explained? (mmap.c:589)

   I cannot tell what it does - but name 'security_vm_enough_memory()' 
sounds promising ;-)

   As I have said - I will check this later when I will get to those box.

-- 
Ihar 'Philips' Filipau  / with best regards from Saarbruecken.
--                                                           _ _ _
  Because the kernel depends on it existing. "init"          |_|*|_|
  literally _is_ special from a kernel standpoint,           |_|_|*|
  because its' the "reaper of zombies" (and, may I add,      |*|*|*|
  that would be a great name for a rock band).
                                 -- Linus Torvalds

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Rik van Riel]

On Tue, 25 Nov 2003, Ihar 'Philips' Filipau wrote:

> > # echo 2 > /proc/sys/vm/overcommit_memory
> > 
> > Then try again.
> 
>    What do you know what is not said in docs?
>    What '2' means?

Strict non-overcommit mode.  You can allocate as much
non-file-backed virtual memory as will fit in swap,
plus /proc/sys/vm/overcommit_percentage worth of memory.

>    this is what 2.6-test10 says:

OK, outdated docs.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Chris Wright]

* Ihar 'Philips' Filipau (filia@softhome.net) wrote:
>    I cannot tell what it does - but name 'security_vm_enough_memory()' 
> sounds promising ;-)

This allows a security module to verify that a process can add mapping
for the new pages it's trying to grab.  And can be used to control
overcommit.

thanks,
-chris
-- 
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Richard B. Johnson]

On Tue, 25 Nov 2003, Ihar 'Philips' Filipau wrote:

As documented, malloc() will never fail as long as there
is still address space (not memory) available. This is
the required nature of the over-commit strategy. This is
necessary because many programs never even touch all the
memory they allocate.

You can turn OFF over-commit by doing:

echo "2" >proc/sys/vm/overcommit_memory

However, you will probably find that many programs fail
or seg-fault when normally they wouldn't. So, if you don't
mind restarting sendmail occasionally, then turn off over-commit.


Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (797.90 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Ihar 'Philips' Filipau]

Richard B. Johnson wrote:
> 
> As documented, malloc() will never fail as long as there
> is still address space (not memory) available. This is
> the required nature of the over-commit strategy. This is
> necessary because many programs never even touch all the
> memory they allocate.
> 

   We are reading different mans? My man malloc(3) clearly states that 
malloc() can return NULL. (*)

   May I ask you one question? Did you were ever doing once graceful
failure of application under memory pressure? Looks like not.

   I can guess why sendmail allocates memory it never touches - memory
pools. There are situations where you really cannot fail - and memory
allocation failures are really nasty. Do you wanna to lose your e-mails? 
No? So then think twice, while implementing lazy allocators.

   So from my tests I see that by default Linux is not safe. You allocate
memory - malloc() != NULL. Then later you try to write to this memory
and you get killed by oom_killer. What is the point of this? Your
reasoning doesn't sound to me.

   Memory pools used by applications exactly to make grace error
handling under memory pressure - but it looks like this stuff under
Linux gets no testing at all. And default settings could make from
simple bug complete disaster.

 > You can turn OFF over-commit by doing:
 >
 > echo "2" >proc/sys/vm/overcommit_memory
 >
 > However, you will probably find that many programs fail
 > or seg-fault when normally they wouldn't. So, if you don't
 > mind restarting sendmail occasionally, then turn off over-commit.
 >

   I shall try overcommit_memory == 2 tomorrow and say what I see.

P.S. For example application I have ported right now to kernel space has
a limitiation - it must never ever allocate memory: memory consumption
is known, protocol just have no situation like ENOMEM - it _must_ fail
to initialize on start-up. No - not to being killed by oom_killer in
middle of processing. think carrier grade and/or just good programming
technics.

(*) Great optimization opportunities: remove from all programmes checks 
of the return value if malloc(). As by your words - why not?

-- 
Ihar 'Philips' Filipau  / with best regards from Saarbruecken.
--                                                           _ _ _
  Because the kernel depends on it existing. "init"          |_|*|_|
  literally _is_ special from a kernel standpoint,           |_|_|*|
  because its' the "reaper of zombies" (and, may I add,      |*|*|*|
  that would be a great name for a rock band).
                                 -- Linus Torvalds

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Oliver]

I thought it did return NULL, now...  Before that i didn't check for NULL :)

--Oliver

Ihar 'Philips' Filipau wrote:

> Richard B. Johnson wrote:
> 
>>
>> As documented, malloc() will never fail as long as there
>> is still address space (not memory) available. This is
>> the required nature of the over-commit strategy. This is
>> necessary because many programs never even touch all the
>> memory they allocate.
>>
> 
>   We are reading different mans? My man malloc(3) clearly states that 
> malloc() can return NULL. (*)
> 
>   May I ask you one question? Did you were ever doing once graceful
> failure of application under memory pressure? Looks like not.
> 
>   I can guess why sendmail allocates memory it never touches - memory
> pools. There are situations where you really cannot fail - and memory
> allocation failures are really nasty. Do you wanna to lose your e-mails? 
> No? So then think twice, while implementing lazy allocators.
> 
>   So from my tests I see that by default Linux is not safe. You allocate
> memory - malloc() != NULL. Then later you try to write to this memory
> and you get killed by oom_killer. What is the point of this? Your
> reasoning doesn't sound to me.
> 
>   Memory pools used by applications exactly to make grace error
> handling under memory pressure - but it looks like this stuff under
> Linux gets no testing at all. And default settings could make from
> simple bug complete disaster.
> 
>  > You can turn OFF over-commit by doing:
>  >
>  > echo "2" >proc/sys/vm/overcommit_memory
>  >
>  > However, you will probably find that many programs fail
>  > or seg-fault when normally they wouldn't. So, if you don't
>  > mind restarting sendmail occasionally, then turn off over-commit.
>  >
> 
>   I shall try overcommit_memory == 2 tomorrow and say what I see.
> 
> P.S. For example application I have ported right now to kernel space has
> a limitiation - it must never ever allocate memory: memory consumption
> is known, protocol just have no situation like ENOMEM - it _must_ fail
> to initialize on start-up. No - not to being killed by oom_killer in
> middle of processing. think carrier grade and/or just good programming
> technics.
> 
> (*) Great optimization opportunities: remove from all programmes checks 
> of the return value if malloc(). As by your words - why not?
> 

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Richard B. Johnson]

On Wed, 26 Nov 2003, Ihar 'Philips' Filipau wrote:

> Richard B. Johnson wrote:
> >
> > As documented, malloc() will never fail as long as there
> > is still address space (not memory) available. This is
> > the required nature of the over-commit strategy. This is
> > necessary because many programs never even touch all the
> > memory they allocate.
> >
>
>    We are reading different mans? My man malloc(3) clearly states that
> malloc() can return NULL. (*)
>

Yes. It returns (void *) NULL if it fails to allocate memory. On
many/most/(all?) Unix memory allocators, this failure occurs when
there is no more address-space available (learn about setting the
break address and/or memory-mapping). Expecting malloc() to
know how much free RAM is available is simply wrong. Dynamic
RAM is dynamic. What is available at this instant will be
different during another, especially when tasks can be created
and die thousands of times before you ever even touch that
memory.

>    May I ask you one question? Did you were ever doing once graceful
> failure of application under memory pressure? Looks like not.
>

Yes. I'm in the business of making embedded systems that cannot
fail. And they do not fail. They allocate memory once during
startup and they never fail or exit. They also do not use malloc()
but that's not an issue.

>    I can guess why sendmail allocates memory it never touches - memory
> pools. There are situations where you really cannot fail - and memory
> allocation failures are really nasty. Do you wanna to lose your e-mails?
> No? So then think twice, while implementing lazy allocators.
>
>    So from my tests I see that by default Linux is not safe. You allocate
> memory - malloc() != NULL. Then later you try to write to this memory
> and you get killed by oom_killer. What is the point of this? Your
> reasoning doesn't sound to me.
>

There is MUCH more to designing a "safe" system than having some
'C' runtime library figure out if there will be memory available
next week when your program touches it. You should learn that
malloc() and friends are not kernel functions. The kernel simply
sets the break address when asked by the caller and/or memory-maps
some address space. In both cases the only reason to fail is
the address-space being out-of-range.

You can ask the kernel to check available dunamic RAM during
this procedure, which is what I showed you how to do, but
that is not a very good idea because there will likely be
no RAM available at this instant, but many GB available when
you actually need it.

Note that if you write a byte at offset 0x1zillion, you do NOT
need to allocate 0x1zillion bytes of memory. You only need
one page. That's a major reason why Virtual RAM systems work.

>    Memory pools used by applications exactly to make grace error
> handling under memory pressure - but it looks like this stuff under
> Linux gets no testing at all. And default settings could make from
> simple bug complete disaster.
>

Wrong. It is up to the application to allocate and deallocate
dynamic memory properly.  FYI, you can always look at /proc/meminfo
yourself instead of expecting malloc() to do it for you. You only
need to look at swap.

[SNIPPED..tirade]


Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (797.90 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Ihar 'Philips' Filipau]

Richard B. Johnson wrote:
> 
>>   May I ask you one question? Did you were ever doing once graceful
>>failure of application under memory pressure? Looks like not.
>>
> 
> Yes. I'm in the business of making embedded systems that cannot
> fail. And they do not fail. They allocate memory once during
> startup and they never fail or exit. They also do not use malloc()
> but that's not an issue.
> 

   So what do you use then in user space to reliably allocate memory?

   As to me - memory is a resource. Is it virtual or is it physical - it 
is still resource. And I need to allocate part of this resource.

   malloc() uses brk() inside. But brk() is "implementation details". I 
honestly do not care about them - I just want to be sure that what ever 
resource I have allocated - I can use it afterwards until I shall free 
it. POSIX even doesn't mention brk() BTW.

   If you can hint me any other method to allocate memory without 
surprises - I will really appreciate.

> 
> 
>>   Memory pools used by applications exactly to make grace error
>>handling under memory pressure - but it looks like this stuff under
>>Linux gets no testing at all. And default settings could make from
>>simple bug complete disaster.
>>
> 
> Wrong. It is up to the application to allocate and deallocate
> dynamic memory properly.  FYI, you can always look at /proc/meminfo
> yourself instead of expecting malloc() to do it for you. You only
> need to look at swap.
> 

   Embedded? with swap?!?
   What you have smoken?! - take me to your dealer!-)))

   And btw Rik already gave me answer - 2.6 kernels + 
/proc/sys/vm/overcommit_memory == 2.
   Work charmfully ;-)
   Enjoy.

-- 
Ihar 'Philips' Filipau  / with best regards from Saarbruecken.
--                                                           _ _ _
  Because the kernel depends on it existing. "init"          |_|*|_|
  literally _is_ special from a kernel standpoint,           |_|_|*|
  because its' the "reaper of zombies" (and, may I add,      |*|*|*|
  that would be a great name for a rock band).
                                 -- Linus Torvalds

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[William Lee Irwin III]

On Wed, Nov 26, 2003 at 02:20:58PM +0100, Ihar 'Philips' Filipau wrote:
>   So what do you use then in user space to reliably allocate memory?
>   As to me - memory is a resource. Is it virtual or is it physical - it 
> is still resource. And I need to allocate part of this resource.
>   malloc() uses brk() inside. But brk() is "implementation details". I 
> honestly do not care about them - I just want to be sure that what ever 
> resource I have allocated - I can use it afterwards until I shall free 
> it. POSIX even doesn't mention brk() BTW.
>   If you can hint me any other method to allocate memory without 
> surprises - I will really appreciate.

Non-overcommit is one such method at the kernel level.
mlockall(MCL_CURRENT|MCL_FUTURE) is another (requiring support at both
levels, in addition to administrative grants of privilege).


-- wli

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Ihar 'Philips' Filipau]

William Lee Irwin III wrote:
> On Wed, Nov 26, 2003 at 02:20:58PM +0100, Ihar 'Philips' Filipau wrote:
> 
>>  So what do you use then in user space to reliably allocate memory?
>>  As to me - memory is a resource. Is it virtual or is it physical - it 
>>is still resource. And I need to allocate part of this resource.
>>  malloc() uses brk() inside. But brk() is "implementation details". I 
>>honestly do not care about them - I just want to be sure that what ever 
>>resource I have allocated - I can use it afterwards until I shall free 
>>it. POSIX even doesn't mention brk() BTW.
>>  If you can hint me any other method to allocate memory without 
>>surprises - I will really appreciate.
> 
> 
> Non-overcommit is one such method at the kernel level.
> mlockall(MCL_CURRENT|MCL_FUTURE) is another (requiring support at both
> levels, in addition to administrative grants of privilege).
> 

   "requiring support at both levels" - what do you mean by this?


   In other words, right after I have allocated all memory I need for 
functioning properly, I can call mlockall(MCL_CURRENT) - and memory of 
my app will be guarantied to be present in memory?
   If I have understood from man - it will not be swapped out?
   Yeah... Nice. Cool. I have no swap in any way ;-)
   I do not need that heavy gun: as I have said looser term of memory 
being really allocated before malloc() returns - is enough for me. But 
as I have guessed overcommit_memory doesn't guarantie this either.

   But it looks like this is more appropriate solution for my situation. 
(more apropriate comparing to overcommit_memory)
   In critical pathes we use only pool based allocators - so we can lock 
them in RAM.

   How can I tell the limit of the RAM which can be locked?
   My test had shown that single application can lock 112MB of RAM, but 
fails to lock 128MB of RAM. (I have 256MB phy RAM - We just cannot find 
smaller memory modules on market in any way :-))
   Is it limited to less than half of physical RAM?
   This would be Ok for me in any way.

   ...

   Little bit more test results (2.4.18, 256MB RAM, Motorola's 
PowerQuiccIII 8280):
   overcommit_memory==0 (default): three memory eater apps run ok. 
fourth app which tryes to mlock() /successfully/ allocated 64MB of 
memory hang my box.
   overcommit_memory==-1: three memory eater apps run ok. fourth app 
fails to allocate its memory. All successful memory allocations do mlock 
Okay. As by my incomplete tests.

   That sounds like results ;-)
   Thanks everyone for help and this results!

-- 
Ihar 'Philips' Filipau  / with best regards from Saarbruecken.
--                                                           _ _ _
  Because the kernel depends on it existing. "init"          |_|*|_|
  literally _is_ special from a kernel standpoint,           |_|_|*|
  because its' the "reaper of zombies" (and, may I add,      |*|*|*|
  that would be a great name for a rock band).
                                 -- Linus Torvalds

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[William Lee Irwin III]

On Wed, Nov 26, 2003 at 03:33:06PM +0100, Ihar 'Philips' Filipau wrote:
>   How can I tell the limit of the RAM which can be locked?
>   My test had shown that single application can lock 112MB of RAM, but 
> fails to lock 128MB of RAM. (I have 256MB phy RAM - We just cannot find 
> smaller memory modules on market in any way :-))
>   Is it limited to less than half of physical RAM?
>   This would be Ok for me in any way.

The limit should be available RAM.


-- wli

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Richard B. Johnson]

On Wed, 26 Nov 2003, Ihar 'Philips' Filipau wrote:

> Richard B. Johnson wrote:
> >
> >>   May I ask you one question? Did you were ever doing once graceful
> >>failure of application under memory pressure? Looks like not.
> >>
> >
> > Yes. I'm in the business of making embedded systems that cannot
> > fail. And they do not fail. They allocate memory once during
> > startup and they never fail or exit. They also do not use malloc()
> > but that's not an issue.
> >
>
>    So what do you use then in user space to reliably allocate memory?
>
>    As to me - memory is a resource. Is it virtual or is it physical - it
> is still resource. And I need to allocate part of this resource.
>
>    malloc() uses brk() inside. But brk() is "implementation details". I
> honestly do not care about them - I just want to be sure that what ever
> resource I have allocated - I can use it afterwards until I shall free
> it. POSIX even doesn't mention brk() BTW.
>
>    If you can hint me any other method to allocate memory without
> surprises - I will really appreciate.
>

Here is a dynamic allocation scheme that doesn't fail with
the usual, i.e., less that 1/2 megabyte temporary storage. It
also automatically frees the RAM it's allocated.

int function(void *what, size_t len)
{
    char tmp[len];
    ;;;;;
    return 0;
}

The additional pointer math is wasteful of time. You can just
do:

int function(void *what, size_t len)
{
    char tmp[0x00100000];
}

... and be done with it. That just subtracts 0x00100000 from the
stack-pointer. Simple, quick. Your access should check against
sizeof(tmp).

If you need really large buffers and have only a single
thread, you can still allocate memory at compile-time, i.e.,

char scratch[0x10000000];

Any single-threaded function can use that scratch space.
Note that since scratch[] was not initialized by the compiler
it is put in the ".bss" segment and initialized to zero
when the program is loaded. Therefore, at least when the
program was started, there was sufficient virtual RAM to
allow the entire buffer to be written.

>
>    Embedded? with swap?!?
>    What you have smoken?! - take me to your dealer!-)))
>

Absolutely. A RAM-Disk on non-paged RAM. It allows individual
tasks to keep track of a valuable resource with minimum
overhead. It would be nicer if there was a "get free pages"
function call but you can make a driver for a virtual device
that returns such information. Then you don't need the
RAM disk to keep track of virtual memory

Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (797.90 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Ihar 'Philips' Filipau]

Richard B. Johnson wrote:
> 
> Here is a dynamic allocation scheme that doesn't fail with
> the usual, i.e., less that 1/2 megabyte temporary storage. It
> also automatically frees the RAM it's allocated.
> 
> int function(void *what, size_t len)
> {
>     char tmp[len];
>     ;;;;;
>     return 0;
> }
> 

    You want to say that space is always allocated?
    And how can you make graceful error handling if there is no memory left?
    malloc() has return value, mlock() has return value. So there is at 
least room for error handling. But in your case if you will by mistake 
will run second instance of application - what will happend? I suppose 
crash or welcome oom_killer. Both not nice.

> 
> If you need really large buffers and have only a single
> thread, you can still allocate memory at compile-time, i.e.,
> 
> char scratch[0x10000000];
> 

   I was thinking about this - it has a lot of adavantages and one very 
big disadvantage: you need to recompile app to scale system, you cannot 
make this a command line/configuration parameter.

> 
>>   Embedded? with swap?!?
>>   What you have smoken?! - take me to your dealer!-)))
>>
> 
> Absolutely. A RAM-Disk on non-paged RAM. It allows individual
> tasks to keep track of a valuable resource with minimum
> overhead. It would be nicer if there was a "get free pages"
> function call but you can make a driver for a virtual device
> that returns such information. Then you don't need the
> RAM disk to keep track of virtual memory
> 

   A-ha. A la mtd/rw flash.
   Intersting idea.

   Let's say: I'm working on embedded system - but it is less embedded 
than other embedded systems ;-)

   Thanks for your advices in any way - probably something can be used 
in future.

-- 
Ihar 'Philips' Filipau  / with best regards from Saarbruecken.
--                                                           _ _ _
  Because the kernel depends on it existing. "init"          |_|*|_|
  literally _is_ special from a kernel standpoint,           |_|_|*|
  because its' the "reaper of zombies" (and, may I add,      |*|*|*|
  that would be a great name for a rock band).
                                 -- Linus Torvalds

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[Tim Connors]

"Richard B. Johnson" <root@chaos.analogic.com> said on Tue, 25 Nov 2003 15:17:28 -0500 (EST):
> On Tue, 25 Nov 2003, Ihar 'Philips' Filipau wrote:
> 
> As documented, malloc() will never fail as long as there
> is still address space (not memory) available. This is
> the required nature of the over-commit strategy. This is
> necessary because many programs never even touch all the
> memory they allocate.
> 
> You can turn OFF over-commit by doing:
> 
> echo "2" >proc/sys/vm/overcommit_memory
> 
> However, you will probably find that many programs fail
> or seg-fault when normally they wouldn't. So, if you don't
> mind restarting sendmail occasionally, then turn off over-commit.

I consider this a bug. If they don't use the memory, don't alloc
it. It's not the responsibility of the kernel to determine whether the
programmer was sane. If the programmer was sane, then he may well have
been trying to keep some memory availble for emergency use, and
wouldn't want his program dying from an untrapable kill signal. 

Or he was just lazy, and if he's lazy enough to allocate too much
memory, he'll also be lazy enough to "forget" to check for malloc()s
return value, and hence his program will crash when derefencing
NULL. Bug reports will be filed against his application, like it was
meant to, because it was his fault. Not the kernel's.

Hence 2 should be the default. 

0 should be left for those poor fools who run closed source software,
and can't get their vendor to fix their bugs, so need to use some
kernel kludges (ie, overcommit) to get around it.

-- 
TimC -- http://astronomy.swin.edu.au/staff/tconnors/
"Consider a spherical bear, in simple harmonic motion..."
                -- Professor in the UCB physics department

Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

[William Lee Irwin III]

On Wed, Nov 26, 2003 at 06:31:37PM +1100, Tim Connors wrote:
> Hence 2 should be the default. 
> 0 should be left for those poor fools who run closed source software,
> and can't get their vendor to fix their bugs, so need to use some
> kernel kludges (ie, overcommit) to get around it.

This suggestion has two rather large problems:

(a) Non-overcommit is a useful reliability feature: the VM guarantees
	(well, with 99% probability) it will not be forced to randomly
	kill processes, but instead return -ENOMEM when there isn't
	enough memory.

(b) Once overcommit is enabled, it can't be reliably disabled.


-- wli