[2.4.26] overcommit_memory documentation clarification

[2.4.26] overcommit_memory documentation clarification

[Vincent Lefevre]

Hi,

The documentation of overcommit_memory in Documentation/sysctl/vm.txt
should be clarified, as with the following simple program, malloc()
never returns 0 on an official 2.4.26 kernel, even if overcommit_memory
has been set to 0. Running it has the effect of having random processes
killed, and eventually this process itself.

/* $Id: malloc.c 2753 2004-03-16 15:23:09Z lefevre $
 *
 * malloc() test
 */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define ONEMB 1048576

int main (void)
{
  char *p;
  int i;

  for (i = 1; (p = malloc(ONEMB)) != NULL; i++)
    {
      printf ("Got %d MB\n", i);
      memset (p, 0, ONEMB);
    }
  printf ("malloc() failed - OK\n");
  return 0;
}

(After some discussions, it appears not to be a bug in the glibc
<http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=247300>.)

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: [2.4.26] overcommit_memory documentation clarification

[Richard B. Johnson]

On Sun, 9 May 2004, Vincent Lefevre wrote:

> Hi,
>
> The documentation of overcommit_memory in Documentation/sysctl/vm.txt
> should be clarified, as with the following simple program, malloc()
> never returns 0 on an official 2.4.26 kernel, even if overcommit_memory
> has been set to 0. Running it has the effect of having random processes
> killed, and eventually this process itself.

What made you think that malloc would return 0 if the system
was "out of memory??" Malloc will return NULL, which is not 0 BTW,
if you are out of address-space or have corrupted it by writing
past a previous allocation. Malloc's return value is a void *. It
should be compared against NULL, not zero.

When malloc() needs new "memory". It just asks the kernel to
set the new break address or, in the case of mmap() mallocs, asks
to extend a mapped region. Until somebody actually uses those
regions, you haven't used any memory. So there is no way for
malloc() to "know" ahead of time.

If you run a malloc() bomb from the root account you should
end up killing off a lot of processes. If you run it from
a normal user account, and you have set the user's resource
quotas properly, only the user should get into trouble.


Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5557.45 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: [2.4.26] overcommit_memory documentation clarification

[Vincent Lefevre]

On 2004-05-08 22:04:30 -0400, Richard B. Johnson wrote:
> What made you think that malloc would return 0 if the system
> was "out of memory??" Malloc will return NULL, which is not 0 BTW,
> if you are out of address-space or have corrupted it by writing
> past a previous allocation. Malloc's return value is a void *. It
> should be compared against NULL, not zero.

You are wrong. They are the same value (this is required by the ISO C
standard), i.e. both NULL == (void *) 0 and NULL == 0 must be true.

> When malloc() needs new "memory". It just asks the kernel to
> set the new break address or, in the case of mmap() mallocs, asks
> to extend a mapped region. Until somebody actually uses those
> regions, you haven't used any memory. So there is no way for
> malloc() to "know" ahead of time.

Again, you are wrong. The goal of malloc is to reserve memory.
This can be seen as used memory. If the implementation behaves
differently, then it is broken.

> If you run a malloc() bomb from the root account you should
> end up killing off a lot of processes. If you run it from
> a normal user account, and you have set the user's resource
> quotas properly, only the user should get into trouble.

The quotas are set properly (i.e. there are no quotas).

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: [2.4.26] overcommit_memory documentation clarification

[Paul Jackson]

Vincent wrote:
> NULL == (void *) 0 and NULL == 0 must be true
Yes - NULL is compares equal to both (void *)0 and 0.
No - not necessarily the _same_ value - one could be
on a system with 32 bit ints, 64 bit pointers, for example.

> The goal of malloc is to reserve memory.
It's up to the kernel whether sbrk (used by malloc to
obtain virtual address space) reserves memory or not.

Check out:
    /proc/sys/vm/overcommit_memory
    Documentation/sysctl/vm.txt - overcommit_memory

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: [2.4.26] overcommit_memory documentation clarification

[Vincent Lefevre]

On 2004-05-09 14:06:11 -0700, Paul Jackson wrote:
> Vincent wrote:
> > NULL == (void *) 0 and NULL == 0 must be true
> Yes - NULL is compares equal to both (void *)0 and 0.
> No - not necessarily the _same_ value - one could be
> on a system with 32 bit ints, 64 bit pointers, for example.

And so?

> > The goal of malloc is to reserve memory.
> It's up to the kernel whether sbrk (used by malloc to
> obtain virtual address space) reserves memory or not.

More old_mmap than brk (BTW, I forgot to say that this was on
an x86 machine, I don't know if this matters...).

> Check out:
>     /proc/sys/vm/overcommit_memory
>     Documentation/sysctl/vm.txt - overcommit_memory

But the documentation is wrong (on an official 2.4.26 kernel).
It seems that there is no way to get malloc() always return 0
when there isn't enough memory, even in simple cases (see my
program posted in the first message).

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: [2.4.26] overcommit_memory documentation clarification

[Marcelo Tosatti]

On Sun, May 09, 2004 at 11:49:41PM +0200, Vincent Lefevre wrote:
> On 2004-05-09 14:06:11 -0700, Paul Jackson wrote:
> > Vincent wrote:
> > > NULL == (void *) 0 and NULL == 0 must be true
> > Yes - NULL is compares equal to both (void *)0 and 0.
> > No - not necessarily the _same_ value - one could be
> > on a system with 32 bit ints, 64 bit pointers, for example.
> 
> And so?
> 
> > > The goal of malloc is to reserve memory.
> > It's up to the kernel whether sbrk (used by malloc to
> > obtain virtual address space) reserves memory or not.
> 
> More old_mmap than brk (BTW, I forgot to say that this was on
> an x86 machine, I don't know if this matters...).
> 
> > Check out:
> >     /proc/sys/vm/overcommit_memory
> >     Documentation/sysctl/vm.txt - overcommit_memory
> 
> But the documentation is wrong (on an official 2.4.26 kernel).
> It seems that there is no way to get malloc() always return 0
> when there isn't enough memory, even in simple cases (see my
> program posted in the first message).

Right. 

We should or merge Alan's strict-overcommit patches (from RH's tree), 
or fix the documentation.

Marc-Christian Petersen has a patch to fixup the documentation.

Re: [2.4.26] overcommit_memory documentation clarification

[Marc-Christian Petersen]

On Thursday 27 May 2004 14:20, Marcelo Tosatti wrote:

Hi Marcelo,

> We should or merge Alan's strict-overcommit patches (from RH's tree),
> or fix the documentation.

I vote for the strict-overcommit thing. Do you have that handy for your 
2.4-bk?

Alan, prolly you? Or do we have to extract it from 
linux-2.4.21-selected-ac-bits.patch? ;)

ciao, Marc

Re: [2.4.26] overcommit_memory documentation clarification

[Marcelo Tosatti]

On Thu, May 27, 2004 at 02:30:11PM +0200, Marc-Christian Petersen wrote:
> On Thursday 27 May 2004 14:20, Marcelo Tosatti wrote:
> 
> Hi Marcelo,

Hi Marc, 

> > We should or merge Alan's strict-overcommit patches (from RH's tree),
> > or fix the documentation.
> 
> I vote for the strict-overcommit thing. Do you have that handy for your 
> 2.4-bk?
> 
> Alan, prolly you? Or do we have to extract it from 
> linux-2.4.21-selected-ac-bits.patch? ;)

Alan is busy with other things.

The strict overcommit fixes need to be extraced and tested.

Dave Jones told me about it the other day.

Still haven't found the time to download RH's srpm.

Re: [2.4.26] overcommit_memory documentation clarification

[Dave Jones]

On Thu, May 27, 2004 at 10:09:25AM -0300, Marcelo Tosatti wrote:

 > > > We should or merge Alan's strict-overcommit patches (from RH's tree),
 > > > or fix the documentation.
 > > I vote for the strict-overcommit thing. Do you have that handy for your 
 > > 2.4-bk?
 > > Alan, prolly you? Or do we have to extract it from 
 > > linux-2.4.21-selected-ac-bits.patch? ;)
 > 
 > Alan is busy with other things.
 > The strict overcommit fixes need to be extraced and tested.
 > Dave Jones told me about it the other day.
 > Still haven't found the time to download RH's srpm.

The overcommit bits in the Fedora SRPM come directly from the
2.4.22-ac1 patch.  You're better off just grabbing  Bero's -pac patch.
He has a 2.4.26 patch, which should save you rediffing,
just chop out the relevant bits.

http://kernel.org/pub/linux/kernel/people/bero/2.4/2.4.27/

		Dave

Re: [2.4.26] overcommit_memory documentation clarification

[Alan Cox]

On Thu, May 27, 2004 at 10:09:25AM -0300, Marcelo Tosatti wrote:
> The strict overcommit fixes need to be extraced and tested.

Bero's tree has them. If you merge it it is probably good to make
the small changes so it works like 2.6