Re: 2.2/2.4/2.6 VMs: do malloc() ever return NULL?

Richard B. Johnson
2003-11-26T13:04:52Z

On Wed, 26 Nov 2003, Ihar 'Philips' Filipau wrote:

> Richard B. Johnson wrote:
> >
> > As documented, malloc() will never fail as long as there
> > is still address space (not memory) available. This is
> > the required nature of the over-commit strategy. This is
> > necessary because many programs never even touch all the
> > memory they allocate.
> >
>
> We are reading different mans? My man malloc(3) clearly states that
> malloc() can return NULL. (*)
>

Yes. It returns (void *) NULL if it fails to allocate memory. On
many/most/(all?) Unix memory allocators, this failure occurs when
there is no more address-space available (learn about setting the
break address and/or memory-mapping). Expecting malloc() to
know how much free RAM is available is simply wrong. Dynamic
RAM is dynamic. What is available at this instant will be
different during another, especially when tasks can be created
and die thousands of times before you ever even touch that
memory.

> May I ask you one question? Did you were ever doing once graceful
> failure of application under memory pressure? Looks like not.
>

Yes. I'm in the business of making embedded systems that cannot
fail. And they do not fail. They allocate memory once during
startup and they never fail or exit. They also do not use malloc()
but that's not an issue.

> I can guess why sendmail allocates memory it never touches - memory
> pools. There are situations where you really cannot fail - and memory
> allocation failures are really nasty. Do you wanna to lose your e-mails?
> No? So then think twice, while implementing lazy allocators.
>
> So from my tests I see that by default Linux is not safe. You allocate
> memory - malloc() != NULL. Then later you try to write to this memory
> and you get killed by oom_killer. What is the point of this? Your
> reasoning doesn't sound to me.
>

There is MUCH more to designing a "safe" system than having some
'C' runtime library figure out if there will be memory available
next week when your program touches it. You should learn that
malloc() and friends are not kernel functions. The kernel simply
sets the break address when asked by the caller and/or memory-maps
some address space. In both cases the only reason to fail is
the address-space being out-of-range.

You can ask the kernel to check available dunamic RAM during
this procedure, which is what I showed you how to do, but
that is not a very good idea because there will likely be
no RAM available at this instant, but many GB available when
you actually need it.

Note that if you write a byte at offset 0x1zillion, you do NOT
need to allocate 0x1zillion bytes of memory. You only need
one page. That's a major reason why Virtual RAM systems work.

> Memory pools used by applications exactly to make grace error
> handling under memory pressure - but it looks like this stuff under
> Linux gets no testing at all. And default settings could make from
> simple bug complete disaster.
>

Wrong. It is up to the application to allocate and deallocate
dynamic memory properly. FYI, you can always look at /proc/meminfo
yourself instead of expecting malloc() to do it for you. You only
need to look at swap.

[SNIPPED..tirade]

Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (797.90 BogoMips).
Note 96.31% of all statistics are fiction.