Re: mmap'ed memory in core files ?

Rob Mueller
2008-07-02T06:34:29Z

>> This is particularly interesting on a 64 bit kernel where a bug in
>> your code causes you to try to read something about 2Gb into your
>> alleged mmaped file (actual size ~500 bytes) and the segfault causes
>> a coredump.
>
> Do you have a ssimple example program for this?

Trying to reproduce the problem, I think it's actually related to sparse
files.

$ cat a.c
#include <stdlib.h>
int main() {
unsigned int i;
char * a = malloc(3000000000u);
*(int *)0 = 0;
}
$ gcc a.c
$ ulimit -c 10240 && ./a.out
$ ls -l
-rw------- 1 root root 3000082432 Jul 2 02:23 core.7761
$

It's clearly sparse, but slightly unintuitive that the ulimit doesn't
actually limit the filesize, just the size of the data written to the file.

If I change the code to include this line after the malloc():

for (i = 0; i < 3000000000u; i++) a[i] = i % 256;

I get:

-rw------- 1 root root 10485760 Jul 2 02:25 core.8992

More what you'd expect.

One interesting side effect of running a 64-bit kernel + 32-bit userland is
that previously bugs that might have previously caused malloc() to fail (eg
underflowing integer to a huge value), now succeed and allocate a huge chunk
of memory rather than failing and causing the program to bailout/crash on
dereference.

Rob