Re: Subtle MM bug (really 830MB barrier question)

Re: Subtle MM bug (really 830MB barrier question)

[Dan Maas]

> 08048000-08b5c000 r-xp 00000000 03:05 1130923
/tmp/newmagma/magma.exe.dyn
> 08b5c000-08cc9000 rw-p 00b13000 03:05 1130923
/tmp/newmagma/magma.exe.dyn
> 08cc9000-0bd00000 rwxp 00000000 00:00 0

> Now, subsequent to each memory allocation, only the second number in the
> third line changes.  It becomes 23a78000, then 3b7f0000, and finally
> 3b808000 (after the failed allocation).

OK it's fairly obvious what's happening here. Your program is using its own
allocator, which relies solely on brk() to obtain more memory. On x86 Linux,
brk()-allocated memory (the heap) begins right above the executable and
grows upward - the increasing number you noted above is the top of the heap,
which grows with every brk(). Problem is, the heap can't keep growing
forever - as you discovered, on x86 Linux the upper bound is just below
0x40000000. That boundary is where shared libraries and other memory-mapped
files start to appear.

Note that there is still plenty (~2GB) of address space left, in the region
between the shared libraries and the top of user address space (just under
0xBFFFFFFF). How do you use that space? You need an allocation scheme based
on mmap'ing /dev/zero. As others pointed out, glibc's allocator does just
that.

Here's your short answer: ask the authors of your program to either 1)
replace their custom allocator with regular malloc() or 2) enhance their
custom allocator to use mmap. (or, buy some 64-bit hardware =)...)

Dan


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Subtle MM bug

[Andrea Arcangeli]

On Mon, Jan 08, 2001 at 03:22:44PM -0800, Wayne Whitney wrote:
> I guess I conclude that either (1) MAGMA does not use libc's malloc
> (checking on this, I doubt it) or (2) glibc-2.1.92 knows of these
> variables but has not yet implemented the tuning (I'll try glibc-2.2) or
> (3) this is not the problem.

You should monitor the program with strace while it fails (last few syscalls).
You can breakpoint at exit() and run `cat /proc/pid/maps` to show us the vma
layout of the task. Then we'll see why it's failing.  With CONFIG_1G in 2.2.x
or 2.4.x (confinguration option doesn't matter) you should at least reach
something like 1.5G.

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Subtle MM bug (really 830MB barrier question)

[Wayne Whitney]

On Mon, Jan 08, 2001 at 03:22:44PM -0800, Wayne Whitney wrote:

> I guess I conclude that either (1) MAGMA does not use libc's malloc
> (checking on this, I doubt it)

I'm still a bit unclear on this one.  I now have two executables,
magma.exe and magma.exe.dyn (ignore the .exe).  magma.exe is statically
linked, and magma.exe.dyn is dynamically linked against libc.so.6.  But
the binaries are the same size!  Well 13.7MB and 13.4MB, respectively.

'strings magma.exe' does not mention MALLOC_MMAP_*, so I conclude it is
statically linked against an older libc (libc.so.5?).  Is it possible that
magma.exe.dyn is statically linked against libc.so.5 and dynamically
linked against libc.so.6, so that the older malloc is getting used?  This
would explain the size similarity.

> or (2) glibc-2.1.92 knows of these variables but has not yet
> implemented the tuning (I'll try glibc-2.2) or

I see the same behavior with glibc-2.2 as with glibc-2.1.92.

On Tue, 9 Jan 2001, Andrea Arcangeli wrote:

> You should monitor the program with strace while it fails (last few
> syscalls).

At the very end of this message is the (slightly edited) strace of the
following test magma session:

  Magma V2.7-4      Mon Jan  8 2001 21:27:34 on modular  [Seed = 3551764170]
  Type ? for help.  Type <Ctrl>-D to quit.
  > M1:=MatrixAlgebra(Rationals(),10000)!1;
  > M2:=MatrixAlgebra(Rationals(),10000)!1;
  > M3:=MatrixAlgebra(Rationals(),10000)!1;

  System error: Out of memory.
  All virtual memory has been exhausted so Magma cannot perform this
  statement.

Here I try three times to allocate a 10000x10000 matrix of a 32bit data
type, so each matrix takes up 4e8 bytes.

My limited understanding of the strace output is that magma.exe allocates
memory by calling brk() to increase the size of its data segment, and
brk() returns the new size of the data segment (on complete success, this
is the size requested), but that eventually this sequence fails with:

brk(0x53567c68)                         = 0x3b807d68

> You can breakpoint at exit() and run `cat /proc/pid/maps` to show us
> the vma layout of the task.

I'm not sure how to set a breakpoint, I didn't see anything in the strace
man page about it handling this.  Do I need to use gdb? I tried 'rbreak
exit' and 'rbreak _exit' with gdb, and those didn't work.

But I did check /proc/pid/maps each time I got MAGMA's > prompt.  Here is
the output the first time (before allocating any matrices):

08048000-08b5c000 r-xp 00000000 03:05 1130923    /tmp/newmagma/magma.exe.dyn
08b5c000-08cc9000 rw-p 00b13000 03:05 1130923    /tmp/newmagma/magma.exe.dyn
08cc9000-0bd00000 rwxp 00000000 00:00 0
40000000-40016000 r-xp 00000000 03:05 393301     /lib/ld-2.2.so
40016000-40017000 rw-p 00015000 03:05 393301     /lib/ld-2.2.so
40017000-40018000 rwxp 00000000 00:00 0
40018000-40019000 rw-p 00000000 00:00 0
40024000-40043000 r-xp 00000000 03:05 393307     /lib/libm-2.2.so
40043000-40044000 rw-p 0001e000 03:05 393307     /lib/libm-2.2.so
40044000-40164000 r-xp 00000000 03:05 393304     /lib/libc-2.2.so
40164000-4016a000 rw-p 0011f000 03:05 393304     /lib/libc-2.2.so
4016a000-4016e000 rw-p 00000000 00:00 0
bfffe000-c0000000 rwxp fffff000 00:00 0

Now, subsequent to each memory allocation, only the second number in the
third line changes.  It becomes 23a78000, then 3b7f0000, and finally
3b808000 (after the failed allocation).

Sorry this is a bit long, I wanted to include the full strace output in
case it would allow one to divine what memory allocation scheme this
program is using.  Did I mention that a different mathematics package,
pari (for which I have the source), does not see this 830MB limit?  It
will happily allocate more memory (I haven't checked whether it hits a
limit around 1.5GB).

Thanks again for all the responses, they are quite helpful and educational
and heart-warming!!

Cheers,
Wayne

execve("/tmp/newmagma/magma.exe.dyn", ["/tmp/newmagma/magma.exe.dyn"], [/* 27 vars */]) = 0
uname({sys="Linux", node="modular", ...}) = 0
brk(0)                                  = 0xbce7460
open("/etc/ld.so.preload", O_RDONLY)    = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=45881, ...}) = 0
old_mmap(NULL, 45881, PROT_READ, MAP_PRIVATE, 4, 0) = 0x40018000
close(4)                                = 0
open("/lib/libm.so.6", O_RDONLY)        = 4
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20J\0\000"..., 1024) = 1024
fstat64(4, {st_mode=S_IFREG|0755, st_size=503435, ...}) = 0
old_mmap(NULL, 128760, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0x40024000
mprotect(0x40043000, 1784, PROT_NONE)   = 0
old_mmap(0x40043000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x1e000) = 0x40043000
close(4)                                = 0
open("/lib/libc.so.6", O_RDONLY)        = 4
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0\301\1"..., 1024) = 1024
fstat64(4, {st_mode=S_IFREG|0755, st_size=4851725, ...}) = 0
old_mmap(NULL, 1217864, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0x40044000
mprotect(0x40164000, 38216, PROT_NONE)  = 0
old_mmap(0x40164000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x11f000) = 0x40164000
old_mmap(0x4016a000, 13640, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4016a000
close(4)                                = 0
open("/lib/libc.so.6", O_RDONLY)        = 4
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0\301\1"..., 1024) = 1024
fstat64(4, {st_mode=S_IFREG|0755, st_size=4851725, ...}) = 0
close(4)                                = 0
munmap(0x40018000, 45881)               = 0
getpid()                                = 13699
brk(0)                                  = 0xbce7460
brk(0xbce7468)                          = 0xbce7468
brk(0xbce7568)                          = 0xbce7568
brk(0xbcffc68)                          = 0xbcffc68
time([979006103])                       = 979006103
open("/etc/localtime", O_RDONLY)        = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=1267, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40018000
read(4, "TZif\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0\4\0"..., 4096) = 1267
close(4)                                = 0
munmap(0x40018000, 4096)                = 0
ioctl(0, TIOCGWINSZ, {ws_row=40, ws_col=120, ws_xpixel=1200, ws_ypixel=800}) = 0
rt_sigaction(SIGWINCH, {0x89ecb4c, [WINCH], SA_RESTART|0x4000000}, {SIG_DFL}, 8) = 0
ioctl(0, TIOCGWINSZ, {ws_row=40, ws_col=120, ws_xpixel=1200, ws_ypixel=800}) = 0
rt_sigaction(SIGWINCH, {0x89ecb4c, [WINCH], SA_RESTART|0x4000000}, {0x89ecb4c, [WINCH], SA_RESTART|0x4000000}, 8) = 0
times({tms_utime=1, tms_stime=0, tms_cutime=0, tms_cstime=0}) = 29238685
rt_sigaction(SIGTSTP, {0x89ec6a8, [TSTP], SA_RESTART|0x4000000}, {SIG_DFL}, 8) = 0
getpid()                                = 13699
time(NULL)                              = 979006103
uname({sys="Linux", node="modular", ...}) = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40018000
ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
write(1, "Magma V2.7-4      Mon Jan  8 200"..., 75) = 75
write(1, "Type ? for help.  Type <Ctrl>-D "..., 41) = 41
rt_sigaction(SIGSEGV, {0x857cde8, [SEGV], SA_RESTART|0x4000000}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGINT, {0x857cbf0, [INT], SA_RESTART|0x4000000}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGQUIT, {0x857ccd0, [QUIT], SA_RESTART|0x4000000}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGILL, {0x857cde8, [ILL], SA_RESTART|0x4000000}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGFPE, {0x857cde8, [FPE], SA_RESTART|0x4000000}, {SIG_DFL}, 8) = 0
ioctl(0, TCGETA, {B38400 opost isig icanon echo ...}) = 0
ioctl(0, TCSETA, {B38400 opost isig -icanon -echo ...}) = 0
write(0, "> ", 2)                       = 2

[ I type M1:=MatrixAlgebra(Rationals(),10000)!1; ]

write(0, "\10\10\10\10\10\10\10\10\10\10\10\10\10\10\10\10\10\10"..., 83) = 83
rt_sigaction(SIGALRM, {0x89eb828, [ALRM], SA_RESTART|0x4000000}, {SIG_DFL}, 8) = 0
alarm(1)                                = 0
gettimeofday({979006159, 644571}, {300, 0}) = 0
brk(0x23a77268)                         = 0x23a77268
--- SIGALRM (Alarm clock) ---
ioctl(0, TCSETA, {B38400 opost isig icanon echo ...}) = 0
sigreturn()                             = ? (mask now [])
gettimeofday({979006164, 153555}, {300, 0}) = 0
ioctl(0, TCGETA, {B38400 opost isig icanon echo ...}) = 0
ioctl(0, TCSETA, {B38400 opost isig -icanon -echo ...}) = 0
write(0, "> ", 2)                       = 2

[ I type M2:=MatrixAlgebra(Rationals(),10000)!1; ]

write(0, "\10\10\10\10\10\10\10\10\10\10\10\10\10\10\10\10\10\10"..., 83) = 83
rt_sigaction(SIGALRM, {0x89eb828, [ALRM], SA_RESTART|0x4000000}, {0x89eb828, [ALRM], SA_RESTART|0x4000000}, 8) = 0
alarm(1)                                = 0
brk(0x23a8f968)                         = 0x23a8f968
gettimeofday({979006171, 207167}, {300, 0}) = 0
--- SIGALRM (Alarm clock) ---
ioctl(0, TCSETA, {B38400 opost isig icanon echo ...}) = 0
sigreturn()                             = ? (mask now [])
brk(0x23a77268)                         = 0x23a77268
brk(0x3b7ef668)                         = 0x3b7ef668
gettimeofday({979006178, 520082}, {300, 0}) = 0
ioctl(0, TCGETA, {B38400 opost isig icanon echo ...}) = 0
ioctl(0, TCSETA, {B38400 opost isig -icanon -echo ...}) = 0
write(0, "> ", 2)                       = 2

[ I type M3:=MatrixAlgebra(Rationals(),10000)!1; ]

write(0, "\10\10\10\10\10\10\10\10\10\10\10\10\10\10\10\10\10\10"..., 83) = 83
rt_sigaction(SIGALRM, {0x89eb828, [ALRM], SA_RESTART|0x4000000}, {0x89eb828, [ALRM], SA_RESTART|0x4000000}, 8) = 0
alarm(1)                                = 0
brk(0x3b807d68)                         = 0x3b807d68
gettimeofday({979006180, 783755}, {300, 0}) = 0
--- SIGALRM (Alarm clock) ---
ioctl(0, TCSETA, {B38400 opost isig icanon echo ...}) = 0
sigreturn()                             = ? (mask now [])
brk(0x53567c68)                         = 0x3b807d68
brk(0x53567c68)                         = 0x3b807d68
gettimeofday({979006186, 446393}, {300, 0}) = 0
write(1, "\n", 1)                       = 1
write(1, "System error: Out of memory.\n", 29) = 29
write(1, "All virtual memory has been exha"..., 78) = 78
rt_sigaction(SIGSEGV, {0x857cde8, [SEGV], SA_RESTART|0x4000000}, {0x857cde8, [SEGV], SA_RESTART|0x4000000}, 8) = 0
rt_sigaction(SIGINT, {0x857cbf0, [INT], SA_RESTART|0x4000000}, {0x857cbf0, [INT], SA_RESTART|0x4000000}, 8) = 0
rt_sigaction(SIGQUIT, {0x857ccd0, [QUIT], SA_RESTART|0x4000000}, {0x857ccd0, [QUIT], SA_RESTART|0x4000000}, 8) = 0
rt_sigaction(SIGILL, {0x857cde8, [ILL], SA_RESTART|0x4000000}, {0x857cde8, [ILL], SA_RESTART|0x4000000}, 8) = 0
rt_sigaction(SIGFPE, {0x857cde8, [FPE], SA_RESTART|0x4000000}, {0x857cde8, [FPE], SA_RESTART|0x4000000}, 8) = 0
ioctl(0, TCGETA, {B38400 opost isig icanon echo ...}) = 0
ioctl(0, TCSETA, {B38400 opost isig -icanon -echo ...}) = 0
write(0, "> ", 2)                       = 2
read(0, "\4", 1)                        = 1
write(0, "\n", 1)                       = 1
ioctl(0, TCSETA, {B38400 opost isig icanon echo ...}) = 0
close(0)                                = 0
times({tms_utime=1455, tms_stime=294, tms_cutime=0, tms_cstime=0}) = 29247102
write(1, "\n", 1)                       = 1
write(1, "Total time: 17.479 seconds\n", 27) = 27
munmap(0x40018000, 4096)                = 0
_exit(0)                                = ?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Subtle MM bug (really 830MB barrier question)

[Szabolcs Szakacsits]

On Tue, 9 Jan 2001, Dan Maas wrote:

> OK it's fairly obvious what's happening here. Your program is using
> its own allocator, which relies solely on brk() to obtain more
> memory.
[... good explanation here ...]
> Here's your short answer: ask the authors of your program to either
> 1) replace their custom allocator with regular malloc() or 2) enhance
> their custom allocator to use mmap. (or, buy some 64-bit hardware =)...)

3) ask kernel developers to get rid of this "brk hits the fixed start
address of mmapped areas" or the other way around complaints "mmapped
area should start at lower address" limitation. E.g. Solaris does
growing up heap, growing down mmap and fixed size stack at the top.

Wayne, the patch below should fix your barrier problem [1 GB physical
memory configuration], I used only with 2.2 kernels. Your app should
complain about out of memory around 2.7 GB (0xb0000000-0x08??????),
but note that only 256 MB (0xc0000000-0xb0000000) left for shared
libraries, mmapped areas.

Good luck,

	Szaka

--- linux-2.2.18/include/asm-i386/processor.h  Thu Dec 14 08:20:17 2000
+++ linux/include/asm-i386/processor.h	Tue Jan  9 17:50:49 2001
@@ -166,7 +166,7 @@
 /* This decides where the kernel will search for a free chunk of vm
  * space during mmap's.
  */
-#define TASK_UNMAPPED_BASE	(TASK_SIZE / 3)
+#define TASK_UNMAPPED_BASE	0xb0000000

 /*
  * Size of io_bitmap in longwords: 32 is ports 0-0x3ff.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Subtle MM bug (really 830MB barrier question)

[Wayne Whitney]

On Tue, 9 Jan 2001, Szabolcs Szakacsits wrote:

> Wayne, the patch below should fix your barrier problem [1 GB physical
> memory configuration].

First, I just wanted to thank you and everyone else (Linus, Andrea, Dan
Maas, Rik and others) who has responded to my emails.  You guys are
wonderful!

On Tue, 9 Jan 2001, Dan Maas wrote:

> OK it's fairly obvious what's happening here. Your program is using
> its own allocator, which relies solely on brk() to obtain more memory.

OK, the statically linked binary I have produces a simpler /proc/pid/maps,
here it is (before I actually try to create any big objects in memory):

08048000-08afb000 r-xp 00000000 03:07 64318      /usr/local/magma-2.7/magma.exe
08afb000-08c3e000 rw-p 00ab2000 03:07 64318      /usr/local/magma-2.7/magma.exe
08c3e000-0bc54000 rwxp 00000000 00:00 0
40000000-40001000 rw-p 00000000 00:00 0
bfffd000-c0000000 rwxp ffffe000 00:00 0

If I understand correctly, the first two lines are the executable
(although I don't know why it shows up twice), the third line is the heap
for this program, the fourth line is where mmap stuff starts and the fifth
line is the boundary between the process address space and the kernel
address space.

First question:  for this statically linked binary, nothing is really
being mmap'ed, is there any way that I can arrange, for this process only,
to get rid of the fourth line?  This would be the ideal solution.

Szabolcs's suggestion (and Mark Hahn's privately, as well) of modifying
TASK_UNMAPPED_BASE does work for me.  Unfortunately, on the same machine
I'd like to both run programs that use brk() allocation and that use
mmap() allocation, so the best I can do is change TASK_UNMAPPED_BASE to
1.5GB from 1GB, this allows a bit under 1.5GB for brk() and 1.5GB() for
mmap().

Second question:  if I understand correctly, the start of the kernel
process space is controlled by PAGE_OFFSET, and under CONFIG_NOHIGHMEM,
the kernel maps all the physical RAM into its address space.  I guess
128MB of this space is used for the 2.4.0 kernel itself, so that the
maximum physical RAM under 2.4.0 with PAGE_OFFSET set to 3GB is 896MB.
One of my machines has 1024MB of RAM, so can I just decrease PAGE_OFFEST
by 128MB to be able to use all of it?

Thanks again,
Wayne





-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Subtle MM bug (really 830MB barrier question)

[Wayne Whitney]

On Tue, 9 Jan 2001, Szabolcs Szakacsits wrote:

> 3) ask kernel developers to get rid of this "brk hits the fixed start
> address of mmapped areas" or the other way around complaints "mmapped
> area should start at lower address" limitation. E.g. Solaris does
> growing up heap, growing down mmap and fixed size stack at the top.

OK, despite knowing nothing of the kernel internals, I looked at doing
this myself :-)

I notice that TASK_UNMAPPED_BASE is only used in get_unmapped_area() in
mm/mmap.c, which is encouraging.  Moreover, get_unmapped_area() is only
called once in mm/mmap.c and once in mm/mremap.c.  So I think I would only
have to change get_unmapped_area() to get the desired effect, and this
change should not affect anything else.

If no address is specified, get_unmapped_area() currently chooses the
first (large enough, unused) region above TASK_UNMAPPED_BASE.  I guess I
would just have to define something like TASK_UNMAPPED_CEILING and arrange
for get_unmapped_area() to allocate the first region below
TASK_UNMAPPED_CEILING.  And I guess TASK_UNMAPPED_CEILING should equal
TASK_SIZE - maximum stack size.  What is the maximum stack size?  I
couldn't quite figure this out myself.

Am I missing something, or should choosing an appropriate value for
TASK_UNMAPPED_CEILING and changing get_unmapped_area() be sufficient?

Cheers,
Wayne






-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/