mmap breakage

mmap breakage

[Jurij Smakov]

Hello,

In Debian we have recently received a number of bug reports [0,1] about 
glibc's ld-linux.so.2 failing (when called via ldd, for example) on some 
binaries. The minimum test case, originally mentioned in [0], was the 
following:

char a[134084860];
int main() { return 0; }

Running the compiled binary under strace on my Ultra1 with 2.6.8 
(same issue is expected to be present for 2.4.x) kernel results in

kundera:~# strace /usr/lib/debug/ld-linux.so.2 ./a.out
execve("/usr/lib/debug/ld-linux.so.2", ["/usr/lib/debug/ld-linux.so.2", "./a.out"], [/* 22 vars */]) = 0
uname({sys="Linux", node="kundera", ...}) = 0
brk(0)                                  = 0x802a000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("./a.out", O_RDONLY)               = 3
read(3, "\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\2\0\2\0\0\0\1\0\1\3P"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size\x19133, ...}) = 0
getcwd("/root", 128)                    = 6
mmap(0x10000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0x10000
mmap(0x20000, 16384, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0x20000
mmap(0x24000, 134077000, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x24000
close(3)                                = 0
open("/etc/ld.so.preload", O_RDONLY)    = -1 ENOENT (No such file or directory)
--- SIGILL (Illegal instruction) @ 0 (0) ---
+++ killed by SIGILL +++

After some debugging I was able to trace the problem to the last mmap 
syscall. It actually manages to overwrite a portion of the ld-linux.so.2 
code, causing the SIGILL. Presumably, this happens due to rounding of the 
size of mmaped chunk upwards to the next full page of 8K, which
is performed in sys_mmap (arch/sparc64/kernel/sys_sparc.c). In my case 
that increased the size of mmapped region from 134077000 to 134078464, so 
that the occupied region of memory extended all the way to 0x8002000
(0x24000 + 134078464). However, according to gdb, the function 
_dl_start_user (where SIGILL actually happens) have been previously 
located at the address 0x8001cac, and so was obliterated by this syscall.

Unfortunately, I don't have a clear idea on how the mmap's memory 
management is supposed to work, so I would appreciate any clues, which 
could lead to a solution.

[0] http://bugs.debian.org/268450
[1] http://lists.debian.org/debian-sparc/2004/12/msg00009.html

Best regards,

Jurij Smakov                                        jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                   KeyID: C99E03CC

Re: mmap breakage

[Richard Mortimer]

On Sat, 2004-12-04 at 05:22, Jurij Smakov wrote:
> Hello,
> 
> In Debian we have recently received a number of bug reports [0,1] about 
> glibc's ld-linux.so.2 failing (when called via ldd, for example) on some 
> binaries. The minimum test case, originally mentioned in [0], was the 
> following:
> 
> 

Well I can demonstrate the behaviour (see attached test program) on sparc64 and i386. It allows mappings to be fully or partially replaced and even allows you to replace bits in the middle of a mapping.

Now the bad news...

This seems to be POSIX defined behaviour

http://www.opengroup.org/onlinepubs/009695399/functions/mmap.html

Specifically

"The mapping established by mmap() shall replace any previous mappings
for those whole pages containing any part of the address space of the
process starting at pa and continuing for len bytes."

and more specifically when MAP_FIXED is supplied

"If a MAP_FIXED request is successful, the mapping established by mmap()
replaces any previous mappings for the process' pages in the range
[pa,pa+len)."

and

"On the other hand, if the program specifies a fixed address mapping
(which requires some implementation knowledge to determine a suitable
address, if the function is supported at all), then the program is
presumed to be successfully managing its own address space and should be
trusted when it asks to map over existing data structures."

I guess that this isn't really what you want to hear but it does
describe why the behaviour exists. I don't have any magic suggestions as
to how to fix it - sorry.

Richard

P.S. You can see the overwriting of mappings if you look at
/proc/<pid>/maps. You will see that the second mapping has totally
superceded the first and the third mapping has split the second one up.

01000000-01002000 rw-p 00000000 00:0b 8               /dev/zero
01002000-01004000 rw-p 00000000 03:02 863336          /var/log/messages
01004000-01020000 rw-p 00004000 00:0b 8               /dev/zero



-- 
richm@oldelvet.org.uk

Re: mmap breakage

[Richard Mortimer]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]

FWIW. I forgot to attach the program that demonstrates the behaviour.

Richard

On Sat, 2004-12-04 at 23:13, Richard Mortimer wrote:
> On Sat, 2004-12-04 at 05:22, Jurij Smakov wrote:

> Well I can demonstrate the behaviour (see attached test program) on sparc64 and i386. It allows mappings to be fully or partially replaced and even allows you to replace bits in the middle of a mapping.


> P.S. You can see the overwriting of mappings if you look at
> /proc/<pid>/maps. You will see that the second mapping has totally
> superceded the first and the third mapping has split the second one up.
> 
> 01000000-01002000 rw-p 00000000 00:0b 8               /dev/zero
> 01002000-01004000 rw-p 00000000 03:02 863336          /var/log/messages
> 01004000-01020000 rw-p 00004000 00:0b 8               /dev/zero
-- 
richm@oldelvet.org.uk

[-- Attachment #2: mmaptest2.c --]
[-- Type: text/x-csrc, Size: 687 bytes --]

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>

int main()
{
	char *addr1;
	char *addr2;
	char *addr3;
	int fd1;
	int fd2;
	int fd3;
	fd1 = open("/var/log/syslog", O_RDONLY);
	fd2 = open("/dev/zero", O_RDWR);
	fd3 = open("/var/log/messages", O_RDONLY);
	addr1 = mmap((void*)0x1000000, 0x2000, PROT_READ, MAP_PRIVATE|MAP_FIXED, fd1,0);
	addr2 = mmap((void*)0x1000000, 0x20000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, fd2,0);
	addr3 = mmap((void*)0x1002000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, fd3,0);
	printf("addr1 is %p\n", addr1);
	printf("addr2 is %p\n", addr2);
	printf("addr3 is %p\n", addr3);

	sleep(100);
	return 0;
}

Re: mmap breakage

[Jurij Smakov]

On Sat, 4 Dec 2004, Richard Mortimer wrote:

> I guess that this isn't really what you want to hear but it does 
> describe why the behaviour exists. I don't have any magic suggestions as
> to how to fix it - sorry.
>
> Richard

Hi,

Thanks for the input. I have forwarded the question to debian-glibc [0], 
asking whether they consider it glibc or kernel problem. So far I have 
received response from Daniel Jacobowitz [1]:

> ld-linux ought to check.  In this particular case, it doesn't.  But if
> it did check, the only thing it could do would be to print an error
> message and quit.  The executable _must_ be mapped at the linked
> address (since it isn't position independent).  ld can't move itself.
> The kernel picked where to load ld; it would be nice if it picked
> somewhere more useful than 0x08000000 on your architecture.  On i386,
> it picks:
>
> 80000000-80016000 r-xp 00000000 09:00 11993980   /lib/ld-2.3.2.so
> 80016000-80017000 rw-p 00015000 09:00 11993980   /lib/ld-2.3.2.so
> 80017000-8006e000 rw-p 80017000 00:00 0 
> 
> which gives you up to 2GB for the executable.

So now the question is why ld-linux.so.2 on sparc64 is mapped at 
0x08000000 instead of something higher, like 0x80000000 on i386? This
actually limits the size of an executable plus all the shared libraries 
required by it (if I understand it correctly) to 128MB. Can this limit be 
easily modified, or are there some special architectural considerations, 
preventing that?

[0] http://lists.debian.org/debian-glibc/2004/12/msg00057.html
[1] http://lists.debian.org/debian-glibc/2004/12/msg00060.html

Best regards,

Jurij Smakov                                        jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                   KeyID: C99E03CC

Re: mmap breakage

[David S. Miller]

On Sun, 5 Dec 2004 23:30:53 -0500 (EST)
Jurij Smakov <jurij@wooyd.org> wrote:

> So now the question is why ld-linux.so.2 on sparc64 is mapped at 
> 0x08000000 instead of something higher, like 0x80000000 on i386? This
> actually limits the size of an executable plus all the shared libraries 
> required by it (if I understand it correctly) to 128MB. Can this limit be 
> easily modified, or are there some special architectural considerations, 
> preventing that?

It only limits the shared libraries as you state for the "medlow"
code model which restricts the binary to the low 32-bits of the
address space.

For the other 64-bit code models, we have large chunks of the rest
of the 64-bit address space in which to map things.

In fact, actually, PIC code such as shared libraries should be
mapped in the upper parts of the 64-bit address space.  The kernel
should start trying to map things for the user at 0xFFFFF80000000000
unless the user specifies a specific address.

Also the stack starts way up high in the 64-bit address space as well,
and this is for all code models.

We could put ld-linux.so.2 somewhere else, that's not too hard.  I
would suggest putting it at that 0xfffff80000000000 address I quoted
above since that is where the rest of the shared objects are going
to end up anyways.

A good person to ask about the logistics of moving ld-linux.so.2 somewhere
else would be Jakub Jelinek <jakub@redhat.com>

Re: mmap breakage

[Jurij Smakov]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2531 bytes --]

Hi Jakub,

On the sparclinux@vger list we have been recently discussing an issue of 
ld-linux.so.2 failing with SIGILL when executed directly with the 
executable name as an argument [0]. This problem was traced to the mmap 
call overwriting portions of ld-linux.so.2 itself. Insight provided by 
glibc people [1] helped to determine that on sparc64 the ld-linux.so.2 
is mapped at 0x08000000, leaving just 128MB for the executable. David 
Miller wrote on the subject of placing it at an alternative location [2]:

> We could put ld-linux.so.2 somewhere else, that's not too hard.  I
> would suggest putting it at that 0xfffff80000000000 address I quoted
> above since that is where the rest of the shared objects are going
> to end up anyways.

I presume that the value above is acceptable for 64-bit systems, but 
since Debian still uses (mostly) 32-bit userspace, I believe that the 
location must be below 4GB boundary. After digging a bit more through the 
code, I've found the following in include/asm-sparc64/elf.h:

/* This is the location that an ET_DYN program is loaded if exec'ed.  Typical
    use of this is to invoke "./ld.so someprog" to test out a new version of
    the loader.  We need to make sure that it is out of the way of the program
    that it will "exec", and that there is sufficient room for the brk.  */

#ifndef ELF_ET_DYN_BASE
#define ELF_ET_DYN_BASE         0x0000010000000000UL
#endif

This appears to be exactly the parameter I've been looking for. However, 
this is the default 64-bit value, for the 32-bit binaries it is redefined 
in arch/sparc64/kernel/binfmt_elf32.c:

#define ELF_ET_DYN_BASE         0x08000000

Since on i386 this is actually set to 0x80000000, I don't see why it 
should be different here. I have built a modified kernel (trivial patch is 
attached) and it seems to work fine. According to /proc/<pid>/maps 
ld-linux.so.2 is now mapped at 0x80000000 and I cannot reproduce the 
SIGILL using the old test case. Since you are an expert in this, could 
you please tell whether it makes sense to you? In particular, is the 
described change enough to remedy the problem, or is there something 
else we need to consider?

[0] http://marc.theaimsgroup.com/?t=110213793300001&r=1&w=2
[1] http://lists.debian.org/debian-glibc/2004/12/msg00060.html
[2] http://marc.theaimsgroup.com/?l=linux-sparc&m=110231073304789&w=2

Thanks and best regards,

Jurij Smakov                                        jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                   KeyID: C99E03CC

[-- Attachment #2: Type: TEXT/PLAIN, Size: 432 bytes --]

--- build-sparc64.orig/arch/sparc64/kernel/binfmt_elf32.c	2004-08-14 01:37:40.000000000 -0400
+++ build-sparc64/arch/sparc64/kernel/binfmt_elf32.c	2004-12-06 20:09:47.000000000 -0500
@@ -79,7 +79,7 @@
 
 #define elf_check_arch(x)	(((x)->e_machine == EM_SPARC) || ((x)->e_machine == EM_SPARC32PLUS))
 
-#define ELF_ET_DYN_BASE         0x08000000
+#define ELF_ET_DYN_BASE         0x80000000
 
 
 #include <asm/processor.h>

Re: mmap breakage

[David S. Miller]

On Tue, 7 Dec 2004 01:00:47 -0500 (EST)
Jurij Smakov <jurij@wooyd.org> wrote:

> On the sparclinux@vger list we have been recently discussing an issue of 
> ld-linux.so.2 failing with SIGILL when executed directly with the 
> executable name as an argument [0]. This problem was traced to the mmap 
> call overwriting portions of ld-linux.so.2 itself. Insight provided by 
> glibc people [1] helped to determine that on sparc64 the ld-linux.so.2 
> is mapped at 0x08000000, leaving just 128MB for the executable.

Actually, Jurij when you run ldd on a 64-bit binary what does
it say?  I can't reproduce that strange 0x08000000 address.

For me it seems to put ld-linux.so.2 at just the right spot:

davem@nuts:~$ file nanotest
nanotest: ELF 64-bit MSB executable, SPARC V9, version 1 (SYSV), for GNU/Linux 2.4.18, dynamically linked (uses shared libs), not stripped
davem@nuts:~$ ldd nanotest
        libc.so.6 => /lib64/libc.so.6 (0xfffff80000120000)
        /lib64/ld-linux.so.2 => /lib64/ld-linux.so.2 (0xfffff80000000000)
davem@nuts:~$ 

And actually executing the binary shows that it is in fact
mapped at 0xfffff80000000000.

davem@nuts:~$ cat /proc/1707/maps
00100000-00102000 r-xp 00000000 08:11 164461                             /home/davem/nanotest
00200000-00202000 rwxp 00000000 08:11 164461                             /home/davem/nanotest
7ffffffe000-80000000000 rw-p 7ffffffe000 00:00 0 
fffff80000000000-fffff8000001c000 r-xp 00000000 08:11 894371             /lib64/ld-2.3.2.so
fffff8000001c000-fffff8000001e000 rw-p fffff8000001c000 00:00 0 
fffff8000011c000-fffff8000011e000 rwxp 0001c000 08:11 894371             /lib64/ld-2.3.2.so
fffff80000120000-fffff80000276000 r-xp 00000000 08:11 895274             /lib64/libc-2.3.2.so
fffff80000276000-fffff80000320000 ---p 00156000 08:11 895274             /lib64/libc-2.3.2.so
fffff80000320000-fffff80000380000 rwxp 00100000 08:11 895274             /lib64/libc-2.3.2.so
fffff80000380000-fffff80000384000 rwxp fffff80000380000 00:00 0 
davem@nuts:~$ 

Re: mmap breakage

[Keith M Wesolowski]

On Mon, Dec 06, 2004 at 10:06:26PM -0800, David S. Miller wrote:

> Actually, Jurij when you run ldd on a 64-bit binary what does
> it say?  I can't reproduce that strange 0x08000000 address.

/lib/ld-linux.so.2 and the programs in question are ELF32.  The
ET_DYN_BASE is the same on sparc64 for ELF32 objects as it is on
sparc32.  Presumably the latter is to accomodate the silly 4c VM
hole...

-- 
Keith M Wesolowski
"Site launched. Many things not yet working." --Hector Urtubia

Re: mmap breakage

[David S. Miller]

On Mon, 6 Dec 2004 22:11:11 -0800
Keith M Wesolowski <wesolows@foobazco.org> wrote:

> On Mon, Dec 06, 2004 at 10:06:26PM -0800, David S. Miller wrote:
> 
> > Actually, Jurij when you run ldd on a 64-bit binary what does
> > it say?  I can't reproduce that strange 0x08000000 address.
> 
> /lib/ld-linux.so.2 and the programs in question are ELF32.  The
> ET_DYN_BASE is the same on sparc64 for ELF32 objects as it is on
> sparc32.  Presumably the latter is to accomodate the silly 4c VM
> hole...

Ok.  On sparc64 32-bit binaries get their ld-linux.so.2 mapped
at:

davem@nuts:~$ ldd ./nanotest 
        libc.so.6 => /lib/ultra3/libc.so.6 (0x7002c000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x70000000)
davem@nuts:~$ 

On sparc32, TASK_UNMAPPED_BASE is defined per-platform.
On sun4c it is set to 0xe0000000, on srmmu systems it is
set to 0x50000000.

So the 0x08000000 address is still a complete mystery to me.

Re: mmap breakage

[Jurij Smakov]

On Mon, 6 Dec 2004, David S. Miller wrote:

> Ok.  On sparc64 32-bit binaries get their ld-linux.so.2 mapped
> at:
>
> davem@nuts:~$ ldd ./nanotest
>        libc.so.6 => /lib/ultra3/libc.so.6 (0x7002c000)
>        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x70000000)
> davem@nuts:~$
>
> On sparc32, TASK_UNMAPPED_BASE is defined per-platform.
> On sun4c it is set to 0xe0000000, on srmmu systems it is
> set to 0x50000000.
>
> So the 0x08000000 address is still a complete mystery to me.

When I run ldd as above, I also get:

kundera:~# ldd ./a.out
         libc.so.6 => /lib/libc.so.6 (0x7002c000)
         /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x70000000)

However, that's somehow different from running '/lib/ld-linux.so.2 ./a.out'
(ldd actually does it). Once you invoke ld-linux.so.2 as an executable, 
it's placement is controlled by ELF_ET_DYN_BASE, which is 0x08000000 for 
ELF32 binaries by default.

Best regards,

Jurij Smakov                                        jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                   KeyID: C99E03CC

Re: mmap breakage

[David S. Miller]

On Tue, 7 Dec 2004 01:31:02 -0500 (EST)
Jurij Smakov <jurij@wooyd.org> wrote:

> However, that's somehow different from running '/lib/ld-linux.so.2 ./a.out'
> (ldd actually does it). Once you invoke ld-linux.so.2 as an executable, 
> it's placement is controlled by ELF_ET_DYN_BASE, which is 0x08000000 for 
> ELF32 binaries by default.

What needs to execute ld-linux.so.2 directly in order to run
and has a huge data section? :-)  Is it some part of the GLIBC
build, perhaps during locale generation?

Re: mmap breakage

[Jakub Jelinek]

On Mon, Dec 06, 2004 at 10:06:26PM -0800, David S. Miller wrote:
> On Tue, 7 Dec 2004 01:00:47 -0500 (EST)
> Jurij Smakov <jurij@wooyd.org> wrote:
> 
> > On the sparclinux@vger list we have been recently discussing an issue of 
> > ld-linux.so.2 failing with SIGILL when executed directly with the 
> > executable name as an argument [0]. This problem was traced to the mmap 
> > call overwriting portions of ld-linux.so.2 itself. Insight provided by 
> > glibc people [1] helped to determine that on sparc64 the ld-linux.so.2 
> > is mapped at 0x08000000, leaving just 128MB for the executable.
> 
> Actually, Jurij when you run ldd on a 64-bit binary what does
> it say?  I can't reproduce that strange 0x08000000 address.
> 
> For me it seems to put ld-linux.so.2 at just the right spot:
> 
> davem@nuts:~$ file nanotest
> nanotest: ELF 64-bit MSB executable, SPARC V9, version 1 (SYSV), for GNU/Linux 2.4.18, dynamically linked (uses shared libs), not stripped
> davem@nuts:~$ ldd nanotest
>         libc.so.6 => /lib64/libc.so.6 (0xfffff80000120000)
>         /lib64/ld-linux.so.2 => /lib64/ld-linux.so.2 (0xfffff80000000000)
> davem@nuts:~$ 

Try LD_TRACE_PRELINKING=1 LD_WARN= /lib64/ld-linux.so.2 ./nanotest

	Jakub

Re: mmap breakage

[Jurij Smakov]

On Mon, 6 Dec 2004, David S. Miller wrote:

> What needs to execute ld-linux.so.2 directly in order to run
> and has a huge data section? :-)  Is it some part of the GLIBC
> build, perhaps during locale generation?

No, this bug has been triggered during autobuilding of at least two 
different Debian packages, kmd [0] and fet [1]. kmd is a graphical 
debugger for ARM boards, and fet is some sort of personal organizer.

When ldd is run on the binaries late in the building stage to extract
the information about the dynamic libraries and generate the correct 
dependencies, it (among other things) invokes the command:

/lib/ld-linux.so.2 --verify ./a.out

which fails with SIGILL, if one is unlucky.

[0] http://bugs.debian.org/268450
[1] http://buildd.debian.org/fetch.php?&pkgþt&ver=3.9.19-1&arch=sparc&stamp\x1101448389&file=log&as=raw

Best regards,

Jurij Smakov                                        jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                   KeyID: C99E03CC

Re: mmap breakage

[Richard Mortimer]

On Tue, 2004-12-07 at 06:00, Jurij Smakov wrote:
> On the sparclinux@vger list we have been recently discussing an issue of 
> ld-linux.so.2 failing with SIGILL when executed directly with the 
> executable name as an argument [0]. This problem was traced to the mmap 
> call overwriting portions of ld-linux.so.2 itself.
...
> 
> > We could put ld-linux.so.2 somewhere else, that's not too hard.  I
> > would suggest putting it at that 0xfffff80000000000 address I quoted
> > above since that is where the rest of the shared objects are going
> > to end up anyways.

Just thinking about the nasty way that this fails (ld-linux.so.2
overwrites itself presumably causing seemingly random failures depending
on what it overwrites with). How about adding a check prior to mapping
to detect when the situation arises and then fail gracefully.

It would probably require adding a few symbols at the start and end of
each executable section (text, data, bss). Then rounding them to page
aligned values and just checking that you aren't overwriting them. Then
again I don't know the code so it might not turn out to be quite that
easy.

Richard

-- 
richm@oldelvet.org.uk

Re: mmap breakage

[David S. Miller]

On Tue, 7 Dec 2004 10:38:02 -0500 (EST)
Jurij Smakov <jurij@wooyd.org> wrote:

> No, this bug has been triggered during autobuilding of at least two 
> different Debian packages, kmd [0] and fet [1]. kmd is a graphical 
> debugger for ARM boards, and fet is some sort of personal organizer.
> 
> When ldd is run on the binaries late in the building stage to extract
> the information about the dynamic libraries and generate the correct 
> dependencies, it (among other things) invokes the command:
> 
> /lib/ld-linux.so.2 --verify ./a.out
> 
> which fails with SIGILL, if one is unlucky.

I understand, and as others pointed it this problem is due to
the ELF_ET_DYN_BASE setting in the kernel used for 32-bit binaries.

This triggered a memory, and now I sort of recall why I choose
this value.  It had something to do with emacs and how it would
dump itself.

Let's turn on the "way back machine" and look into the vger CVS
logs for the answer, and indeed we end up finding this:

CVSROOT:        /vger/u4/cvs
Module name:    linux
Changes by:     davem@vger.rutgers.edu  98/09/14 05:11:15

Modified files:
        arch/sparc64/kernel: binfmt_elf32.c
        include/asm-sparc: elf.h

Log message:
        Tweak ELF_ET_DYN_BASE so that emacs can be run
        properly with the intepreter as the command line.
        Note, this comment above these defines is wrong I believe.
        The brk of ld.so is what is used for sys_brk calls, not that
        of the later mapped main executable.  The reason emacs would
        barf is that it wants the top 4 bits of malloc obtained
        addresses to be zero so it can store lisp type information
        there.

And this explains the value we are using.

When we run /lib/ld-linux.so.2 from the command line as
the executable, the end of this mapping choosen is where
the BRK area of the program begins.

If the addresses used for the BRK area have any of the top bits
set, this makes emacs explode due to how it encodes type
information in the upper bits of pointers.  So at the time
I was debugging a new build of glibc by doing something like:

/path/to/glibc/build-sparc-linux/ld-linux.so.2 ./emacs

and it wouldn't work, which led to the change above.

Before I researched all of this, I was about to just plainly
set ELF_ET_DYN_BASE back to TASK_UNMAPPED_AREA.  That would
fix the given test case we're discussing, but it would also
break emacs per the discussion above.

The situation is pretty dire.  Our 32-bit address space on
different cpus looks like this:

1) sun4c,sun4

   Two virtual segments, one from 0x0 to 0x20000000 and
   another from 0xe0000000 to 0xf0000000.

   We set TASK_UNMAPPED_BASE to 0xe0000000 for this chip.

2) sun4m/sun4d

   One whole virtual segment from 0x0 to 0xf0000000.

   We set TASK_UNMAPPED_BASE to 0x50000000 for this chip.

3) 32-bit binary running on sparc64

   One whole virtual segment from 0x0 to 0xffffffff.

   We set TASK_UNMAPPED_BASE to 0x70000000 for this chip.

The biggest problematic case is sun4c, we simply don't have
much address space to work with.  Because of emacs wanting
the top 4 bits to be clear, we are basically limited to
some place between 0x0 and 0x10000000.

Let me think about this some more.

Re: mmap breakage

[David S. Miller]

[-- Attachment #1: Type: text/plain, Size: 495 bytes --]


Ok, here is what I'm going to do about this.  Patches for
2.6.x and 2.4.x attached.

As I described in a previous email, the ELF_ET_DYN_BASE
value used for 32-bit sparc was choosen simple to make
"/lib/ld-linux.so.2 emacs" work.

That case is obscure, and even i386 doesn't work in this
case either.  In fact, this choice on sparc has shown to
break many more important things.

So I'm going to go back to using TASK_UNMAPPED_BASE for
this and it should fix the test cases in the debian bugs.


[-- Attachment #2: et_dyn_base_24.diff --]
[-- Type: text/plain, Size: 2803 bytes --]

# This is a BitKeeper generated diff -Nru style patch.
#
# ChangeSet
#   2004/12/08 23:08:19-08:00 davem@nuts.davemloft.net 
#   [SPARC]: Adjust 32-bit ELF_ET_DYN_BASE.
#   
#   We were using 0x08000000 instead of TASK_UNMAPPED_BASE
#   so that running something like "/lib/ld-linux.so.2 emacs"
#   would work.
#   
#   The issue there was that wherever /lib/ld-linux.so.2 gets
#   mapped (controlled by ELF_ET_DYN_BASE), that is where the
#   BSS start for the process ends up.  Now, emacs allocates
#   dynamic memory for LISP objects from the BSS, and needs
#   the top 4 bits of the virtual address to be clear so that
#   it can encode LISP type and GC marking information there.
#   
#   But making this obscure emacs case work breaks lots of other
#   stuff.  For example, programs with a reasonably large data
#   section fail to load via direct ld.so interpreter execution
#   because the data section is large enough to begin overlapping
#   with the ELF_ET_DYN_BASE area.
#   
#   The /lib/ld-linux.so.2 emacs case does not work on a lot of
#   platforms due to this issue, including i386, so it is not
#   worth making work on sparc either.  It is indeed useful
#   sometimes when debugging a new experimental build of glibc
#   for example, but people doing that can hack the value of
#   ELF_ET_DYN_BASE in their kernels.  Perhaps at some point
#   we will make a sysctl controllable value.
#   
#   Signed-off-by: David S. Miller <davem@davemloft.net>
# 
# include/asm-sparc/elf.h
#   2004/12/08 23:08:07-08:00 davem@nuts.davemloft.net +1 -1
#   [SPARC]: Adjust 32-bit ELF_ET_DYN_BASE.
# 
# arch/sparc64/kernel/binfmt_elf32.c
#   2004/12/08 23:08:07-08:00 davem@nuts.davemloft.net +1 -1
#   [SPARC]: Adjust 32-bit ELF_ET_DYN_BASE.
# 
diff -Nru a/arch/sparc64/kernel/binfmt_elf32.c b/arch/sparc64/kernel/binfmt_elf32.c
--- a/arch/sparc64/kernel/binfmt_elf32.c	2004-12-08 23:09:43 -08:00
+++ b/arch/sparc64/kernel/binfmt_elf32.c	2004-12-08 23:09:43 -08:00
@@ -81,7 +81,7 @@
 
 #define elf_check_arch(x)	(((x)->e_machine == EM_SPARC) || ((x)->e_machine == EM_SPARC32PLUS))
 
-#define ELF_ET_DYN_BASE         0x08000000
+#define ELF_ET_DYN_BASE         0x70000000
 
 
 #include <asm/processor.h>
diff -Nru a/include/asm-sparc/elf.h b/include/asm-sparc/elf.h
--- a/include/asm-sparc/elf.h	2004-12-08 23:09:43 -08:00
+++ b/include/asm-sparc/elf.h	2004-12-08 23:09:43 -08:00
@@ -81,7 +81,7 @@
    the loader.  We need to make sure that it is out of the way of the program
    that it will "exec", and that there is sufficient room for the brk.  */
 
-#define ELF_ET_DYN_BASE         (0x08000000)
+#define ELF_ET_DYN_BASE         (TASK_UNMAPPED_BASE)
 
 /* This yields a mask that user programs can use to figure out what
    instruction set this cpu supports.  This can NOT be done in userspace

[-- Attachment #3: et_dyn_base_26.diff --]
[-- Type: text/plain, Size: 2805 bytes --]

# This is a BitKeeper generated diff -Nru style patch.
#
# ChangeSet
#   2004/12/08 23:02:03-08:00 davem@nuts.davemloft.net 
#   [SPARC]: Adjust 32-bit ELF_ET_DYN_BASE.
#   
#   We were using 0x08000000 instead of TASK_UNMAPPED_BASE
#   so that running something like "/lib/ld-linux.so.2 emacs"
#   would work.
#   
#   The issue there was that wherever /lib/ld-linux.so.2 gets
#   mapped (controlled by ELF_ET_DYN_BASE), that is where the
#   BSS start for the process ends up.  Now, emacs allocates
#   dynamic memory for LISP objects from the BSS, and needs
#   the top 4 bits of the virtual address to be clear so that
#   it can encode LISP type and GC marking information there.
#   
#   But making this obscure emacs case work breaks lots of other
#   stuff.  For example, programs with a reasonably large data
#   section fail to load via direct ld.so interpreter execution
#   because the data section is large enough to begin overlapping
#   with the ELF_ET_DYN_BASE area.
#   
#   The /lib/ld-linux.so.2 emacs case does not work on a lot of
#   platforms due to this issue, including i386, so it is not
#   worth making work on sparc either.  It is indeed useful
#   sometimes when debugging a new experimental build of glibc
#   for example, but people doing that can hack the value of
#   ELF_ET_DYN_BASE in their kernels.  Perhaps at some point
#   we will make a sysctl controllable value.
#   
#   Signed-off-by: David S. Miller <davem@davemloft.net>
# 
# include/asm-sparc/elf.h
#   2004/12/08 22:56:54-08:00 davem@nuts.davemloft.net +1 -1
#   [SPARC]: Adjust 32-bit ELF_ET_DYN_BASE.
# 
# arch/sparc64/kernel/binfmt_elf32.c
#   2004/12/08 22:56:54-08:00 davem@nuts.davemloft.net +1 -1
#   [SPARC]: Adjust 32-bit ELF_ET_DYN_BASE.
# 
diff -Nru a/arch/sparc64/kernel/binfmt_elf32.c b/arch/sparc64/kernel/binfmt_elf32.c
--- a/arch/sparc64/kernel/binfmt_elf32.c	2004-12-08 23:05:11 -08:00
+++ b/arch/sparc64/kernel/binfmt_elf32.c	2004-12-08 23:05:11 -08:00
@@ -79,7 +79,7 @@
 
 #define elf_check_arch(x)	(((x)->e_machine == EM_SPARC) || ((x)->e_machine == EM_SPARC32PLUS))
 
-#define ELF_ET_DYN_BASE         0x08000000
+#define ELF_ET_DYN_BASE         0x70000000
 
 
 #include <asm/processor.h>
diff -Nru a/include/asm-sparc/elf.h b/include/asm-sparc/elf.h
--- a/include/asm-sparc/elf.h	2004-12-08 23:05:11 -08:00
+++ b/include/asm-sparc/elf.h	2004-12-08 23:05:11 -08:00
@@ -143,7 +143,7 @@
    the loader.  We need to make sure that it is out of the way of the program
    that it will "exec", and that there is sufficient room for the brk.  */
 
-#define ELF_ET_DYN_BASE         (0x08000000)
+#define ELF_ET_DYN_BASE         (TASK_UNMAPPED_BASE)
 
 /* This yields a mask that user programs can use to figure out what
    instruction set this cpu supports.  This can NOT be done in userspace