[Qemu-devel] ARM brk bug

[Qemu-devel] ARM brk bug

[Bernhard M. Wiedemann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

I found that running a debian arm5 bash with qemu runs into varying
problems with -R but works without. Also works fine on both armv5 and
armv7hf hardware.


This happened with both master and 1.0 builds:

curl www.zq1.de/~bernhard/temp/debian-bash-bug-nss-minimal.tar.gz |\
  tar xz
cd debian-bash-bug-nss-minimal
path/to/qemu/arm-linux-user/qemu-arm -R 500M -L . bin/bash
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault


with chroot and our openSUSE qemu-arm-binfmt (which only does some
argv[0] magic) I saw it working after rm lib/libnss*

but otherwise it failed with messages like
bash: xmalloc: ../bash/variables.c:1971: cannot allocate 2 bytes (8192
bytes allocated)


Ciao
Bernhard M.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk9LnmsACgkQSTYLOx37oWT6FgCdEgIriKRVmwkaMQU6jBGN/ABP
s/EAn1cZj4KytK9iC3i6EqqfIXFyEUVX
=DCV8
-----END PGP SIGNATURE-----

Re: [Qemu-devel] ARM brk bug

[Peter Maydell]

On 27 February 2012 15:16, Bernhard M. Wiedemann <bwiedemann@suse.de> wrote:
> I found that running a debian arm5 bash with qemu runs into varying
> problems with -R but works without. Also works fine on both armv5 and
> armv7hf hardware.
>
>
> This happened with both master and 1.0 builds:
>
> curl www.zq1.de/~bernhard/temp/debian-bash-bug-nss-minimal.tar.gz |\
>  tar xz
> cd debian-bash-bug-nss-minimal
> path/to/qemu/arm-linux-user/qemu-arm -R 500M -L . bin/bash
> qemu: uncaught target signal 11 (Segmentation fault) - core dumped
> Segmentation fault

> but otherwise it failed with messages like
> bash: xmalloc: ../bash/variables.c:1971: cannot allocate 2 bytes (8192
> bytes allocated)

So, er, don't do that then? This looks suspiciously like we're
failing an mmap() (because of the limited guest address space you've
asked for with -R) and then bash is either failing to handle it
and crashing or printing a message about the allocation failure.

-- PMM

Re: [Qemu-devel] ARM brk bug

[Bernhard M. Wiedemann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/27/2012 04:32 PM, Peter Maydell wrote:
> On 27 February 2012 15:16, Bernhard M. Wiedemann
> <bwiedemann@suse.de> wrote:
>> I found that running a debian arm5 bash with qemu runs into
>> varying problems with -R but works without. Also works fine on
>> both armv5 and armv7hf hardware.
>> 
>> 
>> This happened with both master and 1.0 builds:
>> 
>> curl www.zq1.de/~bernhard/temp/debian-bash-bug-nss-minimal.tar.gz
>> |\ tar xz cd debian-bash-bug-nss-minimal 
>> path/to/qemu/arm-linux-user/qemu-arm -R 500M -L . bin/bash qemu:
>> uncaught target signal 11 (Segmentation fault) - core dumped 
>> Segmentation fault
> 
>> but otherwise it failed with messages like bash: xmalloc:
>> ../bash/variables.c:1971: cannot allocate 2 bytes (8192 bytes
>> allocated)
> 
> So, er, don't do that then? This looks suspiciously like we're 
> failing an mmap() (because of the limited guest address space
> you've asked for with -R) and then bash is either failing to handle
> it and crashing or printing a message about the allocation
> failure.
> 
> -- PMM

apparently, we patched qemu for openSUSE to always use -R because
otherwise java did not work (with the binfmt chroot magic).

even when I used the highest possible value for -R it did not work:

qemu-arm -R 4026M -L . bin/bash
bash: xmalloc: ../bash/variables.c:1971: cannot allocate 2 bytes (8192
bytes allocated)

even though it worked with any other arm binary I tried (including
openSUSE's bash, Debians dash etc).


Ciao
Bernhard M.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk9Lt14ACgkQSTYLOx37oWTt4ACghJYkfju/iG/jDS/51RasTwzn
HUMAoNyKiuJ3ApeLpWM4EIPAaMHVsGaT
=QN4N
-----END PGP SIGNATURE-----

Re: [Qemu-devel] ARM brk bug

[Peter Maydell]

On 27 February 2012 17:03, Bernhard M. Wiedemann <bwiedemann@suse.de> wrote:
> apparently, we patched qemu for openSUSE to always use -R because
> otherwise java did not work (with the binfmt chroot magic).

Yeah, the ubuntu qemu has a similar patch now. We really should
clean that up so we can put it into upstream qemu...

-- PMM

Re: [Qemu-devel] ARM brk bug

[Peter Maydell]

On 27 February 2012 15:16, Bernhard M. Wiedemann <bwiedemann@suse.de> wrote:
> I found that running a debian arm5 bash with qemu runs into varying
> problems with -R but works without.

So I had a look at this this afternoon, and what seems to be happening
is that with -R, the call to target_mmap() in elfload.c:setup_arg_pages()
(which creates the stack) is putting the stack immediately after the
bash BSS segment in the address space. This means that brk() will
never be able to expand, and it looks like something in either bash
or libc's locale code isn't correctly handling the failure, so we
crash. (The segfault is from a strlen(NULL) from setlocale() I think.)

We should probably try to put the stack somewhere more sensible than
where it currently ends up...

-- PMM

Re: [Qemu-devel] ARM brk bug

[Alexander Graf]

On 02.03.2012, at 18:49, Peter Maydell wrote:

> On 27 February 2012 15:16, Bernhard M. Wiedemann <bwiedemann@suse.de> wrote:
>> I found that running a debian arm5 bash with qemu runs into varying
>> problems with -R but works without.
> 
> So I had a look at this this afternoon, and what seems to be happening
> is that with -R, the call to target_mmap() in elfload.c:setup_arg_pages()
> (which creates the stack) is putting the stack immediately after the
> bash BSS segment in the address space. This means that brk() will
> never be able to expand, and it looks like something in either bash
> or libc's locale code isn't correctly handling the failure, so we
> crash. (The segfault is from a strlen(NULL) from setlocale() I think.)
> 
> We should probably try to put the stack somewhere more sensible than
> where it currently ends up...

Yikes - I just realized that git on armv7 also breaks due to this:

  https://build.opensuse.org/package/live_build_log?arch=armv7l&package=git&project=openSUSE%3AFactory%3AARM&repository=standard

So how could we go with this? Give target_mmap a hint that we want to map at some randomly defined address rather than let it to its automatic thing?


Alex

Re: [Qemu-devel] ARM brk bug

[Alexander Graf]

On 02.03.2012, at 18:49, Peter Maydell wrote:

> On 27 February 2012 15:16, Bernhard M. Wiedemann <bwiedemann@suse.de> wrote:
>> I found that running a debian arm5 bash with qemu runs into varying
>> problems with -R but works without.
> 
> So I had a look at this this afternoon, and what seems to be happening
> is that with -R, the call to target_mmap() in elfload.c:setup_arg_pages()
> (which creates the stack) is putting the stack immediately after the
> bash BSS segment in the address space. This means that brk() will
> never be able to expand, and it looks like something in either bash
> or libc's locale code isn't correctly handling the failure, so we
> crash. (The segfault is from a strlen(NULL) from setlocale() I think.)
> 
> We should probably try to put the stack somewhere more sensible than
> where it currently ends up...

I wrote a small test case to reproduce the breakage as far as I understood it at least:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/fcntl.h>

int main(int argc, char **argv)
{
    void *curbrk;
    void *tmp;
    char buf[1024];
    int fd, r;

    tmp = mmap((void*)0x5000UL, 0x10000, PROT_READ|PROT_WRITE,
               MAP_PRIVATE|MAP_ANONYMOUS, 0, 0);
    printf("mmap: %p\n", tmp);

    curbrk = sbrk(8 * 1024 * 1024);
    printf("current brk: %p\n", curbrk);

    fd = open("/proc/self/maps", O_RDONLY);
    while ((r = read(fd, buf, sizeof(buf))) > 0) {
        write(1, buf, r);
    }

    return 0;
}

On real hw:

$ ./brk
mmap: 0x40056000
current brk: 0x12000
00008000-00009000 r-xp 00000000 b3:02 29374      /root/brk-bug/brk
00010000-00011000 r--p 00000000 b3:02 29374      /root/brk-bug/brk
00011000-00012000 rw-p 00001000 b3:02 29374      /root/brk-bug/brk
00012000-00812000 rw-p 00000000 00:00 0          [heap]
40014000-40015000 rw-p 00000000 00:00 0 
40035000-4004d000 r-xp 00000000 b3:02 6823       /lib/ld-2.15.so
4004d000-4004e000 rw-p 00000000 00:00 0 
40054000-40055000 r--p 00017000 b3:02 6823       /lib/ld-2.15.so
40055000-40056000 rw-p 00018000 b3:02 6823       /lib/ld-2.15.so
40056000-40066000 rw-p 00000000 00:00 0 
4008e000-4008f000 rw-p 00000000 00:00 0 
400a8000-400b1000 r-xp 00000000 b3:02 6846       /lib/libgcc_s.so.1
400b1000-400b8000 ---p 00009000 b3:02 6846       /lib/libgcc_s.so.1
400b8000-400b9000 r--p 00008000 b3:02 6846       /lib/libgcc_s.so.1
400b9000-400ba000 rw-p 00009000 b3:02 6846       /lib/libgcc_s.so.1
40150000-4022d000 r-xp 00000000 b3:02 6830       /lib/libc-2.15.so
4022d000-40234000 ---p 000dd000 b3:02 6830       /lib/libc-2.15.so
40234000-40236000 r--p 000dc000 b3:02 6830       /lib/libc-2.15.so
40236000-40237000 rw-p 000de000 b3:02 6830       /lib/libc-2.15.so
40237000-4023a000 rw-p 00000000 00:00 0 
bee3f000-bee60000 rw-p 00000000 00:00 0          [stack]
ffff0000-ffff1000 r-xp 00000000 00:00 0          [vectors]

strace says:
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40082000
[...]
mmap2(0x5000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0, 0) = 0x40018000
[...]
brk(0)                                  = 0x12000
brk(0x812000)                           = 0x812000

In QEMU with -R <high>

$ qemu-arm -R $(( 0x10000000 )) ./brk
mmap: 0x935000
current brk: 0xffffffff
00000000-00008000 ---p 00000000 00:00 0
00008000-00009000 r-xp 00000000 08:09 1248935408          /brk
00009000-00010000 ---p 00000000 00:00 0
00010000-00011000 r--p 00000000 08:09 1248935408          /brk
00011000-00012000 rw-p 00001000 08:09 1248935408          /brk
00012000-00013000 ---p 00000000 00:00 0
00013000-00813000 rw-p 00000000 00:00 0
00813000-0082b000 r-xp 00000000 08:09 1248675559          /lib/ld-2.15.so
0082b000-00832000 ---p 00000000 00:00 0
00832000-00833000 r--p 00017000 08:09 1248675559          /lib/ld-2.15.so
00833000-00834000 rw-p 00018000 08:09 1248675559          /lib/ld-2.15.so
00834000-00835000 rw-p 00000000 00:00 0
00835000-00838000 ---p 00000000 00:00 0
00838000-00915000 r-xp 00000000 08:09 1248675566          /lib/libc-2.15.so
00915000-0091c000 ---p 000dd000 08:09 1248675566          /lib/libc-2.15.so
0091c000-0091e000 r--p 000dc000 08:09 1248675566          /lib/libc-2.15.so
0091e000-0091f000 rw-p 000de000 08:09 1248675566          /lib/libc-2.15.so
0091f000-00922000 rw-p 00000000 00:00 0
00922000-0092b000 r-xp 00000000 08:09 1248678464          /lib/libgcc_s.so.1
0092b000-00932000 ---p 00009000 08:09 1248678464          /lib/libgcc_s.so.1
00932000-00933000 r--p 00008000 08:09 1248678464          /lib/libgcc_s.so.1
00933000-00934000 rw-p 00009000 08:09 1248678464          /lib/libgcc_s.so.1
00934000-00946000 rw-p 00000000 00:00 0
00013000-00813000 rw-p 00000000 00:00 0          [stack]

qemu's strace says:
15545 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x00834000
[...]
15545 mmap2(0x00005000,65536,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,0,0) = 0x00935000
[...]
15545 brk(NULL) = 0x00012000
15545 brk(0x00812000) = 0x00012000

I wonder who allocates the chunk from 00012000-00013000? I don't see any syscall returning an address in that range on the trace.

Full trace: http://paste.debian.net/158450/

In QEMU with -R 0

$ qemu-arm -R 0 ./brk
mmap: 0xffffffff
current brk: 0x12000
00008000-00009000 r-xp 00000000 08:09 1248935408          /brk
00009000-00010000 ---p 00000000 00:00 0
00010000-00011000 r--p 00000000 08:09 1248935408          /brk
00011000-00012000 rw-p 00001000 08:09 1248935408          /brk
00012000-00812000 rw-p 00000000 00:00 0
0fff8000-101b8000 r-xp 00000000 08:09 1248674338          /usr/bin/qemu-arm
102b8000-102e7000 rw-p 002c0000 08:09 1248674338          /usr/bin/qemu-arm
102e7000-122e8000 rwxp 00000000 00:00 0
122e8000-12336000 rw-p 00000000 00:00 0
12336000-12337000 rwxp 00000000 00:00 0
12337000-12370000 rw-p 00000000 00:00 0          [heap]
18000000-18001000 ---p 00000000 00:00 0
18001000-18801000 rw-p 00000000 00:00 0
18801000-18819000 r-xp 00000000 08:09 1248675559          /lib/ld-2.15.so
18819000-18820000 ---p 00000000 00:00 0
18820000-18821000 r--p 00017000 08:09 1248675559          /lib/ld-2.15.so
18821000-18822000 rw-p 00018000 08:09 1248675559          /lib/ld-2.15.so
18822000-18823000 rw-p 00000000 00:00 0
18826000-18903000 r-xp 00000000 08:09 1248675566          /lib/libc-2.15.so
18903000-1890a000 ---p 000dd000 08:09 1248675566          /lib/libc-2.15.so
1890a000-1890c000 r--p 000dc000 08:09 1248675566          /lib/libc-2.15.so
1890c000-1890d000 rw-p 000de000 08:09 1248675566          /lib/libc-2.15.so
1890d000-18910000 rw-p 00000000 00:00 0
18910000-18919000 r-xp 00000000 08:09 1248678464          /lib/libgcc_s.so.1
18919000-18920000 ---p 00009000 08:09 1248678464          /lib/libgcc_s.so.1
18920000-18921000 r--p 00008000 08:09 1248678464          /lib/libgcc_s.so.1
18921000-18922000 rw-p 00009000 08:09 1248678464          /lib/libgcc_s.so.1
18922000-18924000 rw-p 00000000 00:00 0
ffff0000-ffff1000 r--p 00000000 00:00 0
18001000-18801000 rw-p 00000000 00:00 0          [stack]

qemu's strace says:
15659 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x18822000
[...]
15659 mmap2(0x00005000,65536,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,0,0) = 0xfffffff4
[...]
15659 brk(NULL) = 0x00012000
15659 brk(0x00812000) = 0x00812000

Full trace: http://paste.debian.net/158451/

Don't be surprised about the duplicate [stack] entry at the end. I manually add that to the /proc/self/stat output at the end of the real sanitized /proc/self/maps data from the host in my maps hijack code.


Alex

Re: [Qemu-devel] ARM brk bug

[Alexander Graf]

On 03.03.2012, at 22:02, Alexander Graf wrote:

[...]

> 
> $ qemu-arm -R $(( 0x10000000 )) ./brk
> mmap: 0x935000
> current brk: 0xffffffff
> 00000000-00008000 ---p 00000000 00:00 0
> 00008000-00009000 r-xp 00000000 08:09 1248935408          /brk
> 00009000-00010000 ---p 00000000 00:00 0
> 00010000-00011000 r--p 00000000 08:09 1248935408          /brk
> 00011000-00012000 rw-p 00001000 08:09 1248935408          /brk
> 00012000-00013000 ---p 00000000 00:00 0
> 00013000-00813000 rw-p 00000000 00:00 0
> 00813000-0082b000 r-xp 00000000 08:09 1248675559          /lib/ld-2.15.so
> 0082b000-00832000 ---p 00000000 00:00 0
> 00832000-00833000 r--p 00017000 08:09 1248675559          /lib/ld-2.15.so
> 00833000-00834000 rw-p 00018000 08:09 1248675559          /lib/ld-2.15.so
> 00834000-00835000 rw-p 00000000 00:00 0
> 00835000-00838000 ---p 00000000 00:00 0
> 00838000-00915000 r-xp 00000000 08:09 1248675566          /lib/libc-2.15.so
> 00915000-0091c000 ---p 000dd000 08:09 1248675566          /lib/libc-2.15.so
> 0091c000-0091e000 r--p 000dc000 08:09 1248675566          /lib/libc-2.15.so
> 0091e000-0091f000 rw-p 000de000 08:09 1248675566          /lib/libc-2.15.so
> 0091f000-00922000 rw-p 00000000 00:00 0
> 00922000-0092b000 r-xp 00000000 08:09 1248678464          /lib/libgcc_s.so.1
> 0092b000-00932000 ---p 00009000 08:09 1248678464          /lib/libgcc_s.so.1
> 00932000-00933000 r--p 00008000 08:09 1248678464          /lib/libgcc_s.so.1
> 00933000-00934000 rw-p 00009000 08:09 1248678464          /lib/libgcc_s.so.1
> 00934000-00946000 rw-p 00000000 00:00 0
> 00013000-00813000 rw-p 00000000 00:00 0          [stack]
> 
> qemu's strace says:
> 15545 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x00834000
> [...]
> 15545 mmap2(0x00005000,65536,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,0,0) = 0x00935000
> [...]
> 15545 brk(NULL) = 0x00012000
> 15545 brk(0x00812000) = 0x00012000
> 
> I wonder who allocates the chunk from 00012000-00013000? I don't see any syscall returning an address in that range on the trace.

Oooh that's our reserved memory range. Yikes. I think I'm slowly grasping what's going wrong here.


Alex

Re: [Qemu-devel] ARM brk bug

[Paul Brook]

> In QEMU with -R <high>
> 
> $ qemu-arm -R $(( 0x10000000 )) ./brk
> mmap: 0x935000
> current brk: 0xffffffff
> 00000000-00008000 ---p 00000000 00:00 0
> 00008000-00009000 r-xp 00000000 08:09 1248935408          /brk
> 00009000-00010000 ---p 00000000 00:00 0
> 00010000-00011000 r--p 00000000 08:09 1248935408          /brk
> 00011000-00012000 rw-p 00001000 08:09 1248935408          /brk
> 00012000-00013000 ---p 00000000 00:00 0
> 00013000-00813000 rw-p 00000000 00:00 0

Would tell us a lot more if (a) you gave us the map before the strace call, 
and (b) the requested size wasn't the same as the default stack size.  
> 
> qemu's strace says:
> 15545 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)
> = 0x00834000 [...]
> 15545
> mmap2(0x00005000,65536,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,0,0)
> = 0x00935000 [...]
> 15545 brk(NULL) = 0x00012000
> 15545 brk(0x00812000) = 0x00012000

This is your bug.  According to the trace above, the brk call fails, but 
returns success anyway.  A quick look at do_brk confirms this suspicion.

Paul

Re: [Qemu-devel] ARM brk bug

[malc]

On Sat, 3 Mar 2012, Alexander Graf wrote:

> 
> On 02.03.2012, at 18:49, Peter Maydell wrote:
> 
> > On 27 February 2012 15:16, Bernhard M. Wiedemann <bwiedemann@suse.de> wrote:
> >> I found that running a debian arm5 bash with qemu runs into varying
> >> problems with -R but works without.
> > 
> > So I had a look at this this afternoon, and what seems to be happening
> > is that with -R, the call to target_mmap() in elfload.c:setup_arg_pages()
> > (which creates the stack) is putting the stack immediately after the
> > bash BSS segment in the address space. This means that brk() will
> > never be able to expand, and it looks like something in either bash
> > or libc's locale code isn't correctly handling the failure, so we
> > crash. (The segfault is from a strlen(NULL) from setlocale() I think.)
> > 
> > We should probably try to put the stack somewhere more sensible than
> > where it currently ends up...
> 
> I wrote a small test case to reproduce the breakage as far as I understood it at least:
> 
> #include <stdio.h>
> #include <stdlib.h>
> #include <unistd.h>
> #include <sys/mman.h>
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <sys/fcntl.h>
> 
> int main(int argc, char **argv)
> {
>     void *curbrk;
>     void *tmp;
>     char buf[1024];
>     int fd, r;
> 
>     tmp = mmap((void*)0x5000UL, 0x10000, PROT_READ|PROT_WRITE,
>                MAP_PRIVATE|MAP_ANONYMOUS, 0, 0);

You are setting a bad example here (not that i know if that makes a
dent of a practical difference in this case): passing 0 as an fd even
in MAP_ANONYMOUS case is unsafe.

[..snip..]

-- 
mailto:av1474@comtv.ru

Re: [Qemu-devel] ARM brk bug

[Andreas Schwab]

Paul Brook <paul@codesourcery.com> writes:

>> 15545 brk(NULL) = 0x00012000
>> 15545 brk(0x00812000) = 0x00012000
>
> This is your bug.  According to the trace above, the brk call fails, but 
> returns success anyway.  A quick look at do_brk confirms this suspicion.

This is not a bug.  The brk syscall always returns the break value,
either the new one (on success) or the current one (on failure).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."