[Qemu-devel] qemu-arm fails on test-mmap

[Qemu-devel] qemu-arm fails on test-mmap

[Jan-Simon Möller]

Hi!

I've spotted an bug in mmap for the qemu-arm . It causes a segfault of qemu or at least the running process.

Its reproducible here with "test-mmap" .

legolas:/> qemu-arm ./test-mmap
pagesize=4096 pagemask=fff
check_aligned_anonymous_unfixed_mmaps passed
check_aligned_anonymous_unfixed_colliding_mmapsSegmentation fault (core dumped)


A lengthy trace with debugging on in mmap.c is at
http://filebin.ca/yxypzq/qemu_mmap_segfault.bz2

[... last mmap call ...]
munmap: start=0x5fffd000 len=0x00001000
mmap: start=0x00000000 len=0x00008000 prot=r-- flags=MAP_ANON MAP_PRIVATE fd=-1 offset=00000000
ret=0x5ffff000
start    end      size     prot
00008000-0000b000 00003000 r-x
00012000-00013000 00001000 r--
00013000-00037000 00024000 rw-
40000000-40080000 00080000 rw-
40080000-40081000 00001000 ---
40081000-4009f000 0001e000 r-x
4009f000-400a6000 00007000 ---
400a6000-400a7000 00001000 r--
400a7000-400a8000 00001000 rw-
400a8000-42081000 01fd9000 ---
42085000-421bf000 0013a000 r-x
421bf000-421c6000 00007000 ---
421c6000-421c8000 00002000 r--
421c8000-421ce000 00006000 rw-
5fffe000-60007000 00009000 r--

Segmentation fault

It seems to hit > 0x5fffffff -> segfault.

Best,
Jan-Simon

Re: [Qemu-devel] qemu-arm fails on test-mmap

[Laurent Desnogues]

2009/7/31 Jan-Simon Möller <dl9pf@gmx.de>:
> Hi!
>
> I've spotted an bug in mmap for the qemu-arm . It causes a segfault of qemu or at least the running process.
>
> Its reproducible here with "test-mmap" .
>
> legolas:/> qemu-arm ./test-mmap
> pagesize=4096 pagemask=fff
> check_aligned_anonymous_unfixed_mmaps passed
> check_aligned_anonymous_unfixed_colliding_mmapsSegmentation fault (core dumped)
>
>
> A lengthy trace with debugging on in mmap.c is at
> http://filebin.ca/yxypzq/qemu_mmap_segfault.bz2
>
> [... last mmap call ...]
> munmap: start=0x5fffd000 len=0x00001000
> mmap: start=0x00000000 len=0x00008000 prot=r-- flags=MAP_ANON MAP_PRIVATE fd=-1 offset=00000000
> ret=0x5ffff000
> start    end      size     prot
> 00008000-0000b000 00003000 r-x
> 00012000-00013000 00001000 r--
> 00013000-00037000 00024000 rw-
> 40000000-40080000 00080000 rw-
> 40080000-40081000 00001000 ---
> 40081000-4009f000 0001e000 r-x
> 4009f000-400a6000 00007000 ---
> 400a6000-400a7000 00001000 r--
> 400a7000-400a8000 00001000 rw-
> 400a8000-42081000 01fd9000 ---
> 42085000-421bf000 0013a000 r-x
> 421bf000-421c6000 00007000 ---
> 421c6000-421c8000 00002000 r--
> 421c8000-421ce000 00006000 rw-
> 5fffe000-60007000 00009000 r--
>
> Segmentation fault
>
> It seems to hit > 0x5fffffff -> segfault.

I tested qemu-i386 and qemu-arm on both i386 and x86_64 and
the four pass that test.  That's with git head, and CSL 2009q1 on
my Fedora 11 x86_64 machine.


Laurent

Re: [Qemu-devel] qemu-arm fails on test-mmap

[Jan-Simon Möller]

Am Friday 31 July 2009 11:58:04 schrieb Laurent Desnogues:
> I tested qemu-i386 and qemu-arm on both i386 and x86_64 and
> the four pass that test.  That's with git head, and CSL 2009q1 on
> my Fedora 11 x86_64 machine.

SMP or single-core host ?
What's "cat /proc/sys/vm/mmap_min_addr" in your environment ?

Best
Jan-Simon

Re: [Qemu-devel] qemu-arm fails on test-mmap

[Laurent Desnogues]

2009/7/31 Jan-Simon Möller <dl9pf@gmx.de>:
>
> SMP or single-core host ?

Core i7 with SMT enabled so 8 core.

> What's "cat /proc/sys/vm/mmap_min_addr" in your environment ?

32768


Laurent

Re: [Qemu-devel] qemu-arm fails on test-mmap

[Jan-Simon Möller]

Hi !

The funny thing is - i can run the test with the same binaries without problem _outside_ the arm chroot, but not inside:

dl9pf@legolas:~> /var/tmp/build-root/usr/bin/qemu-arm /var/tmp/build-root/test-mmap
pagesize=4096 pagemask=fff
check_aligned_anonymous_unfixed_mmaps passed
check_aligned_anonymous_unfixed_colliding_mmaps passed
check_aligned_anonymous_fixed_mmaps addr=0x6a271000 passed
check_file_unfixed_mmaps passed
check_file_fixed_mmaps addr=0x6a2c9000 passed
check_file_fixed_eof_mmaps addr=0x6a369000 passed
check_file_unfixed_eof_mmaps passed

legolas:/home/dl9pf # chroot /var/tmp/build-root

legolas:/> ./test-mmap
pagesize=4096 pagemask=fff
check_aligned_anonymous_unfixed_mmaps passed
check_aligned_anonymous_unfixed_colliding_mmapsSpeicherzugriffsfehler

legolas:/> file test-mmap
test-mmap: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, for GNU/Linux 2.6.16, not stripped
legolas:/> file usr/bin/qemu-arm
usr/bin/qemu-arm: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for GNU/Linux 2.6.4, not stripped


*scratching head*

Best,
Jan-Simon

Re: [Qemu-devel] qemu-arm fails on test-mmap

[Jan-Simon Möller]

Am Friday 31 July 2009 15:03:31 schrieb Jan-Simon Möller:
> Hi !
> 
> The funny thing is - i can run the test with the same binaries without problem _outside_ the arm chroot, but not inside:
> 
> dl9pf@legolas:~> /var/tmp/build-root/usr/bin/qemu-arm /var/tmp/build-root/test-mmap
> pagesize=4096 pagemask=fff
> check_aligned_anonymous_unfixed_mmaps passed
> check_aligned_anonymous_unfixed_colliding_mmaps passed
> check_aligned_anonymous_fixed_mmaps addr=0x6a271000 passed
> check_file_unfixed_mmaps passed
> check_file_fixed_mmaps addr=0x6a2c9000 passed
> check_file_fixed_eof_mmaps addr=0x6a369000 passed
> check_file_unfixed_eof_mmaps passed
> 
> legolas:/home/dl9pf # chroot /var/tmp/build-root
> 
> legolas:/> ./test-mmap
> pagesize=4096 pagemask=fff
> check_aligned_anonymous_unfixed_mmaps passed
> check_aligned_anonymous_unfixed_colliding_mmapsSpeicherzugriffsfehler
> 
> legolas:/> file test-mmap
> test-mmap: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, for GNU/Linux 2.6.16, not stripped
> legolas:/> file usr/bin/qemu-arm
> usr/bin/qemu-arm: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for GNU/Linux 2.6.4, not stripped
> 
> 
> *scratching head*

Inside the arm-chroot it segfaults when it hits 0x60000000 . No problems with that outside.

Any idea on this boundary ?

Best,
Jan-Simon

Re: [Qemu-devel] qemu-arm fails on test-mmap

[Jan-Simon Möller]

Am Freitag 31 Juli 2009 11:58:04 schrieb Laurent Desnogues:
I prepared bundles for::

qemu-arm  built for i586  which  can  run the test on a i586 machine, but not 
when its run on an x86_64 box.

Link for this package :  

http://www.stud.uni-hannover.de/~jsm/qemu-mmap_testbundle_prebuilt-i586.tar.bz2

qemu-arm build for x86_64 which can't run the test _inside_ my target chroot, 
but it runs on some hosts when its called standalone. (min_addr = 0 , runs 
probably)

Link: 
http://www.stud.uni-hannover.de/~jsm/qemu-mmap_testbundle_prebuilt_amd64.tar.bz2


Lets sum it up:

* qemu-arm compiled for i586 can run the test in the chroot @32bit host

* qemu-arm compiled for i586 can _not_ run the test on a 64bit host at all

* qemu-arm compiled for x86_64 can run the test only outside the target-chroot 

* The mmap tests fail  if they reach values >0x5fffffff  - starting from
    0x40000000 thats a 0x20000000 margin


It seems to me an issue on 64bit hosts with 32bit target, but I've no idea atm 
what causes this - especially the fact that it runs outside the target chroot 
on x86_64 .

Lest see if someone can reproduce this on his machine.

Best,
Jan-Simon

[Qemu-devel] qemu-arm fails on test-mmap - take #2

[Jan-Simon Möller]

Hi !

This is a follow-up to my tests about test-mmap failing inside an arm chroot 
when using qemu-arm in user-mode.

Here are 2 snippets running   "qemu-arm ./test-mmap"  outside and inside the 
ARM chroot env.

I turned DEBUG_MMAP on in linux-user/mmap.c .

legolas:/var/tmp/build-root # ./usr/bin/qemu-arm test-mmap
------snip-------
mmap: start=0x00000000 len=0x00001000 prot=r-- flags=MAP_ANON MAP_PRIVATE 
fd=-1 offset=00000000
ret=0x40b77000
start    end      size     prot
00008000-00081000 00079000 r-x
00088000-0008a000 00002000 rw-
0008a000-0008c000 00002000 rwx
0008c000-000af000 00023000 rw-
40000000-40080000 00080000 rw-
40080000-40081000 00001000 ---
40081000-40082000 00001000 rw-
40339000-40347000 0000e000 ---
407d6000-407d7000 00001000 ---
4096b000-40974000 00009000 ---
40b76000-40b78000 00002000 r--
40e8a000-40e8c000 00002000 ---
41dc7000-455ca000 03803000 ---
60000000-60166000 00166000 ---
60266000-6231d000 020b7000 ---
------snip-------

Note the last 2 lines !!

Now same procedure inside the chroot ...
legolas:/> chroot /var/tmp/build-root
legolas:/> ./test-mmap

------snip-------
mmap: start=0x00000000 len=0x00001000 prot=r-- flags=MAP_ANON MAP_PRIVATE 
fd=-1 offset=00000000
ret=0x40b74000
start    end      size     prot
00008000-00081000 00079000 r-x
00088000-0008a000 00002000 rw-
0008a000-0008c000 00002000 rwx
0008c000-000af000 00023000 rw-
40000000-40080000 00080000 rw-
40080000-40081000 00001000 ---
40081000-40082000 00001000 rw-
40191000-40192000 00001000 ---
40b73000-40b75000 00002000 r--
40c6c000-40c75000 00009000 ---
41105000-41106000 00001000 ---
41283000-44a95000 03812000 ---
------snip-------


The lines with 60000000-60166000 are gone ... 
Thus as soon as those pages will get allocated  it will segfault.

This is reproducible on different machines:
1)
openSUSE 11.1 64bit on Core 2 Duo  with 2GB Ram, qemu git head
The ARM chroot uses gcc4.4 and glibc 2.10.1 .

2)
openSUSE 11.0 32bit  Athlon XP 2GB RAM, same qemu, same chroot


Now my question is: how is the data of the already blocked pages 
processed/aquired ? It seems to me that the pages get "lost" somewhere.


Best,
Jan-Simon

Re: [Qemu-devel] qemu-arm fails on test-mmap - take #2

[Jan-Simon Möller]

A quick hack around is this:

diff --git a/linux-user/mmap.c b/linux-user/mmap.c
index e05caa0..a04d6b1 100644
--- a/linux-user/mmap.c
+++ b/linux-user/mmap.c
@@ -316,8 +316,14 @@ abi_ulong mmap_find_vma(abi_ulong start, abi_ulong size)
         if (addr == addr_start)
             return (abi_ulong)-1;
     }
-    if (start == 0)
-        mmap_next_start = addr + size;
+    if (start == 0) {
+        abi_ulong mynext = addr + size ;
+        if ( (mynext > 0x5f000000) && (mynext < 0x65000000) ) {
+          mmap_next_start = 0x65000000 ;
+        } else {
+          mmap_next_start = addr + size;
+        }
+    }
     return addr;
 }


But this is for sure not the real solution.

Best,
Jan-Simon

Re: [Qemu-devel] qemu-arm fails on test-mmap

[Martin Mohring]

Hi,

yesterday, we tracked down the problem to a kernel bug in this area:
http://bugzilla.kernel.org/attachment.cgi?id=17219 What do I want to say
with this: the user mode memory allocator does not work because it runs
into a kernel bug.

Kudos to Jan-Simon. Has anyone an idea how to handle such a case....
Patching the kernel is not the best idea. We will check which kernels
are affected.

Cheers, Martin

Jan-Simon Möller wrote:
> A quick hack around is this:
>
> diff --git a/linux-user/mmap.c b/linux-user/mmap.c
> index e05caa0..a04d6b1 100644
> --- a/linux-user/mmap.c
> +++ b/linux-user/mmap.c
> @@ -316,8 +316,14 @@ abi_ulong mmap_find_vma(abi_ulong start, abi_ulong size)
>          if (addr == addr_start)
>              return (abi_ulong)-1;
>      }
> -    if (start == 0)
> -        mmap_next_start = addr + size;
> +    if (start == 0) {
> +        abi_ulong mynext = addr + size ;
> +        if ( (mynext > 0x5f000000) && (mynext < 0x65000000) ) {
> +          mmap_next_start = 0x65000000 ;
> +        } else {
> +          mmap_next_start = addr + size;
> +        }
> +    }
>      return addr;
>  }
>
>
> But this is for sure not the real solution.
>
> Best,
> Jan-Simon
>
>
>   

Re: [Qemu-devel] qemu-arm fails on test-mmap - take #2

[Jan-Simon Möller]

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

Am Montag 10 August 2009 08:33:38 schrieb Kirill A. Shutemov:
> > Now my question is: how is the data of the already blocked pages
> > processed/aquired ? It seems to me that the pages get "lost" somewhere.
>
> Current mmap() implementation requires /proc to obtain all mappings.
>
> I have alternative implementation of mmap_find_vma() which doesn't
> require /proc:
>
> http://www.archivum.info/qemu-devel@nongnu.org/2008-10/01118/%5BQemu-devel%
>5D_%5BPATCH,_v2%5D_Rewrite_mmap_find_vma%28%29_to_work_fine_on_64-bit_hosts_
>with_32-bit_targets
>
> It is probably out of date.

This does the trick !  I've attached a version against trunk.

@Riku: can we apply this to git ?


Acked-by:  Jan-Simon Möller <dl9pf@gmx.de>

Original post: 
http://www.archivum.info/qemu-devel%40nongnu.org/2008-10/01118/[Qemu-devel]_[PATCH,_v2]_Rewrite_mmap_find_vma()_to_work_fine_on_64-bit_hosts_with_32-bit_targets

Attachement:
Patch rebased to trunk.


Best,
Jan-Simon

[-- Attachment #2: qemu-0.11-git-user-linux-mmap_find_vma.patch --]
[-- Type: text/x-diff, Size: 3284 bytes --]

diff --git a/linux-user/mmap.c b/linux-user/mmap.c
index 6f300a0..8cec230 100644
--- a/linux-user/mmap.c
+++ b/linux-user/mmap.c
@@ -275,52 +275,59 @@ abi_ulong mmap_next_start = 0x40000000;
 
 unsigned long last_brk;
 
-/* find a free memory area of size 'size'. The search starts at
-   'start'. If 'start' == 0, then a default start address is used.
-   Return -1 if error.
-*/
-/* page_init() marks pages used by the host as reserved to be sure not
-   to use them. */
-abi_ulong mmap_find_vma(abi_ulong start, abi_ulong size)
+/*
+ * Find and reserve a free memory area of size 'size'. The search
+ * starts at 'start'.
+ * It must be called with mmap_lock() held.
+ * Return -1 if error.
+ */
+abi_ulong mmap_find_vma(abi_ulong start, abi_ulong size)
 {
-    abi_ulong addr, addr1, addr_start;
-    int prot;
-    unsigned long new_brk;
-
-    new_brk = (unsigned long)sbrk(0);
-    if (last_brk && last_brk < new_brk && last_brk == (target_ulong)last_brk) {
-        /* This is a hack to catch the host allocating memory with brk().
-           If it uses mmap then we loose.
-           FIXME: We really want to avoid the host allocating memory in
-           the first place, and maybe leave some slack to avoid switching
-           to mmap.  */
-        page_set_flags(last_brk & TARGET_PAGE_MASK,
-                       TARGET_PAGE_ALIGN(new_brk),
-                       PAGE_RESERVED); 
-    }
-    last_brk = new_brk;
+    void *ptr;
+    abi_ulong addr;
 
     size = HOST_PAGE_ALIGN(size);
-    start = start & qemu_host_page_mask;
+    start &= qemu_host_page_mask;
+
+    /* If 'start' == 0, then a default start address is used. */
+    if (start == 0)
+        start = mmap_next_start;
+
     addr = start;
-    if (addr == 0)
-        addr = mmap_next_start;
-    addr_start = addr;
+
     for(;;) {
-        prot = 0;
-        for(addr1 = addr; addr1 < (addr + size); addr1 += TARGET_PAGE_SIZE) {
-            prot |= page_get_flags(addr1);
-        }
-        if (prot == 0)
+        /*
+         * Reserve needed memory area to avoid a race.
+         * It should be discarded using:
+         *  - mmap() with MAP_FIXED flag
+         *  - mremap() with MREMAP_FIXED flag
+         *  - shmat() with SHM_REMAP flag
+         */
+        ptr = mmap((void *)(unsigned long)addr, size, PROT_NONE,
+                   MAP_ANONYMOUS|MAP_PRIVATE|MAP_NORESERVE, -1, 0);
+
+        /* ENOMEM, if host address space has no memory */
+        if (ptr == MAP_FAILED)
+            return (abi_ulong)-1;
+
+        /* If address fits target address space we've found what we need */
+        if ((unsigned long)ptr + size - 1 <= (abi_ulong)-1)
             break;
+
+        /* Unmap and try again with new page */
+        munmap(ptr, size);
         addr += qemu_host_page_size;
-        /* we found nothing */
-        if (addr == addr_start)
+
+        /* ENOMEM if we check whole of target address space */
+        if (addr == start)
             return (abi_ulong)-1;
     }
-    if (start == 0)
-        mmap_next_start = addr + size;
-    return addr;
+
+    /* Update default start address */
+    if (start == mmap_next_start)
+        mmap_next_start = (unsigned long)ptr + size;
+
+    return h2g(ptr);
 }
 
 /* NOTE: all the constants are the HOST ones */