[Qemu-devel] qemu-user + networking issues / segfaults

[Qemu-devel] qemu-user + networking issues / segfaults

[Christof Schulze]

[-- Attachment #1: Type: text/plain, Size: 2428 bytes --]

Hello qemu-devel list,

This is my first post to this list and I am not sure whether this
actually is the correct Mailinglist.  I recently compiled qemu-1.6.0
on an arm platform for the purpose of running the binary only
otrdecoder-software which is available for a 64bit linux only. I
pursued the following steps:
* creating a chroot on my x64-box that contained the otrdecoder and
  all libraries it needs to run
* test-run the otrdecoder from within the chroot (it works)
* copying this chroot to my arm box, where I compiled qemu previously
* copying qemu and all required libs to the chroot
* copying a shell to the chroot
* copying libnss* libraries from my 64bit system and from my arm
  system to the chroot
* test network connectivity from within the chroot using native
  nslookup and native ping (it works)
* from within the chroot I ran the otrdecoder using qemu-x86_64 which
  works up to a point where it segfaults.
  
running qemu using the -strace flag and comparing the output with a
successful strace from my 64bit-box reveals that the segfault happens
after an munmap and before (or at) the spot where a socket() operation
is run.

This is the operation that should be run:
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3

I am not sure if qemu segfaults because 
* there are actually libs missing in the chroot 
* the syscall is not supported
* the binary does crazy things and is not supported by qemu-user
  
At the same time running the 64bit version of ping results in a
segfault as well which might be related.

this is what the segfault of the otrdecoder shows:
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
SYS_369(0, 0x4, 0, 0xbe9f6d48, 0x4)     = 0
SYS_369(0, 0x4, 0, 0xbe9f8dd8, 0x4)     = 0
SYS_369(0, 0x4, 0xbe9f8dd8, 0, 0xbe9f8dd8) = 0
futex(0xb6dcf7d0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(2, "qemu: uncaught target signal 11 "..., 67qemu: uncaught target signal 
11 (Segmentation fault) - core dumped
) = 67
rt_sigaction(SIGSEGV, {SIG_DFL, ~[RTMIN RT_1], SA_INTERRUPT|SA_NODEFER|
0x5199d28}, NULL, 8) = 0
kill(30161, SIGSEGV)                    = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++

What can I do to investigate further and get this fixed besides trying
to emulate a full-blown system?

Christof

-- 
()  ascii ribbon campaign - against html e-mail
/\  www.asciiribbon.org   - against proprietary attachments

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Qemu-devel] qemu-user + networking issues / segfaults

[Christof Schulze]

[-- Attachment #1: Type: text/plain, Size: 5571 bytes --]

Am Samstag, 24. August 2013, 00:21:57 schrieb Christof Schulze:
> Hello qemu-devel list,

> This is my first post to this list and I am not sure whether this
> actually is the correct Mailinglist.  I recently compiled qemu-1.6.0
> on an arm platform for the purpose of running the binary only
> otrdecoder-software which is available for a 64bit linux only. I
> pursued the following steps:
> * creating a chroot on my x64-box that contained the otrdecoder and
>   all libraries it needs to run
> * test-run the otrdecoder from within the chroot (it works)
> * copying this chroot to my arm box, where I compiled qemu previously
> * copying qemu and all required libs to the chroot
> * copying a shell to the chroot
> * copying libnss* libraries from my 64bit system and from my arm
>   system to the chroot
> * test network connectivity from within the chroot using native
>   nslookup and native ping (it works)
> * from within the chroot I ran the otrdecoder using qemu-x86_64 which
>   works up to a point where it segfaults.

> running qemu using the -strace flag and comparing the output with a
> successful strace from my 64bit-box reveals that the segfault happens
> after an munmap and before (or at) the spot where a socket() operation
> is run.

> This is the operation that should be run:
> socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3

> I am not sure if qemu segfaults because
> * there are actually libs missing in the chroot
> * the syscall is not supported
> * the binary does crazy things and is not supported by qemu-user

> At the same time running the 64bit version of ping results in a
> segfault as well which might be related.

> this is what the segfault of the otrdecoder shows:
> --- SIGSEGV (Segmentation fault) @ 0 (0) ---
> rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
> SYS_369(0, 0x4, 0, 0xbe9f6d48, 0x4)     = 0
> SYS_369(0, 0x4, 0, 0xbe9f8dd8, 0x4)     = 0
> SYS_369(0, 0x4, 0xbe9f8dd8, 0, 0xbe9f8dd8) = 0
> futex(0xb6dcf7d0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
> write(2, "qemu: uncaught target signal 11 "..., 67qemu: uncaught target
> signal 11 (Segmentation fault) - core dumped
> ) = 67
> rt_sigaction(SIGSEGV, {SIG_DFL, ~[RTMIN RT_1], SA_INTERRUPT|SA_NODEFER|
> 0x5199d28}, NULL, 8) = 0
> kill(30161, SIGSEGV)                    = 0
> --- SIGSEGV (Segmentation fault) @ 0 (0) ---
> +++ killed by SIGSEGV +++

> What can I do to investigate further and get this fixed besides trying
> to emulate a full-blown system?
After reading this https://bugs.launchpad.net/qemu/+bug/739785, I
started another try with a statically built version of qemu-1.6.0 with
enabled debugging options. As discussed on irc I am cc'ing rth.

After I copied qemu-x86_64-static to the chroot things still lead to a
segfault. This happens as well when qemu-user is run with "-R 32M".

A 64bit version of ping and the otrdecoder (the application that I am
actually trying to run) show the same symptoms and apart from the memory
addresses exactly the same backtrace.

Please find below a backtrace of the crash:
#0  0x600dcb8c in ldub_p (ptr=0xff600400) at /mnt/data/build/qemu-1.6.0-
ministatic/include/qemu/bswap.h:223
#1  0x60104fc4 in disas_insn (env=0x612def20, s=0xbeffeb30, 
pc_start=18446744073699066880)
    at /mnt/data/build/qemu-1.6.0-ministatic/target-i386/translate.c:4692
#2  0x60113abc in gen_intermediate_code_internal (cpu=0x612dee60, 
tb=0xb5a869a0, search_pc=false)
    at /mnt/data/build/qemu-1.6.0-ministatic/target-i386/translate.c:8367
#3  0x60113ea0 in gen_intermediate_code (env=0x612def20, tb=0xb5a869a0)
    at /mnt/data/build/qemu-1.6.0-ministatic/target-i386/translate.c:8433
#4  0x60128638 in cpu_x86_gen_code (env=0x612def20, tb=0xb5a869a0, 
gen_code_size_ptr=0xbeffecc8)
    at /mnt/data/build/qemu-1.6.0-ministatic/translate-all.c:155
#5  0x6012a100 in tb_gen_code (env=0x612def20, pc=18446744073699066880, 
cs_base=0, flags=4243635, cflags=0)
    at /mnt/data/build/qemu-1.6.0-ministatic/translate-all.c:964
#6  0x600402c4 in tb_find_slow (env=0x612def20, pc=18446744073699066880, 
cs_base=0, flags=4243635)
    at /mnt/data/build/qemu-1.6.0-ministatic/cpu-exec.c:145
#7  0x60040508 in tb_find_fast (env=0x612def20) at /mnt/data/build/qemu-1.6.0-
ministatic/cpu-exec.c:172
#8  0x60040c60 in cpu_x86_exec (env=0x612def20) at /mnt/data/build/qemu-1.6.0-
ministatic/cpu-exec.c:599
#9  0x6006fedc in cpu_loop (env=0x612def20) at /mnt/data/build/qemu-1.6.0-
ministatic/linux-user/main.c:283
#10 0x60072110 in main (argc=13, argv=0xbefff714, envp=0xbefff74c) at 
/mnt/data/build/qemu-1.6.0-ministatic/linux-user/main.c:4079

A 64bit version of ls does not crash however the output is mangled and
incomplete - also a sign of corruption. So many programs seem to be
affected.

Additional information that might be useful:
cat /proc/cpu/alignment 
User:           411600089
System:         5057
Skipped:        0
Half:           61365313
Word:           350239832
DWord:          1
Multi:          0
User faults:    2 (fixup)

# cat /proc/cpuinfo 
processor       : 0
model name      : Feroceon 88FR131 rev 1 (v5l)
BogoMIPS        : 1191.11
Features        : swp half thumb fastmult edsp 
CPU implementer : 0x56
CPU architecture: 5TE
CPU variant     : 0x2
CPU part        : 0x131
CPU revision    : 1

Hardware        : Marvell GuruPlug Reference Board
Revision        : 0000
Serial          : 0000000000000000

What further input is required? If needed, I can provide my chroot that
has only ~50MB via torrent.

Christof
-- 
()  ascii ribbon campaign - against html e-mail
/\  www.asciiribbon.org   - against proprietary attachments

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Qemu-devel] qemu-user + networking issues / segfaults

[Richard Henderson]

[-- Attachment #1: Type: text/plain, Size: 511 bytes --]

On 08/29/2013 02:27 PM, Christof Schulze wrote:
> #5  0x6012a100 in tb_gen_code (env=0x612def20, pc=18446744073699066880, 
> cs_base=0, flags=4243635, cflags=0)
>     at /mnt/data/build/qemu-1.6.0-ministatic/translate-all.c:964

In know exactly what this is -- the fallback vsyscall page.

I've had a patch set around for the last three years or so to provide
a real vdso for x86_64.  I repost it every so often, usually to no
response.

See git://github.com/rth7680/qemu.git elfload-vdso


r~


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 555 bytes --]

[Qemu-devel] qemu-user-x86_64 segfaults on armv5 (WAS: qemu-user + networking issues / segfaults)

[Christof Schulze]

[-- Attachment #1: Type: text/plain, Size: 2044 bytes --]

Hello

Am Mittwoch, 4. September 2013, 08:35:37 schrieb Richard Henderson:
> On 08/29/2013 02:27 PM, Christof Schulze wrote:
> > #5  0x6012a100 in tb_gen_code (env=0x612def20, pc=18446744073699066880,
> > cs_base=0, flags=4243635, cflags=0)

> >     at /mnt/data/build/qemu-1.6.0-ministatic/translate-all.c:964

> In know exactly what this is -- the fallback vsyscall page.

> I've had a patch set around for the last three years or so to provide
> a real vdso for x86_64.  I repost it every so often, usually to no
> response.

> See git://github.com/rth7680/qemu.git elfload-vdso
after trying the patch itself and still getting slightly other crashes
on irc we decided to try this patch based on the master branch. This
allowed for debugging with gdb as it circumvented a bug of the early
1.6rcs of qemu where the g packet was transmitting two many registers.


Having set a breakpoint at *0x0000000040816725 (the memory position
where the segfault happens) I got the following
output.

Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x0000000040802650 in _start () from /lib64/ld-linux-x86-64.so.2
(gdb) b *0x0000000040816725
Haltepunkt 1 at 0x40816725: file ../sysdeps/x86_64/dl-trampoline.S, line 46.
(gdb) cont
Continuing.
Breakpoint 1, _dl_runtime_resolve () at ../sysdeps/x86_64/dl-trampoline.S:46
46	../sysdeps/x86_64/dl-trampoline.S: Datei oder Verzeichnis nicht gefunden.
it says dl-trampoline.S: file or directory not found and gives me the
gdb shell

Breakpoint 1, _dl_runtime_resolve () at ../sysdeps/x86_64/dl-trampoline.S:46
46	in ../sysdeps/x86_64/dl-trampoline.S
(gdb) 
Continuing.
Program received signal SIGSEGV, Segmentation fault.
0xffffffffff600400 in ?? ()

RTH mentioned that the runtime resolution of the symbol was into the
vsyscall page.

Now I am unsure where I should go from this point.
Should I collect more data? If so, what exactly is needed? 
What else could I do to get this resolved?

Christof

-- 
()  ascii ribbon campaign - against html e-mail
/\  www.asciiribbon.org   - against proprietary attachments

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Qemu-devel] qemu-user-x86_64 segfaults on armv5 (WAS: qemu-user + networking issues / segfaults)

[Christof Schulze]

[-- Attachment #1: Type: text/plain, Size: 2549 bytes --]

Hello

Richard, using your vdso patch from today that takes into account vtime
as well, running linux-x86_64 code on arm via qemu-user works! Thank you
very much for your support and energy! 

As promised on IRC the software is dead-slow but I do not mind since it
is a batch application anyways.

I only wonder why this is not in the master branch.

Christof

Am Freitag, 6. September 2013, 12:38:21 schrieb Christof Schulze:
> Hello

> Am Mittwoch, 4. September 2013, 08:35:37 schrieb Richard Henderson:
> > On 08/29/2013 02:27 PM, Christof Schulze wrote:
> > > #5  0x6012a100 in tb_gen_code (env=0x612def20, pc=18446744073699066880,
> > > cs_base=0, flags=4243635, cflags=0)

> > >     at /mnt/data/build/qemu-1.6.0-ministatic/translate-all.c:964

> > In know exactly what this is -- the fallback vsyscall page.

> > I've had a patch set around for the last three years or so to provide
> > a real vdso for x86_64.  I repost it every so often, usually to no
> > response.

> > See git://github.com/rth7680/qemu.git elfload-vdso

> after trying the patch itself and still getting slightly other crashes
> on irc we decided to try this patch based on the master branch. This
> allowed for debugging with gdb as it circumvented a bug of the early
> 1.6rcs of qemu where the g packet was transmitting two many registers.


> Having set a breakpoint at *0x0000000040816725 (the memory position
> where the segfault happens) I got the following
> output.

> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> 0x0000000040802650 in _start () from /lib64/ld-linux-x86-64.so.2
> (gdb) b *0x0000000040816725
> Haltepunkt 1 at 0x40816725: file ../sysdeps/x86_64/dl-trampoline.S, line 46.
> (gdb) cont
> Continuing.
> Breakpoint 1, _dl_runtime_resolve () at ../sysdeps/x86_64/dl-trampoline.S:46
> 46	../sysdeps/x86_64/dl-trampoline.S: Datei oder Verzeichnis nicht
> gefunden. it says dl-trampoline.S: file or directory not found and gives me
> the gdb shell

> Breakpoint 1, _dl_runtime_resolve () at ../sysdeps/x86_64/dl-trampoline.S:46
> 46	in ../sysdeps/x86_64/dl-trampoline.S
> (gdb)
> Continuing.
> Program received signal SIGSEGV, Segmentation fault.
> 0xffffffffff600400 in ?? ()

> RTH mentioned that the runtime resolution of the symbol was into the
> vsyscall page.

> Now I am unsure where I should go from this point.
> Should I collect more data? If so, what exactly is needed?
> What else could I do to get this resolved?

> Christof
-- 
()  ascii ribbon campaign - against html e-mail
/\  www.asciiribbon.org   - against proprietary attachments

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]