Re: [Qemu-devel] Fwd: Re: [target-mips] qemu on centos

Re: [Qemu-devel] Fwd: Re: [target-mips] qemu on centos

[Brendan Kirby]

[-- Attachment #1: Type: text/plain, Size: 2309 bytes --]

Attached are three MIPS binaries that I have seen segfault
intermittently on CentOS 6 machines.  Just run them with no arguments
several times.

Brendan

On Fri, 2011-12-23 at 09:00 -0800, Reed Kotler wrote:
> 
> Please work with this guy.
> 
> If possible, submit some test cases to them of some code that causes
> segfaults on Centos 6, even if intermittent.
> 
> Reed
> 
> -------- Original Message -------- 
>                           Subject: 
> Re: [Qemu-devel] [target-mips] qemu
> on centos
>                              Date: 
> Fri, 23 Dec 2011 14:25:57 +0100
>                              From: 
> Andreas Färber <afaerber@suse.de>
>                      Organization: 
> SUSE LINUX Products GmbH
>                                To: 
> Reed Kotler <rkotler@mips.com>
>                                CC: 
> <qemu-devel@nongnu.org>, Khansa
> Butt <khansa@kics.edu.pk>, Richard
> Henderson <rth@twiddle.net>,
> Richard Sandiford
> <rdsandiford@googlemail.com>
> 
> 
> Hi,
> 
> Am 23.12.2011 13:44, schrieb Reed Kotler:
> > We have been seeing various problems running qemu for MIPS target on
> > Centos 5.
> > We are running linux user mode programs.
> > 
> > Qemu segfaults.
> > 
> > We recently tried upgrading to Centos 6 and the problem there is much
> > worse, making it basically unusable.
> > 
> > The same programs have no problem when running on Ubuntu.
> 
> They likely are using different QEMU versions, and distros may have
> different patches applied on top.
> Please provide some more info on what version, what ABI (which
> executable), what -cpu parameter (if any), etc. you are using. Does a
> gdb backtrace indicate any QEMU function or is it from translated code?
> 
> For n64 there were some patches in need of testing, review and fixing.
> [cc'ing Khansa]
> 
> For n32 there's signal handling missing. (openSUSE for one ignores the
> build warning and provides it non the less.)
> 
> No known issues specific to o32 that I'm aware of, but the two Richards
> had patches for some instructions. Shouldn't lead to segfaults though.
> 
> Regards,
> Andreas
> 
> -- 
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg


[-- Attachment #2: ReedSolomon.mipsgcc --]
[-- Type: application/x-executable, Size: 16759 bytes --]

[-- Attachment #3: bisort.llc.mips32r2 --]
[-- Type: application/x-executable, Size: 12406 bytes --]

[-- Attachment #4: bisort.mipsgcc --]
[-- Type: application/x-executable, Size: 13350 bytes --]

Re: [Qemu-devel] Fwd: Re: [target-mips] qemu on centos

[Stefan Weil]

Am 23.12.2011 19:05, schrieb Brendan Kirby:
> Attached are three MIPS binaries that I have seen segfault
> intermittently on CentOS 6 machines. Just run them with no arguments
> several times.
>
> Brendan
>
> On Fri, 2011-12-23 at 09:00 -0800, Reed Kotler wrote:
>>
>> Please work with this guy.
>>
>> If possible, submit some test cases to them of some code that causes
>> segfaults on Centos 6, even if intermittent.
>>
>> Reed
>>
>> -------- Original Message --------
>> Subject:
>> Re: [Qemu-devel] [target-mips] qemu
>> on centos
>> Date:
>> Fri, 23 Dec 2011 14:25:57 +0100
>> From:
>> Andreas Färber <afaerber@suse.de>
>> Organization:
>> SUSE LINUX Products GmbH
>> To:
>> Reed Kotler <rkotler@mips.com>
>> CC:
>> <qemu-devel@nongnu.org>, Khansa
>> Butt <khansa@kics.edu.pk>, Richard
>> Henderson <rth@twiddle.net>,
>> Richard Sandiford
>> <rdsandiford@googlemail.com>
>>
>>
>> Hi,
>>
>> Am 23.12.2011 13:44, schrieb Reed Kotler:
>>> We have been seeing various problems running qemu for MIPS target on
>>> Centos 5.
>>> We are running linux user mode programs.
>>>
>>> Qemu segfaults.
>>>
>>> We recently tried upgrading to Centos 6 and the problem there is much
>>> worse, making it basically unusable.
>>>
>>> The same programs have no problem when running on Ubuntu.
>>
>> They likely are using different QEMU versions, and distros may have
>> different patches applied on top.
>> Please provide some more info on what version, what ABI (which
>> executable), what -cpu parameter (if any), etc. you are using. Does a
>> gdb backtrace indicate any QEMU function or is it from translated code?
>>
>> For n64 there were some patches in need of testing, review and fixing.
>> [cc'ing Khansa]
>>
>> For n32 there's signal handling missing. (openSUSE for one ignores the
>> build warning and provides it non the less.)
>>
>> No known issues specific to o32 that I'm aware of, but the two Richards
>> had patches for some instructions. Shouldn't lead to segfaults though.
>>
>> Regards,
>> Andreas
>>
>> -- 
>> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
>> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg
>

Please add your message at the bottom of the mail, not at the top.

I tried your binaries with latest QEMU. All three fail here each time
with SIGSEGV. This is caused by a jump to address 0 (pc = 0).
Up to now I don't know the reason for this jump.

Call stack:
#0  0x00005555555f62d2 in ldl_le_p (ptr=0x0) at 
/home/stefan/src/qemu/qemu.org/qemu/bswap.h:477
#1  0x000055555561333e in gen_intermediate_code_internal 
(env=0x555557ca0b10, tb=0x7ffff3982108, search_pc=0) at 
/home/stefan/src/qemu/qemu.org/qemu/target-mips/translate.c:12442
#2  0x0000555555613709 in gen_intermediate_code (env=0x555557ca0b10, 
tb=0x7ffff3982108) at 
/home/stefan/src/qemu/qemu.org/qemu/target-mips/translate.c:12528
#3  0x00005555555f5f8d in cpu_mips_gen_code (env=0x555557ca0b10, 
tb=0x7ffff3982108, gen_code_size_ptr=0x7fffffffd748) at 
/home/stefan/src/qemu/qemu.org/qemu/translate-all.c:66
#4  0x00005555555a33b3 in tb_gen_code (env=0x555557ca0b10, pc=0, 
cs_base=0, flags=34, cflags=0) at 
/home/stefan/src/qemu/qemu.org/qemu/exec.c:1003
#5  0x000055555559c5b2 in tb_find_slow (env=0x555557ca0b10, pc=0, 
cs_base=0, flags=34) at /home/stefan/src/qemu/qemu.org/qemu/cpu-exec.c:126
#6  0x000055555559c73c in tb_find_fast (env=0x555557ca0b10) at 
/home/stefan/src/qemu/qemu.org/qemu/cpu-exec.c:153
#7  0x000055555559cb2a in cpu_mips_exec (env=0x555557ca0b10) at 
/home/stefan/src/qemu/qemu.org/qemu/cpu-exec.c:536
#8  0x00005555555bf780 in cpu_loop (env=0x555557ca0b10) at 
/home/stefan/src/qemu/qemu.org/qemu/linux-user/main.c:2160
#9  0x00005555555c16ac in main (argc=7, argv=0x7fffffffe1c8, 
envp=0x7fffffffe208) at 
/home/stefan/src/qemu/qemu.org/qemu/linux-user/main.c:3812

Try running a working version and the bad version with
mipsel-linux-user/qemu-mipsel -d in_asm,cpu -singlestep ...
and compare the resulting output file /tmp/qemu.log.
For programs without external events (timers and other
interrupt sources), the program flow should be identical
(same instructions, same register values).

This should identify the mips code which is handled differently
by the two versions.

Here is an extract of my qemu.log which shows the jump to
address 0.

pc=0x004027d8 HI=0x0000018a LO=0x0000f816 ds 0022 004027c0 0
GPR00: r0 00000000 at fffffff8 v0 4081190c v1 00000814
GPR04: a0 0040248c a1 00000001 a2 4080042c a3 00402650
GPR08: t0 004026f4 t1 0ffffffe t2 00000063 t3 00000002
GPR12: t4 40800180 t5 40800228 t6 ffffffff t7 00400538
GPR16: s0 4083a010 s1 004004f0 s2 00000000 s3 00000000
GPR20: s4 00000000 s5 00000000 s6 00000000 s7 00000000
GPR24: t8 00000002 t9 00000000 k0 00000000 k1 00000000
GPR28: gp 00412804 sp 40800408 s8 00000000 ra 00400538
CP0 Status  0x00000000 Cause   0x00000000 EPC    0x00000000
     Config0 0x80000482 Config1 0x9e190c8f LLAddr 0xffffffff
CP1 FCR0 0x00000000  FCR31 0x00000000  SR.FR 0  fp_status 0x00
  f0: w:3f800000 d:400000003f800000 fd:  4.61169e+18 fs:  1.06535e+09 
psu:  1.07374e+09
  f2: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
  f4: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
  f6: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
  f8: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f10: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f12: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f14: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f16: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f18: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f20: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f22: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f24: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f26: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f28: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f30: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
IN:
0x004027d8:  jalr       t9

An older qemu-mipsel from August fails, too.

Regards,
Stefan Weil

Re: [Qemu-devel] Fwd: Re: [target-mips] qemu on centos

[Brendan Kirby]

On Fri, 2011-12-23 at 23:57 +0100, Stefan Weil wrote:
> Am 23.12.2011 19:05, schrieb Brendan Kirby:
> > Attached are three MIPS binaries that I have seen segfault
> > intermittently on CentOS 6 machines. Just run them with no arguments
> > several times.
> >
> > Brendan
> >
> > On Fri, 2011-12-23 at 09:00 -0800, Reed Kotler wrote:
> >>
> >> Please work with this guy.
> >>
> >> If possible, submit some test cases to them of some code that causes
> >> segfaults on Centos 6, even if intermittent.
> >>
> >> Reed
> >>
> >> -------- Original Message --------
> >> Subject:
> >> Re: [Qemu-devel] [target-mips] qemu
> >> on centos
> >> Date:
> >> Fri, 23 Dec 2011 14:25:57 +0100
> >> From:
> >> Andreas Färber <afaerber@suse.de>
> >> Organization:
> >> SUSE LINUX Products GmbH
> >> To:
> >> Reed Kotler <rkotler@mips.com>
> >> CC:
> >> <qemu-devel@nongnu.org>, Khansa
> >> Butt <khansa@kics.edu.pk>, Richard
> >> Henderson <rth@twiddle.net>,
> >> Richard Sandiford
> >> <rdsandiford@googlemail.com>
> >>
> >>
> >> Hi,
> >>
> >> Am 23.12.2011 13:44, schrieb Reed Kotler:
> >>> We have been seeing various problems running qemu for MIPS target on
> >>> Centos 5.
> >>> We are running linux user mode programs.
> >>>
> >>> Qemu segfaults.
> >>>
> >>> We recently tried upgrading to Centos 6 and the problem there is much
> >>> worse, making it basically unusable.
> >>>
> >>> The same programs have no problem when running on Ubuntu.
> >>
> >> They likely are using different QEMU versions, and distros may have
> >> different patches applied on top.
> >> Please provide some more info on what version, what ABI (which
> >> executable), what -cpu parameter (if any), etc. you are using. Does a
> >> gdb backtrace indicate any QEMU function or is it from translated code?
> >>
> >> For n64 there were some patches in need of testing, review and fixing.
> >> [cc'ing Khansa]
> >>
> >> For n32 there's signal handling missing. (openSUSE for one ignores the
> >> build warning and provides it non the less.)
> >>
> >> No known issues specific to o32 that I'm aware of, but the two Richards
> >> had patches for some instructions. Shouldn't lead to segfaults though.
> >>
> >> Regards,
> >> Andreas
> >>
> >> -- 
> >> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
> >> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg
> >
> 
> Please add your message at the bottom of the mail, not at the top.

> I tried your binaries with latest QEMU. All three fail here each time
> with SIGSEGV. This is caused by a jump to address 0 (pc = 0).
> Up to now I don't know the reason for this jump.

The binaries should be the same on both the CentOS 5 and CentOS 6
machines.  (They're compiled with the same compiler, at least)  But,
I'll verify that.  I neglected to mention the version of QEMU I'm
running.  Here is the version string:
qemu-mips version 0.14.50 (Sourcery CodeBench 2011.03-95), Copyright (c)
2003-2008 Fabrice Bellard, 2008-2011 Mentor Graphics

And, the version string for GCC that the two mipsgcc binaries were
compiled with is:
mips-linux-gnu-gcc (Sourcery CodeBench 2011.03-95) 4.5.2
Copyright (C) 2010 Free Software Foundation, Inc.

> Call stack:
> #0  0x00005555555f62d2 in ldl_le_p (ptr=0x0) at 
> /home/stefan/src/qemu/qemu.org/qemu/bswap.h:477
> #1  0x000055555561333e in gen_intermediate_code_internal 
> (env=0x555557ca0b10, tb=0x7ffff3982108, search_pc=0) at 
> /home/stefan/src/qemu/qemu.org/qemu/target-mips/translate.c:12442
> #2  0x0000555555613709 in gen_intermediate_code (env=0x555557ca0b10, 
> tb=0x7ffff3982108) at 
> /home/stefan/src/qemu/qemu.org/qemu/target-mips/translate.c:12528
> #3  0x00005555555f5f8d in cpu_mips_gen_code (env=0x555557ca0b10, 
> tb=0x7ffff3982108, gen_code_size_ptr=0x7fffffffd748) at 
> /home/stefan/src/qemu/qemu.org/qemu/translate-all.c:66
> #4  0x00005555555a33b3 in tb_gen_code (env=0x555557ca0b10, pc=0, 
> cs_base=0, flags=34, cflags=0) at 
> /home/stefan/src/qemu/qemu.org/qemu/exec.c:1003
> #5  0x000055555559c5b2 in tb_find_slow (env=0x555557ca0b10, pc=0, 
> cs_base=0, flags=34) at /home/stefan/src/qemu/qemu.org/qemu/cpu-exec.c:126
> #6  0x000055555559c73c in tb_find_fast (env=0x555557ca0b10) at 
> /home/stefan/src/qemu/qemu.org/qemu/cpu-exec.c:153
> #7  0x000055555559cb2a in cpu_mips_exec (env=0x555557ca0b10) at 
> /home/stefan/src/qemu/qemu.org/qemu/cpu-exec.c:536
> #8  0x00005555555bf780 in cpu_loop (env=0x555557ca0b10) at 
> /home/stefan/src/qemu/qemu.org/qemu/linux-user/main.c:2160
> #9  0x00005555555c16ac in main (argc=7, argv=0x7fffffffe1c8, 
> envp=0x7fffffffe208) at 
> /home/stefan/src/qemu/qemu.org/qemu/linux-user/main.c:3812
> 
> Try running a working version and the bad version with
> mipsel-linux-user/qemu-mipsel -d in_asm,cpu -singlestep ...
> and compare the resulting output file /tmp/qemu.log.
> For programs without external events (timers and other
> interrupt sources), the program flow should be identical
> (same instructions, same register values).
> 
> This should identify the mips code which is handled differently
> by the two versions.
> 
> Here is an extract of my qemu.log which shows the jump to
> address 0.
> 
> pc=0x004027d8 HI=0x0000018a LO=0x0000f816 ds 0022 004027c0 0
> GPR00: r0 00000000 at fffffff8 v0 4081190c v1 00000814
> GPR04: a0 0040248c a1 00000001 a2 4080042c a3 00402650
> GPR08: t0 004026f4 t1 0ffffffe t2 00000063 t3 00000002
> GPR12: t4 40800180 t5 40800228 t6 ffffffff t7 00400538
> GPR16: s0 4083a010 s1 004004f0 s2 00000000 s3 00000000
> GPR20: s4 00000000 s5 00000000 s6 00000000 s7 00000000
> GPR24: t8 00000002 t9 00000000 k0 00000000 k1 00000000
> GPR28: gp 00412804 sp 40800408 s8 00000000 ra 00400538
> CP0 Status  0x00000000 Cause   0x00000000 EPC    0x00000000
>      Config0 0x80000482 Config1 0x9e190c8f LLAddr 0xffffffff
> CP1 FCR0 0x00000000  FCR31 0x00000000  SR.FR 0  fp_status 0x00
>   f0: w:3f800000 d:400000003f800000 fd:  4.61169e+18 fs:  1.06535e+09 
> psu:  1.07374e+09
>   f2: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
>   f4: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
>   f6: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
>   f8: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
> f10: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
> f12: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
> f14: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
> f16: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
> f18: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
> f20: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
> f22: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
> f24: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
> f26: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
> f28: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
> f30: w:00000000 d:0000000000000000 fd:            0 fs:            0 
> psu:            0
> IN:
> 0x004027d8:  jalr       t9
> 
> An older qemu-mipsel from August fails, too.

Thanks for looking at this.  I'll give this a try too and see if I can
duplicate your results.  

Brendan

> Regards,
> Stefan Weil
> 
> 

Re: [Qemu-devel] [target-mips] qemu on centos

[Stefan Weil]

Am 23.12.2011 23:57, schrieb Stefan Weil:
> Am 23.12.2011 19:05, schrieb Brendan Kirby:
>> Attached are three MIPS binaries that I have seen segfault
>> intermittently on CentOS 6 machines. Just run them with no arguments
>> several times.
>>
>> Brendan
>>
> I tried your binaries with latest QEMU. All three fail here each time
> with SIGSEGV. This is caused by a jump to address 0 (pc = 0).
> Up to now I don't know the reason for this jump.

[snip]

> An older qemu-mipsel from August fails, too.
>
> Regards,
> Stefan Weil

A version from May is better: it also has a jump to address 0,
but handles it correctly:

qemu-mipsel -L /media/vm/tftpboot/mips/malta-le mipsbin/bisort.llc.mips32r2
qemu: unhandled CPU exception 0xc - aborting
pc=0x00000000 HI=0x0000018a LO=0x0000f816 ds 0022 00000000 0
GPR00: r0 00000000 at fffffff8 v0 4081190c v1 00000814
GPR04: a0 0040107c a1 00000001 a2 4080043c a3 004012a0
GPR08: t0 00401344 t1 0ffffffe t2 00000063 t3 00000002
GPR12: t4 40800190 t5 40800238 t6 ffffffff t7 004006a8
GPR16: s0 4083a010 s1 00400660 s2 00000000 s3 00000000
GPR20: s4 00000000 s5 00000000 s6 00000000 s7 00000000
GPR24: t8 00000000 t9 00000000 k0 00000000 k1 00000000
GPR28: gp 00411544 sp 40800418 s8 00000000 ra 00401520
CP0 Status  0x00000000 Cause   0x00000000 EPC    0x00000000
     Config0 0x80000482 Config1 0x9e190c8f LLAddr 0xffffffff
CP1 FCR0 0x00000000  FCR31 0x00000000  SR.FR 0  fp_status 0x00
  f0: w:3f800000 d:400000003f800000 fd:  4.61169e+18 fs:  1.06535e+09 
psu:  1.07374e+09
  f2: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
  f4: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
  f6: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
  f8: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f10: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f12: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f14: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f16: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f18: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f20: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f22: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f24: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f26: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f28: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
f30: w:00000000 d:0000000000000000 fd:            0 fs:            0 
psu:            0
qemu: uncaught target signal 6 (Aborted) - core dumped

Obviously signal handling for SIGSEGV in user code changed.
It now raises a SIGSEGV on the host...

Merry Christmas

Stefan Weil

Re: [Qemu-devel] Fwd: Re: [target-mips] qemu on centos

[Andreas Färber]

Hello Brendan,

Happy New Year.

Am 23.12.2011 19:05, schrieb Brendan Kirby:
> Attached are three MIPS binaries that I have seen segfault
> intermittently on CentOS 6 machines.  Just run them with no arguments
> several times.

Unfortunately those binaries are all dynamically linked. Even if I use
'-L path/to/linux-user-0.3/gnemul/qemu-mipsel' I still get dependency
issues, such as libstdc++.so.6 and libpthread.so.0.

Can you either supply me with a statically linked binary or do the
git-bisect yourself and let us know which commit introduced the
regression you're seeing?

Thanks,

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg