[Qemu-devel] Reporting Heisenbugs in qemu

[Qemu-devel] Reporting Heisenbugs in qemu

[Torbjorn Granlund]

I am trying to use qemu to

1. cover more of the assembly code in GMP
2. check configuration logic of GMP

but I am not as successful as I would like to be.

The 2nd table of http://gmplib.org/devel/testsystems.html shows all
emulated systems I am using, most of which are qemu-based.

Unfortunately, several of the qemu-based systems experience intermittent
but common segfaults:

1. Linux mips64eb 2.6.32-5-5kc-malta #1 Sun Sep 23 12:29:36 UTC 2012 mips64 GNU/Linux
2. Linux mips64el 2.6.32-5-5kc-malta #1 Fri Feb 15 21:38:11 UTC 2013 mips64 GNU/Linux
3. Linux kick.gmplib.org 2.6.18-6-sparc32 #1 Sat Dec 27 09:13:12 UTC 2008 sparc GNU/Linux

An example of a failure is:

gmp/tests/cxx/t-ops2.cc: In function 'void checkz()':
gmp/tests/cxx/t-ops2.cc:86: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See <URL:http://gcc.gnu.org/bugs.html> for instructions.
For Debian GNU/Linux specific bug reporting instructions,
see <URL:file:///usr/share/doc/gcc-4.1/README.Bugs>.
The bug is not reproducible, so it is likely a hardware or OS problem.

(This was from the sparc32 system.)

rootrem.c: In function 'mpn_rootrem_internal':
rootrem.c:120:1: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.6/README.Bugs> for instructions.
The bug is not reproducible, so it is likely a hardware or OS problem.

(From the mips64eb system.)

I am aware of that these systems don't exactly use the
kernel-of-the-week.  Newer kernels I have tried cause non-boot.  (I
don't think I've tried any newer sparc kernel, as building that would
require a stable sparc system...)

I realise that linux might have been debugged until it works on real
hardware, but that qemu might trigger untested linux execution paths.

Yesterday, I disabled GMP testing on these qemu systems, as I got tired
of the many false alarms, and since GMP looked bad.  Is there any hope
that these qemu systems will become stable?  Or aren't these problems
qemu's fault?

-- 
Torbjörn

Re: [Qemu-devel] Reporting Heisenbugs in qemu

[Mark Cave-Ayland]

On 07/05/13 12:38, Torbjorn Granlund wrote:

Hi Torbjorn,

> I am trying to use qemu to
>
> 1. cover more of the assembly code in GMP
> 2. check configuration logic of GMP
>
> but I am not as successful as I would like to be.
>
> The 2nd table of http://gmplib.org/devel/testsystems.html shows all
> emulated systems I am using, most of which are qemu-based.

Wow - that's a really impressive setup :)

> Unfortunately, several of the qemu-based systems experience intermittent
> but common segfaults:
>
> 1. Linux mips64eb 2.6.32-5-5kc-malta #1 Sun Sep 23 12:29:36 UTC 2012 mips64 GNU/Linux
> 2. Linux mips64el 2.6.32-5-5kc-malta #1 Fri Feb 15 21:38:11 UTC 2013 mips64 GNU/Linux
> 3. Linux kick.gmplib.org 2.6.18-6-sparc32 #1 Sat Dec 27 09:13:12 UTC 2008 sparc GNU/Linux
>
> An example of a failure is:
>
> gmp/tests/cxx/t-ops2.cc: In function 'void checkz()':
> gmp/tests/cxx/t-ops2.cc:86: internal compiler error: Segmentation fault
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See<URL:http://gcc.gnu.org/bugs.html>  for instructions.
> For Debian GNU/Linux specific bug reporting instructions,
> see<URL:file:///usr/share/doc/gcc-4.1/README.Bugs>.
> The bug is not reproducible, so it is likely a hardware or OS problem.
>
> (This was from the sparc32 system.)
>
> rootrem.c: In function 'mpn_rootrem_internal':
> rootrem.c:120:1: internal compiler error: Segmentation fault
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See<file:///usr/share/doc/gcc-4.6/README.Bugs>  for instructions.
> The bug is not reproducible, so it is likely a hardware or OS problem.
>
> (From the mips64eb system.)
>
> I am aware of that these systems don't exactly use the
> kernel-of-the-week.  Newer kernels I have tried cause non-boot.  (I
> don't think I've tried any newer sparc kernel, as building that would
> require a stable sparc system...)
>
> I realise that linux might have been debugged until it works on real
> hardware, but that qemu might trigger untested linux execution paths.
>
> Yesterday, I disabled GMP testing on these qemu systems, as I got tired
> of the many false alarms, and since GMP looked bad.  Is there any hope
> that these qemu systems will become stable?  Or aren't these problems
> qemu's fault?

I actually spent quite a bit of time a couple of years ago trying to 
troubleshoot a similar intermittent buildfarm compiler blowout in a Xen 
x86 guest, and in the end it turned out to be simply a lack of memory 
(both physical and virtual) assigned to the VM.

It just so happened that the code in question was tickling a gcc bug 
which caused excess memory usage during compilation, but only when 
compiling with a certain combination of flags. This may not necessarily 
be the problem here, but it's certainly worth a little experimentation 
first before looking deeper into QEMU.


HTH,

Mark.

Re: [Qemu-devel] Reporting Heisenbugs in qemu

[Aurelien Jarno]

On Tue, May 07, 2013 at 01:38:57PM +0200, Torbjorn Granlund wrote:
> I am trying to use qemu to
> 
> 1. cover more of the assembly code in GMP
> 2. check configuration logic of GMP
> 
> but I am not as successful as I would like to be.
> 
> The 2nd table of http://gmplib.org/devel/testsystems.html shows all
> emulated systems I am using, most of which are qemu-based.
> 
> Unfortunately, several of the qemu-based systems experience intermittent
> but common segfaults:
> 
> 1. Linux mips64eb 2.6.32-5-5kc-malta #1 Sun Sep 23 12:29:36 UTC 2012 mips64 GNU/Linux
> 2. Linux mips64el 2.6.32-5-5kc-malta #1 Fri Feb 15 21:38:11 UTC 2013 mips64 GNU/Linux
> 3. Linux kick.gmplib.org 2.6.18-6-sparc32 #1 Sat Dec 27 09:13:12 UTC 2008 sparc GNU/Linux
> 
> An example of a failure is:
> 
> gmp/tests/cxx/t-ops2.cc: In function 'void checkz()':
> gmp/tests/cxx/t-ops2.cc:86: internal compiler error: Segmentation fault
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <URL:http://gcc.gnu.org/bugs.html> for instructions.
> For Debian GNU/Linux specific bug reporting instructions,
> see <URL:file:///usr/share/doc/gcc-4.1/README.Bugs>.
> The bug is not reproducible, so it is likely a hardware or OS problem.
> 
> (This was from the sparc32 system.)
> 
> rootrem.c: In function 'mpn_rootrem_internal':
> rootrem.c:120:1: internal compiler error: Segmentation fault
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <file:///usr/share/doc/gcc-4.6/README.Bugs> for instructions.
> The bug is not reproducible, so it is likely a hardware or OS problem.
> 
> (From the mips64eb system.)
> 
> I am aware of that these systems don't exactly use the
> kernel-of-the-week.  Newer kernels I have tried cause non-boot.  (I
> don't think I've tried any newer sparc kernel, as building that would
> require a stable sparc system...)
> 
> I realise that linux might have been debugged until it works on real
> hardware, but that qemu might trigger untested linux execution paths.
> 
> Yesterday, I disabled GMP testing on these qemu systems, as I got tired
> of the many false alarms, and since GMP looked bad.  Is there any hope
> that these qemu systems will become stable?  Or aren't these problems
> qemu's fault?
> 

Which version of QEMU are you using? This looks like a bug that has
appeared in version 1.3.0 and fixed in version 1.4.1. The corresponding
commit in master is 52ae646d4a3ebdcdcc973492c6a56f2c49b6578f

-- 
Aurelien Jarno                          GPG: 1024D/F1BCDB73
aurelien@aurel32.net                 http://www.aurel32.net

Re: [Qemu-devel] Reporting Heisenbugs in qemu

[Torbjorn Granlund]

Aurelien Jarno <aurelien@aurel32.net> writes:

  Which version of QEMU are you using? This looks like a bug that has
  appeared in version 1.3.0 and fixed in version 1.4.1. The corresponding
  commit in master is 52ae646d4a3ebdcdcc973492c6a56f2c49b6578f

I cannot tell exactly, since I didn't keep sources to each qemu compile.

The compile for mips64e{l,b} is from 2013-03-06, the sources are not
much older.

Older qemu versions determinisitcally crashed for GMP's "make check"
target for both mips64 targets.  (I cannot recall exactly how qemu
crashed, I think it might have been an abort trap.)

The compile for sparc32 is from 2013-03-19.

I now updated these systems to a qemu build, and will re-enable
testing.  (Results tomorrow.)

-- 
Torbjörn

Re: [Qemu-devel] Reporting Heisenbugs in qemu

[Artyom Tarasenko]

On Tue, May 7, 2013 at 1:38 PM, Torbjorn Granlund <tg@gmplib.org> wrote:
> The 2nd table of http://gmplib.org/devel/testsystems.html shows all
> emulated systems I am using, most of which are qemu-based.

Do I read it correct that qemu-system-ppc64 with the slowdown factor
of 33 is ~3 times faster than qemu-system-sparc64 with the slowdown
factor of 96 ?
Do they both use Debian Wheezy guest? You have a remark that ppc64 has
problems with its clock. Was it taken into account when the slowdown
factors were calculated?

Artyom

--
Regards,
Artyom Tarasenko

linux/sparc and solaris/sparc under qemu blog:
http://tyom.blogspot.com/search/label/qemu

Re: [Qemu-devel] Reporting Heisenbugs in qemu

[Torbjorn Granlund]

Artyom Tarasenko <atar4qemu@gmail.com> writes:

  Do I read it correct that qemu-system-ppc64 with the slowdown factor
  of 33 is ~3 times faster than qemu-system-sparc64 with the slowdown
  factor of 96 ?

You read it correctly.

But please see the caveat at the table end.

  Do they both use Debian Wheezy guest? You have a remark that ppc64 has
  problems with its clock. Was it taken into account when the slowdown
  factors were calculated?
  
The time of a job is computed as the difference between a time stamp
made by the test system and the report time.  The first time is affected
by ppc64's slow clock, the latter is correct.

A cron job sets the clock every 30 minutes, so the ppc time might be
*overestimated* < 30 minutes.

-- 
Torbjörn

Re: [Qemu-devel] Reporting Heisenbugs in qemu

[Artyom Tarasenko]

On Tue, May 7, 2013 at 11:43 PM, Torbjorn Granlund <tg@gmplib.org> wrote:
> Artyom Tarasenko <atar4qemu@gmail.com> writes:
>
>   Do I read it correct that qemu-system-ppc64 with the slowdown factor
>   of 33 is ~3 times faster than qemu-system-sparc64 with the slowdown
>   factor of 96 ?
>
> You read it correctly.
>
> But please see the caveat at the table end.
>
>   Do they both use Debian Wheezy guest? You have a remark that ppc64 has
>   problems with its clock. Was it taken into account when the slowdown
>   factors were calculated?
>
> The time of a job is computed as the difference between a time stamp
> made by the test system and the report time.  The first time is affected
> by ppc64's slow clock, the latter is correct.
>
> A cron job sets the clock every 30 minutes, so the ppc time might be
> *overestimated* < 30 minutes.

Meaning it's even faster than estimated? That's fast. I don't see in
the table, how long does the build take?

--
Regards,
Artyom Tarasenko

linux/sparc and solaris/sparc under qemu blog:
http://tyom.blogspot.com/search/label/qemu

Re: [Qemu-devel] Reporting Heisenbugs in qemu

[Torbjorn Granlund]

Artyom Tarasenko <atar4qemu@gmail.com> writes:

  Meaning it's even faster than estimated? That's fast. I don't see in
  the table, how long does the build take?
  
About one hour.  That's for:

  configure + make + make check TESTS=

(The host is a 4X 3.6 GHz Bulldozer Zambezi.)

-- 
Torbjörn

Re: [Qemu-devel] Reporting Heisenbugs in qemu

[Aurelien Jarno]

On Tue, May 07, 2013 at 11:29:20PM +0200, Artyom Tarasenko wrote:
> On Tue, May 7, 2013 at 1:38 PM, Torbjorn Granlund <tg@gmplib.org> wrote:
> > The 2nd table of http://gmplib.org/devel/testsystems.html shows all
> > emulated systems I am using, most of which are qemu-based.
> 
> Do I read it correct that qemu-system-ppc64 with the slowdown factor
> of 33 is ~3 times faster than qemu-system-sparc64 with the slowdown
> factor of 96 ?
> Do they both use Debian Wheezy guest? You have a remark that ppc64 has
> problems with its clock. Was it taken into account when the slowdown
> factors were calculated?
> 

Clock or not, it should be noted that qemu-system-sparc64 is undoubtedly
slower (at least 5 to 10 times) than qemu-system-{arm,ppc,mips,...} on
some type of load like perl scripts.

-- 
Aurelien Jarno                          GPG: 1024D/F1BCDB73
aurelien@aurel32.net                 http://www.aurel32.net

Re: [Qemu-devel] Reporting Heisenbugs in qemu

[Paolo Bonzini]

Il 08/05/2013 00:57, Aurelien Jarno ha scritto:
> On Tue, May 07, 2013 at 11:29:20PM +0200, Artyom Tarasenko wrote:
>> On Tue, May 7, 2013 at 1:38 PM, Torbjorn Granlund <tg@gmplib.org> wrote:
>>> The 2nd table of http://gmplib.org/devel/testsystems.html shows all
>>> emulated systems I am using, most of which are qemu-based.
>>
>> Do I read it correct that qemu-system-ppc64 with the slowdown factor
>> of 33 is ~3 times faster than qemu-system-sparc64 with the slowdown
>> factor of 96 ?
>> Do they both use Debian Wheezy guest? You have a remark that ppc64 has
>> problems with its clock. Was it taken into account when the slowdown
>> factors were calculated?
>>
> 
> Clock or not, it should be noted that qemu-system-sparc64 is undoubtedly
> slower (at least 5 to 10 times) than qemu-system-{arm,ppc,mips,...} on
> some type of load like perl scripts.

I guess that's the register windows.  There's only so much you can do to
optimize them, and heavily recursive workloads (like Perl, or the RTL
half of GCC) pay a hefty price.

Paolo

Re: [Qemu-devel] Reporting Heisenbugs in qemu

[Torbjorn Granlund]

Paolo Bonzini <pbonzini@redhat.com> writes:

  I guess that's the register windows.  There's only so much you can do to
  optimize them, and heavily recursive workloads (like Perl, or the RTL
  half of GCC) pay a hefty price.
  
Two qemu targets stand out for slowness, sparc (32 and 64) and mips (64,
don't know about 32).

x86 (32 and 64), arm, and ppc run with a slowdown of < 30 for my bogus
benchmark of GMP configure+make.

With FreeBSD x86_64 I see a slowdown of just 13.  (My reference system
runs FreeBSD, so running FreeBSD under qemu is only far.)

My claimed slowdown factors are affected by kernel, libraries, and
unfortunately very much by gcc speed, which vary with target.

If the sparc emulation speed is due to register windows, then why does
mips seem just as slow?

If register windows shortage is a problem, it should be easy to pretend
to have lots of them, right?

-- 
Torbjörn

Re: [Qemu-devel] Reporting Heisenbugs in qemu

[Paolo Bonzini]

> Paolo Bonzini <pbonzini@redhat.com> writes:
> 
>   I guess that's the register windows.  There's only so much you can do to
>   optimize them, and heavily recursive workloads (like Perl, or the RTL
>   half of GCC) pay a hefty price.
>   
> Two qemu targets stand out for slowness, sparc (32 and 64) and mips (64,
> don't know about 32).
> 
> x86 (32 and 64), arm, and ppc run with a slowdown of < 30 for my bogus
> benchmark of GMP configure+make.
> 
> With FreeBSD x86_64 I see a slowdown of just 13.  (My reference system
> runs FreeBSD, so running FreeBSD under qemu is only far.)
> 
> My claimed slowdown factors are affected by kernel, libraries, and
> unfortunately very much by gcc speed, which vary with target.
> 
> If the sparc emulation speed is due to register windows, then why does
> mips seem just as slow?

No idea. :)  Of all the architectures you listed, MIPS is really the one
that I have no inkling of...

> If register windows shortage is a problem, it should be easy to pretend
> to have lots of them, right?

That can help to avoid trapping to the kernel.  But you still have to
spill the whole window to the stack on every call to a non-leaf function,
which can be expensive even if you do it in the translated code.

Paolo

Re: [Qemu-devel] Reporting Heisenbugs in qemu

[Blue Swirl]

On Wed, May 8, 2013 at 10:18 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>> Paolo Bonzini <pbonzini@redhat.com> writes:
>>
>>   I guess that's the register windows.  There's only so much you can do to
>>   optimize them, and heavily recursive workloads (like Perl, or the RTL
>>   half of GCC) pay a hefty price.
>>
>> Two qemu targets stand out for slowness, sparc (32 and 64) and mips (64,
>> don't know about 32).
>>
>> x86 (32 and 64), arm, and ppc run with a slowdown of < 30 for my bogus
>> benchmark of GMP configure+make.
>>
>> With FreeBSD x86_64 I see a slowdown of just 13.  (My reference system
>> runs FreeBSD, so running FreeBSD under qemu is only far.)
>>
>> My claimed slowdown factors are affected by kernel, libraries, and
>> unfortunately very much by gcc speed, which vary with target.
>>
>> If the sparc emulation speed is due to register windows, then why does
>> mips seem just as slow?
>
> No idea. :)  Of all the architectures you listed, MIPS is really the one
> that I have no inkling of...
>
>> If register windows shortage is a problem, it should be easy to pretend
>> to have lots of them, right?
>
> That can help to avoid trapping to the kernel.  But you still have to
> spill the whole window to the stack on every call to a non-leaf function,
> which can be expensive even if you do it in the translated code.

The kernel wouldn't be able to spill or fill more than 8 windows
anyway. But maybe we could implement window cleaning in QEMU, if
nobody minds not seeing any CLEANWIN traps. I don't think real HW can
clean the windows.

Register window access in QEMU could be improved slightly, for example
it may be possible to handle wrapping of the registers with sparse
layout and page mapping tricks. Also the translator might be able to
keep track of current CWP and generate direct accesses instead of
going through regwptr.

>
> Paolo
>

Re: [Qemu-devel] Reporting Heisenbugs in qemu

[Aurelien Jarno]

On Wed, May 08, 2013 at 11:45:45AM +0200, Torbjorn Granlund wrote:
> Paolo Bonzini <pbonzini@redhat.com> writes:
> 
>   I guess that's the register windows.  There's only so much you can do to
>   optimize them, and heavily recursive workloads (like Perl, or the RTL
>   half of GCC) pay a hefty price.
>   
> Two qemu targets stand out for slowness, sparc (32 and 64) and mips (64,
> don't know about 32).
> 
> x86 (32 and 64), arm, and ppc run with a slowdown of < 30 for my bogus
> benchmark of GMP configure+make.
> 
> With FreeBSD x86_64 I see a slowdown of just 13.  (My reference system
> runs FreeBSD, so running FreeBSD under qemu is only far.)
> 
> My claimed slowdown factors are affected by kernel, libraries, and
> unfortunately very much by gcc speed, which vary with target.
> 
> If the sparc emulation speed is due to register windows, then why does
> mips seem just as slow?

I guess it's related to your benchmark, as it is not something really
observed (though not scientifically) on other loads. How much memory do
you need for it, couldn't it be that building GMP needs more than 256MB
of RAM at some point and that the system starts swapping?

-- 
Aurelien Jarno                          GPG: 1024D/F1BCDB73
aurelien@aurel32.net                 http://www.aurel32.net

Re: [Qemu-devel] Reporting Heisenbugs in qemu

[Rob Landley]

On 05/08/2013 04:45:45 AM, Torbjorn Granlund wrote:
> Paolo Bonzini <pbonzini@redhat.com> writes:
> 
>   I guess that's the register windows.  There's only so much you can  
> do to
>   optimize them, and heavily recursive workloads (like Perl, or the  
> RTL
>   half of GCC) pay a hefty price.
> 
> Two qemu targets stand out for slowness, sparc (32 and 64) and mips  
> (64,
> don't know about 32).
> 
> x86 (32 and 64), arm, and ppc run with a slowdown of < 30 for my bogus
> benchmark of GMP configure+make.
> 
> With FreeBSD x86_64 I see a slowdown of just 13.  (My reference system
> runs FreeBSD, so running FreeBSD under qemu is only far.)
> 
> My claimed slowdown factors are affected by kernel, libraries, and
> unfortunately very much by gcc speed, which vary with target.
> 
> If the sparc emulation speed is due to register windows, then why does
> mips seem just as slow?
> 
> If register windows shortage is a problem, it should be easy to  
> pretend
> to have lots of them, right?

sh4 is pretty slow too. Unfortunately:

   http://landley.net/aboriginal/bin/system-image-sh4.tar.bz2

Only has 64 megs of memory in the emulated board. (Enough to build  
hello world, not enough to build most packages.) I have a vague todo  
item to add a command line thing to qemu to plug a physical memory  
address range into an aribtrary address and then tell linux  
discontigmem "add memory range HERE" on the command line. That way I  
wouldn't have to hack up each board emulation to get more memory...)

Rob