[Qemu-devel] Crashes of qemu-system-mips64 and qemu-system-mips64el

[Qemu-devel] Crashes of qemu-system-mips64 and qemu-system-mips64el

[Torbjörn Granlund]

Qemu versions 1.7.0, 1.7.1, 2.0.0, 2.1.0 do not run Debian mips64 BE or
LE stably.  While install in 32-bit mode typically works, running the
64-bit kernel causes qemu to segfault or abort after a while.

How to reproduce:

Grab installation kernel and initrd:

ftp://ftp.debian.org/debian/dists/wheezy/main/installer-mips/current/images/malta/netboot/vmlinux-3.2.0-4-4kc-malta
ftp://ftp.debian.org/debian/dists/wheezy/main/installer-mips/current/images/malta/netboot/initrd.gz

Start qemu-system-mips64 and perform an installation.  Choose defaults,
i.e., "Standard system utilities" and "SSH server".

  qemu-system-mips64 -M malta -m 256 -nographic \
    -drive file=disk.img,if=virtio,index=0 \
    -net nic,macaddr=52:54:00:13:06:64 -net user,hostfwd=tcp::20008-:22 \
    -kernel vmlinux-3.2.0-4-4kc-malta \
    -initrd initrd.gz \
    -append "console=ttyS0"

Copy out /boot somehow.  I usually do "Execute a shell" just before the
installation is about to finish, and there do

  mount /dev/vda2 /target
  mount /dev/vda1 /target/boot
  mount -t proc proc /target/proc
  mount --rbind /sys /target/sys
  mount --rbind /dev /target/dev
  chroot /target bash
  /etc/init.d/ssh start

(assuming "put all files in one partition" was chosen; mount commands
might need adjustment for other partitioning schemes).

Then from the host system I do

  scp -pr -P 20008 localhost:/boot .

and then quit the shell and finish up the installation.  (Alternatively
use qemu-nbd or guestfish.)

Then boot the installed system:

  qemu-system-mips64 -M malta -cpu 5Kc -m 256 \
    -drive file=disk.img,if=virtio,index=0 \
    -net nic,macaddr=52:54:00:13:06:64 -net user,hostfwd=tcp::20008-:22 \
    -kernel boot/vmlinux-3.2.0-4-5kc-malta \
    -initrd boot/initrd.img-3.2.0-4-5kc-malta \
    -append "root=/dev/vda1 console=ttyS0" \
    -nographic -serial null -monitor null

Then, log in to the system ("ssh -p 20008 root@localhost) and try the
system.  It will crash within hours, but usually within minutes.

Safe crash strategy:

  package_list="gcc g++ gdb make emacs23-nox postfix sharutils zsh rsync ntp"
  apt-get -y install $package_list gcc-multilib g++-multilib

Not crashed yet?  Let's provoke it somewhat more:

  wget https://ftp.gnu.org/gnu/gmp/gmp-6.0.0a.tar.bz2
  tar xf gmp-6.0.0a.tar.bz2
  cd gmp-6.0.0
  (configure && make && while true; do make check; done) >&/dev/null

Note that I have reproduced this problem on several host machines.  Host
system stability is not the root cause.  The host system has been either
GNU/Linux or FreeBSD.  I have not found a stable setup since before qemu
1.7.0.  I know it was stable around 1.5.x or 1.6.x.  (Even older qemus
executed a few instructions incorrectly, making them inadequate for my
use.)

Other notes:

1. Replacing virtio with ide for the disk spec does not improve things.

2. Both qemu-system-mips64 and qemu-system-mips64el exhibit the same
   instability patterns.

3. At least qemu-system-mips64el 2.1.0 sometimes hangs also during
   initial install, i.e., while running in 32-bit mode.  (I only
   observed this with virtio disk.)

Torbjörn
Please encrypt, key id 0xC8601622

Re: [Qemu-devel] Crashes of qemu-system-mips64 and qemu-system-mips64el

[Torbjörn Granlund]

I forgot to mention one of the popular crashes:

Assertion failed: (len <= 64), function tcg_gen_deposit_i64, file /var/tmp/pkg/usr/ports/emulators/qemu-devel/work/qemu-2.0.0/tcg/tcg-op.h, line 2206.

(This corresponds to qemu 2.1.0)

Note that segfaults also happened in the recent past.  Not sure if they
still occurs, though.


Torbjörn
Please encrypt, key id 0xC8601622

Re: [Qemu-devel] Crashes of qemu-system-mips64 and qemu-system-mips64el

[Aurelien Jarno]

On Sat, Aug 02, 2014 at 10:49:22PM +0200, Torbjörn Granlund wrote:
> Qemu versions 1.7.0, 1.7.1, 2.0.0, 2.1.0 do not run Debian mips64 BE or
> LE stably.  While install in 32-bit mode typically works, running the
> 64-bit kernel causes qemu to segfault or abort after a while.
> 
> How to reproduce:
> 
> Grab installation kernel and initrd:
> 
> ftp://ftp.debian.org/debian/dists/wheezy/main/installer-mips/current/images/malta/netboot/vmlinux-3.2.0-4-4kc-malta
> ftp://ftp.debian.org/debian/dists/wheezy/main/installer-mips/current/images/malta/netboot/initrd.gz
> 
> Start qemu-system-mips64 and perform an installation.  Choose defaults,
> i.e., "Standard system utilities" and "SSH server".
> 
>   qemu-system-mips64 -M malta -m 256 -nographic \
>     -drive file=disk.img,if=virtio,index=0 \
>     -net nic,macaddr=52:54:00:13:06:64 -net user,hostfwd=tcp::20008-:22 \
>     -kernel vmlinux-3.2.0-4-4kc-malta \
>     -initrd initrd.gz \
>     -append "console=ttyS0"
> 
> Copy out /boot somehow.  I usually do "Execute a shell" just before the
> installation is about to finish, and there do
> 
>   mount /dev/vda2 /target
>   mount /dev/vda1 /target/boot
>   mount -t proc proc /target/proc
>   mount --rbind /sys /target/sys
>   mount --rbind /dev /target/dev
>   chroot /target bash
>   /etc/init.d/ssh start
> 
> (assuming "put all files in one partition" was chosen; mount commands
> might need adjustment for other partitioning schemes).
> 
> Then from the host system I do
> 
>   scp -pr -P 20008 localhost:/boot .
> 
> and then quit the shell and finish up the installation.  (Alternatively
> use qemu-nbd or guestfish.)
> 
> Then boot the installed system:
> 
>   qemu-system-mips64 -M malta -cpu 5Kc -m 256 \
>     -drive file=disk.img,if=virtio,index=0 \
>     -net nic,macaddr=52:54:00:13:06:64 -net user,hostfwd=tcp::20008-:22 \
>     -kernel boot/vmlinux-3.2.0-4-5kc-malta \
>     -initrd boot/initrd.img-3.2.0-4-5kc-malta \
>     -append "root=/dev/vda1 console=ttyS0" \
>     -nographic -serial null -monitor null
> 
> Then, log in to the system ("ssh -p 20008 root@localhost) and try the
> system.  It will crash within hours, but usually within minutes.

I have installation running in qemu-system-mips with weeks of uptime
without any problem. I have however tried the above with QEMU 2.1, and
I have been unable to reproduce the issue.

> Safe crash strategy:
> 
>   package_list="gcc g++ gdb make emacs23-nox postfix sharutils zsh rsync ntp"
>   apt-get -y install $package_list gcc-multilib g++-multilib
> 
> Not crashed yet?  Let's provoke it somewhat more:

It didn't crash for me.

>   wget https://ftp.gnu.org/gnu/gmp/gmp-6.0.0a.tar.bz2
>   tar xf gmp-6.0.0a.tar.bz2
>   cd gmp-6.0.0
>   (configure && make && while true; do make check; done) >&/dev/null
> 
> Note that I have reproduced this problem on several host machines.  Host
> system stability is not the root cause.  The host system has been either
> GNU/Linux or FreeBSD.  I have not found a stable setup since before qemu
> 1.7.0.  I know it was stable around 1.5.x or 1.6.x.  (Even older qemus
> executed a few instructions incorrectly, making them inadequate for my
> use.)
> 

It's now running for more than 48 hours, and hasn't crashed yet.

Could you give us more details about your host, especially if it is a
32-bit or a 64-bit one? Also a cat /proc/cpuinfo would be useful as some
instructions are enabled or not depending on the host support.

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                 http://www.aurel32.net

Re: [Qemu-devel] Crashes of qemu-system-mips64 and qemu-system-mips64el

[Aurelien Jarno]

On Sun, Aug 03, 2014 at 02:11:30AM +0200, Torbjörn Granlund wrote:
> I forgot to mention one of the popular crashes:
> 
> Assertion failed: (len <= 64), function tcg_gen_deposit_i64, file /var/tmp/pkg/usr/ports/emulators/qemu-devel/work/qemu-2.0.0/tcg/tcg-op.h, line 2206.
> 
> (This corresponds to qemu 2.1.0)

Hmm it looks quite strange, looking quickly at the code, the only place
were we have a variable and unsafe length passed to tcg_gen_deposit_i64
is when using MIPS64R2 instruction, which is not your case according to
the previous mail as you said you pass -cpu 5Kc to qemu. I remember
there was a patch to fix these instrucions on the mailing list

The best would be to get a backtrace using gdb so that we can now what
calls tcg_gen_deposit_i64 with this too big len.

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                 http://www.aurel32.net

Re: [Qemu-devel] Crashes of qemu-system-mips64 and qemu-system-mips64el

[Torbjörn Granlund]

Aurelien Jarno <aurelien@aurel32.net> writes:

  I have installation running in qemu-system-mips with weeks of uptime
  without any problem. I have however tried the above with QEMU 2.1, and
  I have been unable to reproduce the issue.
  
I have never had these problems with qemu-system-mips. They only happen
with a 64-bit kernel on qemu-system-mips64 or qemu-system-mips64el.

  It's now running for more than 48 hours, and hasn't crashed yet.
  
Which version of qemu did you use for this experiment?  Please confirm
that you used a qemu in the range I claim crashes quickly.

This crashes for me with qemu 1.7.x through 2.1.0 under GNU/Linux and
FreeBSD.  Since my reports about this issue have not until now sparked
visible activity, I haven't tested any newer qemu release.  (I have
tried every qemu release in that range but not all (qemu release) x (OS
flavour) combinations.)

(I have a mips64 and a mips64el system running under qemu 1.6.2; the
current uptime is 52 days.  These systems are used for large daily
tests, thus are quite stable.)

  Could you give us more details about your host, especially if it is a
  32-bit or a 64-bit one? Also a cat /proc/cpuinfo would be useful as some
  instructions are enabled or not depending on the host support.

All my hosts are 64-bit x86.

I don't have the broken stuff set up any longer (2.5 month later) but
since my many attempts over the years trying to get a qemu post 1.6.x to
work I on many different pieces of hardware, the exact x86 hardware is
almost certainly irrelevant.  My hardware platforms are quite diverse.

-- 
Torbjörn
Please encrypt, key id 0xC8601622

Re: [Qemu-devel] Crashes of qemu-system-mips64 and qemu-system-mips64el

[Aurelien Jarno]

On Fri, Oct 17, 2014 at 03:53:35PM +0200, Torbjörn Granlund wrote:
> Aurelien Jarno <aurelien@aurel32.net> writes:
> 
>   I have installation running in qemu-system-mips with weeks of uptime
>   without any problem. I have however tried the above with QEMU 2.1, and
>   I have been unable to reproduce the issue.
>   
> I have never had these problems with qemu-system-mips. They only happen
> with a 64-bit kernel on qemu-system-mips64 or qemu-system-mips64el.

Sorry I meant qemu-system-mips64.

>   It's now running for more than 48 hours, and hasn't crashed yet.
>   
> Which version of qemu did you use for this experiment?  Please confirm
> that you used a qemu in the range I claim crashes quickly.

I am using 2.1.2 under GNU/Linux.

> This crashes for me with qemu 1.7.x through 2.1.0 under GNU/Linux and
> FreeBSD.  Since my reports about this issue have not until now sparked
> visible activity, I haven't tested any newer qemu release.  (I have
> tried every qemu release in that range but not all (qemu release) x (OS
> flavour) combinations.)
> 
> (I have a mips64 and a mips64el system running under qemu 1.6.2; the
> current uptime is 52 days.  These systems are used for large daily
> tests, thus are quite stable.)
> 
>   Could you give us more details about your host, especially if it is a
>   32-bit or a 64-bit one? Also a cat /proc/cpuinfo would be useful as some
>   instructions are enabled or not depending on the host support.
> 
> All my hosts are 64-bit x86.
> 
> I don't have the broken stuff set up any longer (2.5 month later) but
> since my many attempts over the years trying to get a qemu post 1.6.x to
> work I on many different pieces of hardware, the exact x86 hardware is
> almost certainly irrelevant.  My hardware platforms are quite diverse.

I don't think it's irrelevant, that's why I asked. If you don't provide
this information, we won't be able to check which code paths are
involved

Given you are the only one to reproduce the issue, please provide a
backtrace of a crash so that we can proceed further.

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                 http://www.aurel32.net

Re: [Qemu-devel] Crashes of qemu-system-mips64 and qemu-system-mips64el

[Torbjörn Granlund]

Aurelien Jarno <aurelien@aurel32.net> writes:
  
  I am using 2.1.2 under GNU/Linux.
  
Ah, so you're not trying to reproduce the problem!

Are you passing the -cpu 5Kc argument?

  I don't think it's irrelevant, that's why I asked. If you don't provide
  this information, we won't be able to check which code paths are
  involved
  
  Given you are the only one to reproduce the issue, please provide a
  backtrace of a crash so that we can proceed further.

Really?  Has anybody tried to reproduce the issue?

My bug report contains all information for trivially reproducing the
issue.  I kept the setup around for a long time, but now I have cleaned
it up.  I am very busy in this period and cannot afford to set it up
again, also considering that the effort on the developer part seems very
very limited wrt reported problems.  (I am not complaining, I have no
opinion on how volunteer hackers use their time!)

-- 
Torbjörn
Please encrypt, key id 0xC8601622

Re: [Qemu-devel] Crashes of qemu-system-mips64 and qemu-system-mips64el

[Aurelien Jarno]

On Fri, Oct 17, 2014 at 08:57:38PM +0200, Torbjörn Granlund wrote:
> Aurelien Jarno <aurelien@aurel32.net> writes:
>   
>   I am using 2.1.2 under GNU/Linux.
>   
> Ah, so you're not trying to reproduce the problem!

I do. Well you talked about 2.1.0, the latest stable one is 2.1.2. Now
if you prefer, we can conclude the problem is solved in 2.1.2.

> Are you passing the -cpu 5Kc argument?

I used:

qemu-system-mips64 -M malta -cpu 5Kc -m 256 \
    -drive file=disk.img,if=virtio,index=0 \
    -net nic,macaddr=52:54:00:13:06:64 -net user,hostfwd=tcp::20008-:22 \
    -kernel boot/vmlinux-3.2.0-4-5kc-malta \
    -initrd boot/initrd.img-3.2.0-4-5kc-malta \
    -append "root=/dev/vda1 console=ttyS0" \
    -nographic -serial null -monitor null

And it's still running the testsuite in a loop.

>   I don't think it's irrelevant, that's why I asked. If you don't provide
>   this information, we won't be able to check which code paths are
>   involved
>   
>   Given you are the only one to reproduce the issue, please provide a
>   backtrace of a crash so that we can proceed further.
> 
> Really?  Has anybody tried to reproduce the issue?

At least me, and it doesn't crash here.

> My bug report contains all information for trivially reproducing the
> issue.  I kept the setup around for a long time, but now I have cleaned

Yes, but it doesn't mean it's reproducible.

> it up.  I am very busy in this period and cannot afford to set it up
> again, also considering that the effort on the developer part seems very
> very limited wrt reported problems.  (I am not complaining, I have no
> opinion on how volunteer hackers use their time!)

Ok fine, let's consider the issue fixed. 

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                 http://www.aurel32.net

Re: [Qemu-devel] Crashes of qemu-system-mips64 and qemu-system-mips64el

[Torbjörn Granlund]

Aurelien Jarno <aurelien@aurel32.net> writes:

  On Fri, Oct 17, 2014 at 08:57:38PM +0200, Torbjörn Granlund wrote:
  > Aurelien Jarno <aurelien@aurel32.net> writes:
  >   
  >   I am using 2.1.2 under GNU/Linux.
  >   
  > Ah, so you're not trying to reproduce the problem!
  
  I do. Well you talked about 2.1.0, the latest stable one is 2.1.2. Now
  if you prefer, we can conclude the problem is solved in 2.1.2.
  
  > Are you passing the -cpu 5Kc argument?
  
  I used:
  
  qemu-system-mips64 -M malta -cpu 5Kc -m 256 \
      -drive file=disk.img,if=virtio,index=0 \
      -net nic,macaddr=52:54:00:13:06:64 -net user,hostfwd=tcp::20008-:22 \
      -kernel boot/vmlinux-3.2.0-4-5kc-malta \
      -initrd boot/initrd.img-3.2.0-4-5kc-malta \
      -append "root=/dev/vda1 console=ttyS0" \
      -nographic -serial null -monitor null
  
  And it's still running the testsuite in a loop.
  
  >   I don't think it's irrelevant, that's why I asked. If you don't provide
  >   this information, we won't be able to check which code paths are
  >   involved
  >   
  >   Given you are the only one to reproduce the issue, please provide a
  >   backtrace of a crash so that we can proceed further.
  > 
  > Really?  Has anybody tried to reproduce the issue?
  
  At least me, and it doesn't crash here.
  
  > My bug report contains all information for trivially reproducing the
  > issue.  I kept the setup around for a long time, but now I have cleaned
  
  Yes, but it doesn't mean it's reproducible.
  
  > it up.  I am very busy in this period and cannot afford to set it up
  > again, also considering that the effort on the developer part seems very
  > very limited wrt reported problems.  (I am not complaining, I have no
  > opinion on how volunteer hackers use their time!)
  
  Ok fine, let's consider the issue fixed. 

You won't like me saying this, but this is not how professional software
development is done.  It is also discouraging to see a detailed bug
report being treated in such a dismissive manner.

Here is an alternative approach:

1. Is the bug report complete?  If not, ask for more information (or
   perhaps chose to ignore it if you conclude it is bogus).

2. Try to reproduce the problem using the information in the bug report.
   This means that you set up an environment as close as possible in all
   relevant aspects.  Clearly, do not use a different version of the
   software being investigated.

3. Isolate the bug and fix it.  As the reporter for feedback.

4. Make some effort in finding similar bugs in the the software package.

It is not safe to conclude that when a different version of the software
seems to work, then the bug is gone.  Why?  There might be some mistake
in reproducing the bug.  There might be a relevant environment
discrepancy.  You might even be running the wrong binary by mistake, or
use the wrong data file by mistake.  (I have been in the game a long
time now, and I have made all these mistakes at some point.)

-- 
Torbjörn
Please encrypt, key id 0xC8601622

Re: [Qemu-devel] Crashes of qemu-system-mips64 and qemu-system-mips64el

[Aurelien Jarno]

On Wed, Oct 22, 2014 at 10:31:45PM +0200, Torbjörn Granlund wrote:
> Aurelien Jarno <aurelien@aurel32.net> writes:
> 
>   On Fri, Oct 17, 2014 at 08:57:38PM +0200, Torbjörn Granlund wrote:
>   > Aurelien Jarno <aurelien@aurel32.net> writes:
>   >   
>   >   I am using 2.1.2 under GNU/Linux.
>   >   
>   > Ah, so you're not trying to reproduce the problem!
>   
>   I do. Well you talked about 2.1.0, the latest stable one is 2.1.2. Now
>   if you prefer, we can conclude the problem is solved in 2.1.2.
>   
>   > Are you passing the -cpu 5Kc argument?
>   
>   I used:
>   
>   qemu-system-mips64 -M malta -cpu 5Kc -m 256 \
>       -drive file=disk.img,if=virtio,index=0 \
>       -net nic,macaddr=52:54:00:13:06:64 -net user,hostfwd=tcp::20008-:22 \
>       -kernel boot/vmlinux-3.2.0-4-5kc-malta \
>       -initrd boot/initrd.img-3.2.0-4-5kc-malta \
>       -append "root=/dev/vda1 console=ttyS0" \
>       -nographic -serial null -monitor null
>   
>   And it's still running the testsuite in a loop.
>   
>   >   I don't think it's irrelevant, that's why I asked. If you don't provide
>   >   this information, we won't be able to check which code paths are
>   >   involved
>   >   
>   >   Given you are the only one to reproduce the issue, please provide a
>   >   backtrace of a crash so that we can proceed further.
>   > 
>   > Really?  Has anybody tried to reproduce the issue?
>   
>   At least me, and it doesn't crash here.
>   
>   > My bug report contains all information for trivially reproducing the
>   > issue.  I kept the setup around for a long time, but now I have cleaned
>   
>   Yes, but it doesn't mean it's reproducible.
>   
>   > it up.  I am very busy in this period and cannot afford to set it up
>   > again, also considering that the effort on the developer part seems very
>   > very limited wrt reported problems.  (I am not complaining, I have no
>   > opinion on how volunteer hackers use their time!)
>   
>   Ok fine, let's consider the issue fixed. 
> 
> You won't like me saying this, but this is not how professional software
> development is done.  It is also discouraging to see a detailed bug
> report being treated in such a dismissive manner.
> 
> Here is an alternative approach:

Ok, let's do it.

> 1. Is the bug report complete?  If not, ask for more information (or
>    perhaps chose to ignore it if you conclude it is bogus).

You refused to provide the information I asked.

> 2. Try to reproduce the problem using the information in the bug report.
>    This means that you set up an environment as close as possible in all
>    relevant aspects.  Clearly, do not use a different version of the
>    software being investigated.

I later tried to reproduce it with version 2.1.0. I arrived to the same
conclusion that the bug is not reproducible.

> 3. Isolate the bug and fix it.  As the reporter for feedback.

The bug is not reproducible, so that's not something possible.

> 4. Make some effort in finding similar bugs in the the software package.
> 
> It is not safe to conclude that when a different version of the software
> seems to work, then the bug is gone.  Why?  There might be some mistake
> in reproducing the bug.  There might be a relevant environment
> discrepancy.  You might even be running the wrong binary by mistake, or
> use the wrong data file by mistake.  (I have been in the game a long
> time now, and I have made all these mistakes at some point.)

In that case, please provide:
- The QEMU binary you have built.
- The image to reproduce the issue.

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                 http://www.aurel32.net