Re: [Qemu-devel] QCOW(2) image corruption under QEMU 0.9.0 reproducible

Re: [Qemu-devel] QCOW(2) image corruption under QEMU 0.9.0 reproducible

[herbie hancock]

> -----Ursprüngliche Nachricht-----
> Von: qemu-devel@nongnu.org
> Gesendet: 16.03.07 13:11:15
> An: qemu-devel@nongnu.org
> Betreff: Re: [Qemu-devel] QCOW(2) image corruption under QEMU 0.9.0 reproducible


> herbie hancock wrote:
> > Hello, i had also a reproducible disk crash:
> > info of the last good image, size is about 3,5GB
> > 
> > I never experienced such a bad problem with qemu before, maybe it is a problem with qcow2 format ?
> 
> After the problems with qcow2 images which I reported here a few weeks
> ago, I've only been using qcow images (under QEMU 0.9.0), without such
> surprises.  So it seems qemu has some bug related to qcow2 images,
> maybe manifesting itself only after they get larger than 4GB...
> 
> 
> Best regards
>                           J Esteves
> 
> 
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
> 

I am quite sure now that it is a problem with the qcow2 format. 
I can reprodue the bug only on a qcow2 image. I did a disk fill test with dd on a qcow2 Image and the same test on a qcow Image, 
both images had  the same content. (Image 16GB Debian 4.0 Etch, hda1 ext3 15 gb,  hda5 swap 700MB)

The qcow2 Image was totally corrupted and is not bootable any more, the qcow Image passed the tests without any errors
Host system was now winxp sp2  instead of win2k, so i dont think its a problem of the host os.

I think a big  warning for the users is necessary that they should work with qcow instead of qcow2 until the bug is fixed

Best regards
hr

Test Setup:
---------------
Made a copy of the last known good qcow2 image (size 3.3 GB) and converted it to qcow

..\QemuManager\qemu\qemu-img.exe convert debian4_0.dsk -O qcow debian4_0_qcow.dsk

Crashtest with debian4_0.dsk (qcow2 format)
-----------------------------------------------------------

Size of intact debian4_0.dsk 3.551.580.160 Bytes

file format: qcow2
virtual size: 16G (16777216000 bytes)
disk size: 3.3G
cluster_size: 4096
Snapshot list:
ID        TAG                 VM SIZE                DATE       VM CLOCK
1         test                    79M 2007-03-11 18:34:49   00:07:47.475


Start VM: I:\debian4.0\QemuManager\qemu\qemu.exe -L "I:\debian4.0\QemuManager\qemu" -m 256 -hda "I:\debian4.0\debian4\debian4_0.dsk" -cdrom "I:\debian4.0\debian4\debian-testing-i386-DVD-1.iso" -kernel-kqemu -net nic,vlan=0 -net user,vlan=0 -std-vga -localtime

fsck on mounted filesystem: fsck -nv /dev/hda1 show no errors except # of free inodes
df 19% of system used (Total: 15409468 used: 2641020)
fill the rest of the disk: dd if=/dev/zero of=/tmp/bigfile bs=16M
dd stops with disk full, ok
---> now fsck -nv /dev/hda1 shows endless output of errors    <-----    ERROR
rm bigfile
shutdown system did work

---> Restart VM: Fatal not a bootable harddisk ext3 is totally corrupt    <-------  ERROR

In this case there was no crash on the VM, as i had on WIN2K, header of the image file seems to be still intact

Info after crash:
image: debian4_0.dsk
file format: qcow2
virtual size: 16G (16777216000 bytes)
disk size: 16G         <---- 16GB (why so much on a 15GB Partition ?)
cluster_size: 4096
Snapshot list:
ID        TAG                 VM SIZE                DATE       VM CLOCK
1         test                    79M 2007-03-11 18:34:49   00:07:47.475

Size of crashed debian4_0.dsk 16.659.554.304 Bytes



Test with debian4_0_qcow.dsk:
------------------------------------------

Size of debian4_0_qcow.dsk: 3.105.832.960 Bytes

image: debian4_0_qcow.dsk
file format: qcow
virtual size: 16G (16777216000 bytes)
disk size: 2.9G
cluster_size: 4096

Start VM: I:\debian4.0\QemuManager\qemu\qemu.exe -L "I:\debian4.0\QemuManager\qemu" -m 256 -hda "I:\debian4.0\debian4\debian4_0_qcow.dsk" -cdrom "I:\debian4.0\debian4\debian-testing-i386-DVD-1.iso" -kernel-kqemu -net nic,vlan=0 -net user,vlan=0 -std-vga -localtime

fsck on mounted filesystem: fsck -nv /dev/hda1 show no errors except # of free inodes
df 19% of system used (Total: 15409468 used: 2641020)
fill the rest of the disk: dd if=/dev/zero of=/tmp/bigfile bs=16M
dd stops with disk full, ok
now fsck -nv /dev/hda1 no errors
rm bigfile
shutdown system did work

image: debian4_0_qcow.dsk
file format: qcow
virtual size: 16G (16777216000 bytes)
disk size: 15G
cluster_size: 4096

Size of debian4_0_qcow.dsk: 15.919.493.120 Bytes

Start VM: all o.k.
-----------------------------------------------------



_______________________________________________________________________
Viren-Scan für Ihren PC! Jetzt für jeden. Sofort, online und kostenlos.
Gleich testen! http://www.pc-sicherheit.web.de/freescan/?mc=022222

Re: [Qemu-devel] QCOW(2) image corruption under QEMU 0.9.0 reproducible

[Ben Taylor]

---- J M Cerqueira Esteves <jmce@artenumerica.com> wrote: 
> herbie hancock wrote:
> > Hello, i had also a reproducible disk crash:
> > info of the last good image, size is about 3,5GB
> > 
> > I never experienced such a bad problem with qemu before, maybe it is a problem with qcow2 format ?
> 
> After the problems with qcow2 images which I reported here a few weeks
> ago, I've only been using qcow images (under QEMU 0.9.0), without such
> surprises.  So it seems qemu has some bug related to qcow2 images,
> maybe manifesting itself only after they get larger than 4GB...

I suspect I saw problems with qcow2 images as well.  I was able to suspend
a Solaris Nevada B58 install and use savevm about 30% into the install and
restart it later.  As the image completd, the file system went all to hell
with corruption that was impossible to fix.  At the time, I attributed it to
the Solaris install (thinking it might have something to do with the
cmpxchg8b bug that was later fixed), but I suspect with the multiple
reports I've seen, I'm now thinking I saw the same thing.

I'm testing conversion of a qcow image to a qcow2 image.  We'll see how
that goes

Ben

Re: [Qemu-devel] QCOW(2) image corruption under QEMU 0.9.0 reproducible

[Julian Seward]

I ran QEMU on Valgrind for several hours last night, including a couple
of boot-shutdown cycles of RedHat8, and lots of file copying/deletion 
in the guest to get the qcow2 image up to 8GB and generally cause a lot
of disk IO.  I got no memory errors whatsoever from Valgrind and got no
filesystem corruption, so I guess I didn't trigger the bug.

Really the first thing to do is establish a reliable way to reproduce it.

J


On Friday 16 March 2007 17:00, Ben Taylor wrote:
> ---- J M Cerqueira Esteves <jmce@artenumerica.com> wrote:
> > herbie hancock wrote:
> > > Hello, i had also a reproducible disk crash:
> > > info of the last good image, size is about 3,5GB
> > >
> > > I never experienced such a bad problem with qemu before, maybe it is a
> > > problem with qcow2 format ?
> >
> > After the problems with qcow2 images which I reported here a few weeks
> > ago, I've only been using qcow images (under QEMU 0.9.0), without such
> > surprises.  So it seems qemu has some bug related to qcow2 images,
> > maybe manifesting itself only after they get larger than 4GB...
>
> I suspect I saw problems with qcow2 images as well.  I was able to suspend
> a Solaris Nevada B58 install and use savevm about 30% into the install and
> restart it later.  As the image completd, the file system went all to hell
> with corruption that was impossible to fix.  At the time, I attributed it
> to the Solaris install (thinking it might have something to do with the
> cmpxchg8b bug that was later fixed), but I suspect with the multiple
> reports I've seen, I'm now thinking I saw the same thing.
>
> I'm testing conversion of a qcow image to a qcow2 image.  We'll see how
> that goes
>
> Ben
>
>
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel

Re: [Qemu-devel] QCOW(2) image corruption under QEMU 0.9.0 reproducible

[herbie hancock]

Hello, i had also a reproducible disk crash:
info of the last good image, size is about 3,5GB

image: debian4_0.dsk file format: qcow2 virtual size: 16G (16777216000 bytes) disk size: 3.3G cluster_size: 4096

as soon as the image increase to a size of about 7,9 GB the emulator locks up, and after a restart the the image is not readable any more.
the start of the image is filled with zeros, the signature of the file  at the start (QFI ....)  is overwritten.

I tried it two times, started with a intact image with the size above and in both times the image was corrupted.

Host: WIN2K
Guest: Debian 4.0 Etch.
qemu: 0.90 (build date 2007-02-19, the version that comes with http://www.davereyn.co.uk/qem/setupqemuk40.exe)

I tested the image above with virtualbox (installed backup of the qemu disk with acronis trueimage and bart pe boot cd) , 
started with the above image, and the problems are gone, image is now filled with more than 9GB, no problem so far.

I never experienced such a bad problem with qemu before, maybe it is a problem with qcow2 format ?

Bye
HR
_____________________________________________________________________
Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
http://smartsurfer.web.de/?mc=100071&distributionid=000000000066

Re: [Qemu-devel] QCOW(2) image corruption under QEMU 0.9.0 reproducible

[Julian Seward]

Something similar happened to me.  At first I thought it was a hardware
(host) problem and so do not have good details - this is from memory.

- 0.9.0, binary build from qemu.org
- i386 openSUSE 10.2 host
- RedHat 8 guest
- .qcow2 image, max size 8GB
- using the Accelerator but not -kernel-kqemu

- first sign of trouble was the ext3 driver in RedHat8 complaining of
  something about disk geometry (something % 4 was not zero when it
  should have been)

- this is when the disk image was about 3G in size

- shortly thereafter the disk image was so corrupted that e2fsck
  could not fix it

- IIRC, ls -l on the corrupted image showed some implausibly huge size
  (> 100GB ?) leading me to believe the file had some huge block of
  zeroes in the middle

J


On Wednesday 14 March 2007 23:20, herbie hancock wrote:
> Hello, i had also a reproducible disk crash:
> info of the last good image, size is about 3,5GB
>
> image: debian4_0.dsk file format: qcow2 virtual size: 16G (16777216000
> bytes) disk size: 3.3G cluster_size: 4096
>
> as soon as the image increase to a size of about 7,9 GB the emulator locks
> up, and after a restart the the image is not readable any more. the start
> of the image is filled with zeros, the signature of the file  at the start
> (QFI ....)  is overwritten.
>
> I tried it two times, started with a intact image with the size above and
> in both times the image was corrupted.
>
> Host: WIN2K
> Guest: Debian 4.0 Etch.
> qemu: 0.90 (build date 2007-02-19, the version that comes with
> http://www.davereyn.co.uk/qem/setupqemuk40.exe)
>
> I tested the image above with virtualbox (installed backup of the qemu disk
> with acronis trueimage and bart pe boot cd) , started with the above image,
> and the problems are gone, image is now filled with more than 9GB, no
> problem so far.
>
> I never experienced such a bad problem with qemu before, maybe it is a
> problem with qcow2 format ?
>
> Bye
> HR
> _____________________________________________________________________
> Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
> http://smartsurfer.web.de/?mc=100071&distributionid=000000000066
>
>
>
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel

Re: [Qemu-devel] QCOW(2) image corruption under QEMU 0.9.0 reproducible

[Thiemo Seufer]

Julian Seward wrote:
> 
> Something similar happened to me.  At first I thought it was a hardware
> (host) problem and so do not have good details - this is from memory.

I had trouble (disk access errors when installing Debian/mipsel) with a
5 GB qcow2 image with only some hundred MB of payload. I switched to qcow
for the time being.


Thiemo

Re: [Qemu-devel] QCOW(2) image corruption under QEMU 0.9.0 reproducible

[J M Cerqueira Esteves]

herbie hancock wrote:
> Hello, i had also a reproducible disk crash:
> info of the last good image, size is about 3,5GB
> 
> I never experienced such a bad problem with qemu before, maybe it is a problem with qcow2 format ?

After the problems with qcow2 images which I reported here a few weeks
ago, I've only been using qcow images (under QEMU 0.9.0), without such
surprises.  So it seems qemu has some bug related to qcow2 images,
maybe manifesting itself only after they get larger than 4GB...


Best regards
                          J Esteves