[Qemu-devel] Virtual PC images still corrupt - but better than before

[Qemu-devel] Virtual PC images still corrupt - but better than before

[Jamie Lokier]

I recently looked at the release notes for kvm-84, and saw it merged
from qemu-svn in February, and one of the changes was:

   improve Virtual PC disk format support

So, having reported bugs in that before (to the extent a Windows 2003
image couldn't boot; it spent forever listing chkdsk errors), I
decided to try the new code.

Specifically I tried kvm-88.  It didn't work, but was much improved.
I'm presuming it was the contribution below:

Subject: [Qemu-devel] [PATCH 0/7] block-vpc: Improve support for VHD images
Kevin Wolf wrote:
> Jamie Lokier schrieb:
> > Kevin Wolf wrote:
> >> This patch series improves the support for Virtual PC images. Until
> >> now, support for this image format is read-only and broken for large
> >> images. With these patches applied, you can create and write to VHD
> >> Dynamic Disks and some bugs in the read support are fixed. They have
> >> been tested with qemu-img convert from and to raw images and a
> >> successful openSUSE installation on a fresh VHD image.
> > 
> > Ooh, I look forward to trying these patches. :-)
> > 
> > I have some VHD image of Windows which always resulted in lots of
> > scandisk errors when Windows booted from it (using -snapshot), and the
> > same when converted to a QCOW2 file by qemu-img.
> 
> I think there is only one integrity patch in the series which fixes
> images > 4 GB. Is this the case with your image?

Yes, the image is about 80GB.  83,886,080,000 bytes to be exact.

> [My implementation] is the simplest one I could think of and it's
> mostly to have at least something to convert images. I don't think I
> would run a VM on it except for testing.
> 
> > So I look forward to trying these patches on that image, which I don't
> > use know but kept for this very purpose, and seeing if it's fixed the
> > errors.
> 
> Thanks for testing and please report back. I hope it works fine now.

Let's see.  Previously I'd started by trying to run a VM directly from
the VPC image, but that was a bad idea.  Then I tried using "qemu-img
convert" to convert it to raw, but that was corrupt too.  Then I used
Microsoft's tool for conversion to raw, and the raw image was usable
with QEMU and KVM.

There were changes to VPC support merged from qemu-svn into kvm-84.
I used "qemu-img convert" from kvm-88 for testing, except that I
confirmed the size issue occurs with kvm-84 too:

  1. Strikingly, it produces the wrong raw image size.

     Firstly, the new code interprets the disk image with the wrong size:

     $ /usr/local/kvm-83/bin/kvm-img info original.vpc
     image: original.vpc
     file format: vpc
     virtual size: 78G (83886080000 bytes)
     disk size: 6.0G

     $ /usr/local/kvm-84/bin/kvm-img info original.vpc
     image: original.vpc
     file format: vpc
     virtual size: 78G (83884277760 bytes)
     disk size: 6.0G

     $ ls -l expanded_by_VirtualPC.vhd 
     -rw-r--r-- 1 jamie jamie 83886080512 2008-05-26 07:47 expanded_by_VirtualPC.vhd

     The correct image size is probably 83886080000 bytes, which means
     the older code calculated it correctly and the new code does not.

     Microsoft's conversion tool adds an extra 512 bytes on the end,
     which are all zeros and probably aren't required for qemu/kvm.

     Unsurprisingly given the above output, "qemu-img convert"
     produces raw image of the wrong size too - the size reported as
     "virtual size".

     I checked the partition size on this image, and found the
     partition used for real data to be smaller than the smaller size
     produced by the new VPC format code.  So I looked at the extra
     1802240 bytes, and found they were all zeros.

  2. The amount of corrupt sectors is very much improved.

     Whereas kvm-72 produced a huge number of corrupt sectors - more
     than I counted, kvm-88 produced just 3 corrupt sectors of 512
     bytes in the entire 80GB file.  In two of them, the correct file
     has data where the kvm-88 output has zeros; in the other corrupt
     sector, kvm-88 has non-zeros where the correct file has zeros.

     (I'm assuming the output from Microsoft's tool is correct).

     That's close to perfect, but obviously not perfect enough.

     I have the offsets of the corrupt sectors, so I'm hoping it won't
     be hard to reproduce by attempting to read just those sectors
     through the block-vpc.c code and deducing where it goes wrong.

-- Jamie

[Qemu-devel] Re: Virtual PC images still corrupt - but better than before

[Kevin Wolf]

Jamie Lokier schrieb:
> There were changes to VPC support merged from qemu-svn into kvm-84.
> I used "qemu-img convert" from kvm-88 for testing, except that I
> confirmed the size issue occurs with kvm-84 too:
> 
>   1. Strikingly, it produces the wrong raw image size.
> 
>      Firstly, the new code interprets the disk image with the wrong size:
> 
>      $ /usr/local/kvm-83/bin/kvm-img info original.vpc
>      image: original.vpc
>      file format: vpc
>      virtual size: 78G (83886080000 bytes)
>      disk size: 6.0G
> 
>      $ /usr/local/kvm-84/bin/kvm-img info original.vpc
>      image: original.vpc
>      file format: vpc
>      virtual size: 78G (83884277760 bytes)
>      disk size: 6.0G
> 
>      $ ls -l expanded_by_VirtualPC.vhd 
>      -rw-r--r-- 1 jamie jamie 83886080512 2008-05-26 07:47 expanded_by_VirtualPC.vhd
> 
>      The correct image size is probably 83886080000 bytes, which means
>      the older code calculated it correctly and the new code does not.

I think this depends on the definition of "correct". VPC unfortunately
seems to like contradicting values in its image - CHS geometry and image
size rarely match. The guest seemed to see the CHS geometry, so I
changed the code to take the geometry instead of the size header field
to calculate the image size.

This change resulted in md5sum /dev/hda giving the same result in qemu
and VPC, which I consider more correct than it was before. I haven't
tested with a wide range of guests, though.

>      Microsoft's conversion tool adds an extra 512 bytes on the end,
>      which are all zeros and probably aren't required for qemu/kvm.
> 
>      Unsurprisingly given the above output, "qemu-img convert"
>      produces raw image of the wrong size too - the size reported as
>      "virtual size".
> 
>      I checked the partition size on this image, and found the
>      partition used for real data to be smaller than the smaller size
>      produced by the new VPC format code.  So I looked at the extra
>      1802240 bytes, and found they were all zeros.

This is probably just the MS conversion tool taking the image size
header which is usually larger than the CHS geometry, but remains empty
as the guest cannot see it in VPC.

>   2. The amount of corrupt sectors is very much improved.
> 
>      Whereas kvm-72 produced a huge number of corrupt sectors - more
>      than I counted, kvm-88 produced just 3 corrupt sectors of 512
>      bytes in the entire 80GB file.  In two of them, the correct file
>      has data where the kvm-88 output has zeros; in the other corrupt
>      sector, kvm-88 has non-zeros where the correct file has zeros.
> 
>      (I'm assuming the output from Microsoft's tool is correct).
> 
>      That's close to perfect, but obviously not perfect enough.
> 
>      I have the offsets of the corrupt sectors, so I'm hoping it won't
>      be hard to reproduce by attempting to read just those sectors
>      through the block-vpc.c code and deducing where it goes wrong.

You could have a try with qemu-io if you know the offsets. Should be as
easy as "read -v $OFFSET 512". The VHD format actually is pretty easy,
too, so it shouldn't be too hard to calculate the sector's position in
the image file manually for comparison.

I suspect that it's something with a sector set to "(not) allocated" in
the block bitmap. The qemu driver currently ignores this for reading as
it should only be an optimization. So if the bitmap says the block is
empty and in fact it isn't, you'd have a corrupted image.

Kevin