[Qemu-devel] qemu-img: error while compressing, Input/output error

[Qemu-devel] qemu-img: error while compressing, Input/output error

[Ilkka Tengvall]

I run into an issue where qemu-img convert fails repeatedly on 
compressing a raw disk image. It's not all broken, since it many times 
also works. When it fails, it fails always to the same sector.

I run into problem using f18 qemu-img-1.2.2-6.fc18.x86_64, and repeated 
it with the latest qemu-img from qemu git as of 12th of April 2013:
last git log: 93b48c201eb6c0404d15550a0eaa3c0f7937e35e

I discussed the problem on IRC on qemu channel with stefanha, and he 
asked me to provide the info here. Thanks for you help stefanha!

Here is some background snipplets:

"
<ikke> anyone having idea why "qemu-img convert -c -O qcow2 -f raw 
file1 file1.qcow2" would fail to:
<ikke> qemu-img: error while compressing sector 1842944: Input/output error
<stefanha> ikke: My guess is that 
qcow2_alloc_compressed_cluster_offset() is failing
<stefanha> Compressed writes cannot overwrite existing sectors,
<stefanha> perhaps there's a bug in qemu-img convert that causes it to 
touch the same sector twice.
<stefanha> We wouldn't notice it when compression is disabled.
<stefanha> But when it's enabled you get a failure.
<pm215> stefanha: by the way, is it right that qemu-img.c:1372 
calculates cluster_sectors from cluster_size by rounding down rather 
than up? I know nothing about this code but that looks a little suspicious
<stefanha> And the contents of the input image affect the I/O pattern 
during conversion
<stefanha> which would explain why you always get the problem with 
certain images.
<stefanha> pm215: That should be okay.  cluster_size is a power of 2 and 
multiple of 512.
<stefanha> For qcow2 the default cluster_size is 64 KB.
<stefanha> ikke: A post to the mailing list with a link to the file and 
a description of how to trigger the failure would be fine.
<stefanha> I think it won't take long to fix.
"

The image for repeating this is available here (~230MB):

http://hard.ware.fi/~ikke/stuff/ubuntufs.raw.img.gzip

and the command to reproduce the error is here:

$ qemu-img convert -c -O qcow2 -f raw root.fs.copy root.fs.small.new
qemu-img: error while compressing sector 1842944: Input/output error



BR,

Ilkka Tengvall
ilkka.tengvall@cybercom.com

Re: [Qemu-devel] qemu-img: error while compressing, Input/output error

[Fam Zheng]

On Fri, 04/12 14:55, Ilkka Tengvall wrote:
> I run into an issue where qemu-img convert fails repeatedly on
> compressing a raw disk image. It's not all broken, since it many
> times also works. When it fails, it fails always to the same sector.
> 
> I run into problem using f18 qemu-img-1.2.2-6.fc18.x86_64, and
> repeated it with the latest qemu-img from qemu git as of 12th of
> April 2013:
> last git log: 93b48c201eb6c0404d15550a0eaa3c0f7937e35e
> 
> I discussed the problem on IRC on qemu channel with stefanha, and he
> asked me to provide the info here. Thanks for you help stefanha!
> 
> Here is some background snipplets:
> 
> "
> <ikke> anyone having idea why "qemu-img convert -c -O qcow2 -f raw
> file1 file1.qcow2" would fail to:
> <ikke> qemu-img: error while compressing sector 1842944: Input/output error
> <stefanha> ikke: My guess is that
> qcow2_alloc_compressed_cluster_offset() is failing
> <stefanha> Compressed writes cannot overwrite existing sectors,
> <stefanha> perhaps there's a bug in qemu-img convert that causes it
> to touch the same sector twice.
> <stefanha> We wouldn't notice it when compression is disabled.
> <stefanha> But when it's enabled you get a failure.
> <pm215> stefanha: by the way, is it right that qemu-img.c:1372
> calculates cluster_sectors from cluster_size by rounding down rather
> than up? I know nothing about this code but that looks a little
> suspicious
> <stefanha> And the contents of the input image affect the I/O
> pattern during conversion
> <stefanha> which would explain why you always get the problem with
> certain images.
> <stefanha> pm215: That should be okay.  cluster_size is a power of 2
> and multiple of 512.
> <stefanha> For qcow2 the default cluster_size is 64 KB.
> <stefanha> ikke: A post to the mailing list with a link to the file
> and a description of how to trigger the failure would be fine.
> <stefanha> I think it won't take long to fix.
> "
> 
> The image for repeating this is available here (~230MB):
> 
> http://hard.ware.fi/~ikke/stuff/ubuntufs.raw.img.gzip
> 
> and the command to reproduce the error is here:
> 
> $ qemu-img convert -c -O qcow2 -f raw root.fs.copy root.fs.small.new
> qemu-img: error while compressing sector 1842944: Input/output error
> 
> 
> 
> BR,
> 
> Ilkka Tengvall
> ilkka.tengvall@cybercom.com
> 

Thanks for the info. I tried you image, but it's rather commonly
reproducible, just use any non-zero file with 512 bytes as input will
result in the same error. Looking into the cause, it seems like a bug in
block code, the last sectors are refused to be write compressed unless
it makes a whole cluster in output format (see
block.c:bdrv_check_request).

One workaround now is to round up your source raw image to 64k (the qcow2
cluster size):

$ dd if=/dev/zero bs=1 count=$((64*1024 - `stat -c %s ubuntufs.raw.img` %(64*1024))) >> new.img

-- 
Fam

Re: [Qemu-devel] qemu-img: error while compressing, Input/output error

[Stefan Hajnoczi]

On Fri, Apr 12, 2013 at 02:55:45PM +0300, Ilkka Tengvall wrote:
> I run into an issue where qemu-img convert fails repeatedly on
> compressing a raw disk image. It's not all broken, since it many
> times also works. When it fails, it fails always to the same sector.
> 
> I run into problem using f18 qemu-img-1.2.2-6.fc18.x86_64, and
> repeated it with the latest qemu-img from qemu git as of 12th of
> April 2013:
> last git log: 93b48c201eb6c0404d15550a0eaa3c0f7937e35e
> 
> I discussed the problem on IRC on qemu channel with stefanha, and he
> asked me to provide the info here. Thanks for you help stefanha!
> 
> Here is some background snipplets:
> 
> "
> <ikke> anyone having idea why "qemu-img convert -c -O qcow2 -f raw
> file1 file1.qcow2" would fail to:
> <ikke> qemu-img: error while compressing sector 1842944: Input/output error
> <stefanha> ikke: My guess is that
> qcow2_alloc_compressed_cluster_offset() is failing
> <stefanha> Compressed writes cannot overwrite existing sectors,
> <stefanha> perhaps there's a bug in qemu-img convert that causes it
> to touch the same sector twice.
> <stefanha> We wouldn't notice it when compression is disabled.
> <stefanha> But when it's enabled you get a failure.
> <pm215> stefanha: by the way, is it right that qemu-img.c:1372
> calculates cluster_sectors from cluster_size by rounding down rather
> than up? I know nothing about this code but that looks a little
> suspicious
> <stefanha> And the contents of the input image affect the I/O
> pattern during conversion
> <stefanha> which would explain why you always get the problem with
> certain images.
> <stefanha> pm215: That should be okay.  cluster_size is a power of 2
> and multiple of 512.
> <stefanha> For qcow2 the default cluster_size is 64 KB.
> <stefanha> ikke: A post to the mailing list with a link to the file
> and a description of how to trigger the failure would be fine.
> <stefanha> I think it won't take long to fix.
> "
> 
> The image for repeating this is available here (~230MB):
> 
> http://hard.ware.fi/~ikke/stuff/ubuntufs.raw.img.gzip
> 
> and the command to reproduce the error is here:
> 
> $ qemu-img convert -c -O qcow2 -f raw root.fs.copy root.fs.small.new
> qemu-img: error while compressing sector 1842944: Input/output error

I discussed the bug with kwolf on Friday.  Although I sent a patch to
add an error message when the input image length is not a multiple of
cluster_size, Kevin pointed out that we can allow the final cluster to
be zero-padded beyond the end of the virtual disk.

I'll try to send a patch for that this week.

Stefan

Re: [Qemu-devel] qemu-img: error while compressing, Input/output error

[Ilkka Tengvall]

On 15.04.2013 09:52, Stefan Hajnoczi wrote:
> I discussed the bug with kwolf on Friday.  Although I sent a patch to
> add an error message when the input image length is not a multiple of
> cluster_size, Kevin pointed out that we can allow the final cluster to
> be zero-padded beyond the end of the virtual disk.
> Stefan

I second that. As an end user it is just extra complication to do that 
manually. Especially if the device really ends at the non 64K boundary, 
it's extra trouble to extend the device first. Not to talk about 
complicating the auto build scripts by needing to add the size 
calculations before all compressions.

In my case I don't mind if there is some extra bytes of zero at the end 
of a file size of hundreds of megs. I do understand the purity aspect 
though, not getting the exact same copy while doing the conversion back. 
Could there be a compatibility switch, that would handle the error 
automatically if one doesn't care about converting back to original image?

Anyways, adding the missing 64k modulo bytes works, I just tested it. 
Thanks both of you for the analysis and help!

Ilkka Tengvall


PS, evidence of the success :) :

starting filesystem compression
minimizing root partition on [/dev/loop2p1] - file check
/dev/loop2p1: 56193/102544 files (0.1% non-contiguous), 222914/409600 blocks
minimizing root partition on [/dev/loop2p1] - resizing
minimizing root partition done [/dev/loop2p1] - again file check
/dev/loop2p1: 56193/63104 files (0.4% non-contiguous), 220426/230381 blocks
the new size is [230381] of [4096] sized blocks, copying it out to [root.fs]
we need to add [12288] extra bytes to image to make the compression work
943652864+0 records in
943652864+0 records out
943652864 bytes (944 MB) copied, 2304,77 s, 409 kB/s
packing the filesystem
Completed. See 'ls -lah vmlinuz* initrd* root.fs
done building

Re: [Qemu-devel] qemu-img: error while compressing, Input/output error

[Stefan Hajnoczi]

On Mon, Apr 15, 2013 at 1:00 PM, Ilkka Tengvall
<ilkka.tengvall@cybercom.com> wrote:
> On 15.04.2013 09:52, Stefan Hajnoczi wrote:
>>
>> I discussed the bug with kwolf on Friday.  Although I sent a patch to
>> add an error message when the input image length is not a multiple of
>> cluster_size, Kevin pointed out that we can allow the final cluster to
>> be zero-padded beyond the end of the virtual disk.
>> Stefan
>
>
> I second that. As an end user it is just extra complication to do that
> manually. Especially if the device really ends at the non 64K boundary, it's
> extra trouble to extend the device first. Not to talk about complicating the
> auto build scripts by needing to add the size calculations before all
> compressions.
>
> In my case I don't mind if there is some extra bytes of zero at the end of a
> file size of hundreds of megs. I do understand the purity aspect though, not
> getting the exact same copy while doing the conversion back. Could there be
> a compatibility switch, that would handle the error automatically if one
> doesn't care about converting back to original image?

A compatibility option won't be necessary because the virtual disk
size will not be extended.  This means that the image can be converted
to raw without a change in disk size.

The trick is that although we will compress a padded cluster, the
virtual disk size will remain unchanged.  The zero bytes will not be
accessible, they will just automatically pad to the required
alignment.

Stefan