[Qemu-devel] More qcow2 bugs? - qemu-img convert/commit

[Qemu-devel] More qcow2 bugs? - qemu-img convert/commit

[Jamie Lokier]

Today I decided to take a gamble and run "qemu-img commit" on a couple
of qcow2 files that have backing files, because the qcow2s had become
as large as their backing file anyway.

The first one I did without thinking, confident that qcow2 bugs have
been squashed.  Foolish me, I put my faith in the tool and didn't keep
a backup of the VM disk images.  (They are quite large after all).

Then I got nervous, and for the second one I did this:

    qemu-img convert -O raw my_qcow2_branch flattened
    qemu-img commit my_qcow2_branch

    md5sum my_qcow2_backing_file flattened

The two md5sums _should_ be identical, shouldn't they?

In mine, the files have the same size but different md5sums.

The tool I used is qemu-img from kvm-88, which is the most recent
release on kvm's home page.

I ran script to find which sectors differed, and found a large number.
I stopped counting after the first million sectors, which is a lot,
considering the raw image is only 11GB.  Extrapolating, about 37% of
all the sectors did not match.

In all cases, the output of "convert" had all-zeros in those sectors
while the backing file after "commit" had non-zero data.

To which I think *phew*.  At least that gives me some confidence that
I didn't trash the file I ran "commit" on, and thank goodness I didn't
decide to use "convert" instead.  I had in the back of my mind that
"convert" might be safer because it would use more tested code paths
than "commit" - after all it could just use the same code as qemu and
qemu-nbd, whereas "commit" is more special.

37% is large enough to be a clue: Does it simply expand the qcow2 file
without reading the backing file at all?  A quick strace shows that
indeed, it does *open* the backing file and it does read the backing
file's header, but after that it doesn't read it at all.

Did I miss something obvious which means this is ok?

Should the output of "qemu-img convert" be identical to the backing
file after "qemu-img commit", assuming they are the same format?

Thanks,
-- Jamie

[Qemu-devel] Re: More qcow2 bugs? - qemu-img convert/commit

[Paolo Bonzini]

> 37% is large enough to be a clue: Does it simply expand the qcow2 file
> without reading the backing file at all?  A quick strace shows that
> indeed, it does *open* the backing file and it does read the backing
> file's header, but after that it doesn't read it at all.
>
> Did I miss something obvious which means this is ok?
>
> Should the output of "qemu-img convert" be identical to the backing
> file after "qemu-img commit", assuming they are the same format?

I would have thought so.

Paolo

Re: [Qemu-devel] More qcow2 bugs? - qemu-img convert/commit

[Kevin Wolf]

Am 10.09.2009 01:26, schrieb Jamie Lokier:
> 37% is large enough to be a clue: Does it simply expand the qcow2 file
> without reading the backing file at all?  A quick strace shows that
> indeed, it does *open* the backing file and it does read the backing
> file's header, but after that it doesn't read it at all.
> 
> Did I miss something obvious which means this is ok?

No, this definitely looks like a bug. I tend to suspect qemu-img rather
than qcow2, though. After all, your qcow2 image works when a VM reads it
and from a qcow2 perspective there should be no difference from reading
in qemu-img.

I'll have a look at it.

> Should the output of "qemu-img convert" be identical to the backing
> file after "qemu-img commit", assuming they are the same format?

No, same format isn't enough. Assuming they are both raw images (as in
your case), I would agree, though.

Kevin

Re: [Qemu-devel] More qcow2 bugs? - qemu-img convert/commit

[Kevin Wolf]

Am 10.09.2009 10:27, schrieb Kevin Wolf:
> Am 10.09.2009 01:26, schrieb Jamie Lokier:
>> 37% is large enough to be a clue: Does it simply expand the qcow2 file
>> without reading the backing file at all?  A quick strace shows that
>> indeed, it does *open* the backing file and it does read the backing
>> file's header, but after that it doesn't read it at all.
>>
>> Did I miss something obvious which means this is ok?
> 
> No, this definitely looks like a bug. I tend to suspect qemu-img rather
> than qcow2, though. After all, your qcow2 image works when a VM reads it
> and from a qcow2 perspective there should be no difference from reading
> in qemu-img.
> 
> I'll have a look at it.

Actually, first of all, which qemu version are you using?

There is a qemu-iotests test case for qemu-img convert with backing
files and git master passes it for me. However, there was a fix to
qemu-img in July (commit d032044fb97d525ae93fb4c96713a5c467044c17)
concerning conversion with backing files.

Kevin

Re: [Qemu-devel] More qcow2 bugs? - qemu-img convert/commit

[Jamie Lokier]

Kevin Wolf wrote:
> Actually, first of all, which qemu version are you using?

>>The tool I used is qemu-img from kvm-88, which is the most recent
>>release on kvm's home page.

Dated 12 July 2009.

-- Jamie

Re: [Qemu-devel] More qcow2 bugs? - qemu-img convert/commit

[Kevin Wolf]

Am 10.09.2009 11:44, schrieb Jamie Lokier:
> Kevin Wolf wrote:
>> Actually, first of all, which qemu version are you using?
> 
>>> The tool I used is qemu-img from kvm-88, which is the most recent
>>> release on kvm's home page.
> 
> Dated 12 July 2009.

Sorry, seems I missed the kvm-88 part. The date of the fix I mentioned
is 19 July 2009, so I assume your problem is already fixed with current
git master.

Kevin

Re: [Qemu-devel] More qcow2 bugs? - qemu-img convert/commit

[Jamie Lokier]

Kevin Wolf wrote:
> Am 10.09.2009 11:44, schrieb Jamie Lokier:
> > Kevin Wolf wrote:
> >> Actually, first of all, which qemu version are you using?
> > 
> >>> The tool I used is qemu-img from kvm-88, which is the most recent
> >>> release on kvm's home page.
> > 
> > Dated 12 July 2009.
> 
> Sorry, seems I missed the kvm-88 part. The date of the fix I mentioned
> is 19 July 2009, so I assume your problem is already fixed with current
> git master.

It looks like the right fix, and a simple mistake caused the bug.

Thanks,
-- Jamie

Re: [Qemu-devel] More qcow2 bugs? - qemu-img convert/commit

[Jamie Lokier]

Jamie Lokier wrote:
> Kevin Wolf wrote:
> > Am 10.09.2009 11:44, schrieb Jamie Lokier:
> > > Kevin Wolf wrote:
> > >> Actually, first of all, which qemu version are you using?
> > > 
> > >>> The tool I used is qemu-img from kvm-88, which is the most recent
> > >>> release on kvm's home page.
> > > 
> > > Dated 12 July 2009.
> > 
> > Sorry, seems I missed the kvm-88 part. The date of the fix I mentioned
> > is 19 July 2009, so I assume your problem is already fixed with current
> > git master.
> 
> It looks like the right fix, and a simple mistake caused the bug.

I've confirmed that it's fixed.  "kvm-img-88 commit", "qemu-kvm-git
commit" and "qemu-kvm-git convert" all produce the identical raw files
now.

-- Jamie