Re: [Qemu-devel] committing qcow2 to raw backing file

Re: [Qemu-devel] committing qcow2 to raw backing file

[Murali Balcha]

Thanks for the quick reply. The scenario I have is little different. The
backing file is only 40 MB where as the virtual size of qcow2 is 1GB.

project1@cloudvault3:/tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3$ qemu-img
info 2e94bd45-7e4e-4e04-a8b4-74d579cad892_vda.qcow2
image: 2e94bd45-7e4e-4e04-a8b4-74d579cad892_vda.qcow2
file format: raw
virtual size: 39M (41126400 bytes)
disk size: 39M
project1@cloudvault3:/tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3$ qemu-img
info 2bc87796-60e8-4024-8a40-fae817e094da_vda.qcow2
image: 2bc87796-60e8-4024-8a40-fae817e094da_vda.qcow2
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: 3.3M
cluster_size: 65536
backing file: 
/tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3/2e94bd45-7e4e-4e04-a8b4-74d579cad
892_vda.qcow2
Format specific information:
    compat: 0.10
project1@cloudvault3:/tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3$ qemu-img
commit 2bc87796-60e8-4024-8a40-fae817e094da_vda.qcow2
qemu-img: Error while committing image

The qemu-img is bailing out when it detected that commit is happening
beyond raw backing image.


Is this a valid commit operation. This is how openstack sets up vms.

Regards,
Murali Balcah

On 1/29/14, 4:24 AM, "Kevin Wolf" <kwolf@redhat.com> wrote:

>Am 29.01.2014 um 03:09 hat Murali Balcha geschrieben:
>> Hi,
>> My apologies for reaching out directly on your email. However I am new
>>qemu and
>> don¹t know the active mailing list for qemu related issues.
>
>The qemu mailing list is qemu-devel@nongnu.org. You should generally
>direct your questions there, though keeping individual persons in the CC
>list is fine.
>
>> I have seen your
>> name in one of the patches related to this
>>issue.https://github.com/gooselinux/
>> qemu-kvm/blob/master/kvm-block-Fix-image-re-open-in-bdrv_commit.patch
>> 
>>  We are using 1.6.90 and 1.7 of qemu. the qemu-img errors out with
>>exact same
>> message when we try to commit qcow2 delta to raw image. Looking at the
>>code, we
>> realized that that patch you discussed above did not make it 1.6.90
>>branch we
>> are using. Am I missing something.
>
>I am not sure which error message you mean, but the patch that you
>mentioned has been in qemu since version 0.14.0. You won't see it in the
>current code any more because of later changes that switched to using
>bdrv_reopen() instead.
>
>For me, committing to a raw image works fine with current git master:
>
>$ qemu-img create -f raw /tmp/backing.raw 4G
>Formatting '/tmp/backing.raw', fmt=raw size=4294967296
>$ qemu-img create -f qcow2 -b /tmp/backing.raw /tmp/overlay.qcow2
>Formatting '/tmp/overlay.qcow2', fmt=qcow2 size=4294967296
>backing_file='/tmp/backing.raw' encryption=off cluster_size=65536
>lazy_refcounts=off
>$ qemu-io -c 'write 0 4k' /tmp/overlay.qcow2
>wrote 4096/4096 bytes at offset 0
>4 KiB, 1 ops; 0.0009 sec (4.090 MiB/sec and 1047.1204 ops/sec)
>$ qemu-img commit /tmp/overlay.qcow2
>Image committed.
>
>Kevin

Re: [Qemu-devel] committing qcow2 to raw backing file

[Kevin Wolf]

Am 29.01.2014 um 14:50 hat Murali Balcha geschrieben:
> Thanks for the quick reply. The scenario I have is little different. The
> backing file is only 40 MB where as the virtual size of qcow2 is 1GB.
> 
> project1@cloudvault3:/tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3$ qemu-img
> info 2e94bd45-7e4e-4e04-a8b4-74d579cad892_vda.qcow2
> image: 2e94bd45-7e4e-4e04-a8b4-74d579cad892_vda.qcow2
> file format: raw
> virtual size: 39M (41126400 bytes)
> disk size: 39M
> project1@cloudvault3:/tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3$ qemu-img
> info 2bc87796-60e8-4024-8a40-fae817e094da_vda.qcow2
> image: 2bc87796-60e8-4024-8a40-fae817e094da_vda.qcow2
> file format: qcow2
> virtual size: 1.0G (1073741824 bytes)
> disk size: 3.3M
> cluster_size: 65536
> backing file: 
> /tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3/2e94bd45-7e4e-4e04-a8b4-74d579cad
> 892_vda.qcow2
> Format specific information:
>     compat: 0.10
> project1@cloudvault3:/tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3$ qemu-img
> commit 2bc87796-60e8-4024-8a40-fae817e094da_vda.qcow2
> qemu-img: Error while committing image
> 
> The qemu-img is bailing out when it detected that commit is happening
> beyond raw backing image.
> 
> 
> Is this a valid commit operation. This is how openstack sets up vms.

This is completely unrelated to the patch that you mentioned then.

The problem is that qemu-img would have to increase the image size of
the backing file first before it can commit all the data to it. I
believe there was a patch by Jeff (CC'ed) that would do exactly this.

Kevin

> On 1/29/14, 4:24 AM, "Kevin Wolf" <kwolf@redhat.com> wrote:
> 
> >Am 29.01.2014 um 03:09 hat Murali Balcha geschrieben:
> >> Hi,
> >> My apologies for reaching out directly on your email. However I am new
> >>qemu and
> >> don¹t know the active mailing list for qemu related issues.
> >
> >The qemu mailing list is qemu-devel@nongnu.org. You should generally
> >direct your questions there, though keeping individual persons in the CC
> >list is fine.
> >
> >> I have seen your
> >> name in one of the patches related to this
> >>issue.https://github.com/gooselinux/
> >> qemu-kvm/blob/master/kvm-block-Fix-image-re-open-in-bdrv_commit.patch
> >> 
> >>  We are using 1.6.90 and 1.7 of qemu. the qemu-img errors out with
> >>exact same
> >> message when we try to commit qcow2 delta to raw image. Looking at the
> >>code, we
> >> realized that that patch you discussed above did not make it 1.6.90
> >>branch we
> >> are using. Am I missing something.
> >
> >I am not sure which error message you mean, but the patch that you
> >mentioned has been in qemu since version 0.14.0. You won't see it in the
> >current code any more because of later changes that switched to using
> >bdrv_reopen() instead.
> >
> >For me, committing to a raw image works fine with current git master:
> >
> >$ qemu-img create -f raw /tmp/backing.raw 4G
> >Formatting '/tmp/backing.raw', fmt=raw size=4294967296
> >$ qemu-img create -f qcow2 -b /tmp/backing.raw /tmp/overlay.qcow2
> >Formatting '/tmp/overlay.qcow2', fmt=qcow2 size=4294967296
> >backing_file='/tmp/backing.raw' encryption=off cluster_size=65536
> >lazy_refcounts=off
> >$ qemu-io -c 'write 0 4k' /tmp/overlay.qcow2
> >wrote 4096/4096 bytes at offset 0
> >4 KiB, 1 ops; 0.0009 sec (4.090 MiB/sec and 1047.1204 ops/sec)
> >$ qemu-img commit /tmp/overlay.qcow2
> >Image committed.
> >
> >Kevin
> 

Re: [Qemu-devel] committing qcow2 to raw backing file

[Jeff Cody]

On Wed, Jan 29, 2014 at 03:35:16PM +0100, Kevin Wolf wrote:
> Am 29.01.2014 um 14:50 hat Murali Balcha geschrieben:
> > Thanks for the quick reply. The scenario I have is little different. The
> > backing file is only 40 MB where as the virtual size of qcow2 is 1GB.
> > 
> > project1@cloudvault3:/tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3$ qemu-img
> > info 2e94bd45-7e4e-4e04-a8b4-74d579cad892_vda.qcow2
> > image: 2e94bd45-7e4e-4e04-a8b4-74d579cad892_vda.qcow2
> > file format: raw
> > virtual size: 39M (41126400 bytes)
> > disk size: 39M
> > project1@cloudvault3:/tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3$ qemu-img
> > info 2bc87796-60e8-4024-8a40-fae817e094da_vda.qcow2
> > image: 2bc87796-60e8-4024-8a40-fae817e094da_vda.qcow2
> > file format: qcow2
> > virtual size: 1.0G (1073741824 bytes)
> > disk size: 3.3M
> > cluster_size: 65536
> > backing file: 
> > /tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3/2e94bd45-7e4e-4e04-a8b4-74d579cad
> > 892_vda.qcow2
> > Format specific information:
> >     compat: 0.10
> > project1@cloudvault3:/tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3$ qemu-img
> > commit 2bc87796-60e8-4024-8a40-fae817e094da_vda.qcow2
> > qemu-img: Error while committing image
> > 
> > The qemu-img is bailing out when it detected that commit is happening
> > beyond raw backing image.
> > 
> > 
> > Is this a valid commit operation. This is how openstack sets up vms.
> 
> This is completely unrelated to the patch that you mentioned then.
> 
> The problem is that qemu-img would have to increase the image size of
> the backing file first before it can commit all the data to it. I
> believe there was a patch by Jeff (CC'ed) that would do exactly this.
> 
> Kevin

Thanks Kevin,

Yes, the patch is in the qemu-kvm git master as of Jan 24 2014 (so it
is pretty new); here is the commit id info:

commit 72706ea4cd38bfcb151265df0178ba21863d7518
Author: Jeff Cody <jcody@redhat.com>
Date:   Fri Jan 24 09:02:35 2014 -0500

    block: resize backing file image during offline commit, if necessary




There is a corresponding commit for the same fix, for when you are
doing a live online block-commit to the active layer of a block image
chain:

commit 4da83585961631bfc10831dd26c4afda2a8b23e8
Author: Jeff Cody <jcody@redhat.com>
Date:   Fri Jan 24 09:02:36 2014 -0500

    block: resize backing image during active layer commit, if needed


Jeff


> 
> > On 1/29/14, 4:24 AM, "Kevin Wolf" <kwolf@redhat.com> wrote:
> > 
> > >Am 29.01.2014 um 03:09 hat Murali Balcha geschrieben:
> > >> Hi,
> > >> My apologies for reaching out directly on your email. However I am new
> > >>qemu and
> > >> don¹t know the active mailing list for qemu related issues.
> > >
> > >The qemu mailing list is qemu-devel@nongnu.org. You should generally
> > >direct your questions there, though keeping individual persons in the CC
> > >list is fine.
> > >
> > >> I have seen your
> > >> name in one of the patches related to this
> > >>issue.https://github.com/gooselinux/
> > >> qemu-kvm/blob/master/kvm-block-Fix-image-re-open-in-bdrv_commit.patch
> > >> 
> > >>  We are using 1.6.90 and 1.7 of qemu. the qemu-img errors out with
> > >>exact same
> > >> message when we try to commit qcow2 delta to raw image. Looking at the
> > >>code, we
> > >> realized that that patch you discussed above did not make it 1.6.90
> > >>branch we
> > >> are using. Am I missing something.
> > >
> > >I am not sure which error message you mean, but the patch that you
> > >mentioned has been in qemu since version 0.14.0. You won't see it in the
> > >current code any more because of later changes that switched to using
> > >bdrv_reopen() instead.
> > >
> > >For me, committing to a raw image works fine with current git master:
> > >
> > >$ qemu-img create -f raw /tmp/backing.raw 4G
> > >Formatting '/tmp/backing.raw', fmt=raw size=4294967296
> > >$ qemu-img create -f qcow2 -b /tmp/backing.raw /tmp/overlay.qcow2
> > >Formatting '/tmp/overlay.qcow2', fmt=qcow2 size=4294967296
> > >backing_file='/tmp/backing.raw' encryption=off cluster_size=65536
> > >lazy_refcounts=off
> > >$ qemu-io -c 'write 0 4k' /tmp/overlay.qcow2
> > >wrote 4096/4096 bytes at offset 0
> > >4 KiB, 1 ops; 0.0009 sec (4.090 MiB/sec and 1047.1204 ops/sec)
> > >$ qemu-img commit /tmp/overlay.qcow2
> > >Image committed.
> > >
> > >Kevin
> > 

Re: [Qemu-devel] committing qcow2 to raw backing file

[Murali Balcha]

Thanks a lot. Much appreciate your help.

Regards,
Murali Balcha


On Jan 29, 2014, at 9:52 AM, "Jeff Cody" <jcody@redhat.com> wrote:

> On Wed, Jan 29, 2014 at 03:35:16PM +0100, Kevin Wolf wrote:
>> Am 29.01.2014 um 14:50 hat Murali Balcha geschrieben:
>>> Thanks for the quick reply. The scenario I have is little different. The
>>> backing file is only 40 MB where as the virtual size of qcow2 is 1GB.
>>> 
>>> project1@cloudvault3:/tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3$ qemu-img
>>> info 2e94bd45-7e4e-4e04-a8b4-74d579cad892_vda.qcow2
>>> image: 2e94bd45-7e4e-4e04-a8b4-74d579cad892_vda.qcow2
>>> file format: raw
>>> virtual size: 39M (41126400 bytes)
>>> disk size: 39M
>>> project1@cloudvault3:/tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3$ qemu-img
>>> info 2bc87796-60e8-4024-8a40-fae817e094da_vda.qcow2
>>> image: 2bc87796-60e8-4024-8a40-fae817e094da_vda.qcow2
>>> file format: qcow2
>>> virtual size: 1.0G (1073741824 bytes)
>>> disk size: 3.3M
>>> cluster_size: 65536
>>> backing file: 
>>> /tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3/2e94bd45-7e4e-4e04-a8b4-74d579cad
>>> 892_vda.qcow2
>>> Format specific information:
>>>    compat: 0.10
>>> project1@cloudvault3:/tmp/20bcb425-d3c5-4c0f-819c-7c9b1a16fde3$ qemu-img
>>> commit 2bc87796-60e8-4024-8a40-fae817e094da_vda.qcow2
>>> qemu-img: Error while committing image
>>> 
>>> The qemu-img is bailing out when it detected that commit is happening
>>> beyond raw backing image.
>>> 
>>> 
>>> Is this a valid commit operation. This is how openstack sets up vms.
>> 
>> This is completely unrelated to the patch that you mentioned then.
>> 
>> The problem is that qemu-img would have to increase the image size of
>> the backing file first before it can commit all the data to it. I
>> believe there was a patch by Jeff (CC'ed) that would do exactly this.
>> 
>> Kevin
> 
> Thanks Kevin,
> 
> Yes, the patch is in the qemu-kvm git master as of Jan 24 2014 (so it
> is pretty new); here is the commit id info:
> 
> commit 72706ea4cd38bfcb151265df0178ba21863d7518
> Author: Jeff Cody <jcody@redhat.com>
> Date:   Fri Jan 24 09:02:35 2014 -0500
> 
>    block: resize backing file image during offline commit, if necessary
> 
> 
> 
> 
> There is a corresponding commit for the same fix, for when you are
> doing a live online block-commit to the active layer of a block image
> chain:
> 
> commit 4da83585961631bfc10831dd26c4afda2a8b23e8
> Author: Jeff Cody <jcody@redhat.com>
> Date:   Fri Jan 24 09:02:36 2014 -0500
> 
>    block: resize backing image during active layer commit, if needed
> 
> 
> Jeff
> 
> 
>> 
>>> On 1/29/14, 4:24 AM, "Kevin Wolf" <kwolf@redhat.com> wrote:
>>> 
>>>> Am 29.01.2014 um 03:09 hat Murali Balcha geschrieben:
>>>>> Hi,
>>>>> My apologies for reaching out directly on your email. However I am new
>>>>> qemu and
>>>>> don¹t know the active mailing list for qemu related issues.
>>>> 
>>>> The qemu mailing list is qemu-devel@nongnu.org. You should generally
>>>> direct your questions there, though keeping individual persons in the CC
>>>> list is fine.
>>>> 
>>>>> I have seen your
>>>>> name in one of the patches related to this
>>>>> issue.https://github.com/gooselinux/
>>>>> qemu-kvm/blob/master/kvm-block-Fix-image-re-open-in-bdrv_commit.patch
>>>>> 
>>>>> We are using 1.6.90 and 1.7 of qemu. the qemu-img errors out with
>>>>> exact same
>>>>> message when we try to commit qcow2 delta to raw image. Looking at the
>>>>> code, we
>>>>> realized that that patch you discussed above did not make it 1.6.90
>>>>> branch we
>>>>> are using. Am I missing something.
>>>> 
>>>> I am not sure which error message you mean, but the patch that you
>>>> mentioned has been in qemu since version 0.14.0. You won't see it in the
>>>> current code any more because of later changes that switched to using
>>>> bdrv_reopen() instead.
>>>> 
>>>> For me, committing to a raw image works fine with current git master:
>>>> 
>>>> $ qemu-img create -f raw /tmp/backing.raw 4G
>>>> Formatting '/tmp/backing.raw', fmt=raw size=4294967296
>>>> $ qemu-img create -f qcow2 -b /tmp/backing.raw /tmp/overlay.qcow2
>>>> Formatting '/tmp/overlay.qcow2', fmt=qcow2 size=4294967296
>>>> backing_file='/tmp/backing.raw' encryption=off cluster_size=65536
>>>> lazy_refcounts=off
>>>> $ qemu-io -c 'write 0 4k' /tmp/overlay.qcow2
>>>> wrote 4096/4096 bytes at offset 0
>>>> 4 KiB, 1 ops; 0.0009 sec (4.090 MiB/sec and 1047.1204 ops/sec)
>>>> $ qemu-img commit /tmp/overlay.qcow2
>>>> Image committed.
>>>> 
>>>> Kevin
>>> 

Re: [Qemu-devel] committing qcow2 to raw backing file

[Kevin Wolf]

Am 29.01.2014 um 15:52 hat Jeff Cody geschrieben:
> On Wed, Jan 29, 2014 at 03:35:16PM +0100, Kevin Wolf wrote:
> > Am 29.01.2014 um 14:50 hat Murali Balcha geschrieben:
> > The problem is that qemu-img would have to increase the image size of
> > the backing file first before it can commit all the data to it. I
> > believe there was a patch by Jeff (CC'ed) that would do exactly this.
> > 
> > Kevin
> 
> Thanks Kevin,
> 
> Yes, the patch is in the qemu-kvm git master as of Jan 24 2014 (so it
> is pretty new); here is the commit id info:

Heh, I actually looked at the function in master and didn't see it.
I guess it tells you how well I can concentrate with my cold. :-)

Thanks for giving the details, Jeff.

Kevin