[Qemu-devel] [RFC] qcow3 format in libvirt

[Qemu-devel] [RFC] qcow3 format in libvirt

[Ján Tomko]

Before posting another version of my patches [1], attempting to add
support for the new qcow format to libvirt, I would like to know if this
sounds reasonable:

A new format named 'qcow3' would be added, along with a <features>
sub-element for target.

<volume>
  <name>qcow3test</name>
  <source>
  </source>
  <capacity unit='GiB'>8</capacity>
  <target>
    <path>/var/lib/libvirt/images/qcow3test</path>
    <format type='qcow3'/>
    <features>
      <lazy_refcounts/>
    </features>
  </target>
</volume>

I think that libvirt shouldn't care if the features are compatible or
incompatible, as we don't know what features are supported by the
hypervisor. Would the features be any good as tri-state (on, off, default?).

While the qcow3 format is handled by the qcow2 driver in QEMU,
<driver name='qemu' type='qcow2'/> should be enough for domains,
but in snapshot XML we treat the driver type as the format:

<disk name='/path/to/old'>
  <driver type='qcow3'/>
  <source file='/path/to/new'/>
  <features>
    <lazy_refcounts/>
  </features>
</disk>

So I think we should allow the qcow3 driver type as well and translate
it to qcow2 for QEMU.

Jan

[1] v2 here:
https://www.redhat.com/archives/libvir-list/2013-February/msg00212.html

Re: [Qemu-devel] [RFC] qcow3 format in libvirt

[Daniel P. Berrange]

On Mon, Mar 04, 2013 at 01:58:12PM +0100, Ján Tomko wrote:
> Before posting another version of my patches [1], attempting to add
> support for the new qcow format to libvirt, I would like to know if this
> sounds reasonable:
> 
> A new format named 'qcow3' would be added, along with a <features>
> sub-element for target.
> 
> <volume>
>   <name>qcow3test</name>
>   <source>
>   </source>
>   <capacity unit='GiB'>8</capacity>
>   <target>
>     <path>/var/lib/libvirt/images/qcow3test</path>
>     <format type='qcow3'/>
>     <features>
>       <lazy_refcounts/>
>     </features>
>   </target>
> </volume>
> 
> I think that libvirt shouldn't care if the features are compatible or
> incompatible, as we don't know what features are supported by the
> hypervisor. Would the features be any good as tri-state (on, off, default?).
> 
> While the qcow3 format is handled by the qcow2 driver in QEMU,
> <driver name='qemu' type='qcow2'/> should be enough for domains,

We should use qcow3 everywhere IMHO, regardless of whether qcow2
technically works in this context.

> but in snapshot XML we treat the driver type as the format:
> 
> <disk name='/path/to/old'>
>   <driver type='qcow3'/>
>   <source file='/path/to/new'/>
>   <features>
>     <lazy_refcounts/>
>   </features>
> </disk>
> 
> So I think we should allow the qcow3 driver type as well and translate
> it to qcow2 for QEMU.
> 
> Jan
> 
> [1] v2 here:
> https://www.redhat.com/archives/libvir-list/2013-February/msg00212.html
> 

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [RFC] qcow3 format in libvirt

[Kevin Wolf]

Am 04.03.2013 um 14:09 hat Daniel P. Berrange geschrieben:
> On Mon, Mar 04, 2013 at 01:58:12PM +0100, Ján Tomko wrote:
> > Before posting another version of my patches [1], attempting to add
> > support for the new qcow format to libvirt, I would like to know if this
> > sounds reasonable:
> > 
> > A new format named 'qcow3' would be added, along with a <features>
> > sub-element for target.
> > 
> > <volume>
> >   <name>qcow3test</name>
> >   <source>
> >   </source>
> >   <capacity unit='GiB'>8</capacity>
> >   <target>
> >     <path>/var/lib/libvirt/images/qcow3test</path>
> >     <format type='qcow3'/>
> >     <features>
> >       <lazy_refcounts/>
> >     </features>
> >   </target>
> > </volume>
> > 
> > I think that libvirt shouldn't care if the features are compatible or
> > incompatible, as we don't know what features are supported by the
> > hypervisor. Would the features be any good as tri-state (on, off, default?).
> > 
> > While the qcow3 format is handled by the qcow2 driver in QEMU,
> > <driver name='qemu' type='qcow2'/> should be enough for domains,
> 
> We should use qcow3 everywhere IMHO, regardless of whether qcow2
> technically works in this context.

I think it makes much more sense to deal with it the way qemu does
instead of inventing new names. This has much more of an (incompatible)
feature flag than of a different image format. So to fit it in your
proposed syntax:

<target>
  <path>/var/lib/libvirt/images/qcow3test</path>
  <format type='qcow2'/>
  <features>
    <compat version="1.1" />
    <lazy_refcounts/>
  </features>
</target>

Or if you really think that you should refer to the inner workings of
qcow2, you can make it <version>3</version>.

But I guess you call all VMDKs just "vmdk", despite the fact that they
are really just a collection of different subformats. Right?

Kevin

Re: [Qemu-devel] [RFC] qcow3 format in libvirt

[Daniel P. Berrange]

On Mon, Mar 04, 2013 at 03:04:53PM +0100, Kevin Wolf wrote:
> Am 04.03.2013 um 14:09 hat Daniel P. Berrange geschrieben:
> > On Mon, Mar 04, 2013 at 01:58:12PM +0100, Ján Tomko wrote:
> > > Before posting another version of my patches [1], attempting to add
> > > support for the new qcow format to libvirt, I would like to know if this
> > > sounds reasonable:
> > > 
> > > A new format named 'qcow3' would be added, along with a <features>
> > > sub-element for target.
> > > 
> > > <volume>
> > >   <name>qcow3test</name>
> > >   <source>
> > >   </source>
> > >   <capacity unit='GiB'>8</capacity>
> > >   <target>
> > >     <path>/var/lib/libvirt/images/qcow3test</path>
> > >     <format type='qcow3'/>
> > >     <features>
> > >       <lazy_refcounts/>
> > >     </features>
> > >   </target>
> > > </volume>
> > > 
> > > I think that libvirt shouldn't care if the features are compatible or
> > > incompatible, as we don't know what features are supported by the
> > > hypervisor. Would the features be any good as tri-state (on, off, default?).
> > > 
> > > While the qcow3 format is handled by the qcow2 driver in QEMU,
> > > <driver name='qemu' type='qcow2'/> should be enough for domains,
> > 
> > We should use qcow3 everywhere IMHO, regardless of whether qcow2
> > technically works in this context.
> 
> I think it makes much more sense to deal with it the way qemu does
> instead of inventing new names. This has much more of an (incompatible)
> feature flag than of a different image format. So to fit it in your
> proposed syntax:

The issue is that QEMU is not the only thing that implements the qcow
format. There are a number of other impls out there, and we can't just
assume that they will all be providing a qcow2 driver that automagically
opens a qcow3 image format. Just in the same way we didn't assume that
a 'qcow' (version 1) driver would open a version 2 image.

It so happens that with QEMU if you specify format=qcow2 and give it
a qcow3 image, QEMU will open it, but libvirt can't assume that, since
this is a mere implementation detail. Hence libvirt must explicitly
refer to 'qcow3' in the XML and map it to qcow2 if applicable when
talking to QEMU.

> But I guess you call all VMDKs just "vmdk", despite the fact that they
> are really just a collection of different subformats. Right?

Yes, but that is really a bug in our representation of vmdk.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [RFC] qcow3 format in libvirt

[Kevin Wolf]

Am 04.03.2013 um 15:27 hat Daniel P. Berrange geschrieben:
> On Mon, Mar 04, 2013 at 03:04:53PM +0100, Kevin Wolf wrote:
> > Am 04.03.2013 um 14:09 hat Daniel P. Berrange geschrieben:
> > > On Mon, Mar 04, 2013 at 01:58:12PM +0100, Ján Tomko wrote:
> > > > Before posting another version of my patches [1], attempting to add
> > > > support for the new qcow format to libvirt, I would like to know if this
> > > > sounds reasonable:
> > > > 
> > > > A new format named 'qcow3' would be added, along with a <features>
> > > > sub-element for target.
> > > > 
> > > > <volume>
> > > >   <name>qcow3test</name>
> > > >   <source>
> > > >   </source>
> > > >   <capacity unit='GiB'>8</capacity>
> > > >   <target>
> > > >     <path>/var/lib/libvirt/images/qcow3test</path>
> > > >     <format type='qcow3'/>
> > > >     <features>
> > > >       <lazy_refcounts/>
> > > >     </features>
> > > >   </target>
> > > > </volume>
> > > > 
> > > > I think that libvirt shouldn't care if the features are compatible or
> > > > incompatible, as we don't know what features are supported by the
> > > > hypervisor. Would the features be any good as tri-state (on, off, default?).
> > > > 
> > > > While the qcow3 format is handled by the qcow2 driver in QEMU,
> > > > <driver name='qemu' type='qcow2'/> should be enough for domains,
> > > 
> > > We should use qcow3 everywhere IMHO, regardless of whether qcow2
> > > technically works in this context.
> > 
> > I think it makes much more sense to deal with it the way qemu does
> > instead of inventing new names. This has much more of an (incompatible)
> > feature flag than of a different image format. So to fit it in your
> > proposed syntax:
> 
> The issue is that QEMU is not the only thing that implements the qcow
> format. There are a number of other impls out there, and we can't just
> assume that they will all be providing a qcow2 driver that automagically
> opens a qcow3 image format. Just in the same way we didn't assume that
> a 'qcow' (version 1) driver would open a version 2 image.

That's true. Other implementation actually tend to have a 'qcow' driver
that deals with both qcow1 and qcow2. But these two are actually
different enough that calling them two different formats might be
acceptable.

In contrast, version 3 images share _exactly_ the same structure with
version 2 images, the just have additional header fields and support
some new flags in some structures (that were previously reserved).

If you call this a different image format, then scratch that whole
<feature> idea, because then each newly added feature is a new image
format by your standards.

> It so happens that with QEMU if you specify format=qcow2 and give it
> a qcow3 image, QEMU will open it, but libvirt can't assume that, since
> this is a mere implementation detail. Hence libvirt must explicitly
> refer to 'qcow3' in the XML and map it to qcow2 if applicable when
> talking to QEMU.

If you need this information, sure, save it. I'm just requesting that
you don't abuse the format name for it.

> > But I guess you call all VMDKs just "vmdk", despite the fact that they
> > are really just a collection of different subformats. Right?
> 
> Yes, but that is really a bug in our representation of vmdk.

How are you going to fix it? Do you think having ten different format
names all starting with "vmdk" will make tools user friendly?

Kevin

Re: [Qemu-devel] [RFC] qcow3 format in libvirt

[Daniel P. Berrange]

On Mon, Mar 04, 2013 at 03:38:54PM +0100, Kevin Wolf wrote:
> Am 04.03.2013 um 15:27 hat Daniel P. Berrange geschrieben:
> > On Mon, Mar 04, 2013 at 03:04:53PM +0100, Kevin Wolf wrote:
> > > Am 04.03.2013 um 14:09 hat Daniel P. Berrange geschrieben:
> > > I think it makes much more sense to deal with it the way qemu does
> > > instead of inventing new names. This has much more of an (incompatible)
> > > feature flag than of a different image format. So to fit it in your
> > > proposed syntax:
> > 
> > The issue is that QEMU is not the only thing that implements the qcow
> > format. There are a number of other impls out there, and we can't just
> > assume that they will all be providing a qcow2 driver that automagically
> > opens a qcow3 image format. Just in the same way we didn't assume that
> > a 'qcow' (version 1) driver would open a version 2 image.
> 
> That's true. Other implementation actually tend to have a 'qcow' driver
> that deals with both qcow1 and qcow2. But these two are actually
> different enough that calling them two different formats might be
> acceptable.
> 
> In contrast, version 3 images share _exactly_ the same structure with
> version 2 images, the just have additional header fields and support
> some new flags in some structures (that were previously reserved).
> 
> If you call this a different image format, then scratch that whole
> <feature> idea, because then each newly added feature is a new image
> format by your standards.

No, that's not what I'm saying. The version 3 image format introduces
the ability to set a variety of features in an extensible way. Adding
new features to that list doesn't mean the version has changed.

> 
> > It so happens that with QEMU if you specify format=qcow2 and give it
> > a qcow3 image, QEMU will open it, but libvirt can't assume that, since
> > this is a mere implementation detail. Hence libvirt must explicitly
> > refer to 'qcow3' in the XML and map it to qcow2 if applicable when
> > talking to QEMU.
> 
> If you need this information, sure, save it. I'm just requesting that
> you don't abuse the format name for it.

The key distinction is that libvirt XML is recording an generic image
format, while the QEMU cli args are referring to a specific driver
implementation, which are support multiple formats. Typically these
map 1-to-1, but there's no such requirement in general. Hence will
refer to 'qcow3' in all its XML descriptions, and map to 'qcow2' when
talking to QEMU, or even just to 'qcow' if talking to a different impl
which supports all 3 versions in one driver.

> > > But I guess you call all VMDKs just "vmdk", despite the fact that they
> > > are really just a collection of different subformats. Right?
> > 
> > Yes, but that is really a bug in our representation of vmdk.
> 
> How are you going to fix it? Do you think having ten different format
> names all starting with "vmdk" will make tools user friendly?

Well we can't really fix it now, given we've got tools relying on
this naming, but use we ought to have numbered the vmdk formats in
retrospect.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [RFC] qcow3 format in libvirt

[Kevin Wolf]

Am 04.03.2013 um 15:46 hat Daniel P. Berrange geschrieben:
> On Mon, Mar 04, 2013 at 03:38:54PM +0100, Kevin Wolf wrote:
> > Am 04.03.2013 um 15:27 hat Daniel P. Berrange geschrieben:
> > > On Mon, Mar 04, 2013 at 03:04:53PM +0100, Kevin Wolf wrote:
> > > > Am 04.03.2013 um 14:09 hat Daniel P. Berrange geschrieben:
> > > > I think it makes much more sense to deal with it the way qemu does
> > > > instead of inventing new names. This has much more of an (incompatible)
> > > > feature flag than of a different image format. So to fit it in your
> > > > proposed syntax:
> > > 
> > > The issue is that QEMU is not the only thing that implements the qcow
> > > format. There are a number of other impls out there, and we can't just
> > > assume that they will all be providing a qcow2 driver that automagically
> > > opens a qcow3 image format. Just in the same way we didn't assume that
> > > a 'qcow' (version 1) driver would open a version 2 image.
> > 
> > That's true. Other implementation actually tend to have a 'qcow' driver
> > that deals with both qcow1 and qcow2. But these two are actually
> > different enough that calling them two different formats might be
> > acceptable.
> > 
> > In contrast, version 3 images share _exactly_ the same structure with
> > version 2 images, the just have additional header fields and support
> > some new flags in some structures (that were previously reserved).
> > 
> > If you call this a different image format, then scratch that whole
> > <feature> idea, because then each newly added feature is a new image
> > format by your standards.
> 
> No, that's not what I'm saying. The version 3 image format introduces
> the ability to set a variety of features in an extensible way. Adding
> new features to that list doesn't mean the version has changed.

Why does libvirt care whether a new feature is indicated by incrementing
one header field or by setting a bit in a different header field? These
are image format internals, not external interfaces.

> > > It so happens that with QEMU if you specify format=qcow2 and give it
> > > a qcow3 image, QEMU will open it, but libvirt can't assume that, since
> > > this is a mere implementation detail. Hence libvirt must explicitly
> > > refer to 'qcow3' in the XML and map it to qcow2 if applicable when
> > > talking to QEMU.
> > 
> > If you need this information, sure, save it. I'm just requesting that
> > you don't abuse the format name for it.
> 
> The key distinction is that libvirt XML is recording an generic image
> format, while the QEMU cli args are referring to a specific driver
> implementation, which are support multiple formats. Typically these
> map 1-to-1, but there's no such requirement in general. Hence will
> refer to 'qcow3' in all its XML descriptions, and map to 'qcow2' when
> talking to QEMU, or even just to 'qcow' if talking to a different impl
> which supports all 3 versions in one driver.

I'm not talking about the QEMU cli, but about qcow2 as the format as
defined in the spec (which just happens to sit in qemu.git, but isn't
qemu specific at all)

> > > > But I guess you call all VMDKs just "vmdk", despite the fact that they
> > > > are really just a collection of different subformats. Right?
> > > 
> > > Yes, but that is really a bug in our representation of vmdk.
> > 
> > How are you going to fix it? Do you think having ten different format
> > names all starting with "vmdk" will make tools user friendly?
> 
> Well we can't really fix it now, given we've got tools relying on
> this naming, but use we ought to have numbered the vmdk formats in
> retrospect.

Do you actually support more than one subformat today that they could
rely on? It's probably only important for creating images anyway as all
backends implement all VMDK subformats using the same driver.

Anyway, just imagine that these tools didn't exist yet. I still think
that giving the user ten different "vmdk*" formats to choose from would
be the wrong way - and isn't this how it would end up if libvirt treated
it as different formats? The user should select VMDK and then get
additional options that apply to this specific format. The same way the
user should select qcow2 (or even just qcow) and then all options
that are available for it, including version 3 images.

Kevin

Re: [Qemu-devel] [RFC] qcow3 format in libvirt

[Daniel P. Berrange]

On Mon, Mar 04, 2013 at 04:05:50PM +0100, Kevin Wolf wrote:
> Am 04.03.2013 um 15:46 hat Daniel P. Berrange geschrieben:
> > On Mon, Mar 04, 2013 at 03:38:54PM +0100, Kevin Wolf wrote:
> > > Am 04.03.2013 um 15:27 hat Daniel P. Berrange geschrieben:
> > > > On Mon, Mar 04, 2013 at 03:04:53PM +0100, Kevin Wolf wrote:
> > > > > Am 04.03.2013 um 14:09 hat Daniel P. Berrange geschrieben:
> > > > > I think it makes much more sense to deal with it the way qemu does
> > > > > instead of inventing new names. This has much more of an (incompatible)
> > > > > feature flag than of a different image format. So to fit it in your
> > > > > proposed syntax:
> > > > 
> > > > The issue is that QEMU is not the only thing that implements the qcow
> > > > format. There are a number of other impls out there, and we can't just
> > > > assume that they will all be providing a qcow2 driver that automagically
> > > > opens a qcow3 image format. Just in the same way we didn't assume that
> > > > a 'qcow' (version 1) driver would open a version 2 image.
> > > 
> > > That's true. Other implementation actually tend to have a 'qcow' driver
> > > that deals with both qcow1 and qcow2. But these two are actually
> > > different enough that calling them two different formats might be
> > > acceptable.
> > > 
> > > In contrast, version 3 images share _exactly_ the same structure with
> > > version 2 images, the just have additional header fields and support
> > > some new flags in some structures (that were previously reserved).
> > > 
> > > If you call this a different image format, then scratch that whole
> > > <feature> idea, because then each newly added feature is a new image
> > > format by your standards.
> > 
> > No, that's not what I'm saying. The version 3 image format introduces
> > the ability to set a variety of features in an extensible way. Adding
> > new features to that list doesn't mean the version has changed.
> 
> Why does libvirt care whether a new feature is indicated by incrementing
> one header field or by setting a bit in a different header field? These
> are image format internals, not external interfaces.
> 
> > > > It so happens that with QEMU if you specify format=qcow2 and give it
> > > > a qcow3 image, QEMU will open it, but libvirt can't assume that, since
> > > > this is a mere implementation detail. Hence libvirt must explicitly
> > > > refer to 'qcow3' in the XML and map it to qcow2 if applicable when
> > > > talking to QEMU.
> > > 
> > > If you need this information, sure, save it. I'm just requesting that
> > > you don't abuse the format name for it.
> > 
> > The key distinction is that libvirt XML is recording an generic image
> > format, while the QEMU cli args are referring to a specific driver
> > implementation, which are support multiple formats. Typically these
> > map 1-to-1, but there's no such requirement in general. Hence will
> > refer to 'qcow3' in all its XML descriptions, and map to 'qcow2' when
> > talking to QEMU, or even just to 'qcow' if talking to a different impl
> > which supports all 3 versions in one driver.
> 
> I'm not talking about the QEMU cli, but about qcow2 as the format as
> defined in the spec (which just happens to sit in qemu.git, but isn't
> qemu specific at all)

So you're saying that you consider the format name to be "qcow2" regardless
of whether the version numer field is specified as 2 or 3.

So in other words, if an application came along and required libvirt to
set  format=qcow3 on its CLI, we could justifiably refuse to do that in
libvirt claiming this is not in compliance with the spec ?  This is my
big concern.  If we go with 'format=qcow2' in the XML and we did ever
hit a case where we needed to distinguish versions, we'd not have enough
info in the XML todo that. If you are willing so that that such a scenario
is not spec compliant, then I'll be ok using just qcow2 in the libvirt XML
for this.

It would be nice if the spec explicitly stated that the format should be
referred to by any implementation as 'qcow2' regardles of version number
being 2 or 3.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [RFC] qcow3 format in libvirt

[Kevin Wolf]

Am 04.03.2013 um 16:19 hat Daniel P. Berrange geschrieben:
> On Mon, Mar 04, 2013 at 04:05:50PM +0100, Kevin Wolf wrote:
> > Am 04.03.2013 um 15:46 hat Daniel P. Berrange geschrieben:
> > > On Mon, Mar 04, 2013 at 03:38:54PM +0100, Kevin Wolf wrote:
> > > > Am 04.03.2013 um 15:27 hat Daniel P. Berrange geschrieben:
> > > > > It so happens that with QEMU if you specify format=qcow2 and give it
> > > > > a qcow3 image, QEMU will open it, but libvirt can't assume that, since
> > > > > this is a mere implementation detail. Hence libvirt must explicitly
> > > > > refer to 'qcow3' in the XML and map it to qcow2 if applicable when
> > > > > talking to QEMU.
> > > > 
> > > > If you need this information, sure, save it. I'm just requesting that
> > > > you don't abuse the format name for it.
> > > 
> > > The key distinction is that libvirt XML is recording an generic image
> > > format, while the QEMU cli args are referring to a specific driver
> > > implementation, which are support multiple formats. Typically these
> > > map 1-to-1, but there's no such requirement in general. Hence will
> > > refer to 'qcow3' in all its XML descriptions, and map to 'qcow2' when
> > > talking to QEMU, or even just to 'qcow' if talking to a different impl
> > > which supports all 3 versions in one driver.
> > 
> > I'm not talking about the QEMU cli, but about qcow2 as the format as
> > defined in the spec (which just happens to sit in qemu.git, but isn't
> > qemu specific at all)
> 
> So you're saying that you consider the format name to be "qcow2" regardless
> of whether the version numer field is specified as 2 or 3.

Yes.

> So in other words, if an application came along and required libvirt to
> set  format=qcow3 on its CLI, we could justifiably refuse to do that in
> libvirt claiming this is not in compliance with the spec ?

No, you would just check which features this image uses (which, if I
understood correctly, you need to save anyway), and if a version 3
feature is among it (the basic version 3 could be represented by either
a "feature flags" or "zero clusters" feature, which are what version 3
really means), pass it the 'qcow3' command line option it wants.

Of course, I would be disappointed that the tool thought it had to
invent format names, but it's not really blocking any functionality.

Just the same way it could happen that a tool uses different drivers for
other features that we introduce. For example, imagine that we introduce
a flag that modifies the L2 table structure to allow subclusters - a
change that we've been discussing before and that would have a massive
impact on the implementation, even though it's only a feature flag that
has changed, and not the version number. Using a different driver for
this looks much more likely than a different driver for version 2 and 3,
which was really a quite small step.

So the main problem that I see is the assumption "different version =>
big change, new feature flag => small change" and as a conclusion from
that "different version => possibly new driver, new feature flag =>
definitely only old driver". This isn't true at all.

Kevin

Re: [Qemu-devel] [RFC] qcow3 format in libvirt

[Ján Tomko]

On 03/04/2013 04:40 PM, Kevin Wolf wrote:
> Am 04.03.2013 um 16:19 hat Daniel P. Berrange geschrieben:
>> On Mon, Mar 04, 2013 at 04:05:50PM +0100, Kevin Wolf wrote:
>>>
>>> I'm not talking about the QEMU cli, but about qcow2 as the format as
>>> defined in the spec (which just happens to sit in qemu.git, but isn't
>>> qemu specific at all)
>>
>> So you're saying that you consider the format name to be "qcow2" regardless
>> of whether the version numer field is specified as 2 or 3.
> 
> Yes.
>
>> So in other words, if an application came along and required libvirt to
>> set  format=qcow3 on its CLI, we could justifiably refuse to do that in
>> libvirt claiming this is not in compliance with the spec ?
> 
> No, you would just check which features this image uses (which, if I
> understood correctly, you need to save anyway), and if a version 3
> feature is among it (the basic version 3 could be represented by either
> a "feature flags" or "zero clusters" feature, which are what version 3
> really means), pass it the 'qcow3' command line option it wants.
> 

Since libvirt needs to know the feature names, I doesn't seem to be a
problem to know what compat options they need. And the empty (but
present) <features> element could just mean compat=1.1. Would we still
need to support creating compat=0.10 images with older qemu-img not
understanding this option after the default gets changed in the current
version?

> Of course, I would be disappointed that the tool thought it had to
> invent format names, but it's not really blocking any functionality.
> 
> Just the same way it could happen that a tool uses different drivers for
> other features that we introduce. For example, imagine that we introduce
> a flag that modifies the L2 table structure to allow subclusters - a
> change that we've been discussing before and that would have a massive
> impact on the implementation, even though it's only a feature flag that
> has changed, and not the version number. Using a different driver for
> this looks much more likely than a different driver for version 2 and 3,
> which was really a quite small step.

So, the format and the driver is still 'qcow2' now, there's no need to
translate anything at the moment (apart from features->options when
creating the image).

But if the same 'qcow2' format would need different drivers based on the
features, we need to use different set of values for driver types and
image formats, if the features would not be in the domain XML.

> 
> So the main problem that I see is the assumption "different version =>
> big change, new feature flag => small change" and as a conclusion from
> that "different version => possibly new driver, new feature flag =>
> definitely only old driver". This isn't true at all.
> 
> Kevin
> 

Re: [Qemu-devel] [RFC] qcow3 format in libvirt

[Kevin Wolf]

Am 11.03.2013 um 19:03 hat Ján Tomko geschrieben:
> On 03/04/2013 04:40 PM, Kevin Wolf wrote:
> > Am 04.03.2013 um 16:19 hat Daniel P. Berrange geschrieben:
> >> On Mon, Mar 04, 2013 at 04:05:50PM +0100, Kevin Wolf wrote:
> >>>
> >>> I'm not talking about the QEMU cli, but about qcow2 as the format as
> >>> defined in the spec (which just happens to sit in qemu.git, but isn't
> >>> qemu specific at all)
> >>
> >> So you're saying that you consider the format name to be "qcow2" regardless
> >> of whether the version numer field is specified as 2 or 3.
> > 
> > Yes.
> >
> >> So in other words, if an application came along and required libvirt to
> >> set  format=qcow3 on its CLI, we could justifiably refuse to do that in
> >> libvirt claiming this is not in compliance with the spec ?
> > 
> > No, you would just check which features this image uses (which, if I
> > understood correctly, you need to save anyway), and if a version 3
> > feature is among it (the basic version 3 could be represented by either
> > a "feature flags" or "zero clusters" feature, which are what version 3
> > really means), pass it the 'qcow3' command line option it wants.
> > 
> 
> Since libvirt needs to know the feature names, I doesn't seem to be a
> problem to know what compat options they need. And the empty (but
> present) <features> element could just mean compat=1.1.

Sounds like an option if this isn't too much magic for you. If we ever
introduced a compat=2.0, would this be represented by a specific child
element of <features>?

> Would we still
> need to support creating compat=0.10 images with older qemu-img not
> understanding this option after the default gets changed in the current
> version?

If you want to create images that can be read by versions < 1.1, you
need it. I think it's a reasonable expectation that libvirt allows this.

That said, I'm not sure when we should switch the default in qemu. It's
probably too early in 1.5, but we might consider it for 1.6.

> > Of course, I would be disappointed that the tool thought it had to
> > invent format names, but it's not really blocking any functionality.
> > 
> > Just the same way it could happen that a tool uses different drivers for
> > other features that we introduce. For example, imagine that we introduce
> > a flag that modifies the L2 table structure to allow subclusters - a
> > change that we've been discussing before and that would have a massive
> > impact on the implementation, even though it's only a feature flag that
> > has changed, and not the version number. Using a different driver for
> > this looks much more likely than a different driver for version 2 and 3,
> > which was really a quite small step.
> 
> So, the format and the driver is still 'qcow2' now, there's no need to
> translate anything at the moment (apart from features->options when
> creating the image).
> 
> But if the same 'qcow2' format would need different drivers based on the
> features, we need to use different set of values for driver types and
> image formats, if the features would not be in the domain XML.

What driver is used to deal with the format is an implementation detail,
it's not something mandated by the format. This is why some tools have
one driver for both qcow1 and qcow2, and others like qemu have separate
ones.

Kevin